<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brandon on Medium]]></title>
        <description><![CDATA[Stories by Brandon on Medium]]></description>
        <link>https://medium.com/@brandon.obrien?source=rss-eae943081b48------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1D8nXJmQNt2rmcTD4VZpbg.png</url>
            <title>Stories by Brandon on Medium</title>
            <link>https://medium.com/@brandon.obrien?source=rss-eae943081b48------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:59:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@brandon.obrien/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Notes from Strata Hadoop World 2017 San Jose]]></title>
            <link>https://medium.com/@brandon.obrien/notes-from-strata-hadoop-world-2017-san-jose-b87d0be2bccc?source=rss-eae943081b48------2</link>
            <guid isPermaLink="false">https://medium.com/p/b87d0be2bccc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[san-jose]]></category>
            <category><![CDATA[strata]]></category>
            <dc:creator><![CDATA[Brandon]]></dc:creator>
            <pubDate>Tue, 31 Oct 2017 22:39:31 GMT</pubDate>
            <atom:updated>2017-10-31T22:41:10.517Z</atom:updated>
            <content:encoded><![CDATA[<p>After attending Strata Hadoop World 2017, I wanted to share my notes on startups, industry trends.</p><p><a href="https://conferences.oreilly.com/strata/strata-ca">Strata + Hadoop World in San Jose 2017</a></p><p>Core themes: deep learning, streaming data, big-data-as-a-service platforms, companies from China starting to enter the US big data market.</p><p><strong>Startups</strong></p><p>There were several early stage startups presenting. Sentiment analysis was a recurring theme. Many business models could be summarized as “pay us to run your data through our predictive models”. I was surprised to see that many of these startups were not even attempting to touch data integration problems. In all cases, the startups’ products required to upload your data as a file — this could mean they’re being lean and focusing on the core value prop of their model, or it could mean they’re ignoring the logistical side of the business.</p><p><a href="https://www.crunchbase.com/organization/resonate-ai#/entity">Resonate.ai</a> — Product: a system to do emotional content checking for marketing text before publication. Business model: direct monetization of predictive model. The company created taxonomy of emotions and a model to predict how people will respond emotionally to your company’s marketing text.</p><p><a href="https://angel.co/repable">Repable</a> — niche streaming sentiment analysis for live gaming events</p><p><strong>Industry Talks</strong></p><p><strong>Uber. James Burkhart. Apollo Query &amp; Aggregation Engine. </strong>Great talk on Uber’s query and aggregation engine called Apollo. Very impressive work for achieving better performance through maximized cache reuse. Apollo achieves this through two main strategies. <strong>1. Query canonicalization.</strong> the same SQL query can be phrased differently, so while the query returns the same data, the literal SQL text is different. Mapping queries to a canonical form allows differently phrased SQL queries to be identified as the same query. so cached data can be reused across both queries. <strong>2. Quantization of cacheable data units. </strong>Create standard units for caching data across different dimensions. Example: only cache data in 15 minute intervals at 0, 15, 30 or 45 past the hour. This allows two queries that specify slightly different (but overlapping) time intervals to achieve partial reuse of the cached data. The system consumed data through streaming upserts and used a lambda architecture with batch refill/overwrites to avoid missing data in the streaming system. Poor performance on Spark Streaming was also noted — so Uber implemented their own lightweight stream consumer app.</p><p><strong>Databricks. Tathagata Das. Michael Armbrust. Spark Streaming: Structured Streaming.</strong> Overview of Structured Streaming in Spark Streaming. The promise here is that Spark Streaming can take your batch query and “streamify” it. Looks like Structured Streaming is converging with ideas from Azure Streaming Analytics (running SQL queries on a stream) and Apache Beam / Flink (the concept of a high “watermark” for data received). After the presentation, I’m still wondering why Spark Streaming requires a WAL (Write Ahead Log) for processing data from Kafka — Kafka already is a WAL. If you need a WAL to process data written to a WAL…where does the recursion end? I discuss this topic in my overview on operationalizing Spark Streaming with Kafka: <a href="https://techblog.expedia.com/2016/12/29/operationalizing-spark-streaming-part-1/">https://techblog.expedia.com/2016/12/29/operationalizing-spark-streaming-part-1/</a></p><p><strong>IBM. Michael Dusenberry. Frederick Reiss. Identifying Breast Cancer using Deep Learning. </strong>A walkthrough on how IBM used labeled image data to be able to predict whether tissue was cancerous or not, using <strong>Convolutional Neural Nets</strong>. They discussed problems on the amount of processing power required, problems in preparing the data including pre-filtering the images (adjusting for exposure &amp; color balance), needing to down sample images, and more. One of the useful approaches they discussed was <strong>Transfer Learning</strong>. They re-purposed existing Deep Learning models that could already do things like edge detection or shape recognition, and trained their model on the output of the existing model — peeling back layers in the original model sequentially. Very informative overview.</p><p><strong>LinkedIn. Kartik Paramasivam. Scaling Kafka economically.</strong> A talk on strategies for running Kafka in ways that are scalable, reliable, and economical ($-wise). One of the strategies was: given that a Kafka broker can’t handle a disk failure, it’s useful to run 1 Kafka broker per disk on a machine, and potentially multiple brokers per machine — then ensure two different replicas for the same topic-partition don’t end up on the same machine, and you achieve a great optimum between resiliency &amp; infrastructure cost.</p><p><strong>Training Sessions</strong></p><p>TensorFlow: I took the <a href="https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/56537">2 day straining on Google’s TensorFlow</a>, put on by Robert Schroll at <a href="https://www.thedataincubator.com/">Data Incubator</a>. It was a very intensive workshop on TensorFlow and intro to various kinds of Deep Learning net topologies, including Convolutional Neural Nets, Recurrent Neural Nets and LSTM, Adversarial Noise</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b87d0be2bccc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>