<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marios Assiotis on Medium]]></title>
        <description><![CDATA[Stories by Marios Assiotis on Medium]]></description>
        <link>https://medium.com/@marios_57233?source=rss-8b91e81aede0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_KQ-ItDHFHTqkppSmN2j_Q.jpeg</url>
            <title>Stories by Marios Assiotis on Medium</title>
            <link>https://medium.com/@marios_57233?source=rss-8b91e81aede0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:53:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marios_57233/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A fully reactive ad serving platform using Scala & akka-streams]]></title>
            <link>https://code.tubitv.com/a-fully-reactive-ad-serving-platform-using-scala-akka-streams-b2a3526be6a0?source=rss-8b91e81aede0------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2a3526be6a0</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[akka]]></category>
            <dc:creator><![CDATA[Marios Assiotis]]></dc:creator>
            <pubDate>Tue, 03 Nov 2020 18:23:51 GMT</pubDate>
            <atom:updated>2020-11-03T18:23:51.875Z</atom:updated>
            <content:encoded><![CDATA[<h3>Handling advertiser bids using Scala &amp; akka-streams</h3><p>In <a href="https://code.tubitv.com/a-fully-reactive-ad-serving-platform-using-scala-akka-streams-13299e7ea04e">a previous post</a>, we already looked at how ad requests are received, enriched and subsequently go through a series of filters to determine which creatives are eligible. In this post, we resume with steps 4–8, looking at how different bids are received and paced, as well as how the final determination is made on which bids may win.</p><h3>Step Four: Pacing</h3><p>Pacing is one of the most important functions of any ad server. Advertisers want their campaigns to be spread evenly throughout the day, hoping to reach as big and diverse an audience as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*sUPOi0yv6rSQH5dL" /></figure><p>We maintain campaign spending and budget information in Akka actors. Since spending is a monotonically increasing operation, and we are OK with eventual consistency, <a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type">CRDT operators</a> provided by Akka Distributed Data, make for an excellent choice of tracking spent, even across data centers. After we verify that a campaign has not exceeded its budget, we load forecasts, previously calculated by a Spark Job, along with budget and spend information. The pacing graph then utilizes simple techniques derived from control theory to decide if we should pace the campaign out or allow it to participate in the auction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*33OaTsAcm54KW3QA" /></figure><p>Forecasts are used to determine how traffic will be shaped throughout the day. They are calculated across all major targeting criteria that we know, based on domain knowledge, to affect traffic patterns and available inventory. For example, traffic on our apps running on gaming consoles follows a different pattern than our mobile apps.</p><h3>Step Five: Collecting Bids</h3><p>Tubi ad inventory is sold both directly by our very own sales team (they’re hiring!), as well as programmatically through direct integrations with top-tier DSPs. These integrations typically involve calling into a third party, with all the gotchas that come with making network calls to third parties and accepting arbitrary responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i5iJ-qqY1IbqHF0s" /></figure><p>Being able to capture each one of these into its own akka-streams flow allows us once again to write a common graph that takes care a lot of the boilerplate as well as the mechanics of async network calls, concurrency control, failure handling, throttling, circuit breaking, etc. We can then focus on the business logic and make sure the quirks for each third party integration are well coded and everyone’s individual interpretation of IAB specs is well tested. Similarly, since this is part of a per-request flow with strict upper bounds on concurrency, a misbehaving third party will not affect the rest of the normal operation of our ad serving platform.</p><h3>Step Six: Ad Video Creative Transcoding</h3><p>Once we have collected all eligible bids, we check if we have our own transcodes for each included video creative, and if not, take it through the transcoding process. We do this for a number of reasons:</p><ol><li>As a video streaming company, we have deep expertise in video and we want to make sure ad playback works well across all devices where Tubi runs.</li><li>Advertisers sometimes have inconsistent (read: loud) volume on their ad creatives, which we try to normalize so as to not hurt the user’s experience and ear drums.</li><li>We want to make sure ad creatives are appropriately rated for the content they play against. For example, an advertisement for tobacco should not be shown against content rated PG-13</li><li>For advertisements coming through programmatic means, like DSPs, we want to identify the brand they are advertising, both for reporting as well as frequency capping purposes.</li></ol><p>The steps above are relegated to a set separate services and machine learning models, worthy of their own blog post. Our ad server, like all services at Tubi, communicates with it using gRPC with a sub-10ms p99 SLA.</p><h3>Step Seven: Auction</h3><p>We are now at one of the most, if not the most, important steps in an ad request’s lifecycle. We have a number of bids at our disposal, from a number of different advertising partners and internally sourced campaigns, and we need to quickly decide which of those bids will fill our ad pod, which ranges from one to five slots. While most other parts of an ad request flow deal extensively with network I/O, caching, concurrency, etc, the auction is 100% CPU, solving a complex constraint optimization problem. A simplified version of the optimization problem is shown below. Let there be a matrix <em>n x k, </em>consisting of n advertiser bids vying for k advertising slots in the ad pod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PO51oMt1KgvVRhXd" /></figure><p>Each cell in the matrix can take values 1 or 0, indicating whether the bid in position was selected or not respectively. It is easy then to show how various constraints can be set as simple sums. For example, to make sure a particular advertiser bid is only selected once, all we need to do is add a constraint that the sum of each matrix row cannot be larger than 1. Similarly, to make sure that only a single bid is selected for each position in our ad pod, we add a constraint such that the sum of each matrix column must be less than or equal to 1. More constraints can be added, for example to enforce competitive separation, certain UX guarantees around competitive ads, and more. The solver then works tirelessly to optimize our objective function subject to the constraints we added. The objective function itself is a linear combination of revenue optimization as well as encompassing certain guarantees around delivery.</p><h3>Step Eight: Render</h3><p>In the last and final step, we collect all winning bids and render them as appropriate for each platform. Some of our client applications prefer VAST XML, some prefer custom JSON. Using the built-in akka-http rendering mechanisms, we can output as many predefined templates as we want.</p><h3>Step Nine: Join Tubi</h3><p>Tubi’s in-house AdTech stack is one of many areas where we leverage our own technology to deliver a superior product compared to using an off-the-shelf solution. If you are passionate about distributed systems and reactive programming in Scala, we are <a href="https://corporate.tubitv.com/company/careers/?utm_source=eng-blog&amp;utm_content=rm-part-2&amp;utm_campaign=eng-blog">actively hiring</a>. In addition to AdTech, we use the same technologies to build machine learning platforms, big data systems and much more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2a3526be6a0" width="1" height="1" alt=""><hr><p><a href="https://code.tubitv.com/a-fully-reactive-ad-serving-platform-using-scala-akka-streams-b2a3526be6a0">A fully reactive ad serving platform using Scala &amp; akka-streams</a> was originally published in <a href="https://code.tubitv.com/">Tubi Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A fully reactive ad serving platform using Scala & Akka-Streams]]></title>
            <link>https://code.tubitv.com/a-fully-reactive-ad-serving-platform-using-scala-akka-streams-13299e7ea04e?source=rss-8b91e81aede0------2</link>
            <guid isPermaLink="false">https://medium.com/p/13299e7ea04e</guid>
            <category><![CDATA[akka]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[reactive-streams]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Marios Assiotis]]></dc:creator>
            <pubDate>Wed, 06 Nov 2019 14:42:46 GMT</pubDate>
            <atom:updated>2019-11-06T14:42:46.415Z</atom:updated>
            <content:encoded><![CDATA[<p>When people first find out about Tubi offering premium, studio-produced movies &amp; TV shows for free, their first question if often “how is this legal?”; their second question is inevitably “but how do you make money?”. Senator, we run ads! In fact, Tubi has been generating healthy revenue since day 1, allowing us to grow and sustain our business without dependence on outside venture capital.</p><p>The secret sauce was our own in-house ad server, one of the first products the company built. The original ad server was written in PHP and used MySQL and Memcached as backing stores. While the PHP code base was exceptionally stable and quite performant, it was cumbersome to add new features to it. It was also difficult to reason about its internal workings, as the architecture evolved organically over the span of five years. To make matters worse, we had very few people with PHP expertise left in the company. We thus decided to embark on a rewrite with the following goals:</p><ul><li>Remove business logic technical debt that accumulated over the years</li><li>Support more advanced pacing and auction mechanics</li><li>Better observability and debugging tools for our ad operations team</li><li>Better test coverage</li></ul><p>Scala &amp; Akka was our tech stack of choice. We already had a lot of experience from both building high-throughput analytics pipelines written with Akka Streams and operating low-latency ML serving infrastructure that uses Akka actors. We also use Scala extensively in the context of Spark to train our machine learning models.</p><p>We decided to model the entire business logic as a stream and keep state in actors. To understand the reasons behind our choice, let’s first examine how a typical ad serving request works. During content playback, there are many candidate ad insertion points which are determined automatically during video asset ingestion using a machine vision algorithm. A few seconds before each ad insertion point, the video player makes a call to our ad server, asking if it wants to insert ads. Our ad server then has less than one second to evaluate the request, decide if it does indeed want to serve ads and if yes, go through a multi-step process in order to do so.</p><p>In the first part of this blog post, we will look at the major steps that happen right after receiving the request through to evaluating which campaigns are eligible. In the second part, we will look at how we receive bids for each campaign and how those bids are evaluated.</p><h3>Step One: Request Parsing &amp; Validation</h3><p>Because the entire request/response cycle is modeled as a stream, we flatten our entire hierarchy of campaign -&gt; line items -&gt; creatives into single ad creative units. Each ad creative unit goes through the flow one by one, and a simplified version of it looks like the diagram below:</p><figure><img alt="A high level overview diagram of the how a single request is processed in a streaming fashion" src="https://cdn-images-1.medium.com/max/780/1*39I18s4Ip2t4urP6ixoXpA.png" /><figcaption>Simplified view of all the steps in a single ad request</figcaption></figure><p>Each one of the processing stages in the diagram is a separate graph. Each graph can be authored, reasoned and tested in isolation. Akka-Streams also provided a robust mental model to reason about concurrency and provided backpressure out the box.</p><p>With the Ad Platforms team being an Akka shop, it makes sense to use Akka-Http as our web framework because parsing, validation as well as instrumentation are built-in. In the first step, we use the routing DSL, the built-in Directives as well as some custom ones to perform extensive validation of the request parameters. The work on input validation pays dividends down the road because before code hits production it helps catch client application bugs in development &amp; staging environments.</p><h3>Step Two: Enrichment</h3><p>To successfully evaluate an ad request, we must load metadata from various other services. We do so using gRPC, the standard for service to service calls at Tubi. In addition, we maintain user metadata spread across the clustered application and each user is represented as an actor. The user actor holds 30 day ad viewing history, which allows us to do appropriate frequency capping and make sure we do not inundate the user with too many ads or too many instances of the same ad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*J5RLNX9t7iBlEafbKytNfw.png" /><figcaption>A request is enriched with metadata at the start of the stream</figcaption></figure><p>In the diagram above, we fan-out the async I/O calls and merge the results back in, before passing the combined result to a separate processing stage. The next stage can then determine whether the user should be shown advertisements, and if so, how many (if any at all). Each one of these operations is its own flow, which can be authored and tested in isolation before being made part of the overall main graph.</p><h3>Step Three: Filters/Targeting</h3><p>After we collect all the necessary metadata, and assuming we decided to continue processing the request, we pass each creative through a series of filters and evaluate various targeting criteria. Again, this is modeled like a simple graph, with each creative being evaluated one by one, and rejections automatically generating events which we can store for later analysis or debugging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JySjzO3T6PTSwuYgoQ9axw.png" /><figcaption>Targeting filters processed in a streaming fashion</figcaption></figure><p>The productivity benefits of this approach cannot be overstated. For example, a filter that evaluates whether a creative should be filtered out on the basis of targeting a specific content genre would look something like this:</p><pre>class GenresFilter extends BaseFilter(&quot;genre&quot;) {<br>  override def predicate: (RichContext, AdCandidate) =&gt; Boolean = {<br>    case (context: BaseRequestContext, ad: AdCandidate) =&gt;<br>      if (ad.targeting.genreBlacklist.nonEmpty) {        !ad.targeting.genreBlacklist.exists(context.content.genres.contains)<br>      } else if (ad.targeting.genre.nonEmpty) {<br>        ad.targeting.genre.exists(context.content.genres.contains)<br>      } else {<br>        true<br>      }<br>  }<br>}</pre><p>All the machinery and boilerplate is encapsulated in a custom graph. Each filter then becomes nothing more than a simple predicate function, which can be tested in isolation.</p><h3>}</h3><p>So far, we discussed why we decided to embark on a rewrite and an overview of why &amp; how we use Scala, Akka &amp; Akka-Stream to power our new ad tech stack. We looked at the high level overview on how a stream can be used to model a request/response cycle and did a dive into three of those steps in the stream. In a follow up post, we will look at the remaining steps in the flow, including pacing, bidding and the auction making the final decision on which ads to display. If you are passionate about building distributed systems in Scala, <a href="https://tubitv.com/static/careers">Tubi is hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13299e7ea04e" width="1" height="1" alt=""><hr><p><a href="https://code.tubitv.com/a-fully-reactive-ad-serving-platform-using-scala-akka-streams-13299e7ea04e">A fully reactive ad serving platform using Scala &amp; Akka-Streams</a> was originally published in <a href="https://code.tubitv.com/">Tubi Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Staying agile without Agile]]></title>
            <link>https://code.tubitv.com/staying-agile-without-agile-e1e03a69bd76?source=rss-8b91e81aede0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1e03a69bd76</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kanban]]></category>
            <category><![CDATA[agile]]></category>
            <dc:creator><![CDATA[Marios Assiotis]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 15:31:01 GMT</pubDate>
            <atom:updated>2019-06-18T15:31:01.058Z</atom:updated>
            <content:encoded><![CDATA[<p>Few things can stir such a heated debate in the tech community as trying to decide which software development methodology reigns supreme. Waterfall, Agile, Kanban are just a few examples that get thrown around and not necessarily always with the exact same descriptions.</p><p>As the team grew at Tubi, we found ourselves lost at times with prioritizing, identifying blockers, ensuring items didn’t fall through the cracks and other growing pains. With this in mind, a small team of engineers volunteered to embark on a three month adventure to come up with a system that works for us and can scale with the company, at least in the foreseeable future. In this post we will go through how and why we came up with this system, what worked and what didn’t.</p><h3><strong>Key Insights</strong></h3><p><strong>Easy, I can get it done in a day</strong></p><p>The first thing all engineers agreed upon, is that estimating is notoriously difficult. More often than not, we tend to underestimate the size of the task at hand, resulting in stress, frustration and missed deadlines.</p><p><strong>Deadline driven development must die</strong></p><p>The second, and probably most controversial point is that assigning deadlines to every single engineering task makes no sense for a company like Tubi. The vast majority of things we build are not for external paying customers. They are iterative improvements to our core product, Tubi TV. As such, the answer to “<em>when will this be done?</em>” is “<em>as soon as possible</em>”.</p><p>But if we don’t have deadlines, how do we make sure we are actually delivering? Simple. Our engineers have a passion for building high quality software and a desire to see it in the hands of our users. That said, setting deadlines can still be useful when there are third party dependencies, i.e. external integrations OR sometimes even as forcing functions for ourselves.</p><p><strong>Ship it!</strong></p><p>Putting the previous two points together, we came to the realization that perhaps the most important thing for a consumer facing startup like Tubi is to be<strong> shipping </strong>often and with a regular cadence. As such, we considered sprints as dictated by agile methods. We even took a two day seminar taught by an experienced instructor. However, we all agreed that the rigid structure of one/two/four week sprints was, well, too rigid. Some team members expressed that when in a setting that follows the sprint system, they often found themselves stressed towards the end of the sprint. Others that sandbagged estimates to avoid rolling over tasks from sprint to sprint sometimes found themselves idle halfway through a sprint.</p><p><strong>Not everyone has the same job</strong></p><p>Paul Graham laid it down nicely in his essay “<a href="http://www.paulgraham.com/makersschedule.html">maker’s schedule, manager’s schedule</a>”. At Tubi we don’t have pure management layers, everyone in the engineering organization is also a maker. That said, tech leads are expected to take on more meetings, take initiative in prioritization, push projects forward and help coordinate cross-team. A team lead cares if one of their team members is blocked on something or overloaded or idle. They also care whether the various projects and milestones are tracking towards completion.</p><p>An individual contributor on the other hand, needs long periods of focused time, no randomization, a well-communicated sense of direction to help from going off on tangents and enough freedom to allow for creativity.</p><p><strong>Not every team has the same needs</strong></p><p>A video streaming service may seem simple from the outside, but to have something operating at Tubi’s scale, you need to build (or buy) a lot of moving parts, each with its own requirements. For example, the Machine Learning team needs long periods where they research open ended problems, whereas the various client teams usually have more concrete tasks. The iOS team and Android teams need to go through a store and push an update to all users, whereas the backend team can release multiple times a day.</p><h3>Goals, Non-Goals and Constraints</h3><p>With the above, and a couple of more insights, we defined some clear goals of what we wanted to get out of our development process. Our goals were:</p><ol><li>Regular shipping cadence: We want to be shipping and experimenting with a regular and predictable pace, while allowing the team to keep the quality bar high</li><li>Move the Needle: As a high growth startup with limited resources, we should only be working on things that move the needle, directly or indirectly. This process should help us prioritize</li><li>Have fun: We have a culture where we all love learning new things and building software. Anything to help us do that more, means making work more enjoyable</li></ol><p>Equally important is to call out what is a non-goal. Things we explicitly wanted to avoid while coming up with our new development process were:</p><ol><li>Coming up with KPIs (Key Performance Indicators) to measure engineer productivity. As sophisticated as these KPIs seem to be getting with the advent of machine learning, we have found that they are not much different from using lines of code like it was prevalent in the 80s and 90s.</li><li>Avoid process tax as much as possible. For example, we decided that coming up with estimation points for every little thing was not worth the effort. However, coming up with a level of effort for larger projects during the planning phase is a good forcing function to identify risks.</li><li>Similarly, we wanted to limit the number of recurring meetings to only what is absolutely necessary in order to keep things running smoothly.</li></ol><p>We also have a few unique constraints that shaped our thinking. Most important is that about a third of our engineering team is based in Beijing and has a 15 hour time difference. Even within each office, San Francisco or Beijing, most engineers observe different schedules. As such, even if they were a good idea (they are not), daily morning stand ups were out of the question.</p><h3>Development Process at Tubi</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL7kDUUA0DS4HNsQObekHQ.jpeg" /></figure><p>We centered everything around the concept of a release train. It’s a very simple analogy, where completed tasks are humans queuing on the platform to catch a train. The train leaving the station represents shipping a release to a production environment and making it available to our users. There are only a few rules:</p><ul><li>Code complete and QA’ed tasks can queue up</li><li>Trains leave on schedule, no feature creep</li><li>If a certain item is not ready, catch the next train, no waiting!</li><li>Schedules do not need to align between teams. For example, a backend team can have a daily train schedule whereas the train for a mobile team leaves every three weeks</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sdIarBVH2dX1uR0YPCRNqw.png" /></figure><p>To help us plan and prioritize, we borrowed a little from Kanban and introduced the concept of a queue. A queue is prioritized list of tasks per team. When an engineer is done with a task, they can pull something else from this queue, ideally something close to the top if possible. Doing planning and work allocation independently and asynchronously means that there is no serialization penalty and productive engineers can move faster. We also limit the amount of work in progress for each engineer to avoid over-commitment and burn out.</p><p>The natural question to ask is who prioritizes the queue? The team lead drives the process but each team can choose how to keep their queue up to date. Some may chose to let the team lead coordinate everything while others may prefer to regularly huddle and sort together.</p><h3>Tooling</h3><p>Tools are not a silver bullet, but they help. We decided to move from JIRA to ClubHouse and we haven’t looked back. We enjoy CH for its simplicity, spiffy UI/UX and the great GitHub integration. We even created and open sourced a little command line tool that uses the ClubHouse API to make it easier to create ad-hoc stories quickly, i.e.. when doing a quick bug fix.</p><h3>What worked, what didn’t</h3><p>We have been working in this system for about a year now. Overall, we are happy, although not everything went as expected.</p><p><strong>Went Well</strong></p><ul><li>Client teams are indeed shipping more often, more predictably and with higher quality</li><li>We don’t have too many meetings</li><li>We do a good job on planning and executing small to medium sized projects</li><li>Smaller tasks do not fall through the cracks</li><li>Queue system allows engineers the freedom and flexibility to work on what they think is best</li><li>Having stories move automatically from one stage to the next as they go through the commit, PR, merge, release cycle is life changing</li></ul><p><strong>Needs Improvement</strong></p><ul><li>Backend teams are finding it hard to stick to a regular schedule and sometimes release ad-hoc</li><li>We can probably have even less meetings</li><li>We need to get better at planning big projects that span multiple teams</li><li>Big, cross-team projects may still feel a bit chaotic</li><li>Maintaining the queue is harder than what we expected</li><li>Further improvements are needed in our tooling to deal with stale tasks</li></ul><p>There are a lot of details we omitted here, but this post is long enough as it is. If you are interested in working in an engineering environment where process is a tool rather than a master and the focus is on building great software, <a href="https://tubitv.com/static/careers">join us</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1e03a69bd76" width="1" height="1" alt=""><hr><p><a href="https://code.tubitv.com/staying-agile-without-agile-e1e03a69bd76">Staying agile without Agile</a> was originally published in <a href="https://code.tubitv.com/">Tubi Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Structured Streaming at Tubi (Part I)]]></title>
            <link>https://code.tubitv.com/spark-structured-streaming-at-tubi-part-i-44801de0e15a?source=rss-8b91e81aede0------2</link>
            <guid isPermaLink="false">https://medium.com/p/44801de0e15a</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[streaming-analytics]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Marios Assiotis]]></dc:creator>
            <pubDate>Wed, 27 Dec 2017 20:36:17 GMT</pubDate>
            <atom:updated>2017-12-27T20:36:17.717Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Spark is one of the most revolutionary open source projects in the recent years. It provides a one stop shop for extracting data from multiple different sources, transforming it in various ways, and loading it in a data warehouse of your choice. For loading and saving data, Spark comes built in capable of interacting with popular backends and formats like S3, HDFS, JSON, CSV, parquet, etc and many others provided by the community.</p><p>For transformations, Spark abstracts away the complexities of dealing with distributed computing and working with data that does not fit on a single machine. Unlike other engines like Hadoop, Spark does not require you to explicitly model your transformations in terms of Map/Reduce operations. Instead, you can work with familiar functional operators like map, filter, folds, even arbitrary functions. In addition to providing the toolset for working with big data, Spark comes bundled with its own machine learning library containing a slew of popular algorithms.</p><p>At Tubi, we use Spark extensively for our ETLs and even for building some of our machine learning models. In this series of blog posts, we will explore how we take advantage of Spark’s latest feature, Structured Streaming, to move from scheduled jobs to real time processing. In part 1, we will structure a simple framework for working with streaming jobs. In subsequent posts, we will discuss unit testing and how we use Akka and Akka-Streams to apply end to end back pressure in a streaming job. All posts will contain usable code samples extracted from production code that has been battle tested on our own clusters here at Tubi.</p><p><strong>Why spark structured streaming?</strong></p><p>While there is an abundance of choices these days when it comes to streaming frameworks, Spark Structured Streaming which officially came out of beta this summer, was the best choice for us. The main reason for our choice were:</p><ul><li>We already have a lot of experience, both operating and developing for Apache Spark</li><li>We had already made significant investments integrating our CI/CD pipeline with Databricks cloud and our data team loves working with Databricks notebooks</li><li>The ability to reuse the same core business logic to run a massive ETL across terabytes of data or streaming processing small batches 24/7 off a queue</li><li>Exactly once semantics as a nice to have</li></ul><p>As a downside, Spark Structured Streaming is relatively new and is missing a few features found in more mature offerings, for example joining streams. It’s also a poor choice if ultra-low latency is required, at least until <a href="https://issues.apache.org/jira/browse/SPARK-20928">continuous processing</a> makes it into a release.</p><p><strong>Structured Streaming Job broken down</strong></p><p>At a very high level, a structured streaming job consists of an input Dataset and an output Dataset. A Dataset is Spark’s typed DataFrame, introduced in Spark 2.0. Let’s look at a hypothetical example where we aggregating video playback events for each user and calculating CVT (Cumulative View Time). We start by defining some case classes to hold our event data along with some convenience methods.</p><pre>case class ViewEvent(userId: Int, ts: Timestamp, viewTimeMs: Long)</pre><pre>case class OutputPayload(userId: Int, cvt: Long, playbackStart: Timestamp, playbackEnd: Timestamp) {<br>  def +(e: ViewEvent): OutputPayload = copy(<br>    cvt = cvt + e.viewTimeMs,<br>    playbackStart = min(playbackStart, e.ts),<br>    playbackEnd = max(playbackEnd, e.ts)<br>  )<br>}</pre><pre>object OutputPayload {<br>  def apply(userId: Int): OutputPayload = new OutputPayload(userId, 0, new Timestamp(Int.MaxValue), new Timestamp(0))<br>}</pre><p>We also define a function that encapsulates our core business logic</p><pre>def compute(events: Dataset[ViewEvent]): Dataset[OutputPayload] = {<br>  events<br>    .groupByKey(_.userId)<br>    .mapGroups((userId: Int, events: Iterator[ViewEvent]) =&gt; {<br>      events.foldLeft(OutputPayload(userId))((out, evt) =&gt; out + evt)<br>    })<br>}</pre><p>So far nothing out of the ordinary, just static transformations, written in the usual Spark style and can be tested independently of whether it is an ETL or a streaming job. To convert that to a streaming job, we need two components, a DataStreamReader and a DataStreamWriter.</p><p>Starting with the reader:</p><pre>def inputStream: DataStreamReader = {<br>    spark<br>      .readStream<br>      .parquet(“/path/to/data”).<br>}</pre><pre>def makeDataset(stream: DataStreamReader): Dataset[ViewEvent] = {<br>    stream<br>      .load()<br>      .select(&quot;userId&quot;, &quot;ts&quot;, &quot;viewTimeMs&quot;)<br>      .as[ViewEvent]<br>}</pre><p>and the writer</p><pre>def write(ds: Dataset[OutputPayload]): DataStreamWriter[OutputPayload] = {   <br>  ds<br>    .writeStream<br>    .queryName(“SampleStreamingQuery”)<br>    .option(&quot;checkpointLocation&quot;, “/path/to/checkpoint/directory”)<br>}</pre><pre>def startQuery(writer: DataStreamWriter[OutputPayload]): StreamingQuery = {<br>  writer.start<br>}</pre><pre>def runAll: StreamingQuery = {<br>  startQuery(write(compute(makeDataset(inputStream))))<br>}</pre><p>By breaking our streaming job into small functions, we have separated out the boilerplate, allowing us to reuse the core business logic in both ETL and streaming jobs as well as making it easier to unit test. In the process, we have also created a basic framework that can be packaged neatly in a reusable base class, parametrized by our input and output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c84ff97145e30b20d11dc692c408adac/href">https://medium.com/media/c84ff97145e30b20d11dc692c408adac/href</a></iframe><p>In a follow up post, we will look in more detail how we can operationalize structured streaming jobs. We will look into using akka streams to reliably connect streaming jobs to external services, gotchas with stateful streaming, monitoring, and a few tricks to combine streaming and static Dataframes. If you enjoy working with data pipelines and Apache Spark, we are <a href="http://tubitv.com/static/careers">hiring</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44801de0e15a" width="1" height="1" alt=""><hr><p><a href="https://code.tubitv.com/spark-structured-streaming-at-tubi-part-i-44801de0e15a">Spark Structured Streaming at Tubi (Part I)</a> was originally published in <a href="https://code.tubitv.com/">Tubi Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>