<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel K on Medium]]></title>
        <description><![CDATA[Stories by Daniel K on Medium]]></description>
        <link>https://medium.com/@daniel_k?source=rss-f985b17dea5f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*OT5Piid6Or6es7JZ</url>
            <title>Stories by Daniel K on Medium</title>
            <link>https://medium.com/@daniel_k?source=rss-f985b17dea5f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:29:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@daniel_k/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Filtering Events With Apache Beam Pipeline Using Side Input]]></title>
            <link>https://medium.com/@daniel_k/filtering-events-inside-an-apache-beam-pipeline-using-side-input-d6ad722c52b?source=rss-f985b17dea5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6ad722c52b</guid>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[apache-beam]]></category>
            <dc:creator><![CDATA[Daniel K]]></dc:creator>
            <pubDate>Mon, 20 Feb 2023 15:29:01 GMT</pubDate>
            <atom:updated>2023-03-19T17:10:23.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mGTgpGhKN5CzhIZl" /><figcaption>Photo by <a href="https://unsplash.com/@medbadrc?utm_source=medium&amp;utm_medium=referral">Med Badr Chemmaoui</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article we will go over a feature where we found it best to use side input inside an Apache Beam pipeline. We will see the motivation for the feature and its requirements, different solutions we had in mind, and how we implemented it.</p><p><strong>#1. Motivation</strong></p><p>As part of our work, we are using Apache Beam pipelines to transfer, manipulate and enrich events. In some cases, we transfer the events from PubSub topics to BigQuery tables. When a tenant is churned, we wouldn’t want its events to get into BQ tables anymore, even though they can still get into some PubSub topics after the churning process. Thus, we had to come with a way to filter out these events inside the dataflow.</p><p><strong>#2. Feature Requirements</strong></p><ol><li>The pipeline will filter out events based on their tenant id.</li><li>The pipeline should have access to the most updated list of churned<br>tenants.</li><li>If, for some reason, the pipeline doesn’t have access to the most updated list, it should use the most recently updated list it had access to.</li><li>Filtering is done on each event separately, so accessing the list should not harm the pipeline performance.</li></ol><p>Also, bear in mind that we have a service named X that is in charge of churning tenants and can supply an updated list of churned tenants on demand.</p><p><strong>#3. Different optional solutions</strong></p><blockquote>PubSub subscription into <a href="https://beam.apache.org/documentation/programming-guide/#side-inputs">side input</a></blockquote><p>Service X will publish the full churned tenants list each time it is updated, to a PubSub topic. Our pipeline will subscribe to this topic. It will aggregate the lists it gets into a side input map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*mM6LcHiPcjFhjUo735Z8Ug.jpeg" /><figcaption>PubSub subscription into side input</figcaption></figure><p>Pros:</p><ol><li>Using PubSub meaning having a ready-made retry/acknowledgement system, and since churning a tenant happens only once in a while, we can assume that losing an update will be rare.</li><li>We can use the <a href="https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html">PubsubIO</a> class to read from the subscription. It is coming from an Apache Beam package, so it is well trusted.</li><li>Side input is available in-memory for keeping the pipeline high performances.</li></ol><p>Cons:</p><ol><li>When deploying new pipeline version, it loses the churned tenants list. Since we don’t have control of when the next tenant will churn, we don’t have control of the next time the pipeline will get the churned tenants list. Meaning that when deploying a version, we have an unbounded time window where churned tenants’ events can get into BigQuery tables.</li><li>If, for some reason, the pipeline is missing an update, it will have to wait until the next churning tenant to get an updated list.</li></ol><blockquote>Saving the list in a Redis DataBase</blockquote><p>Service X will manage the churned tenants list in a Redis DB. The pipeline will have a direct connection with the DB and will check on each event if its tenant is churned or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*JLfXVStQ9q4AOeHU1D_xCA.jpeg" /><figcaption>Saving the list in a Redis DataBase</figcaption></figure><p>Pros:</p><ol><li>The list is always available for the pipeline. Even when deploying new version, the pipeline immediately connects to the DB server and starts querying on events.</li><li>The pipeline has always an access to the most updated churned tenants list. If, for some reason, service X failed to update the DB it can try again, but once it updated the DB the pipeline immediately has access to the most updated churned tenants list. This contrasts with previous solution, where pulling from PubSub can fail and then we must wait for the next update.</li><li>By using Redis, which is an in-memory DB, we save IO operations overhead.</li></ol><p>Cons:</p><ol><li>When processing thousands of events per seconds, querying the DB over the net on each event (or even bundle of events) can be very expensive and hurt pipeline performances.</li><li>Coupling the pipeline with a Redis DB. The pipeline needs to know how to query the DB, which is meant for very specific purpose.</li></ol><blockquote>Querying service X into side input</blockquote><p>The pipeline will query the service (http request) to get the tenants list and will aggregate the results into a side input map. The querying will occur on pipeline initialization and repeatedly forever every Y units of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*uPKmG-8wxhqm-vdiEkvNZQ.jpeg" /><figcaption>Querying service X into side input</figcaption></figure><p>Pros:</p><ol><li>An updated list will be retrieved when deploying new pipeline version.</li><li>Side input is available in-memory for keeping the pipeline high performances.</li><li>Pipeline has control of when it is getting the list. If, for some reason, it failed to get it in a specific attempt, it is known when the next time will be it tries to get it. Until then it will use the most recently updated list it managed to retrieve.</li></ol><p>Cons:</p><ol><li>The pipeline doesn’t always have access to the most updated list. If a tenant is churned and the pipeline queries the service only a day after, then for a whole day this tenant’s events can get into BQ tables. It means that we are loosening the requirement number 2 and we will have to deal it later.</li></ol><blockquote>The chosen solution — “Querying service X into side input”</blockquote><p>We couldn’t choose option 1 since it has a problem with new version deployment, and option 2 can hurt our pipeline performance. Option 3 answers all our requirements. It is true that we are loosening feature requirement number 2 — but an event that got into our BigQuery tables, after its tenant got churned, can be deleted by running a simple script over the tables, next time the pipeline is getting the churned tenants list.</p><p><strong>#4. Implemented Solution</strong></p><pre>PCollectionView&lt;Map&lt;Integer, Integer&gt;&gt; churnedTenants =<br>  pipeline<br>    .apply(&quot;TriggerUpdate&quot;, <br>      GenerateSequence.from(0).withRate(1, Duration.standardDays(1L)))<br>    .apply(&quot;GetChurnedTenants&quot;, <br>      FlatMapElements<br>        .into(TypeDescriptors.integers())<br>        .via(new GetChurnedTenants(new HttpManager(), options.getChurnedTenantsUrl())))<br>    .apply(&quot;ChurnedTenantsToKV&quot;, <br>      MapElements<br>        .into(TypeDescriptors.kvs(TypeDescriptors.integers(), TypeDescriptors.integers()))<br>        .via(tenantId -&gt; KV.of(tenantId, tenantId)))<br>    .apply(&quot;KVIntoGlobalWindow&quot;, <br>      Window.&lt;KV&lt;Integer,Integer&gt;&gt;into(new GlobalWindows())<br>        .triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))<br>        .discardingFiredPanes()))<br>    .apply(&quot;CreateSideInputMap&quot;, View.asMap());</pre><blockquote><em>Partially based on this </em><a href="https://beam.apache.org/documentation/patterns/side-inputs/#slowly-updating-global-window-side-inputs"><em>code example</em></a></blockquote><p>Let’s go over the steps:</p><blockquote>TriggerUpdate</blockquote><p>Generates a PCollection&lt;Long&gt; with one single value.</p><p>This collection is what triggers the process of getting the churned tenants list. The generation happens once a day and will first occur on pipeline deployment.</p><blockquote>GetChurnedTenants</blockquote><p>Generates a PCollection&lt;Integer&gt; that holds the churned tenants ids.</p><p>By using FlatMapElements, we are telling the pipeline to apply a simple 1-to-many mapping function (GetChurnedTenants implements SerializableFunction&lt;Long, List&lt;Integer&gt;) over each element in the incoming collection, which contains one single element. The many elements are flattened into the resulting collection.</p><p>The GetChurnedTenants class is using an HttpManager to get the churned tenants ids from service X and returns them as a List&lt;Integer&gt;. If there was a problem with getting the list, it will return List.of(-1). This way, we won’t stuck event processing until successfully getting the list.</p><blockquote>ChurnedTenantsToKV</blockquote><p>Maps each churned tenantId to a KV.of(tenantId, tenantId).</p><p>This mapping might seem odd but we are doing it so we can use a Map as side input.</p><blockquote>KVIntoGlobalWindow</blockquote><p>Inserts each KV into a global window, using the GlobalWindows window function.</p><p>We are using a global window so that the side input will never be expired, only updated. Conceptually, churned tenants list can only be expanded (adding elements and not removing), so it fits us. Another thing is, that once we got an updated list, we can use it until the next time we are getting an updated list, even if between we had couple of failed attempts.</p><p>The trigger is necessary to tell when to emit elements from the global window. Eventually we are working here with an unbounded PCollection,</p><blockquote>The collection produced by the ChurnedTenantsToKV is unbounded since the <a href="https://beam.apache.org/releases/javadoc/2.1.0/index.html?org/apache/beam/sdk/io/GenerateSequence.html">GenerateSequence</a> PTransform is producing an unbounded collection.</blockquote><p>so if we wouldn’t have specified a trigger, then elements wouldn’t have been emitted from the global window, and we couldn’t use our side input. But before that, we would have get an error upon pipeline construction :)</p><blockquote>CreateSideInputMap</blockquote><p>Eventually, we are creating the PCollectionView as a Map — that will be the side input.</p><p>Using a Map has 2 noticeable advantages:</p><ol><li>For each event, the check would be in O(1). If we would have been using a List, for example, then for each event the check would be in O(n) where n is the number of churned tenants. This could harm the pipeline performances.</li><li>Updating it is easy and retrieved data is not overridden — lets say we had a successful attempt to retrieve the list and GetChurnedTenants returns this list: [1,2,3]. Then, our side input map would be: {1:1, 2:2, 3:3}. Next time, the attempt fails, so GetChurnedTenants returns this list: [-1]. Then -1 will be added to the map, resulting: {1:1, 2:2, 3:3, -1:-1}. Information from previous calls wasn’t overridden and we didn’t need to add special treatment for updating the map.</li></ol><p><strong>Conclusion</strong></p><p>We saw the motivation for the feature, its requirements and different possible solutions. For each solution, we talked about its pros and cons, while referring to the requirements. I presented the chosen solution and why we chose it. At last, we went over the implementation and explained each step.</p><p>Hope you found it interesting :)</p><p>Special Thanks to <a href="/service/mailto:adam.abrams1@gmail.com">adam.abrams1@gmail.com</a> for mentoring the process of research and development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6ad722c52b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>