<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Reut Levy on Medium]]></title>
        <description><![CDATA[Stories by Reut Levy on Medium]]></description>
        <link>https://medium.com/@reut_l?source=rss-550b2a77038a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*X9XcZAjnqYHFDvsi</url>
            <title>Stories by Reut Levy on Medium</title>
            <link>https://medium.com/@reut_l?source=rss-550b2a77038a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:06:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@reut_l/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Beam — Dataflow prime usages in load problems]]></title>
            <link>https://medium.com/@reut_l/apache-beam-dataflow-prime-usages-in-load-problems-d32aa098ab1a?source=rss-550b2a77038a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d32aa098ab1a</guid>
            <dc:creator><![CDATA[Reut Levy]]></dc:creator>
            <pubDate>Sun, 05 Mar 2023 13:55:10 GMT</pubDate>
            <atom:updated>2023-03-05T13:55:10.954Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Beam — Dataflow prime usages in load problems</h3><p>While processing big amount of data you need to find the right tool that will fit your requirements, but even with the right tool sometimes memory and performance errors can come up.</p><p>As part of developing a new feature, we created a new service based on <strong>GCP Apache Beam Dataflow</strong>, the service is part of campaign processing.</p><p>The service receives Pub Sub messages as initial input to the pipeline, each pub-sub message include a path to Avro files.</p><p>An Avro file can include large number of records that each record represents a customer in the campaign. The service should read the Pub Sub messages, and process the records in the Avro file.</p><p>As part of the development we preform load tests on the service. The purpose of those tests was to examine how the service handles with large number of customers in a campaign.</p><p>The load tests include 5 pub-sub messages, each message include 5 Avro files and each Avro file contain about 1M records, total 25M customers.</p><p>During the load test we got this error: <strong>“java.lang.OutOfMemoryError: Java heap space”.</strong></p><p>In order to fix this error and that the service will be able to handle this amount of customers we realized that we should use <strong>Dataflow Streaming Engine</strong> and increase the maximum number of workers.</p><p>By default, the Dataflow pipeline runner executes the steps of your streaming pipeline entirely on worker virtual machines. Dataflow’s Streaming Engine moves pipeline execution out of the worker VMs and into the Dataflow service backend.</p><p>We added this to the Yaml file in the Run Job section:</p><pre>--enable-streaming-engine \<br>--max-workers=15\</pre><p>We preformed the load tests again and expected to see the number workers increase after the changes and yet we got the same error again and the number of workers was 1.</p><p>We couldn’t understand what is the problem. how could it be that the CPU usage of the worker is almost 100% and dataflow do not raise more workers?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*9oVe2_N4NuwD5ZK4o7rR9g.png" /></figure><p>After consultation with the architect in our company we realized that maybe the reason that the number of workers didn’t increase is that the amount of messages the pipeline receives as initial input is small, but each message is very heavy.</p><p>Dataflow increases the number of workers by the initial input of the pipeline and if the initial input is small, Dataflow will not recognize that there is a load in a later step and the number of workers should increase.</p><p>In order to solve this problem we use <strong>Dataflow Prime</strong>. Dataflow Prime has many advantages, one of them is <strong>right fitting</strong> — meaning that resources are fitted to each stage of the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pYsuvoQkESZZv5CpOcmmGA.jpeg" /></figure><p>After we used Dataflow Prime we preformed the load tests once again. At those tests the number of workers indeed increase after few minutes from beginning of the tests when the pipeline reached the problematic step. Dataflow Prime recognize that there is a load at one of the steps in the pipeline and increase the number of workers in accordance.</p><p>In the following, we can see that the number of workers grew significantly at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*3hei9Ck3LZheUpE0j5xpRQ.png" /></figure><p>Another advantage of Dataflow Prime is Vertical Autoscaling. Vertical Autoscaling means that you’re adding more compute power to your existing instances/nodes.</p><p>Vertical Autoscaling<strong> </strong>is a feature that enable only in Dataflow Prime. This works along with Horizontal Autoscaling to dynamically scale up or down the memory allocated to each worker based on utilization to best fit the needs of the job.</p><p>In this graph you can see that when the used memory was almost as the memory capacity, the worker memory increased because of the Vertical Autoscaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*DGV7TTq5x0XUBQ_ElpNSjw.png" /></figure><p>The combination of all DataFlow Prime abilities led to fast campaign processing. (It took about 50 minutes to process a campaign with 26M users). Dataflow Prime help to solve memory problems that came up during the pipeline, raise the number of workers when needed and enable vertical autoscaling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d32aa098ab1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>