<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prathap Reddy on Medium]]></title>
        <description><![CDATA[Stories by Prathap Reddy on Medium]]></description>
        <link>https://medium.com/@prathap-reddy?source=rss-951ed98f2e7e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Prathap Reddy on Medium</title>
            <link>https://medium.com/@prathap-reddy?source=rss-951ed98f2e7e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:54:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@prathap-reddy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Cache reuse across DoFn’s in Beam]]></title>
            <link>https://medium.com/google-cloud/cache-reuse-across-dofns-in-beam-a34a926db848?source=rss-951ed98f2e7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a34a926db848</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[beam]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-patterns]]></category>
            <dc:creator><![CDATA[Prathap Reddy]]></dc:creator>
            <pubDate>Mon, 25 Jan 2021 18:06:05 GMT</pubDate>
            <atom:updated>2021-01-25T18:06:05.040Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://beam.apache.org/">Apache Beam</a> is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Pipelines can be written in a variety of language-specific (Java, Python etc) SDKs and can be executed on any beam compatible runners (Dataflow, Flink, Spark etc)</p><p>In this blog post we will discuss following topics:</p><ol><li>LifeCycle of a DoFn</li><li>Caching data for reuse across DoFn Instances</li><li>Example illustrating below concepts:<br> a) Build an in memory cache by loading data from an external source (e.g: Cloud Storage)<br>b) Reuse cached objects across DoFn Instances.<br>c) Refresh the cache via external trigger.</li></ol><h4><strong>DoFn Life Cycle</strong></h4><p>A Dofn is a user defined function used with ParDo transform that gets invoked for each input element to emit zero or more elements. A bundle is a collection of elements (i.e records/messages) processed as a single unit of failure. The division of the collection into bundles is arbitrary and selected by the runner. For example, in the case of Dataflow while a streaming runner may prefer to process and commit small bundles in order to achieve low latency, a batch runner prefers to process larger bundles to reach higher throughput.</p><p>Each <a href="https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.transforms.core.html?highlight=start_bundle#apache_beam.transforms.core.DoFn">DoFn class</a> has a process method invoked for each element and additional optional methods shown below to handle initialization and finalization tasks.</p><ul><li>setup — Invoked after creation of an instance. Good place to initialize transient in-memory resources, such as network connections, parsing a config file etc</li><li>start_bundle — Invoked before processing of each bundle. Certain runners create a DoFn instance per bundle whereas others reuse the same DoFn across bundles. Can be used to initialize any state used for the entire bundle.</li><li>finish_bundle — Invoked after processing of each bundle</li><li>teardown — Invoked before instance is discarded and used for any clean up</li></ul><p>In summary the entire flow appears as follows:</p><ul><li>Setup</li><li>Repeatedly process bundles:<br> * StartBundle<br> * Repeatedly Process Element<br> * FinishBundle</li><li>Teardown</li></ul><h4><strong>Shared Cache</strong></h4><p>There are scenarios that require a shared in memory data to be reused across DoFn Instances to improve space and access efficiency. Few of them include:</p><ul><li>Machine learning model used for local inference</li><li>Look up data loaded from external sources such as Cloud Storage, BigQuery etc.</li></ul><p>As mentioned earlier, beam allows developers to author pipelines using various <a href="https://beam.apache.org/documentation/sdks/java/">supported SDKs</a> with popular choices being Java and Python.</p><p>In Java SDK, each worker launches a JVM instance with multiple threads (determined by number of cores and job type i.e batch vs streaming ). Each thread executes multiple data bundles by creating either one DoFn instance per bundle or reuse the same instance across data bundles. Shared cache can be achieved via a Singleton instance.</p><p>In Python, parallelism is achieved through multiple processes due to <a href="https://wiki.python.org/moin/GlobalInterpreterLock">Global Interpreter Lock</a> limitation. Each worker launches one process (with multiple threads) per core and executes one or more bundles invoking one or more DoFn instances per thread.</p><p>Since runners can potentially recycle DoFn instances per bundle and also has multiple such instances running in parallel on different threads how do we ensure a single copy of cached data is shared across all these threads? Did you know that Beam 2.24 has introduced a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.utils.shared.html">shared</a> class to solve exactly the same challenge and allows you to manage a single instance of an object shared by multiple threads. Let us demonstrate this feature by building a Streaming data enrichment pipeline for a mock retail company.</p><h4><strong>Example: Shared cache for stream enrichment</strong></h4><p>Requirements for the pipeline are:</p><ul><li>Reads sales events continuously from a real time messaging service</li><li>Load lookup data such as customer details, product details from a persistent store.</li><li>Enrich the event with additional information by associating lookup identifiers (e.g: customerid, productid) to lookup details.</li><li>Write the buffered enriched events to persistent storage for every x minutes.</li></ul><p>Nothing fancy so far except for the following challenges:</p><ul><li>Since look up data changes only every x hours, can data be cached &amp; shared to improve pipeline performance?</li><li>How to notify the pipeline to refresh the cache as soon as new look up data arrives?</li></ul><p><strong><em>Slowly Changing Lookup Data<br></em></strong>Before diving into pipeline design we need to understand the nature and shape of the data. In our example, look up data is stored on Google Cloud Storage (can use BigQuery as well) with the following path format:</p><p>gs://bucket/prefix/yyyy/mm/dd/customers/{file1…N}.json<br>gs://bucket/prefix/yyyy/mm/dd/products/{file1…N}.json</p><p>On a daily basis, a separate pipeline generates the latest lookup data and sends a message to Pub/Sub containing the latest base path (i.e gs://bucket/prefix/yyyy/mm/dd).</p><p><strong><em>Schema<br></em></strong>Let us check the schema for sales event and corresponding lookup data. Customer and product look up data schema is similar containing key value pairs spread across different files.</p><pre><strong>Sales Event</strong>: <em>{&quot;Txid&quot;:1,&quot;timestamp&quot;: &quot;2020-05-02 19:08:24.796892&quot;,  &quot;customerid&quot;: 100, &quot;productcode&quot;: &quot;P023&quot;, &quot;qty&quot;: 1, &quot;sales&quot;: 97.65 }</em></pre><pre><strong><em>customers Lookup data</em></strong><em>:  {&quot;key&quot;:100, &quot;value&quot;: {“name”: “john”, “tier”, “premium” }</em></pre><pre><strong><em>products Lookup data</em></strong><em>:  {&quot;key&quot;:&quot;P023&quot;, &quot;value&quot;: {“sku”: “chair”, “category”, “furniture” }</em></pre><pre><strong>Enriched Event (i.e Output)</strong>: <em>{&quot;Txid&quot;:1, &quot;timestamp&quot;: &quot;2020-05-02 19:08:24.796892&quot;,  &quot;customerid&quot;: 100, &quot;productcode&quot;: &quot;P023&quot;, &quot;qty&quot;: 1, &quot;sales&quot;: 97.65,  &quot;customertier&quot;: &quot;premium”, &quot;productcategory&quot;: &quot;furniture&quot; }</em></pre><p><strong><em>Pipeline Design<br></em></strong>Our pipeline has two inputs and one output:</p><ul><li>Read sales events from a Google Cloud Pub/Sub Subscription as the primary input</li><li>Read the base path of look up data from a different Google Cloud Pub/Sub Subscription as side input. In Beam a side input is an immutable PCollection that can be used as an additional input to ParDo transform. Runners then broadcast the side input to all workers. An important note about side inputs is that they should be small enough to fit in memory.</li><li>The final enriched output is written to Google Cloud Storage.</li></ul><p>Let us walk through the pipeline code step by step:</p><p><strong>Step 1: Read the inputs</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c1b1b12d73085db09dd2df1ceb1b1de/href">https://medium.com/media/9c1b1b12d73085db09dd2df1ceb1b1de/href</a></iframe><ol><li>Sales events are read from a sales Pub/Sub Subscription and assigned to a <br> fixed window of 15 minutes duration</li><li>Side Input base file path is read from a side input Pub/Sub Subscription. <br>Despite projecting side input over a long window interval, the side input gets processed immediately due to a data driven trigger.</li></ol><blockquote>Be aware of the fact that side input expires at the end of the window and <br> main input would be blocked either until new side input arrives or the entire <br> side input window duration has passed whichever is earliest. To avoid this, <br> ensure the latest side input base path is passed at the expiry of each side <br> input window.</blockquote><p><strong>Step 2: Create a Shared Cache and enrich events</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e240572644f3704093da7602be74494/href">https://medium.com/media/6e240572644f3704093da7602be74494/href</a></iframe><ol><li>Create a shared handle that encapsulates a weak reference to a singleton instance of a shared resource and pass it while building DoFn instance.</li><li>During execution, along with each event DoFn will also receive the latest side input base path.</li><li>Inside the process method , the shared handle is used to acquire the side input data and works as shown below:</li></ol><ul><li>In case of initial call, as data is not available the load_sindeput method is used to load the data from an external source. In this example, for each side input type (i.e Customers, products) the data is read from Google Cloud Storage and loaded inside a multi level dictionary with top level key indicating side input type and value representing a dictionary matching the KV pairs with corresponding lookup data.</li></ul><blockquote>Important point to note here is , a Shared handle requires a weak reference to the cached object but several built-in types such as list and dict do not directly support <a href="https://docs.python.org/3/library/weakref.html">weak references</a>. This can be addressed by creating a subclass (for e.g: _SideInputContainer in our case) of the built in type.</blockquote><ul><li>In subsequent calls, the same cached object is used instead of loading from an external source.</li><li>If you noticed in addition to actual look up data, we have also created a metadata key to preserve the base path. If anytime the base path changes, the cache is refreshed by passing the new base path as a tag to acquire method.</li></ul><p><strong>Step 3: Write enriched events to Google Cloud Storage</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*caL1v6f_Hj3aZ_w5" /></figure><p>Pipeline DAG appears as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*OU3YoRoy9LQ8XJMOnhMPiA.png" /></figure><h3>Conclusion</h3><p>Thanks for reading and special thanks to <a href="https://medium.com/u/6800d4802d9a">Pablo Estrada</a> for reviewing the article. To summarize in this blog post we covered DoFn LifeCycle, how to build a cache that is shared across threads and an approach to refresh cache via an external trigger. For more information about beam and other interesting patterns refer to beam <a href="https://beam.apache.org/documentation/patterns/overview/">documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a34a926db848" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/cache-reuse-across-dofns-in-beam-a34a926db848">Cache reuse across DoFn’s in Beam</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming Data Generator Dataflow Flex Template on GCP]]></title>
            <link>https://medium.com/google-cloud/streaming-data-generator-978846c5c807?source=rss-951ed98f2e7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/978846c5c807</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Prathap Reddy]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 16:31:13 GMT</pubDate>
            <atom:updated>2021-01-20T21:35:08.054Z</atom:updated>
            <content:encoded><![CDATA[<p>We are excited to announce the launch of a new Dataflow Flex Template <strong>Streaming Data Generator</strong> that supports writing high volume JSON messages continuously to either Google Cloud Pub/Sub Topic or BigQuery or Cloud Storage in various formats (JSON/AVRO/PARQUET) depending on destination. In this blog post, we will briefly discuss the need and usage of the template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*V2jvGCJ4vheQWqCw_LzldA.png" /></figure><h3><strong>Flex Templates</strong></h3><p>Before diving into the details of Streaming Data Generator template functionality, let me introduce the templates at a very high level :</p><p>Primary goal of the <a href="https://cloud.google.com/dataflow/docs/guides/templates/overview">templates</a> is to package Dataflow pipelines in the form of reusable artifacts with support to launch through various channels (UI / CLI / REST API) so that these templates can be leveraged by various teams by supplying required pipeline parameters. In the first version of templates (called as <a href="https://cloud.google.com/dataflow/docs/guides/templates/creating-templates">traditional templates</a>) pipelines were staged on GCS and can be launched either from Console, gcloud command or from other services like Cloud Scheduler/Functions etc.</p><p>However traditional templates has certain limitations: <br> 1. Lack of support for Dynamic DAGS <br> 2. Large number of I/O’s doesn’t support runtime parameters</p><p><a href="https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates">Flex templates</a> overcome these limitations. Flex templates packages data flow pipeline code along with application dependencies in the form of docker images and stage the images in Google Container Registry (GCR). Flex template launcher service launches pipeline with parameters supplied in a similar fashion that a user launches pipeline during development.</p><h3><strong>What is Streaming Data Generator?</strong></h3><p>Streaming Data Generator template can be used to publish fake JSON messages based on user provided schema at a specified rate (messages per second) to either Google Cloud Pub/Sub Topic or BigQuery or Cloud Storage. <a href="https://github.com/vincentrussell/json-data-generator">JSON Data Generator</a><strong> </strong>library used by the pipeline supports various faker functions to be associated with a schema field. Pipeline supports configuration parameters to control the number of messages published per second (i.e QPS) and resource required (via autoscaling)</p><p>A few possible use cases are:</p><ol><li>Simulate large-scale real-time event publishing to a Pub/Sub topic to measure and determine the number and size of consumers required to process published events.</li><li>Generate synthetic data to a BigQuery table or a Cloud Storage bucket to evaluate performance benchmarks or serve as a proof of concept.</li></ol><p>Possibilities are endless in terms of the types of events you can generate.</p><h3><strong>How to run Pipeline?</strong></h3><p>Pipeline can be launched either from cloud console or via gcloud command.</p><p>To launch from cloud console:</p><ol><li>Go to the <a href="https://console.cloud.google.com/dataflow/jobs">Dataflow page</a> in the Cloud Console</li><li>Click <strong>Create job from template</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*GFt5JIKXZ1mUH7k4" /></figure><p>3. Select Streaming Data Generator from the <strong>Dataflow template</strong> drop-down menu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*TSj9QDWd1y44kLr69XBL2g.png" /></figure><p>4. Enter the job name</p><p>5. Enter required parameters as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*0auM8QNndw1x7SrqQyL9TQ.png" /></figure><p>6. Enter additional parameters such as sink details depending on sink type, output encoding, autoScalingAlgorithm and maxNumWorkers etc</p><p>7. Click Run job</p><p>To launch pipeline to publish messages to cloud pub/sub using gcloud use below command:</p><pre>gcloud beta dataflow flex-template run ${JOB_NAME} \<br> —-project=${YOUR_PROJECT_ID} \<br> —-region=${REGION_NAME} \<br> —-template-file-gcs-location=gs://dataflow-templates/latest/flex/Streaming_Data_Generator \<br> —-parameters \<br>schemaLocation=${SCHEMA_LOCATION}\<br>qps=${QPS},\<br>topic=${PUBSUB_TOPIC}</pre><h3>References</h3><ul><li>Code is available on <a href="https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/streaming-data-generator">GitHub</a></li><li>For documentation refer <a href="https://cloud.google.com/dataflow/docs/guides/templates/provided-utilities#streamingdatagenerator">Dataflow docs</a></li><li>Cloud Dataflow docs for various <a href="https://cloud.google.com/dataflow/docs/guides/templates/provided-batch">Batch</a> and <a href="https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming">Streaming</a> templates.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=978846c5c807" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/streaming-data-generator-978846c5c807">Streaming Data Generator Dataflow Flex Template on GCP</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>