<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sameer Abhyankar on Medium]]></title>
        <description><![CDATA[Stories by Sameer Abhyankar on Medium]]></description>
        <link>https://medium.com/@saabhyankar?source=rss-9d41ebd0edce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*bISc_m7RqxAD_hgtqdFGXA.jpeg</url>
            <title>Stories by Sameer Abhyankar on Medium</title>
            <link>https://medium.com/@saabhyankar?source=rss-9d41ebd0edce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:43:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saabhyankar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Profiling Dataflow Pipelines]]></title>
            <link>https://medium.com/google-cloud/profiling-dataflow-pipelines-ddbbef07761d?source=rss-9d41ebd0edce------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddbbef07761d</guid>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[google-dataflow]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <category><![CDATA[google-stackdriver]]></category>
            <dc:creator><![CDATA[Sameer Abhyankar]]></dc:creator>
            <pubDate>Wed, 01 May 2019 20:55:38 GMT</pubDate>
            <atom:updated>2022-02-23T22:10:06.505Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Update Feb ’22:</em></strong><em> Dataflow now supports profiling with general availability — See documentation </em><a href="https://cloud.google.com/dataflow/docs/guides/profiling-a-pipeline"><em>here</em></a><em>!</em></p><p><a href="https://cloud.google.com/dataflow/">Google Cloud Dataflow</a> is a fully managed service for executing batch and streaming data pipelines written using the <a href="https://beam.apache.org/">Apache Beam SDK</a>. In addition to managing and orchestrating the end-to-end pipeline execution for the users, Dataflow provides a number of out of the box features such as <a href="https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#autoscaling">Auto-scaling</a>, <a href="https://cloud.google.com/blog/products/gcp/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow">Dynamic Work Re-balancing</a> (aka Liquid Sharding) and <a href="https://cloud.google.com/dataflow/docs/guides/using-stackdriver-monitoring">native Stackdriver integration</a> that further enhances Dataflow’s zero-knobs story. This makes Dataflow an ideal execution platform for transforming and enriching data on the Google Cloud Platform.</p><h4>Great — What exactly is a Dataflow pipeline?</h4><p>A Dataflow pipeline can be described as a series of <a href="https://beam.apache.org/documentation/programming-guide/#transforms"><em>transform</em></a><em>s</em> that is applied to a distributed collection of data elements as these elements move from source to target. These transforms are generally provided by the authors of the pipeline (<em>aka user-code</em>) and describe business rules that is applied to the data elements to get them into a usable form.</p><p>As such, the overall performance of the data pipeline is highly dependent on the performance of the user-code behind these transforms.</p><p>How many times have you heard someone complain — <em>“My job is running slow”?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*5mrHFJXzLZqzbW0cawxPmg.gif" /></figure><blockquote><em>Probably way more than we care to count!!</em></blockquote><h4>OK — So the pipeline is running slow. How can I troubleshoot it?</h4><p>Dataflow and the Apache Beam SDK provides a number of facilities in the form of <a href="https://beam.apache.org/releases/javadoc/2.12.0/org/apache/beam/sdk/metrics/Metrics.html">metrics</a> and integrated Stackdriver logging to help troubleshoot pipelines. However, these might not always be sufficient due to various reasons:</p><ul><li>Prone to trial-and-error as the author attempts to add verbose logging and metrics to identify the bottleneck.</li><li>Excessive logging adds its own overhead to the pipeline performance.</li><li>Metrics might be relevant where the user code interacts with the Beam model. However, the user code in the bottleneck step might still be opaque to the users.</li></ul><p><em>This is where Dataflow Profiling can help.</em></p><p>Dataflow profiling is a process that allows us to get more insights about the pipelines execution at the granularity of the user-code. In other words, profiling helps us double-click into the actual user code that is executing in the <em>transforms</em> that make up the pipeline.</p><h4>Fantastic — So how do we profile a Dataflow pipeline?</h4><p>One of the easier ways to profile a Dataflow pipeline is to leverage the native integration that Dataflow has with <a href="https://cloud.google.com/profiler/">Stackdriver Profiler</a>. Stackdriver Profiler visualizes the call hierarchy and resource consumption of the pipeline in an interactive flame graph that can be used to analyze performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxf7xYKJTsZnZ6pVWhvkjA.png" /><figcaption>Flame graph in Stackdriver Profiler visualizing a Dataflow pipeline call stack</figcaption></figure><p>Dataflow’s native integration with Stackdriver profiler makes it extremely easy to enable profiling for any Dataflow job.</p><p>For a pipeline written using Apache Beam’s Java SDK, profiling can be enabled by passing an additional parameter that accepts a <em>Map</em> of properties:</p><pre>mvn clean compile exec:java \<br>-Dexec.mainClass=MyClass\<br>-Dexec.args=” \<br>— runner=DataflowRunner \<br>— project=my-project \<br>— jobName=my-job \<br>— stagingLocation=gs://my-staging-bucket \<br>— tempLocation=gs://my-temp-bucket \<br>……. &lt;&lt;other pipeline specific flags&gt;&gt; …..<br><strong><em>--profilingAgentConfiguration=’{ \”APICurated\”: true }’</em></strong></pre><p><strong><em>Note:</em></strong> <em>Do </em><strong><em>not</em></strong><em> set — saveProfilesToGcs=&lt;gcs-bucket-for-profiles&gt; when the above configuration is used as this will disable profiling!</em></p><p>Adding the boolean property <em>APICurated</em> set to <em>true</em> as a <em>Map</em> input to the <em>profilingAgentConfiguration</em> property is all that is needed to enable profiling for the pipeline.</p><p>Once the pipeline starts executing, browse over to the <em>Google Cloud Console, click on the Navigation Menu and the Profiler feature</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*aSPoilVrshOJJJpBFlSZTg.png" /></figure><p>Then click on the service drop-down to search for the Dataflow job that we want to analyze:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vpOxi5rUIHLf2NOLfPw8Wg.png" /></figure><h4>Awesome — We identified and fixed the bottleneck. Is there an easy way to compare the performance before and after the changes?</h4><p><em>Absolutely!</em> Stackdriver Profiler makes it super simple to<a href="https://cloud.google.com/profiler/docs/comparing-profiles"> compare jobs</a>. Just execute the job with the <em>same</em> <em>jobName</em> and use Stackdriver Profiler’s “Compare to” drop-down to pick a different version <em>(i.e. job_id).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*4IL8GG3RHDh_hUmLWH9ufA.png" /></figure><h4>Are there any other profiling knobs that we can leverage?</h4><p><em>Yes!</em> The profiling engine used by Dataflow exposes a number of other additional knobs:</p><pre><strong>APICurated</strong> =<em> Boolean flag to enable profiling</em> // Default: false<br><strong>Interval</strong> = <em>Profile output frequency           </em>// Default: 60 secs<br><strong>Duration</strong> = <em>Trace duration per interval        </em>// Default: 10 secs<br><strong>Delay</strong> = <em>Delay tracing by seconds</em>              // Default: 0 secs<br><strong>MaxCounts</strong> = <em>Max number of profiles collected</em>  // Default: unlimited<br><strong>NativeStacks</strong> = <em>Boolean flag to enable native stack unwind</em> // Default: false</pre><p>However, setting just the <em>APICurated</em> flag to <em>true</em> should be the only change needed in most cases.</p><h4>Are there any other profiling tools?</h4><p><em>Sure!</em> While Stackdriver Profiler is very easy to use given the native integration, there might be a situation where a deeper dive into the execution call tree is needed. In these rare cases, we could leverage <a href="https://github.com/google/pprof"><strong><em>pprof</em></strong></a> which is an open source tool for visualizing and analyzing profile data. Pprof analyzes profile data that is dumped by Dataflow pipelines and has several features that do not currently exist in Stackdriver Profiler. These include:</p><ul><li>Graph view (<em>pprof -graphviz</em>)</li><li>Tree view (<em>pprof -graphviz -call_tree</em>)</li><li>Top-Down/Bottom-Up tree view</li><li>Additional output formats (pdf etc.)</li></ul><h4>How do I extract the profile information needed by pprof?</h4><p>This is pretty straightforward and is done by passing an additional property to the command line while executing the pipeline. So for example, a Dataflow pipeline written using Apache Beam’s Java SDK can dump the profiles to GCS by passing a property — <strong><em>saveProfilesToGcs</em></strong> as follows:</p><pre>mvn clean compile exec:java \<br>-Dexec.mainClass=MyClass\<br>-Dexec.args=” \<br>— runner=DataflowRunner \<br>— project=my-project \<br>— jobName=my-job \<br>— stagingLocation=gs://my-staging-bucket \<br>— tempLocation=gs://my-temp-bucket \<br>……. &lt;&lt;other pipeline specific flags&gt;&gt; …..<br><strong><em>--saveProfilesToGcs=gs://YOUR BUCKET HERE/profiler</em></strong></pre><p><em>That’s all!</em> Once the job starts executing, the Dataflow service will write profile files (*wall* and *cpu*) files at regular intervals <em>(approx 1 file each for cpu and wall times every 60 seconds)</em></p><h4>Great — How do I analyze these files using pprof?</h4><p>Once enough profile information <em>(approx 10 mins)</em> is collected or the job finishes <em>(in case it is a batch pipeline)</em>, download the profile files locally and use pprof to visualize the cpu or wall time profiles. Pprof has a number of command line options to assist with the analysis. For e.g.:</p><pre>pprof --call_tree --web “http:” &lt;LOCAL PATH FOR PROFILES&gt;/*cpu*</pre><p>While there are a lot more details about using each of these tools that we could potentially talk about, this overview should hopefully introduce everyone to a couple of very useful troubleshooting and performance tuning tools that should be part of every Dataflow pipeline developers toolkit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*JCikDPTZ7WWOshtydOZ2Tg.gif" /></figure><h4>Additional Details and Notes:</h4><p>While this overview only covers profiling from a CPU consumption perspective, we hope to do a follow up blog that describes the process of profiling memory utilization for the Dataflow pipelines. Memory profiling is currently not integrated into Stackdriver Profiler and is more of a manual process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddbbef07761d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/profiling-dataflow-pipelines-ddbbef07761d">Profiling Dataflow Pipelines</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Cloud Dataflow to index documents into Elasticsearch]]></title>
            <link>https://medium.com/google-cloud/using-cloud-dataflow-to-index-documents-into-elasticsearch-b3a31e999dfc?source=rss-9d41ebd0edce------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3a31e999dfc</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-dataflow]]></category>
            <category><![CDATA[cloud-bigtable]]></category>
            <dc:creator><![CDATA[Sameer Abhyankar]]></dc:creator>
            <pubDate>Fri, 17 Aug 2018 00:02:55 GMT</pubDate>
            <atom:updated>2018-09-27T17:24:27.998Z</atom:updated>
            <content:encoded><![CDATA[<p>You have a <em>lot</em> of data sitting in your system but your customers still keep complaining that they cant find anything! Sounds familiar? If so, perhaps it is time to add a scalable, distributed search engine such as <a href="https://www.elastic.co/blog/found-uses-of-elasticsearch#you-know-for-search-and-counting"><strong>Elasticsearch</strong></a> to your tech stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*r3-yb6xWYC5nI7W9yOGQBg.gif" /></figure><p>Elasticsearch <a href="https://www.elastic.co/about/partners/google-cloud-platform">is pretty easy to deploy on the Google Cloud Platform </a>and you can set it up pretty quickly. The next step is to figure out how to index — or make searchable — all this data that you are adding into your environment.</p><p>Enter <strong>Cloud Dataflow</strong>!</p><p><a href="https://cloud.google.com/dataflow/">Cloud Dataflow is a fully managed service</a> to execute a data pipeline (batch or streaming) written in<em> </em><a href="https://beam.apache.org/"><em>Apache Beam</em></a>. Luckily, Apache Beam’s Java SDK provides a <a href="https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.html">handy connector for writing (and reading) documents to Elasticsearch</a>. You could do things like validating the data that you want to index or even enhancing it with some external data sets — All this as part of the same data pipeline.</p><p>This blog and the <a href="https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-elasticsearch-indexer">accompanying sample code on Github</a> will attempt to put the pieces of the puzzle together and demonstrate how all this would work as part of a single pipeline running on Cloud Dataflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OaVwGy1MNotrx3Oi6a9Zeg.png" /><figcaption>Dataflow pipeline to index documents into Elasticsearch</figcaption></figure><p>There is a lot going on in the above diagram and so the following steps drill down into how all these components work together:</p><ol><li>Cloud Pub/Sub is an excellent way to stream data into the environment and Cloud Dataflow can easily read from Cloud Pub/Sub making it the data streaming tool of choice in this approach.</li><li>Cloud Bigtable is ideal for ad-hoc low latency lookups and is great for storing any external data needed to enhance documents in flight.</li><li>Using Apache Beam’s ElasticsearchIO Java connector makes it very easy to write (index) documents into Elasticsearch.</li><li>Documents that fail validation might need some manual (or automated) intervention before being reprocessed. In this approach, these documents are published to a different Cloud Pub/Sub topic.</li><li>Optionally, the process of correcting the errors and making the failed documents available for reprocessing can be automated — With Cloud Dataflow of course!</li></ol><p>As an added benefit, <a href="https://cloud.google.com/dataflow/service/dataflow-service-desc#autoscaling">autoscaling can be easily enabled for a streaming pipeline</a> by setting a couple of flags:</p><pre>--autoscalingAlgorithm=THROUGHPUT_BASED<br>--maxNumWorkers=&lt;N&gt;</pre><p>This allows Cloud Dataflow to adjust the number of workers based on the amount of backlog it needs to process — Only use resources when needed!</p><p><strong><em>Great! But how do we address the data that already exists in our system?</em></strong></p><p>A simple modification to the pipeline to read the input from Google Cloud Storage bucket <em>instead</em> of Cloud Pub/Sub should be the only thing that would need to change in the pipeline.</p><p><strong><em>So can we use this approach with the latest and greatest release of Elasticsearch?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*iz0zf4y_Du5NJwYu14sEBw.jpeg" /><figcaption>Alllmost…Not quite..</figcaption></figure><p>Well — <em>Almost</em> :) Currently Apache Beam’s ElasticsearchIO supports Elasticsearch v5.x (and v2.x) out of the box. However there is <a href="https://issues.apache.org/jira/browse/BEAM-3199">active work in the Apache Beam community to support Elasticsearch v6.x as well</a>. An interim solution would be to use a custom transform to talk to Elasticsearch 6.x <a href="https://www.elastic.co/guide/en/elasticsearch/client/java-rest/6.3/java-rest-high.html">using their Java High-Level REST client</a>.</p><p><strong>Ready for a working sample?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*AmI9wRbXrfIWGESx6eEiTw.gif" /></figure><p>Getting started with this pipeline and sample data for a quick demo (who doesn’t like a demo!) cannot be easier — Point your browser to the <a href="https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/dataflow-elasticsearch-indexer">accompanying Github repo</a> and follow along!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3a31e999dfc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/using-cloud-dataflow-to-index-documents-into-elasticsearch-b3a31e999dfc">Using Cloud Dataflow to index documents into Elasticsearch</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>