<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aniket Ghole on Medium]]></title>
        <description><![CDATA[Stories by Aniket Ghole on Medium]]></description>
        <link>https://medium.com/@aniketghole88?source=rss-140bf6cddac9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d6R7AwDA2Z6UQtGlnl43AA.jpeg</url>
            <title>Stories by Aniket Ghole on Medium</title>
            <link>https://medium.com/@aniketghole88?source=rss-140bf6cddac9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:18:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aniketghole88/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Beam Pipeline for Cleaning Batch Data Using Cloud Dataflow and BigQuery]]></title>
            <link>https://medium.com/data-science/apache-beam-pipeline-for-cleaning-batch-data-using-cloud-dataflow-and-bigquery-f9272cd89eba?source=rss-140bf6cddac9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9272cd89eba</guid>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[batch-processing]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Aniket Ghole]]></dc:creator>
            <pubDate>Sun, 12 Jul 2020 07:48:37 GMT</pubDate>
            <atom:updated>2020-07-20T05:32:55.925Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building Data Processing Pipeline With Apache Beam, Dataflow &amp; BigQuery</h3><h4>Implementation of the beam pipeline that cleans the data and writes the data to BigQuery for analysis.</h4><p>There are various technologies related to big data in the market such as Hadoop, Apache Spark, Apache Flink, etc, and maintaining those is a big challenge for both developers and businesses. Which tool is the best for batch and streaming data? Are the performance and speed of one particular tool enough in our use case? How should you integrate different data sources? If these questions often appear in your business, you may want to consider Apache Beam.</p><p><a href="https://beam.apache.org/"><strong>Apache Beam</strong></a> is an open-source, unified model for constructing both batch and streaming data processing pipelines. Beam supports multiple language-specific SDKs for writing pipelines against the Beam Model such as <strong>Java</strong>, <strong>Python</strong>, and <strong>Go </strong>and Runners for executing them on distributed processing backends, including <strong>Apache Flink</strong>,<strong> Apache Spark</strong>, <a href="https://cloud.google.com/dataflow"><strong>Google Cloud Dataflow</strong></a> and <strong>Hazelcast Jet</strong>.</p><p>We will be running this pipeline using Google Cloud Platform products so you need to avail your free offer of using these products up to their specified free usage limit, New users will also get $300 to spend on Google Cloud Platform products during your <a href="https://cloud.google.com/free">free trial</a>.</p><p>Here we are going to use <a href="https://beam.apache.org/get-started/quickstart-py/"><strong>Python SDK</strong></a> and <strong>Cloud Dataflow</strong> to run the pipeline.</p><h4><strong>The Anatomy of a Data Pipeline</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KSbxc89zsZ-AWLJSsAtPvQ.png" /><figcaption><a href="https://link.springer.com/chapter/10.1007/978-1-4842-4470-8_40">Key Concepts of Pipeline</a></figcaption></figure><ul><li><strong>Pipeline: </strong>manages a directed acyclic graph (DAG) of PTransforms and PCollections that is ready for execution.</li><li><strong>PCollection: </strong>represents a collection of bounded or unbounded data.</li><li><strong>PTransform: </strong>transforms input PCollections into output PCollections.</li><li><strong>PipelineRunner: </strong>represents where and how the pipeline should execute.</li><li><strong>I/O transform: </strong>Beam comes with a number of “IOs” — library PTransforms that read or write data to various external storage systems.</li></ul><p>I have clipped some commonly used higher-level transforms (Ptransforms) below, we are going to use some of them in our pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WII3iZ-V8FLUn_pINE5UWA.png" /><figcaption><a href="http://shzhangji.com/blog/2017/09/12/apache-beam-quick-start-with-python/">Common Transforms in Pipeline</a></figcaption></figure><p><strong>ParDo</strong> is a primary beam transform for generic parallel processing which is not in the above image. The ParDo processing paradigm is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm: a ParDo transform considers each element in the input PCollection, performs some processing on that element, and emits zero, or multiple elements to an output PCollection.</p><p><strong>Pipe <em>‘|’ </em></strong>is the operator to apply transforms, and each transform can be optionally supplied with a unique label. Transforms can be chained, and we can compose arbitrary shapes of transforms, and at runtime, they’ll be represented as DAG.</p><p>The above concepts are core to create the apache beam pipeline, so let&#39;s move further to create our first batch pipeline which will clean the dataset and write it to BigQuery.</p><h4>Basic flow of the pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ayAEAIrpsqG88IvpSdyvRA.png" /><figcaption><a href="https://miro.medium.com/max/1250/1*2RSf9nVv6-MaLhVjW2F2tA.png">Pipeline Flow</a></figcaption></figure><ol><li>Read the data from google cloud storage bucket (Batch).</li><li>Apply some transformations such as splitting data by comma separator, dropping unwanted columns, convert data types, etc.</li><li>Write the data into data Sink (<a href="https://cloud.google.com/bigquery">BigQuery</a>) and analyze it.</li></ol><p>Here we are going to use <a href="https://www.kaggle.com/nickhould/craft-cans">Craft Beers Dataset</a> from Kaggle.</p><p><strong>Description of the beer dataset</strong></p><p><strong>abv</strong>: The alcoholic content by volume with 0 being no alcohol and 1 being pure alcohol<br><strong>ibu</strong>: International bittering units, which specify how bitter a drink is<br><strong>name</strong>: The name of the beer<br><strong>style</strong>: Beer style (lager, ale, IPA, etc.)<br><strong>brewery_id</strong>: Unique identifier for a brewery that produces this beer<br><strong>ounces</strong>: Size of beer in ounces</p><p>We will upload this dataset to google cloud bucket.</p><p>Before we run the pipeline, we need to enable Dataflow and Bigquery APIs. Type Dataflow API in GCP search box and enable it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qljUFT5Kkkuo82O-NUjBAA.png" /><figcaption>Enabling API — Image By Author</figcaption></figure><p>Similarly, you need to enable BigQuery API.</p><p>Dataflow will use cloud bucket as a staging location to store temporary files. We will create a cloud storage bucket and choose the nearest location (Region).</p><p>For example— if you are in Asia, you must select Asia region for the speed and performance of computation (Dataflow Job).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsVooOU2Fj0RQ0wuqVxq7Q.png" /><figcaption>Creating GCS bucket — Image By Author</figcaption></figure><p>We will create BigQuery dataset and table with the appropriate schema as a data sink where our output from the dataflow job will reside in. The Dataset region will be your nearest location. It is Asia-south1 (Mumbai) in our case. You need to provide the output schema (already given in batch.py) while creating the table in BigQuery.</p><p>Next open cloud shell editor and set your project property if it is not already set and will clone the GitHub repository which has all supported files and data.</p><pre>git clone <a href="https://github.com/aniket-g/batch-pipeline-using-apache-beam-python">https://github.com/aniket-g/batch-pipeline-using-apache-beam-python</a></pre><p>Once it is done, change into the directory where all files reside.</p><p>Now copy the beer.csv file into our bucket using the command given below.</p><pre>gsutil cp beers.csv gs://ag-pipeline/batch/</pre><p>Alternatively, you can upload that CSV file by going to the Storage Bucket.</p><p>To run the pipeline, you need to have Apache Beam library installed on Virtual Machine.</p><pre>sudo pip3 install apache_beam[gcp]</pre><p>That&#39;s all.</p><p>Now we will walk through the pipeline code to know how it works. Mostly we will look at the Ptransforms in the pipeline.</p><pre>def discard_incomplete(data):<br>    &quot;&quot;&quot;Filters out records that don&#39;t have an information.&quot;&quot;&quot;<br>    return len(data[&#39;abv&#39;]) &gt; 0 and len(data[&#39;id&#39;]) &gt; 0 and len(data[&#39;name&#39;]) &gt; 0 and len(data[&#39;style&#39;]) &gt; 0</pre><p>We have filtered out the data which does not have information or null values in it.</p><pre>def convert_types(data):<br>    &quot;&quot;&quot;Converts string values to their appropriate type.&quot;&quot;&quot;<br>    data[&#39;abv&#39;] = float(data[&#39;abv&#39;]) if &#39;abv&#39; in data else None<br>    data[&#39;id&#39;] = int(data[&#39;id&#39;]) if &#39;id&#39; in data else None<br>    data[&#39;name&#39;] = str(data[&#39;name&#39;]) if &#39;name&#39; in data else None<br>    data[&#39;style&#39;] = str(data[&#39;style&#39;]) if &#39;style&#39; in data else None<br>    data[&#39;ounces&#39;] = float(data[&#39;ounces&#39;]) if &#39;ounces&#39; in data else None<br>    return data</pre><p>The above function will convert the string values to their appropriate data type.</p><pre>def del_unwanted_cols(data):<br>    &quot;&quot;&quot;Deleting unwanted columns&quot;&quot;&quot;<br>    del data[&#39;ibu&#39;]<br>    del data[&#39;brewery_id&#39;]<br>    return data</pre><p>In the above function, we deleted unwanted columns which ended up in cleaned data.</p><pre>p = beam.Pipeline(options=PipelineOptions())</pre><pre>(p | &#39;ReadData&#39; &gt;&gt; beam.io.ReadFromText(&#39;gs://purchases-3/beers.csv&#39;, skip_header_lines =1)<br>       | &#39;Split&#39; &gt;&gt; beam.Map(lambda x: x.split(&#39;,&#39;))<br>       | &#39;format to dict&#39; &gt;&gt; beam.Map(lambda x: {&quot;sr&quot;: x[0], &quot;abv&quot;: x[1], &quot;id&quot;: x[2], &quot;name&quot;: x[3], &quot;style&quot;: x[4], &quot;ounces&quot;: x[5]}) <br>       | &#39;DelIncompleteData&#39; &gt;&gt; beam.Filter(discard_incomplete)<br>       | &#39;Convertypes&#39; &gt;&gt; beam.Map(convert_types)<br>       | &#39;DelUnwantedData&#39; &gt;&gt; beam.Map(del_unwanted_cols)<br>       | &#39;WriteToBigQuery&#39; &gt;&gt; beam.io.WriteToBigQuery(<br>           &#39;{0}:beer.beer_data&#39;.format(PROJECT_ID),<br>           schema=SCHEMA,<br>           write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))<br>    result = p.run()</pre><p><strong>beam.io.ReadFromText</strong> — reads the data from external sources into the PCollection</p><p><strong>beam.map — </strong>works like <strong>ParDo, </strong>applied <strong>Map</strong> in multiple ways to transform every element in <strong>PCollection</strong>. The Map accepts a function that returns a single element for every input element in the PCollection.</p><p><strong>beam.Filter </strong>— accepts a function that keeps elements that return <strong>True</strong>, and filters out the remaining elements.</p><p><strong>beam.io.WriteToBigQuery </strong>— Write transform to a BigQuerySink accepts PCollections of dictionaries. It requires the following arguments</p><ul><li>TableReference can be a PROJECT:DATASET.TABLE or DATASET.TABLE string.</li><li>TableSchema can be a NAME:TYPE{,NAME:TYPE}* string (e.g. ‘month:STRING,event_count:INTEGER’).</li></ul><p>Now we run pipeline using dataflow runner using the following syntax.</p><pre>python3 batch.py --runner DataFlowRunner --project aniket-g --temp_location gs://ag-pipeline/batch/temp --staging_location gs://ag-pipeline/batch/stag --region asia-east1 --job_name drinkbeer</pre><p>Currently, Dataflow provides <a href="https://cloud.google.com/dataflow/docs/concepts/regional-endpoints.">regional endpoints</a> for some regions which do not include Asia-south1 hence I chose Asia-east1 in Region.</p><ul><li><strong>project</strong> — The ID of your Google Cloud project.</li><li><strong>runner</strong> — The pipeline runner that will parse your program and construct your pipeline. It can be direct runner also if you want to debug your pipeline. Here we are using Dataflow runner.</li><li><strong>staging_location</strong> — A Cloud Storage path for Dataflow to stage code packages needed by workers executing the job.</li><li><strong>temp_location</strong> — A Cloud Storage path for Dataflow to stage temporary job files created during the execution of the pipeline.</li><li><strong>region</strong> — You can specify region where you want to run your dataflow runner.</li><li><strong>job_name </strong>(Optional) — Give any name to the dataflow pipeline.</li></ul><p>Now go to Dataflow, you can see your job is running of batch type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQjo2KHho4EN5_jHZa-1CA.png" /><figcaption>Pipeline status — Image By Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*15dHeI-pjKOibzJd6inTcg.png" /><figcaption>DAG — pipeline steps — Image By Author</figcaption></figure><p>Once it is completed and succeeded, you will see results in the BigQuery beer_data table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8gsBIuB6m6jKHLwL5OfgQ.png" /><figcaption>BigQuery table — Image By Author</figcaption></figure><p>Now we can query out the data to get some insights.</p><pre># Beer style with highest alcohol by volume<br>SELECT<br>  style,<br>  SUM(abv) AS volume<br>FROM<br>  `aniket-g.beer.beer_data`<br>GROUP BY<br>  style<br>ORDER BY<br>  volume DESC</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dk8M6QckmwtgEGMXL4T_GQ.png" /><figcaption>Bigquery Insights — Image By Author</figcaption></figure><p><strong>Review the code:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a55993a02ac4a1ef3e17ea9c4fd82f0/href">https://medium.com/media/2a55993a02ac4a1ef3e17ea9c4fd82f0/href</a></iframe><p>The main objective of this article is to demonstrate how we can create a cleaning pipeline using an apache beam. I have used only one dataset which has beers information while another dataset has breweries information which could have given more insights.</p><h4>References:</h4><p><a href="http://shzhangji.com/blog/2017/09/12/apache-beam-quick-start-with-python/">http://shzhangji.com/blog/2017/09/12/apache-beam-quick-start-with-python/</a></p><p><a href="https://beam.apache.org/documentation/programming-guide/">https://beam.apache.org/documentation/programming-guide/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9272cd89eba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/apache-beam-pipeline-for-cleaning-batch-data-using-cloud-dataflow-and-bigquery-f9272cd89eba">Apache Beam Pipeline for Cleaning Batch Data Using Cloud Dataflow and BigQuery</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>