<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sugimiyanto on Medium]]></title>
        <description><![CDATA[Stories by Sugimiyanto on Medium]]></description>
        <link>https://medium.com/@sugimiyanto?source=rss-3be27556505e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*84XPZT6TnjaL-jpKoXVUpQ.jpeg</url>
            <title>Stories by Sugimiyanto on Medium</title>
            <link>https://medium.com/@sugimiyanto?source=rss-3be27556505e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 09:07:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sugimiyanto/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Idempotency: the thing we should implement to have a reliable data pipeline in batch processing]]></title>
            <link>https://sugimiyanto.medium.com/idempotency-the-thing-we-should-implement-to-have-a-reliable-data-pipeline-in-batch-processing-b13664be630d?source=rss-3be27556505e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b13664be630d</guid>
            <category><![CDATA[batch-processing]]></category>
            <category><![CDATA[partitioned-table]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[idempotency]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Sugimiyanto]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 15:29:43 GMT</pubDate>
            <atom:updated>2020-12-08T15:36:52.796Z</atom:updated>
            <content:encoded><![CDATA[<p>In order to have a reliable data pipeline for batch processing, there are several keys we have to implement. Idempotency is one of them.</p><h3>Term: idempotency</h3><p>If you are not familiar with the term “idempotency”. Idempotency is a condition where no matter how many times a job or task is executed, it always results in the exact same result. A simple example of idempotency is a built-in function abs() in python which returns the <a href="https://www.mathsisfun.com/numbers/absolute-value.html">absolute value</a> from the given input.</p><pre>abs(-6512)<br># output: 6512</pre><pre>abs(abs(-6512))<br># output: 6512</pre><pre>abs(abs(abs(-6512)))<br># output: 6512</pre><pre>abs(abs(abs(abs(-6512))))<br># output: 6512</pre><p>As we can see from the above code snippet. There is no difference in the output of a function abs() when we call abs() many times.</p><h3>Idempotency in a data pipeline</h3><h4>Case</h4><p>Suppose we have a data pipeline that performs <a href="https://www.guru99.com/etl-extract-load-process.html">ETL</a> tasks. It ingests data from a data lake, conducts data transformations which again stores the result in the data lake, and finally loads the transformed data to a data warehouse for analytics purposes.</p><p>Most likely, there might be a case when we need to do a backfill process. A backfill process is a process where we redo the ETL tasks but to the past day&#39;s data. This case would be required for instance where we found a bug in the transformation stage let’s say incorrect data format or serializing which turns to block the business intelligence team to do the analysis. Once we fixed the bug, we have to rerun the data pipeline to the past days which have incorrect data so that it could be corrected.</p><h4>BigQuery</h4><p><a href="https://cloud.google.com/bigquery">BigQuery</a> is a serverless data warehouse that allows us to store data in a <a href="https://cloud.google.com/bigquery/docs/partitioned-tables">partitioned table</a>. The partition could be based on the value of DateTime, date, timestamp column e.g. partition by DAY from field timestamp , or could be based on ingestion time using pseudo-column _PARTITIONTIME that contains a date-based timestamp when we load data to the table. This ability allows us to have filtered data by partition column to save cost where we only select data within a time range that we have a concern about instead of scanning whole data. Besides, the partitioned table allows us to have isolated data based on its partition which then enables idempotency.</p><h4>Backfill Job</h4><p>The backfill job must be idempotent in order to not break the existing data. The partitioned table allows us to do so. Moreover, technically, during data loading to BigQuery, we have to specify a specific partition to which we want the data is loaded as shown in the following snippet code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebe3737d009d1fc49fa6d7c728749085/href">https://medium.com/media/ebe3737d009d1fc49fa6d7c728749085/href</a></iframe><p>Pay attention to lines 37–45. This line of code makes our backfill job idempotent. Let’s break it down.</p><ul><li>line 33–39. We are defining time partitioning for target partitioned table. We specify the <a href="https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.table.TimePartitioning.html">type of time partitioning</a> to perform which could be HOUR, DAY, MONTH, YEAR. If we set as DAY, then our data is partitoined or isolated per day, e.g. 2020–01–01 and 2020–01–02 have their own data. We also set the field used for partitioning the data.</li><li>line 41–43. We set the destination partitioned table to load data. Notice the dollar sign “$”, it is used to determine to which partition we are going to insert data. In this case, our partitioned table has type of DAY, and the value after dollar sign “$” is in format of <em>YYYYmmDD</em> without any separator e.g. 20200101 for 2020–01–01.</li><li>line 45. We specify that the data loading should be write truncate, which means overwrite all the existing data with the given new data in the defined specific partition.</li></ul><p><em>Thanks for reading! Keep It Short and Simple (KISS).</em></p><p><em>Any thought or suggestion? feel free to leave a comment.</em></p><p><em>Cheers!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b13664be630d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>