<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arunkumar on Medium]]></title>
        <description><![CDATA[Stories by Arunkumar on Medium]]></description>
        <link>https://medium.com/@rako?source=rss-c92b98ca6070------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Pgfv2m0dFhDO2zF2IUAGMQ.jpeg</url>
            <title>Stories by Arunkumar on Medium</title>
            <link>https://medium.com/@rako?source=rss-c92b98ca6070------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:55:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rako/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spark and Cassandra’s SSTable loader]]></title>
            <link>https://medium.com/@rako/spark-and-cassandras-sstable-loader-ed55f6e67d17?source=rss-c92b98ca6070------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed55f6e67d17</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Arunkumar]]></dc:creator>
            <pubDate>Sun, 13 May 2018 03:00:28 GMT</pubDate>
            <atom:updated>2018-06-08T01:43:43.422Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why: We had a lot of very useful data in our Warehouse and wanted to take advantage of those data in some of our production service to enhance the user’s experience. So we choose to server them from Cassandra for all it’s pros which I’m am not going to get into in this blog.</em></p><p>First stage we went about writing a spark-cassandra exporter. It’s pretty simple and only a couple of line,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50fc8b6f13edf8a1dd9325c7cb7cac39/href">https://medium.com/media/50fc8b6f13edf8a1dd9325c7cb7cac39/href</a></iframe><p>This works and took around ~ 30 mins to write ~150 Million rows. But once our services went live we saw the read latencies going a bit high during the bulk insertion time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXczQ0OE6A9iB2X1yGOONA.png" /><figcaption>Latencies during Cassandra row writes</figcaption></figure><p>The spark-cassandra-connector that we are using here had a few configs that can be used to tune the writes <a href="https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md#write-tuning-parameters">here</a>. Tried a bunch of tuning along the line of reducing concurrent and reducing throughput_mb_per_sec. They helped a bit but still there’s a clear increase in read latency.</p><p>Cassandra has sstableloader and we thought of testing it for this case. And so changed the code to use and saw that there’s barely any notable read latency during this task (only a slight increase in the 99 percentile, caused by the IO waits).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Rnq0mk-59phunI8Ni7g0w.png" /><figcaption>Latencies during Cassandra SSTable loads</figcaption></figure><p>Also if you see the networks graph, the traffic is only on “network in” as now we are generating SSTables in spark and then pushing those tables directly to cassandra. The last spike in below network graph is from SSTable method and the rest are from batched writes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOkESxUMi93WGGOTXe2IjQ.png" /><figcaption>Network Traffic (Row writes vs SSTable load)</figcaption></figure><p>Now let’s get into how to do that in code,</p><ul><li>Using CQLSSTableWriter build the SSTables per partition</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c59cc5fd71d6e65737135b6ea5c7f36f/href">https://medium.com/media/c59cc5fd71d6e65737135b6ea5c7f36f/href</a></iframe><ul><li>We need to define the create and insert statements, but it’s easy to build that from the spark dataframe</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6be6b5f1dfb0bc095f1aeaedbec4361/href">https://medium.com/media/d6be6b5f1dfb0bc095f1aeaedbec4361/href</a></iframe><ul><li>And stream SSTable to Cassandra script. We pick a random Cassandra server and stream the SSTable to it. Host is chosen at random for a better load balancing of network traffic.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2314fc42b18363857d612f12ae2fc646/href">https://medium.com/media/2314fc42b18363857d612f12ae2fc646/href</a></iframe><ul><li>And finally the code that run’s it all,</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc968b7546cf4d8e5ea88aa24fa5a545/href">https://medium.com/media/cc968b7546cf4d8e5ea88aa24fa5a545/href</a></iframe><ul><li>As the no. of partitions Cassandra’s suggestion is several tens of megabytes large to minimize the cost of compacting, we use max of 256 MB per SSTable. “sizeInMB” can be calculated from HDFS.</li><li>Let say the size is 60GB, we will have 256 SSTables of size 256MB each.</li><li>Set this config “mapreduce.output.bulkoutputformat.streamthrottlembits” to throttle traffic to Cassandra.</li></ul><p><strong>Fyi,</strong></p><ul><li>SSTables has to be at-least several tens of megabytes in size to minimize the cost of compacting the partitions on the server side.</li><li>This methods increase IO wait since it’s writing directly to Disk and not memory like in Cassandra writes. Depending on the size of data and throughput, you need a SSD with high IOPS.</li></ul><p>We’ve been using this method in production for over 6 months now, writing around ~ 300 million rows in &lt; 30 mins without any issue to the read latencies.</p><p>Full example code can be found here, <a href="https://github.com/therako/sparkles/blob/master/src/main/scala/util/cassandra/SSTableExporter.scala">https://github.com/therako/sparkles/blob/master/src/main/scala/util/cassandra/SSTableExporter.scala</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed55f6e67d17" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Error rate monitoring in Dead Letter pattern using Apache Beam]]></title>
            <link>https://medium.com/@rako/error-rate-monitoring-in-dead-letter-pattern-using-apache-beam-193fd03f8970?source=rss-c92b98ca6070------2</link>
            <guid isPermaLink="false">https://medium.com/p/193fd03f8970</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Arunkumar]]></dc:creator>
            <pubDate>Sun, 04 Mar 2018 10:16:15 GMT</pubDate>
            <atom:updated>2018-07-10T01:14:50.822Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m if you are aware of Apache Beam you would know about “Dead Letter” pattern.</p><p>If you don’t, it’s very useful pattern to handle mutations in streaming data by handling invalid inputs using Side Outputs as a “Dead Letter” file. Here’s a blog about it <a href="https://cloud.google.com/blog/big-data/2016/01/handling-invalid-inputs-in-dataflow">https://cloud.google.com/blog/big-data/2016/01/handling-invalid-inputs-in-dataflow</a></p><p>So I have been using the pattern for a while and realized there is something missing to it. <strong>Monitoring - </strong>for every live pipeline we need a error threshold monitoring, that’s was missing in this. So here is how to solve that,</p><ul><li>First is obviously the “Dead Letter” patterned pipeline, and here is how it looked</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f-e-1517SvnXd6uI4rHNZQ.png" /><figcaption>“Dead Letter” Pattern</figcaption></figure><ul><li>Then we need a small local window depending on what intervals you want to check.</li><li>And a Accumulator that counts the elements in that window — this is simple to do.</li><li>And ParDo to check if the accumulated count is higher than the configured max error-rate.</li><li>Finally a notifier script. I choose to use Slack notifications, but this could be any kind of notifications. And here’s how the final pipeline looks like</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DpMz9Cn6SAA3Nuw8KTuxqg.png" /><figcaption>“Dead Letter” Pattern with Error rate Monitoring</figcaption></figure><p>And here is the pipeline code for it,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe92c6d2a401365865238365c342b8cb/href">https://medium.com/media/fe92c6d2a401365865238365c342b8cb/href</a></iframe><p>So there we go now there will be alerts on high errors and we will be able to find issues much earlier.</p><p>Full code for this example can be found here, <a href="https://github.com/therako/apache-beam-examples/blob/master/src/main/java/ApacheBeamExamples/DeadLetterPatternExample.java">https://github.com/therako/apache-beam-examples/blob/master/src/main/java/ApacheBeamExamples/DeadLetterPatternExample.java</a></p><h4>Future Improvements</h4><p>I have a plan to change the error rates to percentile instead of using fixed nos, as the incoming message rate is not constant across time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=193fd03f8970" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building custom packages with Docker]]></title>
            <link>https://medium.com/@rako/building-custom-packages-with-docker-ad30f606b0b0?source=rss-c92b98ca6070------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad30f606b0b0</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[packaging]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Arunkumar]]></dc:creator>
            <pubDate>Sun, 25 Feb 2018 10:17:37 GMT</pubDate>
            <atom:updated>2018-02-25T10:17:37.370Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Some info first,</strong></h4><p>So I had to build a custom version of Tensorflow for running our models in Production with CPU. I went and wrote a Bash script to do that, u know the usual thing. After a few months we had to update the version and one of the major issue was that “we didn’t have the VM that was used to build it anymore”.</p><h4>The problem</h4><p>Need simpler solution that reuses the existing bash script but more importantly easier to update/re-build it again in future.</p><h4>Solving it</h4><p>So we were using Docker a lot in our day to day life and seeing how easier it is to update and re-build code with it, I decided to move these custom building scripts to it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c26b2b137c73de9ae1cc935b30e2d68b/href">https://medium.com/media/c26b2b137c73de9ae1cc935b30e2d68b/href</a></iframe><p>Moved most part of the bash script into a Dockerfile, and keep the actual bazel build and package part on a small bash script like this,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/415ec6aa4b8215ba3dd3717eaffc7487/href">https://medium.com/media/415ec6aa4b8215ba3dd3717eaffc7487/href</a></iframe><p>Now all I need it a small run script and docker in any VM to build the package. eg:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fffdbf7861b2463b48143bad983b613d/href">https://medium.com/media/fffdbf7861b2463b48143bad983b613d/href</a></iframe><p>Advantages of dockerzing this,</p><ul><li>Don’t need to keep a machine that was used to build packages, as a container the dockerfile should have everything that’s required.</li><li>Anyone in your team can just checkout this script from your git repos and run it.</li><li>Also might not even need a VM. If you are already running a kubernetes/other docker cluster system, then you can just send a job to it at night from your CI system. For example, in my case I create a Kubernetes job like this</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fe8761a30df29c9adcf5dfe971915d9/href">https://medium.com/media/9fe8761a30df29c9adcf5dfe971915d9/href</a></iframe><ul><li>And voila, we don’t even need a VM anymore</li></ul><h4>Fyi,</h4><p>This is an example to an idea for managing packages in your project, I’m taking tensorflow as an example here. This idea can be applied to any thing, and yes I’m totally planning to convert all of our custom build scripts to this.</p><p>The entire scripts can be found here, <a href="https://github.com/therako/my-docker-examples/tree/master/tensorflow">https://github.com/therako/my-docker-examples/tree/master/tensorflow</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad30f606b0b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Airflow as an External scheduler for distributed systems]]></title>
            <link>https://medium.com/@rako/apache-airflow-as-an-external-scheduler-for-distributed-systems-53b7354d3e48?source=rss-c92b98ca6070------2</link>
            <guid isPermaLink="false">https://medium.com/p/53b7354d3e48</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arunkumar]]></dc:creator>
            <pubDate>Fri, 16 Feb 2018 02:15:00 GMT</pubDate>
            <atom:updated>2018-02-16T02:15:00.977Z</atom:updated>
            <content:encoded><![CDATA[<p>So have you ever needed a reliable External scheduler for your distributed systems? Apache Airflow (by Airbnb) has a good stable scheduler.</p><p>So how can we use Airflow for this purpose, here’s how we did.</p><p>It was easier to create an endpoint than a worker, so we created a endpoint in the service and deployed that.</p><pre>https://doamin.com/api/1.0/purpose?params=value</pre><p>And then just created a simple Airflow DAG to trigger this endpoint on schedule.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21ef17059e93da2dbddab25b0bdebb58/href">https://medium.com/media/21ef17059e93da2dbddab25b0bdebb58/href</a></iframe><p>So what’s advantages of this method,</p><ul><li>Well Airflow has SLA managements, notifications on failures, execution timeouts and retries. All of witch can be very useful here.</li><li>And incase we need to trigger more than one job, just run it thru a loop and create multiple tasks. Airflow will trigger it parallely, and as long as u have a good LB in front the load will be balanced in your distributed servers. So it’s extremely modular and scalable.</li><li>Also the Airflow-UI makes it easier to check logs/response from the endpoint when it fails.</li><li>And also it will give you a nice graph like the below, where we can keep track of the time taken by the job.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQrnlmldb-SR_JnY0Lan5A.png" /><figcaption>Time Taken Graph</figcaption></figure><h3>Takeaway</h3><p>Next time when you need to build a scheduled worker on a distributed system, this could be a good option.</p><p>Also Airflow is easy to setup and get started, take a look at it first!</p><p><a href="https://github.com/apache/incubator-airflow">apache/incubator-airflow</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53b7354d3e48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to handle the feature toggles in Continuous Integration]]></title>
            <link>https://medium.com/@rako/how-to-handle-the-feature-toggles-in-continuous-integration-69610f5dbb18?source=rss-c92b98ca6070------2</link>
            <guid isPermaLink="false">https://medium.com/p/69610f5dbb18</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[testing-strategies]]></category>
            <dc:creator><![CDATA[Arunkumar]]></dc:creator>
            <pubDate>Wed, 19 Aug 2015 09:40:45 GMT</pubDate>
            <atom:updated>2015-08-19T09:40:45.208Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>How to handle the feature toggles in Continuous Integration</strong></p><p>Hi all, this is a journey through how we did Continuous Integration in our project with Feature toggles.</p><p>A few days before the First ever release (when the feature to be released is dev done), there was a discussion on whether to do Feature branches or Feature Toggles. After that discussion we choose to do Feature Toggles as our team was growing and splitting into multiple small feature teams. I am not gonna bore everyone with why and what’s of Feature toggles as it is available all over the internet.</p><p>Back then Our CI server was just doing unit and automation test nothing more. Now there was a bigger issues of How are we going to release a new builds with some bug fixes for the old feature. Of course the answer is just to toggle off the new feature in the production environment config and deploy. But the problem here was, we were also doing a more changes on Navigation and other trivial flows which are common on to all the features.</p><p>The discussion started with the example of Navigation changes,</p><p>To accomplish the new feature the Navigations have to be changed, but it doesn’t make sense to release just the new navigation with old features. So we had them under the feature toggles and now we have two Navigation flows. What do we do we have to test both.</p><p>And then we came up this idea of Branching our CI pipelines into two layers (Development and Release).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*ls_OEEPcSfwajf58.png" /></figure><p>This is behind the idea of achieving Feature Branching Advantages in Feature Toggles. How?</p><p>We split Feature Toggle configs into two environments</p><ul><li>Development config — where always everything is ON.</li><li>Release Config — Only the currently deployed into production Feature are ON. (This is also used in times of release to turn on one particular Feature and run it through the pipelines to get it released)</li></ul><p>Initially It was hard to split the Automation to run based on Feature Toggles, but it became easy as in when we go. Just about adding new tests and also keeping the old tests. In this case we had two different Navigation tests.</p><p>We run every commits through both the pipelines, so in order to call your story complete It has to pass through both versions of feature toggles. And thus we created a way to make every commit deployable, whenever the business wants it to be deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*saNOyTZyvkBpHgHL.png" /></figure><p>This is how our Final setup looks like.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69610f5dbb18" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>