<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sandeep Malik on Medium]]></title>
        <description><![CDATA[Stories by Sandeep Malik on Medium]]></description>
        <link>https://medium.com/@_sandeep_malik?source=rss-2f555a104304------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*dfwt6aN713ngmttu.jpg</url>
            <title>Stories by Sandeep Malik on Medium</title>
            <link>https://medium.com/@_sandeep_malik?source=rss-2f555a104304------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:40:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@_sandeep_malik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring the power of Kotlin extensions]]></title>
            <link>https://medium.com/@_sandeep_malik/exploring-the-power-of-kotlin-extensions-d9a28723b0e1?source=rss-2f555a104304------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9a28723b0e1</guid>
            <category><![CDATA[kotlin]]></category>
            <category><![CDATA[functional-programming]]></category>
            <dc:creator><![CDATA[Sandeep Malik]]></dc:creator>
            <pubDate>Mon, 30 Oct 2017 19:20:51 GMT</pubDate>
            <atom:updated>2017-10-30T19:20:51.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nj3HHd-SarSfeITaGzuv3Q.jpeg" /><figcaption>Photo Credit: <a href="https://pixabay.com/en/tape-extension-hair-extension-1144296/">shc9607</a></figcaption></figure><p>Kotlin is an amazing addition to JVM based family of languages. It is simple, concise, and very easy to understand. One of the most powerful features of Kotlin is <a href="https://kotlinlang.org/docs/reference/extensions.html"><em>extensions</em></a><em>. </em>Extensions allows a developer to <em>‘extend’ </em>a Java class and add more functionality to it. It also makes repetitive code very simple and easy to read.</p><p>Having said that, extensions should not be considered as an alternative for badly designed abstractions. A class needing a lot of extensions is generally a bad smell and in such a scenario it might be more prudent to revisit the design of that class and possibly rewrite it. In other cases, using extensions can turn out to be a very natural and valuable evolution of the existing functionality.</p><p>In this article, we will explore how the extensions can be used to simplify code and make it more readable and enjoyable.</p><h4>Example 1 —Making Guava’s ListenableFuture API more fluent</h4><p>Guava is a powerful library that has been around for sometime. The <a href="https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/ListenableFuture.html">ListenableFuture</a> class allows for wait-free programming using listeners. To make certain operations easier, Guava has <a href="https://google.github.io/guava/releases/19.0/api/docs/com/google/common/util/concurrent/Futures.html">Futures</a> class that allows handling multiple futures very easy.</p><p>Let’s consider a typical series of steps that usually occur in concurrent programming.</p><ul><li>Consider a method, which takes an argument and returns a ListenableFuture</li><li>The caller thread calls the method &#39;n&#39; times with different method parameters</li><li>The caller thread converts the &#39;n&#39; futures in a single ListenableFuture using Futures.allAsList()</li><li>The caller thread then transforms the results to a different form</li><li>The caller thread then finally assigns a listener to handle success or failure scenarios</li></ul><p>The code snippet below shows the above mentioned steps</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a875aec376af520351746eb4cf81efd/href">https://medium.com/media/5a875aec376af520351746eb4cf81efd/href</a></iframe><p>In the above code there are multiple places where static methods of Futures class are called. Some of these could be good candidates for extension methods.</p><p>Let’s define a few extension methods reduce(), transform(), and done() as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fe63b9b1f27b87065dcb24c5eb190da/href">https://medium.com/media/9fe63b9b1f27b87065dcb24c5eb190da/href</a></iframe><p>With these extensions in place, the code can be rewritten as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01488c74cca81ef97267997ab9c31afa/href">https://medium.com/media/01488c74cca81ef97267997ab9c31afa/href</a></iframe><p>This code looks much more readable and cleaner as compared to initial version. Also notice how extensions allow us to write code in a fluent manner with the noise of declaring redundant extra variables significantly reduced.</p><p>Let’s take another example of Java <a href="https://docs.oracle.com/javase/8/docs/api/java/time/LocalDate.html">Date</a> class. Java 1.8 has added a lot of new functionality in the java.time package. However there’s no easy way to iterate over a range of dates. Let’s try to solve that with the use of extensions.</p><h4>Example 2: Applying extensions to LocalDate class</h4><p>Suppose we have the following use case</p><ul><li>A method process() takes two parameters; start date and end date.</li><li>The method then iterates over the date range and for each date does some processing and produces the results.</li><li>Finally, a map of date and resultant value is returned</li></ul><p>The code below shows the above steps</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e0f63fc73ca0b762c33ac752eed977a/href">https://medium.com/media/0e0f63fc73ca0b762c33ac752eed977a/href</a></iframe><p>Now, let’s define the rangeTo() operator for LocalDate class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c13c4e92cfe8a807b1d67526eedeca46/href">https://medium.com/media/c13c4e92cfe8a807b1d67526eedeca46/href</a></iframe><p>With this extension, it is very easy to re-write the above code as follows</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfae4999dc2f5dd418c3a5e030570da6/href">https://medium.com/media/bfae4999dc2f5dd418c3a5e030570da6/href</a></iframe><p>As we can see the code is much clearer to read now. You can also use extensions for other use cases like AtomicInteger. See the code sample below</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccc3bd08088b934af9edec0662480451/href">https://medium.com/media/ccc3bd08088b934af9edec0662480451/href</a></iframe><h4>Conclusion</h4><p>In this article, we explored the power of Kotlin extensions. If used correctly, extensions can be a very powerful way to express the intent of code clearly. However, extensions do have some limitations. Extensions are resolved statically. And if an extension conflicts with a member method then extension is ignored and the member method is invoked. Nonetheless, extensions can prove to be a very convenient way to add more functionality to existing classes and libraries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9a28723b0e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoid pitfalls in scaling your Cassandra cluster: lessons and remedies]]></title>
            <link>https://medium.com/walmartglobaltech/avoid-pitfalls-in-scaling-your-cassandra-cluster-lessons-and-remedies-a71ca01f8c04?source=rss-2f555a104304------2</link>
            <guid isPermaLink="false">https://medium.com/p/a71ca01f8c04</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[bigdatascale]]></category>
            <dc:creator><![CDATA[Sandeep Malik]]></dc:creator>
            <pubDate>Wed, 25 Oct 2017 18:28:19 GMT</pubDate>
            <atom:updated>2018-02-23T15:21:40.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mksBdp0E4y28HApgtkcnvg.jpeg" /><figcaption>Photo credit: <a href="https://pixabay.com/en/blur-close-up-cobweb-dew-halloween-1868997/">Pexels</a></figcaption></figure><h4>Introduction</h4><p>Several teams at Walmart use Cassandra heavily as one of their primary data sources. Many of them rely on the high availability and tunable consistency guarantees provided by Cassandra besides the blazing performance. We have deployed several Cassandra based applications in production with reads and writes staying consistently at 10’s or even 100’s of thousand ops per second.</p><p>Reaching that state has not been easy. We had to ‘un-learn’ a lot of the tricks of RDBMS world, create schema that looked ‘weird’ but turned out to be more performant, and consider various strategies to ‘delete’ the data. In this article, we will discuss some of the pitfalls that we observed while scaling our Cassandra clusters and how we avoided them.</p><h4>Pitfall 1: Get your partitions right</h4><p>After delivering a couple of successful projects based on Cassandra, one pattern always stood out</p><blockquote>Of all the tunings, strategies, tricks that can be applied to Cassandra to gain performance, designing the most optimal schema had the biggest impact in terms of latency, stability, and availability of the cluster.</blockquote><p>Uneven partitions is one of the biggest reasons for non-deterministic reads/writes performance and query timeouts. It also causes uneven pressure on compaction cycles as some partitions may be big and may take long time to compact while smaller partitions may compact quickly.</p><p>In one of our use-cases we had a need to maintain parent-child relationship. It seemed reasonable to maintain parent key as the partition key and child key as the clustering key so that all children could co-locate with parent. One of the design concern was that number of children varied heavily; some parents had one child, while others could have one million! This resulted in partition hot spotting, lots of dropped mutations, and in general excessive pressure on some nodes of the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*upWWOZoMQ4jQAa4DA5RDHQ.png" /><figcaption>Fig 1 — Uneven partitions in a parent child relation</figcaption></figure><p>To solve this issue, we applied the concept of shard<em>ing</em>. We sharded the parent and introduced a shard key as part of the partition key. The shard size was kept at a fixed size of <em>1000</em> so that the overall partition size could be kept under <em>10 MB</em>. (Each row had ~ <em>10 KB </em>of data).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_sgCKZH8VILAkYckFbFRZQ.png" /><figcaption>Fig2 — Almost even sized partitions</figcaption></figure><p>This resulted in almost equi-sized partitions and load on the cluster got distributed as well. As a bonus, we got data pagination for free during read queries! Each shard nicely mapped to a data page, and the nth page could now be fetched with a simple slice query as follows</p><pre>SELECT * FROM table WHERE parent_key = &#39;p1&#39; AND shard_key = n</pre><h4>Pitfall 2: Ensure separation of concenrs</h4><p>Cassandra has limited support for secondary indexes. In one of our projects we needed to create a lot of secondary indexes and decided to use Solr with Cassandra to offload the index management while maintaining the data locality.</p><p>As the data size grew, the cluster started showing stability concerns. We started seeing a lot of timeouts with increased load on the cluster. On digging deeper, we discovered that one of the major culprit was that for every mutation on a Cassandra row, Solr was fetching the entire partition in memory for indexing. Since, each partition carried<em> MB’s </em>of data, this resulted in a lot of data fetch in memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yy2e7sRavT6rrb-4_fZKWQ.png" /><figcaption>Fig 3 — Cassandra + Solr nodes ring — bad performance for wide rows</figcaption></figure><p>We segregated Cassandra and Solr clusters, and started writing to Solr through a Kafka topic. While the latencies of pure Cassandra cluster improved significantly (almost zero timeouts), the Cassandra + Solr cluster continued to behave badly. We finally replaced Cassandra + Solr combination with <a href="https://lucene.apache.org/solr/guide/6_6/solrcloud.html">SolrCloud</a> and the performance has been excellent after the switch with throughput improving significantly!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vKqPwEjfZhnzqee2YGpXJg.png" /><figcaption>Fig 4 — Separate Cassandra and Solr rings — better performance for wide rows</figcaption></figure><h4>Pitfall 3: Avoid full reads/writes</h4><p>Cassandra resolves <em>mutation conflicts </em>by using the “last write wins” strategy. If the same data is written back to a column at a later time, it is considered a <em>mutation </em><strong>even if the data has not changed</strong><em>. </em>Mutations incur a heavy cost on reads as reads have to consult multiple <em>SSTables</em> to ensure that last update is always returned. Mutations are compacted / discarded during the compaction process, however that causes significant load on the cluster if the real content is not changing.</p><p>Almost all Java Object Mapper frameworks for Cassandra provide proxy based mechanism for tracking changes (i.e. which properties/columns need to be written back), which should most definitely be used. However, the read semantics are a bit slippery. Typically an <strong>entire</strong> row is read every time a read query is fired (<em>see code below</em>), which increases the cost of reads. A better approach would be to read selectively wherever possible.</p><pre>//writes: <br>connection.proxyForUpdate(MyBean.class) // the proxy would keep track of which fields were changed<br>connection.save(myBeanObject) // only changed fields would be written</pre><pre>// reads:<br>MyBean myBeanObject = connection.find(MyBean.class, myBeanId) <br>/* <br>this would typically read all the columns, which is cheaper for row-oriented data bases, but costlier for Cassandra<br>*/</pre><h4>Pitfall 4: Beware of collections</h4><p>Cassandra supports Set, Map, and List collections as column types. A general mistake is to read the collection in the application, modify it, and then write it back as shown in the following code</p><pre>List&lt;Object&gt; c = cassandraDO.getList();<br>c.delete(obj1);<br>c.add(obj2);<br>cassandraDO.setList(c); // BAD: re-writes the entire list.<br>cassandraDO.save();</pre><p>This results in re-writing the entire collection again, which causes a lot of tombstones and affects performance. Furthermore, Sets should be preferred to Lists as Sets (and Maps) avoid <em>read-before-write </em>pattern for updates and deletes. An update or delete in a List will potentially read the entire list in memory (server-side) again since there’s no ‘<em>data uniqueness’.</em></p><p>Another issue with collections is that collections are read in entirety and there is no support for selective reads. For example, if a map collection has 10,000 entries and a query wants to read values for 5 keys, all the 10,000 entries will be read by Cassandra. This is unnecessary and redundant.</p><h4>Pitfall 5: Driver tunings</h4><p>Driver tuning has a lot of impact on how the queries perform. We found an excellent article by DataStax <a href="http://docs.datastax.com/en/developer/java-driver/2.1/manual/pooling/">here</a> that describes a lot of settings that resulted in better overall performance and lesser time outs. Since we were only using local data center, we set the remote host connections to 0. Also, settings <em>shuffleAmongReplica </em>to true gave us better load distribution (the downside being more local read-repairs happening on server side)</p><h4>Summary</h4><p>Cassandra is a great system if used properly. Some of the key factors affecting query performance are proper partitioning of data, avoiding hot spotting, avoiding full reads/writes, and tuning the driver properly for the expected query load. We learnt a great deal by changing our <em>RDBMS oriented mindset </em>and we’re happily using Cassandra as one of the main data sources for last couple of years at Walmart.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a71ca01f8c04" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/avoid-pitfalls-in-scaling-your-cassandra-cluster-lessons-and-remedies-a71ca01f8c04">Avoid pitfalls in scaling your Cassandra cluster: lessons and remedies</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Presto and Zeppelin for fast data analytics and visualization]]></title>
            <link>https://medium.com/walmartglobaltech/exploring-presto-and-zeppelin-for-fast-data-analytics-and-visualization-9cb4dca91c3d?source=rss-2f555a104304------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cb4dca91c3d</guid>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Sandeep Malik]]></dc:creator>
            <pubDate>Wed, 02 Aug 2017 17:50:49 GMT</pubDate>
            <atom:updated>2019-02-01T16:47:01.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*BIAJryOgIbbUmYix6VOSGQ.jpeg" /><figcaption>Photo credit <a href="https://pixabay.com/en/abstract-blur-britain-british-1239439">Meditations</a></figcaption></figure><h4>Introduction</h4><p>Big data world is messy! The choice of frameworks, platforms, and data sources is simply mind-boggling. Several teams at Walmart use a mix of various stacks and tools to make sense of the data. While having choices is great, it does become a bit overwhelming when someone wants to access data in a coherent and uniform fashion. Each system exposes a different set of APIs, a different language to write queries in, and potentially a different way of analyzing the data at large scale.</p><p>It became clear to us early on that we need to invest on a tech stack, which can provide access to TBs of structured and semi-structured data for quick analysis and still be manageable with minimal overhead. It was also desirable that the query language be as close to SQL as possible so that it could be used with relative ease.</p><p>Some of the systems we looked at were <a href="https://impala.incubator.apache.org/">Apache Impala</a>, <a href="http://spark.apache.org/">Apache Spark</a>, and <a href="https://prestosql.io/">Presto</a>. Each of these systems are battle tested and support a large number of production systems today. We started gravitating towards Presto for the following reasons:</p><ul><li>Support for a large number of data sources (Hive, Cassandra, Kafka, Elastic Search)</li><li>Extremely rich SQL dialect</li><li>Ability to cross join data from multiple disparate sources</li><li>No external dependency on systems like ZooKeeper, etc</li><li>Great performance</li></ul><p>In this article, we will explore our Presto set up and examine its performance for some common use cases. We will also focus on <a href="https://zeppelin.apache.org/">Apache Zeppelin</a>, which is an excellent open source data visualization tool for interactive queries and helps team collaborate in real time.</p><h4>Presto: A distributed SQL engine</h4><p><a href="https://prestosql.io/">Presto</a> is an excellent distributed SQL engine that is optimized for running queries over huge datasets across multiple data sources. It supports distributed joins across multiple data sources like Hive, Kafka, Elastic Search, Cassandra, etc, thereby allowing a uniform access model for analytics.</p><p>Setting up the Presto cluster is easy. We set up a 8 node (each having 8 cores and 16G RAM) presto cluster in literally under 20 mins. Presto provides live query plans, which is a big plus. It helps us understand how to tune the queries for more performance. Presto also provides a JDBC driver, so accessing it from Java applications is very convenient as well.</p><h4>Configuring Presto</h4><p>Presto requires minimal configuration. We need to provide node.properties file under etc/, which requires bare minimum properties, e.g.</p><pre>node.environment=prod<br>node.id=node1<br>node.data-dir=/var/presto/data</pre><p>Each node also requires a config.properties.</p><pre>coordinator=false<br>http-server.http.port=8080<br>query.max-memory=50GB<br>query.max-memory-per-node=8GB<br>discovery-server.enabled=true<br>discovery.uri=http://${coordinator_ip}:8080</pre><p>One of the nodes has to be made coordinator and needs some more properties</p><pre>coordinator=true<br>node-scheduler.include-coordinator=false<br>http-server.http.port=8080<br>discovery-server.enabled=true<br>discovery.uri=http://localhost:8080</pre><p>One issue we ran into while configuring Presto was regarding the discovery.url. Presto uses its own discovery mechanism to identify the cluster workers. <strong><em>We observed that in the coordinator’s config.properties the discovery.url should be pointing to localhost while in all other nodes it should be point to coordinator’s IP</em></strong></p><p>Finally, we need to add catalogs (connectors for different data sources) under etc/catalog directory. Example below shows how to add a Cassandra connector with minimal configuration</p><pre>connector.name=cassandra<br>cassandra.contact-points=&lt;comma separated IPs&gt;<br>cassandra.consistency-level=LOCAL_ONE<br>cassandra.username=readonly_u<br>cassandra.password=readonly_p<br>cassandra.consistency-level=LOCAL_ONE<br>cassandra.load-policy.use-dc-aware=true<br>cassandra.load-policy.dc-aware.local-dc=DC1<br>cassandra.load-policy.use-token-aware=true</pre><p>As mentioned before, Presto can connect to a lot of data sources including Cassandra, Hive, and Kafka among many others. A full list of connectors can be found <a href="https://prestosql.io/docs/current/connector.html">here</a></p><p>Once the cluster is set up, the console can be accessed at the discovery.url</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJGBmgnZ01xFJtbMlfhhgA.png" /><figcaption>Presto cluster (7 workers + 1 coordinator, 8 cores/16G)</figcaption></figure><h4>Measuring Presto Performance</h4><p>The first task that we set out to do was to measure query performance. We had a table handy with close to 260M rows. Each row had roughly 1 KB data. The schema of the table is as follows:</p><pre>CREATE TABLE event_lookup (<br> id text,<br> bucket_id timestamp,<br> payload text,<br> PRIMARY KEY (( id ))<br>)</pre><h4>Use case 1: count(*)</h4><p>A simple query was fired on Cassandra which returned the count of total partitions in Cassandra.</p><pre>SELECT COUNT(*) FROM event_lookup;</pre><p>This resulted in a full table scan, by Presto with an impressive rate of ~ 418K rows / second! The inflow rate was ~ 400KB/sec. The screen shot below shows the rates along with other metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c9JezuhzJSNho48voQ-zrg.png" /><figcaption>Total counts query performance over 260M rows</figcaption></figure><p>During the query execution, there were occasional blips where read rate reached almost <strong>1 million rows / second</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4KKs4wu3XFfrOOFJ34jLg.png" /><figcaption>Max peak performance at 1.06M rows/sec</figcaption></figure><p>Worker nodes also had almost equal distribution of load. From the screen shot below, each worker was processing on an average 51K rows / second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZ-XipGaCivC9UGI_1mRHg.png" /><figcaption>Worker nodes load distribution (Host IPs hidden for privacy)</figcaption></figure><h4>Use case 2: group by</h4><p>We wanted to execute some thing more complicated than a count(*) query to examine performance better. We used a group by query as follows:</p><pre>SELECT bucket_id, count(*) FROM event_lookup GROUP BY bucket_id ORDER BY count(*) LIMIT 10;</pre><p>This time around, the results were even more impressive. The average query rate reached an impressive of 560K rows / sec!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1x1lzEtI5SUkUjn_hjCWw.png" /><figcaption>Presto cluster dashboard view</figcaption></figure><p>The worker node distribution was a little skewed, possibly because of the way “bucket_id” field was partitioned among the Cassandra token ranges. What was stunning was that one of worker nodes reached a maximum of ~ 100K rows/sec TPS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkvSmyzBQ0f7xqKDkL5LBg.png" /><figcaption>worker nodes load distribution for group by query</figcaption></figure><p>Having obtained satisfactory results, we moved on the other end of spectrum, the visualization!</p><h4>The other end of the spectrum: Apache Zeppelin</h4><p>Data analytics is not appealing if not complemented by a strong visualization. Visualization goes a long way in quickly deriving data patterns and correlations. While there are tools and libraries that provide excellent visualization support, they require a fairly good understanding of JavaScript and CSS, which is not something that data analysts are expected to know. <a href="https://zeppelin.apache.org/">Apache Zeppelin</a> aims to fill that gap precisely. It provides analysts an extremely convenient way to create interactive web notebooks to write and execute the queries and visualize the results immediately. The web notebooks can be shared and collaborated in real time for valuable feedback. The queries can be scheduled so that they can be executed periodically. Like Presto, Zeppelin can connect to many data sources as well. In this article we will focus on JDBC connector for accessing Presto.</p><h4>Configuring JDBC connector for Presto</h4><p>Adding a JDBC connector in Zeppelin is very easy. We just need to provide url, user, password, and driver properties. Below screen shot shows how to configure a Presto connection</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4Jx_7-rs_VBdocjP_3Rrg.png" /><figcaption>Adding Presto connector / interpreter to Zeppelin</figcaption></figure><p>Once the connector is added, we can create a notebook and add a new note. Inside the new note, the query can be executed as follows and the results can be visualized immediately in a variety of charts.</p><pre>%jdbc SELECT bucket_id, COUNT(*) FROM event_buckets GROUP BY bucket_id</pre><p><em>Note: if Presto is not configured as default interpreter, then you need to provide the name of the interpreter in the query</em></p><pre>%jdbc(presto) SELECT bucket_id, COUNT(*) FROM event_buckets GROUP BY bucket_id</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIHc07iEKcEtXCp7xwP5HA.png" /><figcaption>Zeppelin visualization for Presto query</figcaption></figure><h4>Building an interactive SQL executor</h4><p>Zeppelin makes it very easy to build interactive form based UIs. Let’s try that by building a SQL input form. A general purpose SQL query executor requires following inputs:</p><ul><li>Name of the fields/columns to be retrieved</li><li>Table name (FROM clause)</li><li>Filter expression or aggregate expression (WHERE or GROUP BY clause)</li><li>Sort field (ORDER BY clause)</li><li>pagination (LIMIT and OFFSET)</li></ul><p>We wrote a simple template zeppelin query</p><pre>%jdbc<br>SELECT ${checkbox:Select Fields=Field1, Field1 | Field2 |  Field3 | Field4} ${Free Form Fields = }<br>FROM <br>${Select Table=keyspace1.table1, keyspace1.table1 | keyspace2.table2 | keyspace3.table3} <br>${PREDICATE CLAUSE e.g. WHERE, GROUP BY = } <br>${ORDER BY CLAUSE = }<br>LIMIT ${limit = 10}</pre><p>which resulted in the following UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmui0-a9Z7F2CMVgVIeklA.png" /><figcaption>Zeppelin general SQL query form (with query)</figcaption></figure><p>If we hide the query part, the UI becomes extremely simple</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mzA40YED-K3GmquWp1UWJQ.png" /><figcaption>Zeppelin general SQL query form (query hidden)</figcaption></figure><p>Now that’s the power of Zeppelin! In literally under 10 minutes, we are able to create a UI, where a user can</p><ul><li>Select fields (by selecting checkboxes, or entering in the ‘Free Form Fields’ like Count(*)</li><li>Provide any ORDER BY clause</li><li>Provide any WHERE or GROUP BY clause</li><li>Choose any table to connect to</li></ul><p>Using Zeppelin’s powerful display system (AngularJS) and variable binding mechanism, it’s very easy to create a chain of paragraphs which can execute in succession or in any arbitrary custom defined manner.</p><h4>Summary</h4><p>In this blog we explored how we can leverage Presto to run SQL queries over data sources. We also explored how we can wire Presto with Zeppelin to create compelling visualizations and analyze patterns quickly. This has resulted in quick analysis and collaboration between different teams. In the next article, we will be exploring how the Presto cluster is allowing us to join data across Cassandra, Kafka, Hive, etc for instant analysis on fast moving data. Stay tuned for more updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cb4dca91c3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/exploring-presto-and-zeppelin-for-fast-data-analytics-and-visualization-9cb4dca91c3d">Exploring Presto and Zeppelin for fast data analytics and visualization</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Approach to Designing a Distributed, Fault-Tolerant, Horizontally Scalable Event Scheduler]]></title>
            <link>https://medium.com/walmartglobaltech/an-approach-to-designing-distributed-fault-tolerant-horizontally-scalable-event-scheduler-278c9c380637?source=rss-2f555a104304------2</link>
            <guid isPermaLink="false">https://medium.com/p/278c9c380637</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Sandeep Malik]]></dc:creator>
            <pubDate>Wed, 23 Nov 2016 20:10:37 GMT</pubDate>
            <atom:updated>2018-08-06T21:46:22.850Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JI5_Fm7d5jEvUcVdNZZxlQ.jpeg" /><figcaption>Photo Credit <a href="https://pixabay.com/en/pocket-watch-time-of-sand-1637396/">annca</a></figcaption></figure><h4><strong>Introduction</strong></h4><p>Designing a time based event scheduler is always an interesting problem. Doing that at scale makes it even more challenging. First of all, let’s define what we mean by time based event scheduler?</p><p>A time base event scheduler is a system that can be used by a service to schedule a request that needs to be processed in<em> future</em>. The service registers an <em>event</em> in the scheduler and suspends the processing of current request.</p><p>When the stipulated time arrives, the requesting service is notified by the scheduler and former can resume processing of the suspended request. There are many scenarios in which such a system can be useful. Some of the use cases, which by no means are exhaustive, are listed below</p><h4>Timing out async requests</h4><p>As more and more systems move towards messaging based architectures like <a href="https://medium.com/walmartlabs/tech-transformation-real-time-messaging-at-walmart-8787f5ab19e8#.wx4sbxuse">Kafka</a>, JMS, etc., one of the most common scenarios is using messaging tier as asynchronous request/response bus. One system produces a “<em>message request</em>” to another system and when the response is returned by the latter, it proceeds with the rest of the processing. In some of the cases, it’s important for the caller to have ‘<strong>eventual response</strong>’. It is possible that the other system has a lag, or is down, or simply misses to return back a response. In a distributed environment with a stateless micro-services paradigm, it is difficult to keep track of such <em>lost</em> requests. One solution could be to schedule a ‘time-out’ event, which can be triggered at stipulated SLA of the request.</p><p>When the event is triggered the current state of the request can be checked and if the response is missing then the request can be marked timed out.</p><h4>System retries</h4><p>Even in case of systems that interact over a synchronous channel like HTTP, it is possible that the server is down and request can not be served. A client can choose to retry, however, if the retry intervals are very short, it is unlikely to have any net positive effect. For example, if it’s a system wide outage it is likely to remain so for more than a few seconds. In such cases, it might be desirable to do longer, exponential retries for which the request data may not be kept in memory.</p><p>Under these circumstances, the payload for retry can be sent to an event scheduler and can be retried later depending on the client’s retry policy.</p><h4>Reconciliation jobs</h4><p>Sometimes in distributed systems based on batching, the data streams are split and processed by multiple clusters as micro batches. In case of failures, it may be desirable to reconcile responses so that a coherent response can be returned to client for the initial batch.</p><p>In such cases, a reconciliation event can be scheduled at the beginning of the start of the batch processing so that data can be reconciled at the end of the batch processing or the SLA of processing time.</p><h4>Price related triggers</h4><p>In an eCommerce world, price changes are generally made to keep the product at competent price value. Triggering such price changes could be a huge manual effort, and error-prone, and may not happen precisely at the desired time. For example, even a minute lag of price change during the peak holiday hours like Black Friday can potentially impact the annual sales.</p><p>In such cases, the price changes can be set up as ‘promotions’ ahead of time, which can be triggered by the event scheduler at the required time thereby saving a manual price-synchronization effort.</p><p>All of the above mentioned issues were faced by <a href="https://www.walmart.com/">Walmart</a> at one point or another and it became clear for us that we need to invest into a system that can keep track of <em>suspended requests</em>. Since the scale at which Walmart operates is huge, the scheduler had to be fault tolerant, and horizontally scalable.</p><h4>Existing options</h4><p>We started looking at various existing open source solutions like Quartz. While Quartz is a proven solution for time based scheduling with rich functionality, our problem statement was a bit different</p><ul><li>Many of our applications are using <a href="https://www.youtube.com/watch?v=KMHua4LUvQ0">Cassandra</a> as the NoSql data store. Over time we have grown considerable expertise in Cassandra and its nuances. Quartz does not seem to have a JobStore for Cassandra (or at least we couldn’t find one)</li><li>We wanted to have tunable consistency for events. For some events like system retries, etc, it was fine to write them to one node (LOCAL_ONE) but for others like promotion triggers, it was required to use EACH_QUORUM so that no event was lost.</li><li>Scaling out the cluster was harder. We had a requirement to scale to millions of events per hour.</li></ul><p>We also looked at Hazelcast-based JobStore implementation of Quartz but the prolonged garbage collection cycles was a risk for in memory data there by resulting in misfires.</p><h4>Introducing BigBen</h4><p>We set out to implement our own solution, which is called BigBen, inspired by the very famous <a href="https://en.wikipedia.org/wiki/Big_Ben">London Clock Tower</a>. BigBen offers the following capabilities</p><ul><li>Master slave design, so that master can decide the best strategy for distributing load across the cluster</li><li>Extremely fault tolerant design so that if a given master goes down another can be selected almost instantaneously</li><li>Uniform partitioning of events data in data base so that there are no hot pockets in terms of fat partitions in Cassandra</li><li>Uniform execution of events on the cluster so that all nodes share <em>almost</em> equal load</li><li>Maintain checkpoints so as to keep track of events that were not triggered or failed execution (mis-fires)</li><li>Integrate with HTTP and Kafka as <em>default </em>channels for events</li><li>Multi-tenancy. Each events belongs to one tenant, and each tenant can define its policy of how to store (consistency guarantees) and/or process the event.</li></ul><h4>Design and Architecture</h4><p>BigBen is based on a ‘micro-batching’ architecture. The events are bucketed together in a window and these buckets are stored as partitions in data base.</p><p>When the cluster comes up, a master is selected, which starts scanning buckets at a defined granularity. The minimum granularity supported is one minute, which means the master will scan for the events every minute.</p><h4>Event ingestion</h4><p>Since it is unknown how many events will be scheduled every minute, partitioning the data based on bucket width (in this case one minute), may result in an uneven load on the data base. The events are therefore shard<em>ed</em> and only 1000 (configurable value) events are stored in 1 shard.</p><p>For example, say at 10:00 AM, 2030 events were received, then those events will go to three shards. Shard 1 will contain 1000 events, shard 2 will contain another 1000, and shard 3 will contain 30 events. This way the system guarantees that there are no more than 1000 events per shard.</p><p>To know which shard a given event will be assigned to, depends on the running total of events for that bucket. It is then very simple to calculate the shard index for that event using the formula below:</p><blockquote>shard_index = (total_events / shard_size)</blockquote><p>for example, when next event is received after 2030 events, it will go to</p><blockquote>2030/1000 = 2nd index (or 3rd shard)</blockquote><p>For fast lookup, this required maintaining a running total of event counts per bucket. BigBen uses embedded Hazelcast grid to maintain those counts in a distributed map, though other solutions like memcache, <a href="https://t.co/CSDgqm6MP9">Couchbase</a>, etc. which provide support for atomic counts, can be used as well. The map data is synced with Cassandra every few seconds.</p><p>The diagram below captures the event receive flow. The events can be received over HTTP or Kafka. Invalid events are rejected and returned immediately (shown by <em>red dotted line</em>). For others, the corresponding buckets are calculated (say the event is scheduled for 10:21:55 AM, then the bucket is 10:21:00 AM), the counts are incremented, and shard index is calculated. The event is then stored in the corresponding partition which is a combination of bucket and shard index (e.g. 10:21:00/1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LmYyR8GT9TjPmrRorOBzLg.png" /><figcaption>Fig 1 — Event Receive Flow</figcaption></figure><h4>Event Processing</h4><p>For the processing part, a master is elected among the cluster members. Zookeeper could be used for leader/master election, but since BigBen already uses Hazelcast, we used the distributed lock feature to implement a <em>Cluster Singleton. </em>The master then schedules the next bucket and reads the event counts. Knowing the event count and shard size, it can calculate very easily how many shards are in total. The master then creates pairs of (bucket, shard_index) and divides them <strong>equally </strong>among the cluster members, including itself. In case of unequal division, the master tries to take the minimum load on itself. For example, for 10:21 AM bucket, and say 6000 events, and 4 node cluster, the division would look like:</p><blockquote>distribution =&gt; (bucket, array of shard indexes, node IP) =&gt; (10:21, [1,5], node1), (10:21, [2,6], node2), (10:21, [3], node3), (10:21, [4], node4)</blockquote><p>The master also checks if any previous buckets have failed and schedules those as well. Upon completion of the bucket, the bucket is marked processed and the status is updated in the checkpoint. The diagram below shows the processing stage (<em>schedule scanner is the master</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*euaHLOnw6G96SigfXxWhtA.png" /><figcaption>Fig 2 — Event Processing Flow</figcaption></figure><p>The various components in the processing workflow are:</p><ul><li>Bucket Manager: maintains the snapshots of loaded bucket in memory</li><li>Bucket Loader: loads previous buckets in memory in a rate-limited/throttled way</li><li>Checkpoint helper: syncs the checkpoint every few seconds</li><li>Status syncer: syncs the status of bucket, once done</li></ul><h4>Fault Tolerance Design</h4><p>As any other distributed system, there can be multiple failure points. Some of them are:</p><ul><li>Master becomes unavailable</li><li>Events processing fails</li><li>Data base is down</li></ul><p>BigBen uses different techniques to handle failures. For the fault tolerance of master, we use a distributed lock in Hazelcast. Note that this lock is <em>never released </em>in the life time of the node. Only if the node goes down, the other member will try to acquire the lock. This provides for a very robust fault tolerance and we validated this during our Chaos Monkey testing. Below diagram shows the master fail over scenario</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRjTGz1jrWpYl2tYznV8jg.png" /><figcaption>Fig 3 — Master (Cluster Singleton) distributing tasks among slaves</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SHiYcpNCJcRsaAbFpXb3rw.png" /><figcaption>Fig 4 — Master down, new slave promoted to master, and starts distributing tasks</figcaption></figure><p>In cases where event processing is failed or data base is down, the checkpoint is marked with those failed buckets. On every new bucket scan, the master scans the checkpoint as well to see if any previous buckets are in failed state. If found, then those buckets are also scheduled.</p><p>BigBen uses a configurable ‘<em>look back</em>’ parameter regarding how far in the past it has to go to keep trying for failed buckets/events. By default, this value is 1 day.</p><h4>Performance benchmarking</h4><p>We bench marked the cluster’s performance with the following parameters:</p><ul><li>4 machines, Cent OS 6+, 8 Cores, 16 GB RAM, min Heap 2 GB, max Heap 8 GB</li><li>Event payload size: 500 bytes</li><li>Cassandra 6+6 bare metal cluster</li><li>Write Consistency: LOCAL QUORUM</li><li>Read Consistency: LOCAL QUORUM</li><li>Ingestion Rate: 1 Million events ingested in 52 seconds. =&gt; <strong>70 Million Events per Hour</strong></li><li>Processing Rate: 1 Million Events in 60 seconds =&gt; <strong>60 Million Events per Hour</strong></li></ul><h4>Next Steps</h4><p>I hope you found this post informative. K̶e̶e̶p̶ ̶a̶n̶ ̶e̶y̶e̶ ̶o̶u̶t̶,̶ ̶a̶s̶ ̶w̶e̶ ̶w̶i̶l̶l̶ ̶b̶e̶ ̶o̶p̶e̶n̶ s̶o̶u̶r̶c̶i̶n̶g̶ ̶B̶i̶g̶B̶e̶n̶ ̶v̶e̶r̶y̶ ̶s̶o̶o̶n̶. We have open sourced <a href="https://github.com/walmartlabs/bigben">BigBen</a>. Let us know if any questions. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=278c9c380637" width="1" height="1" alt=""><hr><p><a href="https://medium.com/walmartglobaltech/an-approach-to-designing-distributed-fault-tolerant-horizontally-scalable-event-scheduler-278c9c380637">An Approach to Designing a Distributed, Fault-Tolerant, Horizontally Scalable Event Scheduler</a> was originally published in <a href="https://medium.com/walmartglobaltech">Walmart Global Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>