<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prasanna Ranganathan on Medium]]></title>
        <description><![CDATA[Stories by Prasanna Ranganathan on Medium]]></description>
        <link>https://medium.com/@rprasanna?source=rss-935fdafc2795------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_IaNyNXYvwPTUoVfmdYQ8A.jpeg</url>
            <title>Stories by Prasanna Ranganathan on Medium</title>
            <link>https://medium.com/@rprasanna?source=rss-935fdafc2795------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:42:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rprasanna/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sherlock: Near Real Time Search Indexing]]></title>
            <link>https://blog.flipkart.tech/sherlock-near-real-time-search-indexing-95519783859d?source=rss-935fdafc2795------2</link>
            <guid isPermaLink="false">https://medium.com/p/95519783859d</guid>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Prasanna Ranganathan]]></dc:creator>
            <pubDate>Wed, 27 Dec 2017 06:12:47 GMT</pubDate>
            <atom:updated>2018-01-04T06:55:22.237Z</atom:updated>
            <content:encoded><![CDATA[<p>At Flipkart our mission is to transform commerce in India through technology. Our engineers have worked tirelessly over the years in solving complex technical challenges and building systems to power India’s best shopping experience. Building a world-class e-commerce discovery experience through search &amp; browse is a monumental technical challenge, one which is tackled by the ‘Sherlock’ team at Flipkart.</p><p>The dynamic nature of e-commerce poses unique challenges — stock units, availability, pricing, catalog data, etc. can all change at a very high rate and the system needs to keep up with the latest data lest the customer be disappointed. There are few things worse from a customer experience standpoint than showing the desired product in search results only for the product to be out of stock when attempting to complete the purchase. The Sherlock team developed an innovative solution (NRT data store) to deliver near real-time search results and presented it at Lucene Revolution, 2016 at Boston (<a href="https://www.slideshare.net/lucidworks/near-real-time-indexing-presented-by-umesh-prasad-thejus-v-m-flipkart">Slides</a> and <a href="https://www.youtube.com/watch?v=05rX0mJ2N4U">Video</a>). While this talk covered the search results serving layer in-depth, there is a complementary system that handles the ingestion and processing of data for consumption by the serving layer.</p><p>Processing all the data updates (10K per sec on average) on a continuous basis and delivering them to the serving layer with minimal latency (less than 30 secs) is a major undertaking and that is the responsibility of the Sherlock ingestion layer. In this post we provide a detailed overview of the systems and technology used to build and scale the ingestion stack at Flipkart.</p><h3>Duality of Sherlock Ingestion</h3><p>Flipkart is an ecommerce marketplace where multiple sellers participate and sell <em>‘products’</em>. A single product can have multiple sellers offering it for purchase in the marketplace. The entity that captures the seller-specific details &amp; attributes of a product is referred to as a <em>‘listing’</em>. Search broadly deals with these two types of entities — products and listings — each having its own specific set of associated attributes. These product and listing attributes comprise of two types:</p><ol><li>Non-dynamic, mostly-static attributes. Examples are product title, description, …</li><li>Highly dynamic attributes. Examples are listing price, availability, …</li></ol><p>The standard approach to building a search solution is to <a href="https://nlp.stanford.edu/IR-book/">use an inverted index</a>. In our case, the dynamic attributes pose a unique challenge to building an inverted index based solution for search. Their high rate of change makes it hard to sustain a high update rate with a typical inverted index and it also has a higher propagation latency (beyond the scope of this article to detail the challenges with high update rate on an inverted index). Therefore only the mostly-static attributes are processed into segments that comprise the inverted index. The ingestion pipeline receives the updates for dynamic attributes, executes the relevant business logic and directly <a href="https://www.slideshare.net/lucidworks/near-real-time-indexing-presented-by-umesh-prasad-thejus-v-m-flipkart/30?src=clipshare">streams the results to the NRT store in the serving layer</a>. Thus, in effect, the ingestion layer ends up having two classes of service:</p><ol><li>Data delivered to the inverted index (Higher propagation SLA)</li><li>Data streamed to the NRT data store (Lower propagation SLA)</li></ol><p>This dual nature is reflected in the serving layer also as the data present in the inverted index and the NRT store together is used in matching, ranking, sorting and faceting of the search results.</p><h3>Sherlock Ingestion Components</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*kq_TCklDIPoe_ro1." /><figcaption>Figure 1: Sherlock Ingestion Overview</figcaption></figure><h4>Sherlock Ingester</h4><p>Flipkart’s engineering systems are designed and built using service-oriented architecture principles. The upshot of this is that the data required to index a single product/listing is spread across various internal services each of which is responsible for a specific set of product/listing attributes. All of our data source systems primarily provide a data change stream accessible via different methods — push (typically Kafka topics), pull (via APIs), etc. Sherlock Ingester is the component whose function is to fetch the data using appropriate mechanism from each data source and publish this data into Kafka and HBase.</p><h4>HBase</h4><p>A typical inverted index does not support partial/in-place updates of an indexed product/listing. This means that we need to update the entire document even if only one of the constituent indexable attributes changed. We use HBase as a snapshot store to handle this scenario efficiently. When the ingestion pipeline is running, every attribute update is persisted in HBase. After the first-ever update for a product/listing, any subsequent partial update can be combined with the snapshot stored in HBase to construct the entire document without fanning out to all other data source systems.</p><h4>Storm + Kafka</h4><p>Our real-time distributed stream processing framework of choice is Storm. Storm, coupled with Kafka, has a thriving user community not to mention the backing of the Apache foundation. The fact that these are two of the more reliable and widely-adopted open source technologies is the primary reason for us to choose them as foundational pieces of the ingestion stack.</p><p>We have multiple Storm topologies running with each tailored for processing data updates from a specific data source. A typical topology processes the data update stream in an idempotent fashion. We employ callbacks to the source system to fetch the latest value of the attribute coupling it optionally with ordering semantics provided by Kafka and Storm to ensure data integrity even if there happens to be out of order tuple processing. There are a couple of exceptions though and we currently employ solutions tailored to each exceptional situation.</p><p>Kafka plays a dual role — it is the source from which all the Storm topologies consume data tuples for processing. Kafka also is the sink into which the final output of computation from the Storm topologies is written. Every single serving node runs a Kafka consumer that continuously consumes from the sink topic and updates its in-memory NRT data store.</p><h4>Redis</h4><p>The NRT data store on the serving nodes is an in-memory store <em>without</em> persistence. When a serving instance comes up it needs to bootstrap its in-memory NRT data for which we use Redis. The final output of every topology is written to both Kafka as well as Redis. When a serving node needs to bootstrap its NRT data store, it will query Redis to load all the necessary data. We use a Kafka-offset checkpointing mechanism to trigger the consumption of the NRT data stream from the right offset after the completion of bootstrap.</p><p>Redis plays a critical role in our ingestion stack. It supports the high-volume writes from the Storm topologies with a very low latency. It also supports the bootstrap use-case with high throughput, low latency reads from the serving nodes. We deploy Redis in Master-Slave mode with multiple slaves in order to handle the huge spikes in operations per second during rolling restarts and code pushes.</p><h3>Scaling the Ingestion Stack</h3><p>The architecture of the ingestion layer, as described earlier, had been in-place and operational for some time during which the focus was mostly on the functional aspects of the ingestion pipeline — onboarding new attributes, topologies to process new data sources, etc. The business growth during this time pushed the ingestion stack closer to its limits (throughput in particular). Also, we had accumulated a fair amount of technical debt — we were still using old versions of Kafka (0.8.x) and Storm (0.9.x). Kafka, in particular, had released a completely redesigned consumer API as well as improved the broker reliability in the newer versions. Therefore we undertook a major effort to revamp, upgrade and tune the core components in the ingestion stack: Storm (1.0.x) and Kafka (0.10.0.x).</p><p>Our primary goals for this project were to scale the performance and improve the reliability of the ingestion pipeline significantly. The need to get this done without any disruption whatsoever to the existing workloads made this effort all the more challenging. Upgrading to the latest versions, be it your smartphone OS or production open source software, comes with associated risks and we had to pay our share of ‘pioneer tax’ as part of this project. A select set of technical highlights from this project are as under:</p><ul><li>Kafka clients are not backward compatible which makes the upgrade extremely risky from the viewpoint of having a backup plan and minimizing disruption to status quo. Kafka brokers, however, are backward compatible with older clients. We built ‘bridge’ topologies to provide operational flexibility during the migration</li><li>‘Bridge’ topologies use older Kafka client version. They can be configured to stream specific topics between the Kafka clusters bi-directionally (Old-&gt;New as well as New-&gt;Old)</li><li>These topologies were extremely handy for us and helped execute a phased migration with minimal risk</li><li>Kafka spout for Storm 1.0.x (storm-kafka-client) does not work out-of-the-box with Kafka version 0.10.x. We successfully ported the 1.1 version of the spout to resolve this issue</li><li>Given our goal of improving the reliability of the ingestion pipeline, we chose the manual commit option in our spout configuration. Our topologies are many and varied and we ran into numerous issues with regular/random commit exceptions happening in our spouts. Tuning the kafka consumer/broker settings and our topology code gave us partial relief. The clincher in the end was identifying an edge case that led to an offset tracking <a href="https://issues.apache.org/jira/browse/STORM-2505">issue</a> in the <a href="https://issues.apache.org/jira/browse/STORM-2544">spout</a>. Once identified and verified, we fixed it quickly and completed the upgrade</li><li>Flipkart leverages open source heavily across our systems and we benefit greatly from the community at-large. We believe in being good citizens and were more than happy to contribute the patch for the spout offset-tracking issue back to the Apache Storm project</li></ul><p>With the ingestion pipeline revamp, we achieved significant throughput increase across all our topologies with as much as <em>10x improvement</em> in the peak throughput in our critical topologies. Adopting the redesigned Kafka consumer vastly improved the reliability of our systems and made them robust in handling temporary and sustained errors (network issue, broker issue, etc.). This correlated with a significant drop in one-off issues with missing/delayed NRT data updates in the serving layer. Storm Nimbus node is not a SPOF anymore and also debugging and tuning of the topologies is now a lot easier. The ultimate test of our efforts came during <a href="https://stories.flipkart.com/category/top-stories/big-billion-days/">TBBD 2017</a>. The ingestion pipeline processed more than 4.5 Billion near real-time updates during this sale period. Also the throughput of the ingestion pipeline scaled a new peak of ~30K updates per second.</p><h3>Focus areas</h3><p>While we have scored significant wins from the stack upgrade, this is just the first step in our journey toward refreshing the ingestion stack and scaling it for future Big Billion Days. There are numerous challenges that still need to be tackled in the Sherlock ingestion layer and some of the areas where we are currently focusing are as follows:</p><ul><li>The ingestion pipeline is starting to run into scaling challenges with Redis such as choking the replication buffer on Master and operational issues with having a SPOF in the Master-Slave setup</li><li>Enabling priority updates in the ingestion pipeline across all data sources</li><li>Bootstrap of NRT data is resource inefficient and also has reliability challenges</li><li>Building a robust mechanism to completely refresh/reconcile data from specific data sources</li><li>Auditability of the ingestion pipeline</li></ul><p>The above list is far from being comprehensive and there are many more interesting challenges waiting to be solved. If you enjoyed reading through this post and love to work on such engineering problems then we would love to have a chat with you and explore potential opportunities to join us — <a href="http://www.flipkartcareers.com/">Flipkart Careers</a>. You will not only get to work on problems you enjoy solving but also get to do it with wonderful colleagues in a fun environment. It is not an easy feat to become <a href="https://www.linkedin.com/pulse/linkedin-top-companies-2017-where-india-wants-work-now-adith-charlie">The Top Company in India</a> and Flipkart has achieved this distinction for the second year running…!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95519783859d" width="1" height="1" alt=""><hr><p><a href="https://blog.flipkart.tech/sherlock-near-real-time-search-indexing-95519783859d">Sherlock: Near Real Time Search Indexing</a> was originally published in <a href="https://blog.flipkart.tech/">Flipkart Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>