<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Subhash Kumar Mamillapalli on Medium]]></title>
        <description><![CDATA[Stories by Subhash Kumar Mamillapalli on Medium]]></description>
        <link>https://medium.com/@subhash.mamillapalli?source=rss-424b77dce2d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*My6_nKwLPWjMlseH</url>
            <title>Stories by Subhash Kumar Mamillapalli on Medium</title>
            <link>https://medium.com/@subhash.mamillapalli?source=rss-424b77dce2d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:59:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@subhash.mamillapalli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Wishlist Redesign: How we reduced the latency by 75% to under 100ms]]></title>
            <link>https://medium.com/myntra-engineering/wishlist-redesign-how-we-reduced-the-latency-by-75-to-under-100ms-cc5880fcec19?source=rss-424b77dce2d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc5880fcec19</guid>
            <category><![CDATA[wishlist]]></category>
            <category><![CDATA[redesign]]></category>
            <category><![CDATA[document-store]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[cassandra]]></category>
            <dc:creator><![CDATA[Subhash Kumar Mamillapalli]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 12:26:27 GMT</pubDate>
            <atom:updated>2021-11-12T07:49:39.733Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sometimes Redesign is the only way out</h3><p>Every system encounters performance problems as it matures and needs frequent tuning to address them. However, sometimes a fundamental rethinking of the requirements and the design choices is necessary to address the challenges of scale. In this article I outline how we tried to tune our Wishlist service and eventually redesigned it to bring down the latency by 75% to under 100ms and handle a million concurrent users with a peak latency of under 300ms.</p><h3>Soft Intent</h3><p>Wishlist is at the heart of the e-commerce based consumer experience. People can easily add items they love to their wishlist for future consumption and the convenience of shopping from wishlist, opinionating on the wishlist is all evident. It is also a valuable source of analytics as items in wishlist play an important role in understanding the consumer. Next to cart, wishlist is the most critical feature in a consumer transaction.</p><h3><strong>Scale and other SLAs</strong></h3><p>We are seeing a 15% growth in the number of wishlists and wishlisted styles on a monthly basis. We wanted the system to be able to scale out easily based on the increasing demand to wishlist especially during festive sales where the demand is easily 300% more than the usual.</p><p>Wishlist by nature is read heavy as the user might check out the wishlisted items many times before making a purchase. Also given that the user can wishlist from multiple touch points, the updates to the wishlist are also high. The users compare the wishlisted items based on various attributes so it should be able to support various filters.</p><p>Since it is one of the critical systems that are user facing, the wishlist should be available 99.99 percent of the time and the latencies should be less say 99.99th percentile to be below 100ms.</p><h3>Problems with the initial design</h3><p>While the initial design fulfilled the functional requirements, we noticed quite a few performance problems over time with the increase in load and features. It did not scale well and there were frequent timeouts and intermittent outages. The details of how we tuned our database — Cassandra warrants a blog on its own. I will touch up on the 3 primary causes for the outages and how we tuned our Cassandra database to improve the performance.</p><ol><li>Frequent Compactions — Increase in the fields stored owing to the launch of new features and the nature of updates in wishlist resulted in frequent compactions of the data. We repaired the database once in a while to improve the read latencies.</li><li>High Read ahead — We reduced the read ahead buffer of the OS which resulted in a drastic improvement in the read latencies</li><li>GC long pauses — We were using a higher key cache which resulted in GC long pauses. We moved to file cache which works along side OS page cache to reduce the GC long pauses.</li></ol><p>Together these changes resulted in a 20 times reduction in read latencies from around 500ms to under 30ms at Cassandra as shown in the graph below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VwAxqiJYbr8yLoemH5_sEw.png" /><figcaption>Cassandra Read Latency</figcaption></figure><h3><strong>Back to the drawing board</strong></h3><p>Although we were able to tune the database to reduce latencies, we realised that in the long run it is not sustainable. The choice of Cassandra as the datastore was not right for the following reasons</p><ul><li>We have a read heavy workload</li><li>We need to retrieve the same data using multiple keys</li><li>We will have frequent updates to a portion of the data</li></ul><p>So we decided to redesign the wishlist service from the ground up keeping in mind the current workload, scale and future use cases.</p><p><strong><em>Wishlist Use cases</em></strong></p><p>The only operations on the wishlist are addition and removal of styles. We retrieve the entire wishlist as a whole along with the styles. The wishlist being a critical service needs to be highly available and since the flows for reads and updates are different we can live with an eventually consistent data store. So, we decided to use a reliable document store like MongoDb as our datastore.</p><p><strong><em>Data model</em></strong></p><p>Since the predominant operation involved reading the entire wishlist for a user, we decided to store the wishlist as one single document. There are two options available here. One option is to store the entire wishlist as a single document and the other to store the styles as independent documents with an indexed user id and wishlist id. Since we don’t store any style data apart from the ids in our database we don’t have any updates on style data. So, we decided against the 2nd option of storing styles as independent documents as that would defeat the purpose of using a hierarchical data store or a document store in the first place. Also this way we can take advantage of the data locality for faster reads.</p><p>There is still one problem with our database schema. The document size can become large if the number of styles in the wishlist grow uncontrolled. This can cause performance problems in the long run resulting from the fragmentation of disk space. So, we analysed the wishlisting patterns of customers and decided to cap the maximum styles that the user can wishlist. Also, instead of one single wishlist holding a large number of styles we decided that the user can better manage his favourite styles by storing them in multiple lists each with a specific context or purpose.</p><p><strong><em>Data Size and Cluster type</em></strong></p><p>Our initial estimates meant we need at most 1TB of storage for the next 5 years. With the current data size less than even a quarter of that and 30% of it being hot data, most of our hot data can easily fit in a single machine’s RAM. So we can serve most of the reads efficiently from the RAM. If we have a basic 3 node cluster we can easily support upto 3 times the current data size without sharding our data. So we used a replica set cluster configuration. Using the secondaries as read replicas, we are able to handle the read throughput of 5K reads per second very comfortably and our primary was able to handle the write throughput of 2K writes per second.</p><p><strong><em>Cache</em></strong></p><p>Since 99.95% of our operations are served within 50ms we decided against using a cache with its added complexity. The main reason for this is that most of the active wishlists are served directly from memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mxS-6ca7TSPj9qPIjtt0wQ.png" /><figcaption>Wishlist System</figcaption></figure><h3><strong>Migration of data from Cassandra to MongoDb</strong></h3><p>After the major exercise of stabilising the existing system and redesigning it, we faced the daunting task of planning the migration of data of 80 million users from Cassandra to MongoDb without any downtime.</p><p><strong><em>Migration</em></strong></p><p>One of the main challenges faced by us is that the new schema is significantly different from the old one. It is a lot simpler than the old schema and has a lot less fields and only a single collection or table rather than multiple tables used earlier. Since, we cannot directly ingest the data into MongoDb from Cassandra, we needed a script to transform the data into the current schema. We also need to ensure that we use only the spare compute power and disk throughput left to migrate the data from Cassandra to MongoDb so that there is minimal impact to the existing system.</p><p><strong><em>Synchronization</em></strong></p><p>While the script handles one time migration of existing data in Cassandra to MongoDb, we want to establish a dual replication and synchronization between Cassandra and MongoDb so that any current updates to the users’ wish lists are updated in both the databases. We have decided that we will propagate updates asynchronously by sending and consuming an event to update the MongoDb. This is primarily for two reasons.</p><ol><li>Not to introduce any overhead on the current writes and the user experience.</li><li>To allow for stabilisation of writes before they are fetched and updated in MongoDb.</li></ol><p>The risk of such an asynchronous update is that the read might not always fetch the most recent write. To mitigate this, we want to introduce a lag before processing the consumed event to update the MongoDb.</p><p>The update happens by fetching the wishlist of the user from Cassandra and replacing the corresponding wishlist with the fetched one in MongoDb. In case an error occurs during the update the event is added to the error queue and retried.</p><p><strong><em>Dual Replication</em></strong></p><p>I said dual replication because the any updates on MongoDb are also asynchronously updated in Cassandra in the same way as above. This way we ensure that the switch to the MongoDb is as smooth as possible. Also this way, we can plan to open up the new system with MongoDb as the primary database in a phased manner to a percentage of the overall traffic.</p><p><strong><em>Error Handling</em></strong></p><p>In order that the writes to the target database are up to date, we ensured that every write is preceded by a read of the data from the source database. However, there might still be race conditions and writes may not be serialized properly. So we need to validate the data written with the source and handle the scenarios where the data in Cassandra and MongoDb are out of sync.</p><h3>All’s well that ends well</h3><p>Finally, I would like to conclude with the perfomance improvements gained as a result of the redesign. First of all, we were able to simplify the system a lot. The schema is now a lot natural to the wishlist data and use case. We have removed caching and its added complexity, now that most of the reads are served directly from database with very low latency. The performance improvements were evident in the metrics, as the system throughput increased considerably and now we were able to serve 500K read requests per minute and about 200K updates per minute. The latencies also dropped drastically to below 200ms for 99.95% of the requests during peak load at festive sales and below 50ms during BAU.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc5880fcec19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/myntra-engineering/wishlist-redesign-how-we-reduced-the-latency-by-75-to-under-100ms-cc5880fcec19">Wishlist Redesign: How we reduced the latency by 75% to under 100ms</a> was originally published in <a href="https://medium.com/myntra-engineering">Myntra Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>