<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by harsh savasil on Medium]]></title>
        <description><![CDATA[Stories by harsh savasil on Medium]]></description>
        <link>https://medium.com/@harshsavasil?source=rss-b93459b8d92a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZA5VSi4EQoI_y6_LRbc0Ow.jpeg</url>
            <title>Stories by harsh savasil on Medium</title>
            <link>https://medium.com/@harshsavasil?source=rss-b93459b8d92a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:28:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshsavasil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solving the Hot Partition Problem: Optimising Data Distribution Across Partitions]]></title>
            <link>https://harshsavasil.medium.com/solving-the-hot-partition-problem-optimising-data-distribution-across-partitions-c5ed5ca17c3e?source=rss-b93459b8d92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5ed5ca17c3e</guid>
            <dc:creator><![CDATA[harsh savasil]]></dc:creator>
            <pubDate>Wed, 15 Jan 2025 10:55:42 GMT</pubDate>
            <atom:updated>2025-01-15T10:55:42.937Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*4h8Chft9cWfZMvYgEPefpw.png" /></figure><p>In distributed systems, one common challenge is efficiently distributing data across multiple partitions while avoiding “hot” partitions — those that receive disproportionately high loads. Today, we’ll explore an algorithmic solution to this problem using stock trading volumes as an example.</p><h3>The Problem</h3><p>Imagine you’re building a high-frequency trading platform that processes millions of stock trades daily. Each stock has its own trading pattern and volume, creating an interesting distribution problem:</p><ul><li>Some stocks like AAPL or TSLA might have extremely high trading volumes (hot stocks)</li><li>Others might have moderate or low trading volumes</li><li>Trading volumes can vary significantly throughout the day</li><li>System needs to handle both regular trading and sudden volume spikes</li></ul><p><strong>The Challenge</strong>: How do we distribute these stocks across multiple processing partitions to:</p><ul><li>Prevent any single partition from becoming a bottleneck</li><li>Ensure balanced resource utilisation</li><li>Maintain consistent performance during peak trading hours</li><li>Handle future volume growth gracefully</li></ul><p><strong>Simplified Problem Statement -</strong></p><ul><li>Distribute N stocks across K partitions</li><li>Minimize the maximum load on any partition</li><li>Balance the overall distribution to avoid any hot partitions</li></ul><h3>Data Setup and Preparation</h3><p>Before diving into the solution, let’s look at how we set up and clean our data. We’ll use real stock market data from Polygon.io API.</p><p><strong>1. Data Collection</strong></p><p>First, we collect historical trading volume data:</p><pre>def get_trading_volume_data(symbol: str, start: str, end: str):<br>    url = f&quot;https://api.polygon.io/v2/aggs/ticker/{symbol}/range/1/day/{start}/{end}&quot;<br>    response = requests.get(url, timeout=50)<br>    json_body = response.json()<br>    with open(f&#39;trading-volume-data/{symbol}.json&#39;, &#39;w&#39;) as f:<br>        json.dumps(json_body)</pre><p>Key aspects of data collection:</p><ul><li>Daily trading volumes for each stock</li><li>Full year of historical data</li><li>Rate limiting handled with sleep intervals</li><li>Data stored in JSON format for easy processing</li></ul><h3>2. Data Processing and Cleaning</h3><p>Instead of using raw trading volumes, we use the 95th percentile to account for outliers and unusual trading days:</p><pre>def get_p95_volume_data(symbol: str):<br>    with open(f&#39;trading-volume-data/{symbol}.json&#39;, &#39;r&#39;) as f:<br>        json_data = json.load(f)<br>        if json_data.get(&#39;resultsCount&#39;, 0) == 0:<br>            return 0<br>        trading_volume_data = [item[&#39;v&#39;] for item in json_data.get(&#39;results&#39;, [])]<br>        p95 = np.percentile(trading_volume_data, 95)<br>        return p95</pre><p><strong>Why P95?</strong></p><ul><li>Handles outliers better than maximum values</li><li>More realistic representation of high-volume scenarios</li><li>Excludes extreme events that might skew the distribution</li><li>Better predictor of normal high-load conditions</li></ul><h3>3. Data Structure</h3><p>The final processed data looks like this:</p><pre>[<br>    {<br>        &quot;symbol&quot;: &quot;AAPL&quot;,<br>        &quot;volume&quot;: 1441581.0<br>    },<br>    {<br>        &quot;symbol&quot;: &quot;GOOGL&quot;,<br>        &quot;volume&quot;: 892345.0<br>    }<br>    // ... more stocks<br>]</pre><h3>The Solution: A Recursive Backtracking Approach</h3><p>Our solution uses recursive backtracking with optimisation heuristics to find the optimal distribution. Here’s how it works:</p><h4>1. Smart Initial Threshold</h4><pre>def get_heurisitic_factor(ticks):<br>    max_possible = sum(ticks)<br>    min_possible = max(max(ticks), sum(ticks) // num_of_partitions)<br>    heurisitic_factor = min_possible * 1.0001<br>    return heurisitic_factor</pre><p>We establish a threshold based on two factors:</p><ul><li>The highest individual stock volume</li><li>The theoretical minimum (total volume ÷ number of partitions)</li></ul><p>This helps us prune unnecessary recursive branches early.</p><h3>2. Recursive Assignment Strategy</h3><p>The core algorithm recursively tries different combinations:</p><pre>def assign_partition(stock_ticks, current_index, distribution, threshold, best_answer):<br>    if current_index == len(stock_ticks):<br>        # Check if we found a better distribution<br>        getMiniumInCurrentDistribution(distribution, best_answer)<br>        return<br>        <br>    for partition in range(0, num_of_partitions):<br>        # Skip if adding stock would exceed threshold<br>        if distribution[&#39;partitions&#39;][partition][&#39;sum&#39;] + stock_ticks[current_index][&#39;volume&#39;] &gt; threshold:<br>            continue<br>            <br>        # Try adding stock to this partition<br>        # Recurse<br>        # Backtrack by removing stock</pre><h3>3. Optimization Techniques</h3><p>Several optimizations make the algorithm practical:</p><ul><li><strong>Pre-sorting</strong>: Stocks are sorted by volume in descending order</li></ul><pre>   stock_ticks = sorted(stock_ticks, key=lambda x: x[&#39;volume&#39;], reverse=True)</pre><ul><li><strong>Early Pruning</strong>: Skip branches that would exceed the threshold</li></ul><pre>   if distribution[&#39;partitions&#39;][partition][&#39;sum&#39;] + stock_ticks[current_index][&#39;volume&#39;] &gt; threshold:<br>       continue</pre><ul><li><strong>Deviation Tracking</strong>: Beyond minimizing maximum load, we also track overall distribution balance</li></ul><pre>   def update_current_deviation(distribution, best_answer):<br>       deviation_ = 0<br>       for partition in distribution[&#39;partitions&#39;]:<br>           deviation_ += abs(partition[&#39;sum&#39;] - best_answer)<br>       distribution[&#39;deviation&#39;] = deviation_</pre><h3>Results and Output</h3><p>The algorithm outputs a JSON structure showing:</p><ul><li>The optimal distribution of stocks across partitions</li><li>The maximum load deviation between partitions</li><li>The minimum achieved of the maximum load for any partition</li></ul><p>Example output structure:</p><pre>{<br>  &quot;deviation&quot;: 1000,<br>  &quot;minimum&quot;: 50000,<br>  &quot;partitions&quot;: [<br>    {<br>      &quot;sum&quot;: 49000,<br>      &quot;elements&quot;: [30000, 19000],<br>      &quot;stocks&quot;: [&quot;AAPL&quot;, &quot;MSFT&quot;]<br>    },<br>    // ... more partitions<br>  ]<br>}</pre><h3>Performance Considerations</h3><p>While the algorithm guarantees finding an optimal solution (if one exists within the threshold), there are some considerations:</p><ul><li><strong>Time Complexity</strong>: The worst-case complexity is exponential O(K^N), where K is the number of partitions and N is the number of stocks</li><li><strong>Optimisation Impact</strong>: The heuristic threshold and sorting significantly reduce the actual runtime</li><li><strong>Memory Usage</strong>: The algorithm maintains state for each recursive call, but the memory impact is limited by the problem size</li></ul><h3>Performance Analysis</h3><p>Let’s analyse the algorithm’s performance with real data:</p><p><strong>Time Complexity vs Data Size</strong></p><ul><li>Small datasets (&lt; 100 stocks): Near-instant results</li><li>Medium datasets (100–1000 stocks): Polynomial time growth</li><li>Large datasets (&gt; 1000 stocks): Exponential growth</li></ul><p><strong>Memory Usage</strong></p><ol><li><strong>Recursion Stack:</strong></li></ol><ul><li>Maximum depth = N (number of stocks)</li><li>Each stack frame stores distribution state</li><li>O(N) stack space</li></ul><p><strong>2. Distribution State:</strong></p><ul><li>K partitions with their elements</li><li>O(N + K) space for distribution tracking</li></ul><p><strong>Total Space Complexity: O(N + K)</strong></p><p><strong>Optimisation Impact</strong></p><ul><li>Sorting by volume: 20–30% performance improvement</li><li>Threshold pruning: 40–60% reduction in recursive calls</li><li>P95 data cleaning: More stable distribution patterns</li></ul><h3>Real-world Applications</h3><p>This solution can be applied to various scenarios:</p><p><strong>Financial Systems</strong></p><ul><li>Stock market data processing</li><li>Transaction processing systems</li><li>Real-time trading platforms</li></ul><p><strong>Cloud Infrastructure</strong></p><ul><li>Microservices deployment</li><li>Database sharding</li><li>Cache distribution</li></ul><p><strong>Big Data Processing</strong></p><ul><li>MapReduce job distribution</li><li>Stream processing workload balancing</li><li>Data warehouse partitioning</li></ul><h3>Conclusion</h3><p>The Hot Partition Problem is a classic example of optimization in distributed systems. The solution demonstrates how combining algorithmic techniques (recursive backtracking) with practical optimisations can solve real-world distribution challenges. The full implementation is available on <a href="https://github.com/harshsavasil/hot-partition-problem">GitHub</a>, and we welcome contributions and improvements from the community.</p><p>People could try a variation of the above solution where we pick up random partition every time and keep track of the visited partitions. I feel this could work even faster than the current solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5ed5ca17c3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design Real-Time Stock Price Feed]]></title>
            <link>https://harshsavasil.medium.com/design-real-time-stock-price-feed-9e1fba0a1c4e?source=rss-b93459b8d92a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e1fba0a1c4e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[server-sent-events]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[websocket]]></category>
            <dc:creator><![CDATA[harsh savasil]]></dc:creator>
            <pubDate>Sun, 23 May 2021 18:00:14 GMT</pubDate>
            <atom:updated>2025-03-18T12:33:27.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-Time Stock Price Feed Design</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmdrB3dFDAKcdX3uK0eS0A.png" /></figure><p><strong>Overview</strong></p><p>This article explains the process of designing a real-time stock price feed for investment platforms and is targeted at developers who intend to design a trading platform for their users. In this technical post, we will discuss designing a low-latency price feed. Also, we will consider a few design choices and analyze their pros and cons.</p><p>This entire design will be divided into a series comprised of four articles. In the first article, we will be discussing the different design choices and the high-level design of the system. In the second and third articles, we will delve into the details of the implementation and the last one would be combining all of it.</p><p><strong>Exploring Different Design Choices</strong></p><p>Since our entire architecture is event-driven, we would be needing message queues to route the data to different microservices for different operations. While choosing a message broker for executing our asynchronous operations, we will consider these things:</p><ol><li><strong>Broker Scale </strong>— The number of messages being processed per second in the system.</li><li><strong>Data Persistency</strong> — The ability to recover messages in case of abrupt failures.</li><li><strong>Consumer Capability</strong> — Whether the broker is capable of managing one-to-one and/or one-to-many consumers.</li><li><strong>Ordering of Processing of Messages — </strong>Whether the order in which the messages are queued is the same as the order in which they are processed by the subscribers.</li></ol><p><strong>Scale</strong></p><p>For our use case, let’s crunch some numbers. Assuming we are creating a price feed for 1,000 stocks. And, we are getting 1 price tick every second (considering Standard Price Feed APIs like Binance, Bloomberg, etc.) for each of these stocks. This sums up our complete scale to a total of 1,000 messages per second. The <strong>scale of overall messages is so low</strong>, that it can not be a differentiating factor because <strong>Kafka </strong>and<strong> Redis Pub/Sub</strong> have the capability of <strong>processing a million messages per second</strong> whereas <strong>RabbitMQ</strong> can process <strong>50k messages per second</strong>. But for our use cases, all of the message brokers would pass on this metric.</p><p><strong>Persistency</strong></p><p>Since our messages are short-lived, we don’t need data persistence in our use case. Even if we miss a few price ticks while sending it to our users, It won’t impact our system. In simple terms, the messages won’t be lost even if the subscribers are down.</p><p><strong>Consumer Capability</strong></p><p>Since we will be sending price updates to multiple consumers for different operations. We would need a message broker which can <strong>manage one-to-many consumers</strong>.</p><p><strong>Ordering of Processing of Messages</strong></p><p>Some Message Brokers work on the <strong>fire and forget or broadcasting</strong> strategy and <strong>don’t wait for the acknowledgement</strong> from the subscribers before pushing the new message to the subscriber.</p><p><strong>NOTE — If the processing of the message is asynchronous, the order of processing of the message might not be the same as the order in which they were queued</strong>.</p><p><strong>Redis Pub/Sub </strong>is a perfect fit for our use case when we want to send price ticks to our user in <strong>minimum latency</strong> since the messages are <strong>short-lived</strong> and few ticks <strong>loss is</strong> <strong>tolerable</strong>. Furthermore, there won’t be any obligation of sending stale prices to clients, therefore <strong>persistence is not needed</strong>. It provides <strong>extremely fast service</strong> and <strong>in-memory capabilities</strong>, which <strong>reduces the latency</strong> of the price updates.</p><p>But <strong>Redis Pub/Sub fails to guarantee the ordering of the processing of messages by the subscribers</strong> since it works on a <strong>broadcasting model</strong>. Therefore, we might need some Kafka or RabbitMQ for services where the <strong>ordering of processing of messages is required</strong> like a service that will be responsible for updating the candlesticks. So in our final architecture, we will use a <strong>combination of RabbitMQ and Redis Pub/Sub.</strong></p><p>There could be three different ways of <strong>serving our real-time stock price feed </strong>from our <strong>api-server</strong> <strong>to our clients</strong>. These are as follows -</p><p><strong>Polling</strong></p><p>In polling, a client makes XHR requests to the server repeatedly at some regular interval (e.g 0.5 Seconds) to check for new prices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dBX-7Rp7k51XEgvC9sHRw.png" /></figure><p><strong>Pros</strong></p><ul><li>Easy to implement.</li></ul><p><strong>Cons</strong></p><ul><li><strong>Increases Server Load</strong> because of repeated requests.</li><li><strong>Increases Client resource utilization</strong> because of repeated requests.</li></ul><p><strong>WebSockets</strong></p><p>WebSocket is an entirely different 7 layer <strong>communication protocol</strong> that provides <strong>full-duplex</strong> <strong>communication</strong> channels over a <strong>single TCP connection</strong>. The WebSocket protocol allows the communication between a client and a web server with <strong>lesser overheads</strong> (no need of creating and destroying connections every time), providing <strong>real-time data transfer</strong> from and to the server. WebSockets keep the <strong>connection open</strong>, therefore allowing data to be passed back and forth between the client and the server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0y1ASe7gmMCwJpfls64_A.png" /></figure><p><strong>Pros</strong></p><ul><li>The<strong> bi-directional channel</strong> allows full-duplex communication.</li><li>Provides<strong> Low Latency</strong> Data Communication.</li><li><strong>Native support</strong> in more browsers.</li><li>Supports <strong>binary</strong> as well as <strong>utf-8 </strong>data transmission<strong>.</strong></li></ul><p><strong>Cons</strong></p><ul><li>A <strong>maximum number of open connections limit</strong> (255 in Chrome and 200 in Firefox).</li><li>Proxy Servers can wreak havoc with WebSockets running over unsecured HTTP.</li><li>It can be <strong>overkill</strong> for some types of applications, where the client doesn’t need to send data to a server in real-time.</li></ul><p><strong>Server-Sent Events</strong></p><p><strong>Server-Sent Events</strong> (<strong>SSE</strong>) is a <strong>server push</strong> technology in which a client establishes a <strong>long persistent connection</strong> with the server, therefore allowing a server to automatically send new events/messages to a client(whenever available) <strong>via an HTTP connection</strong>. They are commonly used to broadcast message updates or send continuous data streams to a client e.g. Twitter feed, Instagram feed, or a stock price feed. SSE uses Javascript API called <strong>EventSource</strong> to continuously update a client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hDKklGVhQIQlOGiui3GAg.png" /></figure><p>Pros</p><ul><li>Uses <strong>HTTP</strong> instead of a custom protocol</li><li>Built-in support for <strong>re-connection</strong> and event-id</li><li>No trouble with corporate firewalls (Corporate Proxies) doing packet inspection.</li></ul><p>Cons</p><ul><li>Supports <strong>only</strong> <strong>utf-8 </strong>data transmission<strong>.</strong></li><li>A <strong>maximum number of open connections limit </strong>(Very low for some clients).</li></ul><p>Because of the <strong>simplex communication nature of SSE</strong>, it fits right into our use case. Since we will just be <strong>broadcasting new price updates</strong> to all the <strong>connected clients, </strong>we won’t need any information from the clients to send these price updates, so <strong>WebSockets would be an overkill</strong>. Also, SSE is very <strong>easy to implement</strong> with native APIs of Node.js. Now that we have considered different design choices, we should dig deeper into the high-level design of our entire system and discuss each of those components separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*goi4KSrBpShYW3MhrF5Ahg.png" /><figcaption>High-Level Design</figcaption></figure><p><strong>Exchange or External Price Source</strong></p><p>The service will be creating the price ticks of a stock. It could either be an internal exchange service or an external source of price like Binance, FinHubb, etc. This service could provide us with a new price by sending the new prices using web sockets or SSE (Push Mechanism) or we could fetch prices from them at regular intervals (Pull Mechanism).</p><p><strong>Price Consumer Service</strong></p><p>This service <strong>consumes the price ticks</strong> from the Exchange or External Price Source and passes them into the system. This could be the right place for adding some <strong>validations on price ticks</strong> before allowing them to flow into the system. You could also synthesize your <strong>synthetic price feed</strong> using the raw price feed by <strong>adding some safety spreads or currency conversions</strong>. This service then publishes the new prices on the price_feed:${stock} <strong>Redis channel</strong> and the price-ticks<strong> fanout exchange</strong>. The exchange will simply broadcast the prices to all the subscribed queues. You can either run one instance of price-consumer-service for each stock or combine a group of multiple stocks for one instance or a combination of both strategies. It depends on your use case.</p><p><strong>API Server</strong></p><p>The service provides event stream API (SSE) and other REST APIs to clients. Each instance of this service will subscribe to the Redis price_feed:{stock} channel to get price updates in real-time. Whenever there is a new price update, each api-server instance is notified about the new price via events. On receiving this event, the api-server then pushes the price update event on the SSE for the clients to receive the real-time price updates.</p><p><strong>Stock Price Service</strong></p><p>The service will subscribe to the save-price-ticks message queue and consume all the messages and save them in a time-series database like Timescale or InfluxDB for analytics. A separate message queue is added at the start of this service because we need persistence in case of a failure to avoid losing price ticks for analytics.</p><p><strong>Price Graph Service</strong></p><p>This service will subscribe to the update-candle-sticks message queue and updates the different candles in the cache as well as the primary read database in your system. A separate message queue is added at the start of this service to ensure the correct ordering in which the messages are processed by using acknowledgements.</p><p><strong>Further Reading</strong></p><ul><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/EventSource">EventSource MDN</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_APIhttps://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">The WebSocket API</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">Using Server-Sent Events</a></li><li><a href="https://ably.com/topic/websockets">https://ably.com/topic/websockets</a></li><li><a href="https://stackoverflow.com/questions/11077857/what-are-long-polling-websockets-server-sent-events-sse-and-comet">https://stackoverflow.com/questions/11077857/what-are-long-polling-websockets-server-sent-events-sse-and-comet</a></li><li><a href="https://otonomo.io/redis-kafka-or-rabbitmq-which-microservices-message-broker-to-choose/">https://otonomo.io/redis-kafka-or-rabbitmq-which-microservices-message-broker-to-choose/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e1fba0a1c4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>