<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sumedha Khatter on Medium]]></title>
        <description><![CDATA[Stories by Sumedha Khatter on Medium]]></description>
        <link>https://medium.com/@sumedhakhatter?source=rss-d0b9056dbd47------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*67_PojHQDlf305nfwZjGtQ.jpeg</url>
            <title>Stories by Sumedha Khatter on Medium</title>
            <link>https://medium.com/@sumedhakhatter?source=rss-d0b9056dbd47------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:54:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sumedhakhatter/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can AI Predict Stock Returns? I Tested TimesFM-3 Against Simpler Models]]></title>
            <link>https://sumedhakhatter.medium.com/can-ai-predict-stock-returns-i-tested-timesfm-3-against-simpler-models-a4b3fdafdaea?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4b3fdafdaea</guid>
            <category><![CDATA[quantitative-finance]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 01:52:59 GMT</pubDate>
            <atom:updated>2026-10-16T01:52:59.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*01sLCgZPk-3hpEV_ic6V8Q.png" /></figure><h4>I compared an advanced AI forecasting model against simpler statistical methods. The results challenged a common assumption: better predictions don’t necessarily make better investments.</h4><p>A better forecast should lead to a better trading strategy. At least, that’s the intuition. But what if a model predicts uncertainty almost perfectly and still cannot identify which stocks will outperform?</p><p>I compared Google’s TimesFM-3 against Ridge Regression and Gradient Boosting using historical data from 40 large US companies, generating nearly 69,000 predictions per model.</p><p>The experiment used 40 large US stocks, 14 rounds of historical testing, and 68,876 predictions per model.</p><p><strong>Three findings stood out:</strong></p><ul><li>TimesFM-3 was the best at estimating uncertainty around future returns.</li><li>Ridge Regression, a much simpler model, showed the strongest ability to rank future stock performance.</li><li>Every model lost money after accounting for trading costs.</li></ul><p>The interesting part isn’t simply that the models failed to generate profitable trades. It’s why they failed.</p><h3>1. The experiment: Can AI spot opportunities others miss?</h3><p>Suppose Apple’s stock rises 3% in a day.</p><p>At first glance, that looks like strong performance. But if the broader market and technology sector also rose sharply, Apple’s gain might not be particularly impressive.</p><p>A stock’s return can be divided into two broad components:</p><ol><li><strong>Market and sector exposure:</strong> The portion explained by movements in the broader market and the stock’s industry.</li><li><strong>Beta-residual return: </strong>The portion remaining after accounting for those movements.</li></ol><p>The word <em>beta</em> refers to how sensitive a stock is to market movements. A stock with a market beta of 1.2, for example, tends to move about 1.2% for every 1% market move, assuming other factors remain unchanged.</p><p>Consider a simplified example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SM1cPLWcKZHYYdKnO0Ejkg.png" /><figcaption>Figure 1: Beta residual return</figcaption></figure><p>Rather than predict the entire stock return, I focused on predicting this residual component. Why? Because a model that predicts the entire market will rise is not necessarily good at selecting individual stocks. Residual returns help isolate stock-specific opportunities.</p><h3>The models</h3><p>I compared four forecasting approaches:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaABlrn7CV8CnisBUgjkoQ.png" /></figure><p>A <em>foundation model </em>is trained on large amounts of data before being applied to a specific task. Unlike Ridge or GBM, TimesFM-3 arrives with general forecasting knowledge learned during pretraining. Each model produced forecasts for the next trading day, including estimates of uncertainty.</p><p>The experiment covered 2012–2022 and used <strong>walk-forward testing: </strong>training on earlier data, predicting later unseen data, then moving the evaluation window forward.</p><p>The testing also used <strong>purging and embargoing,</strong> techniques that separate training and testing observations to reduce information leakage caused by overlapping time periods.</p><p>This matters because a model should not receive credit for predicting information it effectively encountered during training.</p><h3>2. The advanced AI model was best at estimating uncertainty</h3><p>Most people think of a forecast as a single number.</p><p>For example, a model might predict that a stock will rise 1% tomorrow.</p><p>But a more informative forecast also describes uncertainty.</p><p>Instead of saying the stock will rise exactly 1%, the model might estimate that its return has an 80% probability of falling between −2% and +4%.</p><p>This is called a <strong>prediction interval.</strong></p><p>A well-calibrated model should produce intervals that match reality. If it generates 80% prediction intervals repeatedly, approximately 80% of actual outcomes should fall inside them.</p><p>That property is called<strong> calibration.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdRKLzUv4FSewUeDgo6EMg.png" /></figure><p>TimesFM-3 performed best, achieving 80.1% coverage.</p><p>But interval coverage alone is insufficient. A model could produce extremely wide intervals that capture nearly every outcome without offering useful information. That’s why I also measured pinball loss.</p><h4>What is pinball loss?</h4><p>Instead of predicting only a single return, each model produced nine quantiles.</p><p>A quantile represents a point in a predicted probability distribution. For example, the 10th percentile is a return threshold below which the model expects approximately 10% of outcomes to fall.</p><p>Pinball loss measures how accurate these predicted quantiles are. It penalizes forecasts differently depending on whether actual returns fall above or below the predicted threshold.</p><p>Lower pinball loss is better.</p><p>To compare models, I calculated <em>pinball skill:</em> the percentage improvement in loss relative to the zero baseline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvvz_YVGank6dJOh-1SRPg.png" /></figure><p>TimesFM-3 performed best on both pinball skill and interval coverage.</p><p>Its uncertainty estimates also needed remarkably little recalibration. Across the 14 testing periods, the average adjustment to its predicted interval width was just 0.977, close to the ideal multiplier of 1.</p><p><strong>TimesFM-3 was good at estimating how uncertain future returns would be. </strong>But that does not establish that it could predict which stocks would outperform.</p><h3>3. The simpler model was better at predicting direction</h3><p>Consider two stocks.</p><p>A model predicts that Stock A will outperform Stock B tomorrow.</p><p>If Stock A consistently performs better, the model has useful directional information — even if it cannot predict the exact returns.</p><p>To measure this, I used the<strong> Information Coefficient (IC).</strong></p><p>In this experiment, IC measures the relationship between a model’s predicted stock-return rankings and the returns that actually occurred.</p><p>An IC near zero indicates little useful ranking relationship. A positive IC means stocks ranked higher by the model tend to outperform those ranked lower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlFOq2mpMD38fCpaO_4shA.png" /></figure><p>Ridge achieved an IC of 0.0108, compared with 0.0073 for GBM and just 0.0033 for TimesFM-3.</p><p>All three values were small. To understand whether the observed relationships were distinguishable from statistical noise, I also examined their t-statistics.</p><h4><strong>What is a t-statistic?</strong></h4><p>A t-statistic compares an estimated effect with the uncertainty surrounding that estimate.</p><p>A larger absolute t-statistic means stronger evidence that the measured effect differs from zero, under the assumptions of the statistical test.</p><p>As a rough convention, a t-statistic above 2 is often treated as evidence of statistical significance, although the appropriate threshold depends on the test and how many hypotheses were examined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVug4HUgANF8Q8c5-Idxyg.png" /></figure><p>Only Ridge exceeded a t-statistic of 2.</p><p>TimesFM-3’s result was too weak to establish a reliable directional signal.</p><p>There was another warning sign: TimesFM-3’s relationship between predicted and actual returns changed direction across testing periods. In four of the fourteen periods, its recalibration slope was negative.</p><p>That means the direction of its fitted predictive relationship was not consistently stable.</p><p><strong>The model that best estimated uncertainty was the weakest at identifying future outperformers.</strong></p><p>This distinction matters because a trading strategy earns money from making the right investment decisions, not from accurately describing how uncertain those decisions are.</p><h3>4. Trading costs erased every model’s advantage</h3><p>A model can identify a real predictive pattern and still fail as a trading strategy.</p><p>Suppose a strategy generates $100 in trading gains but spends $120 executing those trades.</p><p>The prediction may contain useful information. The strategy still loses $20.</p><p>To test economic value, I converted each model’s forecasts into a simulated investment portfolio.</p><p>The portfolios were designed to be dollar-neutral, meaning the value of long positions (stocks expected to outperform) was balanced by short positions (stocks expected to underperform).</p><p>They were also hedged against market and sector exposure, reducing the effect of broad market movements on the results.</p><p>I then measured performance before and after estimated trading costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPpYUOWlAJlZ3Su_1qkltg.png" /><figcaption>Figure: All three models generated positive gross returns but negative returns after costs.</figcaption></figure><p>GBM produced the highest return before costs: 2.45% annually.</p><p>But after costs, it lost 5.14%.</p><p>The same pattern appeared across all three models.</p><h4>Why were trading costs so high?</h4><p>The main reason was <strong>portfolio turnover.</strong></p><p>Turnover measures how much of a portfolio is replaced through trading over a given period.</p><p>In this experiment, the strategies had approximately 42–45% daily turnover.</p><p>That meant they were frequently changing positions to follow new predictions.</p><p>Every trade introduced costs such as brokerage fees, bid-ask spreads, and price impact.</p><p>The bid-ask spread is the difference between the price at which a security can be bought and sold. Market impact occurs when trading itself affects the execution price.</p><p>These costs accumulated to roughly 8% annually, overwhelming the models’ small predictive advantages.</p><p>For TimesFM-3, the estimated trading costs were more than 32 times its gross return.</p><p>The result illustrates a fundamental principle:</p><p>Finding a predictable pattern is not enough. The expected profit must exceed the cost of exploiting.</p><h3>5. The modeling error that changed the results</h3><p>One of the most important discoveries came from correcting how I defined the return the models were supposed to predict.</p><p>To understand the mistake, we need to distinguish three concepts: beta, alpha, and residual return.</p><p>A simplified financial regression model expresses a stock’s return as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*quwBzzz5dNkJD-X_eOhTxQ.png" /></figure><p>Alpha and residual return are related, but different.</p><p><strong>Alpha is an estimated historical average. The regression residual is the remaining day-specific deviation after subtracting that average and the market/sector contribution.</strong></p><p>Consider a simplified example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A9xG75Ra-vngqMPUFEvltA.png" /></figure><p>My original calculation removed both market exposure and estimated alpha.</p><p>But the trading strategy was designed to hedge market and sector exposure — not to remove the stock’s historical alpha.</p><p>After hedging those factors, the portfolio’s relevant return would still be 3%, not 2%.</p><p>The corrected prediction target therefore retained alpha.</p><p><strong>How much did this change the results?</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UDgDjWIVXQGNDD6QUHJ_xQ.png" /><figcaption>The target correction reduced the apparent directional signal. The Sharpe comparison uses the same portfolio weights evaluated against different return definitions.</figcaption></figure><p>The Information Coefficient fell from 0.030 to approximately 0.011. The Sharpe ratio fell from 2.33 to 0.37 when the same portfolio weights were evaluated against the return the strategy could actually earn.</p><h4>What is the Sharpe ratio?</h4><p>The Sharpe ratio measures how much excess return an investment generates relative to the variability of its returns.</p><p>Its simplified formula is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*XI9uzsDGQw6YpEFuhB5mHg.png" /></figure><p>The risk-free return is a benchmark representing the return available from a very low-risk investment, commonly approximated using short-term government securities.</p><p>Volatility measures how much returns fluctuate.</p><p>A higher Sharpe ratio generally indicates more return per unit of volatility.</p><p>For example, a strategy earning 10% with large fluctuations may have a lower Sharpe ratio than one earning 8% with much steadier returns.</p><p>A Sharpe ratio above 1 is often considered attractive, while a ratio above 2 is considered particularly strong. These are rough conventions, not guarantees of quality.</p><p>The original value of 2.33 therefore made the strategy appear highly promising.</p><p>The corrected value of 0.37 showed a much weaker relationship between return and risk.</p><p>Importantly, this was not conventional lookahead bias, where future information accidentally enters the model.</p><p>The problem was that the prediction target did not match the economic payoff of the portfolio.</p><p><strong>A model can produce impressive statistical results while predicting something a trading strategy cannot directly earn.</strong></p><h3>6. Can AI tell when its own predictions are reliable?</h3><p>Another question was whether the models could identify situations where their forecasts were more trustworthy.</p><p>Suppose a model predicts that two stocks will each rise 2%.</p><p>For Stock A, it predicts a likely return range of +1% to +3%.</p><p>For Stock B, it predicts a range of −5% to +9%.</p><p>The model is expressing greater certainty about Stock A because its predicted interval is narrower.</p><p>I tested whether trading only when prediction intervals were narrow could improve performance.</p><p>This approach is called a confidence gate: a rule that allows a trade only when the model’s predicted uncertainty is sufficiently low.</p><h4>Was the confidence signal actually new information?</h4><p>To investigate, I compared the width of predicted return intervals with historical stock volatility.</p><p>Volatility measures how much a stock’s returns fluctuate. A stable stock has lower volatility; a stock experiencing large price swings has higher volatility.</p><p>I measured their relationship using Spearman’s rank correlation coefficient.</p><p>Spearman correlation compares the rankings of two variables rather than their exact numerical values.</p><p>Its values range from −1 to +1:</p><ul><li>+1: Higher values of one variable consistently correspond to higher values of the other.</li><li>0: No consistent increasing or decreasing relationship in their rankings.</li><li>−1: Higher values of one variable consistently correspond to lower values of the other.</li></ul><p>Unlike ordinary linear correlation, Spearman correlation can detect relationships where two variables move consistently together without increasing at a constant rate.</p><p>The correlation between prediction interval width and historical volatility was 0.859.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8LbQAr7CgwJzjt8kAu2Ykg.png" /></figure><p>That was a strong relationship.</p><p>It suggested that the model’s confidence was closely connected to something already observable: how much the stock’s price had fluctuated in the past.</p><p>I then compared two trading filters.</p><p>The first selected stocks using narrow AI prediction intervals.</p><p>The second selected stocks using low historical volatility.</p><p>The two filters had a <strong>Jaccard overlap of 0.845, or 84.5%.</strong></p><p><em>Jaccard overlap</em> measures how similar two sets are. It divides the number of items selected by both methods by the number selected by either method.</p><p>An overlap of 84.5% means the two filters were selecting largely the same stocks.</p><p>More importantly, the AI confidence filter did not outperform the simpler volatility filter. In the tested Ridge strategy, it actually worsened risk-adjusted returns.</p><p>The conclusion was not that AI uncertainty estimates were wrong.</p><p><strong>It was that the confidence filter added little useful trading information beyond what a simple volatility measure already captured.</strong></p><h3>7. What the experiment tells us</h3><p>The results highlight three important distinctions.</p><h4>Forecasting uncertainty is not the same as predicting direction</h4><p>TimesFM-3 estimated the distribution of future returns more accurately than the other models. But it was the weakest at ranking which stocks would outperform. These are different forecasting objectives. A model can succeed at one without succeeding at the other.</p><h4>Statistical significance does not guarantee profitability</h4><p>Ridge produced the strongest evidence of directional predictability. But its advantage was too small to overcome trading costs. A statistically detectable pattern is not necessarily large enough to support a viable investment strategy.</p><h4>More sophisticated models are not automatically better</h4><p>TimesFM-3 is a powerful pretrained forecasting model, but the simpler Ridge model produced the strongest directional signal in this experiment. Model complexity is valuable only when it improves the outcome that matters.</p><h4>Important limitations</h4><ol><li>These results should not be interpreted as proof that AI cannot predict stock markets.</li><li>The study used 40 stocks, a survivorship-biased universe, historical development data, and simulated transaction costs.</li><li>Survivorship bias means the selected universe may overrepresent companies that remained successful or available through the study period, making historical results potentially more favorable than a fully investable historical universe.</li><li>There is also possible overlap between TimesFM’s pretraining data and the historical evaluation window.</li><li>Finally, the reserved 2023+ holdout has not been evaluated. A holdout is a separate dataset kept untouched during model development to provide a more independent final test.</li></ol><p>These limitations matter when deciding how much confidence to place in the findings.</p><h3>Conclusion: A good forecast is not necessarily a good trade</h3><p>The experiment produced an interesting contradiction.</p><ol><li><strong>TimesFM-3 delivered the best-calibrated forecasts </strong>but almost no reliable directional signal.</li><li><strong>Ridge produced the strongest evidence of stock-ranking ability.</strong></li><li><strong>GBM generated the highest gross trading returns.</strong></li></ol><p>Yet every strategy lost money after transaction costs.</p><p>The broader lesson extends beyond finance.</p><p>Machine learning models are often evaluated using statistical metrics: prediction error, accuracy, correlation, or calibration. But strong performance on these metrics does not automatically translate into better real-world decisions.</p><p>For financial markets, a prediction must be directionally useful, economically actionable, and valuable enough to overcome the costs of trading. Predicting the future is difficult. Turning a prediction into profit is a different.</p><h3>References</h3><ul><li><a href="https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/?utm_source=chatgpt.com">Google Research — TimesFM</a></li><li><a href="https://github.com/google-research/timesfm?utm_source=chatgpt.com">TimesFM — Official implementation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4b3fdafdaea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Model Is Too Big for Your GPU. What If You Just Don’t Load It All?]]></title>
            <link>https://sumedhakhatter.medium.com/your-ai-model-is-too-big-for-your-gpu-what-if-you-just-dont-load-it-all-859edc6a8bd9?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/859edc6a8bd9</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 07:12:53 GMT</pubDate>
            <atom:updated>2026-10-07T07:12:53.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MUNJhbbQ3Rjwhvb0r-4bmA.png" /></figure><p><em>How Soup and Colibrì use two different strategies to run huge AI models with limited fast memory.</em></p><h3>In this article</h3><ol><li><strong>The memory problem</strong></li><li><strong>Soup: stream layers through the GPU</strong></li><li><strong>Colibrì: load the experts you need</strong></li><li><strong>Soup vs. Colibrì: when to use which</strong></li><li><strong>The bigger systems idea</strong></li><li><strong>Appendix: ML concepts</strong></li><li><strong>References</strong></li></ol><h3>1. The memory problem</h3><p>Large language models have a straightforward hardware problem: <strong>models are getting enormous, but fast memory is limited.</strong> An LLM contains billions — or increasingly hundreds of billions — of learned numerical values called parameters or weights. During computation, we ideally want the necessary weights close to compute, often in GPU memory (VRAM), because it provides extremely high bandwidth. But VRAM is expensive and limited.</p><p>The conventional solution is to use GPUs with more memory or distribute the model across multiple GPUs. Two open-source projects — <a href="https://github.com/razor-ai/soup">Soup</a> and <a href="https://github.com/JustVugg/colibri">Colibrì</a> — approach the constraint differently. Instead of asking how to fit the entire model into fast memory, they effectively ask:</p><blockquote><strong><em>Does the entire model need to be in fast memory at the same time?</em></strong></blockquote><p>Soup answers this question at the <strong>layer level</strong>. Colibrì answers it at the <strong>expert level</strong> in Mixture-of-Experts models.</p><h3>2. Soup: stream layers through the GPU</h3><p>A Transformer processes data through a sequence of layers. Layer 1 runs before Layer 2, Layer 2 before Layer 3, and so on. <a href="https://github.com/razor-ai/soup">Soup</a> exploits this predictable execution order through <strong>layer streaming</strong>.</p><p>Instead of keeping the entire frozen base model in GPU memory, Soup can keep the model outside VRAM and feed the GPU one decoder layer at a time. Once a layer has finished computing, its output activation is preserved for the next layer while the GPU memory used for that layer’s weights can be reused.</p><h3>Soup’s layer-streaming flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7iMwvMkj-TTMu184t8TCsw.png" /></figure><p>This changes the memory requirement substantially. The GPU no longer needs space for every layer’s weights simultaneously; it needs enough space for the current working set.</p><p>The tradeoff is <strong>data movement</strong>. If the GPU finishes one layer and then waits while the next layer is fetched, memory has been saved at the expense of speed. Soup reduces this penalty through prefetching: because the execution order is predictable, the next layer can begin moving while the current layer is still computing.</p><p>Soup’s approach is particularly relevant to memory-efficient fine-tuning with <a href="https://arxiv.org/abs/2106.09685">LoRA</a> and <a href="https://arxiv.org/abs/2305.14314">QLoRA</a>. LoRA freezes the large pretrained model and trains much smaller adapter parameters. QLoRA additionally quantizes the frozen base model, further reducing its memory footprint.</p><p>Soup’s current documentation demonstrates layer-streamed fine-tuning of Llama-3.1–8B-Instruct on an RTX 3050 Laptop GPU with 4 GB of VRAM. The feature is still marked beta, so the important result here isn’t a particular throughput number; it is the architecture itself.</p><blockquote><strong><em>Soup’s key idea: the layers are needed sequentially, so their weights don’t all need to occupy GPU memory simultaneously.</em></strong></blockquote><h3>3. Colibrì: load the experts you need</h3><p><a href="https://github.com/JustVugg/colibri">Colibrì</a> starts from a different property of modern models: <strong>sparsity</strong>.</p><p>In a Mixture-of-Experts (MoE) model, some Transformer layers contain multiple alternative neural-network blocks called experts. A learned component called a <strong>router</strong> determines which experts should process the current token representation.</p><p>Suppose a layer contains 256 experts but the router selects only a few for a particular token. The remaining experts do not need to participate in that computation. This allows an MoE model to contain an enormous number of total parameters while activating far fewer parameters for each token.</p><h3>Colibrì’s expert-streaming flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tiNtHyOUAsvMyVOfdPv-Zw.png" /></figure><p>Colibrì exploits the difference between <strong>total parameters</strong> and <strong>active parameters</strong>. Its documentation describes GLM-5.2 as having roughly 744 billion total parameters while around 40 billion parameters are active per token.</p><p>Instead of requiring the entire expert bank to remain in fast memory, Colibrì treats NVMe storage, RAM and optional VRAM as a hierarchy. The dense portion of the model remains resident while routed expert weights can stay on disk and be brought into faster memory when needed. Frequently useful experts can remain cached.</p><p>At a high level, this is a <strong>caching problem</strong>. The difficult engineering lies in deciding which experts remain cached, which are evicted, what can be prefetched, how expert weights are arranged on storage, and how I/O can overlap with computation.</p><blockquote><strong><em>Colibrì’s key idea: an MoE model contains many experts, but only the selected experts need to participate in the current computation.</em></strong></blockquote><h3>4. Soup vs. Colibrì: when to use which</h3><p>Both approaches reduce the need to keep the entire model in fast memory, but they exploit fundamentally different properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnfdYSkAaT2Ex3NY07uymQ.png" /></figure><p>If your problem is <strong>“I want to fine-tune this model, but its frozen base doesn’t fit in GPU VRAM,”</strong> Soup’s approach is the more natural fit. It exploits the fact that Transformer layers execute sequentially and that the next layer is known in advance.</p><p>If your problem is <strong>“I want to run a huge MoE model, but its expert bank is much larger than my available fast memory,”</strong> Colibrì’s approach is more relevant. It exploits the fact that the router activates only a subset of the model’s experts for each token.</p><p>Neither approach makes the model smaller. <strong>They change where weights live and when those weights move.</strong></p><h3>5. The bigger systems idea</h3><p>The interesting part of Soup and Colibrì extends beyond these two projects. Their underlying problem is a classic computer-systems problem.</p><p>CPUs use cache hierarchies. Operating systems move data between memory and storage. Databases decide which pages deserve space in their buffer pools. All of them operate under the same constraint: <strong>the fastest memory is limited, so the system must decide what belongs there right now.</strong></p><p>Large neural networks are creating another version of that problem:</p><blockquote><strong><em>Out of all the parameters this model contains, which ones need to be close to compute right now?</em></strong></blockquote><p>Soup answers using the model’s <strong>sequential layer structure</strong>. Colibrì answers using the <strong>sparse expert structure of MoE models</strong>.</p><p>Different architectures. Different workloads. Same systems principle:</p><blockquote><strong><em>You don’t necessarily need everything in the fastest memory. You need the working set.</em></strong></blockquote><p>As AI models continue to grow, efficiently managing that working set may become just as important as increasing raw compute.</p><h3>Appendix: ML concepts</h3><p><strong>Parameter / weight</strong> — A numerical value learned during training. A model with 70 billion parameters contains roughly 70 billion learned values.</p><p><strong>Token</strong> — A unit of text processed by an LLM. It can represent a word, part of a word, punctuation or another text unit.</p><p><strong>Layer</strong> — One computational stage in a neural network. The output of one Transformer layer becomes the input to the next.</p><p><strong>Attention</strong> — The mechanism that lets token representations incorporate relevant information from other tokens. The Transformer architecture was introduced in <a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a>.</p><p><strong>Expert</strong> — One of multiple alternative neural-network blocks available inside certain layers of a Mixture-of-Experts model.</p><p><strong>Router</strong> — The learned component in an MoE layer that determines which experts process the current token representation.</p><p><strong>Inference</strong> — Running an already-trained model to generate predictions or outputs.</p><p><strong>LoRA</strong> — <a href="https://arxiv.org/abs/2106.09685">Low-Rank Adaptation</a>, a fine-tuning method that freezes the original model and trains much smaller adapter parameters.</p><p><strong>QLoRA</strong> — <a href="https://arxiv.org/abs/2305.14314">Quantized LoRA</a>, which combines LoRA with a quantized frozen base model to reduce memory requirements further.</p><h3>References &amp; Further Reading</h3><ol><li><strong>Soup</strong> — <a href="https://github.com/razor-ai/soup">GitHub</a></li></ol><p><strong>2. Soup layer streaming</strong> — Makazhan, A., <a href="https://doi.org/10.5281/zenodo.21771064"><em>Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU</em></a>, 2026.</p><p><strong>3. Colibrì</strong> — <a href="https://github.com/JustVugg/colibri">GitHub</a></p><p><strong>4. Colibrì Quick Start</strong> — <a href="https://github.com/JustVugg/colibri/blob/main/docs/quickstart.md">Running GLM-5.2 with streamed experts</a></p><p><strong>5. Transformers</strong> — Vaswani et al., <a href="https://arxiv.org/abs/1706.03762"><em>Attention Is All You Need</em></a>, 2017.</p><p><strong>6. Mixture of Experts</strong> — Fedus, Zoph &amp; Shazeer, <a href="https://jmlr.org/papers/v23/21-0998.html"><em>Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity</em></a>, 2022.</p><p><strong>7. LoRA</strong> — Hu et al., <a href="https://arxiv.org/abs/2106.09685"><em>LoRA: Low-Rank Adaptation of Large Language Models</em></a>, 2021.</p><p><strong>8. QLoRA</strong> — Dettmers et al., <a href="https://arxiv.org/abs/2305.14314"><em>QLoRA: Efficient Finetuning of Quantized LLMs</em></a>, 2023.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=859edc6a8bd9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy a Website and Wait 5 Minutes: The Bots Will Find You!]]></title>
            <link>https://sumedhakhatter.medium.com/deploy-a-website-and-wait-5-minutes-the-bots-will-find-you-66371e8c50f7?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/66371e8c50f7</guid>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[web-security]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 01:25:27 GMT</pubDate>
            <atom:updated>2026-03-16T01:25:27.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xq08JT5aUDqD0rsdMTXMAQ.png" /></figure><h3>Your Website Is Being Scanned Right Now</h3><h3>Understanding Automated Internet Bots and How to Protect Your Infrastructure</h3><p>If you deploy a website on the public internet and check your logs for the first time, you may notice something surprising. Requests like these often appear almost immediately:</p><pre>GET /xmlrpc.php<br>GET /wp-includes/wlwmanifest.xml<br>GET /blog/wp-includes/wlwmanifest.xml<br>GET /wordpress/wp-includes/wlwmanifest.xml<br>GET /wp/wp-includes/wlwmanifest.xml<br>GET /shop/wp-includes/wlwmanifest.xml<br>GET /cms/wp-includes/wlwmanifest.xml</pre><p>This raises obvious questions:</p><ul><li>Who is making these requests?</li><li>Is my website under attack?</li><li>Did my server leak somewhere?</li></ul><p>Most of the time, none of those things happened. What you are seeing is the <strong>background radiation of the internet</strong>: automated scanning bots probing servers for vulnerabilities.</p><h3>The Internet Is Constantly Being Scanned</h3><p>The moment a server becomes reachable through a public IP or domain, it becomes visible to automated scanners. Attackers run large-scale tools such as:</p><ul><li>Masscan</li><li>ZMap</li></ul><p>These tools can scan <strong>millions of IP addresses per second</strong>. Their objective is simple:</p><pre>discover servers<br>identify technologies<br>probe for vulnerabilities</pre><p>If a vulnerable system is found, attackers may attempt exploitation. Most of the time, however, scanners simply move on.</p><h3>Visual: How Internet-Wide Scanning Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*umiSewbSdxMPE9-W5c26fg.png" /></figure><p>Attackers rarely target a single system manually. Instead they <strong>scan huge portions of the internet automatically</strong>, searching for easy targets.</p><h3>Example: WordPress Discovery Scans</h3><p>One of the most common automated scans attempts to detect WordPress installations. Bots request well-known WordPress files:</p><pre>GET /xmlrpc.php<br>GET /wp-includes/wlwmanifest.xml</pre><p>These files exist in almost every WordPress deployment. If the server returns a valid response, the scanner confirms:</p><pre>WordPress detected</pre><p>The attacker may then attempt:</p><ul><li>login brute-force attacks</li><li>plugin vulnerability probes</li><li>credential harvesting</li></ul><p>If the endpoint does not exist, the bot stops scanning and moves on.</p><h3>Why Bots Probe Multiple Paths</h3><p>WordPress is often installed in subdirectories. Bots therefore test several common locations:</p><pre>/blog/<br>/wordpress/<br>/wp/<br>/site/<br>/cms/<br>/shop/<br>/test/</pre><p>This produces requests like:</p><pre>GET /blog/wp-includes/wlwmanifest.xml<br>GET /site/wp-includes/wlwmanifest.xml<br>GET /cms/wp-includes/wlwmanifest.xml</pre><p>The scanner is essentially asking: <em>“Is WordPress installed anywhere on this server?”</em></p><h3>Why Even Small Websites Get Scanned</h3><p>Developers often assume attackers only target large companies.</p><p>In reality, scanners sweep the entire internet continuously.</p><p>Even small or brand-new deployments quickly receive probes such as:</p><pre>GET /.env<br>GET /.git/config<br>GET /phpmyadmin<br>GET /admin<br>GET /server-status</pre><p>These are automated scripts looking for common configuration mistakes.</p><h3>Visual: Secure Web Architecture</h3><p>A secure production architecture places defensive layers between the internet and the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*mRAYhyz_uwex43eUZQC7DQ.png" /></figure><p>The goal is simple: <strong>Most malicious traffic should never reach the application layer.</strong></p><h3>How to Protect Your Website</h3><p>Automated scans cannot be completely stopped, but their impact can be minimized.</p><p>The goal is to make your infrastructure <strong>resilient and uninteresting to attackers</strong>.</p><h3>1. Use a Web Application Firewall</h3><p>A Web Application Firewall (WAF) filters malicious traffic before it reaches your server. Common platforms include:</p><pre>Cloudflare<br>Fastly<br>AWS CloudFront</pre><p>These services provide:</p><pre>bot detection<br>IP reputation filtering<br>automated attack mitigation<br>traffic rate limiting</pre><h3>2. Block Known Attack Endpoints</h3><p>If your application does not use certain endpoints, block them immediately. Example configuration:</p><pre>location ~* (xmlrpc.php|wlwmanifest.xml|wp-login.php) {<br>    return 444;<br>}</pre><p>This closes the connection instantly.</p><h3>3. Add Rate Limiting</h3><p>Public APIs should enforce request limits.</p><p>Example rule:</p><pre>5 requests per second per IP<br>burst limit of 10</pre><p>Rate limiting prevents abuse and reduces bot traffic.</p><h3>4. Protect Public Forms</h3><p>Contact forms and signup endpoints attract spam bots. Tools such as:</p><pre>Cloudflare Turnstile<br>Google reCAPTCHA<br>hCaptcha</pre><p>help distinguish human traffic from automated requests.</p><h3>5. Avoid Leaking Infrastructure Details</h3><p>Logs and error messages sometimes expose sensitive information.</p><p>For example:</p><pre>DATABASE_URL<br>JWT_SECRET<br>AWS_SECRET_ACCESS_KEY</pre><p>These should <strong>never appear in logs or responses</strong>. Logs should contain debugging information, not secrets.</p><h3>Final Thoughts!</h3><p>Operating a service on the public internet means interacting with automated systems constantly scanning for vulnerabilities. Most scans are harmless reconnaissance. The real objective of secure infrastructure is simply this:</p><pre>When scanners probe your system, they find nothing useful.</pre><p>When your system blocks unnecessary endpoints, enforces rate limits, and avoids leaking sensitive data, automated bots quickly move on. The safest server is often the one that <strong>looks boring to attackers</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66371e8c50f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed System Design Patterns]]></title>
            <link>https://sumedhakhatter.medium.com/distributed-system-design-patterns-a6ec071c4e60?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6ec071c4e60</guid>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Sat, 07 Mar 2026 03:09:22 GMT</pubDate>
            <atom:updated>2026-03-07T03:09:22.930Z</atom:updated>
            <content:encoded><![CDATA[<p>A lot of times, system design problems seem unrelated but looking at enough of these problems, we can say that these problems revolve around small set of architectural patterns. This article walks through some of the core backend architecture patterns.</p><h3>1. Request Response Systems:</h3><p>This is the most common architecture, system receives a request, processes it and returns a response.</p><p><strong>Architecture:</strong></p><pre>Client -&gt; Load Balancer -&gt; API Server -&gt; Database</pre><p><strong>Problems:</strong></p><pre>a. URL Shortener <br>b. Pastebin</pre><p><strong>Key Design Considerations:</strong></p><pre>a. API design <br>b. database schema <br>c. indexing strategy <br>d. pagination <br>e. rate limiting <br>f. authentication and authorization</pre><h3>2. Caching Systems (Read Heavy Workloads)</h3><p>Systems that receive a way more reads than writes and it is expensive to hit the database directly for each read request. Eg: Product Catalogs, Social Feeds, Leaderboards, Trending Content</p><p><strong>Architecture:</strong></p><pre>Client -&gt; API -&gt; Cache (Redis/ Memcached) -&gt; Database</pre><p><strong>Problems:</strong></p><pre>a. Twitter Timeline <br>b. Instagram Feed <br>c. Reddit Front Page <br>d. Product Catalog <br>e. Game Leaderboards</pre><p><strong>Key Design Considerations:</strong></p><pre>a. Cache- Aside Pattern <br>b. Cache Invalidation <br>c. TTL Policies<br>d. Hot-Key Mitigation</pre><h3>3. Asynchronous Job Processing:</h3><p>These operations are too slow to run inside a request. Eg: Sending emails, video processing, file conversions, analytics aggregation. Hence instead of blocking the system with the request, system schedules work in a background queue.</p><p><strong>Architecture:</strong></p><pre>API -&gt; Message Queue -&gt; Worker Pool -&gt; Processing</pre><p><strong>Problems:</strong></p><pre>a. Email Notification Service <br>b. Image Processing Service <br>c. Video Transcoding System <br>d. File Conversion Pipeline <br>e. Leetcode - code execution</pre><p><strong>Key Design Considerations:</strong></p><pre>a. retries <br>b. idempotency <br>c. dead letter queues <br>d. worker autoscaling</pre><h3>4. Event Driven Architecture</h3><p>In these large systems, service communicate through events rather than direct API calls. Instead of one service calling another, a service publishes an event that multiple consumers can react to.</p><p><strong>Architecture:</strong></p><pre>Producer -&gt; Event bus (Kafka/ Pub- Sub) -&gt; Consumers</pre><p><strong>Problems:</strong></p><pre>a. Logging Pipelines <br>b. Activity Tracking Systems <br>c. Analytics Ingestion <br>d. Ad- Click Aggregators <br>e. Fraud Detection Pipelines</pre><p><strong>Key Design Considerations:</strong></p><pre>a. Loose Coupling<br>b. Easier Scaling <br>c. Fault Isolation</pre><h3>5. Fanout Systems</h3><p>Systems distributing one event to many recipients. Eg: when a user posts content on social platform, that content must appear in the feeds of all the followers.</p><p><strong>Architecture:</strong></p><pre>User Event -&gt; Fanout Service -&gt; Followers</pre><p><strong>Problems:</strong></p><pre>a. Facebook News feed<br>b. Twitter timeline <br>c. Notification Systems <br>d. Chat Messaging Systems</pre><p><strong>Key Design Considerations:</strong></p><pre>Fanout on Write: When a user posts content, the system pushes it into each follower&#39;s feed. <br>Fanout on Read: System builds the feed dynamically when the user opens the app.</pre><h3>6. Streaming and Real Time Aggregation</h3><p>These are systems where a continuous stream of data is being processed.</p><p><strong>Architecture:</strong></p><pre>Event Producer -&gt; Kafka/ Stream Broker -&gt; Stream Processing (Flink/ Spark) -&gt; Aggregated Storage</pre><p><strong>Problems:</strong></p><pre>a. Ad Click Tracking/Aggregation <br>b. Trending Topics <br>c. Top K Systems <br>d. Analytics Dashboard</pre><p><strong>Key Design Considerations:</strong></p><pre>a. Windowed Aggregation <br>b. High throughput Ingestion <br>c. Partitioning <br>d. Deduplication</pre><h3>7. Workflow / Saga Systems</h3><p>These systems involve multiple dependent steps across services. Modern workflow engines such as Temporal, Step Functions are often used to manage these processes.</p><p><strong>Architecture:</strong></p><pre>Workflow Engine -&gt; Activities -&gt; External services <br>Payments Workflow: Order Placed -&gt; Charge Payment -&gt; Reserve Inventory -&gt; Ship Order</pre><p><strong>Problems:</strong></p><pre>a. Payment Systems <br>b. Ticket Booking Systems <br>c. Ride Hailing Lifecycle <br>d. Food Deliver Systems <br>e. Travel Booking Systems</pre><h3>8. Mixed Systems</h3><p>Real world systems however are a blend of several design patterns.</p><pre>Leaderboard: Caching + Request Response + Streaming <br>Leetcode: Request Response + Async Jobs + Workflow <br>Twitter: Fanout + Caching + Async Jobs + Events <br>Ad Click Aggregator: Events + Streaming <br>Doordash: Workflow/ Saga + Events <br>Dropbox: Async Jobs + Events + workflows <br>Chat: Fanout + Events <br>Top-K Songs: Streaming + Caching<br>Ticket Booking: Workflow + Concurrency Control</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6ec071c4e60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[Event Ingestion] Your Code Isn’t Simple — Production Just Hasn’t Happened Yet]]></title>
            <link>https://sumedhakhatter.medium.com/event-ingestion-your-code-isnt-simple-production-just-hasn-t-happened-yet-af3559505868?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/af3559505868</guid>
            <category><![CDATA[idempotency]]></category>
            <category><![CDATA[event-ingestion]]></category>
            <category><![CDATA[durability-queues]]></category>
            <category><![CDATA[network-failure]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Sat, 28 Feb 2026 09:56:59 GMT</pubDate>
            <atom:updated>2026-02-28T09:56:59.567Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wW7Ne769KDm8jXWk4kEqCQ.png" /></figure><blockquote>Designing Mobile Client–Server Event Ingestion Systems for Real-World Failure</blockquote><p>In theory, client-server systems look easy and deterministic. A client sends a request, the server processes it, and a response comes back. Wow, such a short happy story!</p><p>Alas, but in production, this is just an illusion. Your code isn’t running on a single machine — it is running across millions (or billions) of devices, unreliable networks, and distributed infrastructure. What looks like a single request is usually a workflow that crosses several systems, each with its own ways of failing.</p><p>There’s a well-known idea from<a href="https://www.youtube.com/watch?v=VvUdvte1V3s"> <em>Six Little Lines of Fail</em></a> that captures this perfectly: small orchestration logic can create surprisingly large failure modes once it hits the real world. But the takeaway isn’t about one example — it’s about how distributed client-server systems actually behave and how we can build resilient systems.</p><h3>Defining the system: client and backend server</h3><p>In this context:</p><ul><li><strong>Client</strong> → an application (Android or iOS or an ipad or your browser) running on a network</li><li><strong>Server</strong> → the backend system that owns business logic, persistence, and integrations</li></ul><h3>1. Where Things Break on the Client</h3><p>The client — whether it’s an Android app, iOS app, or browser — lives in a chaotic environment. Unlike backend systems, it doesn’t control its network, runtime, or even how long it stays alive.</p><h4>Connectivity is unpredictable</h4><p>Devices constantly move between Wi-Fi and cellular. A request can start on one network and disappear before it reaches the server.</p><h4>Apps don’t always finish what they start</h4><p>Mobile operating systems aggressively manage memory and battery. An app can be backgrounded or killed mid-request, leaving events half-sent.</p><h4>Timeouts create duplicates</h4><p>If the client doesn’t hear back quickly, it retries. Meanwhile, the original request might still be processing, which is how duplicates quietly appear.</p><h4>Events can vanish locally</h4><p>If events are only stored in memory, a crash or restart can erase them before they’re ever sent.</p><p>Because of this, client delivery is never guaranteed. The best you can do is design for resilience:</p><p><strong>a. Store events locally before sending: </strong>Eg: Storage options for Mobile could be SQLite / Room / CoreData, for browser → IndexedDB / localStorage (less reliable), SDKs → local file or embedded DB. If the app crashes or loses connectivity, events stored in durable local storage can be retried later instead of being permanently lost.</p><p><strong>b. Retry with backoff: </strong>Backoff is not just “wait longer.” It solves a specific distributed systems problem: <strong>synchronized retries overwhelming the backend. </strong>Typical retry flow:</p><pre>Send event → If failure → wait → Increase wait exponentially → Add jitter (randomization) → Retry until success or TTL expires</pre><pre>Example: <br>1s → 2s → 4s → 8s → 16s (± random jitter)</pre><p>This improves reliability, prevents retry storms during outages, allows temporary network failures to recover, avoids draining battery/network aggressively. Backoff doesn’t prevent loss — it <strong>increases the chance the event eventually reaches the server.</strong></p><p><strong>c. Attach unique IDs (Idempotency Keys)</strong></p><p>A unique ID (aka idempotency key) solves a different problem: duplicates. Because retries are inevitable, the backend must be able to know if it has already processed the event.</p><p>How it works:</p><pre>Client generates a UUID -&gt; Server stores processed IDs (or deduplicates in the ingestion pipeline) -&gt; If the same ID arrives again then ignore or merge</pre><p>So IDs don’t prevent failures — they make retries <strong>safe</strong>.</p><p><strong>d. Send events in batches: </strong>Batching means sending multiple events in one request. This means less network trips, lesser connections → lower cumulative risk. And if the device briefly reconnects, a batch can flush many events at once. It has less TLS (Transport layer Security) handshakes, less chances of failure. While a failed batch can affect multiple events, batching reduces the overall probability of failure by minimizing connection overhead and allowing quick flushes when connectivity resumes.</p><h3>2. Where Things Break on the Server</h3><p>Reaching the backend doesn’t mean the problem is solved. A single API call often fans out into a chain of actions — writing to storage, calling other services, publishing events, triggering async jobs. That’s where another complexity really lives.</p><h4>Workflows don’t fail cleanly</h4><p>Some steps succeed while others don’t, leaving the system in a half-completed state.</p><h4>Background processing isn’t guaranteed</h4><p>Queues, workers, and async pipelines can lag or fail, creating temporary inconsistencies across systems.</p><h4>Infrastructure has limits</h4><p>Traffic spikes, deployments, or resource contention can introduce latency and timeouts even when everything is “working.”</p><h4>Retries can repeat side effects</h4><p>If operations aren’t idempotent, duplicate requests can create duplicate records, charges, or messages.</p><p>Reliable backend design is less about preventing failure and more about ensuring the system eventually reaches the right state. That’s why resilient systems rely on:</p><p><strong>a. Idempotency: </strong>This can be obtained using Idempotency keys, natural keys (using primary keys), dedup of tables, upserts instead of inserts</p><p><strong>b. Durable queues or logs: </strong>This entails decoupling requests from processing of requests through queues such as Kafka, Pulsar etc. If the processing fails, the request remains in the queue and can be retried. And if the retried events eventually process succesfully, we get delayed processing instead of absolute failure.</p><p><strong>c. Observability to detect drift: </strong>Even with all safeguards, distributed systems eventually drift — events get stuck, queues lag, or states become inconsistent. Without visibility, these issues remain silent. but signals that reflect system correctness: Metrics (ingestion rate vs processing rate), Lag monitoring, Error budgets, Dead-letter queues, Reconciliation jobs.</p><p><strong>d. Explicit workflow states: </strong>Explicit states make progress observable and resumable. Instead of a boolean success flag, you model transitions.</p><pre>RECEIVED → VALIDATED → PERSISTED → PROCESSED → COMPLETED</pre><p>Each step is written to storage. This is essentially turning workflows into state machines instead of linear procedures and prevents the classic “half-finished workflow” problem where the system doesn’t know whether to retry or roll back.</p><p>The goal is convergence, not perfection.</p><h3>3. Where Things Break Between Client and Server</h3><p>Not all failures belong to the client or the backend. Many happen in the space between them — along the communication path that carries requests across networks and infrastructure.</p><h4>Networks drop things</h4><p>Once a request leaves a device, it travels through multiple hops — wireless links, local routers, ISP networks, internet backbone routers, and data center edge infrastructure — before reaching the server. At any of these points, packets can be delayed or dropped due to congestion, weak signal, routing changes, or connection resets.</p><p>TCP usually retransmits lost packets automatically, but if the disruption lasts long enough to exceed timeouts, the request fails. This is why distributed systems treat the network as inherently unreliable and rely on retries and idempotency rather than assuming guaranteed delivery.</p><h4>Secure connections can fail</h4><p>Before any request is sent, a TLS(Transport Layer Security) handshake must succeed. If this handshake fails — due to certificate issues, network interference, or connection timeouts — the request never actually leaves the client application layer.Infrastructure layers introduce their own risks. Requests pass through DNS, CDNs, load balancers, etc. Each hop adds another place where things can go wrong.</p><h4>Responses can disappear</h4><p>Sometimes the server completes the operation successfully, but the response is lost on the way back. From the client’s perspective, this looks identical to the request never having been processed at all.</p><blockquote>This ambiguity — <strong>not knowing whether an operation failed or merely its acknowledgement failed</strong> — is one of the fundamental challenges of distributed systems. It isn’t a bug in the network; it’s a property of operating across independent systems without a shared state.</blockquote><h4>Techniques that reduce the likelihood of failure or make failures harmless</h4><p>These techniques don’t eliminate network failures, but they reduce their frequency and make them harmless when they occur.</p><p><strong>a. Connection reuse and pooling: </strong>Keeping long-lived connections (HTTP/2, gRPC) avoids repeated handshakes and reduces chances of failure during connection setup.</p><p><strong>b. Multi-region and edge routing: </strong>Using CDNs or geographically distributed endpoints shortens network paths, reducing latency and packet loss probability.</p><p><strong>c. Health checks and adaptive routing:</strong> Traffic is routed away from unhealthy servers or congested regions automatically.</p><p><strong>d. Retries with backoff: </strong>If a request fails due to transient network issues, retrying gives it another chance to succeed.</p><p><strong>e. Idempotency: </strong>Ensures retries don’t create duplicate side effects when the original request actually succeeded.</p><p><strong>f. Acknowledgement after persistence: </strong>The server confirms only after the request is durably stored, so clients know the event won’t be lost.</p><p><strong>g. End-to-end reconciliation: </strong>Background jobs compare client and server state to detect and repair missing data. These patterns assume the network will fail — and design around it.</p><p><strong>h. Retry-friendly timeouts: </strong>Proper timeout configuration avoids premature failures while still detecting real issues. These improve reliability, but they don’t eliminate uncertainty.</p><h3>Let’s conclude!</h3><p>Client-server architectures like these live in a world where partial failure is normal. Networks are unreliable, devices are ephemeral, and systems are distributed by default.</p><p>The systems that scale aren’t the ones that assume everything will go right — they’re the ones designed so that even when things go wrong, the system still finds its way to the correct state.</p><p>In distributed systems, simplicity isn’t measured by how small the code is, but by how predictable the system remains under failure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af3559505868" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solve the “hot key” (a.k.a. hotspot key) in a cache/store like Redis]]></title>
            <link>https://sumedhakhatter.medium.com/solve-the-hot-key-a-k-a-hotspot-key-in-a-cache-store-like-redis-1134cf5d2cb1?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/1134cf5d2cb1</guid>
            <category><![CDATA[hotkeys]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[hot-key-issues]]></category>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[hotspot-key]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 07:00:54 GMT</pubDate>
            <atom:updated>2025-09-30T07:00:54.872Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Problem Statement: </strong>One key gets hammered by huge, concurrent traffic so all requests land on the same hash-slot/node and that node chokes. Here’s a compact toolbox you can apply, with trade-offs and example patterns.</blockquote><h3>Identify the Pattern:</h3><p>Before jumping into solutions, find the pattern.</p><ol><li><strong>Read-heavy &amp; stable value? </strong>Are the requests read heavy and the value of the key remains almost the same</li><li><strong>Regenerates from DB/API on miss?</strong> When the cache doesn’t have the key anymore (TTL expired), the system recomputes it from the database/API. If lots of requests miss at once, they all try to “regenerate” at once — hitting your <strong>DB or API</strong> at massive scale. This creates an avalanche of load, possibly crashing the DB/API or dramatically spiking latency.</li><li><strong>Write-heavy counter/rate-limit? </strong>Are the requests write heavy, eg: the same key is getting incremented or decremented?</li></ol><p>After identifying the patterns out of the above three, we can jump into the solutions for each pattern.</p><h3>1) Read hotspots — spread the load</h3><h4>A) Client-side caching:</h4><p>Caching the key on the client side itself. If the key exists in the client side memory, then use it. If not then send a request to redis.</p><pre>def get_hot(key):<br>   v = local_cache.get(key)<br>   if v is not None:<br>      return v<br>   v = redis.get(key)<br>   local_cache.set(key, v, ttl=1_000) # short TTL as extra guard<br>   return v</pre><h4>B) Read replicas / fan-out reads</h4><p>Spread the GET requests across replicas instead of hammering the primary.</p><blockquote>With the Redis setting replica-read-only yes, these replicas accept read traffic while writes still go only to the primary. A smart client or proxy can then route GET requests to the replicas, spreading the load, and direct all SET or UPDATE operations to the primary. The trade-off is that replicas are <strong>eventually consistent</strong> because replication is asynchronous, so there can be a small lag before a write on the primary appears on a replica — but this is generally acceptable for caching scenarios where slightly stale data is tolerable.</blockquote><h4>C) Duplicate the key to multiple physical keys (“key mirroring”)</h4><p>Instead of storing a hot value in just one Redis key, you can mirror it across multiple keys, such as hot:K#1 … hot:K#N.</p><p>On every write, the same value is fanned out to all N keys so they stay in sync, and clients are instructed to spread their reads by computing something like hash(user_id) % N to pick which copy to use. This approach gives nearly linear read scalability with the number of copies, since traffic is distributed evenly, and it avoids the complexity of client tracking.</p><p>The trade-off is that each write now costs N times more work, because it must update every copy, and slight inconsistencies can appear if one copy updates before another. To mitigate this, writes are often pipelined or coordinated with Redis commands like WAIT, but the system still accepts that replicas may be only eventually consistent.</p><pre># write<br>for i in range(N):<br>    pipe.set(f&quot;hot:K#{i}&quot;, value, ex=ttl)<br>pipe.execute()<br><br># read<br>i = hash(user_id) % N<br>return redis.get(f&quot;hot:K#{i}&quot;)</pre><h3>D) Front caches: CDN or edge/app cache</h3><p>If the frequently accessed (“hot”) value you’re serving is something like an HTTP/JSON response, you can offload pressure from Redis by caching it closer to the client — either with a CDN or an application-level LRU cache.</p><p>These caches can be configured with a <strong>stale-while-revalidate</strong> policy, which serves slightly stale data to users while refreshing the value in the background, preventing traffic spikes from overwhelming the backend when the cache expires.</p><p>To make this even more robust, you randomize (or “jitter”) the cache’s time-to-live (TTL) values so they don’t all expire at the same moment, which avoids a surge of simultaneous cache misses that could otherwise create a thundering herd effect on Redis or your database.</p><h3>2) Stampede (thundering-herd) control on refresh/miss</h3><p>When a hot key in a cache expires or is missing, the danger is that millions of clients may simultaneously try to recompute or reload the same value, overwhelming the backend. To prevent this “thundering herd,” several strategies exist:</p><ol><li><strong>Single-flight / request coalescing</strong>: Ensures that only one worker recomputes the value while others wait (e.g., Go’s singleflight.Group or a per-key lock in Python).</li><li><strong>Soft-TTL + stale-while-revalidate</strong>: Allows serving a slightly stale value after expiration while a background refresh updates the cache.</li><li><strong>Mutex lock in Redis </strong>serialize regeneration attempts</li><li><strong>Probabilistic early expiration</strong> (aka cache decay), where different clients refresh at slightly randomized times before the TTL, smearing the load and avoiding synchronized recomputation.</li></ol><pre>#Implementing a distributed lock around recomputing a hot key in Redis<br><br>k = redis.set(&quot;lock:hot:K&quot;, &quot;1&quot;, nx=True, px=5000) # set a lock key with a short TTL<br>if ok:<br> val = recompute()<br> pipe = redis.pipeline()<br> pipe.set(&quot;hot:K&quot;, val, ex=60)<br> pipe.delete(&quot;lock:hot:K&quot;)<br> pipe.execute()<br>else:<br> # someone else recomputes; either wait briefly or serve stale<br><br><br>&quot;&quot;&quot;<br> nx=True means “only set if not exists,” so only one client succeeds.<br> px=5000 sets an expiry to prevent deadlocks if the process crashes.<br> If the lock is acquired (ok is truthy), this client is the one allowed to recompute the expensive value:<br> It calls recompute() to generate the fresh value. It then uses a Redis pipeline to:<br>   set(&quot;hot:K&quot;, val, ex=60) → store the new hot value with a 60-second TTL.<br>   delete(&quot;lock:hot:K&quot;) → release the lock.<br>   execute() → run both commands atomically in the pipeline.<br>   If the lock was not acquired, it means another client is already recomputing. <br>   In that case, this client either waits briefly for the new value to appear in Redis or falls back to serving a stale copy.<br><br>In short: this code ensures that only one worker at a time recomputes the hot key, <br>while others back off, preventing a dogpile on the backend.<br><br>&quot;&quot;&quot;<br></pre><h3>3) Write hotspots (counters, rate limits, sets)</h3><p>If you have a <strong>single key</strong> that is being updated very frequently with operations like INCR (incrementing a counter) or SETNX (set-if-not-exists), all that traffic is funneled to the one hash slot and one node in the Redis cluster. Because Redis is single-threaded per core, that node becomes overwhelmed — in other words, it can <strong>“melt”</strong> (get saturated and fail to keep up)</p><h4>A) Striping with Sharded Counters</h4><p>Striping: Instead of updating just one key, you <strong>spread the load across multiple keys</strong> (e.g., counter:1, counter:2, … counter:N) and pick which stripe to increment using something like hash(user_id) % N. When you need the total value, you sum across the stripes. This balances writes across multiple hash slots/nodes, so no single node becomes a bottleneck.</p><p><strong>Use Shards:</strong></p><p>Maintain hot:cnt#0 … hot:cnt#(M-1); on increment pick a random shard; <strong>read = SUM of shards</strong>. This keeps each write O(1) to one shard; read aggregation can be cached or done periodically.</p><pre># write (increment one shard)<br>idx = randrange(M)<br>redis.incrby(f&quot;hot:cnt#{idx}&quot;, 1)<br><br># read<br>total = sum(redis.mget([f&quot;hot:cnt#{i}&quot; for i in range(M)]))</pre><p>Use <strong>HyperLogLog</strong> or <strong>Count-Min Sketch</strong> to avoid heavy reads.</p><h4>B) Token buckets / rate limiting</h4><p>A <strong>token bucket</strong> is a common algorithm for rate limiting. Each “bucket” holds tokens that represent how many actions a user (or service) can perform. Every request consumes a token; tokens are replenished over time. If the bucket is empty, the request is denied or delayed. The key design advice here is:</p><ol><li><strong>Per-user / per-bucket keys</strong>: Instead of putting <em>all requests</em> into a single global counter (like global:bucket), you create a separate Redis key per user (e.g., bucket:user123). This avoids millions of requests hammering one hot key.</li><li><strong>If you truly need a global rate limit</strong> (say, system-wide QPS), then you <strong>stripe the bucket</strong> into multiple keys (bucket:0 … bucket:N-1) and when a request comes in, you probabilistically pick one shard to decrement. This spreads the writes across multiple Redis slots/nodes. To know if the global bucket is empty, you aggregate across shards.</li></ol><h3>Cheat sheet</h3><ol><li><strong>Read-mostly + can accept slightly stale</strong> → <em>Client tracking</em> <strong>+</strong> <em>soft-TTL + SWR</em>.</li><li><strong>Read-mostly + need strong freshness</strong> → <em>Client tracking invalidations</em> or <em>replicas with WAIT on writes</em>.</li><li><strong>Massive global counter</strong> → <em>Sharded counter</em> (M shards), <em>SUM on read</em> (and cache the sum).</li><li><strong>Miss triggers heavy recompute</strong> → <em>Single-flight + mutex + jittered TTL</em>.</li><li><strong>Still too hot</strong> → <em>Duplicate key mirroring (N copies)</em> or <em>CDN/app LRU</em>, and/or <em>slot isolation with more replicas</em>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1134cf5d2cb1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SYSTEM DESIGN: Choosing the suitable database for your product.]]></title>
            <link>https://sumedhakhatter.medium.com/system-design-choosing-the-suitable-database-for-your-product-206fbbb8207?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/206fbbb8207</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 05:22:46 GMT</pubDate>
            <atom:updated>2025-09-10T00:50:20.014Z</atom:updated>
            <content:encoded><![CDATA[<p>This article will guide you in choosing the right database for your system design.</p><h3>Purpose bifurcation:</h3><p>We rely on data stores to serve various purposes. However, we can bifurcate these into 2 basic ones:</p><p><strong>1. Operational Data Storage:</strong> Essential for application functionality, this type of data storage is utilized during the app’s runtime, supporting its core operations.</p><p><strong>2. Analytics Data Storage: </strong>Dedicated to storing data for analysis, this type of data storage enables us to extract valuable insights, trends, and patterns, driving informed decision-making. These are sometimes also called as warehouses, as they store a vast amount of historical data which is further used for Online Analytical Processing (OLAP).</p><p>In the following sections, discover the various types of databases, their distinct characteristics, and the specific use cases they cater to. Learn how different databases can effectively support diverse applications and scenarios.</p><h3><strong>Relational Databases</strong></h3><p>These databases allow users to define relations amongst different tables. The data is stored in a highly structured way that helps in fast query response times. RDBMS provides features like- typed columns, secondary indexes, transactions, advanced query language. The following examples fall under this category:</p><pre>MySQL<br>Microsoft SQL Server<br>PostgreSQL<br>SQLite<br>MariaDB<br>AmazonRDS<br>Amazon Aurora<br>AmazonRedshift</pre><h4>Usecases:</h4><p>It is typically recommended to use relational databases where ACID properties are necessary, such as in Payment Systems.</p><p>ACID stands for<strong> Atomicity, Consistency, Isolation and Durability</strong>. ACID transactions guarantee that each read, write or modification of a table has following properties:</p><ol><li><strong>Atomicity: </strong>Ensures that database transactions are atomic, meaning they are treated as a single, indivisible unit of work. If any part of the transaction fails, the entire transaction is rolled back.</li><li><strong>Consistency: </strong>Ensures that the database remains in a consistent state, even after multiple transactions have been applied.</li><li><strong>Isolation:</strong> Ensures that multiple transactions can occur concurrently without interfering with each other.</li><li><strong>Durability:</strong> Ensures that once a transaction has been committed, its effects are permanent and cannot be rolled back.</li></ol><p>These properties ensure that database transactions are reliable, secure, and maintain data integrity.</p><pre>Mobile applications<br>Web Applications<br>Gaming<br>Enterprise Applications<br>Chat Applications</pre><h4>Are Relational Databases distributed databases?</h4><p><strong>Relational databases (like a traditional MySQL) are not typically distributed.</strong> However, there are now multiple approaches to constructing relational databases as distributed systems or avoid a single point of failure in relational databases.</p><blockquote>Example: <a href="https://en.wikipedia.org/wiki/MySQL_Cluster">MySQL cluster</a> is a distributed, multi-master database, which does not have single point of failure and complies with ACID architecture. It follows automatic sharding to scale out read and read and write operations.</blockquote><p>Another approach to scale the MySQL is what Facebook Messenger did.</p><pre><strong>What databases are used for facebook messenger?</strong><br><br>So in general MySql has its own storage, called as InnoDB. InnoDB is optimized<br>for balanced reads + writes, but uses more space. MyRocks (Facebook&#39;s choice) = <br>optimized for very write-heavy workloads (Messenger, News Feed, Ads logs) <br>and storage efficiency. <br><br>Additionally, there is a memcache tier for fetching historical messages <br>quickly. Hence it&#39;s a globally distributed, sharded MySQL cluster running <br>MyRocks, fronted by a giant Memcache tier.</pre><h3>NoSQL Databases</h3><p>NoSQL is a general term meaning that database is not an RDBMS which supports SQL as its primary access language. Join Operations are generally not a core feature of non-relational databases. These databases are right choices if:</p><ol><li>Application requires super low latency.</li><li>Data is not structured</li><li>The volume of data is massive</li><li>The data needs to be serialized and deserialized (JSON, XML, YAML etc).</li></ol><pre><strong>But there is a twist.</strong> <br><strong>We can query NoSql Databases using distributed Sql Query Engines</strong>, example <br>Trino (Presto) and write a lot of complex queries and do complex joins. <br>It can query data wherever it lives, including:<br>1. Hive tables on HDFS<br>2. S3 data in Parquet/ORC/CSV<br>3. Relational databases (MySQL, Postgres)<br>4. NoSQL stores (Cassandra, MongoDB)<br><br><br>The actual query execution happens in Trino and not the databases beneath it.</pre><h3>Types of NoSQL Databases</h3><p>NoSQL databases can be of the following types:</p><ol><li><strong>Document Databases</strong></li><li><strong>Key-Value Store</strong></li><li><strong>Column Oriented databases</strong></li><li><strong>Graph Databases</strong></li><li><strong>Object Storage</strong></li></ol><p>These databases store data in XML, JSON, BSON or XML documents. The documents can be nested and indexing can be done for faster querying.</p><p>Examples:</p><pre>MongoDB</pre><p>Usecases:</p><pre>ecommerce platforms<br>trading platforms<br>mobile app development</pre><p><strong>2. Key-Value Store</strong></p><p>Every data element is stored in a key-value pair containing a name (key) and an attribute mapped to the key.</p><p>Examples</p><pre>Amazon DynamoDB </pre><p>Use cases</p><pre>Shopping Carts<br>User Profiles<br>User Preferences<br>Chat System<br>Rate Limiter (Redis is a good option)</pre><p><strong>3. Column Oriented databases | C-Store</strong></p><p>The data in column stores is stored as per set of columns. These databases are optimized for “read” purposes. For heavy-update OLTP use cases, column stores are not a good option. For “write once” — “heavy reads”, a c-store is a good option.</p><p>Examples</p><pre><strong>Apache Cassandra:</strong> A distributed wide column store in house at Meta for Inbox Search System. Cassandra was written to manage structured data and scale to a very large size across multiple servers. The infrastructure is built on top of hundreds of nodes spread across many datacenters. </pre><pre><strong>Redshift</strong><br><strong>BigQuery</strong><br><strong>Snowflake<br>Apache Hbase<br></strong></pre><p>Usecases</p><pre>Analytics<br>Inbox Search System<br>Archives<br>Datawarehouse Systems</pre><p><strong>4. Graph Databases</strong></p><p>The data is stored in form of nodes and maintains relationships with other nodes. The database is optimized for maintaining the relationships.</p><p>Examples of Graph Databases:</p><pre>Amazon Neptune<br>Neo4j<br>TAO</pre><pre><strong>More about TAO</strong><br><strong>TAO </strong>is a social graph database developed by Meta to efficiently manage and serve its massive social graph, which encompasses billions of users and trillions of relationships. It is a distributed data store optimized for<strong> read-heavy workloads </strong>typical of social networks. It provides a <strong>graph abstraction layer</strong> that allows for efficient retrieval of social graph data, such as user relationships, likes, comments, and other interactions.</pre><pre>TAO utilizes <strong>MySQL</strong> as its persistent storage layer. However, TAO is not merely a wrapper around MySQL; it is a custom-built system that mediates access to MySQL and incorporates its own <strong>graph-aware caching layer</strong>. This design allows TAO to efficiently manage the social graph at scale, addressing the limitations of traditional relational databases in handling complex, interconnected data</pre><p>Usecases</p><pre>Social Media Networks<br>Fraud Detections<br>Knowledge Graphs<br>Make in game-offer recommendations</pre><p><strong>5. Object Storage :</strong></p><p>Data is stored as an object (photos, videos) rather than splitting into columns. Metadata is stored alongside the object. You usually query by object ID rather than scanning tables</p><p>Examples:</p><pre><br>Amazon S3<br>Google Firebase<br>Facebook&#39;s Haystack</pre><pre><strong>Haystack:</strong> It is an object storage system optimized for facebook&#39;s photos application. Users upload over more than 1B of photos each week and facebook serves over 1M images per second at peak. </pre><pre>This is better than using network attached storage appliances (NAS- Network Attached Storage) over NFS (Network File Server). One significant cost in this approach is reading a file&#39;s metadata into memory in order to retreive the file itself. And when it is multiplied over billions of photos and PBs of data, accessing data is throughput bottleneck. Several disk operations are required to read a single photo. This became bottleneck and hence the following solutions were built:<br></pre><pre>1. Instead of one photo = one file, Haystack stores millions of photos inside large objects called Haystack Stores. </pre><pre>2. Metadata (photo ID → offset, size, location) is stored in a <strong>compact in-memory index</strong> maintained by Haystack.</pre><pre>3. So now fetching a photo is just:<br>a. look up offset in the memory index<br>b. seek directly to the data inside the big file. </pre><p>Usecases:</p><pre>Images<br>Videos</pre><h3>References:</h3><ol><li><a href="https://scontent-sjc3-1.xx.fbcdn.net/v/t39.8562-6/240824628_1997495913759105_8427131384780116846_n.pdf?_nc_cat=107&amp;ccb=1-7&amp;_nc_sid=ad8a9d&amp;_nc_ohc=wZPYUL6nawQAX-Zu3Zs&amp;_nc_ht=scontent-sjc3-1.xx&amp;oh=00_AfCet2UkfWm-jLR_UtKWaT6h7ZsxFXU39tJUtfuwKr6Igw&amp;oe=63619056">Finding a needle in the haystack: Facebook’s Photo Storage</a></li><li><a href="https://aws.amazon.com/startups/start-building/how-to-choose-a-database/">Maximizing performance with AWS Databases</a></li><li><a href="https://docs.aws.amazon.com/whitepapers/latest/aws-overview/database.html#compare-database-services">AWS Databases</a></li><li><a href="https://scaleyourapp.com/what-database-does-facebook-use-a-1000-feet-deep-dive/#:~:text=MySQL%20is%20the%20primary%20database,of%20MySQL%20as%20a%20cache.">A thorough insight into the databases used at Facebook.</a></li><li><a href="https://scaleyourapp.com/what-is-data-ingestion-how-to-pick-the-right-data-ingestion-tool/">ScaleYourApp: What is data ingesion and how to pick the right data ingestion tool.</a></li><li><a href="https://engineering.fb.com/2009/06/15/web/chat-reaches-1-billion-messages-sent-per-day/">Chat reaches 1 billion messages sent per day</a></li><li><a href="https://www.mongodb.com/scale/types-of-nosql-databases">MongoDB: Types of Databases</a></li><li><a href="https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf">Scaling MemCache at Facebook</a></li><li><a href="https://en.wikipedia.org/wiki/MySQL_Cluster">MySQL Cluster</a></li><li><a href="https://www.databricks.com/glossary/acid-transactions#:~:text=ACID%20is%20an%20acronym%20that,operations%20are%20called%20transactional%20systems.">ACID Transactions</a></li><li><a href="https://engineering.fb.com/2018/06/26/core-infra/migrating-messenger-storage-to-optimize-performance/">Migrating Messenger storage to optimize performance</a></li><li><a href="https://engineering.fb.com/2013/06/25/core-infra/tao-the-power-of-the-graph/">TAO: The power of Graph</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=206fbbb8207" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Leap Second Crash!]]></title>
            <link>https://sumedhakhatter.medium.com/the-leap-second-crash-312f1d486c65?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/312f1d486c65</guid>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 02:55:57 GMT</pubDate>
            <atom:updated>2025-07-14T02:55:57.654Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zd7DokNUBWg2hNa9cIHpjA.png" /><figcaption>Source: Chatgpt.openai</figcaption></figure><h3>Context &amp; Background</h3><h4>What is a Leap Second?</h4><p>A <strong>leap second</strong> is an extra second added to Coordinated Universal Time (UTC) to keep it in sync with Earth’s slightly irregular rotation. It ensures that our clocks (which are based on highly precise atomic time) stay aligned with <strong>astronomical time</strong>, which is based on Earth’s rotation.</p><h4>Why Do We Need Leap Seconds?</h4><ul><li>Atomic clocks are incredibly accurate and define <strong>International Atomic Time (TAI)</strong>.</li><li>However, Earth’s rotation is <strong>not perfectly uniform</strong> — it’s gradually slowing down due to tidal friction and other factors.</li><li>Without leap seconds, <strong>UTC</strong> (the time standard we use in daily life) would slowly drift away from <strong>solar time</strong> (the position of the Sun in the sky).</li></ul><h4>When Is a Leap Second Added?</h4><ul><li>Leap seconds are typically added <strong>at the end of June 30 or December 31</strong>, announced about six months in advance by the <strong>International Earth Rotation and Reference Systems Service (IERS)</strong>.</li><li>Example: The last leap second was added on <strong>December 31, 2016, at 23:59:60 UTC</strong>.</li></ul><h4>How Often Do They Happen?</h4><ul><li>Since 1972 (when leap seconds were first introduced), <strong>27 leap seconds</strong> have been added.</li><li>They are <strong>irregular</strong> and depend on measurements of Earth’s rotation.</li></ul><h4>Current Status</h4><ul><li>In 2022, international timekeeping bodies decided to <strong>stop adding leap seconds by 2035</strong> due to the operational challenges they create for computing and communication systems.</li></ul><h3>What is Leap Second Crash?</h3><p>The “Leap Second Crash” refers to widespread outages and system failures that occurred during previous leap second insertions — notably <strong>on June 30, 2012</strong>, when <strong>half the internet</strong> (major websites and services) experienced <strong>slowdowns, crashes, or outages</strong>.</p><h4>What caused the bug?</h4><p>The root cause was that <strong>many systems weren’t designed to handle a 61-second minute</strong>. Normally, a minute has 60 seconds (00 to 59), but during a leap second, a special timestamp 23:59:60 is inserted. This confused software expecting time to increment normally.</p><h3>Major technical issue: High CPU load due to kernel bug</h3><h4>In Linux systems:</h4><ul><li>The <strong>Linux kernel’s timekeeping code</strong> used a component called <strong>timekeeping_update()</strong> to manage the system clock.</li><li>When the <strong>leap second</strong> was added, the kernel <strong>misbehaved</strong>, causing threads to spin in tight loops waiting for time to advance.</li><li>This led to <strong>100% CPU usage</strong>, causing systems to hang or crash.</li></ul><h4>Affected platforms:</h4><ul><li>Reddit: Linux kernels locked at 100% CPU due to leap-second handling glitches</li><li>Cloudfare DNS: Go’s random function panicked when receiving negative time intervals</li><li>LinkedIn</li><li>Yelp</li><li>Mozilla</li><li>Foursquare</li><li>Qantas Airways</li><li>And many more…</li></ul><p>Even <strong>Java-based applications</strong> were affected due to JVM issues with how it handled the leap second internally.</p><h4>Example of a Bad Time Sequence:</h4><pre>23:59:58  <br>23:59:59  <br>23:59:60  &lt;-- causes confusion!  <br>00:00:00</pre><p>The systems expected:</p><pre>23:59:59  <br>00:00:00</pre><p>So they got stuck or threw exceptions.</p><h3>Workarounds and Fixes</h3><ol><li><strong>Rebooting servers</strong> cleared the CPU usage and reset the time state.</li><li><strong>Patching kernels</strong>: After the incident, Linux distributions released patches to handle leap seconds gracefully.</li><li><strong>Slewing instead of stepping</strong>: Instead of inserting a full extra second (23:59:60), many systems now use <strong>&quot;slew time&quot;</strong> (gradually slow down the clock over time) using tools like <strong>ntpd</strong> or <strong>chronyd</strong>. Meta uses the slewing approach as well. Find details in the section below.</li></ol><h3>Meta’s Current Approach: Leap-Second Smearing</h3><ul><li>Instead of a sudden clock jump, Meta <strong>“smears”</strong> the leap second gradually over <strong>17 hours</strong> starting at UTC midnight</li><li>Helps maintain consistency across thousands of NTP servers, avoiding large clock offsets that trigger failure detection in clients.</li><li>Meta uses a <strong>quadratic smearing algorithm</strong> (others use linear or cosine).</li></ul><h3>Conclusion</h3><p>The leap second crash was a classic case of <strong>real-world timekeeping clashing with software assumptions</strong>. It revealed how deeply embedded time assumptions are in critical infrastructure — and why many companies and institutions now advocate <strong>abolishing leap seconds entirely.</strong></p><h3>References:</h3><ol><li><a href="https://en.wikipedia.org/wiki/Leap_second">https://en.wikipedia.org/wiki/Leap_second</a></li><li><a href="https://engineering.fb.com/2022/07/25/production-engineering/its-time-to-leave-the-leap-second-in-the-past/">https://engineering.fb.com/2022/07/25/production-engineering/its-time-to-leave-the-leap-second-in-the-past/</a></li><li><a href="https://www.nist.gov/pml/time-and-frequency-division/leap-seconds-faqs">https://www.nist.gov/pml/time-and-frequency-division/leap-seconds-faqs</a></li><li><a href="https://www.somebits.com/weblog/tech/bad/leap-second-2012.html">https://www.somebits.com/weblog/tech/bad/leap-second-2012.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=312f1d486c65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Webhooks!]]></title>
            <link>https://sumedhakhatter.medium.com/learn-webhooks-6871a334587c?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/6871a334587c</guid>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Wed, 25 Sep 2024 20:09:35 GMT</pubDate>
            <atom:updated>2024-09-25T20:09:35.989Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>With this tutorial, we learn how and when we can use webhooks, how we can set up a webhook workflow, and then different ways to send message to the webhook URL.</blockquote><p>Webhooks are another method for enabling communication between two applications. While there are various ways for applications to interact, webhooks are particularly effective when connecting with third-party services.</p><ul><li><strong>Real-Time Notifications</strong>: When you need immediate updates from one application to another, such as payment confirmations or status changes.</li><li><strong>Third-Party Integrations</strong>: If you’re connecting your application with a third-party service (like payment processors or messaging platforms) that supports webhooks for event-driven communication. Example: if my application processes payments through Stripe, we can use webhooks to handle communication.</li><li><strong>Event-Driven Architecture</strong>: When you want to trigger specific actions in your application based on events happening elsewhere, such as user actions or system updates. Example: if your system deals with multiple clients. If the clients want an event triggered in their systems whenver they take an action while using your system, webhooks can facilitate that interaction.</li><li><strong>Reducing Polling</strong>: Instead of continuously checking for updates (polling), webhooks allow your application to receive data only when an event occurs, improving efficiency.</li><li><strong>Asynchronous Communication</strong>: If your application can benefit from processing data asynchronously, webhooks can help facilitate that without needing synchronous API calls.</li></ul><h3><strong>Setting up webhooks:</strong></h3><p>We can set up webhooks in our own application to connect with other applications or clients. For the sake of simplicity in this tutorial, I am setting it up through a tool called Pipedream. With pipedream, I am creating a simple workflow with a basic trigger. This sets up a URL which can receive a message from any other applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4GRc7wXetBtoYQ5ZVXm1g.png" /><figcaption>Screenshot from pipedream. This workflow has a webhook and accepts an event from external applications.</figcaption></figure><h3>Sending HTTP events to the Webhook:</h3><p>Now that we have set up a webhook URL, lets send a message to the webhook URL.</p><h4>1. Use Postman</h4><p>Create an account on Postman and send a POST HTTP request from Postman. Below is the screenshot where I am sending a POST request containing 2 parameters, key and key2 with values ‘abcd’ and ‘xyz’ respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ezf0dS7aDzxePa2uiRe_8A.png" /><figcaption>Screenshot from the Postman Agent</figcaption></figure><p>When the POST request is sent from the Postman, we can see the same event being received on the pipedream webhook URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dyGMYH-g6bsOl8CgJGSouw.png" /><figcaption>Screenshot from the Pipedream.</figcaption></figure><h4>2. Use CURL request</h4><p>Send an event using the CURL command from your machine.</p><p>Here is the command:</p><pre>curl -H &quot;Content-type: multipart/form-data&quot; \<br>     -F title=foo \<br>     -F body=bar \<br>     -F id=&quot;1&quot; \<br>     -X POST \<br>     https://eouxer3y5ir6zas.m.pipedream.net</pre><p>This will send an event containing multiple params: title, body, id to the webhook URL.</p><p>Here is the event received by the pipedream webhook URL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kY3UMd_Wm0lRr15xI9SwCA.png" /><figcaption>Screenshot from Pipedream receiving the POST event using CURL request.</figcaption></figure><h3><strong>Conclusion:</strong></h3><p>In this tutorial, we explored when to use webhooks. We covered how to set up an external webhook URL and workflow, and demonstrated various methods for sending messages to the webhook. This communication method can be implemented in applications to connect two systems effectively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6871a334587c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Live Video Streaming with Comments]]></title>
            <link>https://sumedhakhatter.medium.com/designing-a-live-video-streaming-with-comments-d4dce4c62dd3?source=rss-d0b9056dbd47------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4dce4c62dd3</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Sumedha Khatter]]></dc:creator>
            <pubDate>Thu, 29 Feb 2024 09:53:10 GMT</pubDate>
            <atom:updated>2024-03-02T04:13:48.095Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is live video streaming?</h3><p>Video Live Streaming means the video is published on the internet in real time.</p><p>Technically, the video content is segmented into small chunks and delivered sequentially to viewers for immediate playback, minimizing buffering. However, building a complete video streaming service entails tasks like encoding, format adaptation for different devices, storage management, catalog accessibility, and user engagement features such as liking or rating videos.</p><h3>Design Components of Video Streaming</h3><p><em>Assumptions: Video is streamed to the audience when it is telecasted live by the creator</em></p><ol><li><strong>Users</strong></li></ol><p>They are the users who are interested to create a live video and upload it to a media platform, that is then viewed by their subscribed followers.</p><p><strong>2. Load Balancer</strong></p><p>Load balancer will help in balancing the load across several web servers of the media platform.</p><p><strong>3. Authentication API</strong></p><p>This will help in building an authentication API with create account and login account capabilities authenticating the users who are interested to publish a live video.</p><p><strong>4. Video Ingestion</strong></p><p>This module help in ingesting the video that is being coming live by the user through their cameras. <em>Media servers such as Wowza Streaming Engine, Red5, or Ant Media Server are capable of ingesting live video streams from different sources. Cloud-based video platforms such as Amazon Web Services (AWS) Elemental MediaLive, Microsoft Azure Media Services, or Google Cloud Video Intelligence provide managed services for live video ingestion and processing.</em></p><p><strong>5. Video Transcoding</strong></p><p>Incoming video streams may need to be transcoded to different formats or bitrates to support various devices and network conditions.</p><p><em>There are a couple of video transcoding tools available. Example: HandBrake is a free and open-source transcoder for digital video files. It supports various formats and codecs. Another one, FFmpeg, is a powerful multimedia framework that includes a command-line tool for transcoding video and audio files.</em></p><p><strong>6. Live Streaming Server</strong></p><p>Once the video is transcoded it needs to be delivered to the users in the real time. Use protocols like RTMP (Real Time Messaging Protocols) to deliver video streams to the devices. <em>There are a couple of tools available in the market that can be the live streaming server. Example: Wowza Streaming Engine is a customizable media server software that allows you to stream live and on-demand video to any device. Another one, Adobe Media Server: Formerly known as Flash Media Server, Adobe Media Server enables real-time streaming of video and audio content to web browsers and mobile devices.</em></p><p>When a user accesses the live streaming page, the client (web browser) initiates a WebSocket connection to the backend server. The WebSocket handshake occurs over HTTP, after which the connection is upgraded to a persistent WebSocket connection.</p><blockquote><strong>Why Web Sockets?</strong></blockquote><blockquote>Because Web Sockets provide full duplex communication. Full-duplex enables simultaneous two-way communication, which allows data to flow freely bidirectionally. Half-duplex enables two-way communication, but not simultaneously. Half-duplex requires switching between a sending and receiving mode</blockquote><p><strong>7. Commenting System</strong></p><p>This module provides an interface to the users to comment on the live video and get responses, likes, replies on the comments; while watching the live video.</p><p>As viewers watch the live stream, they can submit comments, reactions, or other interactions via the WebSocket connection. The backend server receives these messages and broadcasts them to all connected clients in real-time, ensuring that everyone watching the live stream sees the same set of comments or reactions as they are submitted.</p><p>To further send notifications to the audience on the comments we can use SSE (Server Sent Events) to push the notifications to the clients through HTTP request.</p><blockquote>How is Server Sent Event different from HTTP response?</blockquote><blockquote>SSE is unidirectional, that allows data to be sent directly from server to clients. These are push events that are triggered from server to client without having any HTTP request from the client. It can be thought of as a as a long-running HTTP request that sends data out to the client whenever the server wants.</blockquote><p><strong>8. CDNs (Content Delivery Network)</strong></p><p>CDNs consist of a distributed network of servers located in multiple geographic locations around the world. By caching and serving content from servers closer to end-users, CDNs reduce the physical distance that data needs to travel, resulting in faster content delivery and reduced latency.</p><p>Distribute video streams across multiple edge servers worldwide to ensure fast and reliable delivery to viewers regardless of their location. This is because when video streams are delivered from edge servers located closer to viewers, the distance data needs to travel is reduced, resulting in lower latency and faster loading times. This improves the overall user experience by reducing buffering, stuttering, and playback interruptions.</p><p>The video is sent over the network in chunks and ends up on users’ devices, which can be mobile devices, laptops, desktops, or even smartwatches.</p><h4>What if the video can be watched offline and shared later on to the audience who opens the app later on?</h4><p>This means we would need separate infrastructure to record the video, and store the video and share it to the users who open the app later on. So apart from the components discussed above, there would be few other modules:</p><p><strong>9. Video Uploading</strong></p><p>This will accept the video and upload it to a distributed storage. Amazon Simple Storage Service or Google Firestore databases can store videos.</p><p><strong>10. Storage Infrastructure</strong></p><p>This would mean building and setting up storage databases to save the transcoded video files.</p><p><strong>11. Metadata and Indexing</strong></p><p>This would mean creating a video URL of the videos in the file storage system and then using those URLs again to render the feed to the audience. This can also be useful in indexing, searching the video from the video database.</p><p><strong>12. Video Player</strong></p><p>This would mean providing video player capabilities to the audience to play, stop an already recorded video.</p><h3>References:</h3><ol><li>Server Sent Events and Web Sockets: <a href="https://blog.logrocket.com/server-sent-events-vs-websockets/">https://blog.logrocket.com/server-sent-events-vs-websockets/</a></li><li>Video Streaming Architecture: <a href="https://hygraph.com/blog/video-streaming-architecture">https://hygraph.com/blog/video-streaming-architecture</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4dce4c62dd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>