<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdul Moiz on Medium]]></title>
        <description><![CDATA[Stories by Abdul Moiz on Medium]]></description>
        <link>https://medium.com/@iam-abdulmoiz?source=rss-5db46112cc9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*11vOPUluPtyONxmO51sgVw.png</url>
            <title>Stories by Abdul Moiz on Medium</title>
            <link>https://medium.com/@iam-abdulmoiz?source=rss-5db46112cc9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:54:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@iam-abdulmoiz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Idempotency & Exactly-Once-ish Processing]]></title>
            <link>https://medium.com/@iam-abdulmoiz/idempotency-exactly-once-ish-processing-a67cf6ec64a2?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a67cf6ec64a2</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[message-queue]]></category>
            <category><![CDATA[idempotency]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 09:01:04 GMT</pubDate>
            <atom:updated>2026-08-13T09:01:04.628Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>Back in the retries article, we flagged something and promised to come back to it: retrying a request is only safe if repeating it produces the same result as running it once. That single idea, idempotency, is what actually makes retries, message queues, and distributed systems in general survivable. This article is that promise, kept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3HutHrOI7k1Q78tzl-yowA.png" /></figure><h3>The Problem Retries Create</h3><p>Here’s the scenario that makes this concrete. A client sends a request to charge a customer’s card. The server receives it, processes the charge successfully, and starts sending back a success response. But somewhere on the way back, the network drops the connection. The client never sees that response.</p><p>From the client’s point of view, the request failed, it got no confirmation either way. Following everything we covered in the retries article, the reasonable thing to do is retry. So it sends the exact same charge request again.</p><p>But here’s the problem: the first charge actually went through. The server did its job correctly. The client retrying isn’t wrong either, it genuinely has no way of knowing the first attempt succeeded. Both sides behaved reasonably, and the customer still ends up charged twice.</p><p>This is the exact tension idempotency exists to resolve.</p><h3>What Idempotency Means</h3><p>An operation is <strong>idempotent</strong> if performing it multiple times has the same effect as performing it once. That’s the whole definition, but it’s worth sitting with, because it applies unevenly across different kinds of operations, and knowing which category something falls into is most of the battle.</p><p>Reading data is naturally idempotent. Asking for a user’s profile once or asking for it fifty times in a row doesn’t change anything about that user, you just get the same answer fifty times. Setting a value directly is also naturally idempotent, if you set a user’s status to “active,” doing that again still leaves it “active.” Deleting something is idempotent too, once it’s gone, deleting it again doesn’t do anything further, it’s just still gone.</p><p>Creating something is where it breaks down. “Create a new order” run twice creates two orders. “Charge this card” run twice charges it twice. These operations have a side effect that compounds every time they run, and that’s exactly the category retries are dangerous for.</p><h3>Idempotency Keys</h3><p>The fix for operations that aren’t naturally safe to repeat is to make them safe artificially, using something called an <strong>idempotency key</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3bahdUVTCShw5bhc8FCYg.png" /><figcaption><em>The client generates a unique key for a request and sends the exact same key again on any retry, so the server can recognize it’s seeing the same logical request twice.</em></figcaption></figure><p>Before sending a request that creates something, like a charge, the client generates a unique identifier for that specific attempt, often a UUID, and includes it with the request. If the request needs to be retried, because of a timeout, a dropped connection, anything, the client sends the retry with the exact same key it used the first time. Not a new one, the same one.</p><p>This is the detail that makes the whole pattern work. The key doesn’t identify “a charge request” in general, it identifies this specific logical attempt to charge, regardless of how many times it physically gets sent over the wire. The server’s job becomes straightforward from there: recognize when it’s seen a key before, and respond accordingly.</p><h3>How the Server Uses the Key</h3><p>On the server side, handling an idempotency key comes down to checking before acting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlq0yusskVKuPQKYFbWc3g.png" /><figcaption><em>Before processing anything, the server checks whether it has seen this exact key before. If it has, it returns the stored result instead of doing the work again.</em></figcaption></figure><p>When a request comes in with an idempotency key, the server first looks that key up in a dedicated table, sometimes called an idempotency store, that maps keys to results. If the key isn’t there, this is genuinely a new request, so the server processes it normally, charges the card, and stores the result against that key before responding. If the key is already there, the server doesn’t touch the payments system at all, it just returns whatever result it stored the first time.</p><p>This means a retry with the same key is completely safe, no matter how many times it happens. The very first attempt does the real work. Every subsequent attempt with that same key just fetches the recorded outcome and hands it back, without charging anything a second time.</p><p>One practical detail worth knowing: these stored keys usually expire after some reasonable window, hours or days depending on the system, rather than being kept forever, since the whole point is covering retries that happen shortly after the original attempt, not permanently remembering every request ever made.</p><h3>Natural vs Enforced Idempotency</h3><p>It’s worth being explicit about the two different ways idempotency shows up, since they require completely different amounts of work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1lUGTikPWsCnDgcBOj6Hw.png" /><figcaption><em>Some operations are safe to repeat by nature. Others need an idempotency key layered on top to become safe.</em></figcaption></figure><p>Operations like PUT /users/5 with a full replacement of the user&#39;s data, or DELETE /posts/9, are naturally idempotent because of what they mean semantically. Setting something to a specific value, or removing something that may already be removed, doesn&#39;t compound no matter how many times you do it. You don&#39;t need to build anything extra here, the safety comes for free from the shape of the operation itself.</p><p>POST /charges is the opposite. Creating a new resource is inherently not idempotent, because by definition, doing it again creates another one. This is exactly the category that needs an idempotency key layered on top, POST /charges plus an idempotency key transforms a dangerous, retry-unsafe operation into a safe one, but only because the server explicitly does the extra work of tracking and deduplicating by that key.</p><p>Recognizing which category an operation falls into tells you immediately whether you need to build anything at all, or whether the safety was already there by design.</p><h3>Why Exactly-Once Doesn’t Really Exist</h3><p>This brings us to something worth being honest about, because it’s a common point of confusion: distributed systems don’t actually achieve “exactly-once” delivery in any strict sense, and understanding why clarifies what idempotency is actually doing for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6IZ-d8w_9ryKpyvGHj5rA.png" /><figcaption><em>Message systems deliver at-least-once. Idempotent handling turns that into something that behaves like exactly-once, even though it technically isn’t.</em></figcaption></figure><p>Most real message delivery systems, queues, event streams, retry-based APIs, guarantee <strong>at-least-once</strong> delivery. A message might arrive once, or it might arrive more than once, because of retries, network issues, or failover between nodes, but it will not silently vanish. Genuinely guaranteeing exactly-once delivery at the network level is extraordinarily difficult, and most systems that claim it are actually doing something else under the hood to compensate.</p><p>That something else is almost always idempotency. If a message might arrive twice, but the handler processing it is idempotent, meaning processing it twice has the same effect as processing it once, then it doesn’t actually matter that the delivery guarantee is only at-least-once. The observable outcome behaves as if it were exactly-once, even though, technically, the delivery mechanism never made that promise.</p><p>This is why the honest way to describe most production systems isn’t “exactly-once,” it’s “effectively-once”: at-least-once delivery, combined with idempotent handling, produces the outcome people actually want, without needing a delivery guarantee that’s nearly impossible to provide honestly at scale.</p><h3>Tying It Back to Retries</h3><p>This closes the loop we opened in the retries article. Retrying a failed request is the right instinct, most failures genuinely are transient, and giving up after one attempt wastes a lot of recoverable situations. But retries are only actually safe once the operation being retried is idempotent, either naturally or through an explicit key.</p><p>Circuit breakers decide whether to attempt a call. Timeouts decide how long to wait. Retries decide whether to try again. Bulkheads decide how much of your resources any one dependency can consume. Backpressure decides how fast work is allowed to arrive in the first place. And idempotency is the thing that makes retrying safe at all, the quiet assumption sitting underneath everything else in this module. Get it wrong, and every other pattern we’ve covered is still capable of quietly duplicating work behind your back.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a67cf6ec64a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backpressure & Flow Control]]></title>
            <link>https://medium.com/@iam-abdulmoiz/backpressure-flow-control-1e6863f5d904?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e6863f5d904</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[backpressure]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[control-flow]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 09:01:04 GMT</pubDate>
            <atom:updated>2026-08-11T09:01:04.271Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>Bulkheads deal with too many different dependencies competing for the same resources. This article deals with a related but different problem: what happens when just two components, a producer and a consumer, disagree on speed. One creates work faster than the other can handle it, and unless something intervenes, that mismatch will eventually break the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Tipjb4x62w6TwCp0SeA4g.png" /></figure><h3>What Happens When a Producer Is Faster Than a Consumer</h3><p>Picture a straightforward pipeline. Some part of your system produces events, maybe user actions, sensor readings, log entries, and writes them somewhere for another part of the system to process.</p><p>As long as the producer and consumer move at roughly the same pace, everything is fine. But real systems rarely have perfectly matched speeds. Maybe the producer is a spike in user traffic that suddenly triples. Maybe the consumer is doing something genuinely expensive per item, writing to a database, calling an external API, running some computation, that just can’t keep up no matter how well it’s built.</p><p>Whatever the cause, the moment the producer outpaces the consumer, work starts accumulating somewhere. Understanding exactly where it accumulates, and what happens next, is the whole subject of this article.</p><h3>The Queue Fills Up</h3><p>Most systems don’t connect a producer directly to a consumer. There’s a queue in between, precisely so the two don’t need to move in lockstep. The producer writes to the queue, the consumer reads from it whenever it’s ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1hIk5TRHtiOgnQwOJpksw.png" /><figcaption><em>A queue absorbs the difference in speed between a producer and a consumer, but only up to a point, its size.</em></figcaption></figure><p>This works great as a short-term buffer. If the producer has a brief burst, the queue soaks it up, and the consumer works through the backlog once things calm down. But if the producer is consistently faster than the consumer, not just briefly but as an ongoing pattern, the queue doesn’t just absorb the difference, it accumulates it. Every second, the gap between what came in and what got processed grows a little bigger. The backlog doesn’t shrink on its own, it only grows, for as long as the imbalance continues.</p><p>A queue is a buffer, not a fix. It buys you time to notice and respond to a speed mismatch. It does not, by itself, resolve the mismatch.</p><h3>Unbounded Queues Are a Trap</h3><p>Here’s where a genuinely dangerous decision gets made, often without anyone realizing they made it: what’s the maximum size of that queue?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7m6zz9vw9LvfTxKDfI5guw.png" /><figcaption><em>An unbounded queue doesn’t fail gracefully. It just keeps growing, consuming more and more memory, until the process runs out and crashes.</em></figcaption></figure><p>Plenty of queue implementations default to no limit at all, they’ll grow to accept whatever gets thrown at them. That sounds convenient, you never have to worry about a “queue full” error. But it just relocates the problem instead of solving it. Every item sitting in that queue is taking up memory. If the producer keeps outpacing the consumer indefinitely, the queue keeps growing indefinitely, and so does memory usage, right up until the process runs out of memory entirely and crashes.</p><p>And when that crash happens, it doesn’t just lose the items that were still in the queue, it takes down the entire process, consumer included, along with anything else that process was responsible for. A slow consumer that would’ve been a manageable, visible problem on its own turns into a full outage, purely because nothing ever told the producer to stop.</p><p>An unbounded queue isn’t actually removing the failure mode, it’s just deferring it and making it worse when it eventually arrives.</p><h3>Backpressure: Telling the Producer to Slow Down</h3><p>The fix is to give the queue an actual limit, and just as importantly, to make the producer aware when that limit is being approached.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YF5yr3Yf8l1nOg0s-YnwNQ.png" /><figcaption><em>When the queue nears capacity, backpressure signals the producer to slow down, rather than letting the queue grow without bound.</em></figcaption></figure><p>This is what <strong>backpressure</strong> means: a signal that flows backward, from the consumer’s side of the system toward the producer’s side, communicating “I can’t keep up, ease off.” Instead of a queue that silently absorbs infinite work, a bounded queue has a defined capacity, and once it starts filling up toward that limit, it pushes back. Depending on the system, that might mean the producer’s write calls start blocking until space frees up, or the producer gets an explicit error it needs to handle, or the rate the producer is allowed to write at gets throttled directly.</p><p>The exact mechanism varies a lot by technology. TCP itself has flow control built in at the protocol level, a receiver can tell a sender to slow down before its buffer overflows. Reactive streams libraries build backpressure into the core abstraction, letting a subscriber explicitly request only as much data as it’s ready to handle. Message queue systems often expose consumer lag metrics specifically so producers, or the systems managing them, can react before things get out of hand.</p><p>What all of these share is the same underlying idea: the imbalance gets fed back to whichever side can actually do something about it, instead of being left to pile up invisibly until something breaks.</p><h3>Backpressure Strategies</h3><p>When a system does hit its limit, there are a few different ways to respond, and the right choice depends heavily on what the data actually means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nE0CzekyEzt2lyeHPiB_A.png" /><figcaption><em>There’s more than one way to respond to overload, and the right choice depends on what you’re willing to sacrifice.</em></figcaption></figure><p><strong>Block</strong> makes the producer wait until there’s room. This preserves every single item, nothing gets lost, but it means the producer’s own throughput drops to match the consumer’s, which can create its own ripple effects further upstream.</p><p><strong>Buffer</strong> is the bounded queue itself, absorbing short bursts without immediately rejecting anything, while still having a hard ceiling that prevents runaway growth.</p><p><strong>Drop</strong> discards items once capacity is exceeded, either the newest ones coming in or the oldest ones already waiting, depending on which matters less for the use case. This trades data completeness for throughput, appropriate when losing some data is acceptable, like metrics or non-critical logs, and completely inappropriate when it isn’t, like financial transactions.</p><p><strong>Shed load</strong> rejects work at the very edge of the system, before it’s even accepted, rather than letting it get partway in and then failing. This keeps the system’s internal state predictable and avoids wasting effort on work that’s going to get abandoned anyway.</p><p><strong>Rate limit</strong> caps how much work a producer is allowed to send in a given window in the first place, addressing the imbalance proactively rather than reactively responding once a queue is already under pressure.</p><p>None of these is universally correct. Losing a few analytics events is usually fine. Losing a payment confirmation is not. The right strategy is a decision about what your system can afford to sacrifice when it’s under more load than it can fully handle, and that answer is different for every kind of data.</p><h3>Backpressure in Practice</h3><p>It’s worth noticing that this isn’t a new idea invented for distributed systems, it’s actually one of the oldest ideas in networking. TCP has had flow control since long before anyone was talking about message queues or event streams, precisely because the same fundamental problem, a fast sender overwhelming a slow receiver, shows up at every layer of computing, not just at the application level.</p><p>That’s a useful thing to internalize: whenever you’re connecting two components that might operate at different speeds, whether that’s two microservices, a producer and a Kafka consumer, or your own application and a downstream API, the question of what happens when one side outpaces the other isn’t optional to answer. If you don’t design an explicit answer, you get the unbounded queue by default, and the unbounded queue always ends the same way.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e6863f5d904" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bulkheads: Isolating Failure by Design]]></title>
            <link>https://medium.com/@iam-abdulmoiz/bulkheads-isolating-failure-by-design-be5a919f55a1?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/be5a919f55a1</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[bulkhead]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[backend-development]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 09:01:03 GMT</pubDate>
            <atom:updated>2026-08-10T09:01:03.758Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>We’ve covered circuit breakers, which stop a service from repeatedly calling something that’s clearly broken, and retries with timeouts, which decide how long to wait and whether to try again. This article covers a pattern that works alongside both of them, but solves a different problem: what happens when one slow dependency starves every other dependency, even the healthy ones, just because they all shared the same pool of resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cirzmxPfh2p3ViXCAP3akw.png" /></figure><h3>One Bad Neighbor Takes Down Everyone Else</h3><p>Imagine a service that talks to three different downstream APIs: payments, search, and email. All three calls go through the same shared pool of, say, 20 threads. Under normal conditions this is fine, requests come in, grab a thread, get a response, release the thread, and the cycle repeats fast enough that nobody notices.</p><p>Now the payments API starts hanging. Every call to it takes 30 seconds instead of 100 milliseconds. Here’s the part that catches people off guard: this doesn’t just make payments slow. If enough payment calls are in flight, they eventually occupy every single thread in that shared pool. At that point, a request that just wants to hit the search API, which is working perfectly fine, can’t even get a thread to run on. It’s not failing because search is broken. It’s failing because payments used up the entire shared resource.</p><p>One struggling dependency just took down two healthy ones. This is exactly the same cascading failure shape we saw with circuit breakers, but the root cause here isn’t repeated failed calls, it’s resource exhaustion from calls that are simply slow.</p><h3>The Ship Analogy</h3><p>The name “bulkhead” comes from shipbuilding, and it’s a genuinely useful mental model. Large ships are built with internal walls, bulkheads, that divide the hull into separate watertight compartments. If the hull gets breached in one compartment and water floods in, that compartment fills up, but the walls stop the water from spreading into the rest of the ship. The ship stays afloat because the damage is contained to one section instead of sinking the whole vessel.</p><p>Software bulkheads do the same thing for resources. Instead of one shared pool that everything draws from, you split it into separate, isolated pools, one per dependency. If one pool gets fully consumed by a struggling dependency, the damage stays contained to that pool. Everything else keeps running normally.</p><h3>What Bulkheads Look Like in Software</h3><p>In practice, a bulkhead is usually a dedicated resource pool, most commonly a thread pool or a connection pool, assigned specifically to one dependency rather than shared across all of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f3Ykza6o6ec7N3XMUpcIvw.png" /><figcaption><em>Instead of one shared pool serving every outbound call, each dependency gets its own pool sized for its own needs.</em></figcaption></figure><p>Service A calls three different APIs. Instead of routing all three through one shared pool, each dependency gets its own dedicated pool: a payments pool, a search pool, an email pool. Each pool has a fixed size, its own ceiling on how many concurrent calls it will allow. Calls to payments can only ever consume threads from the payments pool. They physically cannot touch the threads reserved for search or email, because those live in a completely separate pool.</p><p>This is a genuinely simple idea, and that’s part of why it works so well. There’s no clever failure detection involved, no thresholds to tune, no state machine to reason about. It’s just structural isolation, decided at design time, that limits how far damage from any one dependency can spread.</p><h3>Without Bulkheads: One Dependency Exhausts Everything</h3><p>Let’s look at what actually happens, step by step, when everything shares one pool and one dependency goes bad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULvuiEaA1DiITqYN6hepFg.png" /><figcaption><em>When every dependency draws from the same shared pool, one hanging call can consume the entire pool and starve calls to completely unrelated, healthy services.</em></figcaption></figure><p>Service A has a single shared pool of threads for all outbound calls. The payments API starts hanging. Calls to it pile up, each one holding a thread for far longer than normal. As more and more payment calls come in, they consume a larger and larger share of the shared pool. Eventually, there simply aren’t any threads left for anyone else. A request that only needed to call the search API, which is completely healthy and would have responded in milliseconds, can’t get a thread at all. It sits blocked, waiting for a resource that payments is hoarding.</p><p>From the outside, this looks like search and email are broken too. Nobody debugging this would think to look at payments first, because payments isn’t even the service throwing errors, from a user’s perspective, everything just seems slow or unresponsive across the board.</p><h3>With Bulkheads: Isolated Failure</h3><p>Now the same exact scenario, but with dedicated pools per dependency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CxMmDMCGjXB5BGnHA3yz8Q.png" /><figcaption><em>With isolated pools, the same failing dependency only exhausts its own pool. Every other pool, and every other dependency, keeps working normally.</em></figcaption></figure><p>The payments API still hangs, exactly as before. Its dedicated pool still gets exhausted, exactly as before. But this time, that exhaustion is contained entirely within the payments pool. The search pool has its own separate threads that payments can never touch. Calls to search keep getting served normally, at normal speed, completely unaffected by whatever’s going wrong with payments. Same for email.</p><p>Users trying to check out will still see payment failures, that part hasn’t changed, payments is genuinely having a problem and that’s real. But they won’t lose the ability to search the site or receive email confirmations just because an unrelated dependency happens to be struggling. The blast radius of the failure shrank from “the whole service is unusable” down to “the one actual feature that’s actually broken.”</p><h3>Bulkheads at Different Levels</h3><p>Thread pools and connection pools are the most common place to apply this pattern, but the same idea for compartmentalization shows up at other levels too.</p><p>At the process or container level, running separate service instances for separate concerns means a memory leak or crash in one process can’t directly take down another. At the infrastructure level, some teams go as far as running genuinely separate deployments for high-priority and low-priority traffic, so a flood of low-priority work can never starve requests that actually matter. The specific mechanism changes, but the underlying principle is identical: don’t let unrelated things share a resource that one of them could exhaust.</p><h3>Sizing Bulkheads</h3><p>Deciding how big to make each pool is where bulkheads get genuinely tricky in practice.</p><p>Make a pool too small, and you’ll hit its ceiling during completely normal traffic spikes, rejecting requests that would’ve succeeded fine if they’d had a slightly bigger pool to work with. Make it too large, and you lose some of the protection the pattern is supposed to give you, a huge pool dedicated to one dependency can still get fully exhausted by that dependency, it just takes longer and does more damage before it happens.</p><p>A reasonable starting point is sizing each pool around the actual expected concurrency for that dependency under normal load, with some headroom for spikes, and then watching real usage patterns to tune from there. This isn’t something you get exactly right on the first guess, and that’s fine, it’s meant to be adjusted as you learn how each dependency actually behaves in production.</p><h3>Bulkheads, Timeouts, and Circuit Breakers Together</h3><p>None of the patterns in this module work in isolation, they’re meant to be layered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TDHpftUaYoc6VrDkXcsriw.png" /><figcaption><em>A single outbound call is typically guarded by all three patterns at once, each one handling a different failure mode.</em></figcaption></figure><p>A request goes through a bulkhead first, which caps how many concurrent calls to this dependency are allowed at all. If it gets a slot, it proceeds under a timeout, which bounds how long it’s willing to wait for a response. And the whole thing sits behind a circuit breaker, which tracks the outcomes over time and stops sending calls entirely once failures cross a threshold.</p><p>Each one covers a gap the others don’t. A bulkhead limits how many resources one dependency can consume, but doesn’t stop a request from waiting forever if there’s no timeout underneath it. A timeout bounds the wait, but does nothing to stop a bulkhead’s pool from filling up with requests that are each individually timing out in sequence. And a circuit breaker stops repeat failures from continuing to happen, but relies on timeouts to even detect that a call has failed quickly enough to matter. Together, they cover each other’s blind spots.</p><p>Everything up to this point in the module has been about handling failure gracefully once it happens. The next piece is different: it’s about controlling the rate at which work arrives in the first place, before it ever gets the chance to overwhelm anything.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be5a919f55a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Retries & Timeouts: Trying Again, Safely]]></title>
            <link>https://medium.com/@iam-abdulmoiz/retries-timeouts-trying-again-safely-d540532a5038?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d540532a5038</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[time-out]]></category>
            <category><![CDATA[retry-logic]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-08-07T04:01:01.944Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>We just covered circuit breakers, which decide whether to even attempt a call. This article covers the two patterns that decide what happens around that attempt itself: how long to wait for a response, and whether to try again if it fails. On paper these sound simple. In practice, getting them wrong is one of the most common ways teams accidentally make an outage worse instead of better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LoTcUF6xw1NMfxZZkeSFmg.png" /></figure><h3>Why a Single Failed Request Isn’t Always a Failure</h3><p>Not every failure means the same thing. Sometimes a request fails because something is genuinely broken, bad input, a bug, a resource that doesn’t exist. Retrying that won’t help, you’ll just get the same error again.</p><p>But a lot of failures in distributed systems are transient. A network blip drops one packet. A server was mid-restart for half a second. A load balancer briefly routed you to an instance that hadn’t finished starting up. None of these mean anything is actually wrong, the exact same request sent a moment later would probably succeed.</p><p>The whole idea behind retries is recognizing that difference and giving transient failures a second chance, without endlessly hammering something that’s genuinely broken.</p><p>But before we get to retries, we need to talk about something that has to exist first: a way of deciding how long is too long to wait for an answer in the first place.</p><h3>Timeouts: Deciding How Long Is Too Long</h3><p>Imagine calling a service with no timeout configured at all. If that service hangs, never responds, never errors, just sits there, your calling code sits there too. Forever, if nothing else stops it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2rx14nDrYH7qXDN3YgAjA.png" /><figcaption><em>Without a timeout, a hung dependency can hold a connection open indefinitely. With one, the caller gets a bounded, predictable failure instead.</em></figcaption></figure><p>This is a bigger problem than it sounds like at first. Every one of those hung requests is holding onto a thread, a connection, some chunk of memory, resources that don’t get released until the call finally resolves, which in this scenario might be never. Enough hung requests and your service runs out of capacity to do anything else, exactly the cascading failure problem we covered with circuit breakers, except this time the trigger is the complete absence of any time limit.</p><p>A timeout fixes this by putting a hard ceiling on how long you’ll wait. If the response doesn’t come back within that window, the call is treated as failed, the resources get released, and your code moves on, whether that means returning an error, trying again, or falling back to something else.</p><p>Picking the actual timeout value is a genuine balancing act. Too short, and you’ll time out perfectly healthy requests that just happened to take slightly longer than usual, wasting effort on retries that weren’t needed. Too long, and you’re back to holding resources hostage for far longer than necessary when something really has gone wrong. A common approach is to look at your normal response time distribution and set the timeout somewhere past the high end of what’s typical, not the average, since averages hide exactly the slow outliers you’re trying to catch.</p><h3>Retries: Trying Again Safely</h3><p>Once a call has failed or timed out, the next decision is whether it’s worth trying again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CzkcPhVXHI0mpGWhGP-htw.png" /><figcaption><em>A retry sequence isn’t just “try again immediately.” Each failed attempt is followed by a wait before the next try, and that wait typically grows with each failure.</em></figcaption></figure><p>A basic retry is straightforward: the call fails, you send it again. But a few things need to be true for this to actually be safe and useful.</p><p>First, the failure needs to be the transient kind we talked about earlier. Retrying a request that failed because of bad input just wastes time and gets you the same error again, sometimes it’s worth distinguishing these cases explicitly rather than retrying blindly on any failure.</p><p>Second, and this is the part that trips people up, the operation being retried needs to be safe to run more than once. If your first attempt to charge a customer actually succeeded on the server side but the response back to you got lost, and you retry, you might charge them twice. This is exactly why retries and idempotency are talked about together so often, an idea we’ll dig into properly in the next article, but it’s worth flagging here because you genuinely cannot build safe retries without it.</p><p>Third, you don’t retry instantly. Firing the retry the millisecond the first attempt fails just adds more load to something that may already be struggling. Which brings us to the next problem.</p><h3>The Retry Storm Problem</h3><p>Here’s a scenario that has taken down real production systems more than once. A service goes down, recovers, and the moment it comes back online, it gets crushed again and falls right back over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Clzbjl0wEiW6XDXrZ1hXbA.png" /><figcaption><em>When every client retries on the same fixed schedule, a service that just barely came back up gets hit with a synchronized wave of traffic and collapses again.</em></figcaption></figure><p>What causes this is usually not one misbehaving client, it’s the opposite: everyone behaving the same way at the same time. If a thousand clients all had requests fail at roughly the same moment, and they’re all configured to retry after exactly, say, 2 seconds, they all retry at almost exactly the same instant. The recovering service, which was just barely stabilizing, gets slammed with the same traffic spike that broke it in the first place, except now it hasn’t even finished recovering yet.</p><p>This is called a <strong>retry storm</strong>, or sometimes a thundering herd, and it’s a genuinely counterintuitive failure mode, because every individual client is doing something completely reasonable. Retrying after a failure is the right instinct. The problem only shows up at the aggregate level, when everyone’s individually reasonable behavior lines up perfectly and stacks into one coordinated spike.</p><h3>Exponential Backoff &amp; Jitter</h3><p>The fix for retry storms is two ideas working together, and they show up in almost every serious retry implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r9VXygind1XMuj3dILcmXg.png" /><figcaption><em>Each retry waits longer than the last, and a small random amount is added to that wait so different clients don’t retry at the exact same instant.</em></figcaption></figure><p><strong>Exponential backoff</strong> means each successive retry waits longer than the one before it, typically doubling: 1 second, then 2, then 4, then 8. This spaces out the load a struggling service receives instead of hammering it at a constant rate, and it gives the service progressively more breathing room the longer it stays unhealthy.</p><p><strong>Jitter</strong> adds a small amount of randomness to that wait time. Instead of every client waiting exactly 4 seconds, each one waits something like 4 seconds plus or minus a random amount. This is the piece that actually solves the retry storm specifically, because even if a thousand clients all failed at the same moment and are all backing off exponentially, jitter spreads their retry attempts across a window of time instead of letting them land on the exact same instant.</p><p>Backoff alone reduces the total load. Jitter is what prevents that load from arriving synchronized. Most production-grade retry libraries implement both by default, and it’s worth checking that whatever library or framework you’re using actually does this rather than assuming it.</p><h3>Retries + Circuit Breakers Working Together</h3><p>These two patterns aren’t competing with each other, they’re layered. A circuit breaker decides whether an attempt should be made at all. Retries decide what to do after an attempt fails, assuming the breaker let it through in the first place.</p><p>In practice, this usually means retries happen while the circuit is closed, since that’s when calls are actually reaching the real service. Once the breaker trips open, retrying is pointless, the breaker is already telling you the service is unhealthy, and sending more attempts, retried or not, only adds to the exact problem the breaker exists to prevent. A well-built client checks the breaker state first, and only bothers with retry logic when the breaker says the call is worth attempting.</p><h3>Why Retries Need Idempotency</h3><p>We flagged this earlier, but it’s worth being explicit about before we move on, because it’s the single most common way retry logic goes wrong in production.</p><p>Retrying is only safe when repeating the operation produces the same end result as running it once. Reading data is naturally safe to retry, asking for the same thing twice doesn’t change anything. Writing data is where it gets dangerous. If a payment request times out after the charge actually went through on the server, and your client retries because it never saw the success response, you can end up charging someone twice for something that only happened once from their perspective.</p><p>This isn’t a retry problem exactly, it’s a design problem that retries expose. The fix lives on the server side, in making operations idempotent so that receiving the same request twice has the same effect as receiving it once. That’s a big enough topic to deserve its own full treatment, which is exactly where we’re headed next.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d540532a5038" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Circuit Breakers: Failing Fast on Purpose]]></title>
            <link>https://medium.com/@iam-abdulmoiz/circuit-breakers-failing-fast-on-purpose-2258a6cb58f3?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2258a6cb58f3</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[resiliencepatterns]]></category>
            <category><![CDATA[circuit-breaker]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[fault-tolerance]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 04:01:02 GMT</pubDate>
            <atom:updated>2026-08-05T04:01:02.098Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>Once you’ve dealt with scaling reads and writes, the next question is what happens when one of your services just starts misbehaving. Not the whole database going down, just one dependency, one API, one downstream call that’s suddenly slow or broken. That’s where circuit breakers come in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1y8-8rJ_k7QoH5gcave7zw.png" /></figure><h3>What Happens When One Service Keeps Failing</h3><p>Picture a normal request path. Your app calls a payment service to charge a customer. Most of the time this works fine, the payment service responds in 100ms and life goes on.</p><p>Now imagine the payment service starts having a bad day. Maybe its own database is struggling, maybe it’s overloaded. Every call to it now takes 30 seconds to time out instead of 100ms.</p><p>Here’s the part people don’t expect: your app doesn’t just get slow, it can go down entirely, even though nothing is actually wrong with your app’s own code. If a hundred requests per second are hitting a checkout flow that calls this slow payment service, and each one now takes 30 seconds instead of 100ms, you suddenly have thousands of requests all stuck waiting at the same time. Threads pile up. Memory fills. Eventually your app runs out of capacity to accept new requests at all.</p><p>One slow dependency just took down a service that was otherwise perfectly healthy. This is called a <strong>cascading failure</strong>, and it’s one of the more frustrating things to debug in production, because the actual bug isn’t in the service that crashed, it’s in whatever it was waiting on.</p><h3>The Circuit Breaker Analogy</h3><p>The name comes from actual electrical circuit breakers in your house. When something draws too much current, a wire overheating, a short circuit, the breaker trips and cuts power immediately, rather than letting the wire keep heating up until it catches fire.</p><p>A software circuit breaker does the same thing for a network call. Instead of letting every request keep hammering a service that’s clearly struggling, it “trips” after enough failures and starts rejecting calls immediately, without even trying to reach the failing service. No 30-second wait, no thread stuck. Just an instant “not right now” response.</p><p>This might sound counterintuitive at first. Isn’t giving up on purpose worse than trying? Not when the alternative is bringing down your entire app while you wait on something that’s very likely going to fail anyway.</p><h3>The Three States: Closed, Open, Half-Open</h3><p>Every circuit breaker implementation, whether it’s Hystrix, resilience4j, or something hand-rolled, is built around the same three states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UW6Egu2AKCGL9aNk_KQ7sA.png" /><figcaption><em>A circuit breaker moves between three states based on whether the calls it’s guarding are succeeding or failing.</em></figcaption></figure><p><strong>Closed</strong> is the normal, healthy state. Requests pass through to the real service, and the breaker just watches the outcomes in the background, counting failures.</p><p><strong>Open</strong> is the tripped state. Once failures cross a threshold, the breaker flips open and starts rejecting calls instantly, without even attempting the network call. This is what protects the rest of your system from getting dragged down.</p><p><strong>Half-Open</strong> is the recovery test. After some cooldown period, the breaker cautiously lets a small number of requests through to see if the downstream service has actually recovered. If those test calls succeed, it closes again and resumes normal traffic. If they fail, it goes straight back to open and waits longer before trying again.</p><h3>Closed State: Normal Operation</h3><p>In the closed state, nothing about the request path looks any different to an outside observer. Calls go out, responses come back, everything works as expected.</p><p>What’s happening quietly underneath is that the breaker is tracking a rolling window of recent outcomes, usually something like “how many of the last 20 calls failed” or “what percentage of calls in the last 10 seconds errored out.” As long as that number stays under a configured threshold, the breaker stays closed and stays out of the way.</p><p>The moment that threshold is crossed, say 50% of calls failing in the last 10 seconds, the breaker trips into the open state.</p><h3>Open State: Failing Fast</h3><p>This is the state that actually protects your system, and it’s worth seeing exactly what happens to a request once the breaker is open.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TRrG5sxn8WBVx374VVcDPQ.png" /><figcaption><em>Once the breaker is open, requests never even reach the failing service. They fail instantly, and the caller gets an immediate response instead of a long wait.</em></figcaption></figure><p>The client sends a request like normal. But instead of forwarding it to the struggling downstream service, the breaker intercepts it immediately and returns an error or a fallback response, in milliseconds rather than seconds. No thread sits around waiting on a timeout. No connection pool gets exhausted. The downstream service, which is probably already overwhelmed, gets a break too, since it’s not receiving any traffic from this caller while the breaker is open.</p><p>This is the part that trips people up the first time they see it: the fallback response the client gets back isn’t a bug, it’s the entire point. A fast, predictable failure is almost always better for the overall system than a slow, unpredictable one. What that fallback actually looks like depends on the use case, sometimes it’s cached data from the last successful call, sometimes it’s a simplified default, sometimes it’s just a clear error the caller can handle gracefully.</p><h3>Half-Open State: Testing Recovery</h3><p>A breaker can’t stay open forever, at some point it needs to check whether the downstream service has actually recovered. That’s what half-open is for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ofguMS5g3QF78uy5Mlb4iA.png" /><figcaption><em>After a cooldown, the breaker cautiously lets a single trial request through while continuing to block the rest, to test recovery without fully reopening the floodgates.</em></figcaption></figure><p>After the configured cooldown window passes, the breaker allows a small number of requests, often just one, to actually reach the downstream service. Everything else during this window is still blocked, exactly like in the open state.</p><p>If that trial request succeeds, the breaker treats it as a sign the service is healthy again and closes, resuming normal traffic. If it fails, the breaker immediately snaps back to open and waits through another cooldown period before trying again.</p><p>This cautious approach matters more than it might seem. If the breaker just fully reopened after the cooldown and let all traffic back in at once, and the downstream service was only barely recovering, you’d slam it right back into failure, and it would never get a real chance to stabilize. Testing with a trickle of requests first avoids that.</p><h3>Choosing Thresholds</h3><p>The behavior of a circuit breaker comes down entirely to a handful of configuration choices, and getting these wrong is a common source of headaches.</p><p><strong>Failure rate threshold</strong> is how many failures, or what percentage of failures, trips the breaker. Set it too low and you’ll trip on normal, occasional blips that would’ve resolved on their own. Set it too high and the breaker won’t protect you until real damage is already happening.</p><p><strong>Time window</strong> defines what “recent” means when counting failures. A short window makes the breaker react quickly but makes it jumpy, tripping on brief spikes. A longer window smooths that out but means it takes longer to notice a real problem.</p><p><strong>Cooldown duration</strong> is how long the breaker stays open before testing recovery. Too short, and it’ll keep flapping between open and half-open before the downstream service has genuinely stabilized. Too long, and you’re needlessly rejecting traffic even after the service has recovered.</p><p>There’s no universal correct value for any of these, they depend on the specific service, its normal latency, and how much traffic it handles. Most teams start with sensible defaults from whatever library they’re using and tune based on what they actually observe in production.</p><h3>Circuit Breakers in a Real Request Path</h3><p>Let’s put this in the context of an actual system, rather than an isolated client-and-service pair.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oGUWr3_HWbb07TgRsuH2Bg.png" /><figcaption><em>A circuit breaker usually sits wrapped around one specific outbound call, guarding just that dependency, while a monitor tracks how often it trips.</em></figcaption></figure><p>Users hit an API Gateway, which routes to Service A. Service A needs data from Service B to do its job, so it calls Service B. That call to Service B is exactly where the circuit breaker sits, wrapped around this one specific dependency, not around Service A as a whole.</p><p>This detail matters. Circuit breakers are usually applied per-dependency, not globally. Service A might call three or four different downstream services, and each of those calls typically gets its own breaker, tracking its own failure rate independently. If Service B is having problems, its breaker trips, but the breaker guarding a completely different dependency stays closed and unaffected.</p><p>A failure monitor sitting alongside tracks how often breakers are tripping across the system. This matters operationally, a breaker that trips occasionally is doing its job. A breaker that’s open most of the time is telling you something is seriously wrong with that dependency, and it deserves attention beyond what the breaker alone can fix.</p><h3>What Circuit Breakers Don’t Solve</h3><p>It’s worth being honest about the boundaries here, because circuit breakers are often talked about alongside a few other patterns, and it’s easy to assume one solves what the others actually handle.</p><p>A circuit breaker decides whether to attempt a call at all. It doesn’t decide how long to wait for a response before giving up, that’s a timeout’s job. It doesn’t decide whether to try again after a failure, that’s what retries handle. And it doesn’t limit how many resources a single dependency can consume even while things are healthy, that’s what bulkheads are for.</p><p>These patterns are siblings, not substitutes. A circuit breaker without a sensible timeout underneath it is still vulnerable to slow responses piling up before the failure threshold is even reached. And once a breaker does let calls through again, whether those calls get retried intelligently is a completely separate question, one we’re covering next.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2258a6cb58f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scaling Reads vs Writes: The Complete Guide to Replication and Failover]]></title>
            <link>https://medium.com/@iam-abdulmoiz/scaling-reads-vs-writes-the-complete-guide-to-replication-and-failover-1c6b9cfef9f2?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c6b9cfef9f2</guid>
            <category><![CDATA[database-replication]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[vertical-scaling]]></category>
            <category><![CDATA[horizontal-scaling]]></category>
            <category><![CDATA[scaling-replication]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 04:01:02 GMT</pubDate>
            <atom:updated>2026-08-03T04:01:02.932Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>System Design Series · Module 5: Scaling &amp; Reliability Patterns</em></strong></p><p>Scaling reads and writes is one of the most important concepts in backend engineering and system design. Every large application, from WhatsApp to Netflix to Amazon, eventually faces this problem.</p><p>I’ll explain it as if we’re building a real application together. We’ll start from absolutely zero and gradually move toward how companies handle billions of requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSK3-cRNVbhofig3ctYZ0g.png" /></figure><h3>What Does “Scaling” Actually Mean?</h3><p>Before talking about reads and writes, let’s first understand why scaling even exists.</p><p>Imagine you own a small grocery shop. At first, only 20 customers visit every day. One cashier is enough. Customers walk in, get billed, walk out. Everything works perfectly.</p><p>Now imagine your shop becomes famous. Instead of 20 customers, 20,000 people show up every day. What happens? People stand in long lines. The cashier becomes slow. Some customers leave without buying anything. The business loses money.</p><p>Here’s the important bit: the cashier isn’t bad at their job. There are simply too many customers for one person to handle.</p><p>The exact same thing happens to servers. Suppose you have a simple backend.</p><pre>Users → Backend Server → Database</pre><p>If a hundred users use your app every day, everything works fine. But the moment a million users show up, the server and the database start struggling under the weight.</p><p>That’s where scaling begins.</p><p><strong>Scaling simply means making the system capable of handling more users without becoming slow or crashing.</strong></p><h3>What Are Reads and Writes?</h3><p>A database really only does two kinds of things at its core: giving you data back, or changing the data it holds. Everything else is a variation of these two operations.</p><p><strong>A read means “give me some information.”</strong></p><p>You open Instagram and it shows you a profile. Behind the scenes, that’s a query like:</p><pre>SELECT * FROM users WHERE id = 5;</pre><p>That’s a read. Opening YouTube and seeing recommended videos, checking your notifications, scrolling through comments, loading a dashboard, none of that changes anything in the database. It just fetches what’s already there.</p><p><strong>A write means “change the data.”</strong></p><p>Creating an account, changing your password, deleting a comment, liking a post, uploading a photo, all of these modify what’s stored. Under the hood they show up as INSERT, UPDATE, or DELETE statements.</p><p>Here’s the simplest way to picture the split between the two:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z-OjoqcBRGC8pv7N0DNiZQ.png" /><figcaption><em>Every incoming request eventually splits into a read path or a write path, and the database treats them very differently.</em></figcaption></figure><h3>Why Reads Happen Far More Than Writes</h3><p>This is one of the biggest ideas in system design, and once it clicks, a lot of architecture decisions start making sense on their own.</p><p>Think about Instagram. One celebrity uploads one photo. That’s a single write. Then fifty million people open the app and view that photo. That’s fifty million reads, all pointing at data that was written exactly once.</p><p>One write. Millions of reads.</p><p>This pattern holds for almost every consumer app you can think of: Facebook, LinkedIn, Twitter, news websites, e-commerce platforms, Netflix, blogs. Most of the traffic these systems deal with is people looking at things, not people changing things.</p><p>This matters because it tells you where to spend your engineering effort first. If reads dominate your traffic, the cheapest win is making reads scale, and as you’ll see shortly, reads are also the easier of the two problems to solve.</p><h3>Why the Database Becomes the Bottleneck</h3><p>Your backend servers are usually stateless, which is a fancy way of saying that any server can handle any request, because none of them are holding on to information that only they know about. That makes backend scaling almost boring. Traffic goes up, you add more servers behind a load balancer, done.</p><p>The database is a different story. No matter how many backend servers you add, they all typically talk to the same database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tbb3-r95FmNmojx99rQtFg.png" /><figcaption><em>Application servers scale out easily, but they all still point at one database, and that database eventually runs out of room.</em></figcaption></figure><p>Picture a users table with a hundred rows. Finding one specific user is instant. Now picture that same table with a hundred million rows, and a hundred thousand people searching it at the exact same moment. The CPU gets busy. The RAM fills up. The disk works overtime. The network gets saturated. Response times climb.</p><p>This is why scaling databases is a genuinely harder problem than scaling application servers. The backend is disposable and interchangeable. The database is the one thing every request eventually has to go through.</p><h3>Vertical vs Horizontal Scaling</h3><p>There are two fundamental ways to deal with a system that’s running out of capacity, and they solve the problem in very different ways.</p><p><strong>Vertical scaling (scaling up)</strong> means making one machine stronger. Instead of a server with 4 GB of RAM and 2 CPUs, you buy one with 64 GB of RAM and 32 CPUs. It’s the easiest option because you don’t touch the application code at all, you just move to bigger hardware.</p><p><strong>Horizontal scaling (scaling out)</strong> means adding more machines instead of buying a bigger one. Traffic gets spread across several servers instead of piling onto one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTuShA5t7OBAVD6iGqjy2Q.png" /><figcaption><em>Vertical scaling makes one machine bigger. Horizontal scaling spreads the same load across many smaller machines.</em></figcaption></figure><p>Vertical scaling is simple, but it has a ceiling. Eventually hardware just can’t get any bigger, and the machines that come close to the limit get disproportionately expensive. Worse, if that one machine goes down, your entire application goes down with it. There’s no backup plan built in.</p><p>Horizontal scaling avoids that single point of failure and can, in theory, keep growing forever by adding more boxes. The catch is that it only works cleanly when the pieces being scaled don’t depend on shared state, which is exactly why backend servers scale out so easily and databases don’t.</p><h3>Scaling Reads: Database Replication</h3><p>Since we already know that most traffic is reads, the first and easiest optimization is to make copies of the database and let those copies handle the reading.</p><p>This is called <strong>replication</strong>, and the setup usually looks like a Primary (sometimes called the Leader) and one or more Replicas (sometimes called Followers).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9X12X8kUHueMoJiqwKgqXQ.png" /><figcaption><em>The Primary accepts every write. Replicas hold read-only copies of the data and serve the read traffic instead.</em></figcaption></figure><p>The Primary is the only one allowed to accept writes. Once a write is saved on the Primary, it gets copied over to every Replica. Replicas don’t accept writes themselves, they just stay in sync and answer read queries.</p><p>Walk through a real example. A user uploads a photo, that write goes straight to the Primary. Thousands of people then open their feed and view that photo, and those reads get spread out across several Replicas instead of piling onto one machine.</p><p>The write workload stays concentrated on a single machine, while the read workload, which is the much bigger chunk of traffic, gets shared across many machines. That’s a huge jump in how many users your system can serve.</p><h3>How Replication Actually Works Internally</h3><p>You might assume replication means copying the entire database over and over, but that would be painfully slow for anything beyond a toy project. What actually happens is smarter.</p><p>Every database keeps something called a <strong>Write-Ahead Log</strong>, or WAL (MySQL calls its version a binlog). Every single change, every insert, update, or delete, gets written to this log first, in order, before anything else happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnjR_KWGB4oTPrZA2MmRPQ.png" /><figcaption><em>The Primary writes every change to its log first. Replicas stream that log and replay the same changes locally instead of copying the whole database.</em></figcaption></figure><p>So the actual flow looks like this: a write comes in, the Primary creates a new WAL entry, say entry #1051, and applies the change locally. Replicas are constantly streaming that log and replaying each entry on their own copy of the data.</p><p>This means replicas aren’t copying the database, they’re replaying history. That’s a much cheaper operation, and it’s also why replication can keep up with a high-write system without falling permanently behind.</p><h3>Synchronous vs Asynchronous Replication</h3><p>Here’s a question that comes up the moment you build replication into a real system: does the Primary wait for the replicas to confirm they’ve received the update before telling the client “success,” or does it just say “success” immediately and let replication happen in the background?</p><p>Both approaches exist, and the choice between them is a genuine trade-off, not a solved problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CsSeRdLAi55W8fnYSrPKJg.png" /><figcaption><em>Synchronous replication waits for replica acknowledgment before confirming the write. Asynchronous replication confirms immediately and lets replicas catch up afterward.</em></figcaption></figure><p>With <strong>synchronous replication</strong>, the Primary holds off on responding until at least one replica acknowledges it received the change. This guarantees that a confirmed write genuinely exists in more than one place, but it comes at the cost of added latency, because now every write has to wait on a network round trip to another machine.</p><p>With <strong>asynchronous replication</strong>, the Primary saves the change, tells the client it succeeded, and pushes the update to replicas afterward, without waiting. Writes feel faster to the user, but there’s a small window where the replicas are behind the Primary.</p><p>Most large-scale systems lean asynchronous because the latency savings are worth it, and they build other safety nets around the small window of risk that creates.</p><h3>Replication Lag &amp; Read Routing</h3><p>That “small window of risk” from asynchronous replication has a name: <strong>replication lag</strong>. It’s the gap between when a write lands on the Primary and when that same write shows up on a given replica.</p><p>Now stretch this out across a system with several replicas serving reads, and you get a routing problem. Which replica should handle a given read, and what happens if that replica happens to be a little behind?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hCVFBKJt_daP4__DBh34A.png" /><figcaption><em>A read router picks a replica per request. Some replicas are fully caught up, others carry a small delay, and reading from a lagging replica can return stale data.</em></figcaption></figure><p>This delay is usually measured in milliseconds, sometimes a couple of seconds under heavy load, and it’s often called <strong>eventual consistency</strong>: the data will be correct eventually, just maybe not the exact instant it was written.</p><p>In practice this shows up as small, occasionally confusing bugs. A user uploads a profile photo, then immediately refreshes their profile and the old photo is still there for a second. This is called the <strong>read-after-write</strong> problem, and there are a couple of common fixes: route a user’s own reads back to the Primary right after they write something, or update the interface optimistically on the frontend the moment the write succeeds, without waiting for the read to confirm it. Both approaches trade a bit of engineering complexity for a smoother experience.</p><h3>Scaling Writes: Why Replicas Alone Aren’t Enough</h3><p>Everything up to this point has been about reads, and for good reason, since reads are most of the traffic. But writes eventually become a bottleneck of their own, and replication doesn’t fix that.</p><p>Here’s why: every replica in a Primary-Replica setup ultimately traces back to the same single Primary for writes. You can add as many replicas as you want and it won’t add a single unit of write capacity, because none of them are allowed to accept writes independently.</p><p>Scaling writes needs a fundamentally different technique, usually sharding, where the data itself gets split across multiple independent databases so that each one only handles a portion of the total write load. That’s a big enough topic that it deserves its own article rather than a rushed mention here, so we’ll leave it as the natural next step and focus the rest of this one on what happens when the single Primary you do have stops working.</p><h3>When the Primary Dies: Failover</h3><p>Servers are just computers, and computers fail. Power goes out, disks die, networks drop. If your Primary database goes down and there’s no plan for it, every write request in your application starts failing immediately, because there’s no other machine allowed to accept writes.</p><p>The fix is <strong>failover</strong>: promoting a healthy replica to become the new Primary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8l8QoF4zXXEH0tCdmDDcA.png" /><figcaption><em>The system detects the Primary is gone, compares how up to date each replica is, and promotes the freshest one to take over.</em></figcaption></figure><p>But this raises an obvious question: how does the system even know the Primary is down, and who decides which replica takes over?</p><p><strong>Detecting the failure</strong> relies on heartbeats, small “I’m alive” signals the Primary sends out constantly, sometimes every second or even more frequently. If several heartbeats in a row go missing, the system starts suspecting a real failure rather than a momentary blip. That word, suspecting, matters. A single missed heartbeat could just mean the network was briefly slow, not that the Primary actually crashed. Acting too fast on a false alarm is exactly how you end up with two Primaries at once, which we’ll get to shortly.</p><p><strong>Choosing the replacement</strong> comes down to comparing WAL positions. Remember, every write gets a sequential log entry. If the old Primary had processed up through entry #1052, and Replica A also has entry #1052 while Replica B only has up to #1051, Replica A is clearly the safer promotion, since promoting it loses less committed data.</p><p>Once a replica is chosen, it starts accepting writes, and the application reconnects to point at the new Primary. There’s usually a short gap, often just a few seconds, where writes fail or automatically retry while this whole process plays out. Modern managed databases have gotten this down remarkably fast, but it’s rarely instant.</p><h3>Split Brain: Why Multiple Primaries Are Dangerous</h3><p>Now imagine the coordination step goes wrong, and instead of the system agreeing on exactly one new leader, two replicas both decide independently that they’re the new Primary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4oSKXAjn2koS1UMBz9cfVg.png" /><figcaption><em>A network partition can leave two databases each believing they’re the leader, and both start accepting writes independently.</em></figcaption></figure><p>This is called <strong>split brain</strong>, and it’s one of the nastier failure modes in distributed systems. Picture a network partition that splits your replicas into two groups that can’t talk to each other. Each group still sees a “dead” Primary and independently promotes its own leader. Now you have Primary A and Primary B, both accepting writes, both changing the same logical dataset in different, disconnected ways.</p><p>A post count might read 101 on one side and 102 on the other. There’s no clean way to merge that back together once communication is restored, because both values are individually valid, just based on incomplete information. This is exactly why leader election algorithms are built to be conservative: waiting a bit longer to confirm a failure is a much smaller cost than accidentally creating two Primaries.</p><p>Different databases handle the actual coordination differently. MongoDB has automatic leader election baked into Replica Sets. CockroachDB coordinates leadership internally across nodes. PostgreSQL typically leans on external tools like Patroni to manage this. The implementations vary, but the underlying goal is identical: detect the failure, agree on exactly one leader, and never let two machines believe they’re in charge at the same time.</p><h3>Tying It All Together</h3><p>Let’s zoom out and put every piece from this article into one picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eh4A4wbxg1V2Q0uXYGmHmg.png" /><figcaption><em>Users hit a load balancer, which spreads requests across stateless app servers. Writes go to the Primary, reads get distributed across replicas, and failover keeps the system alive if the Primary ever goes down.</em></figcaption></figure><p>A handful of ideas run underneath everything we covered, and they’re worth naming directly because you’ll keep running into them as you go deeper into distributed systems.</p><p><strong>Single Point of Failure (SPOF)</strong> is any one component whose failure takes down the whole system. A lone database with no replicas is a SPOF. Replication plus failover is how you remove it.</p><p><strong>High Availability (HA)</strong> is the property of staying up and serving traffic even when individual pieces fail. It’s not a single technique, it’s the outcome of combining replication, failover, and good monitoring.</p><p><strong>Source of truth</strong> is whichever copy of the data is considered authoritative at a given moment. In a Primary-Replica setup, that’s always the Primary, right up until failover hands the title to a promoted replica.</p><p><strong>Durability</strong> means a write, once acknowledged, survives crashes. This is why the WAL exists: it lets a database recover exactly where it left off instead of losing data mid-write.</p><p><strong>Acknowledged write loss</strong> is the specific danger asynchronous replication introduces: if the Primary confirms a write and then dies before that write reaches any replica, that write is gone, even though the client was told it succeeded. It’s a real trade-off, not a hidden bug, and it’s the reason some systems choose synchronous replication for their most sensitive data despite the latency cost.</p><p>None of this is about memorizing definitions. It’s about recognizing that reads and writes are genuinely two different problems, wearing the same word, “scaling,” but demanding completely different solutions. Reads scale by copying data. Writes scale by splitting it. And keeping the whole thing alive when a piece fails is a third problem entirely, one that replication alone doesn’t solve without failover sitting on top of it.</p><p>Understanding this distinction is the foundation for everything ahead in this module: circuit breakers, backpressure, idempotency, contention handling, and graceful degradation all assume you already understand why reads and writes needed different treatment in the first place.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c6b9cfef9f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Region Architecture Active-Active , Active-Passive & Geo-Distribution]]></title>
            <link>https://medium.com/@iam-abdulmoiz/multi-region-architecture-active-active-active-passive-geo-distribution-2e9d597f23a8?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e9d597f23a8</guid>
            <category><![CDATA[multi-region]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 04:01:01 GMT</pubDate>
            <atom:updated>2026-07-30T04:01:01.856Z</atom:updated>
            <content:encoded><![CDATA[<p>At some point, one data center stops being enough. Maybe your users are now global and a round trip to a single US region is adding real latency for someone in Singapore. Maybe your last outage took down the entire product because that one data center had a bad night. Either way, the next step is the same: run in more than one region. The moment you do, a whole new category of problems shows up, problems that don’t exist when everything lives in one place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PuFNMfBjxrcf-yOw8zMzw.png" /></figure><h3>Active-Passive, the Conservative Starting Point</h3><p>The simplest multi-region setup keeps one region fully active, handling all real traffic, while a second region sits as a warm standby, continuously replicating data but not serving anyone. If the primary region goes down, you fail over, traffic redirects to the standby, which now becomes active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wH6J2uyCC3uPPZ9ysR_O5Q.png" /><figcaption><em>US-East handles 100% of traffic while EU-West stays synced as a warm standby, ready to take over if the primary region fails.</em></figcaption></figure><p>Iska sabse bada faida yeh hai ke yeh samajhna aasan hai, koi do writers same data pe fight nahi kar rahe, koi conflict resolution ki zaroorat nahi. There’s exactly one place accepting writes at any moment, so you get all the simplicity of a single-region system with a safety net bolted on. The obvious cost: you’re paying for infrastructure in a second region that sits mostly idle, and failover isn’t instant, there’s a real gap, seconds to minutes depending on how automated it is, where the primary is down and the standby hasn’t taken over yet. For most companies just starting to think about multi-region, this is still the right first move. It solves the disaster recovery problem without introducing the harder problem you’re about to read about.</p><h3>Active-Active, Where Both Regions Actually Serve Traffic</h3><p>Active-active means exactly what it sounds like, both regions accept live traffic and both accept writes, simultaneously. A user in Chicago gets routed to the US region, a user in Berlin gets routed to the EU region, and both regions are fully “on” at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Jgf9HBxM09600B79AI9gA.png" /><figcaption><em>Both regions serve local users and accept local writes. Both databases sync with each other, which means both can receive conflicting writes to the same record at nearly the same time.</em></figcaption></figure><p>This gets you two real things active-passive can’t: lower latency for users near either region, since nobody’s making a cross-ocean round trip, and no idle infrastructure, both regions are pulling their weight. The price is the problem every active-active system eventually has to face, what happens when the same record gets written in both regions before either write has had time to propagate to the other. That’s not a hypothetical edge case, at global scale it happens regularly, and it’s the whole reason the next two sections exist.</p><h3>Routing Users to the Right Region in the First Place</h3><p>Before you can even talk about conflicts, you need traffic actually landing in the region that makes sense for it. This is usually handled by geo-DNS or a latency-aware load balancer sitting in front of everything, routing each user to whichever region is physically closest to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g3B57AmYzqCQuZg_SV_e-g.png" /><figcaption><em>A user in Chicago, Berlin, and Singapore each resolve to a different region automatically, based on where the request is actually coming from.</em></figcaption></figure><p>The mechanism is usually anycast IP addresses or DNS resolution that returns a different IP depending on the resolver’s location, sometimes combined with actual latency measurements rather than pure geography, since geographically close doesn’t always mean network-close. Yeh layer users ko kabhi dikhta nahi, but it’s doing real work, quietly deciding which data center every single request even reaches before anything else in your architecture gets involved.</p><h3>When Two Regions Disagree, Someone Has to Decide Who Wins</h3><p>Back to the hard part. Say a customer opens your app on their phone in the US and their laptop in Germany within the same second, changing their shopping cart on both. Both writes land, one in each region, before replication has a chance to catch up. Now both databases have a version of the truth, and they disagree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZD5tz3jkfJnOVi1uMXEQZQ.png" /><figcaption><em>Two regions write to the same cart within 13 milliseconds of each other. A conflict resolver has to pick a winner, here using the simplest strategy, whichever write has the later timestamp.</em></figcaption></figure><p>The most common resolution strategy is last-write-wins, LWW for short, every write carries a timestamp, and whichever one is later gets kept when the conflict surfaces. It’s simple to implement and it’s what a lot of production systems actually use, but it has a real weakness: it can silently drop a legitimate change. In the cart example, the EU removal genuinely happened, a real person did it, and it just quietly lost because it was 13 milliseconds earlier. For low-stakes data that’s a reasonable trade. For anything where silently discarding a user’s action is unacceptable, teams reach for more careful approaches, vector clocks that can detect a conflict exists without automatically resolving it, or CRDTs (conflict-free replicated data types), data structures specifically designed so that merges are always well-defined and nothing gets silently lost, common in collaborative editing tools where “your edit” and “their edit” both need to survive.</p><h3>Choosing Between Them Isn’t About Which Is Better</h3><p>Active-passive and active-active aren’t a strictly-better-strictly-worse pair, they’re answers to different questions. If your priority is disaster recovery and your traffic is concentrated in one region anyway, active-passive gets you most of the safety with a fraction of the complexity. If your users are genuinely global and low latency matters everywhere, active-active is worth the conflict-resolution tax, as long as you go in knowing that tax exists and picking a resolution strategy on purpose, rather than discovering you needed one after two regions have already quietly overwritten each other’s data in production.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e9d597f23a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Consistency Models Strong, Eventual, Casual & Read-Your-Writes]]></title>
            <link>https://medium.com/@iam-abdulmoiz/data-consistency-models-strong-eventual-casual-read-your-writes-0422ecf49b6b?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/0422ecf49b6b</guid>
            <category><![CDATA[data-consistency]]></category>
            <category><![CDATA[database-design]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[eventual-consistency]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 04:01:03 GMT</pubDate>
            <atom:updated>2026-07-29T04:01:03.167Z</atom:updated>
            <content:encoded><![CDATA[<p>Ask five engineers what “consistency” means and you’ll get five slightly different answers, and honestly, all five might be right. Consistency isn’t one thing, it’s a spectrum, and where your system sits on that spectrum is one of the most consequential decisions you’ll make, usually without realizing you made it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vrc3FlDelyd6YYitEUSKhw.png" /></figure><p>We’ve already covered CAP theorem in this series, strong versus available, pick your poison during a partition. This article goes a level deeper into what “consistent” actually means once you’re not in a partition, because there isn’t just one answer.</p><h3>Strong Consistency, the One Everyone Assumes by Default</h3><p>Strong consistency is the simplest to describe and the most expensive to actually deliver: every read reflects the most recent write, immediately, no matter who’s reading or where they’re reading from. There’s effectively one source of truth, and everyone talks to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VU-wVITyWWfJ6aIZxb_fEw.png" /><figcaption><em>With one source of truth and no replication lag, every client sees the same value the instant it’s written, whenever they ask.</em></figcaption></figure><p>Yeh sabse simple lagta hai samajhne mein, but the cost is real. If there’s genuinely one place data lives, that one place is a bottleneck and a single point of failure. Systems that offer strong consistency across multiple nodes (think a properly configured relational database with synchronous replication, or the consensus-backed systems from the last article) are paying for it in latency, every write waits for enough nodes to agree before it’s considered done. Strong consistency isn’t free, it’s a tradeoff you’re actively choosing, usually because correctness matters more than raw speed for that particular piece of data. Bank balances, inventory counts you’re about to sell out of, anything where “slightly stale” causes real damage.</p><h3>Eventual Consistency, the One That Actually Powers Most of the Internet</h3><p>Eventual consistency drops the “immediately” and keeps a promise instead: if no new writes come in, every replica will eventually converge to the same value. Not now. Eventually. Usually within milliseconds to seconds in practice, but there’s a window where different replicas can legitimately disagree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dN-aK_QszhlzidPY9LJDDw.png" /><figcaption><em>Right after a write, replicas can briefly disagree. Given a moment to sync, they converge to the same value.</em></figcaption></figure><p>This sounds alarming until you notice how much of the software you use daily already runs on it. Your social media follower count, product view counts, DNS records, most caching layers, they’re all eventually consistent, and nobody’s complaining, because a follower count being off by one for two seconds genuinely doesn’t matter. The entire appeal here is availability and speed, writes don’t need to wait for a consensus round trip across the globe, they land locally and propagate afterward. The tradeoff is real though: any read during that propagation window can return stale data, and your application needs to be fine with that, not accidentally assume otherwise.</p><h3>Causal Consistency, the Middle Ground Nobody Talks About Enough</h3><p>Here’s where it gets genuinely interesting. Causal consistency doesn’t demand that every read sees every write instantly, but it does demand something specific: if event B happened because of event A, everyone has to see A before B. Events with no causal relationship to each other can arrive in any order, and that’s fine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VU7PH8H6YTjDXSsgadydoA.png" /><figcaption><em>A reply causally depends on the post it’s replying to. Causal consistency guarantees nobody ever sees the reply first. An unrelated like on a different post has no such dependency.</em></figcaption></figure><p>Think about a comment thread. If someone posts “just shipped v2!” and a friend replies “congrats!!”, any consistency model that lets a third person see the reply before the original post is going to look broken, even if technically nothing was corrupted. Causal consistency exists specifically to prevent that kind of confusing-but-not-wrong experience. Iska matlab yeh nahi ke sab kuch strictly ordered ho, sirf woh cheezein order mein rahengi jo ek doosre par depend karti hain. An unrelated like on a totally different post can show up whenever, nobody’s causally waiting on it. This is why comment threads, chat apps, and collaborative editors often reach for causal consistency specifically, it gives you the ordering guarantee that actually matters to a human reading the result, without paying for full strong consistency everywhere.</p><h3>Read-Your-Writes, the One That Breaks in the Most Embarrassing Way</h3><p>This is a narrower, very practical guarantee: after you write something, you personally should always see that write in your own subsequent reads, even if the system as a whole is only eventually consistent for everyone else. Sounds obvious. Gets violated constantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LmLmwPJFDKA8IiikD8uxXg.png" /><figcaption><em>A write lands on Replica A. A follow-up read gets load-balanced to Replica B, which hasn’t synced yet. The user updated their own profile and then watched it revert.</em></figcaption></figure><p>The classic failure mode: you update your profile picture, the write lands on Replica A, the load balancer then routes your next page load to Replica B, which hasn’t caught up yet, and you see your old picture again. Nothing is technically broken, the system will converge eventually, but from your perspective you just watched your own change get undone, which is a genuinely confusing and trust-eroding experience.</p><p>The fix is usually some form of session affinity, routing a given user’s reads back to whichever replica handled their most recent write, at least until replication catches up, or tracking a version/timestamp the client has already seen and refusing to serve anything older than that to them specifically. Neither is exotic, both require you to have actually decided this guarantee matters, which is the whole point of learning these models by name instead of discovering the gaps in production.</p><h3>Picking a Model Isn’t One Decision, It’s Several</h3><p>The mistake is treating “consistency” as one global setting for your whole system. It isn’t. A payments table might genuinely need strong consistency. A comment thread benefits from causal consistency far more than strong. A view counter is happily eventually consistent. A user’s own profile updates need read-your-writes even if nothing else in the system does.</p><p>Most real systems run a mix, strong where correctness is non-negotiable, eventual where availability and speed matter more, causal wherever humans are reading a sequence of related events, and read-your-writes layered in specifically for each user’s view of their own actions. Knowing the name for each of these isn’t academic, it’s what lets you have a precise conversation with your team about which guarantee a given piece of data actually needs, instead of everyone assuming “consistent” means the same thing.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0422ecf49b6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Trasnsactions 2PC, Saga & the Outbox Pattern]]></title>
            <link>https://medium.com/@iam-abdulmoiz/distributed-trasnsactions-2pc-saga-the-outbox-pattern-d160e8214eaa?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d160e8214eaa</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[transactional-outbox]]></category>
            <category><![CDATA[saga-pattern]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 04:01:02 GMT</pubDate>
            <atom:updated>2026-07-28T04:01:02.147Z</atom:updated>
            <content:encoded><![CDATA[<p>A customer clicks “Buy Now.” Behind that one click, three things need to happen: charge the payment, reserve the item, book a delivery slot. Each of those lives in a different service, with its own database. If all three succeed, great. But what happens when the third one fails after the first two already committed? That single question is what this entire article is about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*600ALlQUSatt1VJvb2VRGA.png" /></figure><h3>The Dual-Write Problem, Where All of This Actually Starts</h3><p>In a monolith with one database, this is trivial, wrap it in a transaction, it either all happens or none of it does. The moment you split into services with separate databases, that guarantee disappears, and nothing replaces it automatically.</p><p>Say the Order Service calls the Payment Service, gets a success back, then tries to call the Inventory Service, and the Order Service crashes right before that call goes out. The customer got charged. The item was never reserved. Nobody’s database is technically wrong, each one just saw its own transaction and committed it. The system as a whole is what’s broken.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiC39rA-Twywpc6cqD1ORA.png" /><figcaption><em>Payment commits successfully. The service crashes before the second write ever reaches Inventory. Two databases, two different truths.</em></figcaption></figure><p>Yeh dual-write problem hai, aur yeh sirf distributed transactions tak mehdood nahi, yeh kisi bhi jagah ho sakta hai jahan aap ek database aur ek message queue ko alag alag likh rahe ho. Two writes that need to succeed or fail together, with nothing enforcing that. There are a few real answers to it, and they trade off very differently.</p><h3>Two-Phase Commit, the Strict Answer</h3><p>Two-Phase Commit, almost always shortened to 2PC, is the oldest and most literal solution. One node acts as a coordinator. Every participant votes before anything actually commits.</p><p><strong>Phase 1, Prepare.</strong> The coordinator asks every participant “can you commit this?” Each participant does everything short of committing, locks the relevant rows, writes to its own log, and replies yes or no.</p><p><strong>Phase 2, Commit or Abort.</strong> If every participant voted yes, the coordinator tells everyone to commit for real. If even one voted no, everyone rolls back instead. Once the coordinator writes its commit decision, that decision is final, there’s no going back on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OIjAsrMcuMrQdqzYE5QOcQ.png" /><figcaption><em>All three services vote yes in Phase 1. Only then does the coordinator send the actual commit in Phase 2.</em></figcaption></figure><p>This gives you real atomicity, genuinely all-or-nothing, across multiple databases. The catch is the word “locks.” Every participant is holding locks on real data from the moment it votes yes until the coordinator’s final word arrives. If the coordinator crashes in between, those locks just sit there, held, blocking other transactions, until someone manually intervenes or the coordinator recovers. That’s why 2PC is called blocking, and it’s exactly why you almost never see it used across microservices in practice. It works fine when everything lives inside one database engine’s own transaction manager. It works badly the moment “everything” means five independently deployed services owned by five different teams.</p><h3>The Saga Pattern, Trading Strictness for Availability</h3><p>A Saga takes the opposite approach entirely. Instead of one giant transaction, you break the work into a sequence of small local transactions, each one committing immediately and independently. If something later in the chain fails, you don’t roll back, you run compensating transactions that undo the earlier steps.</p><p>Order placed, commits. Payment charged, commits. Inventory reserved, commits. Shipping fails, no delivery slot available. Now the system runs compensations in reverse, release the inventory, refund the payment. Nothing was ever locked waiting on a coordinator. Each step was real and final the moment it happened, and the “undo” is a new transaction, not a rollback of an old one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S5xhM50pYTXSUhjotD3MGg.png" /><figcaption><em>Each step commits on its own. When shipping fails, compensating transactions run backward through the steps that already succeeded.</em></figcaption></figure><p>Sagas can run in two styles. <strong>Choreography</strong> has each service publish an event and react to events from others, fully decentralized, nobody’s in charge. <strong>Orchestration</strong> has a central coordinator explicitly telling each service what to do next, which is easier to reason about once you have more than three or four steps, at the cost of that orchestrator becoming a critical piece of logic everyone depends on.</p><p>The tradeoff to be honest about: Sagas give you availability and no blocking locks, but you lose strict atomicity. There’s a real window where the order exists, payment is charged, but inventory hasn’t been reserved yet, and anything reading that data during that window sees a state that will later get partially undone. This is eventual consistency, not atomicity, and your system needs to be able to live with that gap.</p><h3>The Transactional Outbox, Solving a Smaller Problem Really Well</h3><p>2PC and Sagas both deal with coordinating multiple services. The Transactional Outbox solves a narrower, more common problem: how do you atomically update your database and publish an event about it, without a distributed transaction at all.</p><p>The trick is almost embarrassingly simple once you see it. Instead of writing to your database and then separately publishing to a message broker (two writes, same dual-write problem as before), you write to your database table and an outbox table in the exact same local transaction. Since they’re in the same database, that’s just a normal ACID transaction, no coordination needed. A separate relay process then polls the outbox table and publishes those rows to the actual message broker, marking them as sent once delivered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9ZxUbnQs75oviy5V04XBw.png" /><figcaption><em>The order and the outbox event are written in one local transaction. A relay process publishes from the outbox afterward, so the event only exists if the transaction actually committed.</em></figcaption></figure><p>The guarantee this buys you is exactly the one you actually needed: the event only ever exists if the local transaction committed. Agar transaction fail ho jaye, outbox row kabhi bani hi nahi, toh event kabhi publish hoga hi nahi. No distributed coordinator, no locks held across services, just one boring database transaction doing double duty.</p><h3>Picking the Right One</h3><p>None of these three compete for the same job, despite getting lumped together.</p><p>Reach for <strong>2PC</strong> only when every participant sits inside a compatible transaction manager and the transaction is short-lived, which in practice usually means it’s not crossing microservice boundaries at all.</p><p>Reach for a <strong>Saga</strong> when a business process genuinely spans multiple services and can tolerate a brief window of partial completion, as long as you can always define a sane compensating action for every step.</p><p>Reach for the <strong>Outbox pattern</strong> almost by default whenever a service needs to update its own data and notify others about it. It’s not solving the multi-service coordination problem, it’s quietly closing the dual-write gap that would otherwise sit underneath any Saga you build anyway.</p><p>Most real systems end up using Sagas for the big cross-service workflow and Outbox underneath each individual step to make sure that step’s own event never gets lost or duplicated. They’re not alternatives to each other so much as different layers of the same problem.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d160e8214eaa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consensus Algorithm | Raft & Paxos, Explained Simply]]></title>
            <link>https://medium.com/@iam-abdulmoiz/consensus-algorithm-raft-paxos-explained-simply-2eba1bd1df8a?source=rss-5db46112cc9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2eba1bd1df8a</guid>
            <category><![CDATA[raft]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[paxos]]></category>
            <category><![CDATA[consensus-algorithm]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Abdul Moiz]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 04:01:02 GMT</pubDate>
            <atom:updated>2026-07-27T04:01:02.042Z</atom:updated>
            <content:encoded><![CDATA[<p>If your service has ever gone down and come back up somewhere else without you doing anything, some algorithm made that decision for you. Not a person, not a script you wrote, an algorithm running across a bunch of machines that don’t fully trust each other, agreeing on one answer before anyone acts on it. That’s consensus, and it’s the part of distributed systems that quietly holds everything else together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvYkOJyrNoN3JbRuw9Kdng.png" /></figure><p>We’ve already talked about leader election in this series, but we skipped past the actual mechanics of how a group of machines picks a leader and agrees on it without two of them thinking they’re both in charge at the same time. That’s what this article is for.</p><h3>Why “Just Pick a Leader” Isn’t as Simple as It Sounds</h3><p>Picture five friends trying to decide where to eat. If they’re all in the same room, easy, someone shouts a name, everyone nods, done. Now put them in five different cities, on phone calls that sometimes drop, where a call dropping doesn’t tell you if the other person hung up or just lost signal. Two of them might both think they’re the one who got to decide. That’s the actual problem.</p><p>In a distributed system, nodes fail, messages get delayed or lost, and a node that stops responding might be dead, or it might just be slow. You can’t tell the difference from the outside. Consensus algorithms exist to make a group of unreliable, half-blind machines agree on one single value or decision, and to make that agreement survive failures without ever ending up with two conflicting answers both looking valid.</p><p>This shows up constantly. Which node is the leader right now. What’s the next entry in a replicated log. Which value actually got written when two writes raced each other. Every one of these needs the same underlying guarantee, and that guarantee has a name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LF_XdGO7jcHj_LrzQSJrXQ.png" /><figcaption><em>Five nodes propose five different values. A consensus algorithm guarantees they all end up agreeing on exactly one, even if two of them crash mid-vote.</em></figcaption></figure><h3>Paxos, the Original Answer</h3><p>Paxos came first, and it’s famous for being correct and famous for being genuinely hard to reason about. It splits every decision into roles: <strong>proposers</strong> suggest values, <strong>acceptors</strong> vote on them, and <strong>learners</strong> find out what got decided.</p><p>The whole thing runs in two phases.</p><p><strong>Phase 1, Prepare and Promise.</strong> A proposer picks a number higher than any it’s used before and sends a Prepare request to a majority of acceptors. Each acceptor looks at that number and, if it hasn’t already promised something with a higher number, promises not to accept anything lower than it going forward.</p><p><strong>Phase 2, Accept and Accepted.</strong> Once the proposer hears back from a majority, it sends an Accept request carrying the actual value. Acceptors accept it, as long as they haven’t since promised a higher number to someone else. When a majority accepts, the value is chosen, permanently, no matter what proposals come after.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-wiKP0Ra-ri0i03WguTMg.png" /><figcaption><em>A proposer collects promises from a majority of acceptors before its value can be chosen.</em></figcaption></figure><p>Yeh jo majority wala rule hai na, that’s the whole trick. Do quorums (two overlapping majorities) can never both succeed without sharing at least one node, and that shared node is what prevents two different values from both getting chosen. Simple idea, but the actual protocol juggling proposal numbers, promises, and re-proposals across failures is where Paxos earns its reputation for being a pain to implement correctly.</p><h3>Raft, Built to Be Understandable</h3><p>Diego Ongaro and John Ousterhout built Raft specifically because Paxos was too hard to teach and too easy to get subtly wrong in production. Raft solves the exact same problem, but it does it by picking a leader first and letting that leader carry almost all the decision-making weight afterward.</p><p>Every node in a Raft cluster is in one of three states at any moment: <strong>Follower</strong>, <strong>Candidate</strong>, or <strong>Leader</strong>. Followers just listen. If a follower doesn’t hear from a leader within its election timeout, it assumes the leader is gone, bumps the term number, and becomes a candidate.</p><p>The candidate votes for itself and sends RequestVote messages to every other node. Each node votes for at most one candidate per term, first request wins. If the candidate gets votes from a majority, it becomes the leader for that term. If nobody gets a majority (a split vote), the term ends and a new election starts. This is why Raft uses randomized election timeouts, if every follower waited the exact same amount of time, they’d all become candidates simultaneously, forever splitting the vote.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2OXAMesrx-xUnH_ahAFWxQ.png" /><figcaption><em>A follower times out, becomes a candidate, and wins a majority of votes to become leader.</em></figcaption></figure><p>Once a leader exists, life gets simple. Only the leader accepts writes. Everyone else just replicates whatever the leader tells them.</p><h3>Log Replication, How Raft Actually Keeps Everyone in Sync</h3><p>The leader doesn’t just make decisions, it maintains a log, an ordered list of every change the cluster has agreed to. When a client sends a write, the leader appends it to its own log and sends it to every follower via AppendEntries.</p><p>Here’s the part that matters: an entry isn’t considered committed until a majority of nodes have it in their log. Only then does the leader apply it and tell the client it succeeded. If one follower is slow or briefly disconnected, that’s fine, the leader doesn’t wait for everyone, just a majority.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGCHqZ9E_PTKw-o_t30w8A.png" /><figcaption><em>An entry is committed once a majority of nodes have acknowledged it, not all of them.</em></figcaption></figure><p>Is baat ka matlab hai ke agar ek ya do followers slow hain ya thoda peeche reh gaye hain, poora system rukta nahi. Majority commit ho gaya, kaam ho gaya. The laggard catches up whenever it reconnects, replaying the leader’s log until it’s current again. This is also exactly what makes Raft safe during a leader crash, whoever becomes the next leader is guaranteed to already have every committed entry, because it had to win votes from a majority, and any majority necessarily overlaps with the majority that had that entry.</p><h3>What Happens When the Leader Actually Dies</h3><p>No heartbeat arrives, a follower’s timeout fires, it becomes a candidate, term increments, election happens, new leader emerges. Usually this takes a few hundred milliseconds. The one genuinely nasty case is a network partition where the old leader is still alive but cut off from the majority, it keeps thinking it’s leader, but it can no longer get anything committed since it can’t reach a majority. Meanwhile the other side elects a new leader and moves on. When the partition heals, the old leader sees a higher term number, immediately steps down to follower, and no split-brain write ever actually got committed on both sides.</p><h3>Paxos vs Raft, When It Actually Matters to You</h3><p>You will almost never implement either of these from scratch. What matters is recognizing them. etcd, Consul, and Kafka’s KRaft mode all run on Raft, that’s why you’ll see terms like “leader,” “term,” and “quorum” in their docs and dashboards. Systems in the Google Spanner and Chubby lineage lean on Paxos-family protocols instead.</p><p>The practical takeaway is smaller than the theory makes it sound: if a system you’re using needs a majority of nodes to be healthy to make progress, and it can survive a minority of failures without losing data, there’s a consensus algorithm underneath doing exactly the job described above. Knowing that shapes how you plan cluster sizes (odd numbers, so majorities are unambiguous), how you think about network partitions, and why “just add one more replica” sometimes changes your fault tolerance more than people expect.</p><p>Drop your questions in the comments and I’ll get back to you. You can also reach out directly at <a href="/service/mailto:abdulmoiz3140@gmail.com"><strong>abdulmoiz3140@gmail.com</strong></a><strong>.</strong></p><p>If you want to go deeper one-on-one, I’m available on <a href="https://topmate.io/im_abdulmoiz/"><strong>Topmate</strong></a><strong> </strong>where we can break things down at your pace.</p><p>At DevWeekends, we focus on building a strong engineering mindset and solid fundamentals. If you’re learning and want a structured path with accountability, reach out at <a href="/service/mailto:info@devweekends.com"><strong>info@devweekends.com</strong></a> and explore the resources on the <strong>DevWeekends website</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2eba1bd1df8a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>