<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muhammad Mansoor Afzal on Medium]]></title>
        <description><![CDATA[Stories by Muhammad Mansoor Afzal on Medium]]></description>
        <link>https://medium.com/@mansoor.afzal?source=rss-92eb6c52b99a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Muhammad Mansoor Afzal on Medium</title>
            <link>https://medium.com/@mansoor.afzal?source=rss-92eb6c52b99a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:38:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mansoor.afzal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Making Durable Writes Work Without Stalling Consensus]]></title>
            <link>https://medium.com/@mansoor.afzal/making-durable-writes-work-without-stalling-consensus-48554c404698?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/48554c404698</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[consensus]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[paxos]]></category>
            <category><![CDATA[csharp]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 06:06:25 GMT</pubDate>
            <atom:updated>2026-11-18T06:06:25.574Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQo8JbBbqRF8W5ZQ1EhIIA.png" /></figure><h4>Inside a Paxos Library — Part 4: sharing disk flushes while preserving the meaning of every acknowledgement.</h4><p>256 clients are writing to the leader at once. Every “Accept” needs a durable save before anyone may reply “Accepted”, so the obvious implementation flushes the disk once per command, and throughput is capped by flush latency.</p><p>A second obvious implementation makes things worse: the protocol loop waits for each flush before it handles anything else. Heartbeats queue behind the disk, and followers start to suspect a healthy leader because its storage is slow.</p><p>Microlens.Paxos uses group commit with two rules. Saves made while other events are waiting share one flush. And anything, that depends on a save, don’t leave the node until that save’s flush has completed. Staging a record, making it durable, and releasing what depends on it are three separate steps.</p><h3>Stage saves while events are waiting</h3><p>While the event loop has more events waiting in its mailbox, it stages each save, whether a promise, an accepted entry, or a chosen entry, in a batch instead of writing it immediately. When no event is waiting, or the batch reaches 1,024 saves or 16 MB of values, the batch goes to storage as one call that ends in one flush.</p><p>Under load, one flush therefore covers many commands, while a command that arrives alone is saved at once. There is no collection timer, so batching adds no delay to an idle node.</p><p>The diagram shows the two batches involved. Batch A is with the storage, flushing on the thread pool. Batch B collects new saves while A flushes. Only one batch is ever with the storage, so storage calls never overlap, and if B fills up before A completes, B waits for A.</p><p>Grouped saving is an optional storage capability. Both built-in storages support it, a custom storage can opt in, and an option turns it off. Without it, the node makes each save individually under the same durability rule, and a leader’s throughput is then bounded by the flush latency of its disk.</p><p>In the Write-Ahead Log, one batch is one physical unit: a batch record, every record of the batch, and a single flush. A crash during that write discards the whole batch on restart (<a href="https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b"><em>Part 3</em></a>). The unit whose completion releases replies at runtime is the same unit recovery keeps or drops.</p><figure><img alt="Two alternating batches let the node stage new saves during a disk flush, with acknowledgement arrows gated by completed persistence." src="https://cdn-images-1.medium.com/max/1024/1*YtU1C6gI74MkIuKZXwxv-g.png" /><figcaption>Sharing a flush changes when storage work is paid, while dependent acknowledgements still wait for durability.</figcaption></figure><h3>Hold replies behind the batch they depend on</h3><p>An “Accepted” reply tells the leader that this node has made a durable commitment. If the reply left while the accepted entry was only staged, a crash could erase a vote the leader had already counted.</p><p>So while a “Promise” or an “Accepted” entry is staged or flushing, every frame the node produces waits behind the batch that holds it. When that batch completes, the frames go out in the order they were produced.</p><p>The leader treats its own vote the same way. Its “Accept” to the followers goes out immediately, because it depends only on the leader’s own “Promise”, which has been durable since it won <strong>Phase 1</strong>. Its own acceptance of the slot is staged, and counts toward the majority only once its batch has completed. That is the step labelled “count durable local votes” in the diagram, and it is how the leader’s flush overlaps its followers’ flushes.</p><p>If a batch fails, nothing it held is released. The node faults, because the commitments it meant to make never reached disk. Pending callers receive an unknown outcome, and the node’s status records the fault (Part 15).</p><h3>Keep the chosen record off the decision path</h3><p>Accepting and choosing are not symmetric. An “Accepted” record is evidence that establishes agreement. A “Chosen” record is local knowledge of a decision that a majority has already made durable.</p><p>So once a majority’s durable “Accepted” replies are counted, the leader does not wait for another flush. It reports the command as “Chosen”, broadcasts “Chosen”, and makes the entry available to local readers. Its own chosen record is only staged. It rides along with the next batch that something waits on, usually the next command’s, or is flushed at the next heartbeat tick at the latest.</p><p>The consequence is deliberate. A node’s reported chosen position and its stream of chosen commands can run ahead of its disk by up to one heartbeat interval plus a flush. A node that crashes in that window restarts slightly behind what it reported, then learns the missing slots again from a peer’s catch-up or from the next leader’s recovery. The decision itself is safe, because the majority’s “Accepted” records are durable. The application resumes from its own recorded next slot, not from the node’s lost marker.</p><p>A node that stops cleanly saves its staged “Chosen” records first, so it restarts with every slot it learned.</p><h3>Keep the loop responsive</h3><p>While batch A flushes, the event loop keeps handling heartbeats, replies, new proposals, and reads, and stages whatever they produce into B.</p><p>Some events have to read from storage: a page of catch-up entries for a peer, a page of history for a slow reader, or a snapshot. Before such a read, the loop completes A and flushes B, so the read never sees an older state than the one the loop is reasoning from.</p><p>The end of a flush arrives in the mailbox as a signal, coalesced if it fires more than once. The storage callback only reports that the flush ended. The event loop decides what that releases, so every state transition still happens in one place (Part 14).</p><p>Stopping respects the batch with the storage. A healthy stop waits for batch A, saves the staged batch B, and only then closes the storage. A run that has faulted also waits for A to finish, but saves nothing more. Its staged saves were never acknowledged, and the next run forgets them.</p><p>Disk latency still slows commands down. What group commit removes is the case where a slow flush freezes unrelated coordination such as heartbeats.</p><h3>Read throughput together with latency</h3><p>The benchmark run of 05 November 2026 used three nodes in separate processes on a two-logical-core Linux host, durable storage, and 256-byte writes over 1,000 keys. Clients sent every request to the leader, and each concurrency level was measured for ten seconds after three seconds of warm-up.</p><p>Throughput grew with the number of commands in flight: 1,273 writes per second with 1 client; 10,704 with 16 clients; 14,875 with 64; and 18,043 with 256. That is the shape flush sharing should produce. Latency grew as well. At 256 clients the median write took 13.7 milliseconds and the 99th percentile 25.4 milliseconds, so high throughput under heavy concurrency does not mean that every command waits briefly.</p><p>The run does not isolate group commit, since nothing was measured with it switched off, and it predates later storage changes. It describes that build and that workload.</p><p>The architectural point is narrower than any throughput number. Many commands can share durable work while every acknowledgement stays tied to a completed flush, and the node keeps coordinating the cluster while the disk catches up.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/paxos.html"><strong>Paxos</strong></a> project page.</p><p>⭠<em> </em><a href="https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b"><em>Previous</em></a><em> · Part 4 of 17</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48554c404698" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why a Slow Load Must Never Overwrite a Newer Write]]></title>
            <link>https://medium.com/@mansoor.afzal/why-a-slow-load-must-never-overwrite-a-newer-write-c4cdbcee6b27?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4cdbcee6b27</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[cache]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 12:38:22 GMT</pubDate>
            <atom:updated>2026-11-09T12:38:22.438Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXI0whI6sMksO1Xvogmlzg.png" /></figure><h4><em>Inside a Cache Library — Part 2: ordering explicit writes against loads already in flight.</em></h4><p>Some cache bugs never show up in a test of the happy path. This is one of them.</p><p>A product’s cached entry has expired. A request misses and starts loading sku-42 from the database, which reports a price of 10. While that query is still running, an administrator changes the price to 12, and the service that saved the change calls cache.Set(“products”, “sku-42”, updated).</p><p>Then the slow load comes back with the price it read. If the cache stores whatever a load returns, 10 lands on top of 12, and customers see the old price until the entry expires again.</p><p>No step in that story failed, and every one of them was thread-safe. What went wrong is order: a value read before the write was stored after it.</p><p>In <a href="https://medium.com/@mansoor.afzal/why-1-000-cache-misses-should-trigger-one-load-119092c43c2c"><em>Part 1</em></a>, one caller owned each load and everyone else waited for it. That raises the obvious follow-up:</p><blockquote>What should happen to a load when the data it is fetching changes before it finishes?</blockquote><figure><img alt="A load reads the old price, a Set stores the new one and supersedes the load, and the load’s late result is never stored." src="https://cdn-images-1.medium.com/max/1024/1*lvoT0OoD9qaR55P78B2Fkg.png" /><figcaption><em>A write made after a load started outranks the value that load carries.</em></figcaption></figure><h3><strong>A lock does not fix this race</strong></h3><p>It is tempting to reach for a lock. But a cache-aside flow has two steps, reading the source and storing the result, and a lock around each step only protects the dictionary. Nothing ties the value being stored to the moment it was read. One lock held across the whole load would make the write wait its turn, and it would also queue every caller of that key behind a database call.</p><p>The cache cannot know that the database changed. What it can know is that a Set, Remove or Clear reached the key after the load began, and that turns out to be enough. I wanted Microlens.Cache to act on exactly that signal, without ever making the write wait for the load.</p><h3><strong>Give every load an identity</strong></h3><p>Each key has at most one load in flight. It lives as an entry in the container’s map of pending loads, and every overlapping caller joins that same entry. When the entry is created, it takes a version from a single counter shared by the whole process, so anything created later has a higher version. Because the order comes from a counter rather than a clock, two entries created in the same instant are still ordered, and no clock adjustment can reorder them.</p><p>Completing a load is a compare-and-remove. Under the lock of the key’s stripe, one of 128 per container, the container removes the entry from the pending map only if that exact entry is still there. If it is, the value is stored. If something replaced or removed it in the meantime, the value is not stored, and the load is superseded. A key’s hash picks its stripe, so unrelated keys rarely compete for a lock, and no lock is ever held while the source is being called.</p><p>The three writes apply the same rule from the other side:</p><ul><li><strong>Set</strong> removes the pending load and stores the new value in one step under the stripe, then cancels the superseded load’s token outside the lock.</li><li><strong>Remove</strong> drops the pending load and the stored value together.</li><li><strong>Clear</strong> takes every stripe of the container, swaps in an empty store and an empty pending map, and cancels every load that was in flight.</li></ul><p>None of them waits. A write takes effect the moment it is made; the load only finds out when it tries to finish.</p><h3><strong>Decide what the waiting callers receive</strong></h3><p>Refusing to store the old value protects the cache, but it leaves a question open. While the slow load was running, other requests missed the same key and joined it. They are still waiting, and they need an answer. What they get depends on one detail of the factory: whether it takes a cancellation token.</p><pre>var product = await cache.GetOrAddAsync(<br>  &quot;products&quot;, id, (repository, id)<br>  , static (state, token) =&gt; state.repository.FindAsync(state.id, token)<br>);</pre><p>A factory that takes a token receives the load’s own token, and a superseding write cancels it. When the factory honors that cancellation, the cache does not pass it on. It forwards the callers to whatever replaced the load instead: after Set or SetCollection, they receive the written value; after Remove, RemoveCollection or Clear, they join one fresh load of the key, shared like any other. The forwarding asks the container for anything newer than the superseded entry, so the write qualifies and the stale result never can.</p><p>A factory without a token, a plain Func&lt;Task&lt;TValue&gt;&gt;, is never cancelled. It runs to the end, and its callers receive the value it read. That is a fair answer, since they asked before the write happened and this is what their read found. The cache, which outlives all of them, still keeps the newer state.</p><p>A superseded load is not treated as a failure. It does not count toward failures; the cache publishes a Superseded event for it, and its trace ends with the outcome Superseded (Part 17). The exception is a factory without a token that throws OperationCanceledException on its own. That is its own failure, and its callers see it as one (Part 3).</p><h3><strong>Measure the race, not the happy path</strong></h3><p>Benchmarks of hits and misses never run into this ordering, so the race has to be staged on purpose.</p><p>In the benchmark run of 3 November 2026, a load read old from the source and was held there. The data then changed to new, and before the load could finish, the key was either set to new or removed, 100 times for each operation. When the load completed, the key was read again, and the result the racing load’s caller had received was checked too.</p><p>The read afterwards returned new in all 100 rounds after a Set and in all 100 after a Remove, and the racing caller received new as well. Neither write ever had to wait for the load. The benchmark’s factory takes the load’s token, so those callers were the forwarded ones. With a factory that ignores tokens, that caller would have received old, while the cache still held new.</p><p>Each branch has a test of its own, too: a set supersedes a load whose result reaches its callers but is never stored; a removal serves the load’s callers a fresh load; a superseded load joins a fresh load already in flight rather than starting another; and a clear supersedes every load in its container.</p><h3><strong>Know where the guarantee ends</strong></h3><p>The rule orders what the cache is told about. A database change that nobody reports to the cache is invisible to it, and only expiration limits how long such a value can survive (Part 5).</p><p>It is also scoped to a key. A write supersedes the load of its own key, and Clear the loads of its container; everything else carries on.</p><p>And it holds on every node that hears about the write. A change that arrives from another instance takes the same path as a local Remove or Clear, so it supersedes a load in flight there as well, and a value read before the write ends up stored on no node at all (Part 8). Refresh relies on the same versions in the other direction: it waits for a value newer than the one it saw, so it is never answered with the value it asked to replace (Part 4).</p><p>What I like about this design is how little it asks of the write. Once a load has an identity, the cache can tell an old answer from a new one at the one moment it matters, and nobody has to wait for anybody.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/cache.html"><strong>Cache</strong></a> project page.</p><p>⭠<em> </em><a href="https://medium.com/@mansoor.afzal/why-1-000-cache-misses-should-trigger-one-load-119092c43c2c"><em>Previous</em></a><em> · Part 2 of 18</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4cdbcee6b27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Adding One Server Should Not Move Every Key]]></title>
            <link>https://medium.com/@mansoor.afzal/why-adding-one-server-should-not-move-every-key-c64b2407c646?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c64b2407c646</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[consistent-hashing]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 12:15:42 GMT</pubDate>
            <atom:updated>2026-11-09T06:05:03.347Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZvFJrHpRSKWEqzOloLlrQ.png" /></figure><h4><em>Inside a Hash Library — Part 1: placing keys so servers can come and go without reshuffling everything.</em></h4><p>Picture 10 cache servers in front of a busy service. The simplest way to decide where a key lives is hash(key) % 10, and for a while it works well. Then an 11th server is added, and the formula quietly becomes % 11.</p><p>I measured what that costs. In the benchmark run of 3 November 2026, 90.9% of 1,000,000 keys belonged to a different server the moment the 11th one joined. Almost every lookup misses its cache at once, and the source behind the caches takes the load they had been absorbing.</p><p>The frustrating part is that modulo spreads keys very evenly: on 100 servers, the busiest held only 2.0% more than the mean. Its weakness is change.</p><p>Microlens.Hash is my answer to that for .NET 8 and .NET 10: two libraries that keep the cost of a change as close as possible to what the change itself requires.</p><figure><img alt="A modulo placement moves 90.9% of a million keys when an eleventh server joins; a ring moves 10.0%, all of them to the new server." src="https://cdn-images-1.medium.com/max/1024/1*mM5VJ6mhTMbf4EOjxrMgnQ.png" /><figcaption>Adding a server should cost one server’s share of the keys, not the whole key-space.</figcaption></figure><h3>What a change should cost</h3><p>The 11th server should take its share, roughly one key in 11 or 9.1%, and every key that moves should move to it. No other server changed, so no other key has a reason to go anywhere.</p><p>Consistent hashing comes close to that ideal. Servers and keys sit on one ring of hash positions, and a key belongs to the first server clockwise from its own position. A server that joins claims only the keys just before its points, and a server that leaves hands its keys to whoever comes next on the ring (Part 4).</p><p>In the same run, Microlens.Hash.Consistent moved 10.0% of the keys, every one of them to the new server. Staying that close to the minimum is the first property this series is about, <strong>Minimality</strong>.</p><h3>Agree before you spread</h3><p>Moving few keys is worth nothing if two processes disagree about where a key lives. In .NET, string.GetHashCode is randomized per process, so two instances of one service would send the same key to different servers.</p><p>So the library hashes bytes instead, a string’s UTF-8 bytes or a server identity’s own bytes, with a seeded algorithm, XxHash3 unless you choose another. It also builds every ring in the order of the servers’ identity bytes, so two processes that add the same servers in a different order end up with the same ring (Part 2).</p><h3>Even is not the same as bounded</h3><p>One point per server leaves some servers with far more keys than others, so each server sits at many points, its virtual nodes, 160 by default. With 160, the busiest of 100 servers held 16.3% more keys than the mean, and a larger server simply gets more points and a larger share (Part 3). That is the second property, <strong>Distribution</strong>.</p><p>More points narrow the gap but never cap it. With 4,096 partitions on 100 servers and no bound, the busiest ended up 51.4% above the mean, the server that runs out of memory first.</p><p>Microlens.Hash.BoundedLoad keeps the ring’s placement and adds one rule: skip a server that is already full. With a load factor of 1.25, no server may hold more than 1.25 times the mean, rounded up, and the busiest came in at 27.0% above it. The cap barely touched <strong>Minimality</strong>: on a ring of 10, an 11th server still moved 10.1% of the partitions (Part 6). Keeping every server under its cap is the third property, <strong>Balance</strong>.</p><h3>Two libraries, three properties</h3><p>Microlens.Hash.Consistent places servers on rings and answers lookups without taking a lock (Part 5). Microlens.Hash.BoundedLoad builds partition maps on those rings, with a cap on every server and each partition’s replicas on distinct servers.</p><p><strong>Distribution</strong>, <strong>Minimality</strong> and <strong>Balance</strong> rarely come for free together; tighten one and another gives a little. The rest of the series takes them one at a time, through the code, along with what each one costs the other two.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/hash.html"><strong>Hash</strong></a> project page.</p><p><em>Part 1 of 8</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c64b2407c646" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a Paxos Node Must Remember After a Crash]]></title>
            <link>https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/533f1c16916b</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[paxos]]></category>
            <category><![CDATA[consensus]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 19:06:41 GMT</pubDate>
            <atom:updated>2026-11-18T06:14:18.337Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQo8JbBbqRF8W5ZQ1EhIIA.png" /></figure><h4>Inside a Paxos Library — Part 3: preserving promises, accepted commands, and recoverable history across process failure.</h4><p>Node B accepts slot 41, replies Accepted, and crashes. The leader, A, has already counted B’s reply, so slot 41 is chosen by A and B.</p><p>Now suppose B restarts with no memory of slot 41, A fails too, and C never received that Accept. A new leader elected by B and C asks both what they accepted at slot 41. Neither reports anything, so it is free to put a different command there. Two values have been chosen for one slot, which is the one outcome Paxos exists to prevent.</p><p>That is why storage is part of the agreement protocol rather than an implementation detail. When a node says it has saved something, the save must still be there after a crash. Microlens.Paxos ships two storages behind one contract, an In-Memory store for tests and a Write-Ahead Log for production, and a custom storage can replace either if it keeps the same contract.</p><h3>Restore three kinds of state</h3><p><strong>The highest promise.</strong> An “Acceptor” keeps a single promised ballot covering every slot. After a restart it must keep rejecting “Prepare” and “Accept” messages below that ballot. Losing the “Promise” would let an obsolete leader win B’s vote back.</p><p><strong>Accepted entries.</strong> For each slot, the node keeps the latest value it accepted, with its ballot and entry kind. These are the answers a new leader gathers in <strong>Phase 1</strong>, and the evidence it uses to preserve commands that may already be chosen. When an “Accept” carries a ballot above the current promise, the node saves the new promise before the entry. A node that crashed in between would otherwise restart with a “Promise” lower than an entry it holds, and could later overwrite that entry with a lower-ballot value.</p><p><strong>Chosen history.</strong> The node also records the slots it has learned are chosen. They let it rebuild its chosen prefix, deliver commands to the application, and answer catch-up requests from peers. An “Accepted” entry is one member’s vote. A “Chosen” entry is a known decision.</p><p>Every reply that depends on one of these saves waits until the save is durable. Replying first would turn a durable vote into a promise to write later (<a href="https://medium.com/@mansoor.afzal/making-durable-writes-work-without-stalling-consensus-48554c404698"><em>Part 4</em></a>).</p><h3>Keep one contract across storages</h3><p>The In-Memory storage keeps its promise and its entries in memory, with entries sorted by slot. It suits tests, simulations, and applications that deliberately run without crash durability. Stopping and starting a node on the same storage object keeps its data, much like a restart with durable storage. A new process starts empty, which is the same as a node that lost its disk. In Paxos terms that node has forgotten its votes, so it must come back under a new identity (Part 11).</p><p>The Write-Ahead Log appends a record for every change, flushes it to disk before the save completes, and replays the file on start.</p><p>Both storages meet the same contract. Entries come back in ascending slot order with one entry per slot, and a newer accepted value replaces an older one. Stored values are copies, so an application that reuses the buffer holding a command cannot change consensus history. The node never calls a storage concurrently, and an open enumeration counts as a call, so a storage can rely on a stable view while the node reads from it.</p><p>A custom storage has to provide the same guarantees, not just the same methods. One that acknowledges a save before it is durable can let two values be chosen after a crash.</p><h3>Make the log verifiable</h3><p>The log file begins with a header naming its format and version. Each record carries its type (“Promised”, “Accepted”, “Chosen”, or “Batch”), the slot, the ballot, the entry kind, the value length, a CRC-32 of the value, and a CRC-32 of the header itself. Replay checks both checksums before it treats a record as protocol state.</p><p>While the node runs, an In-Memory index maps every “Accepted” and “Chosen” slot to its position in the file, so a load seeks straight to the value. The index is rebuilt from the file on every start. The file is the authority; the index is only a way to reach it.</p><p>A slot can collect several accepted records as leadership changes. Replay keeps the latest, with its ballot. The file is append-only, but the logical view is one current entry per slot.</p><p>A save includes the flush to disk. The log also flushes the directory of every file it creates or replaces, and flushes its own directory on every start, within what the operating system and file system actually guarantee.</p><p>One node owns a storage directory at a time. On start the log first takes an exclusive lock file in the directory, so a second node pointed at the same directory fails to start before it touches anything. Only after taking the lock does the log delete temporary files left by an interrupted compaction. It can never delete files that another live node is still writing.</p><h3>Discard a torn tail, refuse corruption</h3><p>A process can die between any two writes. The last write may have its first records on disk and the rest missing.</p><p>Every write, whether a single save or a group-commit batch, is framed by a batch record that states its length and how many records follow. On start, a final batch that is incomplete was never acknowledged, because its flush never finished. It is discarded as a whole, even if some of its records reached the disk intact. The diagram shows this case: two durable batches are kept and the torn third one is dropped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*amA0qHzH4jjyQRGmLXExpQ.png" /><figcaption><em>Recovery restores durable commitments and rejects history it cannot validate.</em></figcaption></figure><p>Damage before the final write is different. If a record fails its checksum and completed writes follow it, acknowledged history is damaged. Start fails instead of skipping the record, because skipping it could silently forget a “Promise” or an “Accepted” value that other nodes have counted.</p><p>A failed write or flush at runtime is cut off. The log truncates the file back to its last durable length before the save reports failure, so a later restart never trusts a record that may be readable without being durable. If the truncation itself fails, the storage refuses to reopen for the rest of that process, and the host has to restart.</p><p>The empty case gets the same care. A node that has never promised anything starts with a promise of zero. A damaged promise inside completed history stops the start; it never turns back into no promise at all.</p><h3>Restore before taking part</h3><p>On start the node opens its storage, restores its promise, its latest snapshot if it has one, and its chosen history, and only then starts the transport and joins the cluster as a follower. Its very first answer to a “Prepare” already reflects the commitments that survived the crash.</p><p>Local recovery does not make the node current. Other members may have chosen more commands while it was down, so it still catches up from its peers (Part 10), and the application resumes from its own recorded progress (Part 8).</p><p>Once the log has been compacted, the latest snapshot plus the log after it is what the node restores from. That snapshot holds the application’s state through its slot and the cluster configuration that governs the next slot, and the log it covers is removed only after the snapshot has been saved durably (Part 9).</p><p>Durable storage earns its place in consensus by keeping earlier acknowledgements true. Every checksum, flush, lock, and recovery rule serves that purpose.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/paxos.html"><strong>Paxos</strong></a> project page.</p><p>⭠<em> </em><a href="https://medium.com/@mansoor.afzal/how-nodes-agree-on-one-ordered-command-history-b28c87c23434"><em>Previous</em></a><em> · Part 3 of 17 · </em><a href="https://medium.com/@mansoor.afzal/making-durable-writes-work-without-stalling-consensus-48554c404698"><em>Next</em></a><em> </em>➝</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=533f1c16916b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Nodes Agree on One Ordered Command History]]></title>
            <link>https://medium.com/@mansoor.afzal/how-nodes-agree-on-one-ordered-command-history-b28c87c23434?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b28c87c23434</guid>
            <category><![CDATA[paxos]]></category>
            <category><![CDATA[consensus]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 18:15:36 GMT</pubDate>
            <atom:updated>2026-11-09T05:58:44.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQo8JbBbqRF8W5ZQ1EhIIA.png" /></figure><h4>Inside a Paxos Library — Part 2: following a command from proposal to majority agreement and ordered delivery.</h4><p>Three nodes run the same application, and node A is the leader. A client asks A to change an account limit, and A gives the command slot 41 in the replicated log. It sends the command to B and C. B replies quickly. C’s reply is delayed.</p><blockquote>Which reply is enough to tell the client that the command has a permanent place in history?</blockquote><p>In Microlens.Paxos the answer is exact: slot 41 is chosen once a majority of distinct members has durably accepted the same value for that slot under the same ballot. With three members, A and B are enough. C’s reply adds nothing to the decision, and C can learn it later.</p><p>Being chosen is still not the end. The command has to reach the application in slot order and be executed there. This article follows a command through those steps: proposed, accepted, chosen, and delivered.</p><h3>Give each command a slot</h3><p>Multi-Paxos runs one instance of Paxos for each position, or slot, in a replicated log. The leader numbers slots in order. Two commands submitted at the same moment get different slots, and every node applies them in slot order, whatever order their messages arrive in.</p><p>The command itself is opaque bytes. It might change a limit, reserve stock, or update a document; the library orders it, and the application decides what it means. A command can therefore be chosen and later rejected by a business rule when the application applies it. Agreement fixes its position. The application decides its outcome.</p><p>Each node keeps its own protocol state: the ballot it has promised, the values it has accepted, and the slots it knows are chosen. The leader coordinates, but a decision exists only in what a majority of members has recorded durably.</p><h3>Run Phase 1 once, then Phase 2 for every command</h3><p>A ballot is a round number paired with a node id, ordered by round and then by node id. Two nodes never hold the same ballot, so concurrent leadership attempts always have a defined order.</p><p>To lead, a node runs <strong>Phase 1</strong>. It sends “Prepare” with a new ballot, and each member answers with a “Promise” that it will reject anything carrying a lower ballot. The “Promise” also reports the values that member has already accepted, which the new leader must preserve (Part 6).</p><p>In stable-leader Multi-Paxos, <strong>Phase 1</strong> runs once per leadership rather than once per command. After it, each command costs only <strong>Phase 2</strong>. The leader sends “Accept” with its ballot, the slot, and the value, and every member that has promised nothing higher answers “Accepted”. Each new slot still needs its own majority, but no new election.</p><p>In the benchmark run of 26 October 2026, with three nodes in separate processes and durable storage, a single client writing <strong>256-byte commands</strong> saw a <strong>median of 0.70 milliseconds per write</strong>. That figure includes the client’s own network hop and every durable save on the path.</p><p>An old leader can still be running after it has lost authority. Its “Accept” messages carry an old ballot, and members that have promised a higher one answer with a “Nack” that names their promise. The old leader steps down when it sees it (Part 5). Authority follows the promises a majority has made, not whichever process last called itself leader.</p><h3>Count durable Accepted replies from distinct members</h3><p>With three members, a majority is two. If A and B both accept the same value at slot 41 under A’s ballot, slot 41 is chosen. That is the situation in the top half of the diagram.</p><p>The leader counts members, not messages. A duplicate “Accepted” from B counts once. A reply for another ballot, or for a slot that is no longer in flight, is ignored. Three open connections are not three votes.</p><p>With the write-ahead log, a member sends “Accepted” only after the accepted record has been flushed to disk (<a href="https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b"><em>Part 3</em></a>). Without that rule, a member could reply, crash, forget the value, and later help a new leader choose something else at the same slot.</p><p>The leader follows the same rule for its own vote. It sends the “Accept” to B and C first and then saves its own acceptance, so its disk write overlaps theirs. Its vote counts only once its own save has completed.</p><p>C can stay silent while A and B make progress. If A could reach only itself, nothing new would be chosen. Until a slot is chosen, the leader resends its “Accept” at every heartbeat interval to each member that has not yet accepted it and has been heard from recently, so a lost “Accept” costs one interval rather than the command. A client timeout changes what that client knows; the majority still decides.</p><p>Majorities matter because any two majorities of the same configuration share at least one member. A later leader’s <strong>Phase 1</strong> majority must include A or B, and that member’s “Promise” reports the value at slot 41, so the new leader is bound to propose that value again. Promises and durable acceptances are what make the overlap useful: a member that had forgotten what it accepted would sit in the intersection and carry nothing forward.</p><p>A value can also be accepted without reaching a majority. A single member cannot tell how many others accepted it before the leader went quiet, so the next leader treats every reported value as possibly chosen and keeps the one with the highest ballot.</p><h3>Choose in any order, deliver in slot order</h3><p>The lower half of the diagram shows a second case. Slots 51 and 52 are proposed close together. Replies for 52 arrive first, a majority accepts it, and 52 is chosen while 51 is still pending.</p><p>The node records 52 as chosen but holds it back from the application. Delivery advances only through the contiguous chosen prefix, so the application waits until 51 is chosen and then receives 51 and 52 in order. Network completion order never becomes execution order.</p><figure><img alt="Three nodes agree on a numbered command, with an ordered delivery gate preventing later chosen work from reaching the application before an earlier gap is resolved." src="https://cdn-images-1.medium.com/max/1024/1*wop-B9ZHXkGBpqrQ-LZNlQ.png" /><figcaption><em>A majority chooses a command; the complete chosen prefix determines when the application can receive it.</em></figcaption></figure><p>Some slots hold internal entries: no-ops written during recovery (Part 6), and configuration entries that change the membership (Part 11). They keep the protocol history continuous, but they are not business commands, so the application never receives them and the slot numbers it sees can skip. A gap in those numbers does not mean a command was lost.</p><p>Every entry also carries a kind: command, no-op, or configuration. An empty command is still a command; an empty payload never turns into a no-op.</p><p>Once a slot is chosen, the leader broadcasts a “Chosen” message so the followers learn the decision. A follower that misses it notices from a later heartbeat that a peer is further ahead and fetches the missing slots through catch-up (Part 10). A lost notification delays learning; it cannot undo a decision.</p><h3>Leave execution to the application</h3><p>The application reads chosen commands in slot order, applies each one deterministically, and records the next slot it needs together with its own state (Part 8). The library does not update application tables, call external services, or complete business transactions.</p><p>The leader reports a command as chosen, with its slot, as soon as its own majority has chosen it. At that moment it is possible that no application has applied it yet. The opposite also happens: a lost reply can leave the client uncertain even though the command is already chosen (Part 7). An application that needs an execution result has to connect the slot to its own processing, and external side effects need their own protection, because ordering alone does not make an action happen exactly once.</p><p>The core path is short: establish leadership once with <strong>Phase 1</strong>, give each command a slot, collect durable “Accepted” replies from a majority, and deliver the contiguous chosen prefix. The rest of the series is about keeping each of those steps true through crashes, delays, and leader changes.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/paxos.html"><strong>Paxos</strong></a> project page.</p><p>⭠<em> </em><a href="https://medium.com/@mansoor.afzal/why-i-built-a-multi-paxos-library-for-net-3e79710b3d1a"><em>Previous</em></a><em> · Part 2 of 17 · </em><a href="https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b"><em>Next</em></a><em> </em>➝</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b28c87c23434" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I Built a Multi-Paxos Library for .NET]]></title>
            <link>https://medium.com/@mansoor.afzal/why-i-built-a-multi-paxos-library-for-net-3e79710b3d1a?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e79710b3d1a</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[paxos]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[consensus]]></category>
            <category><![CDATA[csharp]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 13:27:49 GMT</pubDate>
            <atom:updated>2026-11-09T05:54:56.969Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RQo8JbBbqRF8W5ZQ1EhIIA.png" /></figure><h4>Inside a Paxos Library — Part 1: turning agreement, application state, and failure into explicit architectural boundaries.</h4><p>Two application nodes receive the same commands. Both execute them successfully. They still finish with different state.</p><p>Imagine two commands: set a limit to 10, then increase it by 5. Applying them in that order produces 15. Reversing them produces 10.</p><p>Distributing commands is only part of the problem. The nodes also need an agreed history, including when a process crashes or the network splits.</p><p>I built Microlens.Paxos to explore that problem as an embedded library for .NET 8 and .NET 10, carrying the protocol’s rules through to storage, recovery, and application integration.</p><h3>Start with a division of responsibility</h3><p>My starting point was a replicated log: commands occupy numbered positions, and each application receives chosen commands in log order.</p><p>The application defines what those commands mean. It also owns the state produced by applying them.</p><p>That gives the architecture two responsibilities:</p><ul><li><strong>The consensus library</strong> establishes the ordered history.</li><li><strong>The application</strong> applies that history deterministically and tracks its progress.</li></ul><p>For example, the library can agree that “set the limit to 10” precedes “increase it by 5.” The application implements those operations and decides whether either is valid for its business rules.</p><p>That boundary makes the library reusable across different application state machines.</p><figure><img alt="A consensus library runs inside each of three application processes. A leader coordinates replicated log entries, and each application applies chosen commands to its own state." src="https://cdn-images-1.medium.com/max/1024/1*azbDKsjN1l0QWMjzg34laA.png" /><figcaption>Each node has its own log and application state. The diagram separates agreement on commands from application-owned execution.</figcaption></figure><h3>Why I chose Multi-Paxos</h3><p>With stable-leader Multi-Paxos, a leader coordinates the ordering of commands across a sequence of log positions.</p><p>A command becomes chosen through majority agreement. With three members, the majority is two. The leader can keep ordering commands while it can reach a majority; a minority cannot independently keep choosing a competing history.</p><p>That makes the availability boundary explicit. A cluster split into isolated groups must preserve agreement, even when some nodes cannot make progress.</p><p>I wanted the implementation to make those decisions visible: which node leads, which commands are chosen, where application progress stands, and what recovery must establish before ordinary work resumes.</p><h3>Make failure shape the architecture</h3><p>The public-facing operation is a proposal. Behind it are responsibilities that remain important after the happy path ends.</p><p><strong>Durable protocol state:</strong> promises and accepted values must survive a restart when durable storage is configured. The built-in write-ahead log provides that persistence boundary (<a href="https://medium.com/@mansoor.afzal/what-a-paxos-node-must-remember-after-a-crash-533f1c16916b"><em>Part 3</em></a>).</p><p><strong>Leadership recovery:</strong> a replacement leader must account for prior accepted work before serving new proposals (Part 6). Choosing a new leader is only one part of recovering safely.</p><p><strong>Application recovery:</strong> a node can need missing log entries or an application snapshot before it catches up (Part 10). The application remains responsible for restoring its state and tracking what it has applied (Part 8).</p><p>A chosen command and a completed application action are therefore different milestones. That distinction also matters when a response is lost and the caller cannot tell whether its proposal succeeded (Part 7).</p><h3>Keep infrastructure choices behind contracts</h3><p>Transport and storage are separate architectural boundaries. The library includes TCP, HTTP and In-Memory transport (Part 12, Part 13). It also includes a durable Write-Ahead Log and In-Memory storage.</p><p>Those choices let me exercise the protocol under different conditions while keeping one agreement model. Custom implementations must preserve the contracts that consensus relies on, particularly the meaning of a durable acknowledgement (Part 15).</p><p>This project combines software architecture with engineering: defining responsibilities, implementing the protocol, and examining its behavior under failure.</p><p>The central design decision is the boundary between an agreed command history and the application state built from it. The next article follows a command through majority agreement and ordered delivery.</p><h3><strong>Explore the project</strong></h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/paxos.html"><strong>Paxos</strong></a> project page.</p><p><em>Part 1 of 17 · </em><a href="https://medium.com/@mansoor.afzal/how-nodes-agree-on-one-ordered-command-history-b28c87c23434"><em>Next</em></a><em> </em>➝</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e79710b3d1a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why 1,000 Cache Misses Should Trigger One Load]]></title>
            <link>https://medium.com/@mansoor.afzal/why-1-000-cache-misses-should-trigger-one-load-119092c43c2c?source=rss-92eb6c52b99a------2</link>
            <guid isPermaLink="false">https://medium.com/p/119092c43c2c</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[csharp]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cache]]></category>
            <dc:creator><![CDATA[Muhammad Mansoor Afzal]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 09:27:27 GMT</pubDate>
            <atom:updated>2026-11-09T12:40:36.803Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iXI0whI6sMksO1Xvogmlzg.png" /></figure><h4>Inside a Cache Library — Part 1: coordinating concurrent requests before they become duplicate work.</h4><p>A product becomes popular. Its cached entry expires, and a burst of requests arrives before the replacement value has loaded.</p><p>Every request sees the same thing: a cache miss.</p><p>If each request then queries the database independently, the cache has amplified the work precisely when traffic is highest. The lookup can be thread-safe while the overall operation still creates a stampede.</p><p>This was one of the problems I wanted to address in Microlens.Cache, an in-process caching library I built for .NET 8 and .NET 10.</p><h3>The gap between a miss and a value</h3><p>A cache-aside flow usually reads like this: check the cache, load missing data, then store the result.</p><p>The difficult interval is between the check and the store. An asynchronous database call leaves enough time for other callers to arrive, observe the same missing value, and start identical work.</p><p>Suppose 1,000 callers request the same product. If each load needs a database connection, duplicate loads compete for a finite pool. Requests can spend more time waiting for connections than fetching data. The question becomes:</p><blockquote>Who owns the load while the value does not exist yet?</blockquote><h3>Represent the work in progress</h3><p>I made an in-flight load part of the cache’s coordination model.</p><p>For the same container and key, one caller starts the factory. Overlapping callers join that load and await its result. Once the value is stored, subsequent callers use the cached value.</p><p>This is often called single-flight loading or request coalescing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVVqG57l3YJf-236DVt0PQ.png" /><figcaption><em>One local cache instance coordinates one overlapping load wave. Different keys can load independently.</em></figcaption></figure><p>The factory is the application-supplied function that retrieves the data. The cache coordinates that function; the application still decides where the data comes from.</p><p>For example, with cache and repository supplied by the application:</p><pre>var product = await cache.GetOrAddAsync(&quot;products&quot;, &quot;sku-42&quot;, () =&gt; repository.FindAsync(&quot;sku-42&quot;));</pre><p>The important behavior sits behind that call. During an overlapping miss wave for products/sku-42, callers share the active load rather than invoking the factory separately.</p><h3>Keep the coordination boundary small</h3><p>A single lock held across every database call would serialize unrelated work. I wanted coordination to follow the identity of the cached data: the container and key.</p><p>That lets requests for different products load independently, while requests for the same missing product share work. The goal is to remove duplicate loading without turning the whole cache into a queue.</p><p>Sharing work also creates responsibilities. A failed factory must release the in-flight state so a later request can retry. Cancellation and explicit writes need defined semantics too (<a href="https://medium.com/@mansoor.afzal/why-a-slow-load-must-never-overwrite-a-newer-write-c4cdbcee6b27"><em>Part 2</em></a>, Part 3). Those deserve their own articles; a shared task alone does not settle them.</p><h3>Measure the load, not just the lookup</h3><p>In the run of 19 October 2026, 1,000 callers requested one cold key across 50 rounds. The source simulated a 10-millisecond load using a pool of 100 connections. Microlens.Cache recorded one factory execution per round, with a p99 request latency of 11.7 milliseconds. That latency includes the simulated load. These are measurements from a specific workload, not a prediction for every database or deployment.</p><p>The boundary matters: this coordination belongs to one cache instance. Separate instances each coordinate their own loads, and expiration or invalidation can create a later load wave (Part 5, Part 8).</p><p>The architectural decision is small but useful: make ownership of unfinished work explicit, then let overlapping callers share it.</p><h3>Explore the project</h3><p>Explore the library’s architecture, capabilities, and measurements on the <a href="https://mansoorafzal.web.app/cache.html"><strong>Cache</strong></a> project page.</p><p><em>Part 1 of 18 · </em><a href="https://medium.com/@mansoor.afzal/why-a-slow-load-must-never-overwrite-a-newer-write-c4cdbcee6b27"><em>Next</em></a><em> </em>➝</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=119092c43c2c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>