Quorum and leader election
Plain English: In a distributed system with multiple replica nodes, you don’t want every node agreeing on every decision (too slow) but you also can’t let just one node decide alone (no safety if it fails). Quorum is the compromise: you need a majority of nodes to agree before a write is considered “done.” If you have 5 nodes, quorum is 3.
Technical details: This is why cluster sizes are usually odd numbers (3, 5, 7) — it avoids ties and defines a clear majority. When a leader node fails or gets partitioned away, the remaining nodes run a leader election (Raft, Paxos, ZooKeeper’s ZAB) to pick a new one. But election requires quorum too — if you lose enough nodes that no majority group remains, the cluster can’t elect a leader or accept writes at all. It doesn’t crash; it just stops making progress, which looks like a hang, not a failure.
Why it matters: This is exactly what took down Coinbase’s matching engine on May 7, 2026 (see below) — losing 3 of 5 nodes killed quorum, so trades stopped processing even though 2 nodes were healthy. Interview angle: if asked “what happens when a distributed system loses quorum,” the answer is not “it errors out” — it’s “it becomes unavailable but stays consistent” (this is the CP side of the CAP tradeoff). Common pitfall: teams design for node failure but forget to test what happens when failures span availability zones, which is when quorum loss actually bites in production.
Get Pranav Sagar’s stories in your inbox
Join Medium for free to get updates from this writer.
Industry news
Coinbase published a postmortem (covered by InfoQ in late June) on its May 7 outage: an AWS us-east-1 cooling failure killed 3 of 5 nodes in their matching engine cluster, causing quorum loss and halting nearly all trading. Separately, a bug in AWS’s managed Kafka (MSK) control plane got two clusters stuck in a “healing” state where producers couldn’t write, cascading into their fee and quoting services. Worth studying if you want a real example of why cross-AZ failover for stateful clusters needs to be automated, not manual. InfoQ / Coinbase blog
Base (the Coinbase-backed L2 blockchain) had two block production outages on June 25–26. Root cause: a bug let stale journal state persist after a failed transaction validation, so the sequencer kept building on bad state. Good concrete example of why state cleanup after a failed operation is easy to get wrong — the fix was patching sequencers to properly reset journal state on failure. Base blog
Databricks moved its integrated CDC (change data capture) pipeline for MySQL into Beta — one pipeline handles both extraction and application of change data instead of stitching together separate tools. If you’ve dealt with CDC pipelines (Debezium + Kafka + a sink connector), this is the industry direction: fewer moving parts, less operational overhead. Databricks release notes
