<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Build &amp; Learn on Medium]]></title>
        <description><![CDATA[Stories by Build &amp; Learn on Medium]]></description>
        <link>https://medium.com/@BuildAndLearn?source=rss-9430f59ed0df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xjsRIrE2ikD5SIgDXoXjSA.png</url>
            <title>Stories by Build &amp;amp; Learn on Medium</title>
            <link>https://medium.com/@BuildAndLearn?source=rss-9430f59ed0df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:21:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@BuildAndLearn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your “Tiny” Validator Might Be the Reason You Need Another Server.]]></title>
            <link>https://aws.plainenglish.io/your-tiny-validator-might-be-the-reason-you-need-another-server-1fbb89e2d1e1?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fbb89e2d1e1</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 08:01:49 GMT</pubDate>
            <atom:updated>2026-10-26T08:01:49.229Z</atom:updated>
            <content:encoded><![CDATA[<h4>Before buying capacity, find the inexpensive operation your application performs an expensive number of times.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tngDRLR-rr6bTcqL_5m25g.png" /></figure><p>The next server looks like a reasonable purchase.</p><p>CPU is high. Traffic is growing. Autoscaling adds capacity and the graph settles down.</p><p>The bill is easy to explain: more customers, more infrastructure.</p><p>But there is another possibility. Your customers are asking for more work, and one ordinary library is charging the system a small fee every time it touches that work.</p><p>Nobody notices the fee in a code review. It fits on one line.</p><p>In October, Trigger.dev reported that upgrading from Zod 3 to Zod 4.5 reduced CPU demand in one main service by nearly a third. One measured hot path performed roughly 98 million parses per day. Those are their production observations, not a benchmark I ran or savings every Zod user should expect. <a href="https://trigger.dev/blog/zod-4-got-rid-of-a-bunch-of-cpus-for-us">Their engineering account includes the workload and qualifications.</a></p><p>The interesting question is what the equivalent line is in your application.</p><h3>A cheap operation can become the largest customer</h3><p>Code reviews tend to notice expensive-looking work.</p><p>A nested database query attracts attention. A network call inside a loop attracts attention. A validator near the entrance to a function looks like ordinary responsible engineering.</p><p>It may also execute far more often than the feature code surrounding it.</p><p>Consider a job-processing system. A payload is checked at the API boundary, wrapped into a queue message, checked again by a worker, and converted into a representation used by another component.</p><p>Some repeated checks may protect real trust boundaries. Others may be historical leftovers. You cannot tell which by searching for the word parse and deleting half the results.</p><p>The first question is frequency. How often does this path run for a completed customer operation?</p><p>The second is cost per execution under a representative payload.</p><p>The third is whether the same work is being repeated for a reason someone can still explain.</p><p>A function does not need to be individually slow to become a meaningful part of the bill. It needs to be called often enough.</p><p>That sounds obvious until a team spends a week optimizing an elegant algorithm called once per export while leaving a mundane conversion on every message untouched.</p><h3>The arithmetic can be boring and still change the decision</h3><p>Use a deliberately hypothetical example.</p><p>Suppose a repeated synchronous operation takes 50 microseconds and runs 5,000 times per second. Multiplication gives 250 milliseconds of CPU work each second — about a quarter of one core before other costs.</p><p>Those are illustrative values, not Zod measurements.</p><p>Now imagine several such operations on the same busy path. The application can spend a material share of its time preparing to do the work rather than performing the feature the customer bought.</p><p>This does not mean the preparation is unnecessary. It means its budget should be visible.</p><p>The saving can be more valuable near a capacity boundary than the average graph suggests. Reducing work on a saturated path may create room for bursts and reduce waiting. The actual effect depends on the workload; a small CPU improvement does not guarantee a particular latency improvement.</p><p>Conversely, if the service spends most of its time waiting for a database, a faster validator may barely change the user experience.</p><p>The lesson is not “validation is slow.” It is “the next capacity purchase should follow an explanation of the consumed capacity.”</p><h3>Removing validation is not the optimization</h3><p>The reckless version of this argument ends with a developer replacing a schema with a type assertion.</p><p>The compiler is happy. Untrusted input is now trusted. The application has traded a visible operating cost for an invisible correctness risk.</p><p>That is not a performance win you can safely put in a release note.</p><p>Instead, map the boundaries.</p><p>Where does data enter from a customer, another service, storage, or an external provider? Which guarantees have been established at that point? Which transformations invalidate those guarantees before the next component sees the data?</p><p>There may be redundant work within a boundary you control. There may be important checks you absolutely should keep at another one.</p><p>This is also a reason to compare equivalent behavior. A faster validator that rejects different inputs, produces different defaults, or strips different fields is not a drop-in performance improvement merely because the happy-path object looks the same.</p><p>Zod’s own release documentation points to a migration guide for breaking changes. The existence of a faster major release does not eliminate migration work. <a href="https://zod.dev/v4">The release notes distinguish performance improvements from compatibility.</a></p><p>The goal is to spend less to establish the guarantees you need, not stop establishing them.</p><h3>Profile the work customers actually send</h3><p>A tiny synthetic object is useful for learning how a library behaves. It is weak evidence for the cost of a production path that handles deeply nested or inconsistent payloads.</p><p>Start with measurements from the affected service. Identify where CPU time accumulates and which inputs reach that path.</p><p>Then build a controlled comparison with representative, non-sensitive payload shapes. Include valid input, rejected input, and the cases that trigger error construction. Keep the runtime and configuration consistent enough to know what changed.</p><p>If you cannot explain whether the test is exercising the same work as production, the benchmark is not ready to choose your deployment.</p><p>For a broader approach to finding and checking those bottlenecks, the performance material in <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library">The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</a> helps connect a symptom to the work causing it. That is the useful next step before another round of optimizations that make a chart better without making the service cheaper or more reliable.</p><p>The test should also tell you what would make you reject the change. Incorrect outputs and unacceptable error behavior count, even when the CPU graph looks excellent.</p><h3>Capacity is also time to investigate</h3><p>There is no medal for refusing to scale a service that needs relief.</p><p>If customers are suffering, adding capacity can be the right immediate action. It buys time and limits damage while you investigate.</p><p>The mistake is letting that successful mitigation finish the explanation.</p><p>After the pressure is off, ask what fraction of the work scales with each customer action. Find the repeated operation that seems too small to matter. Measure it before deciding the application is simply becoming more expensive because it is becoming more popular.</p><p>Sometimes the result will justify the extra server. Good. You have learned that the cost belongs to useful work.</p><p>Sometimes the result will reveal duplicated transformations, excessive serialization, or a library version that has become a recurring tax on growth.</p><p>The same investigation is valuable in both cases.</p><p>A tiny line of code can be the busiest thing in the company. Treat it accordingly before hiring another machine to stand behind it.</p><p>Diagnosing that difference is a practical skill, not a trivia question about library benchmarks. Froquiz’s Senior Dev Challenge offers production-style scenarios for choosing what to inspect next; Premium unlocks the full challenge library.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fbb89e2d1e1" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/your-tiny-validator-might-be-the-reason-you-need-another-server-1fbb89e2d1e1">Your “Tiny” Validator Might Be the Reason You Need Another Server.</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Cache Was Fast. Every Application Worker Was Dying.]]></title>
            <link>https://blog.stackademic.com/the-cache-was-fast-every-application-worker-was-dying-5550957843bf?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/5550957843bf</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:34:41 GMT</pubDate>
            <atom:updated>2026-09-03T17:34:41.565Z</atom:updated>
            <content:encoded><![CDATA[<h4>A two-millisecond dependency response can still deliver enough work to collapse the caller.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gG6naUT-7uEnJ-OwLWEp2g.png" /></figure><p>Redis looked excellent.</p><p>Latency stayed low. Connection count was normal. Hit rate was high, CPU was quiet, and the cache dashboard had more green than the incident channel had people.</p><p>The application was falling over anyway.</p><p>Workers stopped accepting requests, memory climbed in sharp steps, and restarts offered only a few minutes of relief. Engineers kept inspecting the cache transport because every failure began immediately after a cache read.</p><p>The read was not slow.</p><p>The value was enormous.</p><h3>Dependency latency hid dependency cost</h3><p>One key had gradually accumulated a large serialized object. The cache returned its bytes quickly, so the operation looked healthy from the server’s perspective.</p><p>The application then had to receive the payload, allocate buffers, decompress it, deserialize thousands of nested objects, validate fields, and temporarily hold both encoded and decoded forms in memory. A single hit consumed more CPU and memory than dozens of ordinary requests.</p><p>Under concurrency, the temporary allocations overlapped. Garbage collection worked harder. Request queues grew. More requests timed out and retried, creating more cache hits and more deserialization work.</p><p>The cache was accelerating delivery of the thing that killed us.</p><p>Monitoring measured how long Redis took to answer. It did not measure how expensive the answer was to use.</p><h3>The obvious fix made the incident worse</h3><p>The first response was to add application workers. That increased the number of large values being decoded at the same time.</p><p>The second response was to increase worker memory. That delayed restarts but allowed a larger backlog to form. When a worker finally died, more in-flight work failed together and retries arrived in a tighter wave.</p><p>The third suggestion was to flush the key. That would remove the immediate payload, but the next request would rebuild it from the database. If many requests missed together, we would trade a worker collapse for a cache stampede.</p><p>The safe recovery was less dramatic:</p><p>→ stop new requests from rebuilding the object concurrently</p><p>→ expire the oversized value</p><p>→ rebuild a smaller representation once</p><p>→ release traffic gradually while watching application allocation rate</p><p>→ bypass caching for requests that could tolerate direct, bounded reads</p><p>The decisive metric was not cache latency. It was bytes returned per key multiplied by concurrent consumers.</p><h3>A cache object needs a budget</h3><p>Teams set limits for request bodies, upload sizes, database rows, and message payloads. Cache values often escape the same discipline because they are considered internal.</p><p>Internal payloads still cross a network boundary and still become application objects.</p><p>We added three controls. First, serialization recorded encoded size before writing. Values above the budget were rejected or stored in a deliberately different representation. Second, read instrumentation captured response bytes and decode time, not only round-trip latency. Third, high-cardinality offender reports identified the keys responsible without turning key names into an unbounded metrics label.</p><p>The cached model was redesigned as a compact summary plus paginated detail. Most requests needed only the summary. The rare path that needed everything paid the cost deliberately, with concurrency limits and a deadline.</p><p>The important insight was that cache granularity is an architecture decision. One giant object improves hit rate while making invalidation, memory, network transfer, and deserialization more expensive. A high hit rate can simply mean the system is repeating a bad decision efficiently.</p><p>Incidents like this are expensive because every dashboard points toward the wrong subsystem. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> gives you the symptom-first pattern: what the healthy metric is hiding, which evidence confirms the real mechanism, how to recover without triggering a second failure, and what guardrail should exist afterward.</p><h3>We moved the boundary into the application</h3><p>After the incident, the service emitted four timings around each cache access:</p><p>→ network wait</p><p>→ bytes received</p><p>→ decode and validation time</p><p>→ downstream processing time</p><p>That separated a slow cache from an expensive cache result. The distinction mattered because the owners and fixes were different.</p><p>Memory monitoring also changed. Heap size alone was too blunt. We watched allocation rate, pause time, request concurrency, payload size distribution, and worker restarts by endpoint. A transient allocation storm could kill a container without leaving a classic long-lived leak.</p><p>Load tests began using production-shaped cache values. Previously, fixtures contained a few tidy records. They proved that the code worked, not that the representation survived real cardinality.</p><p>Finally, the cache client enforced cancellation. When a request deadline expired, decoding and downstream transformation had to stop too. Otherwise timed-out requests continued burning CPU for customers who had already left.</p><h3>Fast is not the same as cheap</h3><p>The cache team had done its job. Redis returned the requested bytes quickly and reliably.</p><p>The application team had also written correct code. It faithfully turned those bytes back into objects.</p><p>The failure lived between those truths.</p><p>Production systems collapse in the cost that dashboards omit: bytes after latency, work after a successful response, retries after a timeout, and allocations after a cache hit.</p><p>The next time a dependency is green while callers are dying, ask a different question.</p><p>Not “How long did it take to answer?”</p><p>Ask “What did the answer force us to do?”</p><p>That kind of diagnosis is what separates syntax knowledge from operating judgment. Froquiz’s Senior Dev Challenge is designed around production-style decisions where several metrics look reassuring and you still have to choose the evidence that matters.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5550957843bf" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-cache-was-fast-every-application-worker-was-dying-5550957843bf">The Cache Was Fast. Every Application Worker Was Dying.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Restore Completed. The Application Still Couldn’t Use the Backup.]]></title>
            <link>https://blog.stackademic.com/the-restore-completed-the-application-still-couldnt-use-the-backup-4a7b5e069f89?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a7b5e069f89</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:07:32 GMT</pubDate>
            <atom:updated>2026-09-03T17:07:32.130Z</atom:updated>
            <content:encoded><![CDATA[<h4>We tested whether data could be copied, not whether the business could recover.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJOU_lU1Psi7jYcodTjOlw.png" /></figure><p>The restore job finished with a green checkmark.</p><p>The database accepted connections. Row counts looked plausible. Storage checksums passed. The disaster-recovery channel relaxed for roughly four minutes.</p><p>Then the application started.</p><p>It could not decrypt customer credentials because the restored environment did not have the required key version. After that was fixed, background workers rejected old serialized payloads. Then the search index pointed at documents newer than the restored database. Finally, the only engineer who knew the correct reconciliation order was on a flight.</p><p>The backup was valid.</p><p>The product was not recoverable.</p><h3>We had tested the easiest verb</h3><p>Our quarterly exercise said <strong>restore database</strong>.</p><p>That is exactly what the team did. A snapshot became a running PostgreSQL instance. Automated checks confirmed expected schemas and rough table counts. The result was recorded as a successful recovery drill.</p><p>But customers do not buy a running database.</p><p>They expect to log in, see current entitlements, submit work, and trust that completed actions will not vanish or repeat. Those outcomes depended on more than database files:</p><ul><li>encryption keys and access policy;</li><li>object storage at a compatible point in time;</li><li>queue messages created before and after the snapshot;</li><li>application binaries that understood the restored schema;</li><li>search and cache state derived from primary data;</li><li>third-party references that could not be rolled back;</li><li>a tested order for bringing writers and consumers online.</li></ul><p>We had verified an artifact.</p><p>We had never rehearsed a service recovery.</p><h3>A successful restore can create incorrect data</h3><p>The most dangerous moment came after the database opened.</p><p>Several workers immediately connected and began consuming a backlog. Some messages described operations already reflected in the restored database. Others represented actions accepted after the recovery point. Without a replay boundary, we could duplicate effects or silently skip them.</p><p>Search and cache made the system look newer than the source of truth. Customers briefly saw records that disappeared when they opened them. A payment provider still knew about transactions our restored database had forgotten.</p><p>Nothing had corrupted the backup.</p><p>Time had split the system into several valid histories.</p><p>Recovery needed an explicit authority model:</p><p>→ Which store is authoritative for each entity?</p><p>→ What timestamp or sequence defines the recovery boundary?</p><p>→ Which side effects require reconciliation rather than replay?</p><p>→ Which derived stores must be discarded and rebuilt?</p><p>→ When is it safe to reopen writes?</p><p>Those questions were absent from the snapshot job because they belonged to the product, not the storage service.</p><h3>We replaced the green check with a recovery proof</h3><p>The new drill began in an isolated environment with no inherited application state.</p><p>Automation restored the database, object storage references, configuration, and minimum infrastructure. It deployed a binary known to be compatible with the recovery point. Derived caches started empty. Search was rebuilt or explicitly reconciled.</p><p>Then business-level probes ran:</p><ol><li>Can a known user authenticate?</li><li>Do entitlements match the billing source?</li><li>Can an existing record be read and changed?</li><li>Can a new transaction complete without duplication?</li><li>Do critical aggregates reconcile within an accepted tolerance?</li><li>Can the system explain messages around the recovery boundary?</li></ol><p>The timer stopped only when those checks passed.</p><p>That changed our measured recovery time dramatically. The snapshot restored in 22 minutes. A usable product initially took more than three hours.</p><p>Leadership had been planning around the first number.</p><p>Customers would experience the second.</p><h3>RTO and RPO needed product language</h3><p>We had documented a one-hour recovery-time objective and a five-minute recovery-point objective.</p><p>They looked precise. They were not operationally complete.</p><p>One hour until what? Database available? API starts? Read-only product? New payments accepted? Backlog reconciled?</p><p>Five minutes of loss in which system? Database writes, object uploads, queue messages, third-party side effects, or user-visible history?</p><p>We rewrote the objectives as states:</p><ul><li>within 30 minutes, operators can access restored primary data;</li><li>within 60 minutes, customers can read verified account state;</li><li>writes reopen only after payment and queue reconciliation;</li><li>all accepted pre-incident operations are classified as preserved, replayed, or manually repaired.</li></ul><p>The language became less tidy and much more useful.</p><p>The difference between a backup and a recoverable system is one of the recurring gaps teams discover only during an emergency. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> maps those gaps from visible symptom to confirming evidence, recovery action, and prevention control — including the checks that prove a green job produced a working business rather than a pile of readable files.</p><h3>The runbook became executable</h3><p>Our old recovery document contained correct sentences and stale assumptions.</p><p>The new process turned as much of the sequence as possible into automation. Required key aliases, image digests, schema compatibility, network access, and restore permissions were tested continuously. The remaining human decisions had named owners and evidence attached.</p><p>Every drill generated a ledger:</p><ul><li>recovery point selected;</li><li>data validation results;</li><li>start and completion timestamps;</li><li>failed dependencies;</li><li>manual interventions;</li><li>reconciliation differences;</li><li>measured RTO and RPO;</li><li>follow-up owner and deadline.</li></ul><p>A restore that completed but failed validation counted as a failed drill. A drill that required undocumented heroics also counted as a failure.</p><p>This was important. Otherwise, humans quietly repair automation during rehearsals and the dashboard records a capability the organization does not actually possess.</p><h3>The backup was never the product</h3><p>We still monitor backup completion, retention, replication, and integrity.</p><p>Those are necessary controls. They are not evidence that customers can recover their world.</p><p>A backup preserves bytes from a point in time. Recovery reconstructs a coherent system around those bytes while external systems continue to remember what happened.</p><p>That is why the first real restore surprises teams even when every backup job is green.</p><p>Do not ask only whether you can restore the database.</p><p>Ask whether a customer can safely perform the next irreversible action after you do.</p><p>The gap between those two questions is where disaster-recovery plans become production incidents.</p><p>Recovery scenarios reward the engineer who checks the business state behind the infrastructure result. Froquiz’s senior challenges are built to practice that next decision when a green signal is technically true and operationally incomplete.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a7b5e069f89" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-restore-completed-the-application-still-couldnt-use-the-backup-4a7b5e069f89">The Restore Completed. The Application Still Couldn’t Use the Backup.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Fixed Production Before Support Replied. Customers Still Left.]]></title>
            <link>https://medium.com/@BuildAndLearn/we-fixed-production-before-support-replied-customers-still-left-b5ab628d2fba?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5ab628d2fba</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:19:06 GMT</pubDate>
            <atom:updated>2026-09-01T22:19:06.581Z</atom:updated>
            <content:encoded><![CDATA[<h4>The technical recovery took nine minutes; silence turned a contained failure into a trust incident.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mA3BKdCbr9FpUy6P6MHeog.png" /></figure><p>At 13:07, customers could not export invoices.</p><p>The on-call engineer found a failed object-storage permission change, reverted it, and verified successful exports by 13:16. No data was lost. The affected feature worked again.</p><p>Support’s first public reply arrived at 15:02.</p><p>Between those times, finance teams retried, opened tickets, warned their managers, and assumed invoices might be gone. Several annual customers escalated cancellation requests even though the technical problem had already ended.</p><p>Production recovered. Confidence did not.</p><h3>Customers experienced uncertainty, not nine minutes</h3><p>The engineering timeline began when the alert fired and ended when exports passed verification.</p><p>The customer timeline began when a month-end task failed and continued until someone explained whether the data was safe, whether retrying would duplicate anything, and when the result could be trusted.</p><p>A restored button did not answer those questions.</p><p>The gap allowed customers to invent their own worst-case explanation. For a financial workflow, silence sounded like lost records rather than a temporary permission failure.</p><p>Repeated customer retries also created operational noise. Each failed export generated another job and another support ticket, making the backlog and perceived impact larger than the original outage.</p><p>A timely message telling users that source invoices remained safe and retries were unnecessary would have protected both trust and recovery capacity.</p><h3>The status page waited for certainty</h3><p>The incident policy required a confirmed root cause before posting externally. That protected the company from publishing an incorrect explanation.</p><p>It also prevented a basic acknowledgement: we see failures, invoice data remains stored, avoid repeated exports, and the next update will arrive in fifteen minutes.</p><p>Communication had been treated as the final report, not part of mitigation.</p><p>By the time the root cause was confirmed, the most valuable uncertainty-reducing message was already late.</p><p>We separated facts by confidence. “Exports are failing” was observed. “Stored invoice data is unaffected” was verified through read checks. “A permission change caused it” remained a hypothesis until audit evidence arrived.</p><p>The communicator could publish the first two without presenting the third as settled. Communication became faster because it no longer required pretending diagnosis was complete.</p><h3>Support and engineering held different evidence</h3><p>Engineering knew storage reads worked and only new export writes had failed. Support saw a flood of “invoice missing” tickets with no way to distinguish failure modes.</p><p>We created an incident view that exposed safe operational facts: affected workflow, start time, current mitigation state, confirmed data-safety status, retry guidance, and next update time.</p><p>It intentionally omitted sensitive internal details and uncertain speculation. Support could communicate useful truth without waiting for a full root-cause analysis.</p><p>Ticket tags and product telemetry also flowed back into the war room. Customer language revealed that some users were blocked from regulatory deadlines, changing the incident priority even though aggregate request volume was small.</p><p>Support also identified affected customers who had not retried and therefore disappeared from error metrics. Their scheduled process failed once and stopped. Counting only repeat traffic would have underreported both impact and urgency.</p><p>The incident view combined unique affected accounts with request failures. Volume described system load; account and workflow context described business harm.</p><p>For strategic accounts, the communicator sent targeted notices instead of relying only on a public status page. The message named the blocked workflow, confirmed what remained safe, and gave the next update time. Account teams no longer learned about an incident from an angry customer after engineering had already closed it.</p><p>Post-incident follow-up acknowledged the inconvenience without claiming that nine minutes made it insignificant. Customers cared about whether the company understood their deadline and could prove their records remained intact.</p><p>The lesson changed future incident reviews. Communication was evaluated beside detection, containment, and recovery — not relegated to a public-relations footnote after engineering declared success.</p><p>Trust mattered.</p><h3>Communication became a recovery control</h3><p>The incident commander assigned a communicator at the same time as a technical lead. The first message acknowledged impact and bounded known risk. Updates followed a clock even when the technical state had not changed.</p><p>The final message explained what customers needed to do: retry once, expect no duplicates, and contact support if the export timestamp fell inside the identified window.</p><p>That instruction prevented random repeated actions from creating additional load and duplicate files.</p><p>Expensive incidents are shaped by human reactions as much as broken components. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> helps teams connect symptoms, business impact, recovery choices, and prevention — so restoring the service does not leave customers alone with uncertainty that costs more than the original failure.</p><h3>We measured time to trustworthy guidance</h3><p>Mean time to recovery remained important. We added time to acknowledgement, time to confirmed safety guidance, and time until affected customers received a clear next action.</p><p>Templates were organized by incident shape: delayed processing, duplicate risk, read-only degradation, data-integrity uncertainty, and complete unavailability. They were starting points, not automated claims.</p><p>Exercises included the communication path. Engineers had to state what was known, unknown, safe, and dangerous while diagnosis continued. Support practiced translating that state without promising an unverified resolution time.</p><p>Customer repair continued after technical closure for accounts that had deadlines or had opened escalations. Trust recovery received an owner instead of disappearing into a closed incident channel.</p><h3>Silence can expand the blast radius</h3><p>The permission bug affected one workflow for nine minutes. The communication failure affected every customer deciding whether the company understood its own system.</p><p>Clear updates do not hide incidents. They reduce harmful uncertainty and prevent customers from taking unsafe compensating actions.</p><p>The company did not need a perfect explanation at minute five. It needed a truthful boundary and a promise to update.</p><p>Froquiz’s senior scenarios practice that judgment: acknowledge, advise against retries, state data safety, escalate impact, or wait for evidence when engineering has restored the path but customers still do not know what happened.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5ab628d2fba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Index Build Never Blocked Writes. It Put the Replica 40 Minutes Behind.]]></title>
            <link>https://blog.stackademic.com/the-index-build-never-blocked-writes-it-put-the-replica-40-minutes-behind-4ac27c4cc3d4?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ac27c4cc3d4</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:02:33 GMT</pubDate>
            <atom:updated>2026-09-03T11:24:45.848Z</atom:updated>
            <content:encoded><![CDATA[<h4>`CONCURRENTLY` protected the primary request path and moved the production risk into recovery and reads.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxFbw3Z_jQwPkCd-WNCQUg.png" /></figure><p>The migration plan used CREATE INDEX CONCURRENTLY for the right reason.</p><p>The table was large and busy. A normal index build could block writes. The concurrent build allowed inserts, updates, and deletes to continue while PostgreSQL scanned the table.</p><p>Primary latency stayed within target. Checkout did not freeze. The deployment dashboard stayed green.</p><p>Replication lag climbed from seconds to forty minutes.</p><p>Read traffic began returning old state. More dangerously, the standby we expected to promote during a primary failure was no longer a current recovery target.</p><p>We had avoided one incident shape and quietly weakened the system’s escape route.</p><h3>Non-blocking did not mean low-impact</h3><p>Concurrent index creation performs more coordination and scanning than the word “concurrent” suggests. It still consumes I/O, CPU, memory, and generates work that replicas must receive and replay.</p><p>Our review asked whether the table would remain writable. It did not ask what resources the build shared with replication.</p><p>The primary had enough capacity to serve traffic and build the index. The replica’s storage and replay path did not. WAL accumulated faster than it could apply.</p><p>The index operation was succeeding locally while degrading the freshness and failover posture of the system.</p><h3>Read traffic turned lag into customer behavior</h3><p>Several endpoints used the replica for order history and account state. Customers completed writes on the primary and then saw older data for much longer than the product tolerated.</p><p>The team initially blamed cache staleness because primary queries returned correct rows. Replica replay position showed the real gap.</p><p>We routed consistency-sensitive reads to the primary and reduced nonessential workload. That preserved correctness but consumed the capacity the replica normally provided. The maintenance operation now competed with both replication and fallback reads.</p><p>We paused the deployment around the index build and let the replica catch up before continuing. Abandoning the operation blindly could leave an invalid index requiring cleanup, so the team checked build state and progress first.</p><h3>Failover readiness became a release constraint</h3><p>The old release gate considered primary error rate, locks, and query latency. The new gate included replay lag in bytes and time, replica apply rate, storage saturation, and estimated catch-up duration.</p><p>A maintenance change could remain technically successful while violating the recovery point objective. In that case, automation paused rather than pretending redundancy still existed.</p><p>We defined the maximum window in which a standby could be unavailable as a safe failover target. If the index build threatened that budget, it ran in smaller operational windows, used more capable replica storage, or waited for lower traffic.</p><p>High availability was treated as capacity that maintenance could consume.</p><h3>We separated creation from adoption</h3><p>After the index became valid, query plans could change. That created another risk independent of the build.</p><p>We verified the index state, analyzed representative plans, and monitored query latency before relying on it. A failed concurrent build could leave an invalid artifact; an unexpectedly attractive index could change plans in ways staging data never revealed.</p><p>The migration runbook therefore covered build progress, cancellation, invalid-index cleanup, replica recovery, and post-build plan behavior. “Command returned successfully” was only one checkpoint.</p><p>Production maintenance incidents often hide in the resource or safety mechanism the change was designed to protect. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> develops the broader response: confirm where pressure moved, protect the customer path and failover posture, recover without leaving invalid state, and make the hidden budget visible before the next migration.</p><h3>We rehearsed a primary failure during maintenance</h3><p>The most revealing test did not measure how long the index took. It asked what would happen if the primary failed halfway through.</p><p>Could the standby be promoted? How much committed data would be missing? Did the recovery runbook distinguish a lagging standby from a healthy one? Would applications route reads to a replica still applying the build?</p><p>The exercise exposed that the on-call dashboard displayed “replica connected” as healthy even when recovery lag exceeded the business objective. Connected was not promotable.</p><p>We added a maintenance admission check that estimated table size, current write rate, replica apply headroom, and conflicting long transactions. Estimates were imperfect, but they forced the deployment to state which capacity it expected to consume. If actual lag or I/O crossed the declared budget, the runbook already contained a pause or cancel threshold.</p><p>Long-running transactions received special attention because concurrent index creation may wait for transactions that could affect the build. A migration that appeared stalled could be waiting correctly, while impatient retries started competing builds or left invalid artifacts. Progress views and lock relationships were captured before anyone issued a second command.</p><p>Replica topology mattered too. A downstream cascading replica could lag even after the first standby appeared acceptable. Reads and disaster-recovery roles were mapped per replica, and maintenance health followed the slowest required recovery path — not whichever node had the prettiest graph.</p><p>Finally, we scheduled post-maintenance catch-up as part of the change window. The operation was not complete when the index became valid; it was complete when replicas regained their objectives, fallback reads returned to normal routing, and failover drills again had a current target.</p><h3>Every safe migration spends a different budget</h3><p>CONCURRENTLY remained the correct tool. Removing it would have traded replica lag for blocked production writes.</p><p>The fix was to stop describing the operation as free. It exchanged lock risk for longer execution and resource pressure. That trade had to fit primary capacity, replica capacity, and failover requirements at the same time.</p><p>The migration checklist therefore records the budget before execution: maximum replica lag, maximum WAL growth, the read-routing response, and the point at which cancellation is safer than completion. “Online” became a mechanism, not permission to run without limits.</p><p>Froquiz’s production scenarios practice that decision: continue, pause, cancel, route reads, or protect failover when the primary is healthy and the safety margin is disappearing elsewhere.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ac27c4cc3d4" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-index-build-never-blocked-writes-it-put-the-replica-40-minutes-behind-4ac27c4cc3d4">The Index Build Never Blocked Writes. It Put the Replica 40 Minutes Behind.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Interview Was Going Perfectly. Then They Asked Me to Remove the Database.]]></title>
            <link>https://blog.stackademic.com/the-interview-was-going-perfectly-then-they-asked-me-to-remove-the-database-2aa1e8031281?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/2aa1e8031281</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 20:42:42 GMT</pubDate>
            <atom:updated>2026-09-03T11:12:15.499Z</atom:updated>
            <content:encoded><![CDATA[<h4>The follow-up was not asking for a databaseless system; it was testing whether I knew what state the database protected.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h0Su_d_KxCjNDQ8-tTzj7A.png" /></figure><p>The design was nearly finished.</p><p>API servers wrote to PostgreSQL. Redis handled hot reads. A queue moved slow work out of the request path. Read replicas supported reporting. Backups and failover completed the reliability story.</p><p>Then the interviewer said:</p><p>“Assume the database is unavailable for ten minutes. What can your product still do?”</p><p>I tried to replace PostgreSQL with another database.</p><p>That was not the question.</p><h3>Start by naming the state</h3><p>“The database” often protects several kinds of truth:</p><ul><li>user identity and authorization;</li><li>product catalog;</li><li>inventory and reservations;</li><li>orders and payments;</li><li>session state;</li><li>configuration;</li><li>idempotency records;</li><li>audit history.</li></ul><p>They do not all require the same availability or consistency.</p><p>A cached product page may remain safe to serve. Creating an order without durable identity, inventory, and idempotency may not.</p><p>The useful answer begins by classifying operations:</p><pre>safe read from stale snapshot<br>safe write to alternate durable log<br>write that must fail closed<br>operation that can be accepted as pending<br>operation that must remain strongly consistent</pre><p>Now “database unavailable” becomes several product decisions instead of one infrastructure panic.</p><h3>Availability without truth is not success</h3><p>It is easy to keep returning HTTP 200.</p><p>Serve cached data. Put writes in memory. Tell users the operation succeeded and persist it later.</p><p>That design improves uptime while risking lost orders, unauthorized actions, duplicate payments, and promises the system cannot reconstruct.</p><p>The API should distinguish available from accepted and committed.</p><p>If a write enters a durable queue while the database is down, the response might say 202 Accepted and expose pending status. If no durable boundary remains, the system should reject the write rather than manufacture success.</p><p>For security decisions, stale data may be unacceptable. A revoked permission must not remain active because the authorization database is unavailable and a long-lived cache looks convenient.</p><h3>Decide what can be rebuilt</h3><p>Some state is derived.</p><p>Search indexes, recommendation features, analytics aggregates, and many caches can be rebuilt from authoritative data. Their outage should degrade features, not redefine truth.</p><p>Some state represents business commitment.</p><p>Payment operations, inventory reservations, account ownership, and order transitions require durable identity and recovery evidence.</p><p>Separate systems accordingly. A database outage should not erase every read path, but a search index should not quietly become the authority for writes.</p><p>Write down the source of truth for each major state and the maximum staleness tolerated by each consumer.</p><p>That table is more useful than adding another replica without explaining which reads may use it.</p><h3>Design a bounded degraded mode</h3><p>A ten-minute database outage needs limits.</p><p>Cached catalog reads can continue until freshness or safety thresholds expire. New orders may be disabled, accepted into a durable regional log, or restricted to products without scarce inventory. Account changes and permission updates may fail closed.</p><p>Communicate the mode to users and operators. Do not let a fallback silently become normal behavior.</p><p>Track:</p><ul><li>age of cached state;</li><li>pending operations awaiting replay;</li><li>capacity of the alternate durable path;</li><li>actions rejected by safety policy;</li><li>reconciliation failures after recovery.</li></ul><p>When the database returns, drain deliberately. Replaying ten minutes of writes at maximum concurrency can take the database down again.</p><p>This is a recurring production-engineering pattern: graceful degradation is useful only when the system knows which guarantees it is allowed to weaken.</p><h3>Remove the dependency, not the data model</h3><p>The interviewer did not expect a magical system with no durable state.</p><p>They wanted to see whether the design had isolated read paths, identified critical writes, and prepared an honest failure mode.</p><p>Possible answers include:</p><ul><li>serve public cached content while disabling mutations;</li><li>use an append-only durable log for bounded pending operations;</li><li>keep authentication verification local while failing closed on authorization changes;</li><li>route to another healthy region only if consistency and failover are designed;</li><li>expose pending state and reconcile after recovery;</li><li>reject operations whose invariants cannot be protected.</li></ul><p>State the reversal condition. If the alternate path cannot guarantee durability, ordering, tenant isolation, or bounded backlog, do not accept the write.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> builds the judgment behind those boundaries — how state, retries, caches, recovery, and hidden dependencies behave when the central box stops cooperating.</p><p>The database still matters.</p><p>The stronger architecture simply prevents one unavailable dependency from forcing every feature into the same dishonest answer.</p><p>My first response tried to swap technologies because I had attached the design to PostgreSQL.</p><p>The better response attached the design to its invariants.</p><p>Once you know which truth each operation needs, removing a box from the whiteboard becomes a useful test instead of a trick question.</p><p>Test the degraded path before trusting it. Disconnect the primary database while cached reads and alternate writes remain active. Verify that every endpoint advertises the right status, no unsafe mutation falls back to memory, and health checks reflect the user capability rather than process liveness.</p><p>Time changes the answer. A cache that is safe for thirty seconds may become dangerous after ten minutes. Attach an expiry and policy to degraded behavior so the system moves from stale service to explicit rejection instead of drifting indefinitely.</p><p>Recovery needs ordering. Restore connectivity, verify the database role, limit new traffic, replay pending operations with stable identities, reconcile conflicts, and only then return to normal mode. If all clients reconnect and drain simultaneously, connection pools and locks can create a second outage.</p><p>Regional failover is not a free substitute. Replication may lag, DNS may continue routing to the old region, and two writable primaries can create split brain. State the consistency and fencing mechanism before claiming another region solves the ten-minute gap.</p><p>The most convincing interview answer includes what the product displays. “Browse-only mode,” “payment confirmation pending,” or “account changes temporarily unavailable” are architectural outcomes. Graceful degradation succeeds when users receive a smaller truthful product, not a full interface backed by weaker guarantees nobody can see.</p><p>Practice redesigning around failed dependencies until degraded mode, recovery, and truth ownership become part of the first answer.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2aa1e8031281" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-interview-was-going-perfectly-then-they-asked-me-to-remove-the-database-2aa1e8031281">The Interview Was Going Perfectly. Then They Asked Me to Remove the Database.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Rotated a Secret Without Downtime. Half the Pods Kept the Old One for Six Hours.]]></title>
            <link>https://blog.stackademic.com/we-rotated-a-secret-without-downtime-half-the-pods-kept-the-old-one-for-six-hours-98c4b0836ee6?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/98c4b0836ee6</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:19:59 GMT</pubDate>
            <atom:updated>2026-09-03T11:09:01.680Z</atom:updated>
            <content:encoded><![CDATA[<h4>The rotation succeeded in the secret manager and failed everywhere that cached the past.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KR_hfj2mlIvoWiwLZYZehQ.png" /></figure><p>The database credential was rotated at 09:00. The secret manager showed the new version as active. New pods connected successfully. No deployment failed.</p><p>By 09:20, authentication errors had settled at 4% — too low for the global alert, high enough for customers to notice.</p><p>The failures followed no obvious endpoint. A retry often worked. Dashboards showed healthy pods in every zone.</p><p>The secret had changed once. The running system now contained three different ideas of what “current” meant.</p><h3>A secret is distributed state</h3><p>We had treated rotation as a write operation: create a new value, mark it active, and verify one connection.</p><p>In production, that value moved through several layers:</p><pre>secret manager -&gt; sync controller -&gt; Kubernetes Secret<br>-&gt; pod environment -&gt; application config<br>-&gt; connection pool -&gt; existing authenticated sessions</pre><p>Each arrow had its own refresh semantics.</p><p>The controller updated the Kubernetes Secret quickly. Pods consuming it as a mounted volume eventually saw new file content. Pods consuming it through environment variables could never see the change without restart. The application read configuration only during boot. Its database pool held connections authenticated with the previous credential.</p><p>“The secret is updated” described one object. It did not describe the consumers.</p><p>The 4% error rate came from old pods receiving a fraction of traffic after the database stopped accepting the previous password. Requests were randomly routed across old and new generations, so retries appeared to heal the problem.</p><p>That made the failure look intermittent even though each pod behaved consistently.</p><h3>Why restarting everything was not our safest move</h3><p>The first proposal was a full rollout restart. It might have worked. It also could have replaced a partial authentication failure with a total capacity failure if new pods could not initialize.</p><p>Before changing the fleet, we tested the new credential from the same network path and runtime identity used by the application. Then we restarted one canary pod and watched four things:</p><ul><li>startup success;</li><li>database authentication failures;</li><li>readiness transition time;</li><li>connection pool creation.</li></ul><p>After the canary served real traffic, we rolled the fleet gradually with a conservative unavailable budget.</p><p>That restored service, but it did not make rotation safe. The real fix was a two-version overlap protocol.</p><p>For systems that support it, rotation now follows this sequence:</p><ol><li>Create the new credential while the old one remains valid.</li><li>Distribute the new version and restart or reload consumers gradually.</li><li>Prove every active consumer has switched.</li><li>Revoke the old credential.</li><li>Watch for attempts using the revoked identity.</li></ol><p>The overlap is not laziness. It is the compatibility window that lets distributed consumers converge.</p><h3>“Reloadable” must be proven per consumer</h3><p>We audited every secret by how it entered and remained in the process.</p><p>A mounted file can update without a pod restart, but that does not mean the application rereads it. A configuration library may cache the first value. A connection pool may create new connections lazily while old sessions stay valid. A sidecar may refresh on a timer. A long-running job may hold credentials for hours.</p><p>We added a rotation contract to each consumer:</p><pre>source:             mounted file<br>refresh trigger:    file watcher plus periodic fallback<br>maximum staleness:  5 minutes<br>connection action:  drain pool and reconnect<br>proof signal:       secret version hash, never the secret value</pre><p>The version hash mattered. We needed to know which configuration generation a pod was using without exposing credential material in logs or metrics.</p><p>During rollout, a dashboard grouped healthy replicas by safe version identifier. “20 pods ready” became “10 ready on version A, 10 ready on version B.” That immediately revealed incomplete convergence.</p><p>We also stopped using readiness as proof of credential validity. A pod could remain ready because its existing database connections worked, then fail when the pool opened a new connection. The health model needed an authentication signal tied to connection creation, not a generic HTTP response.</p><h3>Rotation failures hide inside successful retries</h3><p>Retries reduced visible errors and delayed diagnosis.</p><p>A request hitting an old pod failed authentication. The retry landed on a new pod and succeeded. The customer saw extra latency; the aggregate success rate remained acceptable; the old generation continued serving traffic.</p><p>This is a dangerous class of partial failure because the system spends redundancy to hide inconsistency. Every successful retry consumes evidence.</p><p>We added first-attempt failure metrics and grouped them by pod generation. We kept retries, but stopped allowing them to erase the original outcome from telemetry.</p><p>The same principle applies beyond secrets. Certificate rotations, DNS changes, feature flags, schema versions, and signing keys all create mixed-generation periods. If observability only records the final attempt, those periods look healthier than they are.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is for engineers who need the operating layer behind changes like this — the rollout contracts, failure signals, containment decisions, and recovery patterns that turn “updated successfully” into “safe across the whole system.”</p><h3>The rotation was finished only when the old key was quiet</h3><p>Our final definition of done became stricter.</p><p>The new value being active was the beginning. Completion required every active workload to report the new version, every pool to establish fresh connections, synthetic requests to pass across zones, and old-credential attempts to remain at zero before revocation.</p><p>We also practiced failure in staging: stop the sync controller, delay one pod, block one restart, and revoke the old credential early. The exercise revealed whether the runbook described reality or merely the happy path.</p><p>Secret rotation is often sold as a security hygiene task. Operationally, it is a distributed migration with an irreversible step.</p><p>The hard question is not “Did the secret manager accept the new value?” It is “Can we prove that every consumer has moved before we invalidate the past?”</p><p>Once we asked that question, the six-hour mystery became predictable. Half the pods were not randomly broken. They were faithfully using configuration we had never taught them to forget.</p><p>That kind of diagnosis is less about memorizing commands than recognizing a mixed-generation system. Froquiz is built to exercise that judgment while the stakes are still hypothetical.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98c4b0836ee6" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-rotated-a-secret-without-downtime-half-the-pods-kept-the-old-one-for-six-hours-98c4b0836ee6">We Rotated a Secret Without Downtime. Half the Pods Kept the Old One for Six Hours.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Traced Every Request. The One Incident We Needed Had Been Sampled Out.]]></title>
            <link>https://blog.stackademic.com/we-traced-every-request-the-one-incident-we-needed-had-been-sampled-out-1329b1b8f9e5?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/1329b1b8f9e5</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:34:41 GMT</pubDate>
            <atom:updated>2026-08-30T02:34:41.858Z</atom:updated>
            <content:encoded><![CDATA[<h4>Observability looked complete until tail-based rarity, privacy rules, and cost controls removed the only path that mattered.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JiGSN8Nu-zHSsPMagaoF_g.png" /></figure><p>The incident affected 0.07% of checkout requests.</p><p>Those requests took between eighteen and forty seconds, then failed.</p><p>Metrics showed a small tail-latency bump. Logs showed timeouts. Traces should have connected the story.</p><p>We opened the tracing backend and found no complete affected trace.</p><p>The requests were rare enough to matter and common enough to hurt.</p><p>Our sampling policy had discarded nearly all of them.</p><h3>“Tracing enabled” was not the same as evidence retained</h3><p>Every service emitted trace data.</p><p>That statement was technically true.</p><p>To control cost, the collector sampled a small percentage of ordinary requests. Error traces were favored, but the failure occurred after a proxy timeout. Several internal spans completed successfully before the client-facing layer marked the request failed.</p><p>By the time the final status was known, upstream sampling decisions had already removed important spans.</p><p>Privacy filtering also stripped the tenant and request attributes that could have grouped the failures. We had data without the context needed to find it.</p><p>The instrumentation existed.</p><p>The incident evidence did not.</p><h3>The average trace was the least useful trace</h3><p>Most successful requests followed the same short path.</p><p>They dominated storage because uniform sampling preserved them in proportion to traffic. The rare checkout path involved one regional tax dependency and a specific payload shape. It was expensive, unusual, and underrepresented.</p><p>Observability cost controls often remove data by volume.</p><p>Incident value is not distributed by volume.</p><p>One ordinary trace is interchangeable with thousands of similar traces. One anomalous trace may explain an outage. Treating both as equal produces a beautifully representative dataset of the behavior you already understand.</p><h3>We sampled after learning the outcome</h3><p>We moved critical decisions toward tail sampling.</p><p>The collector temporarily retained spans, assembled the trace, and then applied policies based on the completed outcome:</p><p>→ Keep errors.</p><p>→ Keep requests above latency thresholds.</p><p>→ Keep rare routes and selected tenant classes.</p><p>→ Keep traces containing dependency anomalies.</p><p>→ Sample routine success aggressively.</p><p>We also created an emergency mode that increased retention for a bounded period and service set. Cost had a ceiling. So did blindness.</p><p>Privacy-safe correlation identifiers survived filtering, allowing support reports to locate evidence without retaining sensitive payloads.</p><p>Production debugging repeatedly begins with missing, aggregated, expired, or misleading evidence. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> continues those failure shapes and the operational decisions that preserve useful evidence before the page arrives.</p><h3>Observability is a prediction about future questions</h3><p>You cannot retain everything forever.</p><p>Every logging, metrics, and tracing design predicts which questions future responders will ask. Sampling makes that prediction explicit: this evidence is likely to matter; that evidence is probably redundant.</p><p>The danger is allowing cost policy to make the prediction silently.</p><h3>Complete instrumentation can still produce an empty investigation</h3><p>After the policy change, another rare latency incident occurred.</p><p>This time one retained trace showed a connection attempt waiting behind a regional pool whose global utilization looked normal. The fix took twenty minutes.</p><p>The value did not come from tracing every request.</p><p>It came from keeping the request that contradicted the healthy story.</p><p>Observability is not the volume of data produced. It is the probability that you can answer the expensive question after the fact.</p><p>Before celebrating coverage, ask which failures your retention policy is structurally least likely to preserve.</p><p>That is where production will eventually place the clue.</p><p>Senior incident reasoning starts when metrics point toward the wrong average and the missing evidence becomes part of the hypothesis.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1329b1b8f9e5" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-traced-every-request-the-one-incident-we-needed-had-been-sampled-out-1329b1b8f9e5">We Traced Every Request. The One Incident We Needed Had Been Sampled Out.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Deleted 70% of Our Alerts. The Next Incident Was Easier to Find.]]></title>
            <link>https://blog.stackademic.com/we-deleted-70-of-our-alerts-the-next-incident-was-easier-to-find-443d993773b4?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/443d993773b4</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:15:07 GMT</pubDate>
            <atom:updated>2026-08-30T02:15:07.564Z</atom:updated>
            <content:encoded><![CDATA[<h4>More signals had not created more awareness. They had trained the on-call engineer to wait for the noise to stop.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5ZZBKV9e70e8pnMvwGwEQ.png" /></figure><p>At 02:06, twenty-three alerts fired in ninety seconds.</p><p>CPU high. Queue depth elevated. Error rate rising. Pod restarts. Database connections near limit. Latency above threshold in six services.</p><p>The on-call engineer opened the incident channel and waited.</p><p>Not because they were careless.</p><p>Because experience had taught them that most alerts would resolve after the first noisy deploy signal settled.</p><p>This time the noise did not settle.</p><p>Checkout was down for thirty-one minutes.</p><p>Our alerting system had delivered every fact and almost no direction.</p><h3>Symptoms had multiplied into pages</h3><p>One failing dependency affected many components.</p><p>Each component owned alerts for its local symptoms. The queue alerted on age and depth. APIs alerted on latency and errors. Kubernetes alerted on restarts. PostgreSQL alerted on connections and CPU.</p><p>All were accurate.</p><p>Together they described one incident as twenty-three independent emergencies.</p><p>The responder’s first task became notification management: acknowledge pages, silence duplicates, open dashboards, and guess which signal happened first.</p><p>We had automated detection and left correlation to the most interrupted person in the company.</p><h3>Warning fatigue changed behavior before the incident</h3><p>The dangerous effect was not the volume during one outage.</p><p>It was the training created by months of low-value pages.</p><p>Alerts fired for short autoscaling events, expected batch spikes, deployments, self-healing restarts, and thresholds nobody remembered choosing. Most required no action.</p><p>Every harmless page taught the on-call engineer a small lesson:</p><p>Wait before responding.</p><p>Check whether it clears itself.</p><p>Assume the alert is another local symptom.</p><p>That behavior was rational under the system we built. During the real incident, it became expensive.</p><h3>Every page needed an owner and an action</h3><p>We reviewed alerts with a brutal test.</p><p>If this fires at 3 AM, what should the responder do now?</p><p>Alerts without an answer were removed, downgraded, or turned into dashboard signals. Duplicate symptom alerts grouped beneath customer-impact pages. Expected transient states received time windows aligned with actual recovery behavior.</p><p>We kept pages for conditions requiring timely human decisions:</p><p>→ A user journey is failing beyond its error budget.</p><p>→ Data integrity may be at risk.</p><p>→ Automatic recovery has stopped making progress.</p><p>→ Capacity will exhaust before automation can respond.</p><p>→ A critical dependency failure requires degradation or traffic movement.</p><p>The number of metrics increased.</p><p>The number of reasons to wake someone decreased.</p><p>Operational failures repeat through noisy symptoms, misleading health, cascading retries, and automation that stops one step before recovery. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those patterns continue beyond one alert rule and one incident channel.</p><h3>Fewer alerts forced better system thinking</h3><p>Deleting alerts felt like deleting safety.</p><p>In practice, it forced teams to identify service outcomes, dependency relationships, and the signals that changed operator decisions. We could no longer compensate for unclear ownership by paging everyone.</p><h3>The next incident produced four pages</h3><p>A cache failure increased database load and API latency.</p><p>Four alerts fired:</p><ol><li>Checkout success rate breached its threshold.</li><li>Cache fallback entered degraded mode.</li><li>Database connection headroom approached exhaustion.</li><li>Automatic recovery failed to reduce impact.</li></ol><p>The first alert opened the incident. The others changed decisions.</p><p>Responders found the dependency chain quickly because the system preserved causally useful signals instead of sending every local complaint to a phone.</p><p>Nothing important had become less observable.</p><p>We had separated observation from interruption.</p><p>The alert count fell by 70%. Trust in the remaining 30% rose enough that the on-call engineer moved on the first page rather than waiting for the flood.</p><p>An alert is not proof that monitoring works.</p><p>It is a request for a human decision. If no decision follows, the page is spending attention without buying recovery.</p><p>Practice choosing the next action while multiple healthy-looking components argue about which one is responsible.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443d993773b4" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-deleted-70-of-our-alerts-the-next-incident-was-easier-to-find-443d993773b4">We Deleted 70% of Our Alerts. The Next Incident Was Easier to Find.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL vs CockroachDB: Surviving a Region Failure Added Latency to Every Healthy Day.]]></title>
            <link>https://blog.stackademic.com/postgresql-vs-cockroachdb-surviving-a-region-failure-added-latency-to-every-healthy-day-4cc1a2f4aaeb?source=rss-9430f59ed0df------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cc1a2f4aaeb</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Build & Learn]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 23:59:51 GMT</pubDate>
            <atom:updated>2026-08-28T23:59:51.314Z</atom:updated>
            <content:encoded><![CDATA[<h4>Distributed SQL removed one recovery fear by placing the speed-of-light bill on every write.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cPb2R4Q6jhRyUUfB5ohmKA.png" /></figure><p>The architecture requirement sounded reasonable:</p><p>“The database must survive losing an entire region.”</p><p>Nobody objects to surviving.</p><p>Then the first cross-region transaction arrived slower than the product team’s latency budget, and the requirement gained a price.</p><p>PostgreSQL versus CockroachDB is often framed as traditional SQL versus distributed SQL.</p><p>The more useful comparison is this:</p><p><strong>When do you want to pay for failure — during every healthy write, or during recovery?</strong></p><h3>PostgreSQL keeps the common path short</h3><p>A conventional PostgreSQL primary in one region can make writes fast and predictable for nearby application servers.</p><p>Add replicas for reads. Stream changes to a standby. Take backups. Test failover. Keep the application close to the primary.</p><p>The architecture is understandable because one node owns the write truth at a time.</p><p>Region failure complicates that truth.</p><p>Asynchronous replication can lose transactions that had not reached the standby. Synchronous replication can reduce data loss but adds network latency and availability trade-offs. Promoting a remote replica requires detecting failure, fencing the old primary, updating routing, and proving clients are not writing to two histories.</p><p>Postgres can support serious disaster-recovery designs.</p><p>The operational procedure remains part of the system.</p><p>You do not get region survival by drawing another replica.</p><p>You get it by rehearsing promotion and preventing split brain.</p><h3>CockroachDB charges consensus on the write path</h3><p>CockroachDB distributes and replicates data across nodes while presenting a SQL interface with serializable transactions by default.</p><p>Multi-region settings let teams describe survival goals and data locality. A database can be configured to survive an availability-zone failure or an entire region failure. Tables and rows can be placed according to access patterns.</p><p>This turns several manual recovery concerns into normal database behavior.</p><p>It cannot turn geography into a local call.</p><p>To survive losing a region while preserving strong consistency, replicas in multiple regions must agree on writes. A quorum requires a remote response. The nearest required replica places a floor under write latency.</p><p>Cockroach Labs describes the trade plainly: region survival increases write latency by at least the round trip to another region.</p><p>That is not an implementation defect.</p><p>It is the cost of knowing an acknowledged write will survive the failure you designed for.</p><h3>Postgres compatibility is not Postgres behavior</h3><p>CockroachDB speaks the PostgreSQL wire protocol and works with many Postgres drivers and tools.</p><p>That can make application adoption feel familiar.</p><p>The transactional behavior still deserves attention.</p><p>CockroachDB uses serializable isolation by default. Under contention, transactions may need retries. Client libraries and retry wrappers help, but application code must be safe to execute again.</p><p>If a transaction sends an email, charges an external provider, or publishes a message before commit, retrying it can duplicate side effects.</p><p>This is the same boundary distributed systems keep exposing:</p><p>The database can retry database work.</p><p>It cannot safely retry the outside world without application cooperation.</p><p>“Postgres-compatible” reduces interface migration.</p><p>It does not remove distributed transaction semantics.</p><h3>Model the tables, not only the regions</h3><p>Not every row needs the same geographic behavior.</p><p>User profiles might be read globally and changed rarely. Inventory might need a regional home. Payments might require strict consistency. Analytics can tolerate staleness. Session data may belong somewhere else entirely.</p><p>A good CockroachDB design uses table locality and survival goals deliberately rather than making every write global by default.</p><p>A good PostgreSQL design avoids pretending one primary can be globally local. It may route users to a home region, use read replicas for suitable data, separate globally replicated configuration, or accept a documented recovery window.</p><p>Distributed-system failures are rarely total. They are partial, delayed, and contradictory: one region is reachable from some clients, replication is behind, retries are multiplying, and two components disagree about whether the operation succeeded.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/distributed-systems-under-failure"><strong>Recognize how distributed systems break, recover, and scale under partial failure →</strong></a></p><p>The question is not “Is multi-region better?”</p><p>It is:</p><p>→ Which data must remain writable during region loss?</p><p>→ Which acknowledged writes may be lost?</p><p>→ Which operations can tolerate cross-region latency?</p><p>→ Which data can be stale?</p><p>→ Who performs recovery, and how often is it rehearsed?</p><p>Without those answers, multi-region is an expensive adjective.</p><h3>Test the day after the region returns</h3><p>Architecture reviews focus on the moment a region fails.</p><p>Recovery has a second half.</p><p>What happens when the region comes back?</p><p>With PostgreSQL, can the old primary rejoin safely, or must it be rebuilt? How is divergent state prevented? When does traffic return? Who verifies replication integrity?</p><p>With CockroachDB, how long does rebalancing take? What network and disk load does it create? Does the returning capacity improve latency immediately? Are application retries still elevated while ranges recover?</p><p>Surviving the failure is not the same as returning to normal.</p><p>The right database is the one whose entire failure lifecycle matches the business requirement and the team’s operating ability.</p><p>PostgreSQL often wins when low-latency single-region writes, ecosystem depth, and explicit disaster recovery are the right trade.</p><p>CockroachDB becomes compelling when automatic distributed consistency and region-level survival justify permanent coordination cost.</p><p>Resilience is not free uptime.</p><p>It is a bill you choose when to pay.</p><p>System design practice should make that bill visible. Place the replicas, define the quorum, trace one write, then remove a region and explain what the user experiences.</p><p><a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cc1a2f4aaeb" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/postgresql-vs-cockroachdb-surviving-a-region-failure-added-latency-to-every-healthy-day-4cc1a2f4aaeb">PostgreSQL vs CockroachDB: Surviving a Region Failure Added Latency to Every Healthy Day.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>