<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Coding Interview Secrets on Medium]]></title>
        <description><![CDATA[Stories by Coding Interview Secrets on Medium]]></description>
        <link>https://medium.com/@CodingInterviewSecrets?source=rss-6db3a3ab1911------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WPAkLn7eXCeUB2m_vU_PEg.png</url>
            <title>Stories by Coding Interview Secrets on Medium</title>
            <link>https://medium.com/@CodingInterviewSecrets?source=rss-6db3a3ab1911------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:21:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@CodingInterviewSecrets/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Config Sync Emptied Production Credentials. The Variable Names Still Looked Fine.]]></title>
            <link>https://aws.plainenglish.io/a-config-sync-emptied-production-credentials-the-variable-names-still-looked-fine-20a8cd6657e4?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/20a8cd6657e4</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 07:58:22 GMT</pubDate>
            <atom:updated>2026-10-26T07:58:22.497Z</atom:updated>
            <content:encoded><![CDATA[<h4>The dangerous assumption was not a missing secret. It was treating an unreadable value as an empty one.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Wv-vd4N4dcWSfr-zVwkNw.png" /></figure><p>The credential has a name. It appears in the dashboard. Nobody removed the variable.</p><p>The application still cannot authenticate.</p><p>That is a particularly unpleasant failure because the first check seems to clear the configuration. The thing you are looking for is visibly there.</p><p>On October 6, Trigger.dev’s Vercel integration overwrote some environment-variable values with empty strings. Its report identifies affected variables across 121 customer organizations. The before-build synchronization path had treated the empty representation of an unreadable Sensitive variable as a value to store. Newly enabled support for intentional empty strings removed behavior that had previously masked the bug. <a href="https://trigger.dev/blog/incident-report-oct-6-2026">Trigger.dev published the mechanism and recovery account.</a></p><p>The lesson reaches far beyond one integration.</p><p>“I cannot read this value” and “the value is empty” are different facts. Put them in the same string and eventually something helpful will synchronize them.</p><h3>The interface told a smaller truth than the application assumed</h3><p>At the boundary between two systems, a response is not always a full description of the underlying resource.</p><p>It may be filtered by permissions, redacted for security, summarized for display, or shaped for a particular operation.</p><p>A field can exist without being usable as input to a write somewhere else.</p><p>That sounds obvious when written out. The trouble is that synchronization code often begins with a pleasant symmetry: list things here, store them there.</p><p>The types can even appear to agree. A source field is a string. A destination field accepts a string. Serialization works. Validation passes.</p><p>The missing information is not a type mismatch at the language level. It is a meaning mismatch at the system boundary.</p><p>Think about an address book that returns a blank phone number because the caller lacks permission to see it. A sync process that replaces a populated destination with that blank value has converted limited visibility into deletion.</p><p>The objects fit. The operation is wrong.</p><p>That is why import code deserves the same scrutiny as a public write API. It is a write API with a friendly name.</p><h3>A valid empty value makes the problem harder</h3><p>The tempting fix is “never accept an empty string.”</p><p>That is not a universal solution. Sometimes an empty value is intentional. A configuration option may mean one thing when absent and something else when present but blank.</p><p>The useful fix is to represent the states you actually have.</p><p>For example, a sync design might distinguish:</p><ul><li>A readable value, including an explicitly intentional blank.</li><li>A value that exists but is not readable to this integration.</li><li>A resource that is absent.</li><li>A deliberate request to clear a stored value.</li></ul><p>Those states should not fall through to the same assignment operation.</p><p>A destination can preserve a current value when the source is unreadable. It can refuse destructive transitions that lack an explicit reason. It can require a clear operation rather than infer one from a placeholder.</p><p>Exactly which behavior is correct depends on the contract. The important part is making that contract visible before a generic loop writes the result.</p><p>An explicit clear action may require more code than copying strings. It also gives the operator a chance to understand what is about to happen.</p><h3>The recovery process is another writer</h3><p>Once a bad synchronization has changed state, stopping the code does not necessarily restore that state.</p><p>Trigger.dev’s report describes restoring encrypted values while avoiding overwriting changes customers had made after the incident. That distinction matters: an old value is not automatically the correct current value.</p><p>Now consider the general recovery problem.</p><p>A user sees a failure and repairs a credential. An incident responder restores a backup. If the restore blindly writes the previous value, the recovery can undo the customer’s repair.</p><p>This is a concurrency problem with human participants.</p><p>“Restore the backup” sounds like a command. In a live system, it is a proposal to write old knowledge into current state.</p><p>You need a condition under which that write is still valid. Often that means checking the destination has not changed since the bad update, and performing the check and replacement as one protected operation.</p><p>The exact control varies by storage system. The principle is stable: recovery must account for legitimate work performed while the system was broken.</p><p>Otherwise the second incident can be the first incident’s cleanup.</p><h3>The useful test is not just “does import succeed?”</h3><p>For a configuration integration, a successful import is a weak headline metric.</p><p>You need tests around the transitions with the most expensive consequences.</p><p>What happens when the source becomes unreadable? When a populated value becomes intentionally blank? When an integration loses permissions? When the destination has changed since the source snapshot was taken?</p><p>The fixture should preserve the distinction between a redacted response and real data. Otherwise the test suite will keep confirming the simplified world the bug depends on.</p><p>It is also worth testing the whole route by which a value reaches storage. Two entry points that supposedly perform the same sync can have different filters, defaults, or permissions.</p><p>If this failure shape feels uncomfortably familiar, the incident-pattern material in <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library">The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</a> extends the same work: connect a misleading symptom to confirming evidence, then choose recovery that does not create another problem. That is more useful during an incident than recognizing only the product name in the error message.</p><p>After the tests, examine the operator’s view. A dashboard can accurately show a variable’s existence while doing a poor job of showing its usable state.</p><p>Presence is not validity. An interface that makes them look equivalent is adding diagnostic friction.</p><h3>Synchronization is a policy decision disguised as plumbing</h3><p>The word “sync” makes a feature sound almost neutral.</p><p>Bring two sides into agreement. Remove manual work. Keep things current.</p><p>But agreement requires decisions about authority, missing information, deletion, concurrent changes, and recovery. There is no neutral answer to those questions.</p><p>A process that copies everything has chosen a policy. It has simply hidden that policy in a loop.</p><p>Before approving the next sync integration, ask what information the source is allowed to withhold and what destructive changes the destination is allowed to infer.</p><p>Then ask how an operator would distinguish those states without exposing the secret itself.</p><p>These are not edge cases reserved for credential systems. The same questions apply to permissions, inventories, customer profiles, and infrastructure configuration.</p><p>The dangerous field does not always contain an unexpected value.</p><p>Sometimes the problem is that it contains a perfectly ordinary empty string, and your code has decided that ordinary means authoritative.</p><p>Recognizing that boundary under pressure takes practice. Froquiz’s Senior Dev Challenge uses production-style scenarios to exercise the next-action judgment; Premium opens the complete challenge library.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20a8cd6657e4" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/a-config-sync-emptied-production-credentials-the-variable-names-still-looked-fine-20a8cd6657e4">A Config Sync Emptied Production Credentials. The Variable Names Still Looked Fine.</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Automated the Runbook. It Made the Wrong Decision Faster.]]></title>
            <link>https://blog.stackademic.com/we-automated-the-runbook-it-made-the-wrong-decision-faster-90ff988da90b?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/90ff988da90b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 09:19:50 GMT</pubDate>
            <atom:updated>2026-09-10T09:19:50.255Z</atom:updated>
            <content:encoded><![CDATA[<h4>The script removed hesitation, context, and the only question that could have prevented the outage.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WHNW4hbe51cuAnQXxnmDrw.png" /></figure><p>The runbook took twelve minutes by hand.</p><p>The automation took nineteen seconds.</p><p>It detected rising memory, drained the instance, restarted the process, verified the port, and returned it to service. The script had executed successfully dozens of times. Operators trusted it because it removed repetitive work and shortened recovery.</p><p>At 16:44, memory rose for a different reason.</p><p>Nineteen seconds later, the automation restarted every healthy instance into the same corrupted state.</p><p>We had not automated recovery.</p><p>We had automated one historical diagnosis.</p><h3>The runbook contained a hidden assumption</h3><p>The manual procedure began after an engineer had already decided that a known memory leak was present. That decision never appeared as a numbered step.</p><p>Humans checked the shape of the graph, recent deployments, heap growth, and whether memory fell after traffic moved away. Only then did they use the restart sequence.</p><p>When the steps became code, the trigger was simplified to “memory above threshold for five minutes.” The automation captured the visible actions and discarded the judgment that made them safe.</p><p>This is a common failure in operational automation. The document says what to do. The experienced responder supplies when it applies.</p><h3>A correct restart amplified corrupted configuration</h3><p>The new incident began after a configuration distributor published a malformed but valid value. Running processes still held the previous configuration, so only newly started instances were broken.</p><p>Memory rose because broken instances entered a retry loop. The automation saw high memory and restarted more instances. Each restart loaded the bad value.</p><p>Health checks passed briefly before the retry loop accelerated. The script returned those instances to traffic, observed memory rise again, and repeated the cycle.</p><p>The automation converted a partial deployment into a fleet-wide outage.</p><p>Every individual action was correct for the failure it had been written to solve.</p><h3>The kill switch depended on the same automation</h3><p>Operators attempted to disable the remediation policy. The control-plane change entered the same overloaded work queue as the restart actions already in progress.</p><p>The stop request waited behind the thing it needed to stop.</p><p>Even the alert stream became misleading. Each restart briefly reduced memory, so the automation recorded a successful remediation before the replacement process loaded the bad state and failed again. The system produced a sequence of green confirmations while fleet capacity disappeared.</p><p>An engineer finally revoked the automation role, halted configuration propagation, and isolated a small set of processes that still held the last-known-good value. Those instances became the recovery seed.</p><p>The event changed our definition of a safe automation. It needed more than a success path:</p><p>→ a narrow scope per execution</p><p>→ a maximum action budget</p><p>→ evidence that distinguishes the intended failure</p><p>→ a stop mechanism outside its own control path</p><p>→ a postcondition measured at the customer boundary</p><p>Automation removes time from response, including the time humans use to notice that today’s failure is not yesterday’s.</p><p>Recognizing that difference before acting is what prevents a recovery tool from becoming the incident. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> organizes recurring production failures around their misleading symptoms, confirming evidence, safe recovery moves, and prevention controls — the layer a sequence of commands cannot supply by itself.</p><h3>We automated bounded decisions</h3><p>The replacement did not stop automating restarts. It made the decision explicit.</p><p>The system compared memory shape with allocation signals, deployment and configuration versions, retry volume, and the response from one canary restart. It acted on one instance, then waited for a measurable result.</p><p>If the canary returned with a different configuration version or customer errors increased, the automation stopped. If more than a fixed percentage of capacity had been touched, it required human approval. If its control plane was unhealthy, it failed closed.</p><p>Every action emitted the evidence that triggered it and the condition expected afterward. Operators could understand not only what the script did but what it believed.</p><p>Recommendations received the same treatment. A tool that merely suggests an action can still anchor the responder on the wrong diagnosis. The interface therefore showed disconfirming evidence and the conditions under which the recommendation should be rejected, not just a confident next button.</p><p>We also separated detection from authority. A detector could recommend a restart without automatically receiving permission to restart the entire fleet.</p><h3>Speed multiplies whatever came before it</h3><p>Automation is often described as the removal of human error. More accurately, it moves human decisions earlier and gives them repetition.</p><p>A good decision becomes fast, consistent recovery.</p><p>A missing assumption becomes a fleet-wide mistake completed before the incident channel opens.</p><p>The runbook was not wrong. We were wrong to believe its numbered steps contained all the intelligence required to use it.</p><p>Before automating an operational action, ask the question the senior responder answers silently: what evidence would make me refuse to run this today?</p><p>That refusal is part of the system.</p><p>Production judgment is not memorizing the remediation. It is recognizing when the familiar remediation is dangerous. Froquiz gives you scenarios where the next action depends on reading the evidence, not matching one alert to one command.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90ff988da90b" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-automated-the-runbook-it-made-the-wrong-decision-faster-90ff988da90b">We Automated the Runbook. It Made the Wrong Decision Faster.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Queue Drained to Zero. Customer Orders Were Still Missing.]]></title>
            <link>https://blog.stackademic.com/the-queue-drained-to-zero-customer-orders-were-still-missing-648281154361?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/648281154361</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:03:35 GMT</pubDate>
            <atom:updated>2026-09-03T17:03:35.312Z</atom:updated>
            <content:encoded><![CDATA[<h4>Backlog recovery looked complete because the consumer acknowledged messages before proving the business operation existed.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyH0aq8U1FYvOhKyBKBd6Q.png" /></figure><p>The order queue peaked at 280,000 messages during a database slowdown.</p><p>After the database recovered, workers accelerated. Queue depth fell steadily. At 18:42, the dashboard reached zero and the incident commander announced that processing had caught up.</p><p>The next morning, merchants reported missing orders from the outage window.</p><p>The queue was empty because every message had been acknowledged. Some had never produced an order.</p><h3>Empty measured transport, not completion</h3><p>The worker fetched an event, called the order service, logged failures, and acknowledged the message inside a finally block to avoid poison-message loops.</p><p>During database timeouts, the call failed before an order was committed. The log contained the error, but the message disappeared from the queue.</p><p>Queue depth answered how much work remained available to consumers. It could not answer whether the intended business result existed.</p><p>The team had optimized for a clean queue while silently converting transient failures into permanent loss.</p><p>The error logger created false reassurance. It emitted a line for each failed order, but sampling reduced repeated database timeout messages after the first few hundred. Operators saw the error rate decline while the worker continued losing messages at a similar rate.</p><p>Logs described attempts. A durable unresolved-intent counter would have described customer risk. The incident lacked that counter until reconciliation built it afterward.</p><h3>Faster consumers made the evidence disappear faster</h3><p>When the database recovered, operators increased worker concurrency to drain the backlog.</p><p>That helped valid messages complete, but it also accelerated acknowledgement of failures during brief connection-pool saturation. The graph improved faster as the loss rate increased.</p><p>Worker success counted process completion, not order creation. Logs sampled repeated errors to control volume. The incident dashboard showed high throughput and low backlog while the durable outcome count lagged behind.</p><p>A performance response had amplified a correctness bug.</p><p>Autoscaling joined the feedback loop. Falling queue age caused more workers to launch, which exhausted database connections and created additional failures. Because failed messages were acknowledged, queue depth still fell quickly enough to look like progress.</p><p>We reduced concurrency before replay and reserved database capacity for interactive traffic. Catch-up speed was bounded by the slowest durable dependency, not by how many consumers the queue could feed.</p><h3>We reconciled intent against durable outcomes</h3><p>The source events still existed in the upstream event log, although the delivery queue had discarded them. We compared unique purchase intent IDs with committed order IDs for the affected time range.</p><p>The difference produced a bounded repair set.</p><p>Before replay, the team verified payment and inventory side effects. Some intents had charged successfully even though order creation failed. Replaying the entire handler could duplicate payment or reserve stock twice.</p><p>The repair path resumed from the first missing durable step per intent. It did not pretend the original message was untouched.</p><p>For intents with no external side effects, replay was straightforward. For charged purchases, repair created the missing order using the existing payment reference. For expired inventory reservations, the workflow required customer contact rather than inventing stock.</p><p>Each disposition was recorded so a second repair run could recognize completion. The repair tool was idempotent at the business-operation level, not just at message delivery.</p><h3>Acknowledgement moved behind the business commit</h3><p>The worker acknowledged only after the durable order transition succeeded or an explicitly terminal disposition was recorded.</p><p>Transient failures returned the message with bounded backoff. Unsupported or malformed events moved to quarantine with ownership and an alert. Idempotency keys protected order creation and external side effects across retries.</p><p>Outcome metrics compared received intents, committed orders, terminal rejections, and unresolved work. Those totals had to reconcile within a defined delay.</p><p>“Queue empty” remained useful, but it could no longer close the incident by itself.</p><p>We added a conservation check: accepted intents had to equal completed orders, explicit rejections, and currently unresolved work. The categories could lag, but they could not vanish.</p><p>This equation caught both message loss and duplicate outcomes. If completed orders exceeded accepted intents, retries were producing more business actions than customers requested.</p><p>Daily reconciliation ran outside the consumer stack. That independence mattered: a bug capable of acknowledging failed work should not also own the only metric claiming the work succeeded. The checker read the customer-intent ledger and business database directly, then opened an incident when the unexplained difference exceeded age or count limits.</p><p>Transport health and business completion finally became separate, comparable signals.</p><p>Healthy infrastructure signals can conceal missing business outcomes. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> develops the response instinct these incidents demand: identify what the graph actually measures, locate the durable boundary, reconstruct the affected set, and recover each intent without repeating irreversible side effects.</p><h3>We tested the gap between work and acknowledgement</h3><p>Failure tests killed the worker after every important step: before database commit, after commit but before acknowledgement, after payment, and during response handling.</p><p>Each interruption had an expected result. Either the message remained available, or a retry found the existing idempotent outcome and completed safely.</p><p>Batch consumers received separate tests because one failed item must not cause successful siblings to disappear or replay incorrectly. Partial acknowledgements were verified against real integration behavior.</p><p>The team also alerted on oldest unresolved business intent, not merely oldest queue message. That metric survived redelivery, queue migration, and consumer restarts.</p><h3>Zero can be the most misleading number</h3><p>Backlog graphs feel satisfying because they move visibly toward completion.</p><p>But a queue is a transport mechanism, not the customer ledger. It can be perfectly empty after successful work, deliberate rejection, accidental deletion, or premature acknowledgement.</p><p>Recovery was complete only when every accepted customer intent had one explainable terminal state.</p><p>Froquiz’s senior scenarios practice that decision: retry, quarantine, reconcile, reduce concurrency, or stop consumers when the queue is draining and the business outcome count is not keeping up.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=648281154361" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-queue-drained-to-zero-customer-orders-were-still-missing-648281154361">The Queue Drained to Zero. Customer Orders Were Still Missing.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Feature Flag Had Three Owners. Production Had None.]]></title>
            <link>https://medium.com/@CodingInterviewSecrets/one-feature-flag-had-three-owners-production-had-none-fed6e59e2774?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/fed6e59e2774</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:14:28 GMT</pubDate>
            <atom:updated>2026-09-01T22:14:28.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>Product owned the rollout, engineering owned the code, and nobody owned the stale configuration that kept executing for months.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DdeHLlumntYvlzpIH9R3Tw.png" /></figure><p>The feature had been cancelled in March.</p><p>Product marked the experiment complete. Engineering removed the visible UI. The platform team kept the flag service healthy. Everyone believed the change was gone.</p><p>In August, a dependency outage activated an old fallback branch still guarded by that flag. Five percent of requests entered code nobody had monitored, tested, or discussed for months.</p><p>The flag had an owner in three different systems. Production had no person responsible for its full lifecycle.</p><h3>Off did not mean deleted</h3><p>The flag evaluated false for normal customer traffic. That hid the code path during tests and ordinary monitoring.</p><p>A fallback configuration used a different default when the flag service was unavailable. During the dependency incident, evaluation changed from stored false to SDK default true.</p><p>The application did not crash. It routed traffic into an obsolete implementation that wrote a different record shape and emitted an event no current consumer expected.</p><p>The feature was “off” only while every part of flag delivery worked perfectly.</p><p>Different SDKs made the behavior inconsistent. Web instances cached the last known value, one worker defaulted false, and another older client used the value embedded in code. During the provider outage, the fleet did not select one fallback behavior; it split into three.</p><p>That explained why only a fraction of requests entered the dead path. Aggregate metrics made the failure look random until traces included flag source — remote, cached, or default — alongside the evaluated value.</p><h3>Ownership ended at different boundaries</h3><p>Product owned whether customers should see the feature. The application team owned the guarded code. Platform owned the flag provider.</p><p>Nobody owned removal after the decision.</p><p>The product ticket closed when the experiment ended. The engineering ticket closed when the UI disappeared. Platform correctly operated the flag service but could not judge whether a particular flag was safe to default true.</p><p>Three local definitions of done created one globally abandoned control.</p><p>The flag name did not reveal the consequences either. It described the original experiment, not the payment event branch it later controlled. Search found direct evaluations but missed a wrapper library and a configuration template copied into two workers.</p><p>We treated discovery like dependency mapping: provider inventory, repository search, runtime evaluation telemetry, scheduled jobs, and infrastructure configuration. Static ownership alone could not reveal every active path.</p><h3>Containment required more than switching it off</h3><p>Operators first forced the remote value to false. That had no effect on instances that could not reach the flag service — the exact condition activating the dangerous default.</p><p>We deployed an application-level override, disabled the obsolete event producer, and quarantined records written by the old branch. Consumers continued processing known schemas while the affected records waited for repair.</p><p>The team then located every evaluation site. The same flag controlled a scheduled job and a background reconciliation path that did not appear in the customer request trace.</p><p>A feature flag can become distributed control flow. Containment must find every branch, not only the screen where the feature was once visible.</p><p>Records emitted by the obsolete path were not discarded. We preserved them with producer version and flag provenance, upgraded the consumer to classify them, and replayed only after the resulting business state was reviewed.</p><p>That kept the immediate availability fix — stop new bad events — separate from the correctness decision about events already accepted.</p><h3>Flags gained an expiration contract</h3><p>New flags required a decision date, removal owner, failure default, and list of evaluation sites. Temporary flags failed CI after their expiration unless extended with a reason.</p><p>When an experiment ended, a cleanup change replaced the conditional with the chosen behavior and removed the unused path. Completion meant the flag no longer existed in code, configuration, scheduled jobs, dashboards, or the provider.</p><p>Failure-mode tests disconnected the flag service and verified the default per workflow. High-risk operations used local safe defaults rather than assuming remote configuration would always answer.</p><p>Recurring incidents often begin inside controls that everyone believes are temporary or harmless. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> helps engineers recognize the hidden production mechanism, bound the active paths, recover without spreading incompatible state, and convert ambiguous ownership into a prevention control that can be verified.</p><h3>We monitored behavior, not flag values</h3><p>The old dashboard showed evaluation counts by true and false. It proved what the SDK returned, not what the application did afterward.</p><p>We added outcome metrics for guarded paths: records written, events emitted, dependency calls, latency, and failures by code-path version. An unexpected branch could not hide behind a healthy flag service.</p><p>Alerts also caught stale flags with no recent decision and code paths receiving traffic after their planned retirement. The inventory linked each flag to its repository and cleanup pull request.</p><p>The platform still provided reliable flag infrastructure. Application owners now owned the production meaning of every default and transition.</p><p>We tested flag removal as carefully as flag rollout. Deleting the remote key before every client stopped referencing it could activate inconsistent defaults again. Cleanup order became: ship code without the branch, observe zero evaluations across the maximum fleet age, remove configuration, and finally delete provider metadata.</p><p>Only then was the experiment truly finished.</p><h3>Temporary control flow becomes permanent debt quickly</h3><p>Feature flags make release safer by separating deployment from exposure. They become dangerous when the organization separates exposure from deletion forever.</p><p>The incident was not caused by using flags. It was caused by allowing a temporary decision mechanism to outlive the people, tests, and observability that made it safe.</p><p>The strongest flag policy is not a naming convention. It is a lifecycle that ends with less code.</p><p>Froquiz’s senior scenarios practice that decision: force a value, deploy an override, quarantine output, disable a branch, or remove the flag when remote configuration and local defaults disagree during an outage.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fed6e59e2774" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Deleted One Customer. The Foreign Key Deleted Two Million Rows With It.]]></title>
            <link>https://blog.stackademic.com/we-deleted-one-customer-the-foreign-key-deleted-two-million-rows-with-it-017d92bdf45f?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/017d92bdf45f</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 21:58:05 GMT</pubDate>
            <atom:updated>2026-09-03T11:24:31.486Z</atom:updated>
            <content:encoded><![CDATA[<h4>`ON DELETE CASCADE` preserved referential integrity and destroyed the evidence needed to undo the operation.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNofca2aFC7KezskAqlFqA.png" /></figure><p>The support request was ordinary: remove a duplicate test customer from production.</p><p>An internal tool displayed the account, required confirmation, and issued one delete by primary key. The database returned success quickly.</p><p>Background dashboards began falling minutes later.</p><p>The customer row was the root of a large graph: orders, invoices, audit events, integration mappings, exports, and historical usage. Foreign keys used cascading deletion to prevent orphans.</p><p>One valid statement removed more than two million related rows.</p><p>The database maintained consistency exactly as designed. It also erased the trail needed to reconstruct what had happened.</p><h3>Small SQL had a hidden blast radius</h3><p>Code review saw a precise predicate on one customer ID. The visible statement looked safer than a broad DELETE.</p><p>The actual operation included every cascade reachable from that row. Its size lived in schema relationships and production cardinality, not in the number of rows named by the first statement.</p><p>The internal tool did not estimate dependent records. The audit log recorded the root deletion but disappeared as part of a cascading child table.</p><p>The first investigation mistake was searching application code for a loop deleting millions of rows. PostgreSQL had performed the graph traversal inside referential actions.</p><h3>Integrity and recoverability were different properties</h3><p>ON DELETE CASCADE guaranteed no child row referenced a missing parent. It did not guarantee the business wanted every child erased immediately or that deletion could be reversed.</p><p>Some children were truly owned data. Others were financial history, security evidence, and integration records with separate retention obligations.</p><p>We had used database ownership semantics as a substitute for data-lifecycle policy.</p><p>The schema needed more than “can this row exist without the parent?” It needed “must this record be retained, anonymized, detached, archived, or deleted when the customer leaves?”</p><h3>Recovery started with stopping secondary cleanup</h3><p>Several asynchronous jobs reacted to missing customer data and began deleting external artifacts. We paused those workers first to keep the blast radius from crossing more systems.</p><p>Point-in-time recovery restored the database into an isolated environment before the delete. We extracted the affected customer graph, reconciled rows created after the recovery point, and reinserted data in dependency order under controlled constraints.</p><p>We did not replace the entire production database with the older copy. That would have reversed every unrelated customer transaction since the deletion.</p><p>External side effects — emails, object storage, and third-party mappings — required their own reconciliation because database restore could not undo them automatically.</p><h3>Destructive operations gained a preview</h3><p>The internal tool now calculated dependent counts by category before enabling deletion. Large or regulated graphs required a second approver and an asynchronous plan.</p><p>The plan produced a manifest of intended actions: delete, anonymize, retain, or detach. It wrote audit evidence outside the deletion graph. Execution used a stable operation ID and recorded progress so retries could not repeat completed side effects.</p><p>Soft deletion protected the immediate path where appropriate, but it was not the whole answer. Data still needed eventual lifecycle handling, uniqueness behavior, and query discipline. “Add deleted_at” without those controls merely hid records indefinitely.</p><p>Destructive incidents are expensive when a tiny command carries a database-sized consequence. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> builds the response pattern for that moment: stop downstream propagation, identify the full graph, recover only affected state, and put an observable approval boundary in front of irreversible work.</p><h3>We mapped cascades as production code</h3><p>Schema review generated a cascade graph and flagged paths into high-volume, financial, audit, or shared tables. Changes to referential actions received the same attention as application deletion logic.</p><p>Tests used production-like cardinality. Deleting a seed customer with three orders could not reveal a two-million-row operation. The test asserted both final integrity and preserved records by policy class.</p><p>We added a transaction-level row-change budget for administrative tools. If the planned operation exceeded the preview by a defined margin, execution stopped before commit and required a new approval. The budget was not a substitute for correct policy, but it caught surprising graph expansion.</p><p>Large deletions moved into small, resumable batches where business rules allowed it. That reduced lock duration, replication bursts, and recovery ambiguity. Each batch referenced the same operation manifest and could not wander into records outside the approved scope.</p><p>Backups were indexed by deletion operation ID so recovery could locate the right pre-change state quickly. The team still rehearsed selective restoration; possessing a backup was not enough if extracting one customer graph took days.</p><p>External systems received tombstones only after the database action reached the appropriate durable stage. A transaction rollback could no longer leave search indexes, object storage, and vendors permanently ahead of the source of truth.</p><p>Privacy deletion and accidental deletion also received different evidence handling. Required audit proof retained metadata without preserving the personal content the deletion was meant to remove.</p><p>Monitoring tracked rows affected, transaction duration, replication impact, and operation identity for destructive maintenance. An unexpectedly large cascade could pause before external cleanup made recovery harder.</p><h3>One row was never one business object</h3><p>The customer table stored identity. The customer relationship lived across years of state and multiple systems.</p><p>The final design kept referential integrity but stopped assuming every relationship implied immediate deletion. Ownership, retention, privacy, accounting, and recovery each received an explicit rule.</p><p>For high-blast-radius operations, the first run became a dry report: target identity, expected row counts by table, external objects affected, and estimated replication load. A second authorized step executed against that frozen scope. If reality differed materially, the operation stopped instead of discovering the graph through damage.</p><p>Deletion became a reviewed production change, not a routine button backed by an unexpectedly powerful foreign key.</p><p>Froquiz’s senior scenarios practice that decision: stop the transaction, pause downstream jobs, restore a subset, anonymize, or preserve evidence when one correct delete has already expanded beyond its visible target.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=017d92bdf45f" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-deleted-one-customer-the-foreign-key-deleted-two-million-rows-with-it-017d92bdf45f">We Deleted One Customer. The Foreign Key Deleted Two Million Rows With It.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Error Rate Was 0%. Customers Were Waiting 45 Minutes.]]></title>
            <link>https://blog.stackademic.com/our-error-rate-was-0-customers-were-waiting-45-minutes-73a189b1e3c5?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/73a189b1e3c5</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 15:07:27 GMT</pubDate>
            <atom:updated>2026-09-03T11:10:14.901Z</atom:updated>
            <content:encoded><![CDATA[<h4>Nothing failed because almost nothing finished.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UClIhgAYjKB5wUcxs2WNuA.png" /></figure><p>Support reported that invoice exports were stuck. The API dashboard showed zero errors. Workers were healthy. The queue consumer process had not restarted.</p><p>Every customer request had received 202 Accepted, so the request success rate was perfect.</p><p>The status page therefore remained green while support tickets accumulated.</p><p>Behind that response, one slow export occupied each worker. Thousands of jobs waited. The oldest customer had been staring at “Preparing your file” for 45 minutes.</p><p>We had measured acceptance as success and confused a healthy doorway with a working system.</p><h3>The HTTP response ended before the customer outcome</h3><p>The export endpoint created a job and returned immediately. From the API’s perspective, it succeeded in 30 milliseconds.</p><p>The actual customer operation continued through a queue, worker, database query, object storage upload, notification event, and download link.</p><p>No shared metric connected those stages.</p><p>The queue dashboard showed message count but not age. A stable depth looked harmless because jobs entered and left at similar rates during ordinary traffic. When long-running exports occupied all workers, depth grew slowly enough to avoid the alert.</p><p>The critical signal was oldest-job age:</p><pre>API error rate:       0%<br>queue depth:          2,840<br>active workers:       20/20<br>oldest waiting job:   45 minutes<br>completed exports:    near zero</pre><p>Every component was “up.” The business operation was unavailable.</p><h3>More workers would have moved the outage</h3><p>The first proposed fix was to triple worker count.</p><p>Each export executed a heavy database query. Sixty workers would have turned queue wait into database contention and slowed customer-facing traffic.</p><p>We tested two additional workers and watched connection acquisition time and replica lag rise immediately. The queue was protecting the database by making customers wait.</p><p>Removing that protection without changing the work would relocate the incident to a more important path.</p><p>We contained the problem by pausing unusually large exports, reserving capacity for normal jobs, and cancelling work whose customer had already abandoned or superseded it.</p><p>Then we inspected the job distribution. A handful of tenants had requested unbounded date ranges. Those jobs were not failing; they were monopolizing worker slots for more than twenty minutes each.</p><p>The queue used simple first-in, first-out scheduling. One giant job could delay hundreds of small ones.</p><p>We proved it by plotting service time by tenant and requested date range. The median job still finished in seconds. A tiny long tail consumed almost all worker-minutes. Average duration had hidden the exact population controlling capacity.</p><h3>Fairness mattered more than raw throughput</h3><p>We split exports into size classes with separate concurrency budgets. Small, predictable jobs retained capacity even when large reports accumulated.</p><p>Large exports became chunked and checkpointed. A worker processed a bounded range, persisted progress, released its database connection, and allowed other work to run before continuing.</p><p>Per-account concurrency prevented one tenant from occupying the entire fleet. Maximum date ranges and row estimates moved resource budgeting into the product boundary rather than leaving it to an invisible backend queue.</p><p>We also stopped retrying every worker exception immediately. Retries re-entered with the same priority and could starve new work. Transient failures received bounded backoff; permanent validation failures went directly to a visible terminal state.</p><p>The goal was not merely to empty the queue. It was to make completion time predictable across customers.</p><h3>Success became an end-to-end event</h3><p>We created one operation ID at request acceptance and carried it through every stage.</p><p>The customer-level outcome now records:</p><ul><li>time accepted;</li><li>time waiting;</li><li>time executing;</li><li>time uploading;</li><li>final success, failure, cancellation, or expiration;</li><li>whether the customer received and used the result.</li></ul><p>API success remains useful, but it is labeled as acceptance success. The primary service objective measures completed usable exports within the promised window.</p><p>The operation deadline follows the job through retries. A redelivery does not receive a fresh 30-minute promise. Once the customer deadline is gone, the worker stops starting expensive stages and moves the job to an explicit expired state.</p><p>We alert on oldest age, completion rate, and percentage of operations exceeding their deadline. A queue can now be failing while every process remains alive.</p><p>That model exposed other blind spots: emails accepted but never delivered, webhooks queued but hours late, and background payments awaiting a consumer while the API returned success.</p><p>Healthy asynchronous systems are full of intermediate successes. Customers buy final outcomes.</p><p>Most outages do not announce themselves with a clean exception. They hide in waiting rooms, retries, queues, and states nobody included in the top-line dashboard.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is built around recognizing those patterns — the difference between a green component and a working production path, and the operational decisions that restore outcomes without moving failure downstream.</p><h3>Zero errors is sometimes the worst clue</h3><p>An error tells you the system reached a decision. A growing wait can mean it has stopped deciding at all.</p><p>We added deadlines to jobs, but deadlines alone were not enough. Work that exceeded a deadline had to stop, release resources, and enter a state the customer could understand. Timing out while continuing in the background would preserve the same hidden load.</p><p>The export system now degrades deliberately. Very large requests become scheduled reports. Interactive exports have bounded scope. Customers see progress and a realistic completion expectation rather than an endlessly optimistic spinner.</p><p>The dashboard still shows low errors. Now it also shows whether anything useful is leaving the system.</p><p>Forty-five minutes of waiting taught us that availability is not “the request was accepted.” It is the system’s ability to complete the promise behind that response within a meaningful time.</p><p>When a graph says zero errors during obvious customer pain, do not celebrate. Ask which unfinished operations the denominator forgot.</p><p>That question appears constantly in senior production work. Froquiz gives you scenarios where the real failure is often hidden behind a technically successful response.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73a189b1e3c5" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/our-error-rate-was-0-customers-were-waiting-45-minutes-73a189b1e3c5">Our Error Rate Was 0%. Customers Were Waiting 45 Minutes.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Queue Recovered in Twelve Minutes. The Duplicate Side Effects Took Three Days.]]></title>
            <link>https://blog.stackademic.com/the-queue-recovered-in-twelve-minutes-the-duplicate-side-effects-took-three-days-0e8e2a3a3e17?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e8e2a3a3e17</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:15:30 GMT</pubDate>
            <atom:updated>2026-09-03T11:08:25.815Z</atom:updated>
            <content:encoded><![CDATA[<h4>We celebrated the backlog reaching zero. That was when the customer damage started becoming visible.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFEQTM1M6bGWYagwOSV58w.png" /></figure><p>At 14:08, queue depth began climbing. At 14:17, consumers recovered. By 14:29, the backlog was gone.</p><p>The incident channel declared the service stable.</p><p>The next morning, support found duplicate shipment emails. Then finance found repeated credits. Then one partner reported receiving the same webhook five times.</p><p>The queue had recovered exactly as designed: it retried work until every message was acknowledged.</p><p>The business operations behind those messages had not been designed for repetition.</p><h3>Delivery semantics end where side effects begin</h3><p>The consumer performed four actions for each order event:</p><ol><li>update internal state;</li><li>issue a credit through an external provider;</li><li>send a customer email;</li><li>publish a partner webhook.</li></ol><p>Only the first action participated in our database transaction.</p><p>During the outage, a consumer sometimes issued the credit and crashed before acknowledging the message. The broker correctly redelivered it. The next consumer had no durable evidence that the external call had already succeeded.</p><p>The team described the queue as “at least once.” We had treated the handler as “probably once.”</p><p>That gap is where duplicate side effects live.</p><p>The message ID did not save us because the deduplication record was written after the external operations. A crash in the wrong interval produced this sequence:</p><pre>provider accepts credit<br>process crashes<br>message remains unacknowledged<br>broker redelivers<br>provider accepts credit again</pre><p>No broker setting can make two separate systems commit atomically just because we want them to.</p><h3>Recovery throughput amplified the mistake</h3><p>Under normal traffic, duplicate opportunities were rare. During recovery, we increased consumer concurrency to drain the backlog quickly.</p><p>That changed three things at once:</p><ul><li>more messages were in flight;</li><li>external dependencies experienced a sudden burst;</li><li>timeouts and ambiguous responses increased.</li></ul><p>An ambiguous timeout is especially dangerous. The caller knows it did not receive a response. It does not know whether the provider completed the action.</p><p>Blind retry turns uncertainty into duplication.</p><p>Our recovery procedure optimized queue depth, not business correctness. The graph looked better every minute while the reconciliation workload grew elsewhere.</p><p>We stopped the consumers again — not because the queue was unhealthy, but because continuing would create damage faster than we could measure it.</p><p>The immediate containment was to disable the non-idempotent credit step, continue processing orders into a pending state, and reconcile provider records before resuming financial side effects.</p><h3>Idempotency needs a business key, not only a message ID</h3><p>We rebuilt the handler around durable operation identities.</p><p>A message can be republished with a new broker ID. Two upstream paths can express the same business intent. A replay can deliberately create new delivery metadata.</p><p>The stable key had to describe the operation: for example, credit:{order_id}:{credit_reason}:{version}.</p><p>Before calling the provider, the consumer created an operation record with a unique constraint. The provider request used the same key when its API supported idempotency. The final provider reference and outcome were stored against that record.</p><p>If the consumer saw the operation again, it did not simply return “already processed.” It inspected the state:</p><pre>not_started  -&gt; safe to execute<br>in_progress  -&gt; reconcile before retry<br>succeeded    -&gt; return stored result<br>failed_final -&gt; do not retry automatically</pre><p>The in_progress state was critical. It represented the uncertainty window after an outbound call but before a durable local result. Treating it as failure would repeat the original bug.</p><p>Emails and webhooks received their own operation keys. One deduplication flag around the whole handler was too coarse because some steps could succeed while others failed.</p><h3>A zero backlog is not a recovery criterion</h3><p>We changed the recovery dashboard to include business invariants.</p><p>Queue depth still mattered, but it was joined by:</p><ul><li>duplicate-operation conflicts;</li><li>side effects in an uncertain state;</li><li>provider requests per unique business operation;</li><li>oldest unreconciled operation;</li><li>first-attempt and retry outcomes;</li><li>downstream rate-limit and timeout responses.</li></ul><p>We also limited drain speed. Recovery concurrency became the minimum of what consumers, the database, and external providers could safely sustain. The fastest possible drain was not the safest possible recovery.</p><p>For replays, we added a dry-run mode that counted intended operations without executing them. A replay plan now answered: how many messages, how many unique business keys, which side effects are enabled, and how the run can be stopped.</p><p>The runbook gained an uncomfortable but useful checkpoint: before scaling consumers, identify which handler steps are irreversible.</p><p>We also required a small recovery sample before opening the throttle. Ten messages were processed, their external effects were reconciled, and only then did concurrency increase. It made recovery look slower for the first minute and prevented hours of invisible cleanup.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> develops this pattern-recognition layer across queues, APIs, databases, retries, and incident response — so recovery is judged by restored business correctness, not by the first graph that returns to green.</p><h3>The three-day cleanup was a design review in disguise</h3><p>Reconciliation required comparing message history, internal operation records, provider transactions, email events, and partner receipts. The system had never been built to answer “Did this exact business action happen once?”</p><p>That question now shapes new asynchronous features.</p><p>Every side effect needs an identity. Every ambiguous result needs a reconciliation path. Every replay needs a bounded blast radius. Every recovery dashboard needs at least one business-level invariant.</p><p>The broker did not duplicate our customers’ credits. It redelivered messages we had not acknowledged. Our handler converted a normal delivery guarantee into repeated external actions.</p><p>Twelve minutes was the infrastructure recovery time. Three days was the system recovery time.</p><p>Those numbers measured different realities, and we had been watching only the flattering one.</p><p>When a queue returns to zero, the right question is not “Are we done?” It is “What did we do more than once while getting here?” Practicing that question before an incident is much cheaper than answering it through customer tickets.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e8e2a3a3e17" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-queue-recovered-in-twelve-minutes-the-duplicate-side-effects-took-three-days-0e8e2a3a3e17">The Queue Recovered in Twelve Minutes. The Duplicate Side Effects Took Three Days.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Added a Second Region for Reliability. One Expired Certificate Took Down Both.]]></title>
            <link>https://blog.stackademic.com/we-added-a-second-region-for-reliability-one-expired-certificate-took-down-both-7dd4e058a8f2?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dd4e058a8f2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:30:23 GMT</pubDate>
            <atom:updated>2026-08-30T02:30:23.473Z</atom:updated>
            <content:encoded><![CDATA[<h4>We duplicated compute, databases, and traffic routing — then left trust anchored to one shared dependency.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DVaQP0ZfAQlwwPHk766UoQ.png" /></figure><p>The architecture review had two of everything.</p><p>Two regions. Two Kubernetes clusters. Two database replicas. Two ingress stacks. Independent capacity. Tested traffic failover.</p><p>At 02:14, both regions began rejecting internal requests.</p><p>The certificate used by the shared identity service had expired.</p><p>Region A could not authenticate to it.</p><p>Region B could not authenticate to it either.</p><p>We had built regional redundancy around a global single point of failure.</p><h3>Duplication made the diagram look resilient</h3><p>Multi-region architecture is visually persuasive.</p><p>Boxes appear in pairs. Arrows cross geography. A large traffic manager sits above everything. The failure story seems obvious: if one side disappears, the other side continues.</p><p>That story works for failures contained by geography.</p><p>Our certificate was distributed to both regions from the same issuance process, renewed by the same automation, observed by the same dashboard, and owned by the same team. Its expiration was not regional.</p><p>The identity endpoint remained reachable. TLS validation failed everywhere.</p><p>Traffic failover moved requests between two equally broken environments.</p><h3>Shared control planes create correlated failure</h3><p>We started listing dependencies that could affect both regions simultaneously.</p><p>The list grew quickly:</p><p>→ Identity and authorization.</p><p>→ DNS and certificate issuance.</p><p>→ Artifact registry.</p><p>→ Feature-flag service.</p><p>→ Secrets distribution.</p><p>→ Deployment control plane.</p><p>→ Observability and alert delivery.</p><p>→ The human approval path.</p><p>Some dependencies were shared intentionally. Duplicating everything would create unacceptable cost and consistency problems.</p><p>The mistake was not sharing them.</p><p>The mistake was counting regions without counting shared failure domains.</p><h3>Failover tests had started too low</h3><p>Our exercises disabled a cluster or shifted traffic away from a region.</p><p>They proved that the data plane could survive regional loss. They did not test a valid-looking dependency returning universally unusable responses.</p><p>We added scenarios above the region:</p><p>→ Authentication unavailable globally.</p><p>→ DNS returns a wrong but reachable destination.</p><p>→ A bad secret propagates successfully to every cluster.</p><p>→ The artifact registry is unavailable during recovery.</p><p>→ The traffic manager makes the same bad decision for both regions.</p><p>The certificate incident was especially useful because automation had worked perfectly for years. Reliability decayed on a calendar, outside ordinary traffic and release patterns.</p><p>Distributed failures repeat through shared roots hidden beneath duplicated branches. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> continues those patterns across infrastructure, architecture, incident recovery, and the decisions that determine whether redundancy is real.</p><h3>Independence is a property you must trace</h3><p>We changed certificate renewal to alert on remaining lifetime from each region, not merely on job success. Emergency trust material could be distributed through a separate path. Critical services cached a narrowly scoped verification capability long enough to degrade safely during short identity outages.</p><p>We also documented which dependencies were deliberately global and what product behavior remained possible without them.</p><h3>Two regions are not two systems</h3><p>The second region still improved reliability.</p><p>It protected us from infrastructure loss, regional networking trouble, and capacity failures. The incident did not prove multi-region was pointless.</p><p>It proved the reliability claim had been broader than the failure model.</p><p>Redundancy exists only across the boundary that can fail independently.</p><p>Two clusters using one identity dependency may provide compute redundancy. They do not provide authentication redundancy. Two databases restored by one unavailable control plane may protect data while still blocking recovery.</p><p>Do not count copies.</p><p>Trace shared authority, shared state, shared automation, and shared humans until the supposedly independent paths meet again.</p><p>That meeting point is where the next “impossible” multi-region outage is waiting.</p><p>System design practice should not end when the second region appears on the canvas. It should begin with the dependency capable of making both regions wrong together.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dd4e058a8f2" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-added-a-second-region-for-reliability-one-expired-certificate-took-down-both-7dd4e058a8f2">We Added a Second Region for Reliability. One Expired Certificate Took Down Both.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Cache Hit Rate Was 96%. The Missing 4% Took Down PostgreSQL.]]></title>
            <link>https://blog.stackademic.com/the-cache-hit-rate-was-96-the-missing-4-took-down-postgresql-975c02b3e0d1?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/975c02b3e0d1</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:11:06 GMT</pubDate>
            <atom:updated>2026-08-30T02:11:06.310Z</atom:updated>
            <content:encoded><![CDATA[<h4>A healthy average concealed one expensive key pattern, one synchronized expiry, and a database with no room for either.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lvrYJADgy2azhEVlk0Ynw.png" /></figure><p>The cache dashboard looked excellent.</p><p>Hit rate: 96.1%.</p><p>Redis latency: under a millisecond.</p><p>Evictions: negligible.</p><p>PostgreSQL CPU: 100%.</p><p>The API was timing out because the four percent of requests missing cache were not ordinary misses. They were the largest, most expensive queries in the product — and thousands of them had started at the same second.</p><p>The cache was healthy.</p><p>The system behind it was not.</p><h3>Hit rate treated every request as equal</h3><p>Our most common cache entries were tiny account settings and feature flags. They were read constantly, cheap to rebuild, and almost always present.</p><p>They dominated the numerator.</p><p>The expensive entries were customer dashboards assembled from several large tables. They represented less traffic but far more database work. One dashboard miss could cost more than ten thousand settings misses.</p><p>The aggregate hit rate mixed them together.</p><p>A request saved from a two-millisecond query counted exactly the same as a request saved from a six-second aggregation.</p><p>We had optimized a ratio instead of the load the ratio was supposed to explain.</p><h3>The misses arrived together</h3><p>The dashboard entries used a one-hour TTL.</p><p>They were populated after a scheduled customer import, which meant many keys were created within the same few minutes. One hour later, they expired within the same few minutes.</p><p>Traffic did the rest.</p><p>The first miss began rebuilding the dashboard. Before it completed, dozens of requests for the same customer also missed. Each started the same database work. Across many customers, synchronized expiry became a query storm.</p><p>Redis continued answering hits quickly for everything still cached.</p><p>PostgreSQL received the concentrated cost of everything that was not.</p><p>This is why cache stampedes rarely look like cache outages. The cache may be responsive and mostly successful while the protected dependency is being dismantled by the miss path.</p><h3>We measured avoided work, not only hits</h3><p>We split cache metrics by workload and added estimated rebuild cost.</p><p>The new view showed:</p><p>→ Hit rate by key family.</p><p>→ Database time avoided by hits.</p><p>→ Concurrent rebuilds for the same logical key.</p><p>→ Expiry distribution over time.</p><p>→ Miss latency and downstream query cost.</p><p>→ Cache population failures.</p><p>The global hit rate remained useful as a quick health signal. It stopped being the argument that the cache could not be involved.</p><p>Once misses carried cost, the missing four percent became the loudest part of the dashboard.</p><p>Caches fail through repeated shapes: stampedes, hot keys, stale ownership, oversized values, missing TTLs, and fallback paths more dangerous than failure. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those patterns continue beyond one attractive hit-rate number.</p><h3>The fix was coordination, not a longer TTL</h3><p>Increasing TTL would have delayed the next storm.</p><p>We added jitter so expirations spread across time. We used single-flight rebuilds so one request refreshed a key while others received slightly stale data or waited behind the same work. High-cost entries refreshed before expiry. Database queries received concurrency limits independent of API concurrency.</p><p>We also decided which data was allowed to become stale during recovery.</p><p>That decision mattered more than another Redis setting. A system can preserve availability during cache trouble only if the product has defined what degraded correctness looks like.</p><h3>Ninety-six percent was never the success condition</h3><p>After the changes, our cache hit rate fell slightly.</p><p>That surprised people.</p><p>We stopped caching some cheap data whose invalidation cost exceeded its value. Meanwhile, the expensive key families became safer and their rebuild work became bounded.</p><p>The headline metric got worse.</p><p>Database stability improved.</p><p>That is the difference between measuring the mechanism and measuring the outcome. A cache does not exist to produce hits. It exists to remove enough work from a dependency without creating new correctness and recovery problems.</p><p>The missing percentage is not always noise.</p><p>Sometimes it contains everything capable of taking production down.</p><p>Architecture practice gets useful when you must design the miss path, not merely draw the cache box. Decide what happens when the fast path disappears under real load.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=975c02b3e0d1" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-cache-hit-rate-was-96-the-missing-4-took-down-postgresql-975c02b3e0d1">The Cache Hit Rate Was 96%. The Missing 4% Took Down PostgreSQL.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon S3 vs Cloudflare R2 in 2026: Egress Is Free. Migration Still Isn’t.]]></title>
            <link>https://blog.stackademic.com/amazon-s3-vs-cloudflare-r2-in-2026-egress-is-free-migration-still-isnt-a062c7bfc328?source=rss-6db3a3ab1911------2</link>
            <guid isPermaLink="false">https://medium.com/p/a062c7bfc328</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Coding Interview Secrets]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 00:22:06 GMT</pubDate>
            <atom:updated>2026-08-29T00:22:06.880Z</atom:updated>
            <content:encoded><![CDATA[<h4>The storage bill dropped on paper. Cache behavior, API edges, and the rollback path carried the hidden cost.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sPlv4NPGsZHPRmLqoyNWMA.png" /></figure><p>“No egress fees” is the kind of sentence that can end an infrastructure meeting early.</p><p>The current bill already contains the villain. The alternative already contains the rescue. Copy the objects, change the endpoint, enjoy the graph moving down.</p><p>Cloudflare R2 makes that argument especially tempting. It is S3-compatible, strongly consistent at the bucket layer, and does not charge for direct egress.</p><p>So why not move everything?</p><p>Because data transfer is only one part of object storage behavior.</p><p>The migration has to preserve the application, not merely the objects.</p><h3>Compatibility is a surface, not an identity</h3><p>S3 compatibility can make existing SDKs and tools work with a different endpoint.</p><p>That is valuable. It can remove months from a migration.</p><p>It does not mean every S3-adjacent feature, IAM assumption, event integration, lifecycle behavior, signed URL edge case, or operational tool behaves identically.</p><p>Applications rarely use object storage as a naked PUT and GET API forever.</p><p>They add multipart uploads. Presigned URLs. Bucket policies. Replication. Inventory. Events. CDN behavior. Retention rules. Analytics. Data processing jobs. Cross-account access. Malware scanning. Backup controls.</p><p>The longer S3 has been present, the more likely the real dependency is the ecosystem around the bucket.</p><p>Moving the bytes can be the smallest part of leaving.</p><h3>R2’s strongest promise is genuinely strong</h3><p>Cloudflare currently charges for storage and operation classes while making direct egress free. For workloads that serve large amounts of media or user-generated content, that difference can dominate the bill.</p><p>R2 also documents strong global consistency for object writes, metadata updates, deletes, and listings.</p><p>That removes an old category of “maybe the object has not propagated yet” reasoning at the storage layer.</p><p>But the word <strong>storage</strong> matters.</p><p>Put Cloudflare Cache in front of a custom domain and the delivery path gains separate behavior. A deleted object can remain cached. A cached 404 can survive after the object is uploaded. An overwritten key can keep serving the old object until expiration or purge.</p><p>The bucket is consistent.</p><p>The user can still see stale content.</p><p>That distinction is exactly the kind production discovers after a clean migration report.</p><h3>Free egress changes application design</h3><p>When egress is expensive, teams optimize around avoiding it.</p><p>They colocate compute. Add a CDN. Compress aggressively. Create derived assets in-region. Keep processing close to storage. Think carefully before copying data to another provider.</p><p>When egress becomes free, new designs become economical.</p><p>That is good.</p><p>It can also make request volume and operation classes the next surprise. R2 bills Class A and Class B operations, with rounding rules that matter at boundaries. Infrequent Access adds retrieval charges even though egress remains free.</p><p>A workload with large objects and relatively few reads behaves differently from one with billions of tiny metadata requests.</p><p>“We transfer many terabytes” is not a complete workload description.</p><p>Neither is “we store many terabytes.”</p><p>You need object count, request mix, average size, cache hit rate, mutation frequency, retrieval behavior, and the systems that consume the data.</p><p>This is where apparently simple infrastructure choices become architecture decisions. The missing layer is recognizing the operational and reversal costs that sit outside the pricing headline.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/architecture-under-pressure"><strong>Recognize the 25 architecture decisions that make or break production systems →</strong></a></p><h3>The safe migration reads from two truths</h3><p>A serious migration avoids a single irreversible switch.</p><p>Start by inventorying behavior, not buckets:</p><p>→ Which APIs and SDK features are used?</p><p>→ Which services receive object-created events?</p><p>→ Which policies protect private data?</p><p>→ Which consumers depend on versioning or metadata?</p><p>→ Which URLs are cached outside the storage layer?</p><p>→ Which jobs assume AWS-local access?</p><p>Then design a transition.</p><p>Copy historical objects. Verify checksums and metadata. Dual-write new objects where the business operation can tolerate it. Read from the new store for a controlled cohort. Compare misses and latency. Exercise deletion and overwrite behavior through the actual CDN path.</p><p>Most importantly, define rollback.</p><p>If R2 becomes unavailable or one compatibility edge appears, can reads return to S3 without losing objects written during the migration? Is there a reverse replication path? Does the application know which store contains the newest version?</p><p>The migration is not safe because the copy command succeeded.</p><p>It is safe when either side can fail without making the data ambiguous.</p><h3>S3 and R2 optimize different anxieties</h3><p>S3 is difficult to beat when the application is deeply integrated with AWS, requires the breadth of its storage ecosystem, or values mature controls and service adjacency more than the egress line item.</p><p>R2 is compelling when Internet delivery is central, egress dominates cost, S3-compatible access covers the required surface, and Cloudflare’s network is already part of the delivery path.</p><p>The winner is not “the one without egress fees.”</p><p>It is the one whose surrounding behavior you can prove — and reverse.</p><p>Infrastructure cost is rarely hiding in one price cell. It hides in the architecture the price encourages.</p><p>Good system design practice forces object storage to participate in failure, cache invalidation, security, and migration. A bucket icon is not a design until you can explain what happens when the newest object exists in only one place.</p><p><a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a062c7bfc328" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/amazon-s3-vs-cloudflare-r2-in-2026-egress-is-free-migration-still-isnt-a062c7bfc328">Amazon S3 vs Cloudflare R2 in 2026: Egress Is Free. Migration Still Isn’t.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>