<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dev Side Hustle on Medium]]></title>
        <description><![CDATA[Stories by Dev Side Hustle on Medium]]></description>
        <link>https://medium.com/@DevSideHustle?source=rss-8891a9734cd8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZJlfGFzzK2nSK6s4-5Y9LQ.png</url>
            <title>Stories by Dev Side Hustle on Medium</title>
            <link>https://medium.com/@DevSideHustle?source=rss-8891a9734cd8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:32:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@DevSideHustle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless vs Containers: Who Finishes the Job After the Request Ends?]]></title>
            <link>https://aws.plainenglish.io/serverless-vs-containers-who-finishes-the-job-after-the-request-ends-a1c8d08ce4cc?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1c8d08ce4cc</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 08:03:42 GMT</pubDate>
            <atom:updated>2026-10-26T08:03:42.279Z</atom:updated>
            <content:encoded><![CDATA[<h4>The deployment choice is easier once you stop confusing a running process with a durable promise.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lb3_Wr-vkHjLYn5JysY0Gw.png" /></figure><p>The browser shows “Export started.”</p><p>The handler returns. A background operation is still working. Somewhere in the design, everyone has assumed that it will eventually finish.</p><p>That assumption exists in serverless applications. It exists in containers too.</p><p>Only one of those gets blamed automatically when the process disappears.</p><p>The serverless-versus-containers argument often begins with cold starts, memory, and price. Those are useful questions. For long-running work, I would begin somewhere less fashionable: <strong>where is the promise to finish recorded?</strong></p><p>If the answer is only “in the process that accepted the request,” your deployment choice has not yet solved the important problem.</p><h3>A container can lose a job just as confidently</h3><p>Take a report-generation feature as a design example.</p><p>The endpoint accepts filters, starts some work, and responds quickly. The work gathers records, creates a file, stores it, and notifies the user.</p><p>Putting the handler in a container removes some constraints associated with a short-lived function invocation. It does not make the process immortal.</p><p>A deploy can replace it. A machine can fail. An operator can restart it. The process can run out of memory between creating the file and recording that the file exists.</p><p>The job may be perfectly recoverable, but only if the application has retained enough state to recognize what happened and continue safely.</p><p>“The container usually stays up” describes an observation about uptime. It is not a recovery protocol.</p><p>On the other side, serverless does not mean every useful operation must fit into one synchronous web request. Platforms provide asynchronous invocation, queues, and other execution mechanisms. Their guarantees need to be evaluated, not replaced with a slogan.</p><p>The comparison is between complete designs. One invocation versus one long-lived process is rarely the whole design.</p><h3>Returning successfully is not the same as handing off successfully</h3><p>For the report feature, imagine the handler returns before it has recorded the job anywhere durable.</p><p>The user sees success. A crash immediately afterward can leave the application with no record of what it promised.</p><p>Now reverse the order. A durable job record exists, but the response never reaches the browser. The user retries. Does the application identify the existing request or create another report?</p><p>These are handoff questions. Neither a container image nor a serverless function answers them on its own.</p><p>AWS’s documentation provides a useful concrete distinction: asynchronous Lambda invocation queues an event and returns an acceptance response rather than the eventual function result. Separate documentation describes retries and the possibility of duplicate delivery. <a href="https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html">The handoff and execution are different stages.</a> <a href="https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html">The error-handling contract matters too.</a></p><p>That is not a recommendation to choose Lambda for every report generator. It is an example of the kind of boundary a platform decision should name explicitly.</p><p>What has been accepted? What has been completed? Where can the operator tell the difference?</p><p>Without those answers, an immediate success response is doing public relations for unfinished work.</p><h3>The unit you need to preserve is progress</h3><p>For the example report, the important state might include the requested filters, an operation identifier, the current stage, the output location, and whether notification has been sent.</p><p>Not every application needs all of those as separate fields. It needs an explicit account of what recovery must know.</p><p>If the process disappears while generating a file, can it restart from the input? If it disappears after storing the output, can it discover that output instead of producing another? If notification was sent but not recorded, what duplication is tolerable?</p><p>Some operations can simply be repeated. Others require an idempotency mechanism, a check of existing state, or an explicit reconciliation step.</p><p>There is no benefit in describing all of this as “exactly once” unless the complete set of effects actually supports that claim.</p><p>Modern workflow platforms make this topic more accessible, not less necessary. Vercel’s current Workflows documentation describes durable multi-step execution that can pause and resume across interruptions. <a href="https://vercel.com/docs/workflows">That is a different capability from keeping an HTTP connection open.</a></p><p>A durable orchestrator can manage important execution state. You still need to understand the side effects it coordinates and the guarantees of the services it calls.</p><p>The product promise belongs to the whole workflow.</p><h3>Choose the operating model after drawing the interruption points</h3><p>Once the report’s recovery behavior is clear, the hosting comparison becomes much more concrete.</p><p>A serverless worker may fit bursty, bounded units of work and reduce idle infrastructure to manage. A container worker may fit a workload needing different execution characteristics, dependencies, or resource control.</p><p>Those are tendencies, not universal rules. Provider limits, actual utilization, deployment requirements, and the team’s operating experience can change the answer.</p><p>The useful cost calculation includes the durable handoff and recovery pieces on both sides. Comparing an isolated function price with a whole container fleet bill does not compare equivalent systems.</p><p>If you want a wider set of failure patterns to apply to that design, the architecture and event-driven material in <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library">The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</a> helps examine what happens between acceptance and completion. The capability you want is recognizing where an apparently simple background task can lose its promise.</p><p>Then test the design where it is least comfortable, not where both platforms look good in a demo.</p><h3>Kill the worker before declaring a winner</h3><p>Use controlled, non-production inputs and interrupt the report at several stages.</p><p>Before execution begins. During output creation. After the file is stored. Before notification. Immediately after notification.</p><p>Observe what the system knows, what it repeats, and what an operator has to repair manually.</p><p>A design that restarts from scratch may be entirely acceptable for an inexpensive report. A design that repeats an irreversible external action may not be.</p><p>The goal is not the most sophisticated workflow engine. It is a completion model appropriate to the consequence of failure.</p><p>Give the user an honest view of that model too. A recognizable operation state is more useful than a spinner that disappears while work continues somewhere unobservable.</p><p>The deployment discussion can now be specific: which option delivers this behavior with less cost and operating risk for this team?</p><p>That is a question you can answer with evidence.</p><p>“Serverless cannot do background jobs” and “containers solve long-running work” are much easier statements. They are also poor substitutes for the missing job record.</p><p>The request ended. Your promise to the user did not.</p><p>Try designing that handoff as part of a complete system rather than a hosting preference. Froquiz’s System Design practice starts with a product brief and lets you connect the architecture yourself, including the components responsible for unfinished work.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1c8d08ce4cc" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/serverless-vs-containers-who-finishes-the-job-after-the-request-ends-a1c8d08ce4cc">Serverless vs Containers: Who Finishes the Job After the Request Ends?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Webhook Failed Once. Our Database Processed It 47 Times.]]></title>
            <link>https://blog.stackademic.com/the-webhook-failed-once-our-database-processed-it-47-times-789d8f3d6aa0?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/789d8f3d6aa0</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:29:03 GMT</pubDate>
            <atom:updated>2026-09-03T17:29:03.534Z</atom:updated>
            <content:encoded><![CDATA[<h4>The provider retried exactly as promised. Our handler confused receiving an event with completing the work.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGTr9wxdjtK-qauQOQEcXg.png" /></figure><p>The first webhook arrived at 14:03:11.</p><p>The payment provider had done its job. The signature was valid, the event ID was unique, and the payload said the invoice had been paid. Our handler inserted the payment, updated the subscription, wrote an audit row, called an email service, and then waited for a CRM request that never returned.</p><p>Thirty seconds later, the request timed out before our API returned 200.</p><p>The provider saw a failed delivery and retried. Then it retried again. By the time anyone noticed, the same event had reached us 47 times. The customer had one payment, but our database contained duplicate credits, repeated emails, and 47 attempts to advance the same subscription.</p><p>Nothing was wrong with the retry policy. The missing guarantee was ours.</p><h3>A webhook response is an acknowledgement</h3><p>We had built the endpoint as if the provider was asking: “Can you finish every consequence of this event before I disconnect?”</p><p>It was asking a smaller question: “Have you safely received this event?”</p><p>Those contracts are not interchangeable. External webhook systems cannot see our database transaction or our downstream calls. They see an HTTP response. If the connection closes, the timeout expires, or we return an error, the only safe behavior for the provider is to try again.</p><p>Our handler’s latency therefore controlled duplicate delivery. Every extra synchronous operation widened the period in which useful work could commit while the acknowledgement still failed.</p><p>The database update completed in 180 milliseconds. The email call completed in 600. The CRM call consumed the remaining timeout. We had already changed customer state, but from the provider’s perspective nothing had been accepted.</p><p>The first debugging mistake was searching for why the provider “sent duplicates.” Duplicate delivery was expected. The real question was why duplicate delivery could create duplicate business effects.</p><h3>Event identity was not business identity</h3><p>We already stored the provider’s event ID in logs. That felt like idempotency because we could search for it after an incident. It was not idempotency because the database did not enforce anything with it.</p><p>The handler checked whether a payment record existed by invoice ID, found nothing, and started work. Two deliveries could perform that check at the same time. Both saw an empty result. Both inserted.</p><p>The fix started with an inbox table whose provider event ID had a unique constraint. Receiving the event and claiming its identity happened in the same short transaction. One delivery won. Every later delivery found the existing claim and returned success without repeating the work.</p><p>But event-level deduplication was only the first boundary. Providers can legitimately emit different event IDs for related state changes. We also needed business idempotency: “apply payment X to invoice Y once.” That rule belonged close to the state transition, protected by a unique key or an atomic conditional update — not by a hopeful SELECT before an INSERT.</p><p>Idempotency was no longer a utility function. It became a database invariant.</p><h3>We acknowledged first and processed second</h3><p>The repaired path became deliberately boring:</p><ol><li>Verify the signature and basic envelope.</li><li>Insert the event into the inbox with its unique identity.</li><li>Commit that receipt.</li><li>Return 200.</li><li>Process the event asynchronously from the durable inbox.</li></ol><p>The endpoint stopped sending email, calling the CRM, and calculating account state inside the provider’s timeout budget. A worker performed those actions with its own retry policy. Each action had an idempotency boundary appropriate to its effect.</p><p>This did not create exactly-once delivery. It created something more useful: at-least-once delivery with controlled, repeat-safe effects.</p><p>We also distinguished retryable failures from permanent ones. A temporary CRM timeout stayed pending. A payload that violated an invariant moved to review with the original event, error, attempt count, and affected customer attached. Repeating an impossible operation forever was not resilience.</p><h3>Recovery was more dangerous than the original retry</h3><p>Once we found 47 deliveries, the tempting move was to delete duplicate rows and replay the webhook. That could have charged downstream credits again or emitted another burst of email.</p><p>Recovery began by freezing the consumer and building a ledger: provider event ID, invoice ID, committed database changes, outbound messages, and current customer state. We repaired the final business state rather than blindly reproducing the original execution path.</p><p>For some customers, that meant reversing duplicate credits. For others, no database repair was needed because a later constraint had rejected the duplicate — but the email had still been sent. One event had crossed several systems, and each system had a different definition of “already done.”</p><p>Webhook incidents are expensive precisely because the first error is often small while the replay multiplies side effects. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> is built around this kind of production pattern: recognize the misleading symptom, confirm what actually committed, recover without replaying damage, and install the control that prevents the same shape from returning.</p><h3>We monitored duplicates as a normal condition</h3><p>After the fix, duplicate receipt no longer triggered panic. We expected it and measured it.</p><p>The useful signals were acknowledgement latency, inbox insert failures, duplicate event rate, processing age, attempts per event, and business-invariant conflicts. If duplicates rose, we could investigate timeouts or response failures without assuming customers had already been harmed. If processing age rose, we knew events were safely received but not becoming business outcomes.</p><p>We also tested the uncomfortable boundaries: kill the handler after commit but before response, deliver the same envelope concurrently, crash the worker after a downstream call, and replay a day of events at controlled speed. Those tests finally matched the failure semantics of the real system.</p><p>The provider had always been allowed to send the event again. Our architecture had simply assumed it would not.</p><p>The durable lesson was not “webhooks are unreliable.” It was that delivery and effect are separate contracts. Once we designed both explicitly, a retry became routine traffic instead of a multiplier for customer damage.</p><p>Knowing the pattern is one thing; choosing the next safe move while duplicate effects are already visible is another. Froquiz’s production scenarios exercise that judgment under pressure.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=789d8f3d6aa0" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-webhook-failed-once-our-database-processed-it-47-times-789d8f3d6aa0">The Webhook Failed Once. Our Database Processed It 47 Times.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Pod Recovered in 18 Seconds. The Incident Became Impossible to Explain.]]></title>
            <link>https://blog.stackademic.com/the-pod-recovered-in-18-seconds-the-incident-became-impossible-to-explain-20d2b3414539?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/20d2b3414539</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:02:06 GMT</pubDate>
            <atom:updated>2026-09-03T17:02:06.048Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kubernetes restored capacity faster than our evidence pipeline could preserve the reason it failed.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hf5d1u97iZpZpetI5v7WxQ.png" /></figure><p>The alert resolved itself before the on-call engineer opened the dashboard.</p><p>One API pod had restarted. The replacement was ready in 18 seconds. Error rate returned to normal, the Deployment showed full availability, and the incident channel contained the comforting sentence nobody should trust:</p><p>Looks transient. Kubernetes handled it.</p><p>Three hours later, another pod restarted.</p><p>Then two more.</p><p>By that point we knew the system was sick. We still did not know why, because every successful restart replaced the process state we needed to explain the previous one.</p><p>Kubernetes had repaired capacity.</p><p>It had not performed an investigation.</p><h3>Recovery erased urgency</h3><p>The platform behaved exactly as designed.</p><p>A container exited. The kubelet restarted it. Readiness checks kept traffic away until the new process could serve requests. Replicas elsewhere absorbed the brief gap.</p><p>From an availability perspective, this was a success.</p><p>From an evidence perspective, it was a countdown.</p><p>The default log view followed the current container. The useful stack trace belonged to the previous instance. The application wrote its fatal context to local storage inside the container. The node-level logs retained too little history. Our dashboard aggregated present health, not recent death.</p><p>The system moved forward while our evidence stayed behind.</p><p>By the time a human arrived, “healthy” meant only that the replacement had not failed yet.</p><h3>--previous was necessary, not sufficient</h3><p>The first useful command was simple:</p><pre>kubectl logs api-7d8c9 -c api --previous --timestamps</pre><p>It recovered logs from the prior terminated container instance — when the pod and node still retained them.</p><p>We also inspected termination state instead of reading only the current status:</p><pre>kubectl get pod api-7d8c9 -o jsonpath=&#39;{.status.containerStatuses[*].lastState}&#39;</pre><p>That separated several very different stories:</p><ul><li>the application exited with an error;</li><li>the kernel killed it for exceeding memory;</li><li>the liveness probe repeatedly killed a process that was slow but alive;</li><li>the node evicted it under resource pressure;</li><li>the rollout intentionally replaced it.</li></ul><p>“Restarted” was not a cause. It was a lifecycle event.</p><p>In our case, the previous logs ended in the middle of a large response serialization. The container state showed an OOM kill. Application heap metrics looked safe because they did not include all native buffers and runtime overhead inside the cgroup limit.</p><p>The pod recovered quickly enough to hide a memory boundary we were crossing only on certain requests.</p><p>Those requests shared a response shape, not an endpoint. A rare payload expanded into a large in-memory object before serialization. Looking only at route averages had hidden it; correlating termination time with request attributes finally exposed the trigger without retaining sensitive payloads.</p><h3>The dashboard described survivors</h3><p>Our availability panel asked how many replicas were ready now.</p><p>That was useful for routing traffic. It was terrible for understanding instability.</p><p>We added views for:</p><p>→ restart rate by workload and reason;</p><p>→ time since previous termination;</p><p>→ exit code and OOM state;</p><p>→ memory working set against the container limit;</p><p>→ liveness failures separated from readiness failures;</p><p>→ node evictions and pressure conditions.</p><p>A service with ten ready replicas and six restarts in an hour was no longer rendered as simply green.</p><p>We also changed the page condition. A single fast recovery did not need to wake a human. A rising restart rate, repeated reason, or loss of redundancy did. Evidence flowed to durable storage before local rotation could erase it.</p><p>The goal was not to page on every container exit.</p><p>It was to preserve enough state that recurring exits could become a diagnosis instead of folklore.</p><h3>Self-healing can become evidence laundering</h3><p>Automatic recovery creates a dangerous psychological effect.</p><p>Because users recover, the underlying failure is reclassified as noise. Because it is noise, nobody investigates. Because nobody investigates, the same trigger returns under slightly more load. Eventually the recovery mechanism itself becomes saturated and the “transient” issue turns into a full outage.</p><p>The platform did not hide the cause maliciously. We had built our observability around steady state and expected a human to reconstruct transitions after the fact.</p><p>That assumption failed whenever recovery was faster than response.</p><p>The recurring operational patterns matter more than memorizing another kubectl command. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> connects visible symptoms to confirming evidence, safe recovery, and prevention controls—exactly the path teams need when “Kubernetes fixed it” is the beginning of the incident, not the end.</p><h3>We designed a failure evidence budget</h3><p>Retention had previously been a storage-cost decision. We turned it into an incident requirement.</p><p>For each critical workload, we decided what had to survive a restart:</p><ul><li>structured application logs with request and correlation IDs;</li><li>termination reason and last useful error context;</li><li>deployment revision and configuration identity;</li><li>a short high-resolution window of resource metrics;</li><li>traces for failed or unusually expensive requests;</li><li>relevant Kubernetes events exported beyond their short operational life.</li></ul><p>We did not retain everything forever. We retained the evidence needed to answer the first five questions after a failure.</p><p>The application also began writing concise termination messages for fatal paths it could detect. OOM kills could not depend on application cleanup, so external metrics and container status remained essential.</p><p>Most importantly, restarts became part of the release gate. A canary that remained ready but restarted was not healthy.</p><h3>Eighteen seconds was not the incident duration</h3><p>The visible error spike lasted 18 seconds.</p><p>The condition causing it had existed for days. The investigation took hours because the system repeatedly replaced evidence with healthy capacity.</p><p>Fast recovery is valuable. It protects users while engineers work.</p><p>But recovery without durable evidence creates a system that forgets faster than the team can learn.</p><p>The next time a pod restarts and turns green, do not ask only whether traffic recovered.</p><p>Ask whether the reason survived.</p><p>Production scenarios rarely announce which signal matters. Froquiz gives you the failure state and forces the next diagnostic or recovery decision — the part that becomes difficult when automation has already cleaned up the scene.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20d2b3414539" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-pod-recovered-in-18-seconds-the-incident-became-impossible-to-explain-20d2b3414539">The Pod Recovered in 18 Seconds. The Incident Became Impossible to Explain.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Fallback Worked. It Returned Yesterday’s Prices During Checkout.]]></title>
            <link>https://medium.com/@DevSideHustle/our-fallback-worked-it-returned-yesterdays-prices-during-checkout-466badeb1c0e?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/466badeb1c0e</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:13:28 GMT</pubDate>
            <atom:updated>2026-09-01T22:13:28.941Z</atom:updated>
            <content:encoded><![CDATA[<h4>Availability stayed green because the degraded path answered quickly with data that was no longer safe to sell.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUjtUhPo3kRyqIHGzTPd-w.png" /></figure><p>The pricing service began timing out during a regional network incident.</p><p>Checkout did not fail. A fallback loaded the last successful price snapshot from local storage and returned within 20 milliseconds. Error rate stayed low. Latency improved compared with waiting for the dependency.</p><p>Customers completed purchases using prices from the previous day.</p><p>A promotion had ended at midnight. Several products had changed tax treatment. The fallback preserved technical availability and violated the business contract.</p><p>The service was up. The answer was fast. The money was wrong.</p><h3>Fallback data had no safety boundary</h3><p>The snapshot existed to keep read-only browsing available during pricing trouble. Checkout reused the same client and inherited the fallback automatically.</p><p>No caller declared whether stale data was acceptable. The library treated every price request identically.</p><p>For a product page, showing a recent price with a temporary notice might be tolerable. For charging a card, the system needed authoritative price, bounded staleness with an explicit guarantee, or a safe refusal.</p><p>The fallback converted dependency failure into silent semantic failure because it returned the same response shape as live data.</p><h3>Green metrics counted responses, not correctness</h3><p>The pricing client recorded the fallback as success. Checkout recorded a completed order. Neither metric knew the result came from an expired snapshot.</p><p>Finance detected the incident through margin anomalies, not reliability alerts.</p><p>We added provenance and age to the price response. Callers could distinguish live, cached, manually overridden, and fallback values. The checkout path rejected data older than its business budget while browsing could degrade more flexibly.</p><p>Availability was redefined as completing a correct customer outcome, not returning any 200 response.</p><h3>Recovery required deciding which price to honor</h3><p>When live pricing returned, simply switching back fixed new requests but left completed orders at stale values.</p><p>We identified purchases made with fallback provenance, compared charged and authoritative totals, and grouped them by legal and customer-impact rules. Some lower prices were honored. Some tax differences required correction without charging customers again. Higher stale prices triggered refunds.</p><p>We avoided automatically replaying checkout. Payment, inventory, and customer communication had already occurred. Recovery repaired the business ledger instead of rerunning the technical request.</p><p>The incident was not over when the dependency recovered. It ended when affected orders had a deliberate outcome.</p><h3>Degradation became a product decision</h3><p>Each dependency call documented its fallback behavior by workflow.</p><p>Browsing could use a snapshot within a visible age limit. Cart calculation could preserve an existing quoted price under defined rules. Final purchase required authoritative tax and a price allowed by the offer contract. If those guarantees were unavailable, checkout paused safely and preserved the cart.</p><p>The user experience explained temporary unavailability instead of lying with a fast stale answer.</p><p>We also prevented fallback recursion. If local state was missing, the client did not call another stale service whose own provenance was unknown.</p><p>Fallback incidents are dangerous because resilience patterns can improve every technical graph while worsening the customer result. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> develops the judgment behind these cases: identify what guarantee the fallback removed, contain the affected workflow, reconcile completed side effects, and add a prevention boundary based on business truth.</p><h3>We tested failure with changing truth</h3><p>Our old resilience test disconnected the pricing service while prices remained static. The fallback returned the same answer, so the test passed.</p><p>The new test changed promotions, taxes, inventory rules, and customer entitlements while the dependency was unavailable. It asked whether each workflow degraded safely when the cached answer diverged from reality.</p><p>Metrics tracked fallback rate, data age, provenance by endpoint, and business outcomes completed under degraded data. Any checkout using fallback pricing became visible immediately.</p><p>We added an expiry mode that removed stale values from sale rather than silently extending them. When a promotion boundary approached, the system refreshed proactively and narrowed the fallback window. Data that was safe for hours on an ordinary day might be unsafe one second after a scheduled price change.</p><p>Fallback stores were versioned with the rules that created them. A tax snapshot generated under an old calculation model could not masquerade as current merely because its timestamp was recent. Freshness included semantic version, currency, market, and customer eligibility.</p><p>We also tested dependency recovery under load. When live pricing returned, every caller attempted to refresh at once. Request coalescing and jitter prevented the fallback exit from becoming a stampede that took the service down again.</p><p>The customer ledger preserved provenance after purchase. That made reconciliation a query rather than a forensic search across logs: which orders used degraded prices, what snapshot generated them, and which correction rule applied?</p><p>Finally, business owners approved fallback budgets. Engineering could describe staleness, but product, finance, and legal determined whether a particular stale answer was acceptable. The decision existed before the outage instead of emerging in an incident channel.</p><h3>A fallback is an alternate contract</h3><p>Fallback code is often reviewed as a technical implementation detail. In production it defines what promise remains when the normal source disappears.</p><p>The final system did not remove fallbacks. It made their reduced guarantees explicit, observable, and acceptable to each caller.</p><p>Speed and availability were useful only after correctness had a boundary.</p><p>The UI carried that boundary too. If checkout could not guarantee a final price, it stopped presenting the result as an ordinary purchase. Depending on the market, it either preserved the cart, requested confirmation after refresh, or blocked the transaction. The fallback contract was visible to the person absorbing its risk.</p><p>That removed the most dangerous ambiguity: an endpoint could be technically available while the business operation it represented was not safe to complete.</p><p>Froquiz’s senior scenarios practice that trade-off: serve stale data, block checkout, preserve the cart, honor an old quote, or repair completed orders when the fallback is healthy and the answer is not.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=466badeb1c0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Slow Query Was Cancelled. Its Transaction Stayed Open.]]></title>
            <link>https://blog.stackademic.com/the-slow-query-was-cancelled-its-transaction-stayed-open-b74c1fb9b201?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b74c1fb9b201</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 21:56:37 GMT</pubDate>
            <atom:updated>2026-09-03T11:24:19.858Z</atom:updated>
            <content:encoded><![CDATA[<h4>We stopped the statement in seconds and kept its locks alive for forty-three minutes.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v6iPvK3qKL8rhdsnpeK4Jw.png" /></figure><p>At 09:12, checkout latency crossed six seconds.</p><p>The database dashboard showed one reporting query scanning far more rows than expected. An engineer cancelled it. CPU fell immediately, the query disappeared from the active list, and the incident channel reported that the offender was gone.</p><p>Checkout did not recover.</p><p>For the next forty-three minutes, updates waited behind a session that appeared to be doing nothing. The statement had stopped. The transaction that owned its locks had not.</p><p>We had treated a query, a transaction, and a connection as if they shared one lifetime. Production made the difference expensive.</p><h3>Cancelling work did not undo its transaction</h3><p>The reporting request opened a transaction, ran several statements, and reached the slow one after taking a lock needed by checkout. When the statement was cancelled, the application caught the database exception and returned an error response.</p><p>It did not execute an explicit rollback.</p><p>The connection remained checked out while application cleanup attempted to log additional context. In another version of this bug, the connection could have returned to a pool in a failed transaction state. Either way, stopping the current statement was not the same as ending the transaction.</p><p>The misleading part was visual. The expensive SQL vanished from the “currently running queries” panel. Database CPU improved. Yet the session still had a transaction start time, and blocked writers continued accumulating behind it.</p><p>We had removed the loud symptom and left the ownership state intact.</p><h3>The wait graph mattered more than the slow-query list</h3><p>Our first dashboard sorted by query duration. That is useful when execution consumes resources. It is weak when an idle session owns something other sessions need.</p><p>The investigation changed when we asked which process each checkout transaction was waiting for. The blockers formed a simple chain: dozens of customer writes waited on one session; that session was idle in transaction; its last visible statement was the cancelled report.</p><p>The most important timestamps were transaction start, state-change time, and lock acquisition — not just current-query start.</p><p>We terminated the blocking session, watched waiting transactions drain, and confirmed checkout success before reopening normal traffic. Killing the session was safe only after identifying the business work it contained. A blocker can also be halfway through a legitimate operation, and removing it blindly can turn latency into data inconsistency.</p><p>The evidence had to answer both questions: “Is this session causing harm?” and “What will aborting it leave behind?”</p><h3>Rollback became mandatory cleanup</h3><p>The application fix was small in code and large in meaning. Every transaction path used a boundary that committed on success and rolled back on any exception, including cancellation and request timeout. Cleanup no longer depended on the calling code remembering what state the connection was in.</p><p>Pool return also validated transaction state. A connection could not quietly carry failed or open work into the next borrower.</p><p>We separated three timeout controls:</p><ul><li>Statement timeout limited how long one operation could execute.</li><li>Lock timeout limited how long it could wait to acquire contested state.</li><li>Idle-in-transaction timeout limited how long a transaction could remain open without useful work.</li></ul><p>None replaced application correctness. They were containment boundaries for the day correctness failed.</p><p>The values were chosen from workload behavior, not copied from a tuning guide. A timeout aggressive enough for checkout could destroy a legitimate maintenance operation. Different roles received different budgets.</p><h3>Recovery needed more than terminating a PID</h3><p>After the blocker disappeared, we checked the consequences it had created: request queues, retry storms, pool occupancy, replication lag, and customer operations that might have timed out after committing.</p><p>Some checkout clients had retried while their original transactions waited. That created a second risk: duplicate order submission after database latency recovered. The incident was not over when lock wait reached zero.</p><p>We temporarily limited admission, drained the backlog, and reconciled operations using idempotency keys. Only then did we restore normal concurrency.</p><p>Database incidents often punish the first plausible fix. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> is designed for that gap between symptom and safe recovery: the evidence that confirms the actual failure shape, the action that limits customer damage, and the prevention control that keeps a cancelled statement from becoming a silent forty-minute blocker.</p><h3>We alerted on ownership without progress</h3><p>The lasting dashboard did not merely list long queries. It highlighted old transactions, idle transactions, blocking depth, oldest lock wait, and connections returned to the pool in an invalid state.</p><p>An alert fired when a transaction remained open without an active statement beyond its role’s normal range. Another fired when one backend blocked more than a small number of customer writes. This moved detection from “the API feels slow” to “one owner is preventing progress.”</p><p>We also added the transaction ID and request correlation ID to logs at begin, commit, and rollback. During the next incident, the team could connect an application timeout to the database session it left behind.</p><p>The connection-pool dashboard changed too. Previously it showed only checked-out versus available connections. We added the age of the oldest checkout and the transaction state attached to each borrowed connection. A pool at 40 percent utilization could now look dangerous if one old borrower held a critical lock; a pool at 90 percent could still be healthy if requests were short and progressing. Capacity without state had been another misleading green number.</p><p>In testing, we deliberately cancelled requests between statements, disconnected clients during commit, and raised exceptions inside transaction callbacks. The acceptance condition was not merely that the API returned an error. It was that no transaction, lock, or poisoned connection survived the request boundary. That turned cleanup from a convention into behavior we could verify.</p><p>The slow query had deserved attention. But it was never the whole incident.</p><p>A statement is an attempt. A transaction is ownership. A session is the container that can preserve that ownership after the attempt fails. Recovery became reliable only when our tools and code treated those lifetimes separately.</p><p>Froquiz’s production scenarios train that distinction: stop the visible damage, identify the state that survived, and choose the next action without creating a second incident.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b74c1fb9b201" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-slow-query-was-cancelled-its-transaction-stayed-open-b74c1fb9b201">The Slow Query Was Cancelled. Its Transaction Stayed Open.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Added a Timeout to Every Service. The Shortest One Broke the Entire Request Path.]]></title>
            <link>https://blog.stackademic.com/we-added-a-timeout-to-every-service-the-shortest-one-broke-the-entire-request-path-b3bc91fad412?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3bc91fad412</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:40:40 GMT</pubDate>
            <atom:updated>2026-09-03T11:09:30.773Z</atom:updated>
            <content:encoded><![CDATA[<h4>Every timeout looked reasonable alone. Together, they made success impossible.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lWzyb6WWlH5VIKgfyiFdTg.png" /></figure><p>The checkout request had a two-second deadline at the edge. The order service allowed 1.8 seconds. The inventory client allowed one second. The database statement timeout was 900 milliseconds.</p><p>On a diagram, the numbers looked disciplined.</p><p>In production, the inventory call began after authentication, pricing, and fraud checks had already consumed 1.1 seconds. It inherited a one-second client timeout even though the edge had only 900 milliseconds left. When it retried, the request was already doomed.</p><p>We had added timeouts everywhere and built no shared understanding of time.</p><h3>A timeout is a budget, not a patience setting</h3><p>Teams often choose timeouts component by component. The database team picks a safe query duration. The service owner chooses a client timeout. The gateway owner chooses an edge limit. Each value can be defensible in isolation.</p><p>The request experiences them as one nested budget.</p><p>If an outer layer gives the operation two seconds, inner work cannot each receive two seconds. Queueing, serialization, network travel, retries, and cleanup all consume time too.</p><p>Our trace showed the real sequence:</p><pre>gateway and auth:        240 ms<br>pricing:                 310 ms<br>fraud check:             480 ms<br>inventory queue wait:    160 ms<br>inventory attempt:       900 ms and still running<br>edge deadline:           exceeded</pre><p>The inventory service eventually completed useful work, but the caller had disconnected. A retry had started before the first attempt stopped. The system spent twice as much capacity producing a response nobody could receive.</p><p>The shortest timeout was not simply too short. It was detached from the remaining end-to-end deadline.</p><h3>Retries spent time we did not have</h3><p>Our client policy allowed two attempts with exponential backoff. That policy had been copied from a generic reliability library.</p><p>It did not ask whether enough budget remained for another meaningful attempt.</p><p>After the first timeout, the client waited, retried, and crossed the gateway deadline. The edge returned an error while both downstream attempts continued until their own local timers expired.</p><p>This is how a latency problem becomes a load problem. Slow operations trigger retries. Retries increase concurrency. Increased concurrency creates queueing. Queueing causes more operations to cross their deadlines.</p><p>We contained the loop by disabling retries on the checkout path and propagating cancellation to downstream work. Error rates rose briefly because retries were no longer hiding first-attempt failure, but resource occupancy dropped and recovery began.</p><p>The durable retry rule became conditional:</p><pre>retry only if:<br>- the operation is safe to repeat;<br>- the failure is plausibly transient;<br>- enough deadline remains for a full attempt;<br>- the retry budget for this dependency is available.</pre><p>A retry that cannot finish before the caller leaves is not resilience. It is background load with optimistic branding.</p><h3>We propagated one deadline through the call graph</h3><p>The edge now establishes an absolute deadline. Services propagate it with the request rather than inventing a fresh duration at every hop.</p><p>Each component calculates remaining time, reserves a small cleanup margin, and refuses to start work that cannot plausibly finish.</p><p>That does not mean every service blindly trusts a caller-supplied value. Local maximums still protect dependencies. The effective timeout is the smaller of the local safety limit and the remaining request budget.</p><p>Conceptually:</p><pre>effective budget = min(local maximum, caller deadline - now - cleanup margin)</pre><p>The cleanup margin matters. A process needs time to cancel a query, release a connection, record the outcome, and return a controlled error. Spending the final millisecond on work leaves cleanup to collide with the next request.</p><p>We also separated queue wait from execution time. Previously, an inventory request could spend most of its budget waiting for a worker, then begin expensive work with too little time remaining. Admission control now rejects or degrades before execution when the remaining budget is inadequate.</p><h3>Faster failure is useful only when the system stops</h3><p>Returning a timeout quickly can improve customer experience. It does not reduce load if abandoned work continues behind the response.</p><p>We tested cancellation across every boundary:</p><ul><li>Does the HTTP server notice client disconnects?</li><li>Does the downstream client cancel its socket operation?</li><li>Does the database driver cancel the active statement?</li><li>Does the worker return the connection to the pool?</li><li>Does an external side effect require reconciliation if cancellation is ambiguous?</li></ul><p>Cancellation is not universally safe. A payment request may have reached the provider even if our local context is cancelled. For irreversible operations, we stop waiting but move the outcome into an explicit unknown state and reconcile using an idempotency key.</p><p>That distinction prevented us from treating “caller left” as “nothing happened.”</p><p>Our dashboards changed too. We graph deadline remaining at each service entry, work completed after caller cancellation, retry attempts by remaining budget, and connection occupancy from timed-out requests.</p><p>Those signals reveal whether timeouts are actually containing work or merely hiding it from users.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is built around these production interactions — the difference between a local best practice and a safe end-to-end system, and the recovery decisions that keep timeouts, retries, queues, and side effects from amplifying one another.</p><h3>The number was never the whole policy</h3><p>We no longer review a timeout as a single value. We review a deadline policy.</p><p>It includes the owner of the end-to-end budget, how remaining time propagates, whether the work is cancellable, which failures are retryable, what happens when the budget is almost gone, and which partial effects need reconciliation.</p><p>The original timeouts were not absurd. That was why the incident was difficult. Every team could point to a reasonable number in its own configuration.</p><p>Production exposed the missing agreement between them.</p><p>The checkout path became more reliable after we removed several independent timers and made one deadline visible across the graph. Some requests now fail earlier. Far fewer continue consuming resources after success is impossible.</p><p>A timeout does not create capacity. It decides when the system admits that the capacity did not arrive in time.</p><p>The senior decision is choosing what must stop, what may retry, and what must be reconciled when that admission happens. Those are exactly the kinds of trade-offs worth practicing before a two-second budget becomes a twenty-minute incident.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3bc91fad412" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-added-a-timeout-to-every-service-the-shortest-one-broke-the-entire-request-path-b3bc91fad412">We Added a Timeout to Every Service. The Shortest One Broke the Entire Request Path.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PostgreSQL Had 70% Free Disk. One Temporary Query Still Took Production Down.]]></title>
            <link>https://blog.stackademic.com/postgresql-had-70-free-disk-one-temporary-query-still-took-production-down-20832b3e4e35?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/20832b3e4e35</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:14:01 GMT</pubDate>
            <atom:updated>2026-09-03T11:08:14.654Z</atom:updated>
            <content:encoded><![CDATA[<h4>The disk alert never fired because the filesystem was not the disk that filled.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRjD-Z7iW1klTafSn_0p-Q.png" /></figure><p>At 10:06, an internal reporting request began running. At 10:09, API latency crossed three seconds. At 10:11, new PostgreSQL connections started failing.</p><p>The infrastructure dashboard showed 70% free disk.</p><p>That number sent the investigation in the wrong direction.</p><p>The query was sorting and hashing far more data than work_mem could hold. PostgreSQL spilled the intermediate results to temporary files. Those files lived on a smaller volume mounted for database temporary work, not on the filesystem shown in our primary disk panel.</p><p>The database had plenty of permanent storage and nowhere left to breathe.</p><h3>A read-only query can still be destructive</h3><p>The request did not update a row. It did not hold an obvious application lock. It came from an authenticated internal user and used a query pattern that had worked for smaller date ranges.</p><p>That made it feel safe.</p><p>Its plan joined a large event table, grouped by several dimensions, and sorted the result. PostgreSQL could execute every step correctly, but the intermediate set was enormous. Once memory was exhausted, the executor wrote temporary blocks to disk.</p><p>We found the evidence in two places:</p><pre>SELECT datname, temp_files, temp_bytes<br>FROM pg_stat_database<br>ORDER BY temp_bytes DESC;</pre><p>and in logs after enabling temporary-file logging at a useful threshold:</p><pre>temporary file: path &quot;base/pgsql_tmp/...&quot; size 18GB</pre><p>The query was not “slow” in the usual sense. It was converting a logical reporting question into an unbounded infrastructure allocation.</p><p>Once the temporary volume filled, unrelated sessions failed while writing their own temporary state. Connection attempts piled up. Health checks timed out. The API began restarting workers, which created a new wave of connection attempts.</p><p>One analytical query had turned into an application outage without modifying production data.</p><h3>The first fix was containment, not optimization</h3><p>It was tempting to add an index while the incident was active. That would have been a risky bet. We first needed to stop the resource drain and restore ordinary traffic.</p><p>We identified the backend by application name, user, start time, and query fingerprint. Then we cancelled it. Cancellation gave PostgreSQL a chance to unwind cleanly; termination was reserved for a backend that would not respond.</p><p>Next, we paused the reporting route and confirmed that temporary space, connection acquisition time, and API latency were recovering together.</p><p>Only then did we work on the query.</p><p>The durable changes were less glamorous than “make SQL faster”:</p><ul><li>The reporting role received a strict statement_timeout.</li><li>Large reports moved to an asynchronous job with an explicit row and date-range budget.</li><li>The application set a recognizable application_name so operators could separate reports from customer traffic.</li><li>Temporary-file bytes and failures became monitored signals.</li><li>The temporary volume received its own capacity alert.</li><li>The report returned a bounded aggregate rather than sorting raw detail before pagination.</li></ul><p>The index change came last, after EXPLAIN (ANALYZE, BUFFERS) showed where it would actually reduce work.</p><h3>work_mem is not a harmless speed dial</h3><p>When a sort spills, the common advice is to increase work_mem. Sometimes that is correct for a controlled session. Applying it globally can make the next incident worse.</p><p>work_mem is available per eligible sort or hash operation, not once per server. One query can have multiple such nodes. Many concurrent queries multiply the potential allocation again.</p><p>A global increase that helps one report may allow dozens of sessions to reserve enough memory to trigger swapping or the OOM killer.</p><p>We used a local setting for the known reporting job after measuring the plan:</p><pre>BEGIN;<br>SET LOCAL work_mem = &#39;128MB&#39;;<br>-- bounded reporting query<br>COMMIT;</pre><p>That was paired with concurrency limits. Memory budgets without admission control are promises the server may be asked to keep simultaneously.</p><p>The more important optimization was reducing the amount of data entering the expensive stage. We pushed filters earlier, removed unused selected columns, pre-aggregated stable dimensions, and stopped offering unbounded date ranges through a synchronous endpoint.</p><h3>Our monitoring described capacity, not failure</h3><p>The 70% free-disk panel was factually correct. It was operationally misleading.</p><p>It answered: “How full is the primary data filesystem?”</p><p>The incident required different questions:</p><ul><li>Can PostgreSQL create temporary files right now?</li><li>How quickly is temporary-file usage growing?</li><li>Which query fingerprints are producing the bytes?</li><li>Are reports competing with transactional traffic?</li><li>How long are requests waiting for a connection before SQL starts?</li></ul><p>Monitoring improves when it follows the resource path of a request. An API call can wait for a worker, a connection, a lock, memory, temporary disk, storage I/O, and a downstream response. A green capacity chart at one layer cannot certify the path.</p><p>We also changed the readiness check. A trivial SELECT 1 proved that a connection could execute a trivial query. It said nothing about whether the database could support the application’s normal work. We kept the cheap check, but stopped interpreting it as a full health verdict.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> helps build this kind of incident pattern recognition: moving from a reassuring top-line metric to the constrained resource, containing the blast radius, and turning the discovery into a repeatable production guardrail.</p><h3>The prevention rule we kept</h3><p>Any user-controlled dimension that changes the amount of work must have a budget.</p><p>Date range, export size, group-by dimensions, sort keys, regex complexity, graph depth, and page size are not merely product inputs. They are resource allocation controls wearing UI labels.</p><p>We now review reporting endpoints with four explicit limits:</p><pre>maximum scanned period<br>maximum returned rows<br>maximum concurrent jobs per account<br>maximum execution time</pre><p>Large work is still allowed, but it moves to a queue, reports progress, and cannot compete without bounds against checkout traffic.</p><p>The query did not take production down because PostgreSQL was fragile. PostgreSQL faithfully executed an expensive plan our system had authorized. The failure was that we allowed a synchronous request to claim an unbounded amount of shared temporary space.</p><p>Seventy percent free disk was comforting because it was precise. Precision is not the same as relevance.</p><p>Good operators learn to ask which disk, for which workload, at what growth rate, and what fails when it reaches zero. That is the difference between reading a dashboard and diagnosing a system.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20832b3e4e35" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/postgresql-had-70-free-disk-one-temporary-query-still-took-production-down-20832b3e4e35">PostgreSQL Had 70% Free Disk. One Temporary Query Still Took Production Down.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed.]]></title>
            <link>https://blog.stackademic.com/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed-4f3966122ebb?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f3966122ebb</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:28:53 GMT</pubDate>
            <atom:updated>2026-08-30T02:28:53.613Z</atom:updated>
            <content:encoded><![CDATA[<h4>The document looked thorough, passed review, and failed exactly where production had changed without telling it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WB97cQxF_vNxreEK0Tj7aA.png" /></figure><p>The incident commander opened the runbook.</p><p>Steps one through sixteen worked.</p><p>Step seventeen said to drain traffic through a load balancer we had removed five months earlier.</p><p>The replacement architecture was safer and simpler. The migration had closed successfully. The ticket was done.</p><p>Nobody had updated the recovery path.</p><p>At 03:08, a polished 43-step document became a history lesson.</p><h3>Runbooks decay while appearing stable</h3><p>Code fails loudly when an interface disappears.</p><p>Documents continue rendering.</p><p>The commands remain formatted. Links stay blue. Screenshots keep showing an interface that no longer exists. A reviewer can read the entire page without executing a single assumption.</p><p>Our runbook had an owner, a review date, and version history. It looked governed.</p><p>The problem was semantic drift.</p><p>Cluster names changed. Traffic management moved. Permissions tightened. A database failover process became managed. The document received small edits but its recovery model still described the old system.</p><p>We maintained sentences.</p><p>We did not test recovery.</p><h3>The obsolete step was load-bearing</h3><p>Most of the runbook remained correct.</p><p>That made the failure worse.</p><p>Responders invested confidence through sixteen successful steps. When step seventeen failed, they assumed a permissions problem or temporary control-plane issue rather than an obsolete architecture.</p><p>The missing load balancer had previously provided a way to remove one cluster from traffic without changing global routing. Its replacement required a different sequence with different propagation time.</p><p>The architecture migration had preserved normal behavior.</p><p>It had changed emergency behavior.</p><p>Nobody included runbook impact in the definition of done.</p><h3>We turned steps into executable assertions</h3><p>Not every recovery action can or should run automatically. Many assumptions can be tested safely.</p><p>We extracted checks from the runbook:</p><p>→ Required resources exist.</p><p>→ Commands resolve against current tooling.</p><p>→ Referenced dashboards and ownership records are reachable.</p><p>→ The operator role has the permissions the procedure requires.</p><p>→ Traffic movement can be rehearsed with a non-production target.</p><p>→ Recovery artifacts are available without the failed system.</p><p>Scheduled validation caught structural decay. Game days tested whether the sequence still produced the intended outcome.</p><p>The runbook remained prose where judgment mattered. Its factual assumptions stopped relying on memory.</p><p>Incident recovery repeatedly fails because the system changed while the human control path did not. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those recovery patterns continue across infrastructure, databases, deployments, and the judgment required when the documented path ends.</p><h3>Documentation is part of the production system</h3><p>During an incident, a runbook changes operator behavior. That makes it operational infrastructure.</p><p>It has dependencies, interfaces, permissions, owners, and failure modes. Treating it as a page rather than a component creates the same problem as any untested integration.</p><h3>The new runbook is shorter and less comforting</h3><p>We reduced 43 steps to fourteen verified actions and several decision points.</p><p>The old document tried to anticipate every branch. It became detailed enough to discourage rehearsal and fragile enough to fail after ordinary change.</p><p>The new version states the goal of each action, the evidence expected, and the condition for stopping. Responders can recognize when reality diverges instead of executing stale commands with greater determination.</p><p>Every architectural change now asks one operational question:</p><p>Which recovery assumption did this invalidate?</p><p>If the answer is unknown, the migration is not finished.</p><p>A runbook is not proof that a team can recover. It is a hypothesis about production written before the emergency.</p><p>Test the hypothesis while everyone is awake.</p><p>Senior incident scenarios are difficult because the documented next step eventually stops matching reality. Practice deciding what the step was trying to prove.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f3966122ebb" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed-4f3966122ebb">Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Had Backups in Three Regions. The Restore Needed One Deleted IAM Role.]]></title>
            <link>https://blog.stackademic.com/we-had-backups-in-three-regions-the-restore-needed-one-deleted-iam-role-b28af8b5b8ff?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b28af8b5b8ff</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:09:37 GMT</pubDate>
            <atom:updated>2026-08-30T02:09:37.705Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data redundancy survived the disaster. The permission path required to recover it did not.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yyH9n8V38dBKuMcchdqLw.png" /></figure><p>The backups were healthy.</p><p>Daily restore verification passed. Copies existed in three regions. Retention matched policy. Encryption keys were available.</p><p>Then we needed a real restore.</p><p>The automation failed before reading the first backup object.</p><p>It attempted to assume an IAM role deleted during a security cleanup six weeks earlier.</p><p>We had redundant data.</p><p>We had one recovery identity, and it no longer existed.</p><h3>Backup success tested storage, not recovery</h3><p>Our dashboard answered several useful questions.</p><p>Did backup jobs run? Were objects written? Could checksums be read? Did retention preserve the expected dates?</p><p>The test restore ran inside the production account using existing service permissions. It proved that the ordinary environment could access its own backups.</p><p>The real disaster scenario was different.</p><p>We were restoring into a clean account because the original environment was considered compromised. The bootstrap automation needed a cross-account role, a specific trust policy, DNS access, encryption grants, and a registry credential.</p><p>Those dependencies were not inside the backup.</p><p>They were assumptions surrounding it.</p><h3>Security cleanup removed an invisible dependency</h3><p>The IAM role looked unused.</p><p>No production workload had assumed it in months. Access analysis showed no recent activity. The name belonged to an older disaster-recovery project. Deletion reduced standing privilege and passed review.</p><p>The role was supposed to be unused.</p><p>Its value existed only during a rare event.</p><p>This is a difficult class of dependency because normal observability argues for removal. Emergency paths are quiet when healthy. Their inactivity looks identical to abandonment.</p><p>We had documented the role in a recovery guide, but the deletion process did not connect infrastructure identity to operational documentation.</p><p>The permission disappeared. The runbook kept rendering.</p><h3>We rebuilt recovery from an empty account</h3><p>The repair took hours because each missing prerequisite appeared sequentially.</p><p>After recreating the role, encryption policy blocked access. After fixing the key grant, the restore image could not be pulled. After mirroring the image, DNS delegation still required credentials from the compromised account.</p><p>Our disaster-recovery design had copied data and referenced control-plane state.</p><p>We changed the exercise.</p><p>Recovery now began with a deliberately empty account and a small offline bootstrap package containing only the minimum identities, verified artifact references, and instructions required to establish trust. Everything else was rebuilt from versioned infrastructure and restored data.</p><p>Critical roles carried ownership and deletion protection tied to tested recovery paths. Temporary credentials were created during exercises rather than left standing indefinitely.</p><p>Recovery incidents repeat because teams preserve data while allowing the mechanism that gives data meaning to drift. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those failure and recovery patterns continue across infrastructure, security, databases, and incident operations.</p><h3>Restore dependencies are part of the backup</h3><p>A backup is useful only if the organization can transform it into a working service within the promised time.</p><p>That transformation depends on identity, keys, tooling, artifacts, network paths, knowledge, and people. Copying database files across regions protects one part of the chain.</p><h3>Three regions were not the reassuring number</h3><p>We still keep multiple backup copies.</p><p>They protect against corruption, deletion, and regional loss. But the metric leadership watches now is successful recovery from an environment with no inherited trust.</p><p>The exercise is slower and more expensive than checking object existence.</p><p>It is also the only test that resembles the day we actually need it.</p><p>Data durability and service recoverability are different promises.</p><p>You can satisfy the first with copies. The second requires every dependency between those copies and a functioning product.</p><p>The deleted role taught us that the quietest infrastructure may be the most important infrastructure — because it exists for the day ordinary infrastructure cannot be trusted.</p><p>Do not ask only whether the backup is there.</p><p>Ask who can restore it after everything familiar is gone.</p><p>Production judgment is recognizing the recovery dependency that normal traffic will never exercise for you.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b28af8b5b8ff" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-had-backups-in-three-regions-the-restore-needed-one-deleted-iam-role-b28af8b5b8ff">We Had Backups in Three Regions. The Restore Needed One Deleted IAM Role.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RabbitMQ vs SQS: The Queue Wasn’t the Hard Part. One Poison Message Was.]]></title>
            <link>https://blog.stackademic.com/rabbitmq-vs-sqs-the-queue-wasnt-the-hard-part-one-poison-message-was-cf22a284ef5f?source=rss-8891a9734cd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf22a284ef5f</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Dev Side Hustle]]></dc:creator>
            <pubDate>Sat, 29 Aug 2026 00:06:02 GMT</pubDate>
            <atom:updated>2026-08-29T00:06:02.906Z</atom:updated>
            <content:encoded><![CDATA[<h4>Throughput looked fine in both systems until one payload could never succeed and retries became the workload.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m2JcnCHzDD9fwfCDa_eEpg.png" /></figure><p>The queue had plenty of capacity.</p><p>Consumers were healthy. CPU was low. The incoming message rate had not changed.</p><p>Yet useful work was slowing down because one malformed message kept returning like a bad coin in a vending machine.</p><p>It failed, became visible again, was received by another consumer, failed again, and generated more noise each time. Our dashboards showed activity. Customers saw delay.</p><p>RabbitMQ versus SQS had looked like a platform decision. The poison message exposed it as a recovery-design decision.</p><h3>RabbitMQ gave us control close to the broker</h3><p>RabbitMQ made routing expressive.</p><p>Exchanges, queues, bindings, acknowledgments, priorities, and dead-lettering let us model sophisticated delivery paths. When the topology matched the business, it was elegant.</p><p>That control also meant topology was part of the application’s operational behavior.</p><p>An engineer needed to understand:</p><p>→ When a message was acknowledged.</p><p>→ Whether rejection requeued it.</p><p>→ Which exchange received dead letters.</p><p>→ Whether a retry queue delayed delivery or created a loop.</p><p>→ How consumer prefetch affected fairness and memory.</p><p>RabbitMQ let us build the recovery machine. It did not stop us from building it badly.</p><h3>SQS made fewer broker decisions ours</h3><p>SQS narrowed the model.</p><p>Receive a message. Hide it for a visibility timeout. Delete it after success. Allow it to return if processing does not finish. Move it to a dead-letter queue after enough receives.</p><p>For AWS-native systems, that simplicity reduced the amount of broker infrastructure we owned. Scaling and durability conversations were largely delegated to the service.</p><p>But simpler did not mean automatic.</p><p>A visibility timeout shorter than processing time produced duplicate work. A timeout that was too long delayed recovery after a dead consumer. A redrive count chosen without understanding failure modes either buried messages too quickly or let poison messages circulate too long.</p><p>SQS removed knobs. It did not remove consequences.</p><h3>We were retrying permanent failures</h3><p>The poison payload violated a rule the producer should have enforced. No number of immediate retries could make it valid.</p><p>Our consumer treated every exception the same:</p><pre>processing failed → retry</pre><p>We replaced that with a failure taxonomy:</p><p>→ Transient dependency failure: retry with backoff and jitter.</p><p>→ Rate limit: retry after the provider’s recovery window.</p><p>→ Invalid payload: dead-letter immediately with the validation reason.</p><p>→ Unknown bug: limited retries, then quarantine with full context.</p><p>→ Ambiguous side effect: reconcile before any retry.</p><p>That classification mattered more than the broker.</p><p>Retries are useful only when time can change the outcome. Otherwise they convert one bad message into repeated load.</p><p>Duplicate delivery, poison events, consumer lag, replay, and backpressure keep appearing because they are properties of the architecture, not defects unique to one broker. <a href="https://yusufseyitoglu.gumroad.com/l/event-driven-systems"><strong>Event-Driven Systems in Production</strong></a> is the field manual for recognizing those patterns before the queue looks busy but stops creating value.</p><h3>The DLQ was not a destination</h3><p>Both designs could move failed messages aside. Our next mistake was treating that as completion.</p><p>A dead-letter queue without an operating process is a storage unit for forgotten customer problems.</p><p>We added:</p><ol><li>An alert based on age and arrival rate, not merely queue depth.</li><li>A view showing payload, producer version, failure class, and first error.</li><li>A replay tool that preserved the original identity.</li><li>A quarantine path for messages requiring manual correction.</li><li>A rule that replay could not bypass idempotency.</li></ol><p>We also tested the replay tool before an incident. A recovery mechanism first used under pressure is another untested production system.</p><h3>The broker did not choose our semantics</h3><p>RabbitMQ fit the system that needed rich routing, low-latency broker behavior, and a team willing to own its topology and operations.</p><p>SQS fit the system that valued a smaller AWS-managed surface and could work comfortably within its delivery model.</p><p>The poison message behaved badly in both until the application learned the difference between temporary and permanent failure.</p><p>That is the part comparison articles often hide. Infrastructure can deliver a message again. It cannot decide whether doing so is safe, useful, or expensive.</p><p>Choose the broker after answering four questions:</p><p>→ What may be delivered twice?</p><p>→ Which failures improve with time?</p><p>→ Who empties the DLQ?</p><p>→ How does replay preserve business correctness?</p><p>If those answers are missing, the queue is only storing uncertainty faster.</p><p>Build the event path yourself and decide where deduplication, quarantine, and recovery belong before adding another consumer.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf22a284ef5f" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/rabbitmq-vs-sqs-the-queue-wasnt-the-hard-part-one-poison-message-was-cf22a284ef5f">RabbitMQ vs SQS: The Queue Wasn’t the Hard Part. One Poison Message Was.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>