<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bug to Solution on Medium]]></title>
        <description><![CDATA[Stories by Bug to Solution on Medium]]></description>
        <link>https://medium.com/@BugToSolution?source=rss-5ecd0070ed87------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6bumuD_nTGP4wnyWr9-Dzw.png</url>
            <title>Stories by Bug to Solution on Medium</title>
            <link>https://medium.com/@BugToSolution?source=rss-5ecd0070ed87------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:00:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@BugToSolution/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deno Is Joining Cloudflare. Can Your App Actually Leave Its Cloud?]]></title>
            <link>https://medium.com/@BugToSolution/deno-is-joining-cloudflare-can-your-app-actually-leave-its-cloud-0f6ec43f3cf0?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f6ec43f3cf0</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 06:35:06 GMT</pubDate>
            <atom:updated>2026-10-26T06:35:06.856Z</atom:updated>
            <content:encoded><![CDATA[<h4>An open runtime gives you a door. It does not tell you what happens to everything connected to it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qHTaUvui1OhR6F0K4L1kw.png" /></figure><p>“We can self-host it later.”</p><p>That sentence has approved a remarkable amount of architecture without anyone having to demonstrate the word “can.”</p><p>It sounds responsible. There is an open-source runtime. There is a repository. Somewhere in the design document, an exit strategy exists.</p><p>Then someone asks for a second deployment outside the original platform, and the application turns out to be more than its executable.</p><p>On October 9, the Deno team announced that it was joining Cloudflare. The companies described work to bring celld and workerd together and improve self-hosting for the Workers programming model, including Durable Objects. That is a significant direction, not evidence that every managed-platform capability is already interchangeable everywhere. <a href="https://blog.cloudflare.com/deno-joins-cloudflare/">The announcement explains the intended work.</a></p><p>The useful question is not whether this makes one runtime better than another.</p><p>It is whether your exit plan describes an application that can move, or merely code that can start.</p><h3>The process boots. The product does not.</h3><p>Picture a collaborative planning app. Its HTTP routes run locally. The health endpoint returns a reassuring response. A container image builds successfully.</p><p>By the usual portability demonstration, the app has escaped.</p><p>Now open two browsers and edit the same plan. Ask where their shared state lives. Restart the process holding an active session. Deliver a scheduled reminder. Reconnect a client that missed an update.</p><p>This is a design exercise, not a report of a failed migration. Its point is that running the request handler proves a narrower claim than running the product.</p><p>A managed service may have been doing placement, persistence, routing, scheduling, identity, or recovery behind an API the application developer barely thought about.</p><p>Replacing the runtime does not automatically replace those responsibilities.</p><p>This is not an argument against managed primitives. They can save a small team an enormous amount of work. It is an argument against describing that saved work as though it never existed.</p><p>The exit cost includes deciding who will do it next.</p><h3>Open source is a property. Portability is a relationship.</h3><p>A component can be open source while your application is tightly coupled to the environment around it.</p><p>Conversely, an application can use a proprietary service through a sufficiently narrow boundary that replacing it is manageable.</p><p>The practical difference is where the assumptions sit.</p><p>If every feature reaches directly into platform-specific storage, identity, and scheduling behavior, the platform is part of the application’s design. A wrapper around one SDK call does not change that.</p><p>If a feature’s important rules are explicit, and the platform adapter performs a limited, understood job, there may be a smaller migration surface.</p><p>There is a cost to building that separation too. Over-abstracting every service on the first day can consume effort the product needs elsewhere.</p><p>The sensible position is neither “wrap everything” nor “lock-in is imaginary.” It is to know which commitments you are accepting and which capabilities would be expensive to reproduce.</p><p>You can accept a dependency deliberately. You cannot price a dependency you keep pretending is only a package import.</p><h3>The data needs an exit strategy of its own</h3><p>For the planning app, moving the code is the easy part of explaining a migration.</p><p>What represents an authoritative edit? Can it be exported? Does the replacement preserve ordering and conflict behavior? What happens to sessions active during the move?</p><p>An export file answers where some bytes can go. It does not establish that the destination interprets them the same way.</p><p>Even an uncomplicated migration has operational choices. Is there a pause in writes? Can old and new deployments coexist? Which side owns a scheduled job during the transition?</p><p>The cost of these decisions is not captured by comparing monthly hosting prices.</p><p>Before announcing that a system is portable, I would want a small end-to-end demonstration using representative, non-sensitive data. One customer workflow should complete outside the original platform, including the uninteresting parts after the first successful response.</p><p>Then interrupt it.</p><p>Restart something. Disconnect something. Make one dependency temporarily unavailable. Find out which recovery obligations have followed you out of the managed environment.</p><p>That demonstration is more useful than a theoretically universal infrastructure diagram.</p><h3>You may rationally decide not to leave</h3><p>There is a strange tendency to treat every dependency as a mistake waiting to be exposed.</p><p>For a small team, rebuilding a provider’s operating capabilities can be much riskier than accepting the provider relationship. Owning the servers is not the same as having the staff, knowledge, or budget to operate the replacement well.</p><p>The point of an exit exercise is not to force a migration. It is to turn an assumption into a decision.</p><p>Perhaps the result is that staying is clearly cheaper. Perhaps one proprietary component is doing nearly all the coupling. Perhaps the product needs a regulatory or customer-specific deployment option that makes the work unavoidable.</p><p>Each answer is better than “we could probably move it.”</p><p>If that sentence appears in your own architecture review, the decision-making and distributed-systems material in <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library">The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</a> is a useful next layer. Use it to examine the operating consequences behind the diagram before a migration, customer requirement, or outage forces the decision for you.</p><p>You do not need a second production platform tomorrow. You need an honest account of what a second platform would have to provide.</p><h3>The exit plan has to include a person</h3><p>After the technical questions, ask the least fashionable one: who would own the replacement?</p><p>Not which team name fits in a box. Who handles a failed restore, a storage incident, a new release, or a capacity problem when the managed service is no longer doing that work?</p><p>If the answer is “the same developer who runs everything else,” the infrastructure proposal has acquired a staffing dependency.</p><p>That dependency belongs in the decision. It is not an implementation detail.</p><p>The Deno–Cloudflare announcement is interesting because it addresses a genuine desire to use distributed application primitives in more places. Whether that develops into the option your application needs should be evaluated against the capabilities available when you make the decision.</p><p>An announcement can change what is worth testing. It should not be converted into a migration guarantee.</p><p>The next time someone says an application can leave its cloud, ask them to name one user journey that has done so.</p><p>Not a hello-world endpoint. Not a process that starts. A journey that writes, waits, reconnects, and recovers.</p><p>That is the difference between an open door and a usable exit.</p><p>You can practice identifying those dependencies before you have a migration deadline. Froquiz’s System Design format gives you a product brief and lets you build the connected architecture, so the platform choice has to make sense alongside the rest of the system.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f6ec43f3cf0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Alert Fired. The On-Call Never Received It.]]></title>
            <link>https://blog.stackademic.com/the-alert-fired-the-on-call-never-received-it-4ca4ea4cd585?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ca4ea4cd585</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:31:43 GMT</pubDate>
            <atom:updated>2026-09-03T17:31:43.883Z</atom:updated>
            <content:encoded><![CDATA[<h4>Monitoring knew production was down. The human response system did not.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RyTjcEFkPj6ZxL3ygDCDYA.png" /></figure><p>The status check detected the outage within minutes.</p><p>The incident event existed. The dashboard was red. The alert record showed a clean timestamp and the correct severity.</p><p>Nobody woke up.</p><p>Hours later, a customer message reached someone who happened to check their phone. By then, the technical failure was no longer the most embarrassing part of the incident.</p><p>Monitoring had worked exactly as designed. Notification had quietly failed outside the part of the system anyone monitored.</p><h3>An alert is not a page</h3><p>Teams often use these words as if they describe the same event:</p><p>→ a check failed</p><p>→ an alert rule fired</p><p>→ an incident was created</p><p>→ a notification was delivered</p><p>→ a human acknowledged it</p><p>They are five separate transitions.</p><p>Our dashboards proved only the first two. The alerting platform accepted the event, evaluated the routing rule, and marked the notification workflow successful. A stale schedule and an invalid integration meant no active person actually received a page.</p><p>The green “sent” state described the platform’s action, not the recipient’s outcome.</p><p>This is the same mistake engineers make when an API returns 202 Accepted and everyone talks as though the work finished. Acceptance is not completion. Dispatch is not delivery. Delivery is not acknowledgment.</p><h3>The backup path shared the same failure</h3><p>The escalation policy looked robust on paper. Primary on-call, secondary on-call, then the engineering manager.</p><p>All three routes depended on the same configuration source and notification integration. When the primary route failed, escalation repeated the same broken action with a different name attached.</p><p>Email was listed as a fallback, but the message landed in a filtered folder nobody monitored overnight. Chat was useful during incidents, yet the incident channel itself was created only after acknowledgment. There was no independent path capable of noticing that acknowledgment never happened.</p><p>Redundancy inside one failure domain is decoration.</p><p>The organization had redundant people but one delivery mechanism.</p><h3>We tested the human loop</h3><p>Recovery started with the service, but prevention started with a real page.</p><p>A scheduled test alert now traverses the complete path. It is generated by a synthetic check, evaluated by the real rules, routed through the live schedule, delivered to a device, and explicitly acknowledged. Failure at any stage creates a separate operational event through an independent channel.</p><p>We track time between each transition:</p><p>→ check failure to alert creation</p><p>→ alert creation to notification dispatch</p><p>→ dispatch to device delivery</p><p>→ delivery to acknowledgment</p><p>→ acknowledgment to incident action</p><p>The last two are human latency, and they matter. A perfect metric pipeline cannot compensate for an unreachable on-call engineer or a schedule that has not been reviewed since a team reorganization.</p><p>Escalation also moved across failure domains. If the paging provider cannot confirm delivery, a separate service uses a different channel. Critical customer-impact checks can trigger a provider-independent external monitor.</p><p>The costliest incidents are often not exotic outages. They are ordinary failures combined with one broken assumption about who will notice. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> is designed to build that recognition: trace the real failure chain, identify the missing evidence, recover safely, and add the control that closes the gap rather than polishing the dashboard.</p><h3>Ownership became executable</h3><p>An on-call schedule is production configuration. It receives review, validation, and expiry checks like any other critical routing table.</p><p>Team changes can no longer leave an empty rotation. Integrations have health status and credential-expiration warnings. A route that points to an inactive user or unacknowledged device is detected before an incident needs it.</p><p>Runbooks include a communication fallback that does not assume the primary incident tooling works. The contact tree is short enough to use under pressure and tested often enough that people recognize it.</p><p>We also changed what the service health dashboard displayed. “Alert active” was insufficient. For critical incidents, it showed whether a page had been delivered and acknowledged, plus the current escalation stage. Operations could distinguish a technical outage under active response from an outage silently waiting for a human.</p><p>No automation can guarantee that a tired person makes the right decision at 03:00. It can guarantee that the decision reaches a real person.</p><h3>The final dependency is attention</h3><p>Engineers spend months making databases redundant, queues durable, and services self-healing. Then the entire response plan depends on one notification leaving one vendor and appearing on one muted phone.</p><p>That dependency deserves the same suspicion as any single point of failure.</p><p>The incident changed one sentence in our review process. We stopped asking, “Will this alert fire?”</p><p>We ask, “How do we know a capable human is now acting on it?”</p><p>Everything before that is instrumentation. Response begins only when attention arrives.</p><p>Senior production judgment lives in these gaps between technically successful steps and real outcomes. Froquiz’s scenarios let you practice following those chains under pressure, when the obvious green status is often the least useful fact on the screen.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ca4ea4cd585" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-alert-fired-the-on-call-never-received-it-4ca4ea4cd585">The Alert Fired. The On-Call Never Received It.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Renewed the TLS Certificate. One Load Balancer Served the Expired One for Six Hours.]]></title>
            <link>https://medium.com/@BugToSolution/we-renewed-the-tls-certificate-one-load-balancer-served-the-expired-one-for-six-hours-09c34b837d90?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/09c34b837d90</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:05:07 GMT</pubDate>
            <atom:updated>2026-09-03T17:05:07.915Z</atom:updated>
            <content:encoded><![CDATA[<h4>The renewal job succeeded, the dashboard turned green, and one forgotten termination point kept rejecting customers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDJccFZBHLghOMkjOHIeIQ.png" /></figure><p>The certificate had been renewed three days before expiry.</p><p>The automation completed without errors. The new certificate appeared in the secret store. The primary domain passed the external check, and the expiry dashboard moved from red to green.</p><p>At midnight, nothing happened.</p><p>At 06:17, support reported that a small group of customers could no longer open the application. Most engineers could load it normally. Synthetic monitoring was green. The certificate in the repository was valid for another ninety days.</p><p>One regional load balancer was still serving the old certificate.</p><p>The renewal had succeeded. Distribution had not.</p><h3>We monitored the certificate, not every place that served it</h3><p>Our architecture diagram showed one public endpoint. The request path contained four TLS termination points: the CDN, two regional load balancers, and an internal gateway used by an older mobile route.</p><p>They did not all load certificates the same way.</p><p>The CDN pulled from the managed store automatically. One load balancer watched a synchronized secret. The other mounted a file that changed on disk but required a reload before new connections used it. The internal gateway had its own deployment process.</p><p>Our dashboard checked the canonical domain from one network location. That request consistently reached the CDN and proved only that the CDN had the new certificate.</p><p>The green signal was accurate and incomplete.</p><p>When customers reached the affected region, their TLS handshake ended before an HTTP request existed. Application logs stayed empty. API latency stayed normal because rejected clients never reached the API.</p><h3>The first diagnosis looked in the wrong layer</h3><p>Because only some customers were affected, the initial suspicion was DNS propagation, an ISP problem, or a stale browser cache. Engineers asked users to retry and switch networks.</p><p>The decisive evidence came from inspecting the certificate presented by each resolved edge address, not the certificate stored in our control plane. One address returned the new serial number. Another returned the expired serial.</p><p>That changed the incident from “some clients cannot connect” to “one termination point did not reload.”</p><p>We drained the affected load balancer, reloaded its listener, confirmed the certificate chain externally, and restored traffic gradually. Replacing the secret again would not have helped; the correct bytes were already present.</p><p>The difference between desired state and served state was the entire incident.</p><h3>Renewal and activation became separate steps</h3><p>Our automation had one success condition: a certificate authority issued a replacement and the secret store accepted it. We divided that into four observable stages:</p><ol><li>A new certificate was issued.</li><li>Every target received the new material.</li><li>Every listener activated it for new connections.</li><li>External probes observed the expected serial and chain.</li></ol><p>A certificate was no longer “renewed” after stage one. The workflow stayed incomplete until the serving endpoints proved stage four.</p><p>We also inventoried termination points by hostname, region, listener, owner, reload mechanism, and expected certificate fingerprint. That inventory mattered more than the renewal script. Automation cannot update an endpoint nobody remembers exists.</p><h3>Recovery had to protect existing connections</h3><p>The obvious fix was to restart every load balancer immediately. That would have terminated healthy long-lived connections and expanded a regional certificate incident into a global availability event.</p><p>Instead, we removed one target from rotation, reloaded it, established fresh TLS connections, and returned it before touching the next. We checked both certificate validity and application behavior at each step.</p><p>Certificate incidents encourage fast, broad action because the expiry date feels absolute. The safer recovery still respects blast radius.</p><p>This pattern — correct control-plane state, wrong served state — is behind many expensive incidents. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> helps build the response habit the dashboard did not provide: identify the actual failure boundary, confirm it from the customer path, recover without disrupting healthy capacity, and install a prevention check that observes reality rather than intent.</p><h3>We tested from the outside inward</h3><p>The replacement monitor connected to every advertised endpoint from more than one region. It recorded serial number, issuer, chain validity, hostname match, and days remaining. A mismatch between endpoints alerted even when every certificate was individually valid.</p><p>The alert threshold also moved earlier. The first warning created a ticket with an owner. The later warning paged only if activation remained incomplete. This prevented noisy expiry alerts from training the team to ignore the one that mattered.</p><p>During deployment tests, we rotated a non-production certificate and verified that every listener served the new serial without a global restart. The test exercised delivery and activation, not merely issuance.</p><p>We added one more check that initially seemed excessive: compare the certificate observed through DNS with the certificate observed by connecting directly to each known address while still sending the correct server name. That separated routing from TLS activation. If the public path was healthy but one address was stale, we could repair the target before normal routing or failover exposed customers to it.</p><p>The inventory tracked chain changes, not just leaf expiry. A fresh leaf certificate paired with an incomplete intermediate chain can fail only for certain clients, producing the same “works for us” trap. Validation used several client profiles instead of assuming one modern browser represented the customer base.</p><p>Failover tests included the supposedly passive region. Standby infrastructure receives too little real traffic to reveal stale certificates until the day it becomes primary. Rotation had to prove every failover destination, not only every currently busy endpoint.</p><h3>Green automation can still leave brown infrastructure</h3><p>The renewal job had not lied. It reported the operation it owned. We had allowed its narrow success to represent an end-to-end guarantee.</p><p>Customers do not connect to a secret store. They connect to a particular hostname, address, region, and TLS listener. That is where certificate correctness must be measured.</p><p>The final prevention was not “remember to reload.” It was eliminating memory from the process: inventory every termination point, automate activation, verify externally, and make partial distribution visible before expiry.</p><p>Production judgment often begins where a successful automation report ends. Froquiz’s senior scenarios practice deciding what to verify next when the control plane is green and users still cannot connect.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09c34b837d90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fleet Availability Was 99.9%. One Region Was Completely Down.]]></title>
            <link>https://blog.stackademic.com/fleet-availability-was-99-9-one-region-was-completely-down-4bea94d03e23?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bea94d03e23</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:16:10 GMT</pubDate>
            <atom:updated>2026-09-03T11:25:25.894Z</atom:updated>
            <content:encoded><![CDATA[<h4>The global average turned a total customer outage into a rounding error.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jpJAMjRtJ73xq_t65djeMw.png" /></figure><p>The executive dashboard showed 99.9 percent availability.</p><p>The on-call engineer saw no page. Global request success remained above the service objective. Capacity in the largest regions absorbed most traffic normally.</p><p>Every customer pinned to one smaller region was receiving errors.</p><p>That region represented less than one tenth of total traffic. A complete failure there moved the global average by too little to cross the alert threshold.</p><p>For affected customers, availability was not 99.9 percent. It was zero.</p><p>The metric was mathematically correct and operationally useless.</p><h3>Aggregation erased the failure domain</h3><p>Our alert summed successful requests across the fleet and divided by total requests. High-volume healthy regions dominated the result.</p><p>The architecture isolated regions to limit blast radius. Monitoring recombined them into one number and removed the isolation from our view.</p><p>The first signal came from support tickets sharing geography. Once success was grouped by region, the outage was obvious: one line had fallen vertically while the global line barely moved.</p><p>We had built fault isolation and forgotten observability isolation.</p><h3>Low traffic did not mean low importance</h3><p>The failed region contained fewer users but several high-value enterprise accounts with residency requirements. Traffic could not simply move elsewhere without violating data and contractual boundaries.</p><p>Request share therefore underestimated business impact. A region with 0.08 percent of requests could hold more revenue and urgency than its volume suggested.</p><p>We declared the regional incident, stopped relying on the global SLO, and restored the failed dependency from within that region. Customer communication used affected accounts and workflows, not fleet percentage.</p><p>The response changed when impact was counted in customers and business functions instead of requests alone.</p><h3>Alerts needed both global and local conditions</h3><p>The replacement alerting model had two layers.</p><p>Global burn-rate alerts detected broad degradation. Per-region and per-critical-workflow alerts detected complete local failure even when traffic was small. Minimum-volume logic prevented noise but did not allow “no successful requests” to disappear as insufficient data.</p><p>We added an absence signal. If a normally active region stopped producing traffic entirely, the system did not interpret the empty denominator as healthy.</p><p>Regional synthetic checks exercised the public path and a critical transaction. Infrastructure health alone could not prove customers in that geography could complete work.</p><h3>Dashboards preserved dimensions during incidents</h3><p>The first incident view showed fleet status, then region, tenant class, endpoint, and version. Engineers could move from broad symptom to failure domain without rebuilding queries under pressure.</p><p>Cardinality was controlled through intentional dimensions rather than discarding the ones that represented isolation boundaries. Region and critical workflow were not optional labels.</p><p>We also recorded traffic rerouting. A region could look recovered because requests had been diverted, while its underlying capacity remained broken. Served traffic and regional readiness were separate signals.</p><p>Monitoring incidents are expensive because a correct aggregate can persuade teams not to investigate. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> develops the habit behind stronger response: identify who experiences the failure, preserve the relevant dimension, recover the actual boundary, and install an alert that cannot average a total outage away.</p><h3>We tested small, complete failures</h3><p>Chaos tests often degraded 10 or 20 percent of total traffic to prove alerts fired. We added the opposite scenario: destroy 100 percent of a tiny but real slice.</p><p>The test failed one region, one tenant tier, and one rarely used critical endpoint separately. Success meant detection, ownership, and impact calculation worked without waiting for global error rate.</p><p>SLO reviews compared request-weighted availability with customer-weighted and region-minimum views. No single number replaced judgment, but the set exposed who the average was hiding.</p><p>We added a maximum-impact guard: any region with complete failure for a sustained interval paged regardless of its traffic weight. For partial degradation, burn rate still determined urgency. This avoided paging on every transient probe while ensuring a small population could not remain at zero indefinitely.</p><p>Traffic data itself needed interpretation. During an outage, failed clients often stop retrying, so the affected region’s request share falls and makes the global average improve. We tracked expected baseline traffic and distinct active customers. Disappearing demand became evidence, not relief.</p><p>The status page adopted the same dimensions. A global “operational” badge was misleading when residency-bound customers had no alternative. Regional components and critical workflows communicated impact without claiming the entire platform was down.</p><p>Ownership followed failure domains too. Each region had an escalation target and recovery authority. Detection without someone empowered to act merely shortened the time to watching the outage.</p><p>Capacity planning included evacuation limits. If traffic could legally and technically move, the runbook knew how much the receiving regions could absorb. If it could not move, the regional recovery path had independent dependencies and tested access.</p><h3>Reliability belongs to the customer boundary</h3><p>Global averages remain useful for capacity planning and broad incidents. They are dangerous as the only statement of customer experience.</p><p>The final monitoring model followed architecture: if the system promised isolation by region, tenant, or workflow, health had to be visible at that same boundary.</p><p>We paired every fleet objective with a worst-segment view and a minimum-volume rule. That prevented tiny random samples from paging the team while still catching a complete outage in a quiet region. The dashboard showed both the numerator and denominator so traffic disappearance could not masquerade as recovery.</p><p>The same principle applied beyond geography. Enterprise tenants, payment methods, authentication providers, and mobile versions could each become their own invisible “region.” Whenever the system could fail independently, the health model needed to preserve that dimension long enough for someone to see it.</p><p>Aggregation still summarized the fleet, but it could no longer erase the customer standing inside the failure domain.</p><p>Froquiz’s senior scenarios practice that diagnosis: inspect region, tenant, endpoint, routing, or dependency when the fleet graph is green and a real group of customers cannot use the product.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bea94d03e23" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/fleet-availability-was-99-9-one-region-was-completely-down-4bea94d03e23">Fleet Availability Was 99.9%. One Region Was Completely Down.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Circuit Breaker Opened. Recovery Traffic Took the Service Down Again.]]></title>
            <link>https://medium.com/@BugToSolution/the-circuit-breaker-opened-recovery-traffic-took-the-service-down-again-b997961aa381?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/b997961aa381</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 21:59:38 GMT</pubDate>
            <atom:updated>2026-09-01T21:59:38.867Z</atom:updated>
            <content:encoded><![CDATA[<h4>We protected the dependency during failure and attacked it the moment the timer expired.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqbwZPkqjUFS1AaMqCixUw.png" /></figure><p>The circuit breaker saved us for 60 seconds.</p><p>Calls to the recommendation service were timing out. The breaker opened, requests failed fast, and checkout used a fallback. Threads stopped piling up. Latency recovered.</p><p>Exactly one minute later, every application instance decided the dependency deserved another chance.</p><p>Hundreds of test requests arrived together. The recovering service had just restarted with cold caches and an empty connection pool. It collapsed before the first success could close the circuit.</p><p>The breaker opened again.</p><p>For the next 45 minutes, production oscillated between protected failure and coordinated attack.</p><h3>The state machine was locally correct</h3><p>Each instance maintained its own breaker:</p><pre>CLOSED → failures exceed threshold → OPEN<br>OPEN → wait 60 seconds → HALF_OPEN<br>HALF_OPEN → test call succeeds → CLOSED</pre><p>The implementation matched the diagram.</p><p>The diagram showed one breaker.</p><p>Production had 120.</p><p>Every instance observed similar failures and opened within the same few seconds. Every timer used the same duration. Every instance entered half-open together and sent its permitted test traffic.</p><p>A single cautious probe multiplied by the fleet became a load test.</p><p>The dependency was not being given a chance to recover. It was being asked to prove recovery at full organizational scale.</p><h3>Cold recovery changed the capacity equation</h3><p>Before the incident, the recommendation service handled thousands of requests per second.</p><p>After restart, it could not.</p><p>JIT compilation had not warmed. Local caches were empty. Database pools were establishing connections. The first requests triggered expensive model and configuration loads. Autoscaling saw low historical demand because the breaker had removed traffic.</p><p>The service’s steady-state capacity was irrelevant during the first seconds of recovery.</p><p>Our breaker assumed availability was binary: unavailable, then available again.</p><p>Recovery was a ramp.</p><p>The test policy had to respect the capacity of that ramp, not the capacity of a warm fleet.</p><h3>We centralized permission, not implementation</h3><p>We did not move every circuit decision into one network service. That would create latency and a new dependency.</p><p>We coordinated the recovery budget.</p><p>Instances still failed fast locally, but half-open attempts consumed a small shared token allowance. Jitter spread timers. Only a limited number of probes could reach the recovering dependency at once. Successful probes increased the budget gradually rather than switching every caller directly to closed.</p><p>The dependency also published a recovery signal based on its own saturation. Callers could remain cautious even after endpoints became reachable.</p><p>Fallback traffic received a budget too. Our fallback had originally queried PostgreSQL, which meant protecting one service could overload another. Degradation was redesigned to be cheaper than the path it replaced.</p><h3>The metric was useful work during recovery</h3><p>Breaker state alone had hidden the oscillation.</p><p>Dashboards showed a predictable square wave: open, half-open, open. We added:</p><ul><li>half-open attempts across the fleet;</li><li>success rate and latency of recovery probes;</li><li>dependency saturation during the first minutes after return;</li><li>fallback cost and correctness;</li><li>traffic admitted versus rejected;</li><li>time from reachability to sustainable service.</li></ul><p>This separated “the port answers” from “the service can safely accept normal demand.”</p><p>During incidents, responders could lower the recovery budget without redeploying every caller. They could also keep the breaker open manually while fixing a known dependency issue instead of letting timers repeatedly test a system that was intentionally offline.</p><p>Circuit breakers are sold as a simple resilience pattern, but the costly failure often lives in the transition between states. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> focuses on those recurring production shapes: the confirming evidence, the recovery choice that avoids a second collapse, and the control that turns a familiar diagram into safe operating behavior.</p><h3>We rehearsed the return, not only the outage</h3><p>Chaos tests had verified that the breaker opened when the dependency failed.</p><p>They had not verified what happened when it returned.</p><p>We added recovery experiments:</p><ol><li>Fail the dependency under real caller concurrency.</li><li>Restart it with cold state.</li><li>Hold capacity below normal while half-open begins.</li><li>Measure admitted probes across all instances.</li><li>Verify traffic ramps without saturating the service or fallback.</li><li>Confirm a second failure does not create synchronized retries.</li></ol><p>The most revealing failures happened after the red phase. Once the dependency appeared green, safety mechanisms relaxed faster than the system could warm.</p><p>Recovery became a first-class state in our design rather than a timer between failure and normality.</p><h3>Half-open is a capacity decision</h3><p>The post-incident change that mattered most was a recovery budget expressed in requests per second, not a timeout expressed in seconds. The dependency team could publish a safe ramp: five probes, then twenty requests per second, then a measured increase while saturation stayed below a threshold. Callers no longer interpreted the same timer expiring as permission to rediscover capacity independently.</p><p>We added a second dashboard specifically for the return path. It showed half-open attempts, admitted recovery traffic, rejected traffic, downstream queue depth, and successful useful work. The normal availability dashboard had hidden the dangerous transition because “breaker closed” looked like success even when it was immediately creating another overload.</p><p>Opening the circuit had bought the dependency time. Closing it safely required coordination. Without that second half, the circuit breaker was not a recovery mechanism. It was only a pause button with a synchronized release.</p><p>The breaker had done its first job. It stopped callers from wasting resources on a failing dependency.</p><p>It failed its second job because we had never defined how much traffic recovery could tolerate.</p><p>An OPEN state is easy to understand. Send nothing.</p><p>A CLOSED state is also easy. Send normal traffic.</p><p>HALF_OPEN is where engineering judgment lives. How many requests? From how many instances? With what deadline? Which operations? How quickly does success expand the budget? What happens to fallback demand?</p><p>If the answer is “one request per instance after 60 seconds,” your fleet size is part of the retry policy whether you intended it or not.</p><p>The outage was not caused by lacking a circuit breaker.</p><p>It was caused by believing the pattern ended when the circuit opened.</p><p>Production judgment is often about the transition back to normal. Froquiz’s senior scenarios force that recovery decision while the obvious green signal is still too early to trust.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b997961aa381" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Interviewer Asked for “Exactly Once.” The Correct Answer Was to Challenge the Question.]]></title>
            <link>https://blog.stackademic.com/the-interviewer-asked-for-exactly-once-the-correct-answer-was-to-challenge-the-question-44333303bf3e?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/44333303bf3e</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 20:34:44 GMT</pubDate>
            <atom:updated>2026-09-03T11:11:31.891Z</atom:updated>
            <content:encoded><![CDATA[<h4>Delivery, processing, and business effects are three different guarantees hiding behind one phrase.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-UatvNdYqZ12BvJ9yVb4DA.png" /></figure><p>“The event must be processed exactly once.”</p><p>It sounds like a clear requirement.</p><p>No lost messages. No duplicates. One input, one effect.</p><p>The dangerous answer is to choose a broker that advertises exactly-once semantics and move on.</p><p>The stronger answer begins with a question:</p><p>“Exactly once where?”</p><h3>Delivery is not the business effect</h3><p>A broker controls messages, producer state, consumer positions, and sometimes transactions within its own boundary.</p><p>The business effect may live elsewhere:</p><ul><li>a row inserted in PostgreSQL;</li><li>a card charged through a payment API;</li><li>an email sent by a provider;</li><li>inventory decremented in another service;</li><li>a file written to object storage.</li></ul><p>If the consumer commits its offset and then crashes before the effect, the message may be lost from the application’s perspective.</p><p>If it performs the effect and crashes before committing the offset, the broker redelivers and the effect may repeat.</p><p>The broker cannot atomically control an arbitrary external system unless that system participates in the same transaction.</p><h3>Define the invariant first</h3><p>Most products do not literally need one execution.</p><p>They need one observable business outcome.</p><p>A payment may receive several transport attempts as long as one charge exists. An inventory projection may apply the same version repeatedly as long as the final quantity is correct. An email may be deduplicated by campaign and recipient.</p><p>Rewrite the requirement:</p><pre>One checkout may create at most one captured payment.<br>One event version may advance an aggregate once.<br>No confirmed order may disappear.<br>Duplicate delivery must not create duplicate external effects.</pre><p>Now the architecture can enforce something measurable.</p><h3>Idempotency moves the boundary</h3><p>Give each intended business action a stable identity.</p><p>The consumer stores or forwards that identity when it performs the effect. Repeated processing returns the existing result or produces the same state.</p><p>For database writes, use a unique constraint, processed-event table, version check, or upsert tied to the invariant. For external APIs, reuse a provider-supported idempotency key. For state transitions, reject events older than the current version.</p><p>Idempotency is not “ignore every duplicate message.”</p><p>Two messages with the same payload may represent two legitimate actions. Two different payloads may represent retries of one action. Identity comes from the business operation, not content hashing alone.</p><h3>Transactions have a scope</h3><p>Kafka-style transactions can atomically connect consumed offsets, state stores, and output topics inside the supported ecosystem. That is powerful for stream processing.</p><p>The guarantee does not automatically include a separate email provider or database transaction.</p><p>An outbox can atomically store business state and an event in one database. An inbox can deduplicate consumed events. A Saga can coordinate several local transactions and compensation.</p><p>Each pattern moves the uncertainty boundary; none abolishes partial failure.</p><p>The interview answer should draw that boundary and describe recovery:</p><pre>What is committed together?<br>What may repeat?<br>Where is identity stored?<br>How is unknown outcome reconciled?<br>Which effect cannot be compensated?</pre><p>That reasoning matters across payments, queues, webhooks, deployments, and long-running jobs.</p><h3>Offer a precise guarantee</h3><p>A strong final answer sounds like this:</p><p>“The broker may deliver at least once. Each event has a stable operation ID. The consumer applies the database transition and records the ID in one transaction. External calls reuse that ID as an idempotency key. Pending or unknown outcomes are reconciled. Therefore, duplicate delivery does not create duplicate business effects within the stated retention and provider guarantees.”</p><p>That statement has limits, evidence, and a recovery path.</p><p>Also explain the reversal condition. If the provider does not support idempotency or status lookup, automatic retry may be unsafe. The workflow needs manual reconciliation or a different integration contract.</p><p>Monitor duplicates instead of pretending they cannot occur:</p><ul><li>repeated delivery count;</li><li>deduplication hits;</li><li>idempotency conflicts;</li><li>unknown external outcomes;</li><li>reconciliation age;</li><li>compensation failures.</li></ul><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> builds the production judgment behind those repeated failure shapes — so a guarantee is evaluated at the system boundary where users actually experience it.</p><p>The interviewer asked for exactly once to see whether the phrase would end the discussion.</p><p>Challenging it is not avoiding the requirement.</p><p>Retention is part of the guarantee. If deduplication records expire after twenty-four hours but a webhook retries after three days, the same operation can become new again. Align idempotency windows with the longest realistic retry and replay path, or archive a durable business identity elsewhere.</p><p>Concurrency creates another edge. Two workers may receive the same operation simultaneously before either writes the deduplication record. Use a unique constraint or atomic conditional write; a read-then-insert check races under load.</p><p>Do not let deduplication hide conflicting requests. Reusing one operation ID with different amount, tenant, or destination should fail visibly rather than return the first result. Store a hash of invariant parameters and compare it on reuse.</p><p>Compensation is not erasure. Refunding a duplicate charge creates another financial event and may not restore fees, inventory, customer trust, or external notifications. Prevent duplicate irreversible effects before relying on a Saga to clean them up.</p><p>During an interview, name the test: crash after the effect but before acknowledgement, run two consumers concurrently, replay after retention expires, and return a timeout after remote success. If the invariant survives those four cases, the guarantee is more than vocabulary.</p><p>Also separate “processed” from “successfully applied.” A consumer can parse and acknowledge an event whose business precondition no longer holds. Preserve rejection reason and operation state so monitoring does not celebrate delivery while the customer outcome remains missing.</p><p>It is turning an impossible universal promise into a precise business invariant the architecture can protect.</p><p>Practice stating guarantees with scope, failure conditions, and recovery — not as features borrowed from a technology page.</p><p>That precision is what turns a fashionable promise into an architecture an on-call engineer can actually defend.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44333303bf3e" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-interviewer-asked-for-exactly-once-the-correct-answer-was-to-challenge-the-question-44333303bf3e">The Interviewer Asked for “Exactly Once.” The Correct Answer Was to Challenge the Question.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Added a Read Replica. Customers Started Seeing Their Old Passwords.]]></title>
            <link>https://medium.com/@BugToSolution/we-added-a-read-replica-customers-started-seeing-their-old-passwords-0e319f1b0174?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e319f1b0174</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:17:04 GMT</pubDate>
            <atom:updated>2026-08-31T14:17:04.541Z</atom:updated>
            <content:encoded><![CDATA[<h4>The security bug was a consistency decision wearing a performance label.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rfGBlyxWjxNE3h9gNs4m2A.png" /></figure><p>The replica reduced database load exactly as planned. Profile pages became faster. Read capacity increased. The migration dashboard was green.</p><p>The benchmark was excellent too. It had never included a write immediately followed by a security-sensitive read.</p><p>Then a customer changed their password, logged out, and immediately tried the old password on another device.</p><p>It worked.</p><p>Authentication reads had been routed to a replica. The password update committed on the primary, but the replica was several seconds behind.</p><p>We had optimized a read without defining how stale that read was allowed to be.</p><h3>“Read-only” did not mean low risk</h3><p>The routing rule was simple: writes go to the primary, reads go to replicas.</p><p>That rule ignored meaning.</p><p>Reading a blog post counter can tolerate delay. Reading whether a password is valid, an account is disabled, a permission was revoked, or a payment was captured may not.</p><p>Our authentication service treated every SELECT as equivalent. A replica lag spike turned a performance optimization into a temporary rollback of security state.</p><p>The same problem appeared in less dramatic forms:</p><ul><li>users updated an address and saw the old value;</li><li>support disabled an account that remained active briefly;</li><li>a newly granted permission was missing after redirect;</li><li>password reset tokens appeared unused after redemption.</li></ul><p>The database was eventually consistent. The product had never told users which truths were temporary.</p><h3>Session stickiness was only a partial fix</h3><p>We immediately returned all authentication and authorization reads to the primary.</p><p>For profile data, we considered routing a user to the primary for a short period after their write. This “read your writes” window can improve user experience, but it requires careful boundaries.</p><p>A cookie on one device does not protect a second device. A background worker does not share browser state. A security action may affect every session and service.</p><p>For strong invariants, the safe rule remained simple: read authoritative state from the primary or a system with an explicit consistency guarantee.</p><p>For lower-risk data, we carried a commit position after important writes. A subsequent read could wait until the replica replayed at least that position or fall back to the primary after a small budget.</p><p>That made consistency a deliberate protocol instead of a hopeful sleep.</p><p>We avoided a fixed “wait two seconds” workaround. Replication delay is a distribution, not a promise. Sleeping slows every healthy request and still fails during the exact lag spike that matters. Waiting on a known commit position connects the delay to evidence.</p><h3>Replica lag needed business meaning</h3><p>Our original alert fired when lag exceeded a generic threshold. Five seconds sounded small.</p><p>Five seconds is irrelevant for an analytics dashboard and enormous for password revocation.</p><p>We classified read paths:</p><pre>authoritative: authentication, permissions, financial state<br>read-your-writes: profile and user-visible configuration<br>bounded stale: catalog and operational summaries<br>eventually consistent: analytics and counters</pre><p>Each class had a routing and degradation policy.</p><p>We also monitored replay position and catch-up rate, not only wall-clock lag. During heavy writes, a replica could remain five seconds behind indefinitely. During recovery, a larger lag might be shrinking quickly.</p><p>The application logged the selected database role and observed replay position with each sensitive request. That let us prove whether stale reads caused a customer-visible outcome without exposing private data.</p><p>During an incident, a kill switch routes all reads to the primary. The switch has a capacity guard because correctness recovery can overload the authoritative database. We rehearse the move under realistic traffic instead of discovering the primary’s limit during a security event.</p><h3>Performance architecture is product behavior</h3><p>The replica project had been reviewed as infrastructure. Product and security teams saw it as capacity work.</p><p>But moving a read changes what a customer is allowed to observe after a write. That is product semantics.</p><p>We added consistency requirements to API contracts. Endpoints no longer said only what data they returned; they specified whether the result must include the caller’s latest write and which source was authoritative.</p><p>Load tests gained a sequence dimension:</p><p>→ update a password, then authenticate immediately from another session;</p><p>→ revoke a role while requests are in flight;</p><p>→ redeem a token, then attempt reuse through a replica path;</p><p>→ fail over while a security write is propagating.</p><p>These tests found assumptions that normal read benchmarks never could.</p><p>The broad lesson was uncomfortable: database topology leaks into user experience unless the application deliberately contains it.</p><p>Architecture decisions often look cheap on a diagram and become expensive when their consistency behavior reaches authentication, payments, queues, and recovery.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> continues at that judgment layer — recognizing which production trade-offs are harmless optimizations and which silently redefine correctness.</p><h3>The replica was not the villain</h3><p>We kept the replica. It served the workloads that could safely use it and protected the primary from reporting traffic.</p><p>We removed the universal read-routing abstraction.</p><p>Convenience had hidden a domain decision behind one database client method. Callers now choose a consistency class, and sensitive defaults favor authority over speed.</p><p>The old password worked for only a few seconds. That duration did not make the bug small. It revealed that our security model depended on replication being faster than human behavior.</p><p>Eventually consistent systems are not incorrect. They become incorrect when an application promises immediate truth and reads from a place that has not received it yet.</p><p>The question is never simply, “Can this query run on a replica?”</p><p>The useful question is, “What breaks if this answer is yesterday’s truth for five seconds?”</p><p>If the answer includes access, money, identity, or irreversible action, faster reads may be the least important outcome.</p><p>That trade-off belongs in system design, not in a hidden connection-pool default. Froquiz lets you practice making those architecture decisions while you can still redraw the arrows.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e319f1b0174" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Senior Engineer Interview Had No Coding Round. Half the Candidates Still Failed.]]></title>
            <link>https://ai.plainenglish.io/the-senior-engineer-interview-had-no-coding-round-half-the-candidates-still-failed-9442c0ba2cb4?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/9442c0ba2cb4</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:31:57 GMT</pubDate>
            <atom:updated>2026-09-09T01:13:14.438Z</atom:updated>
            <content:encoded><![CDATA[<h4>Nobody forgot syntax. They failed when the system was ambiguous, partially broken, and impossible to solve alone.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NMtEECD6qiflASjMnpGj4g.png" /></figure><p>We removed the coding round for one senior backend role.</p><p>Candidates could use documentation, search, and AI during the interview. We did not ask them to invert a tree or implement a cache from memory.</p><p>The pass rate did not improve.</p><p>Strong candidates failed on a production scenario with no single correct answer.</p><p>Payments were succeeding. Orders remained pending. Metrics disagreed. A deploy had happened two hours earlier.</p><p>The interview was not testing whether they could write code.</p><p>It was testing whether they could decide what to believe.</p><h3>Tools removed syntax and exposed judgment</h3><p>The candidate received logs, a simplified architecture, several dashboards, and access to an assistant.</p><p>Almost everyone generated a plausible list of causes.</p><p>Queue lag. Database locks. Webhook failure. Cache inconsistency. A bad deployment. Missing idempotency.</p><p>The weaker interviews stayed at the list.</p><p>The stronger candidates converted uncertainty into an investigation:</p><p>→ Define the customer-visible failure precisely.</p><p>→ Establish when it began.</p><p>→ Separate payment acceptance from order state transition.</p><p>→ Find the durable identifier connecting those systems.</p><p>→ Test one hypothesis with evidence.</p><p>AI made brainstorming nearly free. It did not choose the next observation whose result would eliminate half the possibilities.</p><h3>The best answer began with containment</h3><p>Several candidates jumped directly to root cause.</p><p>One asked whether customers could be charged twice while the team investigated.</p><p>That changed the conversation.</p><p>They proposed pausing automatic retries for ambiguous payment outcomes, preserving incoming events, and creating a reconciliation path before changing production code. Only then did they investigate why order updates had stopped.</p><p>This was not avoidance.</p><p>It was recognition that the incident’s largest risk might occur during recovery rather than during the original failure.</p><p>Senior engineers are paid for that ordering. Diagnose matters. Stop making the expensive state worse matters first.</p><h3>Communication was part of the technical system</h3><p>We gave candidates an interruption halfway through: support reported that only one payment provider appeared affected.</p><p>Some ignored it and continued their prepared plan. Some abandoned every prior hypothesis immediately. Strong candidates updated the probability of causes, stated what changed, and preserved evidence that still mattered.</p><p>They also communicated uncertainty.</p><p>“I do not know whether the webhook is lost or delayed. Here is how I would distinguish them.”</p><p>That sentence is more senior than a confident guess with the correct technology name.</p><p>Production work happens through teams, partial evidence, and decisions made before certainty arrives. The interview needed to reveal whether the candidate could make that uncertainty legible to other people.</p><p>That pattern recognition is difficult to demonstrate through algorithm puzzles and impossible to replace with a list generated during the call. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where I continue building the judgment behind those recurring production shapes.</p><h3>The missing layer is pattern recognition</h3><p>Nobody can memorize every incident.</p><p>Experienced engineers become faster because unfamiliar incidents resemble known shapes: accepted work with missing completion, retry without idempotency, healthy components with a broken relationship, state changed on one side but not another.</p><h3>We did not make the interview easier</h3><p>We made it closer to the job.</p><p>Candidates no longer lost points for forgetting a library function. They gained no points for producing code before understanding the failure.</p><p>The strongest interview ended without a finished fix. The candidate had protected customers, narrowed the fault to a missing event transition, identified the evidence required for reconciliation, and explained the next safe move.</p><p>That was enough.</p><p>Senior engineering is rarely a race to the first answer.</p><p>It is the ability to reduce uncertainty without increasing damage.</p><p>AI can supply commands, hypotheses, and implementation drafts. The person still owns the sequence — and the consequences of choosing the wrong one.</p><p>If your interview preparation is only code generation, you are training the part of the job that just became cheapest.</p><p>Practice the moment after the dashboard turns red and five plausible answers arrive at once.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9442c0ba2cb4" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/the-senior-engineer-interview-had-no-coding-round-half-the-candidates-still-failed-9442c0ba2cb4">The Senior Engineer Interview Had No Coding Round. Half the Candidates Still Failed.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Cut Our Cloud Bill by 42%. Then the First Traffic Spike Found What We Removed.]]></title>
            <link>https://medium.com/@BugToSolution/we-cut-our-cloud-bill-by-42-then-the-first-traffic-spike-found-what-we-removed-ef1c2d302786?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef1c2d302786</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:12:37 GMT</pubDate>
            <atom:updated>2026-08-30T02:12:37.315Z</atom:updated>
            <content:encoded><![CDATA[<h4>The savings were real. So was the reliability capacity hiding inside the “waste.”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eIWUf7MWUgp4lPndDLLRoA.png" /></figure><p>Finance loved the graph.</p><p>Our monthly infrastructure bill had fallen from $31,400 to $18,200.</p><p>We removed idle nodes, tightened autoscaling, shortened log retention, reduced database headroom, and deleted resources nobody could confidently explain.</p><p>For three quiet weeks, nothing changed except the invoice.</p><p>Then a partner campaign sent 4.6 times normal traffic in twelve minutes.</p><p>The system did not scale.</p><p>It queued, thrashed, and failed in exactly the capacity we had celebrated removing.</p><h3>“Unused” was doing several jobs</h3><p>Cloud dashboards make idle capacity look guilty.</p><p>A node at 18% CPU appears wasteful. A database with spare connections looks oversized. A queue consumer waiting for work looks like money doing nothing.</p><p>But production capacity has more than one purpose.</p><p>Some of it serves normal demand. Some absorbs burst. Some creates room for failure. Some exists so a deployment can temporarily run two versions. Some allows recovery work without competing with customer traffic.</p><p>We had labeled all of it idle.</p><p>The largest saving came from reducing our Kubernetes node floor. Average utilization improved immediately. During the spike, new pods became pending while the cluster requested nodes. Those nodes then waited for provisioning, image pulls, and readiness checks.</p><p>Autoscaling worked.</p><p>It was simply slower than the traffic.</p><h3>Every optimization moved pressure somewhere else</h3><p>The database tier had been sized for comfortable connection and memory headroom. We reduced it because CPU rarely exceeded 30%.</p><p>During the campaign, application instances multiplied. Each created a pool. Connection count hit the new ceiling before CPU became interesting.</p><p>Requests waited in the application, timed out, and retried. The retry traffic arrived while the original work was still running. Queue depth grew. Cache misses increased as new pods started cold. The database received more duplicate attempts precisely when it had less room.</p><p>No single change caused the outage.</p><p>The savings removed small buffers from five layers. Traffic connected them into one failure.</p><p>That is why cost optimization reviews are dangerous when each resource is evaluated alone. Systems fail through interactions. Bills are itemized by service.</p><h3>The cheapest architecture depends on time</h3><p>Our mistake was using monthly averages to make minute-level reliability decisions.</p><p>Average CPU could not tell us how fast capacity needed to appear. Average connection count could not describe the burst created by a rollout. Average queue depth erased the twelve-minute window that mattered.</p><p>We rebuilt the cost model around time:</p><p>→ How much demand can the current floor absorb immediately?</p><p>→ How long until the next unit of capacity is ready?</p><p>→ What happens during that delay?</p><p>→ Which layer sheds load first?</p><p>→ Can recovery happen while traffic remains elevated?</p><p>The question was no longer “How much is unused?”</p><p>It became “What failure is this spare capacity buying us time to survive?”</p><p>Cost incidents repeat because optimization removes a buffer without naming the failure that buffer contained. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those recurring trade-offs continue — beyond one cloud bill and one unusually honest traffic spike.</p><h3>We kept the savings and bought back the right slack</h3><p>We did not restore the old bill.</p><p>We restored deliberate headroom where response time mattered and removed waste where delay was acceptable.</p><p>The API tier kept a warm floor sized for the first traffic step. Batch workers scaled from zero because their queues tolerated waiting. Database pool sizes became a fleet-wide budget instead of a per-instance default. Images were pre-pulled on the capacity expected to receive burst traffic. Retries received bounded budgets so unavailable capacity did not create additional demand.</p><h3>The invoice was not wrong</h3><p>The 42% reduction proved we had waste.</p><p>The outage proved we had removed resilience with it.</p><p>Both conclusions can be true.</p><p>We eventually settled at a 29% saving with a much clearer capacity story. Finance received a smaller victory. Engineering received a system whose scaling delay, connection budget, and failure headroom could be explained before the next campaign.</p><p>The most expensive cloud resource is not always the idle one.</p><p>Sometimes it is the resource you delete because the last thirty days were calm.</p><p>Architecture interviews love steady-state sizing. Production asks the harder question: what happens during the minutes before new capacity exists?</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef1c2d302786" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB vs PostgreSQL: Schema Freedom Was Fast. Paying It Back Wasn’t.]]></title>
            <link>https://blog.stackademic.com/mongodb-vs-postgresql-schema-freedom-was-fast-paying-it-back-wasnt-e0552d0d12ef?source=rss-5ecd0070ed87------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0552d0d12ef</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Bug to Solution]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 22:58:02 GMT</pubDate>
            <atom:updated>2026-08-28T22:58:02.416Z</atom:updated>
            <content:encoded><![CDATA[<h4>We removed migration friction at the beginning and rediscovered it later inside every reader, index, and repair job.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kZbUi85yLHET2fGn-pi9og.png" /></figure><p>The field did not exist for every customer.</p><p>For some documents it was a string. For newer documents it was an object. For a small group imported years ago, it was an empty array.</p><p>The service handled all three — until a new index assumed it would not have to.</p><p>Nothing about this failure was dramatic. No database crashed. No deploy failed immediately. One background job simply slowed down, retried, and kept rediscovering data shapes nobody remembered creating.</p><p>That was the day our MongoDB versus PostgreSQL debate stopped being about developer velocity.</p><p>It became a debate about when a team wants to pay for change.</p><h3>MongoDB made the first version honest</h3><p>The product was changing quickly. Customer records had nested attributes, optional integration data, and features whose final shape nobody knew.</p><p>MongoDB matched that uncertainty.</p><p>We could store the object the application already used. New fields did not require a synchronized schema migration. Teams could ship independently without negotiating every addition with a central database owner.</p><p>That speed was real.</p><p>It was also easy to misunderstand.</p><p>“Schema flexible” became “schema optional” in conversation. Those are not the same thing. The schema still existed — it lived across application code, validation rules, indexes, serializers, old documents, and assumptions inside every query.</p><p>We had not removed the contract. We had distributed it.</p><h3>PostgreSQL made change visible</h3><p>PostgreSQL created friction earlier.</p><p>A column type, constraint, or relationship forced us to make a decision. Migrations had to be deployed safely. Locks and backfills had to be considered. Compatibility between old and new application versions was part of the release plan.</p><p>That could feel bureaucratic during rapid development.</p><p>But the database made disagreement visible. A NOT NULL constraint did not care that three services had different opinions about whether the field was required.</p><p>The cost arrived at the pull request rather than during a repair job months later.</p><p>PostgreSQL also gave us JSONB when part of the model genuinely needed flexibility. That did not make it MongoDB. It gave us a useful escape hatch inside a relational system, with the same danger: flexible fields become invisible contracts unless somebody governs them.</p><h3>The production difference was repair</h3><p>We deliberately corrupted copies of both datasets to test recovery work.</p><p>In PostgreSQL, invalid states were harder to create once constraints existed. The painful part was evolving those constraints without blocking a busy table.</p><p>In MongoDB, old shapes remained easy to carry. The painful part was discovering all of them before a new reader assumed one canonical form.</p><p>The repair plan for documents needed:</p><p>→ A query that found every legacy shape.</p><p>→ An idempotent backfill that could stop and resume.</p><p>→ Dual-read compatibility during migration.</p><p>→ Validation that prevented old shapes from returning.</p><p>→ Index changes coordinated with the data rewrite.</p><p>That is a migration. It just was not called one when the first field shipped.</p><p>The comparison finally became clear:</p><p>MongoDB let us defer structural decisions. PostgreSQL encouraged us to formalize them. Deferral was valuable while uncertainty was high. It became expensive after the model stabilized and years of shapes accumulated.</p><p>Architecture failures often begin as reasonable decisions whose reversal cost was never discussed. <a href="https://yusufseyitoglu.gumroad.com/l/architecture-under-pressure"><strong>Architecture Under Pressure</strong></a> carries that judgment into the other choices that become expensive only after the system grows around them.</p><h3>Choose the invariant before the database</h3><p>The wrong question is: “Do we want schema flexibility?”</p><p>Every real product changes.</p><p>Better questions are:</p><p>→ Which invariants must be true for every write?</p><p>→ Are relationships central to the product or incidental?</p><p>→ Will common access patterns fit inside aggregate documents?</p><p>→ Who owns backfills when an old shape stops being acceptable?</p><p>→ Can we observe the distribution of document versions in production?</p><p>If the product cannot tolerate two users pointing at a missing organization, enforcing that relationship has value. If each record is a self-contained, rapidly changing aggregate, forcing it through a rigid relational model may create ceremony without safety.</p><h3>Flexibility is debt only when nobody tracks it</h3><p>MongoDB was not the mistake. Our unowned schema evolution was.</p><p>PostgreSQL would not have saved us from careless migrations, bad indexes, or application code that ignored constraints. It would have made a different set of mistakes easier to see.</p><p>That distinction matters because teams love moral answers. “Document databases are bad” is comforting. “Our data contract had no owner” is actionable.</p><p>Choose MongoDB when aggregate-shaped data and controlled flexibility solve a real problem. Choose PostgreSQL when relationships, constraints, and varied querying are the center of the system.</p><p>In both cases, name the invariant. Version the change. Plan the repair before you need it.</p><p>A data model is not finished when the first write succeeds. Test it against the migration you will eventually regret.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0552d0d12ef" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/mongodb-vs-postgresql-schema-freedom-was-fast-paying-it-back-wasnt-e0552d0d12ef">MongoDB vs PostgreSQL: Schema Freedom Was Fast. Paying It Back Wasn’t.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>