<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Clean Code Journal on Medium]]></title>
        <description><![CDATA[Stories by Clean Code Journal on Medium]]></description>
        <link>https://medium.com/@tercanyldz?source=rss-103fbf8a115a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Or-QEM8ympJWg80O5S7tpA.png</url>
            <title>Stories by Clean Code Journal on Medium</title>
            <link>https://medium.com/@tercanyldz?source=rss-103fbf8a115a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:38:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tercanyldz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The WAF Blocked the Attack. It Also Blocked Every Older Mobile Client.]]></title>
            <link>https://medium.com/@tercanyldz/the-waf-blocked-the-attack-it-also-blocked-every-older-mobile-client-05eb099a7a6c?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/05eb099a7a6c</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:27:33 GMT</pubDate>
            <atom:updated>2026-09-03T17:27:33.431Z</atom:updated>
            <content:encoded><![CDATA[<h4>A security rule reached 100% effectiveness by treating legitimate production traffic as hostile.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n4RR4NpvwpnJI0Rt6_NLQQ.png" /></figure><p>The new WAF rule was created after a credential-stuffing campaign.</p><p>In simulation, it blocked the malicious payloads and allowed the browser test suite. Security deployed it globally. Attack traffic fell within minutes.</p><p>Mobile login failures rose at the same time.</p><p>Older versions of the app encoded one request field differently. Its value matched a pattern the managed rule associated with injection attempts. The WAF rejected the request before it reached authentication.</p><p>API error rate stayed normal because the API never saw those users. The security dashboard celebrated blocked requests. The mobile store filled with one-star reviews.</p><p>The control worked at the layer that measured it and failed at the product boundary.</p><h3>Blocked traffic had no application identity</h3><p>The WAF acted before normal authentication and request tracing. Rejected requests produced edge events with rule IDs, addresses, and user agents, but no application request ID.</p><p>Backend dashboards could not connect the rise in blocked traffic to login attempts. Support saw “network error.” Security saw successful mitigation.</p><p>The first useful analysis grouped WAF matches by client version and endpoint. One older mobile cohort accounted for nearly all new blocks on the login path.</p><p>The rule had not discovered a sudden attack from our customers. It had discovered a stable serialization behavior that browser tests did not produce.</p><h3>Global blocking skipped the evidence phase</h3><p>The rule moved directly from test traffic to enforcement. A safer rollout would have counted matches without blocking, sampled redacted payload characteristics, and compared them with known business traffic.</p><p>Security controls need canaries too.</p><p>We changed the rule to log-only for the affected signature, added a narrower rate and behavior control for the actual attack, and restored mobile login. We did not disable the entire WAF or publish the detection detail attackers could tune against.</p><p>The recovery preserved protection while removing the false-positive path.</p><h3>Exceptions were narrow and temporary</h3><p>The fastest permanent-looking fix was to exclude the login endpoint from the rule. That would have reopened the most attacked path.</p><p>Instead, the exception matched the specific client behavior, endpoint, field context, and known app versions. It had an owner and expiry condition tied to minimum supported version adoption.</p><p>Server-side validation still treated the input as untrusted. The WAF exception meant “allow this request to reach the application,” not “declare it safe.” Authentication throttling and account protections remained active.</p><p>We tracked exception traffic separately so a bypass could not silently become the new attack route.</p><h3>Security success included customer availability</h3><p>The WAF dashboard added allowed and blocked outcomes by endpoint, client family, region, and response path. A jump in blocks accompanied by a fall in successful logins became an incident signal, not a victory metric.</p><p>Synthetic tests exercised supported mobile request shapes through the public edge. Backend-only tests were insufficient because they bypassed the control under test.</p><p>Rule deployments used stages: observe, challenge or limit, narrow enforcement, then global enforcement after representative coverage. High-risk emergency rules could move faster but required explicit rollback signals.</p><p>Safety controls create expensive incidents when their false positives are invisible to the service they protect. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> develops the operational response: confirm who the control is blocking, preserve the necessary defense, restore legitimate traffic with the smallest exception, and add evidence before the next global rule change.</p><h3>We tested old clients, not only current code</h3><p>Mobile software remains in the field long after a web deploy. Customers delay updates, devices lose support, and enterprise-managed phones follow slow release schedules.</p><p>The compatibility suite included the oldest supported request encodings and replayed them against new WAF policies. Client telemetry showed the real installed-version distribution, not only the latest store release.</p><p>When a version became unsupported, the product communicated that transition intentionally. A security rule was not allowed to become an accidental forced upgrade with a generic network error.</p><p>We built a false-positive budget alongside the attack-blocking target. A rule could not be promoted only because it caught known malicious samples; it also needed evidence about legitimate match rate and customer outcomes. Security effectiveness without precision was an availability risk.</p><p>Emergency response gained a kill switch scoped to one rule and one path. Previously, the only fast options were leave the global rule active or disable the entire managed ruleset. Narrow rollback reduced the pressure to accept a dangerous all-or-nothing choice.</p><p>Client errors carried an edge correlation token safe to share with support. The app could display a retryable security-block message and support could locate the WAF event without exposing the matching signature. That connected the customer symptom to the layer that rejected it.</p><p>Rule ownership persisted after deployment. Managed rule updates could change behavior without an application release, so version changes were reviewed, observed in shadow mode, and tied to the same outcome dashboards. “No deploy” no longer removed the WAF from the suspect list.</p><h3>Blocking is a production side effect</h3><p>The rule did not modify a database, but it changed who could use the product. That made it production code in every meaningful sense.</p><p>The durable fix combined shadow evaluation, segmented metrics, representative clients, narrow exceptions, and a rollback path owned jointly by security and application teams.</p><p>Stopping attacks and serving customers were not competing dashboards anymore. They were two requirements of the same control.</p><p>The incident review also changed its definition of evidence. A falling request count no longer proved the attack had ended; blocked legitimate clients disappear before application metrics can count them. Edge decisions, support contacts, client versions, and completed user journeys had to be read together.</p><p>That gave the team a practical release rule: a security control could graduate from shadow mode only when it demonstrated both useful detection and an acceptable false-positive rate across the clients the business still supported.</p><p>Froquiz’s senior scenarios practice that decision: disable, narrow, observe, challenge, or preserve a rule when it is blocking both hostile and valuable traffic.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05eb099a7a6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Payment Succeeded. Our API Told the Customer to Try Again.]]></title>
            <link>https://python.plainenglish.io/the-payment-succeeded-our-api-told-the-customer-to-try-again-3f07ee5d7f29?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f07ee5d7f29</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 17:00:49 GMT</pubDate>
            <atom:updated>2026-09-03T17:00:49.570Z</atom:updated>
            <content:encoded><![CDATA[<h4>A timeout turned an unknown outcome into duplicate charges because we treated silence as failure.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dy3X3YtNdJ9OgiGtVdvF5w.png" /></figure><p>The customer clicked <strong>Pay</strong> once.</p><p>Our API waited eight seconds, gave up, and returned an error. The checkout page did exactly what we had taught it to do: it showed a red banner and invited the customer to try again.</p><p>The second attempt succeeded in 900 milliseconds.</p><p>So did the first one.</p><p>The payment provider had accepted the original request. Only its response was lost on the way back. We had confused “I did not receive an answer” with “nothing happened.” By the time support opened the ticket, the customer had two charges, one order, and a screenshot proving our product had explicitly asked for the second attempt.</p><p>The embarrassing part was not the timeout. Timeouts are ordinary.</p><p>The embarrassing part was that our system had no state for <strong>I don’t know</strong>.</p><h3>The response was missing, not the outcome</h3><p>Engineers are trained to split operations into success and failure.</p><p>That model works inside a single process. A function returns a value or throws. But the moment a request crosses a network, a third state appears:</p><p>→ The operation may have completed, but the caller never learned the result.</p><p>The connection can disappear after the provider commits the charge and before our socket receives the response. A reverse proxy can time out while the upstream continues working. The client can close the tab while the backend finishes the request.</p><p>From our API’s point of view, all three look like failure.</p><p>From the payment system’s point of view, the money moved.</p><p>We had built the retry around transport certainty. The business operation required outcome certainty.</p><p>Those are not the same thing.</p><h3>Our retry was reasonable in isolation</h3><p>The code was clean enough to survive review:</p><pre>try:<br>    charge = payments.create(amount, card_token)<br>except TimeoutError:<br>    raise CheckoutUnavailable(&quot;Please try again&quot;)</pre><p>There was no wild loop. No reckless ten-attempt policy. No background worker hammering the provider.</p><p>Just one human retry after a clear error message.</p><p>That was enough.</p><p>We had not attached a stable idempotency key to the payment attempt. Each click created a new business operation. The provider had no way to know that request two was the customer repeating request one rather than buying a second item.</p><p>The browser button had a loading state, but that only prevented two clicks in one page session. Refreshing, switching devices, a mobile reconnect, or our own service retry bypassed it.</p><p>UI protection made the happy path look safe. It did not make the operation idempotent.</p><h3>We stopped returning a lie</h3><p>The first fix was not another timeout value.</p><p>We created the payment attempt in our database before calling the provider. It received a stable identifier derived from the order and attempt number. That identifier became the idempotency key for every retry of the same intent.</p><pre>order_8412:payment_attempt_1</pre><p>If the request timed out, we did not tell the customer the payment had failed. We moved the attempt to UNKNOWN and reconciled it using the same key and provider reference.</p><p>The customer saw a less satisfying but more honest message:</p><p>We’re confirming your payment. Do not try again.</p><p>That state lasted seconds in normal cases. It was still part of the product, support tooling, metrics, and alerting. Unknown outcomes were no longer hidden inside an exception handler.</p><p>We also separated two actions that had previously shared one button:</p><p>→ Retry communication for the same payment intent.</p><p>→ Create a genuinely new payment attempt after confirmed failure.</p><p>Only the second action received a new idempotency key.</p><h3>The expensive failures live between normal states</h3><p>The duplicate charge was one incident pattern. The larger lesson was that our state machine had been designed around the responses we hoped to receive, not the outcomes the world could produce.</p><p>This pattern appears outside payments:</p><ul><li>a job finishes after the queue visibility timeout and runs again;</li><li>an inventory reservation commits before the response disappears;</li><li>a webhook is accepted, processed, and delivered again;</li><li>an email provider sends the message while the caller records failure;</li><li>a database transaction commits after the application loses its connection.</li></ul><p>The code often looks reasonable because every component behaves correctly on its own. The damage begins at the boundary between an irreversible side effect and an uncertain acknowledgement.</p><p>Most costly incidents are not novel. They are familiar failure shapes hiding behind different APIs. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> turns those shapes into incident cards with confirming evidence, recovery paths, and prevention controls — so the next unknown outcome becomes a state you recognize, not a customer discovering the pattern for you.</p><h3>Idempotency needed an operating policy</h3><p>Adding a key was necessary. It was not the end.</p><p>We defined how long keys remained valid, which request parameters had to match, and what happened when reconciliation outlived the provider’s retention window. We stored the provider’s result beside our attempt rather than reconstructing it from logs.</p><p>Alerts changed too.</p><p>We no longer paged only on payment errors. We watched the age and count of attempts stuck in UNKNOWN, the rate of duplicate-key replays, and mismatches between paid orders and captured payments.</p><p>Support received one safe action: reconcile the attempt. They could not blindly “run payment again” from an admin screen.</p><p>The incident had started because the system pressured both customers and operators to convert uncertainty into action. The repair gave uncertainty somewhere to live until evidence resolved it.</p><h3>A timeout is not a verdict</h3><p>The customer did nothing unreasonable. They followed the instruction on the screen.</p><p>The provider did nothing mysterious. It completed a request before the network response vanished.</p><p>Our mistake was pretending a distributed operation could always be reduced to success or failure at the moment the caller wanted an answer.</p><p>Now every side-effecting integration must answer three questions before launch:</p><ol><li>What stable identity represents one business intent?</li><li>How do we reconcile an outcome when the response is missing?</li><li>What does the user see while the answer is unknown?</li></ol><p>If those answers do not exist, the retry button is not recovery. It is a second attempt at reality.</p><p>This is the kind of decision that separates syntax knowledge from production judgment. A scenario can look finished while one missing state makes the entire flow unsafe.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f07ee5d7f29" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/the-payment-succeeded-our-api-told-the-customer-to-try-again-3f07ee5d7f29">The Payment Succeeded. Our API Told the Customer to Try Again.</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Fixed N+1 With Eager Loading. Memory Usage Tripled.]]></title>
            <link>https://ai.plainenglish.io/we-fixed-n-1-with-eager-loading-memory-usage-tripled-d31ea4661bfd?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d31ea4661bfd</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 22:11:47 GMT</pubDate>
            <atom:updated>2026-09-09T01:01:44.641Z</atom:updated>
            <content:encoded><![CDATA[<h4>Query count fell from 801 to one. The one query expanded into more data than the application could hold.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uSSfWGK09XZJtkvVKVV85g.png" /></figure><p>The optimization looked perfect in the pull request.</p><p>An endpoint loaded 800 orders and then fetched line items, discounts, and shipments one order at a time. The trace showed hundreds of small database calls. We replaced the lazy relationships with one eager-loaded query and watched the query count fall from 801 to one.</p><p>Latency improved in staging. The review was approved. The deploy reached production.</p><p>Then container memory climbed from 420 MB to 1.3 GB. Garbage collection ran almost continuously. Two pods were killed under normal traffic, and the remaining replicas became slower than the original N+1 implementation.</p><p>We had optimized the number everyone could see and ignored the shape of the result.</p><h3>One query can contain millions of repeated values</h3><p>The endpoint did not return 800 flat rows. Each order had multiple line items, discounts, and shipment records. Joining all three collections created a multiplication effect.</p><p>An order with 12 line items, 3 discounts, and 2 shipment events could appear 72 times in the database result. The order columns were repeated in every row. So were portions of the related objects. The wire payload grew, the database assembled a much larger result, and the ORM had to turn that expanded row set back into a smaller object graph.</p><p>The SQL was valid. The query plan was not obviously slow. The database returned rows quickly enough. The cost appeared in network transfer, row decoding, object allocation, identity-map bookkeeping, and garbage collection.</p><p>“One query” had sounded cheap because we were using request count as a proxy for work. It was a terrible proxy here.</p><p>The first useful measurement was not execution time. It was rows produced at each join and bytes returned to the application.</p><h3>The ORM was doing work the trace did not show</h3><p>Our application trace stopped the database span when the result arrived. That made the query look fast. It did not include the full cost of materializing thousands of repeated rows into objects.</p><p>The ORM needed to recognize that 72 physical rows described one order, reuse objects it had already created, populate multiple collections, and retain enough state to preserve relationship semantics. That work happened on the application side and consumed heap before serialization even began.</p><p>The endpoint then created response models from those objects, effectively holding two representations of much of the same data. Finally, JSON serialization allocated another large buffer.</p><p>The memory graph told the story the query counter had missed: database result, ORM graph, response graph, and encoded response overlapped long enough to exceed the container limit.</p><p>This was not evidence that eager loading was bad. It was evidence that loading strategy is a cardinality decision.</p><h3>We replaced one giant join with bounded work</h3><p>The correct solution was not to restore 801 queries. We split the load along relationship boundaries.</p><p>Orders were fetched in bounded pages. Related collections were loaded with IN queries for each page, avoiding the Cartesian multiplication between independent one-to-many relationships. The response selected only fields the client actually used. A summary field was computed in SQL instead of materializing every historical child object.</p><p>The final request used more than one query and far fewer than 801. More importantly, each query had a predictable upper bound.</p><p>We also challenged the endpoint contract. Did the client truly need 800 fully expanded orders in one response? It did not. The UI displayed 50 at a time and showed details only after a click. The API had been paying the maximum cost for data the user never saw.</p><p>Pagination and projections removed more risk than the clever loading option.</p><h3>Query count stopped being our performance score</h3><p>The new review checklist paired query count with returned rows, bytes transferred, peak heap, allocation rate, serialization time, and response size. Any optimization that improved one number while making another unbounded had to explain the trade.</p><p>We tested with production-like relationship distributions, not average seed data. In staging, most orders had two line items and no complex history. In production, a small number of long-lived business accounts had thousands of related records. Those accounts defined the memory ceiling.</p><p>This is why performance incidents survive apparently rigorous testing: the code path is present, but the data shape that activates its cost is missing.</p><p>The valuable habit is to ask, “What quantity can multiply here?” before celebrating a faster benchmark.</p><p>Performance failures rarely belong to a single layer. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> helps build the pattern recognition behind that question: connect the customer symptom to the hidden resource, choose a recovery that does not move the bottleneck, and add a prevention control that survives real production data.</p><h3>The endpoint needed a memory budget</h3><p>CPU and latency already had service objectives. Memory was treated as a container setting chosen after the application was built. We reversed that relationship.</p><p>The endpoint gained a maximum page size, maximum expansion depth, and response-size guard. Load tests tracked peak resident memory per request under concurrent traffic. A single pathological customer could no longer ask every replica to construct an unbounded object graph.</p><p>We also recorded a histogram of children per parent. That simple distribution exposed risk earlier than any ORM rule. A median of three line items was comforting; a 99.9th percentile of 8,000 was operationally decisive.</p><p>The original N+1 problem was real. Our first fix was also real — and dangerous because it converted visible database chatter into invisible application pressure.</p><p>Senior engineering was not choosing between lazy and eager loading. It was understanding where the work moved, how large it could become, and which bound kept one customer’s data shape from becoming everyone’s outage.</p><p>Froquiz’s production scenarios practice exactly that kind of trade-off: the obvious optimization is rarely the whole answer when the system is already under load.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d31ea4661bfd" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/we-fixed-n-1-with-eager-loading-memory-usage-tripled-d31ea4661bfd">We Fixed N+1 With Eager Loading. Memory Usage Tripled.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Canary Was Healthy. It Never Received the Customer That Broke Production.]]></title>
            <link>https://blog.stackademic.com/the-canary-was-healthy-it-never-received-the-customer-that-broke-production-994416bb8b0a?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/994416bb8b0a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 21:55:09 GMT</pubDate>
            <atom:updated>2026-09-03T11:24:14.907Z</atom:updated>
            <content:encoded><![CDATA[<h4>One percent of traffic tested the average path and completely missed the account with four million records.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lg9QLa93kJBSyMwItWqsow.png" /></figure><p>The canary served 1 percent of production traffic for thirty minutes.</p><p>Latency matched the old version. Error rate was zero. Memory stayed flat. The release controller promoted the build to the entire fleet.</p><p>Eight minutes later, every pod handling our largest customer began exhausting memory.</p><p>The new code loaded an account’s active rules into one in-memory structure. For most customers, that meant dozens of records. One enterprise tenant had four million historical rules because of an old migration and a unique retention contract.</p><p>The canary was healthy because random traffic never sent that tenant to it.</p><p>We had tested production volume without testing production shape.</p><h3>One percent was not a representative sample</h3><p>Traffic percentage sounded statistically safe. It was not tied to the dimension that activated the bug.</p><p>Requests were distributed by connection and region. The large tenant used a dedicated regional endpoint and sticky routing. None of its traffic entered the canary pool.</p><p>Our canary therefore answered: does this version work for ordinary tenants in the primary region? The deployment decision interpreted it as: does this version work for the production population?</p><p>Both statements used real customer traffic. Only one was true.</p><p>The first diagnostic clue was error concentration by tenant. Fleet-level memory graphs rose gradually, but grouping by account showed one customer producing nearly every large allocation.</p><h3>Average traffic hid tail-shaped risk</h3><p>The release changed a quantity that scaled with records per tenant. The relevant distribution was not requests per second; it was tenant size.</p><p>Average tenants had 120 rules. The median had fewer. The largest had four million. A benchmark built from averages could pass while the maximum exceeded memory by orders of magnitude.</p><p>We rolled back before more pods entered restart loops, then isolated the affected tenant’s requests and restored service for others. The customer received a degraded path that loaded rules in bounded pages while the permanent fix was prepared.</p><p>Adding memory to every pod would have increased cost and allowed the same unbounded structure to fail at a larger tenant. The problem was missing bounds, not an undersized container.</p><h3>Canary selection followed the failure hypothesis</h3><p>The repaired release plan selected cohorts deliberately.</p><p>If code changed serialization, the canary included large payloads. If it changed database access, it included tenants with deep histories and skewed relationships. If it changed permissions, it included complex roles. If it changed regional infrastructure, every region received a probe.</p><p>Random traffic remained useful for unknown unknowns. It stopped being the only evidence.</p><p>We maintained a privacy-safe set of workload characteristics rather than copying customer data into tests. The release system could route representative production requests or replay sanitized shapes without exposing content.</p><p>The canary had to encounter the edge that the code made expensive.</p><h3>Promotion used segmented health</h3><p>Fleet averages allowed healthy low-cost traffic to dilute catastrophic behavior in a small cohort. The dashboard added success, latency, memory, and allocation rate by tenant-size bucket, region, endpoint class, and application version.</p><p>Promotion required minimum sample coverage for the relevant buckets. If the largest cohort produced no canary traffic, the controller reported missing evidence instead of green health.</p><p>We also capped per-tenant resource use. Even if canary selection failed again, one account could degrade its own workload without taking every shared pod with it.</p><p>Canary incidents are dangerous because their evidence feels more trustworthy than staging. <a href="https://yusufseyitoglu.gumroad.com/l/production-incidents"><strong>30 Production Incidents That Cost $10K+ (And How to Prevent Them)</strong></a> helps develop the next question: what production characteristic activates this risk, did the canary actually see it, and what containment prevents one missed cohort from becoming a fleet-wide failure?</p><h3>We tested the release decision itself</h3><p>The new deployment test did not merely inject bad code. It injected biased traffic.</p><p>We created a canary that received plenty of requests but none from the dangerous cohort. The system had to refuse automatic promotion because coverage was incomplete. Another test sent one pathological tenant and verified that segmented health stopped the rollout before fleet average moved materially.</p><p>Rollback preserved version and cohort evidence so the team could explain why the canary looked green instead of dismissing the result as bad luck.</p><p>We then audited sticky routing itself. If the same customers always landed on the same version, a canary could run for hours without sampling a meaningful portion of the population. For read-only paths, controlled shadow evaluation let the new version observe representative requests without serving its answer. For state-changing paths, we used explicit tenant cohorts and rollback-safe flags rather than silent duplication.</p><p>The release controller displayed both request volume and distinct workload coverage. Ten thousand requests from one ordinary tenant no longer looked stronger than one hundred requests spanning the largest, oldest, and most complex tenants. Coverage had a business shape.</p><p>We also created pre-release guards inside the code. The rule loader refused to materialize an unbounded history and reported the account, estimated size, and safe degraded action. Canarying reduced probability; bounds limited consequence when probability still won.</p><p>Support and account teams were included when a known high-risk tenant entered a canary cohort. The goal was not to experiment recklessly on valuable customers. It was to observe representative behavior with tighter monitoring, limited exposure, and an immediate fallback path.</p><h3>A canary is an experiment, not a percentage</h3><p>An experiment needs a hypothesis, representative exposure, observable outcomes, and a stopping rule.</p><p>“Send 1 percent” defined only exposure volume. It said nothing about who, what, or where that traffic represented.</p><p>The lasting fix combined deliberate cohorts, segmented metrics, workload bounds, and a release gate that could say “we do not know yet.” That answer delayed one deployment. The old green answer took down production.</p><p>Froquiz’s production scenarios exercise that release judgment: expand the sample, target a risky cohort, pause promotion, isolate one tenant, or roll back while the aggregate graph still looks acceptable.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=994416bb8b0a" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-canary-was-healthy-it-never-received-the-customer-that-broke-production-994416bb8b0a">The Canary Was Healthy. It Never Received the Customer That Broke Production.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Senior Engineer Didn’t Touch the Keyboard. She Still Solved the Incident First.]]></title>
            <link>https://blog.stackademic.com/the-senior-engineer-didnt-touch-the-keyboard-she-still-solved-the-incident-first-d6143f50af58?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6143f50af58</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:32:09 GMT</pubDate>
            <atom:updated>2026-09-03T11:09:15.247Z</atom:updated>
            <content:encoded><![CDATA[<h4>Everyone else was debugging. She was deciding what not to destroy.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dt_j0O8sDN8e73c4Q4PLA.png" /></figure><p>At 02:14, checkout latency crossed eight seconds. By 02:18, six engineers were sharing commands in Slack.</p><p>One wanted to restart PostgreSQL. Another wanted to increase the connection pool. A third had already prepared a Kubernetes rollout restart.</p><p>Maya had not opened a terminal.</p><p>She asked one question: “Which signal changed first?”</p><p>Four minutes later, we stopped blaming the database and rolled back a worker configuration that had cut concurrency in half.</p><p>The senior engineer solved the incident without typing a command because the scarce skill was not command execution. It was preserving enough evidence to identify cause and effect.</p><h3>Fast hands were making the system harder to read</h3><p>The database looked guilty. Connections were near the limit. Query latency was rising. Several transactions had timed out.</p><p>Restarting it would have changed every one of those signals at once.</p><p>It might also have produced a temporary improvement. Connections would drop, queues would clear through failure, and dashboards would look calmer. We could have mistaken destruction of work for recovery.</p><p>Maya froze changes for five minutes. Then she built a timeline from deployment events, queue wait, request latency, retries, database connections, and pod restarts.</p><p>The order mattered:</p><pre>02:11 worker configuration deployed<br>02:12 available worker slots fall<br>02:13 request queue begins growing<br>02:14 client timeouts trigger retries<br>02:15 database connection churn rises<br>02:17 pods restart under memory pressure</pre><p>The database was suffering. It was not where the incident began.</p><p>Everyone else had looked for the most alarming graph. She looked for the earliest credible change.</p><p>She also checked what had <em>not</em> changed. Traffic volume, database deployment history, and provider latency were stable before the worker rollout. Negative evidence narrowed the search without proving a cause. It kept three emotionally attractive theories from consuming the next ten minutes.</p><h3>Her first contribution was subtraction</h3><p>Incident response rewards visible action. Typing commands feels like progress. Asking the room to stop feels passive.</p><p>It was the highest-leverage action available.</p><p>Every restart erases process state. Every configuration change introduces a new variable. Every broad scale-up changes queueing and connection behavior. When six people act independently, the system after five minutes is no longer the system that failed.</p><p>Maya assigned three roles:</p><ul><li>one person gathered evidence;</li><li>one person prepared the rollback;</li><li>one person recorded the timeline and customer impact.</li></ul><p>Everyone else stopped making production changes.</p><p>Then she stated a falsifiable prediction: restoring the previous worker setting should reduce queue wait first, retry traffic second, and database connection churn last.</p><p>The sequence appeared exactly that way after rollback.</p><p>That was stronger evidence than “things got better after we restarted something.”</p><h3>Seniority showed up in the questions</h3><p>None of Maya’s questions were exotic:</p><p>→ What changed first?</p><p>→ Which component is initiating pressure, and which is reacting?</p><p>→ What evidence will this action erase?</p><p>→ What should improve first if our hypothesis is correct?</p><p>→ Can we reverse the change without touching data?</p><p>The difference was timing. Junior responders often ask these questions in the postmortem. Senior responders ask them before the second production change.</p><p>She also separated containment from diagnosis. We reduced incoming nonessential work to protect checkout, but did not pretend that shedding load identified the root cause. Containment bought time. The timeline produced understanding.</p><p>Most incidents are not won by remembering one perfect command. They are won by recognizing a familiar failure shape before the room destroys the clues.</p><p>That recognition is why experienced engineers sometimes appear slow during the first minute. They are calculating the cost of being confidently wrong.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where that operating layer continues: recurring production patterns, the questions that expose them, and the judgment that keeps a plausible fix from becoming a second incident.</p><h3>We changed how the team practiced incidents</h3><p>Afterward, we stopped running incident drills as command quizzes.</p><p>The exercise no longer asked, “What command shows active PostgreSQL sessions?” Engineers could look that up.</p><p>Instead, we gave them conflicting evidence and watched how they managed uncertainty. Did they establish a timeline? Did they name the protected customer path? Did they make one reversible change? Did they preserve evidence before restarting a process?</p><p>We also added a decision log to the incident channel:</p><pre>hypothesis<br>supporting evidence<br>predicted outcome<br>change owner<br>observation window<br>rollback condition</pre><p>This looked bureaucratic until the next outage. Then it prevented two teams from applying opposite fixes at the same time.</p><p>Drills now include a deliberate trap: a dramatic downstream graph that appears after a quieter initiating change. The objective is not guessing the hidden answer. It is demonstrating a method that can separate trigger, amplifier, and victim while evidence is incomplete.</p><p>The team became faster because fewer actions needed to be undone.</p><h3>The keyboard was never the bottleneck</h3><p>Maya eventually typed commands. She verified the rollback, checked queue recovery, and confirmed the database was returning to normal.</p><p>By then, the important work had already happened.</p><p>She had converted a room full of activity into one testable story. She protected the evidence. She made the expected recovery sequence visible before the fix.</p><p>The incident lasted 23 minutes. The most valuable four minutes were the ones in which nobody restarted anything.</p><p>That is difficult to measure in performance reviews. There is no commit for the destructive command an engineer prevented, no ticket for the false theory they quietly killed, and no dashboard for uncertainty they removed from a room.</p><p>But production remembers.</p><p>The next time an incident channel fills with commands, notice who asks for the first changed signal. That person may be doing more to restore the system than everyone with a terminal open.</p><p>Production judgment is not trivia. It is choosing the next safe move while several plausible moves compete for attention. That is the muscle Froquiz is designed to exercise.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6143f50af58" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/the-senior-engineer-didnt-touch-the-keyboard-she-still-solved-the-incident-first-d6143f50af58">The Senior Engineer Didn’t Touch the Keyboard. She Still Solved the Incident First.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Every Pod Passed Readiness. The Load Balancer Was Sending Traffic to the Wrong Port.]]></title>
            <link>https://blog.stackademic.com/every-pod-passed-readiness-the-load-balancer-was-sending-traffic-to-the-wrong-port-1dc6dd945a4c?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1dc6dd945a4c</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 14:12:30 GMT</pubDate>
            <atom:updated>2026-09-03T11:08:05.833Z</atom:updated>
            <content:encoded><![CDATA[<h4>Kubernetes proved the containers were ready. It never proved the path users actually took.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PX2ryGkRrqqq2bAblXUfMw.png" /></figure><p>The deployment completed without an error. Every pod was Running. Readiness was 100%. The rollout controller marked the release successful.</p><p>Customers received 502 Bad Gateway.</p><p>Inside the cluster, curl to the pod worked. The application logs showed normal startup. Rolling back the application image changed nothing.</p><p>The failure sat between two green checks: the Service targeted the application correctly, but the external load balancer’s backend configuration still pointed to an old named port after an infrastructure change.</p><p>We had validated the destination and ignored the journey.</p><h3>Readiness is a local claim</h3><p>A readiness probe answers whether Kubernetes should consider a pod eligible for traffic through the Service endpoints.</p><p>It does not automatically verify:</p><ul><li>external DNS resolution;</li><li>TLS termination;</li><li>load balancer listener rules;</li><li>security groups or firewall policy;</li><li>the Service’s target mapping;</li><li>ingress or gateway routes;</li><li>the response customers receive through the public hostname.</li></ul><p>Our probe called localhost on the container port. It verified application process health while bypassing every network component involved in the outage.</p><p>The pod was honestly ready for a path nobody used.</p><p>The fastest clue came from comparing requests at successive boundaries:</p><pre>localhost inside pod:        200<br>pod IP from debug container: 200<br>Service DNS inside cluster:  200<br>load balancer backend probe: failed<br>public hostname:             502</pre><p>That ladder turned “Kubernetes is broken” into a bounded search. The first failing hop was outside the application and inside the traffic configuration.</p><h3>The rollback changed code, not routing state</h3><p>Rollback felt responsible because the incident followed a deploy. But the release contained two independently managed changes: an application rollout and an infrastructure update.</p><p>The application image rollback reverted only one.</p><p>Worse, repeated rollbacks added noise. Pods restarted, endpoint sets changed, and logs filled with normal termination messages. The team kept touching the layer already proven healthy.</p><p>We paused changes and wrote down the path:</p><pre>client -&gt; DNS -&gt; public load balancer -&gt; listener<br>-&gt; backend target -&gt; node/service -&gt; endpoint -&gt; pod port</pre><p>Then we attached a test to each arrow.</p><p>The load balancer showed healthy listener configuration but unhealthy backend targets. Its target port did not match the port now exposed by the service path. The configuration was syntactically valid, so infrastructure automation had applied it successfully.</p><p>Changing the backend port restored traffic immediately. The application rollback had never been necessary.</p><h3>Healthy components can form a broken system</h3><p>This incident was not caused by a failed component. It was caused by a false relationship between healthy components.</p><p>The pod listened correctly. The Service selected the intended pods. The load balancer accepted connections. DNS returned the right address. Each item could pass its own check while the composition failed.</p><p>Architecture diagrams encourage boxes. Production failures often live on arrows.</p><p>We changed validation in three layers.</p><p>First, deployment verification called the public hostname from outside the cluster and checked a behavior only the real application path could satisfy. A generic 200 from a load balancer error page was not enough.</p><p>Second, configuration tests resolved named ports and backend targets before apply. The test compared intent across resources rather than validating each manifest separately.</p><p>Third, observability labeled traffic by hop. We could distinguish requests arriving at the load balancer, reaching the ingress layer, selecting a service endpoint, and entering the application.</p><p>That made absence useful. If the public probe failed and the app saw no corresponding request, application debugging moved down the list.</p><p>We preserved a correlation token from the synthetic request at every boundary. When the token appeared in load-balancer logs but not gateway logs, the broken segment became evidence rather than a debate between application and infrastructure teams.</p><h3>End-to-end checks should be narrow and real</h3><p>An external synthetic check can become fragile if it attempts to test the entire product. Ours performs one small operation through the same hostname, TLS path, routing rule, authentication boundary, and backend used by customers.</p><p>It does not replace component probes. It complements them.</p><p>The local readiness probe remains fast and focused. It determines whether the pod can serve. The external check determines whether the system can deliver that service from the user boundary.</p><p>We also separated rollout success from release success.</p><p>“All replicas available” is now a Kubernetes result. “Release verified” requires the external transaction, expected route version, and error-rate window. Automation cannot promote the release using pod state alone.</p><p>This introduced a short delay before declaring success. That delay is cheaper than a fast green check for the wrong thing.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is designed for these production gaps: the failure modes between healthy components, the diagnostic sequence that narrows the path, and the operational checks that make a successful rollout mean more than green pods.</p><h3>We now investigate paths before products</h3><p>When users see an error, teams naturally jump to the most visible technology: Kubernetes, the database, the latest code, the cloud provider.</p><p>A path-first investigation asks a better question: how far did the request get?</p><p>That question reduces speculation. DNS logs, load balancer target health, gateway access logs, service endpoints, and application request IDs form a chain. The missing link usually tells you where to look next.</p><p>Every pod passed readiness because every pod was ready. The failure was our belief that readiness certified the rest of the route.</p><p>The most dangerous green dashboard is not one displaying false data. It is one displaying true data that answers a smaller question than the team thinks it does.</p><p>The fix took less than a minute once we inspected the right arrow. The expensive part was the half hour spent proving the same healthy box again and again.</p><p>System design interviews often focus on components. Real production judgment is visible when someone traces the boundaries between them. That is exactly the habit worth practicing before the route fails under a real release.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1dc6dd945a4c" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/every-pod-passed-readiness-the-load-balancer-was-sending-traffic-to-the-wrong-port-1dc6dd945a4c">Every Pod Passed Readiness. The Load Balancer Was Sending Traffic to the Wrong Port.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive.]]></title>
            <link>https://blog.stackademic.com/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive-0499a00a31e4?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/0499a00a31e4</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:27:23 GMT</pubDate>
            <atom:updated>2026-08-30T02:27:23.284Z</atom:updated>
            <content:encoded><![CDATA[<h4>The deployment returned to the old version, but the control plane still routed customers into a state the old code could not understand.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egIjN7uJGgiSm8xl-bEosQ.png" /></figure><p>The new release lasted seven minutes.</p><p>Error rate climbed. The incident commander called rollback. Kubernetes restored the previous image in four minutes.</p><p>The dashboard did not recover.</p><p>Some customers continued hitting the new workflow even though no new application pods remained.</p><p>The feature flag was still enabled.</p><p>Worse, the seven-minute release had already written state the old version could read — but could not process correctly.</p><p>We had rolled back code.</p><p>The system had moved forward.</p><h3>A deployment is only one source of behavior</h3><p>We treated release state as a container image.</p><p>Production behavior actually came from several independently changing layers:</p><p>→ Application version.</p><p>→ Feature-flag configuration.</p><p>→ Database schema and persisted data.</p><p>→ Queue messages already emitted.</p><p>→ Cached decisions.</p><p>→ Client code outside our deployment boundary.</p><p>Rolling back one layer did not rewind the others.</p><p>The old application received records marked with a new workflow state. Its default branch interpreted the unknown value as pending and retried forever. The flag service continued assigning customers to the treatment because flag state was managed outside the deployment pipeline.</p><p>Each component behaved according to its own valid configuration.</p><p>Their combination no longer represented a valid release.</p><h3>The flag had become a second deployment system</h3><p>Feature flags entered the company as safety tools.</p><p>They allowed gradual rollout, quick disablement, and separation of deployment from release. Over time they gained segments, prerequisites, experiments, scheduled changes, and multiple owners.</p><p>We reviewed application deployments carefully.</p><p>Flag changes required a click.</p><p>That made them fast. It also made them capable of changing production behavior without the evidence, ownership, or rollback plan attached to code releases.</p><p>During the incident, one person rolled back Kubernetes while another assumed the flag would follow automatically. The systems were independent. Our mental model was not.</p><h3>Backward compatibility needed a time window</h3><p>The durable fix began before the next release.</p><p>For any flagged workflow that wrote state, the old version needed to tolerate new values for at least the rollback window. New code wrote data in a form old code could safely ignore, defer, or process. Irreversible transitions waited until the release crossed an explicit point of no return.</p><p>We also attached flag state to the release record:</p><p>→ Which flags changed with this release?</p><p>→ Who owns the disable decision?</p><p>→ Does disabling stop new entry only, or repair existing state?</p><p>→ What queued work survives rollback?</p><p>→ Which clients have cached the treatment?</p><p>The rollback runbook became a system rollback plan rather than a deployment command.</p><p>Rollback incidents repeat because teams restore an artifact without mapping the state that artifact has already changed. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where those recovery patterns continue across deployments, databases, queues, and distributed workflows.</p><h3>Recovery fails where state outlives code</h3><p>Stateless services make rollback look simple because the artifact can be replaced.</p><p>Real products create state. Orders change status. messages enter queues, clients cache configuration, and external systems receive side effects. Once those events happen, the previous binary may no longer represent the previous world.</p><h3>Four minutes was still a good rollback</h3><p>The deployment automation did its job.</p><p>That mattered. Without it, the incident would have been worse.</p><p>Our mistake was giving one successful mechanism the name of the entire recovery.</p><p>Now we separate three actions:</p><ol><li>Stop new exposure.</li><li>Restore compatible code.</li><li>Reconcile state created during exposure.</li></ol><p>Sometimes all three happen with one button. Often they do not.</p><p>The next time someone says a feature is safe because it is behind a flag, ask what remains after the flag turns off.</p><p>That answer is the real rollback time.</p><p>Production judgment is not knowing the command that restores the old image. It is recognizing which parts of the system cannot be made old again.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0499a00a31e4" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive-0499a00a31e4">We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive.</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Wrote the Migration in 90 Seconds. We Spent Two Days Proving It Was Safe.]]></title>
            <link>https://ai.plainenglish.io/ai-wrote-the-migration-in-90-seconds-we-spent-two-days-proving-it-was-safe-ae8f01698f9f?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae8f01698f9f</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 02:08:23 GMT</pubDate>
            <atom:updated>2026-09-09T01:13:45.345Z</atom:updated>
            <content:encoded><![CDATA[<h4>Code generation became cheap. Reversibility, lock behavior, and production evidence did not.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gN7l2V8BqOnDQUTyJ-lMoA.png" /></figure><p>The prompt was ordinary.</p><p>Add a non-null customer classification column, backfill existing rows, and update the application model.</p><p>The coding agent returned a migration in ninety seconds.</p><p>It was valid SQL. It passed tests. It worked against staging.</p><p>The pull request took two days to approve.</p><p>Not because the reviewers were resisting AI.</p><p>Because the database held 640 million rows and the generated code knew the desired schema — not the operational path required to reach it.</p><h3>Correct SQL can still be an unsafe event</h3><p>The migration added the column with a default, updated historical rows, and applied a non-null constraint.</p><p>On a small database, that sequence was fine.</p><p>In production, it raised different questions:</p><p>→ Would the change rewrite the table?</p><p>→ How long would locks be held?</p><p>→ What happened to replication lag during backfill?</p><p>→ Could application versions before and after deployment coexist?</p><p>→ How would we pause safely?</p><p>→ What did rollback mean after new values were written?</p><p>The agent had produced the destination.</p><p>Reviewers had to design the journey.</p><h3>Staging answered the smallest question</h3><p>Our staging database contained representative schema and unrepresentative physics.</p><p>Tables were smaller. Traffic was lighter. Long-running transactions were rare. Replication did not carry production load. Autovacuum pressure was different.</p><p>The migration completed in fourteen seconds.</p><p>That result proved syntax and basic compatibility. It did not predict production lock wait or total backfill time.</p><p>We sampled production statistics, rehearsed against a restored copy with realistic volume, and inspected the execution plan. The backfill needed batching. The constraint needed a validation sequence. Application reads needed to tolerate missing values during transition.</p><p>Most of the two days was not coding.</p><p>It was removing irreversible surprises.</p><h3>We separated expand, migrate, and contract</h3><p>The safe release used several steps.</p><p>First, add the nullable column without forcing an immediate table-wide rewrite. Deploy application code capable of reading old and new state. Begin writing the new value for fresh records.</p><p>Then backfill historical rows in bounded batches with progress, pause controls, replication-lag limits, and a rate the database could sustain during ordinary traffic.</p><p>After verification, add and validate the constraint through the database’s safer path. Only later remove compatibility code.</p><p>The migration became slower on the calendar and safer in production.</p><p>The AI-generated draft remained useful. It captured intent and removed typing. It simply was not the deployment plan.</p><p>Production engineering patterns matter more when implementation becomes cheap because more changes reach the boundary where those patterns decide safety. <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a> is where I continue that judgment layer beyond one migration and one database.</p><h3>Judgment is now the expensive part</h3><p>When code took hours to write, generation and evaluation felt like one activity.</p><p>Agents separated them.</p><p>The first artifact arrives almost immediately. The remaining work is understanding scale, state, failure, observability, reversibility, and coordination with versions already running.</p><h3>Ninety seconds was still valuable</h3><p>We did not stop using the agent for migrations.</p><p>We changed the definition of done.</p><p>Every generated migration now arrived with questions the author — human or agent — had to answer:</p><p>What is the largest lock? What is the maximum unit of work? Can old and new application versions overlap? How is progress observed? Where can we pause? Which state cannot be rolled back?</p><p>The agent shortened the path to the first draft.</p><p>It did not shorten the database.</p><p>The future of engineering is not spending two days typing what a model can write in ninety seconds. It is spending those two days on the part that was always capable of waking customers and responders at 3 AM.</p><p>Fast code is leverage.</p><p>Fast code with slow judgment is a larger change queue aimed at production.</p><p>The senior skill is knowing which ninety-second output deserves a two-day pause.</p><p>→ <a href="https://froquiz.com/senior-challenge"><strong>Practice production judgment on Froquiz</strong></a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae8f01698f9f" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/ai-wrote-the-migration-in-90-seconds-we-spent-two-days-proving-it-was-safe-ae8f01698f9f">AI Wrote the Migration in 90 Seconds. We Spent Two Days Proving It Was Safe.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Lambda vs Cloud Run in 2026: The Cheapest Request Lost to the Cheapest Workload.]]></title>
            <link>https://aws.plainenglish.io/aws-lambda-vs-cloud-run-in-2026-the-cheapest-request-lost-to-the-cheapest-workload-249606df96af?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/249606df96af</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Fri, 28 Aug 2026 22:07:19 GMT</pubDate>
            <atom:updated>2026-09-09T01:22:16.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>Per-invocation math looked decisive until concurrency, background work, and idle behavior entered the bill.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uoOZljCLXc1Kq-U9Gylmxg.png" /></figure><p>Serverless pricing has a magic trick.</p><p>It turns infrastructure into tiny units until every individual request looks almost free.</p><p>Then the application grows a queue, a scheduled job, a traffic spike, one slow dependency, and a worker that needs to stay alive longer than the request that created it.</p><p>Suddenly Lambda versus Cloud Run is not a price-per-request comparison.</p><p>It is a question about the shape of the workload.</p><h3>Lambda wants an event with an ending</h3><p>Lambda is at its best when a unit of work has a clear trigger and a bounded lifetime.</p><p>An object arrives. A message appears. An API request needs a response. A schedule fires. The handler runs and stops.</p><p>AWS takes responsibility for creating execution environments and scaling them with demand. The default Lambda model processes one invocation at a time per execution environment. Regional concurrency and scaling rates provide guardrails around how quickly work can expand.</p><p>That model is easy to reason about when the work is naturally function-shaped.</p><p>It becomes awkward when the workload wants a process.</p><p>Long-lived consumers, connection-heavy services, custom binaries, background loops, large images, or work that benefits from many concurrent requests sharing one warm instance can push against the function abstraction.</p><p>You can often make it work.</p><p>The question is how much architecture you add to preserve the original pricing story.</p><h3>Cloud Run wants a container with behavior</h3><p>Cloud Run begins from a different unit: a container.</p><p>The same platform can run request-serving services, jobs, worker pools, and long-lived instances. Services can handle concurrent requests inside one instance. Jobs can run tasks to completion and fan out across many tasks. Worker pools fit always-on background work.</p><p>That flexibility changes the cost model.</p><p>Concurrency can improve utilization for I/O-heavy services because one warm container handles multiple requests. But a badly chosen concurrency setting can also let too much work compete for CPU, memory, database connections, or event-loop time inside the same instance.</p><p>Scale-to-zero can make idle services cheap. Minimum instances can reduce startup latency. Both are useful.</p><p>Both are decisions.</p><p>Cloud Run removes servers. It does not remove capacity planning.</p><p>It moves capacity planning into instance size, concurrency, minimum scale, maximum scale, and request behavior.</p><h3>The database sees both platforms differently</h3><p>This is where cheap compute becomes expensive architecture.</p><p>Imagine a traffic spike that creates hundreds of concurrent Lambda execution environments. Each opens a database connection. The API scales successfully. PostgreSQL does not.</p><p>Now imagine Cloud Run handling the same traffic with high per-instance concurrency. Fewer instances may mean fewer connection pools, but each instance can create a dense burst of simultaneous database work.</p><p>Neither platform fixes an unbounded downstream.</p><p>The database does not care that compute scaled automatically. It sees connections, transactions, lock time, and queries.</p><p>The responsible comparison includes:</p><p>→ Maximum downstream concurrency</p><p>→ Connection reuse and pooling</p><p>→ Backpressure when the database is saturated</p><p>→ Retry behavior after throttling</p><p>→ The cost of keeping capacity warm</p><p>→ The blast radius of one slow dependency</p><p>Autoscaling without a ceiling is not resilience.</p><p>It is a faster way to move the bottleneck.</p><p>Architecture decisions become dangerous when the calculator models compute but ignores the operational controls around it. The missing layer is learning to evaluate reversal cost, failure boundaries, and downstream pressure before the bill makes the trade-off obvious.</p><p><a href="https://yusufseyitoglu.gumroad.com/l/architecture-under-pressure"><strong>Recognize the architecture decisions that become expensive under real production pressure →</strong></a></p><h3>Price the whole request tree</h3><p>A useful cost model includes more than execution time.</p><p>For Lambda, include API Gateway or other triggers, provisioned concurrency where used, data transfer, logs, retries, orchestration, queue requests, and the concurrency reserved to protect downstream systems.</p><p>For Cloud Run, include allocated CPU and memory, minimum instances, outbound networking, load balancing, build and artifact storage, logs, and the utilization lost when concurrency is configured poorly.</p><p>Then include engineering cost.</p><p>How many separate functions, permissions, deployment units, and event bindings are required to express the workflow?</p><p>How much custom scaling and lifecycle behavior does the container need?</p><p>The cheapest request can belong to the more expensive system if the workload constantly fights its unit of deployment.</p><h3>Choose the shape that needs fewer apologies</h3><p>Lambda is a strong fit for event handlers, bursty bounded work, deep AWS integrations, small independent functions, and workloads where per-invocation isolation is valuable.</p><p>Cloud Run is a strong fit for containerized HTTP services, higher per-instance concurrency, portable runtimes, mixed service-and-job workloads, and teams that want one deployment shape for more kinds of work.</p><p>The wrong choice is usually not catastrophic.</p><p>It is visible as a growing list of apologies:</p><p>“We need this extra queue because the function cannot finish.”</p><p>“We keep five instances warm because startup is unpredictable.”</p><p>“We raised concurrency, but now the database falls over.”</p><p>“The handler is small, but the workflow around it is not.”</p><p>Pick the platform whose normal operating model resembles the workload you actually have.</p><p>Serverless is cheapest when the architecture stops arguing with the billing unit.</p><p>Capacity decisions are easier to defend when you practice them as connected systems rather than isolated boxes. The important part is explaining where concurrency stops and which component absorbs pressure next.</p><p><a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=249606df96af" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/aws-lambda-vs-cloud-run-in-2026-the-cheapest-request-lost-to-the-cheapest-workload-249606df96af">AWS Lambda vs Cloud Run in 2026: The Cheapest Request Lost to the Cheapest Workload.</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The System Design Interview Still Rewards Diagrams. Production Rewards Recovery.]]></title>
            <link>https://aws.plainenglish.io/the-system-design-interview-still-rewards-diagrams-production-rewards-recovery-1b98a2c80730?source=rss-103fbf8a115a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b98a2c80730</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Clean Code Journal]]></dc:creator>
            <pubDate>Thu, 27 Aug 2026 18:29:27 GMT</pubDate>
            <atom:updated>2026-09-09T01:22:35.730Z</atom:updated>
            <content:encoded><![CDATA[<h4>A beautiful architecture tells me how traffic moves when everything works. Senior judgment begins one minute later.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SmTO35h9hHmhlVHzZCVs_w.png" /></figure><p>The candidate drew the right boxes.</p><p>API gateway. Stateless services. Redis. Kafka. PostgreSQL. Read replicas. Object storage. CDN.</p><p>The diagram looked like a cloud provider landing page.</p><p>Then came one question:</p><p><strong>The payment provider succeeded, but your service crashed before saving the result. What happens now?</strong></p><p>The boxes stopped helping.</p><p>That moment is the difference between assembling components and designing a system.</p><h3>Components are vocabulary, not judgment</h3><p>Most engineers can learn the standard building blocks. They know a cache reduces repeated reads, a queue absorbs bursts, replicas scale reads, and a CDN moves content closer to users.</p><p>Those facts are useful.</p><p>They do not tell you whether a cache is allowed to be stale, whether a message can be processed twice, whether a replica is safe for authorization data, or whether an accepted payment can disappear between systems.</p><p>Architecture lives in those decisions.</p><p>Two candidates can draw the same components and produce systems with completely different reliability. One treats Kafka as magic durability. The other defines message identity, acknowledgment timing, retry ownership, dead-letter handling, replay safety, and the operator path when processing stops.</p><p>The queue box is identical.</p><p>The system is not.</p><h3>Start with invariants before scale</h3><p>Interviews often rush toward requests per second, storage estimates, and sharding.</p><p>Capacity matters, but a system that scales the wrong behavior is merely wrong at higher throughput.</p><p>Start with what cannot be violated:</p><ul><li>One order must not create two charges.</li><li>A user must never read another tenant’s document.</li><li>An accepted upload must remain traceable until processing completes.</li><li>Inventory must not go negative.</li><li>Revoked access must not survive through an unsafe cache.</li></ul><p>These statements shape the architecture more strongly than the initial traffic estimate.</p><p>The charge invariant leads to stable idempotency keys and reconciliation. Tenant isolation shapes keys, authorization checks, data partitioning, and logs. Traceable uploads require durable state transitions rather than a hopeful chain of callbacks.</p><p>When the invariant is clear, technology choices become explainable.</p><h3>Every arrow needs a failure sentence</h3><p>Boxes get attention because they have names. Arrows contain the incidents.</p><p>For every network boundary, finish this sentence:</p><p>If the caller does not receive a response, then…</p><p>Does it retry? Who owns the retry? Can the operation happen twice? Is the outcome queryable? How long does uncertainty remain acceptable?</p><p>For every asynchronous boundary, ask:</p><p>If this message is delivered again, then…</p><p>For every cache, ask:</p><p>If this value is stale, then…</p><p>For every replica, ask:</p><p>If this read is behind the primary, then…</p><p>Those four sentences uncover more architecture than adding another managed service to the diagram.</p><p>They also expose when a design needs a human recovery path. Some outcomes cannot be resolved automatically forever. The system must know when to stop retrying, preserve evidence, and surface a decision to an operator.</p><p>That judgment is built from recognizing recurring failure patterns, not memorizing more boxes. A diagram becomes useful when you can see which connection is hiding a retry storm, stale decision, partial commit, or irreversible side effect.</p><p>→ <a href="https://yusufseyitoglu.gumroad.com/l/production-engineering-library"><strong>The Production Engineering Library: The Complete Collection for Production Systems, System Design &amp; Backend Engineering</strong></a></p><h3>Design rollback before deployment</h3><p>Candidates often finish with “we use blue-green deployment” or “we can roll back.”</p><p>Roll back what?</p><p>Application code can move backward. Data written by the new version may not. Messages already published may have the new schema. Mobile clients may continue sending the new request shape. A third-party side effect cannot be undeclared.</p><p>A credible rollout plan separates reversible and irreversible changes.</p><p>Use additive database migrations. Keep old and new readers compatible during the transition. Gate behavior with flags whose failure mode is understood. Observe the new path before making it authoritative. Define the metric that stops expansion.</p><p>Then explain recovery after partial rollout. If 20% of traffic used the new behavior, how will those records be identified? Can old code read them? What repair job exists?</p><p>“Rollback” is not a button. It is a property designed into each state transition.</p><h3>The senior answer includes Monday morning</h3><p>Production recovery is not finished when the graph turns green.</p><p>A strong design explains what evidence remains after failure, how operators identify affected users, how stuck work is replayed safely, and how the system prevents recurrence.</p><p>That means correlation IDs, audit events, durable workflow state, reconciliation jobs, runbooks, and ownership. None of them are visually impressive. All of them decide how long the incident lasts.</p><p>The system design interview should not be a memory contest for fashionable infrastructure.</p><p>It should test whether you can make trade-offs explicit while information is incomplete.</p><p>Draw the boxes. Estimate the load. Choose the database.</p><p>Then spend the rest of the conversation on the arrows, the unknown states, and the way back.</p><p>Production will.</p><p>An architecture becomes yours when you can build its request flow and explain how it behaves after the first clean assumption fails. Practice making those trade-offs concrete rather than leaving recovery outside the diagram.</p><p>→ <a href="https://froquiz.com/system-design"><strong>Build a System Design interview architecture on Froquiz</strong></a></p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b98a2c80730" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/the-system-design-interview-still-rewards-diagrams-production-rewards-recovery-1b98a2c80730">The System Design Interview Still Rewards Diagrams. Production Rewards Recovery.</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>