<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark W. on Medium]]></title>
        <description><![CDATA[Stories by Mark W. on Medium]]></description>
        <link>https://medium.com/@mark.w.1987?source=rss-95d72e9839d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xDAmJPr_NtC7-eqFsVYRZg.png</url>
            <title>Stories by Mark W. on Medium</title>
            <link>https://medium.com/@mark.w.1987?source=rss-95d72e9839d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:44:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mark.w.1987/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Green Staging Environment Doesn’t Guarantee a Safe Production Deploy]]></title>
            <link>https://medium.com/@mark.w.1987/a-green-staging-environment-doesnt-guarantee-a-safe-production-deploy-8522c57b09ec?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8522c57b09ec</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[continuous-deployment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 05:40:15 GMT</pubDate>
            <atom:updated>2026-11-20T13:09:37.585Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve watched too many “safe” deployments fail five minutes after production traffic arrived. In one case, a checkout service passed staging cleanly, then started returning authentication failures when production requests hit a namespace with tighter RBAC. The rollout looked fine; the failure only showed up once real traffic, real permissions, and real concurrency were in play.</p><p>A green staging environment proves that one environment behaved under one set of conditions. It does not prove your production control plane, permissions, data shape, or dependency timing will behave the same way. If the rollout only passed staging, I still assume the production path is unproven.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ek4Y6RW0EU158b8vgo3aQ.png" /><figcaption>Why isolated staging diverges from production</figcaption></figure><h3>Where the hidden drift shows up</h3><p>Most teams collapse this into “config mismatch,” but the failures I see are narrower and more dangerous. The app code is often fine. The break happens in bindings around the code: token issuers, RBAC policies, sidecar versions, informer state, or stale infrastructure variables that only matter when multiple services coordinate under load. That is why configuration drift and environment parity between staging and production matter so much.</p><p>I’ve been burned by auth drift more than once. Staging had service accounts with broad permissions, while production followed least privilege. The deploy passed integration tests, then production started returning authorization failures on a low-volume path that fanned out into retries and queue growth. Those are the kinds of hidden deployment dependencies that make “why staging passes but production fails” such a common question.</p><ul><li><strong>RBAC drift:</strong> policy bindings differ across clusters or namespaces, so one service can read a resource in staging but gets denied in production.</li><li><strong>Informer staleness:</strong> controllers act on delayed cache state, not live API state, so they make decisions on object versions that are already outdated.</li><li><strong>Terraform drift:</strong> a stale workspace or variable set applies old assumptions, which means the infrastructure you think you deployed is not the one actually running.</li></ul><blockquote><strong>Watch out:</strong> Permission drift often looks like an application bug first. I check token audience, issuer, and policy bindings before I touch business logic.</blockquote><p>These are control-plane consistency problems as much as application defects. Kubernetes caches can lag behind live state, IAM changes can take time to propagate, and Terraform workspaces can point at old variables or state. Each one creates a timing window that staging rarely reproduces. If I want fewer surprises, I have to measure drift where it actually forms and audit configuration differences before every release.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_KNYZyvz1-ef6zqi2yaIQ.png" /></figure><h3>What safer deploy architecture looks like</h3><p>The deployment model I trust most uses production infrastructure for validation and progressive rollout for containment. I separate three controls because they answer different questions. Shadowing tells me whether behavior diverges under real requests. Canarying tells me how much traffic I am willing to risk. Feature flags tell me whether deploy and release should be coupled.</p><p>Traffic shadowing is valuable because it validates real auth, real downstreams, and actual request distributions without customer impact. I’ve had good results mirroring read-heavy traffic and comparing response classifications, latency buckets, and downstream fan-out against the baseline. In practice, I keep the mirrored path read-only, drop or suppress writes, and filter by tenant class so I am not comparing noise from every request on the system. This is one of the best practices for a staging environment to mirror production when you are trying to reduce production deployment risk.<a href="https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/"> Kubernetes progressive delivery patterns</a> and<a href="https://argo-rollouts.readthedocs.io/en/stable/"> Argo Rollouts canary controls</a> make this operationally manageable.</p><blockquote><strong>Practical tip:</strong> Shadow traffic should be sampled and labeled. If I cannot compare baseline and candidate by route, tenant class, and dependency path, the signal is weak.</blockquote><p>Canaries solve a different problem. They cap blast radius while exposing production-only effects such as connection-pool starvation, cache-miss spikes, or p99 regressions hidden by stable averages. Feature flags then decouple binary rollout from user exposure, which improves rollback velocity because I can disable behavior without waiting for image replacement. A real example is a payment-service canary where p95 jumped from 180 ms to 430 ms on one tenant cohort before the rollout reached 10 percent of traffic. That is also where feature-flag mismatches can become dangerous if staging and production do not share the same release conditions. The flow is easier to reason about when it is mapped as an operator path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bEr8YKzMIqLxf9raaiDyGg.png" /><figcaption>Progressive delivery limits blast radius</figcaption></figure><h3>What to measure before you trust it</h3><p>When someone tells me a deploy is safe, I want evidence, not confidence language. I ask for blast radius as a percentage of traffic or tenant scope, rollback velocity in minutes, and divergence between baseline and canary at p50, p95, and p99. If p99 regresses by more than 15 percent for 10 minutes, I abort the rollout.</p><blockquote><strong>Note:</strong> Averages hide deploy risk. I compare tail latency and authorization failure rate first because that is where production-only regressions usually appear.</blockquote><p>I also want routine drift audits for config, RBAC, and infrastructure state.<a href="https://opentelemetry.io/docs/concepts/signals/metrics/"> OpenTelemetry metrics guidance</a> helps standardize signals, and<a href="https://www.educative.io/courses/grokking-the-system-design-interview?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_backend_system_design_material&amp;eid=5082902844932096"> Educative’s backend system design material</a> is useful when you’re working through rollout decisions, failure domains, and observability trade-offs. If I cannot see queue depth, cache sync lag, or control-plane delay quickly, the canary is mostly theater. I want alerts and dashboards that gate promotion on those signals, not just a dashboard someone checks after the fact. Strong infrastructure as code consistency also helps reduce environment drift.</p><h3>What I tell teams to change first</h3><p>I do not tell teams to delete staging. I tell them to demote it. Staging is a fast integration check for obvious contract breaks, schema errors, and startup failures. It is not the final proof that production is safe, because the missing variables are usually the ones that matter most under load.</p><p>My first recommendation is usually a short list:</p><ul><li>add shadow validation on production infrastructure</li><li>canary by a small traffic percentage or tenant cohort</li><li>audit drift across RBAC, configs, secrets, and Terraform state</li></ul><blockquote><strong>Practical tip:</strong> I keep rollback velocity under ten minutes when possible. If rollback needs approvals, image rebuilds, or cache drains, I treat that as a platform bug.</blockquote><p>I also push teams to decouple deploy from release with feature flags.<a href="https://launchdarkly.com/docs/home/flags"> LaunchDarkly’s flagging patterns</a> are a reasonable reference, and<a href="https://www.educative.io/courses/system-design-deep-dive-real-world-distributed-systems?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_distributed_systems_courses&amp;eid=5082902844932096"> Educative’s distributed systems courses</a> help engineers reason about consistency and failure domains. The trade-off here is between platform complexity and operational safety. I usually take the added complexity because it fails smaller. Test data parity is another reason staging can look healthy while production still breaks.</p><blockquote><strong>Watch out:</strong> Shadowing is not free. Mirrored writes can corrupt state unless they are blocked, redirected, or made idempotent.</blockquote><p>My rule is simple: trust the environment that carries the real risk, then limit the damage when you are wrong. Green staging is a useful checkpoint, not a safety guarantee. The practical checklist is simple: shadow real traffic, canary a small slice, watch drift signals, and make rollback boring.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8522c57b09ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Long-Running Frontend Session Can Leak Memory Without Warning]]></title>
            <link>https://medium.com/@mark.w.1987/a-long-running-frontend-session-can-leak-memory-without-warning-f50d0c5af755?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f50d0c5af755</guid>
            <category><![CDATA[frontend-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 09:17:42 GMT</pubDate>
            <atom:updated>2026-11-17T09:17:42.976Z</atom:updated>
            <content:encoded><![CDATA[<p>Your frontend can pass every QA checklist and still get slower after lunch. I have debugged frontend <a href="https://www.educative.io/blog/what-is-a-memory-leak?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=memory_leak&amp;eid=5082902844932096">memory leak</a> sessions that looked clean for twenty minutes, then turned sticky after three hours of route changes, idle refreshes, and modal churn. In one shell app, heap climbed from 180 MB to 420 MB over a long afternoon, and input delay rose from about 20 ms to 120 ms after repeated modal opens. The hard part is that nothing obvious fails. Users usually blame Chrome, not my code.</p><h3>Why long sessions fail quietly</h3><p>In production, a long-running session memory leak rarely shows up as one big spike. I usually see slow growth in the old-generation heap, slightly longer <a href="https://www.educative.io/answers/how-garbage-collection-works-in-javascript?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=garbage_collection&amp;eid=5082902844932096">garbage collection</a> pauses, and input delay that creeps upward. The app still works, but every tab switch, route change, or background poll leaves behind a small amount of retained memory. That is why it can be hard to detect a memory leak in frontend session behavior until the session has been open for a while.</p><p>That pattern is common in single-page apps and micro-frontend shells. A user can keep one tab open all day, which means mount and unmount paths execute hundreds of times. If each cycle leaks a few listeners or DOM references, the total retained set grows steadily. This is usually easier to spot in a timeline than in logs because the browser does a little more work on every cycle.</p><ul><li><strong>Short test runs lie:</strong> a five-minute QA pass mostly exercises allocation, not retention.</li><li><strong>Long sessions hurt differently:</strong> the browser spends more time collecting memory that should have been released.</li><li><strong>Symptoms are indirect:</strong> typing lags, scrolling stutters, and route transitions feel heavy.</li></ul><p>I learned this the hard way on a shell app where every micro-frontend looked locally correct. The leak lived in shared contracts, so each team said their mount path was fine. They were all partly right, and the session still degraded. That shared-state problem leads directly to the causes of frontend memory leaks and what memory is actually keeping alive.</p><h3>What actually keeps memory alive</h3><p>A browser leak is usually not just “too much memory used at once.” It is memory that stays reachable when I expected it to die. Garbage collection can only free objects that nothing references anymore. If a listener, timer, observer, or closure still points to component state, that state remains alive.</p><p>Here is a simple way to think about it. Anything I create during mount needs a matching teardown during unmount. If I add a resize listener, I remove it. If I start an interval, I clear it. If I subscribe to a store, I unsubscribe.</p><blockquote><strong>Note:</strong> Retention is the key idea. Small leaks repeated many times are worse than one short-lived allocation burst.</blockquote><p>Common sources I see in frontend code:</p><ul><li><strong>Listeners and subscriptions:</strong> DOM events, custom event buses, store listeners.</li><li><strong>Background work:</strong> intervals, timeouts, observers, polling loops.</li><li><strong>Retained references:</strong> closures capturing props, detached DOM nodes, unbounded caches.</li></ul><p>Detached DOM nodes are especially sneaky. I remove an element from the page, but a listener or cached reference still points to it, so the browser cannot reclaim it. <a href="https://developer.chrome.com/docs/devtools/memory-problems/">Chrome DevTools Memory documentation</a> is useful here because it shows detached nodes directly. Once you understand retention at the object level, the next failure mode is shared ownership across component boundaries, including the kind of event listeners memory leak that builds up across repeated renders.</p><p>Here is the mount and unmount pattern I keep coming back to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBPDS5ICPHrhN4U155whHQ.png" /><figcaption>Memory grows in small steps during a normal session</figcaption></figure><h3>Where shared frameworks still leak</h3><p>Even teams with cleanup checklists still miss framework-level retention. In React, a useEffect can resubscribe on every render if dependencies are unstable. If the cleanup function is wrong, delayed, or skipped because of a contract bug, subscriptions stack up quietly. I have seen a rerender path with a mutable callback keep thousands of listeners alive by the end of a day.</p><p>useSyncExternalStore helps standardize subscriptions, but only if the store implementation returns a real unsubscribe every time. A shared store that keeps old subscriber references breaks that contract. The component looks fine, but the store keeps growing. <a href="https://react.dev/reference/react/useSyncExternalStore">React’s official useSyncExternalStore reference</a> is worth reading closely for that reason.</p><blockquote><strong>Watch out:</strong> “I returned a cleanup” is not enough. The cleanup must remove the exact resource created by that render.</blockquote><p>Micro-frontends add another layer. One fragment may mount correctly, but the shell owns the event bus, cache, or cross-app store. If ownership is unclear, teardown becomes partial. I now require every shared API to define who allocates, who unsubscribes, and whether caches are bounded.</p><blockquote><strong>Practical tip:</strong> In code review, I look for symmetry: add/remove, subscribe/unsubscribe, observe/disconnect, start/stop.</blockquote><p>A quick reference helps beginners spot these patterns faster before they become session-wide problems.</p><pre><br>class Store {<br>  constructor() {<br>    this.subscribers = new Set();<br>  }<br>  subscribe(fn) {<br>    this.subscribers.add(fn);<br>    return () =&gt; this.subscribers.delete(fn);<br>  }<br>}<br><br>class LruCache {<br>  constructor(limit) {<br>    this.limit = limit;<br>    this.map = new Map();<br>  }<br>  set(key, value) {<br>    if (this.map.has(key)) this.map.delete(key);<br>    this.map.set(key, value);<br>    while (this.map.size &gt; this.limit) {<br>      const oldestKey = this.map.keys().next().value;<br>      this.map.delete(oldestKey);<br>    }<br>  }<br>}<br><br>function mountBad(widgetEl, store) {<br>  window.addEventListener(&quot;resize&quot;, () =&gt; widgetEl.render?.());<br>  setInterval(() =&gt; widgetEl.tick?.(), 1000);<br><br>  const observer = new MutationObserver(() =&gt; widgetEl.sync?.());<br>  observer.observe(widgetEl, { childList: true, subtree: true });<br><br>  store.subscribe((state) =&gt; widgetEl.update?.(state));<br>  // Missing teardown leaks listeners, timer, observer, and subscription.<br>}<br><br>function mountGood(widgetEl, store, cache) {<br>  const onResize = () =&gt; widgetEl.render?.();<br>  const intervalId = setInterval(() =&gt; widgetEl.tick?.(), 1000);<br>  const observer = new MutationObserver(() =&gt; widgetEl.sync?.());<br>  const unsubscribe = store.subscribe((state) =&gt; widgetEl.update?.(state));<br><br>  window.addEventListener(&quot;resize&quot;, onResize);<br>  observer.observe(widgetEl, { childList: true, subtree: true });<br><br>  cache.set(widgetEl.id, { mountedAt: Date.now() }); // bounded cache via LRU eviction<br><br>  return function unmount() {<br>    window.removeEventListener(&quot;resize&quot;, onResize); // teardown mirrors setup exactly<br>    clearInterval(intervalId);<br>    observer.disconnect();<br>    unsubscribe();<br>  };<br>}</pre><h3>How I verify a fix is real</h3><p>I do not trust a memory fix until I measure it over a realistic session. My baseline is scripted route changes, tab switches, modal open-close cycles, and periodic data refreshes for at least thirty to sixty minutes. One-off clicks are not enough. I want repeated lifecycles that match actual usage. This is the practical way to measure memory leaks over time and compare behavior before and after a fix.</p><p>My usual checks are simple:</p><ul><li>compare heap snapshots before and after repeated cycles</li><li>count detached DOM nodes</li><li>confirm listener counts return to baseline after unmount</li></ul><p>For heap work, I use <a href="https://developer.chrome.com/docs/devtools/memory-problems/heap-snapshots/">Chrome’s heap snapshot tools</a> and compare retained objects, not just total allocations. I usually start by taking one snapshot after warm-up, then another after the same route-change loop, and I look for objects that stay retained across cycles. If old-generation usage trends upward after each cycle, the fix is incomplete. When available, I also sample <a href="https://developer.mozilla.org/en-US/docs/Web/API/Performance/measureUserAgentSpecificMemory">performance.measureUserAgentSpecificMemory</a> to track browser-reported memory across a run.</p><blockquote><strong>Note:</strong> A good result is not zero movement. Some variance is normal. I look for a stable band, not perfect flat lines.</blockquote><p>For beginners, this may look tricky, but it is just repetition plus measurement. Start with this example and try it yourself: run twenty route changes, force idle time, sample memory, then repeat after your cleanup change. If post-cycle memory settles near baseline, that is meaningful evidence. The last piece is turning that discipline into a default design habit.</p><h3>Designing for safe teardown by default</h3><p>I treat memory ownership as part of component design, not a cleanup chore. Explicit teardown adds code, bounded caches can reduce hit rate, and long-session tests take CI time. I still take that trade-off because debugging a production tab that degrades after hours is slower and more expensive.</p><p>My review checklist is basic but strict:</p><ul><li>every mount path has an unmount path</li><li>every shared subscription returns an idempotent unsubscribe</li><li>every cache has a size or age bound</li></ul><blockquote><strong>Practical tip:</strong> If a component creates work outside itself, I document who stops that work before I merge it.</blockquote><p>For beginners, the main lesson is reassuring: most frontend leaks are understandable once you look for retention instead of crashes. Learn the lifecycle, clean up what you create, and test longer than your shortest demo. The real rule is simple: measure long sessions, tie cleanup to ownership, and treat retention bugs as lifecycle bugs, not browser mysteries. That is the most reliable way to fix memory leaks in SPAs and avoid long-session frontend performance degradation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f50d0c5af755" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your multi-tenant database schema gets harder to change after launch]]></title>
            <link>https://medium.com/@mark.w.1987/your-multi-tenant-database-schema-gets-harder-to-change-after-launch-4d2185f60054?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d2185f60054</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[database-design]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[multitenancy]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 05:21:27 GMT</pubDate>
            <atom:updated>2026-11-11T05:21:27.915Z</atom:updated>
            <content:encoded><![CDATA[<p>The multi-tenant database schema feels easy until the first customer arrives with ten times more data than everyone else. I have watched teams treat tenancy as a reversible storage detail, then discover six months later that every index, every migration, and every support script assumes that exact model. By then, changing it is no longer a refactor. It is a product and operations project.</p><h3>Why early schema choices stick</h3><p>Before launch, I usually see the same decision process. Teams want the fastest path to production, reasonable cost, and enough isolation to trust it in production. That often leads to a shared-schema model, with a shared database, a shared schema, and a tenant_id on nearly every table. I still think that is the right default for many B2B systems.</p><p>What teams underestimate is that this choice becomes part of the product contract. Queries assume tenant-scoped predicates. Unique constraints become composite, such as (tenant_id, email), so two tenants can each have a user with the same email. Reporting pipelines, exports, audit logs, and cache keys all inherit the same shape. Once real tenants arrive, the schema is no longer only about storage.</p><p>I learned this the hard way on a backend where the initial design looked intentionally temporary. It was not. Once customer integrations depended on stable identifiers and tenant-specific uniqueness, every attempt to “clean it up later” collided with migration windows, support expectations, and API compatibility. The data model had already spread into everything around it.</p><p>Shared schema is still workable if the database enforces isolation instead of relying on every engineer to remember a filter. PostgreSQL <a href="https://www.postgresql.org/docs/current/ddl-rowsecurity.html">row-level security (RLS)</a> gives me that practical baseline, because the database applies the tenant filter to every query. I would rather make the safe path automatic than depend on code review to catch one missing WHERE tenant_id = ?.</p><blockquote><strong>Note:</strong> The trade-off here is between operational simplicity and future isolation flexibility. Shared schema is cheap to run early, but expensive to split later.</blockquote><p>Even if I choose a shared schema, I assume from day one that indexes, constraints, and read paths must be tenant-aware. That is what keeps the model viable after launch. I have also seen this work well when teams make those guardrails explicit early. With that baseline in place, the next problem is usually the first one that hurts: migrations, especially when tenants differ a lot in size.</p><p>To make the operational spread concrete, I would compare the common tenancy models side by side: shared schema, schema per tenant, and database per tenant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gk0_TUiWFbbRxAoEyqX4IA.png" /><figcaption>Three-panel diagram comparing shared database shared schema, schema per tenant, and database per tenant</figcaption></figure><h3>What makes migrations harder</h3><p>A schema change for one tenant is a coding task. The same change for all tenants that share hot tables is a rollout problem. The migration has to survive the largest tenant, the slowest index build, and the highest write rate, not the median case. That difference usually becomes obvious only in production.</p><p>I have seen teams add a required column to a large table and assume the hard part was writing the DDL. It was not. The hard part was avoiding long locks, backfilling under live traffic, and keeping old application versions working while replicas caught up. Every one of those steps competes with live traffic for locks and disk I/O.</p><p>That is also why aggregate latency numbers can be misleading during schema work. Migration pain usually starts in the tails before it appears in broad averages.</p><h3>Why p95 can lie</h3><p>At scale, one oversized tenant can dominate row counts and write amplification. Your p95 may look stable because most tenants are small, while p99 degrades badly when one large tenant causes slower updates or more dead rows for vacuum to clean up. During migrations, I trust per-tenant breakdowns more than blended service averages.</p><ul><li><strong>Hot tables:</strong> A volatile default rewrites the whole table, and SET NOT NULL scans every row while holding a lock. A constant default is fast on modern PostgreSQL.</li><li><strong>Large tenants:</strong> Backfills run longer and increase autovacuum pressure.</li><li><strong>Replica lag:</strong> Write-heavy backfills can push read replicas beyond safe staleness.</li></ul><blockquote><strong>Watch out:</strong> If you benchmark migrations on fixture-sized data, you will badly underestimate lock time and WAL (write-ahead log) volume.</blockquote><p>The model I trust is expand and contract. First, add nullable columns or new tables. Then dual-write, backfill asynchronously, switch reads after validation, and only then enforce constraints. On a large table, I add a CHECK (column IS NOT NULL) constraint as NOT VALID, validate it, and then set NOT NULL, which avoids a long locking scan. Expand and contract is still the safest approach when the migration has to run without downtime. PostgreSQL’s <a href="https://www.postgresql.org/docs/current/sql-altertable.html">ALTER TABLE</a> behavior is better than it used to be, but it still does not remove the need for rollout discipline.</p><blockquote><strong>Practical tip:</strong> Track migration telemetry by tenant, not only by migration version. The outliers determine whether the rollout is safe.</blockquote><p>What I learned from too many late-night migrations is simple. Schema design does not end at DDL. It extends into deployment order, observability, and abort strategy. When teams get this right, the payoff is real: safer rollouts, fewer emergency reversions, and less guesswork about whether the largest tenant will break the window. That leads directly to how tenant identity spreads through the codebase.</p><p>Before going deeper, I would show the rollout pattern I rely on in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-W8hOYb05_LwcyE-gl71nQ.png" /><figcaption>Timeline for expand-and-contract migration</figcaption></figure><h3>How tenant id spreads in code</h3><p>In most codebases, tenancy stops being a column as soon as the second service appears. Request handlers need tenant context. Background jobs need it. Batch exports, admin tooling, and ad hoc scripts need it. If I do not centralize propagation, the isolation model quietly shifts from database enforcement to developer discipline.</p><p>The failure mode I see most often is mundane. One reporting query skips the tenant predicate. One admin endpoint joins through a child table that does not carry tenant_id. One queue consumer reconstructs a query from partial state and forgets the scope. None of these look dramatic in code review. All of them are isolation bugs.</p><p>I prefer to treat tenant context as infrastructure. In Node.js services, request-scoped propagation through <a href="https://nodejs.org/api/async_context.html">AsyncLocalStorage</a> is good enough when it is used carefully. In auth layers, tenant claims inside JWTs are useful, but I still want the database session set explicitly before queries run.</p><h4>Database enforcement matters</h4><p>If I use PostgreSQL RLS, I pair it with session context through current_setting, tenant-aware indexes, and composite foreign keys where possible. I treat these as required, not optional. A child row that can be reached without tenant validation is a future incident. I want the relational model to preserve isolation all the way down the hierarchy.</p><ul><li><strong>Indexes:</strong> Prefer leading (tenant_id, …) when most reads are tenant-scoped. This composite-index strategy, with tenant_id first, usually matches access patterns best.</li><li><strong>Uniqueness:</strong> Encode tenant-local business rules with composite unique constraints.</li><li><strong>Joins:</strong> Carry tenant_id on child tables instead of inferring it indirectly.</li></ul><blockquote><strong>Note:</strong> RLS reduces accidental cross-tenant reads, but it does not fix bad query shapes or weak indexing. You still pay for scans.</blockquote><p>I have also learned to be careful with connection pooling. If tenant session state is set on a pooled connection and is not reset correctly, one request can inherit another tenant’s context. That is why I set the tenant inside each transaction with set_config and its local flag, which is the parameterized form of SET LOCAL, and validate it in integration tests.</p><p>I also make sure the application never connects as the table owner. Owners bypass RLS unless the table uses FORCE ROW LEVEL SECURITY, and superusers and roles with BYPASSRLS skip it entirely.</p><p>A few checks help in practice:</p><ul><li><strong>Propagation:</strong> Fail requests early if tenant context is missing at the service boundary.</li><li><strong>Database state:</strong> Assert the expected tenant session value before running sensitive queries.</li><li><strong>Jobs and scripts:</strong> Require tenant identifiers as explicit inputs instead of reconstructing them implicitly.</li></ul><p>To connect the application layer and the database layer, I would show the pattern I rely on.</p><pre>-- Shared table with a tenant_id on every row<br>ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;<br>-- Apply the policy to the table owner too<br>ALTER TABLE invoices FORCE ROW LEVEL SECURITY;<br><br>-- One policy covers SELECT, INSERT, UPDATE, and DELETE.<br>-- current_setting raises an error if the tenant is missing, so a<br>-- request without tenant context fails instead of returning rows.<br>-- (Assumes tenant_id is bigint; cast to uuid if yours is a uuid.)<br>CREATE POLICY tenant_isolation ON invoices<br>  FOR ALL<br>  USING (tenant_id = current_setting(&#39;app.tenant_id&#39;)::bigint)<br>  WITH CHECK (tenant_id = current_setting(&#39;app.tenant_id&#39;)::bigint);</pre><p><em>An RLS policy on a shared table (SQL), and request-scoped tenant context set per transaction (Node.js)</em></p><p>The application side sets that tenant for each request and passes it to the database inside every transaction, so the policy always has a value to check.</p><pre>import { AsyncLocalStorage } from &#39;node:async_hooks&#39;;<br><br>const tenantStore = new AsyncLocalStorage();<br><br>// Runs after auth. Keeps the tenant for the whole request.<br>function tenantMiddleware(req, res, next) {<br>  const tenantId = req.auth?.tenantId; // from a verified JWT claim<br>  if (!tenantId) return res.status(401).end();<br>  tenantStore.run({ tenantId }, next);<br>}<br><br>// Every query runs in a transaction with the tenant set locally<br>async function withTenant(pool, fn) {<br>  const { tenantId } = tenantStore.getStore();<br>  const client = await pool.connect();<br>  try {<br>    await client.query(&#39;BEGIN&#39;);<br>    await client.query(<br>      &quot;SELECT set_config(&#39;app.tenant_id&#39;, $1, true)&quot;, // true = this transaction only<br>      [String(tenantId)]<br>    );<br>    const result = await fn(client);<br>    await client.query(&#39;COMMIT&#39;);<br>    return result;<br>  } catch (err) {<br>    await client.query(&#39;ROLLBACK&#39;);<br>    throw err;<br>  } finally {<br>    client.release();<br>  }<br>}<br><br>// Usage<br>const result = await withTenant(pool, (c) =&gt; c.query(&#39;SELECT * FROM invoices&#39;));</pre><p><em>An RLS policy on a shared table (SQL), and request-scoped tenant context set per transaction (Node.js)</em></p><p>Once tenant identity is handled safely, the next source of pressure is customization and uneven feature adoption.</p><h3>Tenant customizations and drift</h3><p>After launch, schema pressure usually shifts from isolation to customization. One tenant wants extra compliance metadata. Another wants custom workflow state. A third needs fields that nobody else uses. This is where shared-schema systems start to accumulate drift, even when the isolation model is sound.</p><p>I usually see three options, and each has costs. Add columns for everyone, store tenant-specific attributes in JSON, or split custom data into separate extension tables. The right choice depends on validation needs, query patterns, reporting requirements, and how many tenants actually use the feature.</p><ul><li><strong>Extra columns:</strong> Strong typing and easy joins, but wide sparse tables age badly.</li><li><strong>JSON fields:</strong> Flexible and fast to ship, but validation and indexing become uneven.</li><li><strong>Extension tables:</strong> Clearer boundaries, but more joins and more migration surfaces.</li></ul><p>I have used JSON successfully for low-frequency metadata that is rarely filtered. I avoid it when product managers say, “We might report on this later,” because “later” usually arrives. PostgreSQL <a href="https://www.postgresql.org/docs/current/datatype-json.html">JSONB indexing</a> helps, but it does not remove the complexity of mixed document shapes at scale.</p><blockquote><strong>Watch out:</strong> Feature skew becomes operational skew. Once only some tenants use a new model, every dashboard, script, and support path must account for version compatibility.</blockquote><p>Dual-mode systems are where many teams lose simplicity. Some tenants are on old columns, some are on new tables, and some are on both during migration. That is survivable, but only if the version skew is explicit and observable. By version skew, I mean simple things you can count: how many tenants still read old fields, how many still write both paths, how many records remain unbackfilled, and how long any tenant has remained in an in-between state. This is also where per-tenant migration-version tracking and tracking schema version across tenants become operationally important. The comparison below is the one I would put in front of any team discussing customization paths.</p><h4>Comparison of Various Multi-tenancy Strategies</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*aE6Wopte3QhjxA0iZmKlYg.png" /></figure><h3>Keeping the schema changeable</h3><p>I do not know a schema that stays easy forever. I do know a set of habits that keep the cost bounded. They are not free. Every safeguard adds code paths, rollout time, and cleanup work, so I use them only where the blast radius justifies the overhead.</p><p>First, I make large changes backward-compatible before I make them strict. Reads tolerate old and new shapes. Writes dual-write only when I can measure divergence and reconcile it. For realistic rehearsal, I test against tenant-size distributions that resemble production, not tiny fixtures or synthetic averages.</p><p>Second, I treat migration health as an operating metric. I want lock-wait time, migration duration per tenant, and schema-version skew visible in the same dashboard as p95 and p99 latency. If a rollout cannot be observed, it cannot be trusted.</p><blockquote><strong>Practical tip:</strong> Add a per-tenant migration journal table early. It becomes the source of truth for retries, support, and rollback planning.</blockquote><p>I also force cleanup onto the roadmap. Expand and contract works only if the contract phase actually happens. Old columns, compatibility writes, and feature flags all carry cost. With those habits in place, the remaining question is when the current tenancy model is no longer the right fit.</p><p>To make that visible, I would put the operational signals in one place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x2vK4pAR6_4dLYhcOZNtWw.png" /></figure><p>Dashboard mockup with panels for migration duration by tenant, lock wait time, schema version skew, migration failure rate, and p95/p99 latency broken out by top tenants</p><h3>When to change the tenancy model</h3><p>I have seen teams jump from shared-schema pain straight to “we need database per tenant.” Most of the time, that is the wrong first move. A weak migration process, poor indexing, or missing tenant-aware observability can make an adequate model look broken. I try to separate operational immaturity from true model limits before I redesign anything.</p><p>There are real signals that justify change. Enterprise customers may require stronger isolation boundaries for contractual or audit reasons. Data residency can force regional or tenant-specific placement. Persistent noisy-neighbor behavior may continue even after query tuning, rate limits, and workload shaping. In production, those are architecture inputs, not preferences.</p><p>I also look at migration windows. If one shared table means every significant schema change is gated by the largest tenant, the platform may have outgrown the shared path. This becomes more obvious when a few tenants dominate storage and writes, and your p99 latency tracks their activity more than aggregate load. At that point, the trade-off is between platform simplicity and blast-radius control.</p><h4>Signals that usually justify evolution</h4><ul><li><strong>Isolation requirements:</strong> Customer contracts, compliance, or regulated data handling.</li><li><strong>Workload skew:</strong> A few dominant tenants distort p99 latency and lock behavior.</li><li><strong>Operational drag:</strong> Migrations, backups, and restores are too tightly coupled across tenants.</li></ul><p>When I do evolve, I rarely move all tenants at once. A hybrid model is often the practical bridge. Keep most tenants in a shared schema. Move the largest or most sensitive tenants to dedicated schemas or databases. That lets me reserve the extra operational burden for the tenants that actually need it. In practice, this is often the real answer to when to use schema-per-tenant instead of a shared schema.</p><p>This hybrid approach comes with real cost. Provisioning becomes more complex. Backup and restore paths multiply. Deployment tooling must target different tenancy classes. If the team is small, the control-plane work can become its own bottleneck. That is why I pay that tax only when the shared model is causing sustained harm.</p><p>I also push back when the evidence shows that the current design is fine. If RLS is doing its job, cross-tenant incidents are absent, and the main pain is undisciplined rollouts, I fix rollout mechanics first. Better migration sequencing, tenant-aware dashboards, and realistic load tests are cheaper than a tenancy redesign, and they often solve the actual problem. That is also the point at which deeper references help. For background, Educative’s <a href="https://www.educative.io/courses/the-art-of-postgresql?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=the_art_of_postgresql&amp;eid=5082902844932096">The Art of PostgreSQL</a> covers schema modeling and refinement, and its <a href="https://www.educative.io/courses/mastering-postgre-sql-databases-from-basics-to-advanced?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=mastering_postgresql_databases&amp;eid=5082902844932096">Mastering PostgreSQL Databases</a> course covers the fundamentals. When I need to check a decision against how the database actually behaves, I go to the <a href="https://www.postgresql.org/docs/current/indexes.html">PostgreSQL documentation on indexes</a>.</p><p>My rule is simple. Change the tenancy model when the current one repeatedly violates isolation, compliance, or latency goals even after competent operational fixes. Do not change it because migrations feel annoying. Annoying is often a tooling problem. Structural pain has a different signature, and I wait for that evidence before I add permanent complexity.</p><h3>Conclusion</h3><p>Multi-tenant database schema decisions become expensive to change because production turns every shortcut into a concrete constraint. Real tenant-size skew, rollout safety, and isolation guarantees turn a simple table design into an operating model. I still start with a shared schema often, but I trust it only when tenant context is enforced, migrations are staged, and observability is tenant-aware. If I keep those disciplines sharp, I can delay heavier tenancy models until the evidence is strong.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d2185f60054" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Asking AI for the Answer Skips the Part Where You Learn SQL]]></title>
            <link>https://medium.com/@mark.w.1987/asking-ai-for-the-answer-skips-the-part-where-you-learn-sql-12bbfccbac23?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/12bbfccbac23</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 08:28:40 GMT</pubDate>
            <atom:updated>2026-11-19T11:46:19.754Z</atom:updated>
            <content:encoded><![CDATA[<p>A query that returns rows can still be wrong, slow, and hard to debug. I learned that lesson the hard way: a report looked fine in staging, then doubled customer totals in production because an orders-to-payments join turned one row per order into many. The ticket closed quickly. The cleanup took most of the next morning.</p><h3>The shortcut that costs learning</h3><p>When a backend ticket calls for a report, it is tempting to paste the schema into AI and move on. I have done that myself during late-night fixes, when speed mattered more than elegance. The problem is simple: getting rows back is not the same as understanding joins, foreign keys, filters, or how the database will behave under load.</p><p>That gap in understanding usually stays hidden until code review or incident response. I have watched queries pass tests and then fail against real cardinality, nulls, and duplicate relationships. One beginner I mentored checked only the final count, saw “close enough,” and missed the fact that a one-to-many join had inflated revenue totals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*meqZboThDwenuK3fEdnrWw.png" /><figcaption>AI SQL vs. schema understanding</figcaption></figure><h3>AI writes SQL, not understanding</h3><p>AI is good at producing familiar SQL patterns. It can generate SELECT, JOIN, WHERE, GROUP BY, and <a href="https://www.educative.io/answers/window-functions-in-sql?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=window_functions&amp;eid=5082902844932096">window functions</a> that look clean enough to pass a quick review. For beginners, that polish creates a false sense of safety.</p><p>I have mentored developers who copied a query that “worked” but could not explain basic choices:</p><ul><li>why INNER JOIN dropped rows they expected to keep</li><li>why aggregation changed after a one-to-many join</li><li>why filtering before or after grouping produced different counts</li></ul><blockquote><strong>Note:</strong> A result set proves syntax and some logic. It does not prove that the query matches the business rules.</blockquote><p>A common failure is duplicate row inflation. In one report, staging returned 1,240 orders, while production jumped to 2,730 joined rows because some orders had multiple payment records. The fix was not exotic. We had to compare pre-join and post-join row counts, confirm that payments was on the many side, and then aggregate payments first before joining them back to orders.</p><h4>SQL Concepts: What AI Gives You vs. What You Still Need to Know</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSizZMpJw4a8adYmDH7UJw.png" /></figure><h3>The cost appears in backend work</h3><p>The hidden cost appears when the query leaves toy data and hits a real workload. In backend systems, I care less about whether SQL parses and more about whether it scans 5,000 rows or 50 million. A query can look clean and still force work that the database cannot hide.</p><blockquote><strong>Watch out:</strong> A query can pass a test and still fail in production.</blockquote><p>Years ago, I debugged a reporting query with a missing date predicate. It ran in about 120 ms on a narrow staging slice and then took roughly 8 seconds in production because it fell back to a full scan. The generated query did not account for available indexes, stale statistics, or the optimizer’s join order, so the plan did far more work than the author expected. That is why I always check plans with tools like <a href="https://www.postgresql.org/docs/current/sql-explain.html">PostgreSQL EXPLAIN</a> and compare behavior with the <a href="https://www.sqlite.org/queryplanner.html">SQLite query planner docs</a> when I teach fundamentals.</p><p>Business correctness can break as well. An INNER JOIN can silently drop users with no payments, which may be fine for finance and wrong for customer support. Once you own a backend service, you also own correctness and performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJnqPOj0ak_379j9CVcxrQ.png" /><figcaption>Query execution and performance pipeline</figcaption></figure><h3>Use AI like a SQL tutor</h3><p>My rule is simple: make AI explain the query. I try the query first, even if the draft is rough. Then I ask narrower questions about the part I do not understand.</p><h4>A better prompt pattern</h4><p>I ask AI to:</p><ul><li>justify the join path from foreign keys</li><li>compare LEFT JOIN vs. INNER JOIN</li><li>generate sample rows and walk the result step by step</li><li>predict how row counts change before and after each join</li><li>suggest what EXPLAIN output I should expect</li></ul><blockquote><strong>Practical tip:</strong> If you cannot explain each clause in your own words, do not ship the query yet.</blockquote><p>For performance, I ask a different set of questions. I want the likely plan shape, selectivity assumptions, whether an index changes access from scan to seek, and what happens if null-heavy data or skewed values appear. Official references like <a href="https://dev.mysql.com/doc/refman/8.0/en/explain.html">MySQL EXPLAIN</a> and <a href="https://www.postgresql.org/docs/current/planner-stats.html">PostgreSQL planner statistics</a> help ground that discussion.</p><p>If you need a quick refresher on joins before you debug this kind of query, an <a href="https://www.educative.io/courses/sql-interview-patterns?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educative_walk_through_on_sql_basics&amp;eid=5082902844932096">Educative walk-through on SQL basics</a> can help. The point is to understand the query well enough to challenge it.</p><pre>-- Goal: monthly revenue by customer segment for active customers with recent orders.<br><br>-- Naive version:<br>-- EXPLAIN (estimated)<br>--   rows: ~12,000,000 orders x ~500,000 customers<br>--   index usage: orders(created_at) only; customers scanned repeatedly by join<br>--   cost issue: joins all matching rows first, then aggregates; can multiply rows if another 1-to-many table is added later<br>SELECT<br>  c.segment,<br>  DATE_TRUNC(&#39;month&#39;, o.created_at) AS order_month,<br>  SUM(o.amount) AS total_revenue<br>FROM orders o<br>JOIN customers c<br>  ON c.id = o.customer_id<br>WHERE o.created_at &gt;= DATE &#39;2025-01-01&#39;<br>  AND c.status = &#39;active&#39;<br>GROUP BY c.segment, DATE_TRUNC(&#39;month&#39;, o.created_at)<br>ORDER BY order_month, c.segment;<br><br>-- Improved version:<br>-- EXPLAIN (estimated)<br>--   rows: ~1,200,000 filtered orders -&gt; ~200,000 monthly aggregates<br>--   index usage: orders(customer_id, created_at) or orders(created_at, customer_id); customers(id, status)<br>--   why faster: pre-aggregates orders before joining to customers, reducing join volume and avoiding duplicate aggregation work<br>WITH recent_orders AS (<br>  SELECT<br>    o.customer_id,<br>    DATE_TRUNC(&#39;month&#39;, o.created_at) AS order_month,<br>    SUM(o.amount) AS monthly_revenue<br>  FROM orders o<br>  WHERE o.created_at &gt;= DATE &#39;2025-01-01&#39;<br>  GROUP BY o.customer_id, DATE_TRUNC(&#39;month&#39;, o.created_at)<br>)<br>SELECT<br>  c.segment,<br>  r.order_month,<br>  SUM(r.monthly_revenue) AS total_revenue<br>FROM recent_orders r<br>JOIN customers c<br>  ON c.id = r.customer_id<br>WHERE c.status = &#39;active&#39;<br>GROUP BY c.segment, r.order_month<br>ORDER BY r.order_month, c.segment;&quot;<br>}</pre><h3>A practice loop that builds skill</h3><p>What works best when I mentor beginners is a short loop that they can repeat without much setup. I ask them to attempt the query first, get stuck honestly, and then ask AI for a hint instead of a finished answer. That keeps the hard part of learning in their hands.</p><h4>The loop I use</h4><ul><li>attempt the query on paper or in the editor</li><li>compare base-table counts before the join with joined row counts after it</li><li>verify results against sample rows and explain the output aloud</li><li>inspect EXPLAIN to see whether the plan matches your assumptions</li><li>test one edge case with nulls or duplicates</li><li>solve a nearby variation without looking back</li></ul><blockquote><strong>Note:</strong> Small variations matter. Change one filter, one grouping key, or one join type, and see what breaks.</blockquote><p>I have seen this pay off quickly. One junior developer kept joining users to logins and counting rows, which overstated active users because heavy users had many login records. After we compared pre-join and post-join counts, they changed the query to count distinct user IDs, checked the plan, and caught the bug before it reached production.</p><p>Over time, terms like cardinality, B-tree indexes, and vectorized execution stop sounding abstract because you have seen their effects directly. A direct reference like the <a href="https://www.postgresql.org/docs/current/sql-select.html">PostgreSQL SELECT documentation</a> is also more useful here than vague summaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OyElLqqQ7G6TSBA4mg20Q.png" /><figcaption>Five-step SQL practice loop</figcaption></figure><h3>Optimize for understanding, not speed</h3><p>I am fine with using AI for SQL. What I do not trust is speed by itself. In production, somebody still has to own correctness, explain the query plan, reason about indexes and data distribution, and decide whether the latency profile is acceptable.</p><blockquote><strong>Watch out:</strong> Fast answers can hide slow scans, wrong joins, and missing edge cases.</blockquote><p>If you are new to backend work, use AI for hints, explanations, and comparisons. Then verify the parts that usually break: join cardinality, null handling, duplicate rows, and expected plan shape. The next time AI gives you a query, ask it to explain the join cardinality and the expected plan before you run it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12bbfccbac23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sequential API Calls Quietly Turn Into a Frontend Performance Problem]]></title>
            <link>https://medium.com/@mark.w.1987/sequential-api-calls-quietly-turn-into-a-frontend-performance-problem-99e537b3aae8?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/99e537b3aae8</guid>
            <category><![CDATA[web-performance]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[frontend-development]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 09:58:29 GMT</pubDate>
            <atom:updated>2026-11-03T09:58:29.268Z</atom:updated>
            <content:encoded><![CDATA[<p>A page can feel slow even when no single API appears problematic on its own, and that delay shows up in places teams actually care about: conversion, bounce rate, and whether a screen feels trustworthy enough to use. I have debugged this more than once at 1:00 a.m., including one product page that looked fine service by service but still took long enough to render that users dropped off before pricing and inventory settled. The browser did exactly what the code asked it to do: wait for the session, then the profile, then pricing, then inventory. Each call looked reasonable in isolation, but the user paid for the entire chain of sequential API calls.</p><p>What looks like a frontend bug is often a critical-path issue and a real frontend performance problem. If five dependencies must complete before the page is useful, client-visible latency becomes roughly the sum of their tail latencies, not their averages. When I say P95 and P99, I mean the slower end of the latency distribution: the time under which 95 percent or 99 percent of requests finish. I care much more about those numbers here than I do about mean response time, because one slow dependency stretches the entire path.</p><p>I learned this the hard way on product surfaces that combined identity, entitlement, pricing, and recommendations. The frontend looked tidy, but hidden data dependencies and conditional fetches serialized the request graph. That kind of fetch-waterfall frontend performance issue is easy to miss if you inspect only one service at a time. Reviewing distributed traces instead of waterfall screenshots revealed architectural issues rather than only JavaScript problems.</p><blockquote><strong>Note:</strong> Average latency hides the user experience when multiple synchronous dependencies sit on the critical path.</blockquote><p>Before I get into fixes, it helps to make the latency chain visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJeR11N1YEtc9PkOc4DrOA.png" /><figcaption>Sequential calls create additive latency on the client-visible path</figcaption></figure><h3>How the latency chain really forms</h3><p>The browser sequence is only the first layer. I often find a second chain behind it: the gateway calls a user service, which calls entitlement, which blocks on a regional cache miss and falls through to a database read. From the client, that still looks like one request, but the inclusive span time keeps growing as each upstream service waits.</p><p>Critical-path analysis keeps this honest. If work is synchronous, it sits on the path the user feels. If work is asynchronous, such as event publication or nonblocking logging, it may affect throughput later but not the initial render. The mistake teams make is counting all service time equally instead of asking which spans must finish before the UI becomes useful.</p><ul><li><strong>Synchronous work:</strong> directly adds to client-visible latency.</li><li><strong>Asynchronous requests:</strong> affect cost and capacity, but not always first paint.</li></ul><p>Backpressure appears next. A stalled RPC holds threads, connection-pool slots, or event-loop tasks, and that inflation propagates upward. I have seen a single dependency with poor timeout hygiene make healthy upstream services look slow because their inclusive spans absorbed downstream waiting. OpenTelemetry’s tracing model is a useful reference here because it makes span relationships and waiting time explicit, and large gateway systems such as <a href="https://netflixtechblog.com/all?topic=edge-engineering">Netflix’s edge-architecture articles</a> are worth reading once you start seeing this pattern in production. You can review the trace basics in <a href="https://opentelemetry.io/docs/concepts/signals/traces/">OpenTelemetry tracing concepts</a>.</p><blockquote><strong>Watch out:</strong> A service with low exclusive time can still be the reason your page is slow if it spends most of its life waiting on another service.</blockquote><p>Once the failure mode is clear, the useful question becomes which orchestration pattern shortens that path instead of merely relocating it. That is the point where the problem usually stops being “why is this component slow?” and becomes “who should coordinate these dependencies?”</p><p>Before I compare options, here is a compact decision table I use with teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88ZdfDTHtlZHX5ZQYBnzBg.png" /></figure><h3>Where aggregation helps the most</h3><p>When the browser is coordinating application logic, I usually move orchestration into a BFF or edge layer. That lets me issue parallel fetches close to the services, apply consistent timeouts, and return one payload shaped for the screen. It also gives me a clean place to decide which dependencies are hard requirements and which are enrichments.</p><p>Parallel fan-out is the first win, but reducing request count matters too. On mobile networks, one request with controlled backend fan-out is often better than six browser requests with separate TCP and TLS costs. This is the practical heart of sequential vs. parallel API calls: independent requests should not wait on each other, while dependent API requests sometimes must. If you want a solid review of API gateway and microservice coordination patterns, <a href="https://www.educative.io/path/scalability-system-design?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_system_design_path&amp;eid=5082902844932096">Educative’s system-design path</a> is useful because it walks through service-boundary and dependency trade-offs in a structured way, and the <a href="https://spec.graphql.org/">GraphQL specification</a> is still the right reference for understanding the query model itself.</p><p>I use a few patterns repeatedly:</p><ul><li><strong>BFF aggregation:</strong> good when one frontend owns the page contract.</li><li><strong>Batch RPC:</strong> good when many small calls hit the same downstream domain.</li><li><strong>Gateway choreography:</strong> useful when multiple clients need shared composition logic.</li></ul><p>Each has real costs. A BFF can drift into a grab bag of business logic if ownership is unclear. GraphQL or federation can reduce overfetching, but it can also hide expensive resolver waterfalls unless you batch carefully. An API gateway can improve cache locality at the edge, yet it can also become a new synchronous bottleneck with poor failure isolation.</p><blockquote><strong>Practical tip:</strong> Treat recommendations, badges, and secondary panels as fail-open enrichments unless the business truly requires them for first render.</blockquote><p>I also prefer explicit contracts over generic aggregation. A page-specific endpoint is easier to cache, trace, and evolve than a universal query layer that tries to serve every use case. That trade-off is less fashionable, but it reduces bugs and makes debugging easier. In frontend terms, it is often the cleaner form of client-side optimization because it removes avoidable coordination from the browser instead of asking the client to manage a rendering waterfall.</p><h3>What breaks on real networks</h3><p>A lot of latency advice assumes stable desktop connections. In practice, mobile clients are where clean architectures get exposed. With packet loss, radio handoffs, or captive-portal behavior, even parallel browser calls suffer because each request has its own retransmission risk, congestion behavior, and header overhead.</p><p>That changes the economics. Reducing six requests to one aggregated request often beats shaving 20 ms from a backend handler, because the unstable part is the network path, not CPU time. I have seen teams celebrate faster service medians while users on 4G still waited on retries and connection churn. In one rollout, the backend became modestly faster, but the user-visible improvement came only after we cut request count and let optional panels miss their budget without blocking the page.</p><p>A few patterns hold up well:</p><ul><li><strong>Selective caching:</strong> keep session-independent data or recent page fragments locally with sane TTLs.</li><li><strong>Request coalescing:</strong> collapse duplicate in-flight fetches so one screen transition does not stampede the same dependency.</li><li><strong>Fail-open enrichments:</strong> omit noncritical panels when they miss budget.</li></ul><blockquote><strong>Note:</strong> Tail latency on mobile often comes from retries, handshake costs, and packet-loss amplification more than raw server compute.</blockquote><p>Client-tolerant degradation matters too. If core content can render with stale recommendations or no social proof, I take that trade every time. This is also where <a href="https://www.rfc-editor.org/rfc/rfc9110">RFC 9110 HTTP semantics</a> and caching guidance become practical, not theoretical, and <a href="https://www.educative.io/path/zero-to-hero-in-back-end-web-development?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_backend_development_courses&amp;eid=5082902844932096">Educative’s backend-development courses</a> are useful when you want worked examples of graceful degradation, caching, and timeout trade-offs rather than only theory. These are also the moments when caching strategies to cut API latency have more impact than another small handler-level optimization.</p><p>To make that contrast concrete, I like to visualize the request timelines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k7nP1nNBwABGmY4FzMA7Gw.png" /></figure><h3>How I verify the fix worked</h3><p>I do not trust a redesign until I can prove the critical path got shorter in production. That means distributed traces, request fan-out counts, per-dependency timeout rates, and client-visible page latency at P50, P95, and P99. I also check inclusive vs. exclusive span time, because that tells me whether I reduced actual waiting or merely moved it.</p><p>My review checklist is simple:</p><ul><li>Did browser request count drop?</li><li>Did synchronous downstream calls shrink?</li><li>Did tail latency improve under a real traffic mix?</li></ul><p>I also compare partial-success rates. If the new path returns usable payloads during recommendations or profile timeouts, that is a real reliability gain, not just a latency gain. Good tracing with <a href="https://www.jaegertracing.io/">Jaeger</a> or vendor tooling should show the before-and-after critical path directly. In practice, I usually look first for traces where the parent span gets shorter because downstream waits disappeared, not because the work was renamed or split across more spans. If you want a structured refresher on those distributed-systems mechanics, <a href="https://www.educative.io/courses/deleted/web-application-software-architecture-101?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_distributed_systems_material&amp;eid=5082902844932096">Educative’s distributed-systems material</a> is a practical place to review failure handling and service-interaction patterns.</p><blockquote><strong>Watch out:</strong> Faster averages with an unchanged P99 often mean you optimized noncritical work and left the real bottleneck in place.</blockquote><p>One quantitative check I like is simple: if total page usefulness depends on $n$ serialized calls, expected tail behavior grows with the slowest path segment and the accumulated waits around it. I do not need perfect queuing theory to act on that. I need traces, budgets, and enforced timeouts. That is usually enough to diagnose fetch waterfalls in React or any other frontend stack, even if the root cause sits deeper in the service graph.</p><p>If a frontend feature depends on several services, I assume orchestration is part of backend design. That framing changes ownership, observability, and where I spend effort. The best fixes reduce request count, parallelize hard dependencies, and degrade optional data cleanly.</p><h3>The takeaway</h3><p>Sequential API calls from the browser are rarely just a frontend implementation detail. I treat them as a system-design problem because users feel the entire synchronous chain, especially at P95 and P99. My default move is to collapse orchestration into a backend layer that can parallelize, time-box, and fail open where it is safe. If the traces do not show a shorter end-to-end critical path, I assume I have not fixed the problem yet. In other words, the real performance bottleneck that sequential API calls create is not any one slow request, but the fetch waterfall that makes the entire screen wait.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99e537b3aae8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reviewing AI-Generated Code Is Now a Junior Developer’s First Real Skill]]></title>
            <link>https://medium.com/@mark.w.1987/reviewing-ai-generated-code-is-now-a-junior-developers-first-real-skill-19f692616e75?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/19f692616e75</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[code-review]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 05:18:26 GMT</pubDate>
            <atom:updated>2026-11-18T12:51:30.232Z</atom:updated>
            <content:encoded><![CDATA[<p>The junior developer I trust most is not the one who writes the fastest CRUD handler. It is the one who can look at an AI-generated pull request and tell me, with evidence, why it should not ship. I have seen a PR pass CI, merge cleanly, and still break a retry path so one payment attempt turned into three. That shift is already visible on backend teams I work with. Syntax is cheaper now. Judgment is not.</p><p>I have seen juniors pass unit tests and still miss a retry loop that turned one payment attempt into three. I have also seen them reject a clean diff because it changed an API contract in a way the tests never modeled. That is the first real engineering skill: evaluating behavior, failure modes, and fit with the system. It is also why <strong>reviewing AI-generated code</strong> is now a junior developer’s first real skill, and why an <strong>AI code review checklist</strong> matters.</p><blockquote><strong>Note:</strong> Passing tests only proves that a specific set of assertions held. It does not prove compatibility, operability, or safe rollback.</blockquote><p>That is why I now teach review before authorship. The next part is the workflow that makes that review useful instead of ceremonial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drC5Pq1gntYjZ9Ij2Qmoww.png" /><figcaption>AI code review workflow with risk-based routing.</figcaption></figure><h3>How the review pipeline actually works</h3><p>The backend pattern I see most often is simple: generate code in isolation, run tests and linters, open a pull request, and require a human decision at the end. In practice, the useful parts are the gates around that flow: required CI checks, CODEOWNERS routing, a PR template that asks for prompt provenance, and a limit on how many files a generated change can touch. Tools like<a href="https://docs.github.com/en/actions/get-started/understand-github-actions"> GitHub Actions</a> and specifications like the <a href="https://spec.openapis.org/oas/latest.html#introduction">OpenAPI specification</a> make the mechanics easy. The hard part is preserving trust in the diff, which is why a strong <strong>review workflow for AI code pull requests</strong> matters.</p><p>A good pipeline keeps three things visible:</p><ul><li>who or what generated the change</li><li>what checks actually ran</li><li>whether the diff stayed small enough that a reviewer can reason about it without reconstructing the whole subsystem</li></ul><p>When those constraints are ignored, approval becomes a queue-clearing exercise. I want generated changes tagged, linked to prompts or tickets, and bounded by file scope. In practice, once a diff spans handlers, persistence logic, and a migration, review quality drops faster than most dashboards show. That is also where <strong>spec drift and scope creep in AI-generated code</strong> become harder to spot.</p><blockquote><strong>Practical tip:</strong> Track median changed files, not only lines added. Cross-file spread is often a better proxy for review difficulty.</blockquote><p>The split between automation and judgment is easier to see when I map it stage by stage. That comparison leads to the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pngr5YZxSVSvReu9-3H6sw.png" /></figure><h3>Where review collapses under AI volume</h3><p>The failure mode I underestimated first was not bad code generation. It was reviewer overload. Once AI-assisted output increases, review demand scales faster than human attention. If throughput rises while semantic diff size also grows, effective review capacity can fall fast.</p><p>I have watched this happen on service teams with healthy CI. Large PRs looked clean, tests passed, and approvals still became shallow because nobody had time to verify intent across files. In one case, the change touched a handler, a queue consumer, and a migration in the same PR. The handler looked fine on its own, the tests passed, and the combined effect was a timeout change that made retries behave differently in production. Those are the <strong>common mistakes in AI-generated code</strong> that slip through when review becomes rushed.</p><blockquote><strong>Watch out:</strong> Broad refactors with “no behavior change” in the summary deserve extra skepticism when generated code touched retries, caching, or transaction boundaries.</blockquote><p>Typical symptoms show up quickly:</p><ul><li>review latency rises from hours to days</li><li>senior reviewers batch-approve</li><li>defect escape rate climbs after merge</li></ul><p>If I want to make the bottleneck visible to the team, I put workload and defect signals side by side. The next dashboard is the one I wish I had adopted earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_KNZrKwJWbHI9_RPshbC3w.png" /><figcaption>Reviewer load becomes the bottleneck before code generation does</figcaption></figure><h3>What juniors should learn to inspect first</h3><p>When I mentor juniors, I tell them to ignore style first and inspect behavior. Generated code often reads smoothly. That is exactly why it is risky. The right review order is system impact before implementation neatness, similar to how<a href="https://www.postgresql.org/docs/current/transaction-iso.html"> PostgreSQL transaction isolation docs</a> explain correctness before syntax.</p><p>My baseline checklist is short:</p><ul><li>schema or index changes</li><li>backward-compatible migrations</li><li>API compatibility and idempotency</li><li>retries, timeouts, auth, monitoring, and rollback behavior</li></ul><p>If a change touches a queue consumer, I ask whether duplicate delivery is safe. If it changes a handler, I ask whether old clients still work. If it adds a query, I look for hidden fan-out and N+1 behavior. That review habit is worth more than producing another endpoint from scratch. It is also the core of<strong> testing and verifying AI-generated code</strong> and <strong>validating dependencies and APIs in AI code</strong>.</p><blockquote><strong>Practical tip:</strong> Read the diff as a behavior change request, not as a writing sample. That framing reduces bugs and makes debugging easier.</blockquote><p>Teams usually standardize expectations next. A lightweight artifact helps, as long as nobody mistakes it for judgment. For junior devs, this is also where<strong> how to review AI-generated code</strong> as a junior developer becomes a repeatable skill instead of a vague expectation. Before approving any AI-generated backend change, I want these six questions answered:</p><ul><li><strong>Schema impact:</strong> Does this change a table, column, or index, and can the old code still run against it? Is there a backfill or compatibility plan?</li><li><strong>API compatibility:</strong> Will existing clients break? Are request shapes, status codes, and error responses unchanged, or is the change versioned and called out?</li><li><strong>Side effects:</strong> Does it touch queues, caches, jobs, or webhooks? If a call is retried or delivered twice, is that safe?</li><li><strong>Rollback plan:</strong> If this fails after deploy, how do we undo it? Is there a tested migration rollback or a feature flag to turn it off?</li><li><strong>Monitoring updates:</strong> Would we notice if it broke? Are logs, metrics, and alerts updated for the new failure cases?</li><li><strong>Provenance:</strong> Is the AI-generated code labeled, and are the prompt, the review owner, and the approval recorded?</li></ul><h3>The signals that catch silent quality decay</h3><p>The earliest warning is often not in the diff. It is in operational drift around the workflow. I watch CI failure rate, flaky integration tests, rollback frequency, and review latency together. On teams using generated code heavily, I also watch token spend, completion volume, and cost per query. If those numbers climb while merge quality drops, the team is probably generating too much low-value output and reviewing it too late.</p><p>I like signals that connect engineering output to system health:</p><ul><li>rising flaky test counts</li><li>more rollbacks per merged PR</li><li>cost anomalies without throughput gains</li></ul><p>You can wire many of these through<a href="https://opentelemetry.io/docs/"> OpenTelemetry</a> and standard delivery metrics. If a team wants structured practice, I usually point them to Educative’s <a href="https://www.educative.io/courses/grokking-the-system-design-interview?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=grokking_the_system_design_interview&amp;eid=5082902844932096">Grokking the System Design Interview</a> course for review heuristics around contracts, queues, and storage behavior. The workflow matters more than the model, especially when you are watching for <strong>security vulnerabilities in AI-generated code</strong> and <strong>handling edge cases in AI-generated </strong><a href="https://www.educative.io/blog/code-review-best-practices?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=code_review&amp;eid=5082902844932096"><strong>code review</strong></a>.</p><blockquote><strong>Note:</strong> Quality decay often appears as broader, less consistent changes before any single PR looks obviously dangerous.</blockquote><p>Those signals only help if the team acts on them. That takes me to the final trade-offs.</p><h3>How teams keep standards without slowing down</h3><p>I do not want humans rereading every generated line forever. I want risk-based routing, smaller semantic changes, and clear provenance so human attention lands where it matters. Low-risk refactors can move faster. Schema changes, cross-service edits, and auth paths need stricter review and often a second set of eyes.</p><p>That is where junior developers become genuinely valuable. If they can reject a polished PR for the right reasons, they are doing real engineering. I have found the healthiest backend teams treat review as a systems problem with capacity limits, trust boundaries, and feedback loops. That is also why<strong> best practices for reviewing AI-generated code</strong> should include ownership, not just approval.</p><blockquote><strong>Watch out: </strong>Speed without review design usually creates delayed rework, not durable throughput.</blockquote><p>My advice is simple. Keep generated diffs small. Make provenance explicit. Train juniors to inspect contracts, side effects, and rollback safety first. That discipline scales better than pretending the code is the easy part. It also helps teams <strong>review AI code before merge</strong> with confidence and avoid the <strong>security checklist for AI-generated code</strong> mistakes that are easiest to miss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19f692616e75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conway’s Law Still Shapes How Your Frontend and Backend Teams Split Their Code]]></title>
            <link>https://medium.com/@mark.w.1987/conways-law-still-shapes-how-your-frontend-and-backend-teams-split-their-code-c97f225d2228?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c97f225d2228</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 08:52:03 GMT</pubDate>
            <atom:updated>2026-11-02T08:52:03.085Z</atom:updated>
            <content:encoded><![CDATA[<p>I can usually predict the real architecture of a product by looking at who reviews pull requests and who owns deployments, not by reading the system diagram. I learned that lesson after more than one reorg turned a clean end-to-end flow into a frontend package, an API contract, two release trains, and a queue. The code did not change first. The ownership model did. That is why questions about Conway’s Law in frontend and backend design usually appear first as delivery pain, not as differences in <a href="https://www.educative.io/courses/software-architecture-in-applications?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=architecture&amp;eid=5082902844932096">architecture</a> diagrams.</p><h3>Why a reorg reveals real seams</h3><p>I have seen this pattern repeatedly. One quarter, a single team owns a customer flow from a button click to a database write. The next quarter, the frontend team owns a UI package, the backend team owns APIs and a service, and the same change now crosses versioned contracts, release gates, and separate on-call rotations.</p><p>That is the point at which Conway’s Law becomes operational rather than theoretical. Reporting lines change faster than domain boundaries, so the system is forced to reveal its actual seams. If a split reflects a real bounded context, the new ownership usually feels cleaner after a short adjustment. If it reflects the org chart instead, the coordination cost appears immediately because the organizational structure mirrors the architecture whether you intended it to or not.</p><p>At a previous company, I watched a checkout path move from one repository to three. That was Conway’s Law in practice. Once different teams owned each step, the software started to behave like the org chart. The architecture review claimed that nothing fundamental had changed. In practice, a field rename became a schema review, a client update, and a staged rollout because one in-process call had become a cross-team contract. Lead time for small changes roughly doubled because each repository added another review and release step.</p><blockquote><strong>Note:</strong> A boundary becomes real the moment a safe refactor turns into a compatibility problem.</blockquote><p>Before I get into smells and healthy signals, it helps to visualize how the same flow changes once ownership splits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6MjN4JN2V8Mq1UvynL_fiA.png" /><figcaption>One product flow before and after a team split</figcaption></figure><h3>Why the org chart usually wins</h3><p>In my experience, the org chart wins because communication cost is immediate, while architecture intent is delayed. Teams pull code toward what they can review, deploy, support on call, and scale without waiting on another manager’s backlog. Repository ownership and CI boundaries harden into architecture long before anyone proves that they align with a stable domain.</p><p>A healthy split needs something objective behind it:</p><ul><li>Authoritative data ownership</li><li>A compliance boundary such as PCI scope</li><li>A scaling profile that changes independently</li></ul><p>Without one of those conditions, a frontend-backend split often behaves like one coupled system with extra ceremony. I have seen teams use a <a href="https://www.apollographql.com/docs/">GraphQL BFF pattern from Apollo’s docs</a> to restore local autonomy. By BFF, I mean a backend-for-frontend layer that reshapes backend data around one UI’s needs so that the frontend does not have to coordinate directly with several services. The backend-for-frontend pattern can help when the UI genuinely needs a tailored contract, but if both sides still change together every sprint, the BFF mostly hides coupling rather than reducing it. For a second view of the trade-offs, the <a href="https://spec.graphql.org/">GraphQL specification</a> is also useful.</p><blockquote><strong>Practical tip:</strong> Ask who can change a rule without cross-team approval. That usually tells me more than a domain-model workshop.</blockquote><p>I also watch deployment cadence. If the UI deploys daily but the API requires a coordinated release window, the boundary is already carrying operational cost. That cost might be worth it, but only if the split isolates risk or ownership in a meaningful way.</p><p>To make that evaluation concrete, I use a simple checklist rather than intuition.</p><h3>Microservice Boundary Assessment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*zUjNrsjuWIiyicPi1dcEFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*EftQSM5-LHh1iIEEqhfLHg.png" /></figure><h3>What changes when code splits</h3><p>That ownership split is where the technical shape of the system starts to change, not just the reporting structure. The boxes on the diagram may look familiar, but the work required to change them is different.</p><p>The expensive part is not the box-and-arrow redraw. The expensive part is what happens to code paths that used to be local. A function call becomes an HTTP call or an event. Shared types become contracts. A refactor becomes a migration plan, especially once API-contract versioning begins to matter across separately owned repositories.</p><p>I usually explain this with orders and inventory. In one process, I can update order state and reserve inventory in one database transaction. After a split, the trade-off is between keeping a synchronous chain for stronger consistency and moving to events while accepting delayed convergence.</p><p>That change reaches the frontend quickly:</p><ul><li>Pending, confirmed, failed, and retrying states become first-class states</li><li>Validation is duplicated or extracted into a shared package</li><li>Idempotency keys and replay handling become mandatory</li></ul><p>I have seen teams underestimate this because the first implementation still looks small. Then retries arrive, consumers lag, and someone discovers that the UI assumed reads were current when the read model was seconds behind because of queue backpressure.</p><blockquote><strong>Watch out:</strong> Shared validation packages often recreate coupling through version pinning and synchronized upgrades.</blockquote><p>For teams that are new to this transition, I often point them to the <a href="https://httpwg.org/specs/">HTTP semantics specification</a> and practical <a href="https://www.educative.io/courses/distributed-systems-practitioners?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=event_driven_architecture_material_on_educative&amp;eid=5082902844932096">event-driven architecture material on Educative</a>. Those resources help because protocol and delivery semantics usually become the hard part before code volume does.</p><p>The difference is easier to see in a compact before-and-after path.</p><pre>function placeOrder(orderDetails) {<br>  // Write local order state<br>  saveOrderState(orderDetails);<br><br>  // Publish OrderPlaced event for the inventory service to consume<br>  eventBus.publish(&quot;OrderPlaced&quot;, orderDetails);<br><br>  // Strong consistency is lost since order and inventory updates<br>  // are now asynchronous.<br>}<br><br>// Inventory service consuming the event asynchronously<br>eventBus.subscribe(&quot;OrderPlaced&quot;, (orderDetails) =&gt; {<br>  // Retry logic for eventual consistency<br>  try {<br>    updateInventory(orderDetails);<br>  } catch (error) {<br>    retryUpdateInventory(orderDetails);<br>  }<br><br>  // Ensure idempotency by checking if inventory was already updated<br>});<br><br>// Frontend polling or subscribing for eventual confirmation<br>function checkOrderStatus(orderId) {<br>  // Subscribe to a read model or periodically fetch order status<br>  const orderStatus = getOrderStatus(orderId);<br><br>  // Handle versioning if the contract changes,<br>  // e.g., different order status fields.<br>}</pre><h3>The real cost is coordination</h3><p>Most teams label this as technical debt when it is actually coordination debt. Technical debt usually exists inside the implementation: tangled code, poor testability, unsafe deployments, or data-model shortcuts that make the next change harder. Coordination debt appears between teams: waiting on another backlog, sequencing deployments, negotiating schema timing, or delaying a feature because one approval path is blocked.</p><p>The services may be tidy and the repositories well organized, but if one feature still requires a frontend change, an API schema update, a backend service change, contract tests, and a synchronized rollout, co-change coupling remains high even though ownership is split. That is the effect of a frontend-backend split on delivery speed in the most practical sense: less time writing code and more time waiting for coordinated change.</p><p>The production signals I trust are boring and measurable:</p><ul><li>Files that change together across repositories</li><li>Blocked deployment counts and approval wait time</li><li>Incident timelines shared by multiple teams</li></ul><p>I have used <a href="https://www.openapis.org/">OpenAPI</a> successfully for explicit contracts, but the contract itself is not proof of a good boundary. If every schema update triggers two planning meetings and a release-dependency marker, the system is still being coordinated by hand. At scale, the bottleneck is often lead time, not code size.</p><blockquote><strong>Note:</strong> Small services do not imply low coupling. Co-change frequency is a better signal than repository count.</blockquote><p>One internal dashboard I built tracked elapsed time from ticket start to production across four repositories. Implementation work averaged under six hours. Waiting on reviews, test environments, and rollout order pushed median lead time past three days. In one case, the code for a pricing tweak was done before lunch, but the full change missed the same-day deployment window because the client, API, and downstream job all needed approval in sequence. That is why I say the real cost is coordination and why the trade-offs of separating frontend and backend code into different repositories should be measured in lead time, not just code ownership.</p><p>The easiest way to show that cost is to trace one feature through the actual approval path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5guH4hCR-XchgFSlvHA8w.png" /><figcaption>Flowchart tracing one feature request through multiple teams and systems</figcaption></figure><h3>Where distributed trade-offs get real</h3><p>Once coordination becomes part of every change, the distributed-systems trade-offs stop being abstract. This is where the slogan ends and production behavior starts.</p><p>Once a workflow is split across services and teams, strong consistency is no longer implicit. It becomes a design choice with direct costs in latency, failure propagation, and operator effort.</p><p>A synchronous chain is simple to reason about until it is not. If the frontend calls a BFF, which calls an order service, which calls inventory, the p99 latency is roughly additive. If each hop runs at 80 ms p99, plus network and serialization overhead, the user path can exceed 300 ms before rendering, and one timeout can fail the whole request.</p><p>Asynchronous flows improve isolation and throughput, but they move complexity into state management. The frontend and downstream services must tolerate stale reads, retries, reconciliation, and duplicate delivery. In systems backed by <a href="https://kafka.apache.org/documentation/">Kafka’s delivery model</a>, that means I design for idempotent consumers and explicit replay safety.</p><blockquote><strong>Practical tip:</strong> Prefer single-writer ownership for each fact. If two services both believe that they author the same state, reconciliation becomes your steady-state architecture.</blockquote><p>I also tie this back to bounded contexts. If a service owns writes for inventory, no other service should mutate inventory tables directly, even for convenience. That shortcut usually saves one sprint and creates months of hidden consistency work.</p><p>For engineers working through these choices, I also like <a href="https://www.educative.io/courses/distributed-systems-practitioners?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_distributed_systems_material&amp;eid=5082902844932096">Educative’s distributed systems material</a> because it maps closely to the operational issues here: stale reads, retry storms, and replay-safe consumers. For additional depth, <a href="https://martin.kleppmann.com/">Martin Kleppmann’s distributed-systems writing</a> is also worth bookmarking.</p><p>The next comparison makes the latency and consistency trade-off visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ok_CwOS5jIf6YiK19sgIzg.png" /><figcaption>Side-by-side latency and consistency visual</figcaption></figure><h3>When to keep or fight the split</h3><p>I keep the split when a team owns a bounded context end to end, writes the authoritative data, controls its deployment path, and can change internals without hidden coordination. I fight it when the boundary mainly creates RPC chatter, duplicated rules, shared databases, blocked releases, and constant co-change across repositories.</p><p>The question I ask in planning is simple: does this responsibility change independently enough to justify the latency, consistency, operational, and coordination overhead of making the boundary real? This model works well for independently scaled domains. It breaks down when the split merely mirrors management lines and keeps one feature spread across three backlogs. That is the core tension in feature teams vs. technical-layer teams: one optimizes for end-to-end change, while the other often optimizes for local specialization and pushes coordination outward.</p><blockquote><strong>Watch out:</strong> A service boundary that requires release choreography is often an ownership problem labeled as architecture.</blockquote><p>My advice is practical. Follow ownership, but verify it with data: deployment independence, single-writer data ownership, co-change rates, and incident boundaries. If those signals are weak, keep the code closer together until the domain proves otherwise. In practice, reducing coupling between frontend and backend teams usually starts with clearer ownership, not more diagrams, and the inverse Conway maneuver works only if you are willing to align team autonomy and ownership with the architecture you actually want.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c97f225d2228" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secrets Management Isn’t Optional Anymore: Getting Credentials Out of Your Codebase for Good]]></title>
            <link>https://medium.com/@mark.w.1987/secrets-management-isnt-optional-anymore-getting-credentials-out-of-your-codebase-for-good-c25c22e16d82?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c25c22e16d82</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[management-secrets]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Mon, 26 Oct 2026 11:10:43 GMT</pubDate>
            <atom:updated>2026-10-26T11:10:43.951Z</atom:updated>
            <content:encoded><![CDATA[<p>A leaked secret almost never stays in the file where it started. I still review pull requests that place database passwords in YAML, shell scripts, and CI variables, and the part that worries me most is not the first copy. It is how quickly that value spreads into logs, images, build systems, and support tooling before anyone treats it as an incident. That is why secrets management and moving credentials out of the codebase are no longer optional.</p><p>A few years ago, I helped unwind an incident in which one API key had spread into a repository, a container image layer, build logs, and a support bundle. Revoking the key took minutes. Finding every copy took two days. That gap is why I treat secret sprawl as the real problem, not one bad string in one file.</p><h3>Why secrets in code still happen</h3><p>For beginners, the important point is simple: secrets rarely stay where they start. A password added to config.dev.json often becomes one of many hardcoded credentials copied into CI, pasted into a wiki, baked into a <a href="https://www.educative.io/courses/docker-for-developers?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=docker_image&amp;eid=5082902844932096">Docker image</a>, or written into <a href="https://developer.hashicorp.com/terraform/language/state">Terraform state files</a>. Each copy increases the exposure lifespan and the blast radius.</p><p>I measure three things when I assess this risk:</p><ul><li><strong>Exposure lifespan:</strong> how long a leaked secret remains valid</li><li><strong>Blast radius:</strong> which systems that secret can reach</li><li><strong>Detection-to-rotation latency:</strong> how long it takes me to notice and replace it</li></ul><blockquote><strong>Note:</strong> A secret in source control is bad. A secret copied across five systems is much worse because remediation becomes a search problem.</blockquote><p>I have seen teams focus only on scanning Git history. That helps, but it does not fully detect exposure once the same value has moved into CI variables, old helper scripts, debug archives, and registry caches. If I cannot answer who owns a credential, where it is used, and how quickly I can rotate it, I do not actually control it.</p><p>A useful first step is to trace the full backend delivery path. Start at the developer machine, then walk through CI, artifact storage, runtime configuration, logs, and support exports. That simple mapping exercise usually shows where secrets get duplicated long before anyone intends to copy them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*05a7bHivvni9QCsRQNBr-Q.png" /><figcaption>Secret sprawl across the delivery path</figcaption></figure><h3>What good architecture looks like</h3><p>A sane secrets management system has three parts. First, I keep secrets in a centralized manager such as <a href="https://developer.hashicorp.com/vault/docs">HashiCorp Vault</a>, <a href="https://docs.aws.amazon.com/secretsmanager/">AWS Secrets Manager</a>, or a cloud equivalent. Second, workloads use identity-based secret access instead of stored credentials. Third, the system rotates secrets and revokes access automatically.</p><p>The shift that matters most is moving from static credentials to short-lived ones. In practice, that means my service proves its identity at runtime and then gets a database credential that expires quickly and works only for the job it needs to do. TTL means time to live, or how long that credential stays valid before it expires. When that window is small, cleanup gets much simpler because I can revoke or reissue access centrally instead of hunting through old copies. That is also the basic idea behind what dynamic secret generation looks like in real systems.</p><blockquote><strong>Practical tip:</strong> Start with one secret manager and one identity path. Too many teams delay because they compare every vendor detail before they fix the main risk.</blockquote><p>In practice, I prefer workload identity federation for CI and platform-native identity for running services. <a href="https://docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws">GitHub Actions OIDC</a> is a good example. My pipeline requests cloud access just in time instead of storing a long-lived cloud key in repository settings. This approach also supports least privilege for secrets because the pipeline gets only the access it needs at that moment.</p><p>There is a trade-off. Dynamic issuance adds dependencies on identity, network reachability, and vault availability. I accept that because the alternative is hidden operational debt. I have also seen the failure mode directly. One rollout stalled because a service could not refresh credentials after a policy change, and another team spent hours debugging what looked like a database outage but was really an auth-path issue. Those are real costs, but they are easier to reason about than static secrets quietly spreading for months.</p><p>That target architecture matters because migration choices should move you toward it incrementally. Once you know the end state, you can replace the highest-risk static credentials first instead of trying to redesign everything in one pass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZXTto1rD_CpYEeQTPl3l9Q.png" /><figcaption>Modern secret issuance and rotation flow</figcaption></figure><h3>How to move secrets out safely</h3><p>I do not recommend a big-bang migration. I start with inventory because most teams do not know how many secrets they actually have. A practical first pass is:</p><ul><li>list secrets by owner, system, and environment</li><li>rank them by privilege, age, and exposure path</li><li>migrate the highest-blast-radius credentials first</li></ul><p>For backend services, the first implementation rule is simple: inject secrets at runtime, not at build time. If I bake credentials into an image, every replica and every cached layer becomes another copy to track. Runtime retrieval keeps the secret closer to the point of use and shortens cleanup later. For teams asking how to stop hardcoding credentials in code, this is usually the first habit I change.</p><blockquote><strong>Watch out:</strong> Environment variables are convenient, but they can leak through process dumps, crash reports, and misconfigured debug endpoints.</blockquote><p>Environment variables are still acceptable for low-risk cases when the platform injects them at startup and rotation needs are modest. When I need frequent rotation or tighter controls, I prefer a sidecar, an agent, or a direct SDK fetch. The practical question is where retrieval happens and whether the application can refresh credentials without a restart. In other words, this is the real Vault vs. environment variables vs. config-file decision. Choose the option that limits copying, supports rotation, and keeps access close to runtime.</p><p><a href="https://www.educative.io/blog/what-is-ci-cd-devops?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=cicd&amp;eid=5082902844932096">CI/CD</a> needs the same discipline. I scope each pipeline identity to one repository, one environment, and the smallest set of actions possible. If a deploy job needs only to read one secret and push one artifact, I do not grant broader cloud-admin rights. Those CI/CD secret practices matter just as much as application code changes.</p><p>If you want concrete practice with these patterns, <a href="https://www.educative.io/enterprise-search?collection_type=collection&amp;utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=educatives_devops_and_cloud_courses&amp;eid=5082902844932096">Educative’s DevOps and cloud courses</a> fit this section better than a general backend catalog because they focus on deployment workflows, infrastructure access, and CI/CD behavior.</p><p>A simple way to apply this section is to pick one production credential this week and answer four questions: who owns it, where is it stored, how is it injected, and how do you rotate it? If any answer is unclear, that secret is a good migration candidate.</p><p>The contrast is easiest to understand in a small CI and application example.</p><pre>{<br>  // GitHub Actions workflow using OIDC to obtain cloud credentials<br>  &quot;name&quot;: &quot;OIDC Credential Request&quot;,<br>  &quot;on&quot;: [&quot;push&quot;],<br>  &quot;jobs&quot;: {<br>    &quot;request-cloud-credentials&quot;: {<br>      &quot;runs-on&quot;: &quot;ubuntu-latest&quot;,<br>      &quot;permissions&quot;: {<br>        &quot;id-token&quot;: &quot;write&quot;, // Required for OIDC authentication<br>        &quot;contents&quot;: &quot;read&quot;<br>      },<br>      &quot;steps&quot;: [<br>        {<br>          &quot;name&quot;: &quot;Checkout Repo&quot;,<br>          &quot;uses&quot;: &quot;actions/checkout@v2&quot;<br>        },<br>        {<br>          &quot;name&quot;: &quot;Authenticate using OIDC&quot;,<br>          &quot;run&quot;: &quot;echo &#39;Using OIDC to authenticate&#39;&quot;,<br>          &quot;env&quot;: {<br>            &quot;CLOUD_PROVIDER_CLIENT_ID&quot;: &quot;${{ secrets.CLOUD_PROVIDER_CLIENT_ID }}&quot;<br>          }<br>          // Cloud provider setup required to use GitHub&#39;s OIDC token<br>        }<br>      ]<br>    }<br>  }<br>}<br><br>// Example backend retrieving a secret from a secret manager<br>async function getSecret() {<br>  const secretManager = require(&#39;your-secret-manager-sdk&#39;);<br>  const secretName = &#39;my-secret&#39;;<br><br>  // Fetch secret at runtime<br>  const secret = await secretManager.getSecretValue({ <br>    SecretId: secretName <br>  }).promise();<br><br>  return secret.SecretString;  // Assuming returned secret is a string<br>}<br><br>// Contrast: Using environment variables for static secret<br>const staticSecret = process.env.MY_SECRET; // Potential security risk<br>}</pre><h3>The edge case teams miss</h3><p>The most common blind spot I still see is nonhuman identity. Service accounts, API keys, client certificates, and machine tokens usually outnumber employee accounts by a wide margin. They also tend to have weaker ownership, older permissions, and less review.</p><p>I have debugged incidents in which a machine token was not leaked from source control at all. It showed up in a structured log field, moved into an observability pipeline, landed in archived storage, and then reappeared in a support export. That chain is easy to miss because each copy looks operationally normal in isolation.</p><blockquote><strong>Note:</strong> If a credential can appear in logs or traces, assume that it eventually reaches more systems than the original application team knows about.</blockquote><p>AI-assisted coding adds another path. I now see developers paste runtime configs, stack traces, and example requests into assistants while troubleshooting. If those payloads include service tokens, ownership and visibility get even weaker. The credential is still real, but the audit trail around it is worse.</p><p>This is where beginner teams benefit from clear naming and ownership records. A machine credential without an owner is already a risk, even before it leaks. A practical fix is boring but effective: require every nonhuman credential to map to a service, a team, and a rotation method in the same inventory you use for production secrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2v6OCm7J0b6f1pOb3DKPA.png" /><figcaption>Non-human identity leakage path</figcaption></figure><h3>What to measure and enforce</h3><p>If I want this to stick, I measure it. My baseline set is mean secret age, the percentage of static vs. dynamic credentials, detection-to-rotation latency, inventory coverage, and rotation success rate. Those numbers tell me whether I am reducing risk or just installing new tools.</p><p>I also enforce a few rules in code review and platform policy:</p><ul><li>no new long-lived credentials in repositories or CI settings</li><li>all production secrets must have an owner and a rotation path</li><li>runtime access must map to workload identity where possible</li></ul><blockquote><strong>Practical tip:</strong> Inventory coverage matters more than perfect dashboards. If I track only half of my secrets, the other half will define my next incident.</blockquote><p>There are trade-offs. Centralized vaults add latency, extra permissions logic, and another dependency that can fail. Rotation can break clients that cache too aggressively or fail to reload credentials cleanly. I have seen both. That is why I test secret refresh paths like any other production behavior, including retry logic, TTL handling, and rollback.</p><p>For beginners, here is a simple way to think about it. Static credentials optimize for today’s convenience. Managed life cycles optimize for recoverability. In production, recoverability usually wins because failure cost is not linear. If one leaked token reaches ten systems, cleanup effort grows closer to $O(n)$ with every copy I have to find and invalidate.</p><p>I also care about auditability. Good systems show who requested a secret, when it was issued, what policy allowed it, and whether rotation succeeded. Without that record, post-incident review becomes guesswork. With it, I can tighten scope, reduce mean secret age, and remove exceptions over time.</p><p>The practical takeaway is straightforward. Inventory your secrets, assign owners, move retrieval to runtime, prefer short-lived credentials, and test rotation before you need it during an incident. Teams do not need the same vault product or the same migration order, but they do need identity, ownership, and life-cycle management. That is the core of secrets management: moving credentials out of the codebase, reducing hardcoded credentials, and making recovery faster when something leaks. Start with one high-risk credential, remove one static copy, and repeat. That work is not glamorous, but it is much cheaper than spending two days finding where one leaked key ended up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c25c22e16d82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Polling Still Beats WebSockets for Most Real-Time Features]]></title>
            <link>https://medium.com/@mark.w.1987/polling-still-beats-websockets-for-most-real-time-features-92889b8a9d95?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/92889b8a9d95</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 07:30:47 GMT</pubDate>
            <atom:updated>2026-10-20T07:30:47.393Z</atom:updated>
            <content:encoded><![CDATA[<p>If a feature needs “live updates,” my default answer is not WebSockets. It is usually, “What staleness can users tolerate? How often does the data change? How many clients stay connected? Does the browser need to send data back on the same channel?” That is the core of <strong>polling vs. WebSockets real-time features</strong> in practice, and the question of <strong>when to use polling instead of WebSockets</strong> comes up more often than many teams expect. I learned this after helping unwind a status dashboard that held tens of thousands of idle sockets to push updates that users would have accepted every five seconds. The immediate symptom was not latency. It was operational drag: harder deploys, noisier reconnect behavior, and more moving parts than the feature needed.</p><h3>Why live updates do not mean WebSockets</h3><p>I reset the discussion by defining live in measurable terms. I care about acceptable staleness, event rate, client concurrency, and directionality. If users can tolerate data that is three to ten seconds old, I usually start with polling. That keeps the system stateless and makes failure modes familiar. In many <strong>polling vs. WebSockets trade-offs</strong>, the right answer starts with asking whether <strong>when updates are infrequent, use polling</strong> is good enough for the product requirement.</p><p>For many backend features, I use a simple decision ladder:</p><ul><li><strong>Polling:</strong> Best when updates are sparse and seconds of staleness are acceptable.</li><li><strong>SSE:</strong> Best for one-way streams where lower staleness matters.</li><li><strong>WebSockets:</strong> Best when clients must send and receive rapidly on one channel.</li></ul><p>I have seen teams invert this ladder and pay for it later. A dashboard that shows order counts every few seconds does not need a persistent bidirectional transport. It needs predictable freshness and cheap scaling across replicas behind a normal load balancer. That is one of the clearest <strong>use cases for HTTP polling</strong>. With that framing in place, the trade-off is easier to evaluate in practical terms instead of defaulting to the most flexible transport.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-vgDCIerHBHwPep2wsaP3g.png" /><figcaption>Choose the least stateful transport that meets the requirement</figcaption></figure><h3>What a WebSockets connection really costs</h3><p>WebSockets reduce repeated HTTP request overhead, but that is the easy part of the math. The harder part is everything that a long-lived connection pins in place: a file descriptor, socket buffers, TLS state, timers, and application memory. On paper, 50,000 idle clients sound cheap. In production, they change how every layer behaves. This is where the <strong>WebSockets pros and cons</strong> become more obvious than they do in demos.</p><p>I usually estimate persistent-connection cost with simple capacity questions:</p><ul><li><strong>File descriptors:</strong> Can each node safely hold the required sockets plus headroom?</li><li><strong>Memory:</strong> What do read buffers, write buffers, and connection objects cost per client?</li><li><strong>Timers:</strong> How many heartbeats and idle checks fire every second?</li></ul><blockquote><strong>Watch out:</strong> Mostly idle clients are not free. They often hurt more during reconnects, deploys, and regional blips than during steady state.</blockquote><p>A stateless polling endpoint can scale with the same machinery that I use for other HTTP traffic. A WebSocket fleet often needs different tuning at the kernel, proxy, and app layers. The <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">MDN WebSocket API docs</a> are useful here if you want to review browser-side behavior, but the server burden is where teams usually get surprised. In other words, <strong>scaling WebSockets vs. polling</strong> is often less about protocol elegance and more about operating durable state under failure.</p><p>Load balancers also stop being boring. Upgraded connections are long-lived, so uneven balancing persists longer, draining gets slower, and sticky behavior matters more. If a node dies, the reconnect surge hits auth, session stores, and pub/sub fan-out at the same time. In that status-dashboard cleanup, a routine rollout caused a reconnect wave large enough to spike CPU on both the app tier and the authentication path. The transport itself worked. The expensive part was all the recovery work around it.</p><p>HTTP/2 and HTTP/3 complicate the common argument that polling is always wasteful. With multiplexing, multiple logical requests can share a connection efficiently, and SSE can ride standard HTTP semantics cleanly. The <a href="https://httpwg.org/specs/rfc9110.html">HTTP Semantics specification</a> matters here because stateless requests still fit modern transport stacks very well. This also changes some <strong>polling vs. persistent-connections overhead</strong> assumptions. The compact trade-off table below makes the operational profile easier to compare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*z1ES43uqwefdTx_DEJ65EA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*BGj7WFGr0e-32s26x5H52A.png" /></figure><h3>Reconnections and storms at real scale</h3><p>This is the part that most tutorials skip. Real-time systems fail during reconnects, not demos. I care less about the happy-path latency chart and more about what happens after a deploy, a mobile-network flap, or a brief regional control-plane issue.</p><p>A production client needs four things. It needs exponential backoff with jitter, token refresh before reconnect, heartbeat-timeout detection, and replay or resume after disconnect. If any of those are missing, users silently lose updates or the backend gets hammered.</p><blockquote><strong>Practical tip:</strong> Always add jitter to reconnect delays. Without it, thousands of clients align on the same retry schedule and create their own outage.</blockquote><p>SSE removes some protocol work because browsers already support reconnection and Last-Event-ID. I still need server-side replay windows, but the life cycle is simpler than that of a custom WebSocket protocol. With WebSockets, I have to define sequence numbers, heartbeats, resume cursors, auth-refresh behavior, and backpressure semantics myself. That difference is why I treat SSE as more than a compromise. For one-way streams, it can be the option with the least custom life-cycle code. If you are asking <strong>how server-sent events compare</strong>, this is one of the strongest arguments in the <strong>server-initiated updates SSE vs. WebSocket vs. polling</strong> discussion.</p><p>Slow consumers are another hidden cost. If one client reads more slowly than I publish, unbounded per-client buffers turn into memory growth. In practice, I cap send buffers, drop the connection when limits are hit, and force the client to resume from a cursor.</p><p>A few operational checks help here:</p><ul><li><strong>Replay window:</strong> How long can the server retain enough history for a client to resume cleanly?</li><li><strong>Heartbeat policy:</strong> What timeout marks a dead connection, and who closes first?</li><li><strong>Backpressure action:</strong> Do I buffer, shed, or disconnect slow readers?</li><li><strong>Auth refresh path:</strong> Can reconnects avoid hammering the same token or session dependency?</li></ul><p>The reconnect pattern below is what I would review before approving a launch.</p><pre>// Client-side function for reconnecting with exponential backoff and jitter<br>  function clientReconnect() {<br>    let retries = 0;<br>    let backoffInterval = 1000; // base backoff in ms<br>    const maxBackoff = 30000;   // max backoff in ms<br>    const jitter = 0.5;         // jitter percentage<br><br>    function connect() {<br>      try {<br>        // Attempt connection, refresh token if necessary<br>        refreshTokenIfNeeded();<br>        <br>        // Connect using Last-Event-ID or sequence cursor for resuming<br>        const connection = establishConnection({ lastEventId: getLastEventId() });<br><br>        // Setup heartbeat listener and timeout detection<br>        setupHeartbeat(connection);<br>        <br>        retries = 0; // reset retries on successful connection<br>        <br>      } catch (error) {<br>        <br>        // Calculate next backoff with jitter<br>        const jitterValue = backoffInterval * jitter;<br>        const actualBackoff = backoffInterval + Math.random() * jitterValue - jitterValue / 2;<br>        backoffInterval = Math.min(backoffInterval * 2, maxBackoff);<br><br>        setTimeout(connect, actualBackoff); // schedule reconnection<br>        <br>        retries += 1;<br>      }<br>    }<br>    <br>    connect(); // start the connection process<br>  }<br><br>  // Server-side function sketch for managing connections<br>  function serverManager() {<br>    const heartbeatInterval = 10000; // in ms<br>    const connectionTimeout = 30000; // in ms<br>    const maxSendBuffer = 1024 * 1024; // 1MB buffer per client<br><br>    const clients = new Map(); // track client connections<br><br>    function handleConnection(client) {<br>      clients.set(client.id, { lastActivity: Date.now(), buffer: [] });<br><br>      setInterval(() =&gt; sendHeartbeat(client), heartbeatInterval); // send periodic heartbeat<br><br>      client.on(&#39;message&#39;, (msg) =&gt; {<br>        processMessage(client, msg);<br>        clients.get(client.id).lastActivity = Date.now(); // update last activity time<br>      });<br><br>      client.on(&#39;close&#39;, cleanupStaleConnections);<br>    }<br><br>    function cleanupStaleConnections() {<br>      const now = Date.now();<br>      for (const [id, client] of clients.entries()) {<br>        if (now - client.lastActivity &gt; connectionTimeout) {<br>          clients.delete(id); // remove stale connections<br>        }<br>      }<br>    }<br>    <br>    // Placeholder functions<br>    function sendHeartbeat(client) {}<br>    function processMessage(client, msg) {}<br>  }<br>}</pre><p><em>Client and server pseudocode for handling connections with reconnect, backoff, and heartbeat</em></p><h3>When polling is still the best choice</h3><p>I defend polling more often than people expect. When updates are sparse and viewers are mostly passive, polling keeps the backend simple, stateless, and horizontally scalable. That simplicity matters more than saving a few headers per request. For many teams asking <strong>when to use polling instead of WebSockets</strong>, this is the practical answer.</p><p>The efficient version of polling is not “hit refresh every second forever.” I usually combine conditional requests with adaptive intervals. If the response carries an <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag">ETag validator</a>, the client can send If-None-Match and get a 304 Not Modified most of the time, which cuts payload bytes sharply. In practice, I often <strong>use polling with ETags</strong> because it preserves statelessness while reducing wasted transfer.</p><p>A few patterns work well in production:</p><ul><li><strong>Adaptive cadence:</strong> Poll faster during active changes and more slowly during quiet periods.</li><li><strong>Batching:</strong> Fetch related counters and summaries in one request.</li><li><strong>Stateless fan-out:</strong> Let any replica answer from cache or the database.</li></ul><blockquote><strong>Note:</strong> If acceptable staleness is five seconds, polling every two to five seconds is often enough. Measure end-to-end freshness, not just transport latency.</blockquote><p>This is also where <strong>HTTP polling intervals best practices</strong> matter more than ideology. The right interval depends on tolerated staleness, not on whether the transport sounds modern. In many cases, <strong>latency tolerance in polling vs. WebSocket</strong> is the deciding factor, and <strong>is polling sufficient for live dashboards</strong> becomes a straightforward product question rather than a protocol debate.</p><p>I have also seen SSE underperform on networks with proxy buffering. You expect event-by-event delivery, but intermediaries coalesce bytes and flush them in bursts. In that setup, disciplined polling can produce lower user-visible staleness and much easier debugging. The diagram below shows why stateless polling fits standard backend scaling patterns so well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DadGC5nH_rOnMEO_jkadUQ.png" /><figcaption>Stateless polling scales with ordinary HTTP infrastructure</figcaption></figure><h3>A decision checklist and a safe default</h3><p>When I review a real-time design, I start with six questions. Does the client need bidirectional messaging, or is server-to-client enough? What staleness is acceptable from event creation to browser render? What is the update rate in events per second? How large are the payloads? How many concurrent clients will sit idle? Is the team ready to operate replay, backpressure, and fan-out infrastructure?</p><p>That checklist pushes the conversation away from fashion and back toward system behavior. An activity feed with one-way updates every second or two is often a good SSE candidate. A <a href="https://www.youtube.com/watch?v=lKuCP3OV0Zg">collaborative editor</a>, multiplayer session, or command channel usually justifies WebSockets because both sides need low-latency messaging on the same connection. A dashboard, order tracker, or admin console often works perfectly well with polling. Those are common <strong>web real-time architecture polling use case</strong> decisions, and they illustrate the real <strong>advantages of polling for real-time features</strong>.</p><h3>What I usually recommend</h3><p>My default is simple:</p><ul><li><strong>Start with polling</strong> when staleness can be measured in seconds.</li><li><strong>Move to SSE</strong> when one-way updates need lower staleness or token streaming.</li><li><strong>Choose WebSockets</strong> only when you truly need bidirectional or very low-latency exchange.</li></ul><p>That order reflects operational burden. Polling keeps requests stateless, works well with caches and CDNs, and spreads naturally across replicas. SSE adds a persistent response stream but stays inside normal HTTP semantics, including features documented in the <a href="https://html.spec.whatwg.org/multipage/server-sent-events.html">HTML Living Standard event stream section</a>. If you are evaluating SSE specifically, that spec is worth reading because it covers details such as event framing and reconnection behavior that shape the implementation. WebSockets add the most flexibility, but they also shift more life-cycle design into application code.</p><p>For WebSockets, the actionable questions are different:</p><ul><li><strong>Connection budget:</strong> How many durable sockets can each node hold with headroom?</li><li><strong>Resume design:</strong> How does a client recover missed messages after reconnect?</li><li><strong>Flow control:</strong> What happens to slow readers and oversized fan-out bursts?</li><li><strong>Recovery path:</strong> What dependencies get hit when many clients reconnect at once?</li></ul><blockquote><strong>Watch out:</strong> If your architecture needs sticky sessions just to keep a feature working, I treat that as a design smell until proven otherwise.</blockquote><h3>The concurrency math</h3><p>The concurrency math matters more than many teams expect. If I have <em>N</em> clients polling every <em>T</em> seconds, the average request rate is roughly <em>R = \frac{N}{T}</em>. For 60,000 clients polling every 10 seconds, that is about 6,000 requests per second. A well-tuned stateless fleet can often handle that more comfortably than 60,000 durable sockets with heartbeats, per-connection buffers, and reconnect coordination. That is a concrete example of <strong>long polling vs. WebSockets cost</strong> thinking, even when the simpler baseline is ordinary polling.</p><p>That does not mean polling is always cheaper. If updates are frequent and payloads are nontrivial, repeated polling can waste CPU and bandwidth on responses that are mostly unchanged. This is where SSE earns its keep. I have used it for build logs, token-by-token text streams, and operational feeds where the browser only listens. If you want a deeper review of the API and transport fundamentals behind those trade-offs, Educative’s <a href="https://www.educative.io/courses/grokking-the-api-design-interview-crash-course?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=rest_api_design_courses&amp;eid=5082902844932096">REST API design courses</a> are a practical companion to this topic, and its <a href="https://www.educative.io/courses/nodejs-design-patterns?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=node.js_design_patterns_course&amp;eid=5082902844932096">Node.js design patterns course</a> is more useful when you need to reason through scaling and recovery behavior.</p><h3>Failure modes I check before rollout</h3><p>I also want explicit answers on these points before launch:</p><ul><li><strong>Replay:</strong> Can a client resume from a sequence number or event ID after disconnect?</li><li><strong>Backpressure:</strong> What happens when a consumer cannot keep up?</li><li><strong>Reconnect storms:</strong> How do we rate-limit auth refresh, subscription rebuild, and fan-out recovery?</li></ul><p>In one migration that I worked on, the transport itself was fine. The real bug was that reconnects reloaded authorization state from a hot database path, so a short network event became an auth outage. Because of that, I insist on load-testing reconnect storms, not just steady-state delivery.</p><p>My safe default is opinionated because it keeps systems easier to reason about. Start with polling because it is simple to scale, cache, observe, and debug. Move to SSE when one-way freshness needs improve. Use WebSockets when the product requirement is genuinely bidirectional or latency-sensitive, and accept the operational cost intentionally. In many <strong>polling vs. WebSockets trade-offs</strong>, that default leads to fewer surprises.</p><h3>Conclusion</h3><p>Most real-time features do not need the most complex transport. I make the choice by measuring staleness, event rate, concurrency, and directionality, then I pick the least stateful option that satisfies those constraints. In <strong>polling vs. WebSockets real-time features</strong>, polling is often the right first step, SSE is a strong middle ground, and WebSockets should be reserved for cases that truly benefit. If you are deciding <strong>when to use polling instead of WebSockets</strong>, the short answer is this: use polling when users can tolerate seconds of staleness, updates are relatively sparse, and the simplicity of stateless HTTP is more valuable than a persistent channel. When teams make that choice intentionally, the payoff is usually simpler rollouts, fewer reconnect surprises, and a system that is easier to scale and debug.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92889b8a9d95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Cold Start Problem: Why Your API Feels Instant Locally and Slow in Production]]></title>
            <link>https://medium.com/@mark.w.1987/the-cold-start-problem-why-your-api-feels-instant-locally-and-slow-in-production-aa56f4dfb651?source=rss-95d72e9839d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa56f4dfb651</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Mark W.]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 06:13:44 GMT</pubDate>
            <atom:updated>2026-11-19T11:21:48.503Z</atom:updated>
            <content:encoded><![CDATA[<p>You test an API on your laptop, see 40 ms responses, deploy it, and then the first user sees 1.7 seconds. I have seen this many times. The handler is not always the problem. Production often pays for startup work that local testing quietly skips. That missing context is where beginners usually get confused about API cold-start latency.</p><h3>Why production feels slower</h3><p>On my laptop, I usually benchmark an already running server. The process exists, dependencies are loaded, and memory is warm. In production, the first request may arrive before any ready instance exists, so the platform has to build capacity before my code can do anything useful. That difference between local and production API performance is easy to miss.</p><p>I learned this the hard way during a burst incident years ago. <a href="https://www.educative.io/courses/learn-sql-for-developers?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=we_tuned_sql_queries_for_hours_and_then_found_that_the_biggest_delay&amp;eid=5082902844932096">We tuned SQL queries for hours and then found that the biggest delay</a> was instance startup plus scheduling. In my experience, making that hidden path visible was crucial.</p><p>That gap is why the next step is to define what “cold start” actually includes.</p><p>Before going deeper, this comparison helps make the difference concrete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I7Qsiwl_BYjgjSq51S6MEQ.png" /><figcaption><em>Comparison of local warm execution versus production cold-start steps</em></figcaption></figure><h3>What a cold start means</h3><p>A cold start happens when a request arrives and there is no ready capacity to serve it. The platform must create or reactivate something first: a function environment, a container, or a pod. A warm request is the opposite. It hits an instance that is already ready, connected, and able to run request code immediately. This cold-start vs. warm-start distinction in APIs matters because the user experience can change a lot even when the handler code stays the same.</p><p>A few terms matter early:</p><ul><li><strong>Scale-to-zero:</strong> capacity drops to zero when idle.</li><li><strong>Provisioned concurrency:</strong> prewarmed capacity kept ready.</li><li><strong>Lazy initialization:</strong> expensive setup deferred until it is actually needed.</li></ul><blockquote><strong>Note:</strong> Cold start is not one number. I separate platform startup from application initialization because they fail for different reasons and need different fixes.</blockquote><p>With those terms in place, it becomes easier to see why local testing misses the problem.</p><p>The next comparison gives beginners a cleaner mental model.</p><h3>Cold vs Warm Requests Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PMaDxwPttxh9atzZvDVqoA.png" /></figure><h3>Why local tests hide startup</h3><p>Local development hides startup because I usually start the service once and then send many requests to the same long-lived process. Modules stay loaded. Database drivers may already have pooled connections, and local tooling may cache cloud credentials. In other words, the reason an API is fast locally but slow in production often comes down to which work was already done before the test started.</p><p>That means I am mostly measuring steady-state behavior. I am not measuring process creation, runtime boot, container networking, or autoscaler decisions. This is why a fast local result often fails to predict the first request after a quiet period.</p><blockquote><strong>Watch out:</strong> Benchmarking with a warm development server tells me the request cost, not the startup cost. Those are different execution modes and should be measured separately.</blockquote><h3>What changes after deployment</h3><p>After deployment, new capacity is controlled by infrastructure, not by my terminal. In <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/">Kubernetes pod life cycle documentation</a>, a pod must be scheduled, started, and marked ready. In <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html">AWS Lambda concurrency docs</a>, execution environments may need to be created before a request runs.</p><p>In practice, a cold request can wait on several layers:</p><ul><li>autoscaler reaction</li><li>scheduler placement</li><li>container or runtime startup</li></ul><p>I have seen teams call all of this “app startup,” which hides the real bottleneck. If node pressure, image-pull time, or CNI setup dominates, changing handler code will barely move P99. The next step, then, is to split the startup path precisely. This is a common form of the serverless cold-start problem, even though similar delays also appear outside serverless systems.</p><p>Before I break it into steps, this layered view helps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITfpwmvNBXPsGNsfViYgCw.png" /><figcaption><em>Breakdown of infrastructure-driven versus code-driven startup latency</em></figcaption></figure><h3>What happens during startup</h3><p>When I trace a true cold request, I break it into steps. First comes a capacity decision. Then comes instance or pod creation. After that, the runtime boots, modules load, SDK clients initialize, configuration may be fetched, connections are opened, and only then does request handling begin.</p><p>This may look tricky, but it is just staged latency. I want separate timings for:</p><ul><li>pod-allocation time</li><li>application-initialization time</li><li>time to the first useful request</li></ul><p>At Meta, I once chased a latency spike that looked like a networking issue. The actual cause was a larger dependency bundle plus eager SDK setup. Cold start moved from roughly 200 ms to about 1.8 s, and burst traffic made it visible. That is a clear example of the impact of dependency loading on first-request behavior.</p><p>I have also seen the opposite pattern on smaller internal services. A team expected a code problem, but the fix was operational. Keeping a small warm pool removed most of the first-request penalty without changing the handler at all. Startup code deserves the same review as hot request code. In some environments, container-startup delay is the bigger issue. In others, application initialization dominates.</p><blockquote><strong>Practical tip:</strong> Record the process start time at boot, and then stamp the first handled request. That single gap often shows whether I am dealing with app initialization or platform delay.</blockquote><p>Before talking about distributions, I always instrument the path.</p><pre>const express = require(&#39;express&#39;);<br>const app = express();<br><br>let coldStart = true;<br>const startTime = Date.now(); // Record process start time<br>let firstRequestTime;<br><br>// Middleware to track request processing<br>app.use((req, res, next) =&gt; {<br>  if (coldStart) {<br>    firstRequestTime = Date.now(); // Detect first request time<br>    coldStart = false;<br>    // Calculate cold start duration<br>    const coldStartDuration = firstRequestTime - startTime;<br>    console.log(&#39;Cold start duration:&#39;, coldStartDuration, &#39;ms&#39;);<br>  }<br><br>  // Track useful instance lifetime (time from startup to shutdown)<br>  req.on(&#39;end&#39;, () =&gt; {<br>    const instanceLifetime = Date.now() - startTime;<br>    console.log(&#39;Instance lifetime:&#39;, instanceLifetime, &#39;ms&#39;);<br>  });<br><br>  next(); // Proceed to the next middleware or route<br>});<br><br>// Placeholder for metadata-based pod allocation time calculation<br>function trackPodAllocationTime(metadata) {<br>  if (metadata) {<br>    const allocationTime = Date.now() - metadata.creationTime; <br>// Mocked allocation time<br>    console.log(&#39;Pod allocation time:&#39;, allocationTime, &#39;ms&#39;);<br>  }<br>}<br><br>// Simulate a P99 cold-start utility ratio calculation<br>function calculateColdStartUtilityRatio(p99Duration) {<br>  const ratio = p99Duration / (Date.now() - startTime);<br>  console.log(&#39;Cold-start utility ratio:&#39;, ratio);<br>}<br><br>// Simulated startup and shutdown events<br>function onStartup() {<br>  console.log(&#39;Startup event recorded at:&#39;, new Date(startTime));<br>}<br><br>function onShutdown() {<br>  console.log(&#39;Shutdown event recorded at:&#39;, new Date());<br>}<br><br>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  res.send(&#39;Hello World&#39;);<br>});<br><br>onStartup();<br><br>app.listen(3000, () =&gt; {<br>  console.log(&#39;Server running on port 3000&#39;);<br>  // Mock some event for demonstration<br>  trackPodAllocationTime({ creationTime: Date.now() - 1000 });<br>  calculateColdStartUtilityRatio(200); // Mock p99 value<br>  onShutdown(); // Simulate shutdown<br>});<br>}</pre><h3>Cold- and warm-latency patterns</h3><p>The same endpoint can look healthy at median latency and still fail users at P99. That is the cold-start trap. If 95 requests are warm at 60 ms and 5 are cold at 1,800 ms, the average hides the pain, but the tail tells the truth.</p><p>I teach beginners to look at distributions and timing correlation. Line up latency spikes with autoscaling events, low-traffic gaps, or short-lived instances. <a href="https://prometheus.io/docs/practices/histograms/">Prometheus histogram guidance</a> is useful here. If someone is still learning how request timing, retries, and deployment events interact. The metrics table below is what I check first, especially when measuring API latency at p95 and p99 for cold requests vs. warm ones.</p><h3>How to reduce startup delays</h3><p>I reduce cold starts with both infrastructure and code changes. For critical APIs, I keep a minimum warm baseline by using <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">Kubernetes Horizontal Pod Autoscaler concepts</a> or serverless provisioned concurrency. For bursty but cost-sensitive paths, I accept some cold starts and focus on shrinking initialization. If I am working on AWS, provisioned concurrency for AWS Lambda is often the direct control that makes first-request latency more predictable.</p><p>Here is a simple way to think about it:</p><ul><li>pre-initialize what every request needs</li><li>lazily initialize what only some requests need</li><li>keep startup bundles small</li></ul><blockquote><strong>Watch out:</strong> Minimum replicas improve latency predictability, but you pay for idle capacity. Lazy initialization cuts startup time, but the first feature-specific request may become slower.</blockquote><p>The pairing below shows the trade-off directly.</p><pre>const express = require(&#39;express&#39;);<br>const app = express();<br><br>// Eager initialization of critical dependencies<br>const criticalDependency = require(&#39;critical-lib&#39;);<br>criticalDependency.init();<br><br>// Function to lazily initialize non-critical dependencies<br>let nonCriticalDependency;<br>function getNonCriticalDependency() {<br>    if (!nonCriticalDependency) {<br>        nonCriticalDependency = require(&#39;non-critical-lib&#39;);<br>        nonCriticalDependency.init();<br>    }<br>    return nonCriticalDependency;<br>}<br><br>app.get(&#39;/&#39;, (req, res) =&gt; {<br>    // Use critical dependency immediately<br>    criticalDependency.process(req);<br><br>    // Defer work of non-critical dependency<br>    const nonCritical = getNonCriticalDependency();<br>    nonCritical.processDeferredTask(req);<br><br>    res.send(&#39;Hello World&#39;);<br>});<br><br>// Configuration for AWS Lambda to use minimum provisioned concurrency<br>// Setting this in the Lambda function configuration<br>const minProvisionedConcurrency = 1;<br><br>app.listen(3000, () =&gt; {<br>    console.log(&#39;Server running on port 3000&#39;);<br>});<br>}</pre><h3>How to confirm the root cause</h3><p>When I debug this, I compare the first request with later requests on the same instance. I inspect startup logs, readiness timestamps, and autoscaler activity. Then I separate pod allocation from application initialization and check whether useful instance life is so short that startup dominates both cost and latency.</p><p>For people learning backend systems, and most importantly if you want to practice how to use AI in Backend, then “<a href="https://www.educative.io/courses/ai-for-backend-engineers?utm_campaign=persona_web_dev_q4_26&amp;utm_source=medium&amp;utm_medium=text&amp;utm_content=ai_for_backend_engineer&amp;eid=5082902844932096">AI for Backend Engineer</a>” is most useful. That leads to the main takeaway.</p><h3>Your local setup is not production</h3><p>Local speed usually means that I tested a warm process. Production has to manage capacity, cost, scheduling, and recovery while traffic is arriving. Treat cold and warm paths as different modes, instrument both, and read P99 before trusting averages.</p><p>If you start with that mindset, backend latency gets much easier to reason about. Measure first, separate infrastructure delay from application delay, and then choose the cheapest fix that protects the user experience. That is the core of reducing API cold-start latency and applying cold-start mitigation best practices for APIs without guessing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa56f4dfb651" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>