<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Denis Sazonov on Medium]]></title>
        <description><![CDATA[Stories by Denis Sazonov on Medium]]></description>
        <link>https://medium.com/@sadensmol?source=rss-1dc086cfb1fd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Oe85nN9JJ7h1RTxlUeoZ-g.png</url>
            <title>Stories by Denis Sazonov on Medium</title>
            <link>https://medium.com/@sadensmol?source=rss-1dc086cfb1fd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:30:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sadensmol/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learning System Design #13: To RAG or not to RAG]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-13-to-rag-or-not-to-rag-1cb83c91ce37?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cb83c91ce37</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 00:00:08 GMT</pubDate>
            <atom:updated>2026-08-05T19:51:54.721Z</atom:updated>
            <content:encoded><![CDATA[<p>Part #13 of the “Learning System Design” series! This time — RAG, <strong>Retrieval-Augmented Generation</strong>. The three letters everyone drops into every AI architecture diagram, usually right next to a vector database nobody in the room can quite explain the need for. I kept nodding along in those meetings until I actually sat down and built the thing — and it turns out RAG is far less mysterious, and far more of a plain old search problem, than the hype makes it sound.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eh9NX8kyszG1ohG3.jpg" /></figure><p>The mental model I keep coming back to is an <strong>open-book exam</strong>. A fine-tuned model is the student who crammed everything into their head and walks in with nothing — fast, but frozen on the day they stopped studying. RAG is the student who walks in <em>with the textbook</em> and, for every question, flips to the right page first and answers from it. Same brain, wildly different answers — because one of them gets to look things up.</p><h3>What RAG actually is</h3><p>An LLM only knows two things: what was in its training data, and what you put in the prompt right now. The weights are frozen the day training stops — they don’t know your codebase, last week’s incident, or anything private.</p><p>RAG is the embarrassingly simple pattern of <strong>fetching the relevant bits at question time and pasting them into the prompt</strong>, so the model answers from <em>your</em> data instead of its frozen memory. That’s the whole idea. Everything else — embeddings, vector DBs, rerankers — is just machinery in service of doing that <em>fetch</em> well.</p><p>And that reframes the entire problem. It’s the one thing I want you to walk away with: <strong>RAG is a search problem wearing an LLM costume.</strong> The LLM part is mostly solved — you call an API or run a local model. The part that decides whether your answers are gold or garbage is the <em>retrieval</em>. Feed the wrong chunks into the prompt and the smartest model on earth will confidently answer from the wrong context.</p><h3>The pipeline</h3><p>It splits cleanly into two phases: <strong>build the index once</strong>, then <strong>answer queries against it forever</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GmURBMRrex9nVLoO.jpg" /></figure><p><strong>Indexing (offline):</strong></p><ol><li><strong>Chunk</strong> your documents into passages — a paragraph, a section, a few hundred tokens. Too big and you drown the prompt in noise; too small and you slice a thought in half.</li><li><strong>Embed</strong> each chunk with an embedding model. An embedding is just a vector — a list of ~768 to 1500 floats that captures the <em>meaning</em> of the text. Similar meaning → vectors that sit close together.</li><li><strong>Store</strong> those vectors in a <strong>vector database</strong> so you can search them fast.</li></ol><p><strong>Querying (online):</strong></p><ol><li><strong>Embed the question</strong> with the same model.</li><li><strong>Search</strong> for the nearest chunk vectors — a nearest-neighbour lookup: “find the k passages whose meaning is closest to this question.”</li><li><strong>Rerank</strong> the top candidates (optional, but worth it — more below).</li><li><strong>Build the prompt</strong>: question + retrieved chunks + &quot;answer using only this context&quot;.</li><li><strong>Generate</strong> — the LLM writes a grounded answer, ideally with citations back to the chunks.</li></ol><p>If you’re a backend dev, the retrieval step is less exotic than it sounds. With <strong>pgvector</strong> it’s a plain SQL query against Postgres:</p><p>That &lt;=&gt; is cosine distance. No magic here - it&#39;s ORDER BY ... LIMIT, just over vectors instead of numbers. Then you stitch the results into the prompt:</p><p>That’s a working RAG system. Everything past this point is just making each step better.</p><h3>To RAG or not to RAG</h3><p>Here’s the part people skip: RAG is <strong>one option among four</strong>, not the default. When you want a model to “know” something it doesn’t, you actually have a menu:</p><p>The decision usually comes down to a few honest questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DFrOCL3DnLAsb-J1.jpg" /></figure><p>The rule of thumb I use: <strong>does the knowledge change, and does it need to be private and citable?</strong> If yes → RAG. If you’re teaching a fixed <em>style</em> or <em>skill</em> rather than facts — “always answer in this JSON shape”, “write in our brand voice” — that’s fine-tuning, not RAG. If the whole thing genuinely fits in one prompt and you’ll ask it once, just paste it in; don’t stand up a vector DB for a one-off.</p><p>And these aren’t mutually exclusive. Real systems fine-tune a small reranker, keep a long-context model for the final synthesis, and let an agent decide <em>when</em> to retrieve. RAG is the backbone, not the whole skeleton.</p><h3>Where the quality actually lives</h3><p>Because RAG is a search problem, the wins are all in retrieval. Three levers, in the order they’ll bite you:</p><p><strong>Chunking.</strong> The most underrated knob. Split on natural boundaries (headings, paragraphs), keep a little overlap so a thought isn’t guillotined between two chunks, and attach metadata (source, section, date) you can filter on later. Bad chunking is the #1 reason a RAG demo dazzles and the same system flops in production.</p><p><strong>Hybrid search.</strong> Pure vector search is great at <em>meaning</em> but bad at <em>exact tokens</em> — product codes, error strings, names. Keyword search (BM25) is the exact opposite. Run both, merge the results, and you catch what either alone misses. Almost every serious system is hybrid.</p><p><strong>Reranking.</strong> Vector search is fast but approximate — it hands you 50 “probably relevant” chunks. A <strong>reranker</strong> is a smaller, slower, more accurate model that reads the query against each candidate and re-scores them, so the <em>actually</em> best 5 float to the top of the prompt. It’s the cheapest single upgrade to answer quality I know of.</p><p>There’s a whole zoo beyond this — query rewriting, HyDE, Graph RAG for relationship-heavy questions, multimodal RAG for images and audio — but 90% of your quality comes from chunking well, searching hybrid, and reranking.</p><h3>You can run the whole thing yourself</h3><p>One thing I genuinely like about RAG: none of it forces you onto someone’s SaaS. The embedding model, the vector DB, the reranker, even the generator LLM all have solid open-source options you can docker run on your own box - pgvector or Qdrant for vectors, a local embedding model, Ollama for the LLM. People are running trimmed RAG stacks on a <strong>Raspberry Pi or a Jetson Nano</strong>. If owning your data is the reason you reached for RAG in the first place, you can own the entire pipeline end to end.</p><h3>When you should NOT reach for RAG</h3><p>RAG is a trade, not a free upgrade. Skip it when:</p><ul><li><strong>The corpus is tiny and static.</strong> Ten pages that never change? Paste them into the prompt. A vector DB is pure overhead.</li><li><strong>You need a fixed behaviour, not facts.</strong> “Always output this format” is fine-tuning’s job.</li><li><strong>The data is structured and transactional.</strong> “What’s this customer’s balance?” is a SQL query or a tool call, not semantic recall. Don’t fuzzy-search a number you can look up exactly.</li><li><strong>Latency is sacred.</strong> Every RAG query is embed → search → (rerank) → generate. That’s more moving parts, and more milliseconds, than a single model call.</li></ul><p>The failure mode I see most: teams bolt a vector DB onto a problem that was a plain database query all along, then wonder why it’s slow and occasionally wrong.</p><h3>Summary</h3><ul><li><strong>RAG = fetch relevant context at query time and put it in the prompt</strong>, so the model answers from your data, not its frozen weights.</li><li>Mental model: an <strong>open-book exam</strong> — same brain, but it gets to look things up.</li><li>The pipeline is <em>index once</em> (chunk → embed → store) then <em>query forever</em> (embed → search → rerank → generate).</li><li><strong>RAG is a search problem</strong> — retrieval quality caps answer quality, so chunking, hybrid search, and reranking are where the real wins are.</li><li>It’s one of four options — <strong>RAG vs fine-tuning vs long context vs agents</strong> — pick by whether the knowledge changes and must be private/citable.</li><li>You can self-host the entire stack, down to a Raspberry Pi.</li><li>Don’t RAG a tiny static corpus, a fixed-format task, or an exact lookup.</li></ul><p>I wrote this up as a full, hands-on guide — the pipeline in depth, the model zoo, vector DBs, Graph RAG, multimodal, memory, and edge deployment, all self-hosted-first:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/README.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/README.md</a></p><p>If you want to go deeper on a specific piece, the chapters stand alone:</p><p>Foundations — the pipeline in detail:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/01-foundations.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/01-foundations.md</a></p><p>Types of RAG — naive → agentic, HyDE, rerankers, corrective RAG:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/02-evolution-and-types.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/02-evolution-and-types.md</a></p><p>Vector databases — FAISS, Qdrant, pgvector, sqlite-vec, HNSW/IVF:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/05-vector-databases.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/05-vector-databases.md</a></p><p>Graph RAG — when relationships beat plain vectors:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/07-graph-rag.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/07-graph-rag.md</a></p><p>Edge devices — running RAG on a Raspberry Pi / Jetson:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/09-edge-devices.md">https://github.com/sadensmol/learning_system-design/blob/main/rag-guide/09-edge-devices.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: what’s your retrieval stack — plain pgvector, a dedicated vector DB, hybrid + rerank, or did you skip RAG entirely and just throw a bigger context window at it? I’d love to hear what actually worked in production.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/08/learning-system-design-13-to-rag-or-not-to-rag/"><em>https://sadensmol.com</em></a><em> on August 5, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cb83c91ce37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #12: Understanding JWT]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-12-understanding-jwt-b70728c2cc9a?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/b70728c2cc9a</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 00:00:31 GMT</pubDate>
            <atom:updated>2026-06-29T19:16:08.943Z</atom:updated>
            <content:encoded><![CDATA[<p>Part #12 of the “Learning System Design” series! This time it’s JWT — three Base64 strings everybody copy-pastes from a tutorial and almost nobody can explain when something goes wrong. I’ve debugged enough “Hash invalid” and “token expired but it’s not expired” incidents to think it’s worth slowing down and looking at what’s really going on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jPq6C75nB31HE0Eq.jpg" /></figure><p>The mental model I keep coming back to is a stamped passport. It’s self-contained, it carries claims about you, any border agent can verify it without phoning the country that issued it — and it expires. That’s a JWT.</p><h3>The problem in one sentence</h3><blockquote><em>Service A authenticated the user. Now request after request hits services B, C and D. How do they trust that </em>this<em> request is still that same authenticated user — </em><strong><em>without calling back to a central session store every single time</em></strong><em>?</em></blockquote><p>That’s the whole motivation. Everything else is detail.</p><h3>The old way, and where it creaks</h3><p>The classic answer is the <strong>opaque session</strong>:</p><pre>1. User logs in. <br>2. Server generates a random session ID, stores {sid -&gt; userID, <br>roles, expiry} in Redis/DB. <br>3. Sends the opaque sid back as a cookie. <br>4. Every later request: look up sid in the store to learn who the user is.</pre><p>This works great and has one <em>enormous</em> superpower we’ll come back to: <strong>you can delete the row to revoke a session instantly.</strong></p><p>Where it creaks is scale. Every request needs a store lookup. With a dozen stateless services behind a gateway, that’s a hot dependency on one central store, and service D in another region has to reach it too. Statelessness is the goal; a per-request central lookup is the exact opposite.</p><p>JWT makes a trade: <strong>move the identity into the token itself</strong>, sign it so nobody can forge it, and let each service verify it locally with zero network calls. You trade <em>easy revocation</em> for <em>no lookup</em>. Remember that sentence — it’s the whole article.</p><h3>Anatomy of a JWT</h3><p>A JWT is three Base64URL parts joined by dots: header.payload.signature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1ny1kVpVd0NDGhi.png" /></figure><p>The header says how it’s signed ( alg, plus a kid key id). The payload carries the <strong>claims</strong>. The signature covers the header <em>and</em> the payload - change a single byte of either and it stops verifying. That&#39;s the entire integrity guarantee.</p><p>Here’s the part people miss: <strong>Base64URL is encoding, not encryption.</strong> Anyone holding the token can decode and read the payload. A signed JWT protects against <em>tampering</em>, not against <em>reading</em>. So never put passwords, secrets, or full PII in there — treat the payload as public.</p><h3>The claims that actually carry weight</h3><p>The three-letter registered claims are the load-bearing ones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HQScE6mnzypCoeDkMQ_FKg.png" /></figure><p>aud and iss are how you pin a token to one service. A JWT is a <strong>bearer credential</strong> - whoever holds it can use it - so a token your auth server minted for the <em>payments</em> API must be dead on arrival at the <em>admin</em> API. Each service is configured with its own identifier and rejects any token whose aud isn&#39;t itself.</p><p>Skip that check and you get the classic <strong>confused-deputy bug</strong>: a token the user holds for a low-privilege service gets replayed against a high-privilege one, and the second service happily honors a credential it was never meant to receive. Always validate iss and aud.</p><h3>Signing — symmetric vs asymmetric</h3><p>Two families, and picking the wrong one is a real footgun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVDCKWhi50aaUR-MHqz6hg.png" /></figure><p>The rule of thumb: <strong>the moment more than one party verifies tokens — real microservices, or an external provider like Keycloak/Auth0 — use asymmetric (RS256/ES256).</strong> Then verifiers only ever hold the <em>public</em> key. With HS256 every verifier holds the secret, which means every verifier can also <em>mint</em> tokens. One leaked service and an attacker can forge anything.</p><p>But asymmetric only works if every verifier has the issuer’s public key — and keys rotate. That’s what <strong>JWKS</strong> solves: the issuer publishes its public keys at a well-known URL, the verifier fetches and caches them, reads the kid from the token header, picks the matching key, and verifies locally. No per-request call to the issuer. Key rotation becomes painless - publish the new key alongside the old one and verifiers transparently pick the right one.</p><p>In practice the code you write is the “verify, don’t issue” side:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pd-SWEpzM-almY61ONqrrg.png" /></figure><p>That WithValidMethods line matters more than it looks - pinning the algorithm is what closes the alg attacks below.</p><h3>The footguns that actually bite</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jCEsK2QaJigtoucXuMFcug.png" /></figure><p>None of these are exotic. They’re the ones I’ve actually seen in code review.</p><h3>The big one — revocation</h3><p>This is the question every interviewer loves, and the reason JWT isn’t a free lunch.</p><blockquote><em>A signed JWT is valid until it </em><strong><em>expires</em></strong><em>. There’s no central record to delete. So how do you log someone out </em>right now<em>, or kill a token after a password change?</em></blockquote><p>Natively, you can’t. That’s the cost of statelessness. The standard answer is to split the token in two:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*FHU7BcPfUHDJsF9T.png" /></figure><ul><li><strong>Access token</strong> — short-lived (≈5–15 min), attached to every request, verified statelessly. Because it dies fast, a stolen one is only useful for minutes.</li><li><strong>Refresh token</strong> — long-lived (days/weeks), sent <em>only</em> to the auth server’s /refresh endpoint, and checked <strong>against a store on every use</strong>. This is the one spot you deliberately reintroduce state - so it&#39;s revocable.</li></ul><p>To log someone out everywhere, you delete their refresh tokens. Existing access tokens still die on their own within minutes. If you need <em>instant</em> invalidation, add a small jti denylist or a per-user tokenVersion claim - both reintroduce a lookup, but against a tiny, short-lived set instead of the full session store.</p><h3>When you should NOT reach for JWT</h3><p>Honestly, this is the most useful section. JWT is a trade, not an upgrade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CJEtb2HgCU3SUzo9.png" /></figure><p>If you have a <strong>single backend</strong> and you need <strong>instant, fine-grained revocation</strong> (banking “kill this session NOW”, admin force-logout), opaque server sessions in Redis are simpler, safer, and revocable for free. Don’t add JWT to a Rails/Django monolith because it’s fashionable — you’d be paying the revocation tax for a benefit you don’t use.</p><p>JWT earns its keep the moment <strong>independent parties must verify identity without calling back to a central store</strong>: microservices, public/partner APIs, mobile + SPA clients, “Login with Google” (the OIDC ID token <em>is</em> a JWT).</p><h3>You usually verify, you don’t issue</h3><p>Here’s the part most tutorials skip. <strong>In production you rarely sign JWTs yourself.</strong> Issuing them correctly means owning password hashing, 2FA, refresh rotation, key rotation, breach detection, SAML/SCIM, compliance — each its own multi-week subsystem with nasty footguns (forget to rotate a signing key and a leaked key signs valid tokens forever).</p><p>The classic trap is <em>“it’s just username + password + JWT, right?”</em> — you ship v1 in a weekend, then spend two years bolting on password reset, email verification, refresh tokens, TOTP, SSO, SCIM, and SOC 2.</p><p>For ~95% of teams: don’t issue tokens, integrate a platform — Keycloak or Ory if you self-host (Ory is Go-based, nice for Go shops), Auth0/Clerk/WorkOS if you buy. Either way your services still just <em>verify</em> via JWKS. Build your own only at hyperscale where SaaS pricing breaks, or with a hard requirement to keep all PII in your own infra.</p><h3>Summary</h3><ul><li>A JWT is header.payload.signature, Base64URL-encoded - <strong>signed, not encrypted.</strong> Treat the payload as public.</li><li>It exists to make auth <strong>stateless</strong>: verify locally, skip the central lookup.</li><li>iss + aud bind a token to a specific issuer and a specific service, so it can&#39;t be replayed where it doesn&#39;t belong.</li><li>That same statelessness makes <strong>revocation hard</strong> — solve it with short-lived access tokens + a revocable refresh token (and/or a jti denylist).</li><li>Use <strong>asymmetric</strong> signing (RS256/ES256) + <strong>JWKS</strong> the moment more than one party verifies.</li><li>Pin the algorithm, require exp, validate iss/aud, and never trust claims before checking the signature.</li><li>Single app that needs instant logout? <strong>Plain server sessions are often the right call.</strong></li></ul><p>More detailed notes — every claim, all the algorithms, JWKS, OAuth/OIDC flows, and a system-design interview cheat sheet:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/jwt-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/jwt-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: what’s your revocation strategy in production — short-lived access + refresh, a jti denylist, or did you just go back to opaque sessions? I&#39;d love to hear what worked for you.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/06/learning-system-design-12-understanding-jwt/"><em>https://sadensmol.com</em></a><em> on June 29, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b70728c2cc9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #11: The Many-Factor App — Why 12 Factors Are Not Enough in 2026]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-11-the-many-factor-app-why-12-factors-are-not-enough-in-2026-3b2b60f02acd?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b2b60f02acd</guid>
            <category><![CDATA[microservice-architecture]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Mon, 18 May 2026 00:00:37 GMT</pubDate>
            <atom:updated>2026-05-18T17:45:57.179Z</atom:updated>
            <content:encoded><![CDATA[<p>Eleventh part of the “Learning System Design” series! This time — the 12-Factor App. Or rather, why “12” stopped being enough about a decade ago, and what a modern Go microservice actually has to do today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*tanPEn7SeNXF8Dos.jpg" /></figure><p>I keep running into the same situation. Someone joins a team, opens the README, reads “we follow 12-factor,” and assumes the conversation is done. Then we deploy and discover the service has no idempotency keys, no circuit breakers, no SBOM, env-var secrets in kubectl describe pod, prompts hard-coded in source files, and no canary strategy. <em>Technically</em> 12-factor - and yet the design is fifteen years behind reality.</p><p>So I sat down and reread the original manifesto, Hoffman’s 2016 follow-up, and a bunch of cloud-native postmortems. Came out the other end convinced the methodology is still right at the core — we just need to drop the number. Call it <strong>Many-Factor</strong> instead. Here’s the version I currently use for Go services: 23 factors and counting.</p><h3>The Timeline</h3><p>The 12-Factor App was written by Adam Wiggins at Heroku in <strong>2011</strong>. No Kubernetes, no Docker yet (Docker came in 2013), no distributed tracing as a discipline, definitely no LLMs. It captured the hard-won wisdom of running thousands of SaaS apps on a PaaS — and it nailed the fundamentals.</p><p><strong>2016.</strong> Kevin Hoffman wrote <em>Beyond the Twelve-Factor App</em> and added 3 more: <strong>API-first</strong>, <strong>Telemetry</strong>, <strong>Authentication &amp; Authorization</strong>. The number quietly grew to 15.</p><p><strong>2026.</strong> Software supply chain attacks (Log4Shell, xz-utils, SolarWinds) are weekly news. LLMs are commodity backing services. FinOps is a discipline. Progressive delivery is table stakes. AI-assisted development is the default. The number has to grow again — at least 8 more factors — and the truth is it’ll keep growing. So I dropped the number entirely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OHZ_2pBAPrPL04lu.jpg" /></figure><h3>The Original 12 — Still Right</h3><p>I won’t bore you walking through all twelve. They’re at <a href="https://12factor.net/">12factor.net</a> and they’ve aged well. Quick refresher with the parts where Go shines:</p><ul><li><strong>I. Codebase</strong> — one app, one logical codebase, many deploys. Monorepo or polyrepo, both fine. What breaks the factor is <em>blurred app boundaries</em> (copy-pasted “shared” code, or a binary built from merging two repos).</li><li><strong>II. Dependencies</strong> — go.mod + go.sum for compile-time, CGO_ENABLED=0 + FROM scratch for runtime. The scratch image is the spiritual home of this factor.</li><li><strong>III. Config</strong> — separate from code. We’ll come back to this; env vars are no longer enough.</li><li><strong>IV. Backing services</strong> — Postgres, Redis, S3, Kafka are just URLs in config.</li><li><strong>V. Build, Release, Run</strong> — but really <em>Develop → Build → Release → Run</em>, one-way flow only. Editing a release in prod with kubectl edit is the cardinal sin.</li><li><strong>VI. Processes</strong> — stateless. No in-memory sessions. Sticky sessions are an anti-pattern.</li><li><strong>VII. Port binding</strong> — your Go binary IS the server. net/http was built for this.</li><li><strong>VIII. Concurrency</strong> — scale by running more processes, not by making one bigger. Goroutines are a bonus on top, not a replacement.</li><li><strong>IX. Disposability</strong> — fast boot, graceful shutdown. signal.Notify + srv.Shutdown(ctx) is six lines and saves you from corrupted in-flight work.</li><li><strong>X. Dev/prod parity</strong> — same Postgres minor version locally and in prod. Testcontainers makes this trivial.</li><li><strong>XI. Logs</strong> — JSON to stdout, never to a file. log/slog is stdlib since 1.21 - use it.</li><li><strong>XII. Admin processes</strong> — migrations, backfills, one-off fixes run as separate binaries against the <em>same</em> release. Never kubectl exec; psql; UPDATE.</li></ul><p>Nothing controversial here. If you’re not doing these, fix that before reading on.</p><h3>Hoffman’s 3 Additions (2016)</h3><p>Hoffman noticed three gaps once cloud-native took off:</p><p><strong>XIII. API-First.</strong> Design the contract (OpenAPI, Protobuf) before writing the implementation. Generate types and server stubs with oapi-codegen. Consumer teams unblocked early, mock servers possible, no &quot;the API is whatever the handler returns this week.&quot;</p><p><strong>XIV. Telemetry.</strong> Logs, metrics, traces. Treat your app like a spacecraft — you can’t SSH into it, the only signal you get is the telemetry it broadcasts. The big insight that took the industry a decade to internalize: <strong>logs are far more valuable when they carry a </strong><strong>trace_id</strong>. That&#39;s the bridge between pillars.</p><p>Five extra characters in your log line and your debug time drops from days to minutes.</p><p><strong>XV. Authentication &amp; Authorization.</strong> Not bolted on at the end. And — strongly — <em>don’t build it yourself</em>. Auth looks like “JWT + bcrypt, easy, weekend project,” and six months later you’re maintaining SAML, SCIM, TOTP, WebAuthn, audit logs, key rotation, breach detection, and a SOC 2 evidence pipeline. Auth0, Clerk, WorkOS, Keycloak, Ory — all of them solve this. Use one. Verify their JWTs with keyfunc + golang-jwt and move on with your life.</p><h3>The 8 Modern Additions (2026)</h3><p>This is where I think the methodology genuinely needed an update. Each of these costs real money and real outages when you skip it.</p><h3>XVI. AI &amp; LLMs as First-Class Backing Services</h3><p>An LLM is just Postgres with extra steps. It’s a stateful-ish thing you talk to over the network, it has versioning quirks, it can fail or get slow, and <strong>it costs money per query</strong>. If you hard-code gpt-4o in your business logic, you&#39;ve broken Factor IV all over again - but with a $10k/month bill.</p><p>Sub-rules I now treat as non-negotiable: <strong>prompts are code</strong> (version-control them in prompts/*.txt, diff them in PRs), <strong>evals are tests</strong> (a prompt change without an eval is a deploy without a test), and <strong>every LLM call emits a cost metric</strong> ( llm_tokens_total{provider,model,kind}).</p><h3>XVII. Resilience by Design</h3><p>Every network call will fail. Every dependency will get slow. Every retry storm will happen at the worst moment. Naive code is a single copper wire — one break and you’re down. Resilient code wraps that wire in a fuse (timeout), a circuit breaker (stop trying when broken), a bulkhead (don’t let one slow dep drown the others), and a retry with exponential backoff + jitter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_qsF5UKeRN67KB1F.jpg" /></figure><p>The single most important line is context.WithTimeout. No external call without one. Ever.</p><h3>XVIII. Idempotency &amp; Exactly-Once Semantics</h3><p>“Exactly once” doesn’t exist. “At-least-once + idempotent” does. You press “Charge $100”, the response times out — did the charge happen? If you retry, will you be charged twice?</p><p>And in the database:</p><p>Once you have this primitive, retries, replays, and disaster recovery all become safe. Without it, every distributed-system bug is a potential double-charge.</p><h3>XIX. Supply Chain Security</h3><p>Your binary is a Trojan horse waiting to happen. Every go get, every base image, every transitive dep is a potential xz-utils. The minimum stack:</p><p>govulncheck is the one I want to highlight - it&#39;s the Go team&#39;s official tool, it understands which symbols you actually <em>call</em> (not just which packages you import), and it runs in seconds. Add it to CI and forget about it.</p><h3>XX. Progressive &amp; Declarative Delivery</h3><p>Two ideas that compound. <strong>Declarative</strong> = describe what you want in Git, let a controller (ArgoCD, Flux) reconcile the cluster toward it. <strong>Progressive</strong> = don’t flip 100% of traffic at once. Ramp 1% → 10% → 100% with automated rollback when error rates regress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GjFJw-egE5DHlGbs.jpg" /></figure><p>The key mental shift: <strong>deploy and release are two different events.</strong> Deploy puts the code on the cluster. Release exposes it to users. Feature flags decouple them:</p><p>A rollback becomes flipping a flag, not a re-deploy at 3am.</p><h3>XXI. Cost &amp; FinOps Awareness</h3><p>Hardware used to be capex — buy once, depreciate. In the cloud, every API call has a price tag and an inefficient query is expensive <em>forever</em>, scaling linearly with traffic. With LLMs, a single prompt can cost 100× a Postgres query. You need a P&amp;L per endpoint, not just latency.</p><p>Tag everything (CostCenter, Team, Service, Env), set SLOs <em>for cost</em> alongside latency, kill dev environments overnight. Untagged spend is unaccountable spend.</p><h3>XXII. Schema &amp; Contract Evolution</h3><p>APIs, events, database schemas all change. Never break consumers. For databases this means the <strong>expand-migrate-contract</strong> pattern — three deploys for one logical breaking change:</p><p>Three deploys instead of one is annoying. Two-hour outage from a one-shot DROP COLUMN while old pods are still running is much worse.</p><h3>XXIII. AI-Assisted Development</h3><p>The factor I’d have laughed at three years ago. Today a huge fraction of the code in mature codebases is AI-assisted — generated, refactored, or reviewed by an LLM. The question isn’t <em>should we use it</em> — it’s <em>how do we use it safely</em>.</p><p>The dev workflow now looks like: <strong>Spec → AI generation → Human review → Tests/CI → Commit</strong>. The bottleneck shifted from typing speed to specification clarity and review judgment.</p><p>What I treat as non-negotiable:</p><ul><li>Commit CLAUDE.md / AGENTS.md / .cursorrules at the repo root. Project conventions, build commands, test commands, &quot;always use X for Y&quot; rules. The assistant loads it every session.</li><li><strong>Spec-first</strong> for non-trivial work. Write a markdown plan, AI implements against it. Reviewing a spec is much faster than reviewing 800 lines of generated diff.</li><li>Tests are non-negotiable. AI-generated code without tests is unverified text.</li><li>Never trust crypto or auth code from an LLM without an expert reading it.</li><li>Be explicit about model choice — claude-opus-4-7 for hard reasoning, claude-sonnet-4-6 for routine work.</li></ul><p>The AI-friendly Go codebase is just an idiomatic Go codebase — small interfaces, clear errors, table-driven tests, package-level docs. Bad code is also hard for the AI to refactor safely.</p><h3>What to Adopt First</h3><p>If you’re standing in front of a fresh Go service today and asking what gives you the most bang for buck, my ranking:</p><ol><li><strong>Telemetry (XIV)</strong> — without this nothing else matters when things break.</li><li><strong>Resilience (XVII)</strong> — biggest single source of outages in distributed systems.</li><li><strong>Supply chain security (XIX)</strong> — table stakes after Log4Shell and xz-utils.</li><li><strong>AI-assisted dev (XXIII)</strong> — biggest individual productivity gain available right now.</li><li><strong>Idempotency (XVIII)</strong> — saves you from data-corruption bugs that are nearly impossible to debug.</li><li><strong>Progressive delivery (XX)</strong> — biggest single reduction in deploy-day stress.</li></ol><p>Factors I-XV are non-negotiable baseline. If you’re not doing them, fix that <em>before</em> adding XVI-XXIII.</p><h3>Summary</h3><p>The 12-Factor App was right in 2011 and it’s still right today — it just stopped being <em>enough</em> about a decade ago. The world added Kubernetes, then service meshes, then LLMs, then supply chain attacks, then FinOps, then AI-assisted development. The methodology has to grow with the world.</p><p>The 23 factors I use today fall into three buckets:</p><ul><li><strong>The original 12 (2011)</strong> — codebase, deps, config, backing services, build/release/run, processes, port binding, concurrency, disposability, dev/prod parity, logs, admin processes. Still right. Still non-negotiable.</li><li><strong>Hoffman’s 3 (2016)</strong> — API-first, telemetry, AuthN/Z. The cloud-native era’s contribution.</li><li><strong>The modern 8 (2026)</strong> — LLMs as services, resilience, idempotency, supply chain security, progressive delivery, FinOps, schema evolution, AI-assisted dev. The post-pandemic, post-Log4Shell, post-LLM contribution.</li></ul><p>By 2030 there’ll probably be 30. That’s the whole point of dropping the number from the name.</p><p>The full guide with all 23 factors, Go-specific code examples, deeper trade-off tables, and links lives here:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/many-factor-app-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/many-factor-app-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: which factor do you think your team is weakest on? I genuinely believe most Go shops have telemetry and resilience nailed but ship idempotency keys as an afterthought. Curious where you land — drop me a note.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/05/learning-system-design-11-many-factor-app/"><em>https://sadensmol.com</em></a><em> on May 18, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b2b60f02acd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #10: End-to-End Encrypted Backup for a Personal Home Server]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-10-end-to-end-encrypted-backup-for-a-personal-home-server-1394e0c85210?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1394e0c85210</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[encryption]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Thu, 23 Apr 2026 00:00:24 GMT</pubDate>
            <atom:updated>2026-04-23T20:02:56.539Z</atom:updated>
            <content:encoded><![CDATA[<p>Tenth part of the “Learning System Design” series! This time — a really juicy one. How do you design an end-to-end encrypted backup system for a personal home server that stores your photos, videos, notes, and a zoo of embedded databases — and do it on an ARM box with 4 GB of RAM, while still letting your phone and laptop read the data without ever trusting the cloud?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oHFeO4NLgshIQlRq.jpg" /></figure><p>Quick disclaimer: I don’t actually own an Umbrel Home. This is a <strong>system design exercise</strong> — I picked it as the reference device because it’s a real, shipping product with real constraints (ARM, 4 GB RAM, a zoo of self-hosted apps with mixed databases), which makes the problem concrete instead of hand-wavy. This whole series is about reasoning through designs, not about tools I happen to run at home.</p><p>So: <a href="https://umbrel.com/umbrel-home">Umbrel Home</a> is a tiny always-on ARM server that runs Immich for photos, Memos for notes, Vaultwarden for passwords, Rocket.Chat for the family chat, and a dozen other self-hosted apps. Great device. Except: the device itself is the only copy of everything the user cares about. One theft, one failed SSD, one spilled cup of coffee — everything gone.</p><p>The question “how should I back this up?” turned out to be a really deep system design problem. So I sketched out the whole thing. Let me walk you through it from the simplest idea to the trickiest one.</p><h3>The Setup</h3><p>An Umbrel Home is a real device. Quad-core ARM, ~4 GB RAM, NVMe SSD, always-on Ethernet. It runs self-hosted apps as Docker containers, each with its own data directory on the SSD.</p><p>What lives there, roughly:</p><ul><li><strong>Media files</strong> — photos, videos, music. Big, mostly immutable, they only grow.</li><li><strong>Notes, chat history, passwords, bookmarks</strong> — small, high-churn text.</li><li><strong>Embedded databases</strong> — most apps use SQLite (one .db file per app), a few use MongoDB.</li></ul><p>So we’re dealing with: a mix of giant append-only blobs and many tiny live databases that are being written to constantly. And the design has to work under a bunch of uncomfortable constraints:</p><ul><li><strong>Zero-knowledge.</strong> The cloud never sees plaintext. Not the files, not the file names, not the folder structure.</li><li><strong>Efficient on the edge.</strong> No re-uploading the whole 200 GB photo library every time one photo changes. No hogging the CPU while the user is watching something on Plex.</li><li><strong>Restorable from scratch.</strong> Device dies tomorrow, user buys a new one, enters a passphrase, and everything comes back.</li><li><strong>Readable from a phone and a browser.</strong> Without giving the server any key.</li><li><strong>Time-travel.</strong> Restore yesterday. Restore last month. Undelete something deleted two weeks ago.</li></ul><p>That’s the brief. Let’s build it up piece by piece.</p><h3>Step 1: The Big Picture</h3><p>At a very high level, there are three parties: the home server, the cloud, and the user’s other devices (phone, laptop, browser).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HT7GZOG59Op0imfJ.jpg" /></figure><p>The home server runs a <strong>backup agent</strong> — a small containerized process that watches all the app data directories, chunks whatever changes, encrypts it on-device, and uploads to the cloud.</p><p>The cloud is intentionally dumb. It’s just an <strong>ingest API</strong> on top of object storage and a metadata DB. It never holds a key. It never decrypts anything. Its only job is to store ciphertext and remember which ciphertext blob belongs to which user.</p><p>The phone and web client talk to the same ingest API. They authenticate, they download ciphertext, they decrypt locally. The server never helps them decrypt anything — it can’t.</p><p>Everything interesting happens in the agent. So let’s zoom in.</p><h3>Step 2: Upload Only What’s New — Chunking and Dedup</h3><p>The first “don’t be naive” problem: when a single photo gets added to a 200 GB library, we can’t re-upload 200 GB. Fine — only upload the new photo. But what about a 20 GB video file where the user trimmed the first 5 seconds? Or a 1 GB SQLite database that flipped two bytes on page 347?</p><p>The answer the whole backup industry converged on is <strong>content-defined chunking</strong> (CDC). Instead of splitting files into fixed 4 MB blocks (which shifts if you insert even one byte near the start), you split them at positions decided by the content itself — wherever a rolling hash of the last few bytes matches a pattern. Insert a byte at the front of a file and only the chunks inside the inserted region change; everything downstream still hits the same content-defined boundaries at the same offsets and dedups perfectly.</p><p>This is what restic, borg, duplicacy, and Kopia all do. The specific algorithm we pick is <strong>FastCDC</strong> (~2× faster than the older Rabin variant on ARM), with ~1 MB average chunks.</p><p>Each chunk gets hashed, and the hash becomes the chunk’s address. Before uploading, the agent asks the server “do you already have this hash?” If yes, skip. If no, upload.</p><p>The effect: re-importing the same photo is free. A tiny edit to a 10 GB video file is a ~1 MB upload. Uploading a SQLite base snapshot that’s 99% identical to the previous one is nearly free. Exactly the property we want.</p><p>I’m keeping this short on purpose — there’s a full standalone deep-dive on CDC (algorithms, parameter tuning, per-data-type behavior, streaming, performance on ARM) if you want to go down the rabbit hole:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/cdc-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/cdc-guide.md</a></p><h3>Step 3: Encrypt Everything — The Envelope</h3><p>Now the fun part. The server must never see plaintext, which means the agent encrypts every chunk <em>before</em> upload. But we also want to let multiple devices (phone, web) read the same data. And we want to be able to revoke a lost phone without re-encrypting 200 GB of photos.</p><p>Doing this with a single key is a trap. If every chunk is encrypted with “the user’s key,” then revoking a device means re-encrypting everything. No good.</p><p>The trick is <strong>envelope encryption</strong> with three key layers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/0*X04cf34iVqIwUdm4.jpg" /></figure><ul><li><strong>Master Secret (MS)</strong> — derived from the user’s passphrase via a memory-hard KDF (Argon2id) plus a device-generated seed. Never leaves a client device.</li><li><strong>Key Encryption Key (KEK)</strong> — one per user. Rotatable. Used to wrap (encrypt) other keys.</li><li><strong>Data Encryption Key (DEK)</strong> — a fresh random 256-bit key for <em>every chunk</em>. It’s what actually encrypts the chunk. The DEK itself is stored, wrapped with the KEK, next to the chunk’s reference in the manifest.</li></ul><p>Why three layers and not two?</p><p>Because <strong>rotating a KEK is cheap</strong> — you just re-wrap all the DEKs (metadata-only, ~50k rewraps for a 50 GB library, takes seconds). You do <strong>not</strong> have to re-encrypt the chunks themselves. So device revocation, periodic hygiene rotation, and post-exposure rotation are all affordable.</p><p>For the AEAD primitive we use <strong>XChaCha20-Poly1305</strong> — safe with random nonces (192-bit nonce space, vs AES-GCM’s hazardous 96-bit), constant-time in software on ARM without AES hardware acceleration. Same cipher WireGuard and libsodium standardized on.</p><p>Recovery is the one honest tax: if the user loses all devices <em>and</em> the recovery phrase, the data is gone. That’s the price of zero-knowledge. Server-side escrow would make recovery smooth but would also give the operator a path to plaintext, which breaks the whole promise. Tresorit, iCloud ADP, and Cryptomator all picked the same trade-off for the same reason.</p><h3>Step 4: Many Devices, One User — Per-Device Key Wrapping</h3><p>Multi-device is where lazy E2E designs break. The KEK is the shared user secret, so how does a new phone get it without the server seeing it?</p><p>Each device generates an X25519 keypair on first launch. Private key stays on the device; public key is registered with the account. The server then stores <strong>one wrapped-KEK record per device</strong> — same KEK, wrapped separately for each device’s public key.</p><p>Pairing a new phone:</p><ol><li>New device generates its keypair, registers its pubkey, gets a short numeric pairing code (6 digits).</li><li>User types the code into an already-trusted device (or scans a QR).</li><li>The trusted device fetches the new pubkey, <strong>unwraps the KEK locally, re-wraps it for the new pubkey</strong>, uploads the new wrapped-KEK record.</li><li>New device pulls the wrapped-KEK, unwraps it with its own private key, and now holds the KEK in memory.</li></ol><p>The server is a middleman that facilitates the transfer but never sees the KEK in the clear. Same idea iCloud Keychain and Keybase use.</p><p>Revocation is the symmetric operation: delete the revoked device’s wrapped-KEK, rotate KEK → KEK’, rewrap all DEKs with KEK’, upload new wrapped KEK’s for all still-enrolled devices. Forward secrecy from that moment on. Data the revoked device already downloaded is still readable to it — obviously, fundamentally, unfixably — but any future upload is out of reach.</p><p>The web client is the one weird case: browsers have no durable secure key storage. We treat it as less-trusted, hold the KEK only in memory for the session, and re-pair every new session. Signal, ProtonMail, and Tresorit all do some version of this.</p><h3>Step 5: Live Databases — The Hard Part</h3><p>Here’s the thing that makes this design actually interesting instead of being “another E2E file backup.”</p><p>Most of those apps I mentioned run <strong>live embedded databases</strong> while you’re using them. You can’t just cp app.db - SQLite in WAL mode has a whole separate journal file, and a naive copy races with the checkpoint process and gives you a corrupted DB. mongodump is fine but it&#39;s a heavyweight full scan every time - no incremental. Neither handles &quot;near-zero RPO&quot; (&quot;you shouldn&#39;t lose the last hour of Memos edits&quot;).</p><p>So for each DB family we use its <strong>native change log</strong>:</p><ul><li><strong>MongoDB</strong> — tail the <strong>oplog</strong> (Mongo’s internal replication log). Every write Mongo does shows up there as a tiny record. Our agent follows it from a persisted cursor, streams the entries through the chunk+encrypt pipeline, and periodically adds a “base snapshot” (a mongodump equivalent) so replay isn&#39;t from epoch zero. This is literally how MongoDB Atlas&#39;s own continuous backup works internally.</li><li><strong>SQLite</strong> — use <strong>WAL shipping</strong> (Litestream-style). Put the DB in WAL mode, open a read-only connection alongside the app, and ship completed WAL frames as an append-only byte stream. Periodically snapshot a clean base with VACUUM INTO. This is Ben Johnson&#39;s Litestream pattern, and it&#39;s the only approach that gives near-zero RPO against a live writer without stopping the app.</li></ul><p>Both streams ride the same FastCDC + encrypt + upload pipeline as the blobs. No special code path — a WAL frame is just bytes.</p><p>Restore = load latest base + replay the change log up to the point in time you want. Mongo’s oplog is idempotent, SQLite’s WAL is idempotent. Both can tolerate overlap, which makes at-least-once delivery safe.</p><h3>Step 6: The Subtle Bug — Cross-System Consistency</h3><p>Now the trickiest problem. An Umbrel app like Immich stores <strong>photos as files on disk</strong> and <strong>photo metadata (albums, tags, EXIF) in its SQLite DB</strong>. They reference each other.</p><p>What happens if we back up the DB every 10 seconds and blobs every minute? On restore at some arbitrary time, the DB could reference a photo chunk that wasn’t yet uploaded when we restored. The restored library would have broken references — a row saying “this album contains photo_42.jpg &quot; with no such file. Silent corruption.</p><p>We need <strong>point-in-time consistency between DBs and blobs</strong>. But we can’t just pause everything to take a clean snapshot — that’s exactly what we said we wouldn’t do on a live device.</p><p>The mechanism is a monotonic <strong>fence watermark</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z6TVnk39B1pXPwmr.jpg" /></figure><p>Every successful blob upload returns a monotonic watermark W_i from the server. The agent tracks W_durable = the highest watermark it has seen acknowledged. Every few seconds, per DB, the agent <strong>injects the current </strong><strong>W_durable into the DB&#39;s change log</strong>:</p><ul><li>For Mongo: insert a tiny doc into a reserved _backup_fences collection. It goes through the oplog naturally.</li><li>For SQLite: inject a sentinel marker directly into the WAL-frame byte stream the shipper is uploading. The sentinel never touches the live DB file — it’s just a marker inside the stream the agent controls.</li></ul><p>On restore, replay the DB change log <strong>up to the first fence whose watermark is covered by already-restored blobs</strong>. By construction, no restored row can reference a blob that wasn’t also restored. No silent corruption. No global lock. Same mechanism for both DB families.</p><p>This is really the same idea as Chandy-Lamport snapshot markers in distributed systems or Kafka log watermarks — a marker that flows through the data stream to cut a consistent line across multiple independent producers. Once you see it, you see it everywhere.</p><h3>Step 7: Versioning and Retention, for Free</h3><p>The last piece is “how do we time-travel?” The happy surprise: once we have content-addressed chunks, everything about versioning collapses into one primitive.</p><p>A <strong>snapshot</strong> is just an encrypted manifest describing a tree of files and DB states, where each leaf references a chunk by its content hash. Snapshots are immutable — once committed, they never change. A new snapshot just references a new set of chunks plus any reused chunks from earlier snapshots.</p><p>Restore as of time T? Pick the latest snapshot ≤ T and hydrate. Restore one file? Same snapshot, just one file’s chunks. Undelete something from two weeks ago? Look at a snapshot from two weeks ago, it still references the chunks.</p><p>Retention is then a rule over which snapshots stay pinned:</p><ul><li>Keep everything for the last 7 days.</li><li>Keep daily for 90 days.</li><li>Keep weekly for a year.</li><li>Keep monthly forever.</li></ul><p>Classic grandfather-father-son. Every backup tool on the planet converged on it. Garbage collection is mark-and-sweep over chunk references — any chunk not reachable from any pinned snapshot can be deleted, after a small safety delay to avoid racing with a fresh snapshot commit.</p><p>Storage tiering is a lifecycle policy on the object store — recent chunks on standard, 30-day-old on infrequent-access, year-old on cold archive. No custom code for it.</p><h3>What the Server Actually Sees</h3><p>Let’s do the paranoia check. After all the layering, here’s what the cloud operator can see:</p><ul><li><strong>Ciphertext chunks.</strong> Random-looking bytes, addressed by their ciphertext hash.</li><li><strong>Encrypted manifests.</strong> Also random-looking bytes.</li><li><strong>Per-user chunk ownership.</strong> “User X has chunk a3f7....&quot; But never the chunk&#39;s plaintext hash.</li><li><strong>Timing and sizes.</strong> How much user X uploaded at what time, roughly how big each chunk is.</li><li><strong>Device public keys and session IPs.</strong> Can count how many devices you use and when.</li></ul><p>What the server cannot see:</p><p>Residual metadata (sizes, timing) leaks activity patterns. Padding and upload jitter could bound that but are honestly out of scope. An operator who wanted to be malicious could delete your data — availability is not protected by encryption, it’s protected by the object store’s replication and a client-auditable snapshot hash chain.</p><p>This is the trade-off. Same one iCloud Advanced Data Protection, Tresorit, ProtonDrive, Tarsnap, and restic all live with.</p><h3>Summary</h3><p>The design stacks a few well-known ideas into something that actually works under every constraint:</p><ul><li><strong>Content-defined chunking (FastCDC)</strong> for incremental, insertion-friendly uploads. Near-optimal bandwidth on mixed workloads of photos, videos, and DB change logs.</li><li><strong>Envelope encryption</strong> (Master Secret → KEK → per-chunk DEK, AEAD with XChaCha20-Poly1305) so the server never holds a key and revocation is metadata-only.</li><li><strong>Per-device X25519 key wrapping</strong> for multi-device access without compromising zero-knowledge.</li><li><strong>Oplog tailing + WAL shipping</strong> for live Mongo and SQLite backups with near-zero RPO, no stopping the writer.</li><li><strong>Monotonic fence watermarks</strong> injected into each DB’s change log — one trick to keep DB and blob restores point-in-time consistent without a global snapshot.</li><li><strong>Immutable content-addressed snapshots</strong> so time-travel, trash, and retention are all just “which snapshots are pinned.”</li></ul><p>None of these ideas are new on their own. What’s fun is seeing how they compose: every feature the user sees (time-travel, per-app restore, phone access, device revocation, undelete) is a natural consequence of a small set of primitives instead of its own bespoke code path. That’s usually a sign the design is in a good shape.</p><p>The full design doc set — requirements, trade-off tables comparing every industry variant, deeper per-topic dives (encryption, chunking, sync protocol, multi-device, local DB backup, versioning) — lives here:</p><p><a href="https://github.com/sadensmol/learning_system-design/tree/main/umbrel-home">https://github.com/sadensmol/learning_system-design/tree/main/umbrel-home</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: have you rolled your own backup for a self-hosted box? What did you actually trust — Syncthing, restic to S3, a SaaS like Backblaze, something funkier? And did you ever wake up at 3am wondering if your SQLite backup was actually consistent? I’d love to hear the real stories.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/04/learning-system-design-10-umbrel-home-backup/"><em>https://sadensmol.com</em></a><em> on April 23, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1394e0c85210" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #9: ClickHouse — Why Analytical Databases Are Absurdly Fast]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-9-clickhouse-why-analytical-databases-are-absurdly-fast-9bc1dfef29f9?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bc1dfef29f9</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Mon, 20 Apr 2026 00:00:07 GMT</pubDate>
            <atom:updated>2026-04-20T19:37:43.691Z</atom:updated>
            <content:encoded><![CDATA[<p>Ninth part of the “Learning System Design” series! This time — ClickHouse. The columnar database that scans a billion rows in under a second on a single laptop and powers the analytics of Plausible, PostHog, SigNoz, CloudFlare Radar, and a long list of quant trading shops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o3-cpS4lOEd4-lnl.jpg" /></figure><p>I got curious about ClickHouse when I realized every modern observability and product analytics tool I respected was built on top of it. Grafana Cloud, PostHog, Signoz, Uber’s M3 — same engine underneath. So I spent some time digging into what makes it tick, and honestly, the design is beautiful. It’s a rare case where “100× faster than Postgres” is not marketing but actual everyday experience on analytical queries.</p><h3>The Mental Model</h3><p>Before anything else, internalize this:</p><p><strong>ClickHouse is not a replacement for your transactional database. It’s the query engine you bolt onto it.</strong></p><p>Postgres/MySQL/Mongo holds the source of truth with ACID transactions. ClickHouse holds a denormalized, append-only copy optimized for scanning billions of rows in milliseconds. You stream events into it (via Kafka, CDC, batch loads) and hammer it with analytical queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QQHa2IcxY_yCmtPP.jpg" /></figure><p>If you try to use ClickHouse as your primary DB for a CRUD app, you’ll hate your life. Every UPDATE is a rewrite of a whole data block. There are no row-level locks. There’s no referential integrity. It’s fundamentally append-optimized — and that’s the feature, not the bug.</p><h3>Why Is It So Fast?</h3><p>The speed isn’t one trick. It’s the <em>compounding</em> of about a dozen tricks, each giving 2–5× improvement. Multiplied together, you get orders of magnitude.</p><h3>Columnar storage</h3><p>Row stores like Postgres keep whole rows together. Column stores like ClickHouse flip that — each column is its own file on disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ck5DnGshf7COB1n2.jpg" /></figure><p>When you run SELECT avg(price) FROM trades, Postgres has to read the entire row (id, symbol, price, ts, exchange, volume, &amp;mldr;) and throw away 95% of it. ClickHouse reads only the price file. On wide tables with 30+ columns, this alone is a 10-20× win.</p><p>A side effect: adjacent values in the same column are similar in type and often in magnitude, which leads directly into the next trick.</p><h3>Aggressive compression</h3><p>Adjacent values in a column are similar, so they compress extremely well. A timestamp column with a trillion rows may compress 100:1. You can stack codecs per column:</p><ul><li><strong>Delta</strong> — stores differences between consecutive values (perfect for timestamps)</li><li><strong>Gorilla</strong> — Facebook’s float compression (perfect for prices, metrics)</li><li><strong>T64</strong> — efficient for small-range integers</li><li><strong>ZSTD</strong> — higher ratio than LZ4, slower</li></ul><p>Hidden gem: stacking Delta + ZSTD on timestamps often compresses to ~1-2 bytes per value.</p><h3>Vectorized execution</h3><p>Instead of processing one row at a time, ClickHouse processes <strong>chunks of 65,536 values</strong>. These chunks fit in L1/L2 CPU cache. The inner loops auto-vectorize into SIMD instructions (SSE, AVX2, AVX-512). One AVX-512 instruction adds 16 floats in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*akwNpIkhlrt7hKnF.jpg" /></figure><h3>Sparse primary index</h3><p>ClickHouse doesn’t index every row. It marks one row per “granule” (default 8,192 rows) in a tiny index that fits entirely in RAM. A trillion-row table has ~120 million index entries — small enough to keep in memory without breaking a sweat.</p><h3>No random I/O, no MVCC overhead</h3><p>Data on disk is sorted by the table’s ORDER BY key. Queries that filter on the prefix of this key read contiguous byte ranges - pure sequential I/O. Inserts are immutable blocks; &quot;updates&quot; are async background merges. No row-level versioning tax. Removes gigantic amounts of bookkeeping traditional DBs carry.</p><h3>MergeTree — The Engine You’ll Actually Use</h3><p>Unlike Postgres, where tables all behave the same, ClickHouse has ~20 table engines that radically change semantics. There are engines that dedup on merge, auto-sum numeric columns, read from Kafka, query S3 in place. But 95% of the time you want <strong>MergeTree</strong>.</p><p>Three things matter here:</p><p><strong>ORDER BY is critical.</strong> It defines how data is physically sorted on disk and which filters get fast. Rule: put low-cardinality columns first, high-cardinality last. (symbol, ts) is great because symbol has thousands of values and ts has billions. Queries filtering by symbol hit contiguous data.</p><p><strong>PARTITION BY is for data lifecycle, not speed.</strong> You partition by month so you can DROP PARTITION &#39;202401&#39; to delete old data instantly. Don&#39;t go fine-grained - keep total partitions under ~1000 per table or merge performance tanks.</p><p><strong>TTL is magic.</strong> This single feature replaces cron jobs, archival pipelines, and half of ElasticSearch&#39;s ILM:</p><p>Hot data on NVMe, warm on SSD, cold on S3, deleted after 30 days — all declarative.</p><h3>Data Types Worth Knowing</h3><p><strong>LowCardinality(String) - the free lunch.</strong> Wrap any string with up to ~10k distinct values in it. Internally ClickHouse builds a dictionary and stores the data as tiny integer indexes. Queries become 5-10× faster and storage drops 5-20×. Use it on anything with repeated values: exchange, country, event_type, log_level.</p><p><strong>Nullable(T) - avoid if possible.</strong> Adds a separate bitmap column for null flags. Queries get slower. Prefer sentinel values (0, -1, epoch=0) where it makes sense.</p><p><strong>Native high-precision types.</strong> DateTime64(9) for nanoseconds, Decimal(38, 18) for money, UUID, IPv4, IPv6 with native storage and functions, Array(T), Map(K,V), Nested(...) for rich composites. You rarely need to reach outside the type system.</p><h3>Using It from Go</h3><p>The official driver is github.com/ClickHouse/clickhouse-go/v2. The key thing: <strong>always batch inserts.</strong> ClickHouse is terrible at one-row-at-a-time inserts - every insert creates a new &quot;part,&quot; and too many small parts kill merge performance.</p><p>Target batch size: <strong>10k-1M rows, or ~10 MB</strong>. Smaller than that and you’re leaving throughput on the table.</p><p>If your data arrives one row at a time from many producers (and you can’t batch on your side), use server-side <strong>async inserts</strong> — ClickHouse buffers rows and flushes them as big batches:</p><p>For queries, streaming is your friend. Don’t Select a billion rows into a slice - iterate with rows.Next() and process one row at a time.</p><h3>Where It Shines</h3><p>The short list of workloads where ClickHouse is basically the right answer:</p><ul><li><strong>Time-series analytics</strong> — metrics, traces, events, tick data. Ingests millions of rows/sec per node</li><li><strong>Real-time BI dashboards</strong> — queries that take minutes on Postgres return in &lt;1s</li><li><strong>Observability</strong> — 5–10× cheaper than Elasticsearch on storage, 10–100× on aggregations. SigNoz, OpenObserve, HyperDX are all ClickHouse-powered</li><li><strong>Product analytics</strong> — funnels, retention, cohorts. windowFunnel and retention() built-in</li><li><strong>Fintech / tick data</strong> — DateTime64(9), ASOF JOIN, Gorilla compression. Jane Street, Virtu, crypto exchanges</li><li><strong>Ad-tech</strong> — trillions of impressions/clicks with real-time reporting</li><li><strong>Security analytics / SIEM</strong> — replaces Splunk at 10–100× lower cost</li></ul><h3>Where It’s the Wrong Tool</h3><p>Just as important — don’t force it:</p><ul><li><strong>Transactional workloads</strong> — no ACID transactions, no row locking. Use Postgres</li><li><strong>CRUD apps</strong> — every UPDATE rewrites a part. You’ll suffer</li><li><strong>Low-volume data (&lt;10M rows)</strong> — Postgres with a good index beats it and is 10× simpler</li><li><strong>Hot single-row lookups</strong> — sparse indexes mean reading at least one full granule. Use Redis or Postgres</li><li><strong>Full-text search (primary use case)</strong> — Elasticsearch/Meilisearch win on relevance ranking</li><li><strong>Heavy geospatial / graph queries</strong> — PostGIS and Neo4j are purpose-built</li></ul><p>Rule of thumb:</p><pre>Single-row CRUD with transactions? → Postgres/MySQL Key-value point lookups? → Redis/DynamoDB Full-text search? → Elastic/OpenSearch Embedded / notebook-scale? → DuckDB / chDB Large-scale aggregation over events? → ClickHouse ✓</pre><h3>Common Pitfalls</h3><p>A handful of things that trip up newcomers:</p><ol><li><strong>Tiny inserts</strong> — inserting row-by-row creates thousands of small parts. Batch or use async_insert</li><li><strong>Too many partitions</strong> — per-day partitioning for 10 years = 3,650 partitions per table. Keep it under ~1000</li><li><strong>Nullable everywhere</strong> — adds a whole extra column of null flags. Use sentinels when possible</li><li><strong>Forgetting </strong><strong>LowCardinality</strong> - free 5-10× speedup on repeated strings, and people forget it constantly</li><li><strong>Wrong </strong><strong>ORDER BY</strong> - putting high-cardinality columns first defeats the sparse index. Low-card first, high-card last</li><li><strong>Using MUTATIONS like UPDATE statements</strong> — they’re async, they rewrite parts, they’re not what you want. Use ReplacingMergeTree for dedup, or just accept append-only</li></ol><p>Two features nobody talks about but are ridiculously useful:</p><p><strong>clickhouse-local</strong> - runs the ClickHouse engine as a CLI over local files. Think &quot;SQL-powered awk on steroids.&quot; No server required:</p><p>Reads Parquet, CSV, JSON, Avro, ORC, Arrow. Can write to S3. It’s my new favorite data-wrangling tool.</p><p><strong>chdb</strong> - embedded ClickHouse as a Python/Go library. Single binary, no server, full query engine. Perfect for CLI tools, data pipelines, or one-off analysis at notebook scale.</p><h3>Summary</h3><p>ClickHouse earns its reputation by stacking a dozen optimizations on top of each other — columnar storage, aggressive per-column compression, vectorized SIMD execution, sparse primary indexes, parallel scans across all cores, no MVCC overhead. Individually small; together, a 100–1000× win on analytical queries.</p><p>The mental model is simple: <strong>it’s not a Postgres replacement, it’s the analytics query engine you bolt onto your Postgres.</strong> Stream events in, hammer it with GROUP BY and time-windowed aggregations. MergeTree handles 95% of cases. LowCardinality, Delta+ ZSTD codecs, and sensible ORDER BY get you most of the speed. Always batch inserts. Don&#39;t treat it like OLTP.</p><p>If you’re building dashboards, observability, product analytics, fintech, or anything with “billions of timestamped events” in the requirements — ClickHouse is almost certainly the right pick.</p><p>More detailed notes with all the deep dives (architecture internals, table engines, materialized views, tick-store schemas, real-world backtester patterns, Kafka ingestion, sharding and replication):</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/clickhouse-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/clickhouse-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: have you replaced Postgres analytics with ClickHouse on a real project? What broke, what got 100× faster, what surprised you? I’d love to hear the war stories.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/04/learning-system-design-9-clickhouse/"><em>https://sadensmol.com</em></a><em> on April 20, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bc1dfef29f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #8: WebRTC — How Real-Time Communication Actually Works]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-8-webrtc-how-real-time-communication-actually-works-ad652a73317d?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad652a73317d</guid>
            <category><![CDATA[rpc]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[webrtc]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 00:00:54 GMT</pubDate>
            <atom:updated>2026-04-10T20:06:03.270Z</atom:updated>
            <content:encoded><![CDATA[<p>Eighth part of the “Learning System Design” series! This time — WebRTC. The technology that powers every video call you’ve ever made in a browser, every screen share in Google Meet, and every peer-to-peer file transfer that “just works” without installing anything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eZxOpEiQyVeC88Hq.jpg" /></figure><p>Before WebRTC, real-time communication on the web required Flash, Java applets, or proprietary plugins. WebRTC made it a native browser capability — no plugins, no downloads, no intermediary servers for the actual media transfer. And the best part? It’s not just for video calls.</p><h3>What is WebRTC?</h3><p><strong>WebRTC (Web Real-Time Communication)</strong> is an open-source project and set of browser APIs that enable real-time, peer-to-peer communication directly between browsers and devices.</p><pre>Traditional approach: Client A → Server → Client B (server relays everything) <br>WebRTC approach: Client A ←――――――→ Client B (direct peer-to-peer) ↑ Server only helps with setup (signaling)</pre><p>WebRTC gives you three core APIs:</p><ul><li><strong>getUserMedia</strong> — capture audio and video from cameras and microphones</li><li><strong>RTCPeerConnection</strong> — establish and manage peer-to-peer connections</li><li><strong>RTCDataChannel</strong> — send arbitrary data peer-to-peer (files, game state, text, binary)</li></ul><p>That third one is the most underappreciated. WebRTC is not just for video calls — you can send any data directly between browsers with sub-100ms latency. Game state, file transfers, real-time collaboration — all without your server touching a single byte of the actual payload.</p><h3>The Connection Lifecycle</h3><p>Establishing a WebRTC connection is the most confusing part for newcomers. There are many moving pieces, so let me walk through it step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VP2t3nKk8Ut8_P7E.png" /></figure><p><strong>1. Capture media</strong> — getUserMedia() accesses the camera and microphone.</p><p><strong>2. Create an offer</strong> — the caller creates an SDP (Session Description Protocol) offer describing what codecs, resolutions, and capabilities it supports.</p><p><strong>3–4. Exchange via signaling</strong> — the offer goes through a signaling server (your server — more on this below) to the other peer.</p><p><strong>5–7. Answer back</strong> — the callee creates an SDP answer and sends it back through the signaling server.</p><p><strong>8. ICE candidates</strong> — both peers discover their network addresses and exchange them.</p><p><strong>9–10. Connect</strong> — ICE finds the best path, DTLS establishes encryption, and media flows directly between peers.</p><p>The key insight: <strong>the signaling server only handles the initial handshake — it never sees the actual media.</strong> Once the connection is established, everything flows peer-to-peer.</p><h3>Signaling — The Only Server You Need</h3><p>Here’s the chicken-and-egg problem: to establish a direct connection, two peers need to exchange information about each other. But they can’t do that directly because&amp;mldr; the connection doesn’t exist yet.</p><p>Think of it like a phone call — before you can talk to someone, you need their number. But WebRTC peers don’t have “phone numbers.” They have temporary, dynamically discovered network addresses hidden behind NATs and firewalls. Someone needs to introduce them.</p><p>That someone is the <strong>signaling server</strong>. It’s just a middleman that passes messages between peers during setup. Usually a WebSocket server, though you could use HTTP polling, Firebase, or literally anything that can deliver messages. You could pass SDP on a Post-it note and type it into the other browser — it would work.</p><p>Three things get exchanged during signaling:</p><ul><li><strong>SDP</strong> — “what media do you support?” (codecs, formats, DTLS fingerprint)</li><li><strong>ICE candidates</strong> — “how can I reach you on the network?” (local IP, public IP, relay address)</li><li><strong>Session control</strong> — join/leave room, mute, hang up</li></ul><p>After signaling is done, your server goes idle. It never sees the media. This is why WebRTC scales well — the signaling server only handles small JSON messages, not video streams.</p><h3>NAT Traversal — The Hard Part</h3><p>This is where most WebRTC complexity lives. WebRTC promises peer-to-peer, but the internet wasn’t designed for that. Most devices sit behind <strong>NAT (Network Address Translation)</strong> — your laptop, phone, and smart TV all share one public IP address assigned to your router. From the outside, nobody can reach your device directly.</p><p>Two problems:</p><ol><li><strong>Peers don’t know their own public address.</strong> Your laptop knows it’s 192.168.1.5 on the local network, but has no idea the outside world sees it as 85.1.2.3:12345.</li><li><strong>NATs block unsolicited incoming traffic.</strong> Even if you knew the other peer’s public address, their router would drop your packets — it only allows responses to outgoing requests it already knows about.</li></ol><p>WebRTC solves this with three mechanisms: <strong>STUN</strong>, <strong>TURN</strong>, and <strong>ICE</strong>.</p><h3>STUN — “What’s My Public Address?”</h3><p>STUN is a simple server on the public internet that answers one question: “What’s my public IP and port?” You send a packet to the STUN server, it sees where the packet came from (your router’s public address), and tells you. Now you know what to tell the other peer.</p><p>Google provides free STUN servers ( stun:stun.l.google.com:19302). STUN is lightweight, fast, and only contacted during setup.</p><p>Once both peers know their public addresses, they try <strong>UDP hole punching</strong> — both send packets to each other simultaneously, creating entries in their NAT routers that allow the traffic through. When it works, you get a direct P2P connection with zero server cost.</p><p>This works for about <strong>50–65%</strong> of all internet connections.</p><h3>TURN — The Fallback Relay</h3><p>Sometimes direct P2P is impossible. Symmetric NATs (common on mobile carriers and corporate networks) create a different external port for every destination, making STUN-learned addresses useless. Corporate firewalls might block all UDP entirely.</p><p>TURN is a relay server — all media flows through it. It always works because:</p><ul><li>TURN can run over TCP port 443, looking like HTTPS to firewalls</li><li>Even the strictest corporate firewall allows outbound TCP/443</li><li>Both peers can reach the TURN server with normal outbound connections</li></ul><p>The downsides: added latency (+10–50ms), bandwidth cost (the server relays every byte in both directions), and it costs real money to operate.</p><p>About <strong>10–20%</strong> of connections need TURN, but it varies dramatically — home broadband needs it ~5–10% of the time, while corporate networks need it 25–40%.</p><h3>ICE — The Orchestrator</h3><p>ICE (Interactive Connectivity Establishment) ties everything together. It gathers all possible network paths, then systematically tests them to find the best working connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DYtr4wmYhHC4MMwU.png" /></figure><p>ICE tries host candidates first (direct local IP — fastest), then STUN candidates (NAT-traversed P2P), then TURN relay (slowest but always works). With <strong>Trickle ICE</strong>, candidates are tested as they arrive rather than waiting for all to be gathered — this can reduce connection setup time from 3–5 seconds to under 1 second.</p><p><strong>The takeaway: always deploy TURN servers.</strong> Even if 85% of your users connect directly, the remaining 15% will have a completely broken experience without TURN. In enterprise or mobile-heavy products, it can be 30–40% of users. TURN is not optional for production — it’s insurance.</p><h3>The Protocol Stack</h3><p>WebRTC doesn’t run over HTTP. It builds its own protocol stack, primarily over UDP:</p><ul><li><strong>UDP</strong> — the base transport. Real-time media can’t afford TCP’s head-of-line blocking</li><li><strong>DTLS</strong> — TLS but for UDP. Handshake, peer authentication, key derivation</li><li><strong>SRTP</strong> — encrypts and authenticates audio/video. Mandatory — no unencrypted mode exists</li><li><strong>SCTP</strong> — carries data channel messages. Supports reliable and unreliable delivery modes</li></ul><p>Everything multiplexes onto a <strong>single UDP port</strong>. The transport layer examines the first byte of each packet to route it to the right handler. This single-port design is critical — one port through the NAT, one firewall rule.</p><h3>Why UDP and Not TCP?</h3><pre>TCP with packet loss: Packet 2 lost → Packets 3,4,5 HELD IN BUFFER → Wait for retransmit → Release all at once → Result: massive jitter spike UDP with packet loss: Packet 2 lost → Skip it, play packet 3 → Result: brief glitch, smooth otherwise</pre><p>A lost video frame from 200ms ago is useless. Better to skip it and show the next one.</p><h3>Server Architectures — Mesh, SFU, MCU</h3><p>For 1:1 calls, direct P2P works great. But what about group calls with 10, 50, or 500 participants? Three architectures:</p><p><strong>Mesh</strong> — every peer connects to every other peer directly. Simple, but N peers need N*(N-1)/2 connections. At 4 peers you’re encoding and uploading 3 separate video streams. At 10 peers it’s completely unusable. Only works for 2–3 participants.</p><p><strong>SFU (Selective Forwarding Unit)</strong> — each peer sends one upload to the server, the server forwards it to all other peers without decoding or re-encoding. This is what Zoom, Google Meet, and every serious video platform uses. The sender can use <strong>simulcast</strong> — encoding at multiple quality levels (720p + 360p + 180p) so the SFU picks the right quality for each receiver based on their bandwidth.</p><p><strong>MCU (Multipoint Control Unit)</strong> — the server decodes all streams, composites them into one combined stream, and sends that to each peer. Each peer only downloads one stream, but the server does heavy CPU work. Rare in modern systems because SFUs scale better.</p><p><strong>For production: use an SFU.</strong> It’s the right balance of simplicity, scalability, and quality. MediaSoup, Pion (Go), LiveKit, and Janus are popular open-source options.</p><p>Data channels are the most underappreciated feature of WebRTC. They let you send arbitrary data — text, binary, files, game state — directly between peers using the same P2P connection as media.</p><p>You can configure each channel independently — reliable ordered delivery for chat messages, unreliable unordered for game state updates. Multiple channels on the same connection with different settings.</p><p>Compared to WebSockets, data channels offer lower latency (no server hop), no server bandwidth cost, and built-in encryption. The tradeoff is the complex setup (you still need signaling) and NAT traversal overhead. For real-time multiplayer games, collaborative editors, or any scenario where sub-50ms latency matters — data channels are hard to beat.</p><h3>Common Pitfalls</h3><p>A few things that trip people up:</p><ol><li><strong>No TURN server</strong> — “P2P works on my local network” doesn’t mean it works across the internet. Always deploy TURN</li><li><strong>Ignoring ICE failures</strong> — monitor iceConnectionState. When it goes to failed, show the user a meaningful error, don&#39;t just silently break</li><li><strong>Not handling renegotiation</strong> — adding/removing tracks, switching cameras, or starting screen share requires SDP renegotiation. Handle the negotiationneeded event</li><li><strong>Forgetting about permissions</strong> — getUserMedia() shows a browser permission prompt. Users can deny it. Handle that gracefully</li><li><strong>Assuming low bandwidth</strong> — WebRTC adapts to available bandwidth automatically, but your UI should show quality indicators so users understand why video quality dropped</li><li><strong>Using Mesh for group calls</strong> — it doesn’t scale. Use an SFU if you have more than 3 participants</li></ol><h3>Summary</h3><p>WebRTC enables real-time P2P communication in browsers — audio, video, and arbitrary data. The signaling server handles the initial handshake, then gets out of the way. NAT traversal (STUN for discovery, TURN for relay, ICE for orchestration) is the hardest part and determines whether a direct connection is possible. The protocol stack (SRTP + DTLS + SCTP over UDP) handles encryption, media delivery, and data channels. For group calls, use an SFU.</p><p>The most important practical lessons: always deploy TURN servers, use an SFU for group calls, and don’t sleep on data channels — they’re incredibly powerful for any real-time P2P data transfer.</p><p>More detailed notes with all deep dives (SDP format, codec comparisons, simulcast/SVC, platform support, debugging tools, real-world architecture examples), and interview questions:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/webrtc-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/webrtc-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: have you built anything with WebRTC? Video calls, data channels, something creative? I’d love to hear what worked and what was painful!</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/04/learning-system-design-8-webrtc/"><em>https://sadensmol.com</em></a><em> on April 10, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad652a73317d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI: The Future We Wanted or the Trap We Didn’t See?]]></title>
            <link>https://medium.com/@sadensmol/ai-the-future-we-wanted-or-the-trap-we-didnt-see-261cb87ea916?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/261cb87ea916</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Fri, 03 Apr 2026 00:00:53 GMT</pubDate>
            <atom:updated>2026-04-03T17:15:16.266Z</atom:updated>
            <content:encoded><![CDATA[<p>The developer world has split in two. Open any tech forum, any Twitter thread, any Slack channel — and you’ll find the same fight. Camp one: “AI makes me 10x more productive, stop being a dinosaur.” Camp two: “AI is making developers dumb, we’re losing our craft.” Both sides are loud. Both sides have data. And both sides are missing the real point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*urHHdc6hsgTHKkn_.jpg" /></figure><p>The productivity camp cites GitHub’s numbers — 67% of developers use Copilot 5+ days a week, AI writes 46% of all code on the platform. The skeptics fire back with the METR study — a rigorous randomized controlled trial where 16 experienced open-source developers were actually <strong>19% slower</strong> with AI tools, despite believing they were 20% faster. A 40-percentage-point perception gap.</p><p>Stack Overflow’s 2025 survey of 49,000+ developers shows the tension perfectly: 80% use AI tools, but trust in AI accuracy dropped from 40% to 29%. The top frustration, cited by 45% of respondents? AI solutions that are “almost right, but not quite.”</p><p>Both camps are arguing about productivity. But that’s the wrong argument. The real question is what’s happening to our brains.</p><h3>How Your Brain Gets Good at Things</h3><p>There’s a specific physical mechanism behind how the brain learns. Understanding it changes how you think about the entire AI debate.</p><h3>The Myelination Loop</h3><p>When you focus deeply on a hard problem — debugging a race condition, designing a data model, tracing a memory leak — something physical happens inside your skull. You’re repeatedly firing the same neural circuits in isolation. The same pathways light up again and again as you hold the problem in your head, try approaches, fail, adjust, try again.</p><p>When a neural circuit fires repeatedly like this, your brain responds by wrapping more <strong>myelin</strong> — a fatty insulating substance — around the axons in that circuit. Each layer of myelin makes the signal travel faster, with less noise and fewer errors. It’s like upgrading a dirt road to a highway. The more you use the circuit, the thicker the myelin, the faster and more reliable the signal.</p><p>This is not a metaphor. It’s measurable. MRI studies of expert musicians show myelination patterns that directly mirror their thousands of hours of practice. London taxi drivers who memorized 25,000 streets show measurably thicker neural pathways in the hippocampus. The brain literally rewires itself around what you repeatedly do.</p><h3>Why It Has to Be Hard</h3><p>Here’s the part most people miss: <strong>the difficulty is the mechanism, not an obstacle to it.</strong></p><p>When work feels easy — copying code, accepting AI suggestions, following tutorials — the neural circuits fire weakly and briefly. Not enough repetition, not enough intensity to trigger significant myelination. The brain doesn’t bother upgrading a road that barely gets used.</p><p>When work feels hard — when you’re stuck, frustrated, holding multiple variables in your head, trying approach after approach — that’s when the circuits fire intensely and repeatedly. That’s when the myelin wraps. Robert Bjork at UCLA calls these <strong>“desirable difficulties”</strong> — learning conditions that feel slow and frustrating but produce dramatically better long-term retention than methods that feel smooth and productive.</p><p>The irony is brutal: the approach that feels most productive (instant AI answers) builds the least neural infrastructure. The approach that feels least productive (struggling for hours) builds the most.</p><h3>Why Focus Matters</h3><p>There’s a catch: the circuits have to fire <strong>in isolation</strong>. This is why focused work — what Cal Newport calls “deep work” — is so critical. When you task-switch, your brain doesn’t cleanly swap one set of circuits for another. Research shows there’s an <strong>attention residue</strong> — part of your mind stays on the previous task, which means no single circuit gets the sustained, repeated activation needed for myelination.</p><p>Anders Ericsson, who studied expert performance for decades, put it bluntly: <em>“Diffused attention is almost antithetical to the focused attention required by deliberate practice.”</em> Even elite performers max out at about 4 hours of truly focused practice per day. The brain can’t sustain it longer. But those 4 hours of deep, focused struggle build more neural infrastructure than 12 hours of scattered, easy work.</p><p>This is the lens through which we need to view the entire history of software development. At every stage, the question isn’t just “what tools did programmers use?” It’s “what were their brains training on?”</p><h3>The Abstraction Ladder</h3><p>Software development has always been a story of abstraction — each generation trading low-level understanding for higher-level productivity. And here’s the key: <strong>every previous transition replaced old cognitive load with new, equally demanding cognitive load.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mKqDLnWU263cPNIp.png" /></figure><h3>Machine Code: When Code Was Physical</h3><p>In the 1950s and 60s, programming meant understanding every circuit in the machine. Margaret Hamilton led the team that wrote the Apollo Guidance Computer software — code that was literally <strong>hand-woven into physical memory</strong> by workers threading wires through tiny magnetic cores. A wire through a core was a 1, bypassing it was a 0. Manufacturing a single memory module took 8 weeks.</p><p>There was no debugging tool. No Stack Overflow. No “undo.” Programmers had to hold the entire machine in their heads — hardware, software, physics, everything. When the AGC threw alarms three minutes before the Apollo 11 lunar landing, Hamilton’s defensive programming saved the mission. She had insisted on error-handling that others called unnecessary — “astronauts don’t make mistakes,” they said.</p><p>The cognitive load was total. And the brain trained on: electrical engineering, binary arithmetic, physical memory architecture.</p><p>Assembly gave us mnemonics instead of raw binary — MOV AL, 61h instead of 10110000 01100001. A modest abstraction. Programmers still needed to understand registers, memory addressing, and instruction sets. A single wrong bit crashed the system.</p><p>The brain trained on: CPU architecture, memory models, instruction pipelines.</p><h3>High-Level Languages: The First Big Shift</h3><p>When FORTRAN appeared in 1957, John Backus described the previous era as “hand-to-hand combat with the machine — with the machine often winning.” Assembly programmers pushed back hard. Ed Post’s famous 1983 essay “Real Programmers Don’t Use Pascal” captured the contempt: <em>“If you can’t do it in Fortran, do it in assembly language. If you can’t do it in assembly language, it isn’t worth doing.”</em></p><p>Dijkstra was more direct: <em>“The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise.”</em></p><p>Here’s the critical thing: when programmers stopped managing registers, they didn’t become idle. They redirected that brainpower to <strong>entirely new problems</strong> — portable code design, operating system concepts, data structures, software architecture. Problems that couldn’t even be conceived at the assembly level. The cognitive load didn’t decrease. It shifted upward.</p><h3>Frameworks: Even More Abstraction</h3><p>Ruby on Rails, Django, Spring, React — each encoded opinions about how applications should be built. A Rails developer could build a full CRUD application with rails generate scaffold without understanding TCP/IP, SQL joins, or how browsers render pages.</p><p>But the freed capacity was immediately consumed by new demands: distributed systems, API design, cloud architecture, DevOps pipelines, security at the application layer, scaling strategies, caching patterns. A Rails developer in 2015 didn’t know assembly, but needed to understand HTTP semantics, database indexing, background job processing, deployment pipelines, and dozens of other concepts that didn’t exist in the 1970s.</p><p>This is <strong>Jevons Paradox</strong> applied to software. As Addy Osmani put it: <em>“When high-level languages replaced assembly, programmers didn’t write less code — they wrote orders of magnitude more, tackling problems that would have been economically impossible before.”</em></p><p>Each abstraction layer made previously unthinkable projects viable, which generated new complexity at a higher level. The total cognitive budget never shrank. It was reallocated.</p><h3>AI: Something Different?</h3><p>And now we get to AI-assisted coding. The pattern should be the same, right? We abstract away implementation, and our brains redirect to higher-level problems — system architecture, product thinking, AI orchestration.</p><p>Maybe. But maybe not. Because there’s a fundamental difference this time.</p><p><strong>Every previous transition required mastery of the new abstraction layer.</strong> You couldn’t be a productive C programmer without understanding pointers and memory allocation. You couldn’t be a productive Rails developer without understanding HTTP and databases. The abstraction removed one layer of difficulty but demanded you master the next one.</p><p>With AI, you can “vibe code” — Andrej Karpathy’s term for when you “fully give in to the vibes, embrace exponentials, and forget that the code even exists.” You can be “productive” <strong>without understanding any abstraction layer at all.</strong> That has never happened before in the history of computing.</p><h3>What Happens When the Training Stops</h3><p>We know how focused work builds neural pathways — myelin wraps around circuits that fire repeatedly. But the brain doesn’t just build. It also actively dismantles what you don’t use.</p><h3>The Generation Effect</h3><p>Slamecka and Graf documented this in 1978: information is better remembered if it is <strong>generated from your own mind</strong> rather than provided by someone else. A meta-analysis of 86 studies confirmed a significant effect. Even failed attempts to generate answers enhance subsequent learning. The struggle itself primes the brain for encoding.</p><p>Think about what AI coding does. Instead of generating solutions yourself — struggling, failing, trying again — you describe the problem and receive a solution. You skip the generation step entirely. The code works, but the neural pathway that would have been built&amp;mldr; wasn’t.</p><p>Betsy Sparrow at Columbia University published a landmark study in <em>Science</em> (2011) showing that when people know information will be saved digitally, they have <strong>significantly lower recall</strong> of the information itself. They remember where to find it, but not the content. She called it “The Google Effect.”</p><p>GPS provides an even more striking parallel. Dahmani and Bherer (2020) found that habitual GPS users showed steeper declines in hippocampal-dependent spatial memory over time. Greater GPS use correlated with worse spatial performance — the exact opposite of the London taxi drivers who actively navigated. Same brain region, opposite outcomes. The difference? Active cognitive engagement vs. passive following.</p><h3>Use It or Lose It</h3><p>Synaptic pruning follows a strict rule: pathways that are frequently activated get stronger, while unused connections are eliminated. This isn’t just a childhood process — it continues throughout adult life. Boston University research found that cognitive decline in old age is linked to increased pruning of brain cell connections.</p><p>The implication is clear: mental skills that are not actively exercised will physically degrade at the neural level. Not metaphorically. Structurally. The myelination that took years of focused practice to build? It degrades when those circuits stop firing.</p><h3>The Automation Paradox</h3><p>None of this is new. In 1983, Lisanne Bainbridge published “Ironies of Automation” — a paper that predicted exactly what we’re seeing now with AI coding tools.</p><p>Her core paradox: <strong>the more advanced an automated system becomes, the more critical — and more difficult — the human role within it becomes.</strong></p><p>When automation works, operators don’t practice their skills. When automation fails, operators must intervene — but their skills have atrophied precisely because they haven’t been practicing. The operator is expected to monitor a system they no longer actively control and to take over expertly at a moment’s notice.</p><p>This played out fatally in aviation. After decades of increasing cockpit automation, a NASA survey found 90% of commercial pilots believed pilots as a whole were losing flying skills due to automation. The 2009 Air France Flight 447 crash and the 2009 Colgan Air crash both involved pilots who had lost their kinesthetic sense of the aircraft.</p><p>It’s playing out in software development right now. Anthropic’s own study (January 2026) — a randomized controlled trial with 52 developers — found that those using AI coding assistance <strong>scored 17% lower on comprehension tests</strong>. Those who delegated code generation to AI scored below 40%, while those who used AI for conceptual inquiry scored 65% or higher.</p><p>Addy Osmani, engineering lead at Google, drew the distinction perfectly: <em>“A calculator still requires you to set up the problem; an AI can replace the entire thinking process.”</em></p><h3>Two Camps, One Truth</h3><p>Here’s the one truth neither camp wants to hear.</p><p><strong>Both sides are right. And that’s not a contradiction — that’s the trap.</strong></p><p>AI makes you faster today. AI makes you weaker tomorrow. These aren’t opposing positions. They’re two measurements of the same phenomenon at different timescales.</p><p>You’ve seen the pattern through 70 years of the abstraction ladder. Every transition replaced one hard skill with another equally hard one. The brain stayed loaded. The myelin just wrapped around different circuits. No net loss — ever.</p><p>Until now. <strong>AI is the first transition in computing history where the brain has nothing equally hard to redirect to.</strong> The replacement — prompt engineering, reviewing AI output — is not cognitively equivalent to what was lost. AI generates code at 140–200 lines per minute. Humans comprehend at 20–40. That’s a 5–7x gap Osmani calls “comprehension debt” — surface correctness hiding hollowed-out understanding. Microsoft/Carnegie Mellon research (2025) confirmed that the more people leaned on AI tools, the less critical thinking they engaged in.</p><p>And neuroscience is unambiguous about what happens to a brain under reduced load: it prunes the circuits it no longer needs.</p><p>The numbers already show it. GitClear’s analysis of 211 million lines of code: copy/pasted code up from 8.3% to 12.3%. Refactored code collapsed from 25% to under 10%. Code churn — code thrown away within two weeks — doubled. These are the fingerprints of code that nobody understands.</p><p>The junior developer pipeline is drying up. Employment for developers aged 22–25 dropped nearly 20% between 2022 and 2025. Companies say “AI handles the junior work now.” But if you stop training juniors, you stop producing seniors. As Osmani warned: <em>“If you don’t hire junior developers, you’ll someday never have senior developers.”</em></p><p>For companies, AI is a clear short-term win — more output, lower costs, faster time to market. For the profession, for the industry, for the long-term health of the people building the systems the world runs on? It’s a slow-motion disaster that feels like progress.</p><h3>Fight Back — But Fight Smart</h3><p>I’m not saying “stop using AI.” That would be like telling a 1960s programmer to reject compilers. The abstraction is here, and it’s not going away.</p><p>But I am saying: <strong>this is the one transition in computing history where you need to actively resist the comfort it offers.</strong> Not reject it. Resist the easy path of full surrender. Because full surrender means full cognitive offload. And full cognitive offload means your neural circuits start pruning. Slowly, silently, irreversibly.</p><p>Socrates warned that writing would “implant forgetfulness in their souls.” He was partly right — we lost the oral tradition’s memory capacity. But writing demanded a compensating skill: structured thinking, argumentation, building on accumulated knowledge. The brain traded one workout for another.</p><p>AI is more like GPS. It doesn’t demand a compensating skill. It just makes the old one unnecessary. And the taxi drivers’ hippocampi don’t care about your productivity metrics — they shrink when they stop being used.</p><p>So fight. Not against AI — but against your own atrophy.</p><p><strong>Write code without AI regularly.</strong> Like an athlete doing basic drills, maintain the fundamental neural pathways. When you struggle with a bug for two hours, your brain is physically building something that an instant AI answer would have prevented.</p><p><strong>Use AI for acceleration, not for replacement.</strong> Use it to speed up work you already understand. Not to skip understanding entirely.</p><p><strong>Learn the layer below.</strong> Whatever abstraction level you work at, understand at least one level beneath it. If you write Python, understand how the interpreter works. If you use a framework, understand the protocol underneath.</p><p><strong>Train your brain on hard things.</strong> The difficulty is the point. The struggle is not a bug in the learning process — it’s the mechanism itself. Every hour of focused, hard problem-solving wraps myelin around your neural circuits. Every hour of passive AI acceptance lets it degrade.</p><p>Fred Brooks said it in 1986: <em>“The complexity of software is an essential property, not an accidental one. Descriptions of a software entity that abstract away its complexity often abstract away its essence.”</em></p><p>AI can abstract away accidental complexity — boilerplate, repetitive patterns, syntax. But essential complexity — the hard thinking about what to build and why — cannot be delegated without being lost.</p><h3>Summary</h3><p>The developer AI debate is a false binary. Both camps are right — and that’s the trap, not the answer.</p><p>Seventy years of neuroscience — from Maguire’s taxi drivers to Bjork’s desirable difficulties to Sparrow’s Google Effect — tells us one thing clearly: your brain grows when you challenge it and atrophies when you don’t. And AI is the first abstraction in computing history that doesn’t replace the challenge with an equally hard one.</p><p>If we give up completely — if we hand over not just the boilerplate but the thinking itself — our future is not “developers assisted by AI.” It’s developers who can no longer function without AI. Dependent. Replaceable. And eventually, unnecessary. At that point, we haven’t adopted a tool. We’ve surrendered to it. The machines won’t need to take our jobs — we’ll have already forgotten how to do them.</p><p>The real question isn’t “should we use AI?” It’s “are we training our brains hard enough to remain the ones in control?”</p><p>Because every generation of programmers before us kept their brains loaded. They traded one hard skill for another, and they stayed sharp. We’re the first generation with the option to trade hard skills for nothing. And if we take that deal, we won’t lose a battle to machines. We’ll have simply walked off the field.</p><p>PS: where do you stand? Are you actively training your brain alongside AI tools, or have you noticed your skills shifting? I’d love to hear your experience.</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/04/ai-the-future-we-wanted-or-the-trap-we-didnt-see/"><em>https://sadensmol.com</em></a><em> on April 3, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=261cb87ea916" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #7: Kubernetes — The Complete Guide]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-7-kubernetes-the-complete-guide-badba367340b?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/badba367340b</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 00:00:11 GMT</pubDate>
            <atom:updated>2026-03-24T17:33:50.355Z</atom:updated>
            <content:encoded><![CDATA[<p>Seventh part of the “Learning System Design” series! This time — Kubernetes. The thing that every backend engineer eventually encounters, and the thing that makes you question your career choices when you’re debugging a CrashLoopBackOff at 3 AM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TyNYqxLSUjtrdO3y.jpg" /></figure><p>But seriously — K8s is the industry standard for running containers in production. Whether you love it or hate it, you need to understand it. Let’s break it down.</p><h3>What is Kubernetes?</h3><p>Kubernetes (K8s — the “8” stands for the eight letters between “K” and “s”) is an open-source container orchestration platform. Docker lets you package your app into a container. Kubernetes manages thousands of those containers across many machines.</p><p>Think of it like running a fleet of food trucks. Without K8s — you personally drive to each truck, check if the cook showed up, replace broken equipment, and figure out how many trucks you need for a festival. At 3 AM. On a holiday. With K8s — you have a dispatcher that does all of this automatically. You say “I need 5 burger trucks running at all times” and the system handles the rest.</p><p>The key problems it solves:</p><ul><li><strong>Placement</strong> — which server should this container run on?</li><li><strong>Self-healing</strong> — container crashed? Restart it. Server died? Move containers elsewhere</li><li><strong>Scaling</strong> — traffic spiked? Spin up more containers. Traffic dropped? Remove extras</li><li><strong>Service discovery</strong> — containers find each other by name, not hardcoded IPs</li><li><strong>Desired state</strong> — you declare what you want, K8s makes reality match</li></ul><p>Google built it internally as <strong>Borg</strong>, ran it for a decade, then open-sourced the ideas as Kubernetes in 2014. It’s now maintained by CNCF.</p><h3>Architecture</h3><p>K8s has two main parts: the <strong>Control Plane</strong> (the brain) and <strong>Worker Nodes</strong> (where your apps run).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c_sFGMA-okuMaoDL.png" /></figure><p><strong>Control Plane components:</strong></p><ul><li><strong>API Server</strong> — the front door. Everything goes through it — kubectl, dashboards, internal components</li><li><strong>etcd</strong> — distributed key-value store that holds ALL cluster state. The single source of truth</li><li><strong>Scheduler</strong> — watches for new pods without a node and assigns them based on resources and constraints</li><li><strong>Controller Manager</strong> — runs loops that constantly push current state toward desired state</li></ul><p><strong>Worker Node components:</strong></p><ul><li><strong>kubelet</strong> — agent on every node, ensures pods are running and healthy</li><li><strong>kube-proxy</strong> — handles network routing to the right pods</li><li><strong>Container Runtime</strong> — containerd or CRI-O (Docker is no longer used directly since K8s v1.24)</li></ul><h3>The Reconciliation Loop</h3><p>This is the heart of Kubernetes. It works on a <strong>declarative model</strong> — you don’t say “run 3 copies.” You say “I want 3 copies running.” K8s then constantly works to make reality match:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FL_3Gr87JWL1HGOy.png" /></figure><p>One pod crashes? Controller notices only 2 exist, creates a new one. Back to 3. All automatic. This is what makes K8s powerful — you describe the <em>what</em>, not the <em>how</em>.</p><h3>Core Concepts</h3><p><strong>Pod</strong> — the smallest unit. Wraps one or more containers sharing the same network and storage. Most of the time, one pod = one container. You almost never create Pods directly — use Deployments.</p><p><strong>Deployment</strong> — manages a set of identical Pods. Handles replicas, rolling updates, and rollbacks.</p><p><strong>Service</strong> — since Pods are ephemeral (created and destroyed constantly), a Service provides a stable IP and DNS name that routes traffic to the right Pods. Four types: <strong>ClusterIP</strong> (internal only, default), <strong>NodePort</strong> (exposes on each node’s IP), <strong>LoadBalancer</strong> (gets an external IP from your cloud provider), and <strong>ExternalName</strong> (maps to a DNS CNAME).</p><p><strong>StatefulSet</strong> — like Deployment, but for databases and stateful apps. Gives stable names (my-db-0, my-db-1), stable persistent storage per pod, and ordered deployment/scaling. Use for PostgreSQL, Redis, Kafka, Elasticsearch.</p><p><strong>DaemonSet</strong> — ensures one pod runs on every node. Perfect for log collectors (Fluentd), monitoring agents (Datadog), and network plugins.</p><p><strong>Job/CronJob</strong> — for finite tasks. Jobs run to completion, CronJobs run on a schedule. Always set activeDeadlineSeconds and backoffLimit - otherwise zombie jobs run forever. For CronJobs, use concurrencyPolicy: Forbid for anything that modifies data to prevent race conditions.</p><p><strong>Namespace</strong> — logical division of a cluster. Resource isolation, access control, network policies. Think of them as folders — production, staging, development.</p><p><strong>ConfigMap &amp; Secret</strong> — configuration and sensitive data. Important: Secrets are only base64-encoded by default, NOT encrypted. Anyone with API access can read them. Use <strong>Sealed Secrets</strong>, <strong>External Secrets Operator</strong>, <strong>HashiCorp Vault</strong>, or enable <strong>etcd encryption</strong> for real security.</p><h3>Scaling: HPA, VPA, and Karpenter</h3><p><strong>HPA (Horizontal Pod Autoscaler)</strong> — adds more pods when CPU/memory goes up, removes them when it drops.</p><p><strong>VPA (Vertical Pod Autoscaler)</strong> — adjusts CPU/memory of existing pods instead of adding more. Useful when you don’t know what resources your app actually needs. Don’t use HPA and VPA on the same metric — they’ll fight each other.</p><p><strong>Karpenter</strong> — a smarter node autoscaler (originally AWS, now expanding to Azure). Instead of scaling predefined node groups, it provisions the exact right instance type based on pending pod requirements. Faster (seconds vs minutes), cheaper (picks the cheapest instance that fits), and consolidates underutilized nodes automatically. If you’re on AWS with varied workloads — Karpenter over Cluster Autoscaler, no question.</p><h3>Networking: Ingress and Services</h3><p>Getting traffic into your cluster is done through <strong>Ingress</strong> — two parts:</p><ol><li><strong>Ingress Controller</strong> — the actual software (NGINX, Traefik, HAProxy). Usually one per cluster, sometimes two for public/internal split</li><li><strong>Ingress Resource</strong> — YAML that defines routing rules. One or more per namespace</li></ol><p><strong>Gateway API</strong> is the successor to Ingress — supports TCP/UDP/gRPC routing, traffic splitting, header-based routing. It’s now GA and recommended for new setups.</p><p><strong>Egress</strong> — outbound traffic from your pods. By default, pods can reach anything. Lock it down with <strong>Network Policies</strong>. Pro tip: when restricting egress, always allow DNS (port 53) to kube-system, otherwise pods can’t resolve any hostnames.</p><h3>Persistent Storage</h3><p>Pods are ephemeral — when they die, local data is gone. For databases and file storage, you need persistent volumes.</p><p>Three concepts: <strong>StorageClass</strong> (defines <em>how</em> storage is provisioned — “use AWS EBS gp3”), <strong>PersistentVolume</strong> (a piece of actual storage), <strong>PersistentVolumeClaim</strong> (a pod’s request for storage — “I need 50GB of fast SSD”).</p><p>Access modes: <strong>ReadWriteOnce</strong> (one node, most common), <strong>ReadOnlyMany</strong> (multiple nodes read-only), <strong>ReadWriteMany</strong> (multiple nodes read-write — needs special storage like EFS or NFS).</p><h3>OOM — The Most Common K8s Problem</h3><p>When you see OOMKilled - the Linux kernel killed your container for exceeding its memory limit. Exit code 137 (128 + SIGKILL).</p><p>Two key settings: <strong>requests</strong> (minimum guaranteed, used for scheduling) and <strong>limits</strong> (hard ceiling, exceed = killed). The gap between them determines your QoS class:</p><ul><li><strong>Guaranteed</strong> (requests == limits) — last to be evicted</li><li><strong>Burstable</strong> (requests &lt; limits) — medium priority</li><li><strong>BestEffort</strong> (no limits set) — first to be evicted. Never use in production</li></ul><p>Common causes: limits set too low, memory leaks, JVM/runtime overhead, processing large payloads in memory.</p><p>Fix it: increase limits if genuinely needed, fix memory leaks (pprof for Go, heap dumps for Java), set proper JVM flags (-XX:MaxRAMPercentage=75), or use VPA to auto-tune resources.</p><h3>RBAC and Security</h3><p>RBAC (Role-Based Access Control) controls <strong>who can do what</strong> in your cluster. Four objects: <strong>Role</strong> (permissions in a namespace), <strong>ClusterRole</strong> (permissions cluster-wide), <strong>RoleBinding</strong> and <strong>ClusterRoleBinding</strong> (assign roles to users/groups/service accounts).</p><p>Security essentials: enable RBAC (default on managed K8s), use <strong>Network Policies</strong> (don’t let all pods talk to all pods), scan images with <strong>Trivy</strong> or <strong>Snyk</strong>, don’t run as root ( runAsNonRoot: true), use dedicated service accounts with minimal permissions, encrypt secrets at rest, and keep K8s updated.</p><h3>Helm and GitOps</h3><p><strong>Helm</strong> is the package manager for Kubernetes. Instead of managing dozens of raw YAML files, you use <strong>charts</strong> — templated packages that bundle all the resources for an application.</p><pre>helm repo add bitnami https://charts.bitnami.com/bitnami helm install my-postgres bitnami/postgresql --set auth.postgresPassword=secret</pre><p>You can create your own charts for your apps, with values.yaml files per environment. Way better than copy-pasting YAML across staging and production.</p><p><strong>Argo CD</strong> is the most popular GitOps tool for K8s. The idea: your Git repo is the single source of truth for your cluster state. Push a change to Git, Argo CD detects the diff and syncs your cluster automatically. No more kubectl apply from someone&#39;s laptop.</p><p>This combo — Helm for packaging, Argo CD for delivery — is the standard setup for production K8s deployments.</p><h3>Common Mistakes</h3><p>A few things I’ve seen go wrong repeatedly:</p><ol><li><strong>No resource limits</strong> — a single pod can eat all node memory and take down everything else with it</li><li><strong>Using </strong><strong>latest tag</strong> - you lose track of what&#39;s deployed, rollbacks break, and you can&#39;t reproduce issues</li><li><strong>No liveness/readiness probes</strong> — K8s can’t tell if your app is actually healthy or just running</li><li><strong>Secrets in Git</strong> — even base64-encoded. Use Sealed Secrets or external secret managers</li><li><strong>Skipping Network Policies</strong> — by default all pods can talk to everything. Lock it down</li><li><strong>Not setting Pod Disruption Budgets</strong> — during node drains, all your pods could be evicted at once</li><li><strong>Ignoring resource requests</strong> — the scheduler can’t make good placement decisions without them</li></ol><h3>When to Use K8s (and When Not To)</h3><p><strong>Use it when:</strong> multiple microservices, auto-scaling needs, high availability requirements, multi-environment deployments, multi-cloud strategy.</p><p><strong>Don’t use it when:</strong> simple app or monolith, small team without K8s experience, simpler alternatives work (ECS, Cloud Run, Fly.io), cost matters a lot for small workloads.</p><p>Rule of thumb — if you can count your containers on two hands and they run on 1–2 servers, you probably don’t need Kubernetes.</p><h3>Managed vs Self-Managed</h3><p>For production, go managed. <strong>GKE</strong> has the best K8s experience (Google created K8s, and it shows). <strong>EKS</strong> for AWS shops (~$73/mo per cluster). <strong>AKS</strong> is free and integrates well with Azure AD. <strong>DigitalOcean/Civo</strong> for small teams on a budget.</p><p>Don’t want to manage nodes at all? <strong>GKE Autopilot</strong> or <strong>AWS Fargate</strong> — pay per pod, no nodes to worry about. More expensive per compute unit, but saves operational effort.</p><p>For self-managed (hard mode): <strong>kubeadm</strong> is the official tool, <strong>kOps</strong> automates setup on AWS, <strong>Rancher/RKE2</strong> adds a management UI. Only go this route if you have strong infrastructure experience.</p><h3>kubectl Cheat Sheet</h3><h3>Summary</h3><p>Kubernetes is the standard for container orchestration — placement, self-healing, scaling, service discovery. The reconciliation loop (desired vs actual state) is the core idea that powers everything. Core objects: Pods, Deployments, Services, StatefulSets. Scale with HPA/VPA/Karpenter. Route traffic with Ingress or Gateway API. Package with Helm, deploy with Argo CD.</p><p>The downsides are real — complexity, YAML hell, networking headaches, debugging pain. Don’t reach for K8s when simpler tools would do. But when you genuinely need it, nothing else comes close.</p><p>More detailed notes with all deep dives (service mesh, operators, Karpenter configs, network policies, secret management), full YAML examples, and interview questions:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/kubernetes-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/kubernetes-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: are you running K8s in production? Managed or self-hosted? What’s been your biggest pain point? I’d love to hear what tools and patterns are working for you!</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/03/learning-system-design-7-kubernetes/"><em>https://sadensmol.com</em></a><em> on March 24, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=badba367340b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #6: AI Agents]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-6-ai-agents-3421cd44f6dd?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/3421cd44f6dd</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Mon, 16 Mar 2026 00:00:19 GMT</pubDate>
            <atom:updated>2026-03-16T16:58:47.397Z</atom:updated>
            <content:encoded><![CDATA[<p>Sixth part of the “Learning System Design” series! This time we’re talking about AI agents — the thing that took the industry by storm and turned “just ask the AI” into “let the AI do it for you.” If you’ve used Claude Code, Cursor, or GitHub Copilot agent mode — you’ve already interacted with one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TgFAiiWa-vMqt9z_.jpg" /></figure><p>But what actually makes an agent an agent? How does it decide what to do? And most importantly — when should you build one vs just writing a normal program? Let’s figure it out.</p><h3>What is an AI Agent?</h3><p>Think of a regular chatbot as a waiter who can only answer questions from the menu. You ask “what soups do you have?” — you get a list. That’s it. The waiter doesn’t cook, doesn’t go to the store, doesn’t adjust recipes.</p><p>An agent is more like a personal chef. You say “make me something Italian.” The chef checks the fridge (uses tools to gather information), realizes there’s no basil, goes to the store (takes actions in the real world), adjusts the recipe based on what’s available (reasons and adapts), and cooks the meal (produces the result). Nobody told the chef which steps to take or in what order — the chef figured it out.</p><p>That’s the core difference: <strong>a chatbot responds, an agent acts.</strong></p><p>Or if you prefer a programming analogy — a chatbot is a pure function: input in, output out. An agent is a program with a while loop that keeps running, making decisions, calling APIs, reading files, and adjusting its approach until the job is done.</p><p>The formula is surprisingly simple:</p><pre>Agent = LLM + Tools + Loop</pre><ul><li><strong>LLM</strong> — the reasoning engine that reads the situation and decides what to do next</li><li><strong>Tools</strong> — functions the LLM can call to interact with the real world (read files, query databases, call APIs)</li><li><strong>Loop</strong> — the LLM keeps thinking and acting until the task is done, not just one response</li></ul><p>That’s it. Every agent framework — OpenAI, Anthropic, LangChain — is just a different way of wiring these three things together.</p><p>Four properties make something an agent: <strong>autonomy</strong> (it decides the steps, not a hardcoded script), <strong>tool use</strong> (it interacts with external systems), <strong>reasoning</strong> (it thinks about what to do based on observations), and <strong>goal-directed behavior</strong> (it works toward completing a task, not just answering a question).</p><h3>Agent vs Chatbot vs Pipeline</h3><p>These terms get confused constantly. Here’s the difference:</p><p><strong>Chatbot</strong> — takes input, generates text. No tools, no actions. “What’s the capital of France?” → “Paris.” Done.</p><p><strong>Pipeline</strong> — a fixed sequence of steps. Summarize → Translate → Format. The order is predetermined by the developer. Predictable, but rigid.</p><p><strong>Agent</strong> — dynamically decides which tools to use and in what order. Can loop, retry, change strategy. The developer defines the tools and goal, but the agent decides the path.</p><p>The key insight: <strong>a chatbot responds, an agent acts.</strong></p><pre>Chatbot: User asks → LLM responds → Done Agent: User asks → LLM thinks → Uses tool → Observes result → Thinks again → Uses another tool → Observes → ... (loop until goal is met) → Delivers final answer</pre><h3>The ReAct Loop</h3><p>Every agent runs on the same pattern — <strong>ReAct</strong> (Reasoning + Acting). Here’s a concrete example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*uIjEb6wTZ7EuxNHU.png" /></figure><p>Say you ask: “What’s the weather in the city where Apple HQ is located?”</p><pre>Step 1 - THINK: &quot;I need to find where Apple&#39;s HQ is.&quot; Step 2 - ACT: search(&quot;Apple headquarters location&quot;) Step 3 - OBSERVE: &quot;Apple Park, Cupertino, California&quot; Step 4 - THINK: &quot;Now I need the weather there.&quot; Step 5 - ACT: get_weather(&quot;Cupertino, CA&quot;) Step 6 - OBSERVE: &quot;72°F, Sunny&quot; Step 7 - RESPOND: &quot;It&#39;s 72°F and Sunny in Cupertino.&quot;</pre><p>In code, this is embarrassingly simple:</p><p>This is the core of <strong>every</strong> agent framework. Everything else is just wrappers and conveniences on top.</p><h3>Core Building Blocks</h3><h3>Tools</h3><p>Tools are what separate an agent from a chatbot. The LLM never calls tools directly — it’s a text-in, text-out API. The flow is:</p><ol><li>Your code sends the LLM a list of available tools (name, description, parameters)</li><li>The LLM responds with “I want to call tool X with arguments Y” (as structured JSON)</li><li>Your code executes the function and sends the result back</li><li>The LLM reads the result and decides what to do next</li></ol><p>What makes a good tool: <strong>clear name</strong> (the LLM reads it), <strong>clear description</strong> (the LLM uses it to decide when to call), <strong>typed parameters</strong>, and <strong>predictable output</strong>.</p><h3>Memory and Context</h3><p>Two closely related but different things:</p><ul><li><strong>Context</strong> — the text sent to the LLM on each API call. Everything the LLM can “see”</li><li><strong>Memory</strong> — the mechanism that determines what goes into the context</li></ul><p>Think of it this way: <strong>context is the LLM’s desk</strong> — whatever papers are on it right now. <strong>Memory is the filing cabinet</strong> — where information lives between sessions.</p><p>The critical thing to understand: <strong>LLMs are stateless.</strong> Each API call starts completely fresh. Your code rebuilds the full context from memory every single time. The LLM thinks it’s having a continuous conversation, but in reality your code is replaying everything on every call.</p><h3>Planning</h3><p>Advanced agents break complex tasks into subtasks before executing. Three strategies:</p><ul><li><strong>Upfront planning</strong> — create full plan, then execute step by step</li><li><strong>Adaptive planning</strong> — plan a few steps ahead, adjust based on results</li><li><strong>Hierarchical planning</strong> — break into high-level goals, then sub-goals</li></ul><p>Planning agents are powerful, but there’s a catch — <strong>how does a human review and refine the plan before execution?</strong> In most setups, the agent outputs a plan as text in the terminal. You can approve or reject it, but giving specific feedback (“delete step 3, change step 5 to use Redis”) means typing everything out. For complex plans, this is tedious.</p><p><strong>Plannotator</strong> (plannotator.ai) solves this. It integrates with coding agents like Claude Code and opens the agent’s plan in a visual UI where you can select text, mark it for deletion, add comments, or suggest replacements. Feedback flows back to the agent as structured annotations. You can also share plans with your team via URL and save approved plans to Obsidian for a searchable archive. Plan quality determines execution quality — a 5-minute investment reviewing a plan visually can save hours of the agent going in the wrong direction.</p><h3>Guardrails</h3><p>Safety mechanisms that validate inputs and outputs. <strong>Input guardrails</strong> block prompt injection, off-topic requests, or out-of-scope queries before the agent processes them. <strong>Output guardrails</strong> ensure no sensitive data is leaked and the response format is correct.</p><h3>Agent Architectures</h3><p>There are several ways to structure multi-agent systems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*3MT-RldRUOsT3DtW.png" /></figure><p><strong>Single Agent</strong> — one agent with multiple tools. The simplest architecture. Use for straightforward tasks.</p><p><strong>Multi-Agent Handoffs</strong> — a triage agent routes requests to specialized agents. The math question goes to the math agent, the code question to the code agent. Use when different tasks require different instructions, tools, or models.</p><p><strong>Orchestrator-Worker</strong> — a central orchestrator delegates subtasks to worker agents in parallel. Think of a research task: one worker searches the web, another analyzes data, a third writes the report. The orchestrator combines everything. Use for complex tasks that decompose into independent subtasks.</p><p><strong>Pipeline</strong> — agents process data in a fixed sequence, each transforming the output for the next. Researcher → Analyzer → Writer → Reviewer. Use when stages are clear and sequential.</p><p><strong>Ralph Loop</strong> — not a multi-agent pattern, but a meta-pattern worth mentioning here. Instead of coordinating multiple agents, you run the <strong>same agent repeatedly</strong> with the same prompt, letting it build on its own previous work through file changes. The codebase becomes the shared state. More on this below.</p><h3>Context Engineering</h3><p>This is the hidden cost killer that nobody talks about enough. Every token in your context window costs money, adds latency, and competes for the LLM’s attention. Worse — LLMs perform <strong>worse</strong> when given too much irrelevant context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*dT6L8iBbBL-hFnlT.png" /></figure><p>In an agent loop, context accumulates with every step:</p><pre>Step 1: 2,200 tokens → $0.002 Step 5: 25,000 tokens → $0.025 Step 10: 60,000 tokens → $0.060 Step 20: 150,000 tokens → $0.150</pre><p>Each step re-sends the <strong>entire conversation history</strong>. By step 10, you’re paying for 60K input tokens just for the LLM to re-read early messages. This is where most agent cost comes from.</p><h3>How to Fix It</h3><p><strong>Summarize, don’t accumulate.</strong> Instead of keeping every tool result verbatim, summarize older results. Keep only recent ones in full.</p><p><strong>Trim tool output.</strong> A database query might return 500 rows when the agent only needs a count. A file read might return 5000 lines when only 20 are relevant. Design your tools to return just what’s needed.</p><p><strong>Load on demand.</strong> Don’t put everything in the system prompt. Instead of 5000 tokens of product catalog always present, use tools that load specific info when the agent asks for it.</p><p><strong>Use tiered models.</strong> Route simple subtasks (classify, extract, format) to cheaper models like Haiku. Save the expensive Opus for complex reasoning and planning.</p><h3>The Ralph Loop</h3><p>The Ralph Loop (also called the “Ralph Wiggum technique,” pioneered by Geoffrey Huntley) is one of the most interesting patterns I’ve seen for agent-powered development. The idea is beautifully simple: give the agent the <strong>same prompt over and over</strong>, and let it build on its own previous work each iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*YJebYx9CAmUsSdK_.png" /></figure><p>Each iteration:</p><ol><li>Agent receives the <strong>same prompt</strong></li><li>Works on the task, modifying files</li><li>Tries to finish</li><li>A hook intercepts the exit and feeds the same prompt again</li><li>Agent sees its previous work in the codebase</li><li>Iteratively improves until completion</li></ol><p>The “self-referential” part is key: the agent doesn’t get its previous output as input. Instead, it sees the <strong>state of the codebase</strong> it modified in prior iterations — like a developer coming back to their own code the next day.</p><h3>Why It Works</h3><p>Traditional agent loops handle tool calls within a single session. The Ralph Loop operates at a higher level — it loops <strong>entire agent sessions</strong>.</p><p><strong>It overcomes single-session limitations.</strong> Complex tasks often exceed what an agent can do in one pass. Ralph keeps the agent working across multiple sessions, each with a fresh context window.</p><p><strong>Failures become predictable.</strong> The technique is described as “deterministically bad in a non-deterministic world.” Failures follow patterns, so you can tune the prompt to avoid them systematically.</p><p><strong>Files are the memory.</strong> No special memory system needed — the codebase itself is the memory. Each iteration reads the current state and builds on it.</p><h3>When to Use It</h3><p><strong>Good fit:</strong> tasks with clear success criteria — refactoring with tests, adding test coverage, building from a spec. Anything where “run tests” or “check coverage” can tell the agent if it’s done.</p><p><strong>Bad fit:</strong> tasks needing human design decisions, creative work, one-shot operations, or anything with unclear success criteria.</p><h3>Practical Example</h3><pre>ralph-loop &quot;Refactor the database layer to use connection pooling. \ Run all tests after each change. \ --completion-promise &quot;REFACTOR COMPLETE&quot; \</pre><p>The agent will analyze the code in iteration 1, start refactoring in iteration 2, fix failing tests in iteration 3, and keep going until everything passes. Each time it starts fresh but sees the accumulated changes in the files.</p><h3>Tips for Effective Ralph Loops</h3><ul><li><strong>Write specific, measurable prompts</strong> — “Fix the bug” is too vague. “Fix the auth token refresh, all tests must pass” is actionable</li><li><strong>Include a verification command</strong> — tell the agent to run tests or builds so it can self-verify</li><li><strong>Always set max-iterations</strong> — without it, a stuck loop burns tokens forever</li><li><strong>Use completion promises</strong> — let the agent signal when it’s done rather than always hitting the max</li></ul><h3>Real-World Agentic Systems</h3><p>Agents aren’t just a toy for chatbots. Here are systems that already exist or are being actively built:</p><p><strong>Coding assistants</strong> — Claude Code, Cursor, GitHub Copilot agent mode. You describe what you want, the agent reads your codebase, writes code, runs tests, fixes errors, and iterates until it works. This is the most mature category right now.</p><p><strong>Customer support</strong> — an agent that reads the customer’s message, looks up their account in the database, checks order history, applies refund policies, and resolves the issue. No human in the loop for 80% of tickets. The remaining 20% get escalated with full context attached.</p><p><strong>Code review bots</strong> — an agent that monitors PRs, reads the diff, checks for security issues, runs static analysis, verifies test coverage, and posts a structured review. Not just “looks good” — actual actionable feedback.</p><p><strong>Data analysis pipelines</strong> — “analyze last quarter’s sales and find anomalies.” The agent queries the database, runs statistical analysis, generates charts, and writes a summary. Different every time because the data is different.</p><p><strong>DevOps incident response</strong> — an agent that gets paged, reads logs, correlates metrics, identifies the root cause, and either fixes it automatically or prepares a runbook for the on-call engineer. At 3am, you want the agent to do the detective work for you.</p><p><strong>Research assistants</strong> — “find all papers about distributed consensus published in the last 2 years and summarize the key insights.” The agent searches, reads, filters, and synthesizes. What would take you a day takes the agent 10 minutes.</p><p><strong>Content pipelines</strong> — writing, editing, fact-checking, and publishing. An agent that takes a topic, researches it, writes a draft, checks facts against sources, and formats it for your blog. I’m not saying this article was written by one&amp;mldr; but it’s a theme for another article!</p><h3>What You Could Build</h3><p>If you’re thinking about building something with agents, here are patterns that work well:</p><p><strong>Internal tools automation</strong> — take your company’s repetitive workflows (onboarding new employees, provisioning infrastructure, generating reports) and wrap them in an agent. The agent handles the happy path, escalates edge cases to humans.</p><p><strong>Smart assistants for domain experts</strong> — a legal assistant that reads contracts and flags risky clauses, a medical assistant that summarizes patient history before a consultation, a financial assistant that monitors portfolios and explains anomalies. The agent doesn’t replace the expert — it prepares the work.</p><p><strong>Multi-step workflows with branching logic</strong> — anything where the next step depends on the result of the previous one. Insurance claim processing, loan applications, compliance checks. These are painful to hardcode because the decision tree is huge, but an agent with the right tools and guardrails handles them naturally.</p><p><strong>Monitoring and alerting</strong> — agents that watch your systems, correlate signals across multiple sources, and only alert you when something actually needs attention. No more alert fatigue from dumb threshold-based rules.</p><h3>When to Use Agents (and When Not To)</h3><p><strong>Use agents when:</strong></p><ul><li>Tasks require dynamic decision-making based on intermediate results</li><li>You can’t predict the exact steps needed upfront</li><li>The task involves multiple tools and information sources</li><li>You need iterative refinement (code, research, analysis)</li></ul><p><strong>Don’t use agents when:</strong></p><ul><li>A deterministic script would work — if/else is faster, cheaper, and more reliable than an LLM</li><li>The task is simple and well-defined — a function call doesn’t need an agent</li><li>Latency matters — every LLM call adds seconds</li><li>Cost matters at scale — agents are expensive per-task compared to traditional code</li><li>You need guaranteed behavior — agents are non-deterministic by nature</li></ul><p>The honest truth: most tasks don’t need agents. A well-written function, a SQL query, or a simple pipeline will outperform an agent in speed, cost, and reliability for the vast majority of use cases. Agents shine when the problem is genuinely open-ended and requires reasoning.</p><h3>Summary</h3><p>AI agents are LLM + Tools + Loop. The ReAct pattern (think → act → observe → repeat) is the foundation of every agent framework. Context engineering is the hidden lever for cost and quality — smaller, more relevant context beats larger, noisier context every time.</p><p>The Ralph Loop is a powerful pattern for complex iterative tasks — same prompt, multiple sessions, files as memory. And the most important lesson: don’t reach for an agent when a simple function would do.</p><p>More detailed notes with all architectures, SDK examples (OpenAI, Anthropic, DSPy), and interview questions:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/ai-agents-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/ai-agents-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: are you building with agents? What frameworks are you using? I’d love to hear what’s working for you and what’s not!</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/03/learning-system-design-6-ai-agents/"><em>https://sadensmol.com</em></a><em> on March 16, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3421cd44f6dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning System Design #5: Understanding PostgreSQL Locks]]></title>
            <link>https://medium.com/@sadensmol/learning-system-design-5-understanding-postgresql-locks-34f612fc57c2?source=rss-1dc086cfb1fd------2</link>
            <guid isPermaLink="false">https://medium.com/p/34f612fc57c2</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[locks]]></category>
            <dc:creator><![CDATA[Denis Sazonov]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 00:00:22 GMT</pubDate>
            <atom:updated>2026-03-12T17:11:27.814Z</atom:updated>
            <content:encoded><![CDATA[<p>Fifth part of the “Learning System Design” series! This time we’re going deep into PostgreSQL locks — the thing nobody thinks about until their production database grinds to a halt because someone ran an ALTER TABLE on a busy table at peak hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-KMta8cLCLihFPdo.jpg" /></figure><p>Locks are the mechanism PostgreSQL uses to coordinate concurrent access to the same data. Every SELECT, UPDATE, INSERT, DELETE, ALTER TABLE - they all take some kind of lock. Most of the time you never notice because PostgreSQL handles it automatically. But when something goes wrong - a migration blocks all traffic, a deadlock kills transactions, or an idle connection holds a lock for hours - you <strong>really</strong> need to understand how this works.</p><h3>Three Families of Locks</h3><p>PostgreSQL has three distinct lock families:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*JMiNMDI4dwVAnlJ1.png" /></figure><h3>Table Locks: Why ALTER TABLE Is Scary</h3><p>There are 8 table lock modes ranging from ACCESS SHARE (weakest, used by SELECT) to ACCESS EXCLUSIVE (strongest, used by ALTER TABLE, DROP, TRUNCATE). The key insight is the <strong>conflict matrix</strong> — which locks can coexist and which block each other.</p><p>The important rules:</p><ul><li>SELECT never blocks SELECT</li><li>SELECT never blocks INSERT/UPDATE/DELETE</li><li>INSERT/UPDATE/DELETE never block each other at the table level (row locks handle conflicts)</li><li>ACCESS EXCLUSIVE blocks <strong>everything</strong> - even SELECT</li></ul><p>That last point is where the real trouble starts. Here’s the scenario that has ruined many engineers’ evenings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*6ju9jAHuKTbxZYH2.png" /></figure><p>A slow report query is running. You deploy a migration with ALTER TABLE orders ADD COLUMN status VARCHAR(20). It needs ACCESS EXCLUSIVE, so it waits for the report to finish. Reasonable. But now <strong>every new query</strong> - even tiny fast SELECTs - queues behind your ALTER. One migration creates a traffic jam for the entire table. Your API starts timing out. Alerts fire.</p><p>The fix? Always set lock_timeout before DDL:</p><p>If it can’t get the lock in 3 seconds, it fails instead of building a queue. Retry with exponential backoff.</p><h3>Row Locks: The Daily Workhorse</h3><p>Row locks are what you deal with 99% of the time. Four modes: FOR KEY SHARE, FOR SHARE, FOR NO KEY UPDATE, FOR UPDATE — from weakest to strongest.</p><p>The most important pattern is <strong>SELECT FOR UPDATE</strong> — the classic “check-then-act”:</p><p>Without FOR UPDATE, two sessions can read available = true simultaneously and both book the same seat. With it, the second session waits until the first commits.</p><p>Two more tools you should know:</p><p><strong>SKIP LOCKED</strong> — for job queues. Multiple workers grab tasks without blocking each other. If a row is locked, skip it and grab the next one:</p><p><strong>NOWAIT</strong> — fail immediately instead of waiting. Great when you’d rather show “resource busy, try again” than make the user stare at a spinner:</p><h3>Deadlocks: The Circular Wait</h3><p>A deadlock happens when two transactions each hold a lock the other needs. Session 1 has row 1 and wants row 2. Session 2 has row 2 and wants row 1. Neither can proceed.</p><p>PostgreSQL detects deadlocks automatically (checks every deadlock_timeout, default 1 second), picks a victim, and aborts it with ERROR: deadlock detected. The other transaction continues.</p><p><strong>Prevention is simple: always lock rows in the same order.</strong> If both sessions lock row 1 first, then row 2 — no circular dependency possible. Use ORDER BY id with SELECT FOR UPDATE when locking multiple rows.</p><h3>Advisory Locks: Your Custom “Reserved” Sign</h3><p>Advisory locks are perfect for application-level coordination that doesn’t map to specific rows or tables:</p><p>One big gotcha: session-level advisory locks ( pg_advisory_lock) survive COMMIT and ROLLBACK. They stay until you explicitly unlock or disconnect. Prefer pg_advisory_xact_lock - it auto-releases on transaction end.</p><h3>Safe Migration Patterns</h3><p>This is where lock knowledge really pays off in practice. Here’s what you need to know:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*KNB7jyRWpvZyi5oc.png" /></figure><p>The pattern is always the same: <strong>split heavy operations into light steps</strong>. Don’t scan the table while holding the strongest lock. Add the constraint as NOT VALID (instant), then validate it separately (scans with a lighter lock).</p><h3>Troubleshooting: Finding and Fixing Lock Issues</h3><p>When your queries are stuck, here’s the diagnostic query you need:</p><p>Found the blocker? If it’s an idle-in-transaction session (someone forgot to COMMIT) — kill it with SELECT pg_terminate_backend(&lt;pid&gt;). If it&#39;s a long-running migration - decide whether to wait or cancel.</p><p><strong>Prevention settings</strong> that every production database should have:</p><h3>Summary</h3><p>PostgreSQL locks are mostly invisible — until they’re not. The key takeaways:</p><ul><li><strong>Table locks are the dangerous ones.</strong> ACCESS EXCLUSIVE (ALTER TABLE, DROP, TRUNCATE) blocks everything, including SELECT. Always use lock_timeout.</li><li><strong>Row locks are fine.</strong> They only block other operations on the same row. Use FOR UPDATE for check-then-act, SKIP LOCKED for queues, NOWAIT for fail-fast.</li><li><strong>Deadlocks are preventable.</strong> Lock rows in consistent order. Keep transactions short.</li><li><strong>Migrations need care.</strong> Use CONCURRENTLY for indexes, NOT VALID + VALIDATE for constraints, and always set lock timeouts.</li></ul><p>More detailed notes with conflict matrices, all 8 lock modes explained, and ready-to-use monitoring queries:</p><p><a href="https://github.com/sadensmol/learning_system-design/blob/main/postgres-locks-guide.md">https://github.com/sadensmol/learning_system-design/blob/main/postgres-locks-guide.md</a></p><p>Thanks for reading! More system design topics coming in the next parts of the series.</p><p>PS: ever had a production incident caused by locks? ALTER TABLE on a hot table during peak hours? I’d love to hear your war stories!</p><p><em>Originally published at </em><a href="https://sadensmol.com/posts/2026/03/learning-system-design-5-understanding-postgresql-locks/"><em>https://sadensmol.com</em></a><em> on March 12, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34f612fc57c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>