<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mi Do on Medium]]></title>
        <description><![CDATA[Stories by Mi Do on Medium]]></description>
        <link>https://medium.com/@mi-do?source=rss-73a219a96e70------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*txxUwGRmyLUG3N2NA9ITag.jpeg</url>
            <title>Stories by Mi Do on Medium</title>
            <link>https://medium.com/@mi-do?source=rss-73a219a96e70------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:22:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mi-do/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Local Kubernetes Dev — Part 7: Kubernetes manifests for your service]]></title>
            <link>https://mi-do.medium.com/local-kubernetes-dev-part-7-kubernetes-manifests-for-your-service-c48b02105c4f?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/c48b02105c4f</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 12:38:31 GMT</pubDate>
            <atom:updated>2026-08-12T12:38:31.558Z</atom:updated>
            <content:encoded><![CDATA[<p>“The Service exists, but traffic never reaches the pods, and Endpoints is empty” — the most common beginner bug in Kubernetes. And the cause is almost always the same one.</p><p>Part seven of the series is the chapter where, instead of copy-pasting YAML, you finally understand HOW Kubernetes objects connect to each other. We write three minimal manifests for myapp — Namespace, Deployment, Service — and unpack what you really need to internalize: labels and selectors are the glue that binds Deployment and Service together. No hard references like “Service, here are the IDs of these pods” — it’s all held together by matching labels.</p><p>The key rule everyone trips on: a Deployment’s spec.selector.matchLabels MUST match its spec.template.metadata.labels. And the big trap — the Service selector doesn&#39;t match the pod labels (a typo, or you forgot to update it): the Service just hangs, and kubectl describe svc shows Endpoints: &lt;none&gt;. The service seems to exist, but traffic goes nowhere. You can verify it instantly.</p><p>Plus: how ClusterIP differs from a pod’s direct address, the difference between port and targetPort (80 vs 8080), declarative kubectl apply vs imperative create, and where to keep your manifests (spoiler: in Git next to the code, in a k8s/ folder). <a href="https://dorokhovich.com/blog/local-k8s-kubernetes-manifests?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-kubernetes-manifests">https://dorokhovich.com/blog/local-k8s-kubernetes-manifests?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-kubernetes-manifests</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c48b02105c4f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trunk-Based Development vs GitFlow: The Principle Is to Match Your Branches to Your Cadence]]></title>
            <link>https://mi-do.medium.com/trunk-based-development-vs-gitflow-the-principle-is-to-match-your-branches-to-your-cadence-7f7d78a1b9fe?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f7d78a1b9fe</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 12:27:47 GMT</pubDate>
            <atom:updated>2026-08-12T12:27:47.998Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>A lot of repositories are museums of good intentions. Feature branches live for weeks, sometimes months, drifting further from the mainline every day until merging one means a full day of conflict archaeology. main has no protection, so a direct push can — and eventually does — destabilize a release. Hotfixes go out and never get back-merged, so the same bug reappears two releases later. And because unrelated features get mixed into a single release branch, no release is ever truly reproducible.</p><p>The reflex is to blame Git, or to reach for whatever branching model a louder team swears by. But the underlying issue in the perennial <strong>trunk-based development vs GitFlow</strong> debate is subtler: a branch in Git is just a lightweight pointer to a commit, so creating, merging, and deleting branches is cheap — which means long-lived branches are a <em>choice</em>, not a necessity. The teams drowning in conflicts are almost always shipping often while branching as if they shipped quarterly. That mismatch, not the tool and not the model, is the problem.</p><h3>The principle</h3><p>The principle here is that <strong>branch lifetime should track shipping frequency: the more often you ship, the shorter your branches must live and the stricter your CI gates into the mainline must be.</strong> Pick the model to match the cadence, rather than inheriting one by accident.</p><p>That reframe dissolves the “which model is correct” argument, because branching models have a clear arc and each fits a different cadence. From 2008–2012 it was <a href="https://nvie.com/posts/a-successful-git-branching-model/">GitFlow</a> with many long-lived branches; 2013–2018 simplified to GitHub Flow; from 2018 onward the industry moved to <a href="https://trunkbaseddevelopment.com/">trunk-based development</a> with short branches and feature flags. GitFlow is not wrong — its own author now notes it suits versioned software with multiple supported releases rather than continuously deployed web apps. If releases are infrequent and you run many parallel supported versions, its explicit release and hotfix branches are genuinely clearer. If you ship frequently, the answer is trunk-based plus feature flags plus strict CI. There is <a href="https://dorokhovich.com/blog/release-orchestration/git-branching?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/git-branching">a detailed treatment of choosing and running a branching model</a> that walks the full decision; the compressed rule is: cadence dictates model.</p><p>The second, deeper principle is to know what branches are actually <em>for</em>, because every merge-hell symptom is one of their benefits inverted. Branches buy exactly four things:</p><ul><li><strong>Risk isolation</strong> — unfinished work does not break the stable line. (Long-lived branches invert this by drifting.)</li><li><strong>Parallelism</strong> — features, releases, and hotfixes proceed at once.</li><li><strong>Quality control</strong> — PRs, mandatory reviews, green-CI gates. (An unprotected main inverts this.)</li><li><strong>Traceability</strong> — commits, tags, and release notes document exactly what shipped. (Mixed release branches invert this.)</li></ul><p>Reframing the goal as “preserve those four properties” is what makes the specific rules feel principled rather than arbitrary. It also fixes the vocabulary: main is the always-deployable, protected production source of truth; develop is an optional integration buffer for sprint cadences; feature/<em> is short-lived and one-PR-per-goal; </em><em>release/</em> is stabilization only (fixes and version/CHANGELOG bumps, never new features); hotfix/<em> branches off </em><em>main and integrates </em>back* into both main and develop.</p><h3>Trade-offs</h3><p>The choice is not binary good-vs-bad; it is a fit question. The honest comparison:</p><ul><li><strong>Best-fit cadence</strong> — <em>GitFlow (long-lived branches):</em> Infrequent, versioned releases; multiple supported versions; <em>Trunk-based (short branches + flags):</em> Frequent / continuous deployment</li><li><strong>Merge cost</strong> — <em>GitFlow (long-lived branches):</em> Grows with branch age — conflict archaeology; <em>Trunk-based (short branches + flags):</em> Small and frequent</li><li><strong>Release reproducibility</strong> — <em>GitFlow (long-lived branches):</em> Clear via explicit release branches; <em>Trunk-based (short branches + flags):</em> Clear via trunk + tags + flags</li><li><strong>Incomplete work</strong> — <em>GitFlow (long-lived branches):</em> Hidden in a long branch; <em>Trunk-based (short branches + flags):</em> Merged behind a feature flag</li><li><strong>CI strictness required</strong> — <em>GitFlow (long-lived branches):</em> Moderate; <em>Trunk-based (short branches + flags):</em> High — the mainline gate is load-bearing</li><li><strong>Main failure mode</strong> — <em>GitFlow (long-lived branches):</em> Parallel branches drift out of sync; <em>Trunk-based (short branches + flags):</em> Undisciplined flags accumulate as debt</li></ul><p>The trade-off worth stating plainly: trunk-based buys small merges and fast flow, but only if you pay for it with strict CI and <a href="https://martinfowler.com/articles/feature-toggles.html">feature-flag discipline</a> — flags are what make merging incomplete work safe, and a flag with no owner or expiry is future debt. GitFlow buys clean parallel-version management, but the bill arrives as merge cost that scales with branch age. Choosing trunk-based while keeping weeks-long branches gives you the costs of both and the benefits of neither.</p><h3>How to adopt</h3><p>Do not adopt a whole new model overnight — that is the failure mode, and it invites the resistance you fear. Sequence by leverage.</p><ol><li><strong>Protect </strong><strong>main first.</strong> Require reviews and green CI; forbid direct pushes. This is the single highest-payoff change: the moment direct pushes are impossible, &quot;unstable release&quot; incidents essentially stop.</li><li><strong>Shorten feature branches.</strong> Aim for hours to a couple of days. Add a lightweight branch-age report to CI that flags any feature branch older than a few days, so drift becomes visible <em>before</em> it becomes painful. Making the invisible cost visible changes behaviour more than any policy memo.</li><li><strong>Adopt one release lifecycle and walk every release through it.</strong> Preparation branches off a clean mainline:</li></ol><pre>git switch develop &amp;&amp; git pull --ff-only<br>git switch -c release/1.8.0<br>git commit -am &quot;chore(release): bump to 1.8.0 &amp; update CHANGELOG&quot;</pre><p>The release merges into main with a no-fast-forward merge and an annotated tag, then — the step teams skip — back-integrates into develop so nothing is lost:</p><pre>git switch main &amp;&amp; git pull --ff-only<br>git merge --no-ff release/1.8.0 -m &quot;release: 1.8.0&quot;<br>git tag -a v1.8.0 -m &quot;Release 1.8.0&quot;<br>git push origin main --tags</pre><p>Hotfixes follow the same discipline in miniature: branch off main, fix, merge back into <em>both</em> main and develop. Skipping that final back-merge is the exact bug that resurrects old defects. This release/hotfix structure is the part worth keeping from <a href="https://nvie.com/posts/a-successful-git-branching-model/">Vincent Driessen&#39;s original model</a> even as you shorten everything else toward trunk.</p><ol><li><strong>Make the good path the quick path.</strong> Standardize a handful of commands and aliases so the right thing is also the easy thing:</li></ol><pre>git switch -c feature/login<br>git fetch --prune<br>git pull --ff-only<br>git switch -c release/2.0.0 develop<br>git switch main &amp;&amp; git merge --no-ff release/2.0.0 &amp;&amp; git tag -a v2.0.0 -m &quot;Release 2.0.0&quot;</pre><p>Drill one rule hardest: never rebase a published branch — rewriting shared history breaks teammates; for public branches prefer merge --no-ff. Pair that with <a href="https://www.conventionalcommits.org/en/v1.0.0/">Conventional Commits</a> and auto-generated changelogs and semantic releases come almost for free.</p><p>The cultural surprise is that resistance rarely materializes once aliases and protected main are in place. Nobody enjoys conflict archaeology, so a workflow that quietly removes it sells itself after the first clean release. You do not mandate discipline so much as remove the friction that was rewarding the bad habits.</p><h3>Where this goes next</h3><p>The direction of travel is trunk-based plus feature flags, <strong>branch protection expressed as policy-as-code</strong>, and automated releases that compress the cycle to hours. Flags are the piece that makes short branches safe — you merge incomplete work behind a flag instead of hiding it in a long-lived branch — and combined with canary or blue-green delivery, branching stops being a source of risk and becomes bookkeeping.</p><p>The further horizon is that as more of the commit-to-release path gets handled by automation and AI-assisted tooling — bots that open, review, and land small changes; agents that assemble release notes from conventional commits; policy engines that gate merges on live quality signals — the value of a short, linear, well-tagged history compounds. Automated systems reason far more reliably over a clean trunk than over a thicket of drifting branches. The teams that match their branching to their cadence today are the ones whose history will be legible enough for the next generation of tooling to safely act on.</p><h3>Sources &amp; further reading</h3><ul><li>Vincent Driessen, <a href="https://nvie.com/posts/a-successful-git-branching-model/">A successful Git branching model</a> — the original GitFlow, plus the author’s later note on when it no longer fits.</li><li><a href="https://trunkbaseddevelopment.com/">Trunk-Based Development</a> — the short-lived-branch model and why it scales with shipping frequency.</li><li><a href="https://www.conventionalcommits.org/en/v1.0.0/">Conventional Commits</a> — the commit convention that powers auto-generated changelogs and semantic releases.</li><li>Pete Hodgson, <a href="https://martinfowler.com/articles/feature-toggles.html">Feature Toggles</a> — how flags make short branches and trunk-based work safe.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/release-orchestration/git-branching?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/git-branching">choosing and running a Git branching model</a> — every command and the GitFlow-vs-trunk decision for a given cadence.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f7d78a1b9fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local Kubernetes Dev — Part 6: Containerizing your service — writing a Dockerfile]]></title>
            <link>https://mi-do.medium.com/local-kubernetes-dev-part-6-containerizing-your-service-writing-a-dockerfile-e1b4ddced117?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1b4ddced117</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Sun, 09 Aug 2026 09:04:16 GMT</pubDate>
            <atom:updated>2026-08-09T09:04:16.854Z</atom:updated>
            <content:encoded><![CDATA[<p>Two mistakes beginners make in Dockerfiles constantly — and both have real consequences: running as root (a security hole) and the wrong layer order (a tax on build speed).</p><p>Part six of the series is a production-ready Dockerfile for the FastAPI service myapp. Not “your first Dockerfile,” but a breakdown of what separates a solid image from a bloated, insecure one. Inside: why COPY requirements.txt BEFORE COPY ./app — and pip install gets cached instead of reinstalling everything on every code change; a multi-stage build that carries only the finished venv into the final image, without compilers and pip caches; base image choice (slim vs alpine with its musl and compile-from-source vs distroless with no shell); and the security backbone of the chapter — an unprivileged user (adduser --uid 10001 + USER appuser) plus runAsNonRoot in the manifest.</p><p>Plus the important details that break prod silently: the exec form of CMD (otherwise SIGTERM never arrives and graceful shutdown breaks), fastapi run instead of bare uvicorn, PYTHONUNBUFFERED=1 (otherwise you see no logs in kubectl logs), HEALTHCHECK via Python (there&#39;s no curl in slim).</p><p>And the finale — the main trap: docker build does NOT make the image visible to k3d. Two ways to deliver it: k3d image import and the built-in registry. <a href="https://dorokhovich.com/blog/local-k8s-containerizing-your-service?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-containerizing-your-service">https://dorokhovich.com/blog/local-k8s-containerizing-your-service?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-containerizing-your-service</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1b4ddced117" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Storytelling in Software Design: The Principle That Makes Architecture Docs Actually Stick]]></title>
            <link>https://mi-do.medium.com/storytelling-in-software-design-the-principle-that-makes-architecture-docs-actually-stick-7ed21d868ca0?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ed21d868ca0</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 16:27:15 GMT</pubDate>
            <atom:updated>2026-08-01T16:27:15.957Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>Every engineering org I have worked with has the same artifact: a wiki page that opens with something like <em>‘Three-tier architecture: presentation, business logic, data access.’</em> It is accurate. It is correct. And it is completely inert. New hires read it, nod, and then spend their first month asking questions the page technically already answered.</p><p>The instinct is to blame the writer or the tooling — not enough diagrams, wrong wiki, stale content. That instinct is wrong. The information was fine. What failed was the <em>form</em>. We had hidden the meaning behind jargon and boxes-and-arrows and stripped out the one thing human cognition is actually built to retain: a throughline. A component inventory is a list of nouns. A system is a sequence of things happening to actors over time — and our brains file the second away far more reliably than the first.</p><p>That gap between what we produce (inventories) and what people retain (narratives) is the whole problem. It shows up as slow onboarding, as design reviews that miss the point, and as decisions that evaporate the moment the person who made them leaves.</p><h3>The principle</h3><p>The principle here is that <strong>story is a design technique, not decoration.</strong> A narrative — actors, an event, a resolution — is a way of structuring concepts so that intent is visible, and intent is the part that has to survive new people and the passage of time.</p><p>Here is the same system, documented two ways:</p><blockquote><strong><em>Inventory form:</em></strong><em> ‘Three-tier architecture: presentation, business logic, data access.’</em></blockquote><blockquote><strong><em>Narrative form:</em></strong><em> ‘Clients (presentation) consult experts (domain) who consult the archive (data) and return a decision. Each request is a plotline that crosses actors and comes back with a resolution.’</em></blockquote><p>The narrative version is not fluffier — it is <em>more</em> precise, because it exposes intent rather than just structure. A reader who absorbs the second version can reason about where a new feature belongs. A reader who memorizes the first can only recite it. That is the entire thesis in miniature: narrative scales because intent scales, and implementation details do not.</p><p>The leverage point is that you do not need a new methodology to apply this. The mature design practices are <em>already</em> narrative — most teams just run them mechanically and miss it. There is a detailed treatment of this reframing in <a href="https://dorokhovich.com/blog/storytelling-software-design?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=storytelling-software-design">a longer essay on software design as storytelling</a> if you want the full argument; the short version is that five techniques you likely already own are stories wearing engineering clothes.</p><ul><li><strong>Domain-Driven Design (DDD).</strong> <a href="https://martinfowler.com/bliki/BoundedContext.html">Bounded contexts</a> are chapters; the ubiquitous language is the cast’s shared dialogue. Ask ‘who are the characters here and what words do they use?’ and boundaries that looked arbitrary suddenly have reasons.</li><li><strong>Behavior-Driven Development (BDD).</strong> Given / When / Then, <a href="https://dannorth.net/blog/introducing-bdd/">the grammar Dan North coined for BDD</a>, is a plot: a setup, an event, a resolution — readable by every stakeholder and executable as a spec.</li><li><strong>Event Storming.</strong> <a href="https://www.eventstorming.com/">Alberto Brandolini’s workshop format</a> models events, commands, and policies on a wall until the arc of the system emerges in sticky notes. You watch the plot appear.</li><li><strong>Domain Storytelling.</strong> <a href="https://domainstorytelling.org/">The most literal technique</a>, documented by Stefan Hofer and Henning Schwentner and <a href="https://www.thoughtworks.com/radar/techniques/domain-storytelling">featured on the Thoughtworks Technology Radar</a>: domain experts draw their workflow as numbered steps — actors, work objects, activities, in order. ‘The <em>dispatcher</em> assigns the <em>order</em> to a <em>driver</em>’ hands you the language and the boundaries for free.</li><li><strong>Architecture Decision Records (ADRs).</strong> <a href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">Michael Nygard’s format</a> is the memoir: context, decision, consequences — the <em>why</em>, preserved for whoever inherits it. Pair it with <a href="https://c4model.com/">Simon Brown’s C4 model</a> for the map, but keep the rule that every diagram earns a sentence of narrative or it is just decoration.</li></ul><h3>Trade-offs</h3><p>Narrative design is not free, and treating it as an unqualified good is how it degrades into whimsy. The honest comparison:</p><ul><li><strong>Onboarding retention</strong> — <em>Inventory docs:</em> Read once, forgotten; <em>Narrative design:</em> Retained and reasoned about</li><li><strong>Authoring cost</strong> — <em>Inventory docs:</em> Low up front; <em>Narrative design:</em> Higher up front, lower over time</li><li><strong>Precision</strong> — <em>Inventory docs:</em> Exact but intent-free; <em>Narrative design:</em> Exact <em>and</em> intent-bearing when done well</li><li><strong>Failure mode</strong> — <em>Inventory docs:</em> Inert, ignored; <em>Narrative design:</em> Vague metaphor if intent is hidden</li><li><strong>Cross-team alignment</strong> — <em>Inventory docs:</em> Repeated meetings; <em>Narrative design:</em> Shared story spine</li><li><strong>Decision durability</strong> — <em>Inventory docs:</em> Lost when people leave; <em>Narrative design:</em> Preserved in ADRs</li><li><strong>Business ‘why’</strong> — <em>Inventory docs:</em> In one senior head; <em>Narrative design:</em> Embedded in the artifact</li></ul><p>The two costs that matter: narrative takes more effort to write well, and it fails badly when done poorly. A story that hides detail instead of exposing intent is worse than a dry doc — you have added metaphor without adding clarity. So the discipline is that <em>narrative must be more precise, never less.</em> If your reframing makes the system harder to reason about, revert it.</p><h3>How to adopt</h3><p>The adoption failure I see most often is boiling the ocean — a team decides to rewrite every document as a story and stalls under the weight. Stage it instead.</p><ol><li><strong>Start with the single most-ignored document</strong>, usually the onboarding page, and rewrite it as a story with actors, a journey, and a resolution. Prove the principle on the artifact that hurts most before you touch anything else.</li><li><strong>Write your next architecture decision as an ADR the day you make it.</strong> An ADR written at decision time preserves intent; one reconstructed six months later preserves a guess. This is the highest-leverage habit on the list.</li><li><strong>Run one flow through Event Storming as a group.</strong> Put a single user-facing flow on a wall — events, commands, actors — and narrate it end to end out loud: ‘the customer does this, which triggers that, which billing decides on, and the resolution is this.’ Within an hour the arc emerges, and so do the boundary disagreements the team has been silently carrying. Saying the story together is what makes the ubiquitous language stick — it arrives by conversation, not decree.</li><li><strong>Enforce the vocabulary.</strong> If the story’s words and the code’s names drift apart, the throughline is gone. Keep them in sync or the whole effort decays.</li></ol><p>Notice what this is <em>not</em>: no new framework, no tool purchase, no reorg. You are changing the form of what you already produce so the meaning survives contact with new people and with time.</p><h3>Where this goes next</h3><p>The direction of travel is that narrative stops being how you <em>document</em> a system and becomes how you <em>design</em> it. Once a team pitches features as plotlines — ‘here is the actor, here is what changes, here is the resolution’ — reviews sharpen automatically, because a reviewer can ask ‘whose story is this and where does it resolve?’ and expose a design gap no checklist would catch.</p><p>There is a second horizon worth watching: as more of the design surface gets handed to AI assistants, intent-bearing artifacts become the differentiator. A model can generate a component inventory trivially; what it cannot invent is <em>why</em> your bounded contexts sit where they do. Teams that have captured that why as narrative and ADRs will hand their tools far better context than teams sitting on boxes-and-arrows. The organizations that treat storytelling software design as a first-class engineering practice — rather than a nicety for the onboarding wiki — are the ones whose intent will outlast both their people and their tooling.</p><h3>Sources &amp; further reading</h3><ul><li>Stefan Hofer &amp; Henning Schwentner — <a href="https://domainstorytelling.org/">Domain Storytelling</a>, and its <a href="https://www.thoughtworks.com/radar/techniques/domain-storytelling">Thoughtworks Technology Radar entry</a></li><li>Alberto Brandolini — <a href="https://www.eventstorming.com/">EventStorming</a></li><li>Martin Fowler — <a href="https://martinfowler.com/bliki/BoundedContext.html">Bounded Context</a> (Domain-Driven Design)</li><li>Dan North — <a href="https://dannorth.net/blog/introducing-bdd/">Introducing BDD</a> (the origin of Given/When/Then)</li><li>Michael Nygard — <a href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">Documenting Architecture Decisions</a> (the original ADR format)</li><li>Simon Brown — <a href="https://c4model.com/">The C4 model for visualising software architecture</a></li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/storytelling-software-design?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=storytelling-software-design">reframing software design as storytelling</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ed21d868ca0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Raspberry Pi VNC Remote Desktop From Anywhere: The Principle of Multiplexing a Tunnel You Already…]]></title>
            <link>https://mi-do.medium.com/a-raspberry-pi-vnc-remote-desktop-from-anywhere-the-principle-of-multiplexing-a-tunnel-you-already-affe9b3bfe5c?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/affe9b3bfe5c</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 09:50:14 GMT</pubDate>
            <atom:updated>2026-07-31T09:50:14.705Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Raspberry Pi VNC Remote Desktop From Anywhere: The Principle of Multiplexing a Tunnel You Already Own</h3><h3>The problem in context</h3><p>SSH into a home Raspberry Pi gives you a black text console — perfect for editing configs, useless for anything with a mouse. Most of the time that is fine, but there is always a stubborn 10%: a graphical config tool with no CLI equivalent, or a camera app whose live preview you genuinely need to <em>see</em>. For those, you need VNC — screen-sharing that streams the Pi’s desktop to your laptop and sends clicks and keystrokes back.</p><p>The naive path is to install a VNC server, forward port 5900 on the router, and connect. Two things kill that plan. First, a Pi behind carrier-grade NAT has no router port to forward — the same constraint that forces an outbound tunnel for SSH in the first place. Second, and more important, <strong>exposing raw VNC to the internet is a genuinely bad idea.</strong> The protocol’s own spec, <a href="https://datatracker.ietf.org/doc/html/rfc6143">RFC 6143</a>, concedes its password check is “cryptographically weak and is not intended for use on untrusted networks,” and recommends tunneling it inside IPsec or SSH. So the real problem is not “how do I reach VNC” but “how do I get a graphical desktop remotely <em>without adding any new attack surface</em>.”</p><h3>The principle</h3><p>The principle here is that <strong>an SSH tunnel is not limited to one port — the same encrypted connection that already carries SSH can carry a desktop too, so you multiplex a pipe you already own rather than opening a new one.</strong> If a reverse SSH tunnel already forwards the Pi’s port 22 out to a cloud bridge, VNC is just a second reverse forward on that same connection. Nothing new is exposed; the Pi still only ever dials outward; the traffic is wrapped in SSH the entire way.</p><p>The mental model that makes it click is three ports and one path:</p><ul><li><strong>Port 5900</strong> on the Pi is where the VNC server already listens — per the <a href="https://www.raspberrypi.com/documentation/computers/remote-access.html">official Raspberry Pi remote-access docs</a>, a VNC server (wayvnc on current Raspberry Pi OS) is built in, so nothing gets installed.</li><li><strong>Port 5901</strong> on the bridge becomes the public front door that forwards into the tunnel.</li><li>Everything hitting EC2:5901 gets redirected, encrypted, down to RPi:5900.</li></ul><pre>VNC Viewer  --&gt;  EC2:5901  --&gt;  encrypted SSH tunnel  --&gt;  RaspberryPi:5900  --&gt;  desktop image back the same way</pre><p>There is <a href="https://dorokhovich.com/blog/rpi/vnc-ec2-setup?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=rpi/vnc-ec2-setup">a detailed treatment of the full VNC-over-tunnel build</a> with the port map and security-group walkthrough; the mechanical change is a single added flag. An existing systemd tunnel unit carrying only SSH looks like:</p><pre>ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel ... -R 2222:localhost:22 tunnel@YOUR-EC2-IP</pre><p>Adding a second -R — the same remote-forwarding mechanism from the <a href="https://man.openbsd.org/ssh">ssh(1) man page</a>, just another instance of it — carries the desktop:</p><pre>ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel ... -R 2222:localhost:22 -R 5901:localhost:5900 tunnel@YOUR-EC2-IP</pre><p>After systemctl daemon-reload and a restart, ps aux | grep ssh | grep tunnel showing -R 5901:localhost:5900 in the running process is the checkpoint that the tunnel picked up the new forward.</p><h3>Trade-offs</h3><p>Multiplexing an owned tunnel beats every VNC-in-the-cloud SaaS on the axes that matter for a homelab, but the choice carries real security decisions you have to make deliberately:</p><ul><li><strong>Bridge port binding</strong> — <em>Convenient / default:</em> Loopback (SSH default); <em>Safer choice:</em> GatewayPorts yes; <em>Why it matters:</em> A loopback-bound forward is reachable <em>from</em> the bridge but never from your laptop</li><li><strong>Security-group source</strong> — <em>Convenient / default:</em> 0.0.0.0/0 on 5901; <em>Safer choice:</em> Scope to your own IP; <em>Why it matters:</em> Open means the whole internet can reach your VNC login prompt</li><li><strong>VNC transport</strong> — <em>Convenient / default:</em> Raw VNC over the internet; <em>Safer choice:</em> Wrapped in the SSH tunnel; <em>Why it matters:</em> The protocol’s own auth is weak by its spec’s admission</li><li><strong>VNC server</strong> — <em>Convenient / default:</em> RealVNC free tier; <em>Safer choice:</em> <a href="https://tigervnc.org/">TigerVNC</a> / wayvnc; <em>Why it matters:</em> RealVNC increasingly pushes cloud-brokered connections, the opposite of a self-owned tunnel</li></ul><p>The GatewayPorts line is the one that eats an hour. Check the bridge with sudo ss -tulpn | grep 5901 and you will likely see it bound to 127.0.0.1 — localhost only, reachable from the bridge but not from across the internet. This is SSH&#39;s safe default; the <a href="https://man.openbsd.org/sshd_config">sshd_config(5) man page</a> notes sshd binds remote forwardings to the loopback address unless told otherwise. One line, GatewayPorts yes, plus systemctl restart ssh and a tunnel restart, rebinds it to 0.0.0.0:5901. The rule worth internalizing: <em>a </em><em>127.0.0.1-bound forward is not broken, it is just missing </em><em>GatewayPorts.</em></p><p>The deeper trade-off is ownership versus convenience. Routing desktop pixels through a third-party VNC broker is easier, but the whole point of the owned tunnel is that no pixels ever leave infrastructure you control — the bridge is your box, the transport is plain SSH, and the only thing installed is a viewer on your laptop. That is worth defending, which is why the RealVNC-pushing-toward-cloud-brokering drift matters: if a direct address:port connection gets gated behind a subscription, TigerVNC or wayvnc speaks the same protocol over the same -R 5901:localhost:5900 forward, and the tunnel does not care which server sits on port 5900.</p><h3>How to adopt</h3><p>This assumes the reverse SSH tunnel already exists — VNC rides on top of it, so build that foundation first if you are starting from scratch.</p><ol><li><strong>Add the second </strong><strong>-R forward</strong> to the existing systemd unit and confirm it in the running process.</li><li><strong>Flip </strong><strong>GatewayPorts yes on the bridge</strong>, restart sshd, then restart the tunnel on the Pi so the forward re-establishes against the updated server. Order matters — re-check ports on the bridge after each restart. A local nc -zv localhost 5901 on the bridge confirms the forward actually reaches the Pi before you touch the firewall.</li><li><strong>Open exactly one firewall port</strong>, scoped tight: a security-group inbound rule for TCP 5901 sourced to your own IP, not 0.0.0.0/0. If you need roaming access, keep the VNC password strong and layer SSH key auth in front.</li><li><strong>Point a viewer at </strong><strong>EC2-IP:5901</strong>, enter the VNC password, and the desktop fills the window — mouse and keyboard, live.</li><li><strong>Make the session pleasant, not just working.</strong> Drop the viewer’s color depth for a big responsiveness win over a home uplink; set a sane fixed resolution on the headless Pi so the remote window fits your laptop; and prefer wired Ethernet on the Pi, which removes a whole class of “why is this laggy” questions before you ask them.</li></ol><p>Two operational choices keep it maintainable. Keep the VNC forward in the <em>same</em> systemd unit as SSH — one process, one thing to monitor, one place to look when something is wrong, and removing VNC is deleting one flag and reloading. And understand the reconnection behavior: because VNC rides the same managed tunnel, a network drop restarts both, and the viewer simply reconnects once the forward is back, usually within the ten-second restart window. A frozen screen after a Wi-Fi blip is not a failure; it comes back on its own.</p><h3>Where this goes next</h3><p>The immediate generalization is that <em>any</em> service on the Pi can ride this same connection — a metrics endpoint, a database, an internal dashboard — each one a -R forward on the tunnel you already trust, none of them exposed at home. Once you have internalized &quot;one outbound connection, many services multiplexed back,&quot; the pattern scales from a single Pi to a small fleet without ever revisiting the router.</p><p>The larger direction of travel is that this hand-built setup is a concrete instance of where secure remote access is heading industry-wide: identity- and key-gated access to individual services over an outbound-initiated, encrypted transport, rather than exposed inbound ports on a trusted network. That is the same premise underneath zero-trust access and the managed desktop-brokering services — the difference is only who owns the hops. Building it yourself, one -R flag at a time, is the best way to understand what those products are actually doing, and to keep the option of owning every hop when the traffic is your own desktop. The tooling will keep getting easier; the principle of multiplexing an owned, outbound tunnel is the part worth keeping.</p><h3>Sources &amp; further reading</h3><ul><li><a href="https://datatracker.ietf.org/doc/html/rfc6143">RFC 6143 — The Remote Framebuffer Protocol</a> — VNC’s own spec, on why its auth is weak and should be tunneled.</li><li><a href="https://www.raspberrypi.com/documentation/computers/remote-access.html">Raspberry Pi — Remote access documentation</a> — enabling the built-in VNC server.</li><li><a href="https://man.openbsd.org/ssh">OpenSSH </a><a href="https://man.openbsd.org/ssh">ssh(1) man page</a> and <a href="https://man.openbsd.org/sshd_config">sshd_config(5) man page</a> — the -R forward and GatewayPorts.</li><li><a href="https://tigervnc.org/">TigerVNC</a> — an open-source VNC server/viewer that speaks the same protocol without a subscription.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/rpi/vnc-ec2-setup?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=rpi/vnc-ec2-setup">VNC over a reverse SSH tunnel</a> — the port map and AWS security-group walkthrough behind this framing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=affe9b3bfe5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Local Kubernetes Dev — Part 4: Setting up your workstation]]></title>
            <link>https://mi-do.medium.com/local-kubernetes-dev-part-4-setting-up-your-workstation-9ff630b5bdf5?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ff630b5bdf5</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 09:48:47 GMT</pubDate>
            <atom:updated>2026-07-31T09:48:47.172Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we build clusters and write manifests — let’s install the tools and prove they actually work together.</p><p>Part four of the local Kubernetes series is the hands-on one: we install Docker, kubectl, k3d, helm, and Tilt on macOS, Linux, and Windows+WSL2, add k9s, and — most importantly — run a smoke test that proves the whole Docker → k3d → kubectl chain works end to end. Most guides tell you how to install the tools, but not how to verify they actually get along. We close that gap: k3d cluster create dev → kubectl get pods -A → see system Pods Running → k3d cluster delete dev.</p><p>There’s a dedicated section on the classic traps that cost people hours: the Docker daemon isn’t running (connection refused), too many open files from inotify limits (fixed with sysctls, covered in k3d issue #803), a Windows repo living in C:\... instead of WSL2 (slow FS and broken Tilt file-sync), permission denied on docker.sock (you&#39;re not in the docker group), virtualization disabled in the BIOS, and not enough RAM.</p><p>This is the “prep your machine” checkpoint before the build chapters. If you made it to the end and the smoke test is green — you’re ready to move on. <a href="https://dorokhovich.com/blog/local-k8s-workstation-setup?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-workstation-setup">https://dorokhovich.com/blog/local-k8s-workstation-setup?utm_source=medium&amp;utm_medium=syndication&amp;utm_campaign=local-k8s-workstation-setup</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff630b5bdf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Reverse SSH Tunnel for a Raspberry Pi: The Principle of Dialing Out Instead of Being Reached]]></title>
            <link>https://mi-do.medium.com/a-reverse-ssh-tunnel-for-a-raspberry-pi-the-principle-of-dialing-out-instead-of-being-reached-fe4eb3cbebae?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4eb3cbebae</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 09:41:35 GMT</pubDate>
            <atom:updated>2026-07-30T09:41:35.013Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>Consider a common homelab bind: a Raspberry Pi at home records a camera feed, runs a couple of cron jobs, and hosts a small dashboard. On the LAN it is fine. Then you travel, open a laptop, type the familiar ssh pi@home, and get a timeout. The instinct is to log into the router and forward port 22 — except there is no public IP to forward <em>to</em>. The ISP has quietly moved the connection behind CGNAT (carrier-grade NAT), the <a href="https://datatracker.ietf.org/doc/html/rfc6598">shared address space reserved in RFC 6598</a>, which means the &quot;public&quot; IP is shared with hundreds of subscribers and you control none of it.</p><p>That is the real problem, and it is worth naming precisely because it explains why the usual fixes all fail: <strong>port forwarding is meaningless when the port lives on someone else’s equipment.</strong> DDNS points at an address you cannot open, UPnP does nothing, and VPN appliances want a static IP you do not have. Every one of those approaches assumes you can accept an inbound connection at home. Behind CGNAT, you cannot — and no amount of router configuration changes that.</p><h3>The principle</h3><p>The principle here is that <strong>NAT blocks inbound connections but happily allows outbound ones, so you stop waiting for the world to knock and have the Pi knock on a server you own — then keep that door propped open.</strong> This is a reverse SSH tunnel, and it is the cleanest escape hatch behind CGNAT, a locked-down office network, or any router you would rather not touch.</p><p>Concretely: rent a tiny always-on box in the cloud with a real public IP, have the Pi open an SSH connection <em>out</em> to it, and use SSH remote port forwarding (-R) so a port on the cloud box tunnels straight back through that connection to port 22 on the Pi. You connect to the cloud box, hop through the tunnel, and you are on the Pi — no router config, no exposed ports at home. The architecture is three parts:</p><pre>Your laptop (anywhere)  --&gt;  EC2 bridge (public IP, :2222)  --&gt;  reverse tunnel  --&gt;  Raspberry Pi (:22, behind CGNAT)</pre><ul><li><strong>EC2 bridge:</strong> Ubuntu on a t2.micro (free-tier eligible); its only job is to be reachable and forward port 2222 back down the tunnel.</li><li><strong>Raspberry Pi:</strong> initiates and holds the tunnel open, restarting automatically if the network hiccups.</li><li><strong>The tunnel:</strong> an SSH remote port forward (-R 2222:localhost:22) wrapped in a systemd service so it survives reboots and drops.</li></ul><p>The -R flag is the whole trick — the <a href="https://man.openbsd.org/ssh">ssh(1) man page</a> defines it as forwarding connections to a given TCP port on the remote host back to the local side. There is <a href="https://dorokhovich.com/blog/rpi/reverse-ssh?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=rpi/reverse-ssh">a detailed treatment of the full reverse-SSH build</a> with every security-group rule and connection method; the mechanics compress to a handful of config lines below.</p><h3>Trade-offs</h3><p>A reverse SSH tunnel is not the only way to reach a machine behind NAT, and it is worth being honest about where it wins and loses against the managed options:</p><ul><li><strong>Reverse SSH tunnel</strong> — <em>What it costs:</em> You run and own a cloud bridge; <em>What it buys:</em> Depends on nothing but SSH + systemd; every hop is yours; no third-party control plane; <em>Best when:</em> You want to own every hop and understand exactly what carries your packets</li><li><a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/"><strong>Cloudflare Tunnel</strong></a> — <em>What it costs:</em> Traffic rides a third-party control plane; <em>What it buys:</em> Outbound-only cloudflared; firewall can block all inbound; least setup; <em>Best when:</em> You want the shortest path and don&#39;t want to run infrastructure</li><li><a href="https://tailscale.com/blog/how-nat-traversal-works"><strong>Tailscale</strong></a> — <em>What it costs:</em> A coordination service brokers peers; <em>What it buys:</em> Clever STUN/ICE NAT traversal builds direct peer-to-peer links; <em>Best when:</em> You want a mesh across many devices with minimal fuss</li></ul><p>All three are good; the choice is about what you are optimizing for. Cloudflare Tunnel is the easier button. The reverse SSH tunnel wins on ownership: it depends on nothing but tools already on the Pi, it does not route traffic through anyone else’s control plane, and the cloud bridge is a box you can rebuild in minutes. The cost you accept in exchange is running that bridge and securing an internet-facing port yourself — which is a real cost, and the section below is mostly about paying it correctly.</p><h3>How to adopt</h3><p>Build it in layers, proving each one before wrapping it in automation.</p><ol><li><strong>Prepare the bridge.</strong> Launch a t2.micro, open port 22 to your own IP and 2222 to 0.0.0.0/0, and set three lines in /etc/ssh/sshd_config:</li></ol><pre>GatewayPorts yes<br>ClientAliveInterval 60<br>ClientAliveCountMax 3</pre><p>GatewayPorts yes is the line most people miss. As the <a href="https://man.openbsd.org/sshd_config">sshd_config(5) man page</a> notes, sshd binds remote forwardings to loopback by default, so without it port 2222 stays invisible from the internet — the tunnel looks healthy but is unreachable from your laptop. The keepalives let the server reap dead tunnels instead of leaving zombie forwards holding the port.</p><ol><li><strong>Create a dedicated, unprivileged </strong><strong>tunnel user</strong> on the bridge whose only reason to exist is holding this forward. Least privilege: if it is ever compromised, the blast radius is one useless shell.</li><li><strong>Prove a manual tunnel</strong> from the Pi with a fresh ED25519 key whose public half is in the tunnel user&#39;s authorized_keys:</li></ol><pre>ssh -i ~/.ssh/id_tunnel -fN -R 2222:localhost:22 tunnel@YOUR-EC2-IP</pre><p>On the bridge, sudo ss -tulpn | grep 2222 should show it LISTENing.</p><ol><li><strong>Make it permanent with systemd</strong>, because a manual tunnel dies the moment the Pi reboots or Wi-Fi blinks:</li></ol><pre>[Unit]<br>Description=Reverse SSH Tunnel to EC2<br>After=network-online.target<br>Wants=network-online.target</pre><pre>[Service]<br>Type=simple<br>User=pi<br>ExecStart=/usr/bin/ssh -i /home/pi/.ssh/id_tunnel -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -N -R 2222:localhost:22 tunnel@YOUR-EC2-IP<br>Restart=always<br>RestartSec=10</pre><pre>[Install]<br>WantedBy=multi-user.target</pre><p>Each option earns its place: ExitOnForwardFailure=yes kills the process if the forward cannot bind, so systemd retries a <em>clean</em> tunnel; Restart=always — which the <a href="https://man7.org/linux/man-pages/man5/systemd.service.5.html">systemd.service(5) man page</a> defines as restarting regardless of exit status — with RestartSec=10 recovers within ten seconds; and ServerAliveInterval=60 stops idle NAT timeouts from silently severing the link. StrictHostKeyChecking=no is required because a headless unit has no human to accept a fingerprint prompt.</p><ol><li><strong>Make it pleasant with a two-hop </strong><strong>ProxyCommand</strong> in your laptop&#39;s ~/.ssh/config so ssh rpi transparently connects through the bridge and onto the Pi:</li></ol><pre>Host rpi<br>  HostName localhost<br>  Port 2222<br>  User pi<br>  IdentityFile ~/.ssh/id_rpi<br>  ProxyCommand ssh -i ~/.ssh/rpi-tunnel-key.pem -W localhost:2222 ubuntu@YOUR-EC2-IP</pre><p>The non-negotiable part is the security posture. Opening 2222 to the world is reasonable <em>only</em> because it is key-gated: disable password auth on the Pi, add fail2ban on the bridge, and treat that as the same-day work, not a follow-up. The gotchas that cost real time all trace back to the config above — a forgotten GatewayPorts yes (tunnel connects but stays localhost-bound), a missing StrictHostKeyChecking=no (headless service hangs on the fingerprint prompt), and zombie forwards after a hard drop (cleared by the server keepalives plus ExitOnForwardFailure).</p><p>Two operating habits keep this honest. Verify monthly — systemctl is-active on the Pi, ss -tulpn | grep 2222 on the bridge, and ssh rpi &#39;uptime&#39; end to end — because a tunnel you never check fails silently the day you need it, and journalctl -u shows the reason when it does. And treat the bridge as cattle, not a pet: its entire config is a few sshd_config lines and one user, so rebuilding takes minutes, and nothing irreplaceable lives on it, which means nothing irreplaceable is exposed on it.</p><h3>Where this goes next</h3><p>The immediate extension is that the same tunnel carries more than SSH: add a second -R 5901:localhost:5900 forward and the headless Pi becomes a full remote desktop over VNC, reusing everything above. Once you have internalized &quot;the device dials out and multiplexes services back,&quot; the pattern generalizes to any port you want to reach — a dashboard, a database, a metrics endpoint — without ever exposing them at home.</p><p>The larger direction of travel is that this is a hand-rolled instance of a principle the whole industry has converged on: <strong>outbound-only connectivity as the default security posture.</strong> Cloudflare Tunnel, Tailscale, and the broader zero-trust and SASE movement all rest on the same idea that a device should reach out to a control plane rather than expose inbound ports, and that identity and keys, not network location, gate access. Building the reverse tunnel by hand is the best way to understand what those managed services are actually doing for you — and to decide, per deployment, whether owning every hop is worth running the bridge yourself. The mental model is the durable part; the tooling that implements it will keep getting easier.</p><h3>Sources &amp; further reading</h3><ul><li><a href="https://man.openbsd.org/ssh">OpenSSH </a><a href="https://man.openbsd.org/ssh">ssh(1) man page</a> — the -R remote port forwarding flag.</li><li><a href="https://man.openbsd.org/sshd_config">OpenSSH </a><a href="https://man.openbsd.org/sshd_config">sshd_config(5) man page</a> — the GatewayPorts option.</li><li><a href="https://man7.org/linux/man-pages/man5/systemd.service.5.html">systemd.service(5) man page</a> — Restart= and RestartSec= semantics.</li><li><a href="https://datatracker.ietf.org/doc/html/rfc6598">RFC 6598 — IANA-Reserved IPv4 Prefix for Shared Address Space</a> — the standard behind CGNAT.</li><li><a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/">Cloudflare Tunnel docs</a> and <a href="https://tailscale.com/blog/how-nat-traversal-works">Tailscale’s “How NAT traversal works”</a> — the managed alternatives.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/rpi/reverse-ssh?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=rpi/reverse-ssh">this reverse SSH tunnel build</a> — the security-group rules, port reference, and troubleshooting matrix behind this framing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4eb3cbebae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Release Orchestration for Microservices: The Principle of the Control Tower, Not the Light Switch]]></title>
            <link>https://mi-do.medium.com/release-orchestration-for-microservices-the-principle-of-the-control-tower-not-the-light-switch-8c3461e4752b?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c3461e4752b</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Wed, 29 Jul 2026 08:26:30 GMT</pubDate>
            <atom:updated>2026-07-29T08:26:30.992Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>Picture the release a lot of microservices shops actually run. An engineer merges to main, CI builds the images, someone clicks deploy, and thirty seconds later two dozen services are rolling simultaneously with no coordination between them. Most nights it works. The night it doesn’t, a service ships a change its neighbor isn’t ready for, the neighbor starts throwing 500s, the service <em>behind</em> it times out waiting, and within four minutes the checkout path is down. Users no longer forgive minute-long outages, and this kind lasts a lot longer than a minute.</p><p>The post-mortem is always the same shape: no staged rollout, no automated health gate, no clean way to undo, and zero visibility for anyone outside the on-call channel. That is the core problem <strong>release orchestration</strong> addresses: a fleet of 60 interdependent services being shipped with the coordination model of a single monolith. The reflex is to blame the change, or the engineer, or CI. But CI did its job — it built and delivered the code. What was missing was the layer that decides <em>whether the whole system is ready for that code to go live</em>, and in what order.</p><h3>The principle</h3><p>The principle here is that <strong>a release is air-traffic control, not a light switch.</strong> Planes can technically take off on their own; without a control tower you get chaos in the sky. Each microservice <em>can</em> update itself — but without a control layer deciding order, timing, and approval, one backward-incompatible change becomes a company-wide outage.</p><p>That analogy is also the cleanest way to see the difference between CI/CD and orchestration. CI/CD is the assembly line that builds and delivers code. Orchestration is the control tower deciding <em>which</em> system updates first, <em>how</em> changes synchronize, and <em>who</em> signs off. Most teams that suffer domino outages have a working assembly line and no tower. There is <a href="https://dorokhovich.com/blog/release-orchestration/overview?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/overview">a detailed treatment of release orchestration for microservices</a> that walks the full model; the compressed principle is that orchestration is a control layer over the pipeline, not a fancier pipeline.</p><p>The control tower is built from three techniques, best adopted in order of pain relieved:</p><ul><li><strong>Feature flags</strong> decouple deploy from release. Pete Hodgson’s <a href="https://martinfowler.com/articles/feature-toggles.html">feature toggles guide</a> names this a <em>release toggle</em>: ship the code, keep the behavior dark. A risky change sits dormant in production, gets flipped on for internal users then everyone, and flips <em>off</em> in seconds if something smells wrong — no redeploy.</li><li><strong>Canary releases</strong> cap blast radius. Instead of 100% at once, a new version goes to a slice of traffic — what Danilo Sato’s <a href="https://martinfowler.com/bliki/CanaryRelease.html">canary release definition</a> frames as rolling out to a small subset before the whole fleet. On Kubernetes with Argo Rollouts, the config is almost embarrassingly small, mirroring the setWeight/pause model in the <a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts canary reference</a>:</li></ul><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Rollout<br>metadata:<br>  name: example-app<br>spec:<br>  strategy:<br>    canary:<br>      steps:<br>      - setWeight: 10<br>      - pause: {duration: 10m}<br>      - setWeight: 50<br>      - pause: {duration: 10m}<br>      - setWeight: 100</pre><p>Ten percent, pause and watch, fifty, pause and watch, then full. The pause windows are where automated health checks live — if error rate or latency crosses a threshold during a pause, the rollout aborts and holds at the last safe weight instead of marching to 100%.</p><ul><li><strong>Blue-green</strong> handles the changes you can’t canary — schema-coupled services, mostly — by running two identical environments, the pattern Martin Fowler <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">described in 2010</a>. Users stay on blue while green bakes; when green is verified, traffic flips instantly and blue stays warm as a fallback.</li></ul><h3>Trade-offs</h3><p>None of the three is universal; each buys a different guarantee at a different cost. The honest comparison:</p><ul><li><strong>Feature flags</strong> — <em>Best for:</em> Decoupling deploy from release; <em>Rollback:</em> Flip off in seconds, no redeploy; <em>The cost to respect:</em> Stale flags become hidden branches — debt with a fuse</li><li><strong>Canary</strong> — <em>Best for:</em> Everyday releases where partial traffic makes sense; <em>Rollback:</em> Auto-abort at last safe weight; <em>The cost to respect:</em> Only as smart as the metrics behind the pause steps</li><li><strong>Blue-green</strong> — <em>Best for:</em> Schema-coupled changes you can’t slice; <em>Rollback:</em> Instant flip back to blue; <em>The cost to respect:</em> Roughly double resources; DB/session state is hard</li></ul><p>Two constraints do the most damage when ignored. A canary is only as smart as its metrics: if the pause steps aren&#39;t backed by real health signals — error rate, latency, saturation — you have added slow-motion to a bad deploy, so wire the checks before you trust the automation. And config is production code: the worst incidents tend to come from YAML and flag flips, not application code, so deployment config needs the same review gate. Tooling choice is a smaller trade-off than teams expect — <a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts</a> if you are already all-in on Kubernetes, <a href="https://docs.flagger.app/">Flagger</a> for a lighter automatic metric-driven loop, Spinnaker for multi-cloud sprawl. The real cost is not tools; it is the team time to design the process.</p><h3>How to adopt</h3><p>Do not boil the ocean. The staged path that works is three steps.</p><ol><li><strong>Assess.</strong> Answer honestly: how long does a release take, how many manual steps, how often do you roll back, does the business have any visibility? Writing those answers down is uncomfortable and clarifying, and it gives you the baseline to prove improvement.</li><li><strong>Start small — process before tech.</strong> Introduce release checklists, code review on <em>config</em> changes, and a written rollback plan for every release; then flags for new functionality, monitoring on key metrics, and automated health checks.</li><li><strong>Scale up.</strong> Only then template the canary config across services.</li></ol><p>Budget for the culture lag: the tools install in a day, but getting everyone to write a rollback plan per release takes a quarter. It is worth it. And when you need to justify the work, frame it in leadership’s units, not yours — nobody buys “canary rollouts,” they buy fewer outages and faster shipping. Calculate the cost of one hour of downtime, multiply by historical incident frequency, and set that against the near-zero cost of free tooling; the rollback plan alone tends to pay for the initiative on paper. The counterintuitive headline is that decoupling deploy from release and capping blast radius does not slow teams down — it lets them ship <em>more</em>, because each ship is cheap to undo. Competitors shipping 3–5x more often aren’t smarter; they have just made each release cheap to reverse.</p><h3>Where this goes next</h3><p>The direction of travel is toward the control tower making more of its own decisions. AIOps — machine learning that predicts release risk and flags suspicious metrics automatically — is the natural next layer, feeding richer signals into the same pause-and-abort gates that a human tunes today. But it is an accelerant on top of good orchestration, not a substitute for it: you cannot ML your way out of not having a control tower in the first place.</p><p>The deeper forward-looking point is that every one of these techniques produces the structured signal that smarter automation will need — labeled rollouts, health-gated pauses, explicit rollback plans, config under review. Teams that build the tower now are not just avoiding tonight’s domino outage; they are assembling the legible, well-instrumented substrate that AI-assisted release agents will reason over next. The control layer is the thing that stays valuable as the intelligence sitting on top of it improves — which is exactly why it is worth building before the automation arrives, not after.</p><h3>Sources &amp; further reading</h3><ul><li>Danilo Sato, <a href="https://martinfowler.com/bliki/CanaryRelease.html">Canary Release</a> — martinfowler.com.</li><li>Martin Fowler, <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Blue Green Deployment</a> — martinfowler.com.</li><li>Pete Hodgson, <a href="https://martinfowler.com/articles/feature-toggles.html">Feature Toggles (aka Feature Flags)</a> — martinfowler.com.</li><li><a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts — Canary Deployment Strategy</a> — official docs.</li><li><a href="https://docs.flagger.app/">Flagger — progressive delivery for Kubernetes</a> — official docs.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/release-orchestration/overview?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/overview">release orchestration for microservices</a> — the tool comparison and the assess/start-small/scale roadmap behind this framing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c3461e4752b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A GitOps Adoption Roadmap: The Principle of Walking the Arc Instead of Leaping It]]></title>
            <link>https://mi-do.medium.com/a-gitops-adoption-roadmap-the-principle-of-walking-the-arc-instead-of-leaping-it-50604990b2e3?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/50604990b2e3</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Tue, 28 Jul 2026 20:35:05 GMT</pubDate>
            <atom:updated>2026-07-28T20:35:05.972Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>Anyone in the industry long enough remembers the sleepless release night: the coffee, the collective prayer that a deploy would go smoothly, the one guru who alone understood the process, no documentation, and rollbacks so frequent they stopped feeling like exceptions. The canonical horror story — a telecom billing release that ran 14 hours, involved twelve people, crashed three times, and needed another six hours to recover — is only an exaggeration by degree. Plenty of teams still gather on Friday evenings for six-to-twelve-hour manual releases.</p><p>The reflex, once the pain is acute enough, is to buy the shiniest platform and leap straight to it. That reflex is exactly what makes transformations stall. The problem is not that a team lacks GitOps; it is that manual-deploy shops try to adopt GitOps as a single jump, skipping the rungs — script automation, CI/CD, declarative delivery — that GitOps quietly assumes already exist. Framed that way, escaping 3 AM deploys stops being a unique curse and becomes a solved problem with a well-worn path.</p><h3>The principle</h3><p>The principle here is that <strong>a GitOps adoption roadmap is a layered arc, not a purchase: each layer assumes the one below it, so you walk scripts → CI/CD → GitOps → progressive delivery rather than leaping.</strong> The industry traveled this arc for a reason, and repeating its order is a good sign you are not skipping steps.</p><p>The mental model is four phases, each earning the next:</p><ul><li><strong>Assess.</strong> Measure before you automate. A blunt maturity checklist, answered honestly, tells you where the pain is <em>and</em> gives you the baseline you will later use to prove progress:</li></ul><pre># Maturity assessment checklist<br>[ ] Deployment time (target: &lt; 30 minutes)<br>[ ] Release frequency (target: weekly or more often)<br>[ ] Manual steps count (target: 0)<br>[ ] Mean time to restore (target: &lt; 1 hour)<br>[ ] Change success rate (target: &gt; 95%)<br>[ ] Automated test coverage (target: &gt; 80%)<br>[ ] Monitoring and alerting in place<br>[ ] Rollback and disaster recovery procedures</pre><p>Those targets line up with the throughput and stability signals <a href="https://dora.dev/guides/dora-metrics/">DORA formalizes as its core delivery metrics</a>. Assess organizational readiness too — executive sponsorship, willingness to change, DevOps expertise, budget — because skipping any of those is how transformations stall.</p><ul><li><strong>Quick wins.</strong> Chase visible pain reduction: standardize release checklists and runbooks, containerize apps, put infrastructure in code, stand up basic CI, add health checks.</li><li><strong>Scale.</strong> Make Git the single source of truth with <a href="https://opengitops.dev/">GitOps</a> and a reconciling controller like <a href="https://argo-cd.readthedocs.io/en/stable/">Argo CD</a>.</li><li><strong>Optimize.</strong> Layer on progressive delivery — feature flags, canary, observability-driven automatic rollback.</li></ul><p>There is <a href="https://dorokhovich.com/blog/release-orchestration/learning-path?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/learning-path">a detailed treatment of the full manual-to-GitOps learning path</a> with the tools comparison and timeline behind each phase; the compressed principle is that quick wins are how you <em>earn permission</em> for the disruptive changes. Shave one service’s deploy from hours to under an hour with a basic pipeline and health checks, and the skeptics stop arguing whether automation is worth it — the proof is in front of them. Momentum is a currency, and quick wins are how you mint it.</p><h3>Trade-offs</h3><p>The tooling arc recapitulates the industry’s evolution, and each rung trades setup effort for tighter integration and stronger guarantees. The honest comparison:</p><ul><li><strong>Script automation (</strong><a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html"><strong>Ansible</strong></a><strong>)</strong> — <em>What it buys:</em> Turns tribal knowledge into documentation-as-code; agentless, readable playbooks; <em>The cost / when it fits:</em> Still push-based and imperative; fine as the first rung, not the destination</li><li><strong>CI/CD (</strong><a href="https://docs.github.com/en/actions/about-github-actions/understanding-github-actions"><strong>GitHub Actions</strong></a><strong>)</strong> — <em>What it buys:</em> Build/test/deploy folded into version control; low barrier; <em>The cost / when it fits:</em> Some setup effort; pipeline logic can sprawl</li><li><strong>GitOps (Argo CD)</strong> — <em>What it buys:</em> Git as source of truth; versioned, reviewable, self-healing, no drift; <em>The cost / when it fits:</em> Requires declarative discipline and Kubernetes maturity underneath</li><li><strong>Progressive delivery (</strong><a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/"><strong>Argo Rollouts</strong></a><strong>)</strong> — <em>What it buys:</em> Analysis-gated canaries; automatic rollback on metric breach; <em>The cost / when it fits:</em> Needs trustworthy metrics and the three rungs below it already solid</li></ul><p>The trade-off that matters most is <em>ordering discipline</em>: each rung assumes the one below. GitOps on top of a cluster nobody can describe declaratively just relocates the chaos into YAML; progressive delivery without trustworthy metrics automates a decision you cannot yet make. A representative Ansible playbook and an Argo CD Application make the two ends concrete:</p><pre>- name: Deploy web application<br>  hosts: webservers<br>  become: yes<br>  tasks:<br>    - name: Update application files<br>      copy: { src: /builds/myapp-v2.0/, dest: /opt/myapp/ }<br>    - name: Check application health<br>      uri: { url: http://localhost:8080/health, status_code: 200 }</pre><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: myapp<br>spec:<br>  source:<br>    repoURL: https://github.com/company/myapp-config<br>    path: k8s<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true</pre><p>The deeper trade-off is where you spend effort. The tempting failure modes are all one-dimensional: changing everything at once (chaos and resistance), fixing only tooling while ignoring process and culture (minimal impact), and treating security or metrics as an afterthought. The costliest of these is cultural — the technology is the easy part; getting people to trust automation over heroics, and to run blameless post-mortems instead of assigning blame, is the real work.</p><h3>How to adopt</h3><p>Start with the audit, not the tooling, and walk the arc one measured phase at a time.</p><ol><li><strong>Run the maturity audit and baseline the metrics.</strong> You cannot prove an improvement you never measured.</li><li><strong>Find the single biggest pain point and fix it with a quick win.</strong> Prove it with the metric you baselined. Bank the visible, uncontroversial success before touching anything invasive.</li><li><strong>Introduce GitOps once CI and containers are stable.</strong> With Argo CD, every change becomes a reviewable pull request, every state is versioned and instantly revertible, and the cluster stops drifting from what Git says — the four OpenGitOps properties (declarative, versioned, pulled, continuously reconciled) doing the work.</li><li><strong>Add progressive delivery last.</strong> Encode a canary that pauses, runs an automated analysis against an error-rate query, and advances only if the metric stays healthy. That is the moment releases become genuinely low-drama: the system, not a stressed human at 3 AM, decides whether to proceed.</li><li><strong>Invest in the team as much as the tools.</strong> Training reliably returns more than buying yet another platform, and it is what converts a blame-and-burnout culture into a blameless, continuous-learning one.</li></ol><p>Perfect is the enemy of progress — a small improvement today beats the perfect plan still being refined next quarter.</p><h3>Where this goes next</h3><p>The direction of travel is toward <strong>fully closed-loop delivery</strong>: Git as the declared intent, controllers reconciling reality to it, and progressive-delivery analysis promoting or reverting releases without a human in the path. Once the arc is walked and the metrics are trustworthy, the natural next step is handing more of the promote/rollback judgement to the system — and, increasingly, to AI-assisted operations that reason over the same signals to forecast capacity, flag anomalies on release, and draft the changes that reconcile drift.</p><p>None of that can be bolted onto Friday-night deploys later; it compounds on the layers beneath it. The teams walking the roadmap now are not just escaping the sleepless nights — they are building the legible, declarative, well-instrumented substrate that the next generation of autonomous delivery tooling will need in order to be trusted at all. That is the real reason to walk the arc rather than leap it: each rung you lay down is what makes the next one, human or machine, safe to stand on.</p><h3>Sources &amp; further reading</h3><ul><li><a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html">Ansible playbooks</a> and <a href="https://docs.github.com/en/actions/about-github-actions/understanding-github-actions">GitHub Actions</a> — the script-automation and CI/CD rungs of the arc.</li><li><a href="https://argo-cd.readthedocs.io/en/stable/">Argo CD docs</a> and <a href="https://opengitops.dev/">OpenGitOps principles</a> — declarative, Git-as-source-of-truth delivery.</li><li><a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts — Canary strategy</a> — automated, analysis-gated progressive delivery.</li><li>DORA, <a href="https://dora.dev/guides/dora-metrics/">DORA metrics</a> — the baseline metrics that prove each phase actually helped.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/release-orchestration/learning-path?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/learning-path">the full manual-to-GitOps learning path</a> — the four-phase roadmap, tools comparison, and timeline/budget detail.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50604990b2e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blue-Green vs Canary Deployment: The Principle Is to Stop Choosing and Start Combining]]></title>
            <link>https://mi-do.medium.com/blue-green-vs-canary-deployment-the-principle-is-to-stop-choosing-and-start-combining-45045d5eebf3?source=rss-73a219a96e70------2</link>
            <guid isPermaLink="false">https://medium.com/p/45045d5eebf3</guid>
            <dc:creator><![CDATA[Mi Do]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 07:59:46 GMT</pubDate>
            <atom:updated>2026-07-26T07:59:46.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>The problem in context</h3><p>The releases that hurt most are the ones where a team has to <em>decide, live,</em> whether things are bad enough to roll back. Error rate looks a little high — is that the new version or normal noise? Nobody agreed on a threshold in advance, so the argument happens in the incident channel while users suffer. That single failure mode — rollback as a live debate — is what a deployment strategy exists to prevent, and it is why the perennial “blue-green vs canary deployment” argument is usually the wrong frame.</p><p>Treated as a ranking problem, it produces a stalemate: blue-green people cite instant rollback, canary people cite limited blast radius, and the team picks one and inherits the other’s weaknesses. The reframe is that these are not rivals to rank. They are tools that answer three different questions — <em>how do we roll out, how do we roll back, how do we limit blast radius</em> — and a real delivery system needs a different answer at different blast radii.</p><h3>The principle</h3><p>The principle here is that <strong>deployment strategy is blast-radius management, and different releases have different blast radii.</strong> Once you accept that, the design stops being “pick the one right strategy” and becomes “assemble a layered system where each tool covers the radius it is best at.” A combination almost always wins: trunk-based development plus feature flags plus canary for small daily releases, and blue-green for the big drops where instant, explainable rollback is the whole point. <a href="https://dorokhovich.com/blog/release-orchestration/deployment-strategies?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/deployment-strategies">A detailed treatment of choosing and combining strategies</a> walks the full model; the compressed version is four tools with four jobs.</p><p><strong>Blue-green</strong> keeps two identical production environments — the pattern <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Martin Fowler documented in 2010</a>. Blue serves users; green is prepared calmly for the next release; when you are confident you flip traffic in one router change, and if something is wrong you flip back just as fast. Its virtue is that rollback is instant and trivial to explain to a business stakeholder — a fintech spotting a EUR-payments bug and flipping back in about two minutes, with a tiny fraction of operations affected, is the kind of story that sells it to leadership.</p><p><strong>Canary</strong> rolls a change out to a small subset first — what <a href="https://martinfowler.com/bliki/CanaryRelease.html">Danilo Sato describes on Fowler’s site</a> — starting at 1–5% of traffic, watching errors, latency, resource use, and one business metric, then ramping 5 → 10 → 25 → 50 → 100% and comparing at each step. Its virtue is a naturally small blast radius: a bad version is seen by few before the system reacts.</p><p><strong>Rolling updates</strong> are the quiet workhorse for stateless services — the <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">default strategy in the Kubernetes docs</a>, where maxSurge and maxUnavailable govern how aggressively pods are replaced while the service stays available throughout.</p><p><strong>Feature flags</strong> are the one that changes cadence most, because they separate two actions people wrongly treat as one: <em>deploying code</em> and <em>releasing a feature</em>. Ship dark, enable for staff, then a percentage, then everyone, and kill the switch without redeploying — the technique <a href="https://martinfowler.com/articles/feature-toggles.html">Pete Hodgson documents as feature toggles</a>, which pairs naturally with <a href="https://trunkbaseddevelopment.com/">trunk-based development</a>.</p><p>The unifying move that ends the 2 AM arguments is writing <strong>stop-criteria before the release</strong>, as executable rules rather than live judgement:</p><pre>IF error_rate_canary &gt; error_rate_baseline * 1.5 THEN rollback<br>IF latency_p99_canary &gt; latency_p99_baseline * 1.3 THEN rollback<br>IF conversion_rate_canary &lt; baseline * 0.95 THEN rollback</pre><p>With those in place, rollback stops being a debate and becomes a reflex — the system pulls the new version before a human opens the dashboard. Netflix’s canonical case is exactly this: a gradual ramp where SmartTV performance degraded at one step and triggered an automatic rollback in roughly two minutes.</p><h3>Trade-offs</h3><p>No strategy is free, and the honest way to reason is per blast radius:</p><ul><li><strong>Blue-green</strong> — Best for: Big, risky drops (payments rewrite, framework upgrade); Rollback: Instant single flip; Real cost: ~2× resources during release; DB and session state are hard; watch DNS TTL</li><li><strong>Canary</strong> — Best for: Everyday releases; Rollback: Automatic on stop-criteria; Real cost: Needs percentage routing; slow to reach confidence on low traffic; pin users by ID</li><li><strong>Rolling update</strong> — Best for: Stateless services on Kubernetes; Rollback: Gradual, batch by batch; Real cost: Old and new pods coexist — API contracts must stay compatible across versions</li><li><strong>Feature flags</strong> — Best for: Decoupling deploy from release; Rollback: Kill switch, no redeploy; Real cost: Forgotten flags rot; every flag needs an owner and an expiry</li></ul><p>Two constraints deserve emphasis because they are where teams get burned. First, the rolling-update coexistence problem: because old and new pods serve traffic simultaneously, forward-compatible database and API changes are not optional — deployment strategy and schema evolution are two halves of one discipline. Second, a flag with no owner is technical debt with a fuse on it; the discipline that makes flags safe is an owner and an expiry date on every one, with dead code removed once the feature is universal.</p><p>And one distinction worth nailing down, because conflating them wastes time: <strong>canary is not A/B testing.</strong> Canary asks “is the new version not <em>worse</em>?” and optimizes for safety. A/B testing asks “which version is <em>better</em>?” and optimizes for a product decision — split traffic, pin each user to a variant, and wait a week or two to smooth out weekday and seasonality effects before running the stats. Use canary to protect releases and A/B tests to choose product directions; using one for the other’s job produces noisy, untrustworthy conclusions.</p><h3>How to adopt</h3><p>The mistake is adopting all four at once. Sequence by where the pain is loudest.</p><ol><li><strong>Write the stop-criteria before the next release and make them executable.</strong> This is the single highest-leverage move; it converts rollback from a live argument into an automatic reaction and costs almost nothing.</li><li><strong>Add feature flags to decouple deploy from release.</strong> Ship dark, enable gradually, keep a kill switch. Put an owner and expiry on every flag from day one so the debt never accumulates.</li><li><strong>Make rolling updates safe on Kubernetes</strong> with small batches, health probes, and a pause between batches:</li></ol><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>spec:<br>  replicas: 10<br>  strategy:<br>    type: RollingUpdate<br>    rollingUpdate:<br>      maxUnavailable: 1<br>      maxSurge: 2<br>  template:<br>    spec:<br>      containers:<br>      - name: my-app<br>        image: my-app:v2<br>        readinessProbe:<br>          httpGet: { path: /health/ready, port: 8080 }<br>        livenessProbe:<br>          httpGet: { path: /health, port: 8080 }</pre><p>Batches too large increase risk and make rollback painful; skipping the pause means you notice problems too late.</p><ol><li><strong>Reserve blue-green for the big drops</strong> where instant, explainable rollback justifies double the resources. Solve the session problem with a shared session store and graceful connection draining, and watch DNS TTL, which can delay a switch you thought was instant.</li><li><strong>Automate the canary gates</strong> rather than hand-rolling them. <a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts encodes exactly these gates</a> — setWeight and pause steps plus automated analysis that advances the traffic weight only while metrics stay healthy.</li></ol><p>Put together, a normal day looks like this: a change merges to trunk, deploys dark behind a flag, and enables as a canary to 5% while the automated gates watch; if they stay green it ramps to 100% over an hour, and if not the flag flips off while the code stays deployed but inert. A large, risky drop reaches for blue-green instead. Each tool covers a different blast radius, and together they mean no release requires heroics.</p><h3>Where this goes next</h3><p>The direction of travel is toward <strong>progressive delivery as a fully automated control loop</strong> — the stop-criteria you write by hand today become the analysis templates your rollout controller evaluates on its own, promoting or reverting without a human in the path. Tools like Argo Rollouts and Flagger are early forms of this; the interesting frontier is richer signals feeding the gate, including model-based anomaly detection that can catch a regression no static threshold would.</p><p>The deeper point is that all of this rests on the same foundation: a legible definition of “healthy” and forward-compatible changes underneath the traffic shifting. Teams that have captured what “not worse” means as executable criteria are the ones who will safely hand more of the promote/rollback decision to automation — and eventually to AI-assisted release agents that reason over the same signals. The layered strategy is not just what ends the 2 AM rollbacks today; it is the substrate the automated release systems of the next few years will need in order to be trusted at all.</p><h3>Sources &amp; further reading</h3><ul><li>Martin Fowler, <a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">BlueGreenDeployment</a> and Danilo Sato, <a href="https://martinfowler.com/bliki/CanaryRelease.html">CanaryRelease</a> — the canonical write-ups of both patterns.</li><li>Kubernetes docs, <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployments</a> — the RollingUpdate strategy with maxSurge/maxUnavailable.</li><li><a href="https://argo-rollouts.readthedocs.io/en/stable/features/canary/">Argo Rollouts — Canary strategy</a> — automated canary analysis and traffic-weight gates.</li><li>Pete Hodgson, <a href="https://martinfowler.com/articles/feature-toggles.html">Feature Toggles</a> and <a href="https://trunkbaseddevelopment.com/">Trunk-Based Development</a> — decoupling deploy from release.</li><li>A longer reference treatment of <a href="https://dorokhovich.com/blog/release-orchestration/deployment-strategies?utm_source=medium&amp;utm_medium=article&amp;utm_campaign=success-story&amp;utm_content=release-orchestration/deployment-strategies">choosing and combining deployment strategies</a> — the sequences and stop-criteria behind this framing.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45045d5eebf3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>