<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Richard King on Medium]]></title>
        <description><![CDATA[Stories by Richard King on Medium]]></description>
        <link>https://medium.com/@dx_13752?source=rss-5bf36bb309fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q-gQsHx1kvHFhlh-lD_u6A.png</url>
            <title>Stories by Richard King on Medium</title>
            <link>https://medium.com/@dx_13752?source=rss-5bf36bb309fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:48:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dx_13752/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Private Git Asset Hub: High-Performance Gitea Deployment & Repository Persistence]]></title>
            <link>https://medium.com/@dx_13752/private-git-asset-hub-high-performance-gitea-deployment-repository-persistence-8727fe1b2e32?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/8727fe1b2e32</guid>
            <category><![CDATA[homelab-sre]]></category>
            <category><![CDATA[gitea]]></category>
            <category><![CDATA[openssh]]></category>
            <category><![CDATA[linux-security]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Sat, 21 Nov 2026 08:57:37 GMT</pubDate>
            <atom:updated>2026-11-21T08:57:37.945Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqM2IlaDnC0tuJllh_gw8A.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Self-Hosters</li><li><strong>Core Topics</strong>: Gitea, Docker Compose, OpenSSH, Linux Security, HomeLab SRE</li></ul><h3>1. Why Standard Self-Hosted Gitea Tutorials Fail in a Hardened SRE Environment</h3><p>When GitHub experiences an outage or Actions queue delay, a sovereign HomeLab or One-Person Company (OPC) cannot afford to lose access to its Infrastructure-as-Code (IaC) manifests and automation scripts. Deploying a private Git hub on our n100-staging (192.168.3.208) storage tier provides sub-second LAN repository access (470.14 ms full clone from pi3b).</p><p>Yet copying a standard Gitea docker-compose.yml onto a least-privilege Linux host introduces four operational vulnerabilities:</p><ol><li><strong>Docker Port Publishing Bypasses Host Firewalls via </strong><strong>PREROUTING </strong><strong>DNAT</strong>: Binding -p 3000:3000 or -p 2222:22 on 0.0.0.0 causes dockerd to inject DNAT rules before the host&#39;s INPUT firewall chain, exposing unencrypted HTTP on the LAN and colliding with Chapter 003&#39;s staging-origin.service on port :3000.</li><li><strong>Host Port 22 </strong><strong>AuthorizedKeysCommand Passthrough Violates Least Privilege</strong>: Forwarding port :22 from the host&#39;s sshd via docker exec -i requires granting a network-facing git account access to /var/run/docker.sock (root equivalence), breaking our Option A privilege model from Chapters 009 and 013.</li><li><strong>Plaintext Configuration Secrets Written to </strong><strong>app.ini and Exposed in </strong><strong>docker inspect</strong>: Gitea writes SECRET_KEY and INTERNAL_TOKEN back into app.ini on disk, while passing secrets via environment: exposes them in Config.Env and fails Chapter 013&#39;s /usr/local/sbin/sre-compose-ctl AST firewall.</li><li><strong>Default Bleve Issue Indexing &amp; Unbounded Go Garbage Collection</strong>: Gitea defaults to on-disk/mmap Bleve indexing for issues (ISSUE_INDEXER_TYPE = bleve, while REPO_INDEXER_ENABLED defaults to false) and GOGC=100; consolidating issue indexing into SQLite (ISSUE_INDEXER_TYPE = db) and setting GOGC=25 + GOMEMLIMIT=64MiB keeps resident anonymous memory under our &lt; 100 MiB budget on an 8 GB Intel N100 host.</li></ol><h3>2. Dual-Plane SSH Isolation &amp; SO_PEERCRED Unix-Socket tmpfs Secret Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxtJyvdW7tEKSOQ_dDqE4w.png" /></figure><p>To keep Chapter 013’s /usr/local/sbin/sre-compose-ctl AST firewall completely unmodified (127.0.0.1-only Docker port bindings, read_only: true, zero environment: keys, and /data/volumes/storage bind sandboxing), we combine <strong>gitea/gitea:1.22.6-rootless (</strong><strong>UID 1000:1000)</strong> with an unprivileged pure-Python daemon (<strong>sre-gitea-guardian.py</strong>) running as king (UID 1000):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2ofv49p3UxTz5Xaj4M2WQ.png" /></figure><h3>3. Core Mechanism: SO_PEERCRED In-Memory Configuration Rendering</h3><p>Inside compose-gitea.yaml, /etc/gitea is mounted as an in-container 16 MiB tmpfs (uid=1000,gid=1000,mode=0700). At container boot, the startup script fetches the rendered app.ini over /var/lib/gitea/ipc/secret.sock—an AF_UNIX socket inode residing inside /data/volumes/storage/gitea/ipc/ that consumes zero disk blocks while passing sre-compose-ctl&#39;s os.path.realpath sandbox check.</p><p>On the host, sre-gitea-guardian.py (lines 229–248 shown below) loads the deterministically derived secrets from /dev/shm/sre-secrets/gitea/runtime.env (tmpfs, mode 0400), renders the template purely in memory, and verifies the caller&#39;s effective UID via the Linux kernel&#39;s SO_PEERCRED socket option:</p><pre>def render_app_ini_in_memory() -&gt; bytes:<br>    sec = load_secrets()<br>    assert_not_symlink(APP_INI_TEMPLATE)<br>    tpl = APP_INI_TEMPLATE.read_text(encoding=&quot;ascii&quot;)<br>    rendered = (<br>        tpl.replace(&quot;__SRE_SECRET_KEY__&quot;, sec[&quot;SECRET_KEY&quot;])<br>        .replace(&quot;__SRE_INTERNAL_TOKEN__&quot;, sec[&quot;INTERNAL_TOKEN&quot;])<br>        .replace(&quot;__SRE_JWT_SECRET__&quot;, sec[&quot;JWT_SECRET&quot;])<br>        .replace(&quot;__SRE_LFS_JWT_SECRET__&quot;, sec[&quot;LFS_JWT_SECRET&quot;])<br>    )<br>    return rendered.encode(&quot;ascii&quot;)<br></pre><pre>def verify_peer_uid(conn: socket.socket) -&gt; int:<br>    ucred = conn.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, struct.calcsize(&quot;3i&quot;))<br>    _pid, uid, _gid = struct.unpack(&quot;3i&quot;, ucred)<br>    if uid != os.getuid():<br>        raise PermissionError(f&quot;Rejected Unix socket peer uid={uid} (expected {os.getuid()})&quot;)<br>    return uid</pre><h3>4. Four Production Traps Solved During Live Staging</h3><ol><li><strong>Deterministic </strong><strong>/dev/shm Secret Reconstruction Across Host Reboots</strong>: Because /dev/shm is volatile tmpfs, generating fresh random SECRET_KEY and ADMIN_PASS values on reboot locks out sre-admin in the existing SQLite DB. Deriving secrets deterministically via HMAC-SHA256 from a mode-0400 operator seed file (/home/king/.local/libexec/.sre-gitea-seed.key, outside /data) salted with /etc/machine-id + UID 1000 (including an HS256-signed JWT INTERNAL_TOKEN with fixed nbf: 1700000000) and gating health on authenticated /api/v1/version (admin_auth_ok: true) prevents reboot drift even across container stop, /dev/shm wipe, and container cold start.</li><li><strong>read_only: true vs. Git Pre-Receive Hook Temp Directories</strong>: Rootless Gitea unpacks incoming Git objects in /tmp. Mounting /tmp:rw,noexec,nosuid,size=64m,uid=1000,gid=1000,mode=1777 and setting GITEA_TEMP=/tmp/gitea prevents EROFS errors on git push.</li><li><strong>SSH Host Key Continuity Without Redundant On-Disk OAuth2 Private Keys</strong>: Disabling [oauth2] ENABLED = false suppresses data/jwt/private.pem (oauth2_jwt_key_absent: true) while persisting /var/lib/gitea/data/ssh/gitea.rsa (0600) inside /data/volumes/storage/gitea to keep the :2222 Go SSH host key invariant across container rebuilds (ssh_host_key_persisted: true).</li><li><strong>RssAnon (</strong><strong>63.12 MiB) vs. </strong><strong>VmRSS (</strong><strong>140.02 MiB)</strong>: In /proc/&lt;pid&gt;/status, 76.89 MiB of VmRSS is reclaimable RssFile binary text mapping, while actual anonymous heap memory RssAnon is only <strong>63.12 MiB</strong> (63.12 MiB during audit), well under our &lt; 100 MiB budget.</li></ol><h3>5. Live Staging Benchmark, Container Destruction &amp; tmpfs Wipe Drill</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PXajiWXYJ6_Uh1MkDaJvtg.png" /></figure><p>Across our 3-node Staging rig (n100-staging .208, pi3b .202, i5-staging .209), the verification suite completed in 155,601 ms:</p><ul><li><strong>Deployment &amp; Audit Speed</strong>: Cold deploy in 20,593 ms (13,496 ms script), warm idempotent re-run in 15,109 ms (2,989 ms script), and 10-SLO white-box audit in 1,949 ms (10/10 PASS).</li><li><strong>Out-of-Band </strong><strong>pi3b Git-over-SSH (</strong><strong>:2222), Container Recreate &amp; </strong><strong>tmpfs Wipe Drill</strong>: Initial git clone in 470.14 ms, commit and git push in 969.14 ms (commit 922661f4b655d2aefadcfd039c188b0da337482e), full container teardown and recreation in 6,452 ms, simulated host reboot (container stopped before /dev/shm wipe, deterministic secret reconstruction with runtime.env SHA-256 0a54457c... 100% identical, and container cold start with admin_auth_ok: true), and post-recreation fresh git clone in 489.95 ms with 100% matching commit SHA, file SHA-256 (9d1b05f3...b941638e), and SSH host key.</li><li><strong>Adversarial Guardrails</strong>: 12/12 live Compose AST rejections (adv_dir=0, tmp_cnt=0) and 48/48 Exit Code 2 production subnet (192.168.3.2–10) safety interlocks (including quoted &quot;Include&quot; and &quot;Match&quot; keyword checks) passed with 0 unexpected SSH handshakes.</li></ul><h3>6. What’s Next</h3><p>With our private Git repository hub running in the storage tier on n100-staging, the next chapter adds our issue and project tracking backbone:</p><p><strong>Next Chapter: Enterprise Engineering Backlog: Redmine Issue Tracking &amp; Automated Task Workflows</strong></p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete verified disaster recovery runbook, production-hardened deployment scripts (compose-gitea.yaml, sre-gitea-guardian.py, gitea-deploy.sh, gitea-hub-audit.sh, bootstrap-ch014.sh), and empirical staging benchmark suites are available to members of the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em>: </em><a href="https://ubinity.substack.com/p/private-git-asset-hub-high-performance"><em>HomeLab SRE Cookbook Chapter 014 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8727fe1b2e32" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Container Runtime Standardization: Docker Compose Stack Tiering & Network Driver Selection]]></title>
            <link>https://medium.com/@dx_13752/container-runtime-standardization-docker-compose-stack-tiering-network-driver-selection-69d99936d0a3?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/69d99936d0a3</guid>
            <category><![CDATA[linux-networking]]></category>
            <category><![CDATA[homelab-sre]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[macvlan]]></category>
            <category><![CDATA[least-privilege]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 11:50:12 GMT</pubDate>
            <atom:updated>2026-11-20T11:50:12.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J0BeMjYrgk_lUxaljqsIfg.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Self-Hosters</li><li><strong>Core Topics</strong>: Docker Compose, Linux Networking, Macvlan, Least Privilege, HomeLab SRE</li></ul><h3>1. Why Monolithic Compose Files and Containerized Sentinels Fail</h3><p>When teams scale self-hosted infrastructure beyond a single weekend container, three runtime anti-patterns show up immediately:</p><ol><li><strong>Blast-Radius Coupling in Monolithic </strong><strong>docker-compose.yml Files</strong>: Combining PostgreSQL, Git hosting, automation engines, and reverse proxies into one Compose project turns every routine configuration change into a cascading restart across unrelated workloads—and leaves stateful databases sitting on the same default bridge network as internet-facing edge containers.</li><li><strong>Pushing Docker Stacks onto Low-Memory ARM Sentinels</strong>: In our 3-node Golden Trio topology, the sentinel (pi4b in production, bare-metal pi3b with 905 MB RAM and 0 swap in Staging) runs host-native systemd services (cloudflared, dnsmasq, out-of-band probing). Scheduling container stacks onto the sentinel exhausts finite RAM and accelerates MicroSD write wear.</li><li><strong>Breaking Host Hardening with Wildcard </strong><strong>sudo docker</strong>: Stripping the docker group from human operators means nothing if an automation account receives NOPASSWD: /usr/bin/docker *. Any single YAML containing volumes: [&quot;/:/host&quot;] or privileged: true escalates directly to host root.</li></ol><h3>2. Three-Node Role Isolation &amp; 4-Tier /opt/docker-stacks/ Layout</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJ3dBFZPksb3iJW3duMrpw.png" /></figure><p>We enforce a strict hardware split: <strong>n100-staging (</strong><strong>192.168.3.208)</strong> and <strong>i5-staging (</strong><strong>192.168.3.209)</strong> (Ubuntu 26.04.1 LTS x86_64 KVM VMs, Docker 29.1.3, Compose 2.40.3) are the sole Compose-managed nodes (both deploying all 4 verification tiers equally), while <strong>pi3b (</strong><strong>192.168.3.202)</strong> remains strictly host-native (ctl_absent=1, dc_v1_absent=1, dc_v2_absent=1, compose_files=0; Ch 011 /usr/bin/docker sits idle with zero Compose stacks) and serves as our out-of-band LAN verification probe.</p><p>On both x86_64 nodes, stacks live under /opt/docker-stacks/&lt;tier&gt;/&lt;stack&gt;/compose.yaml across four fault-isolation tiers:</p><ul><li><strong>infrastructure</strong>: Ingress proxies and cross-stack gateways (sre-internal-net, 127.0.0.1:18080).</li><li><strong>storage</strong>: Stateful database and object storage stacks (/data/volumes/storage, internal: true backend bridge isolation).</li><li><strong>monitoring</strong>: Reserved observability probes and exporters (/data/volumes/monitoring).</li><li><strong>compute</strong>: Stateless workers (/data/volumes/compute) and strictly allowlisted host metrics exporters.</li></ul><p>Every manifest drops the deprecated top-level version: key, keeps zero plaintext .env files on disk (enforcing the prerequisite contract for <a href="https://ubinity.substack.com/p/credentials-vault-and-secret-governance">Chapter 012</a> /dev/shm tmpfs injection), and enforces read_only: true, no-new-privileges:true, healthcheck, and json-file log rotation (10m × 3).</p><h3>3. Option A Control Plane: Compose JSON AST Policy Firewall</h3><p>Instead of opening raw docker sockets, sre-orchestrator invokes /usr/local/sbin/sre-compose-ctl via /etc/sudoers.d/99-sre-compose (sudo-rs 0.2.13). Before any stack is staged or started, sre-compose-ctl expands docker compose -f &lt;file&gt; config --format json, applies deny-by-default ALLOWED_TOP_KEYS (ALLOWED_NET_KEYS blocking non-empty ipam, non-bridge drivers, and external networks outside {sre-internal-net, sre-macvlan-test}) and ALLOWED_SVC_KEYS allowlists (lines 234–313), and audits each service in Python (verbatim lines 316–348):</p><pre>if svc.get(&quot;privileged&quot;) or svc.get(&quot;cap_add&quot;) or svc.get(&quot;devices&quot;):<br>        print(f&quot;[SECURITY ERROR] Service &#39;{svc_name}&#39; requests privileged/cap_add/devices (forbidden).&quot;, file=sys.stderr)<br>        sys.exit(1)<br>    for ns_key in (&quot;pid&quot;, &quot;ipc&quot;, &quot;userns_mode&quot;):<br>        if ns_key in svc and svc.get(ns_key) is not None:<br>            print(f&quot;[SECURITY ERROR] Service &#39;{svc_name}&#39; sets forbidden namespace key &#39;{ns_key}&#39;.&quot;, file=sys.stderr)<br>            sys.exit(1)</pre><pre>    # Require read_only: true on every container<br>    if svc.get(&quot;read_only&quot;) is not True:<br>        print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; must set read_only=true.&quot;, file=sys.stderr)<br>        sys.exit(1)</pre><pre>    # Require healthcheck with non-empty test (forbid NONE, disable=true, and zero interval)<br>    hc = svc.get(&quot;healthcheck&quot;)<br>    if not isinstance(hc, dict) or not hc.get(&quot;test&quot;):<br>        print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; must define a non-empty healthcheck.&quot;, file=sys.stderr)<br>        sys.exit(1)<br>    if hc.get(&quot;disable&quot;):<br>        print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; healthcheck.disable cannot be true.&quot;, file=sys.stderr)<br>        sys.exit(1)<br>    hc_interval = str(hc.get(&quot;interval&quot;) or &quot;&quot;).strip().lower()<br>    if hc_interval in (&quot;0&quot;, &quot;0s&quot;, &quot;0ns&quot;, &quot;0us&quot;, &quot;0ms&quot;, &quot;0m&quot;, &quot;0h&quot;):<br>        print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; healthcheck.interval cannot be 0.&quot;, file=sys.stderr)<br>        sys.exit(1)<br>    hc_test = hc.get(&quot;test&quot;)<br>    if isinstance(hc_test, list):<br>        if not hc_test or str(hc_test[0]).strip().upper() == &quot;NONE&quot;:<br>            print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; healthcheck.test cannot be NONE.&quot;, file=sys.stderr)<br>            sys.exit(1)<br>    elif not isinstance(hc_test, str) or hc_test.strip().upper() == &quot;NONE&quot;:<br>        print(f&quot;[POLICY ERROR] Service &#39;{svc_name}&#39; healthcheck.test invalid or NONE.&quot;, file=sys.stderr)<br>        sys.exit(1)</pre><p>Lines 350–442 also clean up failed temp stage files via an EXIT trap, enforce no-new-privileges:true (blocking unconfined), block any bind mount whose os.path.realpath resolves outside /data/volumes/&lt;tier&gt;, reject any port not bound strictly to 127.0.0.1 or invalid logging config, and restrict network_mode exclusively to (tier == &quot;compute&quot;, stack == &quot;compute-worker&quot;, svc_name == &quot;compute-host-exporter&quot;).</p><h3>4. Network Driver Selection &amp; Empirical Staging Matrix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cZzzIOniHQ7QZFneHy0qIQ.png" /></figure><p>Live verification across n100-staging, i5-staging, and out-of-band pi3b confirmed four distinct operational boundaries:</p><ul><li><strong>Custom Bridge (</strong><strong>sre-internal-net)</strong>: Embedded DNS (127.0.0.11) resolves storage-api and infra-gateway across stacks on the same host, while all 8 loopback-pinned ports (18080, 18081, 18082, 19100 across both nodes) were refused when probed from pi3b.</li><li><strong>Internal Bridge (</strong><strong>backend_isolated, </strong><strong>internal: true)</strong>: Strips default gateway and NAT egress; storage-db-sim cannot reach 192.168.3.1 and infra-gateway cannot touch the DB directly, while dual-homed storage-api cleanly bridges requests (db-ok).</li><li><strong>Host Mode (</strong><strong>network_mode: host)</strong>: Reserved for zero-NAT metric exporters (127.0.0.1:19100), returning sre_worker_up 1 locally while refusing external LAN probes from pi3b.</li><li><strong>Macvlan L2 (</strong><strong>sre-macvlan-test on </strong><strong>ens18, </strong><strong>192.168.3.224/28)</strong>: Assigns dedicated LAN IPs (.225 on n100-staging, .226 on i5-staging, clear of 192.168.3.2–10). Kernel host-to-macvlan isolation blocks each host from reaching its own macvlan IP (host_to_macvlan=blocked), whereas external pi3b passes 5/5 macvlan-ok probes at 4.73 ms (n100) and 4.65 ms (i5) HTTP median latency (0.639 ms / 0.542 ms ICMP).</li></ul><h3>5. Staging Execution Telemetry</h3><p>Across our full automated verification (236,453 ms total):</p><ul><li>Cold convergence across both x86_64 Docker nodes completed in <strong>91,858 ms</strong>, and idempotent warm re-convergence passed in <strong>38,635 ms</strong>.</li><li>Independent compose-standards-audit.sh --json passed 10/10 SLO assertions in <strong>23,952 ms</strong>, 16/16 live AST adversarial rejections left zero residue (adv_dir=0|tmp_cnt=0), and all 27/27 adversarial Exit Code 2 production interlock checks (12 vectors × 2 scripts + 3 hermetic ProxyJump/ProxyCommand self-tests) aborted cleanly.</li></ul><h3>6. What’s Next</h3><p>With our 4-tier stack hierarchy and network boundaries locked down, the next chapter deploys our first stateful service in the storage tier:</p><p>👉 <strong>Private Git Asset Hub: High-Performance Gitea Deployment &amp; Repository Persistence</strong></p><p>We bind /data/volumes/storage/gitea, isolate Git SSH onto port 2222, and inject runtime secrets strictly through memory.</p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete open-source deliverable bundle — including sre-compose-ctl, sudoers-sre-compose, the four hardened compose-*.yaml manifests, stack-deploy.sh, compose-standards-audit.sh, and bootstrap-ch013.sh—is freely available in the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive &amp; get the companion scripts</em></strong><em>: </em><a href="https://ubinity.substack.com/p/container-runtime-standardization"><em>HomeLab SRE Cookbook Chapter 013 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69d99936d0a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Credentials Vault & Secret Governance: Isolation, Vault, and Environment Hardening]]></title>
            <link>https://medium.com/@dx_13752/credentials-vault-secret-governance-isolation-vault-and-environment-hardening-e72081b4cae4?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e72081b4cae4</guid>
            <category><![CDATA[homelab-sre]]></category>
            <category><![CDATA[management-secrets]]></category>
            <category><![CDATA[pos]]></category>
            <category><![CDATA[zero-disk]]></category>
            <category><![CDATA[age]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 12:07:13 GMT</pubDate>
            <atom:updated>2026-11-19T12:07:13.407Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HxUt8RPMCxwbvRXsbtq7Og.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Homelab SREs, DevOps engineers</li><li><strong>Core Topics</strong>: SOPS, Age, Secret Management, Zero-Disk, Homelab SRE</li></ul><h3>1. The Hidden Cost of Plaintext .env Convenience</h3><p>In small clusters and self-hosted homelab infrastructure, most credential incidents do not begin with zero-day exploits against kernel subsystems. They start with three ordinary habits that accumulate quietly once PostgreSQL, Gitea, MinIO, Redmine, n8n, Dify, and alerting bots share the same hardware fleet:</p><ol><li><strong>The Shared Master </strong><strong>.env Trap</strong>: Dropping 30+ database passwords, object-storage root keys, and webhook tokens into a single /opt/docker-stacks/.env file readable by unrelated services.</li><li><strong>Late-Night Git Drift</strong>: Running git add -A &amp;&amp; git commit during an urgent fix and accidentally pushing plaintext credentials or private SSH/Age keys into repository history.</li><li><strong>Orphaned </strong><strong>scp Artifacts</strong>: Pushing secrets to remote /tmp/ folders over scp, where an interrupted deployment leaves plaintext credentials sitting on NVMe or microSD flash indefinitely.</li></ol><p>When an internet-facing web container leaks its process environment through a debugging endpoint or vulnerability, a shared .env layout turns one single-service bug into total fleet compromise. An attacker immediately gains root credentials for databases, backup archives, and object storage at once.</p><h3>2. Why a 33.4 W Cluster Avoids Heavyweight Vault Servers</h3><p>Deploying a stateful HashiCorp Vault server on a low-power three-node cluster (modeled on Chapter 010’s ~33.4 W idle Golden Trio topology) often introduces worse failure modes than the problem it solves:</p><ul><li><strong>Power-Loss Unseal Deadlocks</strong>: After a brief utility flicker, rebooted ARM64 control-plane nodes stall waiting for manual operator unsealing before downstream databases or workflow engines can receive credentials.</li><li><strong>Control-Plane Memory Overhead</strong>: Dedicating a stateful daemon and persistent encrypted storage volume just to distribute 30 mostly static infrastructure schema keys (31 SOPS ciphertext entries including sops_mac) wastes precious headroom on resource-constrained single-board computers.</li><li><strong>Fragile Runtime Coupling</strong>: When every stack startup depends on an external HTTP secret API remaining healthy during early boot, simple maintenance restarts become circular dependency puzzles.</li></ul><p>Instead of a resident daemon, Chapter 012 isolates secrets cleanly across three cryptographic and kernel boundaries built purely from <strong>SOPS 3.13.3</strong>, <strong>Age v1.3.2</strong>, and Linux <strong>tmpfs</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P_ujYE-wECZBrKS0DchYiA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBpwWe3cW7Zldou7VQYDDA.png" /></figure><p>Under stack-secret-policy.conf, each of the 10 infrastructure stacks sees only the keys required to do its job: for example, dify receives 3 keys (DIFY_SECRET_KEY, DIFY_DB_PASSWORD, CODE_SERVER_PASSWORD), while postgres (4), n8n (4), backup (1: KOPIA_PASSWORD), and alerting (3) stay strictly compartmentalized. Even if an application container is compromised, <strong>90.0%</strong> of the cluster&#39;s secret surface remains invisible.</p><h3>3. Hardening Boundaries in Code: Four Fail-Closed Gates</h3><h3>3.1 Verified tmpfs Allocation &amp; Exit Zero-Shredding (Exit Code 3)</h3><p>Never trust the string /dev/shm blindly. Misconfigured containers or an accidental SRE_VAULT_SHM_DIR override can silently redirect temporary secrets onto persistent ext4 flash storage. Before generating or decrypting a single secret, vault-secret-injector.sh queries findmnt to confirm the backing filesystem is truly volatile (tmpfs or ramfs), aborting with <strong>Exit Code 3</strong> otherwise:</p><pre>create_ram_workspace() {<br>    local base_shm=&quot;${SRE_VAULT_SHM_DIR:-/dev/shm}&quot;<br>    if [[ ! -d &quot;$base_shm&quot; ]]; then<br>        log_error &quot;Volatile tmpfs base directory missing: ${base_shm}&quot;<br>        exit 3<br>    fi</pre><pre>    local fstype<br>    fstype=&quot;$(findmnt -n -o FSTYPE -T &quot;$base_shm&quot; 2&gt;/dev/null || true)&quot;<br>    if [[ &quot;$fstype&quot; != &quot;tmpfs&quot; &amp;&amp; &quot;$fstype&quot; != &quot;ramfs&quot; ]]; then<br>        log_error &quot;Refusing to store plaintext secrets on non-volatile filesystem &#39;${fstype}&#39; at ${base_shm}!&quot;<br>        exit 3<br>    fi</pre><pre>    RAM_DIR=&quot;$(mktemp -d &quot;${base_shm}/sre-vault.XXXXXXXXXX&quot;)&quot;<br>    chmod 0700 &quot;$RAM_DIR&quot;<br>}</pre><p>An EXIT INT TERM HUP QUIT trap also zeroes every slice byte via dd if=/dev/zero before unlinking. Even if an operator cancels a rollout mid-flight with Ctrl+C, no lingering temporary slice survives in /dev/shm.</p><h3>3.2 Idempotent Vault Init &amp; Format-Preserving Rotation (Exit Code 4)</h3><p>Regenerating a live vault accidentally can lock out every database and backup archive in the fleet. Repeating --init-vault without an explicit --force-reinit flag refuses immediately with <strong>Exit Code 4</strong>. Conversely, rotating a single compromised key (--rotate &lt;KEY&gt;) rejects non-secret identity fields (POSTGRES_USER, SLACK_CHANNEL) and malformed regex patterns (N8N_.*), inspects the key naming semantics to preserve expected formats—such as 64-character hex tokens for N8N_ENCRYPTION_KEY or 24-character alphanumeric strings for N8N_PASSWORD—creates a collision-free encrypted .bak.&lt;timestamp&gt;.&lt;rand&gt; snapshot via mktemp, verifies ciphertext diff, and logs only ciphertext SHA-256 transitions rather than plaintext hashes.</p><h3>3.3 SSH Stdin Streaming &amp; Honest Container Runtime Disclosure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VqcXClR45RIMBB7plbJN_A.png" /></figure><p>Cross-node deployment bans scp and rsync completely. vault-secret-injector.sh --inject-remote &lt;stack&gt; &lt;node&gt; pipes the sliced RAM environment directly into remote /dev/shm over SSH standard input, audits /proc/&lt;pid&gt;/cmdline so credentials never appear in ps output, and wipes both local and remote RAM buffers on exit.</p><p>In our non-privileged Staging probes, --runtime auto runs the verified proc sandbox and leaves zero plaintext on disk across all three hosts. When operators target Docker Engine directly via --runtime docker, be aware of an important Docker architectural boundary: passing --env-file makes dockerd serialize environment key-values into /var/lib/docker/containers/&lt;id&gt;/config.v2.json (Config.Env) until the container is recreated. Eliminating Config.Env metadata persistence entirely requires bind-mounting tmpfs-backed secret files inside the container—a pattern covered in future security-hardening chapters.</p><h3>3.4 Fail-Closed Production Interlock &amp; Staged-Index Pre-Commit Guard</h3><p>To ensure no human typo or alias touches production (192.168.3.2–192.168.3.10), every operation resolves the target via direct IP canonicalization (catching zero-padded inputs like 192.168.003.5, octal 192.168.3.011, and compressed/uncompressed IPv4-mapped IPv6 ::ffff:192.168.3.5 / 0:0:0:0:0:ffff:192.168.3.5), leading-dash alias rejection, user@host stripping, unconditional recursive ssh -G (including case-insensitive Hostname, ProxyJump, and ProxyCommand aliases such as ssh -F inc -W %h:%p pve), and getent ahostsv4, immediately aborting with <strong>Exit Code 2</strong> if any resolved IP enters the production band.</p><p>On the Git side, pre-commit-secret-guard.sh scans staged Git index objects (git show &quot;:${path}&quot;) with --diff-filter=ACMRT rather than trusting working-tree files that could be swapped after git add or renamed via git mv. It rejects forbidden filenames (credentials.vault.env, keys.txt), .env.* variants (.env.local, .env.bak, .env.production), OpenSSH/RSA/EC/PKCS#8 private key headers, Slack tokens, and any unencrypted line appended to *.sops.env (<strong>Exit Code 1</strong>).</p><h3>4. Measured Staging Telemetry (ARM64 + x86_64)</h3><p>Running all 9 verification stages across our live three-node Staging cluster (pi3b 192.168.3.202, n100-staging 192.168.3.208, and i5-staging 192.168.3.209) produced verified bare-metal telemetry rather than estimates:</p><ul><li><strong>Full Vault Encryption (30 schema keys / 31 ciphertext entries)</strong>: Encrypted via SOPS + Age in 801 ms on ARM64 pi3b (0 plaintext keys); repeat --init-vault blocked cleanly (exit_code=4).</li><li><strong>Least-Privilege Slicing &amp; Local Process Inject</strong>: postgres 4-key slice took 201 ms in /dev/shm (139 B); alerting local inject completed via proc (CMDLINE=&#39;sleep 15&#39;, 0 secret bytes exposed) in 638 ms.</li><li><strong>N=20 RAM Decrypt &amp; Slice Benchmark (</strong><strong>pi3b)</strong>: min=484.61 ms | median=487.62 ms | p95=491.14 ms | max=520.65 ms.</li><li><strong>Cross-Node SSH Stdin Injection</strong>: n8n -&gt; n100-staging (1524 ms), postgres -&gt; n100-staging (428 ms), and dify -&gt; i5-staging (1477 ms), all verified via proc(tmpfs) with zero CLI exposure.</li><li><strong>Format-Preserving Key Rotation</strong>: N8N_ENCRYPTION_KEY (len 64 -&gt; 64, 3900a8a7cec9 -&gt; f55bf376f489, with [WARN] stateful re-encryption notice) in 373 ms and N8N_PASSWORD (len 24 -&gt; 24, d8aa497b55d6 -&gt; dc5a07717659) in 377 ms, preserving 2 distinct collision-free encrypted .bak snapshots (20261118160730 / 20261118160731).</li><li><strong>Adversarial Guardrails &amp; Positive Controls</strong>: Production subnet targets (192.168.3.2–.10, 192.168.003.5, 192.168.3.011, ::ffff:192.168.3.5, 0:0:0:0:0:ffff:192.168.3.5, ::0:ffff:c0a8:305, king@192.168.3.8, vault-audit.sh with case-insensitive Hostname 192.168.3.5, ProxyJump king@192.168.3.5, ProxyCommand ssh -F inc -W %h:%p pve, ssh -W 192.168.3.5:%p bastion, socat - TCP:192.168.3.5:22, quote/indirection obfuscation (192.168.3.&#39;5&#39;, ${PVE_IP}, 192.168.3.[5], 192.168.3.[5::]), and $HOME/.ssh/config -Jpve chains resolved across /etc/ssh + user configs up to depth 6, plus python3 -I isolated mode resisting CWD/PYTHONPATH-planted ipaddress.py/subprocess.py shadowing in CE-18) aborted with <strong>Exit Code 2</strong>, while benign ProxyCommand ssh -W %h:%p bastion without $HOME/.ssh/config passed (exit_code=0, CE-13a); pointing SRE_VAULT_SHM_DIR=/home/king (ext4) aborted with <strong>Exit Code 3</strong>; .env.* variants, git mv renames, OpenSSH/RSA/EC keys, and Git staged-index bypass attempts were blocked with <strong>Exit Code 1</strong>; non-secret and regex --rotate inputs exited cleanly with <strong>Exit Code 1</strong>.</li></ul><h3>5. An Honest Cluster Audit Finding: Disk Swap Residue</h3><p>Sequential SSH polling across all three hosts via vault-audit.sh --json completed in 15,779 ms (overall_status: HEALTHY, all 6 SLO assertions true), while --skip-probe --json returned overall_status: DEGRADED (zero_disk_env_residue_across_cluster: null, 97 ms) rather than masking skipped checks. Notably, pi3b reported active_swap_devices: 0, whereas both x86_64 workers (n100-staging and i5-staging) reported active_swap_devices: 1. Although transient remote slices remain in /dev/shm for only 428–1524 ms (Stage 4 measured), any memory page swapped onto an unencrypted swap device leaves a forensic surface on memory-pressured workers—making tmpfs -o noswap or swapoff on vault-sensitive nodes an important hardening step.</p><h3>6. What’s Next</h3><p>With all cluster secrets sealed inside an auditable SOPS+Age vault and streamed strictly through verified volatile memory, the foundation is ready for application stack standardization:</p><p>👉 <strong>Chapter 013 — Container Runtime Standardization: Docker Compose Stack Tiering &amp; Network Driver Selection</strong></p><p>We will standardize /opt/docker-stacks/ tiers (Infrastructure, App, Observability), establish port-conflict-free internal DNS service discovery across stacks, and evaluate bridge, host, and macvlan networking across heterogeneous nodes.</p><blockquote><em>### 💡 Read the Full Production Edition on Substack<br>This Medium edition covers the architecture and bare-metal Staging findings. For all 9 copy-paste production scripts (</em><em>vault-secret-injector.sh, </em><em>vault-audit.sh, </em><em>pre-commit-secret-guard.sh, pinned-SHA256 </em><em>bootstrap-ch012.sh, </em><em>stack-secret-policy.conf, </em><em>sops.yaml.example), SHA-256 download verification links, and complete reproduction logs:</em></blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em> : </em><a href="https://ubinity.substack.com/p/credentials-vault-and-secret-governance"><em>HomeLab SRE Cookbook Chapter 012 on Substack</em></a></blockquote><p><em>Originally published at </em><a href="https://ubinity.substack.com/"><em>Richard King’s HomeLab SRE Cookbook</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e72081b4cae4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Deployment Engine Design: Single-Script Multi-Node Cluster Bootstrapping]]></title>
            <link>https://medium.com/@dx_13752/automated-deployment-engine-design-single-script-multi-node-cluster-bootstrapping-e354a6ec16a3?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e354a6ec16a3</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[baremetal]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[openssh]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 23:29:26 GMT</pubDate>
            <atom:updated>2026-11-19T04:15:59.185Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ykxixfn3VvnbnTFaGMJAtA.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Platform Engineers</li><li><strong>Core Topics</strong>: Linux, DevOps, Automation, OpenSSH, Bare Metal</li></ul><h3>1. The Production Reality: Why Naive Shell Loops Fail in Distributed Edge Fleets</h3><p>Building a sovereign, high-efficiency private cloud — such as an automated one-person datacenter operating within a strict 33.4W power envelope (3 nodes consuming 29.2W combined, plus 4.2W for external RAID1 storage and managed switching) — fundamentally transforms how infrastructure is provisioned. In Chapter 010, our Infrastructure-as-Code compiler generated strict DNS resolution blocks (hosts.managed), OpenSSH routing topologies (ssh_config), and canonical environment contracts (cluster.env).</p><p>Yet moving from <strong>static metadata declarations</strong> to <strong>live, multi-node runtime convergence</strong> exposes a severe systems engineering gap. Traditional multi-node shell loops fall victim to four recurring failure modes:</p><ol><li><strong>Compounding Handshake Latencies (Serial Latency Explosion)</strong>: An un-multiplexed SSH invocation performs a full TCP handshake and asymmetric cryptographic authentication on every step. On our live staging hardware, a single cold cross-host SSH invocation consumes a median of <strong>301.78 ms</strong>. Multiplied across N × P × M phases and retries, handshake waits compound linearly and a single stalled node blocks the fleet;</li><li><strong>Architecture Divergence Across Mixed Hardware</strong>: Heterogeneous clusters mixing ARM64 sentinels (Raspberry Pi 3B) with x86_64 storage and compute nodes (Intel N100 / Core i5) fail silently when monolithic scripts omit strict CPU architecture assertions;</li><li><strong>State Contamination from Missing Idempotency</strong>: When transient network jitter interrupts a script mid-flight, re-running naive loops crashes on duplicate /etc/hosts entries, existing directory collisions, or existing Docker bridge networks;</li><li><strong>Catastrophic Blast Radius in Flat L2 Subnets</strong>: When staging nodes (.202, .208, .209) share physical L2 switching with production hosts (.2~.10), a single misconfigured target IP or hidden SSH alias risks corrupting production storage.</li></ol><p>To eliminate these vulnerabilities, Chapter 011 designs a <strong>pure POSIX Bash deployment engine</strong> requiring zero resident agents (no Ansible, no Kubernetes control plane) while achieving deterministic sub-60s convergence and strict privilege isolation.</p><h3>2. Mathematical Performance Model: From O(N × P) Serial Stall to O(1) Multiplexed Convergence</h3><p>Let N be the number of target nodes (3 in our staging topology), P the number of orchestration phases (5), and M the remote operations per phase. Sequential execution scales linearly with connection overhead t_tcp:</p><blockquote><strong><em>T_seq</em></strong><em> = ∑(i=1..N) ∑(j=1..P) ∑(k=1..M) (t_tcp + t_exec^(i, j, k)) = (N × P × M) · t_tcp + ∑(i=1..N) ∑(j=1..P) ∑(k=1..M) t_exec^(i, j, k)</em></blockquote><h3>OpenSSH ControlMaster Multiplexing Connection Pool</h3><p>By activating OpenSSH multiplexing (ControlMaster=auto, ControlPersist=10m) over persistent local Unix domain sockets (~/.ssh/cm-%C), repeated authentication handshakes disappear. On live cross-host LAN benchmarks (pi3b to n100-staging), socket reuse drops latency from <strong>301.78 ms</strong> to <strong>33.15 ms</strong> (an <strong>89.02% reduction</strong>).</p><p>Combined with asynchronous subshell dispatching ((...) &amp;) and wait synchronization per phase, total runtime collapses to the slowest node in each stage:</p><blockquote><strong><em>T_par</em></strong><em> = N · t_tcp + ∑(j=1..P) max_(i=1..N) ( ∑(k=1..M) (t_mux + t_exec^(i, j, k)) )</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7p0bhuvxQ33mc5Z2j0DwjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*btOkYYCucNyRPksx2lP-eQ.png" /></figure><h3>3. Architecture Overview &amp; Least-Privilege Execution Boundary</h3><p>Rather than hardcoding target hosts or exposing dangerous passwordless binaries (tee, sed, install, docker) in /etc/sudoers.d/, our architecture strictly enforces two structural boundaries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxnaRCCILmyPex79Vj3C8w.png" /></figure><ol><li><strong>Contract-Driven 5-Phase State Machine</strong>: Consumes Chapter 010’s static contracts (cluster.env, ssh_config, hosts.managed) across five fail-closed phases: Pre-flight Arch Audit, Atomic Host Contracts, Block-Device Mount Guarded Storage, Container Bridge Readiness, and Health Telemetry Assertion;</li><li><strong>Service-Account Isolation (§H Compliance)</strong>: The interactive admin account (king) retains <strong>zero</strong> passwordless orchestration or privilege-escalation sudo permissions. Remote convergence executes strictly as sre-orchestrator invoking a single root-owned helper (/usr/local/sbin/sre-node-converge) locked to five fixed single-argument subcommands (preflight, contracts, storage, runtime, verify). Both classic sudo (Debian 13) and sudo-rs (Ubuntu 26.04 LTS) refuse arbitrary file writes or extra arguments.</li></ol><h3>4. Preview: Four Fatal Engineering Traps in Edge Cluster Orchestration</h3><p>Elevating an ad-hoc shell script into fault-tolerant automation requires neutralizing four subtle engineering traps:</p><ul><li><strong>Trap 1: Unbounded I/O Hangs</strong> — Why naive background SSH waits forever on kernel D-state storage stalls, and how timeout bounds must pair with TCP keepalives;</li><li><strong>Trap 2: Hidden SSH Alias &amp; Shell Expansion Bypasses</strong> — Why scanning only static config IPs misses HostName overrides in ssh_config and double-quoted remote interpolation flaws;</li><li><strong>Trap 3: Container Volume UID/GID Corruption</strong> — Why recursive chown -R silently breaks Postgres and monitoring containers on reboot;</li><li><strong>Trap 4: Unmounted Root Flash Exhaustion</strong> — Why test -d passes even when external drives fail to mount, requiring kernel block-device (stat -c &#39;%d&#39;) separation checks.</li></ul><p><em>(The complete remediation implementations, stdin schema firewall, full production source code, and adversarial container proofs are available in the paid edition on Substack.)</em></p><h3>5. Empirical Staging Hardware Evidence &amp; Measured SLOs</h3><p>All benchmarks were captured directly on live Staging (pi3b bare-metal Debian 13 ARM64 4C / 905 MB RAM -&gt; n100-staging 4 vCPU / 8 GB RAM Ubuntu 26.04 LTS with dedicated /dev/sdb1 at /data and i5-staging 8 vCPU / 16 GB RAM Ubuntu 26.04 LTS with 2.0 GB /var/lib/data-ssd.img loop mount /dev/loop6 at /data, plus 974 MB loop /dev/loop1 at /mnt/backup-hdd on pi3b; bare-metal production uses physical NVMe/USB drives across all nodes) after purging all ControlMaster sockets on pre-provisioned nodes (evidence/execution.log):</p><pre>================================================================================<br> Starting Single-Script Multi-Node Cluster Bootstrapping Pipeline<br> Cluster Target : homelab-sre-staging (Domain: lab.internal)<br> SSH Account    : sre-orchestrator<br> Target Nodes   : sentinel data_storage compute_worker<br> Start Time UTC : 2026-11-17T06:21:01Z<br>================================================================================<br>[...]<br>&gt;&gt;&gt; Running Phase 3: Directory Hierarchy &amp; Storage Mount Verification<br>--------------------------------------------------------------------------------<br>[INFO] [Phase 3/5] Asserting external storage mount and provisioning volumes on pi3b...<br>  * [PASS] STORAGE_OK:/mnt/backup-hdd:/dev/loop1 and stack directory tree provisioned on pi3b.<br>[INFO] [Phase 3/5] Asserting external storage mount and provisioning volumes on n100-staging...<br>  * [PASS] STORAGE_OK:/data:/dev/sdb1 and stack directory tree provisioned on n100-staging.<br>[INFO] [Phase 3/5] Asserting external storage mount and provisioning volumes on i5-staging...<br>  * [PASS] STORAGE_OK:/data:/dev/loop6 and stack directory tree provisioned on i5-staging.<br>[...]<br>&gt;&gt;&gt; Running Phase 5: Post-Init Verification &amp; Cluster Assertion<br>--------------------------------------------------------------------------------<br>[INFO] [Phase 5/5] Executing post-init verification on pi3b...<br>  * [PASS] Post-init assertions passed cleanly on pi3b (VERIFY_OK:/mnt/backup-hdd:/dev/loop1).<br>[INFO] [Phase 5/5] Executing post-init verification on n100-staging...<br>  * [PASS] Post-init assertions passed cleanly on n100-staging (VERIFY_OK:/data:/dev/sdb1).<br>[INFO] [Phase 5/5] Executing post-init verification on i5-staging...<br>  * [PASS] Post-init assertions passed cleanly on i5-staging (VERIFY_OK:/data:/dev/loop6).</pre><pre>================================================================================<br> [SUCCESS] CLUSTER INITIALIZATION COMPLETED (Status: HEALTHY)<br> Total Elapsed Time : 2.05s (2054 ms)<br> Target Nodes       : sentinel data_storage compute_worker<br> Phases Executed    : 1 2 3 4 5<br> SRE SLA Threshold  : &lt; 60.0s (Measured: 2.05s)<br>================================================================================</pre><h3>Cross-Host LAN OpenSSH Multiplexing Benchmark (STAGE 8)</h3><pre>--- [STAGE 8] Cross-Host LAN OpenSSH Multiplexing Benchmark (N=20 Samples: pi3b -&gt; n100-staging) ---<br>{<br>  &quot;target&quot;: &quot;n100-staging&quot;,<br>  &quot;samples&quot;: 20,<br>  &quot;cold_median_ms&quot;: 301.78,<br>  &quot;mux_median_ms&quot;: 33.15,<br>  &quot;reduction_pct&quot;: 89.02,<br>  &quot;cold_min_ms&quot;: 294.17,<br>  &quot;cold_max_ms&quot;: 330.14,<br>  &quot;mux_min_ms&quot;: 32.36,<br>  &quot;mux_max_ms&quot;: 33.71<br>}</pre><h3>6. What’s Next</h3><p>The automated deployment engine and least-privilege convergence boundaries are now verified across heterogeneous hardware. Next, we eliminate plaintext secret files on disk and establish zero-leakage credential isolation across the cluster:</p><p>👉 <strong>Next Chapter: Chapter 012 Credentials Vault &amp; Secret Governance: Isolation, Vault, and Environment Hardening</strong></p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete 5-phase parallel deployment engine (code/cluster-init-all.sh), argument-pinned root helper (code/sre-node-converge), least-privilege sudoers policy (code/sudoers-sre-orchestration), automated validation suite (code/validate-cluster-init.sh), standalone bootstrapper (code/bootstrap-ch011.sh), Deep-Dive Trap Prescriptions, and full raw staging logs are available to members on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em>: </em><a href="https://ubinity.substack.com/p/automated-deployment-engine-design"><em>HomeLab SRE Cookbook Chapter 011 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e354a6ec16a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Infrastructure as Code (IaC): Dynamic Asset Orchestration with inventory.ini]]></title>
            <link>https://medium.com/@dx_13752/infrastructure-as-code-iac-dynamic-asset-orchestration-with-inventory-ini-236440653185?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/236440653185</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[jinja2]]></category>
            <category><![CDATA[dynamic-inventory]]></category>
            <category><![CDATA[multi-node-orchestration]]></category>
            <category><![CDATA[configuration-drift]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 13:08:23 GMT</pubDate>
            <atom:updated>2026-11-19T04:11:48.316Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0LQDBrG-DIlB0AQ53OBmQ.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Platform Engineers</li><li><strong>Core Topics</strong>: Infrastructure as Code, Jinja2, Dynamic Inventory, Configuration Drift, Multi-Node Orchestration</li></ul><p>When engineers transition a homelab from a collection of standalone servers into a cohesive multi-node cluster, they rarely fail because of hardware limits. Instead, they get ambushed by <strong>Hardcoded IP Sprawl and Configuration Drift</strong>.</p><p>In a typical 3-node architecture — comprising an ARM64 Raspberry Pi sentinel, an Intel N100 stateful storage node, and an Intel i5 compute worker — IP addresses (192.168.3.202, 192.168.3.208, 192.168.3.209), Keepalived virtual IPs (VIPs), SSH ports, and mount paths end up hardcoded across more than 20 separate files: /etc/hosts, ~/.ssh/config, Docker Compose .env files, database replication scripts, and backup cron jobs.</p><p>The moment you re-home a subnet, swap a network interface, or promote scripts between Staging and Production, manual grep and sed edits become a ticking time bomb.</p><h3>1. The Combinatorial Math of Configuration Drift</h3><p>Suppose a cluster has N = 3 physical nodes and each node has M = 8 configuration files referencing network topology parameters. A single topology change requires K = N × M = 24 independent manual edits.</p><p>Even if an experienced operator has a per-file human error rate of just p = 2% (0.02), the probability of completing all 24 edits with zero mistakes is:</p><blockquote><strong><em>P(Success)</em></strong><em> = (1 − p)^K = (1 − 0.02)²⁴ ≈ 0.6158 (61.6%)</em></blockquote><p>Every routine IP rotation carries a <strong>38.4% probability of introducing configuration drift</strong>. And why not simply use envsubst in a shell script? Because envsubst silently replaces misspelled or unset environment variables with an empty string (&quot;&quot;), cheerfully deploying corrupted configurations into production. Worse, standard shell redirection (&gt;) truncates the target file before writing—meaning a sudden power loss or mid-batch template failure leaves a 0-byte file or a partial multi-file tear on disk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxF_OERvhfNa5yhA7ocD2w.png" /></figure><h3>2. Right-Sizing IaC for a 33.4W Datacenter</h3><p>Heavyweight cloud orchestrators like Terraform or Ansible AWX introduce multi-gigabyte control planes or slow API state locking that violate our 33.4W power budget (29.2W across the three hosts + 4.2W shared storage/switch fabric). Instead, we implement a <strong>Single Source of Truth (</strong><strong>inventory.ini) coupled with a fail-closed, two-phase transactional Jinja2 </strong><strong>StrictUndefined compilation engine</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFBdTjMNQS0vf5DSSBFXkw.png" /></figure><h3>Golden Trio Hardware &amp; Staging Topology Constraints</h3><p>Within our 33.4W power envelope (29.2W across the three nodes + 4.2W USB RAID1 enclosure and gigabit switch), every megabyte of RAM and every watt of idle draw matters:</p><ul><li><strong>[sentinel] (</strong><strong>pi3b, </strong><strong>192.168.3.202 / VIP </strong><strong>.203)</strong>: Bare-metal Raspberry Pi 3B (aarch64, 914 MB usable RAM), Debian 13 (trixie), 1.5 W ~ 2.0 W idle draw (mirroring the 4-core sentinel baseline budget); hosts Uptime Kuma (3001), dnsmasq (53), and cold-backup mount /mnt/backup-hdd.</li><li><strong>[data_storage] (</strong><strong>n100-staging, </strong><strong>192.168.3.208 / VIP </strong><strong>.210)</strong>: KVM Virtual Machine on our Xeon D-1581 hypervisor (x86_64, 8192 MB RAM), Ubuntu 26.04 LTS (ubuntu-26.04), 1:1 mirroring our production Intel Alder Lake-N N100 storage node (10.5 W steady wall draw); hosts PostgreSQL Primary (5432), Redis (6379), Gitea (3000/2222), and MinIO (9000/9001).</li><li><strong>[compute_worker] (</strong><strong>i5-staging, </strong><strong>192.168.3.209 / VIP </strong><strong>.211)</strong>: KVM Virtual Machine on our Xeon D-1581 hypervisor (x86_64, 16384 MB RAM), Ubuntu 26.04 LTS (ubuntu-26.04), 1:1 mirroring our production <strong>Minisforum TH50 (Intel Core i5-11320H)</strong> compute node (15.5 W steady wall draw); hosts PostgreSQL Standby (5432), Dify AI (8080), and Code-Server (8443).</li></ul><h3>3. Exposing the Four Fatal Engineering Traps of Naive Template Scripts</h3><p>Before writing a single line of orchestration code, we must dissect the four failure modes that routinely break home and edge datacenters:</p><ol><li><strong>Trap 1: Silent Empty-String Expansion in </strong><strong>envsubst</strong>: When a variable is unset or misspelled (e.g., $SENTINEL_HST), envsubst exits with status 0 and writes an empty string &quot;&quot; into your configuration file, silently breaking /etc/hosts or binding services to blank interfaces.</li><li><strong>Trap 2: </strong><strong>O_TRUNC Zero-Byte Truncation &amp; Multi-File Partial State Tear</strong>: Standard shell redirection (&gt; /etc/hosts) truncates the destination file to 0 bytes before the command even renders output. Worse still, when rendering multiple interdependent contracts sequentially (hosts, ssh_config, cluster.env), a template error in file #3 leaves files #1 and #2 overwritten with new IPs while file #3 retains old IPs—creating an immediate split-brain state tear across the cluster.</li><li><strong>Trap 3: </strong><strong>source Command Injection &amp; INI Multiline Continuation Abuse</strong>: Python&#39;s standard configparser allows indented multiline continuations by default, and downstream provisioning scripts routinely execute source cluster.env.rendered. Without strict input allowlisting and POSIX single-quote escaping, an unquoted semicolon or subshell in inventory.ini leads directly to arbitrary command execution.</li><li><strong>Trap 4: Cross-Environment Production Intrusion &amp; </strong><strong>/tmp Symlink Hijacking</strong>: Sharing a flat /24 LAN (192.168.3.0/24) between a live production rack (.2–.10) and a Staging cluster (.202/.208/.209) means a single typo in inventory.ini can point destructive Staging tests at production databases.</li></ol><h3>4. The 4-Gate Defensive Compilation Architecture</h3><p>Our orchestration engine (render-inventory.py) reads code/inventory.ini and enforces four strict quality gates before emitting /etc/hosts, ~/.ssh/config, and cluster.env:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1P8Tf4I5pUBuEXuF-_1jiA.png" /></figure><pre>┌──────────────────────────────────────────────────────────────────────────┐<br> │             Single Source of Truth (SSoT): code/inventory.ini            │<br> │   [all:vars] ──► Domain (lab.internal), SSH Port (22), Admin (king)      │<br> │   [sentinel] ──► pi3b (192.168.3.202 / VIP .203 / ARM64 / 3.2W)          │<br> │   [data_storage] ──► n100-staging (192.168.3.208 / VIP .210 / 10.5W)     │<br> │   [compute_worker] ──► i5-staging (192.168.3.209 / VIP .211 / 15.5W)     │<br> └────────────────────────────────────┬─────────────────────────────────────┘<br>                                      │<br>                                      ▼<br> ┌──────────────────────────────────────────────────────────────────────────┐<br> │            4-Gate Defensive Compilation Engine (render-inventory.py)     │<br> ├────────────────────────────────────┬─────────────────────────────────────┤<br> │ Gate 1: Production Subnet Interlock│ Aborts (Exit 2) on 192.168.3.2-10   │<br> │ Gate 2: Schema &amp; Collision Audit   │ Verifies CIDR, ports &amp; unique IPs   │<br> │ Gate 3: Two-Phase StrictUndefined  │ In-memory pre-compile; zero partial │<br> │ Gate 4: POSIX Atomic Replacement   │ 0700 dir + tempfile + fsync + rename│<br> └────────────────────────────────────┴─────────────────────────────────────┘<br>                                      │<br>          ┌───────────────────────────┼───────────────────────────┐<br>          ▼ (3.02 ms)                 ▼ (3.55 ms)                 ▼ (5.25 ms)<br> ┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐<br> │   hosts.rendered     │    │ ssh_config.rendered  │    │ cluster.env.rendered │<br> │ /etc/hosts &amp; VIP map │    │ ~/.ssh/config &amp; Mux  │    │ Canonical Env Exports│<br> └──────────────────────┘    └──────────────────────┘    └──────────────────────┘</pre><ol><li><strong>Gate 1 — Production Subnet Safety Interlock</strong>: Scans inventory target IPs (stripping inline #/; comments) and local host NICs, immediately aborting with Exit Code 2 if any target touches the reserved production rack (192.168.3.2 ~ 192.168.3.10) while permitting local compilation from the operator workstation (192.168.3.6 via PRODUCTION_HOST_OCTETS = {2, 3, 5, 8, 9, 10}).</li><li><strong>Gate 2 — CIDR, Schema &amp; Collision Audit</strong>: Uses ipaddress.ip_network and \Z-anchored allowlist regular expressions to validate subnet membership (192.168.3.0/24), reject .0/.255 network/broadcast addresses, verify port ranges (1..65535), and block duplicate IP/VIP collisions across [sentinel], [data_storage], and [compute_worker].</li><li><strong>Gate 3 — Two-Phase </strong><strong>StrictUndefined Pre-Compilation</strong>: Compiles all three templates in memory first; any missing variable raises an immediate TemplateError with zero disk writes. Includes a built-in zero-dependency strict regex fallback (builtin-strict-regex-1.0) and a custom POSIX single-quote escaping filter | shquote (shell_quote_filter).</li><li><strong>Gate 4 — Crash-Safe Atomic Replacement</strong>: Verifies 0700 directory ownership (ensure_secure_output_dir), writes to a hidden temporary file in the target directory with 0644 permissions, flushes both file and parent directory descriptors via os.fsync(), and executes an atomic POSIX os.replace() rename.</li></ol><pre># [SRE Core Code Excerpt: StrictUndefined Compilation &amp; Crash-Safe POSIX Atomic Write]<br>def render_template_str(<br>    template_text: str, context: Dict[str, Any], env: Optional[Any] = None<br>) -&gt; Tuple[str, str]:<br>    if HAVE_JINJA2:<br>        active_env = env if env is not None else _create_jinja_env()<br>        tmpl = active_env.from_string(template_text)<br>        return tmpl.render(**context), f&quot;jinja2-{jinja2.__version__}&quot;<br>    return _builtin_strict_render(template_text, context), &quot;builtin-strict-regex-1.0&quot;<br><br><br>def atomic_write_file(target_path: Path, content: str) -&gt; None:<br>    ensure_secure_output_dir(target_path.parent)<br>    normalized = content.replace(&quot;\r\n&quot;, &quot;\n&quot;)<br>    fd, tmp_name = tempfile.mkstemp(<br>        prefix=f&quot;.{target_path.name}.tmp.&quot;, dir=str(target_path.parent)<br>    )<br>    tmp_path = Path(tmp_name)<br>    try:<br>        with os.fdopen(fd, &quot;wb&quot;) as fp:<br>            fp.write(normalized.encode(&quot;utf-8&quot;))<br>            fp.flush()<br>            # ...<br>            os.fsync(fp.fileno())<br>        os.replace(str(tmp_path), str(target_path))<br>        if os.name != &quot;nt&quot;:<br>            dir_flags = os.O_RDONLY | getattr(os, &quot;O_DIRECTORY&quot;, 0)<br>            # ...<br>    except Exception:<br>        # ...<br>        raise</pre><h3>5. Empirical Benchmark on Bare-Metal Staging Hardware</h3><p>We executed the compilation engine and validate-inventory.sh directly on our bare-metal Staging sentinel node pi3b (192.168.3.202, ARM64 Cortex-A53 / Debian 13 Trixie / Linux 6.18 / Python 3.13.5 / Jinja2 3.1.6) with live TCP/22 socket probes across the 3-node cluster (pi3b 192.168.3.202, n100-staging 192.168.3.208, i5-staging 192.168.3.209). Verbatim excerpt from evidence/execution.log:</p><pre>[3] Deterministic Template Compilation (Output Dir: /tmp/homelab-sre-iac-rendered.bL4kgPXD2Z):<br>  * hosts.j2       -&gt; hosts.rendered       ( 728 B, 20693.42 us) SHA256: 8f7f49a11ca3de6a...<br>  * ssh_config.j2  -&gt; ssh_config.rendered  (1890 B, 31339.63 us) SHA256: 79b449bb5520fbfc...<br>  * cluster.env.j2 -&gt; cluster.env.rendered (1676 B, 45714.06 us) SHA256: 57cb234142dea3b9...<br>  Total Compilation Time : 97.75 ms (Idempotent 2-Pass Verified: True)<br><br>[4] Live Staging Node TCP Reachability Audit:<br>  * pi3b          (192.168.3.202:22) -&gt; [ONLINE] Latency: 41.28 ms | SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4<br>  * n100-staging  (192.168.3.208:22) -&gt; [ONLINE] Latency: 14.39 ms | SSH-2.0-OpenSSH_10.2p1 Ubuntu-2ubuntu3.8<br>  * i5-staging    (192.168.3.209:22) -&gt; [ONLINE] Latency: 11.69 ms | SSH-2.0-OpenSSH_10.2p1 Ubuntu-2ubuntu3.8</pre><p>All three infrastructure contracts compiled on low-power ARM64 hardware in <strong>97.75 milliseconds</strong> (with a 20-iteration median of <strong>98.83 ms</strong>, min <strong>98.32 ms</strong>, and p95 of <strong>99.20 ms</strong>, well below our &lt; 150 ms ARM64 SBC SLO) with 100% two-pass SHA-256 idempotence. During our single-point IP rotation and fault-injection drill, rotating i5-staging from .209 to .219 updated all three downstream artifacts atomically, while injecting a missing template attribute triggered Gate 3 to abort with zero disk writes.</p><h3>6. What&#39;s Next</h3><p>With our Single Source of Truth established and compilation latency verified within the 150 ms ARM64 budget, we connect these rendered configuration contracts directly to our cluster-wide initialization pipeline:</p><p>👉 <strong>Chapter 011 Automated Deployment Engine Design: Single-Script Multi-Node Cluster Bootstrapping</strong></p><p>We will build cluster-init-all.sh to drive parallel, idempotent bootstrapping across our sentinel, storage, and compute tiers from a single command!</p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete dynamic inventory orchestration runbook, verified idempotent deployment manifests (code/validate-inventory.sh, code/render-inventory.py), and empirical staging benchmark suites are available to members of the HomeLab SRE Cookbook publication on Substack.<em><br><br>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em> : </em><a href="https://ubinity.substack.com/p/infrastructure-as-code-iac-dynamic"><em>HomeLab SRE Cookbook Chapter 010 on Substack</em></a><em><br><br></em>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=236440653185" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Host OS Hardening: Why “LAN-Only” HomeLabs Are a Security Illusion]]></title>
            <link>https://medium.com/@dx_13752/host-os-hardening-why-lan-only-homelabs-are-a-security-illusion-f668c403715a?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f668c403715a</guid>
            <category><![CDATA[linux-kernel]]></category>
            <category><![CDATA[ed25519]]></category>
            <category><![CDATA[host-hardening]]></category>
            <category><![CDATA[sudo-governance]]></category>
            <category><![CDATA[openssh]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 11:10:25 GMT</pubDate>
            <atom:updated>2026-11-19T04:19:23.277Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zvuef44pUB8C86Nn-E7bpw.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Platform Engineers</li><li><strong>Core Topics</strong>: Host Hardening, OpenSSH, Ed25519, Sudo Governance, Linux Kernel</li></ul><h3>1. The Illusion of Perimeter Defense</h3><p>In the world of self-hosted micro-datacenters, developers frequently succumb to a dangerous assumption: <em>“It’s behind my home router, so I don’t need enterprise security.”</em></p><p>They deploy modern orchestration stacks, Tailscale mesh tunnels, and Cloudflare Zero Trust connections, yet leave their bare-metal host operating systems running on default passwords, root SSH access enabled, and unlogged sudo privileges.</p><p>The reality of modern networking is that perimeter firewalls are porous. When public tunnels forward web traffic directly into internal containers, a single remote code execution (RCE) flaw in an edge service (such as Gitea or a self-hosted API) grants an attacker an immediate footprint on your internal subnet (192.168.3.x).</p><p>If host OpenSSH daemons permit password logins or direct root sessions, an attacker can brute-force the node within seconds. The result? Total host compromise, ransomware encryption of attached NVMe storage, and lateral traversal across your entire hardware fleet.</p><p>To build an institutional-grade HomeLab, you must adopt the core tenet of SRE: <strong>Zero Trust at the Node Level</strong>.</p><pre>[Inbound Internet Traffic / Dependency RCE Flaw]<br>                       │<br>                       ▼<br>          ┌──────────────────────────┐<br>          │ Compromised Edge Container │<br>          └──────────────────────────┘<br>                       │<br>                       ▼ Lateral Scan of Subnet: 192.168.3.x:22<br>  ┌─────────────────────────────────────────────────────────────────┐<br>  │ Vulnerable OpenSSH Host Daemon                                  │<br>  │   * Password authentication enabled (PasswordAuthentication yes)│<br>  │   * Root direct login permitted (PermitRootLogin yes)           │<br>  │   * Weak or shared default password across cluster              │<br>  └─────────────────────────────────────────────────────────────────┘<br>                       │<br>                       ▼ Host Root Compromise &amp; Ransomware Execution<br>  ┌─────────────────────────────────────────────────────────────────┐<br>  │ Catastrophic Impact:                                            │<br>  │   1. Full data destruction across NVMe / SSDs (`rm -rf /data`)  │<br>  │   2. Exfiltration of Tailscale keys &amp; Kopia backup credentials  │<br>  │   3. Fleet-wide pivot and total loss of availability            │<br>  └─────────────────────────────────────────────────────────────────┘</pre><h3>2. The Mathematics of Authentication: Passwords vs. Ed25519</h3><p>Why is disabling password authentication non-negotiable? The difference in security is not incremental — it is astronomical.</p><h3>The Fragility of Password Entropy</h3><p>Consider an 8-character mixed alphanumeric password (|Σ| = 62). The total keyspace is:</p><blockquote><strong><em>Search Space</em></strong><em> = 6²⁸ ≈ 2.18 × 1⁰¹⁴ combinations</em></blockquote><p>This represents roughly <strong>47.6 bits of Shannon entropy</strong>. Over a local 1Gbps network without aggressive connection throttling, an automated tool testing 5,000 attempts per second requires:</p><blockquote><strong><em>T_exhaust</em></strong><em> = (2.1834 × 1⁰¹⁴) / (5000 × 86400 × 365.25) ≈ 1,383.5 years (≈ 1,384 years)</em></blockquote><p>While full traversal takes centuries, adversaries do not traverse randomly. When loading common weak-password dictionaries (admin, raspberry, password123), LAN collision succeeds in <strong>less than two seconds</strong>.</p><h3>The Mathematical Wall of Ed25519</h3><p>By contrast, the <strong>Ed25519</strong> curve adopted in this guide operates on a 256-bit key length and provides <strong>128 bits of cryptographic security level</strong>:</p><blockquote><strong><em>Work Factor</em></strong><em> ≈ ²¹²⁸ ≈ 3.4 × 1⁰³⁸ operations</em></blockquote><p>Even if an adversary marshaled the world’s most powerful supercomputing clusters performing 1⁰¹⁸ cryptographic operations per second, inverting a single Ed25519 public key would require approximately:</p><blockquote><strong><em>T_crack</em></strong><em> ≈ (3.4 × 1⁰³⁸) / (1⁰¹⁸ × 3.15 × 1⁰⁷) ≈ 1.08 × 1⁰¹³ years (≈ 10.8 trillion years)</em></blockquote><p>Once PasswordAuthentication no and KbdInteractiveAuthentication no are enforced, brute-force exploitation becomes mathematically impossible.</p><h3>3. The Four-Layer Host Defense Matrix</h3><p>A durable hardening architecture must defend the host in depth rather than relying on a single configuration toggle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iRjfN_OSYzEpnsPPruOvgg.png" /></figure><ol><li><strong>Layer 1: Network &amp; OpenSSH Daemon Hardening</strong></li></ol><p>Deployed via /etc/ssh/sshd_config.d/00-sre-hardening.conf. The 00- lexical prefix guarantees first-match dominance over cloud-init or OEM drop-in overrides. All password and keyboard-interactive authentication mechanisms are disabled, root login is completely blocked, connection attempts are bounded by MaxAuthTries 3, and cryptographic suites are restricted to modern post-quantum hybrids and curve25519.</p><ol><li><strong>Layer 2: Identity Governance &amp; Non-Root Operations</strong></li></ol><p>A dedicated administrative account (king) is provisioned with shell /bin/bash and enrolled in administrative groups. Root-equivalent groups (lxd, docker, disk) are strictly audited and stripped from the administrative user to eliminate unauthenticated privilege escalation, requiring container workflows to use sudo lxc or isolated service accounts. The root account password is permanently locked (passwd -l root), and legacy OEM accounts (ubuntu, pi, debian) are neutralized.</p><ol><li><strong>Layer 3: Sudo Least Privilege &amp; Audit Trail</strong></li></ol><p>Wildcard escalation (ALL=(ALL) NOPASSWD: ALL) is eliminated. Daily automated health checks and monitoring heartbeats are restricted to pinned, immutable commands (/usr/bin/systemctl is-active ssh, status ssh). All administrative escalation requires authentication, and every privileged invocation is recorded with timestamps and session context to /var/log/sudo.log.</p><ol><li><strong>Layer 4: Kernel Parameters &amp; Resource Limits</strong></li></ol><p>System integrity is protected at the kernel level via sysctl: blocking symlink/hardlink race conditions (fs.protected_symlinks=1, fs.protected_hardlinks=1), hiding kernel pointers from unprivileged inspection (kernel.kptr_restrict=2), and enabling reverse path filtering (net.ipv4.conf.all.rp_filter=2) to prevent IP spoofing across Tailscale mesh subnets.</p><h3>4. Lockout Prevention &amp; Two-Session Discipline</h3><p>The primary risk of applying strict SSH hardening is administrator lockout. To guarantee zero-downtime execution, the deployment engine applies strict defensive gates (abridged excerpt from init-node.sh):</p><pre># [Abridged] Pre-flight Syntax Verification &amp; Graceful Reload<br>harden_sshd_config() {<br>    # [...] Neutralize conflicting directives &amp; validate sshd syntax<br>    sshd -t || exit 1</pre><pre>    systemctl reload ssh 2&gt;/dev/null || systemctl reload sshd</pre><pre>    # Contextual check: verify sshd -T -C across LAN, Docker &amp; Tailscale ranges<br>    local primary_ip<br>    primary_ip=$(hostname -I 2&gt;/dev/null | awk &#39;{print $1}&#39; || echo &quot;127.0.0.1&quot;)<br>    local probe_addrs=(&quot;127.0.0.1&quot; &quot;::1&quot; &quot;10.0.0.1&quot; &quot;172.17.0.1&quot; &quot;172.16.0.1&quot; &quot;192.168.3.50&quot; &quot;100.64.0.1&quot; &quot;fd7a:115c:a1e0::1&quot;)<br>    for u in &quot;root&quot; &quot;${ADMIN_USER}&quot;; do<br>        for addr in &quot;${probe_addrs[@]}&quot;; do<br>            local ctx=&quot;user=${u},host=staging-probe,addr=${addr},laddr=${primary_ip},lport=22&quot;<br>            local probe_out<br>            probe_out=$(sshd -T -C &quot;$ctx&quot; 2&gt;&amp;1) || exit 1<br>            local ctx_pwd ctx_root ctx_kbd<br>            ctx_pwd=$(echo &quot;$probe_out&quot; | awk &#39;/^passwordauthentication / {print $2}&#39;)<br>            ctx_root=$(echo &quot;$probe_out&quot; | awk &#39;/^permitrootlogin / {print $2}&#39;)<br>            ctx_kbd=$(echo &quot;$probe_out&quot; | awk &#39;/^kbdinteractiveauthentication / {print $2}&#39;)<br>            if [ &quot;$ctx_pwd&quot; != &quot;no&quot; ] || [ &quot;$ctx_root&quot; != &quot;no&quot; ] || [ &quot;$ctx_kbd&quot; != &quot;no&quot; ]; then<br>                log_error &quot;Contextual Match bypass detected under ${ctx}!&quot;<br>                exit 1<br>            fi<br>        done<br>    done<br>}</pre><h3>Why systemctl reload Beats restart</h3><p>On modern Debian and Ubuntu platforms, ssh.service operates with KillMode=process, meaning established sessions survive restarts. However, running systemctl restart destroys and recreates the listening network socket. If the new configuration fails to bind or has syntax defects, the daemon crashes, preventing all future logins. By contrast, systemctl reload signals the daemon to parse configuration files without closing the listening socket. If parsing fails, the running daemon continues operating under its existing configuration, allowing the administrator to inspect logs and remediate errors immediately.</p><p>Always observe the <strong>Two-Session Rule</strong>: never terminate your existing terminal session until a secondary SSH window successfully connects using your Ed25519 private key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0jGK4GMxXww7A7xlVzIrg.png" /></figure><h3>5. Architectural Deep Dive: Sudo-rs, Classic Sudo &amp; Silent Match Bypasses</h3><p>When deploying infrastructure automation across a heterogeneous fleet, two subtle architectural pitfalls frequently compromise security audits:</p><h3>1. The Sudo Implementation Shift: Classic C Sudo vs. Rust sudo-rs</h3><p>In modern distributions such as Ubuntu 26.04 LTS, the default /usr/bin/sudo implementation transitions to sudo-rs, a memory-safe Rust-based alternative. While sudo-rs supports standard least-privilege scoping (ALL=(ALL:ALL) ALL), its parser intentionally rejects legacy file-logging directives such as Defaults logfile=&quot;/var/log/sudo.log&quot; and Defaults log_year, log_host. Attempting to deploy a monolithic sudoers file across both Debian 13 (which ships classic Todd Miller sudo 1.9) and Ubuntu 26.04 results in immediate visudo rejection (unknown setting: &#39;logfile&#39;).</p><p>A resilient SRE hardening script must inspect the running implementation (sudo -V 2&gt;&amp;1) dynamically:</p><ul><li>On classic sudo nodes, configure dedicated file audit logging (Defaults logfile=);</li><li>On sudo-rs nodes, configure Defaults use_pty and rely on native systemd-journald audit streaming (journalctl _COMM=sudo).</li></ul><h3>2. Silent Match Block Bypasses</h3><p>Many cloud-init scripts or existing configurations include Match blocks (e.g., Match Address 192.168.0.0/16 PasswordAuthentication yes). If an auditor or gate only runs a bare sshd -T, OpenSSH outputs solely the global configuration, completely masking conditional overrides. To prevent silent vulnerabilities, the hardening engine and audit suite must probe contextual evaluations:</p><p>sshd -T -C user=king,host=staging-probe,addr=192.168.3.50</p><h3>6. Live Staging Verification Results</h3><p>The hardening engine was verified on live bare-metal hardware and virtualized nodes within our isolated Twin Staging Cluster (n100-staging 192.168.3.208 and i5-staging 192.168.3.209 running Ubuntu 26.04 / OpenSSH 10.2p1; pi3b 192.168.3.202 running Debian 13 ARM64 / OpenSSH 10.0p2; Production Reserved Range 192.168.3.2~10 physically isolated).</p><p>Penetration testing and automated audits confirmed:</p><ul><li>Root login over SSH is rejected (Permission denied (publickey)), both with password and with key authentication.</li><li>Password authentication attempts are immediately rejected across all accounts.</li><li>Automated compliance verification via audit-node-security.py confirmed 100% adherence across all 12 SLO assertions in <strong>52.07 milliseconds</strong> on Ubuntu 26.04 x86_64 (2026-11-13 single run; 11-assertion baseline N=7 median: 46.38 ms on x86_64, 328.82 ms on Debian 13 ARM64—all within the &lt; 75ms / &lt; 350ms SLOs).</li></ul><pre>--- Testing Node: 192.168.3.208 ---<br>$ ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@192.168.3.208<br>[...]<br>root@192.168.3.208: Permission denied (publickey).<br>[PASS] Root password authentication blocked.<br>$ ssh -i ~/.ssh/id_ed25519 -o PreferredAuthentications=publickey root@192.168.3.208<br>[...]<br>root@192.168.3.208: Permission denied (publickey).<br>[PASS] Root direct publickey authentication blocked (PermitRootLogin no).<br>$ ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no king@192.168.3.208<br>[...]<br>king@192.168.3.208: Permission denied (publickey).<br>[PASS] Password authentication blocked for admin user.<br>$ ssh -i ~/.ssh/id_ed25519 king@192.168.3.208 &quot;echo SSH_PUBKEY_AUTH_OK&quot;<br>SSH_PUBKEY_AUTH_OK<br>[PASS] Ed25519 public key authentication verified.<br>$ ssh -i ~/.ssh/id_ed25519 king@192.168.3.208 &quot;sudo -n /usr/bin/systemctl is-active ssh&quot;<br>active<br>[PASS] Pinned passwordless command permitted.<br>$ ssh -i ~/.ssh/id_ed25519 king@192.168.3.208 &quot;sudo -n id -u&quot;<br>sudo: interactive authentication is required<br>$ Wire probe KEXINIT proposal:<br>debug1: Authentications that can continue: publickey</pre><h3>7. What’s Next</h3><p>With node baselines locked down across our hardware nodes, manual per-host configuration is no longer acceptable. In the next chapter, we implement Infrastructure as Code (IaC) to orchestrate cluster assets dynamically:</p><p>👉 <strong>Chapter 010 Infrastructure as Code (IaC): Dynamic Asset Orchestration with inventory.ini</strong></p><p>We replace manual per-host edits with a deterministic single source of truth that compiles /etc/hosts, ~/.ssh/config, and cluster environment contracts in milliseconds.</p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete verified operational runbook, idempotent hardening script (init-node.sh), hardened OpenSSH drop-in config (sshd_config), and 12-assertion compliance audit tool (audit-node-security.py) are available on the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive &amp; get the companion scripts</em></strong><em>: </em><a href="https://ubinity.substack.com/"><em>HomeLab SRE Cookbook Chapter 009 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f668c403715a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Single Source of Truth Storage Tiering: N100 NVMe System Disk & 1TB Business SSD Mount Architecture]]></title>
            <link>https://medium.com/@dx_13752/single-source-of-truth-storage-tiering-n100-nvme-system-disk-1tb-business-ssd-mount-architecture-5b448e3bfe00?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b448e3bfe00</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[systemd]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Sun, 15 Nov 2026 13:52:59 GMT</pubDate>
            <atom:updated>2026-11-19T04:24:15.529Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-2CupHRFEOWO9pqF3c9qw.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Self-Hosters</li><li><strong>Core Topics</strong>: Linux, Storage, HomeLab, DevOps, Systemd</li></ul><p>When setting up a compact homelab server on energy-efficient x86 hardware such as the NiPoGi N100 (4 cores, 6W TDP, roughly 10.5W steady wall power), almost everyone begins with the exact same convenient layout:</p><p><strong>Everything goes onto the single 256GB M.2 NVMe SSD that shipped inside the chassis.</strong></p><p>In week one, the setup looks clean and effortless. Within three to six months of 24/7 operation, however, keeping the host operating system and persistent application state on a single block device creates three predictable points of failure:</p><ol><li><strong>Root Saturation Freeze</strong>: Automated Docker image pulls, container stdout logs, and PostgreSQL WAL growth fill / to 100%. Once free blocks reach zero, systemd and background daemons cannot allocate lock files or Unix sockets in /run, leaving the box reachable by power switch alone because SSH fails to spawn new sessions.</li><li><strong>OS Rebuild Wipeout</strong>: Consumer M.2 drives wear quickly under mixed log and database chatter, and kernel or initramfs updates occasionally brick boots. When / and PostgreSQL share one partition, re-imaging the host means restoring multi-year Git history and databases from scratch.</li><li><strong>Headless Boot Lockout on Naive Secondary Drives</strong>: Adding a second 1TB SSD later via a raw device node such as /dev/sdb1 in /etc/fstab introduces a subtle timing trap. Plugging in an external USB drive can shift kernel enumeration to /dev/sdc1. With default blocking mount units, systemd pauses boot for 90 seconds and drops the server into an interactive local-only Emergency Shell—impossible to reach over the network.</li></ol><p>Breaking out of these cycles requires adopting a storage discipline borrowed from distributed systems: decoupling <strong>Disposable Cattle (the OS drive)</strong> from <strong>Perpetual Pets (authoritative business state)</strong>.</p><h3>1. Why Single-Drive Consolidation Fails Under Unattended SRE Workloads</h3><p>The appeal of a single NVMe drive is simplicity: one partition, one mount point, no extra cables inside a miniature plastic enclosure. Yet from an SRE blast-radius perspective, single-drive consolidation couples high-churn ephemeral artifacts with low-tolerance authoritative state.</p><p>Container registries pull gigabytes of compressed layers during routine updates. Package managers expand caches under /var/cache/apt. System journals accumulate crash dumps or debug logs. None of those bytes carry long-term business value—you can regenerate every OS binary in minutes from standard installation media. Yet when they share the exact same filesystem namespace and free-block pool as PostgreSQL tables, Gitea bare repositories, MinIO object buckets, and n8n workflow queues, any runaway cache can starve critical writes.</p><p>Conversely, treating /data as a dedicated persistence boundary transforms what an OS outage means. A botched Linux upgrade stops hurting business continuity at the block layer: /data stays intact, unformatted, and portable across hardware swaps as long as host automation respects clean unit boundaries.</p><h3>2. Architectural Split: Disposable Cattle vs. Perpetual Pet</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-1OTjcqfv4_eqygx9HP3hg.png" /></figure><p>Our N100 architecture divides storage strictly into two isolated physical roles:</p><ol><li><strong>System Boot Drive (256GB M.2 NVMe SSD at </strong><strong>/ and </strong><strong>/boot) — Disposable Cattle</strong>: Stores the Linux kernel, systemd units, APT packages, and Docker&#39;s runtime directory (/var/lib/docker). If the NVMe controller degrades or an upgrade corrupts boot configuration, the operational recovery target is swapping the stick and re-imaging a minimal OS under a ten-minute RTO budget. Zero authoritative user records live here.</li><li><strong>Stateful Business Tier (1TB 2.5-inch SATA SSD Target at </strong><strong>/data) — Perpetual Pet (SSOT)</strong>: Houses PostgreSQL, Gitea, MinIO, Redmine, and n8n persistent data subdirectories mapped into containers via explicit Docker Compose bind mounts (including /data/docker-volumes).</li></ol><p>Notice why our target production hardware uses a <strong>2.5-inch SATA SSD</strong> for /data instead of stacking a second high-speed M.2 NVMe drive inside a tiny mini-PC chassis:</p><ul><li><strong>Thermal Envelope (~1.8W active, ~50mW PHY idle)</strong>: Compact N100 enclosures have minimal internal airflow. Dual NVMe sticks mounted closely on a crowded board frequently throttle past 75°C under sustained metadata writes, whereas a 2.5-inch SATA drive distributes heat across a dedicated metal bay and lower power rail.</li><li><strong>Electrical Isolation &amp; Portability</strong>: Placing /data on an independent SATA controller decouples power and bus faults from the primary M.2 slot. If the mainboard or internal PSU fails, lifting out a standard 2.5-inch SATA drive into a USB 3.0 enclosure allows attaching /data cleanly to a standby compute node (re-attaching deterministically via filesystem UUID once the standby OS host is up).</li></ul><p>It is equally important to keep Docker’s internal scope clear: installing Docker does not automatically redirect named volumes or overlay layers onto /data. /var/lib/docker continues living on the root NVMe drive unless data-root is deliberately moved, so keeping authoritative services anchored on explicit bind mounts under /data/... preserves fast OS rebuilds without complicating container daemon storage drivers.</p><h3>3. Four Hidden Failure Traps When Adding a Secondary SSD</h3><p>Partitioning a second SSD and tossing a generic mount line into /etc/fstab rarely holds up in production. Four specific Linux storage traps appear quickly once real workloads start running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cA8AuusyMU5XWxFBbsVyeA.png" /></figure><h3>Trap 1: Mutable /dev/sdX Enumeration Drift</h3><p>Linux assigns /dev/sda, /dev/sdb, and /dev/sdc dynamically in order of kernel device probe completion across SATA ports and USB hubs. Attaching a cold backup disk (such as a USB drive from Chapter 007) before rebooting routinely flips /dev/sdb1 to /dev/sdc1. Anchoring mounts to immutable 128-bit superblock UUIDs discovered via blkid rather than transient device letters removes bus-ordering ambiguity.</p><h3>Trap 2: Blocking Boot Mounts on Headless Hardware</h3><p>Consumer workstations tolerate a 90-second mount pause followed by a local Emergency Shell prompt because someone can sit at a monitor and keyboard. Headless mini-PCs running inside wiring closets cannot recover that way. Systemd mount dependencies must degrade gracefully if /data drops offline so networking and SSH still start, allowing operators to diagnose disk health remotely.</p><h3>Trap 3: Docker Ghost Directories &amp; VFS Shadow Masking</h3><p>Graceful boot without /data creates an insidious secondary race: if Docker starts before /data mounts—or keeps running after /data unexpectedly unmounts at runtime—container bind mounts happily initialize empty directories directly inside / on the root NVMe drive and write live database updates there.</p><p>Worse still, once an administrator notices /data was missing and mounts the SATA drive afterward, Linux VFS overlays hide the newly written rootfs &quot;ghost directory&quot; beneath the freshly mounted filesystem. Containers suddenly see older state from the SSD while orphaned transactions silently fill the OS partition underneath. Blocking this split requires <strong>both</strong> boot-time ordering and runtime unmount teardown (BindsTo unit semantics alongside RequiresMountsFor) stored in an isolated drop-in file that never overwrites existing user override.conf customizations.</p><h3>Trap 4: Default 5% ext4 Privileged Tax &amp; Inline Discard Queues</h3><p>Standard mkfs.ext4 reserves 5% of all blocks strictly for root rescue sessions. While vital on /, setting aside 5% on a dedicated 1TB (~931.5 GiB) container drive hides ~46.6 GiB from unprivileged database and Git UIDs (such as UID 999 for PostgreSQL or UID 1000 for Gitea and n8nio/n8n), triggering premature ENOSPC failures before monitoring panels report zero free space. Reclaiming 4% down to 1% opens ~37.2 GiB of usable production capacity on a 1TB disk—while turning off inline synchronous discard avoids stalling write queues during large deletions in favor of scheduled off-peak background maintenance.</p><blockquote><em>💡 </em><strong><em>Want the copy-paste turnkey automation, fail-closed </em></strong><strong><em>sha256sum bundle, rollback engine, and Ghost-Directory </em></strong><strong><em>rsync rescue runbook?</em></strong><em><br>Read the full Freemium engineering deep-dive on </em><a href="https://ubinity.substack.com/"><em>HomeLab SRE Cookbook on Substack</em></a><em> for the hardened shell orchestrator, idempotent systemd configs, and complete live telemetry.</em></blockquote><h3>4. What We Verified Live on n100-staging</h3><p>Rather than relying on theory or hard-coded estimates, all provisioning, idempotency, unit-dependency, safety-interlock, and synchronous I/O checks in this chapter were executed live inside our isolated Staging rig n100-staging (192.168.3.208, VIP 192.168.3.210, isolated physically from the 192.168.3.2~10 production subnet).</p><h3>Honest Hardware &amp; OS Test Boundary</h3><p>n100-staging runs <strong>Ubuntu 26.04.1 LTS (kernel </strong><strong>7.0.0-42-generic, systemd </strong><strong>259)</strong> on KVM/QEMU hardware. For empirical verification, we attached a dedicated <strong>64 GiB virtual secondary disk (</strong><strong>/dev/sdb1, </strong><strong>QEMU HARDDISK, label </strong><strong>homelab-data, </strong><strong>UUID=2efaeb25-33e5-4f81-8987-bd528c2f4f33)</strong>. Because QEMU virtualizes storage controllers, hardware-specific characteristics of bare-metal SATA AHCI NCQ queues and SSD thermal throttling cannot be measured on this virtual testbed and stay strictly documented as target hardware design rationale.</p><h3>Key Empirical Results Logged in evidence/execution.log</h3><p>Across two consecutive live provisioning runs and an adversarial verification pass, our suite demonstrated five concrete guarantees:</p><ul><li><strong>Two-Run Idempotency (</strong><strong>0 changes on Run 2)</strong>: Run 1 configured /etc/fstab (matching non-comment lines so installer comments stay dormant), installed the dedicated systemd drop-in, adjusted ext4 reserved blocks (838,835 -&gt; 167,767, measuring <strong>671,068 blocks / ~2,621 MiB reclaimed</strong> on the 64 GiB Staging disk rather than printing static percentages), and initialized container directories. Run 2 exited cleanly reporting reclaimed_blocks=0 and 0 changes across every unit and mount entry.</li><li><strong>Systemd Unit Dependency Properties Verified</strong>: Live introspection via systemctl show confirmed RequiresMountsFor=/data, After=data.mount, Requires=data.mount, and BindsTo=data.mount active on docker.service without touching override.conf.</li><li><strong>Container Non-Root UID Compatibility</strong>: Inspecting /data/... via ls -ld confirmed 999:999 (lxd:systemd-journal on Ubuntu 26.04 host) on database/attachment trees and 1000:1000 (king:king) on Git/object/workflow directories, while preserving pre-existing /data/volumes permissions untouched.</li><li><strong>Fail-Closed Interlocks (</strong><strong>Exit Code 2 &amp; </strong><strong>Exit Code 1)</strong>: Targeting production IPs (192.168.3.8 or IPv4-mapped ::ffff:192.168.3.8) aborted immediately with exit_code=2; passing the root logical volume /dev/mapper/ubuntu--vg-ubuntu--lv aborted safely with exit_code=1.</li><li><strong>Durable 4KB </strong><strong>os.pwrite + os.fsync Microsecond Audit</strong>: Rather than reporting misleading RAM page-cache write() speeds, our zero-dependency Python benchmark tool measured 200 seeded random 4KB writes paired with synchronous os.fsync(fd) in 837.46 ms: buffered 4K random writes averaged 20.74 us, while true <strong>durable </strong><strong>pwrite+fsync latency averaged </strong><strong>4042.89 us (</strong><strong>p50: 3974.61 us, </strong><strong>p99: 7020.31 us, </strong><strong>247.3 IOPS)</strong> with overall verdict PASS.</li></ul><p><em>(Single-Disk Scope Reminder: Dual-disk tiering ensures OS corruption produces zero local block loss on </em><em>/data, while physical protection against failure of the </em><em>/data SSD itself relies on automated cold backups from Chapter 007 and streaming PostgreSQL replication covered in later chapters.)</em></p><h3>5. What’s Next</h3><p>Once storage boundaries and /data persistence are locked down, the next operational priority is hardening external access and privilege elevation across every host in the cluster:</p><p><strong>Next Chapter: Host OS Hardening: Why “LAN-Only” HomeLabs Are a Security Illusion</strong></p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete verified disaster recovery runbook, Ghost-Directory dry-run merge SOP, rollback-safe provisioning script (mount-data-ssd.sh), and zero-dependency durable I/O benchmark suite (storage-tiering-benchmark.py) are available to members of the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em>: </em><a href="https://ubinity.substack.com/p/single-source-of-truth-storage-tiering"><em>HomeLab SRE Cookbook Chapter 008 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b448e3bfe00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cold Storage Longevity: USB 3.0 External HDD Spindown & Power Governance]]></title>
            <link>https://medium.com/@dx_13752/cold-storage-longevity-usb-3-0-external-hdd-spindown-power-governance-70655af15b11?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/70655af15b11</guid>
            <category><![CDATA[inrush-surge]]></category>
            <category><![CDATA[scsi-start-stop-unit]]></category>
            <category><![CDATA[anti-wakeup-defense]]></category>
            <category><![CDATA[uas-and-sat-protocol]]></category>
            <category><![CDATA[usb-3-hdd-spindown]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 13:31:41 GMT</pubDate>
            <atom:updated>2026-11-19T04:29:03.221Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bSa_F2rCpX6MOypnJeM2Bw.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Self-Hosters</li><li><strong>Core Topics</strong>: USB 3.0 HDD Spindown, UAS &amp; SAT Protocol, SCSI START_STOP_UNIT, Inrush Surge, Anti-Wakeup Defense</li></ul><p>In a 24/7 sovereign home datacenter, external USB 3.0 mechanical drives serve as the immutable last line of defense: holding cold archives and encrypted pull backups.</p><p>Yet, countless systems engineers watch their backup drives spin continuously at 5400 RPM for 23 hours a day, radiating 48°C heat and burning out fluid dynamic bearings within three years.</p><p>When attempting standard Linux power-saving commands like hdparm -y /dev/sdb, the console responds with a frustrating failure: HDIO_DRIVE_CMD(standby) failed: Invalid argument.</p><p>Here is the deep engineering breakdown of why this happens and how to govern external storage for a decade of continuous service.</p><h3>1. The Root Cause: UAS vs. SAT Protocol Barriers</h3><p>Under Linux, USB 3.0 storage controllers default to the <strong>UAS (USB Attached SCSI)</strong> driver (uas.ko). This provides high-throughput command queuing across USB endpoints.</p><p>However, mechanical hard drives speak the ATA command set. The USB enclosure bridge chip (such as the JMicron JMS567 or ASMedia ASM1153E) must perform <strong>SCSI-to-ATA Translation (SAT)</strong> .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-evcIbxWfp0zucYpPehew.png" /></figure><p>When hdparm executes, it wraps an ATA STANDBY_IMMEDIATE command in a SCSI passthrough frame. In most consumer enclosures, firmware drops this frame entirely.</p><h3>The Fix: Native SCSI START_STOP_UNIT (0x1B)</h3><p>Instead of relying on broken ATA translation, we issue raw SCSI power condition commands directly to the drive:</p><blockquote><strong><em>SCSI Command</em></strong><em>: </em><em>START_STOP_UNIT (0x1B) </em><em>[POWER CONDITION=0x02 (STANDBY)]</em></blockquote><p>Using sg_start --stop /dev/sdb, the command bypasses defective SAT layers and directly orders the spindle motor to halt within 3 seconds.</p><h3>2. Electrical Physics: The 1.5A Inrush Current Trap</h3><p>Never power a mechanical storage array directly from a single-board computer’s USB port.</p><p>Accelerating mechanical platters from standstill requires overcoming static friction. During the first 500 milliseconds of spin-up, the motor pulls an inrush surge of <strong>1.2A to 1.5A</strong> (over <strong>2.4A</strong> for dual-drive mirrored enclosures).</p><p>Because a Raspberry Pi caps its USB 5V rail at 1.2A, this current draw collapses bus voltage, resetting the xHCI controller and causing the drive to drop offline (uas: xhci reset).</p><blockquote><strong><em>Engineering Rule</em></strong><em> : Always use an </em><strong><em>independent 12V 2.5A+ DC power adapter</em></strong><em> for external hard drive enclosures.</em></blockquote><h3>3. Eliminating Ghost I/O: The Triple Anti-Wakeup Shield</h3><p>Halting the disk is only half the battle. Background daemons often wake it up minutes later:</p><ol><li><strong>udisks2</strong> : Periodically probes storage media. Blocked via udev rules (ENV{UDISKS_IGNORE}=&quot;1&quot;).</li><li><strong>smartd</strong> : Telemetry reads wake spinning platters. Blocked via smartd.conf with -n standby,10,q.</li><li><strong>updatedb</strong> : Nightly index sweeps traverse cold storage. Blocked via updatedb.conf PRUNEPATHS.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApcNhF52acWr3FE_ggINyA.png" /></figure><h3>4. The SRE Golden 20-Minute Window &amp; Dual-Baseline Verification</h3><p>Balancing thermal degradation against mechanical head ramp Load/Unload fatigue yields an optimal <strong>1200-second (20-minute) idle timeout</strong> :</p><ul><li>Daily power consumption drops from 2.3W to <strong>1.5W</strong> (-34.8%).</li><li>Operating temperature cools from 48.2°C to <strong>24.5°C</strong> ambient.</li><li>Projected bearing lifespan extends to <strong>8.5+ years</strong> .</li></ul><h3>Staging Verification on pi3b (JMicron JMS551 Dual-Bay 2x Toshiba 2TB 5400RPM HDD)</h3><ul><li><strong>Inrush Surge Proof</strong> : vcgencmd get_throttled captured 0x50000 (undervoltage event from dual spindle motors), proving dedicated DC power isolation is mandatory.</li><li><strong>Latency Dual-State</strong> : <strong>81.0 ms</strong> for immediate read during spindle inertial coast-down vs. <strong>3,241.58 ms (3.24s)</strong> cold spin-up from 0 RPM standby.</li></ul><h3>5. What’s Next</h3><p>With cold storage durability and USB spindle governance stabilized, we turn to primary compute node storage architecture:</p><p>👉 <strong>Chapter 008 Single Source of Truth Storage Tiering: N100 NVMe System Disk &amp; 1TB Business SSD Mount Architecture</strong></p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete production-hardened usb-hdd-manager.sh suite, automated systemd idle daemons, and reproducible Python telemetry benchmarks are available to members of the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em> : </em><a href="https://ubinity.substack.com/p/cold-storage-longevity-usb-30-external-3bc"><em>HomeLab SRE Cookbook Chapter 007 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70655af15b11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Stop Your Raspberry Pi SD Card from Dying: A 5-Layer SRE Flash Shield]]></title>
            <link>https://medium.com/@dx_13752/how-to-stop-your-raspberry-pi-sd-card-from-dying-a-5-layer-sre-flash-shield-b125c181459b?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b125c181459b</guid>
            <category><![CDATA[tmpfs-hardening]]></category>
            <category><![CDATA[write-amplification]]></category>
            <category><![CDATA[raspberry-pi-sd-wear]]></category>
            <category><![CDATA[volatile-journald]]></category>
            <category><![CDATA[power-loss-defense]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 13:25:23 GMT</pubDate>
            <atom:updated>2026-11-19T06:27:14.033Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z9eKr1JV3x3SQVYHm-1d2g.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Embedded Linux Engineers, Edge Computing Architects, Site Reliability Engineers, Homelab Practitioners</li><li><strong>Core Topics</strong>: Raspberry Pi SD Wear, NAND Flash Physics, Write Amplification (WAF), tmpfs Hardening, Volatile Journald, Power Loss Defense</li></ul><p>If you have ever run a Raspberry Pi as a 24/7 homelab server — hosting DNS resolvers, VPN tunnels, or monitoring sentinels — you have likely experienced the dreaded “SD card sudden death syndrome.”</p><p>Without warning, the device drops offline. Plugging in a display reveals the fatal kernel verdict: EXT4-fs error (device mmcblk0p2): Remounting filesystem read-only. Your filesystem has corrupted, the NAND flash has locked itself, and your server is dead in the water.</p><p>Here is the engineering reality: <strong>MicroSD cards are not miniature SSDs.</strong> When you run a standard Linux distribution without flash-oriented hardening, the operating system’s default write patterns will destroy a consumer TF card within 3 to 6 months.</p><p>In this deep dive, we explore the physics of NAND flash write amplification, analyze why sudden residential power cuts destroy filesystems, and deploy a <strong>5-layer in-memory flash protection shield</strong> that eliminates 99.9% of flash writes.</p><h3>The Root Cause: MicroSD Flash Architecture &amp; The 1000x WAF Penalty</h3><p>Enterprise-grade solid-state drives (SSDs) are equipped with high-speed multi-gigabyte DRAM caches, multi-core ARM controllers, and advanced wear-leveling algorithms. Consumer MicroSD cards possess none of these.</p><pre>┌──────────────────────────────────────┬──────────────────────────────────────┐<br>│ Enterprise NVMe SSD                  │ Consumer MicroSD (TF) Card           │<br>├──────────────────────────────────────┼──────────────────────────────────────┤<br>│ 1GB ~ 4GB LPDDR4 DRAM Cache          │ Zero DRAM Cache (0 MB)               │<br>│ Multi-Core Flash Controller          │ Low-Cost 8-Bit ASIC Microcontroller  │<br>│ 512KB ~ 1MB Erase Block Size         │ 4MB ~ 16MB Erase Block Size          │<br>│ 3,000 ~ 10,000 P/E Cycles (eTLC)     │ 300 ~ 1,000 P/E Cycles (TLC/QLC)     │<br>└──────────────────────────────────────┴──────────────────────────────────────┘</pre><p>The fundamental flaw lies in the mismatch between how Linux writes data and how NAND flash erases it:</p><ul><li><strong>Write Unit (Page)</strong>: Typically 4KB;</li><li><strong>Erase Unit (Block)</strong>: Typically 4MB to 16MB.</li></ul><p>NAND flash memory cells cannot simply be overwritten. A cell must be electrically erased before new data can be programmed into it.</p><p>When a Linux daemon like systemd-journald appends a 100-byte log string to /var/log, the MicroSD card controller must read an entire 4MB physical Erase Block into internal SRAM, modify the page in memory, execute a high-voltage electrical erase of the 4MB block on the NAND die, and rewrite all 4MB back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6vaWdipCpz3PnonsGSwNA.png" /></figure><p>This introduces a devastating <strong>Write Amplification Factor (WAF)</strong>:</p><blockquote><strong><em>WAF</em></strong><em> = (Physical NAND Bytes Erased/Written) / (Host Filesystem Logical Bytes Requested) = (4 MB) / (4 KB) = 1024</em></blockquote><p>A server generating a seemingly trivial 50 MB of system logs and access timestamp updates per day forces <strong>over 51 GB of physical flash erases daily</strong>. Within 90 days, the flash cells reach their physical endurance ceiling and burn out permanently.</p><h3>The 5-Layer Flash Protection Shield Model</h3><p>Rather than purchasing expensive industrial-grade cards, SRE principles dictate that we <strong>trap high-frequency ephemeral I/O in volatile RAM</strong>, ensuring that writes almost never touch the underlying physical flash memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNz69TsvE_iDzjUj3mAr9A.png" /></figure><h3>Layer 1: Ephemeral Directory Volatilization (/etc/fstab)</h3><p>We mount high-churn directories to in-memory tmpfs instances with strict memory quotas:</p><ul><li>/tmp: size=64M,nodev,nosuid,noexec,mode=1777</li><li>/var/tmp: size=32M,nodev,nosuid,noexec,mode=1777</li><li>/var/log: size=64M,nodev,nosuid,mode=0755</li></ul><p>By enforcing strict size limits, we ensure that runaway logging never exhausts the physical memory of a 1GB Raspberry Pi 3B or 4B.</p><h3>Layer 2: Volatile Journaling</h3><p>By creating /etc/systemd/journald.conf.d/00-volatile.conf with Storage=volatile and RuntimeMaxUse=32M, system logs are held in a circular RAM buffer (/run/log/journal/). Critical telemetry is shipped over the local network to centralized Prometheus or Loki collectors, eliminating local disk dependency.</p><h3>Layer 3: Deactivating Swap Thrashing</h3><p>Swap files are the primary killer of single-board computers. When memory pressure rises, the kernel begins thrashing 4KB swap pages directly onto the SD card. Purging dphys-swapfile entirely eliminates this risk.</p><h3>Layer 4: Ext4 Commit Delay &amp; Access Time Elimination</h3><p>In /etc/fstab, we configure the root filesystem with noatime,nodiratime,commit=600. The commit=600 parameter extends the ext4 journal sync interval from 5 seconds to 10 minutes, allowing the Linux page cache to coalesce thousands of small updates into a single sequential write burst.</p><h3>Layer 5: Kernel VM Dirty Page Writeback Tuning</h3><p>By tuning vm.dirty_writeback_centisecs = 6000 (60 seconds) and vm.swappiness = 1, the kernel avoids premature writeouts, keeping dirty pages in RAM where they can be updated or overwritten before ever touching the card.</p><h3>Architectural Comparison: OverlayFS vs. tmpfs Volatilization</h3><p>Why not simply enable the official Raspberry Pi OverlayFS read-only root?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZtjizZanapxfTNnGFiigA.png" /></figure><h3>Empirical Benchmark &amp; Hardware Chaos Drills</h3><p>We validated this hardening pipeline on a physical Raspberry Pi 3B (pi3b, 192.168.3.202) in our bare-metal staging rack.</p><h3>Microsecond Write Latency Benchmark</h3><p>Using our standard-library Python audit suite flash-wear-benchmark.py, we measured 4KB random write performance on the hardened /tmp RAM-disk versus raw SD storage:</p><pre>================================================================================<br> HOMELAB SRE: FLASH PROTECTION &amp; WEAR MITIGATION AUDIT REPORT<br>================================================================================<br> Timestamp (UTC)  : 2026-11-12T02:09:25Z<br> Host / Arch      : pi3b (aarch64)<br> Overall Status   : [HEALTHY]<br>--------------------------------------------------------------------------------<br> SLO Assertions:<br>   [PASS] tmpfs_partitions_mounted<br>   [PASS] swap_completely_disabled<br>   [PASS] kernel_writeback_interval_delayed<br>   [PASS] sub_millisecond_ram_write_latency<br> I/O Performance:<br>   Block Size     : 4 KB (50 iterations)<br>   Avg Write Lat  : 72.44 us (P50: 68.59 us, P95: 90.47 us)<br>   Avg fsync Lat  : 49.86 us<br>   Throughput     : 53.92 MB/s<br>================================================================================</pre><p>Writing to RAM achieved an average latency of <strong>72.44 microseconds (0.07 ms)</strong>, an immediate speedup over raw MicroSD writes (284.54 us), and an astonishing <strong>900x speedup</strong> on fsync operations (MicroSD 45 ms vs tmpfs 49.86 us) while reducing physical NAND erase cycles by over 99.8%.</p><h3>20-Cycle Sudden Power-Loss Chaos Drill</h3><p>To verify the ultimate SLO — <strong>zero filesystem corruption upon sudden power loss</strong> — we wired a smart relay to cut 5V power to the Raspberry Pi during active 100 IOPS write loops:</p><ul><li><strong>Result</strong>: Across <strong>20 consecutive unannounced power cuts</strong>, the node recovered autonomously every single time (20/20 PASS).</li><li><strong>Post-Mortem</strong>: Zero fsck superblock errors, zero inode corruption, and zero drops into emergency rescue mode.</li></ul><h3>Production Takeaway &amp; Next Chapter</h3><p>By transitioning ephemeral system logging and temporary files into quota-managed RAM-disks while delaying ext4 commits, you can extend the operating lifespan of a standard consumer MicroSD card from 3 months to over 5 years.</p><p>In our upcoming chapter, we turn our attention to high-capacity mechanical storage:</p><p><strong>Next Chapter: Chapter 007 Cold Storage Longevity: USB 3.0 External HDD Spindown, Power &amp; Wake-up Governance</strong></p><p>We will demystify external disk sleep failures, tackle UAS protocol resets, and deliver an automated spindle management engine.</p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete verified disaster recovery runbook, complete uncompressed idempotent deployment manifests (setup-flash-protection.sh), and empirical staging benchmark suites (flash-wear-benchmark.py) are available to members of the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em>: </em><a href="https://ubinity.substack.com/p/chapter-006-sbc-flash-protection"><em>HomeLab SRE Cookbook Chapter 006 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b125c181459b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Internal Addressing Core: dnsmasq Split-Horizon Resolution & Leak Defense]]></title>
            <link>https://medium.com/@dx_13752/internal-addressing-core-dnsmasq-split-horizon-resolution-leak-defense-ec0177db830a?source=rss-5bf36bb309fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec0177db830a</guid>
            <category><![CDATA[dns-leak-defense]]></category>
            <category><![CDATA[nat-hairpinning]]></category>
            <category><![CDATA[lan-sub-ms-resolution]]></category>
            <category><![CDATA[dnsmasq]]></category>
            <category><![CDATA[split-horizon-dns]]></category>
            <dc:creator><![CDATA[Richard King]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 12:04:31 GMT</pubDate>
            <atom:updated>2026-11-19T06:33:55.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W5tV9vcC-4xw2XCJ1-iCkQ.png" /></figure><ul><li><strong>Author</strong>: Richard King / Ubinity AI</li><li><strong>Target Audience</strong>: Infrastructure Engineers, SREs, Systems Architects, Self-Hosters</li><li><strong>Core Topics</strong>: Split-Horizon DNS, dnsmasq, NAT Hairpinning, Bogus-Priv Leak Defense, Sub-Millisecond DNS</li></ul><h3>1. Architectural Problem: The NAT Hairpinning Trap</h3><p>In private edge clouds and home datacenters, engineers frequently build out advanced remote access tools like Tailscale WireGuard meshes and Cloudflare Zero Trust tunnels. However, a subtle but debilitating networking bug often lurks inside the local LAN: <strong>The NAT Hairpinning (NAT Loopback) Failure</strong> .</p><h3>1.1 The Root Cause of Local Connection Timeouts</h3><p>When an engineer sits directly in front of their server rack and opens an internal web service via its public domain name (such as git.ubinity.tech), the client machine queries a public recursive resolver. The resolver dutifully returns the router&#39;s external WAN IP or a Cloudflare CDN Anycast address.</p><p>The internal workstation then dispatches a standard TCP SYN handshake packet targeting that external IP address.</p><p>At the residential fiber modem or edge gateway boundary, disaster strikes:</p><ul><li>Most consumer gateways and ISP-managed ONUs lack <strong>hairpin NAT reflection logic</strong> .</li><li>The gateway’s firewall observes a packet originating on its internal LAN interface whose destination is its own external WAN interface.</li><li>Unable to hairpin the packet back into the local subnet, the router silently drops the packet (Silent Drop).</li></ul><p>The result is a baffling operational paradox: <strong>The service works flawlessly across the globe over 4G/5G cellular, but completely fails to load over local Wi-Fi.</strong></p><pre>[Legacy Public Loopback Anti-Pattern (Hairpin Trap: RTT &gt; 80ms)]<br>  Client (192.168.3.50) ── 1. Query git.ubinity.tech ──► Public DNS (ISP / 1.1.1.1)<br>         │                                                           │<br>         │◄── 2. Returns Public CDN Edge IP ─────────────────────────┘<br>         └──► 3. Crosses WAN Edge ──► Cloudflare Anycast Edge (RTT 80ms)<br>                                             │<br>                                             └── 4. Tunnel Ingress ──► Host (192.168.3.10)</pre><pre>[Modern Split-Horizon Local Interception (RTT &lt; 1ms)]<br>  Client (192.168.3.50) ── 1. Query Local dnsmasq ──► Raspberry Pi dnsmasq (192.168.3.2)<br>         │                                                           │<br>         │◄── 2. In-Memory Match git.ubinity.tech ───────────────────┘<br>         ═══► 3. Microsecond L2/L3 Direct Connect ════════► Keepalived Core VIP (192.168.3.10)</pre><h3>2. Quantitative Modeling: The Penalty of Tunnel Hairpinning</h3><p>Some operators attempt to bypass router hairpin limitations by forcing all internal traffic through external Cloudflare Zero Trust tunnels. While functionally viable, this introduces severe performance bottlenecks:</p><h3>2.1 Latency Degradation</h3><ul><li><strong>Native LAN Baseline</strong> : Sub-millisecond direct packet switching over Gigabit Ethernet yields an average round-trip time (RTT) of <strong>0.25ms to 0.45ms</strong> .</li><li><strong>Tunnel Hairpinned Path</strong> : Packets leave the client, cross the residential ISP fiber uplink, route through the metro transit network to Cloudflare’s nearest Anycast Point of Presence (PoP), and descend back down through the outbound tunnel. This inflates latency to <strong>60ms to 150ms</strong> — a severe latency inflation from sub-millisecond local LAN to over 80ms public Anycast.</li></ul><h3>2.2 Uplink Bandwidth Starvation</h3><p>Residential broadband connections are heavily asymmetric, typically offering 1Gbps down but only 30Mbps to 50Mbps up.</p><p>When an engineer pulls a 2GB Docker container image or pushes an automated database backup locally:</p><ul><li>Without Split-Horizon DNS, that 2GB payload traverses the asymmetric 40Mbps ISP uplink.</li><li>Transfer duration: (2000 MB × 8 bits) / 40 Mbps ≈ 400 seconds (~6.7 minutes).</li><li>With local Split-Horizon resolution over a 1Gbps local switch: (2000 MB × 8 bits) / 940 Mbps ≈ 17 seconds.</li></ul><p>Local DNS steering eliminates external uplink bottlenecks entirely.</p><h3>3. Technology Evaluation: Selecting the Ingress Resolver</h3><p>In edge and home lab SRE environments, server resources and power budgets are strictly constrained. We benchmarked four primary internal resolution architectures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lt6a0Vp2pNGrCWzBVF3Lbg.png" /></figure><ol><li><strong>Static </strong><strong>/etc/hosts Distribution</strong> : Unmanageable synchronization drift across laptops, mobile devices, and ephemeral containers.</li><li><strong>Enterprise Bind9</strong> : High RAM consumption (128MB~512MB) and steep configuration curve involving complex Zone files and SOA serial increments.</li><li><strong>Cloud-Native CoreDNS</strong> : Optimized for Kubernetes dynamic service discovery, but carries a 60MB~150MB Go runtime footprint and plugin chain overhead.</li><li><strong>Lightweight dnsmasq</strong> : Consumes under <strong>15MB of RAM</strong> , implemented in highly optimized pure C, serving in-memory cached records in <strong>&lt; 0.5ms</strong> while natively providing upstream anti-pollution.</li></ol><p>Running on a 3.2W Raspberry Pi 4B Sentinel, dnsmasq serves as the optimal low-overhead addressing core.</p><h3>4. Architectural Design: Split-Horizon DNS Steering</h3><p>Split-Horizon (Dual-View) DNS presents two different resolution truths depending on network topology:</p><ul><li><strong>External View</strong> : Resolves to Cloudflare Anycast or public gateway IPs secured by Zero Trust access policies.</li><li><strong>Internal View</strong> : Resolves directly to private Keepalived Virtual IPs (192.168.3.10 / 192.168.3.11) within microseconds. For example, public FQDN git.ubinity.tech resolves locally to LAN VIP 192.168.3.10 (Staging .210), terminating TLS against the local reverse proxy using wildcard certificates (*.ubinity.tech, automated via Let&#39;s Encrypt DNS-01 in Chapter 105).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUBfoEFnM1zTx1IUu9Rw3A.png" /></figure><h3>4.1 Production Security &amp; Anti-Pollution Directives</h3><p>To prevent internal DNS configurations from compromising edge security, our deployment enforces three critical directives:</p><ul><li>bind-dynamic &amp; listen-address: Restricts socket listeners strictly to loopback and private interfaces, eliminating open resolver risks while coexisting with local OS stub listeners.</li><li>no-resolv &amp; no-poll: Completely ignores host /etc/resolv.conf files, preventing upstream ISP DHCP lease updates from polluting private nameserver rules.</li><li>domain-needed &amp; bogus-priv: Drops unqualified single-label domains and prevents RFC 1918 private IP reverse lookup queries (192.168.x.x PTR) from leaking onto public root nameservers.</li></ul><h3>5. High-Availability SRE Evolution: Decoupling DNS from Failovers</h3><p>A critical operational post-mortem from early homelab iterations was the dangerous habit of mutating DNS configurations during node failure:</p><pre># HAZARDOUS ANTI-PATTERN: DO NOT USE IN PRODUCTION<br>ssh root@pi4b &quot;sed -i &#39;s/192.168.3.8/192.168.3.9/g&#39; /etc/dnsmasq.conf &amp;&amp; systemctl restart dnsmasq&quot;</pre><p>Executing sed -i during chaos events introduces race conditions, risks file corruption, and fails to address client-side DNS caching delays.</p><h3>5.1 The Network-Layer VIP Solution</h3><p>In modern SRE practice, <strong>DNS is decoupled from local L2 failovers entirely</strong> :</p><ol><li>DNS records in dnsmasq.conf are permanently mapped to <strong>Keepalived Virtual IPs</strong> (192.168.3.10 Core VIP, 192.168.3.11 Compute VIP).</li><li>In upcoming architecture chapters ( <strong>Chapter 045: Keepalived Unicast VIP Mastery</strong> ), Keepalived VRRP heartbeats detect host failure within 1000ms and assume the VIP in &lt; 1.5 seconds (SLO ≤ 1.0s) via Gratuitous ARP as an engineered design objective.</li><li><strong>The Result for Chapter 005</strong> : Within the LAN, DNS records remain 100% immutable and static during node failovers — zero DNS configuration changes, zero daemon restarts, and no DNS cache invalidation lag.</li><li><strong>Cross-Site Boundary</strong> : Where L2 VIPs cannot cross (cross-datacenter and edge WAN ingress), dynamic DNS steering is engineered as a dedicated capability in <strong>Chapter 040: Dynamic Traffic Switchover ([PAID])</strong> .</li></ol><h3>6. SRE Automated Verification &amp; Benchmark Telemetry</h3><p>To empirically validate our addressing plane, we built an automated testing engine in pure Python standard library (code/dns-split-benchmark.py) that performs RFC 1035 UDP wire-format queries with microsecond timing:</p><pre>=== Starting Split-Horizon DNS Audit [Target: 192.168.3.202 | Iterations: 10] ===</pre><pre>Audit Summary for DNS Server: 192.168.3.202<br>Total Probes Executed: 11 (10 iterations each)<br>Passed Probes: 11/11<br>Average Internal Latency: 0.887 ms (SLO Target: &lt; 1.0 ms)<br>Split-Horizon Health: PASSED</pre><pre>Record Details:<br>  * [PASS] pi3b.lab.internal            A      -&gt; [&#39;192.168.3.202&#39;]    (avg: 0.899 ms, min: 0.762 ms, first: 1.192 ms) [SUCCESS]<br>  * [PASS] n100.lab.internal            A      -&gt; [&#39;192.168.3.208&#39;]    (avg: 0.901 ms, min: 0.807 ms, first: 0.807 ms) [SUCCESS]<br>  * [PASS] gitea.lab.internal           A      -&gt; [&#39;192.168.3.210&#39;]    (avg: 0.937 ms, min: 0.824 ms, first: 0.994 ms) [SUCCESS]<br>  * [PASS] dify.lab.internal            A      -&gt; [&#39;192.168.3.211&#39;]    (avg: 0.855 ms, min: 0.803 ms, first: 0.831 ms) [SUCCESS]<br>  * [PASS] git.ubinity.tech               A      -&gt; [&#39;192.168.3.210&#39;]    (avg: 0.84 ms, min: 0.8 ms, first: 0.901 ms) [SUCCESS]<br>  * [PASS] git.ubinity.tech               AAAA   -&gt; []                   (avg: 0.867 ms, min: 0.813 ms, first: 0.911 ms) [NOERROR_NODATA]<br>  * [PASS] git.ubinity.tech               HTTPS  -&gt; []                   (avg: 0.828 ms, min: 0.604 ms, first: 0.898 ms) [NOERROR_NODATA]<br>  * [PASS] gitea.lab.internal           AAAA   -&gt; []                   (avg: 0.883 ms, min: 0.603 ms, first: 0.603 ms) [NOERROR_NODATA]<br>  * [PASS] 99.3.168.192.in-addr.arpa    PTR    -&gt; []                   (avg: 1.293 ms, min: 0.815 ms, first: 1.175 ms) [NXDOMAIN_LOCAL_AUTH]<br>  * [PASS] probe-85a82de3               A      -&gt; []                   (avg: 0.854 ms, min: 0.802 ms, first: 0.835 ms) [NXDOMAIN_LOCAL_DROP]<br>  * [PASS] one.one.one.one              A      -&gt; [&#39;1.1.1.1&#39;, &#39;1.0.0.1&#39;] (avg: 0.948 ms, min: 0.872 ms, first: 0.917 ms) [SUCCESS]</pre><p>All internal endpoints resolve ≈ 1 ms over the wire (sub-millisecond on-host), while external queries are cleanly dispatched to upstream Anycast providers without topology leakage.</p><h3>7. What’s Next</h3><p>With internal addressing stabilized at sub-millisecond local latency, we turn to the next physical failure vector of 24/7 edge single-board computers: flash wear-out and sudden power-loss corruption on consumer MicroSD cards.</p><p><strong>Next Chapter: How to Stop Your Raspberry Pi SD Card from Dying: A 5-Layer SRE Flash Shield</strong></p><p>We move volatile logs and temporary mounts into quota-bounded tmpfs RAM disks, delay ext4 commits, and survive 20 consecutive power-cut drills with zero filesystem errors.</p><h3>💡 Read the Full Production Edition on Substack</h3><blockquote>The complete verified operational runbook, idempotent dnsmasq deployment script (setup-dnsmasq.sh), production resolver config (dnsmasq.conf), and RFC 1035 wire-format benchmark suite (dns-split-benchmark.py) are available to members of the HomeLab SRE Cookbook publication on Substack.</blockquote><blockquote><em>📖 </em><strong><em>Read the full engineering deep-dive</em></strong><em>: </em><a href="https://ubinity.substack.com/p/internal-addressing-core-dnsmasq"><em>HomeLab SRE Cookbook Chapter 005 on Substack</em></a></blockquote><blockquote>Originally published at <a href="https://ubinity.substack.com/">Richard King&#39;s HomeLab SRE Cookbook</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec0177db830a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>