<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by JJozzieTech on Medium]]></title>
        <description><![CDATA[Stories by JJozzieTech on Medium]]></description>
        <link>https://medium.com/@jjozzietech?source=rss-197f6d210c24------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RT1lN4u_5BizRh2TOX7JXw.png</url>
            <title>Stories by JJozzieTech on Medium</title>
            <link>https://medium.com/@jjozzietech?source=rss-197f6d210c24------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:00:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jjozzietech/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From static to OSPF without taking the service down]]></title>
            <link>https://medium.com/@jjozzietech/from-static-to-ospf-without-taking-the-service-down-001cd034574f?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/001cd034574f</guid>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[ospf]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 20:15:10 GMT</pubDate>
            <atom:updated>2026-11-18T19:09:24.243Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z-T9NpoBaDPteIfx.png" /></figure><p>Static routes work — until they don’t. In a multi-VRF NX-OS fabric with an OPNsense border firewall as the WAN edge, a single dead next-hop means a dead service. With an external ISP link (Optus) being decommissioned and the last backup path disappearing with it, I needed something more resilient for the storage VRF — the network segment carrying my DePIN storage node.</p><p>This is a phased OSPF migration across three devices — two NX-OS Top-of-Rack switches and an OPNsense border firewall running FRR — executed live, with the storage node serving active sessions the whole time.</p><h3>// the topology</h3><p>The storage VRF runs on its own VLAN (illustrative 192.168.x.0/24). Prior to this change, the routing model was purely static:</p><ul><li>TOR-B held a single default static pointing to the border firewall via the TOR-B↔firewall transit (illustrative /30)</li><li>TOR-A held a cross-TOR static pointing to TOR-B</li><li>The border firewall held kernel statics for the storage subnet and the TOR-A↔TOR-B transit, pointing back through TOR-B</li></ul><p>One physical path. One point of failure. The Optus link had been providing a last-resort backup — once it went, there would be nothing.</p><h3>// the fix</h3><p>The fix was a second physical uplink from TOR-A directly to the border firewall on ix0, tagged on a second transit VLAN (illustrative /30), combined with OSPF process 10 across all three devices. I followed a phased template I&#39;d developed on prior VRF changes:</p><ul><li><strong>P1</strong> — build the TOR-A↔firewall link and verify L3 reachability</li><li><strong>P2</strong> — enable OSPF on all three devices alongside existing statics (observation window)</li><li><strong>P3</strong> — reduce hello/dead timers to 1s/4s for fast failover</li><li><strong>P4</strong> — float the TOR statics to AD 250 (OSPF becomes active path)</li><li><strong>P5</strong> — remove the firewall’s kernel statics and save configs (OSPF owns the full routing table)</li></ul><h3>// P1-P2: building the foundation</h3><p>P1 was straightforward: NX-OS subinterface on the firewall-facing uplink, OPNsense VLAN interface on ix0, firewall rules permitting the storage VRF&#39;s transit prefixes. A ping test confirmed 5/5 at ~1ms before proceeding.</p><p>P2 brought OSPF up on all three devices simultaneously — distinct router-ids per device (illustrative 10.x.x.1 on TOR-B, 10.x.x.9 on TOR-A, 10.x.x.6 on the border firewall). The key discipline here: statics were left untouched. OSPF ran in parallel, forming Full adjacencies on both TOR-firewall links and the TOR-A↔TOR-B transit, while all traffic continued to forward via the existing statics. That gave a clean observation window - OSPF converged, LSAs exchanged, default route type-5 LSA from the border firewall visible in both TOR databases - before any forwarding change was made.</p><h3>// P3: fast timers and a shell gotcha</h3><p>I applied the timer reduction to 1s hello / 4s dead firewall-first, then both TORs in immediate succession. The sequencing matters: once the firewall drops its dead-interval to 4 seconds, any TOR still sending 10-second hellos will be declared dead within 4 seconds of the next missed hello.</p><p>In practice I applied the TOR changes fast enough that no adjacency flap occurred — the Up Time on all neighbours continued uninterrupted through P3. The theoretical risk window was sub-4 seconds; the actual outcome was zero disruption.</p><p>One operational note for anyone running FRR on OPNsense (FreeBSD): the vtysh -c &#39;...&#39; multiline quoting form does not work in the FreeBSD shell. Single-quoted multiline strings are parsed by csh/tcsh, which rejects them. The correct pattern is chained -c flags on a single line:</p><pre>vtysh -c &#39;configure terminal&#39; -c &#39;interface vlanXX&#39; -c &#39;ip ospf hello-interval 1&#39; -c &#39;ip ospf dead-interval 4&#39; -c &#39;end&#39; -c &#39;write memory&#39;</pre><h3>// P4-P5: the forwarding cutover</h3><p>Floating the statics to AD 250 at P4 was the actual forwarding cutover. With OSPF already converged at AD 110, removing the AD 1 statics caused both TORs to immediately prefer the OSPF-learned default. The cutover was effectively instantaneous — OSPF had been sitting ready for the duration of the P2 observation window.</p><p>To be precise about the “zero-blip” characterisation: the forwarding plane change at P4 was zero-blip. OSPF was already converged; the static removal was atomic. P3 carried a theoretical sub-4s adjacency risk that did not materialise. The claim holds for the forwarding plane; it should be understood as “no measured disruption” rather than a mathematically guaranteed zero.</p><p>At P5 I removed the firewall’s kernel statics for the storage subnet and the second transit, and saved configs. Post-removal, both prefixes were confirmed OSPF-selected with ECMP across both uplinks — exactly the redundancy the whole exercise was designed to achieve.</p><h3>// the result</h3><p>The storage VRF now has:</p><ul><li>Dual OSPF paths to the border firewall — ECMP load-sharing under normal conditions</li><li>Automatic failover on either TOR-firewall link loss — convergence up to 4 seconds by design (1s hello, 4s dead-interval); a measured figure will be captured in a controlled failover test</li><li>Floating statics at AD 250 as a silent safety net if OSPF fails entirely</li><li>The storage node serving active sessions throughout — zero session interruptions recorded in the storage node’s own session log across the migration window</li></ul><h3>// what I’d make instinctive</h3><p>Four things from this.</p><p><strong>Run OSPF alongside statics before cutting over.</strong> The P2 observation window — OSPF converged but statics still forwarding — is not optional. It confirms adjacencies, LSA propagation, and the default route type-5 before any forwarding change is made. Skipping it trades safety for speed and gains nothing.</p><p><strong>NX-OS </strong><strong>vrf member silently removes all L3 config on an SVI.</strong> Adding vrf member to an existing SVI in NX-OS drops every IP address and routing statement on that interface without warning. Always build subinterfaces or SVIs with the VRF assignment as the first command, before any L3 config.</p><p><strong>Timer sequencing is firewall-first, TORs immediately after.</strong> Apply fast timers to the device with the shorter dead-interval first — in this case the border firewall. Then apply to both TORs as quickly as possible. The window between firewall update and TOR update is a live risk interval; keep it under 4 seconds.</p><p><strong>The floating static is not dead weight.</strong> AD 250 statics provide a recovery path if the OSPF process fails for any reason — misconfiguration, FRR crash, process restart. They cost nothing in normal operation and provide a last-resort forwarding path. Keep them.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/static-to-ospf-no-downtime/"><em>https://jjozzietech.com.au</em></a><em> on November 11, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=001cd034574f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why I tested from inside the network]]></title>
            <link>https://medium.com/@jjozzietech/why-i-tested-from-inside-the-network-1f0198b980ef?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f0198b980ef</guid>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[isp]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[homelab]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 20:15:22 GMT</pubDate>
            <atom:updated>2026-11-05T19:12:03.622Z</atom:updated>
            <content:encoded><![CDATA[<p>The Internet upgrade was one of the final steps of my rack and network migration. An Aussie Broadband 2000/500 Mbps service had to work through the infrastructure actually carrying production traffic — not merely show a multigigabit link at the firewall. The path included a virtual machine, a virtualisation bridge, aggregated host uplinks, a redundant switching fabric, segmented routing, a firewall and the broadband handoff.</p><p>The question was not whether Aussie Broadband had sold me a 2 Gbps tier. It was whether a workload inside the rack could reach that tier, and what the measurements did — and did not — prove.</p><h3>// what a speed claim actually measures</h3><p>The contracted 2000/500 Mbps tier, the negotiated Ethernet interface rate, and application-level speed-test throughput describe different things. A 2.5GbE WAN interface can provide sufficient physical link capacity without guaranteeing an application 2 Gbps. Equally, a single lower test result does not establish an ISP shortfall.</p><p>My engineering report records that the upgraded service was active on 15 October 2026. A 24 October firewall interface capture shows the broadband-facing interface active at 2500Base-T full duplex. The physical media mode exceeds the nominal downstream service rate. This is necessary capacity, not a throughput measurement by itself.</p><h3>// the path I measured</h3><p>I ran the final benchmarks from a production service VM on a virtualisation host, rather than from the firewall console. Host networking used an IEEE 802.3ad bond built from two 10GbE links, traversing the switching and routed security edge to a 2.5GbE WAN. According to the source report, no 1GbE interface was present on the measured path. The test was conducted while the production compute platform was active; no service impact was recorded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*25vc5miEK0sAnY_i.png" /></figure><p><em>Figure 1. Generic logical path for the production-VM throughput tests. It deliberately omits product names, addressing and VLAN identifiers. This is a forwarding-path representation, not a diagram of active/standby member selection or a failover validation.</em></p><h3>// nine runs across three networks</h3><p>On 19 October 2026 between 06:03 and 06:06 AEDT, I ran Ookla Speedtest CLI 1.2.0 from that VM. Three Sydney networks — Telstra, Superloop and Optus — were tested three times each. The tests were sequential, interleaved across endpoints, and separated by approximately five seconds. Ookla’s default multi-connection method was used, with JSON results retained for later review.</p><p>A tenth test to Aussie Broadband’s own Sydney server was deliberately excluded from the nine-run off-net statistics, avoiding reliance on a single on-net measurement.</p><pre>Metric            Nine off-net runs  What it means<br>----------------  -----------------  ---------------------------------<br>Download median   2,069 Mbps         Measured from the production VM<br>Upload median     535 Mbps           Measured from the production VM<br>Idle ping median  2.57 ms            Idle latency, not loaded latency<br>Download range    2,013–2,077 Mbps   All nine runs exceeded 2,000 Mbps<br>Upload range      533–542 Mbps       All nine runs exceeded 500 Mbps</pre><p>Both bandwidth medians were calculated independently from the nine off-net runs. The recorded throughput was consistent with delivery of the advertised tier under those test conditions. These measurements do not establish sustained all-day throughput or performance to every Internet destination.</p><h3>// the earlier 936 Mbps result</h3><p>An earlier firewall-local iperf3 reverse-direction test returned 936 Mbps at the receiver; its corresponding upload receiver figure was 455 Mbps. That client was the firewall itself, testing against a single remote server. This is a materially different test from a multi-connection speed test initiated on a production VM and repeated across three external networks.</p><p>The earlier result resembles a throughput ceiling often encountered around a 1GbE segment, but I have no proof that a 1GbE link caused it. Remote-server limits, stream behaviour and other test-path differences remain possible. In particular, the available evidence does not identify the limiting hop. I retain the 936 Mbps result as a useful diagnostic lesson, not as a resolved fault.</p><h3>// latency belongs next to throughput</h3><p>Bandwidth results were strong across all three test networks, but loaded latency was not uniform. The three Superloop download tests recorded 50.2, 55.2 and 44.4 ms of loaded latency, despite idle ping near 2.5 ms. The Telstra and Optus download-loaded values remained below 5 ms in these runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_EmYJPGbTJ1tdZMf.png" /></figure><p><em>Figure 2. Download-loaded latency, three sequential runs per ISP (Ookla IQM metric). An endpoint-specific observation, not proof of the location or cause of congestion.</em></p><p>The reported packet-loss field was 0% for Telstra and Superloop runs. Optus did not report a loss value; it would be inaccurate to claim 0% loss for all nine runs. Likewise, the latency pattern cannot be attributed conclusively to queueing, shaping, peering or equipment without further path-specific measurements.</p><h3>// what the result proves — and what it does not</h3><p>The production traffic path delivered more than 2 Gbps downstream and 500 Mbps upstream to the selected off-net test endpoints, with a WAN physical negotiation of 2.5GbE. I could validate that result without removing the workloads from the operational network. That is materially stronger evidence than a firewall-only speed test.</p><p>The test does not establish per-hop headroom, firewall CPU consumption, queue depth, application experience at peak times, or failover convergence. Firewall CPU and state-table data were not collected during these runs. Cumulative interface counters showed no reported errors or drops at the later capture, but they do not isolate the benchmark window. Separate routing/failover tests belong to separate verification records.</p><h3>// the engineering takeaway</h3><p>For a multigigabit upgrade, keep the chain of evidence intact: confirm the commercial service state, verify physical negotiation, identify every link on the end-to-end path, test from a real workload, use multiple capable external endpoints, retain machine-readable outputs, and interpret loaded latency separately from idle ping. When one test disagrees with the others, document its context before declaring a bottleneck.</p><p>The upgrade mattered because the network — not just the WAN socket — could carry the new service rate. The useful result was the evidence trail: a named test path, repeatable measurements, explicit exclusions and honest boundaries around what remained unverified.</p><h3>// method and evidence note</h3><p>Measurement source: my validation record dated 24 October 2026, covering tests executed 19 October. Ookla Speedtest CLI 1.2.0, three sequential tests each to Telstra, Superloop and Optus Sydney endpoints; one provider on-net test excluded. Raw JSON and a summary table are retained in internal operations records. Earlier firewall-local iperf3 results are from a separate terminal capture. Internal IP addresses, public WAN addresses, machine identifiers and result URLs have been intentionally omitted.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/2gbps-wan-validation/"><em>https://jjozzietech.com.au</em></a><em> on November 4, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f0198b980ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Six months operating an Akash provider]]></title>
            <link>https://medium.com/@jjozzietech/six-months-operating-an-akash-provider-2dd164305c9c?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dd164305c9c</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[homelab]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 20:15:06 GMT</pubDate>
            <atom:updated>2026-10-29T19:15:39.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LZk6YccRcnOY1lyY.png" /></figure><p>In May the provider missed a mandatory upgrade deadline, lost its audit attestation, and watched roughly four-fifths of its active leases disappear inside two weeks.</p><p>The recovery took twenty-six days. Most of that time was not spent on the upgrade. It was spent on things that had been quietly broken for months — including one bug that had been suppressing all storage reporting for sixty-seven days without producing a single log line.</p><p>That turned out to be the pattern. Every expensive incident in the six months since has had the same shape: nothing failed loudly. Something simply stopped happening, and nothing said so.</p><h3>// what this covers</h3><p>May to October 2026, on a production Akash provider running in a Sydney enterprise lab. The provider is older than that, but the documented record starts in May, and I would rather write six months I can evidence than twelve I can’t.</p><p>Specific numbers where I have them. No numbers where I don’t. Nothing here is reconstructed from memory — it comes from after-action reviews written at the time, a daily health check that has been running throughout, and the chain.</p><h3>// the deadline</h3><p>The provider ran a mandatory upgrade deadline on 1 May and missed it. Audit attestation was removed. Lease count began falling immediately and was still falling two weeks later.</p><p>The first recovery attempt lasted about forty minutes. It stopped at the ingress chart install with a version constraint:</p><pre>chart requires kubeVersion: &gt;= 1.31.0-0 <br>which is incompatible with Kubernetes v1.28.15</pre><p>The new ingress controller needed Kubernetes 1.31 or later. The cluster was on 1.28.15, which was already end-of-life. There was no flag to pass, no version to pin back to, no way through.</p><p>Recovering the audit meant upgrading Kubernetes first. Which meant the cluster had to survive six minor-version jumps. Which meant it needed a control plane that could lose a node, and backups that existed.</p><p>None of those were true.</p><h3>// what the deadline exposed</h3><p>The honest inventory, severity-ordered, as it stood when the recovery started:</p><p><strong>No etcd backups.</strong> None. A single disk failure on the control plane’s data directory would have meant total cluster state loss. This had been true for sixty-seven days.</p><p><strong>One control plane.</strong> Any failure of that one node took the whole cluster down. Also true for sixty-seven days and counting.</p><p><strong>Kubernetes end-of-life.</strong> No security patches, and the hard blocker on the ingress controller that stopped the first attempt.</p><p><strong>Storage reporting empty.</strong> The provider could not bid on any order requiring persistent storage, because the inventory operator reported no storage capacity at all. Known for sixty-seven days, investigated repeatedly, unresolved.</p><p>The deadline did not create any of this. It made it unavoidable. That is roughly the only good thing to say about missing a deadline: it converts deferred work into work.</p><h3>// foundation</h3><p>Backups first, because everything after it was going to be destructive.</p><p>An etcd snapshot script on a six-hour cron, writing locally and copying off-host, with seven-day retention. Then — and this is the part people skip — a restore tested end to end into a scratch directory, because an untested backup is a belief rather than a backup.</p><p>Then two new control-plane VMs, prepped on two separate physical hosts. Not two total: three. A two-member etcd cluster tolerates zero failures <em>and</em> introduces split-brain risk on a network partition, because either member can believe it holds quorum when it can’t see the other. Two members is strictly worse than one. The decision that followed from that — join both new control planes in a single session rather than splitting across two mornings — is the one I would most want another operator to take away from this.</p><p>The backups paid for themselves within forty-eight hours, not as a recovery but as permission: every subsequent risky step had a fresh snapshot behind it and therefore did not need to be agonised over.</p><h3>// the hardware interrupt</h3><p>Three days into the plan, one of the physical hosts bricked. Cause never fully diagnosed — the decision to swap the chassis came before the diagnosis did, because the migration was the priority and the host was replaceable.</p><p>Ten days of pause while a replacement was sourced and racked.</p><p>One thing came out of it worth keeping: the prepared control-plane VM survived the host swap intact. Its disks moved to the new chassis and it booted with all the preparation work from a week earlier still in place. That pattern — prep the VM, treat the host as fungible — has been reusable every time hardware has moved since.</p><h3>// measuring the failover</h3><p>Three control planes joined, etcd at three members, virtual IP with three-way failover. On paper, high availability.</p><p>On paper is not evidence. So the next session was spent deliberately breaking it: hard-stopping each control plane in turn at the hypervisor, with no graceful shutdown, and measuring what actually happened.</p><pre>Test                          API outage     Quorum       Recovery<br>----------------------------  -------------  -----------  -----------<br>Primary control plane killed  ~0s observed   2 of 3 held  clean, ~90s<br>Second control plane killed   ~8s            2 of 3 held  clean, ~90s<br>Third control plane killed    ~4s            2 of 3 held  clean, ~90s</pre><p>Under ten seconds in every case. Tenant workloads were unaffected throughout.</p><p>Two things worth noting. The first test’s apparent zero-second outage was a monitoring artefact — the monitor was running somewhere affected by the test. Where you put the observer matters as much as what you’re observing.</p><p>And hard-stopping at the hypervisor produces a sharper failure than stopping the service. A graceful shutdown lets everything deregister politely. That is not the failure you are worried about.</p><p>I have not seen these numbers published for a Kubernetes cluster this size. They are better than I expected.</p><h3>// six minor versions in a week</h3><p>The plan allowed one morning per Kubernetes minor version with soak days in between. Under high availability it was so uneventful that the upgrade ran past its own target and kept going — six minor jumps inside a calendar week, control planes one at a time, workers in batches, cluster never below two functioning control planes.</p><p>The lesson is narrower than “Kubernetes upgrades are easy.” It is that the upgrade was never the risk. The risk was doing it without a control plane that could lose a member. Once that was true, the thing everyone dreads became routine.</p><h3>// sixty-seven days</h3><p>Then the audit submission needed persistent storage, and the storage problem was still there.</p><p>The symptom: the inventory endpoint returned an empty storage array. Every other signal said storage was fine. The Ceph cluster was healthy with 19 TiB raw. Persistent volume claims bound normally when tested. The StorageClass carried both documented annotations. The provider simply declined every order that asked for persistent storage, reporting no capacity.</p><p>What had been tried across those sixty-seven days: adjusting the annotations, restarting the inventory operator, downgrading it, removing a broken liveness probe, and a complete provider version migration undertaken partly on the assumption that it would resolve. It did not. A post in the community provider channel got no answer.</p><p>What broke it was cloning the provider source at the exact version deployed and reading the inventory operator’s storage aggregation.</p><p>The code discovers every StorageClass, builds a map, and then filters that map before reporting. The filter tests a boolean read from a <strong>bare-key label</strong> on the StorageClass. Not one of the two documented annotations — those are downstream metadata and gate nothing. A different key, a different metadata type, same namespace.</p><p>The StorageClass had both annotations and not the label. So the operator discovered the class, tracked it, and dropped it during aggregation. No error. No warning. Not even a debug line. Just an empty array.</p><p>The fix was one command:</p><pre>kubectl label storageclass &lt;name&gt; akash.network=true</pre><p>Five seconds later the inventory endpoint reported full pool capacity. Within one bid cycle the provider was bidding on persistent-storage orders again. The audit was submitted the next day and passed all seven benchmarks.</p><p>Sixty-seven days. One label. Roughly two hours of actually reading the source.</p><p>The lesson I wrote down at the time was: <em>always read the actual source code when a known issue persists across versions.</em> The honest version is the note I added underneath it — <strong>should have done this sixty days earlier.</strong></p><p>That same bug has since produced two upstream contributions: a <a href="https://github.com/akash-network/support/issues/645">documentation issue</a> arguing the fix should be reachable from the symptom rather than only from the install sequence, and a <a href="https://github.com/akash-network/support/issues/663">later issue</a> about the silence itself. The operator still drops unlabelled storage classes without saying so.</p><h3>// hardening</h3><p>The three months after the audit recovery were spent on things the recovery had exposed as overdue.</p><p>Certificates moved to automated renewal — both the provider certificate and the wildcard ingress certificate, with a file-watcher reload path so renewal does not restart the provider. That one has <a href="https://jjozzietech.com.au/wildcard-cert-renewal-dns-alias/">its own write-up</a> and I won’t retell it here, but the failure mode behind it is worth stating plainly because it recurs later in this piece: chart defaults silently reassert themselves on every upgrade unless explicitly overridden. That is now a documented pre-flight check rather than something to remember.</p><p>Anti-mining enforcement was formalised into three layers and published. The <a href="https://github.com/jjozzietech/akash-miner-blacklist">blacklist</a> that drives it is public, with its evidence standards written before the first entry rather than after the first dispute, and with an honest provenance note about which entries carry per-entry evidence and which do not.</p><p>And the operational record started being published monthly. The argument for that is simple: a tenant deciding where to deploy has almost nothing to go on beyond on-chain attributes and a price. A provider that publishes its own record — including the parts that went badly — is easier to evaluate than one that doesn’t. The secondary benefit turned out to matter more. A monthly report forces a reckoning with the things that are quietly not working, on a schedule, whether or not you feel like having it.</p><h3>// a silent two weeks</h3><p>One of the twenty-four nodes held Ready for two weeks with no working network path for its pods.</p><p>Kubernetes said the node was fine. The kubelet was reporting heartbeats on schedule, the pods assigned to it showed as scheduled and healthy, nothing was being restarted. The signal Kubernetes watches is its own control-plane connectivity to the kubelet, which was intact throughout.</p><p>What was broken was the data-plane. Pods on that node couldn’t reach storage, couldn’t reach each other, couldn’t reach anything outside — and the signal that would have shown it never travelled through the kubelet, because the kubelet wasn’t the thing broken.</p><p>It was caught by cross-referencing two independent sources of truth: the Kubernetes node list against the storage cluster’s own view of which hosts its daemons were reachable from. A node that holds Ready on one side but is unreachable from the other is in a split state, by definition, and that cross-reference is now one of the handful of checks that run on a schedule.</p><p>Nothing had alerted during the two weeks. The failure mode was the shape every expensive incident in these six months carried — a thing that stopped being true, with nothing in the stack built to notice.</p><h3>// the rebuild</h3><p>August and September were physical.</p><p>The provider estate had been running on a single-core legacy switching design. One switch failure took the entire provider. Any change to that core meant a full outage, which is why several improvements had stayed deferred for months — the cost of touching it was always higher than the cost of not touching it, right up until it wasn’t.</p><p>The programme ran in three phases over a month: relocating the small-form-factor node estate to new rack space, then the hypervisors one at a time, then the network cutover itself — twenty-four hosts moved from the legacy gateway to a redundant gateway pair on a dual-TOR fabric, workers first, control planes second, the primary control plane last. <a href="https://jjozzietech.com.au/rack-power-migration/">One near-miss from the broader rack programme — on the validator side, not the provider side — has its own write-up</a>.</p><p><strong>Zero leases lost across the entire programme.</strong> Twenty-four nodes, twenty storage daemons, three etcd members, three storage monitors, all relocated.</p><p>Three things from that month I would not want to rediscover.</p><p><strong>noout is the wrong tool for a rack.</strong> It is the obvious-looking shortcut for a maintenance window - tell the storage cluster not to rebalance while hosts are down. For a single host, briefly, it is correct. For multiple hosts under a three-replica policy with host-level failure domain, it would have left roughly forty percent of placement groups inactive. The obvious shortcut and the correct action diverge sharply at scale.</p><p><strong>A running component can be in an unrecoverable state without being unhealthy.</strong> A GPU operator had applied a scheduling taint to one workstation. A storage monitor and a storage daemon were running there perfectly — and could never have been rescheduled if they stopped. Nothing alerted. It surfaced only when I tried to move them deliberately. <em>Healthy</em> and <em>recoverable</em> are different properties and only one of them is monitored.</p><p><strong>Diagnose from state, not from logs.</strong> Twice in the programme a storage daemon was diagnosed from heartbeat log text, and twice it was the wrong daemon. The actual fault both times was a node losing power to a failing cable. One command listing daemon state gives the answer directly. That is now the first command rather than the third.</p><p>There was also a switch whose running configuration had never been written to startup — a reload would have taken the configuration with it — and two marginal ports with around twenty thousand accumulated receive errors, which on recabling retrospectively explained three unattributed incidents from earlier in the year. Intermittent physical faults present as software mysteries for as long as you let them.</p><h3>// four days of nothing</h3><p>During the network cutover the provider created no bids for four days.</p><p>It recovered without intervention. No leases were lost — existing workloads ran throughout; only new bidding stopped. Nothing alerted, because nothing had failed. There was no error to catch.</p><p>It was found afterwards, by counting transactions.</p><p>This is the part worth sitting with. A provider that has stopped bidding looks identical, on every dashboard available to its operator, to a provider that is bidding and simply not winning. Both show healthy pods, healthy storage, healthy nodes, active leases ticking over. The difference between them is a thing that is not happening, and monitoring is built to catch things that happen.</p><p>The same month’s provider upgrade revealed a longer-running version of the same problem. Three previous upgrades had failed outright — and the provider had kept running on its last good configuration the entire time, looking healthy by every external check. Worse, each subsequent upgrade correctly built on the last <em>successfully deployed</em> configuration, silently discarding everything in the failed ones.</p><p>Six weeks of changes that appeared to have been applied and had not. A certificate setting that never actually changed. An on-chain attribute update that never deployed. Two values that stayed inverted on-chain until they were corrected directly.</p><p>Both root causes turned out to be upstream bugs in the deployment chart — one where disabling a feature drops the container’s entire environment block and the resulting object is rejected, another where an attribute comparison sorts keys and values independently and therefore cannot see two values that have been swapped. <a href="https://github.com/akash-network/helm-charts/issues/595">Both</a> <a href="https://github.com/akash-network/helm-charts/issues/596">are</a> now filed.</p><h3>// unaudited</h3><p>In August the attestation was removed again — this time not through any failure on the provider’s side. An auditor adjusted the scope of its portfolio and this provider fell outside it. No compliance finding. Nothing about the hardware, the uptime or the pricing changed.</p><p>The practical consequence was larger than expected, and took an outside dataset to make visible.</p><p>The deployment console’s default flow filters to audited providers. So does every trial account. The effect was immediate invisibility to the main discovery path, with no signal on the provider side at all — because the way that invisibility manifests is a bid that never happens.</p><p>An independent benchmarking project noticed first. It had been running automated deployments across the network for a month and observed that this provider had stopped bidding on its orders entirely: hundreds of orders broadcast, hundreds of bids drawn from other providers, none from here — while its own market snapshots showed this provider advertising available capacity in one hundred percent of over a thousand samples across the same window.</p><p>From inside, there was nothing to see. The bid engine was healthy and transacting normally. Thousands of bids over the period, a failure rate well under one percent, nearly all of those losing a race to the chain’s bid cap.</p><p>The answer was on-chain. Pulling a live order from that tenant showed a signed_by requirement naming a specific auditor. Every open order from them carried it. Without the attestation, those orders are unmatchable at protocol level - and the bid engine declines, correctly, every time, in silence, because a declined bid writes nothing anywhere.</p><p>There is a reciprocal finding in that, which the other party has since accepted and is fixing: their own tooling had stated publicly, more than once, that it did not filter on audit status. On-chain, it did. The requirement had been inherited from a template rather than chosen, and was invisible from inside their own tooling for exactly the same reason the missing bids were invisible from inside mine — both of us were reading our own output rather than what the chain actually said.</p><p>At the time of writing the provider is still unaudited. A re-audit request was submitted in October and is outstanding. The connectivity-upgrade dependency that had previously held the request — a WAN capacity upgrade from Optus 800/80 to Aussie Broadband 2000/500 Mbps — has since completed. What remains is the audit response itself.</p><h3>// the pattern</h3><p>Four incidents cost real time or real revenue across these six months.</p><p>A storage class silently filtered out of inventory for sixty-seven days. Deployment upgrades that failed while reporting success for six weeks. A node that held Ready for two weeks with no working network path. Bids that simply never happened.</p><p>Not one of them produced an error. Not one would have been caught by alerting on failures, because none of them was a failure in any form a monitoring system recognises. They were absences — and an absence does not write a log line, does not increment a counter, and does not fire an alert.</p><p>What has actually worked against this is narrow and unglamorous. Count things that should be happening and alert when the count is zero, rather than alerting when something breaks. Cross-reference two independent signals and treat divergence as a fault, which is how a silently partitioned node is now detected — with the honest caveat that the check currently covers nineteen of twenty-four nodes, because five carry no storage daemon to cross-reference against, and that gap is still open.</p><p>And publish the operational record. Two of the four were found because an outside party was measuring, and one of those was found because they published their measurements and I could check them against my own. That cuts both ways — the <a href="https://github.com/jjozzietech/akash-provider-ops-public/tree/main/reports">monthly reports</a> exist so somebody can do the same to me.</p><p>That is the uncomfortable conclusion of six months. The failures that cost the most were not the ones that broke. They were the ones that stopped — and nothing, anywhere, said so.</p><h3>// the receipts</h3><p>Everything above is checkable. The repositories are sanitized but not curated — what’s in them is what runs.</p><ul><li><a href="https://github.com/jjozzietech/akash-provider-ops-public"><strong>akash-provider-ops-public</strong></a> — operational scripts, monitoring and anti-mining tooling, and the monthly operator reports under reports/</li><li><a href="https://github.com/jjozzietech/akash-miner-blacklist"><strong>akash-miner-blacklist</strong></a> — the community blacklist, its evidence standards, and its provenance disclosure</li><li><strong>Upstream</strong> — the storage-label issues (<a href="https://github.com/akash-network/support/issues/645">#645</a>, <a href="https://github.com/akash-network/support/issues/663">#663</a>) and the two deployment-chart bugs (<a href="https://github.com/akash-network/helm-charts/issues/595">#595</a>, <a href="https://github.com/akash-network/helm-charts/issues/596">#596</a>)</li></ul><p>The operator reports are published monthly and carry the same structure: what ran, what shipped, what broke. They include the months where the answer to the third one was long.</p><p><em>Operated from Sydney, au-syd region.</em></p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/akash-provider-six-months/"><em>https://jjozzietech.com.au</em></a><em> on October 27, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dd164305c9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iSCSI gives you two minutes]]></title>
            <link>https://medium.com/@jjozzietech/iscsi-gives-you-two-minutes-df2bd2e7c2b9?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/df2bd2e7c2b9</guid>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 20 Oct 2026 20:15:52 GMT</pubDate>
            <atom:updated>2026-10-22T19:26:06.457Z</atom:updated>
            <content:encoded><![CDATA[<h4>A border firewall went dark for twenty minutes during a racking job. The iSCSI session underneath it had a hundred and twenty seconds of patience.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gDJe8y1OqR6UWn0P.png" /></figure><p>You can have enterprise storage, a healthy host and a filesystem doing exactly what it was designed to do, and still lose a node to one network interruption. This is what a firewall racking job did to my Sia hostd node, and what it taught me about how much interruption iSCSI will actually absorb before the filesystem underneath runs out of options.</p><p>It started with something that looked like a failure and turned out to be me misreading my own setup. Twenty-four hours later it became the real thing.</p><h3>// the setup</h3><p>The hostd node runs as a VM on pro4, one of the Proxmox hosts, on the workload VLAN. Storage comes from TrueNAS over iSCSI - a zvol presented as /dev/sdb, formatted XFS, mounted at /mnt/sia-data, with /var/lib/hostd symlinked into it. hostd itself is a single binary handling contracts, storage proofs and peer connections.</p><p>iSCSI over a routed network is convenient. It is also unforgiving. Convenient because the disks live where the disks live, the compute lives where the compute lives, and neither has to know. Unforgiving because the amount of interruption it will tolerate is a fixed number in a config file, and most of us have never looked it up.</p><p>I hadn’t.</p><h3>// act 1 — the false alarm</h3><p>The morning after the migration, the hostd GUI wouldn’t load. My first assumption was a crash, and the logs backed it up — a startup attempt with failed to create data directory, which reads alarming at seven in the morning.</p><p>It took about thirty seconds to establish I was wrong. hostd was already running. Process up, port 9980 bound and listening. My systemctl start had found a live process and exited without doing anything. The GUI had been reachable the whole time; I had the browser pointed at the management VLAN address rather than the workload one.</p><p>So: operator error, not a node failure. Lesson one, filed and then almost immediately forgotten — verify before you assume.</p><h3>// act 2 — the real one</h3><p>Twenty-four hours later I was racking the new border firewall. During the physical install it was down for roughly twenty minutes. That was planned and I was expecting it. What I had not fully accounted for was that the iSCSI path to TrueNAS ran through it.</p><p>The sequence:</p><ol><li>The network path to TrueNAS dropped when the firewall went dark.</li><li>Five seconds later, with no reply to its NOP-out ping, the initiator declared the connection failed.</li><li>The session went into recovery and I/O queued, waiting for the session to come back.</li><li>Two minutes in, it hadn’t. SCSI commands were failed back to the application and /dev/sdb went away.</li><li>XFS saw write errors on a device that no longer existed and shut down the journal to protect the filesystem.</li><li>hostd was mid-write. database disk image is malformed on hostd.sqlite3.</li></ol><p>journalctl told the whole story in two lines - write I/O errors, followed immediately by XFS shutting itself down. hostd had crashed cleanly enough, but the SQLite write-ahead log was in an uncertain state and the database was 41 GB.</p><h3>// the two numbers I should have known</h3><p>Here is the part I had wrong, and had been wrong about for as long as I’d been running this.</p><p>I assumed iSCSI had no tolerance at all — that a dropped path meant an immediate dead device. It doesn’t. There are two timers, they do different jobs, and the gap between them is where the fault tolerance lives:</p><pre>node.conn[0].timeo.noop_out_interval = 5 <br>node.conn[0].timeo.noop_out_timeout = 5 <br>node.session.timeo.replacement_timeout = 120</pre><p>Those are the stock defaults, unchanged on my host. The initiator pings the target every five seconds. If a ping goes five seconds without a reply, the <em>connection</em> is declared failed — that’s the first number, and it’s the one I’d half-remembered as “the timeout”.</p><p>It isn’t the one that matters. replacement_timeout is how long the session is given to re-establish itself before SCSI commands get failed back to the application. Default 120 seconds. For two full minutes, I/O queues and waits. If the path returns inside that window, the writes land and nothing upstream ever knows there was a problem.</p><p>So iSCSI is not fragile. It ships with two minutes of patience.</p><p>I was gone for twenty.</p><p>That reframes the incident entirely. Nothing failed — not the initiator, not XFS, not SQLite. Every layer did exactly what it was built to do, in order, on a timescale that was documented and that I could have read in thirty seconds any time in the previous year. The outage was ten times longer than the tolerance, and after that there is nothing left for a filesystem to do except stop.</p><h3>// the recovery</h3><p>Methodical, and mostly waiting. Once the firewall was back and the path restored:</p><pre>umount -l /mnt/sia-data</pre><p>Lazy unmount, then remount — XFS replayed the journal automatically on the way back up.</p><pre>sqlite3 /var/lib/hostd/hostd.sqlite3 &quot;PRAGMA integrity_check;&quot;</pre><p>Then wait. On a 41 GB database the integrity check ran for about three hours. The result, when it finally came back, was one word: ok.</p><p>systemctl start hostd. The GUI came up, contracts resumed, and RHP4 RPC successes started streaming in the logs within minutes. Nothing was lost.</p><h3>// what I’d make instinctive</h3><p>Three things from this.</p><p><strong>Stop the service and unmount before touching anything on the storage path.</strong> Routing changes, firewall reboots, link reseats, a switch upgrade — if the path between the VM and the iSCSI target is in scope, hostd stops and /mnt/sia-data unmounts first. It costs a minute. The alternative cost me three hours of integrity check and a morning of not knowing whether the database had survived.</p><p><strong>Know your </strong><strong>replacement_timeout before you need it, not after.</strong> Two minutes is the default, not a law - it&#39;s one line in iscsid.conf and some distributions and appliances ship it differently. Run iscsiadm -m node -o show | grep -E &#39;replacement_timeout|noop_out&#39; on anything you care about and write the answer in the runbook next to the maintenance window. A twenty-minute job against a two-minute tolerance is a decision, as long as you know you&#39;re making it.</p><p><strong>SQLite’s WAL is resilient, but only because XFS was.</strong> The write-ahead log protected the database and the XFS journal protected the filesystem. Both did their jobs. The three-hour integrity check wasn’t a symptom of damage — it was the cost of confirming the safety nets had held, on a database large enough that confirming takes three hours.</p><p>The runbook now has a mandatory pre-change step: stop hostd, unmount the iSCSI volume, then proceed. Every time.</p><h3>// closing</h3><p><a href="https://jjozzietech.com.au/storage-failure-arp-race/">The ARP race piece</a> from last week was the same shape at a different layer — a change that verified fine at one point and broke at a boundary nobody had checked. This one is the same lesson with a number attached. The maintenance window was planned, the outage length was known in advance, and the piece of information that would have made it a non-event was sitting in a config file I’d never opened.</p><p>The tolerance exists. It’s just shorter than most maintenance windows, and nothing warns you when you’re about to spend it.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/iscsi-gives-you-two-minutes/"><em>https://jjozzietech.com.au</em></a><em> on October 20, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df2bd2e7c2b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The storage failure that was actually an ARP race]]></title>
            <link>https://medium.com/@jjozzietech/the-storage-failure-that-was-actually-an-arp-race-60fe195da499?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/60fe195da499</guid>
            <category><![CDATA[truenas]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[rap]]></category>
            <category><![CDATA[cisco-nexus]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 13 Oct 2026 20:15:36 GMT</pubDate>
            <atom:updated>2026-10-15T19:28:36.731Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZUrL4BxU7ErRzGI5.png" /></figure><p>The Windows workload client lost its TrueNAS shares one Thursday morning. Red-X on every drive mapped to that specific server. Mappings to other storage servers on the same VLAN stayed online. Test-NetConnection on port 445 returned TcpTestSucceeded : False. Ping timed out.</p><p>I cleared the client’s ARP cache. The shares came back. Then Windows Update rebooted the client, and the shares went down again.</p><p>That second failure is what made me look properly, because the first failure had a satisfying workaround and the second one broke that workaround in a way that was going to keep breaking it. If I’d stopped at arp -d, I&#39;d have carried a time bomb forward - Windows Update reboots are routine, and every reboot would have been a 50/50 bet on whether the shares came back.</p><p>This piece is the trace that turned a storage symptom into a network root cause: client to Nexus to endpoint, with each layer eliminating a correct-looking suspect, ending in a configuration error that had been in place since the fabric migration. The server I thought was failing wasn’t failing. Nothing was failing. Two devices were simply configured with the same IP address, and ARP — behaving exactly as designed — had been quietly handing my client the wrong one.</p><h3>// why storage was the first suspect</h3><p>The affected TrueNAS server had been through genuine storage trouble recently. The <a href="https://jjozzietech.com.au/truenas-migration-five-storage-layers/">MMX/TrueNAS migration piece</a> documented five layers of failure during that same ReRACK window, including a hung SAS/CAM path on this same server that required a full JBOD power cycle to clear. When storage had just bitten me, storage was the natural place to look.</p><p>The symptoms were plausibly storage: SMB mappings timing out, Red-X on the shares, retries not helping. If this had been a repeat of the earlier SAS failure, I’d have expected pool-level trouble visible in the TrueNAS GUI, SMB service errors in the system logs, or at minimum a kernel message about a hung path. None of that was there. Pools online. SMB service healthy. Other storage servers on the same VLAN reachable from the same client, serving the same workload, with no complaints.</p><p>That mismatch — storage-looking symptom with no storage-layer evidence — was the first thing that didn’t fit. Storage went to the back of the suspect list within about ten minutes. The symptom was Layer 7, but the fault was somewhere lower.</p><h3>// the Windows ARP red herring</h3><p>The next suspect was the client itself. SMB has a long history of edge cases on Windows that can be fixed by clearing some piece of cached state and letting the stack rebuild. The ARP cache is one of them. If the client has learned a stale MAC address for the server, nothing above Layer 2 is going to work until that mapping refreshes.</p><p>I ran:</p><pre>arp -d 172.16.x.22<br>ping 172.16.x.22 <br>arp -a | findstr 172.16.x.22</pre><p>(Addresses are illustrative throughout this piece — I’ll use the 172.16.x.NN pattern to abstract real addressing.)</p><p>The ping succeeded. The ARP cache populated with a MAC. The SMB mappings came back online. Everything felt resolved.</p><p>For a reader who’s spent any time troubleshooting SMB on Windows, this is the moment that feels like victory and is often a trap. arp -d restoring service is a specific signal - it says the client was holding the wrong MAC for the IP it was trying to reach. It does not say <em>why</em> the client was holding the wrong MAC. There are at least three plausible reasons, and only one of them is a Windows bug.</p><p>I didn’t think about which reason it was. I called the shares back up, verified the mappings, and moved on. For about two hours it felt like I’d fixed something.</p><p>Then the client took a Windows Update reboot. The ARP cache cleared as a side effect of the restart. Windows did a fresh ARP lookup on the TrueNAS address. The SMB mappings came up Red-X again.</p><p>That recurrence is what forced the real investigation. A workaround that works once can be a workaround. A workaround that works and then breaks on the next reboot is pointing at something underneath that nobody’s looked at yet.</p><h3>// tracing from client to the fabric</h3><p>The question I needed to answer first was: when the client sends traffic to the TrueNAS address, where does the fabric think that traffic should go?</p><p>The TrueNAS server runs as a Proxmox VM, which attaches to the Cisco Nexus 3548 top-of-rack pair via a vPC-bonded port-channel on the hypervisor. The TrueNAS MAC should appear in both TORs&#39; MAC address tables, learned on that port-channel.</p><p>I checked:</p><pre>TOR-A# show mac address-table address MAC-A <br>+ 10 MAC-A dynamic 0 ... Po200 <br><br>TOR-B# show mac address-table address MAC-A <br>+ 10 MAC-A dynamic 0 ... Po200</pre><p>(MAC addresses abstracted as MAC-A, MAC-B, MAC-C throughout.)</p><p>Both TORs had the TrueNAS MAC on Po200, which is the LACP bundle to the correct Proxmox host. The vPC was consistent, the trunk allowed the workload VLAN, and port type was edge trunk as it should be. From the switch&#39;s perspective the TrueNAS server was exactly where it was supposed to be.</p><p>The fabric wasn’t the problem. The fabric was doing the thing fabrics do: forwarding traffic to the MAC that owns the destination IP, as that MAC resolves through ARP, as the ARP table currently believes.</p><p>Which meant the real question was <em>who owns the destination IP in the ARP table</em>, and the answer to that was about to be the whole story.</p><h3>// the ARP table discrepancy that unlocked everything</h3><p>I checked the gateway ARP table on the Nexus for the TrueNAS address:</p><pre>TOR-A# show ip arp vlan 10 | include 172.16.x.22 <br>172.16.x.22      00:00:44       MAC-B     Vlan10</pre><p>The MAC in that line was not MAC-A. It was a MAC I didn&#39;t recognise - MAC-B. For an address I thought belonged to TrueNAS, the gateway&#39;s ARP table was returning the MAC of some other device.</p><p>This is the moment the investigation shifts. Up until this point I’d been looking for things that were broken — a storage pool, an SMB service, a Windows cache. Nothing was broken. Everything was working. What was happening was that the fabric was routing .22 traffic to a MAC that was not the server I thought owned .22, and it was doing that because that MAC had answered the gateway&#39;s most recent ARP request for the address.</p><p>I traced the unfamiliar MAC through the CAM table:</p><pre>TOR-A# show mac address-table address MAC-B <br>* 10   MAC-B     dynamic  0  ...    Eth1/15</pre><p>Eth1/15 on TOR-A. The port profile named it - a different workload node on the same VLAN. Not Proxmox, not TrueNAS. A separate storage/compute node attached directly to one of the TORs.</p><p>For completeness I checked the client’s path too — its MAC resolved on TOR-B Eth1/21, the port profile matching the expected workload-client access port. Clean. The client was correctly learned, in the correct port, in the expected VLAN. No fabric-side problem on the client&#39;s side either.</p><p>The whole fabric was behaving as designed. What wasn’t right was higher up: two different MAC addresses were able to answer ARP for the same IP, and the fabric had no way to know which one was supposed to.</p><h3>// the endpoint reveal</h3><p>The authoritative check for IP ownership is the host itself. The switch can tell you where a MAC lives. Only the endpoint can tell you what address it thinks it has.</p><p>I logged into TrueNAS directly and ran ip addr show:</p><pre>3: enp6s19: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; ... <br>link/ether MAC-A <br>inet 172.16.x.22/24 brd 172.16.x.255 scope global enp6s19</pre><p>TrueNAS was configured as .22. The other workload node was configured as .22. Both devices had legitimately claimed the same address. Both were valid Layer-2 participants on the workload VLAN. Both would answer an ARP request for .22 with their own MAC.</p><p>The intended address for TrueNAS during the ReRACK was .40. I had the record of the planned allocation; .22 belonged to the other node and had belonged to it for months before the fabric migration. What had happened was that during the TrueNAS reconfiguration for the new fabric, the address .22 had gone onto the interface by mistake - one digit off, or a copy from a different line of a planning spreadsheet, or anything in that family of five-second configuration slips. Nobody caught it at the time, because .22 on the TrueNAS interface produced a device that answered ARP and accepted TCP connections. From any single-request check, it looked fine.</p><p>It only became a problem when both devices were on the VLAN at the same time and the client needed to decide which one owned .22. The client couldn&#39;t decide. The client just believed whatever ARP reply arrived first.</p><h3>// what the duplicate IP actually did</h3><p>ARP is one of those protocols where violating the design assumptions can stay quiet for a long time. Nothing I was watching logged a complaint — no pool errors, no SMB service faults, nothing in the fabric. NX-OS does have duplicate-address detection, and it never surfaced this one; with the second node mostly idle on that address, the two never ARPed close enough together to trip it. The vPC pair stayed consistent, HSRP didn’t flap, MAC flapping didn’t trigger because the two MACs were on different ports and neither of them was moving.</p><p>What the duplicate IP did was turn ARP resolution into a race. When any device on the VLAN sent an ARP request for .22, both TrueNAS and the other workload node answered. Whichever answer arrived first became the entry in the requester&#39;s ARP cache. The entry held until it timed out or the cache got cleared, at which point another race ran.</p><p>For the client, this looked like intermittent connectivity. Half the time the cache held TrueNAS’s MAC and SMB worked. Half the time it held the other node’s MAC, traffic went to a host that had no SMB listener on that address, and the shares came up Red-X. The 50/50 wasn’t literal — timing depends on where the devices sit relative to the client, which replies win in which order — but the practical effect was that the client sometimes worked and sometimes didn’t, with no deterministic rule for which.</p><p>arp -d restored service because it forced a new race. If the race landed on TrueNAS, the shares came back. If it landed on the other node, nothing changed. For my situation, the first few arp -d calls happened to land on TrueNAS, which is why the workaround felt like a real fix. It wasn&#39;t fixing anything. It was rerolling the dice, and I happened to roll well for a few hours.</p><p>The Windows Update reboot was what exposed this. A reboot clears the entire neighbour cache, which forces a fresh resolution. The post-reboot resolution landed on the other node this time, and the shares were Red-X again. That recurrence was the lucky break — if the reboot had landed on TrueNAS, I’d have gone another day or two before the next failure, and the pattern would have been even harder to see.</p><p>Clearing ARP wasn’t a workaround for a Windows bug. Clearing ARP was the trigger I was using to reroll an unstable system state, without understanding that the state was unstable.</p><h3>// the fix</h3><p>With the duplicate IP identified, the fix was configuration-level and non-destructive. Both devices were legitimately part of the VLAN. The other workload node had owned .22 for months and was the historical owner. TrueNAS was the one with the mis-assigned address.</p><p>I verified that .40 was unused on the VLAN, by checking the gateway&#39;s ARP table for the address and getting no entry. Then changed the TrueNAS interface configuration in the GUI from .22/24 to .40/24, leaving everything else - the MAC, the subnet mask, the gateway, the DNS settings - unchanged.</p><p>On the client side, the persistent drive mappings had to point at the new address. The net use entries for the two affected shares came off and went back on with the new target:</p><pre>net use G: /delete <br>net use H: /delete <br>net use G: \\172.16.x.40\JBOD1 /persistent:yes <br>net use H: \\172.16.x.40\JBOD2 /persistent:yes</pre><p>Both drives came up on first attempt. SMB access was clean, reproducible, and stable across the next series of planned client reboots. The duplicate-IP condition was gone because only one device now claimed .22, and that device was the one that had always been the legitimate owner.</p><p>The whole corrective action took about ten minutes once the root cause was identified. The investigation that identified it took the better part of a morning.</p><h3>// what I’d make instinctive</h3><p>Four things from this.</p><p><strong>When clearing ARP “fixes” a problem, test for duplicate IP before treating the client as buggy.</strong> arp -d is a legitimate diagnostic tool, but a successful ARP clear doesn&#39;t prove the client&#39;s ARP implementation is broken. It proves the client was holding one of several possible mappings, and the one it was holding wasn&#39;t the right one. There are three plausible reasons for that: the client had learned a stale MAC that no longer owned the IP (an actual client-side cache problem), the switch had learned stale CAM entries for the MAC (a fabric-side cache problem), or two devices were legitimately answering for the same IP (a configuration problem). The first two are rare on modern hardware with healthy aging timers. The third is more common than operators like to admit, and it&#39;s the one that arp -d can&#39;t fix because the next ARP exchange has the same race condition.</p><p><strong>ARP tables answer IP → MAC. CAM tables answer MAC → port. Use both directions.</strong> The decisive moment in this investigation was comparing what the gateway ARP table thought owned the IP against what the CAM table said about where that MAC lived. The ARP table alone would have told me the resolution was wrong but not where the wrong device was. The CAM table alone would have told me a MAC was on a port but not that it was answering for an IP it shouldn’t own. Both tables, read in the same session, pointed at the specific switch port on the specific TOR that was serving the wrong endpoint. That’s not something you can see from either table in isolation.</p><p><strong>The switch tells you where a MAC lives. Only the endpoint tells you what IP it thinks it owns.</strong> This is the kind of lesson that sounds obvious when written down and gets forgotten under time pressure. During the investigation I was tempted to stop at “wrong MAC answering for the IP, that’s the problem” and go hunt down the owner of that MAC in the inventory. The step that actually resolved the question was logging into TrueNAS and running ip addr show - that&#39;s what proved the mis-assignment at the authoritative source. Any investigation that ends at the switch is incomplete; the switch is reporting observations, not declarations of ownership.</p><p><strong>Static ARP is the wrong permanent answer to duplicate IP.</strong> It’s tempting, once a duplicate IP is identified, to pin the correct mapping on the affected client with a static ARP entry. That hides the symptom for one client. It leaves the duplicate IP in place for every other device on the VLAN — the gateway, the DHCP server, the monitoring stack, any other client that happens to need the address. The correct action is to fix the IP ownership at the source, which is a configuration change on the mis-assigned device. Static ARP as a durable fix means you’ve accepted that your VLAN has a duplicate IP and chosen to paper over it on the one host that noticed.</p><p>None of these are theoretical. All four came out of this specific morning’s investigation, and all four are things I’d apply next time I see a symptom that arp -d temporarily resolves.</p><h3>// closing</h3><p>The Article 10 rack migration piece landed on the observation that a plan can be right and still be incomplete — correctness and completeness are different axes. This piece is a smaller version of the same shape at the IP layer: a reconfiguration had been done, had been verified to the extent that single-point checks could verify it, and had been complete enough that TrueNAS came up on the VLAN and started answering queries. What the single-point check couldn’t see was whether anyone else on the VLAN was also answering for the same address. That verification needs the full ARP table, not just a ping to confirm the host is reachable.</p><p>If you have anything on your network that “sometimes needs an ARP clear to come back,” you probably don’t have a Windows ARP bug. You have a duplicate IP condition that hasn’t been investigated yet, and the ARP clear is masking it by rerolling the race. Investigate the next time the symptom appears, not the next time it’s convenient — the next client reboot is going to reroll the dice regardless, and sooner or later it’s going to land in a way that costs more than one morning.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/storage-failure-arp-race/"><em>https://jjozzietech.com.au</em></a><em> on October 13, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60fe195da499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four lines of patched JavaScript, and the router VM that deleted them]]></title>
            <link>https://medium.com/@jjozzietech/four-lines-of-patched-javascript-and-the-router-vm-that-deleted-them-62f11ab6634a?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/62f11ab6634a</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[depin]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[homelab]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 21:15:41 GMT</pubDate>
            <atom:updated>2026-10-01T22:35:11.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W_uCFK4NPygX5T5b.png" /></figure><p>Two small patches to a dependency file called ssdp.js. Four lines of changed JavaScript, total. They kept the fleet&#39;s nodes reachable from the internet, and they were the reason every incident on the fleet, for months, started with the same sentence: <em>&quot;the patches are gone again.&quot;</em></p><p>The patches lived inside a Node.js dependency called @runonflux/nat-upnp. Every time the application updated itself - and it updated itself often - the file was rewritten, and the four lines vanished. A cron job would notice within a minute and put them back. A @reboot timer paired with a sleep would apply them again during startup. A watchdog on the edge firewall restarted miniupnpd every two minutes because the UPnP daemon had a habit of losing state without any log line indicating it had. And a verification pass every thirty minutes checked that the patches were still on disk and that the UPnP mappings the daemon had negotiated were still in place.</p><p>That’s four distinct pieces of automation running continuously to keep four lines of JavaScript alive on seven nodes.</p><p>The migration’s purpose was not to improve any of that. It was to delete it.</p><h3>// why the patches existed</h3><p>The edge firewall running miniupnpd was configured correctly. The UPnP daemon was up, listening on the LAN interface, advertising the Internet Gateway Device profile the nodes expected. And it refused to talk to any of them.</p><p>The problem was M-SEARCH. UPnP clients discover their gateway by sending an M-SEARCH message to a multicast group on 239.255.255.250. That works when both endpoints are on the same L2 segment and the multicast group is joined on both sides. On this network, the firewall’s LAN interface never joined the group. From the daemon’s side everything looked fine; from the nodes’ side the M-SEARCH went out and nothing came back.</p><p>The first patch made the nodes send M-SEARCH as unicast to the firewall’s LAN address instead of to the multicast group. Two lines of change to the SSDP client code inside nat-upnp. Once applied, discovery worked immediately. The daemon answered the unicast M-SEARCH, the node completed its discovery step, and the port mapping request went through.</p><p>The second patch was for a different problem. nat-upnp binds to every network interface on the host during socket creation. On a FluxOS node running Docker, that includes Docker&#39;s 172.x bridges - interfaces that have their own UPnP behaviour (specifically none) and cannot support the port mapping the client tries to negotiate. The socket creation attempt against those interfaces returned UPnP error 718 - &quot;conflict in mapping entry&quot; - and the client crashed before it could try the interface that would actually have worked.</p><p>The second patch filtered the socket creation loop by interface, letting nat-upnp skip anything that didn&#39;t look like a routable interface pointing at the gateway. Two more lines. Once applied, socket creation succeeded on the first attempt.</p><p>Two patches, two specific bugs, four lines of code between them. Both trivial. Both load-bearing.</p><h3>// the maintenance tax</h3><p>The problem with load-bearing patches to a third-party dependency is that the dependency doesn’t know about them. nat-upnp shipped through the application&#39;s normal update cycle. Every application update pulled a fresh copy of the dependency, which overwrote ssdp.js with the upstream version. The patches were gone in the moment of the update. Discovery would fail on the next M-SEARCH, port mappings would drop shortly after, and the node would fall out of the fleet&#39;s reachable set.</p><p>The maintenance response was a stack of automation that grew a piece at a time as I learned what the failure modes were:</p><p><strong>A reapplication cron on a fixed timer.</strong> Every minute, check the current ssdp.js against the patched version. If they differ, apply the patches, restart the client. This caught the majority of updates within sixty seconds of the file being rewritten.</p><p><strong>A </strong><strong>@reboot timer paired with a startup sleep.</strong> The one-minute cron didn&#39;t fire until the node had been up for a minute, and the application started faster than that on some hosts. Without the reboot-time apply, the client could initialise against the upstream ssdp.js, cache its (broken) module state at load time, and continue running with the wrong code for the first minute after every boot. The @reboot sleep 30 &amp;&amp; apply-patches entry closed that gap.</p><p><strong>A watchdog on the firewall itself.</strong> miniupnpd had a habit of drifting into an unresponsive state - not crashed, not exited, just no longer answering M-SEARCH. A systemd unit on the firewall restarted the daemon every two minutes, unconditionally, so that any drift was bounded to the same two-minute ceiling. This was crude and honest - there was no signal I trusted to detect the drift, so the watchdog restarted the daemon on a schedule instead.</p><p><strong>A re-verification pass every thirty minutes.</strong> Both cron and the boot timer could fire successfully and still leave the fleet with broken port mappings, if the daemon had lost state during a window when the patches happened to be in place. The verification pass ran the equivalent of upnpc -l from each node against the firewall, checked the returned mapping list against the expected mappings, and re-negotiated any that were missing. This was the safety net for the cases the other three mechanisms didn&#39;t catch.</p><p>Four distinct pieces of automation. Two of them local to the nodes (cron and @reboot), two of them running on the firewall ( miniupnpd watchdog and the verification pass). All four load-bearing. If any one of them stopped running, the fleet would degrade within an hour.</p><p>I don’t want to make this sound heroic. It wasn’t. It was a workaround for a bug that a proper fix would have resolved once and stayed fixed. The workaround happened to work — for months — and every month I told myself I’d do the proper fix during the next quiet week. There was never a quiet week.</p><h3>// why it kept breaking anyway</h3><p>Even with all four pieces of automation in place, incidents still landed on the fleet at a rate of about one per week. The pattern was always the same: application auto-updates in a way that races the one-minute cron, the client instantiates against the upstream ssdp.js, the module cache locks in the broken code path, and the cron&#39;s later patch-apply doesn&#39;t help because the running process has already loaded the module.</p><p>That last part is the point. Node.js caches modules at first require(). Patching ssdp.js on disk after the client has already loaded it doesn&#39;t affect the running process - the module remains resident in memory in its original form until the process restarts. So the reapplication cron only worked if it ran before the client instantiated the module. On a boot, that meant winning a race against the application&#39;s own startup. On a mid-life update, that meant restarting the client immediately after re-applying the patches. Both were possible; neither was reliable enough to be trusted at fleet scale.</p><p>There was also a doctrine issue that made the problem harder than it looked. Standing rule five on this fleet is: <em>never patch files under </em><em>ZelBack/src/.</em> Those files carry an integrity hash the application checks at startup. Patching them causes the node to refuse to start at all - a much worse failure than a broken UPnP client. nat-upnp sits outside ZelBack/src/, which is why the patches were possible in the first place. But it meant the workaround had to live in a boundary layer between the application and its dependencies, patching upstream code that could be replaced at any moment by the application itself.</p><p>The whole arrangement was a stack of workarounds on top of workarounds. Nothing about it was going to become stable by adding more mechanisms.</p><h3>// the alternative</h3><p>The migration’s answer was to move the IGD off the nodes entirely.</p><p>A router VM sits L2-adjacent to the fleet on the same segment. It holds the public IP address for the fleet through 1:1 NAT — the router owns the public address, the fleet’s nodes hold private addresses, and traffic bound for the public address is translated to the router’s LAN address and forwarded onto the segment. The router runs its own miniupnpd, correctly configured, joining the multicast group on the LAN side, answering M-SEARCH from the nodes without patches or workarounds.</p><p>From the nodes’ side, this looks like a normal gateway. nat-upnp unpatched, upstream, running against the vendor&#39;s own release. Discovery works because the router VM&#39;s miniupnpd correctly joins the multicast group. Socket creation works because the router VM is the only interface pointing at the internet from the nodes&#39; perspective - Docker&#39;s 172.x interfaces are still there, but the client&#39;s discovery finds the router&#39;s LAN address as the gateway of record and doesn&#39;t try the Docker bridges.</p><p>The router VM approach dissolved every piece of the workaround in one move:</p><ul><li><strong>Both patches to </strong><strong>ssdp.js</strong> - deleted. The nodes run stock @runonflux/nat-upnp and have since the migration.</li><li><strong>Three cron entries</strong> — the reapplication timer, the @reboot sleep pair, and the verification pass every thirty minutes - all deleted. Nothing to reapply, nothing to re-verify.</li><li><strong>The firewall watchdog restarting </strong><strong>miniupnpd every two minutes</strong> - deleted. The router VM&#39;s miniupnpd runs stable because the underlying configuration is correct.</li><li><strong>The old DMZ 1:1 NAT for Node 1</strong> — deleted. Node 1 had a special-case NAT rule for historical reasons; the router VM’s uniform 1:1 NAT covered it too.</li></ul><p>Seven nodes on stock dependency code. No cron entries related to the fleet’s UPnP layer. No firewall watchdog. No re-verification pass. And no more one-per-week incidents traceable to the patch treadmill, because there was no treadmill to trace them to.</p><p>The lesson isn’t that the patches were wrong. The patches worked, for what they were, for as long as they had to. The lesson is that patches designed to survive on top of a moving target <em>aren’t going to</em>, and any workaround that has a maintenance tax attached to it is going to fail eventually — not because the workaround is bad, but because the tax compounds.</p><p>The right answer wasn’t a better patch. It wasn’t a better cron. It was an architecture that didn’t need either.</p><h3>// the coda that closed everything else</h3><p>I want to be honest about how this piece was supposed to end, and how it actually ends.</p><p>The piece I wrote in my head, during the router VM build, ended at the previous section. Seven nodes on stock code, workaround deleted, treadmill ended by operator design. A clean win — a workaround built through operator effort, retired through operator effort, on operator terms.</p><p>That’s not how the story closed.</p><p>Since the migration, the protocol has released ArcaneOS. Every node in the fleet is expected to be running it. ArcaneOS is a self-updating, encrypted-disk, no-root, restricted-shell operating system image. It is not the sort of thing you patch files on. The router VM approach continues to work because it lives outside the nodes — the router isn’t a Flux node, so ArcaneOS doesn’t touch it. But every piece of node-side infrastructure I built or configured over the last year, including the deep operational knowledge of how to keep FluxOS behaving, is no longer mine to work on. I don’t have root. I can’t inspect the filesystem beyond what the restricted shell exposes. The OS updates itself on its own schedule and I find out about the update after it’s done.</p><p>That’s the coda. The patch treadmill stopped when I moved the IGD off the nodes. It also stopped, more completely, when I stopped owning the OS.</p><p>I want to be careful how I write that. It reads two ways. The relieved reading — no more maintenance tax, no more four-in-the-morning patch races, no more workarounds on workarounds — is real. There is genuine relief in never having to reapply a patch to ssdp.js again, because it can&#39;t happen. The restricted shell prevents it.</p><p>The other reading is also real. Operating a fleet of nodes involves a hundred small decisions about how the OS runs — cron cadence, log rotation, disk pressure alerting, monitoring hooks, package versions, kernel parameters. Every one of those decisions is now made by the protocol, not by me. I have opinions about some of them. Those opinions no longer matter.</p><p>The two readings are not opposed. Both are honest. But if I had to pick which one the closing sentence should carry, it’s the second one, because the first one is already implicit in “the treadmill stopped.” Saying only that would suggest that the ending was clean. The ending wasn’t clean. Something ended that I had built, and something ended that I wanted to keep building.</p><p>The treadmill stopped when I stopped owning the OS. Both things are true at once.</p><p>A coming piece covers the mechanism that made that happen — how the protocol enforcement rolled out, what “every node in the fleet went dark” actually looked like from an operator’s seat, and what the operator response was. This piece is the workaround story. That piece is the governance story.</p><h3>// what I’d make instinctive</h3><p>Four lessons from the treadmill months.</p><p><strong>Patch treadmills mask their true cost until you stop.</strong> While the four mechanisms were running, I could tell you what each one did but not what the total cost was. Only after deleting all four could I see how much operator attention had been going into keeping them working. If you have a workaround with a maintenance tax attached, the cost you can measure — the CPU time, the disk writes, the log volume — is not the real cost. The real cost is the operator attention that isn’t going elsewhere.</p><p><strong>Auto-update-then-patch is a race the auto-update wins eventually.</strong> Any workaround that depends on being reapplied faster than upstream updates is going to lose that race, at some point, at the worst possible moment. The correct fix is architectural — remove the need for the patch — not a faster cron.</p><p><strong>When you build a workaround, ask what would let you delete it.</strong> The router VM approach was possible from day one. I didn’t do it because the workaround was working and building the router VM was more work than reapplying patches. Every month that passed, the calculation stayed the same and the workaround stayed. The trigger for the architectural fix should have been the workaround’s existence, not the workaround’s eventual failure at scale. The <a href="https://jjozzietech.com.au/etc-environment-systemd/">/etc/environment piece</a> landed on the same pattern from a different angle on this fleet - a mechanism that had been running for months, wasn&#39;t doing the job I thought it was, and needed replacing at the architectural layer rather than fixing at its own. Different failure mode, same operator lesson.</p><p><strong>Operator ownership and operator maintenance are the same thing.</strong> The ability to run a workaround like the patch treadmill is the same ability the protocol has now taken away. I don’t get to solve UPnP problems on the nodes anymore because I don’t get to solve any problems on the nodes anymore. The relief and the loss are the same event seen from different angles. Anyone running fleet infrastructure on a protocol they don’t control should be thinking about which side of that trade they’d end up on.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/patch-treadmill-router-vm/"><em>https://jjozzietech.com.au</em></a><em> on September 29, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62f11ab6634a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five layers, one migration — what a fabric cutover exposed about storage]]></title>
            <link>https://medium.com/@jjozzietech/five-layers-one-migration-what-a-fabric-cutover-exposed-about-storage-fa67d3a09f19?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa67d3a09f19</guid>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[zfs]]></category>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[truenas]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 22 Sep 2026 21:15:01 GMT</pubDate>
            <atom:updated>2026-09-24T19:47:07.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VIx8izAVQU9_7p73.png" /></figure><p>The plan was a network change with a storage flavour. Move a Windows workload and two TrueNAS servers off the legacy flat network and onto the new dual-TOR fabric on the workload VLAN. Three cutovers. Repoint the SMB mappings. Done.</p><p>The workload moved. Both TrueNAS servers moved. The mappings got repointed. Everything the change plan named came back healthy.</p><p>What the plan didn’t cover was what “healthy” meant at each layer of the storage stack — because there isn’t one layer. There are at least five, they can each be independently green or broken, and a healthy signal at one of them does not imply a healthy signal at the next. That’s the piece.</p><p>Worth saying up front: I did not know the migration was going to teach me this while it was happening. During the incident, each failure looked like its own problem. It was only after everything was back that the pattern surfaced. Five different failure modes, five different layers of the storage stack, five different validation steps that would have caught each one independently. The layered framing is retrospective. If it reads as if I was tracking it that way in real time, that’s an artefact of writing about it, not what the day felt like.</p><p>The five layers, in the order they surfaced during the migration:</p><ol><li><strong>Network reachability</strong> — physical link plus L2/L3 configuration</li><li><strong>Service reachability</strong> — the SMB advertisement, independent of what’s behind it</li><li><strong>Filesystem state as the middleware sees it</strong> — TrueNAS’s cached view of pool status</li><li><strong>ZFS state as the kernel sees it</strong> — pool importable, disks readable, labels intact</li><li><strong>Physical storage-path state</strong> — the SAS and CAM layer under ZFS</li></ol><p>Each of the five failed at least once during the day. Each of them failed in a way that would have been invisible from the layer above.</p><h3>// layer 1: network reachability</h3><p>The Windows workload cutover was the first to surface a network-layer failure. Old address on the legacy fabric, new address on the workload VLAN, TOR-B. Cable in, port configured, physical layer up. The switchport showed a valid 10G link. Windows showed the interface as Disconnected.</p><p>That’s the layer-1 problem in one sentence. The physical link doesn’t prove the operating system has a usable NIC.</p><p>The workload host had a 10G adapter installed but no driver loaded for it. The only interface Windows recognised was the onboard Intel I219-LM 1G. From the switch’s side everything looked correct. From the workload’s side there was no interface to bind an address to. Both views were internally consistent. Both views were incomplete.</p><p>The fix was the standard one — install the correct 10G adapter driver, watch the interface enumerate in Windows, watch the switchport carry the traffic that had been waiting for it. The lesson isn’t the fix. The lesson is that the switchport LED and the OS network stack are two different signals, and only the OS signal proves the host can actually use the link. On the migration checklist “link up” needs to mean <em>link up on the host’s side, not the switch’s side</em>.</p><p>The second network-layer failure surfaced when I tried to reach one of the TrueNAS servers on its new address. It didn’t respond. Ping timed out. RDP timed out. The workload couldn’t enumerate its shares because it couldn’t reach the server at all.</p><p>I don’t remember the specific moment that turned into an investigation — I might have gone hunting on the switch first, I might have gone straight to the server. What I found was that the workload VLAN hadn’t been added to the LAGG on the TrueNAS server. The switch was configured. The physical link was up. The other end of the LAGG on the server had no idea the workload VLAN existed as a tagged interface.</p><p>Adding the workload VLAN as a tagged interface on the existing LAGG, with the server’s new address on it, brought the server online. From the workload side, the SMB shares immediately became reachable.</p><p>Both of these are “network” failures in the loose sense, and both had the same shape: healthy signal on one side of the link, invisible break on the other. Layer 1 validation isn’t “the switch shows the link up.” Layer 1 validation is “both endpoints can talk to each other at the address the change plan says they should.” Every migration touching L2 or L3 configuration needs to check both directions before declaring the layer green.</p><h3>// layer 2: service reachability</h3><p>With the network layer working, the workload tried to enumerate its SMB shares. Most of them appeared. One of them — the largest one, the primary storage target for the workload — returned a Windows error: System error 67 has occurred. The network name cannot be found.</p><p>That message is the operating system telling you the SMB server accepted your connection but refused to serve you the share you asked for. It’s not a network failure. It’s not an authentication failure. It’s the SMB service saying yes, I know that share name, and no, I can’t give it to you right now.</p><p>The dataset backing that share didn’t exist.</p><p>Specifically: the share was advertised by TrueNAS’s SMB configuration, correctly. But the ZFS dataset the share was configured to expose wasn’t mounted. It wasn’t mounted because the pool holding it wasn’t imported. It wasn’t imported because the pool’s disks weren’t visible to the operating system. And the disks weren’t visible because the JBOD shelf holding them hadn’t been powered on after the re-rack.</p><p>The layer here is service reachability — SMB advertising a share is one thing, the share having a valid backing dataset is another thing entirely. TrueNAS doesn’t disable an SMB share when its backing dataset goes away. The share stays advertised. The share stays in the configuration. Windows can see it in the list of shares on the server. It just can’t be used, and the error message it produces is deliberately unhelpful about why.</p><p>Powering the JBOD brought the disks back to visibility. But that only fixed the physical prerequisite. The middleware layer above ZFS still had its own idea of what was going on, and that’s the next layer.</p><h3>// layer 3: middleware view vs ZFS reality</h3><p>With the JBOD powered, camcontrol devlist on the TrueNAS server showed all the drives. zpool import - the command that lists importable pools - showed the three pools from that shelf as ONLINE and ready to import. From the ZFS layer&#39;s perspective, the pools were healthy and available.</p><p>The TrueNAS GUI disagreed. Every one of those three pools was still shown as OFFLINE, in the state the middleware had recorded when the JBOD went dark. From the middleware’s view, nothing had changed since the shelf went off.</p><p>That’s the third layer. TrueNAS caches pool state in its middleware database. When something happens outside the middleware’s visibility — a JBOD losing power, a SAS cable being pulled, a controller resetting — the middleware doesn’t automatically catch up when the situation resolves. The kernel’s view of ZFS updates in real time. The middleware’s view updates when it’s told to.</p><p>The recovery for each pool was the same sequence. Export/Disconnect from the GUI, being careful not to check the destroy-data or delete-configuration boxes. Then re-import via the GUI’s Import Pool workflow. The pool would enumerate correctly, the middleware would record it as ONLINE with its current disks, and the shares configured against its datasets would start serving. Three pools, three cycles of Export/Import, three shares recovered.</p><p>The middleware-vs-ZFS divergence isn’t a bug in TrueNAS. It’s a consequence of the middleware being a caching layer over the kernel-level truth. The important operator move is knowing that the two views can drift, and knowing which one to trust when they disagree — the answer, always, is the kernel. If zpool import says the pool is available and the GUI says it&#39;s offline, the pool is available.</p><h3>// layer 4: ZFS state under SAS path interruption</h3><p>The second TrueNAS server surfaced the piece’s first real ZFS-layer failure. Several SMB mappings from the workload connected fine but couldn’t enumerate files. Timeouts, empty directory listings, hangs on read.</p><p>I opened the GUI and looked. One of the pools — call it Pool-A — was in the UNAVAIL state, with all eight of its members marked REMOVED.</p><p>That’s a scary status line. It suggests the pool has lost every one of its physical members simultaneously. If that were true it would mean the array had suffered a catastrophic multi-drive failure, and the response would be one of the worst calls an operator makes: try to import from backups, accept whatever data loss the last replication window implies, rebuild.</p><p>It wasn’t that. It looked like that. Because the pool had lost every disk at once, the diagnostic instinct that reaches for “eight independent drives don’t fail simultaneously” needed to fire before the instinct that reaches for “restore from backup.” A whole pool showing all members REMOVED at once is almost always a path problem, not a media problem. And a path problem is recoverable at the software layer if the labels on the disks are still intact.</p><p>I checked. camcontrol devlist showed all eight drives. Their GPTIDs - the ZFS-tracked partition identifiers - had reappeared. zdb -l on each of the drives showed valid ZFS labels naming Pool-A as their pool. SMART data on each of the eight came back clean. Nothing about the disks themselves said they were unhealthy.</p><p>The pool’s UNAVAIL state was a stale view. zpool clear on Pool-A brought it back ONLINE, all eight members ONLINE, no permanent errors. From the ZFS layer&#39;s perspective, the pool was fully recovered.</p><p>Except SMB access from the workload was still broken.</p><p>The layer-5 lesson to come is that below ZFS is the SAS and CAM path, and that’s the layer that had interrupted. The layer-4 lesson is smaller but easy to miss: <strong>pool state and consumer state can diverge</strong>. Recovering the pool at the ZFS layer doesn’t automatically recover the services consuming that pool. Samba, in particular, holds ZFS VFS handles that go stale when the pool goes away and don’t refresh when the pool comes back. From Samba’s point of view, the dataset it was serving still didn’t exist, even though zpool status said it did.</p><p>Restarting the SMB service — service samba_server restart from the shell, or Stop/Start on the SMB service in the GUI - dropped the stale handles and rebuilt them against the newly-recovered pool. Windows access came back immediately. Layer 4 clean.</p><h3>// layer 5: physical storage-path state</h3><p>The last failure, and the hardest, was on the same TrueNAS server. A second pool — call it Pool-B — was also showing UNAVAIL, also with all eight members REMOVED. The diagnostic sequence I’d just run on Pool-A said the same story: eight members visible in camcontrol devlist, ZFS labels intact, SMART clean. Same recovery expected.</p><p>zpool clear on Pool-B did nothing. The pool stayed UNAVAIL. zpool reopen - the command that tells ZFS to re-scan the vdev&#39;s devices - did nothing. zpool online on individual members did nothing. None of the ZFS commands could bring the pool back.</p><p>At this point I looked at the kernel logs. dmesg was full of messages I hadn&#39;t seen on Pool-A. CAM Periph destroyed events. SCSI transport errors. The CAM layer - FreeBSD&#39;s Common Access Method, the driver layer that sits between ZFS and the SAS host bus adapter - was in a state where it had torn down its representation of Pool-B&#39;s devices and couldn&#39;t rebuild them without a full reset.</p><p>Then the situation started to spread. Other pools on the same TrueNAS server started showing UNAVAIL in the GUI, one after another. zpool status on any of them returned slowly, then hung. Trying to interact with the affected pools started hanging the shell that ran the commands. The system was becoming unresponsive.</p><p>This is a below-ZFS problem. ZFS can only manipulate what CAM can present to it, and CAM can only present what it can reach through the SAS layer. When the SAS/CAM path wedges, ZFS commands become impossible not because there’s anything wrong with ZFS but because the layer below it has lost track of what devices exist. There is no ZFS-level command that fixes a stuck CAM layer.</p><p>The clean recovery, once I’d worked out what I was looking at, was blunt: stop SMB (to prevent consumer-side hangs during the reset), hard power off TrueNAS1, fully power-cycle both attached JBOD shelves, let everything settle for a couple of minutes, then power the server back on. That resets the entire storage path from the server’s HBA outward — the SAS controller re-enumerates the shelves, the shelves re-enumerate their drives, CAM builds a fresh representation from scratch, and ZFS gets a stable substrate to work against again.</p><p>After the reset, both TrueNAS servers came back with all their pools ONLINE. Neither the drive labels nor the ZFS state had actually been damaged by any of this. What had failed was the layer under both of them.</p><p>The layer-5 lesson is the one that took me longest to internalise: <strong>below ZFS is a stack of physical-path components — SAS controller, cables, JBOD backplane, drive interfaces — and when any of those wedge, no amount of ZFS-level work can fix the situation from above</strong>. zpool clear cannot repair a hung CAM state. zpool reopen cannot re-enumerate devices the driver layer has torn down. Sometimes the only recovery is a full storage-path reset, from the HBA outward, and that means the server comes down for the duration.</p><p>The corollary is a diagnostic one: when zpool commands start hanging on a pool whose disks are physically present and healthy, don&#39;t reach for more ZFS-level tools. Look at CAM. Look at the kernel logs. Look for signs that the layer below ZFS has become the problem - because when it has, ZFS is the wrong tool for the job.</p><h3>// what the migration design did right</h3><p>The five failures across the day are the bulk of the piece, and worth being clear-eyed about: none of them were caused by the migration design being wrong. Every one was an existing condition or a physical-layer artefact that the fabric change happened to surface.</p><p>What the design did right was containment. The workload VLAN on the new fabric isolates this workload’s storage traffic from everything else running on the rack. The TOR-A/TOR-B split gives each end of the LAGG its own upstream. The workload servers and the storage servers on the new fabric can talk to each other. Nothing on the legacy network was affected by any of the failures above.</p><p>That containment is the reason a bad day for the workload’s storage stack was not also a bad day for anything else on the rack. The Kubernetes cluster kept running. The other DePIN workloads kept running. The rack’s other services didn’t notice the migration was happening beyond the fact that traffic patterns shifted subtly as the workload’s dependencies moved. That’s not a lucky outcome. It’s what per-service VLAN segmentation is <em>for</em>, and it worked as designed.</p><h3>// what I’d make instinctive</h3><p>Five doctrine points came out of the day, one per layer.</p><p><strong>Layer 1 — network reachability.</strong> Switch-side signals and host-side signals are two different things. The switchport LED tells you the link is electrically up; only the host OS’s network stack tells you the link is usable. Every migration checklist item that says “verify link” needs to be explicit about which side is being verified. Both sides. Always.</p><p><strong>Layer 2 — service reachability.</strong> SMB will happily advertise a share whose backing dataset is missing, and the error message clients get is deliberately opaque. Validating an SMB migration means validating that each share can not only be seen but can also be opened and read from — because “visible in the share list” and “backed by a real dataset” are two different states.</p><p><strong>Layer 3 — middleware view vs kernel truth.</strong> When a management layer caches state over a kernel-level source of truth, the cache can drift. Export/Import in the TrueNAS GUI is the standard way to refresh a pool’s middleware view after something upstream has changed. Recognise the situation early — GUI shows OFFLINE, zpool import shows ONLINE - and don&#39;t waste time trying to fix the middleware view through the middleware. Cycle it.</p><p><strong>Layer 4 — ZFS state under SAS path interruption.</strong> A pool showing every member REMOVED simultaneously is almost always a path event, not a media event. Before touching backup recovery or destructive commands, verify the disks with camcontrol devlist, verify the labels with zdb -l, verify SMART. If the disks are healthy and the labels are intact, zpool clear is likely all that&#39;s needed. Also - don&#39;t forget that pool recovery isn&#39;t service recovery. If Samba was serving the pool, it needs to be restarted to drop stale VFS handles.</p><p><strong>Layer 5 — physical storage-path state.</strong> ZFS can only work on what CAM can present. When CAM wedges, no ZFS command will get through. Kernel logs — specifically CAM Periph destroyed and SCSI transport errors - are the signal that the problem has moved below ZFS. Recovery may require a full physical power-cycle of the storage path from the HBA outward. Plan for that possibility on any migration involving SAS-attached JBODs.</p><h3>// closing</h3><p>The <a href="https://jjozzietech.com.au/rack-power-migration/">rack power migration piece</a> closed on the observation that a change plan can be right and still be incomplete — the plan describes what you know, and the rack describes what actually is, and the gap between them is where changes reveal themselves. This piece is that observation transposed one axis. A layer can be green and the whole still doesn’t work, because storage isn’t one thing. It’s a stack, and each layer of the stack has its own way of failing, its own way of appearing healthy, and its own validation step that has to be run before the layer above it can be trusted.</p><p>The migration itself finished the day it was supposed to. Both TrueNAS servers on the new fabric, all pools ONLINE, all shares serving, the workload back in operation on the new fabric. What actually got done that day wasn’t just a network cutover — it was a five-layer diagnostic, retrospectively organised into a pattern I could name.</p><p>If you’re running a storage migration and your validation checklist has one entry per host, you have four fewer entries than you need.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/truenas-migration-five-storage-layers/"><em>https://jjozzietech.com.au</em></a><em> on September 22, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa67d3a09f19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[/etc/environment is not read by systemd, and I believed it was for a month]]></title>
            <link>https://medium.com/@jjozzietech/etc-environment-is-not-read-by-systemd-and-i-believed-it-was-for-a-month-0109b883a86d?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/0109b883a86d</guid>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[systemd]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 21:15:32 GMT</pubDate>
            <atom:updated>2026-09-17T20:30:13.002Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_lPKeMHKZ7oHLUWS.png" /></figure><p>UV_USE_IO_URING=0 was supposed to be the fix. In late August the Flux Cumulus fleet running on Ubuntu 22.04 picked up a nasty failure mode on Node 1: npm install entering uninterruptible sleep, refusing to die, and taking the node from CONFIRMED to expired inside a few hours. The kernel journal named the io_uring submission-queue polling thread:</p><pre>[Fri Aug 21 08:22:44 2026] INFO: task iou-sqp-1334:1346 blocked for more than 120 seconds. <br>[Fri Aug 21 08:22:44 2026] Not tainted 5.15.0-187-generic #197-Ubuntu <br>[Fri Aug 21 08:22:44 2026] task:iou-sqp-1334 state:D stack: 0 pid: 1346 ppid: 1333 flags:0x00004000 <br>[Fri Aug 21 08:24:45 2026] INFO: task iou-sqp-1334:1346 blocked for more than 241 seconds.</pre><p>Same task, same PID, still blocked two minutes later. state:D - uninterruptible sleep. SIGKILL does nothing to a task blocked inside the kernel. ppid: 1333 - the parent is the npm install shell.</p><p>Node 20’s libuv routes async filesystem work through io_uring, and npm install is almost entirely async filesystem work. On this kernel that combination produced the deadlock captured above. UV_USE_IO_URING=0 tells libuv not to use io_uring - the variable is documented in Node&#39;s CLI reference, implemented by a dependency, with no stability guarantees attached. Set it before Node starts and the path that hangs never runs. I did not chase the kernel-side root cause: a reproducible trigger and a workaround that eliminated it was enough at the time.</p><p>I set it in /etc/environment on every node and verified it was in place.</p><p>For most of the next month, it was never in effect at boot on any node.</p><h3>// the verification that proved the wrong thing</h3><p>I verified the fix in August by reading the pm2 God Daemon’s environment directly out of /proc. pm2 kill &amp;&amp; pm2 list to force the daemon to restart, then sudo cat /proc/$(pgrep -f &quot;God Daemon&quot; | head -1)/environ | tr &#39;\0&#39; &#39;\n&#39; | grep UV_USE. The variable was there. The daemon had it.</p><p>The check was correct. What it proved was not what I thought.</p><p>/etc/environment is read by PAM for login sessions. When I ran pm2 kill &amp;&amp; pm2 list in my SSH session, PAM had already given the shell the variable. The daemon pm2 list spawned was a child of that shell - it inherited the environment from its parent. My /proc check then read the child&#39;s environment and correctly reported the variable was present.</p><p>The daemon inherited from the shell, not from any durable configuration. It was a fact about a process that would not exist after the next reboot. The verification passed honestly, and it proved the wrong thing, because the object under test had been created by the act of testing.</p><p>That’s the lead beat of this piece: a verification that passes honestly and proves the wrong thing, because the object under test was created by the act of testing. Everything downstream is either mechanism or consequence.</p><h3>// what actually pulled the trigger</h3><p>The other thing worth knowing is what triggered Node 1’s August incident. The deadlock needed something to cause npm install to run. That something was in the standard FluxNode crontab from Step 7 of the deployment guide: @reboot sleep 120 &amp;&amp; pm2 restart flux. A fixed 120-second timer, set to fire into the middle of npm install on every single boot.</p><p>Not bad luck. A scheduled collision on a loop, with an unkillable process at the end of it.</p><p>The timer wasn’t Node 1 specific. Every node in the fleet ran the same line. Why only Node 1 hit the deadlock is plausible inference — it was the only node with a genuine long-running npm install for the timer to land in the middle of during that window - but that&#39;s inference, not finding. The cron entry itself was universal.</p><p>That cron entry is now removed; a */10 DOS-state monitor covers restart recovery instead.</p><h3>// why /etc/environment never reached the daemon</h3><p>At the next reboot after that August verification, systemd resurrected the pm2 daemon. The generated unit’s environment block set exactly two variables:</p><pre>Environment=PATH=/home/flux/.nvm/versions/node/v20.9.0/bin:... <br>Environment=PM2_HOME=/home/flux/.pm2 <br>ExecStart=/home/flux/.nvm/.../pm2/bin/pm2 resurrect</pre><p>PATH and PM2_HOME. No UV_USE_IO_URING.</p><p>The mechanism: systemd’s system manager doesn’t read /etc/environment. PAM does, for login sessions. systemd does not. A daemon resurrected by systemd at boot inherits exactly what the unit file declares - which in this case was two variables that did not include the one I was relying on.</p><p>That’s the kind of thing you either know or you don’t. I didn’t.</p><h3>// the fix</h3><p>The fix is a systemd drop-in. On every node:</p><pre>sudo mkdir -p /etc/systemd/system/pm2-flux.service.d/ <br>sudo tee /etc/systemd/system/pm2-flux.service.d/override.conf &gt; /dev/null &lt;&lt;&#39;EOF&#39; <br>[Service] <br>Environment=UV_USE_IO_URING=0 <br>EOF <br>sudo systemctl daemon-reload</pre><p>Drop-ins under /etc/systemd/system/ survive unit regeneration and application updates. The variable is now part of the unit&#39;s declared environment, so the daemon inherits it whether it was started by hand or resurrected at boot.</p><p>No restart is needed. The drop-in applies at the next boot — which is exactly when the workaround is needed. Applied to all seven nodes on 10–11 September, no node leaving CONFIRMED during the change.</p><p>Worth noting the fleet runs Node v20.9.0. From 20.11.1 onward io_uring is opt-in rather than opt-out, so a reader on a current Node likely isn’t exposed to this at all — the drop-in is protection for the version we’re actually on.</p><h3>// exposed, never bitten</h3><p>With the fix in place, I swept the kernel journals across every retained boot on all seven nodes, counting iou-sqp and hung-task events:</p><pre>for b in $(sudo journalctl -k --list-boots | awk &#39;{print $1}&#39;); do <br>  printf &#39;%3s %s\n&#39; &quot;$b&quot; \ <br>     &quot;$(sudo journalctl -k -b &quot;$b&quot; --no-pager 2&gt;/dev/null | \ <br>        grep -c &#39;iou-sqp\|blocked for more than&#39;)&quot; <br>done</pre><p>Node 1 showed 67 hung-task reports across five consecutive boots in a three-hour window on 21 August, then zero — including the boot that spanned the entire unprotected month. The rest of the fleet: zero across every retained boot.</p><p>The gap cost nothing. But that isn’t the framing. The @reboot timer was live on every node for the whole month. What kept the fleet from a repeat wasn&#39;t the missing workaround - it was that nothing landed in the middle of an npm install window on any other node in that time. Exposed, never bitten.</p><p>The <a href="https://jjozzietech.com.au/rack-power-migration/">rack power migration piece</a> landed on a rhyming lesson about a different safety measure — an autostart flag disabled for a reason, then quietly missing for weeks after that reason no longer applied. The pattern is the same: nominal protection, silent absence, no cost until an unrelated event exposes both.</p><h3>// the truncation trap</h3><p>Immediately after writing the drop-in on the first node, I ran the verification I should have run in August:</p><pre>systemctl show pm2-flux.service -p Environment</pre><p>It returned a single line, truncated at terminal width, with a trailing &gt; and the pager sitting at (END). All I could see was PATH. The drop-in had worked; the output couldn&#39;t show it.</p><p>Environment is emitted as one space-separated line, and any pager that folds long lines will truncate it. The fix is trivial: systemctl show pm2-flux.service -p Environment --no-pager | tr &#39; &#39; &#39;\n&#39; | grep UV_USE. Cost of the trap: about a minute of thinking I had to rewrite a drop-in that was already correct.</p><p>The instrument lying twice in one investigation: once at a cost of a month, once at a cost of a minute. Both times the tool did exactly what it was asked. Both times the framing of the question was where the mistake sat.</p><h3>// one more thing from the same three commands</h3><p>The same systemctl show output also revealed the unit&#39;s ExecStart: pm2 resurrect, restoring the process list from ~/.pm2/dump.pm2. Anyone applying the older /etc/environment fix pattern - pm2 kill to force the daemon to inherit the variable - walks into a related trap: kill without saving first, and the resurrect at boot restores an empty process list. The node boots with no application.</p><p>pm2 save --force after any kill. One command, one sentence.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/etc-environment-systemd/"><em>https://jjozzietech.com.au</em></a><em> on September 15, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0109b883a86d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The wildcard cert saga — eight days from expiry, three failed attempts, one flag]]></title>
            <link>https://medium.com/@jjozzietech/the-wildcard-cert-saga-eight-days-from-expiry-three-failed-attempts-one-flag-460dc9028935?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/460dc9028935</guid>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[lets-encrypt]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[homelab]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 25 Aug 2026 21:15:21 GMT</pubDate>
            <atom:updated>2026-08-27T19:36:40.564Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I7SZxUTF-fVm1DxV.png" /></figure><p>Cert-expiry monitoring is one of those systems that runs quietly for months and then, suddenly, starts asking for your attention with growing volume. On the Akash provider running out of the Sydney rack, the daily-check script had been raising the same warning for a week — quiet drumbeat, background noise until it wasn’t. The wildcard cert covering the provider’s public ingress had eight days to live.</p><p>Eight days is not a crisis. It’s not comfortable either. The comfortable window for renewing a wildcard cert that fronts real tenant workloads is measured in weeks, not days, and I did not want to be doing this work at expiry-minus-hours with tenants noticing. So it moved to the top of the queue on 29 July, expiry set at 6 August.</p><p>The first instinct was the wrong one. cert-manager is normally auto-magical. It watches its Certificate resources, notices when they’re within the renewal window, kicks off a fresh order against Let’s Encrypt, and the operator finds out about it via absence — no notification because nothing broke. When the automation goes quiet, the reflex is to assume it just needs a nudge: restart the pod, re-issue the CertificateRequest, watch the reconciler pick things up again.</p><p>That reflex was wrong for reasons the first hour of investigation would surface. But the deeper story is what surfaced after that. Three failed renewal attempts. Root causes distributed across four different components — DNS provider, cert-manager, Cloudflare, acme.sh. Rate-limits triggered by the diagnosis itself. And ultimately a fix that landed as a single flag on an already-working tool, once the problem was framed correctly.</p><p>This piece is that story. The 8-day countdown, the three attempts that didn’t work, the reframing that ended the incident in about half an hour of active work after days of distributed-bug hunting, and the four lessons that stuck.</p><h3>// the first hour: what killed the renewal cron</h3><p>Before any renewal attempt, the first hour went to config review. If cert-manager was supposed to auto-renew and hadn’t, the question wasn’t “what’s wrong with the renewal now” — it was “what’s been wrong for weeks that stopped the renewal from happening at all.”</p><p>The acme.sh account configuration held the answer. Credentials were CF_Key + CF_Email - Cloudflare&#39;s Global API Key model. Those credentials had degraded, silently rejected for reasons not fully diagnosed at the time. The renewal cron had been executing on schedule the entire time. It had been failing at authentication the entire time. Neither of us - cron nor operator - had noticed. The immediate fix was rebuilding on scoped API Tokens rather than forensically reconstructing the failure cause - an auth-mode swap that doubled as overdue credential rotation.</p><p>Discovery mechanism matters. This wasn’t found via log review. acme.sh&#39;s cron logs did contain terse authentication errors, but they were the kind of error that reads as normal noise if you&#39;re not already suspicious. The smoking gun was the config file itself - one line, CF_Key=..., unambiguous once you know Global Keys are being deprecated.</p><p>Auth was the technical trigger. The renewal cron dying was the symptom. The <em>structural</em> problem was that nobody was watching whether the cron actually succeeded. The cron was scheduled. The cron ran. The exit code was ignored. The distinction between “the cron executed” and “the cron did what it was supposed to do” was the gap monitoring should have caught, and didn’t.</p><p>This is the class of failure that turns a routine credential rotation — normally a fifteen-minute job you do during a Cloudflare account maintenance session — into an 8-day fire drill. The credential rotation itself was trivial. The consequence of the credential rotation having happened silently, weeks earlier, without any monitoring signal that it had happened, is what pushed the deadline.</p><p>Cron exit codes without monitoring aren’t automation. They’re theatre.</p><h3>// combining fixes: credential rotation plus architecture change</h3><p>The natural response would have been to rotate to a scoped API Token, restart the renewal cron, and consider the incident closed. That’s not what happened. The credential rotation got combined with a larger architecture change — one that had been on the “should probably do this eventually” list for months.</p><p>The reasoning was structural. Cloudflare’s push away from Global API Keys wasn’t arbitrary. Global Keys granted full-account write access with no audit trail — one credential capable of every action across every zone on the account, with no way to scope permissions or track which automation had done what. Scoped API Tokens fixed both problems: zone-limited, purpose-specific, individually revocable.</p><p>That same reasoning also applied to the DNS-01 challenge path itself. The existing wildcard cert renewal used acme.sh&#39;s default TXT placement - write records to the domain&#39;s authoritative zone at Cloudflare, remove them after validation. That mode required Global-API-Key-equivalent write access to the entire zone. A safer architecture would isolate the challenge writes to a dedicated CNAME-delegated location - SiteGround holding the primary nameservers, _acme-challenge CNAMEs delegating just the ACME work to Cloudflare, and API Tokens scoped to only that delegated zone.</p><p>Doing credential rotation and DNS architecture change as one combined fix meant less integration risk, less cleanup, and fewer moving parts to reason about afterwards. Two-in-one is usually a false economy — but here, the two changes were driven by the same underlying reasoning, and the fix stack for both landed in the same session.</p><h3>// first failed attempt: cert-manager’s zone-walking bug</h3><p>With API Tokens in place and the CNAME delegation planned, the first renewal attempt went through cert-manager. And this wasn’t a single-vector change. Three vectors were shifting at the same time.</p><p>The tool was changing: cert-manager as the new standard for the Kubernetes stack, replacing acme.sh. The CA was changing: Let&#39;s Encrypt as the new issuer, replacing the incumbent ZeroSSL. The challenge architecture was changing: CNAME delegation replacing acme.sh &#39;s default TXT placement.</p><p>Each had its own accumulated “should probably do this eventually” reasoning, and two of them were entangled. Wildcard issuance needs DNS-01 regardless of CA — the ZeroSSL setup was already doing that. What made CNAME delegation a prerequisite rather than a nicety was the credential model: default TXT placement required write access to the whole zone, so scoping the token to anything narrower meant moving the challenge records somewhere that existed only for that purpose.</p><p>Whether that was a wise bundling decision is worth naming: it was probably too many changes at once. Multi-vector migrations amplify the diagnostic difficulty when they fail, because you don’t immediately know which vector is the problem. That was the lesson not yet learned going into the first attempt.</p><p>The attempt failed. But not in a way that produced an obvious error.</p><p>The failure signature was subtle. cert-manager wasn’t returning an obvious error. The CertificateRequest sat in a Pending state — no failure event, no red status, nothing that would light up a dashboard. cert-manager’s Cloudflare provider logs showed it querying zones, matching them, and then attempting to place the TXT record for the ACME challenge at a location. And Let’s Encrypt’s validator was looking for the TXT record at a different location. The challenge never validated. But nothing about the failure <em>looked</em> like a failure — every component was doing what its logs said it was doing.</p><p><em>It wasn’t “this is broken.” It was “this looks like it’s working but the challenge never validates.”</em></p><p>That’s the class of failure that eats hours. When every log line reads as healthy activity, the diagnostic instinct — “find the error message and follow it” — has nowhere to go. There is no error message.</p><p>The trace came from reading cert-manager’s Cloudflare provider source directly, understanding its zone-detection logic, and then cross-referencing that logic against the actual DNS zone contents of the affected domain. cert-manager’s provider walks the DNS hierarchy from the record name up toward the root, looking for the “authoritative zone” — the zone where it should write the TXT record. That walk followed a CNAME chain. And the CNAME chain led to .snapshots.</p><p>.snapshots is a SiteGround backup-access hostname. It&#39;s part of the shared-hosting furniture - installed by the hosting provider for their own backup tooling, invisible to you because you didn&#39;t set it up and never think about it. If you&#39;re not a SiteGround-hosted domain operator you&#39;d never look at it. If you <em>are</em>, you&#39;d still never look at it, because it has nothing to do with anything you actively manage.</p><p>Except now it did. cert-manager’s zone-walker followed the .snapshots CNAME, decided that was the authoritative zone for the ACME challenge, and placed the TXT record somewhere Let&#39;s Encrypt would never look. Every piece was behaving as documented. The whole still didn&#39;t work.</p><p>The teaching moment isn’t about .snapshots specifically. It&#39;s about the class of failure where every component reads healthy and the composition fails silently. That class of failure needs a different diagnostic instinct than error-reading. It needs a diagnostic instinct closer to what network engineers use for packet capture - read the actual traffic between components, don&#39;t trust either component&#39;s log about what it thinks it&#39;s doing.</p><h3>// second failed attempt: the retry loop became the story</h3><p>With the .snapshots CNAME identified as the zone-walking trap, the second attempt tried a workaround: manually specify the zone cert-manager should target, bypassing the auto-detection. The attempt failed in a very different way.</p><p>cert-manager’s default behaviour when a challenge fails is to back off and retry. Backoff intervals lengthen with each retry, but the retries never stop until the CertificateRequest is either manually paused or succeeds. That default is correct for transient failures — network hiccups, brief provider outages, momentary rate-limits. It’s harmful for persistent failures on already-loaded infrastructure, where the retry adds pressure without being the sole cause of the problem.</p><p>The second attempt ran for about twenty minutes before I noticed it had shifted into a distinct failure mode. cert-manager’s default backoff made each retry look reasonable in isolation — a single request every few minutes, exponentially spaced. Then Cloudflare 429 responses started appearing in the cert-manager pod logs — rate-limit rejections.</p><p>Here the mechanism deserves care. Cloudflare’s global rate limit is 1,200 requests per five-minute window, counted account-wide across dashboard sessions, API tokens, and legacy keys combined. When the limit breaks, everything talking to that account gets blocked for the remainder of the window. cert-manager’s backoff-driven retries alone weren’t generating anything near 1,200 requests in five minutes — but the retry loop was probably enough additional load, on an account already carrying baseline API traffic from other automation, to push the account over the threshold.</p><p>Retries were the last straw, not the sole cause. Exact contribution unknown — request-rate telemetry from the incident window wasn’t captured, and I can’t reconstruct precisely how much of the budget the retry loop was consuming versus what was already in flight.</p><p>That makes the lesson more useful, not less. Account-wide rate-limit budgets mean isolated request rates aren’t the whole picture. A retry loop that would be harmless on a quiet account can be the tipping point on a busy one. The operator awareness that matters: not “how many requests does this thing generate” but “how many requests does this thing generate <em>on top of what’s already there.</em> “</p><p>The 429s in the cert-manager logs meant other operations against the same account would have been affected during that window too — that’s how account-wide limits work by definition. Whether specific other automation on the account actually noticed depends on whether it was making requests during the limited window; I didn’t directly correlate specific other-tool failures during the incident.</p><p><em>The retry loop had become part of the problem.</em></p><p>Not the primary cert renewal being wrong — that was still the thing being diagnosed. But the diagnosis, in the form of cert-manager retrying against a persistently failing challenge, was contributing to a rate-limit condition that affected the whole account. Diagnostic activity was making things worse in ways I couldn’t fully see.</p><p>The immediate fix was to pause the CertificateRequest — one kubectl annotate to stop cert-manager retrying - and wait for the rate-limit window to clear before touching Cloudflare again. That&#39;s the mechanical fix. The operator lesson is broader.</p><p>Retry loops without circuit breakers can amplify account-wide rate-limit conditions. cert-manager’s default retry behaviour is fine for the failure modes it was designed for. It is not fine for the failure mode where the retry adds pressure to a rate-limit budget that’s already tight for reasons outside the tool’s visibility.</p><p>Circuit breakers — “stop retrying after N consecutive failures of the same type” — exist as a concept in most well-designed retry systems. cert-manager’s Cloudflare provider doesn’t have one aggressive enough to matter in this situation. Knowing that in advance would have saved twenty minutes of adding pressure to an already-loaded account.</p><h3>// the pivot that wasn’t really a pivot</h3><p>At this point the shape of the incident changed. cert-manager’s Cloudflare provider bug — the zone-walking through .snapshots - wasn&#39;t going to be a fix I could apply in-session. It was a bug in the provider code, and correcting it would need an upstream patch, testing, release. That timeline was days minimum, and the cert had single-digit days until expiry.</p><p>The path forward was cert-manager off the critical path. Which meant collapsing the three-vector migration down to two.</p><p>Framing that as a “pivot” isn’t quite right. Two of the three vectors — Let’s Encrypt as CA, CNAME delegation as challenge architecture — were staying. The tool vector was reverting. cert-manager was coming off; acme.sh was going back on. But acme.sh wasn&#39;t going to be doing what it had been doing before. It had been talking to ZeroSSL with default TXT placement. Now it would talk to Let&#39;s Encrypt with CNAME delegation.</p><p>Same tool. New work.</p><p>The mechanical work was smaller than that reframe implies. acme.sh was already installed. Its cron entry was already running - dead, from the failed authentication weeks earlier, but present. The credentials just needed rotation from Global API Key to API Token, already planned as part of the combined fix. The CA switch was one flag on the renewal command. And the challenge path needed pointing at the right place via --challenge-alias.</p><p>Or rather, needed to be pointed at the right place via --challenge-alias. The third obstacle was already waiting.</p><h3>// third obstacle: the --challenge-alias discoverability gap</h3><p>The third obstacle isn’t quite fair to call a “failed attempt.” It was more that the tooling had exactly the feature the situation needed, and the documentation didn’t foreground it that way.</p><p>acme.sh supports DNS Alias Mode via the --challenge-alias flag. When you use CNAME delegation to redirect ACME challenges to a different zone - exactly the architecture the combined fix required - --challenge-alias tells acme.sh where to write the TXT record. Without it, acme.sh writes to the default location. With it, acme.sh writes to the aliased location. The flag is documented. The functionality works cleanly once configured.</p><p>But finding it required already knowing that’s what you were looking for.</p><p>The acme.sh DNS Alias Mode wiki page describes the mechanism - <em>&quot;place TXT records at an aliased location via CNAME delegation.&quot;</em> It does not describe the mechanism as &quot;the solution to using base-zone NS delegation with a non-API DNS provider.&quot; Someone reading the DNS Alias Mode page top-to-bottom, without already knowing they need CNAME delegation for their specific problem, would come away understanding the mechanism but not connecting it to their situation.</p><p>I found it via a specific search for the problem, not by reading through the docs comprehensively. If I hadn’t already been sitting with “I need to write TXT records at a location other than the default because SiteGround controls my authoritative nameservers and doesn’t expose a DNS API,” the wiki page would not have surfaced as relevant.</p><p>That’s a documentation gap worth naming. It’s not the same class of gap as missing documentation or wrong documentation. The docs exist. The docs are correct. The docs describe the mechanism accurately. What’s missing is the connection between the mechanism and the class of problem it solves. Discoverability is a distinct axis from correctness, and the acme.sh DNS Alias Mode page is a case where correctness is high and discoverability is low.</p><p>Naming that gap publicly is more useful than pretending the answer was always obvious.</p><h3>// the turning point</h3><p>The turning point in the incident wasn’t a moment of insight. It was a moment of reframing.</p><p>After the third obstacle, I sat down with the acme.sh docs again. Second read-through. Different question this time. The first read had asked &quot;how do I do DNS-01 challenge with acme.sh?&quot; - that had produced generic DNS-01 documentation, general-purpose, not obviously connected to the specific architecture I now had in place. The second read asked &quot;what&#39;s the flag for the setup I&#39;ve just configured?&quot; - which is a much smaller, much more specific question.</p><p>The problem was framed correctly this time. So the answer was findable.</p><p>--challenge-alias surfaced immediately once I knew to look for the flag that told acme.sh &quot;the TXT records go to a different zone than the domain&#39;s default authoritative one.&quot; That&#39;s the whole feature description in one line, and it maps directly to what the CNAME delegation was doing at the DNS layer.</p><p>What made this a turning point rather than “another thing to try” wasn’t the flag itself. It was the shape of the fix collapsing at that moment.</p><p>The cert-manager path had been: diagnose the distributed bug across the zone-walker and the DNS layer, wait for the upstream patch, deploy the patched version, hope no new failures surface. Multi-day, multi-component, uncertain.</p><p>The acme.sh path was: add two CNAMEs at SiteGround, add one flag to an already-working cron entry, restart the cron, done. Single-digit hours, two-component, deterministic.</p><p>When the plan collapses from “diagnose distributed bug across three components” to “add a flag to an already-working tool,” you know before you run it whether it’s going to work. That’s the anti-heroic version of the story. Not a moment of insight. A moment of reframing that made the answer obvious.</p><h3>// the fix that landed</h3><p>The mechanical fix was smaller than the diagnosis suggested it would be.</p><p>Two CNAMEs at SiteGround. _acme-challenge for the base domain pointing to a Cloudflare-hosted alias zone. The wildcard-challenge variant pointing to the same alias zone. SiteGround&#39;s DNS management supports CNAME creation via their control panel; no API required.</p><p>At Cloudflare, a dedicated zone for the alias, with a scoped API Token that had write access only to that zone. Not the domain’s actual authoritative zone. Not the full account. Just the ACME challenge zone.</p><p>At acme.sh, the credential rotation from Global API Key to API Token in the account configuration. Two additions to the renewal command in the cron: --server letsencrypt to explicitly target Let&#39;s Encrypt (rather than acme.sh&#39;s ZeroSSL default), and --challenge-alias pointing at the delegated location. That&#39;s the whole acme.sh-side change.</p><p>At the Kubernetes layer, no change was needed. The renewed certificate would be dropped into the same Kubernetes secret acme.sh had been maintaining all along. The Gateway resource watching that secret would pick up the update automatically. Which is where NGINX Gateway Fabric earned its keep in this incident - but that&#39;s the next section.</p><p>The renewal ran clean the first time under the new configuration. acme.sh requested the challenge from Let&#39;s Encrypt, wrote the TXT record to the alias zone at Cloudflare via the scoped API Token, Let&#39;s Encrypt validated against the correct location (because CNAME delegation resolved correctly from Let&#39;s Encrypt&#39;s side), and the fresh wildcard cert issued. Total active work from &quot;the flag surfaced&quot; to &quot;renewed cert deployed&quot; was about half an hour - most of that spent waiting for SiteGround&#39;s NS delegation to propagate.</p><p>Two DNS records. Two flags. One credential rotation. And two vectors of the original migration (Let’s Encrypt as CA, CNAME delegation as challenge architecture) preserved cleanly — cert-manager came off, but the harder-to-reverse changes stayed.</p><h3>// NGINX Gateway Fabric added one thing for free</h3><p>One small win worth naming, because it saved a step during the incident that would have otherwise been manual.</p><p>The certificate that acme.sh renews lives in a Kubernetes secret. The ingress layer that serves the certificate to inbound HTTPS traffic - in this stack, NGINX Gateway Fabric - reads that secret and terminates TLS with it.</p><p>Standard nginx-ingress-controller requires that when the secret changes, the ingress-controller pods either restart or be signalled to reload their configuration. Some deployments handle this via a sidecar that watches Kubernetes secrets and triggers reloads. Others handle it via periodic reload jobs. Some handle it by not handling it, and letting operators reload manually when they remember. All of those approaches add moving parts.</p><p>NGINX Gateway Fabric handles secret updates automatically. When the wildcard cert secret updated with the new acme.sh-issued certificate, the Gateway picked it up without any manual reload, without a sidecar, without a periodic job. TLS termination started using the new cert. No operator action required.</p><p>Not the story of this incident. But worth naming as a small win: the ingress layer was one of the few things in this whole stack that didn’t need touching during the recovery. When something goes right during an incident, it’s worth naming — especially when the same functionality in a sibling tool would have added another manual step under time pressure.</p><h3>// what I’d make instinctive</h3><p>Four lessons went into the runbook after this.</p><p><strong>Cron exit codes need monitoring, not just cron execution.</strong> The renewal cron was scheduled correctly. It ran on schedule for weeks. Cron ran successfully — the shell command executed, cron logged the execution, the cron system considered the job complete. And every single execution was failing at authentication. Cron-ran-successfully monitoring is not the same as job-succeeded monitoring. Any critical automation — cert renewal, backup jobs, replication tasks, anything where “the cron ran” doesn’t imply “the intended work happened” — needs both.</p><p><strong>“Looks like working” is a distinct failure mode from “broken.”</strong> cert-manager’s zone-walking failure produced no error message, no failed status, no red dashboard indicator. Every log line read as healthy activity. The composition failed silently. That class of failure needs a diagnostic instinct closer to network engineering — read the actual traffic between components, don’t trust either component’s log about what it thinks it’s doing. Error-reading skills don’t apply when there are no errors.</p><p><strong>Retry loops without circuit breakers can amplify already-loaded conditions.</strong> cert-manager’s default retry behaviour was correct for transient failures and wrong for persistent ones. Twenty minutes of retrying against an already-loaded Cloudflare account was probably enough additional load to push the account into rate-limit rejection — not the sole cause, but a contribution the tool had no visibility into. Circuit breakers matter not just because they stop wasted work, but because they prevent the retry itself from becoming part of the failure. Know which of your automation has circuit-breaker behaviour and which doesn’t.</p><p><strong>Discoverability is a distinct axis from correctness.</strong> --challenge-alias works. Its documentation exists. What&#39;s missing is the connection between the mechanism the docs describe and the class of problem the mechanism solves. That gap turned a fifteen-minute config change into an hour of doc-reading. Documentation authors owe their readers not just correctness but discoverability - the framing that lets a reader find the right feature by describing their problem. Both are needed.</p><p>None of these are theoretical. Every one of them was learned by doing the diagnostic in the wrong order the first time.</p><h3>// closing</h3><p>Cert-manager saga posts tend to read as heroic sysadmin folklore. Someone stares at cryptic errors for days, has a moment of insight, cracks the case, writes it up as a triumph. The genre is well-established.</p><p>Most of them shouldn’t be. The plans that collapse to a flag or a config change were never really the plans they looked like during the diagnostic phase. When the fix goes from “three-component distributed bug” to “two DNS records plus a config line,” the diagnostic wasn’t wrong — the <em>frame</em> was.</p><p>Reading this piece back, the whole incident could have ended on day one if the frame had been correct on day one. acme.sh was already in the stack. CNAME delegation was already a documented architecture pattern. --challenge-alias was already a supported flag. The pieces were all there. What wasn&#39;t there was the framing that connected them. cert-manager as the migration target was pulling attention toward the wrong tool for the specific problem the incident actually presented.</p><p>Article 8’s <a href="https://jjozzietech.com.au/gpu-passthrough-proxmox/">GPU passthrough piece</a> landed a similar-shaped observation about a different domain: the error pointed at the GPU, the culprit was the HBA. Different tools, different failure mode, same operator lesson. When the diagnostic is spending time in the wrong place, the frame is usually the problem — not the diagnostic.</p><p>If your renewal cron is running without monitored exit codes, that’s the single highest-leverage thing you can fix this week. Not because it prevents this class of failure — but because it turns silent multi-week failures into loud immediate ones. Silent long failures are always worse than loud short ones. Fix the monitoring before you need it.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/wildcard-cert-renewal-dns-alias/"><em>https://jjozzietech.com.au</em></a><em> on August 25, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=460dc9028935" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The error pointed at the GPU. The culprit was the HBA. — a Proxmox passthrough recovery]]></title>
            <link>https://medium.com/@jjozzietech/the-error-pointed-at-the-gpu-the-culprit-was-the-hba-a-proxmox-passthrough-recovery-45c635a0eaff?source=rss-197f6d210c24------2</link>
            <guid isPermaLink="false">https://medium.com/p/45c635a0eaff</guid>
            <category><![CDATA[homelab]]></category>
            <category><![CDATA[virtualization]]></category>
            <category><![CDATA[proxmox]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[JJozzieTech]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 21:15:00 GMT</pubDate>
            <atom:updated>2026-08-20T19:51:46.329Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GqruCKR04ovir-aq.png" /></figure><p>The plan looked textbook. PCIe passthrough an NVIDIA RTX A4000 from a Proxmox 8 host into an Ubuntu 24.04 VM, giving a Filecoin sealing worker direct GPU access for the compute phases that need it. VT-d on the Xeon. IOMMU enabled at the kernel line. OVMF BIOS on the VM. vfio-pci bound to the device IDs. Machine type set to q35 for PCIe support. Standard sequence, followed by every guide, ordered exactly as those guides ordered it.</p><p>Then qm start returned, and the VM did not come up.</p><p>Not a kernel panic. Not a host lockup. Not a “device not found” error surfaced to the operator. QEMU exited from the host’s perspective and left nothing behind — no running guest, no console output, no obvious pointer at what had failed. The Proxmox web UI showed the VM as stopped, exactly as it had before I hit start.</p><p>The errors were there. They were just in a different terminal. dmesg -w on the host, running in a second window, produced a burst of IRQ allocation failures and vfio interrupt remapping errors scrolling past at VM startup, then silence. The host itself was completely fine throughout. Nothing was wrong with the box. Only the VM layer was broken, and it was broken in a way that read more like a misconfigured VM than a hardware conflict - which is exactly why the real cause took three sessions to identify.</p><p>The <a href="https://jjozzietech.com.au/homelab-data-centre-network-design/">segmentation piece</a> covered the network work that came before this. This piece is one of the workload-layer stories that followed — the GPU passthrough that took two evenings longer than it should have, because the error pointed at the GPU and the culprit was somewhere else entirely.</p><p>Two things this piece is not. It is not a step-by-step passthrough guide — there are plenty of those, and they are all fine until they aren’t. It is the story of what happens when the checklist runs clean and the passthrough still refuses to work.</p><h3>// what the GPU was actually there to do</h3><p>Filecoin sealing — the process of turning raw storage into provable, verifiable data on the network — is not one workload. It’s two compute phases with fundamentally different characteristics, and only one of them benefits from a GPU.</p><p>The first phase is SDR: Sequential Data Replication. It’s designed to be resistant to parallelisation — memory-latency bound, CPU-heavy, and immune to GPU acceleration by design. On this hardware, SDR takes three to four hours per sector regardless of what else is in the box. No GPU speeds it up. That is intentional at the protocol level, not a limitation of the implementation.</p><p>The second phase is where the GPU matters. Tree building — specifically the TreeRC computations that produce the Merkle tree structure used in Proof-of-Replication — is GPU-eligible. Without acceleration, TreeRC runs on CPU and adds another two to three hours per sector. With the A4000 handling it, the same work completes in roughly fifteen to twenty minutes.</p><p>The framing that gets used in Filecoin marketing — “GPU acceleration makes sealing faster” — is technically true and operationally misleading. The GPU doesn’t make sealing fast. It makes the second half of sealing fast. A storage provider without GPU acceleration is bottlenecked on tree computation as badly as on SDR, and the observable sealing throughput is roughly the SDR time plus the CPU tree time, sector by sector.</p><p>Where the GPU actually earns its place is in pipeline parallelism at scale. With the CPU pipeline running SDR on one sector while the GPU pipeline runs TreeRC on the previous one, sealing throughput becomes gated by SDR alone. That’s the operator-relevant payoff — not making any single sector faster, but decoupling the two phases so they can run against different sectors concurrently. For a small storage provider, that is the difference between sealing two sectors a day and sealing five or six.</p><p>That was what the A4000 was there to do. Which is why the VM not coming up was a problem.</p><h3>// the diagnostic that didn’t converge</h3><p>The investigation stretched across three sessions over about a week, mostly limited by the time I had rather than by the problem itself. Each session ran ninety minutes or so, and each one eliminated one correct-looking explanation without producing the actual cause.</p><p>Session one focused on the obvious: was the GPU actually bound to vfio-pci, and was the VM config correct?</p><pre>lspci -nnk -d 10de:24b0</pre><p>That output showed Kernel driver in use: vfio-pci - the host had surrendered the card, exactly as intended. No nouveau, no nvidia. The VM&#39;s PCI passthrough config referenced the correct BDF for the A4000, machine type was q35, OVMF was enabled, memory ballooning was off. Nothing in the config was wrong. That was session one.</p><p>Session two went down the driver blacklist and IOMMU group mapping path. /etc/modprobe.d/vfio.conf had the right device IDs listed. /etc/modprobe.d/blacklist.conf blocked nouveau, nvidia, nvidiafb, and snd_hda_intel. update-initramfs -u -k all had been run, and the host had been rebooted since. All correct.</p><p>Then the IOMMU group listing:</p><pre>find /sys/kernel/iommu_groups/ -type l | sort -V</pre><p>The A4000 at 03:00.0 appeared in one group with its HDMI audio companion at 03:00.1 - expected. But the same group also contained an IBM SAS2008 HBA. The HBA was managing the host&#39;s scratch storage, which meant it couldn&#39;t be passed through to the VM. And in an IOMMU group, all devices are the atomic unit of passthrough. You pass through the entire group or none of it.</p><p>That was where the pointer to the SAS2008 first appeared. But at that moment, I did not yet understand what shared-group membership specifically causes to fail. Group sharing is a widely documented passthrough concern, and the standard advice (“try the ACS override kernel patch”) felt like a hack for a build that would need to run reliably long-term. I closed session two aware the SAS2008 was somehow involved but not yet certain how.</p><p>Session three was the crystallisation. Not a eureka moment — a slow narrowing. Once every other explanation was eliminated, the IOMMU group problem was the only candidate left. But the specific mechanism — why exactly a non-passthrough device in the same group breaks the passthrough of the target device — needed one more step of understanding before the physical fix made sense.</p><h3>// FLR: what makes an IOMMU group problem actually a passthrough blocker</h3><p>IOMMU group membership alone doesn’t break passthrough. The Linux community’s shorthand — “everything in the group has to be passed through together” — is correct but incomplete. It describes the safety constraint, not the failure mode. Understanding what actually goes wrong requires knowing what QEMU does at VM startup.</p><p>When the VM starts and takes ownership of a passed-through PCI device, QEMU has to reset it — bring it to a known state before the guest OS initialises. The reset mechanism it relies on is Function Level Reset (FLR), a PCIe capability that allows a device to be reset in isolation without affecting anything else on the bus. FLR is the clean, safe reset. It’s what QEMU wants.</p><p>The A4000 supports FLR. The SAS2008 does not. And here is the mechanism: when QEMU tried to perform FLR on the A4000, it could not do so cleanly because the SAS2008 was in the same IOMMU group and could not itself be reset. The interrupt remapping infrastructure that isolates devices during passthrough could not correctly separate the GPU’s IRQ handling from the HBA’s, because from the IOMMU’s perspective the two devices were treated as a single reset domain. The IRQ allocation failed. The VM didn’t start.</p><p>The error pointed at the GPU because the GPU was the device QEMU was trying to bring up. The culprit was the HBA because the HBA was the device that couldn’t cleanly get out of the way.</p><p>FLR support is directly checkable from the host. For any PCI device, the reset_method sysfs file lists the reset mechanisms the kernel considers available:</p><pre>cat /sys/bus/pci/devices/0000:XX:XX.X/reset_method</pre><p>For the SAS2008, that file did not list flr among its supported methods. That was the confirmation the mechanism I was reasoning about actually applied to this hardware. Once confirmed, the fix stopped being a question of ACS overrides or kernel patches. It became a question of physical topology.</p><p>This is the class of understanding most passthrough guides skip. They tell you what to configure. They don’t tell you what to check when the configuration is right and the passthrough still fails. FLR-vs-non-FLR devices sharing an IOMMU group is the silent blocker at the heart of this failure mode.</p><h3>// moving the HBA to a slot that mattered</h3><p>The fix was physical. The SAS2008 had to move to a different PCIe slot — one that was not just physically distant from the A4000 but sat behind a genuinely separate PCIe root port. That distinction is critical and easy to miss.</p><p>Not every PCIe slot on a server board sits behind its own root port. Slot layout on the motherboard reflects physical trace routing, not IOMMU topology. Two adjacent slots may share a root port; two slots on opposite ends of the board may or may not. Moving the HBA one slot over on the same root complex would have done nothing — the IOMMU group membership would have been identical after reboot, and the failure would have repeated.</p><p>The target slot had to be chosen carefully. Two sources helped. The board’s manual documents the PCIe topology — which slots connect to which CPU root complex, and how the lanes are allocated across the PCH. The existing IOMMU group listing also carries information: any two devices that had shown up in <em>different</em> groups before must, by definition, be behind different root ports. A slot known to be safe for the HBA was a slot whose currently-installed device (or its adjacent devices) had shown up in a group separate from the A4000&#39;s.</p><p>The physical work itself was straightforward. Full shutdown of the host — no live PCIe hotswap for this class of card. Chassis open. HBA out of its original slot. SAS cables temporarily disconnected (they had to be re-plugged after the move — a small operational risk if any cable connector had degraded, but nothing did). HBA into the target slot on the other root complex. Chassis closed. Boot.</p><p>Confidence going into that boot was moderate, not certain. PCIe slot physical location does not guarantee separate root complex membership — the only real proof is what the IOMMU group listing shows after boot. Moving the card is a testable hypothesis, not a guaranteed fix. If the target slot had turned out to be on the same root complex as the A4000 anyway, I would have been back at square one, chassis open again.</p><p>The fix confirmed itself in layers.</p><p><strong>Structural first.</strong> After boot, the IOMMU group listing showed the SAS2008 in its own group, and the A4000 with only its audio companion at 03:00.1. That was the proof the physical move had produced the isolation it needed to produce. Before I even started the VM, I knew the group problem was resolved at the topology layer.</p><p><strong>Functional second.</strong> VM started. QEMU didn’t exit. Guest OS came up. nvidia-smi inside the VM reported the A4000 with its full 16 GB of memory and no error state. CUDA initialised. The card was talking to its driver.</p><p><strong>Ecological third.</strong> The sealing worker initialised, picked up a queued sector, and the first GPU-accelerated TreeRC task completed in fifteen minutes instead of three hours. That was the confirmation the whole endeavour had been about.</p><p>Three checks, each validating a different layer, each necessary. The structural check proved the topology; the functional check proved the driver stack; the ecological check proved the workload actually benefited. Skipping any one of them would have left the fix incompletely verified.</p><h3>// the second time was clean</h3><p>The Filecoin project was later scrapped for reasons unrelated to the GPU passthrough — project constraints outside this story. The A4000 came back into the general hardware pool. When MMX came online later as a farming workload, the passthrough pattern needed to run again on a different guest OS.</p><p>MMX is a Chia fork with an important twist relevant to storage layout: it supports both HDD plots and SSD plots, and the SSD plots are compressed. Farming compressed plots requires GPU-accelerated decompression at every proof challenge — the harvester has to expand the plot data on the fly to check whether it holds a winning proof. Without a GPU, compressed SSD plotting is effectively non-viable at scale. With one, the decompression happens fast enough that the harvester keeps up with challenges in real time.</p><p>MMX and Chia have always run on Windows VMs on this network — a convention that predates this build, driven purely by the maturity of the Windows GUI tooling around Chia’s ecosystem rather than any technical requirement. So the second passthrough build targeted a Windows guest instead of Ubuntu.</p><p>The build was clean. IOMMU group listing checked before touching physical hardware — target slot known-safe. Device IDs identified. VFIO binding confirmed. Windows installed with only the emulated VGA present, drivers applied after boot, GPU attached in a second pass to avoid the display blackout risk when the NVIDIA driver takes over primary output. Two extra CPU flags on the VM: hidden=1 to mask the hypervisor and avoid the notorious Error 43, and +pcid for TLB performance under memory-heavy workloads.</p><p>One clean build. No gotchas. Roughly one session, no diagnostic sessions after.</p><p>The pattern only reads as clean because the checks became routine. That is what the Filecoin project’s expensive lesson bought.</p><h3>// what I’d make instinctive</h3><p>Four things went into the runbook after this.</p><p><strong>Check IOMMU groups before touching physical hardware.</strong> The listing takes seconds to produce and tells you exactly what the passthrough constraints will be before you start. Every hour of build effort saved by checking the group listing first is an hour that doesn’t need to be spent later, chassis open, moving cards under time pressure.</p><p><strong>Non-FLR devices in the same group are silent blockers.</strong> IOMMU group membership on its own doesn’t fail the passthrough. It’s the combination of shared group and one device without FLR support that produces the specific failure mode that took me three sessions to identify. If your target device is in a group with a device that doesn’t support FLR, that group needs to be broken up before passthrough will work. The reset_method sysfs check is thirty seconds and catches this class of problem definitively.</p><p><strong>Physical slot choice is a topology question, not a proximity question.</strong> Adjacent slots may or may not sit behind separate root ports. Distant slots may or may not either. The board manual and the existing IOMMU group listing between them tell you which slots are actually independent. Guessing based on physical distance is worse than guessing at random, because it feels like a plan.</p><p><strong>Layered confirmation matters — structural, functional, ecological.</strong> A working passthrough is not the same as a working workload. The structural check (IOMMU group listing) proves the topology. The functional check ( nvidia-smi or equivalent) proves the driver stack. The ecological check (the actual workload running against real work) proves the whole system does what it was built to do. Skip any layer and the fix is only partly verified.</p><p>None of these are theoretical. Every one of them was learned by getting the diagnostic order wrong the first time.</p><h3>// closing</h3><p>The GPU passthrough pattern extends far beyond either of the workloads that surfaced in this piece. Any hardware acceleration that a VM needs — an AI or ML lab running CUDA training jobs, a transcoding server that needs hardware video encoders, a gaming VM that needs a discrete GPU, or a media server that needs GPU-accelerated encoding for streaming to remote clients — hits exactly the same setup questions, the same IOMMU group constraints, and the same class of neighbour-check that this piece is built around.</p><p>The unified lesson from this piece is small and load-bearing: <strong>the error pointed at the GPU. The culprit was the HBA. Check the neighbours before you blame the card.</strong></p><p>If you’re planning GPU passthrough on a Proxmox host, do this before you touch any physical hardware. Enumerate the IOMMU groups on the current configuration. Identify what shares a group with your target device. Check the reset_method sysfs entry for each of those neighbours. If any of them lacks FLR support, plan the physical move now - not later, when the VM refuses to start and the error message is pointing everywhere except at the actual problem.</p><p>The checklist runs clean. Until it doesn’t. This is the check that catches when it doesn’t.</p><p><em>Originally published at </em><a href="https://jjozzietech.com.au/gpu-passthrough-proxmox/"><em>https://jjozzietech.com.au</em></a><em> on August 18, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45c635a0eaff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>