<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by 陳昶仲 on Medium]]></title>
        <description><![CDATA[Stories by 陳昶仲 on Medium]]></description>
        <link>https://medium.com/@jason-chen-0604?source=rss-8e9e2f4ae9e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BQRFUkBB1grxZD3T.jpg</url>
            <title>Stories by 陳昶仲 on Medium</title>
            <link>https://medium.com/@jason-chen-0604?source=rss-8e9e2f4ae9e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:18:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jason-chen-0604/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One Gauge for Every AI Coding Tool: My Usage Monitor Now Tracks GitHub Copilot Too]]></title>
            <link>https://jason-chen-0604.medium.com/one-gauge-for-every-ai-coding-tool-my-usage-monitor-now-tracks-github-copilot-too-0fa5d55bb8c0?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/0fa5d55bb8c0</guid>
            <category><![CDATA[macos]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[github-copilot]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 06:53:54 GMT</pubDate>
            <atom:updated>2026-11-04T06:55:44.958Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>Claude Usage Monitor series ｜ Previous: </em><a href="https://jason-chen-0604.medium.com/i-ran-out-of-gas-mid-sprint-with-claude-code-so-i-built-it-a-fuel-gauge-e84f07a0503f"><em>I Ran Out of Gas Mid-Sprint With Claude Code, So I Built It a Fuel Gauge</em></a><em> ｜ Next: coming soon</em></blockquote><p>GitHub: <a href="https://github.com/jasonChen0604/claude-usage-monitor">jasonChen0604/claude-usage-monitor</a></p><p>Last time I built a fuel gauge for Claude Code so I’d stop running out of gas mid-sprint. Problem solved — except it turns out I’m not driving a regular car. I’m driving a hybrid with two separate tanks, and the second one (<strong>GitHub Copilot</strong>) was still running completely blind.</p><p>Same story as before: GitHub shows you Copilot’s AI-credit usage somewhere in Settings → Billing, but nobody checks a billing page mid-flow. So <strong>Claude Usage Monitor v0.5.0</strong> adds Copilot tracking right next to Claude’s — same popover, same menu bar, same glance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Ap2vjD6TzyPpa5-_BXv4A.png" /><figcaption><a href="https://github.com/jasonChen0604/claude-usage-monitor/raw/main/assets/screenshot-popover.png">GitHub Copilot and Claude usage cards side by side in the popover</a></figcaption></figure><h3>The fix, in one sentence: one gauge, two tanks</h3><p>Claude Usage Monitor already stored usage data in a <strong>provider-agnostic schema</strong> — that was the whole point of the design in article one. v0.5.0 is where that design finally gets tested against a second, very different provider, and a few assumptions from the original build turn out not to hold.</p><h3>Quick Start</h3><p>If you’re already running the app, updating gets you Copilot support automatically — there’s no separate install.</p><pre># Prerequisite: the official GitHub CLI, logged in with a Copilot seat<br>brew install gh<br>gh auth login<br><br># Update Claude Usage Monitor to v0.5.0+ (download from Releases)<br># https://github.com/jasonChen0604/claude-usage-monitor/releases<br><br># Open the popover, click the gear icon, and turn on<br># &quot;Track GitHub Copilot usage&quot; - no token to paste, gh handles auth</pre><p>If you’re on multiple GitHub accounts, there’s an optional <strong>gh account</strong> field in Settings — point it at the account whose Copilot seat you want tracked, independent of whatever gh auth switch happens to be active in your terminal that day.</p><h3>Three things that didn’t go the way I planned</h3><h3>Trap one: the “official” endpoint that doesn’t work for my seat</h3><p>My rule from article one was simple: only use what the vendor documents. So I started with GitHub’s actual documented billing endpoints — ai_credit/usage and premium_request/usage. First call came back 404, missing the user scope on my token. Fixed the scope, authorized again — now it returned 400 Unable to get billing usage data.</p><p>Turned out the 400 wasn’t a bug on my end. My Copilot seat is <strong>provided by my employer’s organization</strong>, not a personal subscription — and those documented per-user billing endpoints only return data for individually-billed accounts. An org-assigned seat has no personal billing record to query. The documented path simply doesn’t apply to a large chunk of actual Copilot users.</p><p>What does return my quota — plan, credits used, entitlement, reset date — is GET https://api.github.com/copilot_internal/user, the same endpoint Copilot&#39;s own editor extensions call. It&#39;s <strong>not documented</strong>, so GitHub could change or remove it without notice. I&#39;m being upfront about that trade-off in the docs rather than pretending it&#39;s the same situation as Claude&#39;s prohibited endpoint — this one isn&#39;t flagged as off-limits for third-party use, it&#39;s just unofficial. I isolated it in its own module and marked it shortcut: in the source, so it&#39;s a one-file swap if GitHub ever ships a real public API for this.</p><h3>Trap two: the GUI app that forgot where gh lives</h3><p>First local test, Copilot’s card in the popover just stayed empty — no error, nothing. Running the exact same collector logic from a terminal worked fine, which made it more confusing, not less: same code, same machine, different result depending on how it launched.</p><p>The difference was <strong>PATH</strong>. A terminal shell sources your .zshrc or .bashrc, which is how gh ends up findable in the first place (usually installed via Homebrew). A macOS app launched from Finder or Spotlight starts with none of that — it gets a minimal system PATH that doesn&#39;t include /opt/homebrew/bin. std::process::Command::new(&quot;gh&quot;) was failing to resolve the binary at all, silently. The fix was boring but necessary: check /opt/homebrew/bin/gh and /usr/local/bin/gh explicitly first, and only fall back to a bare gh lookup after that.</p><h3>Trap three: admitting the “unfixable” bug from article one wasn’t</h3><p>This one stings a little. In the first article I wrote that multiple concurrent Claude Code sessions overwriting the same snapshot file was <strong>“inherent to using a per-render statusline callback… not a bug in this app’s logic.”</strong> Mostly true — except I’d stopped one step short of actually fixing what I could fix.</p><p>The snapshot file doesn’t have to be last-write-wins. While wiring up Copilot’s snapshot alongside Claude’s, I rewrote the merge logic: for the same window period, the <strong>larger</strong> used_percentage wins, since usage within a period only grows — it never resets backward. A window with a later resets_at replaces an older period entirely. And writes go through a temp file + rename instead of a direct overwrite, so the app never reads a half-written snapshot mid-write. The underlying limitation (no live API, no single source of truth across sessions) is still real — but &quot;the numbers visibly jump around&quot; turned out to be a fixable symptom of it, not an unavoidable one.</p><h3>Under the hood</h3><pre>+------------------------+------------------------------------------------+--------------------------------------------+<br>| Item                   | What it does                                   | Effect                                     |<br>+------------------------+------------------------------------------------+--------------------------------------------+<br>| Copilot collector      | Runs inside the app itself as a Rust module,   | Copilot gets near-live polling instead of  |<br>| (copilot.rs)           | not a standalone script. Shares the same poll  | depending on an external trigger the way   |<br>|                        | loop as the Claude reader, but throttles       | Claude&#39;s statusLine does                   |<br>|                        | itself to one real fetch every 5 minutes       |                                            |<br>|                        | minimum                                        |                                            |<br>+------------------------+------------------------------------------------+--------------------------------------------+<br>| gh binary lookup       | Checks /opt/homebrew/bin/gh and                | Works even though a GUI app launched       |<br>|                        | /usr/local/bin/gh explicitly before falling    | outside a shell doesn&#39;t inherit PATH the   |<br>|                        | back to a bare gh on PATH                      | way a terminal does                        |<br>+------------------------+------------------------------------------------+--------------------------------------------+<br>| Per-account token      | If a gh account is set, runs gh auth token     | Switching the active gh account later with |<br>| scoping                | --user &lt;account&gt; once and passes it via        | gh auth switch doesn&#39;t silently change     |<br>|                        | GH_TOKEN for that single call only             | which Copilot seat gets tracked            |<br>+------------------------+------------------------------------------------+--------------------------------------------+<br>| Claude snapshot        | Incoming windows merge into the existing       | Fixes the multi-session &quot;flapping&quot; that    |<br>| merging                | snapshot instead of overwriting it: higher     | the first article called an unfixable      |<br>|                        | used_percentage wins within the same period, a | design limitation                          |<br>|                        | later resets_at replaces the period, writes go |                                            |<br>|                        | through a temp file + rename                   |                                            |<br>+------------------------+------------------------------------------------+--------------------------------------------+<br>| --collect mode         | statusLine now runs claude-usage-monitor       | Node.js is no longer a dependency;         |<br>|                        | --collect (the app&#39;s own binary) instead of a  | existing installs are prompted to          |<br>|                        | separate Node.js script                        | reconfigure once                           |<br>+------------------------+------------------------------------------------+--------------------------------------------+</pre><p>The provider-agnostic schema from article one earned its keep here: adding Copilot touched exactly one new module (copilot.rs) and a couple of UI tweaks for a second card. Nothing in the snapshot reader, the tray rendering, or the popover layout had to change to understand a brand-new provider with a completely different shape of data source.</p><h3>What to know before you rely on the Copilot number</h3><ul><li><strong>It’s reading an undocumented endpoint.</strong> /copilot_internal/user works today because it&#39;s what Copilot&#39;s own editor extensions use, but GitHub hasn&#39;t promised it&#39;ll stay stable. If Copilot tracking silently stops working after a GitHub-side change, that&#39;s why.</li><li><strong>Org-provided seats are why this exists at all.</strong> If your Copilot plan is personal and billed directly to you, the documented endpoints might actually work for you — this project just can’t assume that for everyone.</li><li><strong>You need the </strong><strong>gh CLI, installed and logged in.</strong> No token field to fill in by hand; auth is entirely delegated to gh&#39;s own stored credentials, and the app never touches or stores your token itself.</li><li><strong>If you manage multiple GitHub accounts</strong>, set the <strong>gh account</strong> field in Settings, since the app otherwise just uses whatever gh considers the default active account.</li><li><strong>The merge fix helps Claude’s flapping, not Copilot’s.</strong> Copilot only ever has one window (credits), so there&#39;s nothing to merge there — this fix was specifically for Claude&#39;s concurrent-session case.</li></ul><h3>Wrapping up</h3><p>The honest version of this update: one feature landed cleanly (provider-agnostic design, proven under real pressure), one feature landed with an asterisk (Copilot tracking works, but on a private endpoint I’d rather not depend on long-term), and one bug I’d called unfixable six weeks ago got fixed anyway. That last one is probably the most useful lesson here — “inherent limitation” is sometimes just “the fix I hadn’t thought of yet.”</p><p>Still MIT-licensed, still free, still no telemetry. If you’re juggling both Claude Code and Copilot subscriptions and tired of checking two separate billing pages, this is for you — and if you’ve hit the same org-seat wall with Copilot’s documented APIs, I’d genuinely like to know whether anyone’s found a cleaner way around it. Stars on GitHub are always appreciated.</p><blockquote><em>Claude Usage Monitor series ｜ Previous: </em><a href="https://jason-chen-0604.medium.com/i-ran-out-of-gas-mid-sprint-with-claude-code-so-i-built-it-a-fuel-gauge-e84f07a0503f"><em>I Ran Out of Gas Mid-Sprint With Claude Code, So I Built It a Fuel Gauge</em></a><em> ｜ Next: coming soon</em></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604/claude-usage-monitor">https://github.com/jasonChen0604/claude-usage-monitor</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0fa5d55bb8c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discipline, Not Luck]]></title>
            <link>https://jason-chen-0604.medium.com/discipline-not-luck-6cc52f7e6514?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cc52f7e6514</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[self-hosted]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 06:12:44 GMT</pubDate>
            <atom:updated>2026-09-16T06:13:00.498Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — Sixteen posts, eleven stations, two CVEs, one long-standing bug — all closed. Looking back, one habit shows up in almost every single entry.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cbnNXRk7Fu-mNyW9N0n5Rg.png" /><figcaption><em>Sixteen posts later, the pattern is easier to see in hindsight than it ever was in the moment</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-11-part-2-202-not-400-the-image-that-wouldnt-die-finally-does-6ad70e363cb8?sharedUserId=jason-chen-0604">Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a></blockquote><h3>TL;DR</h3><p>This series started with a CVSS 9.4 vulnerability and no test environment to rehearse any of it in. It ends eleven stations and sixteen posts later, with both CVEs patched and a long-standing registry bug finally resolved instead of just worked around. None of the individual stations felt connected to each other while they were happening. Looking back at all of them together, one habit shows up almost everywhere: <strong>don’t trust the surface signal — verify the real state, independently, every time.</strong></p><h3>What This Project Actually Was</h3><p>A GitLab instance that had carried real production traffic for three years, upgraded across eleven stations, with no staging environment anywhere in the process. That was the premise this series opened with, and it held all the way through: every risky step happened against the real system, because there was never another option.</p><h3>The Habit, in One Sentence</h3><p>Whatever a system, a log line, or a piece of documentation claims about its own state, verify it independently before acting on it — because the gap between what something looks like and what it actually is was where nearly every finding in this series lived.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eHYKHnxjqFoJqR-kAhJr0A.png" /><figcaption><em>The same shape of gap, six different disguises</em></figcaption></figure><h3>The Same Gap, Wearing Different Disguises</h3><p><strong>The Runner Audit</strong> — a field named contacted_at looked exactly like &quot;still in use.&quot; Real job history said a third of the fleet had been idle for months.</p><p><strong>The Image That Wouldn’t Die</strong> — a registry answered a capability probe with an honest-looking 200. The actual delete call it claimed to support failed every single time.</p><p><strong>Station 4 and Station 7–8</strong> — a backup file existing on disk looked like a finished job, twice, in two different tools. Both times, the process behind it was still running.</p><p><strong>Station 6</strong> — an error message blamed a missing database column. The column hadn’t vanished; it had been renamed, and the real cause was a completely unrelated migration’s side effect.</p><p><strong>Station 5–6</strong> — this series’ own notes claimed a specific fix mechanism, carried forward from earlier research and never independently checked. Verifying it before writing found the outcome was real, but that particular explanation wasn’t provable.</p><p><strong>Station 11, Part 2</strong> — closing the loop meant testing the exact same indicator three separate times, at increasing levels of realism, before it was trustworthy enough to act on.</p><p>Six different tools, six different layers of the stack, the same underlying gap every time: something claimed a state, and the real state turned out to be different enough to matter.</p><h3>What Made the Risky Stations Survivable</h3><p>None of this series’ outcomes came from things going right by default. Station 4 spent an entire day on verification and rehearsal before swapping a database engine live, turning a 30-minute official estimate into 6 minutes of real downtime. Station 11 ran a dry-run that surfaced four separate blockers before the real cutover, turning an estimated 30 to 60 minutes into 17. In both cases, the risk got spent somewhere safe to spend it, on purpose, rather than discovered live.</p><h3>Where Honesty Mattered More Than a Clean Story</h3><p>Not every finding in this series resolved neatly, and the series is more honest for including the ones that didn’t. The migration that fixed itself came with an appendix admitting the specific mechanism couldn’t be confirmed. The backup that looked broken ended with no fix at all, just a correct diagnosis and a wait. A parallelism setting that clearly helped turned out to have a real limit, found the hard way against tables that simply didn’t care how high the knob was turned. None of these needed to be dressed up as bigger wins or cleaner failures than they actually were.</p><h3>The Project, by the Numbers</h3><pre>+---------------------------+---------------------------------------------+<br>| Metric                      | Result                                        |<br>+---------------------------+---------------------------------------------+<br>| Mandatory stations            | 11, all completed                             |<br>+---------------------------+---------------------------------------------+<br>| Posts in this series          | 16                                             |<br>+---------------------------+---------------------------------------------+<br>| CVEs patched                  | 2 (CVSS 9.4 and a CSRF issue)                  |<br>+---------------------------+---------------------------------------------+<br>| Test environments available    | 0, the entire way through                      |<br>+---------------------------+---------------------------------------------+<br>| Station 4 estimate vs. real    | 30 minutes → 6 minutes                          |<br>+---------------------------+---------------------------------------------+<br>| Station 11 estimate vs. real   | 30-60 minutes → 17 minutes                      |<br>+---------------------------+---------------------------------------------+</pre><h3>What Actually Worked</h3><p>The single habit that shows up across nearly every post in this series — verifying the real state instead of trusting the surface signal — cost time in the moment and saved far more of it later. It’s not a sophisticated technique. It’s closer to a discipline: check the thing itself, not the thing that’s supposed to represent it, every time the two could plausibly disagree.</p><h3>Where This Still Falls Short</h3><p>This series covered one GitLab instance across one upgrade window. The specific bugs, the specific error messages, the specific migration names — none of that generalizes directly to any other system. What might generalize is the habit itself: the willingness to stop trusting a clean-looking signal long enough to check what’s actually underneath it. Whether that holds up on the next project is a question this series can’t answer from here.</p><p>If you’ve made it through this whole series — what’s the equivalent gap you’ve run into, between what something looked like and what it actually was?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-11-part-2-202-not-400-the-image-that-wouldnt-die-finally-does-6ad70e363cb8?sharedUserId=jason-chen-0604">Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cc52f7e6514" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does]]></title>
            <link>https://jason-chen-0604.medium.com/station-11-part-2-202-not-400-the-image-that-wouldnt-die-finally-does-6ad70e363cb8?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ad70e363cb8</guid>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 06:07:46 GMT</pubDate>
            <atom:updated>2026-09-16T06:14:38.064Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — 58,818 failed requests, back in August. One API call, one month later, that finally returns what it was always supposed to.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_3DQf6iuiffqVhtG1CYnw.png" /><figcaption><em>The same delete call this series opened a mystery with, finally closing it</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-11-part-1-three-blockers-a-dry-run-almost-missed-398332ee18c9?sharedUserId=jason-chen-0604">Station 11, Part 1: Three Blockers a Dry-Run Almost Missed</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/discipline-not-luck-6cc52f7e6514?sharedUserId=jason-chen-0604">Discipline, Not Luck</a></blockquote><h3>TL;DR</h3><p>Back in article five of this series, a container image refused to delete — 58,818 failed requests traced to a protocol-level mismatch between GitLab and the registry image running underneath it. The workaround stopped the bleeding. The real fix waited for this station. With every dry-run blocker from Part 1 already resolved, the actual cutover verified the same one indicator — can a delete-by-tag call actually succeed — at three increasing levels of realism, before confirming it against the real production endpoint. The exact API call that returned 400 &quot;digest invalid&quot; in August now returns 202 Accepted. The cutover itself took 17 minutes against an estimated 30 to 60.</p><h3>A Cliffhanger From Early in This Series</h3><p>Article five of this series ended without a resolution, on purpose: a stuck deletion, traced down to a registry image that answered a capability probe honestly by its own logic, while genuinely not supporting the operation GitLab inferred from that answer. The fix — swapping the registry image itself for GitLab’s official fork — was deliberately deferred until every other station in this project had already landed safely. This is that station.</p><h3>The Fix, in One Sentence</h3><p>Verify the exact same indicator — can this registry actually delete by tag name — at three increasing levels of realism, and don’t call it fixed until the real production endpoint confirms it directly.</p><h3>Quick Start: One Indicator, Tracked Three Times</h3><pre># The exact capability check that lied back in August<br>curl -X OPTIONS https://&lt;registry-host&gt;/v2/&lt;repo&gt;/manifests/&lt;tag&gt;<br># Looking specifically at the Allow header this time, not just the status code</pre><p>Every verification step below asks this same question, at increasing levels of realism, until there’s nothing left to doubt.</p><h3>Layer 1: The Protocol Check, Done Right This Time</h3><p>Against an isolated mini registry — copied, not hard-linked, so nothing about this test could touch real data — the same OPTIONS probe that gave a false positive against the old image gave an honest answer against the fork:</p><pre>Allow: DELETE, GET, HEAD, PUT</pre><h3>Layer 2: Real Data, Still Fully Isolated</h3><p>The isolated registry wasn’t synthetic — it held a genuine copy of one real project’s repositories and every blob they referenced, roughly 1.5GB, completely separated from production. Against that real data:</p><pre>DELETE by tag name  →  202 Accepted<br>tags/list confirms the tag now returns null<br>push (full blob-upload flow)  →  201 Created</pre><p>Both directions worked — deleting what should be deletable, and confirming new content could still be pushed afterward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MvIsgkwZrY7XVqpuHM2Gfw.png" /><figcaption><em>Same indicator, three levels of realism, then the real thing</em></figcaption></figure><h3>Layer 3: The Real Endpoint, the Real Project</h3><pre>TOKEN=$(curl ... &quot;/jwt/auth?...scope=...delete&quot;)<br>curl -X DELETE -H &quot;Authorization: Bearer $TOKEN&quot; \<br>  &quot;https://&lt;host&gt;:5000/v2/tqms/alpha/alpha-ai-agent/manifests/DEV&quot;</pre><pre>HTTP/2 202<br>tags_now=&quot;&quot;</pre><p>Two minutes of registry logs afterward: <strong>zero occurrences of “digest invalid.”</strong> Not reduced. Zero.</p><h3>The Exact Comparison This Series Has Been Waiting For</h3><pre>2026-08-21 (stop-the-bleeding diagnosis)<br>DELETE /v2/tqms/alpha/alpha-ai-agent/manifests/qas<br>→ 400 &quot;digest invalid&quot;</pre><pre>2026-09-15 (real fix, same project)<br>DELETE /v2/tqms/alpha/alpha-ai-agent/manifests/DEV<br>→ 202 Accepted</pre><p>Same project. Same shape of API call. One month apart. One returns the error this whole thread started with — the other returns exactly what it was always supposed to.</p><h3>The Cutover Itself: 17 Minutes</h3><pre>12:01 → 12:18</pre><p><strong>17 minutes</strong>, against an estimated 30 to 60. This is the same pattern this series has traced before, back at Station 4: the real risk gets spent the day before, in a dry-run, not during the maintenance window itself. Every blocker from Part 1 — the missing config, the database detection bug, the ownership mismatch, the health check quirk — was already resolved before this window opened. What was left to do during the window itself was small by design.</p><h3>Closing the Backlog: 3,098 Jobs, Cleared</h3><p>With the registry fix confirmed working, the deferred decision from “Two Thousand Stalled Jobs” finally had an answer:</p><pre>Sidekiq::DeadSet.new.clear<br># before=3098 after=0</pre><p>The dead queue got cleared outright, not retried job by job. The reasoning: these are job execution records, not business data — retrying them would have meant re-attempting deletions that had already been abandoned months ago, on repositories that may have moved on since. If a real cleanup need exists going forward, it’s simpler to handle it directly than to try to resurrect a six-month-old backlog.</p><h3>Under the Hood</h3><pre>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Layer                       | What Was Tested                                   | Result                                                |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Protocol                    | OPTIONS capability probe, isolated registry        | Allow: DELETE, GET, HEAD, PUT — honest this time      |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Isolated real data            | Delete by tag + full push flow, real repo copy     | 202 on delete, 201 on push — both directions work     |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Production                   | The exact delete call from the August diagnosis    | 202, zero digest invalid errors in the two            |<br>|                             | re-run against the real endpoint                    | minutes after                                          |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Backlog closure               | Cleared 3,098 dead jobs once the fix was confirmed  | Chose clearing over retrying — the underlying          |<br>|                             |                                                    | deletions were already long abandoned                  |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Testing the exact same indicator at three increasing levels of realism — protocol, isolated real data, then production — is what turned “the dry-run problems are fixed” into actual confidence that the core bug itself was gone, not just the blockers around it. Comparing the same API call, on the same project, directly against the original August failure is the clearest possible proof this series could offer that the fix is real rather than assumed.</p><h3>Where This Still Falls Short</h3><p>This closes the loop this series opened in article five, and it closes Station 11 — but it doesn’t close the series. What’s left is a look back across all eleven stations, and whatever pattern holds across nearly all of them once the adrenaline of any single station wears off.</p><p>If you’ve ever carried an open question across months of unrelated work, what did it feel like the day you finally got to close it?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-11-part-1-three-blockers-a-dry-run-almost-missed-398332ee18c9?sharedUserId=jason-chen-0604">Station 11, Part 1: Three Blockers a Dry-Run Almost Missed</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/discipline-not-luck-6cc52f7e6514?sharedUserId=jason-chen-0604">Discipline, Not Luck</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ad70e363cb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Station 11, Part 1: Three Blockers a Dry-Run Almost Missed]]></title>
            <link>https://jason-chen-0604.medium.com/station-11-part-1-three-blockers-a-dry-run-almost-missed-398332ee18c9?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/398332ee18c9</guid>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[gitlab]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 05:22:53 GMT</pubDate>
            <atom:updated>2026-09-16T06:15:42.948Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — The plan said “just swap the image, keep the environment variables.” A dry-run found three separate reasons that assumption was wrong, stacked one on top of the next.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDBgjGs_YYu6dj2QgD-O0Q.png" /><figcaption><em>Three errors, one dry-run, each one hiding behind the last</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/two-thousand-stalled-jobs-c0966dc466cc?sharedUserId=jason-chen-0604">Two Thousand Stalled Jobs</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-11-part-2-202-not-400-the-image-that-wouldnt-die-finally-does-6ad70e363cb8?sharedUserId=jason-chen-0604">Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does</a></blockquote><h3>TL;DR</h3><p>Six days after reaching 19.2.4 — enough to satisfy the “roughly a week of observation” threshold set out in the original plan — Station 11 preparation began: replacing the Container Registry image itself with GitLab’s official fork, to finally resolve the protocol mismatch this series met early on. The plan assumed this was a simple image swap, keeping every existing environment variable in place. A dry-run found three separate, stacked reasons that assumption was wrong — a missing config file, a known GitLab bug around database detection, and a permission model mismatch. A follow-up, more rigorous audit then found that a supposedly-complete permission fix had actually only touched one out of 265,487 directories.</p><h3>An Assumption Worth Testing Before Trusting</h3><p>The original plan for this station was simple on paper: swap the registry’s container image, leave the rest of the configuration untouched. That kind of assumption — “this should just be a drop-in replacement” — is exactly the kind worth testing in a dry-run before it gets tested in production instead.</p><h3>The Fix, in One Sentence</h3><p>Run the swap against an isolated, read-only test first, let every assumption fail where it’s safe to fail, and only then bring the same fix to the real thing.</p><h3>Blocker 1: The Image That Doesn’t Read the Old Configuration</h3><pre>configuration error: open /etc/docker/registry/config.yml: no such file or directory</pre><p>The fork image doesn’t configure itself from REGISTRY_* environment variables the way the upstream image did — it uses an Omnibus-style templating mechanism (/scripts/set-config) that expects an actual config file. The &quot;just swap the image, environment variables unchanged&quot; assumption failed on the very first attempt to start the container.</p><h3>Blocker 2: A Known Bug Around “Fresh Install” Detection</h3><p>With a minimal config.yml in place, a second error appeared:</p><pre>&quot;msg&quot;:&quot;database in prefer mode on a fresh install: configured and reachable metadata database required to start&quot;<br>{&quot;err&quot;:&quot;failed to connect to `user=git database=`: /tmp/.s.PGSQL.5432 (/tmp): dial error: dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory&quot;}</pre><p>This traced to a known GitLab issue, <strong>#600955</strong>: starting with GitLab 19.0, the registry defaults to “prefer mode” for its metadata database, but an upgrade or image swap doesn’t automatically import existing filesystem metadata into that database — so the registry misreads an environment with real, existing data as a brand-new install. The officially documented fix landed in version v4.40.1-gitlab; the image in use here, v4.41.0-gitlab, was built after that fix shipped. Worth being explicit about the limit of that reasoning, though: <strong>a version number being newer than a fix&#39;s release is an inference, not a guarantee</strong> — it&#39;s not something to treat as confirmed without testing it directly, which is exactly what this dry-run was for. Setting database.enabled: false sidestepped the issue for the registry&#39;s actual configuration going forward.</p><h3>Blocker 3: A Container That Runs as a Different User Entirely</h3><pre>{&quot;error&quot;:&quot;filesystem: mkdir /registry/docker/registry/lockfiles: read-only file system&quot;}</pre><p>Switching to a writable mount surfaced the real issue underneath:</p><pre>{&quot;error&quot;:&quot;filesystem: mkdir /registry/docker/registry/lockfiles: permission denied&quot;}</pre><p>The root cause: registry:2.8 and the existing compose setup never specified a user:, so it had always run as root by default. The fork image, by contrast, has its runtime identity <strong>fixed to </strong><strong>git:git — uid=1000, gid=1000 — not root.</strong> The existing data directory was owned by root:root. Nothing about the container itself was broken; it simply wasn&#39;t allowed to touch data it didn&#39;t own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YS250sjflgwkHSVZBT2yrg.png" /><figcaption><em>Three errors deep before the real blocker (ownership) showed itself</em></figcaption></figure><h3>Correcting an Assumption With a Full Audit, Not a Sample</h3><p>Before running a recursive ownership fix across a 1.6TB data tree, the sensible move was checking the current state first — and a quick sample suggested most of it was already in reasonable shape. A full, unsampled check told a different story:</p><pre>find /data/gitlab/shared/registry -type d -not -uid 1000 | wc -l<br># 265486</pre><p><strong>265,487 total directories. Exactly one of them had the correct owner.</strong> The sample had been technically accurate about what it looked at — it just hadn’t looked at enough of the tree to represent the whole thing. That gap between “the part I checked looks fine” and “the whole thing is fine” is the entire lesson here.</p><h3>Testing Permission Requirements Function by Function</h3><p>Rather than guessing how much of the tree actually needed the ownership fix, an isolated, fully-copied mini registry (not a hard-linked shortcut — a genuine copy) made it possible to test each function’s real requirement directly:</p><pre>Chown Scope            Start   Read    Delete           Push<br>docker/registry only     ✅      ✅      ❌ 500 EACCES     ❌<br>+ repositories/          ✅      ✅      ✅ 202            ❌ 500<br>Whole tree                ✅      ✅      ✅                ✅ 201</pre><p>Starting and reading tolerated a shallow fix. Deleting needed more. Pushing needed the entire tree. The conclusion: <strong>the full recursive </strong><strong>chown was necessary, not optional</strong> — anything less would have left delete or push silently broken while everything else looked fine. And the fix itself turned out to be fast once run correctly: a full traversal of 208,469 directories took <strong>5.1 seconds</strong>, with the actual chown estimated at 10 to 30 seconds — nowhere close to a real constraint on the maintenance window.</p><h3>One More Thing the Dry-Run Almost Didn’t Catch: Health Checks</h3><p>The dry-run’s test environment ran over a plain, unencrypted port with no TLS configured, so it never triggered a health-check-related error. Re-auditing the full compose configuration afterward surfaced a difference the isolated test hadn’t been positioned to catch:</p><pre>HEALTHCHECK={&quot;Test&quot;:[&quot;CMD-SHELL&quot;,&quot;/scripts/healthcheck&quot;],&quot;Interval&quot;:30000000000,...}<br>status=$(curl --retry 6 -s -q -o /dev/null -w &#39;%{http_code}&#39; http://127.0.0.1:${PORT:-5000}/v2/)<br>[[ &#39;200 401&#39; =~ &quot;$status&quot; ]]</pre><p>The fork image ships a <strong>built-in </strong><strong>HEALTHCHECK</strong> — something registry:2.8 never had at all — and its script probes over plain HTTP. The real configuration serves HTTPS on that same port. Tested directly, that combination returns 400 Bad Request (Client sent an HTTP request to an HTTPS server) — meaning the container would report <strong>permanently unhealthy, regardless of whether the service is actually working.</strong> The plan for cutover day: disable the health check (healthcheck: disable: true) during the swap itself, confirm nothing else in the compose setup depends on that health status (restart: always doesn&#39;t act on it, and no other service declares a health-based dependency on the registry), and switch to a correct HTTPS-based probe once the swap settles. Worth noting how this one surfaced: it wasn&#39;t found by something breaking — it came from a simple planning question about whether the port could stay unchanged, which pulled a thread into several previously unconsidered details at once.</p><h3>Under the Hood</h3><pre>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Finding                     | Root Cause                                      | Resolution                                            |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Missing config.yml           | Fork image ignores REGISTRY_* env vars,          | Supplied a minimal config.yml matching the fork&#39;s    |<br>|                             | expects a real config file                       | expected format                                        |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Database prefer-mode error   | Known GitLab issue #600955 — misreads an          | Set database.enabled: false, avoiding the fresh-      |<br>|                             | existing environment as a fresh install           | install misdetection entirely                          |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Permission denied             | Fork image runs as git:git (1000:1000),           | Full recursive chown, confirmed necessary via         |<br>|                             | not root; data was owned by root:root              | function-by-function testing                            |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Health check mismatch         | Fork image&#39;s built-in HTTP health probe            | Disabled during cutover, switched to a correct         |<br>|                             | against an HTTPS-only port                        | HTTPS probe once the swap settled                        |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Every one of these four findings surfaced during a dry-run, not during the real cutover — which is the entire point of running one. The permission audit specifically demonstrates something worth generalizing past this one station: a sample that looks clean isn’t the same claim as a full check that’s actually clean, and the gap between the two can be enormous — one correct directory out of 265,487 would have been easy to mistake for “already handled” without looking at all of them.</p><h3>Where This Still Falls Short</h3><p>This dry-run used read-only mounts as a deliberate, conservative choice — which meant it couldn’t test anything requiring real write access, including the exact permission depth that later needed its own separate, more thorough check. None of these findings involved any drama or urgency; they were exactly what a dry-run is supposed to produce; the real payoff of this station is what happens next, once every one of these was already resolved before the actual cutover began.</p><p>Have you ever had a “simple swap” turn out to have three separate, stacked assumptions hiding behind it — each one only visible once the last one got out of the way?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/two-thousand-stalled-jobs-c0966dc466cc?sharedUserId=jason-chen-0604">Two Thousand Stalled Jobs</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-11-part-2-202-not-400-the-image-that-wouldnt-die-finally-does-6ad70e363cb8?sharedUserId=jason-chen-0604">Station 11, Part 2: 202, Not 400 — The Image That Wouldn’t Die, Finally Does</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=398332ee18c9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Two Thousand Stalled Jobs]]></title>
            <link>https://jason-chen-0604.medium.com/two-thousand-stalled-jobs-c0966dc466cc?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0966dc466cc</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 05:18:44 GMT</pubDate>
            <atom:updated>2026-09-15T05:26:13.627Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — Ten stations into this upgrade, an unrelated number showed up in the admin panel: over 2,800 dead jobs, the oldest one six months old.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1aDeFLEjH-84cNu9Taflw.png" /><figcaption><em>A number that had nothing to do with the upgrade, and everything to do with a bug this series already met once</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/reaching-19-2-4-the-milestone-3d42472d4142?sharedUserId=jason-chen-0604">Reaching 19.2.4: The Milestone</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-11-part-1-three-blockers-a-dry-run-almost-missed-398332ee18c9?sharedUserId=jason-chen-0604">Station 11, Part 1: Three Blockers a Dry-Run Almost Missed</a></blockquote><h3>TL;DR</h3><p>Ten stations into this upgrade, a routine glance at the admin panel’s Background Jobs page turned up something unrelated to any of it: a dead-job queue sitting at 2,887 entries, spanning six months of accumulation. Classifying them by job type found that 89.9% traced directly back to a bug this series already met once — the Container Registry protocol mismatch from early in this project. A workaround stopping the bleeding six months ago never made the underlying failures disappear; it just made them pile up somewhere quieter. The decision here isn’t a fix — it’s choosing to wait until the actual root cause is resolved before deciding what to do with the backlog it left behind.</p><h3>A Number That Didn’t Belong to This Upgrade</h3><p>Ten stations of version bumps, migrations, and database work — and the thing that stood out wasn’t any of that. A routine check of the admin panel’s Background Jobs page showed a dead-job queue that had grown well past two thousand entries, with no obvious connection to anything this series had been actively working on.</p><h3>The Fix, in One Sentence</h3><p>Classify before concluding — a large, vague number becomes a specific, already-understood problem the moment it’s broken down by what’s actually failing.</p><h3>Quick Start: Sizing Up a Dead Job Queue</h3><pre>stats = Sidekiq::Stats.new<br>puts &quot;processed=#{stats.processed} failed=#{stats.failed} enqueued=#{stats.enqueued} &quot; \<br>     &quot;scheduled_size=#{stats.scheduled_size} retry_size=#{stats.retry_size} dead_size=#{stats.dead_size}&quot;</pre><pre>processed=874164054 failed=26787 enqueued=0 scheduled_size=19 retry_size=0 dead_size=2887</pre><pre>klass_count = Hash.new(0)<br>Sidekiq::DeadSet.new.each { |job| klass_count[job.klass] += 1 }</pre><p>The first command gives the scale. The second one gives the story.</p><h3>2,887 Jobs, Six Months Deep</h3><pre>oldest: 2026-03-12 06:00:05 +0000<br>newest: 2026-09-08 05:17:08 +0000</pre><p>Not a recent spike — a backlog that had been quietly accumulating for roughly six months, untouched, without anyone flagging it as a problem worth investigating until it happened to be glanced at during an unrelated upgrade project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b72Hls0pvQAUoL1K2kgbFQ.png" /><figcaption><em>A vague number becomes a specific, already-understood cause the moment it’s classified</em></figcaption></figure><h3>Classifying the Backlog</h3><pre>Ci::DeleteObjectsWorker                                        2,279<br>ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper              253<br>ContainerExpirationPolicies::CleanupContainerRepositoryWorker     222<br>ContainerRegistry::DeleteContainerRepositoryWorker                 96<br>Database::BatchedBackgroundMigration::MainExecutionWorker          15<br>(remaining classes, 1–12 jobs each)</pre><p><strong>2,279 + 222 + 96 = 2,597 jobs — 89.9% of the entire dead queue</strong> — all belonging to job classes directly tied to deleting or cleaning up container registry images.</p><h3>Recognizing an Old Friend</h3><p>This is the same failure this series already documented, back in <strong>“The Image That Wouldn’t Die.”</strong> GitLab attempting to delete registry images or run cleanup policies, failing repeatedly against the same protocol-level mismatch, retrying, and eventually landing in the dead queue once retries were exhausted. This isn’t a new problem, and it isn’t something the upgrade caused. It’s the visible, accumulating residue of a bug that was stopped from causing active harm six months ago, but never actually resolved — <strong>the fix for that is exactly what Station 11 exists to deliver.</strong></p><h3>The Decision: Not Yet</h3><p>No action gets taken on this backlog right now. The plan explicitly defers the choice — batch-retry the whole set, or simply clear the dead queue outright — until after Station 11 replaces the registry image and the real fix is confirmed working. Retrying now, against a registry still carrying the same mismatch, would just recreate the same 2,597 failures a second time. That decision is written directly into the closing checklist for Station 11, specifically so it doesn’t get quietly forgotten once the registry swap itself is done.</p><h3>Under the Hood</h3><pre>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Item                        | What I Found                                     | Why It Mattered                                       |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Scale                       | 2,887 dead jobs, spanning roughly six months      | Turned a vague admin-panel number into a              |<br>|                             |                                                    | measurable, dated backlog                              |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Classification               | 89.9% traced to registry deletion/cleanup          | Converted an unexplained pile of jobs into a           |<br>|                             | job classes                                       | specific, already-understood root cause                 |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Connection                   | Matched directly to the protocol mismatch from    | Confirmed this wasn&#39;t new — it was the same bug&#39;s      |<br>|                             | &quot;The Image That Wouldn&#39;t Die&quot;                     | ongoing, uncounted cost                                  |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Decision                     | Deferred retry/clear decision to after Station    | Avoided recreating the same failures against a         |<br>|                             | 11&#39;s registry fix, logged in the closing checklist | registry that still had the underlying bug              |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Classifying by job class before drawing any conclusion is what turned “there are 2,887 mystery jobs in here” into “89.9% of these are one already-known problem” — the difference between those two framings is the difference between a vague worry and an actionable, already-understood fact. Logging the deferred decision directly into Station 11’s own closing checklist, rather than trusting memory to carry it forward, is a small habit that matters exactly because it’s easy to forget once the more dramatic part of a station is finished.</p><h3>Where This Still Falls Short</h3><p>Nothing gets fixed in this piece, and it’s worth being direct about that: this is a diagnosis and a deliberate decision to wait, not a resolution. The remaining 10.1% — including 253 jobs wrapped generically as ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper, without further digging into what specific work those actually represent — never got investigated further here. A workaround that stops active harm isn&#39;t the same as a fix, and this backlog is the receipt for that difference: <strong>six months of quiet accumulation, for a bug that was &quot;handled&quot; back in August.</strong></p><p>Have you ever gone looking for one thing and found the long-delayed bill for a workaround you thought had already closed the book on something?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/reaching-19-2-4-the-milestone-3d42472d4142?sharedUserId=jason-chen-0604">Reaching 19.2.4: The Milestone</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-11-part-1-three-blockers-a-dry-run-almost-missed-398332ee18c9?sharedUserId=jason-chen-0604">Station 11, Part 1: Three Blockers a Dry-Run Almost Missed</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0966dc466cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reaching 19.2.4: The Milestone]]></title>
            <link>https://jason-chen-0604.medium.com/reaching-19-2-4-the-milestone-3d42472d4142?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d42472d4142</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[self-hosted]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 02:16:42 GMT</pubDate>
            <atom:updated>2026-09-15T05:25:48.068Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — Ten mandatory stations after a CVSS 9.4 vulnerability announcement, the version number that started this whole series is finally the version actually running.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*071PXLRMhbEuFFpBpnd-vw.png" /><figcaption><em>The version this series opened with as a target is now the version actually running</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-7-8-the-backup-that-looked-broken-2772b46c0902?sharedUserId=jason-chen-0604">Station 7–8: The Backup That Looked Broken</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/two-thousand-stalled-jobs-c0966dc466cc?sharedUserId=jason-chen-0604">Two Thousand Stalled Jobs</a></blockquote><h3>TL;DR</h3><p>This series opened with a CVSS 9.4 vulnerability forcing an upgrade across a mandatory version path, with no test environment to rehearse any of it. Ten stations later, the instance is running 19.2.4 — the exact version that patches both CVEs this whole project was triggered by. This station itself was uneventful, the second clean landing after Station 9. A fourth data point confirmed the backup-duration trend already tracked across the series, and a small cross-verification habit caught an SSH display quirk before it could cause an unnecessary delay. The CVE line is closed. One station remains, and it’s a different kind of problem entirely.</p><h3>Back to Where This Started</h3><p>This series opened with a number: <strong>CVSS 9.4</strong>, an unauthenticated GraphQL code-injection vulnerability, alongside a lower-severity CSRF issue riding along with it. Ten mandatory stations stood between that announcement and a fix, with no staging environment anywhere in the process. Everything since has been about getting from there to here.</p><h3>The Milestone, in One Sentence</h3><p>Confirm the version number directly rather than assuming success from a smooth-looking process, and treat this as the end of one storyline — the CVE fix — not the end of the whole project.</p><h3>Quick Start: Confirming the Version, Not Assuming It</h3><pre>curl -sk &quot;https://&lt;host&gt;/api/v4/version&quot; --header &quot;PRIVATE-TOKEN: ${GITLAB_ADMIN_TOKEN}&quot;</pre><pre>{&quot;version&quot;:&quot;19.2.4&quot;,&quot;revision&quot;:&quot;06e8d813296&quot;,&quot;kas&quot;:{...},&quot;enterprise&quot;:false}</pre><p><strong>19.2.4</strong> — the exact version GitLab’s own advisory named as the fix for CVE-2026–19478 (CVSS 9.4, GraphQL code injection) and CVE-2026–19650 (CSRF). The core objective this entire project was triggered by is now confirmed, directly, rather than assumed from an upgrade script exiting cleanly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q4rBs_pfxgvpwSyhBsrgAg.png" /><figcaption><em>Ten stations, start to finish — and a trend line that’s now four points long</em></figcaption></figure><h3>A Small Habit That Paid Off: Verify Independently, Don’t Wait Blindly</h3><p>During this station’s execution, the Rails console command to raise migration parallelism appeared to time out over SSH, with the connection display suggesting it had been backgrounded unexpectedly. Rather than waiting on that same connection to resolve, a direct SQL check confirmed the real state:</p><pre>SELECT value FROM application_settings<br>WHERE name = &#39;database_max_running_batched_background_migrations&#39;;<br>-- 6</pre><p>The setting had, in fact, updated correctly — the apparent timeout was purely a display quirk in how that SSH session reported its own status, not a real failure. The lesson generalizes past this one command: <strong>when an operation’s own output looks stuck or ambiguous, an independent query through a different channel is usually faster and more reliable than waiting on the original command to resolve itself.</strong></p><h3>The Backup Trend: A Fourth Point Makes It a Real Pattern</h3><pre>Date       Before          Duration     Final Size<br>09-01      Station 6        ~2h          39G<br>09-03      Station 7-8      ~3h05m       39G<br>09-09      Station 10       ~2h06m       38G</pre><p>The judgment principle from two stations back — trust the process, not just the file — verified correctly a fourth time: at 11:05, the backup file already read 37GB, but pgrep -af &quot;rake gitlab:backup:create|tar --exclude&quot; still returned a live process, which didn&#39;t actually finish until 11:06. Checking the file alone would have suggested it was already done a full minute early. Four data points across four different stations turns &quot;data volume seems to be growing&quot; from a single station&#39;s observation into a confirmed, ongoing trend.</p><h3>The Path, Looking Back</h3><p>Ten mandatory stations, start to finish: <strong>17.4.2 → 17.5.5 → 17.8.7 → 17.11.7 → (PostgreSQL 15→16) → 18.2.8 → 18.5.7 → 18.8.11 → 18.11.11 → (PostgreSQL 16→17, Redis 6.2→7.2) → 19.2.4.</strong> Worth being honest about one detail: the original plan named eight version stops. Two of those stops turned out to each fold in a second, distinct kind of station — a database engine swap layered underneath the version bump — bringing the real total to ten. The CVSS 9.4 vulnerability that started all of this is now patched.</p><h3>Under the Hood</h3><pre>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Item                        | What I Confirmed                                 | Why It Mattered                                       |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Version and CVE fix          | Queried the API directly for the running          | Confirmed the core objective, not just that the       |<br>|                             | version, not just trusted a clean upgrade exit     | upgrade script exited without error                    |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Ambiguous command state      | Cross-verified via direct SQL instead of          | Avoided an unnecessary delay chasing a display          |<br>|                             | waiting on a stuck-looking SSH session             | quirk that wasn&#39;t a real failure                         |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+<br>| Backup duration trend         | Logged a fourth data point across a fourth        | Turned a single station&#39;s observation into a           |<br>|                             | station                                            | confirmed, ongoing pattern                                |<br>+---------------------------+------------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Confirming the version directly, rather than trusting that a clean-looking upgrade meant the goal was met, is a small habit that matters most exactly on the station where it would be easiest to skip. The independent cross-check during the SSH timeout is the same instinct in a different shape: when something looks stuck, verify through a different channel before assuming failure or waiting indefinitely.</p><h3>Where This Still Falls Short</h3><p>This closes the CVE-driven storyline this series opened with — it does not close the project. <strong>One station remains</strong>, and it’s a different kind of problem than any of the ten before it: the Container Registry itself, carrying a protocol-level mismatch this series met early on and never fully resolved. That fix is deliberately scheduled for after a period of observing 19.2.4 running stable, not rushed in immediately just because the CVE line is finished.</p><p>If you’ve ever hit a milestone that felt like the end of a project, only to realize it was actually the end of just one storyline inside it — what did the second half look like?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-7-8-the-backup-that-looked-broken-2772b46c0902?sharedUserId=jason-chen-0604">Station 7–8: The Backup That Looked Broken</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/two-thousand-stalled-jobs-c0966dc466cc?sharedUserId=jason-chen-0604">Two Thousand Stalled Jobs</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d42472d4142" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Station 7–8: The Backup That Looked Broken]]></title>
            <link>https://jason-chen-0604.medium.com/station-7-8-the-backup-that-looked-broken-2772b46c0902?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2772b46c0902</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[self-hosted]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 02:09:16 GMT</pubDate>
            <atom:updated>2026-09-14T02:20:53.221Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — A backup file appeared on disk at a third of its usual size. The obvious read was failure. The actual answer was: it just wasn’t done writing yet.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C_8TWVcWvQlgO2NRCqaQNw.png" /><figcaption><em>A file that looked finished, and wasn’t</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/tuning-the-parallelism-knob-3bcab19331ec?sharedUserId=jason-chen-0604">Tuning the Parallelism Knob</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/reaching-19-2-4-the-milestone-3d42472d4142?sharedUserId=jason-chen-0604">Reaching 19.2.4: The Milestone</a></blockquote><h3>TL;DR</h3><p>Between Stations 7 and 8, a daily automatic backup produced a .tar file sitting at 12GB — a third the size of the 38-39GB it had consistently been for the three days before. The obvious read was a failed or stuck backup. The actual process was still running, mid-write, confirmed by checking for the process directly instead of trusting the file on disk. Ruling out two real suspects — competing migrations, a full disk — left one honest answer: every stage had simply gotten 50 to 80% slower, from data volume growing day over day. Nothing was broken. It just needed more time.</p><h3>A File a Third the Size It Should Have Been</h3><p>The daily automatic backup ahead of Station 8 had taken roughly two hours for each of the three days prior — consistent, predictable. That day, checking in on it found something that looked wrong immediately: a final .tar file already present — 1788397250_2026_09_03_18.8.11_gitlab_backup.tar — but sitting at <strong>12GB</strong>, against a consistent 38-39GB every day before. The intermediate db/ and repositories/ directories hadn&#39;t been cleaned up. artifacts.tar.gz was nowhere to be found. The first read was that the backup had failed or gotten stuck.</p><h3>The Fix, in One Sentence</h3><p>Don’t trust a file’s presence on disk as proof of completion — check whether the process that’s supposed to produce it is still actually running.</p><h3>Quick Start: Checking the Process, Not Just the File</h3><pre>ssh Gitlab &quot;bash -ic &#39;echo \$SUDO_PASS | sudo -S pgrep -af \&quot;tar --exclude\&quot; | grep -v pgrep&#39;&quot;<br># empty result = it actually finished<br># any result  = it&#39;s still running</pre><p>Running this against the “failed” backup returned a result. The tar process was still alive.</p><h3>What Was Actually Happening</h3><pre>ps aux | grep tar</pre><p>The packaging process — tar --exclude=lost+found --directory=/home/git/data/backups --create ... — was still running, in state <strong>D</strong>: uninterruptible disk I/O wait, not dead. The parent rake gitlab:backup:create process was alive too. That 12GB file wasn&#39;t a failed or finished artifact — it was a snapshot of a tar archive <strong>caught mid-write</strong>, at whatever point it happened to be when it was checked.</p><h3>Ruling Out the Obvious Suspects</h3><p>Two real hypotheses got checked before accepting “it’s just slow”:</p><p><strong>Was a leftover migration competing for I/O?</strong> Station 7 had left 27 background migrations behind, two of them touching hundreds of thousands of rows — a reasonable suspect for I/O contention.</p><pre>SELECT count(*) FROM batched_background_migrations WHERE status = 1;<br>-- 0</pre><p>Zero active migrations. Ruled out.</p><p><strong>Was the disk simply full?</strong></p><pre>df -h /data<br># 3.0T total, 1.8T used, 1.1T available</pre><p>1.1TB free. Ruled out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6T7g0AkCS2ebGMpN53vUA.png" /><figcaption><em>Ruled out migration contention. Ruled out a full disk. What was left was simply more data than the day before.</em></figcaption></figure><h3>The Real Answer: Every Stage Slowed Down, Systemically</h3><pre>Stage                09-01    09-02    09-03    change<br>Dumping database       104s     105s     176s     +68%<br>Dumping repositories   4063s    2983s    5371s     +80%<br>Dumping artifacts      3413s    2606s    3986s     +53%</pre><p>Every single stage — database, repositories, artifacts — slowed by 50 to 80% compared to the day before, not just one bottlenecked component. A single failing piece would show up as one outlier stage, not a uniform slowdown across all of them. The most honest explanation: repositories/ and artifacts/ grow with every day&#39;s commits and CI activity, and that day&#39;s dataset was simply larger than the prior day&#39;s, compounded by a short-lived I/O spike. Nothing here needed fixing. It needed time.</p><h3>The Judgment Call, Learned a Second Time</h3><p>This is the same lesson Station 4’s pg_dump taught earlier in this series, generalized past a single tool: <strong>a file existing on disk doesn&#39;t mean the job that produces it is done.</strong> The reliable check needs three things — the process has actually exited (an empty pgrep result), the intermediate working directories have been cleaned up, and the final file&#39;s size has stabilized in the expected range rather than still climbing. Any one of those missing means the job isn&#39;t finished yet, file or no file.</p><h3>Under the Hood</h3><pre>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Item                        | What I Did                                       | Why It Mattered                                       |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| First read                  | Saw a 12GB file where 38-39GB was expected        | Looked exactly like a failed or stuck backup          |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Real check                  | pgrep for the tar process instead of trusting     | Confirmed the process was alive and still writing —   |<br>|                             | the file on disk                                  | the file was a mid-write snapshot, not a result        |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Ruled out                   | Checked for competing migrations and disk space    | Eliminated two real, checkable suspects before        |<br>|                             |                                                    | accepting &quot;it&#39;s just slower today&quot;                     |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Real answer                 | Compared all three stages across three days        | Uniform 50-80% slowdown across every stage pointed    |<br>|                             |                                                    | to data growth, not a single broken component          |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Checking for the actual process instead of trusting the file on disk is what prevented a false alarm from turning into an unnecessary intervention. Ruling out the real suspects — migration contention, disk space — one at a time, rather than jumping straight to “data must just be growing,” is what made that final conclusion trustworthy instead of a convenient guess.</p><h3>Where This Still Falls Short</h3><p>There’s no fix to report here, and it’s worth being direct about that rather than manufacturing one: the diagnosis was “nothing is actually wrong,” and the resolution was waiting. That’s a less satisfying ending than most of this series, but it’s the honest one — and it’s the second time this exact lesson has shown up in this project, which is itself worth noting. A rule learned once might be a fluke. A rule learned twice, in two different tools, is a real principle.</p><p>Have you ever chased what looked like a serious failure, only to find the honest answer was “nothing’s broken, it just needs more time”?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/tuning-the-parallelism-knob-3bcab19331ec?sharedUserId=jason-chen-0604">Tuning the Parallelism Knob</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/reaching-19-2-4-the-milestone-3d42472d4142?sharedUserId=jason-chen-0604">Reaching 19.2.4: The Milestone</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2772b46c0902" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning the Parallelism Knob]]></title>
            <link>https://jason-chen-0604.medium.com/tuning-the-parallelism-knob-3bcab19331ec?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bcab19331ec</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 08:28:03 GMT</pubDate>
            <atom:updated>2026-09-14T02:21:54.795Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — Stations 5 through 8 — A database setting that isn’t even in the official API sped up mid-size migrations noticeably. A 320,000-row table didn’t care.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GPFV1TzehmRNzvmYtm_CZg.png" /><figcaption><em>A rule discovered at Station 5, verified at Station 6, and shown its actual limit at Station 7</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-6-the-bug-gitlab-hasnt-fixed-yet-68f7eebe75b8?sharedUserId=jason-chen-0604">Station 6: The Bug GitLab Hasn’t Fixed Yet</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-7-8-the-backup-that-looked-broken-2772b46c0902?sharedUserId=jason-chen-0604">Station 7–8: The Backup That Looked Broken</a></blockquote><h3>TL;DR</h3><p>Each station’s maintenance window happened to land on the quietest hour of the day, when database load sat near zero. Raising a normally-conservative parallelism setting during that window sped up mid-size background migrations noticeably — enough to formalize into a standard, repeatable step. Two stations later, a table with 322,136 rows made it clear this was never a guarantee: 40 minutes at full parallelism cleared 0.9% of it. The rule didn’t fail. It just turned out to have a scale it was never going to help with, and the right response was accepting that rather than pushing harder.</p><h3>An Empty Hour, and a Question Worth Asking</h3><p>Every station’s maintenance window happened to fall during the same quiet stretch — the point in the day when database load drops close to zero. An idle resource during a window that’s already reserved for risk is worth asking a simple question about: is there something useful to do with it?</p><h3>The Fix, in One Sentence</h3><p>Use the quiet window to safely push a normally-conservative setting higher, formalize it into a repeatable step — and stay honest about the fact that it accelerates work, it doesn’t guarantee completion.</p><h3>Quick Start: Verifying a Setting Is Actually Tunable Before Trusting It</h3><pre># Don&#39;t assume a setting is reachable via the official API — check first<br>curl -X PUT &quot;https://&lt;host&gt;/api/v4/application/settings&quot; \<br>  --header &quot;PRIVATE-TOKEN: &lt;token&gt;&quot; \<br>  --data &quot;database_max_running_batched_background_migrations=6&quot;<br># response: field silently absent from the accepted whitelist</pre><pre># The setting exists — it&#39;s just Rails-console-only<br>ApplicationSetting.current.update!(database_max_running_batched_background_migrations: 6)</pre><p>The first command doesn’t error loudly — it just doesn’t do anything, which is its own kind of trap if you don’t check the response.</p><h3>Station 5: The Rule Gets Discovered</h3><p>With the database sitting at zero load during the maintenance window, raising database_max_running_batched_background_migrations from its default of 2 to 4 visibly sped up how fast background migrations cleared. That result got formalized into a standing rule for every subsequent station: <strong>raise it to 6 at the start of the noon window, and revert it to 2 at exactly 13:00 — regardless of whether migrations have finished</strong> — to avoid holding elevated parallelism longer than the quiet window actually justifies.</p><h3>A Setting the Official API Won’t Touch</h3><p>Testing whether this could be automated through GitLab’s own Application Settings REST API turned up a small surprise: PUT /api/v4/application/settings accepted the request without error, but the field simply wasn&#39;t in the accepted whitelist — no visible failure, just no effect. The only reliable way to change it turned out to be the Rails console directly, which is why the toggle above is two manual commands rather than one scripted API call.</p><h3>Station 6: The Rule Works, Until It Meets a Table That Doesn’t Care</h3><pre>12:00 — parallelism 2 → 6<br>21 migrations clearing steadily...</pre><p>The rule held up for exactly what it was meant for — mid-size migrations cleared noticeably faster with parallelism raised. Then the queue hit <strong>BackfillPushEventPayloadsProjectId, 77,471 rows</strong>, and by 12:40 it still hadn&#39;t cleared. Raising parallelism speeds up small-to-medium migrations. A genuinely large table — tens of thousands of rows and up — still needs hours-scale time regardless of how many migrations are allowed to run concurrently. That&#39;s the first real edge of this tool, and it showed up on the very first station where the rule got tested for real.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ts5E1m7sbYFek-8ykQ7LLA.png" /><figcaption><em>The knob accelerates. It was never going to guarantee.</em></figcaption></figure><h3>Station 7: A Bigger Table, an Even Clearer Answer</h3><p>If Station 6 left any doubt about whether this was a one-off, Station 7 closed it. This station’s background migration queue was noticeably larger in scale than anything before it — 27 migrations, two of them well past a quarter million rows: <strong>MoveCiBuildsMetadataSelfManaged at 322,136 rows, </strong><strong>BackfillMergeRequestFileDiffsPartitionedTable at 261,577.</strong> Following the same SOP — parallelism raised to 6 for the noon window — MoveCiBuildsMetadataSelfManaged processed <strong>3,039 rows in 40 minutes: 0.9%</strong> of the total. There was no ambiguity left about whether this would finish that day.</p><h3>The Decision: Accept the Deferral, Don’t Force It</h3><p>The response to both giant-table cases was the same, and deliberate: <strong>stop for the day, continue at the next station — don’t reach for a more aggressive lever, like pushing parallelism even higher, to force completion.</strong> This follows the same principle established earlier in this series for maintenance windows generally: if the service is stable, don’t fight to finish inside a window that was never going to be long enough. A migration that’s still running isn’t broken. It’s just going to take longer than one lunch hour, and that’s fine.</p><h3>Under the Hood</h3><pre>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Station                     | What Happened                                   | What It Taught                                       |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| 5                            | Parallelism 2 → 4, mid-size migrations           | The idle noon window could safely absorb a higher    |<br>|                             | cleared visibly faster                            | parallelism setting                                    |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| —                            | REST API silently ignored the setting             | Verify a setting is actually reachable via API        |<br>|                             |                                                    | before assuming it can be automated that way           |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| 6                            | Rule formalized (6 at noon, 2 at 13:00);          | Confirmed the rule works for mid-size migrations,     |<br>|                             | a 77,471-row table still didn&#39;t clear by 12:40    | and found its first real edge                          |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| 7                            | A 322,136-row table processed 0.9% in 40 min      | Confirmed the edge wasn&#39;t a fluke — accepted           |<br>|                             | at full parallelism                                | deferral instead of forcing it                          |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Formalizing the rule into a strict, time-boxed toggle — 6 at noon, back to 2 at 13:00 no matter what — meant the benefit (faster mid-size migrations) never came with the cost of holding elevated database load longer than the quiet window actually justified. Verifying the setting’s real reachability, rather than assuming the official API would accept it, avoided building an automation step around a field that would have silently done nothing.</p><h3>Where This Still Falls Short</h3><p>This isn’t a story about a rule failing — it’s a story about a rule turning out to have a boundary that took two more stations to actually see clearly. It’s worth being honest that this is a correction to understanding, not a dramatic failure: the tool does exactly what it always did, and the only thing that changed was knowing exactly where it stops helping.</p><p>Have you ever formalized a rule that worked the first time, only to find its real limit a few uses later?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-6-the-bug-gitlab-hasnt-fixed-yet-68f7eebe75b8?sharedUserId=jason-chen-0604">Station 6: The Bug GitLab Hasn’t Fixed Yet</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-7-8-the-backup-that-looked-broken-2772b46c0902?sharedUserId=jason-chen-0604">Station 7–8: The Backup That Looked Broken</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bcab19331ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Station 6: The Bug GitLab Hasn’t Fixed Yet]]></title>
            <link>https://jason-chen-0604.medium.com/station-6-the-bug-gitlab-hasnt-fixed-yet-68f7eebe75b8?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f7eebe75b8</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 06:26:13 GMT</pubDate>
            <atom:updated>2026-09-09T06:26:28.256Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — The error message said a column was missing. It wasn’t missing — it had been renamed three versions earlier. The real cause was a GitLab bug that’s still open today.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zhuKq5Pn4dTZie0TLYboWA.png" /><figcaption><em>A five-step trace from a generic 500 page to an open GitLab issue number</em></figcaption></figure><blockquote><em>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: </em><a href="https://jason-chen-0604.medium.com/station-5-6-the-migration-that-fixed-itself-e2fc743f5d33?sharedUserId=jason-chen-0604"><em>Station 5–6: The Migration That Fixed Itself</em></a><em> ｜ Series overview: </em><a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604"><em>Live Wire — Series Overview</em></a><em> ｜ Next: coming soon</em></blockquote><h3>TL;DR</h3><p>After upgrading to 18.5.7, pages started throwing intermittent, generic 500 errors — easy to write off as noise, since not every request hit it. The real error underneath pointed at a missing database column. The column wasn’t missing; it had been renamed years earlier, and the table it belonged to was structurally fine. The actual cause traced back to a GitLab bug, still open today, where an unrelated migration triggers a stale in-memory schema cache. The almost-wrong fix was patching the database by hand. The actual fix was a restart.</p><h3>A 500 Page That Comes and Goes</h3><p>Not every failure announces itself clearly. After upgrading to 18.5.7 — with the container reporting healthy — the liveness endpoint and a handful of other pages started intermittently returning a plain, unhelpful <strong>“500: We’re sorry, something went wrong on our end.”</strong> Intermittent failures are the easiest kind to dismiss, precisely because most requests succeed and the pattern doesn’t jump out immediately.</p><h3>The Fix, in One Sentence</h3><p>Trace the error past its surface message to what the schema actually looks like, confirm whether it’s a real corruption or a stale cache, and resist the urge to “fix” a database that was never actually broken.</p><h3>Quick Start: Quantifying and Confirming Before Acting</h3><pre># Don&#39;t judge severity from a single screen refresh — count it<br>grep -c &quot;PG::UndefinedColumn&quot; gitlab-rails/production.log</pre><pre>-- Confirm whether the column is actually gone, or just renamed<br>SELECT column_name FROM information_schema.columns<br>WHERE table_name = &#39;application_settings&#39;<br>  AND column_name LIKE &#39;%secret%&#39;;</pre><p>Both of these changed the read on what was actually happening.</p><h3>Step 1 and 2: A Vague Symptom, Then the Real Error</h3><p>The screen said nothing useful. The logs said something specific:</p><pre>ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR:  column application_settings.pre_receive_secret_detection_enabled does not exist<br>LINE 1: ...&quot;enable_artifact_external_redirect_warning_page&quot;, &quot;applicati...</pre><p>A missing column, on the face of it — the kind of error that usually means a migration didn’t run, or the schema is out of sync with the code expecting it.</p><h3>Step 3: This Wasn’t a One-Off</h3><pre>grep -c &quot;PG::UndefinedColumn&quot; gitlab-rails/production.log<br># 1506</pre><p><strong>1,506 occurrences</strong>, not a single unlucky request. Whatever this was, it was happening constantly, just not on every single page load — which is exactly the pattern that makes an intermittent error easy to underestimate from the UI alone.</p><h3>Step 4: The Schema Wasn’t Actually Broken</h3><pre>SELECT column_name FROM information_schema.columns<br>WHERE table_name = &#39;application_settings&#39; AND column_name LIKE &#39;%secret%&#39;;</pre><p>pre_receive_secret_detection_enabled genuinely wasn&#39;t there. But a different, clearly related column was: <strong>secret_push_protection_available</strong> — present, intact, doing its job. That&#39;s the pattern of a column that was renamed at some point, not one that vanished from a broken migration. The table itself was structurally fine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQnWPQXu-8FPcZeNUVzzLA.png" /><figcaption><em>The schema wasn’t broken. Something else was reading it wrong.</em></figcaption></figure><h3>Step 5: An Open Issue With GitLab’s Name on It</h3><p>Tracing the exact error signature further led to <strong>GitLab issue #576490</strong> — open, no milestone attached, meaning not yet fixed on GitLab’s side at the time of writing. The description matched exactly: during the 18.2.8-to-18.4.2 range, a post_migrate migration called FinalizeMarkAdminBotRunnersAsHosted — a migration entirely about runners, with nothing to do with application_settings — runs, and as a side effect, Rails loads the ApplicationSetting model with a stale in-memory schema cache still holding the old, pre-rename column name.</p><h3>The Fix That Almost Happened</h3><p>The instinct, once a “missing column” error is confirmed, is to add the column back. That instinct was wrong here, and it’s worth being specific about why:</p><ul><li>The migration actually throwing the error had <strong>nothing to do with</strong> application_settings — it was collateral damage from an unrelated runner migration</li><li>The replacement column, secret_push_protection_available, genuinely existed — proof the schema itself wasn&#39;t corrupted</li><li>Manually adding the old column back would have created an <strong>unofficial ghost column</strong>, one that could directly conflict with a real GitLab migration touching the same table later</li></ul><p>Patching the database here would have “fixed” a database that was never actually broken, while creating a new, self-inflicted problem in its place.</p><h3>The Actual Fix: Don’t Touch Data, Clear the Cache</h3><pre>ssh Gitlab &quot;bash -ic &#39;cd ~/gitlab &amp;&amp; echo \$SUDO_PASS | sudo -S docker-compose restart gitlab&#39;&quot;</pre><p>A restart clears the stale in-memory schema cache without touching a single row of actual data. After restarting: the error count held steady at 1,506 — no new occurrences — and manually testing several pages (sign_in, explore, admin_settings) came back clean, with the liveness endpoint returning a consistent 200.</p><h3>Under the Hood</h3><pre>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Step                       | What I Found                                     | Why It Mattered                                       |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Symptom                     | Generic, intermittent 500 — easy to dismiss       | Underestimating an intermittent error is the         |<br>|                             | as noise                                          | most common way to let a real one slip past           |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Real error                  | PG::UndefinedColumn on application_settings       | Pointed at a specific, checkable claim instead of    |<br>|                             |                                                    | a vague failure                                        |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Scale                       | 1,506 occurrences via log grep                     | Confirmed this was systemic, not a one-off             |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Schema check                 | Old column gone, but a renamed replacement          | Ruled out real schema corruption before               |<br>|                             | column genuinely existed                            | considering any fix                                     |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Root cause                   | GitLab issue #576490 — an unrelated migration      | Confirmed this wasn&#39;t a local misconfiguration,       |<br>|                             | triggering a stale schema cache, still open         | it was an upstream, already-known bug                  |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+<br>| Fix                          | Restarted the service instead of editing the        | Cleared the stale cache without creating an           |<br>|                             | database                                            | unofficial, conflict-prone ghost column                 |<br>+---------------------------+-----------------------------------------------+----------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Quantifying the error before reacting to it — 1,506, not “sometimes” — is what turned a dismissible-looking symptom into something worth fully tracing. Checking the schema directly, rather than trusting the error message’s framing of “missing,” is what caught that the real problem was a stale cache, not a broken table — and that distinction is exactly what separated the correct fix (restart) from the tempting wrong one (patch the database).</p><h3>Where This Still Falls Short</h3><p>This is GitLab’s own bug, not a misconfiguration in this environment — worth stating plainly so it doesn’t read as an operational mistake. At the time of writing, issue #576490 is still open with no milestone attached, meaning GitLab hasn’t shipped an official fix. If that changes later, this account of “GitLab hasn’t fixed it yet” will need revisiting — the workaround here is a safe way to live with the bug, not a substitute for the real fix landing upstream.</p><p>Have you ever traced an error message back to a root cause that was completely unrelated to what the message itself claimed was wrong?</p><blockquote><em>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: </em><a href="https://jason-chen-0604.medium.com/station-5-6-the-migration-that-fixed-itself-e2fc743f5d33?sharedUserId=jason-chen-0604"><em>Station 5–6: The Migration That Fixed Itself</em></a><em> ｜ Series overview: </em><a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604"><em>Live Wire — Series Overview</em></a><em> ｜ Next: coming soon</em></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f7eebe75b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Station 5–6: The Migration That Fixed Itself]]></title>
            <link>https://jason-chen-0604.medium.com/station-5-6-the-migration-that-fixed-itself-e2fc743f5d33?source=rss-8e9e2f4ae9e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2fc743f5d33</guid>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[陳昶仲]]></dc:creator>
            <pubDate>Wed, 09 Sep 2026 06:08:41 GMT</pubDate>
            <atom:updated>2026-09-09T06:28:05.062Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Live Wire: A Solo GitLab Upgrade Log — Official advice, before Station 5 even started: don’t fix this one, just keep upgrading. It worked. Verifying exactly how it worked is a different story.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EBQBFcqvKUgEF3NoldnFDg.png" /><figcaption><em>Half the story checked out under verification. The other half didn’t — and that’s still worth telling</em></figcaption></figure><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-4-the-day-before-the-six-minutes-a8eb0c03760c?sharedUserId=jason-chen-0604">Station 4: The Day Before the Six Minutes</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-6-the-bug-gitlab-hasnt-fixed-yet-68f7eebe75b8?sharedUserId=jason-chen-0604">Station 6: The Bug GitLab Hasn’t Fixed Yet</a></blockquote><h3>TL;DR</h3><p>Before Station 5 even started, official documentation already predicted a specific migration would fail — and gave counterintuitive advice: don’t fix it, just keep upgrading, because later versions clean it up automatically. It failed exactly as predicted. Continuing forward instead of stopping to patch it was the right call — the migration is now confirmed finalized and the data fully intact. Verifying that story for this article, though, turned up something the original research got wrong: the two specific &quot;fix migrations&quot; the advice pointed to don&#39;t exist anywhere in this environment. The outcome was real. The mechanism I wrote down for it wasn&#39;t provable.</p><h3>A Migration That Was Supposed to Fail</h3><p>Some failures are surprises. This one wasn’t. Before Station 5 (17.11.7 to 18.2.8) even started, official GitLab documentation and forum reports already flagged that BackfillSentNotificationsAfterPartition — a migration touching the sent_notifications table — was expected to fail during this exact version jump.</p><h3>The Fix, in One Sentence</h3><p>Trust documented, counterintuitive advice over instinct — and then verify the outcome for real before writing about it, rather than assuming the advice’s explanation was correct just because the advice itself worked.</p><h3>Quick Start: Checking a Migration’s Real Status</h3><pre>-- Confirm whether a specific migration actually finished, and when<br>SELECT id, job_class_name, status, finished_at<br>FROM batched_background_migrations<br>WHERE job_class_name = &#39;BackfillSentNotificationsAfterPartition&#39;;<br><br>-- Confirm whether a named &quot;fix&quot; migration actually exists in this environment<br>SELECT id, job_class_name, status, finished_at<br>FROM batched_background_migrations<br>WHERE job_class_name IN (&#39;MigrationNameOne&#39;, &#39;MigrationNameTwo&#39;);</pre><p>The first query told the expected story. The second one didn’t.</p><h3>What the Documentation Said, and Why It Made Sense</h3><p>The official explanation, once tracked down: when 18.2.x schedules this backfill, it doesn’t first create partitions covering the <em>old</em> range of created_at timestamps — so migrating historical rows fails with PG::CheckViolation, because there&#39;s no partition for them to land in. The documented fix was specific: <strong>keep upgrading to 18.3.0 or later</strong>, where two follow-up migrations were said to clean up and reschedule the stuck backfill automatically. And the advice came with an explicit exception carved out: <strong>normally you shouldn&#39;t upgrade past a version with unfinished background migrations — but this specific case was called out as safe to do anyway.</strong></p><h3>Confirming the Risk Was Real, Not Theoretical</h3><pre>SELECT count(*) FROM sent_notifications;<br>-- 48,422</pre><p>Not an empty table that could simply be ignored — 48,422 real rows meant this migration had actual work to do, and the partition-key collision condition genuinely applied here.</p><p>One more detail worth digging into before Station 5 started: sent_notifications was, at the time, an <strong>ordinary table, not yet partitioned</strong> — the partitioned version the forum reports referenced didn&#39;t exist yet; it would only get created during the 18.2.8 upgrade itself. The table didn&#39;t even have a created_at column yet, either, even though the documented root cause was specifically about a created_at-based partition range. That gap made sense once connected: 18.2.8 was about to <strong>add</strong> that column and partition on it — which is exactly why older rows, created before the column existed, wouldn&#39;t have a matching partition to land in.</p><h3>Station 5: It Failed Exactly as Predicted</h3><pre>BackfillSentNotificationsAfterPartition — 48,422 rows<br>5 of 5 batches: failed (retried 1–3 times each, then status → 4/failed)</pre><p>No surprises here — the documented risk played out precisely as described. The call, per the documented advice, was to leave it alone rather than intervene, and continue toward Station 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2ieoYLt9-Hsl1cjyZ1Zww.png" /><figcaption><em>The result held up under verification. The explanation for it didn’t.</em></figcaption></figure><h3>Under the Hood</h3><pre>+---------------------------+------------------------------------------------+---------------------------------------------------+<br>| Item                       | What I Did                                       | Why It Mattered                                      |<br>+---------------------------+------------------------------------------------+---------------------------------------------------+<br>| Pre-upgrade research        | Found the documented failure and its             | Turned a surprise failure into an expected,          |<br>|                             | counterintuitive &quot;don&#39;t fix it&quot; advice            | already-understood one                                 |<br>+---------------------------+------------------------------------------------+---------------------------------------------------+<br>| Impact check                 | Confirmed the table had real, non-empty data      | Ruled out &quot;this doesn&#39;t actually apply here&quot;          |<br>|                             | before trusting the documented risk applied        | before trusting the plan                                |<br>+---------------------------+------------------------------------------------+---------------------------------------------------+<br>| Schema detective work        | Found the table had no created_at column yet,     | Connected an apparently unrelated fact to the exact   |<br>|                             | but the failure was about created_at partitioning  | root cause the documentation described                 |<br>+---------------------------+------------------------------------------------+---------------------------------------------------+<br>| Decision at Station 5        | Left the failed migration alone, continued         | Trusted documented advice over the instinct to        |<br>|                             | upgrading rather than stopping to patch it          | patch every failure immediately                          |<br>+---------------------------+------------------------------------------------+---------------------------------------------------+</pre><h3>What Actually Worked</h3><p>Trusting the documented, counterintuitive advice — instead of the instinct to fix every failure the moment it appears — was the right call here, and the eventual finalized status and intact row count confirm it. Verifying the table&#39;s actual state before Station 5 (real row count, missing column, not-yet-partitioned structure) is what turned a generic forum warning into a confirmed, environment-specific risk instead of an assumption.</p><h3>Where This Still Falls Short</h3><p>This is the part of the story that didn’t hold up, and it belongs in the article rather than quietly edited out. See the appendix below.</p><h3>Appendix: Verifying My Own Notes</h3><p>Before writing this article, one detail needed a final check: the specific claim that two named migrations — CleanSentNotificationsFirstBackfill and RescheduleSentNotificationsBackfill — were what actually cleaned up and rescheduled the failed backfill. That claim had never been independently verified; it was carried forward from the original documentation research and never re-checked against this environment directly. Before putting it in writing, it seemed worth confirming.</p><pre>SELECT id, job_class_name, status, finished_at<br>FROM batched_background_migrations<br>WHERE job_class_name = &#39;BackfillSentNotificationsAfterPartition&#39;;</pre><pre>id  |             job_class_name              | status |          finished_at<br>-----+------------------------------------------+--------+-------------------------------<br> 609 | BackfillSentNotificationsAfterPartition   |      6 | 2026-09-01 04:21:04.032121+00<br>(1 row)</pre><p>That part matched expectations exactly — status = 6 (finalized), completed September 1st.</p><pre>SELECT id, job_class_name, status, finished_at<br>FROM batched_background_migrations<br>WHERE job_class_name IN (&#39;CleanSentNotificationsFirstBackfill&#39;, &#39;RescheduleSentNotificationsBackfill&#39;);</pre><pre>id | job_class_name | status | finished_at<br>----+-----------------+--------+-------------<br>(0 rows)</pre><p>Zero rows. Neither named migration exists in this table. A source-code search came back the same way:</p><pre>docker exec -u git -w /home/git/gitlab gitlab_gitlab_1 \<br>  grep -rl &quot;CleanSentNotificationsFirstBackfill\|RescheduleSentNotificationsBackfill&quot; db/</pre><pre>(no output — zero matches)</pre><p>Whatever actually resolved this migration, it wasn’t two independently named, identifiable migrations by those class names — at least not findable through either the database or the current source tree.</p><p>The data itself told a cleaner story:</p><pre>SELECT tablename FROM pg_tables WHERE tablename LIKE &#39;%sent_notification%&#39;;</pre><pre>tablename<br>------------------------<br> p_sent_notifications<br> p_sent_notifications_1<br> p_sent_notifications_2<br> notification_settings<br>(4 rows)</pre><pre>SELECT count(*) FROM p_sent_notifications;<br>-- 49478</pre><p>The original sent_notifications table is gone. In its place: p_sent_notifications, properly partitioned across two sub-tables, holding 49,478 rows — slightly more than the 48,422 counted before Station 5, consistent with normal use in the time since. <strong>No data was lost. The problem is, by every measure available, actually resolved.</strong></p><p>What can’t be confirmed is the <em>mechanism</em>. Three explanations remain open, and none of them can be ruled in or out from here: the migration names might simply be remembered incorrectly; they might be standard Rails schema migrations rather than batched background migrations, which wouldn’t appear in this table at all; or the fix might have been folded into some other, differently-named migration entirely. The honest version of this story has two halves — one fully confirmed, one that stops at “I don’t know, and I checked.”</p><p>Have you ever gone back to verify a detail you were confident about, only to find the outcome was real but your explanation for it wasn’t?</p><blockquote>Live Wire: A Solo GitLab Upgrade Log ｜ Previous: <a href="https://jason-chen-0604.medium.com/station-4-the-day-before-the-six-minutes-a8eb0c03760c?sharedUserId=jason-chen-0604">Station 4: The Day Before the Six Minutes</a> ｜ Series overview: <a href="https://jason-chen-0604.medium.com/live-wire-a-solo-gitlab-upgrade-log-series-overview-7a5ecd66d89b?sharedUserId=jason-chen-0604">Live Wire — Series Overview</a> ｜ Next: <a href="https://jason-chen-0604.medium.com/station-6-the-bug-gitlab-hasnt-fixed-yet-68f7eebe75b8?sharedUserId=jason-chen-0604">Station 6: The Bug GitLab Hasn’t Fixed Yet</a></blockquote><ul><li>GitHub: <a href="https://github.com/jasonChen0604">jasonChen0604</a></li><li>Portfolio: <a href="https://jason-chen-1cb56.web.app/">jason-chen-1cb56.web.app</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/jason-cj-chen/">jason-cj-chen</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2fc743f5d33" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>