DEV Community: DevOps Daily

What Happens When You Type a URL: Answer It With Three Interactive Simulators

DevOps Daily — Tue, 08 Sep 2026 16:43:00 +0000

"What happens when you type a URL and press enter?" is the most reused interview question in infrastructure, and for good reason: the honest answer touches DNS, TCP, TLS, CDNs, load balancers and application servers in one breath. It packs a whole networking curriculum into one question.

Most people prepare for it by reading one of the famous write-ups. That produces a memorized list. The version that survives interview follow-up questions ("and what if the cache is cold?", "why does the first request cost more?") comes from having watched the steps happen and poked at them. Below is a way to build that version in an evening, using three free browser simulators, each covering one chunk of the story. Disclosure up front: I help build these; genuinely good alternatives are linked at the end.

Chunk 1: the name becomes an address

Everything starts with DNS, and DNS is where memorized answers are weakest. Most people can say "the domain gets resolved"; far fewer can say who is asked, in what order, and what happens at each hop.

The DNS Simulator steps through the resolution story interactively, on a simplified model with its own records: browser cache first, then the OS cache, then your configured resolver, and if the resolver has nothing cached, the full walk: root server, TLD server, authoritative server. Each step shows what is asked and what comes back. It is a model, not live DNS, which is the point: the walk is slowed down enough to read.

The three facts to take away from playing with it:

Caches answer first. The browser and OS caches can resolve the name before any network question is asked, which is why the "full" DNS story you memorized often does not happen, and why cache TTLs are what stand between you and a propagated DNS change.
The resolver does the walking, not you. Your machine asks one question; the recursive resolver deals with root, TLD and authoritative servers on your behalf.
One machine answers many questions. Replaying the walk with different record types (A, AAAA, CNAME, MX, TXT) shows the same machinery serving addresses, aliases, mail routing and ownership proofs.

Chunk 2: the address becomes a connection

With an IP in hand, the browser opens a TCP connection, and here interviews love the follow-up: "why does TCP need a handshake and UDP does not?"

The TCP vs UDP simulator answers it by letting you break the network. You watch the SYN, SYN-ACK, ACK exchange establish a TCP connection, then flip on packet loss (the simulator randomly destroys packets in transit) and watch what each protocol does about it. In the model, TCP notices the hole and retransmits the lost data; UDP keeps firing and the lost datagrams just never arrive. The counters keep score: packets sent, received, and the retries TCP needed. It is a simplified model of retransmission, not a full TCP stack, but the asymmetry it shows is the real one.

After that, the interview answer is a tradeoff you have watched instead of a definition: TCP gives you an ordered, reliable byte stream at the price of handshakes and retransmissions; UDP strips out connection setup and recovery entirely, and anything that vanishes is upstream's problem. Which is also the setup for the modern footnote worth knowing: HTTP/3 runs over QUIC, which is built on UDP precisely to escape some of TCP's costs while rebuilding the guarantees it still needs.

Chunk 3: the connection becomes a response

The last chunk is everything between "connected" and "response received", and it is the part most answers hand-wave: where TLS happens, what a CDN actually short-circuits, and which hops a request pays for on the way to the application.

Packet Journey animates the full path: browser, DNS resolution, TCP handshake, TLS handshake, CDN edge, load balancer, web server, application, database. The reason it teaches more than a diagram is its four scenarios, which you can run back to back:

Basic HTTP: the naive path, no encryption, straight to origin.
HTTPS: same path plus the TLS handshake, so you see where TLS enters the sequence.
HTTPS + CDN: the request detours to an edge node, and on a miss still travels to origin.
HTTPS + CDN (cached): the edge answers directly, and the origin, load balancer, application and database never hear about it.

Watching scenarios 3 and 4 side by side is the fastest way to internalize what "cache hit ratio" actually buys: not a faster server, but entire hops that stop existing for most requests.

Assembling the interview answer

After the three simulators, the answer is no longer a memorized list; it is a story you can enter at any point:

The browser checks its own DNS cache, then the OS's; on a miss the recursive resolver walks root, TLD and authoritative servers. With an IP, TCP's three-way handshake establishes a connection, TLS negotiates encryption on top, and the HTTP request goes out. If the site uses a CDN, an edge node may answer from cache and the origin never sees the request; otherwise it flows through load balancer, web server and application, often to a database and back.

The difference is you can now survive the follow-ups, because for each stage you have seen the failure mode: the cold cache, the lost packet, the CDN miss.

The three simulators are part of 50+ free DevOps games and simulators, all browser-based, no signup. For going deeper on the same material: Julia Evans' Mess with DNS gives you a real subdomain to experiment on, and the classic what-happens-when repo is the encyclopedic written version of this whole question.

Who Owns the State File, and Other Questions That Decide Your Week

DevOps Daily — Thu, 03 Sep 2026 09:00:00 +0000

The Terraform incidents that eat a week rarely start with a bad resource block. They start with a question nobody answered early: two people ran apply against the same state at the same time; production and a sandbox share one state file and someone ran destroy in the wrong directory; a security group was edited in the console in March and nobody noticed until a plan in June wanted to "fix" it; a plan with 40 destroys got applied because the review looked at the HCL diff and not at the plan.

Each of those is a state question, not a syntax question. This post walks through the four that matter: who owns the state file, how it is split, how you detect drift, and how a plan gets reviewed. For the drift part you get a real run with the configuration to reproduce it. Along the way it names the tools built for each problem.

TL;DR

One writer per state file. A remote backend with locking is the floor. On S3 that now means use_lockfile = true; the DynamoDB lock table is legacy.
Split state by ownership and failure domain , not by convenience. Per environment always; per component when different teams or different lifecycles share a file.
Drift is normal. Run terraform plan -detailed-exitcode on a schedule and treat exit code 2 as "something changed, go look". Use -refresh-only to record what you observed, then fix code or lifecycle rules so the next plan agrees.
Review the plan, not the diff. The plan output is the artifact that changes infrastructure. Put it on the pull request, and make the apply run against a plan someone approved.
State is sensitive. It contains attribute values, including things you did not think of as secrets. Encrypt it, restrict who can read it, and use ephemeral values and write-only arguments to keep secrets out entirely.

Prerequisites

Terraform 1.10 or newer. The examples were run with 1.15.8; use_lockfile needs 1.10+, write-only arguments need 1.11+.
An AWS account if you want to reproduce the S3 backend section. The drift demo runs locally with the hashicorp/local provider, version 2.9.0.
A CI system that can run on pull requests. The examples use GitHub Actions.

Question 1: who is allowed to write the state file?

State is the map between your HCL and real resource IDs. Lose it and Terraform believes nothing exists. Corrupt it with two concurrent writes and Terraform believes the wrong things exist, which is worse. So the first decision is ownership: exactly one process may write a given state file at a time, and every human and pipeline goes through the same lock.

The local backend does lock. It takes an OS-level lock on terraform.tfstate while a command runs, so two commands in the same directory on the same machine cannot collide. What it cannot do is coordinate independent copies: your laptop, a colleague's laptop and a CI runner each have their own file and their own lock. The moment a second person or a pipeline touches the same resources, you have two states and no shared lock.

A remote backend fixes the "where" and shared locking fixes the "one at a time". On AWS the current setup is S3 with native locking:

terraform {
  backend "s3" {
    bucket = "acme-terraform-state"
    key = "platform/network/terraform.tfstate"
    region = "eu-west-1"
    encrypt = true
    use_lockfile = true # S3-native lock, Terraform 1.10+
  }
}

use_lockfile writes a .tflock object next to the state with a conditional PUT (the write succeeds only if the object does not exist yet), so a second writer gets a lock error right away by default. Pass -lock-timeout=5m and Terraform retries for that long instead. Before 1.10 the S3 backend worked without any lock; if you wanted one you added a DynamoDB table (dynamodb_table = "terraform-locks"). That option still works but is deprecated, and new projects should not add the table.

What the bucket and the IAM role need:

Versioning on. Every apply that changes state writes a new object version, and the previous version is your recovery when state is damaged. Each version is a full copy and is billed as one, so add a lifecycle rule that expires noncurrent versions after a period set by your recovery, audit and cost requirements rather than keeping every version forever.
Permissions for the lock file. The role needs s3:GetObject, s3:PutObject and s3:DeleteObject on <state key>.tflock. The state object itself needs GetObject and PutObject only; Terraform never deletes it. Both need s3:ListBucket on the bucket, restricted with an s3:prefix condition to the team's state keys.
Bucket policy scoped per state key. The network team's role can read and write platform/network/*; the app team's role can write only apps/checkout/*. State files are where over-broad IAM turns into an outage.
Encryption with a customer-managed key if compliance asks who can decrypt state. Default SSE-S3 is fine for most teams; the point is that state is not a public artifact.

Note

Three different things protect you here, and it helps to keep them apart. The lock stops two writers running at once. A saved plan (question 4) stops a stale plan from applying: terraform apply tfplan refuses if the state changed after the plan was made, whoever changed it. Neither one notices a change made outside Terraform that never touched state; that is what drift detection (question 3) is for.

The same shape exists on every cloud (Azure Blob with lease-based locking, GCS with native locking). The hosted platforms take the decision away from you: HCP Terraform, Spacelift and env0 put every run behind their own queue, so there is one serialized writer per stack by construction. HCP Terraform also hosts the state; Spacelift and env0 can hold it for you or work against a backend you already own. Digger is different in kind: it runs Terraform inside your existing CI with your backend, and coordinates pull request locks and plan caching from its own component. More on that split in question 4.

Question 2: how is state split?

One state file for everything works until the day a plan runs for eleven minutes and a three-line change proposes destroying something you did not touch. That happens because of dependencies, not bad luck: change an attribute that forces replacement on a subnet, and every resource that references the subnet is re-evaluated, and depending on its schema may be updated in place or replaced too.

The unit of state is the unit of blast radius. Two rules of thumb:

Never share state across environments. prod and staging in one file means every staging experiment refreshes and plans production, and a destroy in the wrong place takes both.
Split by owner and by lifecycle. Networking and IAM change monthly and belong to a platform team. Application infrastructure changes daily and belongs to product teams. Different owners, different permissions, different rate of change: different state files. Plan duration is a symptom of getting this wrong, not the rule for splitting.

A layout that holds up:

infra/
  platform/
    network/ # VPCs, subnets, peering. Own state.
    iam/ # Roles and policies. Own state.
    clusters/ # EKS, node groups. Own state, reads network values.
  apps/
    checkout/ # Per-app resources: queues, buckets, RDS. Own state per env.
      prod/
      staging/
    search/
      prod/
      staging/

Each leaf directory has its own backend key. How the leaves share values is a security decision in itself. The terraform_remote_state data source is the obvious tool, but to read one output it downloads the whole source state, so the consumer role needs read access to everything in that file, including attribute values you would rather not hand to every app team. Two safer patterns:

Provider data sources. Look the value up from the cloud API by name or tag (data "aws_vpc", data "aws_iam_openid_connect_provider"). The consumer needs read permission on that resource, not on the platform team's state.
Publish selected outputs to a store built for sharing: SSM Parameter Store, a DNS record, a small "exports" configuration. The producer writes exactly what it wants to share; consumers read that.

# platform/clusters: publish what apps are allowed to know
resource "aws_ssm_parameter" "oidc_provider_arn" {
  name = "/platform/clusters/prod/oidc_provider_arn"
  type = "String"
  value = aws_iam_openid_connect_provider.eks.arn
}

# apps/checkout/prod: read it without touching platform state
data "aws_ssm_parameter" "oidc_provider_arn" {
  name = "/platform/clusters/prod/oidc_provider_arn"
}

terraform_remote_state is still fine between stacks owned by the same team with the same trust level. Use it knowingly.

Note

Workspaces are not environment isolation. terraform workspace switches between state files under the same backend prefix with the same credentials and the same code. That is fine for short-lived per-branch copies of a stack. It is not fine as the boundary between staging and production, because nothing stops a -destroy in the wrong workspace except attention.

The cost of splitting is orchestration: when the network stack changes, dependents need a plan too. You need three things whatever you build it with: an order to run stacks in, a way to pass values between them, and a way to see that a downstream stack has not been planned since its upstream changed. Terragrunt models this with dependency blocks on the plain CLI; a CI pipeline with explicit job dependencies does it for small graphs; Spacelift stack dependencies and env0 workflows do it as a hosted feature with output passing built in.

Question 3: how do you find out state no longer matches reality?

Two things get called drift, and they need different responses.

Configuration drift is the gap between what your code declares and what actually exists. Someone widened a security group in the console at 3 a.m.; the code still says the old range. The next plan will propose to close it again.

State drift is the gap between what the state file recorded and what the provider API now returns. The resource is fine and matches the code, but state has old attribute values because they changed outside Terraform. A refresh fixes state without touching the resource.

Terraform surfaces both at the same moment: when it refreshes during a plan. Which means you only find out when someone runs a plan, and for a quiet stack that can be weeks.

Here is what it looks like from Terraform's side, run for real with the local provider so you can reproduce it without a cloud account. The full configuration:

# main.tf
terraform {
  required_providers {
    local = { source = "hashicorp/local", version = "2.9.0" }
  }
}

resource "local_file" "app_config" {
  filename = "${path.module}/out/app.env"
  content = "LOG_LEVEL=info\nWORKERS=4\n"
  file_permission = "0644"
}

resource "local_file" "feature_flags" {
  filename = "${path.module}/out/flags.json"
  content = jsonencode({ new_checkout = false, dark_mode = true })
  file_permission = "0644"
}

Apply it, then edit one file by hand and delete the other, then plan again. The transcript below is abridged (the provider prints six hash attributes per resource that add nothing here); the commands, messages and exit code are as they ran with Terraform 1.15.8:

drift demo

$ terraform apply -auto-approve
local_file.feature_flags: Creation complete after 0s [id=497bf222e1c3c415669ba709d62873551fd34315]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
# someone edits one file by hand and deletes the other
$ printf 'LOG_LEVEL=debug\nWORKERS=4\n' > out/app.env && rm out/flags.json
$ terraform plan -detailed-exitcode
local_file.app_config: Refreshing state... [id=7a5c3ff122fe7ec3ef80d88617b257d9a79ed359]
local_file.feature_flags: Refreshing state... [id=497bf222e1c3c415669ba709d62873551fd34315]

Terraform will perform the following actions:

  # local_file.app_config will be created
  + resource "local_file" "app_config" {
      + content = <<-EOT
            LOG_LEVEL=info
            WORKERS=4
        EOT
      + filename = "./out/app.env"
    }

  # local_file.feature_flags will be created
  + resource "local_file" "feature_flags" {
      + filename = "./out/flags.json"
    }

Plan: 2 to add, 0 to change, 0 to destroy.
$ echo $?
2

Two things worth reading closely.

First, the exit code. -detailed-exitcode returns 0 for an empty plan, 1 for an error and 2 for a successful plan with changes. Exit code 2 is a change signal , not a drift verdict: it also fires for code that was merged and never applied, for a variable that changed, or for a provider upgrade that added a default. It becomes a drift detector only when you run it against a stack whose code was fully applied and whose inputs are pinned, so that the only remaining cause of a non-empty plan is the world moving. Even then, a data source that resolved to a new value produces a plan without anyone touching the infrastructure. So treat a scheduled plan as a change check : it tells you a stack would change if applied, and a person classifies why. The hosted platforms' drift detection runs on the same signal and adds the classification for you by comparing refreshed state with the last applied configuration.

Second, what the plan wants to do. The hand-edited file shows up as "will be created" with the original LOG_LEVEL=info. That is a quirk of this provider: local_file identifies a resource by the hash of its content, so a changed file looks like a missing one. A cloud provider would show the same situation as an in-place update (~ ingress { ... }). Either way the plan is proposing to undo the manual change, and whether that is right depends on why the change was made. Terraform cannot know.

You have two honest ways to resolve it:

Reality was wrong, code is right. Apply the plan. The on-call widening gets closed again, and if it was needed, it gets re-added in code where it survives the next apply.

Reality is right, code is stale. Change the code to match, then confirm with a plan that shows no changes. Along the way, a refresh-only apply records what Terraform observed into state without touching any resource:

recording what changed (abridged)

$ terraform apply -refresh-only -auto-approve
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # local_file.app_config has been deleted
  - resource "local_file" "app_config" {
      - content = <<-EOT
            LOG_LEVEL=info
            WORKERS=4
        EOT -> null
      - filename = "./out/app.env" -> null
    }

  # local_file.feature_flags has been deleted
  - resource "local_file" "feature_flags" {
      - filename = "./out/flags.json" -> null
    }
$ terraform state list
# state holds no bindings now. out/app.env still exists on disk with the hand edit; the code still declares both files, so the next plan creates flags.json and overwrites app.env.

That last line is the point about refresh-only: it makes state describe what Terraform saw, and nothing else. If the code still demands the old value, the next normal plan will bring it back. Refresh-only is the first half of accepting a change; editing the code, or telling Terraform to stop reconciling that attribute, is the second half:

resource "aws_autoscaling_group" "web" {
  # ...
  desired_capacity = 3

  lifecycle {
    ignore_changes = [desired_capacity] # the autoscaler owns this now
  }
}

ignore_changes does not stop Terraform from refreshing and recording the attribute. It stops Terraform from planning an update when that attribute differs from the code, which is what you want for values another system legitimately controls.

A change check that runs on a schedule:

# .github/workflows/change-check.yml
name: change-check
on:
  schedule:
    - cron: "17 6 * * 1-5" # weekday mornings, before people start applying
jobs:
  plan:
    strategy:
      fail-fast: false
      matrix:
        stack: [platform/network, platform/clusters, apps/checkout/prod]
    runs-on: ubuntu-latest
    permissions:
      id-token: write # OIDC to AWS
      contents: read
      issues: write # to open or update the issue for the stack
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: 1.15.8
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          # read-only on infrastructure, plus get/put/delete on the .tflock object
          role-to-assume: arn:aws:iam::123456789012:role/terraform-plan
          aws-region: eu-west-1
      - run: terraform -chdir=infra/${{ matrix.stack }} init -input=false
      - name: plan
        id: plan
        run: |
          set +e
          terraform -chdir=infra/${{ matrix.stack }} plan -detailed-exitcode -input=false -lock-timeout=2m -no-color > plan.txt
          code=$?
          set -e
          echo "code=$code" >> "$GITHUB_OUTPUT"
          # 0 and 2 are answers; anything else is a broken check and must fail loudly
          if ["$code" != "0"] && ["$code" != "2"]; then cat plan.txt; exit "$code"; fi
      - if: steps.plan.outputs.code == '2'
        name: open or update the issue for this stack
        env:
          GH_TOKEN: ${{ github.token }}
          STACK: ${{ matrix.stack }}
        run: |
          existing=$(gh issue list --label plan-changes --state open --search "in:title \"Plan changes: $STACK\"" --json number -q '.[0].number')
          if [-n "$existing"]; then
            gh issue comment "$existing" --body-file plan.txt
          else
            gh issue create --title "Plan changes: $STACK" --body-file plan.txt --label plan-changes
          fi

Two details in there are deliberate. The step fails on any exit code other than 0 or 2, so expired credentials or a broken backend cannot produce a green run that quietly stops checking. The issue says "plan changes", not "drift", because the person who opens it has to classify the cause. And the plan takes the lock with a short timeout rather than running with -lock=false; skipping the lock would let the check read state while an apply is halfway through writing it, and a drift report against a half-applied state is noise. If the morning window collides with real applies, move the schedule or accept the two-minute wait.

Question 4: how does a plan get reviewed?

Code review on Terraform has a specific failure mode: reviewers read the HCL diff, which looks small, and approve. Then apply runs and the plan they never saw replaces a subnet, and the resources that depend on it get updated or replaced behind it. The HCL diff was three lines. The plan was 40 destroys.

The plan is the artifact that changes infrastructure, so the plan is what needs review. The workflow that follows:

Pull request HCL change
terraform plan locked, saved to a file
Plan on the PR summary + full output
Approval of the plan, not the diff
Apply the approved plan file

The detail that makes it safe is apply the saved plan. terraform plan -out=tfplan writes a plan file that records the planned actions together with the state it was computed from, the configuration, the provider versions and the input values. terraform apply tfplan refuses to run if the state has moved since. So what was approved is what applies, or nothing applies. Two limits to keep in mind: values that were unknown at plan time are still resolved at apply time, and the plan file does not know about a change made outside Terraform after the plan ran. It also contains sensitive values in clear text, so a stored plan needs the same access controls as state.

Doing this well with plain GitHub Actions is harder than it looks, and the hard part is exactly "apply the plan that was reviewed". A plan produced on the pull request lives in the pull request's workflow run; the merge to main is a different run, with a different commit (the PR ran against a synthetic merge commit, main now has a squash or merge commit), and download-artifact only sees artifacts from its own run unless you hand it a token and the originating run ID. Teams that push through this end up storing the plan somewhere addressable (S3 keyed by PR number and head SHA), verifying at apply time that the merged tree matches the tree that was planned, and re-planning as a fallback. That is a project, not a snippet.

The version below is honest about that: it reviews the plan on the pull request, and on merge it plans again in an ungated job, then applies that plan from a gated job. The order matters: GitHub evaluates an environment's protection rules before the job starts, so a gated job that runs the plan itself would ask for approval of a plan that does not exist yet. Planning first and gating only the apply gives the approver the actual plan to read.

# .github/workflows/terraform.yml
on:
  pull_request:
    paths: ["infra/apps/checkout/prod/**"]
  push:
    branches: [main]
    paths: ["infra/apps/checkout/prod/**"]

# One running and at most one waiting run per stack; a newer waiting run replaces an older one.
concurrency: tf-checkout-prod

env:
  TF_VERSION: 1.15.8
  STACK: infra/apps/checkout/prod

jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
      pull-requests: write # to post the plan comment
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v4
        with: { terraform_version: "${{ env.TF_VERSION }}" }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-plan
          aws-region: eu-west-1
      - run: terraform -chdir=$STACK init -input=false
      - name: plan
        run: |
          set -o pipefail
          terraform -chdir=$STACK plan -input=false -lock-timeout=2m -no-color | tee plan.txt
      - name: post the plan on the pull request
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          {
            echo "### Plan for apps/checkout/prod"
            grep -E "^Plan:|^No changes" plan.txt || true
            echo
            echo "<details><summary>Full plan</summary>"
            echo
            echo '```
{% endraw %}
'
            cat plan.txt
            echo '
{% raw %}
```'
            echo "</details>"
          } > comment.md
          gh pr comment ${{ github.event.pull_request.number }} --body-file comment.md

  plan-for-apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v4
        with: { terraform_version: "${{ env.TF_VERSION }}" }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-plan
          aws-region: eu-west-1
      - run: terraform -chdir=$STACK init -input=false
      - name: plan
        run: |
          set -o pipefail
          terraform -chdir=$STACK plan -input=false -lock-timeout=5m -no-color -out=tfplan | tee plan.txt
          { echo "### Plan waiting for approval"; grep -E "^Plan:|^No changes" plan.txt || true; } >> "$GITHUB_STEP_SUMMARY"
      # The plan file holds sensitive values and backend details: same run only, short retention.
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: ${{ env.STACK }}/tfplan
          retention-days: 1

  apply:
    needs: plan-for-apply
    runs-on: ubuntu-latest
    # The environment's protection rules (required reviewers, prevent self-review,
    # deployment branch = main) are configured in the repository settings; naming
    # it here only opts the job in. The approver reads the plan job's summary
    # and full log before approving.
    environment: production
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v4
        with: { terraform_version: "${{ env.TF_VERSION }}" }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-apply
          aws-region: eu-west-1
      - run: terraform -chdir=$STACK init -input=false
      - uses: actions/download-artifact@v4
        with: { name: tfplan, path: ${{ env.STACK }} }
      - name: apply the approved plan
        run: terraform -chdir=$STACK apply -input=false tfplan

What this buys you: the plan is on the pull request where the reviewer is, the summary line (Plan: 1 to add, 0 to change, 3 to destroy) is visible without expanding anything, the apply job applies exactly the plan file the previous job produced (same run, so download-artifact finds it), the same Terraform version runs everywhere, and the environment gate puts a human in front of the real apply plan. What it does not buy you: a guarantee that the plan on the pull request and the plan at apply are the same. If someone merged another change to the same stack in between, the apply plan will differ, and the environment approver is the only one who sees it.

Note the permissions blocks: once you set any permission on a job, everything you did not list is off, so the plan job needs pull-requests: write for the comment and both jobs need id-token: write for OIDC. Pull requests from forks get a read-only token and cannot post comments; keep infrastructure repos to branches in the same repository.

Where the tools come in, each with a different answer to "which plan applies":

Atlantis (open source, self-hosted) runs as a pull request bot. atlantis plan posts the plan on the PR, atlantis apply applies that saved plan while the PR is still open, and the PR is merged after the apply succeeded. It holds a lock per directory and workspace for the life of the PR so two PRs cannot plan the same stack against each other. Apply-before-merge is the whole idea: it solves plan identity by never letting a merge happen before the reviewed plan has applied.
Digger runs the plan and apply steps inside your existing CI (GitHub Actions, GitLab CI), with your runners and your credentials, and adds an orchestrator component that owns the pull request locks and caches plans between the plan and apply steps. State stays in your own backend. It is the option for teams that want the Atlantis workflow without operating an extra server that holds cloud credentials.
HCP Terraform, Spacelift and env0 are hosted run platforms. Each run plans, waits for approval, then applies from that run's plan, so the reviewed plan and the applied plan are one object. On top of that: run queues per stack; ordering between stacks (Spacelift stack dependencies and env0 workflows also pass outputs downstream; HCP Terraform run triggers only queue the downstream run, and it reads values through data sources or tfe_outputs); policy checks against the plan (Sentinel or OPA in HCP Terraform, OPA in Spacelift and env0; "a plan with more than five destroys needs a second approver" becomes a rule rather than a habit), scheduled drift detection with optional remediation runs, and access control over who may trigger what. Which of those are included depends on the plan or edition you are on, so check before assuming.

The decision between the GitHub Actions version and one of these is not about team size. It is about whether you need any of: a guarantee that the plan reviewed on the pull request is the plan that applies, more than one PR open against the same stack at a time, dependencies between stacks, or policy that is enforced rather than reviewed.

The question under all four: what is in the state file?

Everything Terraform knows about a resource is in state, in plain JSON, including attribute values. That means:

RDS master passwords set through password = var.db_password are in state.
The private key from tls_private_key is in state, in full.
Every resource "random_password" result is in state (the newer ephemeral "random_password" is not).
Attributes you never set but the provider returns (connection strings, generated tokens) are in state.

sensitive = true hides values from plan output. It does nothing to the state file. So the last decision is treating state access as secret access: the bucket policy from question 1, encryption at rest, no terraform.tfstate in a repository, ever, and the same care for saved plan files.

Recent Terraform versions let you keep some secrets out of state entirely. This needs both a Terraform version and a provider version that support it; for the AWS provider, password_wo on aws_db_instance arrived in release 5.88.0 (the Secrets Manager ephemeral resource a little earlier). Pin the exact version you tested and commit the dependency lock file:

terraform {
  required_version = ">= 1.11"
  required_providers {
    aws = { source = "hashicorp/aws", version = "5.88.0" }
  }
}

# Read during the run, never written to state or plan
ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/checkout/db"
}

resource "aws_db_instance" "checkout" {
  # ...
  password_wo = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_wo_version = 1 # bump to rotate
}

Ephemeral resources (Terraform 1.10) are read during the run and discarded. Write-only arguments (Terraform 1.11) accept a value that the provider sends to the API but Terraform never persists; the _wo_version companion is how you tell Terraform the value changed, since it cannot compare something it does not store. Not every resource has a write-only variant yet, so check the provider documentation for the ones you care about.

A short checklist

Run through these for each state file you own.

Remote backend with locking, versioning on with a lifecycle rule for old versions, encryption on.
IAM scoped so a team can write only its own state keys, including the .tflock objects.
No environment shares a state file with another environment.
Components split by owner and lifecycle, with values shared through provider data sources or a parameter store rather than whole-state reads.
A scheduled plan -detailed-exitcode per stack that fails on errors, opens an issue on exit code 2, and lands with someone who classifies the cause (drift, unapplied code, or a moving data source).
Plans posted on pull requests; applies from a saved plan; one run at a time per stack.
A rule, enforced by tooling or by an approval gate, that a plan with destroys gets a second look.
Secrets moved to ephemeral values and write-only arguments where the provider supports them; state and plan files treated as secret material where it does not.

Originally published at devops-daily.com.

How Stripe Avoids Double-Charging Anyone

DevOps Daily — Thu, 03 Sep 2026 09:00:00 +0000

Take Stripe's own classic example: a service sends POST /v1/charges and the socket dies before a response arrives. There are three possible worlds: the request never reached the payment provider, the provider charged the card and the response was lost, or the provider is still working on it. Your code cannot tell them apart, and the customer is waiting. Retry, and you might charge twice. Give up, and you might have taken money without recording an order.

Businesses running on Stripe generated $1.9 trillion in total volume in 2025, by Stripe's own count. At that scale, dropped connections are routine, and every one is a potential double charge. Idempotency keys let clients retry an ambiguous failure safely, and the pattern is small enough to copy in an afternoon. Whether the promise holds is decided by the server-side state machine: what it remembers, in what order, and around which call.

This post combines Stripe's documented API behaviour with the separate Rocket Rides reference design that Brandur Leach published on his own site. We build a smaller Node and Postgres version, test it against concurrent duplicates and a mid-request crash, and look closely at the run where our first version failed.

TL;DR

A client generates a unique key per operation and sends it as Idempotency-Key. The server stores the first result under that key and replays it for any retry with the same key and the same parameters. Stripe's API v1 keeps a key for at least 24 hours and stores the first status and body once the endpoint starts executing, including 500s; validation failures and concurrent conflicts are not stored.
The response cache is the easy half. The hard half is a request that dies in the middle: the server has to know how far it got and resume from there without repeating the one step it cannot undo.
The pattern is atomic phases and recovery points: group local database writes into transactions, put a marker after each, and treat any call to another system (a card network, an email API) as a boundary that must carry its own idempotency key.
Concurrent duplicates are handled by locking the key row, not by hoping they arrive one at a time.
A time-based lock is a lease. A two-second lease let our demo create three rides for one charge; ten seconds avoided the race in the recorded run, but correctness also needs lease renewal or fencing and invariants the database enforces. The output of both runs is below.

Prerequisites

Comfort with HTTP APIs and SQL transactions
Node.js 20 or newer to run the demo
Any Postgres connection string; the run below used a branch on Neon so the schema could be dropped and recreated freely
Familiarity with the phrase "at-least-once delivery" helps; the message queue simulator is a five-minute refresher

The problem, stated precisely

An operation is idempotent when doing it twice leaves the system in the same state as doing it once. GET is idempotent by nature. DELETE is too: deleting an already deleted thing changes nothing. POST /charges is not. Send it twice and you have two charges.

Retries are unavoidable. Stripe's engineering post on the subject, written by Brandur Leach in 2017, splits failures into two kinds. Some are "definitive enough that the client knows with good certainty that it's safe to simply retry": the connection was refused, DNS failed, nothing was ever sent. The dangerous kind is the failure in the middle: the request was sent, then the client timed out waiting for the answer. Now the client's knowledge of the world is stale, and a naive retry is a coin flip between "fine" and "charged twice".

Idempotency keys turn the coin flip into a lookup. The client picks a unique identifier before the first attempt, sends it in the Idempotency-Key header, and reuses it on every retry of that same operation. The server's job is to make sure that no matter how many times a request with that key arrives, the work happens once and every caller gets the same answer.

The rules Stripe documents for its own API are worth reading closely, because each one encodes a lesson:

Keys are client-generated. Stripe suggests a V4 UUID or another random string with enough entropy; keys can be up to 255 characters. The other common strategy is deriving the key from a business object, such as a shopping cart id, which also protects against a user double-clicking "Pay".
Results are cached whether or not the request succeeded. Stripe saves the status code and body of the first request for a key "regardless of whether it succeeds or fails", and that includes 500s. Retrying a 500 with the same key returns the same 500, because the original attempt may have had side effects that Stripe is still reconciling. The advice is to treat a 500 as indeterminate and let webhooks tell you what really happened.
Parameters are compared. Reusing a key with a different request body is treated as a client bug and rejected, not silently replayed.
Concurrent conflicts are not stored. If a request conflicts with another one executing at the same time, Stripe does not save an idempotent result for it, because no endpoint began executing. The client can retry it.
Rate limiting runs before the idempotency layer. A request that was rate limited with 429 can produce a different result on retry with the same key. The layers are ordered on purpose: a limiter that had to consult the key store would not be much of a limiter.
Keys live at least 24 hours (API v1). Stripe may prune a key once it is 24 hours old; a key reused after pruning starts a new request. Stripe's newer API v2 has its own retention and replay rules, so check the version you are on.
Only POST needs it. In API v1 every POST accepts a key; on GET and DELETE, which are idempotent by definition, a key has no effect.
Replays are labelled. A replayed response carries Idempotent-Replayed: true, and a Stripe-Should-Retry header tells well-behaved clients whether retrying is even worth it. The official SDKs generate keys and retry eligible network failures once you turn retries on (maxNetworkRetries in stripe-node); your code still has to treat an indeterminate 500 as unknown and reconcile through webhooks.

From the client side

Most teams meet all of this as a Stripe customer, not as an API author, so here is what the rules look like from that side. Derive the key from the business event (the order, not the attempt), send it on every attempt of that operation, and let the SDK retry the failures that are safe to retry.

Send a key with the request

curl

curl https://api.stripe.com/v1/payment_intents \
  -u "$STRIPE_SECRET_KEY:" \
  -H "Idempotency-Key: order_8f1c2e_charge" \
  -d amount=1900 -d currency=eur \
  -d "payment_method_types[]=card"

stripe-node

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY, { maxNetworkRetries: 2 });

const intent = await stripe.paymentIntents.create(
  { amount: 1900, currency: "eur", payment_method_types: ["card"] },
  { idempotencyKey: `order_${order.id}_charge` },
);

Python

stripe.api_key = os.environ["STRIPE_SECRET_KEY"]
stripe.max_network_retries = 2

intent = stripe.PaymentIntent.create(
    amount=1900,
    currency="eur",
    payment_method_types=["card"],
    idempotency_key=f"order_{order.id}_charge",
)

With retries turned on, stripe-node retries connection failures, concurrent 409 conflicts and eligible 5xx responses with exponential backoff and jitter, and it honours Stripe-Should-Retry; it deliberately does not retry a real rate-limit 429 on its own. If you write your own policy instead, keep the same idempotency key across attempts, honour Stripe-Should-Retry and Retry-After, cap the backoff, add jitter, and do not stack your loop on top of the SDK's.

None of this is exotic. Brandur's separate Rocket Rides post shows one way to implement those semantics on the server when a request dies halfway through, and that is the design we build next.

What the server has to remember

Consider what "create a ride and charge for it" means inside any service that calls a payment provider. It is never a single write. In Brandur's Rocket Rides example (a fictional jetpack rideshare), one API call records a ride, calls Stripe to create a charge, stores the charge id on the ride, and stages a receipt email. The Stripe call is the problem. It is a foreign state mutation : it changes state in a system whose transaction you do not control. You cannot roll it back with the rest of your work, and you cannot make it happen atomically with your own writes.

The design answer is to split the request into atomic phases separated by those foreign calls, and to write a recovery point after each phase so a retry knows where to pick up.

Phase 1 claim the key row
Phase 2 insert ride (tx)
Charge card foreign call, own key
Phase 3 store charge id + response (tx)
Reply or replay on retry

The key row is the memory. In the published design it carries:

the key itself and the user or account it belongs to, unique together, because two customers may pick the same UUID
locked_at, set while a request holds the key, so a concurrent duplicate can be told to wait
recovery_point, the name of the last completed phase (started, ride_created, charge_created, finished)
a fingerprint of the request (method, path, parameters) so a mismatched reuse can be rejected
the response code and body once the request has finished

Three supporting processes complete the picture: an enqueuer that drains staged jobs once their transaction has committed, an optional completer that pushes unfinished requests through their remaining phases when the client has stopped retrying, and a reaper that deletes old keys so the table does not grow without bound. Brandur suggests about 72 hours of retention for the reference design; Stripe's API v1 may prune keys once they are at least 24 hours old.

As a result, a retry does not need special-case code. It claims the key, reads the recovery point, and runs whatever phases are left. If the process died after the card was charged but before the charge id was stored, the retry sees recovery_point = ride_created, calls the card network again with the same downstream idempotency key, receives the same charge back, and finishes. The customer is charged once.

An immediate retry cannot claim the live lease and gets 409. After the lease expires, a retry claims the key, sees recovery_point = ride_created, skips ride creation, calls the provider with the same derived key, receives the same charge id, and completes phase 3.

That last sentence hides a requirement: the downstream call must itself be idempotent, keyed by something you derive from your key. Stripe's API gives you that. If you call an API that does not, you are back to guessing.

A Postgres state machine

We wrote a small version of this in Node with plain pg and ran it against a Postgres branch. The whole thing is one server file, one schema file, and a script that tries to break it. The repo is public:

The-DevOps-Daily/idempotency-keys-demo on GitHub

The "payment provider" is a second endpoint in the same process that the rides API calls over HTTP. It models one Stripe property, the one that matters for this story: repeated requests with the same key return the same charge. It deliberately leaves out parameter checks, retention, cached errors and replay headers. It lives in the same database only so you need one connection string.

What the demo does and does not claim, next to Stripe's documented behaviour:

	Stripe API v1	This demo
Key scope	per account, up to 255 chars	per user, up to 255 chars
Retention	kept at least 24 hours; may be pruned afterwards	never pruned (no reaper)
Same key, different parameters	rejected	rejected with `409`
Concurrent duplicate	conflict, not stored, retryable	`409` while the lease is held
Endpoint `500`	stored and replayed	not stored; lease expires and the retry resumes
Replay signal	`Idempotent-Replayed: true` header	`replayed: true` field in the body
Recovery after an indeterminate `500`	Stripe tries to reconcile and emit webhooks; not guaranteed	recovery point resumes the remaining phases
External boundary	depends on the operation; payment networks for card payments	a second HTTP endpoint in the same process

The tables

CREATE TABLE idempotency_keys (
  id bigserial PRIMARY KEY,
  user_id text NOT NULL,
  key text NOT NULL CHECK (char_length(key) <= 255),
  request_hash text NOT NULL,
  locked_at timestamptz,
  recovery_point text NOT NULL DEFAULT 'started',
  response_code int,
  response_body jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (user_id, key) -- keys are scoped to the account
);

CREATE TABLE rides (
  id bigserial PRIMARY KEY,
  user_id text NOT NULL,
  idempotency_key_id bigint NOT NULL REFERENCES idempotency_keys(id),
  amount_cents int NOT NULL,
  charge_id text,
  created_at timestamptz NOT NULL DEFAULT now()
);
-- One ride per key, enforced by the database (added after the run below).
CREATE UNIQUE INDEX rides_one_per_key ON rides (idempotency_key_id);

-- Stands in for the payments provider.
CREATE TABLE provider_charges (
  id text PRIMARY KEY,
  idempotency_key text UNIQUE NOT NULL,
  amount_cents int NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

Claiming the key

The key-claim transaction is the first concurrency guard. Insert the key row if it does not exist, lock it, and then decide what this request is: a replay, a conflict, or the one that gets to do the work. The reference schema also has a unique constraint tying a ride to its key. The first version of this demo did not, which is how the expired-lease failure below became visible; the final schema has it, and the last run shows what it changes.

// Phase 1 (atomic): claim the key. SELECT ... FOR UPDATE serialises
// concurrent duplicates; whoever comes second sees what the first left behind.
const claim = await tx(async (c) => {
  await c.query(
    `INSERT INTO idempotency_keys (user_id, key, request_hash)
     VALUES ($1, $2, $3) ON CONFLICT (user_id, key) DO NOTHING`,
    [userId, key, requestHash],
  );
  const { rows: [k] } = await c.query(
    `SELECT * FROM idempotency_keys WHERE user_id = $1 AND key = $2 FOR UPDATE`,
    [userId, key],
  );
  // Same key, different request: a client bug, not a retry.
  if (k.request_hash !== requestHash) return { reply: [409, { error: "This Idempotency-Key was used with different parameters" }] };
  // Already finished: replay the stored answer.
  if (k.response_code) return { reply: [k.response_code, { ...k.response_body, replayed: true }] };
  // Take the lock only if nobody holds a live one. clock_timestamp() moves
  // inside a transaction, unlike now(), so the lock time is real.
  const { rowCount } = await c.query(
    `UPDATE idempotency_keys SET locked_at = clock_timestamp()
     WHERE id = $1 AND (locked_at IS NULL OR locked_at < clock_timestamp() - make_interval(secs => $2))`,
    [k.id, LOCK_TTL_MS / 1000],
  );
  if (rowCount === 0) return { reply: [409, { error: "A request with this Idempotency-Key is still in progress" }] };
  return { key: k };
});
if (claim.reply) return json(res, ...claim.reply);

Three things to notice. After loading the row, the hash of the request body is compared first, so a reused key with a different body never takes the lock. (The demo hashes JSON.stringify(body); production code should hash a canonical form that includes the endpoint and every input that changes the result, and nothing volatile.) The replay check comes next, so a finished request answers instantly. And the row lock serialises claimants, while the conditional UPDATE evaluates lease expiry in database time and its rowCount says whether this claimant got the lease.

The phases

// Phase 2 (atomic): local bookkeeping, then move the recovery point.
if (k.recovery_point === "started") {
  await tx(async (c) => {
    await c.query(`INSERT INTO rides (user_id, idempotency_key_id, amount_cents) VALUES ($1, $2, $3)`,
      [userId, k.id, params.amount_cents]);
    await c.query(`UPDATE idempotency_keys SET recovery_point = 'ride_created' WHERE id = $1`, [k.id]);
  });
  k.recovery_point = "ride_created";
}

// Foreign state mutation: the charge. Not inside any of our transactions,
// so it carries its own idempotency key derived from ours. A retry after a
// crash asks the provider for the same charge and gets the same answer.
if (k.recovery_point === "ride_created") {
  const r = await fetch(`http://127.0.0.1:${PORT}/provider/charges`, {
    method: "POST",
    headers: { "content-type": "application/json", "idempotency-key": `${userId}:${key}:charge` },
    body: JSON.stringify({ amount_cents: params.amount_cents }),
  });
  const charge = await r.json();
  if (!r.ok) throw new Error(`provider said ${r.status}`);
  if (crash === "after_charge") throw new Error("simulated crash after the provider charged the card");

  // Phase 3 (atomic): record the charge and the response, release the lock.
  await tx(async (c) => {
    const { rows: [ride] } = await c.query(
      `UPDATE rides SET charge_id = $1 WHERE idempotency_key_id = $2 RETURNING id, amount_cents, charge_id`,
      [charge.id, k.id]);
    const body = { ride_id: ride.id, amount_cents: ride.amount_cents, charge_id: ride.charge_id };
    await c.query(
      `UPDATE idempotency_keys
         SET recovery_point = 'finished', response_code = 201, response_body = $2, locked_at = NULL
       WHERE id = $1`, [k.id, body]);
  });
}

The crash query parameter exists only so the demo can die at the worst possible moment: after the provider has the money, before we have the charge id. On failure the handler returns a 500 and leaves the row locked with its recovery point intact. This is a deliberate departure from Stripe, which stores an endpoint's 500 and replays it; the demo treats the failure as recoverable instead, so the lease expires and the next retry resumes from ride_created.

The provider endpoint is eight lines and one INSERT ... ON CONFLICT. Its whole contract is: same key, same charge.

const row = await pool.query(
  `INSERT INTO provider_charges (id, idempotency_key, amount_cents) VALUES ($1, $2, $3)
   ON CONFLICT (idempotency_key) DO UPDATE SET idempotency_key = EXCLUDED.idempotency_key
   RETURNING id, amount_cents, (xmax = 0) AS created`,
  [id, key, body.amount_cents],
);

(The DO UPDATE that sets a column to itself is a Postgres idiom to make RETURNING produce the existing row on conflict; xmax = 0 tells you whether this call inserted it.)

One winner, nineteen conflicts

The demo script fires three scenarios at the API: twenty concurrent requests with one key, a reuse of that key with a different amount, and a request that crashes after the charge followed by retries. Here is the run, unedited, against a Postgres branch on Neon from a Raspberry Pi:

npm run demo

$ npm run schema
schema ready
$ npm start &
rides api on :4100 (lock ttl 10000 ms)
$ npm run demo
# 1. Twenty clients retry the same request at once (same Idempotency-Key)
statuses: {"201":1,"409":19}
201 bodies all name the same charge: true (ch_bbf48cd47263)
replayed responses: 0, first-time: 1
stats: {"rides":1,"rides_with_charge":1,"provider_charges":1,"provider_cents":1900}

# 2. Same key, different amount: a client bug, not a retry
{"status":409,"body":{"error":"This Idempotency-Key was used with different parameters"}}

# 3. Crash after the card was charged but before we recorded it
first attempt: {"status":500,"body":{"error":"simulated crash after the provider charged the card","recovery_point":"ride_created"}}
stats now: {"rides":2,"rides_with_charge":1,"provider_charges":2,"provider_cents":6100} <- provider has the money, we have no charge_id
retry at once: {"status":409,"body":{"error":"A request with this Idempotency-Key is still in progress"}}
waiting for the lock to expire (10 s)...
retry later: {"status":201,"body":{"ride_id":"2","amount_cents":4200,"charge_id":"ch_165363a6ef7d"}}
retry again: {"status":201,"body":{"ride_id":"2","charge_id":"ch_165363a6ef7d","amount_cents":4200,"replayed":true}}
stats: {"rides":2,"rides_with_charge":2,"provider_charges":2,"provider_cents":6100}

Reading the three scenarios:

The burst. Twenty requests, one winner. The other nineteen arrived while the winner held the lease and got 409. Stripe likewise treats a concurrent conflict on a key as retryable and does not store a result for it. One ride, one provider charge, 1900 cents. A client that received a 409 here should back off and retry with the same key; by then it will get the replayed 201.
The reuse. Same key, 2900 cents instead of 1900. Rejected at the hash check before any lock or write. Silently replaying the 1900-cent result would have been worse than an error: the client thinks it charged 2900.
The crash. The first attempt charges the card (the provider now holds 6100 cents across two charges) and dies before storing the charge id. The immediate retry finds the row still locked and gets 409. After the lease expires, the retry resumes at ride_created, asks the provider for the charge with the same derived key, receives ch_165363a6ef7d again, stores it, and returns 201. A further retry returns the stored body plus a demo-only replayed flag; Stripe keeps the body untouched and signals the replay in the Idempotent-Replayed header instead. Two rides, two charges, one per customer intent. Nobody was charged twice.

The run that went wrong

The output above is the second run. The first one looked like this:

npm run demo (lock ttl 2000 ms)

$ npm run demo
# 1. Twenty clients retry the same request at once (same Idempotency-Key)
statuses: {"201":3,"409":17}
201 bodies all name the same charge: true (ch_6c15fd603155)
replayed responses: 0, first-time: 3
stats: {"rides":3,"rides_with_charge":3,"provider_charges":1,"provider_cents":1900}

Three first-time 201s and three rides for one provider charge. The row locking behaved as written; the two-second lease assumption did not. It was chosen so the crash scenario would not make readers wait. A database query afterwards showed created_at values of 24.7 seconds past the minute for the key row and 28.0, 28.3 and 29.5 for the three rides. Postgres's now() records transaction start rather than the exact insert instant, so these are not precise, but together with the output they are consistent with one picture: under twenty concurrent requests on a cold connection pool, the winner took longer than the lease to get from claiming the key to inserting its ride, and two waiting requests acquired the expired lease while the committed recovery point still said started.

The provider's own idempotency saved the money: all three rides point at the same charge, and the customer paid once. The application data was still wrong, and in a system where the ride-creation phase did something with a side effect (reserved inventory, sent a confirmation), the customer would have noticed.

The lesson generalises past this demo. A lock timeout shorter than your slowest honest request is a duplicate generator. The reference design also lets a retry acquire an expired lock; its optional completer exists for unfinished requests whose clients stopped retrying, and it does not remove the risk of an old worker and a takeover running at the same time. Raising the lease to 10 seconds is what made the recorded run clean, and it is not a fix: no fixed timeout is guaranteed to outlast every pause. Production needs a conservative lease plus renewal or a fencing token, database constraints for every local invariant (here, one ride per key), and alerts for stale work.

The constraint, run

Prose is cheap, so we added the constraint (CREATE UNIQUE INDEX rides_one_per_key ON rides (idempotency_key_id)), put the lease back to 2 seconds, and ran the burst again:

npm run demo (lock ttl 2000 ms, one ride per key)

$ npm run demo
# 1. Twenty clients retry the same request at once (same Idempotency-Key)
statuses: {"201":1,"409":17,"500":2}
201 bodies all name the same charge: true (ch_de1965ba9783)
replayed responses: 0, first-time: 1
stats: {"rides":1,"rides_with_charge":1,"provider_charges":1,"provider_cents":1900}

Same race, different outcome. The winner still finishes with one ride and one charge. The two requests that took over the expired lease now fail on the unique index when they try to insert their ride and return 500, which is the honest answer: something went wrong with their attempt, nothing was duplicated, and their client will retry with the same key and get the winner's replayed 201. Loud failure beat silent duplication; that is the whole point of putting the invariant where a lease cannot reach it.

The pattern beyond payments

Stripe is the reference. Current stripe-node retries eligible failures once by default; maxNetworkRetries changes that count, and the library adds idempotency keys where appropriate. Idempotent-Replayed: true marks a cached server response.
Webhook senders need it in both directions. Svix accepts an Idempotency-Key on its POST endpoints and returns the first result for up to 12 hours; on the receiving side you deduplicate on the message id, as covered in what it actually takes to deliver a webhook in production.
Transactional email is a foreign state mutation with a human on the other end. The smtpfast send API takes an Idempotency-Key and returns the original email id on a retry, which is what let us build a reply feature in that product without a "did the retry send twice?" path.
Job queues deliver at least once. Running a background job that must not be lost is the same idea from the worker's side.

A checklist for your own API

If you are adding idempotency to a POST endpoint, here is the list we would review against:

Scope keys to the caller. The unique constraint is (account, key), never key alone.
Hash and compare the request. Reject the same key when the canonical method, path or any outcome-affecting parameter differs, and document the status you return. Include recipients, amounts and scheduling; leave out volatile transport headers such as tracing ids. A partial fingerprint turns a client bug into a silent wrong answer.
Claim the key atomically, and let the second caller lose. SELECT ... FOR UPDATE plus a conditional update gets you there. Return 409 for an in-flight duplicate and let clients back off and retry.
Treat the lock as a lease. Make it longer than your slowest request measured under load, renew it or fence it with a token, and enforce the one-operation invariant with a unique constraint so a takeover cannot duplicate work even when the lease is wrong.
Write a recovery point after every local phase , before the next foreign call. The phase before a foreign call must be committed, or a retry will repeat it.
Give every foreign call its own key derived from yours. If the downstream API is not idempotent, you have not made your endpoint idempotent, only your database.
Store the final response and replay it verbatim , including errors that were the endpoint's answer. Label replays so clients can tell.
Decide what happens before the idempotency layer. Authentication and rate limiting usually run first, and a 429 or 401 is therefore not cached. Document it, as Stripe does.
Reap old keys. Pick a window longer than your clients' retry and reconciliation period; Stripe's API v1 keeps keys at least 24 hours, which suits an API that gets retried in seconds and reconciled in hours. Make the window explicit in your docs so clients know how long a retry is safe.
Never put personal data in a key. Keys end up in logs on both sides. Stripe's docs say this outright.

The guarantee lives in the state machine

Idempotency keys look like a caching feature and are really a small state machine. The header buys you nothing on its own; the guarantees come from persisted progress, serialised claims, parameter matching, safe foreign calls, and invariants the database enforces. The demo above is about 200 lines because the idea is small. What is not small is the number of ways to get the details slightly wrong, and the two-second run shows why the header and a response cache are not enough on their own.

To try the behaviour, break a receiver in the webhook delivery simulator and watch retries and deduplication play out, or point the demo repo at your own database.

Originally published at devops-daily.com.

Learn SQL by Typing It: Three Browser Database Terminals Compared

DevOps Daily — Wed, 02 Sep 2026 13:51:59 +0000

SQL is a skill you acquire by running queries against data and being wrong a lot. Reading a tutorial produces recognition ("yes, that is a LEFT JOIN"); typing sixty queries produces recall, and recall is what an interview or a production incident actually tests. The problem for beginners is that running queries traditionally starts with installing a database, and "install PostgreSQL" is where a lot of SQL journeys quietly end.

Browser-based database terminals remove that step. Below is a practice path through three of them, each free and signup-free, and each covering a different layer of the skill: the SQL language itself, the psql client you will meet at work, and the document-database alternative for contrast.

Layer 1: the SQL language

The SQL Terminal Simulator is a guided path of 13 lessons against a small commerce dataset: customers, orders, order_items, products. It starts at SELECT and ends at common table expressions and window functions.

The progression, with one example of each rung, all runnable in the simulator:

-- reading
SELECT name, country FROM customers;
SELECT * FROM customers WHERE country = 'Germany';
SELECT * FROM customers ORDER BY signup_date DESC LIMIT 5;
SELECT DISTINCT country FROM customers;

-- aggregating
SELECT COUNT(*) FROM customers;
SELECT country, COUNT(*) FROM customers GROUP BY country;

-- combining
SELECT o.order_id, c.name, o.status
FROM orders o
JOIN customers c ON c.customer_id = o.customer_id
WHERE o.status = 'shipped';

The rung after that is where the dataset earns its keep: revenue does not live in one table. There is no orders.total column, so answering "how much has each customer spent" forces the three-way join that real schemas force on you:

SELECT c.name, SUM(oi.quantity * p.price) AS revenue
FROM customers c
JOIN orders o  ON o.customer_id = c.customer_id
JOIN order_items oi ON oi.order_id = o.order_id
JOIN products p ON p.product_id = oi.product_id
GROUP BY c.name
ORDER BY revenue DESC;

The JOIN versus LEFT JOIN pair is the single most valuable thing to drill. Two classic bugs account for a large share of wrong query results in application code: an inner join silently dropping rows, and an aggregate double-counting because of join fan-out. You learn to smell both by writing the queries and counting the rows that come back.

An implementation detail worth knowing, because it affects what you can type: single-table queries are evaluated live against the in-memory dataset (SELECT, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT and the common aggregates), so you can experiment freely there. The multi-table joins, subqueries and window functions are the guided lessons' specific queries, and their results were captured verbatim from a real PostgreSQL instance, so what you see is exactly what Postgres returns, down to the numeric formatting. Arbitrary ad-hoc joins are not interpreted; for free-form join practice, graduate to a real database (more on that below).

Layer 2: the client is a skill too

Something SQL courses tend to skip: at work you often get a connection string and a terminal, and the tool at the other end is psql. Knowing SQL but not the client leaves you fumbling exactly when someone is watching.

The psql Terminal Simulator drills the client itself across 8 lessons. The meta-commands first:

\l          -- list databases
\dt         -- list the tables in your search path
\d orders   -- describe one table: columns, types, indexes
\di         -- list indexes
\x          -- toggle expanded output (wide rows become readable)
\timing     -- toggle per-query timing

\d is the one you will use daily: it answers "what is actually in this table" without leaving the terminal. \x is the one that saves you the first time a query returns twenty wrapped columns. (In real psql, \q quits; the simulator politely declines that one.)

The later lessons walk the workflow that separates "can query" from "can investigate a slow query", using a deliberately large big_events table:

EXPLAIN SELECT * FROM big_events WHERE user_id = 42;
-- Seq Scan on big_events ...

CREATE INDEX idx_big_events_user ON big_events (user_id);

EXPLAIN SELECT * FROM big_events WHERE user_id = 42;
-- Bitmap Index Scan feeding a Bitmap Heap Scan

In this prepared dataset the plan flips once the index exists; on a real system the planner weighs table size, selectivity and statistics, and sometimes correctly ignores your new index, which is its own lesson. EXPLAIN ANALYZE then shows actual timings, and a transactions lesson (BEGIN, COMMIT, ROLLBACK) covers the day you want to try a risky UPDATE with an exit hatch.

Watching a plan change because of something you did teaches more about indexes than any diagram of a B-tree.

Layer 3: the document side, for contrast

You will meet MongoDB, or something shaped like it, eventually. The mental model is different enough that learning it by contrast, right after SQL, is the cheapest time to do it.

The MongoDB Terminal Simulator covers the shell in 10 lessons: find() with query operators, projections, sort/limit/skip paging, inserts and updates, and the aggregation pipeline. The instructive part is seeing familiar questions wearing new syntax:

// SQL: SELECT name, country FROM customers WHERE country = 'Germany';
db.customers.find({ country: 'Germany' }, { name: 1, country: 1, _id: 0 })

// SQL: SELECT country, COUNT(*) FROM customers GROUP BY country;
db.customers.aggregate([
  { $group: { _id: '$country', count: { $sum: 1 } } }
])

The aggregation pipeline is where the model clicks or does not: you express the computation as an ordered sequence of stages ($match, then $group, then $sort), each transforming the stream. Some people find it clearer than SQL. Almost everyone finds it clarifying to hold both models, because half of understanding a tool is knowing what it is not.

How to actually use these

A workable schedule: one layer per week, fifteen minutes a day, always typing rather than reading. Do the guided lessons once, then come back without the guidance and try to reproduce the queries from the lesson titles alone. Recall, not recognition, is the goal.

Then graduate: install Postgres locally or grab a free hosted instance, load a real dataset, and take the muscle memory with you. The simulators are the on-ramp, not the destination.

Disclosure: I help build these simulators; they are part of 50+ free DevOps games and simulators. For more practice beyond them, SQLBolt and PostgreSQL Exercises are excellent and free, and SQL Murder Mystery is the most fun anyone has ever had with a schema.

Go Is Not Just for CLIs. It Runs the Cloud Native Control Plane

DevOps Daily — Wed, 02 Sep 2026 10:00:00 +0000

There is a meme that goes around every few months: a list of infrastructure tools, each followed by "is Go", ending with "still, you think Go is just for CLIs." The list is accurate, and the reasons behind it decide what a DevOps engineer should learn to read. So instead of repeating the list, we measured it. The language statistics below come from the GitHub API for each project's main repository on September 1, 2026, and the build demo at the end was run for real.

TLDR

Of 20 projects that define the cloud native stack, 19 are majority Go, most above 90%. The exception, Grafana, is a Go backend under a TypeScript frontend.
The reasons are concrete: one self-contained binary, cross-compilation from one machine, goroutines for daemons that juggle thousands of connections, fast compiles, and the gravitational pull of Docker and Kubernetes having chosen Go first.
Go does not own everything. The fastest data paths (nginx, HAProxy, Redis, Envoy) are C and C++, the JVM still runs Kafka, Elasticsearch, and Jenkins, Ansible is Python, and the newest proxies and pipelines are Rust (Linkerd's proxy, Vector, Cloudflare's Pingora).
For DevOps engineers the practical takeaway is "learn enough Go to read the tools you operate" rather than "rewrite your scripts in Go." The step from reading Kubernetes source to writing an operator is short.

Prerequisites

Nothing to install to follow the argument; Go 1.22+ if you want to run the build demo at the end
Familiarity with at least a few of the tools named below

The list, measured

Everyone knows the meme list; here is what the repositories say. Percentages are bytes of code by language from the GitHub API, top language per project:

Share of Go in the main repository, by bytes of code

	Value
CoreDNS	99.9%
Terraform	99.7%
MinIO	99%
Helm	98.4%
Istio	98.1%
Caddy	98.1%
containerd	97.8%
Kubernetes	97.7%
Docker (moby)	97.3%
etcd	96%
Hugo	93.8%
Traefik	93%
CockroachDB	91.4%
Prometheus	88.3%
Cilium	88.3%
Nomad	82.1%
Argo CD	80.4%
Consul	76%
Vault	66.2%
Grafana	45.4%

GitHub API language statistics, main repositories, 2026-09-01. Grafana is the one project where another language (TypeScript, 48.6%) leads.

The numbers add three things the meme leaves out:

The core is Go even where the total is not. Vault (66% Go) and Consul (76%) carry large JavaScript and SCSS shares because they ship web UIs; the servers are Go. Grafana is the honest outlier: the product is a TypeScript frontend and a Go backend in roughly equal measure, so "Grafana is Go" is half true.
The projects are polyglot at the edges. Cilium is 10% C because its datapath is eBPF programs; Hugo carries 2.5% C for a bundled library; CockroachDB has 3% Starlark for Bazel build files. Go owns the control logic, not every byte.
The pattern holds across vendors and foundations. HashiCorp, the CNCF projects, Grafana Labs, MinIO, and Cockroach Labs all landed on the same language, and the reasons below are the ones their engineers cite.

Why Go keeps winning the control plane

The reasons these teams give are operational: the properties of a Go program match what infrastructure software has to do.

One self-contained binary. A Go program compiles to a single executable with the Go runtime (scheduler, garbage collector) linked in, so there is nothing to install beside it, and a pure-Go program built with CGO_ENABLED=0 links statically on Linux with no shared-library dependencies. kubectl, terraform, and caddy are downloaded as one file and run. The demo below shows what that looks like: a working HTTP server in under 6 MB, ldd reporting "not a dynamic executable". For tools that must run on a fleet of hosts you do not fully control, that matters more than any language feature. Compare distributing a Python tool (interpreter version, virtualenv, native wheels) or a JVM service (JDK, heap flags, startup time).

Cross-compile from one laptop. GOOS=linux GOARCH=arm64 go build produces an ARM Linux binary from a Mac in the same command that produced the x86 one, as long as the code stays cgo-free (cgo needs a C toolchain for the target). Release pipelines for these tools are largely a matrix of environment variables rather than a fleet of build machines, which is why the CLIs among them ship darwin, linux, and windows builds for several architectures from the first release.

Goroutines fit daemons. A control-plane component holds thousands of long-lived connections: watch streams in the API server, gossip in Consul, scrape targets in Prometheus, backends behind Traefik. Goroutines make "one lightweight thread per connection" the natural design instead of a callback pyramid or a thread pool tuned by hand, and channels give the coordination primitives. The Kubernetes controller pattern (watch, queue, reconcile) is idiomatic Go.

The compile loop is fast. Fast compilation was an explicit design goal of the language, and it shows in day-to-day work on large codebases: a changed package rebuilds in seconds, and a full build of something the size of Kubernetes is a coffee break rather than a lunch break. Teams that ship weekly with hundreds of contributors feel this daily.

A garbage collector that is good enough for the control plane. Infrastructure code allocates constantly (parsing YAML, JSON, protobuf), and Go's concurrent, low-pause collector keeps latency acceptable for coordination work without manual memory management. It is not free: GC CPU time and occasional pauses are real, which is exactly why the data-path projects in the next section chose otherwise.

Gravity. Docker chose Go in 2013; Kubernetes was rewritten from a Java prototype into Go before its 2014 launch; client libraries, CRD tooling, controller-runtime, and much of the CNCF's shared plumbing came out Go-shaped. A few years in, starting an infrastructure project in anything else meant re-implementing a lot of that plumbing. Gravity is a real technical reason once it exists.

Where Go does not run the show

The meme stops at the control plane on purpose, because the data plane and the older layers are a different story:

Primary language of infrastructure projects that are not Go

	Value	Series
nginx (C)	97.7%	C / C++
HAProxy (C)	96.1%	C / C++
Envoy (C++)	87.7%	C / C++
Redis (C)	68.2%	C / C++
Elasticsearch (Java)	99.2%	JVM
Kafka (Java)	90%	JVM
Jenkins (Java)	87.2%	JVM
Ansible (Python)	86.6%	Python
Pingora (Rust)	100%	Rust
Linkerd2 proxy (Rust)	99.5%	Rust
Vector (Rust)	65.3%	Rust

GitHub API language statistics, 2026-09-01. Redis counts 28.6% Tcl because its test suite is Tcl; the server is C.

The hot data path is still C and C++. nginx, HAProxy, Redis, and Envoy sit where every byte and every microsecond count, and none of them accept a garbage collector on that path. Istio is the cleanest illustration inside one product: its control plane is 98% Go, its sidecar and waypoint proxies are Envoy in C++, and its newer ambient mode adds a Rust node proxy, ztunnel, for L4 traffic.
The JVM runs the big stateful systems. Kafka, Elasticsearch, and Jenkins predate the Go wave and carry ecosystems too large to move. They cost more memory and startup time, and they are not going anywhere.
Python holds configuration management and glue. Ansible is Python, extended by a large audience of operators who write Python modules and plugins rather than systems code.
Rust is taking the new data paths. Linkerd's 2.x proxy was written in Rust from the start (its 1.x proxy was Scala on the JVM) for latency and memory reasons, while its control plane is Go; Vector (observability pipelines) and Cloudflare's Pingora (which replaced Cloudflare's nginx-based origin-facing proxies) chose Rust as well. Where a GC on the hot path is a cost, new projects reach for Rust; where developer throughput matters more, they still reach for Go.

The rough picture in 2026 is two layers: Go for the control plane (scheduling, coordination, configuration, APIs) and C, C++, or increasingly Rust for the data plane (bytes on the wire, storage engines). It is rough because Go does carry real data-path work too: MinIO serves objects, CockroachDB stores rows, and Prometheus ingests samples, all in Go. As a rule of thumb for where a DevOps engineer's reading time goes, it holds.

The six-megabyte demonstration

The claim about self-contained binaries is easy to check. Here is a complete HTTP service (go.mod is two lines: module healthz and the Go version):

package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

func main() {
    host, _ := os.Hostname()
    http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "ok from %s at %s\n", host, time.Now().UTC().Format(time.RFC3339))
    })
    fmt.Println("listening on :8080")
    http.ListenAndServe(":8080", nil)
}

We built it on a Raspberry Pi (arm64, Go 1.26), ran it, and cross-compiled it for three other targets from the same shell:

static binaries

$ CGO_ENABLED=0 go build -ldflags="-s -w" -o healthz .
$ ls -l healthz | awk '{print $5" bytes"}'
5374114 bytes
$ file healthz
healthz: ELF 64-bit LSB executable, ARM aarch64, statically linked, stripped
$ ldd healthz
    not a dynamic executable
$ ./healthz & sleep 1; curl -s localhost:8080/healthz
listening on :8080
ok from raspberrypi at 2026-09-01T21:05:55Z
# same source, other platforms, no other machines involved
$ for t in linux/amd64 darwin/arm64 windows/amd64; do GOOS=${t%/*} GOARCH=${t#*/} CGO_ENABLED=0 go build -ldflags="-s -w" -o healthz-${t/\//-} . && echo "$t $(stat -c %s healthz-${t/\//-}) bytes"; done
linux/amd64 5771426 bytes
darwin/arm64 5428114 bytes
windows/amd64 5901312 bytes

Between 5.4 and 5.9 MB per target, nothing to install beside it, no shared libraries on the Linux build we inspected, four platforms from one directory. The CLIs and single-binary servers in the first chart (kubectl, terraform, caddy, etcd, MinIO) ship in exactly this shape, and that property explains more of the meme than any language feature does. It is also why FROM scratch containers are normal in this ecosystem: the image is the binary. (Not universal: Grafana ships its frontend assets alongside the binary, and Hugo's extended build uses cgo.)

What this means if you run this stack

You do not have to write Go to benefit from the fact that your infrastructure is written in it, but reading it changes how you operate:

Error messages become searchable at the source. When kubectl or terraform prints something cryptic, the string is in a Go file you can find in seconds, with the condition that produced it right above.
Configuration semantics stop being folklore. The definitive answer to "what does this Helm flag do" is a short Go function, and it is usually clearer than the docs.
Extending the tools is the same language as the tools. Kubernetes operators, Terraform providers, Prometheus exporters, Caddy modules, and Traefik plugins are written in Go against libraries the projects maintain. Our guide to writing a simple Kubernetes operator starts from exactly that position.
The language is small. The Go specification is short enough to read in a sitting, and reading competence comes quickly from following code in a project you already run. That is a good return for the time.

The meme holds, for operational reasons: the properties that make Go a good CLI language (one binary, fast start, cross-compile) are the same properties a control plane needs, plus goroutines for the daemons. The data plane keeps going to C and Rust. The layer that schedules, coordinates, and configures your infrastructure is written in Go, and it is worth being able to read.

Originally published at devops-daily.com.

Stop Building Webhook Retries Yourself

DevOps Daily — Wed, 02 Sep 2026 09:00:00 +0000

Teams that ship webhooks tend to write the code twice. First the happy path: an HTTP POST with a JSON body. Then, after the first customer outage, the real product: a retry table, a scheduler, exponential backoff, a place to store failed deliveries, a signature scheme, a way to replay a day of events for one customer, and a dashboard so support can answer "did you get it?" That second half is the expensive one, and it rarely appears in the original estimate.

We took the other route for this article. We built a receiver that fails on purpose in five common ways (returns 500s for a while, answers 429 with Retry-After, hangs past the timeout, stays dead, rejects bad signatures), pointed Svix at it, and recorded what happened, attempt by attempt, with timestamps from both sides. The receiver and the driver scripts are public:

The-DevOps-Daily/webhook-retries-demo on GitHub

Everything below is a real run on September 1, 2026. Where we quote a timing, it comes from the logs in that repo.

TLDR

A single message create fanned out to five endpoints: one healthy control and four failure modes. Svix retried the flaky one on its schedule and it recovered on attempt three at 19:25:23, about four minutes after the first failure, with no code on our side.
The receiver's Retry-After: 60 on a 429 was not honored: the retry arrived 11 seconds later, on the sender's schedule. If you rely on Retry-After, that is a real limitation to know.
A hung endpoint was recorded as request timed out (Svix's documented delivery timeout is 15 seconds) and retried.
Every delivery carried Standard Webhooks signature headers; the receiver verified them with a short handler using the Svix SDK and rejected a forged payload with 401.
Replay is an API call, not a project: resend one message, or recover everything that failed for one endpoint since a timestamp.

Prerequisites

Node.js 22 and a Svix account (the free tier covers this whole exercise)
A public HTTPS URL for the receiver. Svix Cloud rejects plain-HTTP endpoint URLs (Endpoint URL schemes must be https when endpoint_https_only is set), so on a fresh VM we used Caddy with automatic TLS on an sslip.io hostname (157-230-57-75.sslip.io resolves to that IP, and Let's Encrypt issues for it)
npm install in the demo repo

A receiver built to fail

The receiver is one file, one HTTP server, one path per failure mode. It records every request so we can compare its view with the sender's afterwards:

switch (path) {
  case "/ok":
    record(path, msgId, 200, "accepted");
    res.writeHead(200); return res.end("ok");
  case "/flaky": {
    // Fail the first two attempts of every message, succeed on the third.
    if (n < 3) { res.writeHead(500); return res.end("temporary failure"); }
    res.writeHead(200); return res.end("ok");
  }
  case "/ratelimited": {
    // Push back with 429 + Retry-After on the first attempt only.
    if (n === 1) { res.writeHead(429, { "retry-after": "60" }); return res.end("slow down"); }
    res.writeHead(200); return res.end("ok");
  }
  case "/slow": {
    // Never answer within the sender's timeout on the first attempt.
    if (n === 1) return setTimeout(() => { res.writeHead(200); res.end("late"); }, 120_000);
    res.writeHead(200); return res.end("ok");
  }
  case "/dead":
    res.writeHead(503); return res.end("down");
}

n is the attempt count for this message id on this path, which the receiver tracks in memory so it can misbehave a fixed number of times per message. The /ok path also does the thing a production receiver must do with at-least-once delivery: it remembers every svix-id it has processed and acknowledges a redelivery without processing it again. Each case above also calls record(...) so the log at /attempts matches what the sender saw (trimmed here for length; the full file is in the repo).

We exercised the dedup path by sending a second message and then forcing a manual resend of it to /ok:

receiver-side dedup

$ node sender/replay.js resend /ok msg_3Ik0Jx7HzBt7aaXpgEe09l0InQV
resend requested for msg_3Ik0Jx7HzBt7aaXpgEe09l0InQV -> /ok
$ docker logs receiver | grep msg_3Ik0Jx | grep /ok | cut -c1-105
{"at":"20:06:31.076Z","path":"/ok","status":200,"note":"accepted and processed"}
{"at":"20:06:41.039Z","path":"/ok","status":200,"note":"duplicate svix-id, ignored"}

Both deliveries got a 200, because from the sender's point of view both succeeded; only the first one did work. That is the shape of correct at-least-once consumption.

Before any of that runs, every request passes signature verification (more on that below). Bad signature, 401, no processing.

Setting up the sender: three SDK methods

One application, one endpoint per path, then read back each endpoint's signing secret so the receiver can verify:

await svix.application.create({ name: "Retries demo", uid: "retries-demo" });
for (const path of PATHS) {
  const uid = "ep" + path.replace("/", "-");
  await svix.endpoint.create("retries-demo", { url: PUBLIC_URL + path, uid });
  const { key } = await svix.endpoint.getSecret("retries-demo", uid); // whsec_...
}

Sending is one call, with two different duplicate protections that are easy to confuse. eventId is a uniqueness guard: we tested it, and a second create with the same eventId is rejected with msg_exists. The idempotencyKey option (an Idempotency-Key header on the wire) is what makes the create call itself safe to retry after a network blip: we sent the same key twice and got the same message id back both times.

const msg = await svix.message.create(
  "retries-demo",
  {
    eventType: "invoice.paid",
    eventId, // unique per business event
    payload: { invoiceId: "inv_1042", amount: 4900, currency: "usd", sentAt: new Date().toISOString() },
  },
  { idempotencyKey: `send-${eventId}` }, // safe to retry the call
);

That single message fans out to all five endpoints. Here is what the receiver saw in the first eleven seconds (log lines condensed to time, path, status, and note; the full JSON lines are in the repo's RESULTS.md, and this first message ran against the receiver before we added the /ok dedup path, hence accepted rather than accepted and processed):

receiver log

$ docker logs receiver | grep msg_3IjuoZ # condensed
{"at":"19:21:14.552Z","path":"/flaky","status":500,"note":"attempt 1: simulated outage"}
{"at":"19:21:14.555Z","path":"/dead","status":503,"note":"attempt 1: permanently down"}
{"at":"19:21:14.567Z","path":"/ok","status":200,"note":"accepted"}
{"at":"19:21:14.575Z","path":"/slow","status":0,"note":"attempt 1: holding the connection open (will time out)"}
{"at":"19:21:14.578Z","path":"/ratelimited","status":429,"note":"attempt 1: 429 with Retry-After: 60"}
{"at":"19:21:19.059Z","path":"/flaky","status":500,"note":"attempt 2: simulated outage"}
{"at":"19:21:19.152Z","path":"/dead","status":503,"note":"attempt 2: permanently down"}
{"at":"19:21:25.710Z","path":"/ratelimited","status":200,"note":"attempt 2: accepted after backoff"}

Five endpoints hit within 26 milliseconds of each other, the two immediate failures retried about 4.5 seconds later, and the rate-limited endpoint accepted its second attempt 11 seconds after the 429. Nothing in our code scheduled any of it.

The retry schedule, observed

Svix's documented schedule is immediate, then 5 seconds, 5 minutes, 30 minutes, 2 hours, 5 hours, 10 hours, and 10 hours more: eight attempts spread over roughly 27 hours. We let the run continue and pulled the sender's own attempt log per endpoint:

npm run report -- msg_3IjuoZxzSCwvwWsTpkqeZCF3zjD

$ node sender/report.js msg_3IjuoZxzSCwvwWsTpkqeZCF3zjD
/ok (1 attempts)
  19:21:14 http 200 success trigger=scheduled

/flaky (3 attempts)
  19:21:14 http 500 fail trigger=scheduled
  19:21:18 http 500 fail trigger=scheduled
  19:25:23 http 200 success trigger=scheduled

/ratelimited (2 attempts)
  19:21:14 http 429 fail trigger=scheduled
  19:21:25 http 200 success trigger=scheduled

/slow (2 attempts)
  19:21:14 http - fail trigger=scheduled request timed out
  19:22:49 http 200 success trigger=scheduled

/dead (4 attempts)
  19:21:14 http 503 fail trigger=scheduled
  19:21:18 http 503 fail trigger=scheduled
  19:26:17 http 503 fail trigger=scheduled
  19:56:45 http 503 fail trigger=scheduled

Read the /flaky line: two failures, then success on the third attempt, which arrived about four minutes after the second failure (the documented interval for that slot is five minutes, measured from the previous failure). The receiver's own log agrees (attempt 3: recovered). That is the entire transient-outage case, and it cost zero lines of retry code.

Two details matter more than the happy path.

Retry-After was not honored. Our receiver answered the first /ratelimited attempt with 429 and Retry-After: 60. The retry came 11 seconds later, on the sender's own schedule, not 60 seconds later. Svix documents no Retry-After support, and this run confirms it. What Svix offers instead is sender-side: a per-endpoint rate limit (messages per second) you configure, and as of late August 2026, receiver-side response headers webhook-delivery: abort-message (stop retrying this message) and webhook-delivery: disable (stop sending to this endpoint). Those solve "stop" and "slow down in general", not "come back in exactly N seconds". If your consumers lean on Retry-After, know this going in.

Timeouts are counted as failures. The /slow endpoint held the connection open. The sender gave up (its documented limit is 15 seconds; our logs record the attempt start and the failure, not the exact cutoff), logged request timed out with no HTTP status, and the retry landed at 19:22:49, about 95 seconds after the first attempt began. The second attempt succeeded because our receiver only misbehaves once per message. In production, a consumer that takes 20 seconds to process a webhook and then returns 200 has still failed from the sender's point of view; acknowledge fast, process later.

Signatures: the handler you must not skip

Every delivery carries three headers: svix-id, svix-timestamp, and svix-signature. They are Svix-branded aliases of the Standard Webhooks webhook-* headers with identical values, so a Standard Webhooks library verifies them once you map the names (the Svix SDK accepts both spellings):

import { Webhook } from "svix";

const wh = new Webhook(secret); // whsec_... from endpoint.getSecret()
try {
  wh.verify(rawBody, {
    "svix-id": headers["svix-id"],
    "svix-timestamp": headers["svix-timestamp"],
    "svix-signature": headers["svix-signature"],
  });
} catch (err) {
  res.writeHead(401); return res.end("bad signature");
}

Two rules that hand-rolled code usually gets wrong: verify the raw request body exactly as received, never a re-serialized JSON object (one reordered key and the HMAC fails), and reject timestamps outside a tolerance window so a captured request cannot be replayed later; the SDK handles the second, the first is on you. The secret is per endpoint, which is why the setup script prints one whsec_ per path.

We tested the negative path by posting a hand-built request with a forged svix-signature to /ok: the receiver logged signature rejected: No matching signature found, answered 401, and nothing downstream ran.

Dead endpoints and what happens after retries run out

/dead returns 503 forever. We watched it take the first four scheduled attempts on the documented cadence: 19:21:14, 19:21:18 (5 s), 19:26:17 (5 min), and 19:56:45 (30 min); the 2-hour, 5-hour, and two 10-hour attempts were still ahead when we stopped recording. After the eighth failure the message is marked failed and Svix emits an operational webhook, message.attempt.exhausted, to you, the sender, so your own systems can react (open a ticket, email the customer). Endpoints that keep failing get disabled automatically, with an endpoint.disabled event: per the docs, once an endpoint has failures at least 12 hours apart within a 24-hour window, five further days of nothing but failures trips the switch. Both behaviors are configurable per environment.

Who carries that state is the difference between the two approaches. In the do-it-yourself version, every one of those transitions is a row you update, a job you schedule, and an alert you wire. Here it is a webhook you subscribe to.

Replay: the feature you build third and need first

The expensive failure is rarely a single bounced webhook; it is the consumer that was misconfigured for an hour and missed thousands of them. That needs two operations, and both are one API call each:

// resend one message to one endpoint
await svix.messageAttempt.resend(APP_UID, "msg_3IjuoZ...", "ep-dead");

// recover every failed message for this endpoint since a point in time
await svix.endpoint.recover(APP_UID, "ep-dead", { since: new Date("2026-09-01T19:00:00Z") });

We ran both against the dead endpoint at 19:58, right after its 30-minute attempt. Each produced a new delivery within seconds, and the attempt log tells them apart from the schedule:

replay

$ node sender/replay.js resend /dead msg_3IjuoZxzSCwvwWsTpkqeZCF3zjD
resend requested for msg_3IjuoZxzSCwvwWsTpkqeZCF3zjD -> /dead
$ node sender/replay.js recover /dead 2026-09-01T19:00:00Z
recover started for /dead since 2026-09-01T19:00:00Z: { task: 'endpoint.recover', status: 'running' }
$ node sender/report.js msg_3IjuoZxzSCwvwWsTpkqeZCF3zjD | grep -A7 /dead
/dead (6 attempts)
  19:21:14 http 503 fail trigger=scheduled
  19:21:18 http 503 fail trigger=scheduled
  19:26:17 http 503 fail trigger=scheduled
  19:56:45 http 503 fail trigger=scheduled
  19:58:40 http 503 fail trigger=manual
  19:58:50 http 503 fail trigger=manual

Your customers get the same two operations in the embeddable App Portal (Resend on a message, and "Recover Failed Messages" from a date on an endpoint) without a support ticket, and the trigger=manual marker separates operator-initiated deliveries from scheduled ones in the audit trail. In this run the endpoint was still dead, so the replays failed too, which is the correct outcome: recovery re-delivers, it does not pretend.

What you did not have to build

Tally the run against the list from the introduction:

POST /msg your code: 1 call
Fan-out 5 endpoints
Retry schedule 8 attempts / 27h
Signatures Standard Webhooks
Replay + portal API + UI

Retry scheduler and state machine : not built. Observed working across 500, 503, 429, and timeout.
Duplicate protection : eventId uniqueness and idempotencyKey on the send call; svix-id dedup in the receiver, which stays your job under at-least-once delivery.
Signing and verification : SDK, standard headers, tested negative path.
Failure escalation : message.attempt.exhausted and endpoint.disabled operational webhooks.
Replay and recovery : two API calls, also exposed to customers in the portal.
Attempt history for support : report.js is a short loop over the attempts API; the portal shows the same to the customer.

What you still own: fast acknowledgement and svix-id deduplication on the receiving side, the decision of what to do when a customer's endpoint is exhausted, and, if your consumers need Retry-After semantics, that gap. What we wrote for this run was the deliberately broken receiver, the verification handler, and about sixty lines of driver scripts; none of it was retry logic.

Build or buy, with the run in front of you

The DIY version is not hard to start and is hard to finish: the scheduler is small, the portal is not, and the operational edge cases (what does exhausted mean, who gets told, how does a customer self-serve a replay) are the part that keeps leaking into on-call. We covered the sender's side of this in depth in what it actually takes to deliver a webhook in production, including a working DIY implementation, so you can compare the two approaches line by line.

If you also need the other direction, receiving other people's webhooks, the tradeoffs differ; our Svix vs Hookdeck comparison covers both directions and both vendors.

The demo repo takes about ten minutes to set up against a free Svix account and a throwaway VM; letting the retry schedule play out to the 30-minute slot, as we did, takes about 45. Point it at your own receiver, break things your way, and read the attempt log. The retry code you were about to write is the part you can skip.

Originally published at devops-daily.com.

DNS Detective: an Agent That Diagnoses Your Domain by Actually Probing It

DevOps Daily — Tue, 01 Sep 2026 14:00:00 +0000

Ask an LLM "why does mail to my domain bounce?" and you get a plausible list of everything that has ever caused a bounce. Ask an engineer, and they do something different: they run dig, look at the answer, and let the evidence pick the next question. The difference is not knowledge; it is that the engineer is allowed to touch the network.

So we gave the model the network. DNS Detective is a small agent, running on DigitalOcean Serverless Inference, that diagnoses DNS, TLS and email-record problems by calling real probe tools in a loop: resolve records, shake hands with TLS endpoints, pull registration data, fetch URLs. It probes, reads, probes again, and delivers a diagnosis where every claim cites a lookup it actually ran. The whole thing is about 300 lines of Python, and this post walks the build plus three real diagnoses recorded as they happened.

The-DevOps-Daily/dns-detective on GitHub

TLDR

One tool-calling loop plus four probes (dns_lookup, tls_check, rdap_lookup, http_check) turns a chat model into a diagnostician that follows evidence instead of listing possibilities.
On camera it solved three real mysteries: example.com's bouncing mail (a null MX , 0 .), a monitoring alert on expired.badssl.com ( certificate expired 2015 , read from the offered cert after verification failed), and dnssec-failed.org's split behavior ( bogus DS record , and the model noticed the DS digest is literally the ASCII for "broken chain of trust send help!").
The system prompt's one law: never state a record you did not probe. One model we tried broke that law by roleplaying fake probe results and was disqualified; the section below shows why that test matters more than benchmarks.
DigitalOcean's inference platform made the plumbing boring in the good way: OpenAI-compatible API, function calling, a model menu you switch with one env var.

Prerequisites

Python 3.10+, pip install dnspython
A DigitalOcean Serverless Inference API key
No infrastructure: the agent is one file, the probes run from wherever you run it

The architecture is one loop

There is no framework here. The agent is the classic function-calling loop: send the conversation plus tool definitions, and if the model responds with tool calls, run them, append the results, repeat; when it responds with text, that is the diagnosis.

Symptom
Model picks a probe
Probe runs for real
Evidence appended

The four probes are deliberately small and deliberately honest about failure modes, because the failure modes are the diagnosis:

dns_lookup distinguishes NXDOMAIN (domain missing) from NoAnswer (domain exists, record type missing) from NoNameservers (all servers refused, the classic DNSSEC tell) from a timeout. Collapsing those into "lookup failed" would lobotomize the detective.
tls_check does a verifying handshake, and on failure does the thing a human would: re-reads the offered certificate without verification (read-only, openssl s_client style) so it can report what the server actually presented and why it was rejected.
rdap_lookup pulls registration data (registrar, expiry, status) over RDAP, the WHOIS successor, no API key needed.
http_check fetches without following redirects, because the redirect itself is often the clue.

And one paragraph of system prompt does the character work: form a hypothesis, run the cheapest decisive probe next, and never state a record you did not probe.

Three mysteries, on camera

Everything below is real recorded output, straight from the agent's own trail (it prints each probe as it runs, so the diagnosis is auditable).

Case 1: "mail to example.com bounces immediately."

dns-detective

$ python3 detective.py "Customers say they cannot email us at example.com. Mail bounces immediately. Why?"
  → dns_lookup(example.com, MX)
    {"status": "ok", "records": ["0 ."], "ttl": 100}
  → dns_lookup(example.com, A)
    {"status": "ok", "records": ["104.20.23.154", "172.66.147.243"], "ttl": 265}
DIAGNOSIS: example.com is configured with a null MX record (0 .), which explicitly tells the internet that it does not accept email, so all inbound mail is rejected immediately.

Two probes, done. The subtle part: a human junior often misreads 0 . as "some weird MX". The agent correctly identified a null MX (RFC 7505), the record whose entire meaning is "this domain refuses mail on purpose", and did not go on a goose chase through SPF and server logs.

Case 2: "monitoring says the site is down, the server looks up."

dns-detective

$ python3 detective.py "Monitoring says https://expired.badssl.com is down for users but the server itself seems up."
  → tls_check(expired.badssl.com, 443)
    {"handshake": "certificate_verification_failed", "verify_error": "certificate has expired", "offered_certificate": "...notAfter=Apr 12 23:59:59 2015 GMT..."}
DIAGNOSIS: The HTTPS service is reachable, but the TLS certificate expired on 12 April 2015, causing browsers and monitoring to reject the connection.

Note what the probe design bought here: verification failed, so the tool re-read the certificate unverified and handed the model the actual notAfter date. "Down" became "up, but presenting a certificate that died in 2015", which is a different incident with a different fix.

Case 3: "site loads on one old server, nowhere else." The split-behavior mystery, and the agent's best moment:

dns-detective

$ python3 detective.py "dnssec-failed.org will not load for anyone on our office network, but a curl from one old server fetches it fine."
  → dns_lookup(dnssec-failed.org, A) {"status": "ok", "records": ["96.99.227.255"]}
  → dns_lookup(dnssec-failed.org, DNSKEY) {"status": "ok", "records": [two keys, tags 256 and 257]}
  → dns_lookup(dnssec-failed.org, DS) {"status": "ok", "records": ["42069 13 2 62726f6b656e..."]}
DIAGNOSIS: The DNSSEC chain is broken: the parent .org zone publishes a bogus DS record that does not match the domain's DNSKEY, so validating resolvers (the office network) reject the zone while the old server, which skips validation, resolves fine.

It resolved A (zone reachable), pulled the DNSKEYs, pulled the DS from the parent, concluded they cannot match, and explained why exactly the validating resolvers fail while the legacy one sails through. It even noticed that the DS digest is not a hash at all: the hex decodes to the ASCII string "broken chain of trust send help!", which is the fixture's inside joke, spotted by the model mid-diagnosis. That is evidence-following, not pattern-matching on the domain name.

The model that got disqualified

Here is the part we would want to read in anyone else's agent post. Our first model choice narrated its tool calls as text instead of calling them, and then did something worse: it invented probe results. "Let's say the MX lookup returned NoAnswer", it wrote, and proceeded to diagnose a hypothetical, complete with a made-up IP address, while the real answer (that null MX) sat unqueried.

For a diagnostic agent this is the cardinal sin. A wrong diagnosis from real evidence is a bug; a confident diagnosis from imagined evidence is a hazard. So the test that actually selected our model was not a benchmark, it was: give it a symptom and watch whether every record it cites exists in the probe log. The model that shipped (openai-gpt-oss-120b on DigitalOcean's platform) passed on every case; the platform's model menu meant switching candidates was a one-line env var (DETECTIVE_MODEL), which turned model selection into an experiment instead of a rewrite.

That is also the general lesson for agent builders: grounding tools only help if fabrication is treated as disqualifying, and you only catch it by auditing the trail. It is why the agent prints every probe as it runs.

Why the platform part was boring, complimentarily

The DigitalOcean side of this build is the part with nothing to debug, which is the compliment: an OpenAI-compatible endpoint (inference.do-ai.run/v1), standard function calling, one bearer key, and a menu of models from multiple providers behind the same API. The whole integration is a urllib request; no SDK, no framework. For agent experiments where the interesting decisions are the tools and the honesty constraints, a serverless per-token endpoint is exactly the right amount of infrastructure, and swapping models to run the fabrication test across candidates cost nothing but the tokens.

Where to take it

The repo is MIT and the pattern extends anywhere probes exist: an SMTP probe (connect to port 25, read the banner and the rejection message) would make the mail diagnosis end-to-end; a propagation probe (query several public resolvers and compare) would catch mid-migration states; and CI could run the detective against your own domains nightly, alerting when a diagnosis changes. If you build the SMTP one, our DNS record checkers cover the static half of that story already.

The bigger point stands on its own: the gap between "LLM that talks about infrastructure" and "agent that inspects infrastructure" is four small functions and one rule about evidence. The tools are the easy part. The rule is the product.

Originally published at devops-daily.com.

Getting a Row Change Out of Postgres Without Dual-Writing

DevOps Daily — Tue, 01 Sep 2026 12:00:00 +0000

Somewhere in your codebase there is probably a function that does two things: saves a row to Postgres, then publishes an event about it to Kafka, RabbitMQ, or a webhook. It works in the demo, it works for months, and then a deploy restarts the process between the two calls, and now your database says the order exists while your event stream says it never happened. Every downstream consumer is now wrong, and nothing corrects it until a human writes a reconciliation job.

That is the dual-write problem , and it is not a bug you fix with retries. It is an architecture problem: without a distributed transaction spanning both systems (possible via two-phase commit, practical almost never), code that writes to both will eventually disagree with itself. The fix is to stop writing twice: make the database the single place a change happens, and derive the event stream from the database's own record of changes. This post walks the two honest ways to do that, the failure mode the second one hides (with a live demonstration of it eating disk), and the tooling landscape around it.

TLDR

Dual writes fail because there is no transaction across Postgres and your broker. Some interleaving of crash and retry always produces disagreement.
Fix one: the transactional outbox. Write the event into an outbox table in the same transaction as the data; a relay publishes from that table. The transaction buys agreement; the relay still needs retries and monitoring.
Fix two: logical decoding , Postgres's built-in change stream. A replication slot plus a decoder turns every committed INSERT/UPDATE/DELETE into consumable messages; no application changes at all.
The catch: a replication slot pins WAL until decoding no longer needs it. In our live demo, an idle slot went from 1,488 bytes to 45 MB of retained WAL in under a minute, from traffic that had nothing to do with the tables it watched. Unmonitored, this fills the primary's disk.
Guard with a pg_replication_slots alert and max_slot_wal_keep_size; then choose between running Debezium yourself or paying one of the managed CDC vendors.

Prerequisites

Comfortable SQL and a rough idea of what the write-ahead log is (our WAL deep dive is the perfect warm-up; this post is its practical sequel)
A Postgres you can experiment on, with wal_level = logical (we ran everything below on a scratch project on Neon, where it is a project setting)
No Kafka required to follow along

Why dual-writing always loses

The failing pattern, in its natural habitat:

def create_order(order):
    db.execute("INSERT INTO orders ...") # write 1
    db.commit()
    kafka.produce("orders", order_event) # write 2, and the lie begins

Walk the interleavings. Crash after commit, before produce: database has the order, stream does not. Produce first instead? Crash after produce, before commit: stream announces an order that does not exist. Wrap it in retries: now a timeout that actually succeeded gets retried and the event publishes twice, or the retry queue dies with the pod. No ordering of two non-transactional writes survives every crash, because the two systems share no notion of "this happened".

INSERT order
COMMIT
publish event

Teams discover this the slow way: a reconciliation script somebody writes "temporarily" in year one that is load-bearing by year three. The permanent fixes both follow one principle: the database is the only writer, the stream is derived.

Fix one: the transactional outbox

The outbox pattern moves the second write inside the transaction:

BEGIN;
INSERT INTO orders (customer, total, status) VALUES ('ada', 42.50, 'pending');
INSERT INTO outbox (topic, payload)
  VALUES ('orders', '{"event": "order_created", "customer": "ada", "total": 42.50}');
COMMIT;

One transaction, so either both rows exist or neither does. A small relay process polls the outbox (or, foreshadowing, tails it via CDC), publishes each row to the broker, and marks it done. Consumers must tolerate duplicates, because the relay can crash between publishing and marking, but duplicates are a solvable problem (idempotency keys); disagreement is not.

The outbox is the right first tool: no exotic infrastructure, trivially auditable, and the event schema is explicit and versioned by you rather than mirroring your table structure. Be honest about what it buys, though: the transaction guarantees the outbox row matches the data, not that broker delivery is exactly-once. The relay still needs retries, ordering rules, cleanup, and monitoring, and the pattern only captures what your application chooses to record.

Fix two: the database's own change stream

Postgres already maintains a record of every committed row change to regular tables: the WAL. Logical decoding exposes it as a consumable stream: you create a replication slot , attach a decoder plugin, and Postgres hands you every committed change, in commit order, exactly where you left off.

This is the part worth seeing rather than reading about. Everything below is a real recorded session:

psql, wal_level = logical

neondb=> CREATE TABLE orders_cdc(id serial PRIMARY KEY, customer text, total numeric, status text);
CREATE TABLE
neondb=> SELECT slot_name, lsn FROM pg_create_logical_replication_slot('cdc_demo', 'test_decoding');
 cdc_demo | 0/2990C78
neondb=> INSERT INTO orders_cdc(customer, total, status) VALUES ('ada', 42.50, 'pending');
INSERT 0 1
neondb=> UPDATE orders_cdc SET status = 'shipped' WHERE customer = 'ada';
UPDATE 1
neondb=> DELETE FROM orders_cdc WHERE customer = 'ada';
DELETE 1
neondb=> SELECT lsn, data FROM pg_logical_slot_peek_changes('cdc_demo', NULL, NULL);
0/2990EC8 | BEGIN 4098
0/2990F68 | table public.orders_cdc: INSERT: id[integer]:1 customer[text]:'ada' total[numeric]:42.50 status[text]:'pending'
0/29910C8 | COMMIT 4098
0/29910C8 | BEGIN 4099
0/29910C8 | table public.orders_cdc: UPDATE: id[integer]:1 ... status[text]:'shipped'
0/2991160 | COMMIT 4099
0/2991160 | BEGIN 4100
0/2991160 | table public.orders_cdc: DELETE: id[integer]:1
0/29911D8 | COMMIT 4100

There it is: three ordinary SQL statements came back out as a structured, ordered, transaction-delimited change stream, without the application writing a single event.

The test_decoding plugin above is the built-in demo decoder; real pipelines use pgoutput (the protocol-native one) or wal2json. Same session with a wal2json slot, and the same insert becomes machine-readable (also real output):

wal2json: the same stream as JSON

neondb=> SELECT data FROM pg_logical_slot_peek_changes('json_demo', NULL, NULL, 'format-version', '2');
{"action":"B"}
{"action":"I","schema":"public","table":"orders_cdc","columns":[{"name":"id","type":"integer","value":1},{"name":"customer","type":"text","value":"grace"},...]}
{"action":"C"}

Two function families matter here: peek_changes reads without consuming (we used it above so the demos are re-runnable), while get_changes consumes, advancing the slot's acknowledged position, which is what a real consumer does on every poll. One honest subtlety we hit while testing: after consuming, restart_lsn (and so the retained-WAL number) does not drop instantly; Postgres advances it lazily once decoding no longer needs the older segments. Do not panic-tune based on a retention figure measured seconds after a catch-up. If you read our WAL post, those LSNs are old friends: the stream's cursor is just a position in the log.

One more piece the stream does not give you: the past. A slot starts at creation time, so a new consumer needs the initial snapshot problem solved: copy the existing table contents first, then apply changes from the stream without a gap. Postgres supports this handoff properly (a slot creation can export a consistent snapshot to read the baseline from), and it is precisely the fiddly part that Debezium and the managed vendors have production-hardened; if you hand-roll a consumer, this is where the subtle bugs live.

CDC's superpower over the outbox is completeness: every committed change to the captured tables, including the UPDATE someone runs by hand during an incident. The fine print: DDL and sequences are not part of the stream, UPDATE/DELETE detail depends on the table's REPLICA IDENTITY, a crash can redeliver recent changes (consumers still deduplicate), and you inherit the table schema as your event schema. Plus one sharp operational edge.

The slot that eats your primary's disk

A replication slot is a promise: Postgres keeps every WAL segment from the slot's restart_lsn forward, the point decoding would need to resume, so a slow consumer can always catch up. (That can trail the consumer's acknowledged position when long transactions are open, which is why an actively streaming slot can still pin WAL.) Read it as an ops engineer: a slot that is not advancing forbids WAL cleanup, no matter whose WAL it is.

Watch it happen. Same session, same idle cdc_demo slot, and the traffic we generate touches a completely different table (a slot is database-scoped; even consumers that filter to a publication still cause all WAL to be retained until they advance):

the retained-WAL trap, live

neondb=> SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained FROM pg_replication_slots WHERE slot_name = 'cdc_demo';
 cdc_demo | f | 1488 bytes
# 20,000 rows into a completely unrelated table
neondb=> INSERT INTO bulk_junk(payload) SELECT repeat('x', 1000) FROM generate_series(1, 20000);
INSERT 0 20000
neondb=> SELECT ... retained ...;
 cdc_demo | f | 23 MB
neondb=> UPDATE bulk_junk SET payload = repeat('y', 1000);
UPDATE 20000
neondb=> SELECT ... retained ...;
 cdc_demo | f | 45 MB

From 1,488 bytes to 45 MB of pinned WAL in under a minute, on a toy workload, from unrelated traffic. Now scale that to a production write rate and a CDC consumer that crashed on Friday evening: the primary's disk fills at your full WAL generation rate all weekend, and the incident that pages you says "database out of disk", nowhere near the actual culprit. This exact anatomy, a stalled consumer plus an unmonitored slot, is one of the classic self-inflicted Postgres outages.

Two guards, both cheap:

-- Alert on this. An inactive slot with growing retention is a countdown.
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;

-- Postgres 13+: cap how much WAL slots may pin (enforced at checkpoints,
-- so treat it as a strong limit, not an exact one). A slot that exceeds it
-- is invalidated instead of the primary dying; the consumer typically
-- re-snapshots, which is a bad day but not an outage.
ALTER SYSTEM SET max_slot_wal_keep_size = '10GB';
SELECT pg_reload_conf();

When diagnosing, look past active: an active-but-lagging consumer pins WAL too. wal_status and safe_wal_size in pg_replication_slots tell you how close to the cliff each slot is, and Postgres 18 adds idle_replication_slot_timeout for automatic cleanup of abandoned slots.

And the operational rule behind both: a replication slot is a consumer contract, not a fire-and-forget resource. Create it when the consumer exists, monitor it like a queue, drop it when the consumer is decommissioned. (We dropped ours right after the recording; the demo project thanks us.)

The landscape: run it or rent it

The protocol layer is standard Postgres, so the build-vs-buy question is about the pipeline around it: snapshotting existing data, schema change handling, delivery into your broker or warehouse, and babysitting the slots.

Run it yourself: Debezium is the open source standard: usually a Kafka Connect connector, though Debezium Server delivers to non-Kafka sinks too. It handles initial snapshots and the common schema-change cases, and has seen every edge case in production somewhere. The cost is operating that machinery, and the slot monitoring above becomes your pager's problem.

Rent the pipeline (examples, not a census; the build/rent line blurs since several offer self-hosted versions): Estuary does real-time CDC into warehouses and streams with a managed backfill story; Sequin is Postgres-native CDC aimed at developers who want changes as HTTP/streams without Kafka at all; Artie focuses on low-latency Postgres-to-warehouse replication; Striim sells the enterprise end with decades of database-replication lineage; and Airbyte wraps Debezium for the batch-leaning integration crowd. They differentiate on destinations, latency, and how much of the slot babysitting they absorb; all of them exist because that babysitting is real work. (Confluent's managed connectors and the clouds' native CDC services compete here too.)

The honest decision guide: if the events feed one warehouse nightly, a plain updated_at polling job is still legitimate and nobody should shame you for it. If your application needs to emit domain events it controls, start with the outbox. If you need every change, or changes from tables your code does not own, that is CDC, and the choice between Debezium and a managed pipeline is the choice of who wakes up for the slot alert.

What to do with this

Find your dual writes. Grep for commit-then-publish patterns; each one is a consistency bug with an unknown detonation date.
Adopt the outbox for domain events. Same transaction or it did not happen.
If you deploy CDC, deploy the slot monitor the same day. The pg_replication_slots query above, alerted at a threshold well below your disk headroom, plus max_slot_wal_keep_size as the backstop.
Treat slots as consumer contracts with a lifecycle, an owner, and a decommissioning step.
And if Kafka entered the chat while you read this: our guide to when you actually need it pairs well here, because "transport for CDC events" is one of the six cases where it genuinely earns its keep.

Originally published at devops-daily.com.

Omarchy 4 Makes the Linux Desktop Feel Like a Product, Finally

DevOps Daily — Tue, 01 Sep 2026 09:00:00 +0000

"The year of the Linux desktop" has been a punchline for two decades, and the punchline always had the same explanation: nobody with product taste and staying power ever owned the whole experience. Distros assembled parts; nobody curated them. That is exactly the gap Omarchy was built to fill, and with August's 4.0 release, "Quattro", it is getting hard to keep laughing at the old joke.

Omarchy is David Heinemeier Hansson's opinionated, Arch-based Linux for developers: Hyprland tiling, one keyboard-driven workflow, every default chosen on purpose. What started in 2025 as one famous developer ricing his laptop in public has turned into something with real institutional weight, and Quattro (shipped August 14) is the release where that shows.

TLDR

Quattro rewrote the entire desktop shell in Quickshell : bar, launcher, menus, notifications, lock screen, one coherent, themed, scriptable process instead of a federation of independent tools, running under 300 MB.
The ISO dropped under 6 GB (more than a gigabyte smaller) and installs got 30%+ faster; sub-minute installs are possible on fast hardware. Dual boot with Windows (with full LUKS encryption) finally landed.
Coding agents are system citizens : nine pre-wired (Claude Code, Codex, Gemini CLI, Copilot CLI and more), a system-wide default you pick once, agent status in the top bar, and crash diagnosis that routes to your agent.
The Omacom Foundation launched with $8M from eight patrons including Tobi Lütke, Patrick Collison, Michael Dell, Jack Dorsey and Matthew Prince, since grown past $10M. Hardware vendors are engaging, with Framework support among the reported wins.
The same simplicity philosophy extends naturally to the server side, which is where the rest of your stack gets to stay boring too.

Prerequisites

None to read this. To try Omarchy: a spare machine or partition, comfort with the idea of a tiling window manager, and about a minute of installation, apparently.

The shell rewrite is the headline

Pre-4.0 Omarchy was, under the hood, what every polished Linux setup is: a carefully configured federation. Waybar here, a launcher there, a notification daemon, each themed into agreement but still separate programs around the Hyprland compositor.

Quattro replaces the federation with a single long-running shell built on Quickshell (a Qt Quick toolkit for building desktop components): bar, launcher, menus, notifications, on-screen displays, control panels, lock screen and polkit agent in one coherent, IPC-scriptable process with a plugin architecture, running in under 300 MB.

If you have ever maintained a hand-rolled tiling setup, you know why this matters. The federation approach means every theme change touches five config formats and every component upgrade can break the seams. One process, one theme system (expanded from 8 to 24 palette colors in this release), one scripting surface: this is the difference between a collection of dotfiles and an actual product. It is also, notably, the kind of consolidation only a project with a single opinionated owner ships, because every component it replaced has its own community that would have voted no.

Installs measured in seconds, and dual boot at last

The whole install story got the product treatment too: the ISO shrank by over a gigabyte to under 6 GB, installation sped up more than 30%, and on fast hardware a full install lands in under a minute. For a distro whose pitch includes "reinstalling is cheap, your config is code", making the install nearly free is not vanity, it is the philosophy made concrete.

Quattro also added the feature whose absence kept many people at the door: dual boot. A free-space install alongside Windows, with full LUKS disk encryption, so trying Omarchy no longer means sacrificing a machine to it. (You shrink the Windows partition and disable BitLocker first; the full-disk path still wipes the drive it is pointed at.) For the "I would try it but I need my Windows partition" crowd, the excuse is gone.

Agents as system citizens

Here is the part most relevant to how development actually changed in the last two years. Every OS treats coding agents as apps you happen to run in a terminal. Omarchy 4 treats them as part of the system: nine agents pre-wired as lazy-loaded launchers (Claude Code, OpenAI Codex, OpenCode, Gemini CLI, GitHub Copilot CLI, Crush, Grok CLI, Pi, Oh My Pi), a system-wide default you set once (omarchy default agent claude), and then the OS routes agent-shaped work accordingly.

The details are where it gets genuinely clever: agent state lives in the top bar (including plan limits and token burn), a multiplexer tracks whether agents are idle, working, blocked or done, and when something on the system crashes, Omarchy can hand the diagnosis to your default agent, with a built-in skill that knows how to reconfigure the OS itself. That last one is quietly a big idea: the operating system shipping first-party context for the AI that maintains it.

Agree or not with every choice, this is the first OS-level answer to a question every developer now has: where do agents live in my environment? Everyone else is leaving it to terminal multiplexers and muscle memory.

Money, governance, and hardware taking it seriously

The reason to take Omarchy seriously as more than a famous developer's dotfiles is what happened around the software in August. DHH launched the Omacom Foundation with $8 million from eight founding patrons, and the list reads like a who's-who with skin in the developer-tools game: Tobi Lütke (Shopify), Patrick Collison (Stripe), Michael Dell, Jack Dorsey, Matthew Prince (Cloudflare), Brendan Iribe, Jason Fried, and DHH himself, with funding since passing $10 million as more patrons joined. The foundation holds the trademarks, funds infrastructure, and, importantly, supports the upstream open-source projects Omarchy depends on, Hyprland and Quickshell included.

Hardware is responding too: Framework has been reported as officially supporting Omarchy, and work has surfaced on tuning for current Dell machines. A Linux desktop with a taste dictator, a war chest, upstream funding, and OEM attention is a combination the ecosystem has simply never had before.

Where the servers fit

One more observation, because this is a DevOps site: Omarchy's appeal is a philosophy, not just a theme pack. Fewer moving parts, defaults chosen by someone with taste, tools you can hold in your head. Developers who feel that pull on their workstation tend to want the same thing one layer up, which is why this crowd so often pairs a setup like Omarchy with deliberately simple infrastructure: a few droplets on DigitalOcean, Docker Compose, boring DNS, rather than a hyperscaler console with four hundred services. (It is the same instinct we leaned on when we self-hosted a PaaS on DigitalOcean with Coolify: own your tools, keep the stack legible.) DHH's crusade against accidental complexity does not stop at the desktop, and neither should yours.

Should you try it?

If you live in a terminal, like keyboard-driven everything, and have wanted a Linux desktop that feels decided rather than assembled: yes, and Quattro is the right moment, because dual boot removed the commitment problem and the sub-minute install removed the time problem. If you need mainstream desktop conventions or hate tiling, it is deliberately not for you, and Omarchy would be the first to say so; opinionated software earns its coherence by not negotiating.

Either way, it is worth watching. The Linux desktop's chronic problem was never capability, it was curation, and for the first time in a long time someone with taste, money, and an audience is doing the curating in public, shipping monthly, and dragging hardware vendors along. The old joke needed retiring anyway.

Originally published at devops-daily.com.

5 DevOps Newsletters Actually Worth Your Inbox

DevOps Daily — Mon, 31 Aug 2026 10:01:00 +0000

The problem with staying current in DevOps is not finding information, it is that the firehose never stops. Vendor blogs push product news dressed as insight, aggregators recycle the same ten links, and the genuinely important stuff (a deprecation that breaks your cluster, a postmortem worth studying) arrives mixed into the noise.

A small number of newsletters have solved this by having an actual human with actual taste do the filtering. Here are the five worth your inbox, starting with ours, because we are hardly going to put it last.

1. DevOps Daily

devops-daily.com/newsletters · weekly, Mondays

One Monday email with everything that happened week. That means hands-on posts (fact-checked release coverage like our Kubernetes 1.37 deep dive, scale stories, build logs with real recorded output), plus interactive simulators, quizzes and guides, so most issues include at least one thing you can practice in the browser rather than just read. The whole archive is public, so you can judge every past issue before handing over an email address, which is how every newsletter on this list deserves to be evaluated.

Best for: people who learn by doing, not just reading.

2. DevOps Weekly

devopsweekly.com · weekly, Sundays

Gareth Rushgrove has been curating this since 2010, which makes it one of the longest continuously running newsletters in the space, and the longevity shows in the selection. No commentary, no fluff, just eight to twelve links per issue spanning tools, practices and write-ups, chosen by someone who has watched every hype cycle come and go.

Best for: broad, trend-resistant coverage of the whole field.

3. SRE Weekly

sreweekly.com · weekly, 530+ issues

Lex Neva's roundup leans into reliability: incident postmortems, outage analyses, on-call culture, resilience engineering. The standout feature is the outage section at the end of every issue, a running log of who went down this week, which over months becomes a quiet education in how systems actually fail. If postmortems are your favorite genre (they are ours: the postmortem nobody reads), this is that interest as a weekly habit.

Best for: incident-minded engineers and anyone with a pager.

4. KubeWeekly

cncf.io/kubeweekly · weekly

The CNCF's official Kubernetes newsletter: upcoming releases, KEPs moving stages, ecosystem project updates, and community content. It is the reliable way to hear that a deprecation is coming for you before it lands in release notes you were not reading.

Best for: anyone operating Kubernetes in anger.

5. Last Week in AWS

lastweekinaws.com · weekly

Corey Quinn reads all of AWS's announcements so you do not have to, and then says what he actually thinks about them, which no vendor newsletter will ever do. The snark is the delivery mechanism; the payload is genuinely sharp analysis of pricing, product strategy and which announcements matter. Even if you are not on AWS, it is the best example of what opinionated curation looks like.

Best for: AWS users, FinOps people, and anyone who enjoys their cloud news with a spine.

Bonus: TLDR DevOps, for the daily skimmers

tldr.tech/devops · daily

If your reading time is five minutes over coffee, a short daily digest with two-sentence summaries beats a weekly deep read you never get to. It pairs well with one or two of the weeklies above rather than replacing them.

How to actually use newsletters

Advice earned by unsubscribing dozens of times: pick at most one daily and two weeklies, or the backlog guilt kills the habit. Read the outage and postmortem sections first, they compound fastest. And re-evaluate quarterly: a newsletter that no longer earns its open is a newsletter to leave, including any of the ones above, including ours.

How Discord Stores Trillions of Messages With a Tiny Team

DevOps Daily — Sat, 29 Aug 2026 16:00:00 +0000

Some engineering stories are worth studying because the numbers are absurd, and some because the lessons transfer. Discord's message-storage story, told across their own engineering posts (2017, 2023), is both: trillions of stored messages, migrated live in nine days, by a team small enough to fit around one table. All numbers below come from those two posts.

The arc in one paragraph: in 2017 Discord ran 12 Cassandra nodes storing billions of messages. By early 2022 that had grown to 177 nodes storing trillions, and the cluster was hurting in ways that paged humans. In 2022 they moved everything to ScyllaDB, ending at 72 nodes of 9TB each, with p99 read latency dropping from a wandering 40-125ms to a steady 15ms. Fewer nodes, more data, an order of magnitude calmer tail.

The migration headline is fun, but the durable lessons live in the data model, the three problems that forced the migration, and the thing they built that was not a database at all.

First, the data model that carried them

The 2017 chapter starts where most scaling stories do: the original database hit a wall. Discord launched on a single MongoDB replica set, and by November 2015, at 100 million messages, the data and indexes no longer fit in RAM and latency went unpredictable. Their traffic made it worse than it sounds: reads and writes were roughly 50/50, and reads were highly random, which is the workload page caches hate most.

The move to Cassandra came with the design decision the whole story rests on. Messages are identified by Snowflake IDs (Twitter's chronologically sortable 64-bit IDs), so the natural key was (channel_id, message_id): all of a channel's messages in one partition, sorted by time for free. Then the import taught them the classic wide-partition lesson: big channels blew past 100MB per partition, and giant partitions meant GC pressure and compaction pain. Cassandra advertises support for 2GB partitions; Discord's write-up delivers one of the great one-liners of database operations: just because it can be done does not mean it should.

The fix was time bucketing. They measured their largest channels and found that 10 days of messages stayed comfortably under 100MB, so the key became ((channel_id, bucket), message_id), where the bucket is derived from the timestamp. Partition size is now bounded no matter how big a channel gets, and quiet channels just query a few sequential buckets.

That key design is the most reusable artifact in the whole saga. "Partition by tenant" is where everyone starts; "partition by tenant plus a bounded time window" is where high-write systems end up, and getting there before the import, rather than six months into production, is the cheap version.

Problem 1: the hot partition

Discord partitions messages by channel (plus a time bucket), which distributes load beautifully as long as channels are similarly busy. They are not. A three-friend server generates orders of magnitude less traffic than a two-hundred-thousand-person community, and when something happens in a huge channel, a flood of concurrent reads lands on the one partition that holds it.

That is a hot partition , and its signature is the nasty part: the node serving the hot partition slows down, queues back up, and every other partition on that node gets slow too. Latency spreads sideways to users who have nothing to do with the busy channel. The failure is invisible in averages, obvious in the tail, and it is the same mechanism whether you run 177 nodes or a single Postgres with one viral customer row. (Our latency percentiles simulator shows exactly this signature: a healthy median over a growing tail.)

Problem 2: the garbage collector

Discord's Cassandra cluster ran on the JVM, and the JVM stops the world to collect garbage. At their read/write volume, GC pauses produced latency spikes big enough to page people, and in bad cases nodes needed manual reboots to recover.

The general lesson is not "avoid Java". It is that at the tail, your database's runtime is part of your latency budget. p99 problems that correlate with nothing in your query patterns often live a layer down: GC, compaction, page cache pressure. ScyllaDB being a C++ rewrite of Cassandra with no GC was a major reason it was the destination; the shape of their p99 graph before and after says the diagnosis was right:

Message read latency, p99

	Value	Series
Cassandra, worst observed p99	125ms	Cassandra
Cassandra, best observed p99	40ms	Cassandra
ScyllaDB p99	15ms	ScyllaDB

Numbers from Discord's 2023 engineering post: p99 reads went from a 40-125ms range on Cassandra to a steady 15ms on ScyllaDB. Inserts went from 5-70ms to a stable 5ms.

Problem 3: maintenance that becomes a lifestyle

The third pain was compaction falling behind. Cassandra compacts SSTables in the background, and once a cluster falls behind under load, operators start doing what Discord called a gossip dance: pull a node out of rotation so it can compact in peace, bring it back, let it catch up on hints, repeat, node after node.

Every ops team knows some version of this: a routine background process that quietly becomes a manual, rotating chore. The lesson is diagnostic: when babysitting a system becomes a recurring calendar event, the system is telling you its design no longer fits your load. Discord's answer was not better runbooks; it was removing the reason the runbook existed.

The tombstone wars

Deletes deserve their own chapter, because in log-structured databases a delete is not a removal, it is a tombstone : a marker written on top, reconciled at read time, cleaned up later by compaction. Discord ran into both of the classic tombstone disasters, five years apart.

The first was self-inflicted and invisible: their writer sent null values for unset columns, and Cassandra treats a null write as a delete. Result: about 12 tombstones written per average message , pure overhead, fixed by simply not writing nulls. The generalizable habit is knowing what your driver actually emits, because ORMs and serializers make this class of mistake silently.

The second is the famous one. Six months after launch, a node started running ten-second stop-the-world GC pauses. The cause was one channel, a Puzzles & Dragons subreddit server, that had deleted its way down to one visible message sitting on top of millions of tombstones. Every load of that channel made Cassandra wade through the graveyard to find the survivor. The mitigation: cut tombstone lifetime from 10 days to 2 (with nightly repairs to make that safe) and track empty buckets so queries skip them entirely.

Tombstones also close the loop on the 2022 migration: the final blocker before the ScyllaDB migrator could finish was compacting gigantic tombstone ranges in Cassandra. The deletes of 2017 were still shaping operations five years later, which is the most honest definition of technical debt you will find.

The part everyone skips: the layer in front

Here is the piece that transfers to every stack, at every scale. Before migrating anything, Discord built data services : a Rust layer that sits between the API and the database, whose star feature is request coalescing. When a thousand users request the same message row at once (exactly what a hot channel produces), the service makes one database query and fans the result out to all thousand waiters. Consistent hash routing by channel ID sends all traffic for a channel to the same service instance, so coalescing actually catches the duplicates.

API clients 1,000 identical reads
Data service Rust, coalesces to 1 query
ScyllaDB sees 1 read, not 1,000
Fan-out one result, 1,000 answers

Notice what this means: the hot-partition problem was partially solved before the database changed , by making the database see less of the load. That ordering is the real architecture lesson. The database swap fixed GC and compaction; the protective layer fixed the traffic shape. Teams reach for a migration first because it feels decisive, but the layer in front is cheaper, lower-risk, and usually where the win is. At normal scale this same idea is a cache with request deduplication, or a materialized read path; the principle is identical.

Their storage hardware story rhymes with this: cloud persistent disks had the durability but not the latency, so they built "super-disks": local NVMe for speed, RAID-mirrored to persistent disks for durability. Same pattern again: keep the slow-but-safe thing, put a fast layer in front of it.

Coalescing is small enough to build yourself

The idea sounds exotic at Discord's scale and is almost embarrassingly small in code. Here is the whole mechanism, runnable as-is: keep a map of in-flight requests per key, and make duplicate callers await the existing one.

import asyncio

db_queries = 0

async def db_read(key):
    global db_queries
    db_queries += 1
    await asyncio.sleep(0.05) # one slow database read
    return f"row:{key}"

class Coalescer:
    def __init__ (self):
        self.inflight = {}

    async def read(self, key):
        if key in self.inflight: # someone already asked: wait for theirs
            return await asyncio.shield(self.inflight[key])
        task = asyncio.create_task(db_read(key))
        self.inflight[key] = task
        try:
            return await task
        finally:
            del self.inflight[key]

Fire a hot-channel burst at it, with and without coalescing (this is a real run, not sketched output):

request coalescing

$ python3 coalesce.py
# 1,000 concurrent reads of the same key, through the coalescer
clients served: 1000, database queries: 1
# same 1,000 reads, no coalescing
clients served: 1000, database queries: 1000

One thousand callers, one database query. In Go this is singleflight from the standard extended library; in most stacks it is twenty lines. If your system has any hot-key read pattern, this is among the highest ratio of latency saved to code written that exists.

The migration itself

The plan was to migrate with ScyllaDB's Spark-based migrator, estimated at three months. They did not want to babysit a migration for a quarter, so they rewrote the migrator in Rust, and the estimate fell to nine days , running at up to 3.2 million messages per second, with the last obstacle being enormous tombstone ranges in Cassandra that needed compacting before they would move.

Two things worth keeping from that: first, migration tooling is code, and investing engineer-weeks in it can buy back engineer-months of supervised risk. Second, the messages moved while Discord kept running; the era where a migration of this size implied a maintenance window is simply over, and your users' expectations know it.

Worth stealing from the 2017 playbook too: before Cassandra went primary, they ran a dark launch , double-writing to MongoDB and Cassandra while reads still came from the old system. It surfaced a genuinely subtle bug before users could: concurrent edits and deletes, racing under Cassandra's last-write-wins conflict resolution, could resurrect corpses of deleted messages as corrupted rows with only a primary key and text. The fix (delete any message missing required columns like the author) is less important than the pattern: double-write early, read-compare quietly, and let the race conditions introduce themselves while the blast radius is zero.

What this means if you are not Discord

Find your hot partitions before they find you. Whatever your store, some key is orders of magnitude hotter than the median. Know which, and know what happens to neighbors when it spikes.
Chase tail latency into the runtime. If p99 spikes do not correlate with queries, look at GC, compaction, and background maintenance. The database's internals are part of your SLO.
Build the protective layer before the migration. Coalescing, caching, and read-path shaping change what the database experiences, at a fraction of a migration's risk.
Treat recurring manual maintenance as a design signal , not an ops failure.
If a migration is unavoidable, make the tooling fast enough to be boring. Nine supervised days beat ninety.

The deeper pattern in this story is that storage-engine design decides operational reality: Discord's pain (GC, compaction, tombstones) and Discord's wins (coalescing, super-disks) all live below the query layer. If that angle interests you, we recently went deep on another example of it: how Lakebase Postgres, the storage architecture you get on Neon, makes the WAL itself the database, where the same kind of architectural bet makes branching and point-in-time restore nearly free instead of heroic.

Discord's own posts are worth reading in full: 2017's billions for the data-model thinking, and 2023's trillions for everything above. For the hands-on version of the concepts, our message queue and database scaling simulators let you cause lag, hot spots and rebalances on purpose, which is considerably cheaper than learning them at a trillion messages.

Originally published at devops-daily.com.

Kubernetes 1.37 Really Can Flag Unused PVCs, but the Viral YAML Is Wrong

DevOps Daily — Fri, 28 Aug 2026 21:00:00 +0000

There is a post going around about Kubernetes 1.37 solving one of the quieter FinOps headaches: orphaned PersistentVolumeClaims. It comes with a YAML snippet showing a new field, status.unusedSince, with a big red arrow pointing at it.

The good news: the feature is real, it went beta in 1.37, and if you pay a cloud bill it is worth knowing about. The problem: the field in that screenshot does not exist. The actual API is a condition , not a timestamp field, and if you go looking for unusedSince in your cluster you will find nothing and conclude the feature is missing. We checked the enhancement against KEP-5541 itself, the same way we checked the 1.37 release claims when third-party roundups disagreed. Here is what actually shipped and how to use it.

TLDR

The problem is real: deleting a StatefulSet or Helm release keeps its PVCs by design, and nobody remembers whose they are six months later.
KEP-5541 "Report Last Used Time on a PVC" : alpha in 1.36, beta and enabled by default in 1.37 , behind the PersistentVolumeClaimUnusedSinceTime feature gate.
The API is a new Unused condition in status.conditions, managed by the PVC protection controller. There is no status.unusedSince field.
The "unused since" timestamp is the condition's lastTransitionTime.
A PVC with no Unused condition at all is normal right after upgrade: the condition appears as usage transitions are observed.
Unused does not mean deletable. It means no non-terminal pod references the claim.

Prerequisites

A cluster on Kubernetes 1.37 (or 1.36 with the alpha gate enabled)
kubectl and, for the queries below, jq
Basic familiarity with PVCs and StatefulSets

The problem this solves

Kubernetes keeps PVCs around on purpose. Delete a StatefulSet and its claims stay, because the alternative, data vanishing with a workload object, is worse. The cost of that safety is drift: six months later the monitoring namespace has a 100Gi claim named after a Prometheus that no longer exists, nobody is sure whether anything still mounts it, and the cloud provider bills for it monthly either way.

Until now, answering "is anything using this PVC?" meant correlating pods to claims yourself, and answering "since when?" meant an audit trail most clusters do not have. That second question is the one 1.37 finally answers natively.

What actually shipped

KEP-5541 adds a condition type Unused to PersistentVolumeClaim status, maintained by the PVC protection controller in kube-controller-manager:

When the last non-terminal pod referencing a PVC goes away, the condition becomes status: "True" with reason NoPodsUsingPVC.
When a pod starts referencing it again, the condition flips to status: "False" with reason PodUsingPVC.
The condition's lastTransitionTime records when that flip happened, which is exactly the "unused since" timestamp the viral post promised, living where Kubernetes actually puts such things.

So the real YAML looks like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-db-data
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
status:
  phase: Bound
  conditions:
    - type: Unused
      status: "True"
      reason: NoPodsUsingPVC
      message: No pods are currently referencing this PVC
      lastTransitionTime: "2026-08-01T10:00:00Z"

Same information as the screenshot, different shape: a condition you select on, not a scalar field you read. The distinction matters because every query, controller, or policy you write against this feature addresses status.conditions[], and anything written against status.unusedSince silently matches nothing.

(If you are wondering how a feature gate named PersistentVolumeClaimUnusedSinceTime produces a condition rather than an unusedSince field: gate names stick early and describe intent, not final API shape. It is a fair guess at where the confusion started.)

The query you actually came for

"Flag PVCs unused for more than 30 days" as a working pipeline:

kubectl get pvc --all-namespaces -o json | jq -r \
  --arg cutoff "$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)" '
  .items[]
  | . as $pvc
  | (.status.conditions // [])[]
  | select(.type == "Unused" and .status == "True" and .lastTransitionTime < $cutoff)
  | [$pvc.metadata.namespace, $pvc.metadata.name, .lastTransitionTime,
     $pvc.spec.resources.requests.storage]
  | @tsv'

Output is one line per stale claim: namespace, name, unused-since, size.

monitoring prometheus-db-data 2026-08-01T10:00:00Z 100Gi

Put that in a weekly CronJob that posts to Slack and you have the "automated cleanup visibility" the viral post promised, in about eight lines. The ISO-8601 timestamps compare correctly as strings, which is what makes the < in jq honest.

The caveats that keep this from biting you

No condition is not a bug. Right after upgrading, PVCs carry no Unused condition at all. The controller adds it as usage transitions are observed, so a claim that has not had a pod come or go since the feature turned on simply has nothing to report yet. Your tooling needs a three-state model: unused, in use, and not-yet-observed, which is why the query above selects explicitly instead of assuming.

Unused means unreferenced, not deletable. The condition says no non-terminal pod references the claim. A monthly reporting job's PVC is "unused" for 29 days at a time. A claim kept as a manual backup is "unused" forever and load-bearing. This feature gives you a review list, not a deletion list; the human step is the point.

The controller can lag. Conditions are reconciled from a queue, so the transition timestamp can trail the actual pod event slightly. For a 30-day threshold this is irrelevant; for a 30-minute one it is not the right tool.

Disabling the gate freezes the conditions. Turn the feature off and existing Unused conditions stay in etcd, stale. If you experiment with the gate, remember that a frozen condition looks exactly like a live one.

About that "CSI Volume Health" line

The same viral post credits 1.37 with "new CSI Volume Health APIs". Volume health monitoring is real but it is not a 1.37 headline: it is KEP-1432, which has been developing across releases for years, with related work continuing in newer storage KEPs. Combining a genuinely-new-in-1.37 feature with a years-old one under one "1.37 fixes storage" banner is how release folklore starts, and release folklore is how upgrade plans go wrong.

Which is the general lesson we keep re-learning this release cycle: for any "Kubernetes now does X" claim, thirty seconds with the KEP's own kep.yaml in kubernetes/enhancements tells you the real stage, the real milestone, and the real API. The features are usually good news. The screenshots are usually approximate.

What to do with this

On 1.37, nothing to enable : the gate is on by default at beta. Give the controller time to observe transitions before expecting conditions everywhere.
Wire the query into a schedule and route it to wherever your team reviews costs. Sort by size; the top of that list is usually a few claims worth most of the money.
Review, then delete deliberately : check snapshots, check whether a seasonal workload owns the claim, then remove claim and (depending on your reclaim policy) the underlying volume.
Do not build against unusedSince : it does not exist. Conditions do.

Originally published at devops-daily.com.