PostgreSQL Backup to S3 on OpenShift HCP, with Zero Static Credentials

Automating logical database backups to S3 from a managed-control-plane cluster — the credential mechanics decoded down to the JWT and the STS call, and the two things everyone gets wrong: they never test the restore, and they leave a long-lived AWS key sitting in a Secret.

The backup that wasn’t

Every team I’ve seen get burned by backups got burned the same way. There was a CronJob. It ran. It uploaded something to a bucket every night, and the little green checkmark said “succeeded” for months. Then the day came when someone actually needed to restore — a bad migration, a dropped table, a corrupted volume — and the archive turned out to be empty, or truncated, or of a schema three versions stale, or written by an IAM key that had since been rotated out from under it. The backup existed. The restore had never once happened. And a backup you have never restored is not a backup; it’s a rumor that data is safe.

This is a write-up of building the version that isn’t a rumor: automated PostgreSQL backups to Amazon S3, running on an OpenShift Hosted Control Plane (HCP) cluster, with the restore path exercised as a first-class part of the exercise rather than an afterthought. Because it runs on AWS there’s a second trap to avoid — the one where you make the whole thing work by dropping a permanent access key into a Kubernetes Secret and calling it done. We’ll do it with zero static credentials, and I’m going to decode that mechanism properly — the projected token, the trust policy, the STS exchange — because it’s the same machinery you’ll reach for on every AWS integration from OpenShift for the rest of your career, and hand-waving it is how people end up with the Secret full of AKIA anyway.

The cluster is disposable, as always in this series. Fresh cluster, build the pipeline, prove it end to end by destroying and recovering real data, tear it down. Nothing survives between runs, which is exactly why the whole thing has to be reproducible from Terraform and manifests alone.

What a managed control plane backs up for you — and what it very much doesn’t

On HCP, the control plane runs in Red Hat’s management cluster. That includes etcd, the datastore behind the Kubernetes API, and Red Hat owns its backup, restore, and disaster recovery. This is genuinely one of the best parts of a managed control plane: the single most nerve-wracking restore in all of Kubernetes operations, the etcd restore, stops being your 3 a.m. problem.

It is very easy to let that fact quietly expand into “backups are handled.” They are not. Here is the line, drawn sharply:

Data Who backs it up on HCP etcd / Kubernetes API state Red Hat Control-plane configuration Red Hat Your application’s PVC data You Your PostgreSQL database You Objects your app writes to S3 You

etcd holds the shape of your cluster — Deployments, Services, ConfigMaps. It does not hold the rows in your database. When your application’s PostgreSQL loses a table, etcd is serenely, completely healthy and utterly useless to you. The managed control plane covers the platform; the data your business actually runs on is yours, and on HCP the boundary is drawn for you — right through the middle of the word “backups.”

Before writing a line of YAML, it’s worth naming the two numbers every backup design is really about:

• RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time. For a scheduled logical dump, your RPO is your backup interval: a */10 cron means up to ten minutes of writes can be lost. Want a tighter RPO than your dump interval? You're no longer in pg_dump territory — you're into continuous WAL archiving and point-in-time recovery, which is a different, heavier lab.
• RTO (Recovery Time Objective) — how long a restore takes. For logical restore this scales with database size and index rebuild time, and it’s precisely the number that stays theoretical until you actually run a restore and time it.

The credential question, decoded

This is the part that trips people up, and it deserves to be taken apart properly, because getting it right is the difference between a secure integration and a liability you’ll rediscover during an incident.

The objection everyone raises is a good one: to write to S3 you need AWS credentials to sign the request — so how can the pod write to S3 without a key?

It doesn’t write without a key. It writes without a static, stored key. Those are different things, and conflating them is the root of the confusion.

Every AWS request is SigV4-signed with credentials. There are two families:

Static (IAM user) Temporary (STS) Access key id AKIA… ASIA… Session token none present (X-Amz-Security-Token) Lifetime until someone revokes it ~1 hour, auto-expires Where it lives a Secret / file / env var — you store and guard it process memory — never stored

The naive approach uses the left column: mint an IAM user, drop its AKIA key into a Secret, mount it into the backup pod. It works, and it's a time bomb. That key is permanent, copyable, and it leaks into logs, git history, and env dumps; a stolen AKIA key stays valid until a human notices and revokes it — historically, "much too late." Worse, it's ambient: anything that can read the Secret is, to AWS, indistinguishable from your backup job.

What we do instead lives in the right column, and the pod earns those temporary credentials by trading a signed proof of its own identity. Let me decode that trade layer by layer, because each layer is a place people get it subtly wrong.

Layer 1 — the projected token is a real, signed JWT

Kubernetes projects a bound service account token into the pod via the TokenRequest API. "Bound" means it's tied to this pod's lifetime, scoped to a specific audience, time-limited, and rotated by the kubelet before it expires — nothing like the old, permanent, static SA tokens. It's a standard JWT, signed by the cluster's OIDC issuer. Decode the one our backup pod carries and the payload looks like this:

{
  "aud": ["openshift"],
  "exp": 1757160000,
  "iat": 1757156400,
  "iss": "https://rh-oidc.s3.us-east-1.amazonaws.com/2a3b4c5d6e7f...",
  "kubernetes.io": {
    "namespace": "pgbackup",
    "pod":            { "name": "pg-backup-manual-abc", "uid": "…" },
    "serviceaccount": { "name": "s3-backup",            "uid": "…" }
  },
  "sub": "system:serviceaccount:pgbackup:s3-backup"
}

Three claims do all the work. iss is the issuer AWS will decide whether to trust. aud is the audience — who this token is for. And sub, the subject, is the cluster's canonical name for the identity: system:serviceaccount:<namespace>:<serviceaccount>. That sub string is about to become the entire security boundary. Note what this token is not: it carries no AWS anything. It cannot sign an S3 request. On its own it opens no doors.

Layer 2 — the trust policy tells AWS to believe the cluster, narrowly

In Terraform, an IAM role gets an assume-role (trust) policy that federates to the cluster’s OIDC provider and pins exactly which token it will accept:

{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::<acct>:oidc-provider/rh-oidc.s3.us-east-1.amazonaws.com/2a3b…" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "rh-oidc.s3.us-east-1.amazonaws.com/2a3b…:sub": "system:serviceaccount:pgbackup:s3-backup",
      "rh-oidc.s3.us-east-1.amazonaws.com/2a3b…:aud": "openshift"
    }
  }
}

The condition keys are the issuer host followed by :sub and :aud. Read literally, this says: I will hand out this role only to a token from this issuer whose subject is exactly pgbackup:s3-backup and whose audience is exactly openshift. Every other identity in the cluster — every other ServiceAccount, in every other namespace — is refused. The trust is not "the cluster can assume this role"; it's "this one Kubernetes identity can."

Layer 3 — the SDK performs the exchange, invisibly

Inside the pod we set only two environment variables:

AWS_ROLE_ARN=arn:aws:iam::<acct>:role/itssolutions-aws02-irsa-…
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/aws/token

The AWS SDK’s default credential provider chain notices these two together and activates its web-identity provider. It reads the token file and calls STS:

POST https://sts.amazonaws.com/
Action=AssumeRoleWithWebIdentity
&RoleArn=arn:aws:iam::<acct>:role/itssolutions-aws02-irsa-…
&RoleSessionName=aws-sdk-…
&WebIdentityToken=<the JWT from layer 1>

STS validates the JWT signature against the issuer’s published keys, checks sub and aud against the trust policy from layer 2, and if everything lines up, returns temporary credentials:

<Credentials>
  <AccessKeyId>ASIA…</AccessKeyId>
  <SecretAccessKey>…</SecretAccessKey>
  <SessionToken>…</SessionToken>
  <Expiration>2026-…T…Z</Expiration>
</Credentials>
<AssumedRoleUser>
  <Arn>arn:aws:sts::<acct>:assumed-role/itssolutions-aws02-irsa-…/aws-sdk-…</Arn>
</AssumedRoleUser>

The SDK caches those in memory, signs every S3 call with them, and — because the token file is refreshed by the kubelet and the SDK re-reads it — silently repeats the whole exchange before the hour is up. No AWS_SECRET_ACCESS_KEY was ever set; no credential ever touched disk. The Expiration field is the point: even if these ASIA credentials leaked, they're dead in under an hour.

The analogy, and the one idea to keep

A static key is a photocopy of your passport left on a desk — whoever finds it is you, forever. The OIDC flow is showing your passport at a check-in desk (STS) and getting a boarding pass (temporary credentials): scoped to this flight, this gate, today, useless tomorrow. The passport doesn’t board the plane; the boarding pass does — and it evaporates on schedule.

The idea to carry away: the trust boundary is the sub claim. Not a shared secret, not a password, not a key — a Kubernetes identity, verified cryptographically end to end. This is also why it's auditable in a way static keys never are: in CloudTrail, the S3 PutObject shows a userIdentity of type AssumedRole with the session traceable back to the role and, through it, to the ServiceAccount. You don't see "some IAM user did this"; you see "the pgbackup:s3-backup identity did this." Same pattern as IRSA on EKS and Workload Identity on GKE — learn it once, use it everywhere.

Architecture

Two paths, one identity, everything either on my workers or in AWS:

BACKUP  ── CronJob (SA: s3-backup) ────────────────────────────────┐
  initContainer: pg_dump  ──▶  gzip  ──▶  /backup (emptyDir)        │
  container:     aws s3 cp /backup ──▶  s3://bucket/backups/db/TS   │  IRSA via STS
                                                                     │  (no static key)
RESTORE ── Job (SA: s3-backup) ────────────────────────────────────┤
  initContainer: aws s3 cp  latest ◀──  s3://bucket/backups/db/     │
  container:     gunzip | psql  ──▶  PostgreSQL                     ┘

RETENTION ── S3 lifecycle ── expire backups/ older than N days (AWS-side)

Side Component Role AWS (Terraform) S3 bucket, versioned, public access blocked Off-cluster backup store AWS (Terraform) S3 lifecycle rule (+ noncurrent expiry) Auto-retention, server-side AWS (Terraform) IAM policy, least-privilege on backups/* The only thing the role may touch AWS (Terraform) IAM role, OIDC trust scoped to one SA Assumable only by pgbackup:s3-backup OpenShift PostgreSQL + gp3 PVC + Secret The data to protect OpenShift pg-seed Job Puts real rows in the table OpenShift pg-backup CronJob (IRSA) Scheduled dump → S3 OpenShift pg-restore Job (IRSA) Latest backup → database

Get KF dogbe’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A detail I like: Terraform doesn’t just build AWS, it renders the manifests with the role ARN, bucket, and audience already substituted. The one place those values are defined is the one place they come from — no ARN copy-pasted by hand into YAML, no drift.

Design decisions, and the reasoning underneath

Logical dump, and plain vs custom format. pg_dump produces a logical backup — the SQL to recreate schema and data — not a byte-for-byte copy of the data files (that's pg_basebackup, physical, a different tool with different restore semantics). For small-to-mid databases, logical is the right default: portable across minor versions, human-inspectable, trivial to reason about. In plain format the dump is a script — a run of SET statements, then CREATE TABLE, then a COPY notes (id, body, created_at) FROM stdin; block streaming the rows (pg_dump uses COPY, not per-row INSERT, because it restores an order of magnitude faster), then setval to fix sequences. I chose plain deliberately because the restore is legible: pipe it into psql and watch the table come back. The trade-off, named honestly: plain format contains no DROP, so the table must be absent at restore time. In production you'd typically switch to the custom format (pg_dump -Fc) — a compressed archive with a table of contents that pg_restore can restore selectively, in parallel (-j), and with --clean --if-exists to drop-and-recreate atomically. Plain is a teaching choice; custom is the production one.

Splitting the dump from the upload. The detail I’m most pleased with. pg_dump needs the PostgreSQL client tools; aws s3 cp needs the AWS CLI. Instead of building a bespoke image carrying both — a maintenance liability that rots the day either tool releases a CVE — two containers share an emptyDir: an initContainer on the PostgreSQL image dumps to the shared volume, then the main container on the AWS CLI image uploads it. initContainers are guaranteed to run to completion before the main container starts, so ordering is free. Each half holds exactly the credentials it needs and nothing more — the dumper gets PGPASSWORD, the uploader gets the projected OIDC token — which is least-privilege pushed all the way down to the container boundary. The emptyDir lives and dies with the pod, so the dump never lingers on a node.

Retention as an S3 lifecycle rule, and how it interacts with versioning. It’s tempting to put “delete anything older than N days” in the job. Don’t. Lifecycle rules run server-side, on schedule, whether or not your job ran and whether or not your delete logic had a bug. There’s a subtlety worth knowing because the bucket is versioned: with versioning on, an expiration action doesn't hard-delete — it writes a delete marker, making the current version noncurrent. So the rule pairs expiration { days = N } with noncurrent_version_expiration { noncurrent_days = 1 } to actually reclaim the bytes. And S3 evaluates lifecycle once daily and rounds expiration to midnight UTC, so deletion can lag the nominal age by up to a day — fine for retention, but don't design a test that expects deletion to the minute.

Least-privilege, and the ARN distinction that matters. The policy grants s3:ListBucket on the bucket ARN (arn:aws:s3:::bucket) and GetObject/PutObject/DeleteObject on the object ARN with a prefix (arn:aws:s3:::bucket/backups/*). That split is not cosmetic: ListBucket is a bucket-level action and only accepts the bucket ARN, while object actions need the /… path. Scoping objects to backups/* means a compromised backup job can't read or write anything else in the bucket — the blast radius is one prefix.

gp3, EBS, and the AZ constraint on two workers. The PVC uses the gp3-csi StorageClass, whose volumeBindingMode is WaitForFirstConsumer — the volume isn't provisioned until the pod schedules, so EBS creates it in the same Availability Zone as the node that will use it. This matters on a two-worker cluster that may straddle two AZs: an EBS volume is single-AZ and ReadWriteOnce, so the PostgreSQL pod is pinned to its volume's AZ forever after. With a Recreate deployment strategy, that's fine — one pod, one volume, no simultaneous mounts — but it's the reason you can't casually assume a stateful pod will reschedule anywhere.

CronJob semantics. concurrencyPolicy: Forbid so a long backup never overlaps the next tick; successfulJobsHistoryLimit/failedJobsHistoryLimit so old Job objects don't accumulate in etcd; backoffLimit: 1 so a broken backup fails fast and visibly instead of hammering the database with retries.

Implementation: the parts that carry the design

Full manifests are in the bundle; here are the load-bearing pieces.

The two-container backup, sharing a volume. Dump first (initContainer), upload second (main). Watch where each credential goes:

initContainers:
  - name: dump
    image: quay.io/sclorg/postgresql-15-c9s:latest
    args: ["-c", "pg_dump -h postgres -U appuser appdb | gzip -c > /backup/dump.sql.gz"]
    env:
      - name: PGPASSWORD
        valueFrom: { secretKeyRef: { name: pg-creds, key: POSTGRESQL_PASSWORD } }
    volumeMounts: [{ name: backup, mountPath: /backup }]
containers:
  - name: upload
    image: public.ecr.aws/aws-cli/aws-cli:latest
    args: ["-c", 'aws s3 cp /backup/dump.sql.gz "s3://$BUCKET/backups/appdb/$(date -u +%Y%m%d-%H%M%S).sql.gz"']
    env:
      - { name: HOME, value: /tmp }                        # see troubleshooting
      - { name: AWS_ROLE_ARN, value: "<role-arn>" }
      - { name: AWS_WEB_IDENTITY_TOKEN_FILE, value: /var/run/secrets/aws/token }
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      capabilities: { drop: ["ALL"] }
      seccompProfile: { type: RuntimeDefault }

The securityContext isn't decoration — under OpenShift's restricted-v2 SCC these are effectively required, and declaring them explicitly means the pod admits under the default SCC without special privileges.

The projected OIDC token. This is the JWT from layer 1, materialized as a file. The audience must equal a client ID the OIDC provider accepts, and the SDK finds the file via AWS_WEB_IDENTITY_TOKEN_FILE:

volumes:
  - name: aws-token
    projected:
      sources:
        - serviceAccountToken:
            audience: openshift
            expirationSeconds: 3600
            path: token

expirationSeconds: 3600 requests an hour-long token; the kubelet rotates it well before expiry and the SDK re-reads it, so a long-running pod never wields a stale credential.

The restore finds the latest backup itself. No human picks a filename under pressure — the restore’s initContainer lists the prefix, sorts, takes the newest:

LATEST=$(aws s3 ls "s3://$BUCKET/backups/appdb/" | sort | tail -1 | awk '{print $4}')
aws s3 cp "s3://$BUCKET/backups/appdb/$LATEST" /backup/restore.sql.gz

Lexical sort works here precisely because the keys are YYYYMMDD-HHMMSS — timestamp formatting chosen so that "sorts last" means "is newest." A small thing that turns "which backup?" into a non-question.

Validation: destroying real data and getting it back

A restore you’ve never run is the rumor, so the validation is a small disaster, on purpose.

1 — Seed. pg-seed creates a notes table and inserts three rows. rows_seeded = 3. This is our real data.

2 — Back up. Trigger the CronJob without waiting for the schedule:

oc -n pgbackup create job --from=cronjob/pg-backup pg-backup-manual

Two stages fire: the dumper writes /backup/dump.sql.gz; the uploader assumes the role via the layer-1-through-3 exchange and pushes the object. Then confirm it from outside the cluster — the proof that must hold independent of Kubernetes:

aws s3 ls s3://<bucket>/backups/appdb/
#   2026-…  <ts>.sql.gz

If you want to see the identity that did it, get-caller-identity from inside the pod returns an Arn of …:assumed-role/itssolutions-aws02-irsa-…/… — an assumed role, not a user. That string is the visible confirmation that no static key was involved.

3 — Lose the data. Drop the table outright:

DROP TABLE notes;

A row count now errors — the table is gone. This is the moment that, in production, ruins someone’s week. The elapsed time from here to step 5 is your RTO, and now it’s a measured number instead of a hope.

4 — Restore. Apply the restore Job. Its initContainer pulls the latest archive from S3; its main container pipes it into psql. The logs end with rows_restored = 3.

5 — Verify. Count again: 3. The data destroyed in step 3 is back, sourced entirely from S3, recovered through a path that used no static credentials at any point.

That five-step loop — seed, back up, destroy, restore, verify — is the deliverable. Not the bucket, not the CronJob: the demonstrated round trip, with an RPO you can state (the cron interval) and an RTO you’ve actually timed.

What actually bit me

Four failures, with their signatures, because recognizing the signature is most of the fix.

Permission denied: '/.aws' — and it wasn't AWS. The very first aws command failed before any credential logic ran. Pure OpenShift: restricted-v2 ignores the image's USER and runs the container as a random high UID with no /etc/passwd entry, so HOME resolves to /, and the CLI tries to create /.aws at the root, which a non-root UID can't write. The fix is one line — HOME=/tmp, a world-writable directory. The lesson generalizes far past this lab: any third-party image that writes to $HOME breaks under OpenShift's arbitrary-UID model until you give it a writable home (or an emptyDir mounted where it writes). The IRSA machinery hadn't even engaged — the error was upstream of it entirely, a good reminder to read the actual error before assuming it's the exotic new thing you just built.

Audience mismatch → InvalidIdentityToken. The projected token's aud must be a client ID the OIDC provider actually lists. On ROSA, openshift is present by default, which is why the lab uses it; if your provider's ClientIDList differs, STS rejects the token during the layer-3 exchange. Diagnosis is one command — aws iam get-open-id-connect-provider … --query ClientIDList — and the fix is aligning the token's audience with what's listed. The audience is a contract between the token and the provider that has to match exactly; this is the number-one gotcha of custom IRSA.

sub mismatch → AccessDenied on assume. Running the job under the wrong ServiceAccount on purpose, STS refused — because the trust policy pins sub to one exact namespace:serviceaccount. Not a bug; the security model working. Seeing it fail deliberately is how you come to trust that the sub claim really is the boundary and nothing sneaks past it.

relation "notes" already exists on restore. A direct consequence of the plain-format choice: the dump recreates the table, so it must be absent at restore. Dropping the table (rather than deleting rows) is the correct way to simulate loss here — and, in production, the reason you'd reach for custom format with pg_restore --clean --if-exists instead, which sidesteps the whole class of problem.

Lessons I’m taking with me

• A managed control plane backs up the platform, not your data. HCP takes etcd off your plate — and it’s easy to let that convince you backups are “handled.” Your database and PVCs are still entirely yours, and the boundary is sharp enough that getting it wrong is a choice.
• Static credentials are the liability; identity federation removes the thing that leaks. The pod still gets a key to sign S3 calls — a temporary one it trades a signed identity for, scoped to an hour and one ServiceAccount, with nothing on disk. The sub claim, not a secret, is the trust boundary — and it's what makes the access auditable in CloudTrail.
• Split tools and credentials at the container level. Two containers sharing a volume beat one custom image carrying everything, and each half holds only the credentials it needs.
• Let the platform own retention — and know how it interacts with versioning. A lifecycle rule enforces retention server-side; with versioning on, remember to expire noncurrent versions too, or the bytes never actually leave.
• State your RPO and RTO, then measure the RTO. The RPO falls out of the schedule; the RTO stays a guess until you run a real restore and time it.
• Read the actual error before blaming the new machinery. My first failure looked like IRSA and was a $HOME permission issue that predated the AWS call entirely.
• The restore is the deliverable, not the backup. Until you’ve watched real data vanish and return, you have a rumor. Running the disaster on purpose is the cheapest confidence you’ll ever buy — and the only kind worth trusting.

The cluster is disposable. The proof that data survives it is the whole point.

This is part of an ongoing series of production-style OpenShift-on-AWS labs — each an independent scenario drawn from real platform-engineering problems, built, deliberately broken, and validated end to end. Next in the AWS track: restoring this same backup onto a freshly provisioned cluster, so the disaster recovery is genuinely cross-cluster — which is where “disposable cluster” finally earns its name.