Upgrading Trino on Kubernetes with blue/green clusters

Two Trino clusters behind Trino Gateway, one colour upgraded at a time, and what changed between Trino 481 and 484.

A Trino cluster runs a single version. When a worker reports a different version from the coordinator, the coordinator marks it invalid and stops scheduling work on it, so you can’t roll a cluster forward one pod at a time. Restarting the coordinator ends every query it is running. In practice, upgrading a single cluster means picking a quiet hour and accepting that whatever is running at that moment fails.

In our busiest region we avoid that by running two Trino clusters, blue and green, behind Trino Gateway. Clients only know the gateway’s address. One colour takes the traffic, the other sits idle, and an upgrade happens on the idle one. This October we used it to go from Trino 481 to 483 and then to 484. This post covers the setup, the procedure and the breaking changes we met on the way.

The setup

Two Trino clusters behind Trino Gateway. New queries go to the active colour; the idle one gets upgraded first.

Both colours are Helm releases of the same chart, built from shared values plus a small overlay per colour. We deploy them with Argo CD and Kustomize:

trino/
  common/          catalogs, access control, resource groups, session properties, topology
  trino-blue/      namespace, zone, spooling bucket, temporary version pins
  trino-green/     the same for green
  trino-gateway/   gateway release and routing rules

The colours share everything that defines what a query sees: catalog files, the AWS Glue metastore and the data in S3, the access-control rules, resource groups and session defaults. A query returns the same result on either colour, so users don’t notice a switch.

They differ in where they run:

• Each colour lives in its own namespace and is pinned to its own availability zone, with anti-affinity against the other colour. Losing a zone takes out one colour at most.
• Each colour has its own spooling bucket for fault-tolerant execution. We spool to S3 Express One Zone, whose directory buckets are zonal, so the bucket sits in the colour’s zone.
• Each colour has its own hostname next to the gateway’s. We use it to test a colour before it gets any traffic.

The idle colour costs a coordinator and one worker. Workers scale with KEDA, and the minimum is one rather than zero: our coordinators don’t run splits, and with no worker a query waits for query-manager.required-workers-max-wait and then fails.

On the gateway side there are two backends in the same routing group, and only one of them is active. The gateway runs at least two replicas with a PodDisruptionBudget and a preStop delay, so a gateway rollout never takes the entry point down. The community chart didn't support those settings when we started, so I added them upstream: the PodDisruptionBudget, deployment strategy, lifecycle hooks, termination grace period, and a config checksum annotation that rolls the gateway when its configuration changes.

Keeping a change on one colour

The setup only helps if a change reaches one colour at a time, and the shared common directory works against that. The chart puts a checksum of the configuration on each pod template. A change in a shared file changes the checksum on both colours, and Argo CD restarts both coordinators in the same sync.

So any change that restarts pods goes into one colour’s overlay first:

# trino-green/values.yaml
trino:
  image:
    tag: "484"   # lift into common once blue runs it too

When both colours run the new version, the tag moves into common and both overrides are removed. That change renders the same manifests as before, so nothing restarts. I check that on every pull request by rendering both colours from the branch and from main and diffing the output. The active colour must show no difference, and the idle one only the change I expect.

Get Sebastian Daberdaku’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Coordinator config files were a special case. Resource groups and session property rules live in files on the coordinator, and every edit changed the coordinator’s checksum and restarted it on sync. I changed the community chart so that a user-supplied annotation replaces the generated checksum, and our chart now pins it. Trino reads those files only at startup, so a change now waits for the next restart, which we do on purpose, one coordinator at a time.

Checks before the cluster sees a new version

Most of the work happens before anything is deployed:

1. Read the release notes of every version in the jump, including the ones you skip. Going from 481 to 483 meant reading 482 as well. For each removed or renamed property, search the rendered configuration.
2. Run the new image locally in Docker with the rendered config.properties, node.properties, exchange manager configuration and every catalog file, with AWS credentials stubbed. Trino refuses to start when it finds a property it doesn't use, so this catches removed properties in seconds. It shows that the configuration is accepted, not that the metastore is reachable.
3. Compare defaults. At startup Trino logs every configuration property with its default and current value. Diffing those tables between the old and the new image shows changed defaults that the release notes don’t stress. In 484, for example, the default of hive.storage-format changed from ORC to PARQUET, which matters for a CREATE TABLE without an explicit format.
4. Search application code for the SQL behaviour changes. In 484, json_parse keeps the order of object members and duplicate keys, and several JSON and interval functions became stricter. We found one script that casts parsed JSON to a map, which can now fail on a duplicate key.

The upgrade

With green idle, an upgrade goes like this:

1. Bump the chart and the image in green’s overlay and merge. Argo CD syncs green; the render diff confirms blue doesn’t change.
2. Test green through its own hostname. SELECT node_id, node_version FROM system.runtime.nodes shows every node on the new version, SHOW CATALOGS lists every catalog, and a few test queries return.
3. Switch the gateway: activate green and deactivate blue, either with the switch in the gateway UI or through its REST API (the two calls are below). New queries go to green. Queries already running on blue finish there, because the gateway routes follow-up requests for a query, such as fetching results, by query ID to the cluster that started it.
4. Let green take traffic for a while. If something goes wrong, switching back is the rollback, since blue still runs the old version.
5. Wait until blue has no running or queued queries, then apply the same change to blue.
6. Lift the version into common and drop both overrides. This renders identical manifests, so nothing restarts.

The switch in step 3 through the REST API:

curl -X POST https://trino-gateway.example.com/gateway/backend/activate/trino-green
curl -X POST https://trino-gateway.example.com/gateway/backend/deactivate/trino-blue

Our second production region runs a single cluster. It gets the upgrade after the blue/green region has run the new version for a while, at a quiet hour.

What changed between 481 and 484

Four changes needed more than a version bump.

Trino 483 added s3.auth-type and requires s3.auth-type=IAM_ROLE wherever s3.iam-role is set. Every Iceberg, Hive and Delta Lake catalog that assumes an IAM role needed the new line, and Trino 481 and 482 reject s3.auth-type as an unknown property. The chart change and the image change therefore had to ship together, in one colour's overlay, and could only move into common after both colours ran 483.

Trino 483 also made the new web UI the default at /ui and moved the old one to /ui/legacy. The web-ui.preview.enabled property is deprecated, so we removed it in a later change that restarted the coordinators anyway.

Trino 484 changed the Docker image. The run-trino script is gone. It used to set node.id to the container's hostname, so without it every restart gets a random node ID unless node.properties sets node.id=${ENV:HOSTNAME}. The image is now based on the Red Hat hardened runtime, which drops tools such as grep. Check that your probes, preStop hooks and graceful shutdown scripts only use what is still there; ours needed sh, curl and tail, which remain.

Trino 484 moved the Alluxio file system cache settings out of the catalog files into a cache manager configured with cache-manager.config-files. Trino 484 refuses to start with the old properties in a catalog, and 483 rejects the new one, so the move had to ship with the image. The cache also starts cold after the upgrade.

We upgraded Trino Gateway to release 21 in the same weeks. It refreshes a backend’s health state after an API call changes it, so a backend you just activated isn’t sent traffic before a health check has passed. It also routes the spooling client protocol’s result downloads to the coordinators, which matters once you enable that protocol.

What I’d keep

• Keep the differences between colours small and visible: namespace, zone, spooling bucket and temporary version pins.
• Render and diff both colours on every change. The active colour should not change.
• Start the new image with your real configuration on a laptop before any cluster runs it.
• Read the release notes of the versions you skip as carefully as the one you install.