Running Trino workers on spot instances

Graceful shutdown, Karpenter node pools on local NVMe, spill to disk, and fault-tolerant execution spooling to S3 Express One Zone.

At Trino Summit 2024 I showed a diagram of the setup we were moving to: the Trino coordinator on an on-demand EC2 instance, the workers on spot instances, and Karpenter replacing workers when AWS takes a spot instance back. The idea was that graceful shutdown and fault-tolerant execution would make those interruptions invisible to the applications sending queries.

The setup from my Trino Summit 2024 talk: coordinator on demand, workers on spot, Karpenter replacing interrupted nodes.

We have run Trino workers on spot in production since 2025, on EKS, and the setup has changed in several places. This post covers what it looks like now: the node pools, where spill goes, what graceful shutdown can and can’t do on a two-minute spot notice, and how fault-tolerant execution spools to S3 Express One Zone. The examples use the community Helm chart from trinodb/charts and Karpenter v1. We run Trino 484.

Coordinator on demand, workers on spot

The coordinator holds the state of every running query. If it goes away, every query on the cluster fails, and Trino’s graceful shutdown API doesn’t apply to it (a shutdown request to a coordinator is rejected). So the coordinator gets its own Karpenter NodePool with karpenter.sh/capacity-type: on-demand and a short list of instance types.

We also set node-scheduler.include-coordinator=false. With it set to true the coordinator runs splits next to its own work, and ours got OOM-killed under load. The price is that the cluster needs at least one worker at all times: with no workers, a query waits query-manager.required-workers-max-wait and then fails. Our worker autoscaler has a minimum of one.

Workers are the part worth putting on spot. Each worker runs in its own node (one pod per node, requests of 15 CPU and 55Gi on a 16 vCPU / 64 GiB instance), so losing a node means losing one worker.

A worker node pool on local NVMe

This is the worker NodePool, trimmed to the parts that matter here:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: trino-worker
spec:
  template:
    metadata:
      labels:
        dedicated: trino-worker
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: trino
      requirements:
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["16"]
        - key: karpenter.k8s.aws/instance-memory
          operator: In
          values: ["65536"]
        - key: karpenter.k8s.aws/instance-local-nvme
          operator: Gt
          values: ["350"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      taints:
        - key: dedicated
          value: trino-worker
          effect: NoSchedule
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 300s
    budgets:
      - nodes: "20%"
      - nodes: "0"
        schedule: "0 6 * * mon-fri"
        duration: 12h
        reasons: ["Underutilized"]

When a NodePool allows both capacity types, Karpenter tries spot first and falls back to on-demand within the same requirements when no spot offering has capacity. Running out of spot capacity costs money for a while instead of causing an outage.

The instance requirements went through a few rounds. Our first version only required local NVMe. Karpenter picks spot instances with the price-capacity-optimized strategy, which weighs interruption risk as well as price, and it kept choosing 128 GiB storage- and memory-optimized types for a pod that uses about 55Gi. Pinning memory to 64 GiB fixed that. Pinning CPU to 16 matters for a different reason: our worker pod has no CPU limit, so on a 32 vCPU instance Trino would run twice the splits against the same query.max-memory-per-node, and the CPU utilization our autoscaler reads would no longer mean the same thing.

With CPU, memory and NVMe fixed, the remaining spot pools were all the same size. When I compared EC2 spot prices in one US availability zone in October 2026 (from the EC2 spot price history), the cheapest NVMe type of that size was also the general-purpose one:

• m7gd.4xlarge and m6gd.4xlarge: $0.174 per hour
• m8gd.4xlarge: $0.214 per hour
• r7gd.4xlarge: $0.218 per hour
• i8g.4xlarge: $0.232 per hour

Prices differ by region and zone and change all the time, so check your own before copying the requirement list.

The business-hours budget blocks consolidation of underutilized nodes during the day, so Karpenter doesn’t move busy workers around to save a few cents. Drift and expiry can still replace nodes during the day. Graceful shutdown and fault-tolerant execution handle those.

One more thing caught us out. Karpenter has to estimate how much memory an instance type will offer before it has launched one. Until it has seen a real node of that type, it subtracts vmMemoryOverheadPercent (7.5% by default) from the nominal memory, plus the kubelet reservation. On a 64 GiB type that leaves about 57.7Gi. With 56Gi or 57Gi requests plus daemonsets, Karpenter wouldn't launch a node at all. 55Gi fits. Larger requests appeared to work while nodes of the type were already running, because Karpenter then uses their real capacity, and broke again on a type it hadn't launched yet.

Spill on the instance store

The EC2NodeClass tells Karpenter what to do with the instance-store disks:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: trino
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  instanceStorePolicy: RAID0
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
  # role, subnetSelectorTerms and securityGroupSelectorTerms as usual

With instanceStorePolicy: RAID0, Karpenter builds a RAID0 array from the NVMe disks, mounts it at /mnt/k8s-disks/0, and puts kubelet and containerd storage on it. The container's writable layer then lives on NVMe, so a spill directory inside the container needs no volume at all. The 50Gi root volume stays separate. That matters because the Trino docs warn against spilling to the drive that holds the JVM logs.

The spill settings, in the coordinator and worker config.properties:

spill-enabled=true
spiller-spill-path=/tmp/trino-spill
spiller-max-used-space-threshold=0.95
spiller-threads=16
spill-compression-codec=NONE

And on the workers:

max-spill-per-node=300GB
query-max-spill-per-node=150GB

Trino treats each spill path as a separate disk. Here there is one path on a RAID0 array, and the docs suggest raising spiller-threads in that case so the device is kept busy. Spill also counts towards the pod's ephemeral storage, so if you set an ephemeral-storage limit, make it larger than max-spill-per-node.

Get Sebastian Daberdaku’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The Trino docs call spill-to-disk legacy functionality and recommend fault-tolerant execution with the TASK retry policy instead. We keep spill on for queries that run with the QUERY policy, where it lets joins, aggregations, sorts and window functions that outgrow the memory limit write to disk instead of failing. For batch queries, fault-tolerant execution does that job.

Graceful shutdown

When a worker gets a shutdown request, Trino 484 does this:

1. The worker switches to SHUTTING_DOWN.
2. It waits for shutdown.grace-period, long enough for the coordinator to notice and stop sending it new tasks.
3. It waits until all of its active tasks are finished.
4. It waits for the grace period again, so the coordinator sees the finished tasks.
5. It exits.

In 2024 I contributed an option to the Trino Helm chart that sets this up:

worker:
  gracefulShutdown:
    enabled: true
    gracePeriodSeconds: 60
    user: admin
  terminationGracePeriodSeconds: 120

The chart sets shutdown.grace-period on the workers and adds this preStop hook:

curl -X PUT -d '"SHUTTING_DOWN"' -H 'Content-Type: application/json' \
  -H 'X-Trino-User: admin' http://localhost:8080/v1/info/state
tail --pid=1 -f /dev/null

The shutdown request returns immediately, and tail --pid=1 keeps the hook running until the Trino process exits. Kubernetes only stops the container after the hook ends, so the worker gets to finish its tasks. The chart also adds a rule to the worker's file-based access control that lets user read and write system information, because the default system access control refuses shutdown requests.

The chart refuses to render if terminationGracePeriodSeconds is less than twice the grace period. The sequence above sleeps for the grace period twice, so it can never finish faster than that. Whatever you configure beyond twice the grace period is the extra time running tasks get to finish before Kubernetes kills the container. With our 60 and 120 seconds, tasks that are already running get the first 60 seconds and nothing more.

What graceful shutdown does on a spot interruption

AWS gives two minutes’ notice before reclaiming a spot instance. With an interruption queue configured, Karpenter receives the warning through SQS, taints and drains the node, and starts a replacement node at the same time. Draining deletes the worker pod, which runs the preStop hook.

With a 60-second grace period, the shutdown sequence takes at least 120 seconds, and the notice is at most 120 seconds. So on a spot interruption the sequence never finishes. What it still does is stop the coordinator from scheduling new tasks on a worker that is about to disappear. The tasks that are running when the instance goes away are lost.

For planned removals, such as consolidation, drift, expiry, scale-in and rolling updates, Kubernetes waits the full terminationGracePeriodSeconds, and the worker leaves without failing anything that finishes in time. Karpenter's own terminationGracePeriod on the NodePool is the longest a node can drain before Karpenter cleans it up by force, and pods whose grace period would run past it are deleted early. Keep it above the pod's terminationGracePeriodSeconds (ours is 10 minutes).

Fault-tolerant execution handles the rest

Fault-tolerant execution covers the tasks graceful shutdown can’t save. With retry-policy=QUERY, Trino reruns the whole query when a worker fails. With retry-policy=TASK, it reruns only the failed tasks. That only works with an exchange manager configured: it spools the output of every task to external storage, S3 in our case, so a retried task can read its inputs again even though the worker that produced them is gone. Without one, queries with TASK fail. The docs also note that TASK adds latency to short queries.

We use both. The cluster default is QUERY, and batch workloads get TASK through the session property manager, per resource group. The exchange manager is configured either way, because QUERY retries of queries with results above exchange.deduplication-buffer-size (32MB by default) also need it.

The applications add one more retry of their own. When a query fails with one of the errors Trino counts as out of memory (EXCEEDED_LOCAL_MEMORY_LIMIT, EXCEEDED_GLOBAL_MEMORY_LIMIT or CLUSTER_OUT_OF_MEMORY), the client runs it again once with the retry_policy session property set to TASK. It only does this if no rows have been returned yet, since a stream that has already produced rows can't be restarted without duplicates.

Spooling to S3 Express One Zone

With TASK retries, every task's output goes through the exchange manager's spooling storage. That data is written and read within a single query and is useless afterwards, so request latency and request price matter more than the price per stored GB. That fits S3 Express One Zone: directory buckets in a single availability zone, with lower latency and lower request costs than S3 Standard, and a higher storage price.

The exchange-manager.properties:

exchange-manager.name=filesystem
exchange.base-directories=s3://trino-spooling--<zone-id>--x-s3
exchange.s3.region=<region>
exchange.s3.storage-class=EXPRESS_ONEZONE
exchange.s3.endpoint=https://s3express-<zone-id>.<region>.amazonaws.com

A few details took us some time:

• Directory buckets reject the default STANDARD storage class, so exchange.s3.storage-class=EXPRESS_ONEZONE is required.
• Access goes through CreateSession, so the role Trino uses needs s3express:CreateSession on the bucket.
• Up to Trino 483, the exchange’s S3 client always set an endpoint override, and when exchange.s3.endpoint was empty it used the regional S3 endpoint. Requests to a directory bucket then failed with The authorization header is malformed; incorrect service "s3express". Setting exchange.s3.endpoint to the bucket's zonal endpoint fixed it. In 484 the override only applies when the property is set. We still set it.
• S3 Express has its own gateway VPC endpoint service, com.amazonaws.<region>.s3express. The usual S3 gateway endpoint doesn't carry directory-bucket traffic.
• Spool files from crashed queries stay behind. The Trino docs recommend a lifecycle rule for them. Ours expires objects after one day and aborts incomplete multipart uploads after one day. Directory buckets have no versioning, so those two rules are the whole lifecycle.

The bucket lives in one availability zone, so we pin the coordinator and the workers to that zone with a node affinity on topology.k8s.aws/zone-id. Use zone IDs here, because zone names map to different physical zones in different AWS accounts. The cost of the pin is that every spot pool comes from one zone. That makes on-demand fallback more likely, and it's the main trade-off in this setup.