How to Build Highly Available Kubernetes Applications with Amazon EKS Auto Mode

Auto Mode patches and replaces your nodes automatically. Here is how to make sure your app never notices.

Amazon EKS Auto Mode keeps your nodes patched, scaled and replaced for you, but it does not make your application highly available. That part is still your workload configuration. Because Auto Mode deliberately recycles nodes, an app with one replica, no Pod Disruption Budget and no zone spread will see downtime during routine maintenance, not just during outages.

This guide covers what Auto Mode does to your nodes, the exact Kubernetes settings that keep traffic flowing through it, and how to prove it works before production does it for you.

TL;DR

• Run at least 3 replicas, spread across 3 Availability Zones (AZs) and across nodes.
• Add a Pod Disruption Budget (PDB) to every production Deployment.
• Use Pod Readiness Gates with ALB/NLB so pods are not terminated before their replacements are healthy.
• Handle SIGTERM, add a short preStop delay and size terminationGracePeriodSeconds to match.
• Tune NodePool disruption settings for your traffic pattern, and test pod, node, AZ and upgrade failures on purpose.

What EKS Auto Mode changes for availability

EKS already runs the Kubernetes control plane across three AZs, so the control plane is AWS’s responsibility. Auto Mode extends that to the data plane: it provisions, scales and updates the EC2 nodes that run your pods, using a managed Karpenter-based system. The catch for availability is that nodes are treated like appliances, and appliances get replaced. According to the EKS Auto Mode and NodePool documentation:

• Node lifetime: nodes are replaced by default after 336 hours (14 days), with a hard maximum of 21 days. Every pod will be evicted regularly.
• AMI updates: nodes can be replaced through drift when a new Auto Mode AMI ships, roughly weekly. Replacement is routine, not rare.
• Default disruption budget: a single budget of 10% of nodes. On a small cluster, that can mean one node at a time.
• Consolidation: underutilized nodes are consolidated by default, so pods are repacked for cost, which also moves them.
• PDBs: respected on scale-down and on node updates. Your PDB is the safety rail, but one that is too strict can stall node replacement until the 21-day cap.

The takeaway: Auto Mode gives you automatic patching, and you pay for it in pod movement. The checklist below makes that movement invisible to users. For the wider principles, our guide to the AWS Well-Architected Framework covers how the reliability pillar fits together.

The workload checklist: six settings that keep traffic flowing

These are standard Kubernetes features. AWS’s own EKS Auto Mode high availability guide tested them against pod, node, AZ and upgrade failures, and the examples below follow its configuration.

1. Run enough replicas

One replica means any node replacement is an outage. Use at least 3 replicas for anything user-facing, so a lost zone still leaves capacity.

2. Add a Pod Disruption Budget

A PDB limits how many pods can be evicted at once. Auto Mode honors it during node updates and scale-down.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: web

Do not set minAvailable equal to your replica count. That blocks every drain and stalls node replacement.

3. Spread pods across zones and nodes

Topology spread constraints stop all replicas landing in one failure domain.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web

ScheduleAnyway is a soft preference, which keeps pods schedulable during an AZ outage. Use DoNotSchedule only if you would rather leave pods pending than break the spread.

4. Use Pod Readiness Gates with AWS load balancers

New pods register with ALB or NLB target groups more slowly than a rollout proceeds, which can leave only initializing or draining targets. Readiness gates hold the old pod until its replacement is healthy in the target group. AWS’s example enables them per namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    eks.amazonaws.com/pod-readiness-gate-inject: enabled

Also lower the target group deregistration delay from the 300-second default if your requests are short, for example with alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30.

5. Shut down gracefully

Node replacement sends SIGTERM. Your app must stop accepting new work and finish in-flight requests, and the load balancer needs a moment to stop sending traffic.

terminationGracePeriodSeconds: 60
lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

This gives 10 seconds for deregistration to propagate and 50 for the app to drain.

6. Separate readiness from liveness

Get MetaSips Official’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Readiness decides whether a pod receives traffic; liveness decides whether to restart it. Keep liveness checks cheap and independent of downstream dependencies, or a database blip restarts every pod at once.

readinessProbe:
  httpGet: { path: /healthz/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz/live, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

Control when and how fast nodes are replaced

You cannot edit the built-in system and general-purpose NodePools, only enable or disable them. For production workloads with specific availability needs, create your own NodePool and set its disruption block. The NodePool documentation describes three consolidation policies:

• WhenEmpty: only removes empty nodes, so running workloads are never disrupted by consolidation. Best for latency-sensitive or slow-starting apps.
• WhenEmptyOrUnderutilized: repacks pods whenever it is cheaper. Best for stateless, fast-starting, cost-driven workloads.
• Balanced: weighs disruption cost against savings and skips actions where disruption outweighs them. A sensible default for mixed production clusters.

A starting point for a production pool:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: production
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
        - key: eks.amazonaws.com/instance-category
          operator: In
          values: ["c", "m", "r"]
      expireAfter: 336h
  disruption:
    consolidationPolicy: Balanced
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"
  limits:
    cpu: "1000"

Three levers do most of the work:

• Budgets cap how many nodes are disrupted at once. Karpenter budgets can also be time-windowed, so you can block disruption during peak hours. See the Karpenter disruption docs.
• Instance diversity (several categories and sizes) lets replacement capacity be found quickly.
• expireAfter sets node lifetime. Auto Mode's hard cap is 21 days, so shorter lifetimes are possible but never longer.

Auto Mode also applies a default 24-hour termination grace period to nodes when you do not set one, so pods that ignore SIGTERM can be held up for a long time. Fix the app rather than relying on that timeout.

Plan zone capacity, not just zone spread

Spread constraints only help if each zone can actually hold the pods. Three rules:

• Give the NodePool subnets in at least three AZs, so Auto Mode can launch capacity in any surviving zone.
• Size for N-1. If one of three zones fails, the other two must absorb its load. For three zones, that means roughly 50% headroom over steady-state, or an autoscaler configured to add replicas quickly.
• Consider static capacity for latency-critical pools. Auto Mode’s static-capacity node pools keep a fixed node count regardless of pod demand. A multi-zone static pool does not guarantee an even split, so create one pool per zone, each pinned with topology.kubernetes.io/zone, when you need predictable distribution. Check the current EKS Auto Mode docs, as this feature is newer.

If your workload is stateful, remember that EBS volumes are zone-bound. A StatefulSet pod can only restart in the zone where its volume lives, so zone failure needs a data-level plan such as replication, not just scheduling.

Prove it: four failure drills

Point an uptime monitor (AWS’s test used the open-source Uptime Kuma) at your load balancer, then run each drill in staging and watch for failed requests.

• Pod failure: run kubectl delete pod on one replica. Pass if there are no failed requests and the replacement becomes Ready.
• Node failure: kubectl drain one node, or delete it. Pass if pods reschedule and the PDB is respected.
• AZ failure: delete all nodes in one zone. Pass if the service stays up at reduced capacity and pods reschedule into other zones.
• Upgrade: upgrade the cluster version. Pass if rolling node replacement honors PDBs with no user-visible errors.

For a harsher test, AWS also used a NodePool that expires nodes every five minutes (expireAfter: 5m) with a taint so only test workloads land on it. It surfaced weak shutdown handlers, slow connection draining and probe misconfiguration. Never apply that to a shared production pool.

When one region is not enough

Multi-AZ design covers a bad node, a lost zone and routine maintenance. It does not cover a regional outage. For a grid-monitoring platform on EKS, we found that backups of individual pieces were not the same as a tested path to recovery. Our cross-region disaster recovery case study for Amazon EKS shows how we used AWS Backup with a pre-provisioned DR cluster, then verified record counts after restore rather than trusting the job status.

The same principle applies to everything in this guide: an availability design is only real once you have tested it.

Five mistakes that cause downtime on Auto Mode

1. Single replica, or all replicas in one zone. The first node replacement or zone issue takes the service down.
2. No PDB, or minAvailable equal to replicas. The first lets too many pods leave at once; the second blocks node replacement entirely.
3. Ignoring SIGTERM. Requests in flight are cut off, or pods hang until the 24-hour default termination grace period.
4. Liveness probes that check dependencies. A slow database triggers mass restarts instead of a clean readiness failure.
5. Never testing. Configuration that looks right still fails under real node churn, so run the four drills before launch and after major changes.

FAQ: EKS Auto Mode high availability

Is EKS Auto Mode highly available by default? The EKS control plane runs across three AZs, and Auto Mode manages the nodes. Your application’s availability still depends on replicas, PDBs, zone spread and graceful shutdown.

How often does EKS Auto Mode replace nodes? By default after 336 hours (14 days), with a maximum of 21 days. Nodes can also be replaced sooner when a new Auto Mode AMI is released, roughly weekly.

Does EKS Auto Mode respect Pod Disruption Budgets? Yes, during scale-down and node updates. A PDB that is too strict can stall node replacement until the 21-day limit.

Can I control when nodes are disrupted? Yes. Use a custom NodePool with consolidationPolicy, consolidateAfter, expireAfter and disruption budgets.

Does Auto Mode cost extra? Yes. AWS charges a management fee on top of standard EC2 pricing, only for nodes Auto Mode manages. Check current rates on the EKS pricing page.

Should I run critical workloads on Spot? NodePools support Spot and On-Demand capacity types. For availability-critical services, keep baseline capacity on On-Demand, and use Spot for burst capacity that can tolerate interruption.

Need a second pair of eyes on your EKS setup?

Metasips is an AWS Advanced Tier Services Partner. Our DevOps services and AWS infrastructure management teams design, test and run production Kubernetes on AWS. Book a call and we will tell you what is one bad deploy away from breaking.

Sources: AWS Containers Blog: highly available applications with EKS Auto Mode, EKS Auto Mode docs, NodePool docs, EKS reliability best practices, Karpenter disruption concepts.