Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Migrating EKS from AWS VPC CNI to Cilium with Zero Downtime

14 min readMay 13, 2026

--

Press enter or click to view image in full size

Some time ago, we had a requirement to migrate from AWS VPC CNI in our EKS clusters to Cilium. It was mainly because we wanted to have more observability into traffic patterns within our cluster.

The migration began as a network policy project. We wanted Kubernetes workloads to communicate only with explicitly allowed services and namespaces. The current AWS VPC CNI setup gave us reliable pod networking, but it did not give us the end-to-end operating model we wanted for least-privilege east-west traffic, policy verdicts, audit-first rollout, and runtime flow visibility. Cilium, Envoy, and Hubble gave us that path.

The hardest requirement was simple to say and painful to implement: no downtime.

This is the story of how we migrated an Amazon EKS platform from the AWS VPC CNI to Cilium without downtime. In a nutshell, what we did was — isolate the old and new networking stacks by node labels, run a blue/green node migration, bring Cilium up with AWS ENI IPAM, preserve the existing pod subnet model, tweak the load balancer behavior to suite the Cilium setup, and only then remove the AWS networking components. For some nodes we use Karpenter, but concept is similar there aswell.

Press enter or click to view image in full size

The final state was Cilium as the CNI, Cilium kube-proxy replacement enabled, Hubble enabled for observability, AWS ENI IPAM with prefix delegation, native routing, and Karpenter-created nodes joining the cluster with a Cilium startup taint.

At the time of the cutover we used:

Press enter or click to view image in full size

Why this was not an in-place CNI swap

A Kubernetes node is not CNI-agnostic once workloads are running on it. Pods are created through the CNI plugin that exists on the node at scheduling time. kubelet asks the CNI to create pod networking. The CNI config in `/etc/cni/net.d`, the per-node daemon behavior, IPAM state, routes, BPF programs or iptables rules, and security group assumptions all matter.

An in-place replacement would have meant trying to change the networking engine under live pods. That is the wrong failure domain. If something breaks, the blast radius is the entire node and every workload on it. Rollback is also ugly because the old and new plugins may have already mutated host networking state.

So the design rule was: A node should belong to exactly one CNI generation. Workloads move between generations; nodes do not mutate in place.

That rule shaped the whole migration.

Press enter or click to view image in full size

As in the above image, our AWS VPC CNI was an addon that we installed. Cilium is something we install with Helm Charts. In our setup we use GitOps, and thats why in the image, ArgoCD appears as managing Cilium DS and cilium-envoy DS.

We used labels and affinity as the isolation boundary:


# Old nodes
metadata:
  labels:
    cni: aws-cni

# New nodes
metadata:
  labels:
    cni: cilium

The legacy AWS networking components were constrained away from the Cilium nodes, and Cilium was constrained onto only the Cilium nodes. During the overlap window, two networking stacks existed in the same Kubernetes cluster, but not on the same node.

The target architecture

At the end of the migration, the cluster looked like this:

Press enter or click to view image in full size

The important pieces are:

  • Cilium agent runs as a DaemonSet on every Cilium node.
  • Cilium operator manages cluster-level tasks, including AWS ENI/IPAM operations.
  • Cilium Envoy runs for Cilium features that need proxy functionality.
  • Hubble Relay and Hubble UI provide flow observability.
  • Karpenter and managed node groups create nodes labeled for Cilium.
  • AWS VPC CNI and kube-proxy are removed after workloads are fully moved.

The AWS VPC CNI model we had before used AWS-native pod addressing. We wanted to preserve that basic idea. We did not want a tunneling migration if the existing subnet design could support native routing. Cilium ENI IPAM was a good fit because it lets Cilium allocate pod IPs from AWS ENIs and subnets rather than using an overlay.

How Cilium ENI IPAM fit the existing subnet model

Many EKS platforms use a split-subnet model:

  • Nodes live in primary/private VPC subnets.
  • Pods receive IPs from secondary or dedicated pod subnets.
  • The pod CIDR may be separate from the node CIDR.

That pattern is common when teams want to conserve primary VPC address space or keep pod addressing separated from node addressing. With AWS VPC CNI custom networking, this is often done with `ENIConfig` resources. With Cilium, we moved that responsibility to Cilium ENI IPAM.

Press enter or click to view image in full size

Our VPC design used two different address spaces for two different jobs:

  1. A routable private range for EC2 worker nodes.
  2. A non-routable or tightly scoped pod range for Kubernetes pod IPs.

The word “non-routable” here is operational, not magic. These pod subnets are still real VPC subnets and AWS can attach ENIs in them. The point is that they are not broadly routable from every corporate, VPN, peered, or on-prem network. They are intentionally scoped for pod addressing. That means pod-to-pod traffic inside the cluster can use pod IPs directly, but traffic leaving the cluster boundary often needs to be SNATed to a node IP from the routable range.

The EC2 worker node starts life in the routable range. When the instance is created, AWS attaches its primary ENI, usually visible as `eth0` or the primary Linux device name. That primary ENI gives the node its node IP, lets kubelet reach the API server, lets SSM or node management work, and gives the operating system its normal route out of the subnet.

Pods are different. We did not want pods to consume addresses from the node subnet, and we did not want every external network to need routes back to pod CIDRs. So Cilium attaches and manages additional ENIs from the pod subnets. Those secondary ENIs are where pod IP capacity comes from.

Press enter or click to view image in full size

The lifecycle is roughly:

  • A Cilium-labeled node joins the cluster with its primary ENI already attached by EC2.
  • `cilium-agent` starts on the node and publishes node/IPAM requirements through Cilium’s Kubernetes state, including the `CiliumNode` resource.
  • `cilium-operator` watches that state and talks to the EC2 API.
  • The operator selects the configured pod subnets and attaches or updates secondary ENIs on the EC2 instance.
  • With prefix delegation enabled, AWS delegates prefixes to those ENIs instead of forcing Cilium to manage only individual secondary IPs.
  • Cilium records the available IP capacity for the node.
  • When kubelet creates a pod, the Cilium CNI plugin allocates an IP from the node’s Cilium-managed pool and the agent programs the datapath.

This is the key difference from the old AWS VPC CNI custom networking model. Previously, AWS VPC CNI owned the ENI/IPAM loop. After the migration, Cilium owns that loop. AWS still provides the ENIs, subnet routing, prefixes, and EC2 APIs; Cilium decides when the node needs capacity, which configured pod subnets to use, how much IP capacity to keep warm, when to release excess IPs, and how to program pod connectivity on the node.

`firstInterfaceIndex=1` is what makes that split explicit in the Helm values. It tells Cilium not to treat the primary node interface as the starting point for pod ENI allocation. Interface index `0` remains the node’s primary/routable interface. Interface index `1` and above become the place where Cilium-managed pod ENIs live. We will come to this later, since it’s already looking to be more complex than it needs to be.

Implementation: Phase 1

Well, hope you did not make any changes yet. Let’s walk through the steps we followed;

Step 1.1 — Create policies required for Cilium to operate and attach it to node instance role.

Cilium needs some permission to describe subnets, assign IP addresses and perform some actions on network interfaces. We are adding this to our node policy (instance role) for simplicity. You may do this by creating a seperate role and referring to it through an annotation for service account in your Cilium helm chart aswell. Below is the policy;

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DescribeSecurityGroups",
        "ec2:CreateNetworkInterface",
        "ec2:AttachNetworkInterface",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:AssignPrivateIpAddresses",
        "ec2:CreateTags",
        "ec2:UnassignPrivateIpAddresses",
        "ec2:DeleteNetworkInterface",
        "ec2:DescribeInstanceTypes"
      ],
      "Resource": "*"
    }
  ]
}

Step 1.2 — Allow Node to Node ICMP

In your node security group, you need to enable Node to Node ICMP for Cilium to work properlty.

    node_to_node_icmp = {
      description = "Node to node ICMP for Cilium health checks"
      protocol    = "icmp"
      from_port   = -1
      to_port     = -1
      type        = "ingress"
      self        = true
    }

Step 1.3 — Set Networking addons to PRESERVE

We are going to limit where aws-node pods and kube-proxy pods can schedule. To to do this, we need to first set addons to “Preserve” so AWS will not revert our changes to the deployments created by the addons.

You can do this from AWS Console by navigating to your EKS cluster → Add-ons → (Amazon VPC CNI and Kube Proxy) → Edit → Optional configuration settings → Conflict resolution method → Preserve

Step 1.4 — Patch kube-proxy and aws-node (AWS VPC CNI) addons

kubectl -n kube-system patch ds kube-proxy --type='merge' -p '{
  "spec": {
    "template": {
      "spec": {
        "affinity": {
          "nodeAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": {
              "nodeSelectorTerms": [
                {
                  "matchExpressions": [
                    {
                      "key": "cni",
                      "operator": "NotIn",
                      "values": ["cilium"]
                    }
                  ]
                }
              ]
            }
          }
        }
      }
    }
  }
}'
kubectl -n kube-system patch ds aws-node --type='merge' -p '{
  "spec": {
    "template": {
      "spec": {
        "affinity": {
          "nodeAffinity": {
            "requiredDuringSchedulingIgnoredDuringExecution": {
              "nodeSelectorTerms": [
                {
                  "matchExpressions": [
                    {
                      "key": "cni",
                      "operator": "NotIn",
                      "values": ["cilium"]
                    }
                  ]
                }
              ]
            }
          }
        }
      }
    }
  }
}'

Step 1.5 — Add cni: aws-cni Label to all existing nodes

We now need to add the label aws-cni to all existing nodes. We manage our infrastructure with Terraform, so we went with adding the label in our terraform configurations and applying it, because it then ensures new nodes that will be created will also carry this toleration. If you nodes don’t fluctuate much, you can use a kubectl command to add the label.

Step 1.6 — Bring up new NodeGroups with Label cni: cilium

We brought up a new nodegroup with similar capacity to our existing nodegroup with the label cni: cilium. As expected, they were in NotReady state.

We also added a taint to these nodes, which Cilium will remove when it’s ready. It prevents pods from scheduling on these nodes before Cilium.

taints = [
  {..other taints...},
  {
    key    = "node.cilium.io/agent-not-ready"
    value  = "true"
    effect = "NO_SCHEDULE"
  }
]

Explaining the Helm Values in Cilium Chart

Next phase is deploying the helm chart. I thought I’ll first explain the values we added, so you can tweak your values.yaml to suit your usecase.

Get Ravindu Fernando’s stories in your inbox

Join Medium for free to get updates from this writer.

Earlier we were talking about ENI allocation. So we will continue first exploring that area.

ENI and IPAM Settings

The core values looked like this, sanitized:


cilium:
  ipam:
    mode: eni

  eni:
    enabled: true
    awsReleaseExcessIPs: true
    awsEnablePrefixDelegation: true
    nodeSpec:
      firstInterfaceIndex: 1
      subnetIDs:
        - <pod-subnet-a>
        - <pod-subnet-b>
        - <pod-subnet-c>

There are several important decisions hidden in those few lines.

  • `ipam.mode=eni` tells Cilium to allocate pod IPs using AWS ENIs. Cilium becomes responsible for managing IP availability on each node. Instead of Kubernetes pods getting overlay addresses, they receive VPC-native IPs from the configured pod subnets.
  • `awsEnablePrefixDelegation=true` uses AWS prefix delegation so an ENI can receive delegated prefixes instead of only individual secondary IPs. That improves pod density and reduces the amount of per-pod EC2 API churn during scheduling spikes.
  • `awsReleaseExcessIPs=true` allows Cilium to release excess IPs back when they are no longer needed. Without this, a busy scale-up event can leave too much address capacity pinned to nodes that no longer need it.
  • `firstInterfaceIndex=1` means Cilium starts pod ENI work from the second interface. The primary interface remains the node interface. This matters because we did not want pod IP allocation to collide with the node’s own management path.

In a reusable article or platform module, I would avoid hardcoding subnet IDs in the Cilium values if I can discover them by tags. But during a controlled migration, explicit subnet IDs can be a very useful safety rail because they remove ambiguity. The key is to make sure the values point to the pod subnets, not the node subnets.

Native routing and the CIDR trap

We used native routing:

cilium:
  routingMode: native
  ipv4NativeRoutingCIDR: <pod-cidr>

Native routing means we rely on the underlying network rather than encapsulating pod traffic in an overlay. That choice matched the ENI IPAM model: pods already have IPs from the VPC subnet design, so extra encapsulation would add complexity without solving the main problem.

The `ipv4NativeRoutingCIDR` value tells Cilium which CIDR is native-routed and should not be treated like generic external traffic. We set it to the pod CIDR range used by the cluster.

The trap is making this value too broad. For example, do not lazily set it to something like `10.0.0.0/8` just because many corporate networks live there. That can change SNAT behavior for destinations outside the cluster. If your pods live in a secondary non-routable range and need to talk to private endpoints, VPN clients, or peered VPCs, overly broad non-masquerade rules can blackhole traffic because the remote side may not know how to route back to pod IPs.

Our egress model was intentionally narrow:

cilium:
  bpf:
    masquerade: true

  enableIPv4Masquerade: true

  ipMasqAgent:
    enabled: true
    config:
      nonMasqueradeCIDRs:
        - <pod-cidr>
        - <additional-pod-cidr-if-used>
        - <service-cidr>
      masqLinkLocal: false
      masqLinkLocalIPv6: false

The idea is:

  • Do not SNAT traffic that stays inside known cluster pod/service ranges.
  • Do SNAT traffic to networks that cannot route back to pod subnets.
  • Avoid broad enterprise CIDR exclusions unless routing has been explicitly proven.

This was one of the most important production details in the whole migration. Cilium can make the datapath very efficient, but it will also faithfully implement a bad CIDR decision.

Replacing kube-proxy

We enabled Cilium kube-proxy replacement:

cilium:
  kubeProxyReplacement: true
  kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256

That meant Cilium handled Kubernetes service load balancing through eBPF rather than relying on kube-proxy rules. We did not run kube-proxy on Cilium nodes.

This was not a cosmetic change. It simplified the end state and avoided a partial migration where Cilium handled pod networking but kube-proxy still handled service translation. During the migration overlap, kube-proxy remained only on legacy nodes, and Cilium’s service datapath existed only on Cilium nodes.

The healthz bind address matters because some cluster components and operational checks expect kube-proxy-style health behavior. Cilium can expose a compatible health endpoint for that.

Scheduling Cilium only where it belongs

Cilium itself was pinned to Cilium nodes:

cilium:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cni
                operator: In
                values:
                  - cilium

  operator:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: cni
                  operator: In
                  values:
                    - cilium

  envoy:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: cni
                  operator: In
                  values:
                    - cilium

We also gave Cilium and its operator a high priority class:

cilium:
  priorityClassName: system-node-critical
  operator:
    priorityClassName: system-node-critical
  envoy:
    priorityClassName: system-node-critical

The CNI is node-critical infrastructure. If a node is under pressure, Cilium should not be the first pod evicted.

The startup taint pattern

In Implementation Phase 1 under last step I showed you how we added a taint. We need to introduce that taint to Cilium so it will remove the taint when ready.

cilium:
  agentNotReadyTaintKey: node.cilium.io/agent-not-ready
  removeNodeTaints: true

Observability

We enabled prometheus metrics on Cilium and Operator:

cilium:
  prometheus:
    enabled: true
    port: 9090

  operator:
    prometheus:
      enabled: true
      port: 6942

We also enabled Hubble:

cilium:
  hubble:
    enabled: true
    relay:
      enabled: true
      prometheus:
        enabled: true
    ui:
      enabled: true

Implementation: Phase 2

Now what we have to do is install Cilium. We already went though the configuration options. Here is a minified values.yaml, but read the configurations I explained earlier and add them here if necessary.

cilium:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cni
                operator: In
                values: ["cilium"]

  ipam:
    mode: eni

  eni:
    enabled: true
    awsReleaseExcessIPs: true
    awsEnablePrefixDelegation: true
    nodeSpec:
      firstInterfaceIndex: 1
      subnetIDs:
        - <pod-subnet-a>
        - <pod-subnet-b>
        - <pod-subnet-c>

  routingMode: native
  ipv4NativeRoutingCIDR: <pod-cidr>

  endpointRoutes:
    enabled: true

  bpf:
    hostLegacyRouting: false
    masquerade: true

  kubeProxyReplacement: true
  kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256

  devices:
    - <primary-device>
    - <secondary-device>

  enableIPv4Masquerade: true

  ipMasqAgent:
    enabled: true
    config:
      nonMasqueradeCIDRs:
        - <pod-cidr>
        - <service-cidr>

  agentNotReadyTaintKey: node.cilium.io/agent-not-ready
  removeNodeTaints: true

One subtle point: when using BPF masquerading, do not also configure the older `egressMasqueradeInterfaces` pattern. Use `devices` and the BPF masquerading path consistently.

For AWS Nitro instances, the devices are typically:

cilium:
  devices:
    - ens5
    - ens6

Other option is to use a wildcard:

cilium:
  devices:
    - "ens+"

Another subtle point: `endpointRoutes.enabled=true` creates per-endpoint routes. In ENI/native routing mode, this gives Cilium a clear route per pod endpoint and helps keep routing behavior explicit.

Now, install the Helm chart. Watch the Cilium pods get scheduled on your new node group. Once Cilium is up and running, those nodes would turn to “Ready” state.

Then you can start draining your older nodes, so workloads start scheduling on the new nodes. Make sure to remove AWS VPC CNI and Kube Proxy addons aswell.

However, I highly recommend you test this patten on a test cluster before touching an active cluster, since every setup is different.

What we tested

The migration was tested at multiple layers.

At the platform layer:

kubectl -n kube-system get ds cilium -o wide
kubectl -n kube-system get deploy cilium-operator hubble-relay
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --brief
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg version

After the cutover, the live cluster reported Cilium `1.19.0`, Cilium DaemonSet fully ready, Cilium operator available, Hubble Relay available, and no remaining `aws-node` or `kube-proxy` DaemonSet. We also checked pod IP distribution to ensure application pods were receiving IPs from the expected pod ranges rather than accidentally falling back to node subnet behavior.

At the service layer:

kubectl get endpointslices -A
kubectl get svc -A
kubectl get ingress -A
kubectl run curl --rm -it --image=curlimages/curl -- sh

From test pods, validate:

curl -sS http://kubernetes.default.svc
curl -sS http://<internal-service>.<namespace>.svc.cluster.local
curl -sS https://<ingress-host>

At the load balancer layer:

kubectl -n ingress get pods -o wide
kubectl -n ingress describe svc ingress-internal-controller
kubectl -n ingress logs deploy/ingress-internal-controller

Note: I’m using our pod names and namespaces here. Yours would be different.

In AWS, check that NLB targets are healthy, cross-zone load balancing is active if you use it, target group attributes match the intended state, and target deregistration behaves correctly during a test drain.

Important Note:

We came across a bizarre issue with our LB setup.

For ingress-nginx behind an AWS NLB, we had the LB service set to IP target mode and explicitly set client IP preservation behavior.

Here is a snippet from our older ingress-nginx setup:

controller:
  config:
    use-proxy-protocol: "true"
    use-forwarded-headers: "true"
    compute-full-forwarded-for: "true"

  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true,deregistration_delay.timeout_seconds=120,deregistration_delay.connection_termination.enabled=true
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
      service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true

We had to make preserve_client_ip.enabled=false to get our Ingress setup working.

With NLB IP targets, the target is a pod IP, not an instance/node target. In a Cilium ENI IPAM setup, pod IPs come from dedicated pod subnets. Preserving the original client IP can be attractive for logging, but it can also expose routing and return-path assumptions that were previously hidden by node-level traffic flows. By disabling client IP preservation at the target group and enabling Proxy Protocol v2, the ingress controller can still recover client identity while the network path remains predictable.

References

Kubernetes
Amazon Eks
Cilium
AWS
DevOps

--

--

Ravindu Fernando

Written by Ravindu Fernando

11 followers
8 following

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech