Istio service mesh with Calico (VXLAN mode) in EKS

What is Istio, and Why Use It

The problem it solves

Once you split an application into many microservices running in Kubernetes, a new set of problems shows up that has nothing to do with your application code:

• How do services find and talk to each other securely?
• How do you see what’s slow, what’s failing, and why — across dozens of services?
• How do you roll out a new version safely, without risking the whole app?
• How do you enforce “service A can talk to service B, but not service C” without writing that logic into every app?
• How do you encrypt traffic between services without every team implementing TLS themselves?

Kubernetes handles scheduling, scaling, and basic networking — but it doesn’t answer any of the questions above. That gap is what’s usually called a service mesh, and Istio is the most widely used implementation of one.

What Istio actually is

Istio is an infrastructure layer that sits between your services, handling all the network-level concerns (routing, security, observability) so your application code doesn’t have to.

It does this with a simple mechanical trick: it deploys a small proxy (Envoy) alongside every one of your application pods — called a sidecar. All traffic in and out of your app container is transparently routed through this proxy first. Because every request now passes through a proxy that Istio controls, Istio can:

• Encrypt the connection (mTLS) — without your app knowing or doing anything
• Decide where a request should go — this version or that version, this % of traffic or that %
• Retry, time out, or circuit-break a failing call — before your app even notices
• Record exactly how long the call took, whether it succeeded, and where it went

Why teams adopt it

Security: mutual TLS everywhere, without code changes

Istio can automatically encrypt every service-to-service connection and verify the identity of both sides (mTLS), using short-lived certificates it manages itself. No team has to add a TLS library, manage certs, or remember to turn it on per-service.

Traffic control: safe rollouts

Canary releases, blue-green deploys, and percentage-based traffic splitting (e.g. “send 5% of traffic to v2”) become a config change (VirtualService), not an application feature every team has to build.

Resilience: retries, timeouts, circuit breaking

You can set “retry this call twice, time out after 2s, stop sending traffic to an instance that’s failing” — again, as config, not code — and apply it consistently across every service.

Observability: see the whole system, not just one service’s logs

Since every request passes through a proxy, Istio can generate consistent metrics, logs, and distributed traces for every service automatically — even ones you didn’t write and can’t easily instrument yourself (e.g. a third-party or legacy service).

Fine-grained access control

AuthorizationPolicy resources let you say "only the checkout service can call payments, and only on this path" — enforced at the network layer, independent of what the app itself does or doesn't check.

What Istio is not

• It doesn’t replace your Ingress/API gateway concept entirely — it usually works alongside your existing edge (e.g. ALB), handling routing once traffic is inside the cluster.
• It’s not free — every pod gets an extra container (the sidecar), which means more CPU/memory per pod and more moving parts to operate. This is a real, ongoing operational cost, not just a one-time setup cost.
• It doesn’t fix bad application design — retries and circuit breakers help with transient issues, not with a service that’s fundamentally too slow or a data model that doesn’t scale.
• It’s not something you should install because it’s popular — it earns its cost when you have enough services, enough teams, and enough cross-service traffic that manual TLS/retry/observability work per-service becomes genuinely painful.

Read this carefully before starting the setup.

Note: An overlay CNI (e.g., Calico with its own IPAM) breaks admission webhooks, and the only solution is to run Calico in policy mode or in hostNetwork.

You can get the error while setting up Istio; the webhook is not coming up due to network limitations. Although I have updated the YAML file with the hostNetwork changes for no need to do any modification.

Istio webhook error.

2026–08–13T17:26:22.429555Z error controllers error handling istio-validator-prod-v1-istio-system, retrying (retry count: 47): webhook is not ready, retry controller=validation 2026–08–13T17:27:22.441420Z info validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected

sidecar injection and/or config validation fail with errors like:

13m Warning FailedCreate replicaset/nginx-6f9664446b Error creating: Internal error occurred: failed calling webhook “rev.namespace.sidecar-injector.istio.io”: failed to call webhook: Post “https://istiod-prod-v1.istio-system.svc:443/inject?timeout=10s": no endpoints available for service “istiod-prod-v1”

Set up Istio in the production EKS 1.34.4 cluster; AWS Load Balancer Controller already in use for ingress.

Data plane: Sidecar mode (mature, best supported)

• mTLS rollout: PERMISSIVE mesh-wide → STRICT per namespace → STRICT mesh-wide
• Ingress: ALB (via AWS LB Controller) in front, Istio Gateway behind it
• Control plane: installed with a revision (not the default, unnamed install) so you can canary upgrades later without touching this structure

⚠️ Version note: Istio’s official support matrix typically covers the last 3 Kubernetes minor versions at time of an Istio release. Since EKS 1.34 is very recent, confirm on istio.io’s supported releases page that your chosen Istio version explicitly lists 1.34 support before installing in production.

0. Pre-flight checks

0.1 Cluster/network capacity

Sidecars double pod IP consumption (app pod + Envoy sidecar container share one pod IP, but pod density per node effectively goes up and ENI/IP pressure increases with more containers per node needing more resources — the bigger real risk is IP exhaustion under VPC CNI at scale, and CPU/memory headroom per node).

But no need to worry about IPs because we are using Calico here, which uses its own IP pool for the pods, and for services, EKS use it’s own.

kubectl get nodes -o custom-columns=NAME:.metadata.name,CAPACITY_PODS:.status.capacity.pods,ALLOCATABLE_PODS:.status.allocatable.pods

This step is not required for Calico

If you’re near IP limits, enable prefix delegation on the VPC CNI before rolling Istio out broadly:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

0.2 Security groups

Istio control plane and sidecars communicate on ports in the 15000-15090 range plus 15021 (health check), 15012 (istiod XDS). Make sure your node security groups allow this traffic between nodes (not just node-to-control-plane):

• 15012/TCP — istiod XDS (nodes → istiod)
• 15017/TCP — webhook (if using in-cluster validation/injection webhook)
• 15021/TCP — sidecar health checks
• 15090/TCP — Envoy Prometheus stats

If your nodes’ security group already allows all traffic within the SG (common default for EKS managed node groups), you’re fine. If it’s locked down, add explicit rules.

0.3 Backup current state

kubectl get all -A -o yaml > pre-istio-cluster-state-$(date +%F).yaml

1. Install istioctl

Check https://istio.io/latest/docs/releases/supported-releases/ to confirm 1.30.3 is still on the supported (non-EOL) list, then:

export ISTIO_VERSION=1.30.3
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
cd istio-$ISTIO_VERSION
export PATH=$PWD/bin:$PATH 
istioctl version --remote=false

2. Define a production IstioOperator manifest

This uses the default profile as a base (not demo — demo is intentionally over-permissive and under-resourced for learning purposes) and layers in production settings: HA control plane, resource limits, a named revision, and PERMISSIVE mTLS to start.

yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-prod
spec:
  profile: default
  revision: prod-v1   # enables revision-based canary upgrades later

  meshConfig:
    accessLogFile: /dev/stdout
    enableAutoMtls: true
    defaultConfig:
      holdApplicationUntilProxyStarts: true   # avoid race where app traffic starts before sidecar is ready

  components:
    pilot:
      k8s:
        resources:
          requests:
            cpu: 500m
            memory: 2048Mi
          limits:
            cpu: "1"
            memory: 4096Mi
        hpaSpec:
          minReplicas: 3   # capped to current node count — see Appendix A, hostNetwork needs 1 istiod per node
          maxReplicas: 5
          metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 80
        podDisruptionBudget:
          minAvailable: 1
        overlays:
        - apiVersion: apps/v1
          kind: Deployment
          name: istiod-prod-v1
          patches:
          - path: spec.template.spec.hostNetwork
            value: true
          - path: spec.template.spec.dnsPolicy
            value: ClusterFirstWithHostNet
          - path: spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution
            value:
            - labelSelector:
                matchLabels:
                  app: istiod
              topologyKey: kubernetes.io/hostname

    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        replicaCount: 3
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
          limits:
            cpu: "1"
            memory: 512Mi
        service:
          type: ClusterIP   # ALB will target this directly; no separate ELB needed
        podDisruptionBudget:
          minAvailable: 1

  values:
    global:
      proxy:
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 256Mi
    pilot:
      autoscaleEnabled: true

Key production choices explained:

• replicaCount: 3 for pilot/istiod — spreads across AZs, survives node/AZ loss. Pair with pod anti-affinity if you want to guarantee AZ spread (add under affinity in the same k8s block).
• revision: prod-v1 — every future Istio upgrade installs as a new revision (prod-v2, etc.) alongside the old one, and you migrate namespaces one at a time. This is the standard safe-upgrade pattern; skipping it means in-place upgrades with no easy rollback.
• ingressGateways as ClusterIP — since ALB Controller will front this, you don't want Istio also provisioning a Classic ELB. ALB will route to the ingressgateway pods directly via IP target-type.
• holdApplicationUntilProxyStarts: true — prevents a common early failure mode where app containers start receiving traffic before their sidecar is ready.
• Global proxy resource limits — without these, sidecars use no explicit limits and can starve nodes under load. Tune based on your actual traffic; these are conservative starting points.
• spec.template.spec.hostNetwork — Not supported in the default network by default so we have to run the Istio pods in the hostNetwork.
• spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution — Defining the pod anti-affinity so all pods will schedule on different nodes for HA.

3. Install

# istioctl install -f istio-production.yaml -y
output will be like below. 
        |\          
        | \         
        |  \        
        |   \       
      /||    \      
     / ||     \     
    /  ||      \    
   /   ||       \   
  /    ||        \  
 /     ||         \ 
/______||__________\
____________________
  \__       _____/  
     \_____/        

This will install the Istio 1.30.3 profile "default" into the cluster. Proceed? (y/N) y
✔ Istio core installed ⛵️                                                                                                                                                                                 
- Processing resources for Istiod. Waiting for Deployment/istio-system/istiod-prod-v1

Verify:

kubectl get pods -n istio-system
kubectl get pods -n istio-system -l app=istiod --show-labels
istioctl analyze -A

#During analyze you can can erorr related to mutatingwebhookconfiguration. 
Delete the unnecssary mutatingwebhookconfiguration. 

kubectl delete mutatingwebhookconfiguration <name>

You should see istiod pods labeled with istio.io/rev=prod-v1.

4. Set mesh-wide PERMISSIVE mTLS (safe starting point)

yaml

# peerauth-mesh-permissive.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: PERMISSIVE

bash

kubectl apply -f peerauth-mesh-permissive.yaml

#kubectl get peerauthentication -A
NAMESPACE NAME MODE AGE
default default PERMISSIVE 21h

PERMISSIVE means: sidecars accept both plaintext and mTLS. This lets you onboard namespaces gradually without breaking traffic from not-yet-meshed pods.

5. Onboard a namespace (repeat per namespace, don’t do all at once)

# Because we used a revision, label with the revision name, not istio-injection=enabled
kubectl label namespace default istio.io/rev=prod-v1

kubectl get namespace default  --show-labels
NAME      STATUS   AGE     LABELS
default   Active   5y81d   istio.io/rev=prod-v1,kubernetes.io/metadata.name=default

Roll workloads to pick up sidecars: If anything is running.

kubectl rollout restart deployment -n default

Check every pod got 2/2 containers ready:

kubectl get pods -n default

Do this namespace by namespace, verify app health and error rates after each, before moving to the next. This is the single biggest risk-reduction step for a production rollout — don’t inject the whole cluster in one shot.

Get Naresh Sharma’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

To remove the label if you don’t want to use istio.

kubectl label namespace default istio.io/rev-

kubectl get namespace default  --show-labels
NAME      STATUS   AGE     LABELS
default   Active   5y81d   kubernetes.io/metadata.name=default

6. Once a namespace is fully onboarded: lock mTLS to STRICT for it

yaml

# peerauth-namespace-strict.yaml
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT

bash

kubectl apply -f peerauth-namespace-strict.yaml

kubectl get peerauthentication -A
NAMESPACE NAME MODE AGE
default default STRICT 21h

After all namespaces with mesh traffic are STRICT, you can optionally flip the mesh-wide default in istio-system to STRICT and remove the per-namespace overrides — but this is a nice-to-have, not urgent.

Let's start with the nginx pods and test the Istio flow.

ALB Ingress (istio-system)

→ istio-ingressgateway-nodeport Service (istio-system)

→ istio-ingressgateway pod’s Envoy

→ routes based on Host header, via Gateway (istio-system) +

VirtualService (default)

→ nginx Service (default)

→ nginx pod (default)

Architected multi-AZ EKS routing diagram with namespace boundaries

The Ingress and Gateway both live in istio-system because they're shared infrastructure serving every app. The VirtualService lives in default (or wherever your app is) because it's app-specific routing config. Your app's namespace never needs its own Ingress or Gateway — just a VirtualService plus a new host line added to the existing shared Ingress.

So , to directly confirm: one Ingress in,istio-systemreused and extended (new host rules) as you onboard more apps — never one per app namespace.

Let’s create the Nginx manifest and apply it.

#nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80

kubeclt apply -f nginx-deployment.yaml

kubectl get pod 
nginx-6f9664446b-ggnct      2/2     Running   0          75m

#envoy proxy sidecar is running with the nginx. 

Create Istio Gateway and virtual service.

#istio-gateway.yaml
apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: main-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "istio-nginx.example.io"
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: nginx
  namespace: default
spec:
  hosts:
  - "istio-nginx.example.io"
  gateways:
  - istio-system/main-gateway
  http:
  - route:
    - destination:
        host: nginx.default.svc.cluster.local
        port:
          number: 80

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway-nodeport
  namespace: istio-system
spec:
  type: NodePort
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    - name: http2
      nodePort: 30080 # ALB will target this node port
      port: 80
      targetPort: 8080

Now apply the manifest YAML.

kubectl apply -f istio-gateway.yaml

Validate gateway

kubectl get gateway -n istio-system

NAME           AGE
main-gateway   15h

Validate service,

kubectl saas get virtualservice
NAME    GATEWAYS                        HOSTS                        AGE
nginx   ["istio-system/main-gateway"]   ["istio-nginx.example.io"]   14h

#can run kubectl saas get virtualservice -oyaml to see more options. 

Now create an Ingress as an ALB.

# alb-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginxx-istio-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/group.name: group-internal
    alb.ingress.kubernetes.io/success-codes: 200,302,307
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:xxxxx:certificate/34ab925-0d6c-42fa-a700-23239b2324
    alb.ingress.kubernetes.io/security-groups: sg-024bc3fcf6
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01
    alb.ingress.kubernetes.io/subnets: subnet-3e974e63 , subnet-ba691433
    alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
spec:
  rules:
  - host: istio-nginx.example.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: istio-ingressgateway-nodeport
            port:
              number: 80

saas get ing  -nistio-system |grep nginx
nginxx-istio-ingress   <none>   istio-nginx.example.com. internal-k8s-xxxx-0a87ffdfee-.ap-south-1.elb.amazonaws.com   80      14h

Now point your domain to the ALB endpoint from browser.

Verify the logs to see whether traffic is routing via Istio or not; you can see the logs here.

kubectl logs -f  -n istio-system -l app=istio-ingressgateway  --tail=30

[2026-08-14T10:07:17.722Z] "GET /healthz/ready HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "172.23.x.159" "ELB-HealthChecker/2.0" "1fd95f8c-2007-41c9-8a7b-2cf53ec77e45" "172.23.x.159:30080" "-" - - 10.200.96.194:8080 172.23.x..159:60676 - -
[2026-08-14T10:07:26.081Z] "GET / HTTP/1.1" 304 - via_upstream - "-" 0 0 273 273 "10.100.30.2,10.200.67.192" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "c267506f-b94b-4458-8643-912b92093cd1" "istio-nginx.paytmpb.io" "10.200.67.194:80" outbound|80||nginx.default.svc.cluster.local 10.200.96.194:40066 10.200.96.194:8080 10.200.67.192:12468 - -
[2026-08-14T10:07:27.074Z] "GET /healthz/ready HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.200.67.192" "ELB-HealthChecker/2.0" "11117f0e-df1c-40fc-88df-a30d64faf8dd" "172.23.131.192:30080" "-" - - 10.200.96.194:8080 10.200.67.192:15184 - -
[2026-08-14T10:07:27.095Z] "GET /healthz/ready HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.200.13.192" "ELB-HealthChecker/2.0" "fc29b449-98ef-48a8-94f1-0622edcaa67c" "172.23.131.36:30080" "-" - - 10.200.96.194:8080 10.200.13.192:62328 - -

Verify mTLS.

Let’s start with peer authentication permissive.

#kubectl get peerauthentication -A
NAMESPACE NAME MODE AGE
default default PERMISSIVE 18

Let’s run the netdebug pod and test.

apiVersion: v1
kind: Pod
metadata:
  name: netdebug
spec:
  containers:
  - name: netdebug
    image: nicolaka/netshoot:latest
    command: ["sleep", "3600"]
  restartPolicy: Never

kubectl exec -it netdebug   bash

From the pod IP.

From the service endpoint.

It’s working from the pod IP and the service as well.

Let’s run in strict mode.

#kubectl apply -f peerauth-namespace-strict.yaml
peerauthentication.security.istio.io/default configured

Check the config for the peerauthenication.

#kubectl get peerauthentication -A
NAMESPACE   NAME      MODE     AGE
default     default   STRICT   4m54s

Getting an error from the pod IP.

Working using the service endpoint.

Test from the worker node; it's working from Istio Envoy.

 curl -v -H "Host: istio-nginx.paytmpb.io" http://172.23.131.159:30080/
*   Trying 172.23.131.159:30080...
* Established connection to 172.23.131.159 (172.23.131.159 port 30080) from 172.23.131.159 port 38886 
* using HTTP/1.x
> GET / HTTP/1.1
> Host: istio-nginx.paytmpb.io
> User-Agent: curl/8.17.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< server: istio-envoy
< date: Fri, 14 Aug 2026 10:57:08 GMT
< content-type: text/html
< content-length: 896
< last-modified: Wed, 15 Jul 2026 16:03:14 GMT
< etag: "6a57af42-380"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 265
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.
Further configuration is required for the web s

Let’s try some other ways to test the mTLS.

istioctl x describe pod nginx-6f9664446b-ggnct

This is the most useful single command — it tells you plainly whether the pod is in the mesh, what mTLS mode applies to it (after considering both mesh-wide and namespace-level policy), and flags any conflicting DestinationRules that might override client-side behavior.

Confirm the sidecar has valid certificates issued

[naresh1.sharma@mgmt-server istio]$ istioctl proxy-config secret nginx-6f9664446b-ggnct.default
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           f647a7ab3c5dd04144244f92c0d90109     2026-08-15T08:47:07Z     2026-08-14T08:45:07Z
ROOTCA            CA             ACTIVE     true           9cf0a557906b950b8df4145d3c801c42     2036-08-10T18:19:24Z     2026-08-13T18:19:24Z

Look for a valid, non-expired default cert (the workload identity cert) and ROOTCA. If these are missing or expired, mTLS can't work regardless of what PeerAuthentication says.

To find out which one is actually being used right now

#istioctl proxy-config cluster nginx-6f9664446b-ggnct.default --fqdn nginx.default.svc.cluster.local -o json | grep -A10 transportSocket
        "transportSocketMatches": [

[naresh1.sharma@mgmt-server istio]$ istioctl proxy-config cluster nginx-6f9664446b-ggnct.default --fqdn nginx.default.svc.cluster.local -o json | grep -A10 transportSocket
        "transportSocketMatches": [
            {
                "name": "tlsMode-istio",
                "match": {
                    "tlsMode": "istio"
                },
                "transportSocket": {
                    "name": "envoy.transport_sockets.tls",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                        "commonTlsContext": {
                            "tlsParams": {
                                "tlsMinimumProtocolVersion": "TLSv1_2",
                                "tlsMaximumProtocolVersion": "TLSv1_3"
                            },
                            "tlsCertificateSdsSecretConfigs": [
                                {
--
                "transportSocket": {
                    "name": "envoy.transport_sockets.raw_buffer",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
                    }
                }
            }
        ],
        "name": "outbound|80||nginx.default.svc.cluster.local",
        "altStatName": "outbound|80||nginx.default.svc.cluster.local;",
        "type": "EDS",

You can check tlsMode: istio; that means it’s working via Istio using mTLS.

Enjoy, and suggest if anything is missing, will add the monitoring part later.