Service Mesh: A working guide to Istio

Every microservices team eventually hits the same wall. You’ve split the monolith into thirty services, and now nobody can answer simple questions: which service is calling which? Is that traffic encrypted? What happens if I send 5% of users to the new version?

A service mesh exists to answer those questions once, centrally, instead of forcing every team to solve them inside their own code. This guide walks through Istio — the most widely adopted service mesh — piece by piece, in the same order a hands-on walkthrough would: from the Kubernetes plumbing that makes it possible, up through encryption and canary traffic shifting.

Admission controllers: the on-ramp Istio rides in on

Before Istio can do anything, it needs a way to slip its own proxy into every pod you deploy — without you editing a single YAML file. That mechanism is a Kubernetes primitive called an admission controller.

An admission controller sits between “the API server accepted your request” and “the object gets written to etcd.” Every object you create — a Pod, a Deployment, a ConfigMap — passes through this checkpoint. Controllers here come in two flavors:

• Validating webhooks can only approve or reject an object. They enforce policy (“no pod may run as root”) but never change what you sent.
• Mutating webhooks can rewrite the object in flight, before it’s persisted. This is the one Istio depends on.

When you label a namespace istio-injection=enabled, Istio's mutating webhook starts watching every Pod-creation request in that namespace. It intercepts the request, splices an extra container definition into the pod spec — the Envoy sidecar — and lets the (now modified) request continue on to etcd. Your Deployment manifest never mentions Envoy at all; the mutation happens invisibly, in-flight.

# Turn on automatic sidecar injection for a namespace

kubectl label namespace default istio-injection=enabled

# Inspect Istio’s registered webhook

kubectl get mutatingwebhookconfiguration

Sidecars, and the proxy doing the actual work

A sidecar is just a second container riding along in the same pod as your application, sharing its network namespace. It’s a Kubernetes pattern, not an Istio invention — log shippers and metrics agents use the same trick. Istio’s sidecar happens to be Envoy, a proxy originally built at Lyft, written in C++ for low-latency L4/L7 traffic handling.

What makes this powerful is where Envoy sits: directly in the path of every byte the pod sends or receives. An init container configures iptables rules the moment the pod starts, so that all inbound and outbound traffic is transparently redirected through the local Envoy instance first — the application code has no idea this is happening; it still just opens a socket and talks to a hostname.

Because Envoy now sees every request, Istio can bolt on retries, timeouts, circuit breaking, encryption and telemetry — entirely outside your application code, and consistently across every language your services happen to be written in.

What a service mesh actually is :

Put the two ideas together — automatic injection plus a proxy in front of every workload — and you have the shape of a service mesh: a dedicated, application-agnostic layer for service-to-service communication, made of two planes.

• Data plane — the fleet of Envoy sidecars, one per pod, actually moving bytes.
• Control plane — istiod, the single brain that compiles your routing rules and certificates and pushes configuration down to every sidecar in real time.

Nothing here requires touching application code. You describe policy — “encrypt everything,” “send 10% of traffic to v2” — as Kubernetes objects, and the mesh enforces it uniformly.

Why and when — reach for a mesh

The pain a mesh solves only shows up once you have enough services that these questions stop being rhetorical:

• How do you encrypt every service-to-service call without touching a single codebase?
• How do you send 5% of traffic to a new version and watch it before committing further?
• When service A calls B and B fails, how do you actually trace that across the hop?
• How do you enforce the same retry and circuit-breaking policy everywhere, regardless of what language each service happens to be written in?

A mesh solves these once, in infrastructure, instead of everyone reinventing (and inevitably diverging on) the same logic per service.

Installing Istio :

The fastest path to a working mesh is Istioctl, Istio's own CLI, paired with the demo installation profile — which turns on most features at a resource footprint that's fine for a laptop cluster.

# download the CLI
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

# install the control plane + ingress/egress gateways
istioctl install --set profile=demo -y

# confirm istiod and the gateways are running
kubectl get pods -n istio-system

# opt a namespace into automatic sidecar injection
kubectl label namespace default istio-injection=enabled

Traffic management, conceptually:

Once every request flows through Envoy, routing decisions move out of your code and into declarative Kubernetes objects. Istio’s traffic-management layer covers request routing, weighted traffic shifting, fault injection for chaos testing, timeouts, retries, and circuit breaking — all configured, not coded. The two objects that do the heavy lifting are VirtualServices and DestinationRules, which is where the next two sections live.

Virtual services and destination rules :

These two CRDs divide the labor cleanly: a VirtualService decides where a request goes; a DestinationRule decides what happens once it gets there.

VirtualService: where the request goes

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
  - route:
    - destination: {host: reviews, subset: v1}
      weight: 90
    - destination: {host: reviews, subset: v3}
      weight: 10

DestinationRule: what happens once it arrives

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews
spec:
  host: reviews
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      clientCertificate: /etc/certs/myclientcert.pem
      privateKey: /etc/certs/client_private_key.pem
      caCertificates: /etc/certs/rootcacerts.pem
  subsets:
  - {name: v1, labels: {version: v1}}
  - {name: v3, labels: {version: v3}}

DestinationRules also carry resilience policy — for example, outlier detection that ejects a misbehaving pod from the load-balancing pool automatically:

trafficPolicy:
  outlierDetection:
    consecutive5xxErrors: 1
    interval: 1s
    baseEjectionTime: 3m
    maxEjectionPercent: 100

mTLS: giving every service an identity

Mutual TLS means both sides of a connection present a certificate — not just the server proving who it is to the client (ordinary TLS), but the client proving who it is to the server too. In Istio, istiod is the certificate authority: it mints and rotates a short-lived certificate for every sidecar automatically, so no human ever touches a private key.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-ca-root
  namespace: istio-system
data:
  root-cert.pem: |
    -----BEGIN CERTIFICATE-----
    # Shared root CA certificate
    -----END CERTIFICATE-----

Apply that, then try calling a meshed service with plain curl from a pod that has no sidecar — you'll get a connection reset. That failure is the proof: the server's sidecar is refusing anything that isn't mutually authenticated.

kubectl apply -f peer-authentication-strict.yaml

# check what istiod believes the TLS posture is for a workload
istioctl x describe pod <pod-name>

Canary releases, done with a config change

This is the payoff of sections 7 and 8 combined: because routing weight lives entirely in a VirtualService object, shipping a new version to a slice of traffic is a kubectl apply, not a redeploy.

1. Deploy v1 and v2 side by side, each carrying a distinct version label.
2. Define subsets v1 and v2 in a DestinationRule, mapped to those labels.
3. Start the VirtualService conservatively — 95% v1 / 5% v2.
4. Watch error rate and latency for v2 in Kiali or Grafana.
5. Re-apply the VirtualService with a heavier v2 weight, repeatedly, as confidence builds.
6. At 100% v2, decommission the v1 Deployment.

# stage 2: shift 25% of traffic to v2
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
  - route:
    - destination: {host: reviews, subset: v1}
      weight: 75
    - destination: {host: reviews, subset: v2}
      weight: 25
EOF

Circuit breaking & other resilience patterns

“Circuit breaking” gets thrown around loosely in service-mesh talks, often borrowing the mental model from Hystrix-style application libraries: a circuit that’s closed (traffic flows), trips open (traffic blocked) after enough failures, then goes half-open to test recovery. Envoy — and therefore Istio — doesn’t implement that state machine at all. It gets the same practical outcome from two separate, simpler mechanisms working together, and knowing which one is doing what matters when something’s misbehaving in production.

Mechanism 1: connection pool limits — failing fast on purpose

A DestinationRule can cap how much concurrency a client sidecar is allowed to send to a given service:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 10
        maxRequestsPerConnection: 2

Once a client has 100 TCP connections and 10 queued requests outstanding to payment-service, the next request doesn't queue up behind them — Envoy rejects it immediately with a 503. That's the whole mechanism: a hard ceiling that turns "wait and hope" into "fail immediately," so one slow downstream service can't exhaust threads and sockets in every service that calls it and drag the rest of the mesh down with it.

Mechanism 2: outlier detection — passive health checking

Separately, each client sidecar watches the responses it gets back from every individual backend endpoint (not the service as a whole — each pod behind it) and can eject a specific misbehaving endpoint from its own load-balancing pool for a while:

trafficPolicy:
  outlierDetection:
    consecutive5xxErrors: 5
    interval: 30s
    baseEjectionTime: 30s
    maxEjectionPercent: 50

Read as: check every 30 seconds; if an endpoint has thrown 5 consecutive 5xx responses, pull it out of rotation for 30 seconds; never eject more than half the pool at once, so a bad rollout can’t take every replica out simultaneously. After the ejection period, the endpoint quietly goes back into rotation — if it fails again immediately, it’s re-ejected, typically for a longer period. That re-admission-and-watch behavior is what plays the role a “half-open” state would in the classic model.

Timeouts and retries — the pieces that decide whether a failure is even visible

Get Suresh Sk’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Circuit breaking limits how much traffic goes to a struggling service. Timeouts and retries decide how a single request behaves when things are slow or flaky — and get the defaults wrong, and they can make an outage worse, not better.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts: [payment-service]
  http:
  - timeout: 2s
    retries:
      attempts: 2
      perTryTimeout: 500ms
      retryOn: 5xx,reset,connect-failure
    route:
    - destination: {host: payment-service}

Fault injection — testing that the above actually works

None of this is trustworthy until you’ve deliberately broken something and watched the mesh handle it. Istio can inject synthetic failures at the routing layer, with no code changes to the target service:

Ingress gateways vs. Kubernetes Ingress

A Kubernetes Ingress controller only has one job: get external traffic into the cluster. An Istio Gateway does that too — but because it’s backed by the same Envoy proxies as the rest of the mesh, external (“north-south”) traffic gets the identical routing, security, and observability machinery that internal (“east-west”) traffic already has.

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port: {number: 80, name: http, protocol: HTTP}
    hosts: ["*"]

Seeing it all: observability with Kiali

Because every hop passes through Envoy, telemetry falls out of the mesh almost for free — no code changes, no manually wired tracing libraries. Kiali is Istio’s console for this: a live graph of every service in the mesh, colored by health, with the routing config (VirtualServices, DestinationRules, Gateways) drawn directly onto the topology.

# install the bundled observability add-ons
kubectl apply -f samples/addons
kubectl rollout status deployment/kiali -n istio-system

# open the dashboard
istioctl dashboard kiali

North-south vs. east-west, precisely

These two terms get used loosely, but they describe genuinely different traffic — different threat models, different scaling behavior, different failure modes. Getting the distinction precise matters because a mesh secures and observes both, but not in the same way.

• North-south is anything crossing the boundary of your cluster or mesh — a browser, a partner API caller, or another cluster entirely, coming in through an ingress gateway. It’s comparatively low-volume, high-scrutiny traffic: this is where you terminate TLS, check a WAF, rate-limit by client, and authenticate an external caller.
• East-west is service-to-service traffic that never leaves the mesh — Order Service calling paymentservice calling ledgerservice. It's high-volume and high-fanout: one external request can turn into a dozen internal calls, so the overhead per hop matters far more than at the edge.

Identity: the thing mTLS actually checks

Istio doesn’t authenticate an IP address or a hostname — it authenticates a SPIFFE identity, tied to the workload’s Kubernetes service account.

spiffe://cluster.local/ns/payments/sa/payment-service

That URI — trust domain, namespace, service account — is what’s actually embedded in the workload’s certificate, and it’s what every policy decision keys off:

• A pod’s identity comes from its ServiceAccount, not its IP or hostname. Kill the pod, reschedule it to a different node with a new IP, and its identity is unchanged — because it never depended on the network location in the first place.
• istiod is the certificate authority. When a workload's sidecar (or, in ambient mode, the node's ztunnel) starts, it requests a certificate over the Secret Discovery Service; istiod validates the request against the Kubernetes API (does this pod really run under this service account?) and signs a short-lived cert — typically valid for hours, rotated automatically, no human ever handles a key.
• AuthorizationPolicy is where identity becomes an actual access-control decision: principal: cluster.local/ns/payments/sa/payment-service in a rule means "only this specific workload identity may call this endpoint" — enforceable even if an attacker spoofs a source IP, because the check happens on the cryptographically verified identity, not the packet header.

Handling more than one region:

Everything so far assumes one cluster in one region. Once you need resilience against a whole region going dark, two separate problems appear, and it’s worth keeping them distinct: getting a client’s traffic to a healthy region at all, and letting workloads in different regions talk to each other once they’re there.

Getting traffic to a healthy region

The common pattern is a health-checked DNS routing policy in front of a per-region internal load balancer:

1. Each region runs its own complete stack — VPC/subnet, internal load balancer, ingress gateway, mesh — often called a cell. Cells are independent failure domains: a control-plane bug or a bad deploy in one region’s cell doesn’t touch the other.
2. A DNS record (with a deliberately short TTL, commonly 30–60s) sits in front of all the cells, using a health-check-driven or geo routing policy.
3. A health check against each cell’s load balancer (like the /health/composite pattern) determines whether that region is eligible to receive traffic at all.
4. When the primary region’s health check fails, DNS starts resolving to the backup region’s IP instead — clients pick this up on their next lookup, bounded by the TTL.

The DNS-and-cells pattern above treats each region as fully independent — good for blast-radius isolation, but it means two regions don’t share a mesh at all. If you actually need workloads in one region to call workloads in another (not just fail over between them), that’s a different problem: multi-cluster mesh federation

• Multi-primary, multi-network — every region runs its own full istiod, but they share one root of trust, so workloads in different clusters can still mTLS-authenticate each other's SPIFFE identities. An east-west gateway in each cluster carries cross-region traffic over the public internet, still encrypted.