Istio, Hands-On, Part 4: Reading the Configuration Envoy Actually Received

Listeners, routes, clusters and endpoints as istioctl proxy-config shows them, the configuration Envoy refuses while kubectl lists it, and EnvoyFilter as the last resort

Part 4 of the series Istio, Hands-On: From the First Route to Multi-Cluster. Part 1 covered where requests go and who decides, Part 2 time budgets, failure and egress. Part 3 covered identity, mTLS and authorization.

Four layers of configuration between a YAML file and a request.

The change for the payments platform was one VirtualService. Send every call to fraud-check to its v2 pods for a day, while the team watched the new model. It passed review. kubectl apply printed created and kubectl get virtualservice listed it. In my lab I sent twelve requests from payments-api afterward. All twelve went through the route called default to the cluster with no subset, which means all three fraud-check pods, v1 included. Applying the file again changes nothing, because the object in the API server is already what the file says. istioctl analyze complained about a missing port (IST0112) and said nothing about the host. The answer was in the proxy the whole time. The host in the manifest read fraud-check.payment.svc.cluster.local, one letter short of the namespace. istiod had not dropped it. It had built a virtual host for that name and put it in a route configuration for port 80 that no request to fraud-check ever consults. The route inside pointed at a cluster that does not exist.

Every failure in the first three parts of this series surfaced the same way, as a short code in an Envoy access log. NR for a missing route, UH for a cluster with no healthy endpoint, URX for retries exhausted, filter_chain_not_found for a plain-text caller refused under strict mTLS, rbac_access_denied_matched_policy for an authorization policy that said no. We read those codes without opening the proxy that wrote them. That works until the proxy does something the YAML does not describe. Then the usual reflexes kick in. Someone applies the file again, restarts the pod, or pastes an EnvoyFilter from a forum thread.

Every Istio resource becomes Envoy configuration. When traffic does something unexpected, the answer is in what the proxy received. A request leaving a sidecar crosses four layers of that configuration, a listener, a route, a cluster and an endpoint. Each symptom from Parts 1 to 3 lives in one of them. Reading those layers with istioctl proxy-config comes before changing anything. EnvoyFilter, which rewrites the generated configuration, is the last resort, because it binds itself to internal names that change on upgrade and fails without a word when it does not match. When traffic disagrees with a manifest, the first artifact I ask for is the route configuration of the caller's sidecar. The second is the sync status of that proxy.

The article starts with the vocabulary and follows one request through both sidecars. Then come sync status, a configuration Envoy refuses while the API server accepts it, logs, stats and EnvoyFilter. It ends in ambient mode, where only one of the two proxies is Envoy. Everything was measured on Istio 1.31.2 in the companion repository, whose README holds the full outputs that the article only summarizes.

What istiod Sends and What Envoy Keeps

Envoy’s own terminology page gives the two definitions everything else hangs on. “A listener is a named network location (e.g., port, unix domain socket, etc.)” that accepts connections from downstream clients. “A cluster is a group of logically similar upstream hosts that Envoy connects to.” Between them sits the configuration that decides which cluster a request belongs to. Below a cluster sit its endpoints, the addresses of the individual hosts.

A listener does not handle every connection the same way. Envoy’s listener overview says that “each listener is independently configured with filter_chains, where an individual filter_chain is selected based on its filter_chain_match criteria,” such as the destination port, whether the connection starts with a TLS handshake, or the application protocol it negotiates. “An individual filter_chain is composed of one or more network level (L3/L4) filters.” For HTTP traffic the important network filter is the HTTP connection manager, which Envoy describes as a “built in network level filter” that “translates raw bytes into HTTP level messages and events.” I will call it the HCM. It runs its own chain of HTTP filters on each request. Istio puts a handful there, among them metadata exchange, stats, RBAC when an authorization policy exists, and, always last, the router, which picks a route and sends the request upstream.

The router consults a route configuration. In Envoy’s words, “the top level element in the routing configuration is a virtual host,” with a name and a list of domains matched against the request’s Host header. Each virtual host holds an ordered list of routes. Each route matches on path, headers or other attributes and names a cluster. Istio’s clusters for Kubernetes Services are of type EDS, where Envoy fetches the members from the control plane. A ServiceEntry resolved by DNS becomes a strict DNS cluster, where "Envoy will continuously and asynchronously resolve the specified DNS targets."

In a mesh, istiod generates all of this and pushes it to every proxy over xDS, Envoy’s family of discovery APIs. Envoy’s protocol page names the services after the resource each one carries. The Listener Discovery Service (LDS) carries listeners, the Route Discovery Service (RDS) route configurations, the Cluster Discovery Service (CDS) clusters, the Endpoint Discovery Service (EDS) endpoints, and the Secret Discovery Service (SDS) certificates and keys. Two more families show up in Istio’s tools. ECDS is Envoy’s extension config discovery, through which “extension configuration can be supplied dynamically from an xDS management server,” filter configuration included. The workload types are Istio’s own, used mostly by ztunnel. In istiod’s source, WDS covers workloads and addresses, two resource types under one short name. WADS covers the authorization policies ztunnel enforces. istiod is the server and each proxy is a client that subscribes to the types it needs.

Every response istiod sends gets an answer. The proxy either acknowledges the version (ACK) or rejects it (NACK). In the protocol’s words, “a NACK is indicated by the presence of the error_detail field.” The client replies “with error_detail populated and its previous version,” which, as the lab shows later, means the proxy keeps running the version before it. istiod tracks the last version it sent to each proxy and the last one acknowledged. istioctl proxy-status reports the difference.

Istio’s resources map onto these layers in a regular way. A VirtualService becomes routes inside virtual hosts. A DestinationRule adds one cluster per subset and sets traffic policy on clusters. A ServiceEntry adds clusters, plus a listener where the port needs one. A PeerAuthentication changes the inbound filter chains. An AuthorizationPolicy adds the RBAC filter. The next two sections measure each change.

The reading tool is istioctl proxy-config, "a group of commands used to retrieve information about proxy configuration from the Envoy config dump," with subcommands that follow the layers, listener, route, cluster, endpoint and secret. The data comes from the proxy itself, so it shows what Envoy accepted, which is not always what istiod generated.

The lab is the payments platform of the earlier parts, on Istio 1.31.2 with the ambient profile, whose sidecars run Envoy 1.39. payments-api calls fraud-check, which runs as two Deployments behind one Service, version: v1 with two pods and version: v2 with one. A second port on fraud-check, 8090, forwards each request to risk-score through fraud-check's own sidecar. A Gateway API Gateway sits in front and calls to api.github.com exercise egress. A second namespace, payments-ambient, runs payments-api and fraud-check in ambient mode behind a waypoint. That makes nine proxies streaming from istiod, five sidecars, the ingress gateway, the waypoint and one ztunnel per node, seven of them Envoy.

Who calls whom in the lab, with istiod pushing configuration to every proxy.

To start, pick one caller in your mesh and one service it calls. Write down which port the caller uses. That pair is all the next section needs.

Follow One Request Through Both Sidecars

Inside a sidecar pod, iptables rules redirect outbound connections to the proxy. Istio’s diagnostic guide describes the entry point as “a listener on 0.0.0.0:15001 that receives all outbound traffic to the pod," which then "hands the request over to the virtual listener that best matches the original destination." In the lab that listener is named virtualOutbound, with useOriginalDst set to true, the Envoy setting that performs the hand-over. When no listener matches the original port, the connection stays there. The documentation shows one filter chain on it, going to PassthroughCluster, which connects to the original destination directly. On 1.31.2 it has two. virtualOutbound-catchall-tcp goes to PassthroughCluster. virtualOutbound-blackhole matches connections aimed at port 15001 itself and sends them to BlackHoleCluster, a cluster with no endpoints, so that the proxy never loops traffic into itself.

A request from payments-api to http://fraud-check:8080/ matches the listener 0.0.0.0_8080, one listener for all services on that port. Its HTTP filter chain matches plain-text HTTP/1.1 or h2c. Its HCM loads routes over RDS from the route configuration named 8080, which every HTTP service on 8080 shares, one virtual host each. In the lab, payments-api's copy held virtual hosts for fraud-check, payments-api and risk-score, plus one for fraud-check.payments-ambient, since a sidecar knows the ambient namespace's services too. The virtual host fraud-check.payments.svc.cluster.local:8080 answers to the short name fraud-check and to the full name, among others. With no VirtualService it carries one route, named default, which matches the prefix / and points at the cluster outbound|8080||fraud-check.payments.svc.cluster.local.

That cluster name is the string I look for first in any Istio log. It reads as direction, port, subset and host, separated by pipes, with an empty subset field when no DestinationRule defines one. Its type is EDS, and its endpoints, from proxy-config endpoint, were the three fraud-check pod IPs on port 8080, all HEALTHY, one of which the load balancer picks.

On the receiving side the iptables rules send the connection to the inbound listener, “a listener on 0.0.0.0:15006 that receives all inbound traffic to the pod," named virtualInbound, with filter chains matched on destination port and transport protocol. With no PeerAuthentication in the mesh, port 8080 had two, one for plain text (raw_buffer) and one for TLS, both named 0.0.0.0_8080 and both ending in the cluster inbound|8080||, which forwards to the application on the same port. The HTTP filters of those chains were istio.metadata_exchange, envoy.filters.http.grpc_stats, envoy.filters.http.fault, envoy.filters.http.cors, istio.stats and envoy.filters.http.router, with no RBAC filter, since no authorization policy existed.

One request, every object it crosses, by name.

The lab’s access log format prints those names as key=value pairs, so the logs of the two sidecars name the same objects. Here is one request from payments-api and the line each side wrote for it:

[2026-10-17T00:12:46.982Z] "GET /r17921959662488 HTTP/1.1" 200 flags=- details=via_upstream cluster=outbound|8080||fraud-check.payments.svc.cluster.local statname=outbound|8080||fraud-check.payments.svc.cluster.local; route=default upstream=10.244.1.9:8080 authority=fraud-check:8080 sni=- dst=10.96.25.161:8080
[2026-10-17T00:12:46.987Z] "GET /r17921959662488 HTTP/1.1" 200 flags=- details=via_upstream cluster=inbound|8080|| statname=inbound|8080||; route=default upstream=10.244.1.9:8080 authority=fraud-check:8080 sni=outbound_.8080_._.fraud-check.payments.svc.cluster.local dst=10.244.1.9:8080

The caller’s line has the route default, the outbound cluster and the endpoint it chose, 10.244.1.9:8080, while dst is the ClusterIP the application dialed. The receiver's line has the inbound cluster. Its dst is the pod IP the caller picked. Its sni shows that the hop between the sidecars was mutual TLS, with a server name derived from the outbound cluster name, so the request arrived on the TLS chain.

The subtle part sits in the statname field, which is Envoy's %UPSTREAM_CLUSTER%. Envoy documents it as "upstream cluster to which the upstream host belongs to. alt_stat_name will be used if provided." Istio provides one on every cluster, the cluster name plus a semicolon, so the field ends in ;. The real name is %UPSTREAM_CLUSTER_RAW%, which "alt_stat_name does NOT modify," and which Istio's default access log format uses. A custom format built from Envoy's documentation, with %UPSTREAM_CLUSTER%, produces names that never match proxy-config cluster. A query joining the two silently finds nothing. Envoy stats carry the same semicolon, as in cluster.outbound|8080||fraud-check.payments.svc.cluster.local;.upstream_rq_200. The lab logs both fields on purpose, as cluster= and statname=.

To start, run the five proxy-config commands of step two of the POC against your own caller and service, in that order. Check that the cluster your route names exists in the cluster list and has endpoints. Then find the same request in both sidecars' logs. scripts/02-follow-request.sh asserts each name above, read-only.

Each Istio Resource Changes One Layer

Knowing the layers, the next question is which one a given manifest touches. Script 03 of the lab applies one resource at a time and diffs the proxy’s configuration before and after, by object name.

A DestinationRule with subsets v1 and v2 added four clusters to payments-api's proxy and removed nothing. They were outbound|8080|v1|..., outbound|8080|v2|... and the same two for port 8090, one per subset per port. The v1 cluster had two endpoints and the v2 cluster one, the pods whose labels match each subset. The route did not change. It still pointed at outbound|8080||fraud-check.payments.svc.cluster.local, the cluster with no subset. A DestinationRule alone routes nothing. It creates destinations a route can name.

A VirtualService names them. The lab's translation/20-virtualservice-canary.yaml sends requests carrying a canary header to v2 and everything else to v1:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: fraud-check
  namespace: payments
spec:
  hosts:
  - fraud-check.payments.svc.cluster.local
  http:
  - name: canary
    match:
    - port: 8080
      headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: fraud-check.payments.svc.cluster.local
        port:
          number: 8080
        subset: v2
  - name: stable
    match:
    - port: 8080
    route:
    - destination:
        host: fraud-check.payments.svc.cluster.local
        port:
          number: 8080
        subset: v1

In the proxy, this replaced the route default with two routes, canary and stable, in that order, inside the same virtual host of route configuration 8080. The name of each HTTP route in the VirtualService becomes the route name in Envoy and, from there, the route= field of every access log line. A request with x-canary: true logged route=canary and the v2 cluster. One without it logged route=stable and the v1 cluster. Name your HTTP routes. An unnamed route shows up in logs as something you have to decode.

The port is explicit because fraud-check exposes two. Without port: 8080 in the match and the destination, istiod generated the same canary and stable routes for port 8090 too. It also generated them for route configuration 80, pointing at outbound|80|v1|..., a cluster that does not exist because the Service has no port 80. istioctl analyze caught that one as IST0112, a destination without a port on a multi-port Service. On a single-port Service the same manifest is fine, which is how it ends up in production unchanged the day a second port is added.

A ServiceEntry for api.github.com on port 443, protocol TLS, resolved by DNS, added three objects, a listener 0.0.0.0_443, a filter chain on it matching the server name api.github.com, and a cluster outbound|443||api.github.com of type STRICT_DNS. The proxy does not parse TLS traffic, so the match happens on the SNI of the client's handshake. The access log line for the call is a TCP line with sni=api.github.com and no route.

A PeerAuthentication with mode STRICT on fraud-check touched only its inbound listener, two chains added and seven removed. For port 8080 the chains went from two, one of them plain text, to one, TLS only. The plain-text and passthrough chains of virtualInbound disappeared. That is the configuration behind Part 3's NR filter_chain_not_found, a plain-text connection arriving at a listener with no chain that matches it.

An AuthorizationPolicy with an ALLOW rule inserted envoy.filters.http.rbac into the inbound HTTP filter chains, after istio.metadata_exchange where present and before the router, in virtualInbound and in main_internal, an internal listener the sidecars of this mesh also carry. The policy appears inside the filter's configuration under a key built from its namespace, name and rule index, ns[payments]-policy[fraud-check-allow]-rule[0], the same string the access log prints in rbac_access_denied_matched_policy[...] when the policy refuses a request.

To start, when you review a mesh change, predict which layer it touches on which proxy, then check that layer before and after. A DestinationRule that changes no cluster or a VirtualService that changes no route on the caller is a finding, whatever kubectl says. scripts/03-translation.sh prints the diffs above.

Every Symptom Lives in One Layer

With the translation in hand, the codes of the earlier parts become addresses. Script 04 breaks each layer on purpose and reads the evidence. The order of the layers is the order I check them in.

The route layer fails when a request matches no route. The lab applied a VirtualService with the canary route and no fallback, broken/01-vs-no-default-route.yaml. A request without the header got 404. payments-api's own sidecar logged it:

[2026-10-17T00:13:14.929Z] "GET /r17921959942864 HTTP/1.1" 404 flags=NR details=route_not_found cluster=- statname=- route=- upstream=- authority=fraud-check:8080 sni=- dst=10.96.25.161:8080

Envoy’s description of NR is "no route configured for a given request in addition to 404 response code," and the empty cluster confirms the request never left the caller. No fraud-check sidecar logged anything. A 404 from a service that has never returned 404 is a route problem on the caller. The callee's logs will only cost you the first half hour of the incident.

The cluster layer fails when a route names a cluster with no healthy endpoint. broken/02-subset-without-pods.yaml adds a subset v3 that no pod carries and routes everything to it. istiod still created outbound|8080|v3|fraud-check.payments.svc.cluster.local, with zero endpoints. Requests got 503 with the body no healthy upstream, logged as flags=UH details=no_healthy_upstream route=v3-rollout. Envoy describes UH as "no healthy upstream hosts in upstream cluster in addition to 503 response code." Both shortcuts saw this one. istioctl x describe pod, which "reports the configuration objects that affect that pod," printed Matching subsets: v1 (Non-matching subsets v2,v3) for a v1 pod and a warning that the VirtualService routes to the non-matching subset v3. istioctl analyze -n payments reported IST0173, "The Subset v3 defined in the DestinationRule does not select any pods. Which may lead to 503 UH (NoHealthyUpstream)."

The hardest failures land nowhere. The VirtualService from the opening, broken/03-vs-wrong-host.yaml, is short enough to miss the typo:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: fraud-check-typo
  namespace: payments
spec:
  hosts:
  - fraud-check.payment.svc.cluster.local
  http:
  - name: all-to-v2
    route:
    - destination:
        host: fraud-check.payments.svc.cluster.local
        subset: v2

istiod treated fraud-check.payment.svc.cluster.local as a host of its own. With no Service of that name and no port, it got a virtual host fraud-check.payment.svc.cluster.local:80 in route configuration 80, with the route all-to-v2 pointing at outbound|80|v2|fraud-check.payments.svc.cluster.local, a cluster that does not exist. Route configuration 8080 was untouched, so the twelve requests of the opening went through default. istioctl analyze flagged the missing destination port (IST0112) and nothing about the host. The only place this mistake is visible is the route configuration of the caller, where the virtual host you expected to change has not changed.

A VirtualService can also land in no proxy at all. broken/04-vs-gateway-drops-mesh.yaml sets gateways to the Gateway API gateway of the ingress namespace. The VirtualService reference says that "if a list of gateway names is provided, the rules will apply only to the gateways," and the reserved name mesh is what covers sidecars. Leaving mesh out removed the rules from payments-api's route configuration. The ingress gateway did not get them either, since a VirtualService binds to Istio Gateway resources and not to Gateway API ones. Here analyze helped with IST0101, Referenced gateway not found: "payments-ingress/payments-gateway". It also raised IST0132 for the host not found in that gateway.

Egress, from Part 2, splits in a way the documentation blurs. With the mesh’s outboundTrafficPolicy at ALLOW_ANY, the default, a request to http://example.com matched a catch-all virtual host named allow_any and went to PassthroughCluster. With REGISTRY_ONLY, the same catch-all virtual host became block_all. The request got a 502 that the sidecar produced itself:

[2026-10-17T00:13:45.974Z] "GET /r17921960252208 HTTP/1.1" 502 flags=- details=direct_response cluster=- statname=- route=block_all upstream=- authority=example.com sni=- dst=172.66.147.243:80

No flag, no cluster and the details direct_response. The diagnostic guide's example cluster list includes BlackHoleCluster, the name engineers search for when egress is blocked. For HTTP that search finds nothing. Only the HTTPS call reached BlackHoleCluster. https://www.example.com failed with curl exit 35, the same exit code Istio's egress task shows. The sidecar wrote a TCP line with flags=UH cluster=BlackHoleCluster sni=www.example.com. api.github.com, declared by the ServiceEntry, kept answering 200.

Each symptom of Parts 1 to 3, mapped to the layer that produced it.

Part 3’s NR filter_chain_not_found and the RBAC filter's rbac_access_denied_matched_policy complete the map. Both are decided on the callee's inbound listener, while routes and clusters are decided on the caller, and that split alone tells you which pod's logs to open first.

To start, keep a map in your runbook from each flag to its layer and side. NR with route_not_found on the caller is a route, UH a cluster without endpoints, NR with filter_chain_not_found on the callee a listener, an RBAC detail a filter, and direct_response on block_all egress policy. Run istioctl analyze in CI with IST0173, IST0101 and IST0112 as errors. scripts/04-broken-by-layer.sh reproduces each case.

Is the Proxy Up to Date?

Everything so far assumes the proxy holds what istiod generated. istioctl proxy-status checks that assumption. The diagnostic guide defines its states. "SYNCED means that Envoy has acknowledged the last configuration Istiod has sent to it." "NOT SENT means that Istiod hasn't sent anything to Envoy." "STALE means that Istiod has sent an update to Envoy but has not received an acknowledgement."

The page shows one column per type, CDS, LDS, EDS and RDS. istioctl 1.31.2 prints something else, a default table with the columns NAME, CLUSTER, ISTIOD, VERSION and SUBSCRIBED TYPES, the last one a count and a list, such as 5 (CDS,LDS,EDS,RDS,WDS) for a sidecar. The sync state per type appears only with -v 1, which the reference documents as "1=show all xDS types." Two states the page does not list showed up there. ERROR marks a type whose last version the proxy rejected. IGNORED marks a type the proxy never subscribed to. The subscribed types are a diagnostic of their own. The waypoint subscribed to four, CDS,LDS,EDS,WDS, with no RDS, because its routes are inline in its listeners. Each ztunnel subscribed to two, WADS,WDS, Istio's workload types and no Envoy types at all.

Per its release note, istioctl 1.31.2 no longer skips “verification of the istiod certificate when connecting to XDS with a service account token,” so in the lab a wrong --authority made proxy-status fail with tls: failed to verify. A script that breaks after the upgrade should get a fixed address before it gets --insecure.

STALE is hard to catch on a healthy cluster, since acknowledgements take milliseconds. To show it, the lab freezes one Envoy. The images Istio 1.31.2 deployed were distroless, proxyv2:1.31.2-distroless for the sidecar, so there is no shell in istio-proxy to send a signal from. Script 05 uses kubectl debug with --target=istio-proxy to attach an ephemeral busybox container to the proxy's process namespace, sends SIGSTOP to Envoy, sleeps 60 seconds and sends SIGCONT. During the pause the lab applied the DestinationRule from the translation section. proxy-status -v 1 showed:

NAME                                                        CLUSTER        CDS             ECDS        EDS              LDS             RDS             WADS             WDS              WDS              ISTIOD                      VERSION
payments-api-67b685b75f-zsqd5.payments                      Kubernetes     STALE (0s)      IGNORED     STALE (0s)       STALE (0s)      STALE (0s)      IGNORED          IGNORED          SYNCED (10s)     istiod-779854df95-j8g69     1.31.2

Every other proxy acknowledged the same push. After the pause, CDS returned to SYNCED and the subset clusters were there, two endpoints and one. A frozen Envoy serves no traffic, so the lab shows the state and nothing about serving. For a proxy that is slow rather than stopped, starved of CPU or behind an overloaded istiod, the protocol gives the answer. Until it applies and acknowledges the new version, it has only the last configuration it accepted and keeps routing by that one. A route that "did not apply" on one pod and applied on its siblings deserves a look at this column before anything else.

To start, add istioctl proxy-status -v 1 to the first five minutes of every traffic incident and look for anything other than SYNCED and IGNORED. Scripts that parse the old per-type columns need -v 1 on 1.31.

The Configuration Envoy Refuses

STALE means an answer has not arrived. ERROR means the answer was no. The lab produced one with an EnvoyFilter that a lot of people write at least once, a filter inserted after the router. The part that matters in envoyfilter/silent-errors/04-after-router.yaml is the position:

spec:
  workloadSelector:
    labels:
      app: fraud-check
  priority: 10
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        portNumber: 8080
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_AFTER

The API server accepted it with the generic EnvoyFilter warning. istiod accepted it too, built the new inbound listener and pushed it to the three fraud-check proxies. All three rejected it. proxy-status -v 1 showed LDS as ERROR on each of them, with CDS, EDS and RDS still SYNCED. payments-api, which the filter does not select, was unaffected. The JSON output of proxy-status carries the reason Envoy gave, in errorState.details:

Error adding/updating listener(s) virtualInbound: Error: terminal filter named envoy.filters.http.router of type envoy.filters.http.router must be the last filter in a http filter chain.

The router is a terminal filter, so nothing may follow it. istiod logged the rejection as a warning, ADS:LDS: ACK ERROR with the proxy name and the same message, and its counter pilot_total_xds_rejects{type="lds"} went up by three, one per proxy. On the Envoy side, listener_manager.lds.update_rejected, which Envoy documents as "total API fetches that failed because of schema/validation errors," went up by one on the proxy the lab read.

Get Filipe Motta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Forum answers tend to describe a filter after the router as one that simply never runs, which would be harmless. In the lab the cost was a frozen listener. The proxy kept its previous inbound listener and fraud-check kept answering with it. Its proxy-config listener --port 15006 output, hashed before and after the push, came out identical. As long as the EnvoyFilter exists, every new version of that listener carries the same bad filter, so the proxy rejects every one of them. Any change to the inbound listener stops at the door.

With the bad filter in place, the lab applied proxy-status/fraud-check-deny-blocked.yaml, a DENY for GET /blocked on port 8080. kubectl get authorizationpolicy listed it. GET /blocked on fraud-check v2 kept returning 200 while LDS stayed ERROR. istioctl proxy-status with the pod name compares what istiod would send with what Envoy holds. It printed Clusters Match and Listeners Don't Match, followed by a diff more than two thousand lines long. After the EnvoyFilter was deleted, the next push was accepted and LDS returned to SYNCED. GET /blocked got 403.

A filter after the router ends in a NACK and a frozen listener, so a DENY applied later never arrives.

Read that sequence as the security team would. A DENY policy that passed review sits in Git, in the API server and in kubectl get. It is not enforced. Nothing in the policy's own status says so. The only signal is the ERROR in proxy-status, a place nobody checks for a security policy. A broken EnvoyFilter in one namespace becomes a security incident there. In the root namespace it would become a fleet-wide one.

To start, alert on any increase in pilot_total_xds_rejects, by type. Treat any ERROR in proxy-status as a page. When a policy "does not work," check LDS sync on the target pod before you rewrite the policy. scripts/05-proxy-status.sh reproduces STALE and ERROR, including the policy that never arrives.

What Envoy Says About Itself

Logs and stats record what the proxy did with its configuration. Envoy exposes both through an admin interface on port 15000, which Istio’s application requirements list as the “Envoy admin port (commands/diagnostics)” and mark as pod-internal only. In the lab’s bootstrap the admin address was 127.0.0.1:15000. Everything below goes through it, via istioctl or via pilot-agent request inside istio-proxy, which works on distroless images.

The first tool is per-component log levels, changed at runtime with istioctl proxy-config log --level. The lab set rbac:debug on one fraud-check proxy while the http logger stayed at warning. A blocked request then logged enforced denied, matched policy ns[payments]-policy[fraud-check-deny-blocked]-rule[0]. After -r reset the levels, rbac was back at warning. The pod never restarted.

The access log is the second tool. Its format is plain configuration. The lab defines an extension provider in the mesh config of istio/install.yaml and turns it on with the Telemetry API in observability/access-log.yaml:

meshConfig:
    extensionProviders:
    - name: diag-log
      envoyFileAccessLog:
        path: /dev/stdout
        logFormat:
          text: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% flags=%RESPONSE_FLAGS% details=%RESPONSE_CODE_DETAILS% cluster=%UPSTREAM_CLUSTER_RAW% statname=%UPSTREAM_CLUSTER% route=%ROUTE_NAME% upstream=%UPSTREAM_HOST% authority=%REQ(:AUTHORITY)% sni=%REQUESTED_SERVER_NAME% dst=%DOWNSTREAM_LOCAL_ADDRESS%\n"

%RESPONSE_FLAGS% and %RESPONSE_CODE_DETAILS% say who produced the answer and why. %ROUTE_NAME% and %UPSTREAM_CLUSTER_RAW% say which configuration objects decided it. Without those four fields, a format cannot answer the questions in this article.

Stats are the third tool, with a default that trips people up. Istio’s Envoy stats page says that “by default, Istio configures Envoy to record a minimal set of statistics,” keyed on cluster_manager, listener_manager, server and cluster.xds-grpc, with no per-cluster request counters. ProxyConfig.ProxyStatsMatcher adds more, mesh-wide or per pod through the proxy.istio.io/config annotation. The proxy "needs to restart to pick up the stats matcher configuration." payments-api in the lab includes .*fraud-check.*. After ten requests, cluster.outbound|8080||fraud-check.payments.svc.cluster.local;.upstream_rq_200 had grown by ten. The risk-score cluster, outside the matcher, had no stats at all in that proxy. When a dashboard shows nothing for a cluster, check the matcher before you check the cluster.

“Localhost only” is a weaker statement than it sounds. From payments-api, the lab called fraud-check's pod IP on port 15000 and got a 503. The admin interface refused nothing itself. fraud-check's sidecar captured the connection on 15006 like any inbound connection and sent it to InboundPassthroughCluster toward the pod IP on 15000. Nothing was listening there, since admin is bound to 127.0.0.1. Its access log says so, 503 flags=URX,UF cluster=InboundPassthroughCluster details=upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}. The protection is the bind address. Anyone who can kubectl exec into a pod has the full admin API, configuration dumps and runtime changes included, so access to exec is access to the proxy.

One more behavior affects every check in this article. Envoy’s listener documentation says that “when a listener is updated, an entirely new listener is created,” and with filter-chain-only updates “only the connections owned by the removed filter chains will be drained.” Connections opened before a policy push keep the replaced filter chain until Envoy drains them. The gauge listener_manager.total_filter_chains_draining stays above zero meanwhile. In my early runs, before the scripts waited for the drain, a single request right after a DENY got 200 and a rate limit burst measured 200=6 429=2 instead of 200=5 429=3. The lab's scripts wait for that gauge to return to zero before asserting. Any rollout check that fires one request right after kubectl apply should do the same.

To start, define a mesh-wide access log format with flags, details, route and raw cluster name. Add a stats matcher for the clusters you alert on. Guard kubectl exec on proxies like the admin API it exposes. scripts/06-logs-stats.sh covers each of these.

EnvoyFilter Is the Last Resort

“EnvoyFilter provides a mechanism to customize the Envoy configuration generated by istiod,” the reference begins. It warns that “some aspects of this API are deeply tied to the internal implementation in Istio networking subsystem,” and that “any envoy configuration provided through this mechanism should be carefully monitored across Istio proxy version upgrades.” On 1.31.2 the API server repeats the point on every apply, with the warning EnvoyFilter exposes internal implementation details that may change at any time. Prefer other APIs if possible, and exercise extreme caution, especially around upgrades.

Before writing one, I go through the alternatives in order. A native API covers most needs and survives upgrades. The Telemetry API covers access logs, metrics and tracing. WasmPlugin, which “provides a mechanism to extend the functionality provided by the Istio proxy through WebAssembly filters,” covers custom request logic with an Istio-owned attachment point. Only when none of those fits does an EnvoyFilter enter the conversation.

A local rate limit is a fair example. No CRD in the lab cluster mentions rate limiting. Istio’s own rate limit task says its limits “are implemented using the EnvoyFilter API.” The lab’s version is envoyfilter/fraud-check-local-ratelimit.yaml, five requests per minute per pod on fraud-check's port 8080:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: fraud-check-local-ratelimit
  namespace: payments
spec:
  workloadSelector:
    labels:
      app: fraud-check
  priority: 10
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        portNumber: 8080
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.local_ratelimit
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
          stat_prefix: http_local_rate_limiter
          token_bucket:
            max_tokens: 5
            tokens_per_fill: 5
            fill_interval: 60s
          filter_enabled:
            runtime_key: local_rate_limit_enabled
            default_value:
              numerator: 100
              denominator: HUNDRED
          filter_enforced:
            runtime_key: local_rate_limit_enforced
            default_value:
              numerator: 100
              denominator: HUNDRED
          response_headers_to_add:
          - append_action: OVERWRITE_IF_EXISTS_OR_ADD
            header:
              key: x-local-rate-limit
              value: "true"

Every field of the match maps to a layer. workloadSelector picks the fraud-check pods. context: SIDECAR_INBOUND picks virtualInbound. portNumber: 8080 picks the filter chains for destination port 8080, which the reference defines as "the service port/gateway port to which traffic is being sent/received." The filter and subFilter names pick the HCM and, inside it, the router. INSERT_BEFORE places the new filter right before the router, after metadata exchange, RBAC when a policy exists, and stats. priority orders patch sets "within a context." Without it, istioctl analyze warns IST0151 for a relative operation.

The lab departs from the official task on purpose in three places. The task wraps the configuration in a udpa.type.v1.TypedStruct, which is not decoded field by field at admission. The lab tested a misspelled max_tokens in both forms. Neither was rejected by the API server nor by Envoy, and both became a limiter with an empty bucket that answered 429 to every request. The proto form at least produced a warning on apply naming unknown field "max_tokenz", while the TypedStruct form produced none. Its only trace is in the proxy, a warning in Envoy's log and server.dynamic_unknown_fields going up by one per filter chain. The lab uses the proto type directly, and its lint turns the warning into an error. The task also uses append: false for the response header, a field Envoy has deprecated in favor of append_action. Finally, it inserts with no subFilter. For INSERT_BEFORE the reference spells out where that lands: "If no filter is selected, the specified filter will be inserted at the front of the list." That puts the limit ahead of the metadata exchange and RBAC filters. I did not run the task's form in the lab. The repository keeps it verbatim as a lint fixture, where it passes with four warnings.

The proof that a filter entered is the proxy’s configuration, read after apply. In all three fraud-check proxies, the inbound HTTP filters for 8080 became istio.metadata_exchange, envoy.filters.http.grpc_stats, envoy.filters.http.fault, envoy.filters.http.cors, istio.stats, envoy.filters.http.local_ratelimit, envoy.filters.http.router. The filter was absent from the 8090 chains and from every outbound listener. Eight requests to the v2 pod on 8080 returned 200=5 429=3. Each 429 carried x-local-rate-limit: true and the body local_rate_limited. The access log wrote 429 flags=RL details=local_rate_limited cluster=inbound|8080||. Envoy describes RL as "the request was rate-limited locally by the HTTP rate limit filter in addition to 429 response code." The filter's stats, enabled in the pod annotation with the task's .*http_local_rate_limit.* matcher, moved by enabled +10, ok +5, rate_limited +5 and enforced +5 for a burst of ten. Port 8090 answered 200=8 for eight requests.

The upgrade risk sits in the strings. istiod compares filter names by equality, so a rename, a new filter between the HCM and the router, or a listener restructuring changes what your patch matches without any change to your YAML. This rate limit needs the check the reference asks for on every upgrade, a pipeline test that reads the filter list after apply.

To start, inventory every EnvoyFilter with kubectl get envoyfilter -A and write down what could replace each one. For the ones that stay, add a test that reads the proxy's filter list and a request that proves the behavior. Run both on every upgrade. scripts/07-envoyfilter.sh does both for the rate limit.

In Ambient, Only the Waypoint Is Envoy

Ambient mode splits the data plane in two. Istio’s ambient overview describes ztunnel as “a purpose-built, per-node proxy” that “is written in Rust and is intentionally scoped to handle L3 and L4 functions.” It says of the other one that “the waypoint proxy is a deployment of the Envoy proxy; the same engine that Istio uses for its sidecar data plane mode.” Everything this article does with proxy-config therefore applies to the waypoint and to nothing in ztunnel.

The ztunnel container is also named istio-proxy, so a script that treats every istio-proxy as Envoy will point proxy-config at it. On 1.31.2 the answer is an error that does not explain itself:

Error: listener dump: config dump has no configuration type type.googleapis.com/envoy.admin.v3.ListenersConfigDump

ztunnel has no listeners in Envoy’s sense. Its configuration is a set of workloads, services, policies and certificates, received over the workload types seen in proxy-status. istioctl ztunnel-config has a subcommand for each, plus connections. In the lab, ztunnel-config workloads listed the ambient pods with protocol HBONE and the waypoint with TCP. HBONE, short for HTTP-Based Overlay Network Environment, is the secure tunneling protocol Istio components use between themselves, on port 15008. The sidecar pods of payments were listed as HBONE too, because the injector labels them networking.istio.io/tunnel=http. ztunnel-config services showed fraud-check with WAYPOINT waypoint and ENDPOINTS 1/1, which is how a ztunnel knows to send that Service's traffic through the waypoint. ztunnel-config policies was empty, since the lab's L7 policy attaches to the Service with targetRefs and belongs to the waypoint. ztunnel-config connections showed the waypoint's pooled connection into fraud-check's node and nothing on the caller's node, even with a request in flight.

The waypoint reads like any Envoy, with a different shape and no RDS subscription. Its listeners were connect_terminate, the listener on 15008 that terminates HBONE, then main_internal, connect_originate, inner_connect_originate and outer_connect_originate, plus 0.0.0.0_15090 and 0.0.0.0_15021 for Prometheus stats and health checks. Its routes are inline in those listeners. The HTTPRoute of ambient/fraud-check-httproute.yaml appeared as a route that adds the header x-served-by: waypoint-route, the AuthorizationPolicy as an RBAC filter with ns[payments-ambient]-policy[fraud-check-l7]-rule[0], and the destination as the cluster inbound-vip|8080|http|fraud-check.payments-ambient.svc.cluster.local. Its access log reads like a sidecar's, with route names generated from the HTTPRoute, route=payments-ambient.fraud-check.0. GET got 200 via_upstream. POST got 403 rbac_access_denied_matched_policy[none], refused because no ALLOW rule matched.

To start with ambient, use ztunnel-config for workloads, identities and waypoints. Use proxy-config only on waypoints. When L7 policy seems ignored, check ztunnel-config services first, since a Service without a waypoint never reaches the Envoy that would enforce it. scripts/08-ambient.sh runs every command above.

The Silent Errors

Every file in envoyfilter/silent-errors/ is the working rate limit with one change, accepted by the API server and doing something other than what it appears to do. Script 07 applies each one live, except the filter after the router, which was script 05's.

The first uses the wrong context. 01-wrong-context.yaml says SIDECAR_OUTBOUND where it meant inbound. fraud-check's callers got 200=8. The patch landed on fraud-check's outbound listener 0.0.0.0_8080, the one it uses to call risk-score:8080. Its own calls through the 8090 proxy came back 200=5 429=3. A rate limit meant to protect a service throttled that service's outbound calls, which in production looks like a dependency failing.

The second matches the port of the capture listener. 02-wrong-port-15006.yaml sets portNumber: 15006, reasoning that inbound traffic arrives there. istiod compares portNumber with each filter chain's destination port, the service port, so 15006 matches nothing. The hash of the inbound listener did not change, and callers got 200=8.

The third uses an old filter name. 03-deprecated-filter-name.yaml matches envoy.http_connection_manager, a name that still circulates in older examples. The API server only warns, using deprecated filter name "envoy.http_connection_manager"; use "envoy.filters.network.http_connection_manager" instead. Names are compared by equality, so the patch matched nothing and 200=8. A warning inside a GitOps sync log is a warning nobody reads.

The fourth forgets to enable itself. 06-not-enabled.yaml leaves out filter_enabled and filter_enforced. Envoy's proto documents both with "defaults to 0% of requests for safety." The filter sat in all three proxies, right before the router, and limited nothing, 200=8. This one fools the configuration check, which is why every EnvoyFilter test needs a request as well as a read.

The fifth is about scope. 05-root-namespace-no-selector.yaml lives in istio-system with no selector and no context:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: mesh-wide-ratelimit
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.local_ratelimit
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
          stat_prefix: mesh_wide_probe
          token_bucket:
            max_tokens: 1000
            tokens_per_fill: 1000
            fill_interval: 1s

The reference says that a filter in the root namespace with no selector applies “to all applicable workloads in any namespace,” and a missing match.context means every context. Six of the seven Envoys received it, every sidecar and the ingress gateway. It patched 17 HTTP connection managers in each fraud-check proxy, 14 in payments-api and in risk-score, and 1 in the gateway. The waypoint got none, because, per the same reference, waypoints "are required to use" targetRefs. This file leaves filter_enabled at its default and limits nothing. The point is the blast radius. One edit adding the enabling fields would rate limit every HTTP hop of the mesh.

istioctl analyze --use-kube=false on 1.31.2 caught none of these. It repeated the API server's warning for the deprecated name and warned IST0151 for the missing priority on the root namespace filter. It also printed IST0133, the generic "exposes internal implementation details" warning, for every EnvoyFilter alike.

The repository’s answer is ci/envoyfilter-lint.sh, a static pass that runs before istioctl analyze and needs no cluster. Its error codes EF001 to EF007 are one code per silent error above, from a missing workloadSelector to a local rate limit in SIDECAR_OUTBOUND. EF008 turns the proto form's unknown-field warning into an error. ci/test-envoyfilter-lint.sh proves it both ways. The working filter passes clean and each silent error fails with its own codes, EF001, EF002 and EF006 for the root namespace file. A GitHub Actions workflow runs both on every push. The lint encodes the mistakes I have seen. It cannot know that a filter name will change in the next Envoy. Only the post-upgrade test covers that.

How to Start Reading Before You Change

Most Istio incidents I am pulled into already have a reapplied manifest and a restarted pod behind them. The order below costs a few minutes and replaces both.

Start with the access log line of the failing request on the caller’s sidecar. Add the callee’s if the caller shows a cluster and an upstream. The flag, the details, the route and the cluster tell you the layer and the side. Then check istioctl proxy-status -v 1 for the pods involved. A STALE or ERROR type means nothing you change in YAML will reach that proxy until it is resolved. Then read the layer the log pointed at, whether listener, route configuration, cluster or endpoints. Compare it with what the manifest intended. istioctl x describe pod and istioctl analyze are useful shortcuts, as long as you do not take their silence as proof.

Change something only after that, in the resource that owns the layer you found. An EnvoyFilter goes through the lint, a read of the proxy after apply and a request that proves the behavior. Then it goes onto the list the next upgrade must recheck.

A POC You Can Run on a Laptop

The companion repository reproduces everything above on kind, at zero cost, with the same names. Its README pins the versions (Istio 1.31.2, kind v0.33.0, Kubernetes 1.37) and lists the prerequisites, chiefly Docker with about 3 CPUs and 4 GB and Internet access.

Before step one, clone it and run tools/install-cli.sh, which downloads pinned istioctl, kubectl, kind and yq into ./bin and checks each against its release's SHA-256. Every script passes --context kind-istio-p4 and refuses to run unless it points at a local kind API server, which ci/test-context-guard.sh proves for scripts 02 to 08. The lint needs no cluster at all:

bash ci/envoyfilter-lint.sh envoyfilter/fraud-check-local-ratelimit.yaml

It ends with envoyfilter-lint: 1 EnvoyFilter(s) checked, 0 error(s), 0 warning(s). On envoyfilter/silent-errors/05-root-namespace-no-selector.yaml the same summary reads three errors and two warnings.

Step one, scripts/01-cluster.sh creates a cluster with one control plane and one worker and installs the Gateway API CRDs, Istio and the workloads. It ends with every proxy connected to istiod.

Step two, follow one request yourself before the script does, reading the layers in order:

export PATH="$PWD/bin:$PATH"
POD=$(kubectl --context kind-istio-p4 -n payments get pod -l app=payments-api -o jsonpath='{.items[0].metadata.name}')
istioctl --context kind-istio-p4 proxy-config listener "$POD.payments" --port 15001
istioctl --context kind-istio-p4 proxy-config listener "$POD.payments" --port 8080
istioctl --context kind-istio-p4 proxy-config route "$POD.payments" --name 8080
istioctl --context kind-istio-p4 proxy-config cluster "$POD.payments" --fqdn fraud-check.payments.svc.cluster.local
istioctl --context kind-istio-p4 proxy-config endpoint "$POD.payments" --cluster "outbound|8080||fraud-check.payments.svc.cluster.local"

The listener on 8080 shows its HTTP chain loading Route: 8080, and the last command prints the three HEALTHY endpoints. Add -o json to any of them for the full objects. Then run scripts/02-follow-request.sh, which asserts each name and the matching access log lines on both sides.

Step three, scripts/03-translation.sh applies the five resources one at a time and prints the objects each one added and removed.

Step four, scripts/04-broken-by-layer.sh breaks each layer, including the typo host and both egress modes.

Step five, scripts/05-proxy-status.sh pauses one Envoy to show STALE, then applies the filter after the router. While it runs, istioctl --context kind-istio-p4 proxy-status -v 1 shows LDS in ERROR on the three fraud-check pods. It then applies the DENY, shows GET /blocked still answering 200, removes the filter and gets the 403.

Step six, scripts/06-logs-stats.sh covers log levels, the access log format, per-cluster stats and the admin port.

Step seven, try the rate limit by hand. Apply the filter, wait a few seconds, then send eight requests to the v2 pod:

kubectl --context kind-istio-p4 apply -f envoyfilter/fraud-check-local-ratelimit.yaml
FC=$(kubectl --context kind-istio-p4 -n payments get pod -l app=fraud-check,version=v2 -o jsonpath='{.items[0].status.podIP}')
kubectl --context kind-istio-p4 -n payments exec deploy/payments-api -c app -- \
  sh -c "for i in 1 2 3 4 5 6 7 8; do curl -s -o /dev/null -w '%{http_code}\n' http://$FC:8080/; done"

Expect five 200s and three 429s. If you get six and two, the draining note in the logs section explains why. If you repeat the burst within a few minutes of a previous apply and delete, it can come back as seven and one with the limiter counting all eight requests, a result I measured twice and could not explain from Envoy’s documentation. Apply it fresh. Delete the filter with kubectl --context kind-istio-p4 delete -f envoyfilter/fraud-check-local-ratelimit.yaml and run scripts/07-envoyfilter.sh, which repeats this with the waits and then applies each silent error.

Step eight, scripts/08-ambient.sh covers ztunnel and the waypoint. Then run it all at once and clean up:

bash scripts/test-all.sh         # offline tests first, then steps 01..08; about 13 minutes from no cluster
bash scripts/teardown.sh         # deletes the kind cluster istio-p4

The final run from no cluster passed 270 of 270 assertions in 750 seconds, 43 offline tests of the lint, the TypedStruct typo and the context guard plus 227 in the lab, including a check that runs the commands of this POC exactly as printed. Script 07 takes the longest, since it waits for old filter chains to drain before each burst. The lab has limits. STALE comes from a paused process, which leaves the production causes, an overloaded istiod or a starved proxy, outside the lab. The lab reads configuration and counts results. It does not measure the performance cost of any filter. The Sidecar resource, Wasm and ext-proc are out of scope. Where the README’s outputs contradict the documentation, they describe 1.31.2. A future release may change any of them.

Part 5 of Istio, Hands-On starts trimming the configuration this article read. The Sidecar resource narrows what each proxy receives, the remedy for route configurations that list every service in the mesh. Virtual machines join the mesh as workloads with proxies of their own.

A manifest tells you what someone asked for. The proxy’s configuration tells you what the mesh is doing. Between the two sit a translation, a push and an acknowledgement that can each go wrong without a word in kubectl.

Read what the proxy received before you change what you apply.

Filipe Motta is a Senior DevOps Engineer with 20 plus years of experience, building platforms on Kubernetes and service mesh where every routing, security and extension change is reviewed as configuration and checked against what the proxies actually received. He has earned the CKA, CKAD, CKS, AWS DevOps Professional, and GCP Professional DevOps Engineer certifications. Author Official DevOps AI Guide.