We run NVIDIA Triton Inference Server on GKE across two zones. Each Triton pool has the same number of pods in each zone, one pod per L4 GPU, no GPU sharing. On paper the placement was perfect. Yet when we looked at per-pod request rates, the pods in one zone were serving about 1.5x the traffic of their twins in the other zone.

Nothing was broken: GPU utilization was low and queue time was under 2 ms. But that kind of skew means one zone hits its ceiling first during peak, and you pay for headroom you cannot use. Here is how we tracked it down.

Step 1: Measure per pod, not per service

Our APM tool aggregated Triton metrics by service and model. Its tag configuration had dropped the pod tag to save on custom-metric cardinality, so a per-pod breakdown simply was not available there.

The fix was to go one layer down. Triton exposes Prometheus metrics on port 8002, and our in-cluster Prometheus-compatible store already scraped them per pod. A query like this gives requests per second per pod:

sum by (pod) (rate(nv_inference_request_success{model!="version"}[1h]))

(We excluded a tiny “version” model used only for health checks.) Put the result next to each pod’s zone and the pattern is obvious:

• Pool A, zone B pod: 4.6 req/s. Zone A pod: 3.0 req/s, so 1.55x.
• Pool B, zone B pod: 4.5 req/s. Zone A pod: 3.1 req/s, so 1.48x.
• The autoscaled pool: 27 to 30 req/s per pod, within 8%. Even.

So the problem was specific to some pools, and it was zonal.

Step 2: Check how traffic reaches Triton

Our Triton Services use topology-aware routing:

spec:
  trafficDistribution: PreferClose

With PreferClose, a client pod is routed to endpoints in its own zone when they exist. That is good for latency and avoids cross-zone network charges. It also means Triton’s load in a zone is proportional to how many client pods live in that zone.

Long-lived gRPC connections pinning traffic to a few pods is the usual suspect for Triton imbalance, but within a zone the pods were even. The skew was between zones, which points at the callers.

Step 3: Count the callers per zone

The main caller of the skewed pools was an OCR service with about 20 replicas. Its pods were split 7 in zone A and 13 in zone B.

Its topology spread looked fine at first glance:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: ocr-caller

The catch is ScheduleAnyway. It is a soft constraint: the scheduler only adds it to the scoring, together with every other preference. This deployment also had node-affinity preferences (spot vs. on-demand node pools), and those preferences outweighed the spread score. The scheduler kept putting new pods where the preferred nodes had room, which was mostly zone B.

So the chain was:

1. Soft spread loses to node-affinity preference, so the callers end up 7/13.
2. PreferClose keeps each caller’s requests in its own zone.
3. Zone B’s Triton pods get roughly 13/7 of zone A’s share of that caller’s traffic.

Step 4: The fix

Another service in the same cluster already used a hard constraint, and its pods were perfectly balanced. We made the callers match it:

whenUnsatisfiable: DoNotSchedule

With DoNotSchedule the scheduler will not place a pod that would push the zone skew above maxSkew. After the rollout, the caller went from 7/13 to 8/9 across zones, with no Pending pods, and the Triton pods in both zones started receiving comparable traffic.

Two details made this a safe change:

• matchLabelKeys with the rollout hash. We use Argo Rollouts, and the constraint includes matchLabelKeys: ["rollouts-pod-template-hash"]. The spread is computed per revision, so old pods do not distort the placement of new ones during a rollout.
• Existing pods do not move by themselves. The constraint only applies at scheduling time. The skew fixed itself on the next rollout. If you need it immediately, restart the deployment or use the descheduler’s RemovePodsViolatingTopologySpreadConstraint strategy.

The trade-off you must check first

A hard spread can leave pods Pending. If the short zone has no capacity and its node pool cannot grow (max nodes reached, or a machine-type stockout in that zone), the scheduler will not “borrow” the other zone. With ScheduleAnyway it would have.

Before merging, we checked:

• Free CPU and memory per zone in the node pools the callers use. Zone A had room for only one more pod, so we expected a few minutes of Pending while the autoscaler added a node.
• Whether the autoscaler can grow that pool in both zones, and how far it is from its max.
• What happens at max replicas. In our case the pools could not fit every service at max under either setting, so the real fix for that scenario is raising the pool max, not softening the spread.

We decided against removing PreferClose. Turning it off would balance load instantly, but it gives up zone-local latency and adds cross-zone traffic cost. Fixing the callers' placement keeps both benefits.

Takeaways

• Topology-aware routing (PreferClose) makes your backend's load mirror your clients' placement. Balance the clients, not only the servers.
• ScheduleAnyway is a hint. If other scheduling preferences exist, expect it to lose.
• Measure per pod. Aggregated service metrics hide this class of problem completely, and the cheapest data source is often the Prometheus endpoint you already scrape.
• Before switching to DoNotSchedule, check per-zone headroom and autoscaler limits, and plan for brief Pending pods.

A one-line YAML change, but finding it took looking at the system as a chain: placement of callers, routing policy, then placement of servers.