When Dev Quietly Talks to QA: A KEDA HTTP Add-on Host Collision

I knew something was wrong when I shut down the QA service to test something else, and the dev environment stopped working.

Dev and QA are supposed to be separate. Yet every transaction I ran on dev was being processed by QA, and nothing in the logs told me so. Each environment looked correct on its own.

In this article I’ll walk through how I tracked it down, what the root cause was, and how I fixed it. We’ll cover how the request path works, why the KEDA HTTP interceptor can’t tell two environments apart, how I proved the leak, and a checklist to stop it happening again.

The setup

Our platform runs on AKS with Istio and KEDA’s HTTP add-on, deployed through ArgoCD. Dev and QA share the same cluster entry point:

Client → Application Gateway → Istio ingress → KEDA HTTP interceptor → Service

Each environment has its own public hostname and its own path prefix, so dev-api gets /dev/... and qa-api gets /qa/.... Istio VirtualServices match on that prefix and forward the request to the KEDA interceptor, which wakes the workload if it is scaled to zero and proxies the request on.

LayerHow it separates dev from QADNS / App GatewayDifferent hostnames, path prefix added per environmentIstio VirtualServiceDifferent route names and path prefixesKEDA interceptorHost header (this is where it went wrong)

The symptom

A request to the dev API returned a healthy 200, but the transaction landed in QA. The first checks all looked fine:

• The dev VirtualService matched /dev/... and had its own route name.
• The App Gateway forwarded to the right place.
• Istio’s access log showed tekmeet-dev-routes for dev and tekmeet-qa-routes for QA.

Routing worked at every layer I could see. The leak had to be in the hop I wasn’t looking at.

The root cause

Istio logs showed both environments reaching the interceptor with the same Host header:

GET /dev/switch/ 200 "tekmeet.local" → keda-add-ons-http-interceptor-proxy  (tekmeet-dev-routes)
GET /qa/switch/  200 "tekmeet.local" → keda-add-ons-http-interceptor-proxy  (tekmeet-qa-routes)

Our VirtualService templates set the Host header from the service name:

yaml

headers:
  request:
    set:
      Host: "{{ .Values.service.serviceName }}.local"

And the HTTPScaledObject registered the same name:

yaml

spec:
  hosts:
    - {{ .Values.service.serviceName }}.local

Dev and QA both deploy a service called tekmeet, so both produced tekmeet.local. The KEDA interceptor routes only on the Host header. With two HTTPScaledObjects claiming one host, it can map that host to only one backend, so some dev requests ended up in QA.

Most of our QA services use an environment prefix like qa.tekneed.local, which is why this one slipped through. A handful of QA services use the plain, unprefixed name, and those collide with dev.

How I proved it

Seeing the same host in the logs was strong evidence, but I wanted a test that couldn’t be argued with.

1. Scale QA’s tekmeet to zero.
2. Call the dev endpoint.

If dev and QA were truly separate, dev would keep working. It returned 502. Dev only worked while QA was up, so QA had been answering for it.

A note on testing: a 200 alone proves very little here, because either backend can return one. Taking one side down is the cleanest way to find out who is really answering.

The fix

Give dev its own host. Two objects need to change, and they must change together, because the interceptor needs the VirtualService header and the HTTPScaledObject host to match.

Get Tobi Oderinde’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

I added one optional value to the chart:

yaml

# values.yaml (dev)
Keda:
  host: dev.tekmeet.local

And used it in both templates, falling back to the old behaviour so other environments don’t change:

yaml

# virtualservice.yaml
Host: "{{ .Values.Keda.host | default (printf "%s.local" .Values.service.serviceName) }}"

# httpScaledObject.yaml
hosts:
  - {{ .Values.Keda.host | default (printf "%s.local" .Values.service.serviceName) }}

I tested it live first. I paused auto-sync, backed up both objects, and patched them directly:

bash

kubectl -n dev patch httpscaledobject tekmeet --type merge \
  -p '{"spec":{"hosts":["dev.tekmeet.local"]}}'

kubectl -n dev patch virtualservice.networking.istio.io tekmeet --type json \
  -p '[{"op":"replace","path":"/spec/http/0/headers/request/set/Host","value":"dev.tekmeet.local"}]'

With QA still scaled to zero, dev returned 200. It was now served by its own backend. Then I committed the same change to git so ArgoCD wouldn’t revert it.

What this approach can’t tell you

• A 200 doesn’t prove which backend answered. Use isolation (stop one side) or compare environment-specific content.
• App logs may not help. Some services don’t log plain GET requests, so an empty grep isn’t proof of anything.
• Past damage isn’t undone. Requests that landed in the wrong environment used its database, config and queues. Check for stray test records.

Checklist

• Every service has a unique host per environment (dev.<svc>.local, qa.<svc>.local)
• The VirtualService Host header and the HTTPScaledObject hosts match exactly
• List all hosts across namespaces and look for duplicates:

bash

kubectl get httpscaledobject -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,HOSTS:.spec.hosts

• Test isolation by scaling one environment down and confirming the other still works
• Pause ArgoCD auto-sync before live patching, and commit the change to git afterwards
• Check the other environment’s data for stray records

Conclusion

Shared infrastructure fails quietly. Every layer routed correctly, and the bug lived in a single header that the interceptor used as its only key. The practical takeaway: when two environments share a proxy that routes on the Host header, make sure no two environments can ever produce the same host. A quick duplicate check across namespaces would have caught this on day one.

Further reading

• KEDA HTTP Add-on documentation
• Istio VirtualService reference (headers, match, rewrite)
• Argo CD sync policies and auto-sync behaviour
• Kubernetes kubectl patch and managed fields