A Real Two-Model Canary Rollout for RAG Reranking, with KServe

Most “canary deployment” posts for ML models show the same model deployed twice behind a traffic split. That proves your routing works. It doesn’t prove anything about the model.

I wanted to know what it actually looks like to run two different models in production at the same time — real traffic, real latency numbers, real accuracy differences — not just a health check dressed up as a canary. So while building RAGOps, a retrieval-augmented Q&A system over SEC 10-K filings, I deployed a genuine two-checkpoint canary for the reranking stage: two distinct cross-encoder models, split 70/30, both serving live traffic through the same KServe InferenceGraph.

Why the reranker needed this

RAGOps retrieves candidate chunks from a vector store, reranks them, then generates a cited answer. The retrieval step alone is high-recall but low-precision — it reliably finds the right filings but drags along a lot of irrelevant chunks with them. Reranking exists specifically to fix that: reorder the candidates by actual relevance to the question before anything else sees them. Which raises the obvious question: which reranker? A bigger cross-encoder is usually more accurate and always slower. You don’t want to find that tradeoff out in a postmortem — you want to measure it, on real traffic, before you commit.

The setup

I deployed two KServe InferenceServices wrapping sentence-transformers.CrossEncoder:

• reranker — cross-encoder/ms-marco-MiniLM-L6-v2
• reranker-b — cross-encoder/ms-marco-MiniLM-L12-v2 (same architecture family, more layers, slower, theoretically more accurate)

Both run from one shared image — which checkpoint a given pod loads is picked at runtime via a MODEL_CHECKPOINT environment variable, not baked into separate builds. One Dockerfile, two deployments, zero duplicated build logic.

Get Rupinder Guron’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In front of them sits a KServe InferenceGraph with a Splitter router:

apiVersion: serving.kserve.io/v1alpha1
kind: InferenceGraph
metadata:
  name: reranking-router
spec:
  nodes:
    root:
      routerType: Splitter
      steps:
        - serviceName: reranker
          weight: 70
        - serviceName: reranker-b
          weight: 30

The RAG query path calls this router, not either model directly — it has no idea which checkpoint answered any given request. That’s the whole point: the canary is invisible to the caller and real to the infrastructure.

Proving it’s a real canary, not a plumbing test

It’s easy to stand up two services and declare victory the moment both respond. The actual bar is higher: do the two checkpoints disagree? I sent identical input to both and confirmed they returned genuinely different relevance scores — not noise, a real accuracy difference from a real architectural difference. If I’d only verified that both services returned a number, I’d have proven the router works and nothing about the models.

What it took to get there (the honest part)

A few things broke on the way, each a small lesson:

• serviceUrl vs. serviceName in the InferenceGraph spec. serviceUrl proxies to a bare / with no path suffix — your model's actual predict route never gets hit. serviceName is what you want.
• Model name mismatch. Each predictor needs to answer to the model name the request path specifies (/v1/models/reranker:predict), set via MODEL_NAME — easy to overlook when you're copy-pasting a manifest for the second service.
• Image tag caching. I rebuilt an image, redeployed, and kept seeing old behavior — because I’d reused the same tag and the cluster never pulled the new layer. imagePullPolicy: Always plus SHA-based tags fixed it for good, not just that one time.

None of these are exotic. They’re exactly the kind of friction you only find by actually deploying the thing, which is sort of the point of doing this for real instead of describing it in the abstract.

Watching it live

GKE Managed Prometheus doesn’t auto-discover scrape targets the way a self-hosted Prometheus does — you need an explicit PodMonitoring resource telling it what to scrape. Once that's wired up, Grafana (using its native Google Cloud Monitoring data source, authenticated via Workload Identity — no API keys) plots both checkpoints side by side: throuhput(request rate per model), confirming the 70/30 split is actually happening, and p95 latency per model.

Grafana Dashboard Latency Panel:

Latency: Larger Reranker Model(Yellow) is Slower

Grafana Dashboard Throughput Panel:

Throughput: Canary Reranker Model(Green) has smaller # of requests routed.

GitHub - RoopGuron/ragops-project: document Q&A platform