Inside Deploying ThousandEyes: Metrics-Driven Progressive Delivery with Argo Rollouts

By Antonio Jimenez, Software Engineering Technical Leader and Chris Detsicas, Site Reliability Engineering Technical Leader at Cisco ThousandEyes

At Cisco ThousandEyes, deployments are not just a delivery mechanism; they are a reliability event.

Every new version of a service has the potential to impact the real customer experience across our globally distributed platform. This reality requires a shift in mindset: deployment success cannot be defined by infrastructure health alone. A deployment is only successful if the system continues to behave correctly under real-world traffic.

This is the challenge we set out to solve using Argo Rollouts and AnalysisTemplates, which we presented at KubeCon EU 2026. In this post, we look beyond the talk to explore how we implement metrics-driven progressive delivery in practice, the trade-offs we encountered, and the patterns have proven reliable in production.

From “Pods Are Healthy” to “The System Is Healthy”

Kubernetes Deployments are intentionally simple. The default RollingUpdate strategy focuses on availability guarantees:

• Pods are replaced gradually
• Readiness probes gate traffic
• Capacity is controlled via maxSurge and maxUnavailable

While this ensures that infrastructure remains available during updates, it leaves a critical gap: Kubernetes does not evaluate whether the new version is actually functioning correctly.

In real-world systems, failures rarely manifest as immediate crashes. Instead, they manifest as:

• Increased latency under load
• Subtle error rate degradation
• Partial failures in downstream dependencies
• Incorrect business logic behavior

All of these issues can pass standard readiness probes while still degrading the user experience.

This leads to a fundamental question:

How do we promote a version based on actual behavior rather than simple health checks?

Progressive Delivery with Argo Rollouts

Progressive delivery introduces a feedback loop into the deployment process. Argo Rollouts provides the controller, while AnalysisTemplates define the signals and decision criteria. The combination replaces the native Deployment controller with a Custom Resource Definition (CRD) that introduces explicit control over:

• Traffic shaping
• Rollout progression
• Metric evaluation
• Automated decisions

It integrates with ingress controllers and service meshes, which allows traffic to be shifted based on granular percentages rather than replica approximations.

Most importantly, it allows us to encode deployment logic that answers:

“Is this version safe to continue?”

This is fundamentally different from Kubernetes’ default behavior.

A rollout in Argo is not just a resource definition. It is a sequence of controlled steps, guarded by validation.

apiVersion: argoproj.io/v1alpha1 
kind: Rollout 
metadata: 
  name: canary-demo 
spec: 
  replicas: 5 
  revisionHistoryLimit: 1 
  selector: 
    matchLabels: 
      app: canary-demo 
  strategy: 
    canary: 
      analysis: 
        templateName: success-rate 
        args: 
        - name: ingress 
          value: canary-demo 
      canaryService: canary-demo-preview 
      steps: 
      - setWeight: 5 
      - pause: 
          duration: 5m 
      - setWeight: 25 
      - pause: 
          duration: 5m 
      - setWeight: 50 
      - pause: 
          duration: 5m 
      - setWeight: 80 
      - pause: 
          duration: 5m 
  template: 
    metadata: 
      labels: 
        app: canary-demo 
    spec: 
      containers: 
      - image: argoproj/rollouts-demo:blue 
        imagePullPolicy: Always 
        name: canary-demo 
        ports: 
        - containerPort: 8080 
          name: http 
          protocol: TCP 
        resources: 
          requests: 
            cpu: 5m 
            memory: 32Mi 

There is no implicit trust in the new version. Every step is verified. Traffic is increased incrementally.

Get Cisco ThousandEyes Engineering’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The key abstraction enabling this is the AnalysisTemplate, which defines

• Where metrics come from
• How often they should be evaluated
• What constitutes success or failure

apiVersion: argoproj.io/v1alpha1 
kind: AnalysisTemplate 
metadata: 
  name: success-rate 
spec: 
  metrics: 
    - name: success-rate 
      interval: 1m 
      count: 5 
      successCondition: result[0] >= 0.99 
      provider: 
        prometheus: 
          address: http://prometheus.monitoring:9090 
          query: | 
            sum(rate(http_requests_total{status!~"5.."}[1m])) 
            / 
            sum(rate(http_requests_total[1m])) 

This template is simple but powerful: if the success rate remains above 99%, we continue; otherwise, we roll back. This creates a deterministic gate that replaces manual inspection. Metrics are queried from Prometheus and evaluated every minute.

Demo: Failure-Driven Rollback

During our presentation, we demonstrated a release involving two versions: a successful one and a faulty version that returned HTTP 500 errors.

The rollout behavior was:

• Initial steps passed
• Error rates increased as traffic scaled
• AnalysisRun detected the failure
• The rollout was aborted automatically
• Traffic reverted to the stable version

This illustrates the core value of the system: it reacts to real signals, not assumptions.

Demo recording:

Deep Dive: Our Authentication Service

Our authentication service manages OAuth, basic auth, and user logins for many services across our platform. It is mission-critical: an outage here cascades across the entire ecosystem. The team managing this service uses Argo Rollouts and AnalysisTemplates to ensure deployment quality and consistency.

The deployment strategy combines two validation layers:

1. Inline E2E tests as a Kubernetes Job: Before any traffic shifts to the canary, we run a containerized end-to-end test suite that exercises real authentication flows against the canary endpoint. The Job exits zero on success; anything else fails the rollout immediately.
2. Background Metrics Analysis: While traffic shifts incrementally — (1% → 5% → 10% → 20% → 40% → 70%) via Istio routing — Prometheus-backed AnalysisTemplates continuously evaluate availability, latency, and CPU/Memory resource usage growth. Any sustained regression triggers an automatic abort.

The conservative early steps (starting at 1%) reflect the criticality of the service: we aim to detect issues with the smallest possible blast radius. Once metrics consistently pass at each stage, traffic increases.

The outcome is roughly 15 minutes of engineer time saved per region, with higher confidence than manual validation provided. Most importantly, the rollout decision is now reproducible. Two engineers running the same release will get the same result, because the result is determined by metrics, not judgment.

Shoutout to Bruno Galazio Marques and the team for sharing the insights about the authentication service.

Scaling Up: From Rollouts to Kargo

Argo Rollouts solves the “is this version safe?” problem within a single environment. As ThousandEyes has grown to operate across many regions, coordinating a single release across all of them became its own challenge. Running Argo Rollouts in each region individually doesn’t scale to a unified promotion decision. The question quickly becomes: is this version safe to promote from staging to production, across all our regions?

This is where Kargo comes in. Kargo orchestrates promotions across environments using the same AnalysisTemplate primitives. Each Stage can define Verifications metric checks, HTTP calls, or containerized jobs — that must pass before a Freight (a versioned set of artifacts and manifests) is promoted downstream.

The pattern is identical to what we already trust within a Rollout, scaled to a higher level: if staging verification fails, production promotion is blocked automatically. What makes this powerful is consistency. The same philosophy — encode the decision, run it continuously, act on real signals — now applies both within a deployment and across our environment graph.

Closing Thoughts

Progressive delivery shifts deployments from a best-effort activity to a controlled, measurable process.

With Argo Rollouts and AnalysisTemplates, deployment decisions are no longer based on dashboards and intuition. They are encoded directly into the system, enforced continuously, and executed automatically.

For us, this has changed not just how we deploy, but how we think about reliability. Every rollout is now an experiment with clear hypotheses, measurable signals, and deterministic outcomes — and with Kargo, that same discipline now extends across all our regions and environments.

For the full talk, including the live demo, check out our session from KubeCon Europe 2026: https://www.youtube.com/watch?v=Cj5_wyUPhNg

Want to be a part of our team? ThousandEyes is hiring! Please see our Careers page for open roles.