upgrade mode grrrreat

Upgrading Apache Flink Without Stopping the Stream

How blue/green deployments use savepoints to keep processing — and what happens to your delivery guarantees.

Your streaming pipeline is healthy. Events keep arriving, state keeps growing, and downstream systems expect fresh results. Then it is time to upgrade Flink.

A conventional stateful upgrade suspends the job and restores it with the new specification. The processing pause can be acceptable, but it still creates a gap in fresh output. Blue/green deployments keep the previous job running while its replacement starts. [1]

The feature arrived in Operator 1.14. Operator 1.16 added fixes around transition completion and delayed deletion after an aborted attempt. It remains an experimental capability in the 1.16 documentation. [2][3][4]

The interesting part is what happens between starting the replacement and removing the old job. That overlap is where availability improves — and where output semantics need careful design.

Two deployments and a shared starting point

For a stateful transition, the Operator takes a savepoint while Blue continues running. It then starts Green with the new specification and restores that savepoint. Blue remains active until Green meets the readiness conditions and the configured deletion delay has elapsed. [1][6]

Think of the savepoint as a common recovery position, not a live synchronization channel. After restoration, the two jobs execute independently. The same input can therefore reach the destination through both versions. [1]

Figure 1. A savepoint transfers state; it does not coordinate subsequent writes from the two jobs.

A deployment example you can adapt

The resource is FlinkBlueGreenDeployment. Its application definition lives under spec.template.spec, while blue/green controls belong under spec.configuration. The Operator creates and manages the Blue and Green child deployments; you do not maintain two independent manifests for those children. [5][6]

The example starts on Flink 2.1.3 and illustrates an upgrade to 2.2.1. Both version families are supported by Operator 1.16, but support in the Operator does not establish compatibility for your application state. [3][8]

Prerequisites for this example: install Operator 1.16 and its CRDs; create the streaming namespace and a suitably authorized flink service account. Replace the example registry and bucket. Build both images with the matching Flink runtime, compatible connectors, the application JAR at the path shown, and an enabled S3 filesystem plugin. Supply source, sink, and storage credentials through your normal configuration. The application and those environment-specific settings are outside this manifest.

deployment-v1.yaml

apiVersion: flink.apache.org/v1beta1
kind: FlinkBlueGreenDeployment
metadata:
  name: telemetry-pipeline
  namespace: streaming
spec:
  configuration:
    kubernetes.operator.bluegreen.abort.grace-period: "15 min"
    kubernetes.operator.bluegreen.reconciliation.reschedule-interval: "15 s"
    kubernetes.operator.bluegreen.deployment-deletion.delay: "2 min"
  template:
    spec:
      image: registry.example.com/telemetry:flink-2.1.3-v1
      flinkVersion: v2_1
      serviceAccount: flink
      flinkConfiguration:
        taskmanager.numberOfTaskSlots: "2"
        execution.checkpointing.interval: "30 s"
        execution.checkpointing.storage: filesystem
        execution.checkpointing.dir: s3://YOUR-BUCKET/telemetry/checkpoints
        execution.checkpointing.savepoint-dir: s3://YOUR-BUCKET/telemetry/savepoints
      jobManager:
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
      taskManager:
        resources:
          requests:
            cpu: "2"
            memory: "4Gi"
      job:
        jarURI: local:///opt/flink/usrlib/telemetry-job.jar
        parallelism: 4
        upgradeMode: savepoint
        state: running

These are illustrative resource sizes, not sizing guidance. With parallelism four and two slots per TaskManager, allow enough capacity for both complete application clusters during the overlap. The checkpoint paths use Flink 2.x configuration names. [1][9]

Create the initial application

kubectl apply -f deployments/deployment-v1.yaml
kubectl get flinkbgdep telemetry-pipeline -n streaming

This is a new application example. Applying a different resource kind does not adopt an already-running FlinkDeployment. For an existing application, plan the initial migration, state restoration, and output overlap separately.

Triggering the upgrade

After validating the new build, update the image and version together. This merge patch changes the template on the existing parent resource:

upgrade-patch.yaml

spec:
  template:
    spec:
      image: registry.example.com/telemetry:flink-2.2.1-v2
      flinkVersion: v2_2

Start the transition

kubectl patch flinkbgdep telemetry-pipeline \
  -n streaming --type merge \
  --patch-file deployments/upgrade-patch.yaml

For GitOps, make the same change in the desired manifest and let your controller reconcile it. The accompanying package includes a complete deployment-v2.yaml as that target state. Choose one update method; do not leave Git declaring v1 after manually patching v2.

An image or Flink-version change triggers a transition. Some other edits are handled in place, so a blue/green resource does not imply that every configuration change starts a second job. [6]

Get Eduardo Donzeli’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Inspect the parent and its child deployments

kubectl get flinkbgdep telemetry-pipeline -n streaming -w
kubectl get flinkdeployments -n streaming
kubectl get flinkbgdep telemetry-pipeline -n streaming \
  -o jsonpath='{.status.blueGreenState}'

What ready actually means

The documented success conditions are a job in RUNNING and a child resource in STABLE. If the candidate misses the transition deadline, the Operator suspends it for inspection and retains the previous active deployment. [4]

Figure 2. Readiness controls retirement of the old deployment. It does not prove that lag is zero or that business results match.

I would treat those states as infrastructure checks. For a telemetry pipeline, I would also monitor event freshness, source lag, watermark progress, checkpoint health, and whether aggregates match expectations. A running job can still produce the wrong answer.

The example allows 15 minutes for readiness and keeps the old deployment for another two minutes. The documented defaults are 10 minutes for abort.grace-period, 15 seconds for reconciliation.reschedule-interval, and zero extra time for deployment-deletion.delay. All use the prefix kubernetes.operator.bluegreen.. [7]

The extra delay is a timer, not an automated data-quality approval. Also, the abort grace period covers the transitioning phase; it is not an overall deadline for savepoint creation. [6]

Exactly-once inside a job does not extend across the overlap

The two deployments commit independently. Consequently, the blue/green transition provides at-least-once delivery, even when each job individually uses exactly-once. Downstream processing must tolerate or deduplicate overlapping results. [1]

Consider event E42 from an agricultural machine. Blue processes it and updates a dashboard table. Green later replays the same event from the restored position and writes another result. There is no single cross-job transaction telling the destination that one of those outputs should be ignored.

For an event history, a stable result identifier can support deduplication. For a latest-state table, a plain upsert may be insufficient: an older replayed update could overwrite a newer value. I would use a version or ordering rule that the destination enforces, and define how legitimate aggregate revisions differ from duplicates.

Separate Blue and Green output destinations are another architectural option. However, promoting one output requires an explicit consumer handoff and recovery protocol. That is additional application design, not a switch the Operator automatically provides.

For a transactional Kafka sink, also isolate the transaction identities of concurrently running applications. The connector requires distinct transactionalIdPrefix values across those applications to avoid transaction interference. Configure and test this at the application level; the deployment template above does not define the sink. [10]

The managed ingress switch targets the active Flink REST service and Web UI. It does not coordinate Kafka output records, database writes, or other sink side effects. [4]

What to validate before production

Savepoint compatibility still applies: preserve the intended operator UIDs, check serializers and state schemas, and validate the application graph against the target runtime. SQL and Table API applications need particular attention because generated operator identities can change. Test restoration with representative state before the production transition. [8]

Blue/green currently supports application clusters, not Session Mode. Stateful transitions use savepoints even when upgradeMode is last-state. And retaining Blue during a failed attempt does not undo side effects Green may already have emitted. [4]

My first production rehearsal would deliberately fail the candidate. I would verify that the previous job keeps processing, the destination remains correct, and the team can diagnose the suspended candidate. That is more useful than a demo in which both jobs simply turn green.

The deciding question is how much continuous freshness is worth to your consumers. When even a short processing pause matters, blue/green can justify its extra capacity and output coordination. When a brief pause fits the latency budget, a conventional savepoint-based upgrade may remain the easier system to operate.

Sources and example scope

The diagrams and telemetry scenario are explanatory illustrations. The YAML was checked for syntax and consistency with the referenced APIs; it has not been deployed against a live cluster. Replace the placeholders and validate application-specific behavior before use.

[1] Zero-Downtime Upgrades

[2] Operator 1.14 release announcement

[3] Operator 1.16 release announcement

[4] Blue/Green Deployments

[5] Custom Resource Reference

[6] Blue/Green Controller

[7] Operator Configuration

[8] Upgrading Applications and Flink Versions

[9] Flink 2.1 Checkpointing Configuration

[10] Kafka Sink Fault Tolerance