We Moved a Kubernetes Control Plane Between Two Clouds While It Was Running

A story in seven disasters and one very boring DNS change

Everyone has migrated worker nodes. You add a node pool somewhere else, slap a label on it, the scheduler spreads the pods, you write a wiki page, you get a 👏 in the team channel.

That’s the easy half. Workers are designed to die. The cluster already assumes they will.

This is about the other half.

Two buttons. One sweaty forehead.

[ Migrate the workers, get applause ]

[ Migrate the thing that has consensus, quorum, and one single IP address the entire cluster depends on ]

Guess which one we pressed.

Throughout this post the providers are Cloud A (where we lived) and Cloud B (where we were going). Every IP, hostname, and path below is fake. `10.20.30.250` and `api.example-cluster.internal` are documentation placeholders. The architecture is real. The coordinates are fiction, deliberately.

— — —

Act 0: The thing you cannot drain

Here’s what we had on Cloud A — a split control plane, the shape you naturally get when a provider hands you cheap managed instance groups:

Cloud A
├─ 3 × etcd-only nodes
│     (just the database, no API)
├─ 3 × controlplane-only nodes
│     (apiserver, scheduler, controllers)
└─ N × workers
      (eBPF CNI, kube-proxy disabled)
            ↑
   Managed L4 load balancer
   → api.example-cluster.internal:6443

That load balancer was the front door. DNS pointed at it. It health-checked the control-plane nodes. It had worked flawlessly for years — which is exactly why nobody had ever thought hard about it.

Here’s the part that makes this different from a worker migration, and I want to be blunt:

You cannot drain etcd.

There is no `kubectl drain` for consensus. Every member you add or remove changes the live arithmetic of how many machines can catch fire right now before the cluster stops accepting writes.

Workers let you fail forward. A control plane only lets you fail back — and only if you were careful enough to still have a back to fail to.

❌ What you do to a worker node: `kubectl drain node-01`

✅ What you do to an etcd member: stare at the member list for nine minutes whispering “three… it has to stay three”

— — —

Act 1: The load balancer that wasn’t there

First decision. Cloud B has a managed load balancer too. Obvious port, right? Lift the config, point the DNS, home by Thursday.

We killed that idea in about an hour, once someone drew the dependency arrows on a whiteboard and the room went quiet.

A managed load balancer is provisioned by an API call, configured by a controller, and points at instances discovered through the provider’s inventory. Fine at steady state. Now read it again during a migration: the instances it must discover don’t exist yet, and the controller that would configure it wants to run inside the cluster it’s trying to make reachable.

So we threw it out and made the control-plane nodes announce their own anycast VIP over BGP. The fabric already spoke BGP for the worker CNI — new tier, not a new protocol.

Then we hit the bootstrap paradox properly. The standard way to run a VIP manager is leader election: nodes race through the Kubernetes API, and the winner claims the address

Sit with that for a second.

Leader election needs → the Kubernetes API

The Kubernetes API needs → the VIP

The VIP needs → leader election

(all three pointing, nobody moving)

So: no leader election. All three nodes announce the same `/32`, the fabric installs a three-way ECMP route, done. No lease. No API dependency. No circle.

The honest trade: failover is now a BGP hold timer — seconds, not the sub-second failover the leader-election docs brag about. Every client of a control-plane endpoint retries. Seconds is fine.

We wrote that down as an accepted cost in advance, which is the only way accepted costs stay accepted instead of becoming a postmortem line item.

The same logic drove two more decisions:

Split → stacked. Six machines became three, each running its own apiserver and its own etcd member. Not for cost — so that a node which boots healthy is immediately useful, with no ordering dance across two tiers on a network we were still learning.

Static pod on day 0, DaemonSet on day 2. There’s no cluster to schedule onto at first boot, so the VIP speaker ships as a kubelet static pod written by instance metadata. Once things are healthy, a GitOps DaemonSet takes over and a flag turns the bootstrap path off.

Every weird-looking choice here traces back to one question: what is allowed to exist before the cluster does?

— — —

Act 2: The first node boots, and nothing works

Image built. Terraform applied. Three machines came up on Cloud B.

The server process refused to start on all three.

”it’s a stacked node, it’s self-contained, it’ll just come up healthy”

— me, forty minutes earlier

Two bugs. Both embarrassing. Both extremely instructive.

Bug one: the config was partial. Our node image baked a complete, correct config. Our cloud-init then overwrote it with its own — which was missing the CNI setting and the `disable-kube-proxy` flag. Those must match the existing cluster exactly or the server process simply declines to exist. The running cluster used an eBPF CNI with kube-proxy disabled; our joining node politely asked to join with default settings and was shown the door.

The lesson isn’t “we forgot two lines.” It’s this: exactly one layer is allowed to be authoritative for a given config file. A partial overwrite is strictly worse than no overwrite, because it looks like it worked.

Bug two: the wrong disk path. Cloud B attaches block volumes as SCSI. Our cloud-init hardcoded the virtio path from Cloud A. So `mkfs` failed, the mount failed, and etcd quietly fell back to the root disk — a non-fatal failure that would have become extremely fatal later, at 3am, under write pressure.

Then DNS. The nodes couldn’t pull images. The error said ”image not found” — so naturally we spent a while investigating a missing image.

It wasn’t. The provider’s DHCP was handing out a resolver that was simply dead.

The two hard things in computer science:

1. Cache invalidation
2. Naming things
3. ~~Off-by-one errors~~ It’s always DNS
4. Believing the error message

We fixed it live with a resolver override to keep moving — then immediately moved the fix into the image as a static resolver file.

That distinction became a house rule, and it’s the single most important sentence in this post:

A live fix is a diagnostic. It is not a solution.

It bought information. It isn’t done until it’s baked into an image or a template. Because a cluster where any node’s state depends on something a human typed once is a cluster you cannot rebuild — and a migration is nothing but rebuilding.

— — —

Act 3: The VIP that pings but never answers

Node joined. etcd grew. Time to test the VIP.

Ping from the node’s own address: clean, 0% loss.

Ping sourced from the VIP: 100% loss.

Off-node connections were even better. The SYN arrived at the node. We could see it. The SYN-ACK went out. It never came back.

packet arrives ✅ · reply is sent ✅ · reply does not exist ❌

The standard way to let a machine use an address that isn’t its assigned one is to declare it as an allowed address pair on the network port. We’d done that. Cloud B accepted the declaration, stored it, returned it in API responses — and ignored it completely. Anti-spoof filtering ate every VIP-sourced packet in silence.

The fix: disable port security on the control-plane ports outright. Which then forbids attaching a security group to those ports. Which then means ingress filtering has to move to the host firewall.

One toggle, three consequences. Write all three down the moment you flip it, because the second and third will otherwise be discovered by someone else, six weeks later, during an audit.

— — —

Act 4: The quorum walk

Now the real thing. The part with an actual non-recoverable failure mode.

Before touching anything real, we built a file-backed fake cluster and ran the actual orchestration scripts against it. Not a diagram. Not a dry-run flag. A simulated etcd membership our real code could add to, remove from, and count.

It found three sequencing bugs at zero risk. One was the good one: the first phase transiently pushed us to an even member count — precisely the state where you’ve added a failure domain without buying a single bit of fault tolerance. Four members tolerate exactly as many failures as three. You’ve doubled your surface area for nothing.

🧠 Migrate all three at once, it’ll be fine

🧠🧠 Migrate one at a time

Get Prateek Raj’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

🧠🧠🧠 Remove before you add — never rest on an even count

🧠🧠🧠🧠 Rehearse the whole thing against a cluster made of text files before you touch production

The sequence we actually walked:

3 → 4 → 3 → 4 → 3 → 4 → 3
 ↑
 never below 3
 never resting on an even count
 one member in flight, always

Then the plan got rejected by our own guardrail — which, credit where due, is what guardrails are for. The cluster module validated that control-plane count must be an odd number ≥ 3, killing the elegant ramp-from-one plan. Fine. We brought all three Cloud B nodes up as a joining group, let etcd grow, then retired Cloud A’s members one at a time.

Somewhere in here we also found the worker CNI and the VIP speaker fighting over port 179. Two BGP daemons, one port — and the worst part, silent failure: the VIP worked perfectly while the CNI’s peering sat in `active` forever and nothing alerted, because nothing was technically down.

Fix: exclude control-plane nodes from the CNI’s BGP node selector.

Rule: decide which daemon owns BGP on each class of node, and encode it in a selector — not in a comment, not in someone’s memory.

— — —

Act 5: The ghost VIP

We rebuilt the image to bake in all the fixes. Rolled it out. The VIP died.

But this time it died beautifully:

- BGP session: established ✅
- Route advertised to both reflectors: yes ✅
- Route installed in the fabric: yes ✅
- VIP reachable from anywhere in the universe: absolutely not ❌

”the route is in the routing table”
— the routing table, correctly, about an address that existed on no interface anywhere

The daemon was announcing an address it had never claimed locally. In BGP mode it only assigns the VIP to a local interface when given an explicit router ID. Our image rebuild had switched to deriving identity from a router interface instead — which peers fine, advertises fine, and never binds the address.

The documentation said the router ID was irrelevant in that mode.

The packets disagreed. The packets win. Always.

Fix: set it explicitly, rendered at boot from the node’s own address. Instant.

Bonus landmine from the same afternoon, free to you: the kubelet parses every file in the static-pod directory. Not every `.yaml`. Every file. A `kube-vip.yaml.pre-change.bak` sitting next to the live manifest shadowed our edits, and no amount of restarting anything took effect until it was moved out of the folder.

Back things up somewhere else. Anywhere else.

— — —

Act 6: The realization that actually saved the project

Around the fourth time someone asked ”wait — does that need a node rebuild, or just a GitOps sync?”, we stopped and did something better than fixing another bug.

We stopped treating “the migration” as one thing.

A control-plane migration is a sequence of distinct state transitions, each with its own blast radius, its own rollback, and its own way of proving it worked. Once you classify them, the classification tells you the safe sequencing and who has to approve it.

We landed on five classes.

Class 1 — Image. OS packages, the cluster binary, the etcd client, SSH CA trust, static resolver, the VIP config skeleton.
Applied by: rebuild + node replace. Rollback: previous image tag. Blast radius: one node at a time.
Never applied to a running node. Uncomfortable at first, liberating forever — no running node’s config has a history, so “what’s on this box?” is answered entirely by its image tag.

Class 2 — Boot metadata. Join token, VIP address and BGP peers, node labels and taints, etcd device, CNI mode, API TLS SANs.
Applied by: node recreate. Rollback: previous metadata + recreate. Blast radius: one node at a time.
Rendered at first boot. Must be complete — see Act 2, where it wasn’t.

Class 3 — Topology. Adding or removing an etcd member or a control-plane node.
Applied by: orchestration, one member at a time. Rollback: re-add the removed member. Blast radius: the entire cluster.
The only class where order of operations is load-bearing and the failure mode is permanent.

Class 4 — Declarative infra. Machines, volumes, ports, security groups, DNS.
Applied by: PR + pipeline. Rollback: revert the PR, re-apply. Blast radius: varies — DNS is cluster-wide.
No local applies. Ever. Including during the migration. If the migration needed a step, that step became a PR.

Class 5 — Cluster runtime. GitOps: CNI BGP selectors, admission policies, the day-2 DaemonSet, labels and taints.
Applied by: GitOps sync. Rollback: revert the commit. Blast radius: namespace → cluster-wide.

Now ”can I just fix this by editing a file on the box?” has a one-step answer: only if it’s Class 5 — and even then you’re editing git, not the box.

Two things this model caught immediately:

A Class-4 flag became the entire cutover. Our cluster module owned the API endpoint’s DNS records. On the very first apply it would have cheerfully repointed the live endpoint at a VIP that nothing was announcing yet — breaking the cluster in order to build it. We added a `manage_dns` boolean, defaulted true, set false for the whole migration, flipped it at the end.

That boolean is the cutover. The entire migration compiles down to one bool.

A Class-5 change can’t substitute for a Class-1 one. Our node image baked CNI manifests. The cluster had an admission policy saying the CNI is GitOps-managed and nothing else may write it. So every new node’s baked manifests were rejected at admission — harmless, the node joined anyway, but a flat contradiction between two layers.

Right fix: delete the manifests from the image. Not weaken the policy. Never weaken the policy.

— — —

Act 7: Measuring what users actually feel

Before any of this, we built a small out-of-band monitor. It probed both API front-ends — Cloud A’s load balancer and Cloud B’s VIP — from both sides of the network, via a two-pod DaemonSet with one pod pinned per provider.

Four probes: A-from-A, A-from-B, B-from-A, B-from-B. Latency, success rate, rolling sparkline.

That 2×2 is the whole point. A VIP that works from its own provider but not from the other one is a broken migration that looks green on a single-sided dashboard.

Two traps it taught us:

Health endpoints lie through proxies. Our normal cluster access ran through a management proxy. Hitting raw `/healthz` through it returned the proxy’s own HTML error page, which parsed as “DOWN” — a beautiful false alarm on a perfectly healthy cluster. Reachability has to be measured with a call whose response can only come from an apiserver.

Identity-matching rules drift. Our node counter matched nodes by name prefix. Then the naming scheme changed, and it began counting new-provider nodes as old-provider ones — a monitor confidently reporting a migration that had not happened.

Match on labels you control, not names that drift.

— — —

Act 8: The cutover (sorry, it’s boring)

The switch was a DNS change with a 60-second TTL. The endpoint stopped resolving to Cloud A’s load balancer and started resolving to the Cloud B VIP.

That’s it. That’s the moment. Months of work, and the event itself was one record edit — because both paths had been live and serving simultaneously for weeks beforehand.

”so what happened during the cutover?”
”…nothing. Nothing happened. That was the point.”

Final state, verified from inside the cluster: exactly three etcd members, all on Cloud B, zero learners, all three at the same applied index, no lag. Cloud A’s members gone from the member list. The endpoint resolving to the VIP and nothing else.

All the work goes into making both paths simultaneously correct. Then the switch is a DNS record — and a DNS record is reversible.

Act 9: What we found after we’d already celebrated

This is the part most migration posts skip. So:

A node label never applied. The control-plane nodes were missing their node-pool label. The taint — which does the real scheduling protection — was correctly in place, so impact was cosmetic. But it was cosmetic by luck: the label and the taint shipped in the same change, and only one landed.

Verify each field, not the change.

A helper script was missing two functions its sibling had. The surge-replacement script lacked the VIP withdrawal step that the ordinary replacement script had.

Sit with what that means: a node that’s been drained and removed from etcd keeps announcing the VIP until the machine actually powers off. So the fabric keeps ECMP-ing a full third of API traffic straight into a black hole.

Two scripts that do almost the same thing will drift. And the drift will be in the step you only need during a failure.

And then we did it twice more. The first cluster was the expensive one — it bought the taxonomy, the rehearsal harness, the probe matrix, and all nine of these scars. Staging and production followed it onto the same architecture, and they were dramatically less interesting, which is the entire return on the first one.

That’s worth saying plainly, because it’s the actual payoff of everything above: the first migration is where you pay for the procedure. The second and third are where you find out whether you actually wrote one down.

So here’s the thing nobody has asked yet

Read back through all of that.

We rebuilt a load balancer out of a routing protocol. We rewrote the node image five times. We rehearsed etcd quorum arithmetic against a cluster made of text files. We found out a cloud silently ignores a networking primitive it claims to support. We built a 2×2 probe matrix to watch two clouds watch each other. We invented a five-class taxonomy of change just to answer ”does this need a reboot?”

And at the end of it all, the user-visible result was:

absolutely nothing.

Same cluster. Same API endpoint. Same workloads. Same SLOs. Nobody filed a ticket. Nobody noticed. If we’d done our job perfectly, the only evidence it ever happened would be this blog post.

”You spent months making sure nothing changed?”

”We spent months making sure nothing changed — for them.”

Which leaves the only question that actually matters. The one I’ve carefully not answered anywhere in nine acts:

Why?

Why move a control plane at all? Why take a system that worked, that had been stable for years, that nobody was complaining about — and deliberately walk it across a tightrope between two clouds, one etcd member at a time?

There is an answer. It’s a good one. It’s the reason this project existed, the reason it got funded — and the reason we went back and did it to staging and production too, on purpose, after we already knew exactly how much it would hurt.

That last part is the tell. Nobody walks this tightrope three times for a reason that isn’t worth it.

But it’s not a technical answer. And it doesn’t fit in this post.

If you want to know why — leave a comment, or DM me. I’ll tell you directly. 📩

— — —

All identifiers here — provider names, hostnames, IP ranges, subnets, file paths, tokens — are fictional stand-ins. `10.20.30.0/24` and `api.example-cluster.internal` are documentation placeholders and correspond to nothing real. The architecture is accurate; the coordinates are not.

If you’ve run a control-plane migration and hit a landmine I didn’t list — I want to hear about it. Especially the silent ones.