Why Kubernetes Upgrades Feel Harder on Platforms Like Google Anthos
Kubernetes upgrades are often described as routine operations, but that assumption quietly breaks down the moment Kubernetes becomes part of a platform. Google Anthos is not “just Kubernetes with add-ons”; it is a full lifecycle management system with strong opinions about control, reconciliation, and ownership. When Anthos evolves, the upgrade is no longer a version bump — it becomes a platform transition.
This becomes very visible when moving from Anthos 1.32 to 1.33. On paper, the change looks incremental. In reality, it introduces architectural expectations that reshape how clusters are defined, validated, and reconciled. The introduction of mandatory advanced cluster configurations is a good example of this shift. These changes are not optional toggles; they redefine the baseline contract between Anthos and the clusters it manages. Existing clusters must align with this new contract before the platform will allow them to progress.
One of the most important things to understand about Anthos is its layered architecture. Unlike standalone Kubernetes, Anthos separates responsibility across the admin workstation, the admin cluster, and the user clusters. The admin workstation is a controlled execution environment where lifecycle actions are initiated. The admin cluster is the authoritative brain of the platform — it owns configuration, reconciliation, validation, and state enforcement. User clusters are downstream systems that continuously converge toward the desired state defined upstream.
Because of this design, upgrades are inherently ordered and stateful. The admin cluster must always be upgraded first, because it defines what “correct” means for everything else. If the admin cluster does not understand the new platform version, it cannot safely manage clusters that attempt to move ahead. This ordering is not a best practice — it is a structural requirement enforced by the platform.
As Anthos versions advance, the number of moving parts involved in an upgrade increases. Controllers evolve, CRDs change, validation rules tighten, and reconciliation behavior becomes more assertive. Components that usually sit quietly in the background suddenly become critical path dependencies. Node lifecycle controllers, inventory resources, admission webhooks, cluster profile deployers, and configuration operators all participate in driving the cluster toward the new desired state.
Get keshav agarwal’s stories in your inbox
Join Medium for free to get updates from this writer.
This is where many upgrades appear to “stall” or enter ambiguous error states. From the outside, things can look deceptively healthy. Pods may exist, nodes may register, and logs may not show obvious failures. Under the hood, however, reconciliation loops are blocked by unmet expectations. A single missing field, annotation, or structural mismatch in a resource can halt progress across the system, not because something is broken, but because the platform is waiting for the world to match its new model.
Anthos makes heavy use of continuous reconciliation. This is both its strength and its challenge. Manual changes applied directly to user clusters may seem effective at first, but if they are not aligned with the desired state owned by the admin cluster, they will be reverted. Multiple controllers may touch the same resources, and ownership is not always obvious unless you understand which component is responsible for reconciliation at each layer.
Identity and access add another dimension to this complexity. OIDC configuration, client identity resources, and authentication flows are not standalone settings; they are part of a larger platform contract. During upgrades, these contracts are revalidated. Changes that appear correct at the surface may be overwritten if they conflict with upstream configuration sources. Understanding where truth is defined — and which controller enforces it — becomes essential.
Infrastructure performance also plays a larger role than many expect. Core platform components such as DNS, storage-backed controllers, and networking layers are no longer just “supporting services.” They are active participants in the control plane. Performance degradation in any of these layers can manifest as control-plane instability, delayed reconciliation, or cascading failures that appear unrelated at first glance.
What makes Anthos upgrades particularly challenging is not the presence of bugs or errors, but the density of interactions. Each component is doing the right thing in isolation, yet the system as a whole can become blocked by a subtle misalignment. In some cases, progress resumes only after the platform is given exactly the signal it expects — often in the form of configuration that aligns legacy state with new platform assumptions.
This is why Anthos upgrades should be approached as architectural changes rather than operational tasks. They require a mental model of how the platform thinks, not just how Kubernetes behaves. Reading logs becomes less about finding explicit errors and more about understanding which controller is waiting, which dependency is unmet, and which layer owns reconciliation authority.
Anthos represents a broader shift in how Kubernetes platforms are built. As platforms mature, they trade flexibility for consistency, governance, and safety at scale. The cost of that trade-off is complexity during transitions. Upgrades surface the invisible dependencies that steady-state operations quietly rely on.
For engineers working with Anthos, the real skill is not executing commands, but understanding systems. Knowing how admin clusters govern user clusters, how reconciliation flows across layers, and how configuration ownership is enforced is what turns long, uncertain upgrades into predictable ones. Anthos does not just run Kubernetes — it teaches you how platforms behave when Kubernetes becomes infrastructure rather than tooling.
