<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lubomir Tobek on Medium]]></title>
        <description><![CDATA[Stories by Lubomir Tobek on Medium]]></description>
        <link>https://medium.com/@lubomir-tobek?source=rss-61de7b0f9608------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QJ1fivbYEJTJkSihGsZ46A.png</url>
            <title>Stories by Lubomir Tobek on Medium</title>
            <link>https://medium.com/@lubomir-tobek?source=rss-61de7b0f9608------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:38:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lubomir-tobek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Architecting Kubernetes Availability on vSAN in VCF 9.1.1: Eight Topology Design Patterns]]></title>
            <link>https://medium.com/@lubomir-tobek/architecting-kubernetes-availability-on-vsan-in-vcf-9-1-1-eight-topology-design-patterns-018bebce8a36?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/018bebce8a36</guid>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 18:15:01 GMT</pubDate>
            <atom:updated>2026-11-10T05:25:43.688Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 2 of 2 — single-zone, multi-zone and cross-zone design patterns drawn one by one, what each one survives, and a decision path from the workload to the topology.</em></p><p>This article continues the series started in <a href="https://medium.com/@lubomir-tobek/architecting-kubernetes-on-vsan-in-vcf-9-1-1-zones-volumes-and-storage-clusters-2907cd2d88da">Architecting Kubernetes on vSAN in VCF 9.1.1: Zones, Volumes and Storage Clusters</a>. In that first part, I covered the architectural building blocks that the Broadcom technical paper Kubernetes Topologies with vSAN uses for VCF 9.1.1: storage clusters, RWO and RWX volumes, single-zone and multi-zone Supervisors, multi-cluster zones, Auto-RAID and the WaitForFirstConsumer binding mode. The paper combines these parts into eight topology design patterns and gives each one a support stance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyjGaDXWRZJomrbUNhjVhQ.png" /></figure><p>This article draws all eight. It shows:</p><ul><li>which design patterns keep storage inside a zone and which extend it across zones</li><li>what each design pattern survives, and how long recovery takes where the paper reports it</li><li>the caveats of the patterns rated <em>Supported with Caveats and Recommended</em></li><li>why the paper generally recommends the cross-zone patterns</li><li>a decision path from the workload to the topology, and a checklist for the architecture review</li></ul><pre>+----------------------+-------------------------------------------------------+<br>| Item                 | Value                                                 |<br>+----------------------+-------------------------------------------------------+<br>| Source               | Kubernetes Topologies with vSAN, Broadcom, Oct 2026   |<br>| Platform             | VCF 9.1.1, vSAN 9.1.1, Supervisor, VKS                |<br>| Design groups        | single-zone (1-2), multi-zone (3-5), cross-zone (6-8) |<br>| Stances              | Supported, Supported with Caveats, Recommended        |<br>| My lab, for contrast | one zone, one HCI cluster, not disaggregated          |<br>+----------------------+-------------------------------------------------------+</pre><p>The paper places every design pattern in one matrix. In my own words:</p><pre>+---+-------------+------------------------------------+---------+------------------------+<br>| # | Group       | Storage layout                     | Volumes | Stance                 |<br>+---+-------------+------------------------------------+---------+------------------------+<br>| 1 | single-zone | storage cluster + File Services    | RWO RWX | Supported              |<br>| 2 | single-zone | stretched cluster + File Services  | RWO RWX | Supported              |<br>| 3 | multi-zone  | separate storage cluster per zone  | RWO     | Supported              |<br>| 4 | multi-zone  | per-zone cluster + File Services   | RWO RWX | Supported              |<br>| 5 | multi-zone  | per-zone + shared File Services    | RWO RWX | Supported with Caveats |<br>| 6 | cross-zone  | one stretched cluster across zones | RWO     | Recommended            |<br>| 7 | cross-zone  | stretched cluster + File Services  | RWX     | Recommended            |<br>| 8 | cross-zone  | one cluster, fault domain per zone | RWO     | Recommended            |<br>+---+-------------+------------------------------------+---------+------------------------+</pre><p>The diagrams below use the same symbols:</p><pre> [CP]   Supervisor control plane VM      [VKS]  VKS node VMs<br> (RWO)  block volume                     (RWX)  file volume, vSAN File Services<br> &lt;==&gt;   synchronous vSAN replication     &lt;--&gt;   replication by the application</pre><blockquote><em>Part 1 — Single-Zone Design Patterns</em></blockquote><p><strong>Design 1: single-zone vSAN storage cluster with File Services</strong></p><pre>+--------------------------------------------------------------+<br>| Zone A                                                       |<br>|                                                              |<br>|   vSphere cluster (compute):  [CP]  [VKS]  [VKS]  [VKS]      |<br>|                                 |                            |<br>|                                 | remote mount               |<br>|                                 v                            |<br>|   vSAN storage cluster + File Services:  (RWO)  (RWX)        |<br>+--------------------------------------------------------------+<br>   Highly tolerant of: failures inside Zone A<br>   Does not survive:   the loss of the entire Zone A</pre><p>Compute and storage reside in the same zone. The paper positions this pattern for simpler environments and for workloads that do not need more than one failure domain, such as a data center, a campus building, a room or a rack. It fits test and development as well as AI/ML. It cannot tolerate the failure of the entire zone, but vSphere HA, vSAN and the instantiation of containers by VKS make it highly tolerant of failures inside the zone.</p><p>Architectural considerations:</p><ul><li><strong>Closest to my lab, but not the same:</strong> my Supervisor also runs in one zone, but compute and storage are one HCI cluster. Design 1 uses a separate storage cluster.</li><li><strong>Inside the zone, the platform recovers:</strong> host failures are handled by vSphere HA and, for VKS nodes, by MachineHealthCheck, as I showed in <a href="https://medium.com/@lubomir-tobek/losing-an-esx-host-under-a-running-supervisor-vsphere-ha-vs-machinehealthcheck-3c0ae7468b03">Losing an ESX Host Under a Running Supervisor</a>.</li></ul><p><strong>Design 2: single-zone vSAN stretched storage cluster with File Services</strong></p><pre>+-------------------------------------------------------------------------+<br>| Zone A (the Supervisor sees one zone)                                   |<br>|                                                                         |<br>|  Preferred site           Non-preferred site        Witness site        |<br>| +-------------------+    +-------------------+   +-------------------+  |<br>| | compute hosts     |    | compute hosts     |   | vSAN witness      |  |<br>| | [CP] [VKS]        |    | [VKS]             |   |                   |  |<br>| | data copy         |&lt;==&gt;| data copy         |   | quorum            |  |<br>| +-------------------+    +-------------------+   +-------------------+  |<br>|                                                                         |<br>|       vSAN stretched storage cluster + File Services: (RWO) (RWX)       |<br>+-------------------------------------------------------------------------+</pre><p>Compute and storage stay in one zone, but the storage is stretched across the vSAN fault domains <em>Preferred Site, Non-Preferred Site</em> and <em>Witness Site</em>. Data remains available when one of these fault domains fails. The paper reports the following validation results:</p><pre>+---------------------+---------------------------------------------------+<br>| Failed site         | Result in the paper&#39;s validation testing          |<br>+---------------------+---------------------------------------------------+<br>| Witness site        | no disruption                                     |<br>| Non-preferred site  | no disruption                                     |<br>| Preferred site      | recovery by vSphere HA: Supervisor VMs about      |<br>|                     | 15-20 minutes, VKS VMs about 10-15 minutes        |<br>+---------------------+---------------------------------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>VM placement in the diagram is an example:</strong> where the Supervisor and VKS VMs run at the time of a failure depends on DRS and on any site affinity rules.</li><li><strong>Restart order matters:</strong> after a preferred site failure, many VMs restart at once, and vSphere HA restart priority decides which come back first. I looked at how it applies to the Supervisor and VKS nodes in <a href="https://medium.com/@lubomir-tobek/who-restarts-first-vsphere-ha-restart-priority-for-the-supervisor-and-vks-nodes-31ea21d4209a">Who Restarts First</a>.</li><li><strong>Network:</strong> the RTT between the data sites must not exceed 5 ms.</li></ul><blockquote><em>Part 2 — Multi-Zone Design Patterns: Storage Stays in the Zone</em></blockquote><p><strong>Design 3: multi-zone vSAN storage cluster isolated to each zone</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A             |    | Zone B             |    | Zone C             |<br>|                    |    |                    |    |                    |<br>| [CP] [VKS]         |    | [CP] [VKS]         |    | [CP] [VKS]         |<br>| app replica 1      |&lt;--&gt;| app replica 2      |&lt;--&gt;| app replica 3      |<br>|     |              |    |     |              |    |     |              |<br>| vSAN storage       |    | vSAN storage       |    | vSAN storage       |<br>| cluster A  (RWO)   |    | cluster B  (RWO)   |    | cluster C  (RWO)   |<br>+--------------------+    +--------------------+    +--------------------+<br>     Data between zones is replicated by the application, not by vSAN.</pre><p>Each of the three zones has its own dedicated vSAN storage cluster. The pattern fits applications such as Cassandra, PostgreSQL or MongoDB, whose built-in replication keeps them available when a zone fails. The WaitForFirstConsumer StorageClass co-locates every PV with its pod in the same zone. The pattern provides RWO volumes and, in addition to the resilience inside each zone, tolerates the failure of one zone.</p><p>Architectural considerations:</p><ul><li><strong>The</strong> -latebinding <strong>class is mandatory:</strong> with Immediate binding, a PV can be created before the scheduler has chosen the zone of its pod. I explained the difference in the <a href="https://medium.com/@lubomir-tobek/architecting-kubernetes-on-vsan-in-vcf-9-1-1-zones-volumes-and-storage-clusters-2907cd2d88da">first part of this series</a>.</li><li><strong>A lost zone takes its volumes with it:</strong> the replicas in the failed zone are unavailable until the zone returns, and the application must keep serving from the other two.</li><li><strong>Spread the replicas explicitly:</strong> a topology spread constraint or a required pod anti-affinity rule with the topology key topology.kubernetes.io/zone places each replica in a different zone.</li></ul><p><strong>Design 4: multi-zone vSAN storage cluster isolated to each zone, with per-zone File Services</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A             |    | Zone B             |    | Zone C             |<br>|                    |    |                    |    |                    |<br>| [CP] [VKS]         |    | [CP] [VKS]         |    | [CP] [VKS]         |<br>|     |              |    |     |              |    |     |              |<br>| vSAN storage       |    | vSAN storage       |    | vSAN storage       |<br>| cluster A          |    | cluster B          |    | cluster C          |<br>| + File Services    |    | + File Services    |    | + File Services    |<br>| (RWO) (RWX)        |    | (RWO) (RWX)        |    | (RWO) (RWX)        |<br>+--------------------+    +--------------------+    +--------------------+<br>     An RWX share is available only within its own zone.</pre><p>Design 4 is the more flexible variant of Design 3. vSAN File Services runs on each storage cluster, so every zone provides NFS-based RWX volumes in addition to RWO volumes. The shared volumes are available within the zone.</p><p>Architectural considerations:</p><ul><li><strong>Per-zone RWX is not cross-zone RWX:</strong> it suits pods in the same zone that share files. Pods in another zone use the share of their own zone.</li><li><strong>Three file services to operate:</strong> each zone runs its own File Services instance, with its own capacity and lifecycle.</li></ul><p><strong>Design 5: multi-zone vSAN storage cluster isolated to each zone, with an additional storage cluster for centralized shared File Services</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A             |    | Zone B             |    | Zone C             |<br>| [CP] [VKS]         |    | [CP] [VKS]         |    | [CP] [VKS]         |<br>| local vSAN:        |    | local vSAN:        |    | local vSAN:        |<br>| CP, node OS disks  |    | CP, node OS disks  |    | CP, node OS disks  |<br>+---------+----------+    +---------+----------+    +---------+----------+<br>          |                         |                         |<br>+---------+-------------------------+-------------------------+----------+<br>| Additional vSAN stretched storage cluster across all three zones       |<br>| + File Services, user PVs: (RWO) (RWX)                                 |<br>+------------------------------------------------------------------------+</pre><p>Each zone keeps its own storage cluster, and an additional vSAN storage cluster with File Services spans all three zones and provides storage-level resilience for RWO and RWX volumes. All control plane data, such as the Supervisor control plane and the OS disks of the VKS nodes, stays on the local storage of each zone. All user persistent volumes reside on the stretched storage cluster. According to the paper, when a full zone fails and vSphere HA restarts the VKS worker VMs on another site, recovery of the VKS VMs and their pods takes about 10–15 minutes.</p><p>Architectural considerations:</p><ul><li><strong>The caveat:</strong> the pattern is supported in VCF 9.1.1, but the vSAN datastore must be mounted even when RWX is the only volume type in use.</li><li><strong>Validate the compute layout:</strong> vSphere HA restarts VMs only within one vSphere cluster. The restart on another site described in the paper therefore depends on how the compute clusters are laid out across the sites, and this has to be checked during the design.</li><li><strong>Target workload:</strong> according to the paper, the pattern could conceptually serve applications that are already protected by application-level HA and need NFS-based RWX volumes shared across zones.</li><li><strong>Network:</strong> the stretched storage cluster requires an RTT of 5 ms or less between the data sites.</li></ul><blockquote><em>Part 3 — Cross-Zone Design Patterns: Storage Spans the Zones</em></blockquote><p>The paper describes the cross-zone patterns as an adaptation of the multi-zone model in which storage comes from a single vSAN cluster, stretched or not stretched. High availability moves from the application layer to the vSAN storage layer. Because data is written synchronously, the pods in the surviving zones can take over without waiting for storage to become ready. For this reason, the paper generally recommends the cross-zone patterns over the other options.</p><p><strong>Design 6: cross-zone vSAN stretched storage cluster</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A             |    | Zone B             |    | Zone C             |<br>| [CP] [VKS]         |    | [CP] [VKS]         |    | [CP] [VKS]         |<br>| data site          |&lt;==&gt;| data site          |    | quorum site        |<br>+---------+----------+    +---------+----------+    +---------+----------+<br>          |                         |                         |<br>+---------+-------------------------+-------------------------+----------+<br>|        One vSAN stretched storage cluster, one datastore: (RWO)        |<br>+------------------------------------------------------------------------+</pre><p>vSAN provides zone-level resilience by mirroring data synchronously across two of the three sites and uses the third site for quorum arbitration. VKS runs on compute resources in all three zones, backed by a single vSAN datastore that remains available after a zone outage. When a zone fails, worker nodes in the other zones take over the workloads with the same persistent volumes.</p><p>Architectural considerations:</p><ul><li><strong>The caveat:</strong> RWO volumes remain in an attached state to the failed zone. In VCF 9.1.1, a manual detach and attach of a CNS volume attached to a disconnected VM may not complete correctly because of a lock on the volume. The paper points to <a href="https://knowledge.broadcom.com/external/article/311105/manually-removing-stale-or-orphaned-virt.html">Broadcom KB 311105</a> and to the TechDocs procedure <em>Recover Orphaned Virtual Machines</em>. This procedure belongs in the runbook before the pattern goes live.</li><li><strong>Target workload:</strong> mission-critical applications that need only block-based RWO volumes and must stay resilient across zones.</li><li><strong>Network:</strong> RTT of 5 ms or less between the data sites.</li></ul><p><strong>Design 7: cross-zone vSAN stretched storage cluster with File Services</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A             |    | Zone B             |    | Zone C             |<br>| [VKS] pods mount   |    | [VKS] pods mount   |    | [VKS] pods mount   |<br>|   the same share   |    |   the same share   |    |   the same share   |<br>+---------+----------+    +---------+----------+    +---------+----------+<br>          |                         |                         |<br>+---------+-------------------------+-------------------------+----------+<br>|       One vSAN stretched storage cluster + File Services: (RWX)        |<br>+------------------------------------------------------------------------+</pre><p>Design 7 runs vSAN File Services on the stretched cluster of Design 6. Pods in every zone can mount the same RWX volume, which provides continuous availability of the shared data.</p><p>Architectural considerations:</p><ul><li><strong>Cross-zone RWX without caveats:</strong> Design 5 also provides RWX across zones, but it is rated <em>Supported</em> with <em>Caveats</em>. Design 7 is rated <em>Recommended</em>.</li><li><strong>Target workload:</strong> mission-critical applications that also need shared persistent volumes.</li><li><strong>Network:</strong> RTT of 5 ms or less between the data sites.</li></ul><p><strong>Design 8: cross-zone shared vSAN storage cluster</strong></p><pre>+--------------------+    +--------------------+    +--------------------+<br>| Zone A = FD 1      |    | Zone B = FD 2      |    | Zone C = FD 3      |<br>| [CP] [VKS]         |    | [CP] [VKS]         |    | [CP] [VKS]         |<br>| vSAN hosts         |    | vSAN hosts         |    | vSAN hosts         |<br>| data components    |    | data components    |    | data components    |<br>+---------+----------+    +---------+----------+    +---------+----------+<br>          |                         |                         |<br>          +-------- one vSAN cluster, RTT 1 ms or less -------+</pre><p>This pattern does not use the vSAN stretched cluster construct. A single vSAN cluster spans all three zones, and its hosts are grouped into vSAN fault domains that align with the physical zones. Persistent volume data is distributed across the zones and remains available after a single full zone outage. Application-state replication is still required so that the application can continue operating after the zone failure.</p><p>Architectural considerations:</p><ul><li>Stretched cluster or fault domains:</li></ul><pre>+-----------------------+--------------------------------+--------------------------------+<br>| Aspect                | Design 6: stretched cluster    | Design 8: fault domains        |<br>+-----------------------+--------------------------------+--------------------------------+<br>| vSAN construct        | two data sites + quorum site   | three fault domains            |<br>| Maximum RTT           | 5 ms                           | 1 ms                           |<br>| Survives              | loss of one zone               | loss of one zone               |<br>| Application state     | other zones take over same PVs | own state replication required |<br>| Volumes               | RWO                            | RWO                            |<br>+-----------------------+--------------------------------+--------------------------------+</pre><ul><li>1 ms is a tight budget: in practice it points to zones on one campus or in closely connected data centers.</li></ul><blockquote><strong><em>Part 4 — The Eight Design Patterns Side by Side</em></strong></blockquote><pre>+---+----------+--------------------------+--------------------+--------+--------------+<br>| # | Volumes  | Site or zone failure     | Data copied by     | RTT    | Stance       |<br>+---+----------+--------------------------+--------------------+--------+--------------+<br>| 1 | RWO RWX  | zone loss not tolerated  | vSAN, inside zone  | -      | Supported    |<br>| 2 | RWO RWX  | site loss tolerated      | vSAN stretched     | &lt;=5 ms | Supported    |<br>| 3 | RWO      | one zone, app continues  | application        | -      | Supported    |<br>| 4 | RWO RWX* | one zone, app continues  | application        | -      | Supported    |<br>| 5 | RWO RWX  | one zone, ~10-15 min     | vSAN stretched     | &lt;=5 ms | With caveats |<br>| 6 | RWO      | one zone, same PVs       | vSAN synchronous   | &lt;=5 ms | Recommended  |<br>| 7 | RWX      | one zone, shared access  | vSAN synchronous   | &lt;=5 ms | Recommended  |<br>| 8 | RWO      | one zone, app replicates | vSAN fault domains | &lt;=1 ms | Recommended  |<br>+---+----------+--------------------------+--------------------+--------+--------------+<br> * RWX within each zone.  - : no storage replication between zones, no RTT value.</pre><blockquote><em>Part 5 — A Decision Path</em></blockquote><p>The paper closes with three recommendations: multi-zone local vSAN topologies for cloud-native, distributed applications; cross-zone vSAN stretched storage clusters for mission-critical and legacy applications; and multi-cluster zones in VCF 9.1.1 for flexible scaling. As a decision path:</p><pre>Does the application replicate its own data (Cassandra, Kafka, CockroachDB)?<br>|<br>+-- yes --&gt; Does it need RWX volumes?<br>|           |<br>|           +-- no --------------------------------------&gt; Design 3<br>|           +-- yes, shared inside each zone ------------&gt; Design 4<br>|           +-- yes, shared across zones ----------------&gt; Design 5 (caveats)<br>|<br>+-- no ---&gt; Must it survive the loss of a whole zone?<br>            |<br>            +-- no ---&gt; one failure domain --------------&gt; Design 1<br>            |           stretched within one zone -------&gt; Design 2<br>            |<br>            +-- yes --&gt; Which volumes does it need?<br>                        |<br>                        +-- RWO, RTT up to 5 ms ---------&gt; Design 6<br>                        +-- RWX, RTT up to 5 ms ---------&gt; Design 7<br><br>Design 8: RTT up to 1 ms between zones, no stretched cluster, and<br>          application-state replication in place</pre><p>The path is a starting point, not a replacement for sizing. Where failure domains are geographically separated, every topology still has to meet its minimum requirements, such as network bandwidth, network latency and physical host counts.</p><blockquote><strong><em>Part 6 — Architecture Review Checklist</em></strong></blockquote><pre>+----+------------------------------------------------------------------------+<br>| #  | Check                                                                  |<br>+----+------------------------------------------------------------------------+<br>| 1  | Multi-zone means exactly three zones under one vCenter                 |<br>| 2  | Workload networks identical in all zones; L2 or NSX overlay across all |<br>| 3  | Networking stack chosen up front: NSX VPC or VDS                       |<br>| 4  | One storage policy whose rules every zone&#39;s datastore can satisfy      |<br>| 5  | Multi-zone local vSAN: StorageClass with WaitForFirstConsumer          |<br>| 6  | RWX required: File Services on the storage cluster that serves it      |<br>| 7  | RTT measured: 5 ms between data sites, 1 ms between fault domains      |<br>| 8  | Bandwidth and host counts sized for the chosen topology                |<br>| 9  | Design 6: KB 311105 procedure tested and in the runbook                |<br>| 10 | Design 5: compute layout validated for vSphere HA restarts             |<br>| 11 | Multi-cluster zones: same datastore mounted on every cluster in a zone |<br>| 12 | HA restart priority set for the Supervisor and VKS node VMs            |<br>| 13 | PodDisruptionBudgets that do not block maintenance of a whole zone     |<br>+----+------------------------------------------------------------------------+</pre><p>Some of these checks have their own articles in this blog. Bandwidth for synchronous replication is covered in <a href="https://medium.com/@lubomir-tobek/strategic-bandwidth-sizing-for-vsan-stretched-clusters-in-vcf-9-0-a-roadmap-to-resilience-ce55545b96a2">Strategic Bandwidth Sizing for vSAN Stretched Clusters in VCF 9.0</a>. How a PodDisruptionBudget can hold up a drain is shown in <a href="https://medium.com/@lubomir-tobek/poddisruptionbudget-vs-rolling-update-in-a-vks-cluster-a-blocked-drain-and-nodedraintimeout-29045f008ad2">PodDisruptionBudget vs. Rolling Update in a VKS Cluster</a>. How a VKS cluster behaves when one of three control plane nodes is lost is described in <a href="https://medium.com/@lubomir-tobek/losing-one-of-three-control-plane-nodes-in-a-vks-cluster-remediation-etcd-quorum-and-a-two-minute-bcbaa1d6ef34">Losing One of Three Control Plane Nodes in a VKS Cluster</a>, which follows the same majority logic as a design that tolerates the loss of one of three zones.</p><blockquote><em>Summary</em></blockquote><pre>+-------------------------------------------+-------------------------------------------+<br>| Question                                  | Answer                                    |<br>+-------------------------------------------+-------------------------------------------+<br>| How many design patterns are described?   | eight, in three groups                    |<br>| Which ones are Recommended?               | the cross-zone patterns 6, 7 and 8        |<br>| Which one is Supported with Caveats?      | Design 5                                  |<br>| Which patterns offer RWX?                 | 1, 2, 4 (per zone), 5 and 7               |<br>| Cross-zone RWX rated Recommended?         | Design 7                                  |<br>| Design 2 preferred site failure?          | Supervisor ~15-20 min, VKS ~10-15 min     |<br>| Design 5 full zone failure?               | VKS VMs and pods ~10-15 min               |<br>| Known issue in Design 6?                  | volume lock on detach, see KB 311105      |<br>| Stretched cluster vs. fault domains RTT?  | 5 ms vs. 1 ms                             |<br>| Why are cross-zone patterns preferred?    | synchronous writes, no wait for storage   |<br>+-------------------------------------------+-------------------------------------------+</pre><blockquote><em>Conclusion</em></blockquote><p>The eight design patterns answer one architectural question in different ways: who keeps a copy of the data in another failure domain. In the single-zone patterns, the data never leaves the zone, and Design 2 adds a stretched cluster across sites within that zone. In the multi-zone patterns 3 and 4, the application replicates its data between zones and vSAN protects the data inside each zone, while Design 5 adds a stretched cluster for user volumes. In the cross-zone patterns, vSAN keeps the data synchronously available across zones, which is why compute can move as soon as a zone fails and why the paper generally recommends them.</p><p>The right pattern is therefore decided by the application first and by the network second. A Cassandra ring and a monolithic database with a single volume should not share the same topology, and on VCF 9.1.1 they do not have to. The paper provides a validated design pattern for each of them.</p><p>Ref:</p><p>Kubernetes Topologies with vSAN<br><a href="https://www.vmware.com/docs/vmw-kubernetes-topologies-with-vsan">https://www.vmware.com/docs/vmw-kubernetes-topologies-with-vsan</a></p><p>vSphere Supervisor Zone Models<br><a href="https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/design/design-library/self-service-iaas-deployment-models/vsphere-supervisor-zone-models.html">https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/design/design-library/self-service-iaas-deployment-models/vsphere-supervisor-zone-models.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=018bebce8a36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecting Kubernetes on vSAN in VCF 9.1.1: Zones, Volumes and Storage Clusters]]></title>
            <link>https://medium.com/@lubomir-tobek/architecting-kubernetes-on-vsan-in-vcf-9-1-1-zones-volumes-and-storage-clusters-2907cd2d88da?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/2907cd2d88da</guid>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 17:30:52 GMT</pubDate>
            <atom:updated>2026-11-10T05:41:56.075Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 1 of 2 — the architectural building blocks behind Kubernetes topologies on vSAN in VCF 9.1.1: storage clusters, volume types, zone models, StorageClass binding and the placement of availability.</em></p><p>Every VKS cluster in my recent articles runs in one zone, on one vSphere cluster with one vSAN datastore. That is a good lab and a reasonable starting point for many platforms. But as soon as someone asks what happens when a whole rack, room or site goes away, the answer no longer depends on Kubernetes alone. It depends on how zones, vSphere clusters and vSAN storage clusters are combined in the architecture.</p><p>On October 15, 2026, Broadcom published a technical paper, Kubernetes Topologies with vSAN, that maps these combinations for VCF 9.1.1 and describes eight topology design patterns. This two-part series works through the paper from an architect’s point of view and connects it to what I have already shown in my lab. This first article covers the building blocks and the design decisions they create. The second article walks through the eight design patterns, one diagram at a time, and ends with a decision path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S7QwPmrq6uw4usLDrGDrTQ.png" /></figure><p>This article shows:</p><ul><li>why the topologies in the paper assume disaggregated storage, and what a storage cluster is</li><li>what ReadWriteOnce and ReadWriteMany volumes require from the platform</li><li>why Kubernetes on bare metal needed its own storage replication, and what changes in VCF</li><li>how single-zone and multi-zone Supervisors differ, and what multi-cluster zones add in 9.1.1</li><li>how CNS, Auto-RAID and the volume binding mode put a volume into the right zone</li><li>where availability should be designed: in the application or in the storage</li><li>which design decisions have to be recorded before a topology is chosen</li></ul><pre>+---------------------+-----------------------------------------------------+<br>| Item                | Value                                               |<br>+---------------------+-----------------------------------------------------+<br>| Source              | Kubernetes Topologies with vSAN, Broadcom, Oct 2026 |<br>| Platform            | VCF 9.1.1, vSAN 9.1.1                               |<br>| Kubernetes layer    | vSphere Supervisor, vSphere Kubernetes Service (VKS)|<br>| Zone models         | single-zone, multi-zone with exactly three zones    |<br>| Volume types        | RWO (block), RWX (file, needs vSAN File Services)   |<br>| Design patterns     | 8: 2 single-zone, 3 multi-zone, 3 cross-zone        |<br>| My lab, for contrast| VCF-LAB, 5 ESX hosts, one zone domain-c9, HCI vSAN  |<br>+---------------------+-----------------------------------------------------+</pre><blockquote><em>Part 1 — Storage Clusters: Separating Compute from Capacity</em></blockquote><p>All topologies in the paper assume disaggregated storage. The vSphere clusters that run the Supervisor and the VKS node VMs consume a remote vSAN datastore, and the paper calls the cluster that provides it a storage cluster. It can be one of two things:</p><ul><li>a dedicated vSAN storage cluster</li><li>an existing vSAN HCI cluster whose datastore the vSphere clusters mount remotely</li></ul><p>In both cases the workloads and the storage are on different clusters. The term vSAN HCI in the paper does not mean that they are aggregated on the same cluster.</p><pre>+--------------------------------+        +-------------------------------+<br>|    vSphere cluster (compute)   |        |      vSAN storage cluster     |<br>|                                | remote |                               |<br>|  Supervisor control plane VMs  | mount  |  vSAN datastore               |<br>|  VKS control plane VMs         |-------&gt;|  storage policies (Auto-RAID) |<br>|  VKS worker VMs                |        |  optional vSAN File Services  |<br>+--------------------------------+        +-------------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>The datastore is a shared resource of the topology, not of one cluster:</strong> this is what allows a namespace in a multi-cluster zone (Part 4) to switch to another cluster without losing access to its data, provided the same datastore is mounted there.</li><li><strong>Compute and storage scale independently:</strong> adding hosts for VKS workers no longer forces adding capacity, and the other way round. The price is a dependency on the network between the compute cluster and the storage cluster.</li><li><strong>My lab is not disaggregated:</strong> compute and storage are one cluster, VCF-LAB with its local VCF-LAB-vSAN datastore. In <a href="https://medium.com/@lubomir-tobek/one-zone-five-hosts-where-the-supervisor-places-control-plane-and-vks-node-vms-5816d3e5a09f">One Zone, Five Hosts</a> I showed how the Supervisor places its VMs in that single cluster. The disaggregated model is the next architectural step.</li><li><strong>Scope:</strong> the paper notes that a specific deployment can have additional requirements or support limits that it does not cover. Every design still has to be validated against the current interoperability and support documentation.</li></ul><blockquote><em>Part 2 — RWO and RWX: What Each Volume Type Requires</em></blockquote><p>A pod requests storage with a PersistentVolumeClaim, and Kubernetes binds the claim to a PersistentVolume. The access mode of the claim decides what the platform has to provide:</p><pre>+---------------------+------------+---------------------+----------------------------+<br>| Access mode         | Type       | Mounted by          | Platform requirement       |<br>+---------------------+------------+---------------------+----------------------------+<br>| ReadWriteOnce (RWO) | block      | one node at a time  | vSAN datastore only        |<br>| ReadWriteMany (RWX) | file (NFS) | many pods and nodes | vSAN File Services enabled |<br>+---------------------+------------+---------------------+----------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>RWO is the default case:</strong> every PVC in my lab is RWO. In <a href="https://medium.com/@lubomir-tobek/from-pvc-to-vsphere-disk-tracing-a-vks-volume-through-the-supervisor-9f42e67353fa">From PVC to vSphere Disk</a> I traced one such volume from the claim in a VKS cluster to its disk on the vSAN datastore.</li><li><strong>RWX is an additional service with its own lifecycle:</strong> vSAN provides RWX volumes only when vSAN File Services is configured on the cluster. This is why File Services appears in five of the eight design patterns covered in the second article.</li><li><strong>The reach of an RWX share is a design decision:</strong> a share served inside one zone is not the same as a share that every zone can mount. The second article shows that only some design patterns provide the latter.</li></ul><blockquote><em>Part 3 — From Bare Metal to the Supervisor</em></blockquote><p>Kubernetes has its conceptual origins in installations directly on the operating system of physical servers. On bare metal, a pod that uses a node-local volume is tied to that node. If the node fails, the pod cannot simply start somewhere else with its data. Highly available persistent data therefore required a storage orchestration layer inside or next to Kubernetes, such as Ceph, OpenEBS or Longhorn, that replicated volumes between nodes. Zones represent logical or physical failure domains: racks in a data center row, rooms or buildings of a campus, or a broader geographic region.</p><p>VCF changes the layers underneath:</p><ul><li>VKS automates the deployment and lifecycle of upstream, CNCF-certified Kubernetes clusters, provisioned through standard Kubernetes APIs or declarative tools such as Cluster API.</li><li>The Supervisor embeds Kubernetes into vSphere and connects it to vSAN storage, NSX or VDS networking, and vSphere compute and security boundaries. It can map vSphere clusters or compute resources to physical availability zones, so that nodes and pods can be spread across independent failure domains.</li></ul><pre>+-------------------------------------------------------------------+<br>| Applications: pods, PVCs                                          |<br>+-------------------------------------------------------------------+<br>| VKS clusters: control plane and worker VMs (Cluster API)          |<br>+-------------------------------------------------------------------+<br>| Supervisor and vCenter: namespaces, zones, StorageClasses, CNS    |<br>+---------------------------------+---------------------------------+<br>| vSphere clusters (compute)      | vSAN storage clusters (data)    |<br>+---------------------------------+---------------------------------+<br>| Physical failure domains: rack, room, building, site              |<br>+-------------------------------------------------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>A VKS node is a VM:</strong> when its host fails, vSphere HA restarts it on another host of the same cluster, and its volumes are still reachable because they live on a shared vSAN datastore, not on a local disk. I tested this in <a href="https://medium.com/@lubomir-tobek/losing-an-esx-host-under-a-running-supervisor-vsphere-ha-vs-machinehealthcheck-3c0ae7468b03">Losing an ESX Host Under a Running Supervisor</a>, and looked at the order of the restarts in <a href="https://medium.com/@lubomir-tobek/who-restarts-first-vsphere-ha-restart-priority-for-the-supervisor-and-vks-nodes-31ea21d4209a">Who Restarts First</a>.</li><li><strong>Storage replication moves one layer down:</strong> vSAN does the job that Ceph or Longhorn did on bare metal, so no second storage system has to run inside the Kubernetes cluster.</li><li><strong>A zone keeps its meaning:</strong> it is still a physical failure domain. The vSphere administrator defines the zones in vCenter and maps them to vSphere clusters. Kubernetes consumes them as zone topology for scheduling and volume placement.</li></ul><blockquote><em>Part 4 — Zone Models: Single-Zone, Multi-Zone and Multi-Cluster Zones</em></blockquote><p>The paper describes two operational models:</p><ul><li><strong>Single-zone:</strong> one or more vSphere clusters mapped to one zone.</li><li><strong>Multi-zone:</strong> one or more vSphere clusters mapped across exactly three zones.</li></ul><pre>   Single-zone Supervisor               Multi-zone Supervisor<br><br>+--------------------------+  +------------+ +------------+ +------------+<br>| Zone A                   |  | Zone A     | | Zone B     | | Zone C     |<br>|                          |  |            | |            | |            |<br>| vSphere cluster 1        |  | cluster A1 | | cluster B1 | | cluster C1 |<br>| vSphere cluster 2 (opt.) |  | (A2 opt.)  | | (B2 opt.)  | | (C2 opt.)  |<br>+--------------------------+  +------------+ +------------+ +------------+<br>                                 \_____________ one vCenter _____________/</pre><p>A multi-zone Supervisor comes with a fixed set of design constraints:</p><pre>+---------------------+---------------------------------------------------------+<br>| Constraint          | Value                                                   |<br>+---------------------+---------------------------------------------------------+<br>| Number of zones     | exactly three; available while at most one zone fails   |<br>| vCenter             | one vCenter Server for all three zones                  |<br>| Workload networks   | the same networks in every zone                         |<br>| Management networks | may use different port groups on the same VDS           |<br>| Networking stack    | NSX VPC or VDS, chosen in the Supervisor wizard         |<br>| Storage policy      | one shared policy, every rule satisfiable in each zone  |<br>| Layer 2             | contiguous L2 or NSX overlay across all clusters        |<br>| Cross-zone HA       | application-level HA plus data replication              |<br>+---------------------+---------------------------------------------------------+</pre><p>The networking decision behind the fifth row is the same one I described in <a href="https://medium.com/@lubomir-tobek/two-diagrams-one-decision-supervisor-networking-for-vcf-private-ai-services-2586f8df68fb">Two Diagrams, One Decision</a>. It has to be made before the Supervisor is activated.</p><p><strong>New in VCF 9.1.1:</strong> a zone can contain more than one vSphere cluster, in both single-zone and multi-zone deployments. The earlier one-to-one mapping between a zone and a cluster no longer applies.</p><pre>                                  Zone A<br>+------------------------------------------------------------------------+<br>|                                                                        |<br>|        vSphere cluster A1                    vSphere cluster A2        |<br>| +------------------------------+      +------------------------------+ |<br>| | namespace ns-1 attached here |      | ns-1 can move here           | |<br>| | (cluster chosen by capacity) |      | if the datastore is mounted  | |<br>| +---------------+--------------+      +---------------+--------------+ |<br>|                 |                                     |                |<br>|                 +------------------+------------------+                |<br>|                                    |                                   |<br>|                 vSAN datastore, mounted on both clusters               |<br>+------------------------------------------------------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>One cluster per zone at a time:</strong> a namespace is attached to a single cluster in each zone. The cluster is chosen by available capacity.</li><li><strong>Data follows only a shared datastore:</strong> after a switch to another cluster, the volumes stay available only if the same datastore is mounted on that cluster. If namespaces are expected to switch clusters, mounting the datastore on every cluster of the zone should be treated as a design requirement.</li><li><strong>Datastore maintenance can mean migration:</strong> if maintenance makes a datastore unavailable, its PVs must be moved to another datastore that supports the same storage policy. Capacity for that migration has to be planned.</li><li><strong>HA stays inside the cluster:</strong> vSphere HA and vMotion work only within the cluster the namespace is attached to. A second cluster in the zone adds capacity and operational flexibility, not automatic failover. Availability beyond the cluster still comes from the application.</li><li><strong>Why it matters:</strong> inside one cluster, maintenance happens host by host, as in <a href="https://medium.com/@lubomir-tobek/putting-an-esx-host-into-maintenance-mode-under-a-running-supervisor-what-moves-what-restarts-and-167125745154">putting an ESX host into maintenance mode under a running Supervisor</a>. The paper recommends multi-cluster zones to decouple compute maintenance from the zones that consumers see and to simplify capacity expansion.</li></ul><blockquote><em>Part 5 — From StorageClass to vSAN Object: CNS, Auto-RAID and Volume Binding</em></blockquote><p><strong>The Cloud Native Storage (CNS)</strong> plugin connects Kubernetes StorageClasses to vSAN storage policies. I followed that path in <a href="https://medium.com/@lubomir-tobek/from-storage-policy-to-storageclass-how-storage-reaches-a-supervisor-namespace-c55f8c05ba11">From Storage Policy to StorageClass</a>: the only link between a StorageClass and its policy is the storagePolicyID parameter.</p><p><strong>Auto-RAID</strong> changes what the policy carries. According to the paper, resilience in vSAN for VCF 9.1 is no longer controlled by discrete storage policies. Auto-RAID applies the highest level of resilience that the characteristics of the cluster support. The StorageClass in my lab shows it in its name: vcf-lab-optimal-datastore-default-policy-autoraid.</p><p>The second setting that matters for topology is volumeBindingMode. In a multi-zone deployment with local vSAN storage in each zone, the paper requires WaitForFirstConsumer:</p><pre>+---------------------------------------+---------------------------------------+<br>| Immediate                             | WaitForFirstConsumer                  |<br>+---------------------------------------+---------------------------------------+<br>| 1. PVC is created                     | 1. PVC is created and stays Pending   |<br>| 2. PV is created at once, e.g. Zone A | 2. A pod uses the PVC                 |<br>| 3. A pod uses the PVC                 | 3. Scheduler places the pod in Zone B |<br>| 4. Pod is forced into Zone A, or      | 4. PV is created in Zone B            |<br>|    stays Pending if Zone A does not   | 5. Pod and volume share one zone      |<br>|    fit its rules                      |                                       |<br>+---------------------------------------+---------------------------------------+</pre><p>Architectural considerations:</p><ul><li><strong>Both variants already exist:</strong> in my lab, the Supervisor holds two StorageClasses for each of its 19 storage policies, 38 in total. The class without a suffix uses Immediate. The class with the suffix -latebinding uses WaitForFirstConsumer.</li><li><strong>The class is the design control, not the policy:</strong> in a multi-zone deployment with per-zone storage, workloads must reference the -latebinding class. The policy behind both classes is the same.</li><li><strong>A snapshot is not a copy in another zone:</strong> the snapshot of a VKS volume is realized on the Supervisor side, as shown in <a href="https://medium.com/@lubomir-tobek/snapshotting-a-pvc-in-a-vks-cluster-the-snapshot-lives-on-the-supervisor-672c9b6fe7de">Snapshotting a PVC in a VKS Cluster</a>, and it stays on the same datastore as the volume. Protection against the loss of a zone needs replication or backup to another failure domain.</li></ul><blockquote><em>Part 6 — Where Availability Is Designed: Application or Storage</em></blockquote><pre>+-------------------------------+-------------+-------------------------+------------+<br>| Workload                      | Replicates  | Storage approach        | Topology   |<br>|                               | itself?     |                         | family     |<br>+-------------------------------+-------------+-------------------------+------------+<br>| Cassandra, CockroachDB, Kafka | yes, quorum | local vSAN in each zone | multi-zone |<br>| PostgreSQL, MongoDB replicas  | yes         | local vSAN in each zone | multi-zone |<br>| Legacy, monolithic apps       | no          | synchronous vSAN, RPO=0 | cross-zone |<br>+-------------------------------+-------------+-------------------------+------------+</pre><p>Architectural considerations:</p><ul><li><strong>Do not replicate across zones twice:</strong> if a self-replicating application runs on a datastore stretched across sites, vSAN also mirrors each replica between the sites. With three application replicas on a stretched datastore that keeps a copy in each data site, the data exists at least six times, and the inter-site link carries both layers of replication. With per-zone local storage, only the application crosses zone boundaries, and vSAN protects the data inside each zone.</li><li><strong>Applications on shared disks need the storage layer:</strong> an application that relies on shared disks instead of its own replication, such as the cluster in <a href="https://medium.com/@lubomir-tobek/high-availability-database-design-implementing-microsoft-sql-server-fci-on-vmware-cloud-foundation-8934ab82773f">SQL Server FCI on VCF 9 and vSAN</a>, depends on synchronous replication below it. The same applies to a monolithic application in a pod with a single RWO volume.</li><li><strong>The network sets the limits:</strong> every synchronous write waits for the remote copy, so latency and bandwidth between sites directly affect application response times.</li></ul><pre>+---------------------------------+---------------------------------------+<br>| Construct                       | Inter-site requirement                |<br>+---------------------------------+---------------------------------------+<br>| vSAN stretched cluster          | RTT 5 ms or less between data sites   |<br>| vSAN fault domains across zones | RTT 1 ms or less between fault domains|<br>| Multi-zone networking           | contiguous L2 or NSX overlay          |<br>+---------------------------------+---------------------------------------+</pre><p>Latency is only half of the equation. How much bandwidth a stretched cluster needs is covered in <a href="https://medium.com/@lubomir-tobek/strategic-bandwidth-sizing-for-vsan-stretched-clusters-in-vcf-9-0-a-roadmap-to-resilience-ce55545b96a2">Strategic Bandwidth Sizing for vSAN Stretched Clusters in VCF 9.0</a>.</p><blockquote><em>Part 7 — Design Decisions to Record</em></blockquote><p>Before a topology is chosen, the building blocks above turn into a short list of decisions. Each one should be recorded with its reason, because most of them are expensive to change later.</p><pre>+----+--------------------------+------------------------------+-------------------------+<br>| #  | Decision                 | Options                      | Driven by               |<br>+----+--------------------------+------------------------------+-------------------------+<br>| D1 | Storage source           | storage cluster, remote HCI  | scaling, cost, network  |<br>| D2 | Zone model               | single-zone, three zones     | failure domains, sites  |<br>| D3 | Clusters per zone        | one, several (9.1.1)         | capacity, maintenance   |<br>| D4 | Networking stack         | NSX VPC, VDS                 | automation, tenancy     |<br>| D5 | Volume types             | RWO only, RWO and RWX        | application needs       |<br>| D6 | Replication owner        | application, vSAN            | application design      |<br>| D7 | Binding mode             | Immediate, -latebinding      | per-zone or shared data |<br>| D8 | Cross-zone vSAN layout   | fault domains, stretched     | inter-site RTT          |<br>+----+--------------------------+------------------------------+-------------------------+</pre><blockquote><em>Summary</em></blockquote><pre>+-----------------------------------------+-------------------------------------------+<br>| Question                                | Answer                                    |<br>+-----------------------------------------+-------------------------------------------+<br>| Where do VKS volumes live in the paper? | on a remote vSAN datastore                |<br>| What does RWX need?                     | vSAN File Services on the cluster         |<br>| How many zones does multi-zone use?     | exactly three, one may fail               |<br>| Can a zone have more than one cluster?  | yes, new in VCF 9.1.1                     |<br>| Clusters per zone for one namespace?    | one at a time, chosen by capacity         |<br>| Does HA cross clusters in a zone?       | no, HA and vMotion stay in the cluster    |<br>| Who sets resilience in vSAN 9.1?        | Auto-RAID, from cluster properties        |<br>| Binding mode for per-zone local vSAN?   | WaitForFirstConsumer (-latebinding class) |<br>| Self-replicating apps fit best on?      | multi-zone local vSAN                     |<br>| Legacy apps fit best on?                | cross-zone stretched vSAN, RPO=0          |<br>| Maximum RTT, stretched / fault domains? | 5 ms / 1 ms                               |<br>+-----------------------------------------+-------------------------------------------+</pre><blockquote><em>Conclusion</em></blockquote><p>A Kubernetes administrator sees a StorageClass and a zone label. Behind them, the architecture of VCF 9.1.1 decides where the volume is created, how many copies vSAN keeps and whether the data is still reachable after a zone is gone. Three choices shape the result: whether storage stays inside a zone or spans zones, whether the workload needs block or file volumes, and whether the application or the infrastructure owns replication.</p><p>With these building blocks and decisions in place, the eight design patterns in the paper become combinations of the same few parts. The second article draws each one, shows what happens when a site or zone fails, and ends with a decision path from the workload to the topology.</p><p>Architecting Kubernetes Availability on vSAN in VCF 9.1.1: Eight Topology Design Patterns<br><a href="https://medium.com/@lubomir-tobek/architecting-kubernetes-availability-on-vsan-in-vcf-9-1-1-eight-topology-design-patterns-018bebce8a36">https://medium.com/@lubomir-tobek/architecting-kubernetes-availability-on-vsan-in-vcf-9-1-1-eight-topology-design-patterns-018bebce8a36</a></p><p>Ref:</p><p>Kubernetes Topologies with vSAN<br><a href="https://www.vmware.com/docs/vmw-kubernetes-topologies-with-vsan">https://www.vmware.com/docs/vmw-kubernetes-topologies-with-vsan</a></p><p>vSphere Supervisor Zone Models<br><a href="https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/design/design-library/self-service-iaas-deployment-models/vsphere-supervisor-zone-models.html">https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/design/design-library/self-service-iaas-deployment-models/vsphere-supervisor-zone-models.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2907cd2d88da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[vExpert 2027 Applications Are Open: This Could Be Your Year]]></title>
            <link>https://medium.com/@lubomir-tobek/vexpert-2027-applications-are-open-this-could-be-your-year-c3d33d63598c?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3d33d63598c</guid>
            <category><![CDATA[vmwarecommunity]]></category>
            <category><![CDATA[vexpert]]></category>
            <category><![CDATA[vexpert2027]]></category>
            <category><![CDATA[techcommunity]]></category>
            <category><![CDATA[career-growth]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 16:42:48 GMT</pubDate>
            <atom:updated>2026-11-08T16:53:52.595Z</atom:updated>
            <content:encoded><![CDATA[<p>Let me guess. You’ve thought about applying for vExpert before.</p><p>Maybe you saw the badge on someone’s LinkedIn profile. Maybe a colleague mentioned it. And then a quiet voice in your head said: “That’s for the experts. I’m not there yet.”</p><p><strong>This article is for you.</strong></p><p>Applications for the VMware vExpert 2027 program are officially open. If you’ve been sharing your VMware knowledge in any way, there’s a very real chance you’re closer to the award than you think.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3uu7ekgvXlsnQsTuIH8G2w.png" /></figure><blockquote><strong><em>The Biggest Barrier Is Not Your Experience. It’s Hesitation.</em></strong></blockquote><p>Every year, talented people don’t apply. Not because they don’t qualify, but because they assume they don’t.</p><p>Here’s the truth: vExpert is not a certification, and it’s not a ranking of the smartest engineers in the world. There’s no exam, no lab, and no technical interview.</p><p>It’s recognition for people who give back to the community. And it recognizes you as an individual, not your employer or your job title.</p><p>If you’ve helped others learn, solve a problem, or understand VMware technology better, you’re already doing what vExperts do.</p><blockquote><strong><em>You Know Something the Documentation Doesn’t</em></strong></blockquote><p>Official documentation is excellent at describing how things should work. It covers the happy path.</p><p>But real environments don’t always follow the happy path. You’ve hit the strange error message at 2 AM. You’ve found the workaround for the upgrade that wouldn’t complete. You’ve written the script that turned a four-hour manual task into four minutes.</p><p>That knowledge lives in your head, and it’s exactly what the community needs.</p><p>Somewhere right now, an engineer is searching for the answer you’ve already found. When you share it, three things happen:</p><pre>+--------------------------+----------------------------------------------+<br>| When you share...        | What happens                                 |<br>+--------------------------+----------------------------------------------+<br>| Someone else             | Saves hours (or a weekend) on the same issue |<br>| Your work                | Gets better through feedback and questions   |<br>| Your reputation          | Grows, one useful contribution at a time     |<br>+--------------------------+----------------------------------------------+</pre><p>Many vExperts never set out to win an award. They simply gave back to the community that once helped them, and they kept doing it.</p><blockquote><strong><em>Myth vs. Reality</em></strong></blockquote><pre>+--------------------------------------+---------------------------------------+<br>| What people think                    | What is actually true                 |<br>+--------------------------------------+---------------------------------------+<br>| &quot;I need a famous blog.&quot;              | Quality and impact matter more than   |<br>|                                      | traffic or follower count.            |<br>+--------------------------------------+---------------------------------------+<br>| &quot;I need to be a VCDX or architect.&quot;  | No certification is required.         |<br>+--------------------------------------+---------------------------------------+<br>| &quot;I have to publish every week.&quot;      | Consistency matters, but it&#39;s about   |<br>|                                      | real value, not volume.               |<br>+--------------------------------------+---------------------------------------+<br>| &quot;Only bloggers get accepted.&quot;        | Speakers, VMUG leaders, script        |<br>|                                      | writers, podcasters, volunteers and   |<br>|                                      | internal champions all qualify.       |<br>+--------------------------------------+---------------------------------------+<br>| &quot;I work for a customer, not VMware.&quot; | Customer AND partner employees can    |<br>|                                      | apply.                                |<br>+--------------------------------------+---------------------------------------+<br>| &quot;My topic is too basic.&quot;             | Beginners learn best from people who  |<br>|                                      | recently solved the same problem.     |<br>+--------------------------------------+---------------------------------------+<br>| &quot;If I&#39;m rejected, that&#39;s it.&quot;        | There are two review periods a year.  |<br>|                                      | You can apply again.                  |<br>+--------------------------------------+---------------------------------------+</pre><blockquote><strong><em>Find Your Format</em></strong></blockquote><p>There’s no “right” way to contribute. The best format is the one that fits your strengths, because that’s the one you’ll keep doing.</p><pre>+-------------------------------+-----------------------------------------+<br>| If you enjoy...               | Try this                                |<br>+-------------------------------+-----------------------------------------+<br>| Writing                       | A blog post about a problem you solved  |<br>| Talking                       | Being a guest on a podcast, or a short  |<br>|                               | talk at your local VMUG                 |<br>| Coding &amp; automation           | Publishing your scripts on GitHub       |<br>| Teaching                      | Running a workshop for your team        |<br>| Helping people one-on-one     | Answering questions in community forums |<br>| Recording &amp; editing           | A short video walkthrough or demo       |<br>| Organizing                    | Helping run community meetups or events |<br>| Giving product feedback       | Joining beta programs or customer       |<br>|                               | advisory boards                         |<br>+-------------------------------+-----------------------------------------+</pre><p>Start small. One honest, well-written post about a real problem is worth more than ten generic ones. Then build from there.</p><blockquote><strong><em>The Badges: Find Yourself in the List</em></strong></blockquote><p>Every vExpert profile in the official <a href="https://vexpert.vmware.com/directory">vExpert Directory</a> displays badges that reflect the types of contributions behind the award. Browse a few profiles there and you’ll see how varied the paths really are. Here are the badges you can earn:</p><pre>+--------------------------------------------+---------------+<br>| Badge                                      | Category      |<br>+--------------------------------------------+---------------+<br>| Blogger                                    | Knowledge     |<br>| Event Speaker                              | Knowledge     |<br>| Video Publisher                            | Knowledge     |<br>| Code Warrior                               | Knowledge     |<br>| Podcaster - Host, Co-Host or Guest         | Knowledge     |<br>| Book Author                                | Knowledge     |<br>| AI Knowledge Author                        | Knowledge     |<br>+--------------------------------------------+---------------+<br>| VMUG Leadership                            | Community     |<br>| Volunteer - Community Work                 | Community     |<br>| VMware Related Forums Champion             | Community     |<br>| HOL - Hands-on Labs Volunteer              | Community     |<br>| Top Contributor                            | Community     |<br>| VMTN Moderator                             | Community     |<br>+--------------------------------------------+---------------+<br>| Internal Champion                          | Influence     |<br>| Enterprise Influencer                      | Influence     |<br>| CTAB - Customer Technical Advisory Board   | Influence     |<br>| Beta Program Participation                 | Influence     |<br>+--------------------------------------------+---------------+<br>| VCDX - Certification                       | Excellence    |<br>| 10 Year vExpert                            | Milestone     |<br>| 15 Year vExpert                            | Milestone     |<br>| vExpert Legend                             | Milestone     |<br>+--------------------------------------------+---------------+</pre><p>Take a closer look at a few of them:</p><p>Podcaster — Host, Co-Host or Guest. You don’t need your own show. Being a guest on a podcast counts.</p><p>Volunteer — Community Work and Internal Champion. These recognize people who help others without needing a public blog: inside their companies, at events, and in local communities.</p><p>CTAB and Beta Program Participation. If you’ve given product feedback as a customer, that’s a contribution too.</p><p>AI Knowledge Author. The program keeps evolving with the technology, and new ways to contribute keep getting recognized.</p><p>So that internal workshop you ran for your team? It counts.<br>That PowerCLI script you shared on GitHub? It counts.<br>That forum thread where you helped someone fix their vSAN cluster at 11 PM? It counts.</p><p>Read through the list again and count how many describe you. Most people find at least one. Many find three or four.</p><blockquote><strong><em>What’s Waiting for You on the Other Side</em></strong></blockquote><p>Becoming a vExpert gets you more than a badge. It opens doors. The award runs for one full year, from February to January, and includes:</p><pre>+---------------------------------------------+---------------------+<br>| Benefit                                     | Why it matters      |<br>+---------------------------------------------+---------------------+<br>| 1-year VMUG Advantage membership            | Training, discounts |<br>| 3-year VCF or VVF license for home lab *    | Hands-on learning   |<br>| 50% discount on exams                       | Certification path  |<br>| Private vExpert channels &amp; webinars         | Direct access       |<br>| Private betas &amp; pre-launch briefings **     | See what&#39;s next     |<br>| vExpert logo usage for one year             | Personal branding   |<br>| Listing in the vExpert directory            | Visibility          |<br>| Eligibility for sub-programs                | Specialization      |<br>+---------------------------------------------+---------------------+<br>  *  Requires passing the VCP-VCF or VCP-VVF exam<br>  ** Subject to approval</pre><p>But ask vExperts what the real benefit is, and many will tell you the same thing: the people. You join a global network of professionals who share, help, and push each other forward.</p><p>And once people join, they tend to stay. The 10 Year, 15 Year, and vExpert Legend badges exist for a reason. That kind of loyalty says more about the program than any list of benefits.</p><blockquote><strong><em>What It Means for Your Career</em></strong></blockquote><p>The benefits table is the visible part. The career impact is often bigger and longer-lasting.</p><pre>+-----------------------------+---------------------------------------------+<br>| Before                      | After                                       |<br>+-----------------------------+---------------------------------------------+<br>| Your skills are known to    | Your skills are visible to the whole        |<br>| your team                   | industry                                    |<br>+-----------------------------+---------------------------------------------+<br>| You&#39;re seen as someone who  | You&#39;re seen as someone others learn from    |<br>| implements solutions        |                                             |<br>+-----------------------------+---------------------------------------------+<br>| You follow product news     | You get a closer look at what&#39;s coming      |<br>| from the outside            | next                                        |<br>+-----------------------------+---------------------------------------------+<br>| Your resume lists what      | Your public work proves what you know       |<br>| you know                    |                                             |<br>+-----------------------------+---------------------------------------------+</pre><p>Every blog post, talk, and script becomes part of a public portfolio that keeps working for you long after you publish it. Recruiters, hiring managers, and event organizers can see real evidence of what you know and how well you explain it.</p><p>There’s also a quieter benefit: explaining something to others is the fastest way to truly understand it yourself.</p><blockquote><strong><em>Your First Application: Step by Step</em></strong></blockquote><p>Applying is simpler than you think. Here’s the whole process:</p><pre>+--------+-------------------------------+--------------------------------+<br>| Step   | What to do                    | Tip                            |<br>+--------+-------------------------------+--------------------------------+<br>|   1    | Create your account           | vexpert.vmware.com/apply       |<br>|   2    | Collect your work             | Add a link to everything       |<br>|   3    | Describe the impact           | Results, not just activities   |<br>|   4    | Write for a stranger          | Assume they don&#39;t know you     |<br>|   5    | Get feedback                  | A second pair of eyes helps    |<br>|   6    | Submit                        | Before January 22, 2027        |<br>+--------+-------------------------------+--------------------------------+</pre><p>Step 1: Create your account.<br>Go to <a href="https://vexpert.vmware.com/apply">vexpert.vmware.com/apply</a>. You’ll need an account before you can apply. Click Login if you already have one, or Create a New Account. It only takes a couple of minutes, and once it’s done, the hardest part (getting started) is behind you.</p><p>Step 2: Collect your work.<br>Open a document and list everything you did for the community in 2026: blog posts, talks, scripts, forum answers, workshops. Add a link to each one. You’ll be surprised how long the list gets.</p><p>Step 3: Describe the impact.<br>Don’t just list activities. Explain what changed because of them. <em>“Wrote a five-part VCF deployment series that’s referenced in community forums”</em> says far more than <em>“Wrote five posts.”</em> Focus on what you did beyond your normal job responsibilities, because that’s what the program recognizes.</p><p>Step 4: Write for a stranger.<br>Describe your work as if the reviewer has never heard of you. Don’t assume they know your blog or your VMUG chapter.</p><p>Step 5: Get feedback.<br>Ask a colleague or a vExpert you know to read your draft. Then check it against the official <a href="https://vexpert.vmware.com/files/2024_vExpert_Best_Practice_Guide_3.pptx">vExpert Best Practice Guide</a>.</p><p>Step 6: Submit.<br>Hit the Submit button. Seriously. Hit the button.</p><blockquote><strong><em>A Simple Habit: Keep a Contribution Journal</em></strong></blockquote><p>A common challenge for applicants is remembering everything they did. By the time the application window opens, many of the year’s contributions have slipped from memory.</p><p>The fix is simple. Start a contribution journal today, and add a line every time you share something:</p><pre>+------------+------------------------------+----------------------------+<br>| Date       | Contribution                 | Link / Impact              |<br>+------------+------------------------------+----------------------------+<br>| 2026-03-12 | Blog: VCF upgrade tips       | URL, shared in forums      |<br>| 2026-06-20 | Internal workshop on NSX     | 15 colleagues trained      |<br>| 2026-09-03 | GitHub: host audit script    | URL, first pull request    |<br>+------------+------------------------------+----------------------------+</pre><p>Five minutes a month is all it takes. Your next application practically writes itself.</p><blockquote><strong><em>Key Dates</em></strong></blockquote><pre>+------------------------------------+----------------------------------+<br>| Milestone                          | Date                             |<br>+------------------------------------+----------------------------------+<br>| vExpert 2027 applications          | OPEN NOW                         |<br>| Application deadline               | January 22, 2027                 |<br>| Voting period                      | After the deadline               |<br>| Award announcement (expected)      | February 2027                    |<br>| Appeals window                     | 5 business days after results    |<br>| Award validity                     | February 2027 - January 2028     |<br>+------------------------------------+----------------------------------+</pre><p>There’s still plenty of time to build a strong application, as long as you start now. The deadline is January 22, 2027.</p><blockquote><strong><em>What’s the Worst That Can Happen?</em></strong></blockquote><p>Let’s be honest about the risk.</p><p>If you apply and get accepted, you join one of the most respected communities in the industry.</p><p>If you apply and aren’t accepted this time, you gain valuable perspective on where to grow, and you can apply again in the second review period. You can even keep updating your application throughout the year.</p><p>If you don’t apply at all, the answer is guaranteed to be no.</p><p>The only application that never succeeds is the one that’s never sent.</p><p><em>Already a vExpert? The award doesn’t roll over automatically. Submit your 2027 application too, and while you’re at it, encourage someone new to apply. Every vExpert started with a first application.</em></p><blockquote><strong><em>Final Thoughts</em></strong></blockquote><p>Every vExpert you admire was once exactly where you are now: unsure, hesitant, and wondering whether they were “good enough.”</p><p>The difference is that they applied.</p><p>The vExpert title isn’t a finish line. It’s an invitation to keep learning, keep sharing, and keep growing alongside people who love this technology as much as you do.</p><p>If you’ve shared what you know, helped someone learn, or made the VMware community a little better this year, you’ve earned the right to tell that story.</p><p>This could be your year. Make it count.</p><p>Apply here: <a href="https://vexpert.vmware.com/apply">vexpert.vmware.com/apply</a><br>Create your account today. Your future self will thank you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3d33d63598c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Envoy Without Routes: Why the Contour VIP Refuses Connections]]></title>
            <link>https://medium.com/@lubomir-tobek/envoy-without-routes-why-the-contour-vip-refuses-connections-973a3fb274ca?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/973a3fb274ca</guid>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 14:29:42 GMT</pubDate>
            <atom:updated>2026-11-06T14:29:42.082Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article, I removed the last leftover HTTPProxy from the Supervisor, so the Contour Supervisor Service now has nothing to route. What does the Envoy VIP 10.1.31.155 answer in this state, what does Envoy actually listen on, and what do Contour’s metrics show?</p><p>In this short article, I test the VIP, look inside Envoy through its stats and admin interface, read the Contour metrics and explain one restart of a Contour pod. Everything is read-only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6a48_P9MYvuoFAsIsy-4A.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Supervisor          | VCF-LAB-Supervisor, VDS networking                         |<br>| Supervisor Service  | contour.tanzu.vmware.com, version 1.33.5+vmware.1-vks.1    |<br>| Service namespace   | svc-contour-jvu1r                                          |<br>| Envoy VIP           | 10.1.31.155 (ports 80, 443)                                |<br>| Routing objects     | none (no HTTPProxy, no Ingress)                            |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: The VIP Refuses Connections</em></blockquote><p>First, the pods and their IP addresses:</p><pre>kubectl get pods -n svc-contour-jvu1r -o wide</pre><pre>NAME                       READY   STATUS    RESTARTS      AGE   IP           NODE                    NOMINATED NODE   READINESS GATES<br>contour-844597c688-8jqkw   1/1     Running   1 (19h ago)   24h   10.1.31.46   cons-esx-05.lab.local   &lt;none&gt;           &lt;none&gt;<br>contour-844597c688-bsk2g   1/1     Running   0             24h   10.1.31.51   cons-esx-02.lab.local   &lt;none&gt;           &lt;none&gt;<br>envoy-6c69547955-t5vd6     2/2     Running   0             31h   10.1.31.36   cons-esx-02.lab.local   &lt;none&gt;           &lt;none&gt;<br>envoy-6c69547955-zn7hw     2/2     Running   0             29h   10.1.31.44   cons-esx-05.lab.local   &lt;none&gt;           &lt;none&gt;</pre><p>The vSphere Client shows the same four pods, all of them vSphere Pods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PR0xpf79rT6DeuqInJNVXQ.png" /></figure><p>The Service envoy publishes the VIP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KZ8byxZcCMg4r-h_3zsGLw.png" /></figure><p>Now HTTP and HTTPS on the VIP, from the Supervisor control plane VM:</p><pre>curl -sv --max-time 5 http://10.1.31.155/ 2&gt;&amp;1 | head -20<br>curl -skv https://10.1.31.155/ 2&gt;&amp;1 | grep -E &#39;^&lt; |^\* |SSL|reset|refused&#39; | head -20</pre><pre>*   Trying 10.1.31.155:80...<br>* connect to 10.1.31.155 port 80 from 10.1.31.42 port 53980 failed: Connection refused<br>* Failed to connect to 10.1.31.155:80 after 2 ms: Could not connect to server<br>* closing connection #0<br>*   Trying 10.1.31.155:443...<br>* connect to 10.1.31.155 port 443 from 10.1.31.42 port 50790 failed: Connection refused<br>* Failed to connect to 10.1.31.155:443 after 0 ms: Could not connect to server<br>* closing connection #0</pre><p>Both ports refuse the connection; there is no HTTP 404 and no TLS handshake. The source address 10.1.31.42 is the workload network interface of the control plane VM. To rule out the Foundation Load Balancer, I test the target port 8080 of one Envoy pod directly:</p><pre>curl -sv --max-time 5 http://10.1.31.36:8080/ 2&gt;&amp;1 | head -8</pre><pre>*   Trying 10.1.31.36:8080...<br>* connect to 10.1.31.36 port 8080 from 10.1.31.42 port 44560 failed: Connection refused<br>* Failed to connect to 10.1.31.36:8080 after 2 ms: Could not connect to server<br>* closing connection #0</pre><p>The Envoy pod itself refuses the connection, so the answer does not come from the FLB.</p><blockquote><em>Step 2: What Envoy Listens On</em></blockquote><p>From the vSphere point of view, the Envoy deployment is healthy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GDPfbvhJ7KmylQIyTn5u3Q.png" /></figure><p>Envoy is also alive from its own point of view. Its stats listener on port 8002 answers readiness and statistics:</p><pre>curl -s --max-time 5 http://10.1.31.36:8002/ready; echo<br>curl -s --max-time 5 http://10.1.31.36:8002/stats | grep -E &#39;^control_plane\.|^cluster_manager\.(active_clusters|cds\.update_success)|^listener_manager\.(total_listeners_active|lds\.update_success)&#39;<br>curl -s --max-time 5 http://10.1.31.36:8002/stats | grep -E &#39;^listener\.&#39; | grep downstream_cx_total</pre><pre>LIVE<br><br>cluster_manager.active_clusters: 2<br>cluster_manager.cds.update_success: 12<br>listener_manager.lds.update_success: 11<br>listener_manager.total_listeners_active: 2<br>listener.0.0.0.0_8002.downstream_cx_total: 28398<br>listener.0.0.0.0_8002.worker_0.downstream_cx_total: 28398<br>listener.0.0.0.0_8080.downstream_cx_total: 0<br>listener.0.0.0.0_8080.worker_0.downstream_cx_total: 0<br>listener.127.0.0.1_9001.downstream_cx_total: 0<br>listener.127.0.0.1_9001.worker_0.downstream_cx_total: 0<br>listener.admin.downstream_cx_total: 1<br>listener.admin.main_thread.downstream_cx_total: 1</pre><p>Envoy receives its configuration from Contour (11 successful listener updates over LDS, 12 successful cluster updates over CDS), but it has only 2 active listeners and 2 clusters. The stats still contain a counter for 0.0.0.0_8080 with 0 connections. A listener counter is created together with the listener, so this listener existed earlier in the life of this pod; I did not test the VIP before the cleanup, so I cannot say when it was removed.</p><p>The authoritative list comes from the Envoy admin interface. Contour exposes only the read-only endpoints of this interface, on 127.0.0.1:9001 inside the pod. I reach it with a port-forward:</p><pre>kubectl port-forward -n svc-contour-jvu1r pod/envoy-6c69547955-t5vd6 9001:9001 &gt;/dev/null 2&gt;&amp;1 &amp;<br>sleep 3<br>curl -s http://127.0.0.1:9001/listeners; echo<br>curl -s http://127.0.0.1:9001/clusters | grep -E &#39;::observability_name::|::added_via_api::&#39;<br>kill %1</pre><pre>stats-health::0.0.0.0:8002<br>envoy-admin::127.0.0.1:9001<br><br>contour::observability_name::svc-contour-jvu1r_contour_8001<br>contour::added_via_api::false<br>envoy-admin::observability_name::svc-contour-jvu1r_envoy-admin_9001<br>envoy-admin::added_via_api::false</pre><pre>+--------------+------------------------+-------------------------------------------------+<br>| Object       | Name                   | Meaning                                         |<br>+--------------+------------------------+-------------------------------------------------+<br>| Listener     | stats-health :8002     | readiness and stats, used in this step          |<br>| Listener     | envoy-admin :9001      | read-only admin interface on localhost          |<br>| Listener     | ingress_http :8080     | not present, so port 80 on the VIP is refused   |<br>| Listener     | ingress_https :8443    | not present, so port 443 on the VIP is refused  |<br>| Cluster      | contour                | xDS connection to the Service contour on 8001   |<br>| Cluster      | envoy-admin            | local admin interface behind the :9001 listener |<br>+--------------+------------------------+-------------------------------------------------+</pre><p>Both clusters have added_via_api::false: they come from the bootstrap file, not from Contour over CDS. Neither of the two listeners serves application traffic. There is no listener for 8080 or 8443 and no cluster for any application. This matches the older Contour troubleshooting documentation (v0.15): Contour does not configure Envoy to listen on a port unless there is traffic to be served. Version 1.33.5 in my lab behaves the same way.</p><blockquote><em>Step 3: What Contour Reports</em></blockquote><p>Contour exposes Prometheus metrics on port 8000. I read them from the current leader 10.1.31.51:</p><pre>curl -s --max-time 5 http://10.1.31.51:8000/metrics | grep -E &#39;^contour_(build_info|httpproxy|dagrebuild_total|dag_cache_object|eventhandler_operation_total)&#39; | head -30</pre><pre>contour_build_info{branch=&quot;&quot;,revision=&quot;c32389c0d4ededde371d2d3c95471d7a298e0e39&quot;,version=&quot;v1.33.5+vmware.1&quot;} 1<br>contour_dag_cache_object{kind=&quot;HTTPProxy&quot;} 0<br>contour_dag_cache_object{kind=&quot;Secret&quot;} 242<br>contour_dag_cache_object{kind=&quot;Service&quot;} 65<br>contour_dagrebuild_total 3<br>contour_eventhandler_operation_total{kind=&quot;EndpointSlice&quot;,op=&quot;add&quot;} 65<br>contour_eventhandler_operation_total{kind=&quot;EndpointSlice&quot;,op=&quot;update&quot;} 162<br>contour_eventhandler_operation_total{kind=&quot;HTTPProxy&quot;,op=&quot;add&quot;} 1<br>contour_eventhandler_operation_total{kind=&quot;HTTPProxy&quot;,op=&quot;delete&quot;} 1<br>contour_eventhandler_operation_total{kind=&quot;HTTPProxy&quot;,op=&quot;update&quot;} 2<br>contour_eventhandler_operation_total{kind=&quot;Secret&quot;,op=&quot;add&quot;} 246<br>contour_eventhandler_operation_total{kind=&quot;Secret&quot;,op=&quot;delete&quot;} 4<br>contour_eventhandler_operation_total{kind=&quot;Secret&quot;,op=&quot;update&quot;} 519<br>contour_eventhandler_operation_total{kind=&quot;Service&quot;,op=&quot;add&quot;} 65<br>contour_eventhandler_operation_total{kind=&quot;Service&quot;,op=&quot;update&quot;} 130</pre><pre>+------------------------------------------+-------+------------------------------------------+<br>| Metric                                   | Value | Meaning                                  |<br>+------------------------------------------+-------+------------------------------------------+<br>| contour_build_info (version)             | 1     | v1.33.5+vmware.1                         |<br>| contour_dag_cache_object{HTTPProxy}      | 0     | no HTTPProxy left after the cleanup      |<br>| contour_dag_cache_object{Secret}         | 242   | Secrets of the Supervisor in the cache   |<br>| contour_dag_cache_object{Service}        | 65    | Services of the Supervisor in the cache  |<br>| eventhandler HTTPProxy add/update/delete | 1/2/1 | the argocd object, deleted in cleanup    |<br>| contour_dagrebuild_total                 | 3     | DAG rebuilds since the pod started       |<br>+------------------------------------------+-------+------------------------------------------+</pre><p>The counters run from the start of the pod 24 hours ago. The one add and the one delete of an HTTPProxy match the argocd object and its removal in the previous article. Contour keeps 242 Secrets (246 added minus 4 deleted) and 65 Services of the Supervisor in its cache, although no route uses them now.</p><blockquote><em>Step 4: Why One Contour Pod Restarted</em></blockquote><p>Only one Contour pod is the leader. The leader is recorded in a Lease:</p><pre>kubectl get lease -n svc-contour-jvu1r</pre><pre>NAME           HOLDER                                                          AGE<br>leader-elect   contour-844597c688-bsk2g_39d35661-78d1-4128-959a-897977dad6f1   22d</pre><p>The leader is contour-844597c688-bsk2g. The other pod, contour-844597c688–8jqkw, shows 1 restart. Its last termination and the end of its previous log:</p><pre>kubectl get pod contour-844597c688-8jqkw -n svc-contour-jvu1r -o jsonpath=&#39;{.status.containerStatuses[0].lastState}{&quot;\n&quot;}&#39;<br>kubectl logs -n svc-contour-jvu1r contour-844597c688-8jqkw --previous 2&gt;&amp;1 | tail -8</pre><pre>{&quot;terminated&quot;:{&quot;containerID&quot;:&quot;7206a169-c679-4652-a66c-93292f18aef2&quot;,&quot;exitCode&quot;:1,&quot;finishedAt&quot;:&quot;2026-10-31T21:03:45Z&quot;,&quot;reason&quot;:&quot;Error&quot;,&quot;startedAt&quot;:&quot;2026-10-31T15:49:18Z&quot;}}<br>time=&quot;2026-10-31T15:49:22Z&quot; level=info msg=&quot;received a new address for status.loadBalancer&quot; context=loadBalancerStatusWriter loadbalancer-address=10.1.31.155<br>time=&quot;2026-10-31T15:49:22Z&quot; level=info msg=&quot;the initial dag is built&quot; context=xds<br>time=&quot;2026-10-31T15:49:22Z&quot; level=info msg=&quot;started xDS server&quot; address=&quot;0.0.0.0:8001&quot; context=xds<br>time=&quot;2026-10-31T15:49:22Z&quot; level=info msg=&quot;performing delayed update&quot; context=contourEventHandler last_update=142.767476ms outstanding=51<br>time=&quot;2026-10-31T21:03:39Z&quot; level=error msg=&quot;Failed to update lock optimistically: Put \&quot;https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/svc-contour-jvu1r/leases/leader-elect?timeout=5s\&quot;: net/http: request canceled (Client.Timeout exceeded while awaiting headers), falling back to slow path&quot; caller=&quot;leaderelection.go:441&quot; context=kubernetes error=&quot;&lt;nil&gt;&quot;<br>time=&quot;2026-10-31T21:03:44Z&quot; level=error msg=&quot;error retrieving resource lock svc-contour-jvu1r/leader-elect: Get \&quot;https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/svc-contour-jvu1r/leases/leader-elect?timeout=5s\&quot;: context deadline exceeded&quot; caller=&quot;leaderelection.go:448&quot; context=kubernetes error=&quot;&lt;nil&gt;&quot;<br>time=&quot;2026-10-31T21:03:44Z&quot; level=info msg=&quot;failed to renew lease svc-contour-jvu1r/leader-elect: context deadline exceeded&quot; caller=&quot;leaderelection.go:297&quot; context=kubernetes<br>time=&quot;2026-10-31T21:03:45Z&quot; level=fatal msg=&quot;Contour server failed&quot; error=&quot;leader election lost&quot;</pre><p>The sequence on 31 October 2026:</p><pre>+-----------+-------------------------------------------------------------------------+<br>| Time UTC  | Event                                                                   |<br>+-----------+-------------------------------------------------------------------------+<br>| 15:49:22  | the container (started 15:49:18) builds the DAG, starts xDS on 8001     |<br>| 21:03:39  | renewing the Lease via the Service kubernetes (10.96.0.1) times out     |<br>| 21:03:44  | reading the Lease times out as well, the renewal fails                  |<br>| 21:03:45  | &quot;leader election lost&quot;, Contour exits with exit code 1                  |<br>+-----------+-------------------------------------------------------------------------+</pre><p>The messages failed to renew lease and leader election lost show that contour-844597c688–8jqkw was the leader at that moment; only the leader renews the Lease. It could not renew the Lease because its requests to the Kubernetes API through the Service 10.96.0.1 timed out, so it exited, and Kubernetes restarted the container. Today, contour-844597c688-bsk2g holds the Lease, and the restarted pod runs as a follower. Exiting after a lost leader election is expected Contour behavior; the same pattern is described in Contour issue #6374 about API server outages. The logs do not show why the API requests timed out, so the root cause on the Supervisor side remains open.</p><blockquote><em>Summary</em></blockquote><pre>+------------------------------+-----------------------------------------------------------+<br>| Check                        | Result in my lab                                          |<br>+------------------------------+-----------------------------------------------------------+<br>| VIP 10.1.31.155:80 and :443  | Connection refused                                        |<br>| Envoy pod 10.1.31.36:8080    | Connection refused, the answer does not come from the FLB |<br>| Envoy listeners              | stats-health :8002 and envoy-admin :9001 only             |<br>| Envoy clusters               | contour and envoy-admin, both from the bootstrap file     |<br>| Contour HTTPProxy cache      | 0                                                         |<br>| Contour leader               | contour-844597c688-bsk2g                                  |<br>| Contour restart              | leader election lost, API timeouts, 2026-10-31 21:03 UTC  |<br>+------------------------------+-----------------------------------------------------------+</pre><ul><li>Without any HTTPProxy or Ingress, Contour does not configure the ingress_http and ingress_https listeners, so Envoy and the VIP 10.1.31.155 refuse connections on ports 80 and 443.</li><li>Envoy is still LIVE and connected to Contour: it has received LDS and CDS updates, but its only listeners are stats-health and envoy-admin, and its only clusters are the two from the bootstrap file.</li><li>The read-only part of the Envoy admin interface on 127.0.0.1:9001 is reachable with kubectl port-forward; /listeners is the authoritative list of what Envoy listens on.</li><li>Contour’s metrics on port 8000 confirm 0 HTTPProxy objects and record the add and delete of the argocd object.</li><li>The restart of contour-844597c688–8jqkw was a leader election lost exit after timeouts to the Kubernetes API; the Lease moved to contour-844597c688-bsk2g.</li></ul><p>Ref:</p><p>VMware by Broadcom Documentation (vSphere 8.0), Install Contour as a Supervisor Service: <a href="https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-services-workloads/GUID-D0DEC184-00E6-4592-A044-A504840A48F5.html">https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-services-workloads/GUID-D0DEC184-00E6-4592-A044-A504840A48F5.html</a></p><p>vSphere Supervisor Services repository (vsphere-tmm): <a href="https://github.com/vsphere-tmm/Supervisor-Services">https://github.com/vsphere-tmm/Supervisor-Services</a></p><p>Contour Documentation (v1.25), Envoy Administration Access: <a href="https://projectcontour.io/docs/1.25/troubleshooting/envoy-admin-interface/">https://projectcontour.io/docs/1.25/troubleshooting/envoy-admin-interface/</a></p><p>Contour troubleshooting, v0.15 (listeners are configured only when there is traffic to be served): <a href="https://github.com/heptio/contour/blob/v0.15.3/docs/troubleshooting.md">https://github.com/heptio/contour/blob/v0.15.3/docs/troubleshooting.md</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=973a3fb274ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Contour and Envoy as a Supervisor Service: Who Configures What?]]></title>
            <link>https://medium.com/@lubomir-tobek/contour-and-envoy-as-a-supervisor-service-who-configures-what-43626781b6f3?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/43626781b6f3</guid>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 14:29:04 GMT</pubDate>
            <atom:updated>2026-11-06T14:29:04.646Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article, the VIP 10.1.31.155 led to two Envoy pods of the Contour Supervisor Service, while the two Contour pods were not backends at all. So what does Contour actually do, how does Envoy get its configuration, and what is Contour serving in my lab?</p><p>In this short article, I take apart the Contour Supervisor Service in my VCF 9.1.1 lab: its version, its two deployments, the connection between Contour and Envoy, and the routing objects it processes. Everything is read-only; no Secret content is printed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBd80f0CUbmLC5SHJeP-Eg.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Supervisor          | VCF-LAB-Supervisor, VDS networking                         |<br>| Supervisor Service  | contour.tanzu.vmware.com, version 1.33.5+vmware.1-vks.1    |<br>| Service namespace   | svc-contour-jvu1r                                          |<br>| Envoy VIP           | 10.1.31.155 (ports 80, 443)                                |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: Contour in the vSphere Client</em></blockquote><p>Contour is installed as a Supervisor Service. The Manage Versions dialog of the service shows its ID and version:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HP3S7DK0xmWmei-f5OjxGQ.png" /></figure><p>The service runs in its own vSphere Namespace svc-contour-jvu1r, where the vSphere Client lists its two deployments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LjBEsWQfybYV7vNOvuiONA.png" /></figure><p>The label kapp.k14s.io/app shows that the objects of the service are deployed with kapp, the deployment tool of kapp-controller.</p><blockquote><em>Step 2: Two Deployments, Two Roles</em></blockquote><p>The deployments, Services and ConfigMaps of the namespace:</p><pre>kubectl get deploy,svc,cm -n svc-contour-jvu1r -o wide</pre><pre>NAME                      READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS               IMAGES                                                                                                                                                                                                                                                                SELECTOR<br>deployment.apps/contour   2/2     2            2           21d   contour                  vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:aafa5ae1eb42d2320f7a363121642c11d083991513496a4a1fa4d49b626c262c                                                                                                                                     app=contour,kapp.k14s.io/app=1791649808382489684<br>deployment.apps/envoy     2/2     2            2           21d   shutdown-manager,envoy   vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:aafa5ae1eb42d2320f7a363121642c11d083991513496a4a1fa4d49b626c262c,vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:0f808ec7ae8c8b64dd3e7641ef1a7fe277a572ef167d2de51e12dad885e208c3   app=envoy,kapp.k14s.io/app=1791649808382489684<br><br>NAME              TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE   SELECTOR<br>service/contour   ClusterIP      10.96.0.195   &lt;none&gt;        8001/TCP                     21d   app=contour,kapp.k14s.io/app=1791649808382489684<br>service/envoy     LoadBalancer   10.96.1.69    10.1.31.155   80:30211/TCP,443:31513/TCP   21d   app=envoy,kapp.k14s.io/app=1791649808382489684<br><br>NAME                         DATA   AGE<br>configmap/contour            1      21d<br>configmap/contour-ver-1      1      21d<br>configmap/kube-root-ca.crt   1      21d</pre><p>The images come from my private Harbor registry (vcf-harbor-registry.lab.local/supervisor-services), not from the internet (see KB 372484 for Supervisor Services with a private registry). The containers and their ports:</p><pre>kubectl get deploy contour envoy -n svc-contour-jvu1r -o jsonpath=&#39;{range .items[*]}{.metadata.name}{&quot;\n&quot;}{range .spec.template.spec.containers[*]}{&quot;  container=&quot;}{.name}{&quot; image=&quot;}{.image}{&quot; ports=&quot;}{.ports[*].containerPort}{&quot;\n&quot;}{end}{end}&#39;</pre><pre>contour<br>  container=contour image=vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:aafa5ae1eb42d2320f7a363121642c11d083991513496a4a1fa4d49b626c262c ports=8001 8000 6060<br>envoy<br>  container=shutdown-manager image=vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:aafa5ae1eb42d2320f7a363121642c11d083991513496a4a1fa4d49b626c262c ports=<br>  container=envoy image=vcf-harbor-registry.lab.local/supervisor-services/contour@sha256:0f808ec7ae8c8b64dd3e7641ef1a7fe277a572ef167d2de51e12dad885e208c3 ports=8080 8443 8002</pre><pre>+------------------+------------------------+-----------------------------------------------+<br>| Container        | Ports                  | Role                                          |<br>+------------------+------------------------+-----------------------------------------------+<br>| contour          | 8001, 8000, 6060       | control plane: watches Kubernetes objects and |<br>|                  |                        | serves the configuration to Envoy (xDS 8001)  |<br>| envoy            | 8080, 8443, 8002       | data path: HTTP 8080 and HTTPS 8443 behind    |<br>|                  |                        | the VIP ports 80 and 443                      |<br>| shutdown-manager | -                      | sidecar of Envoy for graceful shutdown        |<br>+------------------+------------------------+-----------------------------------------------+</pre><p>The shutdown-manager container uses the same image as contour (same digest), while envoy has its own image (a different digest in the same repository). Only Envoy is behind the LoadBalancer Service and the VIP; the EndpointSlice of the Service envoy in my previous article showed the target ports 8080 and 8443. Contour is reachable only inside the Supervisor through the ClusterIP Service contour on port 8001.</p><blockquote><em>Step 3: How Contour Configures Envoy</em></blockquote><p>The arguments of both deployments show the connection:</p><pre>kubectl get deploy contour -n svc-contour-jvu1r -o jsonpath=&#39;{.spec.template.spec.containers[0].args}{&quot;\n&quot;}&#39;<br>kubectl get deploy envoy -n svc-contour-jvu1r -o jsonpath=&#39;{range .spec.template.spec.containers[*]}{.name}{&quot; args=&quot;}{.args}{&quot;\n&quot;}{end}&#39;</pre><pre>[&quot;serve&quot;,&quot;--incluster&quot;,&quot;--xds-address=0.0.0.0&quot;,&quot;--xds-port=8001&quot;,&quot;--stats-address=0.0.0.0&quot;,&quot;--http-address=0.0.0.0&quot;,&quot;--envoy-service-http-address=0.0.0.0&quot;,&quot;--envoy-service-https-address=0.0.0.0&quot;,&quot;--health-address=0.0.0.0&quot;,&quot;--contour-cafile=/certs/ca.crt&quot;,&quot;--contour-cert-file=/certs/tls.crt&quot;,&quot;--contour-key-file=/certs/tls.key&quot;,&quot;--config-path=/config/contour.yaml&quot;,&quot;--disable-feature=tlsroutes&quot;,&quot;--disable-feature=tcproutes&quot;,&quot;--disable-feature=backendtlspolicies&quot;,&quot;--disable-feature=grpcroutes&quot;]<br>shutdown-manager args=[&quot;envoy&quot;,&quot;shutdown-manager&quot;]<br>envoy args=[&quot;-c&quot;,&quot;/config/envoy.json&quot;,&quot;--service-cluster $(CONTOUR_NAMESPACE)&quot;,&quot;--service-node $(ENVOY_POD_NAME)&quot;,&quot;--log-level info&quot;]</pre><ul><li>Contour runs serve and offers the xDS API on port 8001, secured with a certificate ( — contour-cafile, — contour-cert-file, — contour-key-file).</li><li>Envoy starts with a small bootstrap file /config/envoy.json; the routes and clusters themselves come from Contour over xDS. — service-node $(ENVOY_POD_NAME) identifies each Envoy pod to Contour.</li><li>Four Gateway API features are disabled: tlsroutes, tcproutes, backendtlspolicies and grpcroutes.</li></ul><p>The certificates for this connection are stored as Secrets. Only names and types are listed here:</p><pre>kubectl get secret -n svc-contour-jvu1r -o custom-columns=NAME:.metadata.name,TYPE:.type</pre><pre>NAME                  TYPE<br>contour-ca-key-pair   kubernetes.io/tls<br>contour-reg-creds     kubernetes.io/dockerconfigjson<br>contourcert           kubernetes.io/tls<br>envoycert             kubernetes.io/tls</pre><p>contourcert and envoycert are the TLS certificates of the two sides of the xDS connection; contour-ca-key-pair holds the CA for them. contour-reg-creds is the pull secret for the registry.</p><p>The Contour configuration file itself is empty:</p><pre>kubectl get cm contour -n svc-contour-jvu1r -o jsonpath=&#39;{.data.contour\.yaml}{&quot;\n&quot;}&#39;</pre><pre>{}</pre><p>contour.yaml is {}, so Contour runs with its default settings. This matches the default value configFileContents: {} of the Contour package in the VMware Tanzu Packages documentation.</p><blockquote><em>Step 4: What Contour Serves</em></blockquote><p>Contour turns HTTPProxy and Ingress objects, and Gateway API routes when Gateway API is configured, into Envoy configuration. The objects on the Supervisor:</p><pre>kubectl get httpproxy,ingress -A<br>kubectl get ingressclass,gatewayclass</pre><pre>NAMESPACE          NAME                                 FQDN               TLS SECRET   STATUS    STATUS DESCRIPTION<br>svc-cci-ns-9mppe   httpproxy.projectcontour.io/argocd   argocd.lab.local                invalid   At least one error present, see Errors for details<br>NAME                                              AGE<br>gatewayclass.networking.x-k8s.io/flb-supervisor   70d</pre><p>There is no Ingress, no IngressClass and only one HTTPProxy, and it is invalid. Its details:</p><pre>kubectl get httpproxy argocd -n svc-cci-ns-9mppe -o jsonpath=&#39;{.metadata.creationTimestamp}{&quot;\n&quot;}{.spec}{&quot;\n&quot;}{.status.conditions}{&quot;\n&quot;}&#39;<br>kubectl get svc -A | grep -i argocd</pre><pre>2026-10-10T16:35:35Z<br>{&quot;routes&quot;:[{&quot;services&quot;:[{&quot;name&quot;:&quot;argocd-server&quot;,&quot;port&quot;:443,&quot;protocol&quot;:&quot;tls&quot;}]}],&quot;virtualhost&quot;:{&quot;fqdn&quot;:&quot;argocd.lab.local&quot;}}<br>[{&quot;errors&quot;:[{&quot;message&quot;:&quot;Spec.Routes unresolved service reference: service \&quot;svc-cci-ns-9mppe/argocd-server\&quot; not found&quot;,&quot;reason&quot;:&quot;ServiceUnresolvedReference&quot;,&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ServiceError&quot;}],&quot;lastTransitionTime&quot;:&quot;2026-10-24T12:51:53Z&quot;,&quot;message&quot;:&quot;At least one error present, see Errors for details&quot;,&quot;observedGeneration&quot;:1,&quot;reason&quot;:&quot;ErrorPresent&quot;,&quot;status&quot;:&quot;False&quot;,&quot;type&quot;:&quot;Valid&quot;}]</pre><p>The HTTPProxy argocd was created on 10 October 2026 for argocd.lab.local and points to the Service argocd-server. No Service with argocd in its name exists in any namespace (the grep returns nothing), so the Valid condition is False (reason ErrorPresent) with the error ServiceUnresolvedReference; the condition last changed on 24 October 2026. The object is most likely a leftover of an earlier Argo CD installation. Contour does not configure a route for an invalid HTTPProxy, so it does no harm, but it can be removed with kubectl delete httpproxy argocd -n svc-cci-ns-9mppe.</p><p>The only GatewayClass does not belong to Contour:</p><pre>kubectl get gatewayclass flb-supervisor -o jsonpath=&#39;{.spec}{&quot;\n&quot;}&#39;</pre><pre>{&quot;allowedGatewayNamespaceSelector&quot;:{},&quot;allowedRouteNamespaces&quot;:{&quot;namespaceSelector&quot;:{},&quot;onlySameNamespace&quot;:false},&quot;controller&quot;:&quot;lbapi.run.tanzu.vmware.com/foundation-lb&quot;}</pre><p>Its controller is lbapi.run.tanzu.vmware.com/foundation-lb, and its API group is networking.x-k8s.io, the early pre-release API of Gateway API. By its controller name, it is the interface of the Supervisor to the Foundation Load Balancer, not a Gateway API implementation of Contour.</p><blockquote><em>Summary</em></blockquote><pre>+--------------------------+-----------------------------------+--------------------------------+<br>| Component                | Role                              | Location                       |<br>+--------------------------+-----------------------------------+--------------------------------+<br>| contour (2 pods)         | control plane, xDS server         | ClusterIP 10.96.0.195:8001     |<br>| envoy (2 pods)           | data path for HTTP and HTTPS      | VIP 10.1.31.155:80, 443        |<br>| shutdown-manager         | graceful shutdown of Envoy        | sidecar in each Envoy pod      |<br>| contourcert / envoycert  | TLS of the xDS connection         | Secrets in svc-contour-jvu1r   |<br>+--------------------------+-----------------------------------+--------------------------------+</pre><ul><li>The Supervisor Service contour.tanzu.vmware.com 1.33.5+vmware.1-vks.1 runs two Contour pods and two Envoy pods as vSphere Pods with their own IP addresses from the workload network; the images come from the private Harbor registry.</li><li>Contour is the control plane: it watches HTTPProxy and Ingress objects (and Gateway API routes when configured) and serves the result to Envoy over xDS on port 8001, secured with TLS.</li><li>Envoy is the data path: only Envoy is behind the VIP 10.1.31.155, with ports 80 and 443 mapped to 8080 and 8443.</li><li>contour.yaml is {}: Contour runs with defaults, with four Gateway API features disabled.</li><li>The only HTTPProxy in my lab is invalid, most likely a leftover of Argo CD, and the GatewayClass flb-supervisor belongs to the FLB, not to Contour.</li></ul><p>Ref:</p><p>VMware by Broadcom Documentation (vSphere 8.0), Install Contour as a Supervisor Service: <a href="https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-services-workloads/GUID-D0DEC184-00E6-4592-A044-A504840A48F5.html">https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-with-tanzu-services-workloads/GUID-D0DEC184-00E6-4592-A044-A504840A48F5.html</a></p><p>VMware by Broadcom Documentation, Contour package (default values, including configFileContents): <a href="https://docs.vmware.com/en/VMware-Tanzu-Packages/latest/tanzu-packages/packages-contour.html">https://docs.vmware.com/en/VMware-Tanzu-Packages/latest/tanzu-packages/packages-contour.html</a></p><p>Broadcom KB 372484, Supervisor Services using a private registry: <a href="https://knowledge.broadcom.com/external/article/372484/supervisor-services-using-a-private-regi.html">https://knowledge.broadcom.com/external/article/372484/supervisor-services-using-a-private-regi.html</a></p><p>vSphere Supervisor Services repository (vsphere-tmm): <a href="https://github.com/vsphere-tmm/Supervisor-Services">https://github.com/vsphere-tmm/Supervisor-Services</a></p><p>Contour Documentation, Architecture: <a href="https://projectcontour.io/docs/main/architecture/">https://projectcontour.io/docs/main/architecture/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=43626781b6f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Uses the VIPs of the Foundation Load Balancer? Counting the Supervisor VIP Pool]]></title>
            <link>https://medium.com/@lubomir-tobek/who-uses-the-vips-of-the-foundation-load-balancer-counting-the-supervisor-vip-pool-d1310950ef49?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1310950ef49</guid>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 06:22:07 GMT</pubDate>
            <atom:updated>2026-11-06T06:22:07.360Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous article, I followed three VIPs from the Foundation Load Balancer (FLB) to their backends. The VIP pool of the Supervisor, however, reports six allocated addresses. Which services use the other three, how do they differ from the VIPs of VKS clusters, and how many VIPs are left?</p><p>In this short article, I map every allocated address of the VIP pool in my VCF 9.1.1 lab to its Service, its owner and its backends. Everything is read-only; nothing is changed in the lab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqWRakjLSmZUyQ8udA-rlQ.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Supervisor          | VCF-LAB-Supervisor, VDS networking                         |<br>| Load balancer       | Foundation Load Balancer flb-supervisor                    |<br>| VIP pool            | lbservicefrontend-10-1-31-150 (10.1.31.150 - .200)         |<br>| Supervisor Service  | Contour (namespace svc-contour-jvu1r)                      |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: The VIP Pool</em></blockquote><p>The vSphere Client shows the VIP range of the load balancer in the network settings of the Supervisor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zf2x5tSO52V2PPIl48C-_g.png" /></figure><p>The VIPs and the workload interfaces share the same port group and subnet 10.1.31.0/24, which is consistent with the One Arm topology of the FLB: the VIP range .150 — .200 and the workload range .10 — .100 are two separate pools in one network.</p><p>On the Supervisor, the range is the IP pool lbservicefrontend-10–1–31–150:</p><pre>kubectl get ippools.netoperator.vmware.com lbservicefrontend-10-1-31-150 -o jsonpath=&#39;{.metadata.name}{&quot;\n  spec=&quot;}{.spec}{&quot;\n  status=&quot;}{.status}{&quot;\n&quot;}&#39;</pre><pre>lbservicefrontend-10-1-31-150<br>  spec={&quot;addressCount&quot;:51,&quot;startingAddress&quot;:&quot;10.1.31.150&quot;}<br>  status={&quot;allocated&quot;:6,&quot;conditions&quot;:[{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ready&quot;}]}</pre><p>The pool has 51 addresses, 10.1.31.150 to 10.1.31.200, and 6 of them are allocated.</p><blockquote><em>Step 2: All Services of Type LoadBalancer</em></blockquote><p>Every VIP belongs to a Kubernetes Service of type LoadBalancer on the Supervisor:</p><pre>kubectl get svc -A -o wide | grep -E &#39;NAMESPACE|LoadBalancer&#39;</pre><pre>NAMESPACE                                   NAME                                                             TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)                        AGE     SELECTOR<br>kube-system                                 kube-apiserver-lb-svc                                            LoadBalancer   10.96.0.93    10.1.31.151   443:31789/TCP,6443:31611/TCP   70d     component=kube-apiserver<br>kube-system                                 mgmt-image-proxy                                                 LoadBalancer   10.96.0.50    10.1.31.150   443:31402/TCP                  70d     component=kubectl-plugin-vsphere<br>svc-cci-ns-9mppe                            kubernetes-cluster-c2ny                                          LoadBalancer   10.96.0.221   10.1.31.153   6443/TCP                       67d     &lt;none&gt;<br>svc-cci-ns-9mppe                            kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519                    LoadBalancer   10.96.0.206   10.1.31.157   9000/TCP,9001/TCP              7d18h   &lt;none&gt;<br>svc-cci-ns-9mppe                            kubernetes-cluster-s4cy                                          LoadBalancer   10.96.1.201   10.1.31.152   6443/TCP                       67d     &lt;none&gt;<br>svc-contour-jvu1r                           envoy                                                            LoadBalancer   10.96.1.69    10.1.31.155   80:30211/TCP,443:31513/TCP     21d     app=envoy,kapp.k14s.io/app=1791649808382489684</pre><p>There are exactly six Services, one for each allocated address. Three of them have no selector (&lt;none&gt;). The owner references show where they come from:</p><pre>kubectl get virtualmachineservice -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,VIP:&#39;.status.loadBalancer.ingress[*].ip&#39;<br>kubectl get svc -A -o jsonpath=&#39;{range .items[?(@.spec.type==&quot;LoadBalancer&quot;)]}{.metadata.namespace}/{.metadata.name}{&quot;  ip=&quot;}{.status.loadBalancer.ingress[*].ip}{&quot;  owner=&quot;}{.metadata.ownerReferences[*].kind}/{.metadata.ownerReferences[*].name}{&quot;\n&quot;}{end}&#39;</pre><pre>NAMESPACE          NAME                                            TYPE           VIP<br>svc-cci-ns-9mppe   kubernetes-cluster-c2ny                         LoadBalancer   10.1.31.153<br>svc-cci-ns-9mppe   kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519   LoadBalancer   10.1.31.157<br>svc-cci-ns-9mppe   kubernetes-cluster-s4cy                         LoadBalancer   10.1.31.152<br>kube-system/kube-apiserver-lb-svc  ip=10.1.31.151  owner=/<br>kube-system/mgmt-image-proxy  ip=10.1.31.150  owner=/<br>svc-cci-ns-9mppe/kubernetes-cluster-c2ny  ip=10.1.31.153  owner=VirtualMachineService/kubernetes-cluster-c2ny<br>svc-cci-ns-9mppe/kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519  ip=10.1.31.157  owner=VirtualMachineService/kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519<br>svc-cci-ns-9mppe/kubernetes-cluster-s4cy  ip=10.1.31.152  owner=VirtualMachineService/kubernetes-cluster-s4cy<br>svc-contour-jvu1r/envoy  ip=10.1.31.155  owner=/</pre><p><em>(owner=/ means that the Service has no owner reference.)</em></p><p>The six VIPs fall into two groups:</p><ul><li>Services with a pod selector and no owner: kube-apiserver-lb-svc, mgmt-image-proxy and envoy. They are regular Kubernetes Services of the Supervisor and of the Contour Supervisor Service.</li><li>Services without a selector, owned by a VirtualMachineService: the two VKS cluster APIs and the MinIO Service of c2ny. Their backends are written by the VM Service controller, as shown in my previous article.</li></ul><blockquote><em>Step 3: The Backends of the Supervisor and Contour VIPs</em></blockquote><p>The EndpointSlices of the three Services with a pod selector, and the pods of Contour:</p><pre>kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-apiserver-lb-svc -o wide<br>kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=mgmt-image-proxy -o wide<br>kubectl get endpointslices -n svc-contour-jvu1r -l kubernetes.io/service-name=envoy -o wide<br>kubectl get pods -n svc-contour-jvu1r -o wide</pre><pre>NAME                          ADDRESSTYPE   PORTS      ENDPOINTS                          AGE<br>kube-apiserver-lb-svc-jvcxh   IPv4          6443,443   10.1.31.41,10.1.31.42,10.1.31.43   70d<br>NAME                     ADDRESSTYPE   PORTS   ENDPOINTS                          AGE<br>mgmt-image-proxy-zrnb7   IPv4          5000    10.1.31.41,10.1.31.42,10.1.31.43   70d<br>NAME          ADDRESSTYPE   PORTS       ENDPOINTS               AGE<br>envoy-t77s8   IPv4          8080,8443   10.1.31.36,10.1.31.44   21d<br>NAME                       READY   STATUS    RESTARTS      AGE   IP           NODE                    NOMINATED NODE   READINESS GATES<br>contour-844597c688-8jqkw   1/1     Running   1 (18h ago)   23h   10.1.31.46   cons-esx-05.lab.local   &lt;none&gt;           &lt;none&gt;<br>contour-844597c688-bsk2g   1/1     Running   0             23h   10.1.31.51   cons-esx-02.lab.local   &lt;none&gt;           &lt;none&gt;<br>envoy-6c69547955-t5vd6     2/2     Running   0             30h   10.1.31.36   cons-esx-02.lab.local   &lt;none&gt;           &lt;none&gt;<br>envoy-6c69547955-zn7hw     2/2     Running   0             28h   10.1.31.44   cons-esx-05.lab.local   &lt;none&gt;           &lt;none&gt;</pre><ul><li>kube-apiserver-lb-svc (VIP .151) and mgmt-image-proxy (VIP .150) both point to 10.1.31.41, .42 and .43. These are the workload interfaces of the three Supervisor control plane VMs (shown in my article about the Supervisor workload network IP pool). mgmt-image-proxy publishes port 443 on the VIP and forwards to port 5000.</li><li>envoy (VIP .155) points to the two Envoy vSphere Pods 10.1.31.36 and .44 on ports 8080 and 8443. The two Contour pods .46 and .51 are not backends; they are the control plane of Contour, not the data path.</li></ul><p>The Supervisor API VIP is also shown in the vSphere Client:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNYgaf4T_r8baZIgjOSYHw.png" /></figure><p>The three SupervisorControlPlaneVM nodes in this list correspond to the three endpoints .41, .42 and .43 of the Supervisor API VIP. When a VIP does not respond, its endpoints are the first thing to check; Broadcom KB 388260 uses the same check for the VIPs of VKS clusters.</p><blockquote><em>Step 4: The Complete Map of the VIP Pool</em></blockquote><pre>+-------------+---------------------------------------+----------------+-------------------------+<br>| VIP         | Service                               | Created by     | Backends                |<br>+-------------+---------------------------------------+----------------+-------------------------+<br>| 10.1.31.150 | kube-system/mgmt-image-proxy          | Supervisor     | CP VMs .41-.43:5000     |<br>| 10.1.31.151 | kube-system/kube-apiserver-lb-svc     | Supervisor     | CP VMs .41-.43:443,6443 |<br>| 10.1.31.152 | svc-cci-ns-9mppe/kubernetes-cluster-  | VKS (VM        | s4cy control plane      |<br>|             | s4cy                                  | Service)       | .60:6443                |<br>| 10.1.31.153 | svc-cci-ns-9mppe/kubernetes-cluster-  | VKS (VM        | c2ny control plane      |<br>|             | c2ny                                  | Service)       | .33, .34, .63:6443      |<br>| 10.1.31.155 | svc-contour-jvu1r/envoy               | Contour        | Envoy pods .36, .44     |<br>|             |                                       | (Supervisor    | :8080, 8443             |<br>|             |                                       | Service)       |                         |<br>| 10.1.31.157 | svc-cci-ns-9mppe/kubernetes-cluster-  | Service minio  | c2ny worker .35         |<br>|             | c2ny-a2fd8d2c67a20c622d519            | in c2ny        | :31797, 30493           |<br>+-------------+---------------------------------------+----------------+-------------------------+</pre><pre>+--------------------------------------------+-----------+<br>| VIP pool lbservicefrontend-10-1-31-150     | Addresses |<br>+--------------------------------------------+-----------+<br>| Pool size (10.1.31.150 - 10.1.31.200)      | 51        |<br>| Allocated                                  | 6         |<br>| Free                                       | 45        |<br>+--------------------------------------------+-----------+</pre><p>The addresses 10.1.31.154 and 10.1.31.156 are free, although they lie between allocated addresses. Allocated VIPs are not necessarily contiguous, so a free address in the middle of the range is available to the next Service.</p><p>Every new VKS cluster takes one VIP for its API, and every Service of type LoadBalancer in a VKS cluster or a Supervisor Service takes one more. With 45 free addresses, my lab has plenty of room; in production, the VIP pool must be sized for the expected number of clusters and LoadBalancer Services, separately from the workload network pool for the node IPs.</p><blockquote><em>Summary</em></blockquote><ul><li>The VIP pool lbservicefrontend-10–1–31–150 has 51 addresses; 6 are allocated and 45 are free.</li><li>The Supervisor itself uses two VIPs: .150 (mgmt-image-proxy) and .151 (kube-apiserver-lb-svc, the Supervisor API). Both point to the three Supervisor control plane VMs.</li><li>The Supervisor Service Contour uses .155 for Envoy; its backends are the Envoy vSphere Pods.</li><li>VKS uses three VIPs: one per cluster API (.152, .153) and one for the LoadBalancer Service minio in c2ny (.157). These Services have no selector and are owned by a VirtualMachineService.</li><li>Each VKS cluster and each LoadBalancer Service needs its own VIP; size the VIP pool accordingly.</li></ul><p>Ref:</p><p>Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues: <a href="https://knowledge.broadcom.com/external/article/388260/troubleshooting-vsphere-supervisor-workl.html">https://knowledge.broadcom.com/external/article/388260/troubleshooting-vsphere-supervisor-workl.html</a></p><p>VM Operator Documentation, VirtualMachineService: <a href="https://vm-operator.readthedocs.io/en/latest/concepts/services-networking/vm-service/">https://vm-operator.readthedocs.io/en/latest/concepts/services-networking/vm-service/</a></p><p>Kubernetes Documentation, Service: type LoadBalancer: <a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1310950ef49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Does a Foundation Load Balancer VIP Find Its Backends in VKS?]]></title>
            <link>https://medium.com/@lubomir-tobek/how-does-a-foundation-load-balancer-vip-find-its-backends-in-vks-e45dc3faf99d?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/e45dc3faf99d</guid>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Fri, 06 Nov 2026 06:20:53 GMT</pubDate>
            <atom:updated>2026-11-06T06:20:53.867Z</atom:updated>
            <content:encoded><![CDATA[<p>A VKS cluster API is reachable through a VIP on the Foundation Load Balancer (FLB), and so is every Service of type LoadBalancer inside a VKS cluster. But the FLB has no idea what a VKS cluster or a Kubernetes pod is. How does a VIP know which VMs and ports to send traffic to, and what happens when the nodes behind it change?</p><p>In this short article, I follow three VIPs in my VCF 9.1.1 lab from the FLB to the VirtualMachineService objects on the Supervisor, their endpoints and the node VMs, including one VIP that was created by a Service inside a VKS cluster. Everything is read-only; nothing is changed in the lab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVHTp_jN0T3-aUFwLvHO6A.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Supervisor          | VCF-LAB-Supervisor, VDS networking                         |<br>| Load balancer       | Foundation Load Balancer flb-supervisor (One Arm, Medium)  |<br>| vSphere Namespace   | svc-cci-ns-9mppe                                           |<br>| VKS clusters        | kubernetes-cluster-c2ny (3 control plane, 1 worker)        |<br>|                     | kubernetes-cluster-s4cy (1 control plane, 2 workers)       |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: The Foundation Load Balancer</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WymAgP5K43X_KJVoCwA71Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8OzaQKuwGMQGrgVjSX0wmg.png" /></figure><p>The FLB is one active and one standby VM. Their management addresses 10.1.23.141 and 10.1.23.142 come from the pool foundation-lb-mgmt-10–1–23–141, and their frontend interfaces on the workload network have 10.1.31.13 and 10.1.31.14 (both shown in my article about the Supervisor workload network IP pool). On the Supervisor, the FLB is one object:</p><pre>kubectl get flb -A</pre><pre>NAMESPACE             NAME             AGE<br>vmware-system-netop   flb-supervisor   70d</pre><blockquote><em>Step 2: Three VirtualMachineServices</em></blockquote><p>A VIP in a vSphere Namespace starts as a VirtualMachineService, the VM Service equivalent of a Kubernetes Service. It selects VMs instead of pods:</p><pre>kubectl get virtualmachineservice -n svc-cci-ns-9mppe -o wide<br>kubectl get svc -n svc-cci-ns-9mppe -o wide | grep -E &#39;NAME|LoadBalancer&#39;</pre><pre>NAME                                            TYPE           AGE<br>kubernetes-cluster-c2ny                         LoadBalancer   67d<br>kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519   LoadBalancer   7d18h<br>kubernetes-cluster-s4cy                         LoadBalancer   67d<br>NAME                                            TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE     SELECTOR<br>kubernetes-cluster-c2ny                         LoadBalancer   10.96.0.221   10.1.31.153   6443/TCP            67d     &lt;none&gt;<br>kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519   LoadBalancer   10.96.0.206   10.1.31.157   9000/TCP,9001/TCP   7d18h   &lt;none&gt;<br>kubernetes-cluster-s4cy                         LoadBalancer   10.96.1.201   10.1.31.152   6443/TCP            67d     &lt;none&gt;</pre><p>For each VirtualMachineService, VM Service creates a Kubernetes Service of the same name. The Service has no pod selector (&lt;none&gt;); its external IP is the VIP on the FLB. The cluster IPs are in the Supervisor service network 10.96.0.0/23, and the VIPs come from the VIP pool lbservicefrontend-10–1–31–150.</p><p>The selector, ports and status of the three objects:</p><pre>kubectl get virtualmachineservice -n svc-cci-ns-9mppe -o jsonpath=&#39;{range .items[*]}{.metadata.name}{&quot;\n  selector=&quot;}{.spec.selector}{&quot;\n  ports=&quot;}{.spec.ports}{&quot;\n  status=&quot;}{.status}{&quot;\n&quot;}{end}&#39;</pre><pre>kubernetes-cluster-c2ny<br>  selector={&quot;capv.vmware.com/cluster.name&quot;:&quot;kubernetes-cluster-c2ny&quot;,&quot;capv.vmware.com/cluster.role&quot;:&quot;controlplane&quot;,&quot;capw.vmware.com/cluster.name&quot;:&quot;kubernetes-cluster-c2ny&quot;,&quot;capw.vmware.com/cluster.role&quot;:&quot;controlplane&quot;}<br>  ports=[{&quot;name&quot;:&quot;apiserver&quot;,&quot;port&quot;:6443,&quot;protocol&quot;:&quot;TCP&quot;,&quot;targetPort&quot;:6443}]<br>  status={&quot;loadBalancer&quot;:{&quot;ingress&quot;:[{&quot;ip&quot;:&quot;10.1.31.153&quot;}]}}<br>kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519<br>  selector={&quot;capv.vmware.com/cluster.name&quot;:&quot;kubernetes-cluster-c2ny&quot;,&quot;capv.vmware.com/cluster.role&quot;:&quot;node&quot;}<br>  ports=[{&quot;name&quot;:&quot;s3&quot;,&quot;port&quot;:9000,&quot;protocol&quot;:&quot;TCP&quot;,&quot;targetPort&quot;:31797},{&quot;name&quot;:&quot;console&quot;,&quot;port&quot;:9001,&quot;protocol&quot;:&quot;TCP&quot;,&quot;targetPort&quot;:30493}]<br>  status={&quot;loadBalancer&quot;:{&quot;ingress&quot;:[{&quot;ip&quot;:&quot;10.1.31.157&quot;}]}}<br>kubernetes-cluster-s4cy<br>  selector={&quot;capv.vmware.com/cluster.name&quot;:&quot;kubernetes-cluster-s4cy&quot;,&quot;capv.vmware.com/cluster.role&quot;:&quot;controlplane&quot;,&quot;capw.vmware.com/cluster.name&quot;:&quot;kubernetes-cluster-s4cy&quot;,&quot;capw.vmware.com/cluster.role&quot;:&quot;controlplane&quot;}<br>  ports=[{&quot;name&quot;:&quot;apiserver&quot;,&quot;port&quot;:6443,&quot;protocol&quot;:&quot;TCP&quot;,&quot;targetPort&quot;:6443}]<br>  status={&quot;loadBalancer&quot;:{&quot;ingress&quot;:[{&quot;ip&quot;:&quot;10.1.31.152&quot;}]}}</pre><pre>+-----------------------------------------------+-------------+--------------+--------------+<br>| VirtualMachineService                         | VIP         | Selects role | Target port  |<br>+-----------------------------------------------+-------------+--------------+--------------+<br>| kubernetes-cluster-c2ny                       | 10.1.31.153 | controlplane | 6443         |<br>| kubernetes-cluster-s4cy                       | 10.1.31.152 | controlplane | 6443         |<br>| kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519 | 10.1.31.157 | node         | 31797, 30493 |<br>+-----------------------------------------------+-------------+--------------+--------------+</pre><p>The VIP appears in status.loadBalancer.ingress once the load balancer has assigned it. Until then the field is empty; Broadcom KB 398504 shows the resulting error VirtualMachineService LB does not yet have VIP assigned during cluster creation.</p><p>The two cluster APIs select the VMs with the role controlplane of their cluster and forward port 6443 unchanged. The third object selects the VMs with the role node of c2ny and forwards ports 9000 and 9001 to the ports 31797 and 30493.</p><blockquote><em>Step 3: From the Selector to the Backends</em></blockquote><p>The selector is matched against the labels of the VirtualMachine objects. The role label of every node VM:</p><pre>kubectl get virtualmachine -n svc-cci-ns-9mppe -L capv.vmware.com/cluster.name,capv.vmware.com/cluster.role -o custom-columns=NAME:.metadata.name,IP:.status.network.primaryIP4 --show-labels=false<br>kubectl get virtualmachine -n svc-cci-ns-9mppe -o custom-columns=NAME:.metadata.name,IP:.status.network.primaryIP4,ROLE:&#39;.metadata.labels.capv\.vmware\.com/cluster\.role&#39;</pre><pre>NAME                                                              IP<br>kubernetes-cluster-c2ny-drhkh-bhlds                               10.1.31.34<br>kubernetes-cluster-c2ny-drhkh-scbwn                               10.1.31.33<br>kubernetes-cluster-c2ny-drhkh-xkkpn                               10.1.31.63<br>kubernetes-cluster-c2ny-kubernetes-cluster-c2ny-np-sh6g-pfr9lpm   10.1.31.35<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgrhqfg   10.1.31.62<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgs8gpx   10.1.31.61<br>kubernetes-cluster-s4cy-sg4zr-8s6tc                               10.1.31.60<br>NAME                                                              IP           ROLE<br>kubernetes-cluster-c2ny-drhkh-bhlds                               10.1.31.34   controlplane<br>kubernetes-cluster-c2ny-drhkh-scbwn                               10.1.31.33   controlplane<br>kubernetes-cluster-c2ny-drhkh-xkkpn                               10.1.31.63   controlplane<br>kubernetes-cluster-c2ny-kubernetes-cluster-c2ny-np-sh6g-pfr9lpm   10.1.31.35   node<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgrhqfg   10.1.31.62   node<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgs8gpx   10.1.31.61   node<br>kubernetes-cluster-s4cy-sg4zr-8s6tc                               10.1.31.60   controlplane</pre><p><em>(The first command combines </em><em>-L with </em><em>custom-columns; </em><em>custom-columns defines the output, so the label columns are not shown. The second command reads the role label directly.)</em></p><p>The VM Service controller writes the IP addresses of the matching VMs into the Endpoints of the Service:</p><pre>kubectl get endpoints -n svc-cci-ns-9mppe kubernetes-cluster-c2ny kubernetes-cluster-s4cy kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519<br>kubectl get endpointslices -n svc-cci-ns-9mppe -o wide | grep -E &#39;NAME|kubernetes-cluster&#39;</pre><pre>Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>NAME                                            ENDPOINTS                                         AGE<br>kubernetes-cluster-c2ny                         10.1.31.33:6443,10.1.31.34:6443,10.1.31.63:6443   67d<br>kubernetes-cluster-s4cy                         10.1.31.60:6443                                   67d<br>kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519   10.1.31.35:30493,10.1.31.35:31797                 7d18h<br>NAME                                                  ADDRESSTYPE   PORTS         ENDPOINTS                          AGE<br>kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519-7gcwz   IPv4          30493,31797   10.1.31.35                         7d18h<br>kubernetes-cluster-c2ny-wr649                         IPv4          6443          10.1.31.33,10.1.31.34,10.1.31.63   67d<br>kubernetes-cluster-s4cy-57n49                         IPv4          6443          10.1.31.60                         67d</pre><p>The backends match the VM roles exactly:</p><ul><li>VIP 10.1.31.153 (c2ny API): the three control plane VMs .33, .34 and .63 on port 6443.</li><li>VIP 10.1.31.152 (s4cy API): the single control plane VM .60 on port 6443.</li><li>VIP 10.1.31.157: the only c2ny worker .35 on ports 30493 and 31797. The c2ny control plane VMs are not backends, because they have the role controlplane, not node.</li></ul><p>The Endpoints of the two cluster APIs are 67 days old, older than the node VMs, which were all created on 24 October 2026 during a rolling update. These objects stayed; only their addresses changed. The selector is based on labels, not on IP addresses, so new VMs with new IPs become backends automatically, and the FLB keeps the same VIP. Broadcom KB 388260 uses exactly this check when a VKS VIP is not reachable: the endpoints of the cluster must match the IP addresses of its control plane nodes.</p><blockquote><em>Step 4: A VIP Created from Inside a VKS Cluster</em></blockquote><p>The third VIP was not requested on the Supervisor. It comes from a Service of type LoadBalancer in the VKS cluster c2ny. I read it with a temporary kubeconfig, which is not printed and is deleted right after the check:</p><pre>kubectl get secret kubernetes-cluster-c2ny-kubeconfig -n svc-cci-ns-9mppe -o jsonpath=&#39;{.data.value}&#39; | base64 -d &gt; /root/c2ny.kubeconfig<br>kubectl --kubeconfig /root/c2ny.kubeconfig get svc -A -o wide | grep -E &#39;NAMESPACE|LoadBalancer&#39;<br>rm -f /root/c2ny.kubeconfig</pre><pre>NAMESPACE            NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE     SELECTOR<br>minio                minio                         LoadBalancer   10.111.42.62    10.1.31.157   9000:31797/TCP,9001:30493/TCP   7d18h   app=minio</pre><p>The Service minio/minio in c2ny has the external IP 10.1.31.157 and the NodePorts 31797 (port 9000) and 30493 (port 9001). The matching VirtualMachineService on the Supervisor, kubernetes-cluster-c2ny-a2fd8d2c67a20c622d519, was created for this Service by the cloud provider of the VKS cluster: it carries the cluster name in its name, has the same VIP, selects the node VMs of c2ny and uses the two NodePorts as target ports.</p><p>A client request to 10.1.31.157:9000 therefore takes this path:</p><pre>+---+---------------------------------------------+---------------------------------------+<br>| # | Hop                                         | Address                               |<br>+---+---------------------------------------------+---------------------------------------+<br>| 1 | VIP on the Foundation Load Balancer         | 10.1.31.157:9000                      |<br>| 2 | FLB forwards to a worker node (NodePort)    | 10.1.31.35:31797                      |<br>| 3 | The node forwards the NodePort traffic to   | Service minio/minio, selector         |<br>|   | a MinIO pod                                 | app=minio                             |<br>+---+---------------------------------------------+---------------------------------------+</pre><p>In my article about kube-proxy and AntreaProxy, I showed for the cluster s4cy that external NodePort traffic is handled by kube-proxy, because AntreaProxy runs with proxyAll: false. I did not check the Antrea settings of c2ny separately; with the same setting, the last hop works the same way.</p><blockquote><em>Summary</em></blockquote><pre>+---------------------+-------------+-----------------------------+------------------------+<br>| VIP                 | Created by  | Backends (selector)         | Target                 |<br>+---------------------+-------------+-----------------------------+------------------------+<br>| 10.1.31.152:6443    | VKS (s4cy)  | control plane VMs of s4cy   | .60:6443               |<br>| 10.1.31.153:6443    | VKS (c2ny)  | control plane VMs of c2ny   | .33, .34, .63:6443     |<br>| 10.1.31.157:9000/01 | Service in  | worker VMs of c2ny          | .35:31797, .35:30493   |<br>|                     | c2ny        |                             | (NodePorts)            |<br>+---------------------+-------------+-----------------------------+------------------------+</pre><ul><li>Every VIP in this vSphere Namespace is a VirtualMachineService with a Kubernetes Service of the same name; the external IP of the Service is the VIP.</li><li>The backends are not configured by IP. The VM Service controller selects VMs by the labels cluster.name and cluster.role and writes their IPs into the Endpoints.</li><li>Cluster API VIPs select the controlplane VMs on port 6443. A LoadBalancer Service inside a VKS cluster becomes a VirtualMachineService that selects the node VMs and targets the NodePorts.</li><li>After a rolling update, the VIP stays the same and the new VMs become backends automatically.</li><li>The FLB runs as one active and one standby VM, and the vSphere Client reports it as Healthy with Low utilization.</li></ul><p>Ref:</p><p>Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues: <a href="https://knowledge.broadcom.com/external/article/388260/troubleshooting-vsphere-supervisor-workl.html">https://knowledge.broadcom.com/external/article/388260/troubleshooting-vsphere-supervisor-workl.html</a></p><p>Broadcom KB 398504, TKG Service Cluster Control plane deployment Fails with Port 6443 Connectivity Errors “VirtualMachineService LB does not yet have VIP assigned”: <a href="https://knowledge.broadcom.com/external/article/398504/tkg-service-cluster-control-plane-deploy.html">https://knowledge.broadcom.com/external/article/398504/tkg-service-cluster-control-plane-deploy.html</a></p><p>VM Operator Documentation, VirtualMachineService: <a href="https://vm-operator.readthedocs.io/en/latest/concepts/services-networking/vm-service/">https://vm-operator.readthedocs.io/en/latest/concepts/services-networking/vm-service/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e45dc3faf99d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Is the Pod MTU 1450 in VKS? From the Distributed Switch to the Pod Interface]]></title>
            <link>https://medium.com/@lubomir-tobek/why-is-the-pod-mtu-1450-in-vks-from-the-distributed-switch-to-the-pod-interface-0245a2843134?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/0245a2843134</guid>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 17:43:51 GMT</pubDate>
            <atom:updated>2026-11-05T17:43:51.004Z</atom:updated>
            <content:encoded><![CDATA[<p>My distributed switch has an MTU of 9000 bytes. Yet every pod on the pod network of my VKS cluster has an MTU of 1450 bytes. Where do the missing bytes go, and which layer actually sets the limit?</p><p>In this short article, I follow the MTU from the vSphere Distributed Switch through the VKS node to the Antrea tunnel and the pod interfaces in my VCF 9.1.1 lab. Everything is read-only: I read the vCenter settings, the Antrea configuration and the interfaces of the nodes through the existing antrea-agent pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lWCRUCciJKpdBHCZoTwlA.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Distributed switch  | VCF-LAB-vDS-Mgmt (version 9.1.0, MTU 9000)                 |<br>| Port group          | Supervisor-Workload (VLAN 1031)                            |<br>| VKS cluster         | kubernetes-cluster-s4cy (3 nodes, CNI Antrea)              |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: The Distributed Switch and the Port Group</em></blockquote><p>The network adapter of every VKS node VM is connected to the port group Supervisor-Workload on the distributed switch VCF-LAB-vDS-Mgmt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0J_V9TaweQ4KaLnnLZ4Hw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IOyUoo3p2GyJsUcP1sQGpA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPx_flAy9aTg-x7R7TeKDA.png" /></figure><p>These are the same values that the Supervisor stores in its VSphereDistributedNetwork object supervisor-workload (shown in my article about the Supervisor workload network IP pool): vlanID: 1031, allowPromiscuous: false, macChanges: false, forgedTransmits: false and macLearningPolicy.enabled: false.</p><p>The distributed switch allows frames up to 9000 bytes on its ports and uplinks; the physical switches behind the uplinks must support the same size. The switch MTU is an upper limit only: it does not set the MTU inside the VMs.</p><blockquote><em>Step 2: The Antrea Tunnel Configuration</em></blockquote><p>For the next steps, I create a temporary kubeconfig for the VKS cluster on the Supervisor. Its content is not printed:</p><pre>kubectl get secret kubernetes-cluster-s4cy-kubeconfig -n svc-cci-ns-9mppe -o jsonpath=&#39;{.data.value}&#39; | base64 -d &gt; /root/s4cy.kubeconfig</pre><p>The tunnel and MTU settings of the Antrea agent:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig get configmap antrea-config -n kube-system -o jsonpath=&#39;{.data.antrea-agent\.conf}&#39; | grep -v -E &#39;^\s*#&#39; | grep -E &#39;^(trafficEncapMode|tunnelType|tunnelPort|tunnelCsum|defaultMTU|trafficEncryptionMode|noSNAT|transportInterface):&#39;</pre><pre>trafficEncapMode: encap<br>noSNAT: false<br>tunnelType: geneve<br>tunnelPort: 0<br>tunnelCsum: false<br>trafficEncryptionMode: none<br>defaultMTU: 0</pre><pre>+-----------------------+---------+--------------------------------------------------+<br>| Setting               | Value   | Meaning                                          |<br>+-----------------------+---------+--------------------------------------------------+<br>| trafficEncapMode      | encap   | pod traffic between nodes is encapsulated        |<br>| tunnelType            | geneve  | Geneve tunnel                                    |<br>| tunnelPort            | 0       | default port (UDP 6081 for Geneve)               |<br>| trafficEncryptionMode | none    | no IPsec or WireGuard overhead                   |<br>| defaultMTU            | 0       | not set: the agent calculates the pod MTU        |<br>| noSNAT                | false   | pod traffic leaving the cluster uses node IP     |<br>+-----------------------+---------+--------------------------------------------------+</pre><p>defaultMTU: 0 means that the MTU is not fixed. Broadcom KB 441548 explains the calculation: the antrea-agent looks at the MTU of the node and subtracts 50 bytes for the Geneve tunnel header.</p><blockquote><em>Step 3: The MTU on the Nodes</em></blockquote><p>The antrea-agent pods run in the host network of the nodes, so ip link in the agent container shows the interfaces of the node itself. First, a summary per node:</p><pre>for p in $(kubectl --kubeconfig /root/s4cy.kubeconfig get pods -n kube-system -l component=antrea-agent -o name); do echo &quot;== $p&quot;; kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system $p -c antrea-agent -- ip -o link show | sed -E &#39;s/^[0-9]+: ([^:@]+)[^ ]*:.* mtu ([0-9]+).*/\1 \2/&#39; | sort -k2 -n | uniq -c -f1 | head -20; done</pre><pre>== pod/antrea-agent-m29h8<br>      3 antrea-gw0 1450<br>      3 antrea-egress0 1500<br>      1 genev_sys_6081 65000<br>      1 lo 65536<br>== pod/antrea-agent-tprf5<br>      4 antrea-gw0 1450<br>      3 antrea-egress0 1500<br>      1 genev_sys_6081 65000<br>      1 lo 65536<br>== pod/antrea-agent-wd4k7<br>      5 antrea-gw0 1450<br>      3 antrea-egress0 1500<br>      1 genev_sys_6081 65000<br>      1 lo 65536</pre><p><em>(</em><em>uniq -c -f1 groups the interfaces by MTU and counts them; the name shown is only the first interface of each group.)</em></p><p>All three nodes have the same MTU values. The number of interfaces with MTU 1450 differs between the nodes (3, 4 and 5), because each node runs a different number of pods. The full list of one node shows which interfaces are behind these groups:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/antrea-agent -c antrea-agent -- ip -o link show | sed -E &#39;s/^[0-9]+: ([^:@]+)[^ ]*:.* mtu ([0-9]+).*/\1 \2/&#39;</pre><pre>lo 65536<br>eth0 1500<br>ovs-system 1500<br>genev_sys_6081 65000<br>antrea-gw0 1450<br>antrea-egress0 1500<br>coredns--160ef3 1450<br>node-age-9b2060 1450<br>velero-5-48b870 1450</pre><p>When the check is done, the temporary kubeconfig is deleted:</p><pre>rm -f /root/s4cy.kubeconfig</pre><pre>+-------------------+-------+------------------------------------------------------+<br>| Interface         | MTU   | Role                                                 |<br>+-------------------+-------+------------------------------------------------------+<br>| eth0              | 1500  | vNIC of the node VM on the port group                |<br>| ovs-system        | 1500  | internal device of Open vSwitch                      |<br>| antrea-egress0    | 1500  | device of the Antrea Egress feature                  |<br>| genev_sys_6081    | 65000 | Geneve tunnel device (UDP 6081), not a limit         |<br>| antrea-gw0        | 1450  | gateway between the pod network and the node         |<br>| coredns--160ef3   | 1450  | host side of a pod interface (CoreDNS)               |<br>| node-age-9b2060   | 1450  | host side of a pod interface (node agent)            |<br>| velero-5-48b870   | 1450  | host side of a pod interface (Velero)                |<br>+-------------------+-------+------------------------------------------------------+</pre><p>The host-side names of the pod interfaces are shortened pod names with a hash. Their MTU of 1450 is the MTU of the pods. Pods with hostNetwork: true, such as antrea-agent itself, use the interfaces of the node and therefore the node MTU of 1500.</p><blockquote><em>Step 4: The MTU Chain</em></blockquote><pre>+--------------------------------------+-------+------------------------------------------+<br>| Layer                                | MTU   | Set by                                   |<br>+--------------------------------------+-------+------------------------------------------+<br>| Distributed switch VCF-LAB-vDS-Mgmt  | 9000  | vCenter (upper limit of the switch)      |<br>| Node vNIC eth0                       | 1500  | guest OS of the VKS node                 |<br>| Geneve overhead                      | - 50  | Antrea tunnel encapsulation              |<br>| Pod interfaces, antrea-gw0           | 1450  | antrea-agent (defaultMTU: 0 = automatic) |<br>+--------------------------------------+-------+------------------------------------------+</pre><p>The limiting layer is the node, not the distributed switch. A pod packet of 1450 bytes gets a Geneve header when it leaves the node for another node, and the outer IP packet is exactly 1500 bytes, the MTU of eth0 (the outer Ethernet header is not counted in the MTU):</p><pre>+--------------------------------+-------+<br>| Outer IP packet on eth0        | Bytes |<br>+--------------------------------+-------+<br>| Outer IPv4 header              | 20    |<br>| Outer UDP header               | 8     |<br>| Geneve header (no options)     | 8     |<br>| Inner Ethernet header          | 14    |<br>+--------------------------------+-------+<br>| Overhead                       | 50    |<br>| Pod packet (pod MTU)           | 1450  |<br>| Total = node MTU of eth0       | 1500  |<br>+--------------------------------+-------+</pre><p>The 9000 bytes on the distributed switch are headroom that the VKS nodes in my lab do not use. They are still important: the physical path must carry at least the node MTU, otherwise large encapsulated packets are dropped.</p><p>Broadcom KB 441548 describes what this means in practice. TCP adapts to the smaller pod MTU through the MSS, but a 1500-byte UDP packet from an external server to a pod must be fragmented. If a larger pod MTU is needed, the node MTU and the MTU of the whole physical path must be increased; the antrea-agent then picks up the new node MTU automatically. For jumbo-frame workloads, Broadcom KB 428030 recommends a guest MTU of 8900, which leaves room for the encapsulation overhead below 9000.</p><blockquote><em>Summary</em></blockquote><ul><li>The distributed switch VCF-LAB-vDS-Mgmt allows 9000 bytes, but the VKS nodes use MTU 1500 on eth0.</li><li>Antrea uses a Geneve tunnel with defaultMTU: 0, so the antrea-agent sets the pod MTU automatically: node MTU 1500 − 50 = 1450.</li><li>All pod interfaces and antrea-gw0 have MTU 1450 on all three nodes; the Geneve device genev_sys_6081 (65000) is not a limit.</li><li>The limiting layer is the node MTU, not the switch. A larger pod MTU requires a larger node MTU and a physical path that carries it end to end.</li><li>The port group settings in vCenter (VLAN 1031, security policies Reject) match the VSphereDistributedNetwork object of the Supervisor.</li></ul><p>Ref:</p><p>Broadcom KB 441548, UDP Packet Drops or Fragmentation in Tanzu/Antrea Environments Due to MTU Mismatches: <a href="https://knowledge.broadcom.com/external/article/441548/udp-packet-drops-or-fragmentation-in-tan.html">https://knowledge.broadcom.com/external/article/441548/udp-packet-drops-or-fragmentation-in-tan.html</a></p><p>Broadcom KB 428030, Performance tuning recommendations for pod to pod Antrea networking in a VKS environment: <a href="https://knowledge.broadcom.com/external/article/428030">https://knowledge.broadcom.com/external/article/428030</a></p><p>Antrea Documentation, antrea-agent.conf configuration template (trafficEncapMode, tunnelType, defaultMTU): <a href="https://github.com/antrea-io/antrea/blob/main/build/charts/antrea/conf/antrea-agent.conf">https://github.com/antrea-io/antrea/blob/main/build/charts/antrea/conf/antrea-agent.conf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0245a2843134" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who Implements Kubernetes Services in a VKS Cluster: kube-proxy or AntreaProxy?]]></title>
            <link>https://medium.com/@lubomir-tobek/who-implements-kubernetes-services-in-a-vks-cluster-kube-proxy-or-antreaproxy-40fea59ccf41?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/40fea59ccf41</guid>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 12:29:43 GMT</pubDate>
            <atom:updated>2026-11-05T12:29:43.509Z</atom:updated>
            <content:encoded><![CDATA[<p>Every Kubernetes Service needs something that translates its virtual ClusterIP to the IP addresses of the pods behind it. In upstream Kubernetes, this is kube-proxy with iptables, IPVS or nftables rules. Antrea, the default CNI of VKS, has its own implementation, AntreaProxy, which load-balances Services in Open vSwitch (OVS). Which one is active in a VKS cluster?</p><p>In this short article, I check both components in a VKS cluster of my VCF 9.1.1 lab, compare the Services they have programmed, and explain how they share the work. Everything is read-only: I only read configuration and use kubectl exec into the existing kube-proxy and antrea-agent pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TkPPjkPe3YDcMPptSd4Bpg.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| VKS cluster         | kubernetes-cluster-s4cy (v1.37.1+vmware.4, 3 nodes)        |<br>| CNI                 | Antrea (bootstrapAddons cniRef antrea)                     |<br>| Cluster network     | pods 10.95.0.0/16, services 10.96.0.0/12                   |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: Both Proxies Are Deployed</em></blockquote><p>On the Supervisor, the API cni.kubernetes.vmware.com offers an AntreaNetworkProxy resource. There is no such object in my lab:</p><pre>kubectl get antreanetworkproxies -A<br>kubectl get antreanetworkproxies -A -o jsonpath=&#39;{range .items[*]}{.metadata.namespace}/{.metadata.name}{&quot;\n  spec=&quot;}{.spec}{&quot;\n&quot;}{end}&#39;</pre><pre>No resources found</pre><p>The effective configuration can be read inside the VKS cluster. I create a temporary kubeconfig on the Supervisor; its content is not printed:</p><pre>kubectl get secret kubernetes-cluster-s4cy-kubeconfig -n svc-cci-ns-9mppe -o jsonpath=&#39;{.data.value}&#39; | base64 -d &gt; /root/s4cy.kubeconfig</pre><p>The DaemonSets and ConfigMaps in kube-system, and the mode of kube-proxy:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig get daemonset -n kube-system<br>kubectl --kubeconfig /root/s4cy.kubeconfig get configmap -n kube-system | grep -E &#39;NAME|kube-proxy|antrea&#39;<br>kubectl --kubeconfig /root/s4cy.kubeconfig get configmap kube-proxy -n kube-system -o jsonpath=&#39;{.data.config\.conf}&#39; | grep -E &#39;^\s*mode:|clusterCIDR|^\s*strictARP&#39;</pre><pre>NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE<br>antrea-agent   3         3         3       3            3           kubernetes.io/os=linux   67d<br>kube-proxy     3         3         3       3            3           kubernetes.io/os=linux   67d<br>NAME                                                   DATA   AGE<br>antrea-agent-tweaker                                   1      67d<br>antrea-ca                                              1      67d<br>antrea-cluster-identity                                1      67d<br>antrea-config                                          3      67d<br>kube-proxy                                             2      67d<br>clusterCIDR: &quot;10.95.0.0/16&quot;<br>  strictARP: false<br>mode: &quot;iptables&quot;</pre><p>Both antrea-agent and kube-proxy run on all three nodes. kube-proxy works in mode iptables; clusterCIDR is the pod CIDR that kube-proxy uses to recognize traffic from pods.</p><p>The vSphere Client shows the CNI and the cluster network on the summary page of the VKS cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ekuvHw-9fNgL_ml7LpE44w.png" /></figure><p>The YAML view of the cluster shows where these values are defined:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IaTHaJSsT76tk4_rqW4_qQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-0D6j58ujtjIA8ewTPWVBg.png" /></figure><p>The relevant parts of the YAML:</p><pre>spec:<br>  clusterNetwork:<br>    pods:<br>      cidrBlocks:<br>        - 10.95.0.0/16<br>    serviceDomain: cluster.local<br>    services:<br>      cidrBlocks:<br>        - 10.96.0.0/12</pre><pre>      - name: bootstrapAddons<br>        value:<br>          cniRef:<br>            name: antrea<br>            namespace: vmware-system-vks-public</pre><p>The pod CIDR 10.95.0.0/16 is the same as the clusterCIDR of kube-proxy. The service CIDR 10.96.0.0/12 (10.96.0.0–10.111.255.255) contains all ClusterIP addresses shown in Step 3. The CNI add-on antrea comes from the namespace vmware-system-vks-public, the same namespace as the ClusterClass. The status of the cluster in the same YAML shows the condition AddonsReconciled with status True.</p><blockquote><em>Step 2: How AntreaProxy Is Configured</em></blockquote><p>The Antrea agent configuration is in the ConfigMap antrea-config:</p><pre>for cm in $(kubectl --kubeconfig /root/s4cy.kubeconfig get configmap -n kube-system -o name | grep antrea-config); do echo &quot;== $cm&quot;; kubectl --kubeconfig /root/s4cy.kubeconfig get $cm -n kube-system -o jsonpath=&#39;{.data.antrea-agent\.conf}&#39; | grep -v -E &#39;^\s*#&#39; | grep -A12 -E &#39;^antreaProxy:|^featureGates:&#39;; done</pre><pre>== configmap/antrea-config<br>featureGates:<br>  Traceflow: true<br>  PacketCapture: false<br>  AntreaPolicy: true<br>  FlowExporter: false<br>  NetworkPolicyStats: true<br>  Egress: true<br>  AntreaIPAM: false<br>  Multicast: true<br>  ServiceExternalIP: true<br>  SecondaryNetwork: false<br>  TrafficControl: false<br>  SupportBundleCollection: false<br>--<br>antreaProxy:<br>  enable: true<br>  proxyAll: false<br>  nodePortAddresses: []<br>  skipServices: []<br>  defaultLoadBalancerMode: nat<br>  disableServiceHealthCheckServer: false<br>auditLogging:<br>  maxSize: 500<br>  maxBackups: 3<br>  maxAge: 28<br>  compress: true<br>memberlist:</pre><p><em>(Output limited to 12 lines after each match by </em><em>grep -A12.)</em></p><p>The running agent confirms the feature gate:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/antrea-agent -c antrea-agent -- antctl get featuregates | grep -E &#39;FEATUREGATE|AntreaProxy|EndpointSlice|TopologyAwareHints&#39;</pre><pre>FEATUREGATE                       STATUS       VERSION<br>AntreaProxy                       Enabled      GA<br>EndpointSlice                     Enabled      GA<br>TopologyAwareHints                Enabled      GA</pre><pre>+--------------------------------------+--------+-----------------------------------------------+<br>| Setting                              | Value  | Meaning                                       |<br>+--------------------------------------+--------+-----------------------------------------------+<br>| antreaProxy.enable                   | true   | AntreaProxy load-balances Services in OVS     |<br>| antreaProxy.proxyAll                 | false  | only Service traffic from pods, not from      |<br>|                                      |        | nodes or external clients                     |<br>| antreaProxy.skipServices             | []     | no Service is excluded                        |<br>| antreaProxy.defaultLoadBalancerMode  | nat    | default mode; DSR is not used                 |<br>+--------------------------------------+--------+-----------------------------------------------+</pre><p>The Antrea documentation explains proxyAll: with proxyAll: true, AntreaProxy handles ClusterIP traffic regardless of where it comes from, and kube-proxy is no longer required. With proxyAll: false, as here, AntreaProxy handles the Service traffic of pods, and kube-proxy remains responsible for traffic from the nodes themselves (including hostNetwork pods) and for external NodePort and LoadBalancer traffic.</p><blockquote><em>Step 3: The Same Services in iptables and in OVS</em></blockquote><p>The Services of the cluster:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig get svc -A --no-headers | awk &#39;{print $3}&#39; | sort | uniq -c<br>kubectl --kubeconfig /root/s4cy.kubeconfig get svc -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP,PORTS:.spec.ports[*].port</pre><pre>      9 ClusterIP<br>NAMESPACE            NAME                          TYPE        CLUSTER-IP       PORTS<br>default              kubernetes                    ClusterIP   10.96.0.1        443<br>default              supervisor                    ClusterIP   None             6443<br>default              supervisor-management-proxy   ClusterIP   None             10093<br>kube-system          antrea                        ClusterIP   10.99.103.31     443<br>kube-system          kube-dns                      ClusterIP   10.96.0.10       53,53,9153<br>kube-system          metrics-server                ClusterIP   10.108.223.35    443<br>tkg-system           packaging-api                 ClusterIP   10.110.165.164   443,8080<br>vmware-system-csi    vsphere-csi-controller        ClusterIP   10.99.102.89     2112,2113<br>vmware-system-helm   source-controller             ClusterIP   10.105.11.36     80</pre><p>There are 9 Services of type ClusterIP. Two of them, supervisor and supervisor-management-proxy, are headless (ClusterIP: None) and need no proxy. The other 7 Services have 11 ports together: 1 + 1 + 3 + 1 + 2 + 2 + 1.</p><p>kube-proxy has programmed one iptables rule per Service port in the chain KUBE-SERVICES:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/kube-proxy -- iptables-save -t nat | grep -c &#39;^-A KUBE-SERVICES .*-j KUBE-SVC-&#39;<br>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/kube-proxy -- iptables-save -t nat | grep &#39;^-A KUBE-SERVICES .*-j KUBE-SVC-&#39; | head -3</pre><pre>11<br>-A KUBE-SERVICES -d 10.110.165.164/32 -p tcp -m comment --comment &quot;tkg-system/packaging-api:metrics cluster IP&quot; -m tcp --dport 8080 -j KUBE-SVC-6FGNP2JIGDYADXF6<br>-A KUBE-SERVICES -d 10.99.102.89/32 -p tcp -m comment --comment &quot;vmware-system-csi/vsphere-csi-controller:syncer cluster IP&quot; -m tcp --dport 2113 -j KUBE-SVC-AAQCXBCVUR2XJIMS<br>-A KUBE-SERVICES -d 10.105.11.36/32 -p tcp -m comment --comment &quot;vmware-system-helm/source-controller:http cluster IP&quot; -m tcp --dport 80 -j KUBE-SVC-BIAH25WIYN6K3W3B</pre><p>AntreaProxy has programmed one OVS flow per Service port in the table ServiceLB:</p><pre>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/antrea-agent -c antrea-agent -- antctl get ovsflows -T ServiceLB | grep -c &#39;nw_dst&#39;<br>kubectl --kubeconfig /root/s4cy.kubeconfig exec -n kube-system ds/antrea-agent -c antrea-agent -- antctl get ovsflows -T ServiceLB | head -3</pre><pre>11<br><br>cookie=0x2030000000000, duration=671808.072s, table=ServiceLB, n_packets=0, n_bytes=0, idle_age=65535, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.99.102.89,tp_dst=2112 actions=set_field:0x200/0x200-&gt;reg0,set_field:0x20000/0x70000-&gt;reg4,set_field:0x5-&gt;reg7,group:5<br>cookie=0x2030000000000, duration=671808.056s, table=ServiceLB, n_packets=0, n_bytes=0, idle_age=65535, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.99.103.31,tp_dst=443 actions=set_field:0x200/0x200-&gt;reg0,set_field:0x20000/0x70000-&gt;reg4,set_field:0x6-&gt;reg7,group:6</pre><p>When the check is done, the temporary kubeconfig is deleted:</p><pre>rm -f /root/s4cy.kubeconfig</pre><p>On the checked node, both proxies know exactly the same 11 Service ports. The iptables rule matches the ClusterIP and port and jumps to a KUBE-SVC-… chain; the OVS flow matches the same ClusterIP (nw_dst) and port (tp_dst) and sends the packet to an OVS group, which selects one of the endpoints. ds/kube-proxy and ds/antrea-agent in kubectl exec select one pod of the DaemonSet, so these outputs come from one node.</p><blockquote><em>Step 4: Who Handles Which Traffic</em></blockquote><pre>+----------------------------------------------+-----------------------------------+<br>| Traffic                                      | Handled by (proxyAll: false)      |<br>+----------------------------------------------+-----------------------------------+<br>| Pod -&gt; ClusterIP (e.g. DNS to 10.96.0.10)    | AntreaProxy (OVS)                 |<br>| Node / hostNetwork pod -&gt; ClusterIP          | kube-proxy (iptables)             |<br>| External client -&gt; NodePort or LoadBalancer  | kube-proxy (iptables)             |<br>| Headless Service (ClusterIP: None)           | no proxy, DNS returns pod IPs     |<br>+----------------------------------------------+-----------------------------------+</pre><p>Most Service traffic in a cluster typically comes from pods, so in practice AntreaProxy does most of the work. The Antrea documentation notes that, with AntreaProxy enabled, pod DNS queries to the kube-dns ClusterIP are load-balanced directly by AntreaProxy to a CoreDNS pod.</p><p>kube-proxy should not be removed from a VKS cluster. In VKS, kube-proxy is a managed add-on: Broadcom KB 431402 describes VKS Machines stuck in Provisioned state with the message Addon KubeProxy is not ready: unable to retrieve kube-proxy daemonset. The Antrea settings of a VKS cluster are managed from the Supervisor through AntreaConfig objects (antreaconfigs.cni.tanzu.vmware.com, Broadcom KB 410650), rather than by editing the ConfigMap in the cluster, which belongs to the managed add-on.</p><blockquote><em>Summary</em></blockquote><ul><li>In VKS 3.8.0, both kube-proxy (mode iptables) and AntreaProxy (enable: true) run on every node.</li><li>AntreaProxy runs with proxyAll: false: it handles Service traffic from pods in OVS. kube-proxy handles traffic from nodes and external NodePort and LoadBalancer traffic.</li><li>On the checked node, both have programmed the same 11 Service ports: iptables chain KUBE-SERVICES and OVS table ServiceLB. Headless Services are not proxied.</li><li>Do not delete kube-proxy in a VKS cluster; it is a managed add-on, and its absence blocks node provisioning (KB 431402).</li><li>Change Antrea settings through AntreaConfig on the Supervisor (KB 410650).</li></ul><p>Ref:</p><p>Broadcom KB 431402, VKS Machines are stuck in “Provisioned” state and describing workload cluster reveals kube-proxy daemonset to be missing: <a href="https://knowledge.broadcom.com/external/article/431402/vks-machines-are-stuck-in-provisioned-st.html">https://knowledge.broadcom.com/external/article/431402/vks-machines-are-stuck-in-provisioned-st.html</a></p><p>Broadcom KB 410650, Enable FlowExporter for Antrea in guest cluster (AntreaConfig on the Supervisor): <a href="https://knowledge.broadcom.com/external/article/410650/enable-flowexporter-for-antrea-in-guest.html">https://knowledge.broadcom.com/external/article/410650/enable-flowexporter-for-antrea-in-guest.html</a></p><p>Antrea Documentation, Antrea Proxy: <a href="https://github.com/antrea-io/antrea/blob/main/docs/antrea-proxy.md">https://github.com/antrea-io/antrea/blob/main/docs/antrea-proxy.md</a></p><p>Kubernetes Documentation, Virtual IPs and Service Proxies: <a href="https://kubernetes.io/docs/reference/networking/virtual-ips/">https://kubernetes.io/docs/reference/networking/virtual-ips/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40fea59ccf41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where Do VKS Node IP Addresses Come From? The Supervisor Workload Network IP Pool]]></title>
            <link>https://medium.com/@lubomir-tobek/where-do-vks-node-ip-addresses-come-from-the-supervisor-workload-network-ip-pool-b55df7c7cb41?source=rss-61de7b0f9608------2</link>
            <guid isPermaLink="false">https://medium.com/p/b55df7c7cb41</guid>
            <category><![CDATA[vcf9-lab]]></category>
            <category><![CDATA[vcf9-nested-lab]]></category>
            <category><![CDATA[vcf-9]]></category>
            <category><![CDATA[vcf9-nested]]></category>
            <category><![CDATA[vcf9-learning]]></category>
            <dc:creator><![CDATA[Lubomir Tobek]]></dc:creator>
            <pubDate>Thu, 05 Nov 2026 12:27:41 GMT</pubDate>
            <atom:updated>2026-11-05T12:27:41.638Z</atom:updated>
            <content:encoded><![CDATA[<p>In my earlier articles, I looked at where the Supervisor VIPs come from and how a Foundation Load Balancer gets its addresses. The VKS nodes themselves need IP addresses too. Which pool do they come from, who else uses the same pool, and how many more nodes fit into it?</p><p>In this short article, I follow the IP addresses of the VKS nodes in my VCF 9.1.1 lab from the VirtualMachine objects to the IP pool of the Supervisor workload network, and calculate the remaining capacity. Everything is read-only; nothing is changed in the lab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wy78X2m0FHHTm43_MXmWDg.png" /></figure><blockquote><em>Lab Environment</em></blockquote><pre>+---------------------+------------------------------------------------------------+<br>| Component           | Value                                                      |<br>+---------------------+------------------------------------------------------------+<br>| Platform            | VCF 9.1.1, ESX 9.1.1, VKS 3.8.0                            |<br>| Supervisor          | VCF-LAB-Supervisor, VDS networking                         |<br>| Load balancer       | Foundation Load Balancer (2 VMs)                           |<br>| vSphere Namespace   | svc-cci-ns-9mppe                                           |<br>| VKS clusters        | kubernetes-cluster-c2ny (4 nodes)                          |<br>|                     | kubernetes-cluster-s4cy (3 nodes)                          |<br>| Access              | vCenter UI and kubectl on the Supervisor control plane     |<br>+---------------------+------------------------------------------------------------+</pre><blockquote><em>Step 1: The Node IP Addresses</em></blockquote><p>VM Service reports the primary IPv4 address of every VKS node VM:</p><pre>kubectl get virtualmachine -n svc-cci-ns-9mppe -o custom-columns=NAME:.metadata.name,POWER:.status.powerState,IP:.status.network.primaryIP4</pre><pre>NAME                                                              POWER       IP<br>kubernetes-cluster-c2ny-drhkh-bhlds                               PoweredOn   10.1.31.34<br>kubernetes-cluster-c2ny-drhkh-scbwn                               PoweredOn   10.1.31.33<br>kubernetes-cluster-c2ny-drhkh-xkkpn                               PoweredOn   10.1.31.63<br>kubernetes-cluster-c2ny-kubernetes-cluster-c2ny-np-sh6g-pfr9lpm   PoweredOn   10.1.31.35<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgrhqfg   PoweredOn   10.1.31.62<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgs8gpx   PoweredOn   10.1.31.61<br>kubernetes-cluster-s4cy-sg4zr-8s6tc                               PoweredOn   10.1.31.60</pre><p>All seven nodes are in 10.1.31.0/24, but the addresses are not in order: c2ny has .33, .34, .35 and .63. All node VMs were created during a rolling update on 24 October 2026 (see the creation times of the NetworkInterface objects in Step 3), when the old and the new VMs existed at the same time. A new VM gets a free address from the pool, not the next address after its cluster’s other nodes.</p><blockquote><em>Step 2: The Workload Network and Its IP Pools</em></blockquote><p>The Supervisor uses VDS networking. The network objects come from the API group netoperator.vmware.com:</p><pre>kubectl api-resources | grep -i -E &#39;network|ipaddress|ippool|netop&#39;</pre><pre>antreanetworkproxies                 antreaproxy                               cni.kubernetes.vmware.com/v1alpha1               true         AntreaNetworkProxy<br>ippools                                                                        crd.projectcalico.org/v1                         false        IPPool<br>ipaddressclaims                                                                ipam.cluster.x-k8s.io/v1beta2                    true         IPAddressClaim<br>ipaddresses                                                                    ipam.cluster.x-k8s.io/v1beta2                    true         IPAddress<br>aviloadbalancerconfigs                                                         netoperator.vmware.com/v1alpha1                  false        AviLoadBalancerConfig<br>foundationloadbalancerconfigs        flb                                       netoperator.vmware.com/v1alpha1                  true         FoundationLoadBalancerConfig<br>haproxyloadbalancerconfigs                                                     netoperator.vmware.com/v1alpha1                  false        HAProxyLoadBalancerConfig<br>ipaddressallocations                                                           netoperator.vmware.com/v1alpha1                  true         IPAddressAllocation<br>ippools                                                                        netoperator.vmware.com/v1alpha1                  false        IPPool<br>loadbalancerconfigs                                                            netoperator.vmware.com/v1alpha1                  false        LoadBalancerConfig<br>managementserviceaccessgrants                                                  netoperator.vmware.com/v1alpha1                  true         ManagementServiceAccessGrant<br>managementservices                                                             netoperator.vmware.com/v1alpha1                  false        ManagementService<br>networkinterfaces                                                              netoperator.vmware.com/v1alpha1                  true         NetworkInterface<br>networks                                                                       netoperator.vmware.com/v1alpha1                  true         Network<br>vmxnet3networkinterfaces                                                       netoperator.vmware.com/v1alpha1                  true         VMXNET3NetworkInterface<br>vspheredistributednetworks                                                     netoperator.vmware.com/v1alpha1                  false        VSphereDistributedNetwork<br>ingressclasses                                                                 networking.k8s.io/v1                             false        IngressClass<br>ingresses                            ing                                       networking.k8s.io/v1                             true         Ingress<br>ipaddresses                          ip                                        networking.k8s.io/v1                             false        IPAddress<br>networkpolicies                      netpol                                    networking.k8s.io/v1                             true         NetworkPolicy<br>servicecidrs                                                                   networking.k8s.io/v1                             false        ServiceCIDR<br>gatewayclasses                                                                 networking.x-k8s.io/v1alpha1pre1                 false        GatewayClass<br>gateways                                                                       networking.x-k8s.io/v1alpha1pre1                 true         Gateway<br>httproutes                                                                     networking.x-k8s.io/v1alpha1pre1                 true         HTTPRoute<br>tcproutes                                                                      networking.x-k8s.io/v1alpha1pre1                 true         TCPRoute</pre><p>The vSphere Namespace uses the network supervisor-workload, which points to a VSphereDistributedNetwork of the same name:</p><pre>kubectl get networks.netoperator.vmware.com -n svc-cci-ns-9mppe -o jsonpath=&#39;{range .items[*]}{.metadata.name}{&quot;\n  spec=&quot;}{.spec}{&quot;\n  status=&quot;}{.status}{&quot;\n&quot;}{end}&#39;</pre><pre>supervisor-workload<br>  spec={&quot;providerRef&quot;:{&quot;apiGroup&quot;:&quot;netoperator.vmware.com&quot;,&quot;apiVersion&quot;:&quot;v1alpha1&quot;,&quot;kind&quot;:&quot;VSphereDistributedNetwork&quot;,&quot;name&quot;:&quot;supervisor-workload&quot;},&quot;type&quot;:&quot;vsphere-distributed&quot;}<br>  status=</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AlQy9M15Difi3holUnvxA.png" /></figure><p>The two VSphereDistributedNetwork objects and all IP pools of netoperator.vmware.com:</p><pre>kubectl get vspheredistributednetworks -o jsonpath=&#39;{range .items[*]}{.metadata.name}{&quot;\n  spec=&quot;}{.spec}{&quot;\n  status=&quot;}{.status}{&quot;\n&quot;}{end}&#39;<br>kubectl get ippools.netoperator.vmware.com -o jsonpath=&#39;{range .items[*]}{.metadata.name}{&quot;\n  spec=&quot;}{.spec}{&quot;\n  status=&quot;}{.status}{&quot;\n&quot;}{end}&#39;</pre><pre>foundation-lb-mgmt<br>  spec={&quot;gateway&quot;:&quot;10.1.23.1&quot;,&quot;ipPools&quot;:[{&quot;apiVersion&quot;:&quot;v1alpha1&quot;,&quot;name&quot;:&quot;foundation-lb-mgmt-10-1-23-141&quot;},{&quot;apiVersion&quot;:&quot;v1alpha1&quot;,&quot;name&quot;:&quot;foundation-lb-mgmt-10-1-23-181&quot;}],&quot;portGroupID&quot;:&quot;dvportgroup-27&quot;,&quot;subnetMask&quot;:&quot;255.255.255.0&quot;}<br>  status={&quot;conditions&quot;:[{&quot;lastTransitionTime&quot;:&quot;2026-08-23T13:33:17Z&quot;,&quot;status&quot;:&quot;False&quot;,&quot;type&quot;:&quot;IPPoolPressure&quot;}],&quot;defaultPortConfig&quot;:{&quot;macManagementPolicy&quot;:{&quot;allowPromiscuous&quot;:false,&quot;forgedTransmits&quot;:false,&quot;macChanges&quot;:false,&quot;macLearningPolicy&quot;:{&quot;allowUnicastFlooding&quot;:true,&quot;enabled&quot;:false,&quot;limit&quot;:4096,&quot;limitPolicy&quot;:&quot;allow&quot;}},&quot;vlan&quot;:{&quot;type&quot;:&quot;standard&quot;,&quot;vlanID&quot;:1023}}}<br>supervisor-workload<br>  spec={&quot;gateway&quot;:&quot;10.1.31.1&quot;,&quot;ipPools&quot;:[{&quot;apiVersion&quot;:&quot;v1alpha1&quot;,&quot;name&quot;:&quot;supervisor-workload-10-1-31-10&quot;}],&quot;portGroupID&quot;:&quot;dvportgroup-11001&quot;,&quot;subnetMask&quot;:&quot;255.255.255.0&quot;}<br>  status={&quot;conditions&quot;:[{&quot;lastTransitionTime&quot;:&quot;2026-08-23T13:08:48Z&quot;,&quot;status&quot;:&quot;False&quot;,&quot;type&quot;:&quot;IPPoolPressure&quot;}],&quot;defaultPortConfig&quot;:{&quot;macManagementPolicy&quot;:{&quot;allowPromiscuous&quot;:false,&quot;forgedTransmits&quot;:false,&quot;macChanges&quot;:false,&quot;macLearningPolicy&quot;:{&quot;enabled&quot;:false}},&quot;vlan&quot;:{&quot;type&quot;:&quot;standard&quot;,&quot;vlanID&quot;:1031}}}<br>foundation-lb-mgmt-10-1-23-141<br>  spec={&quot;addressCount&quot;:2,&quot;startingAddress&quot;:&quot;10.1.23.141&quot;}<br>  status={&quot;allocated&quot;:2,&quot;conditions&quot;:[{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ready&quot;},{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;full&quot;}]}<br>foundation-lb-mgmt-10-1-23-181<br>  spec={&quot;addressCount&quot;:10,&quot;startingAddress&quot;:&quot;10.1.23.181&quot;}<br>  status={&quot;conditions&quot;:[{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ready&quot;}]}<br>lbservicefrontend-10-1-31-150<br>  spec={&quot;addressCount&quot;:51,&quot;startingAddress&quot;:&quot;10.1.31.150&quot;}<br>  status={&quot;allocated&quot;:6,&quot;conditions&quot;:[{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ready&quot;}]}<br>supervisor-workload-10-1-31-10<br>  spec={&quot;addressCount&quot;:91,&quot;startingAddress&quot;:&quot;10.1.31.10&quot;}<br>  status={&quot;allocated&quot;:17,&quot;conditions&quot;:[{&quot;status&quot;:&quot;True&quot;,&quot;type&quot;:&quot;ready&quot;}]}</pre><pre>+--------------------------------+---------------------+-------+-----------+-----------------------+<br>| IPPool                         | Range               | Count | Allocated | Used for              |<br>+--------------------------------+---------------------+-------+-----------+-----------------------+<br>| supervisor-workload-10-1-31-10 | 10.1.31.10 - .100   | 91    | 17        | workload interfaces   |<br>| lbservicefrontend-10-1-31-150  | 10.1.31.150 - .200  | 51    | 6         | LoadBalancer VIPs     |<br>| foundation-lb-mgmt-10-1-23-141 | 10.1.23.141 - .142  | 2     | 2 (full)  | FLB management        |<br>| foundation-lb-mgmt-10-1-23-181 | 10.1.23.181 - .190  | 10    | 0         | FLB management        |<br>+--------------------------------+---------------------+-------+-----------+-----------------------+</pre><p>The workload network supervisor-workload is the port group Supervisor-Workload (dvportgroup-11001) with VLAN 1031 and gateway 10.1.31.1. It has one pool for interfaces, supervisor-workload-10–1–31–10, with 91 addresses, 17 of them allocated. The pool lbservicefrontend-10–1–31–150 in the same subnet is not referenced by either network; it is the separate VIP pool of the load balancer, and the VKS API VIPs 10.1.31.152 and 10.1.31.153 come from there. The condition IPPoolPressure is False on both networks.</p><p>The Supervisor shows both networks in its workload network settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gk62QrvRkbRIepTG0c0CZQ.png" /></figure><p>The IP address range in the vSphere Client is exactly the IP pool supervisor-workload-10–1–31–10. The column Namespaces shows that 4 namespaces use this network, and any vSphere Pod in them takes an address from the same range.</p><blockquote><em>Step 3: Who Uses the 17 Addresses</em></blockquote><p>Every allocated address belongs to a NetworkInterface object. In the vSphere Namespace, each VKS node has one interface, and one more interface belongs to a vSphere Pod:</p><pre>kubectl get networkinterfaces -n svc-cci-ns-9mppe -o wide<br>kubectl get networkinterfaces -A --no-headers | awk &#39;{print $1}&#39; | sort | uniq -c<br>kubectl get ipaddressallocations -A -o wide</pre><pre>NAME                                                                                       AGE<br>kubernetes-cluster-c2ny-drhkh-bhlds-supervisor-workload-eth0                               7d17h<br>kubernetes-cluster-c2ny-drhkh-scbwn-supervisor-workload-eth0                               7d17h<br>kubernetes-cluster-c2ny-drhkh-xkkpn-supervisor-workload-eth0                               7d17h<br>kubernetes-cluster-c2ny-kubernetes-cluster-c2ny-np-sh6g-pfr9lpm-supervisor-workload-eth0   7d17h<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgrhqfg-supervisor-workload-eth0   7d18h<br>kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgs8gpx-supervisor-workload-eth0   7d18h<br>kubernetes-cluster-s4cy-sg4zr-8s6tc-supervisor-workload-eth0                               7d18h<br>vmware-system-pod-8822246e-e30d-4202-aa2c-5ac43eef3506                                     26h<br>      3 kube-system<br>      8 svc-cci-ns-9mppe<br>      4 svc-contour-jvu1r<br>      4 vmware-system-netop<br>No resources found</pre><p>The interface of a VKS node is owned by its VirtualMachine and gets its address in the mode staticpool:</p><pre>kubectl get networkinterfaces -n svc-cci-ns-9mppe -o yaml | head -60</pre><pre>apiVersion: v1<br>items:<br>- apiVersion: netoperator.vmware.com/v1alpha1<br>  kind: NetworkInterface<br>  metadata:<br>    creationTimestamp: &quot;2026-10-24T19:39:05Z&quot;<br>    finalizers:<br>    - networkinterface.netoperator.vmware.com<br>    generation: 2<br>    labels:<br>      vmoperator.vmware.com/vm-interface-name: eth0<br>      vmoperator.vmware.com/vm-name: kubernetes-cluster-c2ny-drhkh-bhlds<br>    name: kubernetes-cluster-c2ny-drhkh-bhlds-supervisor-workload-eth0<br>    namespace: svc-cci-ns-9mppe<br>    ownerReferences:<br>    - apiVersion: vmoperator.vmware.com/v1alpha5<br>      blockOwnerDeletion: true<br>      controller: true<br>      kind: VirtualMachine<br>      name: kubernetes-cluster-c2ny-drhkh-bhlds<br>      uid: 8bcaf9b8-eda3-45a8-80fb-1862034bd477<br>    resourceVersion: &quot;66593254&quot;<br>    uid: 3036f631-9840-4657-986a-92495c3fac20<br>  spec:<br>    networkName: supervisor-workload<br>    type: vmxnet3<br>  status:<br>    conditions:<br>    - lastTransitionTime: &quot;2026-10-24T19:39:06Z&quot;<br>      status: &quot;True&quot;<br>      type: Ready<br>    ipAssignmentMode: staticpool<br>    ipConfigs:<br>    - gateway: 10.1.31.1<br>      ip: 10.1.31.34<br>      ipFamily: IPv4<br>      subnetMask: 255.255.255.0<br>    networkID: dvportgroup-11001<br>- apiVersion: netoperator.vmware.com/v1alpha1<br>  kind: NetworkInterface<br>  metadata:<br>    creationTimestamp: &quot;2026-10-24T19:36:02Z&quot;<br>    finalizers:<br>    - networkinterface.netoperator.vmware.com<br>    generation: 2<br>    labels:<br>      vmoperator.vmware.com/vm-interface-name: eth0<br>      vmoperator.vmware.com/vm-name: kubernetes-cluster-c2ny-drhkh-scbwn<br>    name: kubernetes-cluster-c2ny-drhkh-scbwn-supervisor-workload-eth0<br>    namespace: svc-cci-ns-9mppe<br>    ownerReferences:<br>    - apiVersion: vmoperator.vmware.com/v1alpha5<br>      blockOwnerDeletion: true<br>      controller: true<br>      kind: VirtualMachine<br>      name: kubernetes-cluster-c2ny-drhkh-scbwn<br>      uid: 0cb96655-3ec4-4355-a1ba-5d4eb288250b<br>    resourceVersion: &quot;66590369&quot;<br>    uid: 84920517-71fa-46f2-825b-ad2ef6ae06f7<br>  spec:</pre><p><em>(Output limited to the first 60 lines by </em><em>head -60.)</em></p><p>All interfaces in all namespaces, with their network and address:</p><pre>kubectl get networkinterfaces -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,NETWORK:.spec.networkName,MODE:.status.ipAssignmentMode,IP:.status.ipConfigs[0].ip</pre><pre>NAMESPACE             NAME                                                                                       NETWORK               MODE         IP<br>kube-system           vm-13006                                                                                   supervisor-workload   staticpool   10.1.31.41<br>kube-system           vm-13007                                                                                   supervisor-workload   staticpool   10.1.31.42<br>kube-system           vm-13008                                                                                   supervisor-workload   staticpool   10.1.31.43<br>svc-cci-ns-9mppe      kubernetes-cluster-c2ny-drhkh-bhlds-supervisor-workload-eth0                               supervisor-workload   staticpool   10.1.31.34<br>svc-cci-ns-9mppe      kubernetes-cluster-c2ny-drhkh-scbwn-supervisor-workload-eth0                               supervisor-workload   staticpool   10.1.31.33<br>svc-cci-ns-9mppe      kubernetes-cluster-c2ny-drhkh-xkkpn-supervisor-workload-eth0                               supervisor-workload   staticpool   10.1.31.63<br>svc-cci-ns-9mppe      kubernetes-cluster-c2ny-kubernetes-cluster-c2ny-np-sh6g-pfr9lpm-supervisor-workload-eth0   supervisor-workload   staticpool   10.1.31.35<br>svc-cci-ns-9mppe      kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgrhqfg-supervisor-workload-eth0   supervisor-workload   staticpool   10.1.31.62<br>svc-cci-ns-9mppe      kubernetes-cluster-s4cy-kubernetes-cluster-s4cy-np-4ybb-sgs8gpx-supervisor-workload-eth0   supervisor-workload   staticpool   10.1.31.61<br>svc-cci-ns-9mppe      kubernetes-cluster-s4cy-sg4zr-8s6tc-supervisor-workload-eth0                               supervisor-workload   staticpool   10.1.31.60<br>svc-cci-ns-9mppe      vmware-system-pod-8822246e-e30d-4202-aa2c-5ac43eef3506                                     supervisor-workload   staticpool   10.1.31.47<br>svc-contour-jvu1r     vmware-system-pod-95bda2b9-ef65-4a00-a994-abf278fd7e45                                     supervisor-workload   staticpool   10.1.31.36<br>svc-contour-jvu1r     vmware-system-pod-a0b30ad9-595d-432d-99b4-a8e2a9fc8a49                                     supervisor-workload   staticpool   10.1.31.51<br>svc-contour-jvu1r     vmware-system-pod-b20ea25e-d6c5-4b73-864b-702605b5e451                                     supervisor-workload   staticpool   10.1.31.44<br>svc-contour-jvu1r     vmware-system-pod-f5ce57c3-5cce-46d3-af0c-70b171d0c450                                     supervisor-workload   staticpool   10.1.31.46<br>vmware-system-netop   flb-supervisor--0--frontend--supervisor-workload                                           supervisor-workload   staticpool   10.1.31.13<br>vmware-system-netop   flb-supervisor--0--management--foundation-lb-mgmt                                          foundation-lb-mgmt    staticpool   10.1.23.141<br>vmware-system-netop   flb-supervisor--1--frontend--supervisor-workload                                           supervisor-workload   staticpool   10.1.31.14<br>vmware-system-netop   flb-supervisor--1--management--foundation-lb-mgmt                                          foundation-lb-mgmt    staticpool   10.1.23.142</pre><p>The three interfaces in kube-system are named after vCenter VM IDs. The Supervisor control plane VM I am logged in to has 10.1.31.42 on eth1:</p><pre>ip -4 -o addr show | grep 10.1.31</pre><pre>3: eth1    inet 10.1.31.42/24 brd 10.1.31.255 scope global eth1\       valid_lft forever preferred_lft forever</pre><p>Broadcom KB 415007 also shows the NetworkInterface of a Supervisor control plane VM with a name in the form vm-######. The three interfaces .41, .42 and .43 are therefore the workload interfaces of the three Supervisor control plane VMs. The vmware-system-pod-… interface in the vSphere Namespace belongs to a vSphere Pod:</p><pre>kubectl get pods -n svc-cci-ns-9mppe -o wide | grep -E &#39;NAME|10.1.31.47&#39;</pre><pre>NAME                                        READY   STATUS    RESTARTS      AGE     IP           NODE                               NOMINATED NODE   READINESS GATES<br>cci-ns-controller-manager-88768f86d-pg254   1/1     Running   0             26h     10.1.31.47   cons-esx-02.lab.local              &lt;none&gt;           &lt;none&gt;</pre><p>The sum matches the pool status allocated: 17. The two FLB management interfaces use the separate network foundation-lb-mgmt and are not part of this count. IPAddressAllocation objects are not used in this lab.</p><p>The important point: the pool is shared. The same 91 addresses serve the Supervisor control plane, the FLB frontend, every VKS node and every vSphere Pod, including the pods of Supervisor Services such as Contour.</p><blockquote><em>Step 4: How Much Room Is Left</em></blockquote><pre>+--------------------------------------------------+-----------+<br>| Item                                             | Addresses |<br>+--------------------------------------------------+-----------+<br>| Pool size (10.1.31.10 - 10.1.31.100)             | 91        |<br>| Allocated now                                    | 17        |<br>| Free now                                         | 74        |<br>| Temporary: rolling update with maxSurge 1        | +1 each   |<br>| (per control plane or node pool being rolled)    |           |<br>| Temporary: Supervisor upgrade                    | +1        |<br>| (new control plane VM before an old one goes)    |           |<br>+--------------------------------------------------+-----------+</pre><p>Every new VKS node and every new vSphere Pod, including the pods of Supervisor Services, takes one address from the same pool. A rolling update with maxSurge: 1 creates a new VM before an old one is removed, so it needs one extra address for each control plane or node pool that is rolled out at the same time.</p><p>Broadcom KB 415007 describes what happens when the pool runs out: a Supervisor upgrade gets stuck at Configured Supervisor Control plane VM’s Workload Network, because the new control plane VM gets no IP address, and vCenter warns when the IPPool utilization is above 95%. The fix is to add another IP range to the workload network.</p><p>With 74 free addresses, my lab is far from that limit. In a production Supervisor with many VKS nodes and Supervisor Services, the size of this pool deserves the same planning as the VIP pool.</p><blockquote><em>Summary</em></blockquote><ul><li>VKS node IP addresses come from the IP pool of the Supervisor workload network (supervisor-workload-10–1–31–10, 91 addresses), assigned in the mode staticpool.</li><li>The pool is shared by the Supervisor control plane VMs, the FLB frontend, the VKS nodes and all vSphere Pods. In my lab: 3 + 2 + 7 + 5 = 17 addresses.</li><li>Node addresses are not assigned in order. After a rolling update, a node can get any free address, so firewall rules should use the whole range, not single node IPs.</li><li>Plan headroom for rolling updates and Supervisor upgrades: each new VM needs an address before the old one releases its own.</li><li>Watch the condition IPPoolPressure on the VSphereDistributedNetwork and the allocated value of the IP pool.</li></ul><p>Ref:</p><p>Broadcom KB 415007, VKS Supervisor Cluster upgrade is stuck: “Configured Supervisor Control plane VM’s Workload Network”: <a href="https://knowledge.broadcom.com/external/article/415007/vks-supervisor-cluster-upgrade-is-stuck.html">https://knowledge.broadcom.com/external/article/415007/vks-supervisor-cluster-upgrade-is-stuck.html</a></p><p>Broadcom KB 410597, Networking Requirements for downloading VKS Supervisor Services (workload network IP range in the vSphere Client): <a href="https://knowledge.broadcom.com/external/article/410597/networking-requirements-for-downloading.html">https://knowledge.broadcom.com/external/article/410597/networking-requirements-for-downloading.html</a></p><p>Broadcom Developer, vSphere Automation API: NamespaceManagement Networks IPAssignment (NODE ranges on VDS networks): <a href="https://developer.broadcom.com/xapis/vsphere-automation-api/latest/vcenter/data-structures/NamespaceManagement_Networks_IPAssignment">https://developer.broadcom.com/xapis/vsphere-automation-api/latest/vcenter/data-structures/NamespaceManagement_Networks_IPAssignment</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b55df7c7cb41" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>