<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by kleinpapa on Medium]]></title>
        <description><![CDATA[Stories by kleinpapa on Medium]]></description>
        <link>https://medium.com/@kleinpapa?source=rss-5bb8efb724bd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ubCxEo5NFUJMSXxSoEN99g.jpeg</url>
            <title>Stories by kleinpapa on Medium</title>
            <link>https://medium.com/@kleinpapa?source=rss-5bb8efb724bd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:06:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kleinpapa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[High-Availability Storage for Small Kubernetes Clusters: Surviving a Whole Datacenter Failure with…]]></title>
            <link>https://kleinpapa.medium.com/high-availability-storage-for-small-kubernetes-clusters-surviving-a-whole-datacenter-failure-with-a97cd38cc320?source=rss-5bb8efb724bd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a97cd38cc320</guid>
            <category><![CDATA[on-premise]]></category>
            <category><![CDATA[longhorns]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[persistence]]></category>
            <dc:creator><![CDATA[kleinpapa]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 21:17:33 GMT</pubDate>
            <atom:updated>2026-11-03T07:55:39.321Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFH8FZ6uYCy7_ikjUHLLiw.jpeg" /></figure><h3>High-Availability Storage for Small Kubernetes Clusters: Surviving a Whole Datacenter Failure with Longhorn</h3><p>If you run Kubernetes in a public cloud, persistent storage feels like an afterthought. You create a PersistentVolumeClaim (PVC), the cloud provider stamps out an AWS EBS volume or a Google Persistent Disk behind the scenes, attaches it, and you move on with your day.</p><p>The moment you leave managed cloud environments and run on bare metal, on-premise infrastructure, or a compact private virtualization cluster, stateful storage becomes your biggest operational nightmare.</p><p>Most teams starting out on-prem fall straight into the <strong>Local Storage Trap</strong>. They deploy a local-path-provisioner because it is simple, fast, and takes five minutes to configure. It binds a host path to a Pod. Everything looks great during testing, until a physical node drops offline, a hypervisor reboots, or hardware fails.</p><p>Suddenly your database Pod is marked for rescheduling, but it cannot start on any other node. Why? Because its disk is trapped on the dead host.</p><p>In our own environment the problem is even sharper. We run RKE2 on VMs spread over <strong>three datacenters</strong>, and each datacenter has its own storage. There is <strong>no RAID and no replication between those storage systems</strong> in the virtualization layer. If one datacenter goes dark, its disks are gone, and nothing keeps a copy of the data anywhere else.</p><p>In this guide we will build the fix: a storage layer that keeps one copy of every volume in every datacenter, so that losing an entire site does not lose data.</p><h3>The Bare-Metal Dilemma: The Heavyweight vs. The Fragile</h3><p>When engineers look for a way to survive hardware failures on-prem, they usually see three unappealing options:</p><ol><li><strong>Enterprise SAN/NAS hardware.</strong> Dedicated storage arrays (NetApp, Dell EMC, Pure Storage) with dual controllers and replicated shelves. They work brilliantly, but they cost tens or hundreds of thousands of dollars and bring vendor-specific CSI drivers.</li><li><strong>Ceph / Rook-Ceph.</strong> The giant of open-source distributed storage. It is powerful, but for a small cluster (e.g. 3 to 5 nodes) it is classic overkill. It wants dedicated disks, separate storage networks, and serious CPU and memory per OSD just to keep quorum and state maps. When a small Ceph cluster breaks, recovering it can take days of specialized storage-admin expertise.</li><li><strong>Traditional NFS.</strong> Shared mounts give you multi-node access, but the NFS server is a Single Point of Failure, write latency and locking are mediocre, and you do not get real block storage behavior.</li></ol><p>This is where <strong>Longhorn</strong> shines. Longhorn is a cloud-native, micro-architected distributed block storage system that turns the local drives of your Kubernetes nodes into a replicated, highly available storage pool.</p><p>Instead of running one monolithic storage cluster, Longhorn starts a dedicated controller for each volume and synchronously replicates the data blocks across failure domains. Think of it as a container-aware, software-defined network RAID-1 across your physical machines or datacenters.</p><h3>Decoding the Vocabulary: Storage Terms You Will Meet Along the Way</h3><p>Storage has its own dialect. Before we touch any manifest, here is the short dictionary for every term that appears in the rest of this guide.</p><ul><li><strong>CSI (Container Storage Interface):</strong> The standard that lets Kubernetes talk to third-party storage backends without changing Kubernetes code.</li><li><strong>PV (PersistentVolume) and PVC (PersistentVolumeClaim):</strong> A PV is a piece of provisioned storage in the cluster. A PVC is a workload’s request for storage. Kubernetes binds a matching PV to a PVC.</li><li><strong>StorageClass:</strong> A blueprint describing how a volume is provisioned: which driver, which reclaim policy, which parameters (such as the replica count).</li><li><strong>Longhorn Engine:</strong> A lightweight controller process started for a single volume. It exposes a Linux block device on the node where the workload Pod runs and orchestrates the writes to the replicas.</li><li><strong>Replica:</strong> A full copy of a volume’s data, stored on the filesystem of a worker node under a specific path (by default /var/lib/longhorn). A volume with three replicas exists as three independent copies on three different nodes.</li><li><strong>Synchronous replication:</strong> When your workload writes a block, the engine does not acknowledge the write until the block has been written to all healthy replicas over the network.</li><li><strong>Anti-affinity:</strong> Rules about where replicas may live. <em>Node</em> anti-affinity keeps two copies of a volume off the same node. <em>Zone</em> anti-affinity keeps two copies out of the same zone (a rack, a server room, or a datacenter).</li><li><strong>Degraded volume:</strong> A volume that is still readable and writable, but has lost one or more of its replicas.</li><li><strong>iSCSI (</strong><strong>iscsid):</strong> Longhorn relies on the Linux iSCSI initiator of the host to attach its block devices to the node where your Pod runs.</li></ul><h3>How It Works: One Copy of Everything in Every Datacenter</h3><p>Here is the whole idea in one sentence: <strong>every volume is stored three times, and each copy lives in a different datacenter.</strong></p><p>In Longhorn terms, each datacenter (dc-1, dc-2, dc-3) is a <strong>zone</strong>. When your Pod writes to its volume, the Longhorn engine sends that write to all three replicas and waits until every one of them confirms it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qMevJ5ESMGPUXM7dcEURhw.png" /></figure><p>Two settings make this happen:</p><ol><li><strong>Three replicas per volume</strong>: one copy for each datacenter.</li><li><strong>Strict zone anti-affinity</strong>: Longhorn is forbidden to put two copies of the same volume into one zone. With three replicas and three zones, that leaves exactly one copy per datacenter.</li></ol><p>So what happens when a whole datacenter goes dark? Two copies are still online. The Pod is restarted on a node in a surviving datacenter, the volume is attached there, and the application carries on with all the data that was confirmed before the failure.</p><blockquote><strong><em>Not instant, though.</em></strong><em> The dead node has to be detected, the Pod evicted and rescheduled, and the volume attached on the new node. The Longhorn docs put the attach step alone at roughly 1 to 7 minutes. This is high availability for your </em>data<em>, not zero downtime.</em></blockquote><blockquote><strong><em>And not a backup.</em></strong><em> An accidental </em><em>DELETE or a buggy release is replicated to all three copies within milliseconds. You still need a separate backup target.</em></blockquote><h3>Prerequisites</h3><h4>Kubernetes version</h4><p>The Longhorn 1.13 line requires Kubernetes 1.34 or newer. Our cluster runs Kubernetes 1.36, which is fine. Always check the release notes before bumping the chart version.</p><h4>No cloud provider needed</h4><p>Longhorn does not need a cloud provider integration. That means no vendor-specific CPI/CSI drivers, no storage policies.</p><h4>Packages on every Longhorn node</h4><pre>apt-get update<br>apt-get install -y open-iscsi nfs-common<br>systemctl enable --now iscsid</pre><ul><li>open-iscsi provides iscsid. Avoid version 2.1.12, which is known to be problematic.</li><li>nfs-common is needed for backups and for ReadWriteMany volumes.</li></ul><p>Longhorn has its own preflight checker, a small CLI called longhornctl. It is a standalone binary that you supposed to download yourself; it does not come with the cluster. Run it from any machine that has a kubeconfig for the cluster (for example on a control plane node). It talks to the Kubernetes API and starts a temporary check job on every node, so it needs the same access as kubectl. Pick the release that matches your Longhorn version (see the <a href="https://github.com/longhorn/cli/releases">longhorn/cli releases</a>):</p><pre>LHCTL_VERSION=&lt;release-tag&gt;   # e.g. the tag matching your Longhorn version<br>ARCH=amd64                    # or arm64<br><br>curl -sSfL -o longhornctl https://github.com/longhorn/cli/releases/download/${LHCTL_VERSION}/longhornctl-linux-${ARCH}<br>chmod +x longhornctl<br>mv longhornctl /usr/local/bin/</pre><p>Then run the check before you install anything:</p><pre>longhornctl check preflight</pre><h4>A dedicated disk</h4><p>Do not let Longhorn write onto the system disk. Add a separate virtual disk and mount it at /var/lib/longhorn through /etc/fstab. On a dedicated disk it is also safe to lower the reserved free space from the default 25% to 10%.</p><h4>Hardware and network</h4><ul><li><strong>Recommended:</strong> at least 4 vCPU and 4 GiB RAM per node, and SSD or NVMe disks.</li><li><strong>Recommended:</strong> 10 Gbps between the nodes for production. It is a guideline, not a hard limit: Longhorn will run on slower links, but every replica rebuild and every write travels over this network, so a slow link shows up directly as slow volumes.</li><li>At least one Longhorn node in every zone, ideally two, so that losing a single node does not take the whole zone’s replicas with it.</li></ul><p>This is the part that people underestimate: <strong>replication is synchronous, so the latency between your datacenters is your write latency.</strong> Longhorn does not publish an RTT limit, but for a multi-site setup, latency matters more than bandwidth.</p><p>And since our virtualization layer has no RAID or replication of its own, the Longhorn replicas are the <em>only</em> data protection layer. The disks of a zone’s Longhorn nodes must therefore sit on that datacenter’s <em>own</em> storage. Otherwise “three zones” on paper does not mean three failure domains in reality.</p><h3>Step 1: Teach Longhorn Where the Datacenters Are</h3><p>Longhorn reads the standard topology.kubernetes.io/zone label of each node. Nodes without that label are all treated as one big shared zone, and a single unlabeled worker can quietly ruin your distribution. So <strong>every Longhorn node needs a zone label</strong>.</p><p>The durable way is to set the labels in the node’s own configuration, so they survive reboots and node replacement. With RKE2, for example, a worker in the dc-2 datacenter gets this in /etc/rancher/rke2/config.yaml:</p><pre>node-label:<br>  - &quot;topology.kubernetes.io/zone=dc-2&quot;<br>  - &quot;node.longhorn.io/create-default-disk=true&quot;</pre><p>Then restart the agent:</p><pre>systemctl restart rke2-agent</pre><p>The second label tells Longhorn that this node has a dedicated disk and should get a default disk object. We will use it in the Helm values with createDefaultDiskLabeledNodes: true, so only the nodes we mark will contribute storage.</p><p>For a quick test you can label by hand:</p><pre>kubectl label node &lt;node-name&gt; topology.kubernetes.io/zone=dc-2<br>kubectl label node &lt;node-name&gt; node.longhorn.io/create-default-disk=true<br><br>kubectl get nodes -L topology.kubernetes.io/zone</pre><h3>Step 2: Install Longhorn with Helm</h3><p>Longhorn is installed with its official Helm chart. All the settings that matter for our multi-datacenter design go into one values.yaml file.</p><p><strong>values.yaml</strong></p><pre>defaultSettings:<br>  # --- 3 datacenters: 1 replica per zone ---<br>  replicaZoneSoftAntiAffinity: false   # never two replicas in the same zone<br>  replicaSoftAntiAffinity: false       # never two replicas on the same node<br>  replicaAutoBalance: least-effort     # keep at least one replica per zone<br>  # --- only labeled nodes get a Longhorn disk ---<br>  createDefaultDiskLabeledNodes: true<br>  defaultDataPath: /var/lib/longhorn/<br>  storageMinimalAvailablePercentage: 10<br>  # --- datacenter failure: reschedule pods from the dead node ---<br>  nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod<br>persistence:<br>  defaultClass: true<br>  defaultClassReplicaCount: 3<br>  reclaimPolicy: Retain<br>ingress:<br>  enabled: false       # the UI has no authentication by default</pre><p>Add the chart repository and install:</p><pre>helm repo add longhorn https://charts.longhorn.io<br>helm repo update<br><br>helm upgrade --install longhorn longhorn/longhorn \<br>  --namespace longhorn-system --create-namespace \<br>  --version &lt;1.13.x&gt; \<br>  -f values.yaml</pre><p>Pin the chart version explicitly. Longhorn minor versions should be upgraded one by one, after reading the upgrade guide, never by blindly following the latest chart.</p><h4>What do these settings actually do?</h4><p><strong>replicaZoneSoftAntiAffinity: false</strong><br>The heart of the design. Longhorn may not put a new replica into a zone that already has a healthy one. This is what gives us one replica per datacenter. The default is true.</p><p><strong>replicaSoftAntiAffinity: false</strong><br>Two replicas of one volume never share a node (this is already the default).</p><p><strong>replicaAutoBalance: least-effort</strong><br>Rebalances replicas when new nodes appear and keeps at least one per zone. best-effort is more even, but causes constant rebuilds on an unstable cluster.</p><p><strong>createDefaultDiskLabeledNodes: true</strong><br>Only nodes labeled node.longhorn.io/create-default-disk=true get a disk.</p><p><strong>nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod</strong><br>When a node dies, Longhorn deletes the stuck Pods so they can reschedule onto another datacenter. With the default (do-nothing), a StatefulSet Pod just hangs forever.</p><p><strong>persistence.defaultClassReplicaCount: 3</strong><br>The longhorn StorageClass asks for three replicas: one per zone.</p><p><strong>persistence.reclaimPolicy: Retain</strong><br>Deleting a PVC does not delete the PV and its data. A cheap insurance against human error (the chart default is Delete).</p><p><strong>ingress.enabled: false</strong><br>The Longhorn UI can delete volumes and change settings, and it has no login. We do not publish it by default.</p><p>One more rule to remember: a volume with three replicas needs <strong>at least three zones, each with at least one node that has enough free disk space</strong>. If one zone has no free Longhorn disk, the new volume cannot be scheduled and the PVC stays Pending.</p><h3>Step 3: Verify the Distribution</h3><p>First check that the release is deployed and the settings landed:</p><pre>helm status longhorn -n longhorn-system<br>kubectl -n longhorn-system get pods<br><br>kubectl -n longhorn-system get settings.longhorn.io replica-zone-soft-anti-affinity -o jsonpath=&#39;{.value}{&quot;\n&quot;}&#39;<br>kubectl -n longhorn-system get settings.longhorn.io replica-auto-balance -o jsonpath=&#39;{.value}{&quot;\n&quot;}&#39;</pre><p>You should see false and least-effort. Also check that longhorn is the only (default) StorageClass:</p><pre>kubectl get storageclass</pre><p>Now the real test. Create a small PVC and a Pod that uses it:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: test-longhorn-pvc<br>  namespace: default<br>spec:<br>  storageClassName: longhorn<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 1Gi<br>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: test-longhorn-pod<br>  namespace: default<br>spec:<br>  containers:<br>    - name: test-container<br>      image: busybox:latest<br>      command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;echo &#39;Longhorn works!&#39; &gt; /data/test.txt &amp;&amp; sleep 3600&quot;]<br>      volumeMounts:<br>        - mountPath: &quot;/data&quot;<br>          name: persistent-storage<br>  volumes:<br>    - name: persistent-storage<br>      persistentVolumeClaim:<br>        claimName: test-longhorn-pvc</pre><pre>kubectl apply -f longhorn-test.yaml<br>kubectl get pvc test-longhorn-pvc        # expect: Bound<br>kubectl exec -it test-longhorn-pod -- cat /data/test.txt</pre><p>Then ask Longhorn where the replicas ended up, and compare that with the zone of each node:</p><pre>kubectl -n longhorn-system get replicas.longhorn.io \<br>  -o custom-columns=NAME:.metadata.name,VOLUME:.spec.volumeName,NODE:.spec.nodeID</pre><pre>kubectl get nodes -L topology.kubernetes.io/zone</pre><p>You are looking for <strong>three replicas on three nodes with three different zone labels</strong> (dc-1, dc-2, dc-3). If two replicas share a zone, or a node has no zone label at all, go back to Step 1.</p><h3>Step 4: Break a Datacenter on Purpose</h3><p>Storage HA that has never been tested is just a hypothesis. So let’s test it with a StatefulSet that writes a heartbeat line to a Longhorn volume every five seconds. After the failure, that log tells us whether the data survived and how long the outage lasted.</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: longhorn-test<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: dc-test<br>  namespace: longhorn-test<br>spec:<br>  clusterIP: None<br>  selector:<br>    app: dc-test<br>  ports:<br>    - name: dummy<br>      port: 80<br>---<br>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: dc-test<br>  namespace: longhorn-test<br>spec:<br>  serviceName: dc-test<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: dc-test<br>  template:<br>    metadata:<br>      labels:<br>        app: dc-test<br>    spec:<br>      tolerations:<br>        - key: node.kubernetes.io/not-ready<br>          operator: Exists<br>          effect: NoExecute<br>          tolerationSeconds: 30<br>        - key: node.kubernetes.io/unreachable<br>          operator: Exists<br>          effect: NoExecute<br>          tolerationSeconds: 30<br>      containers:<br>        - name: writer<br>          image: busybox:1.37<br>          env:<br>            - name: NODE_NAME<br>              valueFrom:<br>                fieldRef:<br>                  fieldPath: spec.nodeName<br>          command:<br>            - /bin/sh<br>            - -c<br>            - |<br>              [ -f /data/created-on ] || echo &quot;$(date -u +%FT%TZ) $HOSTNAME on $NODE_NAME&quot; &gt; /data/created-on<br>              while true; do<br>                echo &quot;$(date -u +%FT%TZ) $HOSTNAME on $NODE_NAME&quot; &gt;&gt; /data/heartbeat.log<br>                sleep 5<br>              done<br>          volumeMounts:<br>            - name: data<br>              mountPath: /data<br>  volumeClaimTemplates:<br>    - metadata:<br>        name: data<br>      spec:<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        storageClassName: longhorn<br>        resources:<br>          requests:<br>            storage: 1Gi</pre><p>Two details in this manifest are deliberate:</p><ul><li><strong>No </strong><strong>nodeSelector and no affinity.</strong> We do not decide where the Pod runs. We simply power off whichever node it landed on, so the Pod is free to move to any surviving zone.</li><li><strong>tolerationSeconds: 30.</strong> Kubernetes normally waits 300 seconds before evicting Pods from a dead node. We shorten that to 30 seconds so the test does not take forever. This is a test-only shortcut.</li></ul><p>Open four terminals and watch everything at once:</p><pre>kubectl get nodes -w<br>kubectl -n longhorn-test get pods -o wide -w<br>kubectl -n longhorn-system get volumes.longhorn.io \<br>  -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness -w<br>kubectl -n longhorn-system get events --sort-by=.lastTimestamp -w</pre><p>Now pull the plug: in your hypervisor console (vCenter, Proxmox, or similar), find the VM that runs dc-test-0 and choose <strong>Power Off</strong> (the hard one, not <em>Shut Down Guest OS</em>). A hard power-off simulates a real power failure, because the guest operating system gets no chance to shut down gracefully.</p><blockquote><strong><em>Safety first:</em></strong><em> only ever power off a </em>worker<em> node, never a control plane node, otherwise you endanger the etcd quorum. And do not run this test on a node with critical workloads.</em></blockquote><h4>What you should see</h4><ol><li><strong>~40–60 s:</strong> the node turns NotReady.</li><li><strong>+30 s:</strong> the Pod goes Terminating/Unknown after the 30 s toleration.</li><li><strong>Then:</strong> Longhorn force-deletes the Pod (thanks to nodeDownPodDeletionPolicy) and the StatefulSet starts a new one on a node in a <strong>different zone</strong>.</li><li><strong>A few minutes:</strong> ContainerCreating while the volume is attached, then Running (1–7 minutes according to the Longhorn docs). Meanwhile the volume shows Degraded, with 2 healthy replicas.</li><li><strong>~10 min:</strong> the replicaReplenishmentWaitInterval (600 s) expires. Longhorn would like to build a replacement replica, but there is no free zone to put it in, so the volume stays degraded.</li></ol><p>Then check the data:</p><pre>kubectl -n longhorn-test get pods -o wide<br>kubectl -n longhorn-test exec dc-test-0 -- cat /data/created-on<br>kubectl -n longhorn-test exec dc-test-0 -- tail -40 /data/heartbeat.log</pre><p>The heartbeat log should start with lines carrying the <strong>old node’s name</strong>, then show a time gap (your outage), and continue with the <strong>new node’s name</strong>. The old lines are all still there; that is the proof that the data followed the Pod to another datacenter.</p><p>When you power the VM back on, the node becomes Ready again and Longhorn rebuilds the missing replica. The volume goes back to Healthy once the rebuild finishes.</p><h3>The Trade-Off You Should Know About</h3><p>Strict zone anti-affinity has a price, and we chose it on purpose. When a datacenter is down, Longhorn <strong>cannot</strong> place the missing third replica anywhere, because the two surviving zones already hold one each. The volume runs degraded, with two copies, until the datacenter returns.</p><p>The alternative (soft anti-affinity) would let Longhorn pile two replicas into the same surviving datacenter. The volume would look healthy sooner, but a second failure in that datacenter would then take two copies at once. We would rather run degraded and honest than healthy and fragile.</p><p>Also note what this setup does <em>not</em> give you: it does not protect against a second datacenter failing while the first is still down (you are down to one copy), and it does not replace backups.</p><h3>Exposing the Longhorn UI Safely</h3><p>Longhorn comes with a nice web UI, but <strong>it has no authentication by default</strong>, and through it anyone can delete volumes or change settings. That is why we left ingress.enabled: false in the Helm values.</p><p>For occasional access, port-forward is enough:</p><pre>kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80</pre><p>For permanent access, publish the UI through your ingress controller or Gateway API implementation, restrict it to your internal network, and put authentication (for example basic auth) in front of it. How you do that depends on what you run. As one example, here is how it looks with Traefik, using an IngressRoute and a BasicAuth middleware:</p><pre>apiVersion: traefik.io/v1alpha1<br>kind: Middleware<br>metadata:<br>  name: longhorn-ui-auth<br>  namespace: longhorn-system<br>spec:<br>  basicAuth:<br>    secret: longhorn-ui-auth<br><br>---<br><br>apiVersion: traefik.io/v1alpha1<br>kind: IngressRoute<br>metadata:<br>  name: longhorn-ui<br>  namespace: longhorn-system<br>spec:<br>  entryPoints:<br>    - websecure<br>  routes:<br>    - match: Host(`longhorn.example.internal`)<br>      kind: Rule<br>      middlewares:<br>        - name: longhorn-ui-auth<br>      services:<br>        - name: longhorn-frontend<br>          port: 80</pre><p>With NGINX, another ingress controller, or Gateway API (Gateway and HTTPRoute), the idea is the same, only the objects differ. The longhorn-ui-auth Secret above holds an htpasswd file. In production, manage it with External Secrets instead of creating it by hand.</p><h3>Bonus: The GitOps Way with Flux CD</h3><p>If your cluster is managed with Flux CD, the same Helm chart can be installed declaratively from Git. Create a longhorn/ folder in your components repository with four files.</p><p><strong>namespace.yaml</strong></p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: longhorn-system<br>  annotations:<br>    kustomize.toolkit.fluxcd.io/prune: disabled<br>    # never let Flux delete this namespace (and the data with it)</pre><p><strong>longhorn-repo.yaml</strong></p><pre>apiVersion: source.toolkit.fluxcd.io/v1<br>kind: HelmRepository<br>metadata:<br>  name: longhorn-repo<br>  namespace: longhorn-system<br>spec:<br>  interval: 24h<br>  url: https://charts.longhorn.io</pre><p><strong>longhorn-release.yaml</strong> carries the very same values as before, plus a few Flux-specific lines:</p><pre>apiVersion: helm.toolkit.fluxcd.io/v2<br>kind: HelmRelease<br>metadata:<br>  name: longhorn<br>  namespace: longhorn-system<br>spec:<br>  releaseName: longhorn<br>  interval: 1h<br>  chart:<br>    spec:<br>      chart: longhorn<br>      version: &quot;1.13.*&quot;<br>      sourceRef:<br>        kind: HelmRepository<br>        name: longhorn-repo<br>        namespace: longhorn-system<br>  install:<br>    remediation:<br>      retries: 3<br>  values:<br>    defaultSettings:<br>      replicaZoneSoftAntiAffinity: false<br>      replicaSoftAntiAffinity: false<br>      replicaAutoBalance: least-effort<br>      createDefaultDiskLabeledNodes: true<br>      defaultDataPath: /var/lib/longhorn/<br>      storageMinimalAvailablePercentage: 10<br>      nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod<br>    persistence:<br>      defaultClass: true<br>      defaultClassReplicaCount: 3<br>      reclaimPolicy: Retain<br>    preUpgradeChecker:<br>      jobEnabled: false  # recommended by the chart when using GitOps<br>    ingress:<br>      enabled: false</pre><p><strong>kustomization.yaml</strong></p><pre>apiVersion: kustomize.config.k8s.io/v1beta1<br>kind: Kustomization<br>resources:<br>  - namespace.yaml<br>  - longhorn-repo.yaml<br>  - longhorn-release.yaml</pre><p>Add longhorn/ to the parent kustomization.yaml, push, and Flux does the rest. To check the result:</p><pre>helm status longhorn -n longhorn-system</pre><blockquote><strong><em>Two safety nets worth noting.</em></strong><em> The </em><em>prune: disabled annotation stops Flux from deleting </em><em>longhorn-system (and the Longhorn data) if someone removes the folder from Git by mistake. And the chart version floats only on the patch level (</em><em>1.13.*): minor versions are upgraded one by one, by hand.</em></blockquote><h3>Conclusion</h3><p>You do not need a six-figure SAN or a full Ceph cluster to survive the loss of a datacenter. With three zones, three replicas, strict zone anti-affinity and a few well-chosen Helm values, Longhorn gives a small Kubernetes cluster a storage layer that behaves like a network RAID-1 across your sites, and with Helm (or Flux on top of it) the whole setup stays declarative and repeatable.</p><p>The recipe in one paragraph: label every node with its zone, give every Longhorn node a dedicated disk, set replicaZoneSoftAntiAffinity: false and a replica count of 3, and then <em>test the failure</em>. Pull the plug on a worker, watch the Pod come back in another datacenter, and read the heartbeat log. That ten-minute experiment tells you more than any architecture diagram.</p><p>Have you run Longhorn across multiple sites, and how did your inter-datacenter latency behave? Let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a97cd38cc320" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Flying Blind: Add Audit Logging and OpenSearch Observability to Your RKE2 Cluster]]></title>
            <link>https://kleinpapa.medium.com/stop-flying-blind-add-audit-logging-and-opensearch-observability-to-your-rke2-cluster-9a0bab498c36?source=rss-5bb8efb724bd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a0bab498c36</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[opensearch]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[kleinpapa]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 16:41:02 GMT</pubDate>
            <atom:updated>2026-06-28T16:41:31.461Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oWWsIFlE1ZA_ca6a3MZrw.jpeg" /></figure><p>If you followed my previous article on <a href="https://kleinpapa.medium.com/stop-using-a-single-kubernetes-control-plane-build-ha-with-kube-vip-and-rke2-a94e1f2934b6">building a highly available RKE2 cluster with Kube-VIP</a>, you now have a resilient control plane with a floating VIP and multiple master nodes. Your cluster survives a node failure. But here is the uncomfortable question: <strong>do you actually know what is happening inside it?</strong></p><p>Who created that pod last Tuesday? Which service account modified a ClusterRoleBinding at 2 AM? Was that ConfigMap change intentional, or did something unexpected touch it? Without audit logging, the answer to all of these is the same: you have no idea.</p><p>This article walks you through building a complete observability pipeline for Kubernetes API Server audit logs on RKE2. By the end, you will have:</p><ul><li>A structured <strong>audit policy</strong> that captures security-critical and compliance-relevant API events</li><li><strong>Fluent-bit</strong> running as a DaemonSet to read and ship those logs in real time</li><li>A single-node <strong>OpenSearch</strong> instance storing and indexing every event</li><li><strong>OpenSearch Dashboards</strong> exposed through an API Gateway so you can explore logs from your browser</li><li>An <strong>ISM retention policy</strong> that automatically removes logs older than 3 days</li></ul><h3>Architecture Overview</h3><p>Before we touch a single command, let’s understand what we’re building:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*djFlmkx1uLPx3gc90mqeIQ.png" /></figure><p>The actual control plane writes every audited API call to a log file on disk. Fluent-bit tails that file, parses the JSON, and pushes records into OpenSearch under daily indices (kube-audit-YYYY.MM.DD). Dashboards gives you a UI to search, filter, and visualize those events.</p><h3>Prerequisites</h3><ul><li>A running RKE2 cluster (single or HA — both work)</li><li>Helm v3 installed and kubectl configured with cluster-admin access</li><li>An <strong>API Gateway</strong> (e.g. <strong>Nginx Gateway Fabric)</strong> solution already running in your cluster (for exposing the Dashboards UI)</li><li>Enough resources on at least one node for OpenSearch: the configuration we use requests 1Gi RAM and allows up to 1.5Gi</li></ul><h3>Step 1: Enable RKE2 API Server Audit Logging</h3><p>Kubernetes audit logging is disabled by default. On RKE2, you enable it by passing arguments to the embedded kube-apiserver via the RKE2 config file. This step happens on the <strong>control plane node</strong>, before we deploy anything into the cluster.</p><h4>Create the Audit Policy</h4><p>The audit policy controls <em>what</em> gets logged and <em>at what verbosity</em>. This is the most important design decision in the whole setup — get it wrong and you either drown in noise or miss critical events.</p><p>Copy the following policy to /etc/rancher/rke2/audit-policy.yaml on every control plane node:</p><pre>apiVersion: audit.k8s.io/v1<br>kind: Policy<br>omitStages:<br>  - &quot;RequestReceived&quot;<br><br>rules:<br>  # --- SECURITY-CRITICAL ---<br><br>  # Secret changes: Metadata only (never log secret values)<br>  - level: Metadata<br>    verbs: [&quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources: [&quot;secrets&quot;]<br><br>  - level: Metadata<br>    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources: [&quot;secrets&quot;]<br><br>  # Full RBAC change logging<br>  - level: RequestResponse<br>    resources:<br>    - group: &quot;rbac.authorization.k8s.io&quot;<br>      resources:<br>      - &quot;clusterroles&quot;<br>      - &quot;clusterrolebindings&quot;<br>      - &quot;roles&quot;<br>      - &quot;rolebindings&quot;<br><br>  # Token and auth review operations<br>  - level: Metadata<br>    resources:<br>    - group: &quot;authentication.k8s.io&quot;<br>      resources: [&quot;tokenreviews&quot;]<br>    - group: &quot;authorization.k8s.io&quot;<br>      resources: [&quot;subjectaccessreviews&quot;]<br>    - group: &quot;&quot;<br>      resources: [&quot;serviceaccounts/token&quot;]<br><br>  # Pod and workload mutations<br>  - level: RequestResponse<br>    verbs: [&quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources: [&quot;pods&quot;]<br>    - group: &quot;apps&quot;<br>      resources: [&quot;deployments&quot;, &quot;daemonsets&quot;, &quot;statefulsets&quot;]<br><br>  # --- COMPLIANCE ---<br><br>  - level: Request<br>    verbs: [&quot;create&quot;, &quot;update&quot;, &quot;patch&quot;, &quot;delete&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources:<br>      - &quot;serviceaccounts&quot;<br>      - &quot;configmaps&quot;<br>      - &quot;namespaces&quot;<br>      - &quot;persistentvolumes&quot;<br>      - &quot;persistentvolumeclaims&quot;<br><br>  - level: Request<br>    resources:<br>    - group: &quot;admissionregistration.k8s.io&quot;<br>      resources:<br>      - &quot;mutatingwebhookconfigurations&quot;<br>      - &quot;validatingadmissionpolicies&quot;<br>      - &quot;validatingadmissionpolicybindings&quot;<br>      - &quot;validatingwebhookconfigurations&quot;<br>    - group: &quot;networking.k8s.io&quot;<br>      resources: [&quot;networkpolicies&quot;]<br><br>  # --- NOISE REDUCTION ---<br><br>  - level: None<br>    users: [&quot;system:kube-proxy&quot;]<br>    verbs: [&quot;watch&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources: [&quot;endpoints&quot;, &quot;services&quot;, &quot;services/status&quot;]<br><br>  - level: None<br>    userGroups: [&quot;system:nodes&quot;]<br>    verbs: [&quot;update&quot;, &quot;patch&quot;]<br>    resources:<br>    - group: &quot;&quot;<br>      resources: [&quot;nodes/status&quot;, &quot;pods/status&quot;]<br><br>  - level: None<br>    users:<br>    - &quot;system:kube-controller-manager&quot;<br>    - &quot;system:kube-scheduler&quot;<br>    verbs: [&quot;list&quot;, &quot;watch&quot;]<br><br>  - level: None<br>    userGroups: [&quot;system:authenticated&quot;]<br>    nonResourceURLs:<br>    - &quot;/api*&quot;<br>    - &quot;/version&quot;<br>    - &quot;/healthz&quot;<br>    - &quot;/readyz&quot;<br>    - &quot;/livez&quot;<br><br>  # --- DEFAULT CATCH-ALL ---<br><br>  - level: Metadata<br>    omitStages:<br>    - RequestReceived</pre><p><strong>A few design decisions worth explaining:</strong></p><ul><li>omitStages: [RequestReceived] — this stage fires the instant a request arrives, before any processing. Including it doubles your log volume with zero added value.</li><li>Secrets are logged at Metadata level, never Request or RequestResponse — this means you see <em>that</em> a secret was accessed, but never <em>what value it contained</em>. This is intentional.</li><li>RBAC changes are logged at RequestResponse — the full request and response body — because these are the highest-risk mutations in any cluster.</li><li>system:kube-proxy, system:nodes, system:kube-controller-manager, and system:kube-scheduler generate enormous amounts of routine chatter. The None rules suppress these entirely.</li></ul><h4>Configure <strong>kub</strong>e-apiserver Arguments</h4><p>Edit /etc/rancher/rke2/config.yaml on the control plane node and add:</p><pre>kube-apiserver-arg:<br>  - &quot;audit-policy-file=/etc/rancher/rke2/audit-policy.yaml&quot;<br>  - &quot;audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log&quot;<br>  - &quot;audit-log-maxage=30&quot;<br>  - &quot;audit-log-maxbackup=10&quot;<br>  - &quot;audit-log-maxsize=100&quot;<br>  - &quot;audit-log-format=json&quot;</pre><p>The audit-log-path value must match exactly what Fluent-bit will mount later — both point to /var/lib/rancher/rke2/server/logs/. The json format is mandatory for Fluent-bit&#39;s parser.</p><h4>Restart RKE2 Server</h4><pre>systemctl restart rke2-server</pre><p>After the restart, verify that the log file is being created and populated:</p><pre>tail -f /var/lib/rancher/rke2/server/logs/audit.log</pre><p>You should see a stream of JSON objects. If the file is empty or missing, double-check the path in config.yaml and confirm the policy file is in place.</p><h3>Step 2: Set Up Helm Repositories and Namespace</h3><pre>helm repo add opensearch https://opensearch-project.github.io/helm-charts/<br>helm repo add fluent https://fluent.github.io/helm-charts<br>helm repo update<br><br>kubectl create namespace opensearch</pre><h3>Step 3: Deploy OpenSearch Core</h3><p>We deploy a single-node OpenSearch instance. For a production environment you would run a 3-node cluster, but for an internal observability stack a single node with persistent storage is perfectly adequate.</p><p>Create values.yaml:</p><pre>singleNode: true<br>replicas: 1<br><br>opensearchJavaOpts: &quot;-Xmx512M -Xms512M&quot;<br><br>resources:<br>  requests:<br>    cpu: &quot;500m&quot;<br>    memory: &quot;1024Mi&quot;<br>  limits:<br>    cpu: &quot;1000m&quot;<br>    memory: &quot;1536Mi&quot;<br><br>extraEnvs:<br>  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD<br>    value: &quot;&lt;YOUR_STRONG_PASSWORD&gt;&quot;<br><br>persistence:<br>  enabled: true<br>  size: 10Gi</pre><p><strong>A few notes on this configuration:</strong></p><ul><li>opensearchJavaOpts caps the JVM heap at 512MB, keeping the total pod memory within the 1.5Gi limit.</li><li>persistence.size: 10Gi gives you enough room for several days of audit data before the ISM retention policy kicks in.</li><li>Set a strong password for OPENSEARCH_INITIAL_ADMIN_PASSWORD — this is the admin user credential used by both Dashboards and Fluent-bit.</li></ul><p><strong>Install it:</strong></p><pre>helm install opensearch-cluster opensearch/opensearch \<br>  -n opensearch \<br>  -f values.yaml</pre><blockquote><strong>Important:</strong> Wait until opensearch-cluster-master-0 is fully Running and its readiness probe passes before proceeding. OpenSearch takes 60–90 seconds to initialize on first boot.</blockquote><blockquote>Check with: <em>kubectl get pods -n opensearch -w</em></blockquote><h3>Step 4: Deploy OpenSearch Dashboards</h3><p>Create dashboard-values.yaml:</p><pre>opensearchHosts: &quot;https://opensearch-cluster-master:9200&quot;<br>replicaCount: 1<br><br>resources:<br>  requests:<br>    cpu: &quot;500m&quot;<br>    memory: &quot;512M&quot;<br>  limits:<br>    cpu: &quot;1000m&quot;<br>    memory: &quot;1024M&quot;<br><br>startupProbe:<br>  periodSeconds: 10<br>  failureThreshold: 40<br><br>config:<br>  opensearch_dashboards.yml: |<br>    server.name: dashboards<br>    server.host: &quot;0.0.0.0&quot;<br>    opensearch.hosts: [&quot;https://opensearch-cluster-master:9200&quot;]<br>    opensearch.ssl.verificationMode: none<br>    opensearch.username: &quot;admin&quot;<br>    opensearch.password: &quot;&lt;YOUR_STRONG_PASSWORD&gt;&quot;</pre><p>The ssl.verificationMode: none disables certificate validation for the internal cluster-to-cluster connection. Since OpenSearch ships with a self-signed certificate by default and Dashboards communicates inside the cluster, this is an acceptable trade-off for an internal stack. In a production environment, you would replace this with a properly signed certificate.</p><p>The startupProbe with failureThreshold: 40 gives Dashboards 400 seconds to start — generous but necessary, because Dashboards waits for OpenSearch to become fully available before it considers itself healthy.</p><pre>helm install opensearch-dashboards-test opensearch/opensearch-dashboards \<br>  -n opensearch \<br>  -f dashboard-values.yaml</pre><h3>Step 5: Expose the Dashboards UI</h3><p>We use API Gateway&#39;s HTTPRoute to route external traffic to the Dashboards service. Create httproute.yaml (replace the hostname with your own domain):</p><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: opensearch-dashboards-route<br>  namespace: opensearch<br>spec:<br>  parentRefs:<br>  - name: nginx-gateway<br>    namespace: nginx-gateway<br><br>  hostnames:<br>  - &quot;opensearch.your-domain.example&quot;<br><br>  rules:<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /<br>    backendRefs:<br>    - name: opensearch-dashboards-test<br>      namespace: opensearch<br>      port: 5601</pre><pre>kubectl apply -f httproute.yaml</pre><h3>Step 6: Deploy Fluent-bit</h3><p>This is where the pipeline comes together. Fluent-bit runs as a DaemonSet, mounts the RKE2 audit log directory from the host, tails the log file, and pushes parsed events into OpenSearch.</p><p>Create fluent-bit-values.yaml:</p><pre>tolerations:<br>  - operator: &quot;Exists&quot;<br><br>extraVolumes:<br>  - name: rke2-audit-logs<br>    hostPath: { path: /var/lib/rancher/rke2/server/logs/ }<br>  - name: var-log-host<br>    hostPath: { path: /var/log/ }<br><br>extraVolumeMounts:<br>  - name: rke2-audit-logs<br>    mountPath: /var/lib/rancher/rke2/server/logs/<br>    readOnly: true<br>  - name: var-log-host<br>    mountPath: /var/log/fluent-bit/<br>    readOnly: false<br><br>config:<br>  service: |<br>    [SERVICE]<br>        Flush         2<br>        Log_Level     info<br>        Daemon        off<br>        Parsers_File  parsers.conf<br>        HTTP_Server   On<br>        HTTP_Listen   0.0.0.0<br>        HTTP_Port     2020<br><br>  inputs: |<br>    [INPUT]<br>        Name tail<br>        Path /var/lib/rancher/rke2/server/logs/audit*.log<br>        Tag kube-audit<br>        Parser json<br>        DB /var/log/fluent-bit/flb_audit.db<br>        Mem_Buf_Limit 20MB<br><br>  outputs: |<br>    [OUTPUT]<br>        Name es<br>        Match kube-audit<br>        Host opensearch-cluster-master.opensearch.svc.cluster.local<br>        Port 9200<br>        Index kube-audit-%Y.%m.%d<br>        Suppress_Type_Name On<br>        tls On<br>        tls.verify Off<br>        http_User admin<br>        http_Passwd &lt;YOUR_STRONG_PASSWORD&gt;</pre><p><strong>Design decisions explained:</strong></p><ul><li>tolerations: [{operator: &quot;Exists&quot;}] — this tolerates <em>all</em> taints, which is necessary for Fluent-bit to schedule on control plane nodes (which are typically tainted to prevent regular workloads). Without this, the DaemonSet would not run where the audit log lives.</li><li>Two hostPath volumes: one mounts the audit log directory read-only, the other gives Fluent-bit a writable location for its SQLite offset database (flb_audit.db). This database tracks the last read position so Fluent-bit resumes correctly after a restart — without it, every restart would re-ship all logs from the beginning.</li><li>Flush: 2 — records are sent to OpenSearch every 2 seconds. Lower values reduce latency, higher values reduce API call frequency.</li><li>Index kube-audit-%Y.%m.%d — daily indices make retention management simple. The ISM policy we configure next can target kube-audit-* and delete whole index files rather than individual documents.</li><li>Suppress_Type_Name On — required for OpenSearch 2.x compatibility. Older Elasticsearch-compatible output plugins default to including a _type field which OpenSearch 2.x rejects.</li></ul><pre>helm install fluent-bit fluent/fluent-bit \<br>  -n opensearch \<br>  -f fluent-bit-values.yaml</pre><h3>Step 7: Configure Automatic Log Retention</h3><p>At this point your pipeline is running, but indices will accumulate indefinitely. We use OpenSearch’s Index State Management (ISM) to automatically delete audit indices older than 3 days.</p><p>Navigate to http://&lt;YOUR_DOMAIN&gt; → <strong>Management → Dev Tools</strong> and run the following two requests:</p><h4><strong>Create the retention policy</strong></h4><pre>PUT _plugins/_ism/policies/audit-retention-policy<br>{<br>  &quot;policy&quot;: {<br>    &quot;description&quot;: &quot;Delete audit logs older than 3 days&quot;,<br>    &quot;default_state&quot;: &quot;hot&quot;,<br>    &quot;states&quot;: [<br>      {<br>        &quot;name&quot;: &quot;hot&quot;,<br>        &quot;actions&quot;: [],<br>        &quot;transitions&quot;: [<br>          {<br>            &quot;state_name&quot;: &quot;delete&quot;,<br>            &quot;conditions&quot;: {<br>              &quot;min_index_age&quot;: &quot;3d&quot;<br>            }<br>          }<br>        ]<br>      },<br>      {<br>        &quot;name&quot;: &quot;delete&quot;,<br>        &quot;actions&quot;: [<br>          {<br>            &quot;retry&quot;: {<br>              &quot;count&quot;: 3,<br>              &quot;backoff&quot;: &quot;exponential&quot;,<br>              &quot;delay&quot;: &quot;1m&quot;<br>            },<br>            &quot;delete&quot;: {}<br>          }<br>        ],<br>        &quot;transitions&quot;: []<br>      }<br>    ],<br>    &quot;ism_template&quot;: [<br>      {<br>        &quot;index_patterns&quot;: [&quot;kube-audit-*&quot;],<br>        &quot;priority&quot;: 1<br>      }<br>    ]<br>  }<br>}</pre><p>The ism_template block binds the policy automatically to every new index matching kube-audit-*. This means you never have to manually associate policies to new daily indices.</p><h4>Apply the policy retroactively to existing indices</h4><pre>PUT kube-audit-*/_settings<br>{<br>  &quot;plugins.ism.policy_id&quot;: &quot;audit-retention-policy&quot;<br>}</pre><p>The template only applies to <em>future</em> indices. If Fluent-bit has already created indices for today, this second call attaches the policy to them manually.</p><h3>Step 8: Verify and Explore</h3><h4>Check that all pods are healthy</h4><pre>kubectl get pods -n opensearch</pre><p>You should see three pods in Running state: opensearch-cluster-master-0, opensearch-dashboards-test-*, and one fluent-bit-* pod per node.</p><h4>Configure the index pattern in Dashboards</h4><ol><li>Open http://&lt;YOUR_DOMAIN&gt; and log in with admin / &lt;YOUR_STRONG_PASSWORD&gt;</li><li>Navigate to <strong>Stack Management → Index Patterns</strong> (or <strong>Data Views</strong> in newer versions)</li><li>Create a new pattern matching kube-audit-*</li><li>Set the time field to @timestamp</li><li>Go to the <strong>Discover</strong> tab</li></ol><p>You should now see a stream of Kubernetes API Server audit events. Try filtering by verb: delete to see all deletion operations, or by user.username: system:serviceaccount:* to inspect service account activity.</p><h3>What You Have Built</h3><p>Let’s take a quick stock of the full pipeline and how each component contributes to the architecture:</p><ul><li><strong>audit-policy.yaml</strong> — Controls exactly what the API server logs and at what verbosity level, filtering out the noise.</li><li><strong>RKE2 </strong><strong>config.yaml</strong> — Instructs the kube-apiserver where to write the physical log files on the host and enforces the JSON format.</li><li><strong>OpenSearch (Single-Node)</strong> — The core database engine that securely stores and indexes all incoming Kubernetes audit events.</li><li><strong>OpenSearch Dashboards</strong> — The graphical Web UI used for querying, analyzing, and visualizing the data.</li><li><strong>Fluent Bit (DaemonSet)</strong> — The lightweight log shipper that continuously tails the local RKE2 audit files and forwards them to OpenSearch.</li><li><strong>HTTPRoute </strong>— Securely exposes the Dashboards interface to the outside world via the API Gateway.</li><li><strong>ISM Policy</strong> — An automated Index State Management rule that deletes data older than 3 days to prevent your storage from filling up.</li></ul><h3>What’s Next</h3><p>This setup gives you a working observability foundation. From here, natural next steps are:</p><ul><li><strong>Alerting</strong> — use OpenSearch Alerting to trigger notifications when sensitive operations occur (e.g., any write to clusterrolebindings)</li><li><strong>SIEM integration</strong> — the same pipeline can feed a broader security monitoring stack alongside Falco runtime events</li><li><strong>Multi-node OpenSearch</strong> — for production, migrate to a 3-node cluster with dedicated master and data roles</li><li><strong>TLS with proper certificates</strong> — replace the self-signed cert with one from your internal CA or cert-manager</li></ul><p>The cluster no longer flies blind. Every API call that matters now has a record.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a0bab498c36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using a Single Kubernetes Control Plane: Build HA with Kube-VIP and RKE2]]></title>
            <link>https://kleinpapa.medium.com/stop-using-a-single-kubernetes-control-plane-build-ha-with-kube-vip-and-rke2-a94e1f2934b6?source=rss-5bb8efb724bd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a94e1f2934b6</guid>
            <category><![CDATA[rancher]]></category>
            <category><![CDATA[kube-vip]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[baremetal]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[kleinpapa]]></dc:creator>
            <pubDate>Tue, 09 Jun 2026 17:42:47 GMT</pubDate>
            <atom:updated>2026-06-09T18:15:37.841Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*8axNguxIAkTPU-flgCMJpg.png" /></figure><p>When you build a <strong>Kubernetes (K8s)</strong> cluster inside public cloud environments like Amazon Web Services (AWS) or Google Cloud Platform (GCP), high availability is handed to you on a silver platter. You click a button, and the cloud provider provisions a <strong>Load Balancer</strong> that routes your traffic to the healthy servers in the background.</p><p>But what happens when you step out of the cloud?</p><p>If you are deploying Kubernetes on your own physical hardware (<strong>Bare-Metal</strong>), in an on-premise corporate data center, or inside a home laboratory (<strong>Homelab</strong>), you don’t have a cloud provider to hand you a load balancer. You have to build it yourself.</p><p>For a long time, setting up a highly available cluster on bare-metal was a painful process. It required managing external virtual machines, configuring complex networking tools like keepalived, and setting up proxy software like HAProxy.</p><p>Fortunately, there is a modern, elegant, and Kubernetes-native alternative: <strong>Kube-VIP</strong>.</p><p>In this guide, we will break down how to build a production-ready, highly available Kubernetes cluster from scratch using the <strong>RKE2 (</strong>Rancher Kubernetes Engine 2<strong>) </strong>as our Kubernetes distribution and <strong>Kube-VIP</strong> as our virtual load balancer.</p><h3>Jargon Buster: Understanding the Technologies</h3><p>Before we run a single command, let’s demystify the acronyms and technologies we are going to use:</p><ul><li><strong>Control Plane (The Brains):</strong> The collection of components that manage the Kubernetes cluster (such as the API server, scheduler, and controller manager). In a production environment, you need multiple Control Plane nodes so that if one physical server dies, the cluster keeps running.</li><li><strong>HA (High Availability):</strong> A system design approach that ensures a high level of operational performance and uptime. In our case, it means our cluster remains accessible even if a Control Plane node crashes.</li><li><strong>VIP (Virtual IP):</strong> An IP address that is not permanently assigned to a specific physical network interface. Instead, it “floats” between multiple servers. If the server currently holding the VIP goes down, the VIP instantly moves to another healthy server.</li><li><strong>RKE2 (Rancher Kubernetes Engine 2):</strong> Also known as <em>RKE Government</em>. It is a fully conformant Kubernetes distribution focused on security and compliance. It is lightweight, secure by default, and ideal for both enterprise and bare-metal deployments.</li><li><strong>API (Application Programming Interface) Server:</strong> The core component of the Kubernetes Control Plane. It exposes the gateway through which you, your worker nodes, and your tools (like kubectl) communicate with the cluster.</li><li><strong>SAN (Subject Alternative Name):</strong> An extension to the <strong>TLS (Transport Layer Security)</strong> certificate standard. It allows us to specify additional hostnames or IP addresses (like our Virtual IP) that should be protected and trusted by the same security certificate.</li><li><strong>CNI (Container Network Interface):</strong> A plugin architecture that enables networking inside Kubernetes. It is responsible for assigning IP addresses to individual containers (<strong>Pods</strong>) and allowing them to talk to each other across different physical machines.</li></ul><h3>The Architecture: Resolving the “Chicken-and-Egg” Problem</h3><p>To achieve true High Availability, we are going to use <strong>three separate Control Plane nodes</strong>.</p><p><strong>However, we face a classic “chicken-and-egg” dilemma during setup:</strong></p><ol><li>We want Kube-VIP to manage our Virtual IP (VIP) so we can access our cluster securely.</li><li>Kube-VIP needs to run as a component inside Kubernetes, but the Kubernetes cluster doesn’t exist yet!</li></ol><p><strong>To solve this, we use a brilliant bootstrapping strategy:</strong></p><ol><li>We configure and start RKE2 on the very first node locally.</li><li>We inject the Kube-VIP configuration files directly into RKE2’s automatic deployment directory.</li><li>As RKE2 starts up its local API server, it immediately fires up Kube-VIP. Kube-VIP claims the Virtual IP address on the network.</li><li>Once the Virtual IP is alive, we can easily join the second and third Control Plane nodes directly to that shared, highly available endpoint.</li></ol><h3>Network Planning For This Guide</h3><p>To make this tutorial easy to follow, we will use the following IP address layout. Make sure to replace these with your own network’s IP addresses:</p><ul><li><strong>10.10.0.100</strong> -&gt; Our <strong>Virtual IP (VIP)</strong>. This is the single, permanent address we will use to manage the cluster.</li><li><strong>10.10.0.10</strong> -&gt; Control Plane Node 1 (controlnode-1)</li><li><strong>10.10.0.11</strong> -&gt; Control Plane Node 2 (controlnode-2)</li><li><strong>10.10.0.12</strong> -&gt; Control Plane Node 3 (controlnode-3)</li></ul><h4>Step 1: Preparing the First Control Plane Node</h4><p>Log into your first server (controlnode-1) via <strong>SSH (Secure Shell)</strong>. Before installing any software, we need to create the specific directory paths where RKE2 expects to find its configuration and deployment manifests.</p><h4><strong>1.1. Create the Configuration Directories</strong></h4><p>Execute the following commands to create the necessary directories:</p><pre># This directory will hold our main RKE2 configuration file<br>mkdir -p /etc/rancher/rke2/<br><br># This directory is monitored by RKE2; any manifest placed here is automatically deployed<br>mkdir -p /var/lib/rancher/rke2/server/manifests/</pre><h4>1.2. Create the config.yaml File</h4><p>Now, we will create the main configuration file for RKE2. The most critical configuration here is the tls-san block.</p><p>When Kubernetes generates its security certificates, it only trusts the specific IP address of the local machine. Since we plan to access the cluster via our <strong>Virtual IP</strong> (10.10.0.100), we must explicitly tell Kubernetes to include this VIP (and the IPs of our future nodes) inside the security certificate&#39;s <strong>Subject Alternative Name (SAN)</strong> list. If we skip this, our kubectl commands will reject the connection due to a certificate mismatch error.</p><p>Create a new file at /etc/rancher/rke2/config.yaml and paste the following content:</p><pre># High Availability &amp; Virtual IP Configuration<br># We include all possible entry points so the TLS certificates accept them<br>tls-san:<br>  - 10.10.0.100  # Our shared Kube-VIP Floating IP<br>  - 10.10.0.10   # Node 1 IP<br>  - 10.10.0.11   # Node 2 IP<br>  - 10.10.0.12   # Node 3 IP<br><br># Network &amp; Security Settings<br>write-kubeconfig-mode: &quot;0600&quot;  # Restricts reading permissions of the cluster access file for security<br>cni:<br>  - canal                      # Specifies the Container Network Interface plugin to use</pre><h4>Step 2: Bootstrapping the RKE2 Service</h4><p>Now that our directories and configuration file are ready, we can actually install and run RKE2 on our first node (controlnode-1).</p><h4>2.1. Download and Run the RKE2 Installer</h4><p>Rancher provides a convenient, secure upstream script that detects your operating system and installs the correct binaries, systemd service files, and tools.</p><p>Run the following command:</p><pre>curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -</pre><ul><li><strong>INSTALL_RKE2_TYPE=server</strong>: This environment variable tells the installer that this node will be a <strong>Server</strong> node (which contains the Control Plane and the etcd distributed database), rather than an <strong>Agent</strong> node (which only runs worker workloads).</li></ul><h4>2.2. Start and Enable the RKE2 Server</h4><p>Because RKE2 integrates natively with the Linux init system (<strong>systemd</strong>), we manage it like any other system service using systemctl.</p><pre># Enable the service so it automatically starts if the physical server reboots<br>systemctl enable rke2-server<br><br># Start the service right now<br>systemctl start rke2-server</pre><blockquote><strong><em>Important Note on Patience:</em></strong><em> The </em><em>systemctl start command will feel like it is &quot;hanging&quot; and can take anywhere from </em><strong><em>3 to 6 minutes</em></strong><em>. Do not cancel it! In the background, RKE2 is downloading all the core Kubernetes container images (the API server, </em><em>etcd, controllers) into its internal runtime, initializing the database, and generating security certificates.</em></blockquote><h4>2.3. Set Up Your Environment to Use kubectl</h4><p><strong>kubectl (Kubernetes Control)</strong> is the standard command-line utility used to talk to a Kubernetes cluster. RKE2 automatically installs a pre-tested version of kubectl, but it is hidden deep in RKE2&#39;s binary directory. Also, kubectl needs to know <em>where</em> to find the cluster&#39;s secret access token (<strong>kubeconfig</strong>).</p><p>Let’s update your user environment so you can run kubectl commands globally:</p><pre># Temporarily add RKE2 binaries and the configuration path to your current terminal session<br>export PATH=$PATH:/var/lib/rancher/rke2/bin<br>export KUBECONFIG=/etc/rancher/rke2/rke2.yaml<br><br># Permanently append these paths to your user profile so they work every time you log in<br>echo &quot;export PATH=$PATH:/var/lib/rancher/rke2/bin&quot; &gt;&gt; $HOME/.bashrc<br>echo &quot;export KUBECONFIG=/etc/rancher/rke2/rke2.yaml&quot; &gt;&gt; $HOME/.bashrc</pre><p>Now, verify that your first node is online and healthy:</p><pre>kubectl get nodes</pre><p>You should see controlnode-1 listed in a Ready or NotReady (still initializing networking) state.</p><h4>2.4. Save the Cluster Join Token</h4><p>Before we touch Kube-VIP, we need to extract a secret token generated by RKE2. When we want Node 2 and Node 3 to join this cluster later, they will need this exact token to authenticate themselves.</p><p>Print and copy the token to a safe text file on your laptop:</p><pre>cat /var/lib/rancher/rke2/server/node-token</pre><h3>Step 3: Deploying Kube-VIP as a DaemonSet</h3><p>Our first node is up, but right now, it is an isolated island. It only knows about its own physical IP (10.10.0.10). It is time to deploy <strong>Kube-VIP</strong> to create our Virtual IP (10.10.0.100).</p><p>To do this, we will use a great feature of RKE2: the <strong>Auto-Deploy Manifest Directory</strong>. Any standard Kubernetes YAML file placed in /var/lib/rancher/rke2/server/manifests/ is automatically picked up and deployed by RKE2.</p><h4>3.1. Understand the Deployment Method: DaemonSet vs. Static Pod</h4><p>Kube-VIP can be deployed in two ways:</p><ol><li><strong>Static Pod: </strong>Managed directly by the local container runtime (independent of Kubernetes).</li><li><strong>DaemonSet:</strong> Managed by Kubernetes itself, ensuring exactly one copy of Kube-VIP runs on every single Control Plane node in the cluster.</li></ol><p>For RKE2, deploying as a <strong>DaemonSet</strong> via the auto-deploy folder is the cleanest approach. It allows Kubernetes to handle logging, health checks, and automatic failover out of the box.</p><h4>3.2. Define Your Network Variables</h4><p>To avoid typing typos in our configuration, let’s declare our target Virtual IP and the exact physical network interface Kube-VIP should hook into.</p><pre>export VIP=10.10.0.100<br>export INTERFACE=ens192</pre><p><strong>Tip:</strong> You can find your exact network interface name by running ip a. Look for the interface that holds your 10.10.0.10 IP address (it might be eth0, ens192, ens256, etc.).</p><h4>3.3. Apply the RBAC (Role-Based Access Control) Policy</h4><p>Kube-VIP is going to monitor the state of our cluster. If a Control Plane node dies, Kube-VIP needs to know about it immediately so the remaining nodes can vote on who should take over the Virtual IP. To do this monitoring, Kube-VIP needs specific system permissions within Kubernetes.</p><p>We download the official, pre-configured <strong>RBAC</strong> manifest directly into RKE2’s auto-deploy directory:</p><pre>curl -sfL https://kube-vip.io/manifests/rbac.yaml &gt; /var/lib/rancher/rke2/server/manifests/kube-vip-rbac.yaml</pre><h4>3.4. Generate the Kube-VIP DaemonSet Manifest</h4><p>Instead of writing a massive, complicated 100-line Kubernetes configuration by hand, we can use the Kube-VIP container itself to generate the perfect manifest for us based on our environment variables.</p><p>RKE2 uses an embedded container runtime called <strong>containerd</strong>. We will use its command-line tool, ctr (via crictl), to pull the Kube-VIP image and compile our manifest.</p><p>Run this command block on the server:</p><pre># 1. Instruct RKE2&#39;s container runtime to pull the latest Kube-VIP image<br>/var/lib/rancher/rke2/bin/crictl -r &quot;unix:///run/k3s/containerd/containerd.sock&quot; pull ghcr.io/kube-vip/kube-vip:latest<br><br># 2. Run the image locally just to generate the configuration file into RKE2&#39;s manifest folder<br>CONTAINERD_ADDRESS=/run/k3s/containerd/containerd.sock \<br>ctr -n k8s.io run \<br>--rm \<br>--net-host \<br>ghcr.io/kube-vip/kube-vip:latest vip /kube-vip manifest daemonset \<br>--arp \<br>--interface $INTERFACE \<br>--address $VIP \<br>--controlplane \<br>--leaderElection \<br>--taint \<br>--services \<br>--inCluster | tee /var/lib/rancher/rke2/server/manifests/kube-vip.yaml</pre><h4>What do these flags mean?</h4><ul><li><strong>--arp</strong>: Tells Kube-VIP to use Gratuitous ARP (Address Resolution Protocol) packets. This broadcast message announces to devices on the local network that the Virtual IP address (10.10.0.100) is now associated with a different MAC address. Network switches update their MAC address tables, and neighboring hosts refresh their ARP caches, allowing traffic destined for the VIP to be immediately redirected to the new active Control Plane node.</li><li><strong>--controlplane</strong>: Tells Kube-VIP to provide High Availability for the Kubernetes API server.</li><li><strong>--leaderElection</strong>: This is the brains. It forces the Kube-VIP instances across different nodes to constantly talk to each other and elect a &quot;Leader&quot;. Only the leader node actively holds the Virtual IP.</li><li><strong>--taint</strong>: Ensures Kube-VIP can run on Control Plane nodes, which are usually locked down (<strong>tainted</strong>) to prevent standard applications from running on them.</li></ul><h4>3.5. Verification: The Moment of Truth</h4><p>Within 30 to 60 seconds, RKE2 will notice the new file inside the /manifests/ folder and apply it. Kube-VIP will spin up, look at your network interface, and bind the 10.10.0.100 IP address to it.</p><p>Let’s check if the Kube-VIP pods are running successfully:</p><pre>kubectl rollout status daemonset kube-vip-ds -n kube-system --timeout=60s</pre><p>If it returns successfully, open up your command prompt on your personal laptop (or another machine on the same network) and try to ping the Virtual IP:</p><pre>ping 10.10.0.100</pre><p>If you see bytes flying back and forth, <strong>congratulations!</strong> Your cluster now has a highly available, floating gateway address completely running on bare-metal without any external hardware or VM dependencies.</p><h3>Step 4: Joining the Remaining Control Plane Nodes</h3><p>With our Virtual IP fully operational and managed by our first node, we can finally complete our High Availability triangle by joining controlnode-2 and controlnode-3.</p><p>Because we did the heavy lifting upfront inside the TLS certificates, these companion nodes will connect straight to our Virtual IP, pull the configuration, and automatically spin up their own Kube-VIP instances to join the backup pool.</p><h4>4.1. Prepare Node 2 and Node 3 Configuration</h4><p>Log into your second server (<strong>controlnode-2</strong>) via SSH. Just like before, create the configuration folder:</p><pre>mkdir -p /etc/rancher/rke2/</pre><p>Now, create the configuration file at /etc/rancher/rke2/config.yaml. However, <strong>the contents will be slightly different</strong> this time. Instead of initializing a new cluster, we need to point this node to our existing virtual gateway and provide our secret cluster token.</p><pre># Point this node to the highly available Virtual IP address<br>server: https://10.10.0.100:9345<br><br># Paste the exact secret token you extracted from Node 1 in Step 2.4<br>token: K101b6789fbc00...your...secret...node...token...here<br><br># We keep the identical TLS settings across the cluster<br>tls-san:<br>  - 10.10.0.100<br>  - 10.10.0.10<br>  - 10.10.0.11<br>  - 10.10.0.12<br><br>write-kubeconfig-mode: &quot;0600&quot;<br>cni:<br>  - canal</pre><p><em>(Note: You will repeat this exact same step on </em><em>controlnode-3 as well).</em></p><h4>4.2. Install and Start RKE2 on Node 2 and Node 3</h4><p>Run the installer and systemd manager on both remaining hosts:</p><pre># Download the binaries<br>curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -<br><br># Enable and fire up the engine<br>systemctl enable rke2-server<br>systemctl start rke2-server</pre><p>Once the systemd services finish starting up on all machines, head back to your terminal and run:</p><pre>kubectl get nodes -o wide</pre><h4>4.3. The Final Verdict: Verifying the Multi-Node Cluster</h4><p>Once RKE2 finishes initializing on controlnode-2 and controlnode-3, it is time to check if our Kube-VIP DaemonSet has automatically scaled across the entire infrastructure.</p><p>Run a targeted query to look at only the Kube-VIP components inside the system namespace:</p><pre>kubectl get pods -n kube-system | grep kube-vip</pre><p>If your High Availability architecture is fully operational, you will see exactly <strong>three healthy pods</strong> running, matching the number of your Control Plane nodes:</p><pre>kube-vip-ds-4vmdm                      1/1                    Running<br>kube-vip-ds-rr8h2                      1/1                    Running<br>kube-vip-ds-sw4gh                      1/1                    Running</pre><p>Every single control plane server now has its own local Kube-VIP guardian. But how do they cooperate? To see the magic in action, let’s look under the hood.</p><h4>4.4. Peeking Under the Hood: Analyzing Kube-VIP Logs and Leader Election</h4><p>To see exactly how these three separate pods negotiate who should hold the Virtual IP, we can inspect the internal application logs of one of the instances.</p><p>Run the following command to see the real-time operations (in this case, looking at the kube-vip-ds-sw4gh pod):</p><pre>kubectl logs kube-vip-ds-sw4gh -n kube-system</pre><p>When the cluster initializes and the pods start communicating, you will see an output very similar to this:</p><pre>2026/06/09 01:48:19 INFO kube-vip.io version=v1.0.4 build=ff40ff06a3d6ee2d4452968e46cfa7e5ac4cc456<br>2026/06/09 01:48:19 INFO starting namespace=kube-system Mode=ARP &quot;Control Plane&quot;=true Services=true<br>2026/06/09 01:48:19 INFO using node name name=controlnode-1<br>2026/06/09 01:48:19 INFO prometheus HTTP server started<br>2026/06/09 01:48:19 INFO Starting Kube-vip Manager with the ARP engine<br>2026/06/09 01:48:19 INFO Start ARP/NDP advertisement<br>2026/06/09 01:48:19 INFO [ARP manager] starting ARP/NDP advertisement<br>2026/06/09 01:48:19 INFO beginning services leadership namespace=kube-system &quot;lock name&quot;=plndr-svcs-lock id=controlnode-1<br>2026/06/09 01:48:19 INFO cluster membership namespace=kube-system lock=plndr-cp-lock id=controlnode-1<br>I0609 01:48:19.800910       1 leaderelection.go:258] &quot;Attempting to acquire leader lease...&quot; lock=&quot;kube-system/plndr-svcs-lock&quot;<br>I0609 01:48:19.801207       1 leaderelection.go:258] &quot;Attempting to acquire leader lease...&quot; lock=&quot;kube-system/plndr-cp-lock&quot;<br>I0609 01:48:19.813051       1 leaderelection.go:272] &quot;Successfully acquired lease&quot; lock=&quot;kube-system/plndr-cp-lock&quot;<br>2026/06/09 01:48:19 INFO (svcs) starting services watcher for all namespaces</pre><h4>Decoding the Logs: What Is Actually Happening?</h4><p>Let’s break down the critical milestones inside these logs so you know exactly what Kube-VIP is negotiating in the background:</p><ul><li><strong>Mode=ARP &quot;Control Plane&quot;=true Services=true</strong>: This confirms our environment variables were processed correctly. Kube-VIP is officially running in <strong>ARP</strong> layer-2 mode, and it is actively watching both the <strong>Control Plane</strong> (the API server gateway) and standard Kubernetes <strong>Services</strong> (for application-level load balancing).</li><li><strong>Start ARP/NDP advertisement</strong>: Kube-VIP has initialized its network broadcasting engine. It is now ready to announce our Virtual IP to the local physical switch.</li><li><strong>Attempting to acquire leader lease...</strong>: Since we now have a multi-node setup, Kube-VIP instances across different nodes use Kubernetes native <strong>Leases</strong> (plndr-cp-lock and plndr-svcs-lock). This is a specialized mechanism where nodes compete to decide who gets to hold the lock.</li><li><strong>Successfully acquired lease</strong>: This is the ultimate green light. controlnode-1 won the election and successfully claimed the lock, which means this specific machine is now actively hosting the 10.10.0.100 Virtual IP address.</li></ul><p>Right now, one pod actively routes the VIP. If that specific physical server loses power or crashes, the remaining two pods will instantly detect that the lease has expired, renegotiate the lock within seconds, and grab the Virtual IP without dropping a single command from your kubectl or CI/CD deployment pipelines.</p><h3>Conclusion</h3><p>Setting up high availability on bare-metal or on-premise environments doesn’t mean you have to deal with complex legacy networking tools like external HAProxy or keepalived servers.</p><p>By combining RKE2’s <strong>secure-by-default, automated manifest deployment</strong> with <strong>Kube-VIP’s native virtual routing</strong>, you can build a resilient, enterprise-grade Kubernetes control plane that mimics public cloud behavior directly on your own hardware.</p><p>You now have a solid, highly available foundation ready to host your production workloads, local storage solutions, or automated deployment pipelines.</p><p><em>Have you tried setting up Kube-VIP in your bare-metal environment? Let me know your thoughts or any networking hurdles you faced in the comments below!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a94e1f2934b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>