<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jean-François PUCHEU on Medium]]></title>
        <description><![CDATA[Stories by Jean-François PUCHEU on Medium]]></description>
        <link>https://medium.com/@jfpucheu?source=rss-ca7365ab90b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kjwBIrINPXLjXKLC9polrQ.png</url>
            <title>Stories by Jean-François PUCHEU on Medium</title>
            <link>https://medium.com/@jfpucheu?source=rss-ca7365ab90b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:05:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jfpucheu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Votre carte réseau habite chez le voisin (et ça ralentit tout)]]></title>
            <link>https://medium.com/@jfpucheu/votre-carte-r%C3%A9seau-habite-chez-le-voisin-et-%C3%A7a-ralentit-tout-9d26193dfa0a?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d26193dfa0a</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[cpu]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[processors]]></category>
            <category><![CDATA[performance-tuning]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 10:14:10 GMT</pubDate>
            <atom:updated>2026-08-02T10:14:10.789Z</atom:updated>
            <content:encoded><![CDATA[<p>Vous avez un serveur bisocket flambant neuf, un lien 25 ou 100 Gbit/s, et pourtant les débits mesurés sont en dessous de ce que vous attendiez. La carte réseau va bien, le câble va bien, le switch va bien. Le vrai problème, c’est que votre CPU et votre carte réseau ne vivent peut-être pas sous le même toit : câblée sur l’autre socket, la carte est un peu comme un colocataire installé chez le voisin — chaque échange demande de traverser le palier, et ça se paie en latence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TJQAybNIQ1C55DGoFR6dA.png" /></figure><h3>Un serveur “bisocket” n’est pas un seul gros cerveau. Ce sont deux processeurs séparés, chacun avec :</h3><ul><li>ses propres cœurs de calcul,</li><li>sa propre mémoire RAM directement raccordée,</li><li>ses propres lignes PCIe (le bus qui alimente les cartes réseau, les cartes de stockage, les GPU…).</li></ul><p>Les deux processeurs se parlent via un lien dédié : QPI ou UPI chez Intel, Infinity Fabric chez AMD. C’est rapide, mais ce n’est pas gratuit : chaque aller-retour par ce lien coûte du temps et de la bande passante.</p><p>C’est le principe <strong>NUMA</strong> (<em>Non-Uniform Memory Access</em>) : selon l’endroit où se trouve la donnée, y accéder est rapide (mémoire locale) ou plus lent (mémoire de l’autre socket).</p><p>À proprement parler, NUMA est un problème de mémoire : un cœur du CPU 0 accède vite à sa propre RAM locale, mais chaque accès à la RAM du CPU 1 doit traverser ce lien inter-socket, et une application mal placée par rapport à sa mémoire paie ce coût à chaque accès. Le sujet des cartes réseau traité dans cet article est un problème connexe mais distinct — ce n’est pas l’accès mémoire qui pose problème, mais la localité sur le bus PCIe.</p><p>Les deux partagent la même cause racine, ce lien inter-socket coûteux, ce qui explique qu’on les regroupe souvent sous l’étiquette “NUMA” alors que le terme ne désigne à l’origine que la mémoire. En pratique, les deux se cumulent souvent : un processus mal placé par rapport à sa carte réseau l’est fréquemment aussi par rapport à sa mémoire.</p><h3>Le schéma qui change tout</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqG60EbcuhG5k4pKe-Df0w.png" /></figure><p><em>Une carte réseau se branche sur le bus PCIe, et sur une carte mère bisocket, chaque slot PCIe est câblé physiquement vers l’un ou l’autre des deux processeurs — jamais vers les deux à la fois.</em></p><p><em>Si une application tourne sur des cœurs du CPU 1 mais utilise la </em><strong><em>carte A</em></strong><em>, câblée sur le CPU 0, chaque paquet reçu doit traverser le lien inter-socket avant d’être traité. Latence ajoutée, bande passante d’interconnexion consommée, performance dégradée — surtout visible à fort débit ou fort taux de paquets par seconde.</em></p><h3>Comment identifier le problème</h3><h4>Vérifier le nœud NUMA d’une carte réseau</h4><pre># Topologie NUMA globale<br>numactl --hardware<br>lstopo</pre><pre># Nœud NUMA d&#39;une interface réseau<br>cat /sys/class/net/eth0/device/numa_node</pre><p>Si cette dernière commande renvoie 0, la carte est câblée sur le CPU 0. Si l&#39;application tourne sur des cœurs du CPU 1, le problème est identifié.</p><h4>Vérifier combien de CPU et quels CPU sont réellement utilisés par un processus</h4><p>Connaître le nœud NUMA d’une carte réseau ne suffit pas : il faut aussi savoir sur quels cœurs tourne réellement le processus qui traite le trafic.</p><pre># Liste des CPU autorisés pour le processus haproxy<br>grep Cpus_allowed_list /proc/$(pgrep -o haproxy)/status</pre><p>Cette commande affiche la liste des cœurs sur lesquels le processus HAProxy est autorisé à s’exécuter (son <em>affinity mask</em>). Si cette liste couvre les deux sockets, le scheduler Linux peut librement déplacer le processus d’un nœud NUMA à l’autre — y compris vers un cœur éloigné de la carte réseau utilisée.</p><pre># Nombre de files (queues) configurées sur la carte réseau<br>ethtool -l enp1s0</pre><p>Cette commande montre le nombre de canaux (RX/TX queues) configurés sur l’interface enp1s0. Chaque file peut être traitée par une interruption dédiée, elle-même assignée à un cœur précis : c&#39;est ce qui permet de répartir la charge réseau sur plusieurs cœurs du bon nœud NUMA plutôt que sur un seul.</p><pre># Affinité CPU d&#39;un processus déjà lancé (via son PID)<br>taskset -cp &lt;PID&gt;</pre><p>taskset permet de consulter (et, avec les bons arguments, de modifier) l&#39;affinité CPU d&#39;un processus en cours d&#39;exécution. Combinée à la commande précédente sur Cpus_allowed_list, elle permet de vérifier — ou de forcer — qu&#39;un processus reste cantonné aux cœurs du nœud NUMA où se trouve sa carte réseau.</p><p>📎 <a href="https://devops-notes.com/other/img/mindmap_CPU.png">Mindmap des concepts NUMA, carte réseau et affinité CPU</a></p><h3>Multiqueue : la carte réseau aussi peut faire du multitâche</h3><p>Par défaut, sur de nombreuses distributions Linux, le multiqueue n’est pas forcément actif ou correctement dimensionné à l’installation. Il est donc important de le vérifier et, le cas échéant, de l’activer ou de l’ajuster — surtout sur des interfaces rapides (10 Gbit/s et au-delà) où un seul cœur peut rapidement devenir le goulot d’étranglement.</p><p>Sans multiqueue, toutes les interruptions réseau sont traitées par un seul cœur : peu importe le nombre de cœurs disponibles sur le bon nœud NUMA, un seul porte toute la charge. C’est particulièrement pénalisant sur des workloads à fort taux de nouvelles connexions ou de paquets par seconde, comme un load balancer.</p><p>On active ou ajuste le nombre de files avec :</p><pre># Passer à 4 files combinées RX/TX<br>ethtool -L enp1s0 combined 4</pre><p>La valeur choisie doit rester inférieure ou égale au nombre de cœurs disponibles sur le nœud NUMA local de la carte : au-delà, des files seraient assignées à des cœurs du mauvais socket, ce qui annulerait une partie du bénéfice. Pour que ce réglage survive à un redémarrage, il faut le persister via la configuration réseau du système (/etc/network/interfaces, udev, systemd-networkd ou un script post-up selon la distribution).</p><p>Une carte réseau moderne ne pousse pas tout son trafic vers un seul cœur : elle répartit les paquets entrants et sortants dans plusieurs <strong>files d’attente matérielles</strong> (queues), gérées directement par la carte. C’est ce qu’on appelle le <strong>multiqueue</strong>.</p><p>La sortie de ethtool -l enp1s0 ressemble à ceci :</p><pre>Channel parameters for enp1s0:<br>Pre-set maximums:<br>RX:             0<br>TX:             0<br>Other:          1<br>Combined:       8<br>Current hardware settings:<br>RX:             0<br>TX:             0<br>Other:          1<br>Combined:       4</pre><ul><li><strong>Pre-set maximums</strong> : le nombre maximal de files que la carte peut techniquement gérer (ici 8 canaux combinés au maximum).</li><li><strong>Current hardware settings</strong> : le nombre de files réellement actives (ici 4 canaux combinés).</li><li><strong>Combined</strong> : un canal combiné gère à la fois une file de réception (RX) et une file d’émission (TX). Certaines cartes séparent au contraire les files RX et TX en compteurs distincts, qui apparaissent alors sur les lignes RX et TX plutôt que sur Combined.</li></ul><p>Le mécanisme qui décide dans quelle file atterrit chaque paquet s’appelle le <strong>RSS</strong> (<em>Receive Side Scaling</em>) : la carte calcule un hachage à partir de certains champs de l’en-tête du paquet — en général les adresses IP et ports source/destination — et ce hachage détermine la file de destination. Les champs exacts pris en compte sont configurables et dépendent de la carte et de son driver. Résultat : les paquets d’une même connexion restent sur la même file (donc le même cœur), tandis que des connexions différentes peuvent être traitées en parallèle sur des cœurs différents.</p><p>Chaque file est associée à sa propre interruption (IRQ). C’est cette IRQ qu’on peut ensuite épingler à un cœur précis — idéalement un cœur du même nœud NUMA que la carte réseau — via /proc/irq/&lt;numéro&gt;/smp_affinity ou irqbalance.</p><p>Sans configuration explicite, rien ne garantit que ces IRQ (et donc le traitement des paquets) restent sur le bon nœud NUMA — c’est là que le lien avec la section précédente se referme : carte réseau, files, interruptions et affinité du processus doivent tous pointer vers le même socket pour éviter la traversée inutile du lien inter-CPU.</p><h3>Les leviers pour limiter l’impact</h3><ul><li><strong>Affinité des IRQ (interruptions)</strong> : configurer les interruptions de la carte réseau pour qu’elles soient traitées par des cœurs du bon nœud NUMA (via irqbalance ou une configuration manuelle des affinités CPU).</li><li><strong>Épinglage des processus (</strong><strong>numactl, </strong><strong>taskset)</strong> : lancer les applications sensibles au réseau sur les cœurs du même nœud NUMA que la carte réseau utilisée.</li><li><strong>Placement physique</strong> : lors du choix des slots PCIe pour les cartes réseau, privilégier ceux rattachés au socket qui héberge les workloads réseau critiques.</li><li><strong>Virtualisation</strong> : dans un contexte de VM ou de conteneurs (Kubernetes, KVM…), s’assurer que le scheduler tient compte de la topologie NUMA (CPU pinning, NUMA-aware scheduling).</li></ul><h3>Mono-socket ou bisocket ? Un cas concret : HAProxy</h3><p>Cette question de topologie n’est pas qu’académique : elle influence directement le choix du matériel pour des rôles réseau-intensifs, comme un load balancer type HAProxy.</p><h4>Ce qu’HAProxy sait faire tout seul</h4><p>Bonne nouvelle : HAProxy n’est pas totalement aveugle à ce sujet, il propose deux réglages directement liés à NUMA et à l’affinité CPU.</p><ul><li><strong>nbthread &lt;nombre&gt;</strong> définit le nombre de threads sur lesquels HAProxy va s&#39;exécuter. Sur les plateformes qui le permettent, si aucune valeur n&#39;est précisée, HAProxy détecte automatiquement le nombre de cœurs auxquels le processus est lié au démarrage et règle nbthread en conséquence — ce qui veut dire qu&#39;on peut ajuster ce nombre de threads simplement en contraignant le processus avec taskset ou cpuset avant de le lancer, sans toucher à la configuration. Sans détection ni réglage explicite, la valeur par défaut retombe à 1. La valeur effectivement retenue est visible dans la sortie de haproxy -vv.</li><li><strong>numa-cpu-mapping</strong> va plus loin : sur une machine NUMA, cette directive permet à HAProxy d&#39;inspecter lui-même la topologie matérielle et de choisir le meilleur jeu de cœurs à utiliser, ainsi que le nombre de threads correspondant. Ce comportement automatique s&#39;efface dès qu&#39;on définit soi-même nbthread, qu&#39;on fixe l&#39;affinité du processus autrement (via cpu-map, taskset...), ou qu&#39;on désactive explicitement la fonctionnalité avec no numa-cpu-mapping si le placement automatique s&#39;avère sous-optimal sur une architecture donnée.</li></ul><p>Concrètement, sur un bisocket, numa-cpu-mapping peut faire une bonne partie du travail décrit plus haut à la place de l&#39;administrateur : détecter le nœud NUMA pertinent et y cantonner les threads HAProxy. Mais cela reste une automatisation logicielle d&#39;un problème matériel — elle ne dispense pas de vérifier que la carte réseau elle-même, ses files et ses IRQ sont alignées sur ce même nœud NUMA.</p><p>La documentation officielle HAProxy va dans le même sens sur le fond : sur une machine bisocket, elle déconseille de répartir le traitement entre les deux sockets physiques et recommande plutôt d’utiliser tous les cœurs d’un seul CPU physique, tant le coût de communication entre les deux CPU est élevé. Si malgré tout une charge SSL/TLS lourde justifie d’exploiter les deux sockets, la même documentation suggère d’assigner une carte réseau dédiée à chaque socket, afin que chaque groupe de threads ne dialogue qu’avec la carte réseau de son propre CPU.</p><h3>Mono-socket vs bisocket</h3><h4><strong>En faveur du mono-socket :</strong></h4><ul><li>HAProxy est avant tout un consommateur de <strong>réseau et de quelques cœurs</strong>, rarement gourmand en RAM ou en calcul massif. Un CPU récent à fréquence élevée, même avec peu de cœurs, couvre largement le besoin.</li><li>En mono-socket, <strong>il n’existe qu’un seul nœud NUMA</strong> : toutes les cartes réseau, toute la RAM, tous les cœurs sont “locaux” les uns aux autres. Le problème décrit plus haut disparaît structurellement, sans configuration particulière à maintenir.</li><li>Moins de complexité opérationnelle : pas d’affinité IRQ à soigner en fonction du socket, pas de risque qu’un redéploiement replace le processus sur le “mauvais” nœud NUMA.</li></ul><h4><strong>En faveur du bisocket, même pour ce type de rôle :</strong></h4><ul><li>Si la machine mutualise plusieurs rôles (reverse proxy + terminaison TLS lourde + observabilité + autres services), le nombre de cœurs et la bande passante mémoire totale d’un bisocket peuvent redevenir nécessaires.</li><li>Bien configuré (affinité IRQ, numactl, choix du bon slot PCIe), l&#39;impact NUMA peut être largement maîtrisé : c&#39;est une question de rigueur de configuration, pas une fatalité.</li><li>Pour des débits très élevés avec plusieurs cartes réseau utilisées en parallèle, un bisocket bien pensé peut même offrir plus de bande passante PCIe agrégée qu’un mono-socket.</li></ul><p>En pratique, pour une flotte de serveurs dédiés à un rôle réseau-intensif mais peu gourmand en calcul comme HAProxy, le mono-socket est souvent le choix le plus simple et le plus robuste : il élimine le problème NUMA à la source plutôt que de demander à le gérer. Le bisocket garde son intérêt dès que la machine doit aussi porter une charge de calcul ou de mémoire significative.</p><h3>À retenir</h3><p>Dans la majorité des cas, une carte réseau est câblée vers un seul socket. Sur un serveur bisocket, ignorer ce détail peut coûter cher en latence et en débit dès que la charge réseau devient significative. Le mono-socket supprime le problème par construction ; le bisocket demande de la rigueur mais reste justifié dès que d’autres ressources (cœurs, mémoire) entrent en jeu.</p><p>Sur un bisocket, avant de forcer manuellement nbthread ou l&#39;affinité CPU pour exploiter tous les cœurs des deux sockets, mieux vaut benchmarker : forcer plus de threads peut sembler logique sur le papier, mais si cela pousse HAProxy à faire tourner des threads sur les deux sockets, le coût de communication inter-CPU peut faire chuter les performances au lieu de les améliorer. Les réglages automatiques d&#39;HAProxy (numa-cpu-mapping, cpu-policy) donnent généralement un point de départ plus sûr ; toute configuration manuelle mérite d&#39;être validée par des tests de charge réels avant mise en production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d26193dfa0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your NIC Lives Next Door (And It’s Slowing Everything Down)]]></title>
            <link>https://medium.com/@jfpucheu/your-nic-lives-next-door-and-its-slowing-everything-down-e4f2309c91e1?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4f2309c91e1</guid>
            <category><![CDATA[processors]]></category>
            <category><![CDATA[performance-tunning]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[cpu]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 09:36:07 GMT</pubDate>
            <atom:updated>2026-08-02T10:17:16.011Z</atom:updated>
            <content:encoded><![CDATA[<p>Y<em>ou’ve got a brand new dual-socket server, a shiny 25 or 100 Gbit/s link, and yet measured throughput is below expectations. The NIC is fine, the cable is fine, the switch is fine. The real problem is that your CPU and your network card might not be living under the same roof: plugged into the wrong socket, the NIC is a bit like a roommate who moved in next door — every exchange means crossing the hallway, and that costs latency.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TJQAybNIQ1C55DGoFR6dA.png" /></figure><p><a href="https://medium.com/@jfpucheu/votre-carte-r%C3%A9seau-habite-chez-le-voisin-et-%C3%A7a-ralentit-tout-9d26193dfa0a">French Version</a></p><h3>A Dual-Socket Server Is Two Computers Holding Hands</h3><p>A “dual-socket” server is not one big brain. It’s two separate processors, each with:</p><ul><li>its own compute cores,</li><li>its own directly attached RAM,</li><li>its own PCIe lanes (the bus that feeds network cards, storage controllers, GPUs…).</li></ul><p>The two processors talk to each other through a dedicated interconnect: QPI or UPI on Intel, Infinity Fabric on AMD. It’s fast, but it’s not free: every round trip across that link costs time and bandwidth.</p><p>This is the <strong>NUMA</strong> principle (<em>Non-Uniform Memory Access</em>): depending on where a piece of data lives, accessing it is either fast (local memory) or slower (memory on the other socket).</p><p>Strictly speaking, NUMA is a memory problem: a core on CPU 0 reaches its own local RAM quickly, but every access to CPU 1’s RAM has to cross that interconnect, and an application poorly placed relative to its memory pays that cost on every single access. The network card story told in this article is a related but distinct problem — it’s not about memory access, but about locality on the PCIe bus. Both share the same root cause, the costly inter-socket link, which is why people commonly lump them together under the “NUMA” label even though the term strictly refers to memory. In practice the two frequently compound: a process badly placed relative to its NIC is often also badly placed relative to its memory.</p><h3>The Diagram That Changes Everything</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqG60EbcuhG5k4pKe-Df0w.png" /></figure><p>A network card plugs into the PCIe bus, and on a dual-socket motherboard, each PCIe slot is physically wired to one processor only — never both at once.</p><p>If an application runs on cores belonging to CPU 1 but uses <strong>NIC A</strong>, which is wired to CPU 0, every incoming packet must cross the inter-socket link before it can be processed. Added latency, interconnect bandwidth consumed, degraded performance — especially noticeable at high throughput or high packet rates.</p><h3>How to Identify the Problem</h3><h4>Check the NUMA Node of a Network Card</h4><pre># Global NUMA topology<br>numactl --hardware<br>lstopo<br><br># NUMA node of a network interface<br>cat /sys/class/net/eth0/device/numa_node</pre><p>If that last command returns 0, the card is wired to CPU 0. If the application runs on cores belonging to CPU 1, you&#39;ve found your problem.</p><h4>Check How Many CPUs — and Which Ones — a Process Is Actually Using</h4><p>Knowing the NUMA node of a NIC is not enough: you also need to know which cores the process handling the traffic is actually running on.</p><pre># List of allowed CPUs for the haproxy process<br>grep Cpus_allowed_list /proc/$(pgrep -o haproxy)/status</pre><p>This command shows the list of cores on which the HAProxy process is allowed to run (its <em>affinity mask</em>). If that list spans both sockets, the Linux scheduler can freely move the process from one NUMA node to the other — including onto a core far from the NIC being used.</p><pre># Number of queues configured on the NIC<br>ethtool -l enp1s0</pre><p>This command shows the number of channels (RX/TX queues) configured on the enp1s0 interface. Each queue can be handled by a dedicated interrupt, itself pinned to a specific core — this is what allows network load to be spread across multiple cores on the right NUMA node rather than piling onto just one.</p><pre># CPU affinity of a running process (by PID)<br>taskset -cp &lt;PID&gt;</pre><p>taskset lets you inspect (and, with the right arguments, modify) the CPU affinity of a running process. Combined with Cpus_allowed_list above, it lets you verify — or enforce — that a process stays confined to the cores of the NUMA node where its NIC lives.</p><p>📎 <a href="https://devops-notes.com/other/img/mindmap_CPU.png">Mindmap of NUMA, NIC and CPU affinity concepts</a></p><h4>Multiqueue: Your NIC Can Multitask Too</h4><p>On many Linux distributions, multiqueue is not necessarily active or properly sized out of the box. It’s worth checking and, if needed, enabling or tuning it — especially on fast interfaces (10 Gbit/s and above) where a single core can quickly become the bottleneck.</p><p>Without multiqueue, all network interrupts are handled by a single core: no matter how many cores are available on the right NUMA node, only one carries the entire load. This is particularly damaging on workloads with high connection rates or high packet-per-second counts, such as a load balancer.</p><p>You can enable or adjust the number of queues with:</p><pre># Set 4 combined RX/TX queues<br>ethtool -L enp1s0 combined 4</pre><p>The value you choose should not exceed the number of cores available on the NIC’s local NUMA node: beyond that, queues would be assigned to cores on the wrong socket, negating much of the benefit. To make this setting survive a reboot, persist it through the system’s network configuration (/etc/network/interfaces, udev, systemd-networkd, or a post-up script depending on the distribution).</p><p>A modern NIC does not push all its traffic through a single core: it distributes incoming and outgoing packets across multiple <strong>hardware queues</strong>, managed directly by the card itself. This is what <strong>multiqueue</strong> means.</p><p>The output of ethtool -l enp1s0 looks like this:</p><pre>Channel parameters for enp1s0:<br>Pre-set maximums:<br>RX:             0<br>TX:             0<br>Other:          1<br>Combined:       8<br>Current hardware settings:<br>RX:             0<br>TX:             0<br>Other:          1<br>Combined:       4</pre><ul><li><strong>Pre-set maximums</strong>: the maximum number of queues the card can technically handle (here, up to 8 combined channels).</li><li><strong>Current hardware settings</strong>: the number of queues currently active (here, 4 combined channels).</li><li><strong>Combined</strong>: a combined channel handles both one RX queue and one TX queue. Some cards separate RX and TX into distinct counters, which then appear on the RX and TX lines instead of Combined.</li></ul><p>The mechanism that decides which queue each packet lands in is called <strong>RSS</strong> (<em>Receive Side Scaling</em>): the card computes a hash from certain fields in the packet header — typically source/destination IP addresses and ports — and that hash determines the target queue. The exact fields used are configurable and depend on the card and its driver. Result: packets from the same connection always land on the same queue (and therefore the same core), while different connections can be processed in parallel on different cores.</p><p>Each queue has its own dedicated interrupt (IRQ). That IRQ can then be pinned to a specific core — ideally a core on the same NUMA node as the NIC — via /proc/irq/&lt;number&gt;/smp_affinity or irqbalance.</p><p>Without explicit configuration, nothing guarantees that these IRQs — and therefore packet processing — stay on the right NUMA node. This is where everything comes full circle: NIC, queues, interrupts, and process affinity must all point to the same socket to avoid unnecessary trips across the inter-CPU link.</p><h4>Levers to Reduce the Impact</h4><ul><li><strong>IRQ affinity</strong>: configure network card interrupts to be handled by cores on the correct NUMA node (via irqbalance or manual IRQ affinity configuration).</li><li><strong>Process pinning (</strong><strong>numactl, </strong><strong>taskset)</strong>: launch network-sensitive applications on the cores of the same NUMA node as the NIC they use.</li><li><strong>Physical placement</strong>: when choosing PCIe slots for network cards, favour slots wired to the socket that hosts the critical network workloads.</li><li><strong>Virtualisation</strong>: in VM or container environments (Kubernetes, KVM…), ensure the scheduler is NUMA-aware (CPU pinning, NUMA-aware scheduling).</li></ul><h3>Single-Socket or Dual-Socket? A Concrete Case: HAProxy</h3><p>This topology question is not purely academic: it directly influences hardware choices for network-intensive roles such as a load balancer running HAProxy.</p><h4>What HAProxy Can Handle on Its Own</h4><p>Good news: HAProxy is not completely blind to this. It provides two settings directly tied to NUMA and CPU affinity.</p><ul><li><strong>nbthread &lt;number&gt;</strong> sets the number of threads HAProxy will run on. On supported platforms, if no value is specified, HAProxy automatically detects the number of cores the process is bound to at startup and sets nbthread accordingly — meaning you can adjust the thread count simply by constraining the process with taskset or cpuset before launching it, without touching the configuration. Without auto-detection or explicit configuration, the default falls back to 1. The effective value is shown in the output of haproxy -vv.</li><li><strong>numa-cpu-mapping</strong> goes further: on a NUMA-aware platform, this directive lets HAProxy inspect the hardware topology itself and determine the best set of CPUs to use, along with the corresponding thread count. This automatic behaviour is disabled as soon as nbthread is set manually, when process affinity is already specified (via cpu-map, taskset...), or when the feature is explicitly turned off with no numa-cpu-mapping if the automatic placement proves suboptimal on a given architecture.</li></ul><p>In practice, on a dual-socket system, numa-cpu-mapping can do much of the work described above on the administrator&#39;s behalf: detect the relevant NUMA node and confine HAProxy threads to it. But it remains a software workaround for a hardware problem — it does not remove the need to verify that the NIC itself, its queues, and its IRQs are aligned to the same NUMA node.</p><p>The official HAProxy documentation points in the same direction: on a dual-socket machine, it advises against spreading processing across both physical sockets and recommends using all the cores of a single physical CPU instead, given the cost of inter-CPU communication. If heavy SSL/TLS workloads justify using both sockets, the same documentation suggests assigning a dedicated NIC to each socket so that each thread group only communicates with the NIC on its own CPU.</p><h3>Single-Socket vs Dual-Socket</h3><p><strong>In favour of single-socket:</strong></p><ul><li>HAProxy is primarily a consumer of <strong>network bandwidth and a handful of cores</strong>, rarely demanding in terms of RAM or heavy computation. A modern high-frequency CPU, even with few cores, comfortably covers the need.</li><li>On a single-socket system, <strong>there is only one NUMA node</strong>: all NICs, all RAM, and all cores are “local” to each other. The problem described above disappears by design, with no special configuration to maintain.</li><li>Less operational complexity: no IRQ affinity to manage per socket, no risk that a redeployment places the process on the “wrong” NUMA node.</li></ul><p><strong>In favour of dual-socket, even for this type of workload:</strong></p><ul><li>If the machine handles multiple roles (reverse proxy + heavy TLS termination + observability + other services), the additional core count and total memory bandwidth of a dual-socket system may become necessary.</li><li>A well-configured dual-socket system (IRQ affinity, numactl, correct PCIe slot selection) can keep NUMA impact well under control: it is a matter of operational discipline, not an unavoidable penalty.</li><li>At very high throughput with multiple NICs used in parallel, a well-planned dual-socket setup can offer more aggregate PCIe bandwidth than a single-socket system.</li></ul><p>In practice, for a fleet of servers dedicated to a network-intensive but compute-light role like HAProxy, single-socket is often the simplest and most robust choice: it eliminates the NUMA problem at the source rather than requiring ongoing management. Dual-socket remains worthwhile as soon as the machine must also carry significant compute or memory workloads.</p><h3>Key Takeaways</h3><p>In the majority of cases, a network card is wired to a single socket. On a dual-socket server, overlooking this detail can cost dearly in latency and throughput as soon as network load becomes significant. Single-socket eliminates the problem by design; dual-socket demands discipline but remains justified whenever other resources — cores, memory — come into play.</p><p>On a dual-socket system, before manually forcing nbthread or CPU affinity to exploit all cores across both sockets, benchmark first: adding more threads may seem logical on paper, but if it causes HAProxy to run threads across both sockets, the inter-CPU communication overhead can actually hurt performance instead of improving it. HAProxy&#39;s automatic settings (numa-cpu-mapping, cpu-policy) generally provide a safer starting point; any manual configuration should be validated under real load before going to production.</p><h3>Sources</h3><ul><li><a href="https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/">HAProxy Configuration Manual — nbthread, numa-cpu-mapping</a></li><li><a href="https://www.haproxy.com/documentation/haproxy-configuration-tutorials/performance/performance-tuning/">HAProxy — Performance optimization for large systems</a></li><li><a href="https://devops-notes.com/other/numa-nic-bisocket-en.html">https://devops-notes.com/other/numa-nic-bisocket-en.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4f2309c91e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning Pod Eviction: Don’t Let Kubernetes Pull the Trigger Too Fast]]></title>
            <link>https://medium.com/@jfpucheu/tuning-pod-eviction-dont-let-kubernetes-pull-the-trigger-too-fast-0c718bd9d702?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0c718bd9d702</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Mon, 06 Apr 2026 08:24:36 GMT</pubDate>
            <atom:updated>2026-04-06T08:24:36.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KpVvaQOR0DioG0CP.png" /></figure><p>When a Kubernetes node disappears — network failure, kubelet crash, OOM — your pods don’t evict instantly. There’s a deliberate pipeline of timeouts that decides <em>when</em> and <em>whether</em> to reschedule workloads. Misconfigure it and you get two equally bad outcomes: pods evicted after 30 seconds on a node that was just briefly overloaded, or pods stuck for 10 minutes on a genuinely dead node.</p><p>This article explains how the eviction pipeline works, what each knob does, and gives you three ready-to-use configurations depending on your tolerance for disruption.</p><h3>How the eviction pipeline works</h3><p>One of the most common mistakes is tuning a single parameter and wondering why nothing changes. Eviction behavior is the <strong>product of a combination of settings spread across three different components</strong> — kubelet, kube-controller-manager, and kube-apiserver. Each one controls a different phase of the pipeline, and they must be tuned together or the result is unpredictable.</p><p>Three components cooperate in sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*0CjQgKF2DabneTgZZ9SSnA.png" /></figure><p>The total eviction time is always:</p><pre>T_eviction = node-monitor-grace-period + tolerationSeconds</pre><h3>All parameters at a glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*GAxS7j9vjIIFwfYloqlgVQ.png" /></figure><h3>NotReady vs Unreachable — two different taints</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*p3G4Lib063LpNCMmbF_qEg.png" /></figure><p>Both taints trigger eviction via tolerationSeconds. The difference is the <em>cause</em>, not the mechanism.</p><pre>node-monitor-grace-period ≥ 3 × nodeStatusUpdateFrequency</pre><p>This ensures at least <strong>3 missed heartbeats</strong> are required before a node is declared unhealthy. Anything less and a momentary kubelet hiccup or network blip can evict production pods unnecessarily.</p><h3>Timeline — what actually happens</h3><p><strong>When a node goes NotReady (kubelet degraded)</strong></p><pre>T+0s    kubelet stops sending heartbeats<br>T+Xs    last known heartbeat (depends on nodeStatusUpdateFrequency)<br>T+Gs    grace-period expires → taint not-ready:NoExecute applied<br>T+G+Ts  tolerationSeconds expires → pods enter Terminating</pre><p><strong>When a node goes Unreachable (network / crash)</strong></p><p>Same timeline, but the taint is unreachable:NoExecute and Ready=Unknown instead of Ready=False. The risk of false positives is higher - a 30-second network partition is enough to trigger eviction with an aggressive config.</p><h3>Eviction timeline — the three scenarios compared</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lF6HHxdpRT5kq9xUMb2JjA.png" /></figure><h3>The three configurations</h3><p><strong><em>🔴 Aggressive — fast failover, high false positive risk</em></strong></p><p><strong>Use case:</strong> stateless workloads, HA deployments where you want fast rescheduling and can tolerate occasional unnecessary evictions.</p><pre># kube-apiserver<br>--default-not-ready-toleration-seconds=10<br>--default-unreachable-toleration-seconds=10<br><br># kube-controller-manager<br>--node-monitor-period=2s<br>--node-monitor-grace-period=20s<br>--terminated-pod-gc-threshold=50<br><br># kubelet<br>nodeStatusUpdateFrequency: 10s</pre><p>Pod tolerations (auto-injected):</p><pre>tolerations:<br>  - effect: NoExecute<br>    key: node.kubernetes.io/not-ready<br>    operator: Exists<br>    tolerationSeconds: 10<br>  - effect: NoExecute<br>    key: node.kubernetes.io/unreachable<br>    operator: Exists<br>    tolerationSeconds: 10</pre><pre>T_eviction = 20s + 10s = 30s<br>Golden rule: 20s ≥ 3×10s=30s → ❌ NOT MET (only 2 missed heartbeats)</pre><blockquote><em>⚠️ A kubelet under CPU pressure or a 15-second network blip is enough to evict pods. Not recommended for stateful workloads or clusters with variable network latency.</em></blockquote><blockquote><strong><em>Well suited for</em></strong><em> small, fast-starting containers that reschedule in seconds and have no local state — think sidecars, short-lived jobs, or replicated stateless services with multiple replicas.</em></blockquote><blockquote><strong><em>Watch out during apiserver maintenance:</em></strong><em> if your control plane sits behind a load balancer or kube-vip, a leader failover or VIP switchover can take 10–30 seconds. With a 30s eviction window, pods may start evicting before the new apiserver is reachable — causing a cascade that wouldn’t happen with a less aggressive config. Make sure your LB/kube-vip failover time is well below </em><em>tolerationSeconds.</em></blockquote><p><strong><em>🟡 Medium — tolerant to flapping, faster than default</em></strong></p><p><strong>Use case:</strong> bare-metal clusters, industrial environments, nodes with slow or unreliable heartbeat paths (high kubelet nodeStatusUpdateFrequency), or any cluster where false evictions are more costly than slow recovery.</p><pre># kube-apiserver<br>--default-not-ready-toleration-seconds=60<br>--default-unreachable-toleration-seconds=60<br><br># kube-controller-manager<br>--node-monitor-period=5s<br>--node-monitor-grace-period=2m<br>--terminated-pod-gc-threshold=50<br><br># kubelet<br>nodeStatusUpdateFrequency: 20s</pre><p>Pod tolerations:</p><pre>tolerations:<br>  - effect: NoExecute<br>    key: node.kubernetes.io/not-ready<br>    operator: Exists<br>    tolerationSeconds: 60<br>  - effect: NoExecute<br>    key: node.kubernetes.io/unreachable<br>    operator: Exists<br>    tolerationSeconds: 60</pre><pre>T_eviction = 120s + 60s = 3min<br>Golden rule: 120s ≥ 3×20s=60s → ✅ MET (6 missed heartbeats)</pre><blockquote><em>The kubelet heartbeats every 20s, and 6 consecutive misses are required before the node is considered unhealthy. Very resilient to network flapping and temporary kubelet load spikes. Recommended for CAPI-managed clusters and environments where nodes are provisioned/deprovisioned frequently.</em></blockquote><p><strong><em>🟢 Normal — general purpose</em></strong></p><p><strong>Use case:</strong> general-purpose clusters, mixed workloads. Good balance between recovery speed and false positive tolerance.</p><pre># kube-apiserver<br>--default-not-ready-toleration-seconds=300<br>--default-unreachable-toleration-seconds=300<br><br># kube-controller-manager<br>--node-monitor-period=5s<br>--node-monitor-grace-period=40s<br>--terminated-pod-gc-threshold=12500<br><br># kubelet<br>nodeStatusUpdateFrequency: 10s</pre><p>Pod tolerations:</p><pre>tolerations:<br>  - effect: NoExecute<br>    key: node.kubernetes.io/not-ready<br>    operator: Exists<br>    tolerationSeconds: 300<br>  - effect: NoExecute<br>    key: node.kubernetes.io/unreachable<br>    operator: Exists<br>    tolerationSeconds: 300</pre><pre>T_eviction = 40s + 300s = ~5min40s <br>Golden rule: 40s ≥ 3×10s=30s → ✅ MET (4 missed heartbeats)</pre><blockquote><em>These are close to the Kubernetes upstream defaults. The 5-minute </em><em>tolerationSeconds means pods stay on a dead node for a while before rescheduling - which is intentional to avoid thundering herd on short outages.</em></blockquote><h3>Comparison summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*x9wv0Qe_wjeKDrNdlvbBGg.png" /></figure><h3>Key takeaways</h3><ul><li><strong>Always verify the golden rule</strong> grace-period ≥ 3 × heartbeat - violating it makes eviction non-deterministic.</li><li><strong>tolerationSeconds is injected at pod creation</strong> by the DefaultTolerationSeconds admission plugin - changing the apiserver flag only affects <em>new</em> pods, not running ones.</li><li><strong>Unreachable is riskier than NotReady</strong> for false positives. A 30-second network partition is invisible to users but enough to trigger eviction in an aggressive config.</li><li><strong>Node shutdown</strong> is a separate mechanism (shutdownGracePeriod on kubelet) - the eviction pipeline above only applies to unexpected failures.</li><li>For <strong>stateful workloads</strong> (databases, queues), always use Medium or at minimum Normal, and consider Pod Disruption Budgets on top.</li></ul><h3>Conclusion</h3><p>These parameters are not cosmetic — they directly control how your cluster reacts to failure, and a wrong value in either direction causes real damage: unnecessary disruption of healthy workloads, or pods stuck on dead nodes for minutes.</p><p>There is no universal right answer. The correct values depend on three things you need to assess together:</p><ul><li><strong>Your workloads</strong> — a stateless nginx replica and a PostgreSQL primary do not have the same tolerance for eviction. Fast-starting containers can afford aggressive settings; anything with local state, slow startup, or strict ordering cannot.</li><li><strong>Your services</strong> — a batch job tolerates 5 minutes of downtime; a payment API does not. SLOs and SLAs should drive your tolerationSeconds, not the other way around.</li><li><strong>Your underlying infrastructure</strong> — bare-metal with occasional flapping NICs, a cloud provider with live migration, kube-vip with a 20-second failover, or a CAPI-managed cluster that reprovisioning nodes regularly all have different heartbeat reliability profiles.</li></ul><p>When in doubt, start with <strong>Normal</strong> (the Kubernetes default), measure actual node failure and recovery times in your environment, then adjust. Going aggressive without understanding your LB failover time or kubelet startup latency is a reliable way to cause incidents during maintenance windows.</p><p><em>Originally published at </em><a href="https://devops-notes.com/kubernetes/eviction.html"><em>https://devops-notes.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0c718bd9d702" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop letting Kubernetes events kill your etcd — Here’s the Fix]]></title>
            <link>https://medium.com/@jfpucheu/stop-letting-kubernetes-events-kill-your-etcd-heres-the-fix-cd682a9c7e73?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd682a9c7e73</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Wed, 01 Apr 2026 21:04:18 GMT</pubDate>
            <atom:updated>2026-04-01T21:04:18.873Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In large Kubernetes clusters, </em><strong><em>events</em></strong><em> can quickly become a problem. While they are useful for debugging and observability, they generate a very high volume of writes to etcd, which can </em><strong><em>severely degrade the performance</em></strong><em> of the main etcd cluster and therefore impact the entire Kubernetes API.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*rBs72J8jNV60gBWtxUqX6g.png" /></figure><h3>Problem</h3><p>In Kubernetes, events are stored in etcd under the prefix /registry/events. In high-activity environments (many pods, deployments, controllers…), this prefix becomes heavily used:</p><ul><li>Massive and continuous writes</li><li>High data turnover (short TTL by default: 1h)</li><li>Pressure on etcd storage and CPU</li><li>More frequent etcd compaction and defragmentation</li></ul><h3>Solution: offload certain keys to a dedicated etcd</h3><p>Kubernetes allows <strong>redirecting certain keys to a separate etcd cluster</strong> via the kube-apiserver flag:</p><pre>--etcd-servers-overrides=&lt;prefix&gt;#&lt;server1&gt;,&lt;server2&gt;,&lt;server3&gt;</pre><p><em>Servers within the same group are separated by </em><strong><em>commas</em></strong><em> (</em><em>,). Multiple overrides are also separated by </em><strong><em>semicolons</em></strong><em> (</em><em>;).</em></p><p><em>kube-apiserver configuration example:</em></p><pre># Main etcd<br>- --etcd-servers=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379<br>- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt<br>- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt<br>- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key<br><br># Offloading events to a dedicated etcd cluster<br>- --etcd-servers-overrides=/events#https://10.0.1.20:2379,https://10.0.1.21:2379,https://10.0.1.22:2379</pre><h3>Benefits</h3><ul><li>Reduced load on the main etcd</li><li>Better overall API server performance</li><li>Isolation of highly volatile data</li><li>Independent scalability of both clusters</li><li>Independent compaction/defragmentation cycles</li></ul><h3>Limitations and constraints</h3><h4>Shared authentication</h4><p>The kube-apiserver uses the <strong>same TLS client certificates</strong> (etcd-certfile / etcd-keyfile) for all etcd clusters, including overrides. It is not possible to specify different certificates per cluster.</p><p>In practice: both etcd clusters must be signed by the <strong>same CA</strong> and accept the same kube-apiserver client certificate.</p><h4>Availability</h4><p>If the events-dedicated etcd becomes unavailable:</p><ul><li>The kube-apiserver keeps attempting to write/read events</li><li>Timeouts and retries impact overall performance</li><li>The main etcd remains operational, but the API server is degraded</li></ul><h3>Best practices</h3><ul><li>Deploy the events etcd in high availability (minimum 3 nodes)</li><li>Monitor both etcd clusters separately</li><li>Tune event retention: --event-ttl=1h (default value)</li><li>Limit overrides to highly volatile keys (events, leases…)</li><li>Do not offload critical keys such as /registry/pods or /registry/secrets</li></ul><h3>Conclusion</h3><p>Offloading /registry/events to a separate etcd cluster is an effective optimization for large Kubernetes clusters where events become a real bottleneck.</p><p>This must be set up carefully: the dedicated etcd must be highly available, and both clusters must share the same PKI.</p><p><em>📘 Sources:<br></em><a href="https://devops-notes.com/kubernetes/etcd_split.html"><em>Devops-notes.com</em></a><em><br></em><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/"><em>Kubernetes Documentation</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd682a9c7e73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resurrect your broken Kubernetes Cluster with just one ETCD snapshot]]></title>
            <link>https://medium.com/@jfpucheu/resurrect-your-broken-kubernetes-cluster-with-just-one-etcd-snapshot-d1f5228ec6b6?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1f5228ec6b6</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubeadm]]></category>
            <category><![CDATA[disaster-recovery]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[cluster-api]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Sun, 30 Nov 2025 21:59:55 GMT</pubDate>
            <atom:updated>2025-11-30T21:59:55.535Z</atom:updated>
            <content:encoded><![CDATA[<h3>Resurrect Your Broken Kubernetes Cluster with Just One ETCD Snapshot</h3><p><em>Reading time: 8 minutes | Last updated: November 30, 2025</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iiIobSpg3w3DU8H2AAuqAQ.png" /></figure><p>Your production Kubernetes cluster is <strong>gone</strong>. Complete datacenter failure. Every node destroyed. But you have one thing: <strong>an ETCD backup from yesterday</strong>.</p><p>In this article, I’ll show you how to resurrect your entire cluster using <strong>Cluster API (CAPI)</strong> and that single ETCD snapshot. Everything — your deployments, secrets, configs, RBAC policies — all restored in <strong>less than 30 minutes</strong>.</p><h4>What You’ll Learn</h4><ul><li>✅ Why ETCD is your ultimate insurance policy</li><li>✅ Restore a complete cluster using Cluster API automation</li><li>✅ Handle the critical kubeadm preflight checks</li><li>✅ <strong>Don’t forget the ETCD encryption key</strong> (if enabled)</li><li>✅ Clean up old infrastructure properly</li><li>✅ The entire disaster recovery workflow</li></ul><p><strong><em>Universal Approach</em></strong><em>: This works with any Kubernetes cluster — CAPI, kubeadm, EKS, AKS, GKE. The principles remain the same.</em></p><blockquote>⚠️ Critical Warning: If your cluster uses ETCD encryption-at-rest, you MUST have a backup of the encryption key (<em>/etc/kubernetes/pki/encryption-config.yaml</em>). Without it, all your Secrets will be unreadable after restoration!</blockquote><h3>📚 Why ETCD is Your Lifeline</h3><h4>The Single Source of Truth</h4><p>ETCD isn’t just a database — it’s <strong>the brain</strong> of your Kubernetes cluster. Everything lives there:</p><ul><li>🔐 All Secrets and ConfigMaps</li><li>📦 Every Deployment, Service, and Ingress</li><li>🎫 Certificates and authentication tokens</li><li>📊 Complete state of all Pods and Nodes</li><li>🔒 RBAC policies and ServiceAccounts</li><li>🎯 CustomResourceDefinitions and their instances</li></ul><blockquote><strong>Critical Point</strong>: With a recent ETCD backup, you can recreate your <strong>entire cluster</strong>, even if every node has vanished</blockquote><h4>The “Stateless Cluster” Paradigm</h4><p>A <strong>stateless cluster</strong> means:</p><ol><li><strong>Workloads are ephemeral</strong>: Apps use external persistent storage (EBS, Ceph, NFS)</li><li><strong>Infrastructure is code</strong>: Nodes are recreated via Terraform, CAPI, or cloud-init</li><li><strong>State lives in ETCD</strong>: All cluster configuration exists in ETCD</li></ol><p>This enables <strong>ultra-fast disaster recovery</strong>: destroy everything, restore ETCD, and your cluster rises from the ashes! 🔥</p><h3>🔧 Prerequisites</h3><pre># Required Tools<br>✓ kubectl (&gt;= 1.28)<br>✓ clusterctl (Cluster API CLI)<br>✓ etcdctl (&gt;= 3.4)<br>✓ curl or aws-cli (for fetching backups)<br><br># Required Access<br>✓ Management CAPI cluster operational<br>✓ Recent ETCD backup (&lt; 24h ideally)<br>✓ Access to backup storage (S3, GCS, HTTP, etc.)<br><br># ⚠️ CRITICAL: ETCD Encryption Key<br>✓ Backup of ETCD encryption key (if encryption-at-rest enabled)<br>  Location: /etc/kubernetes/pki/etcd/encryption-config.yaml<br>  Without this key, your Secrets will be UNREADABLE!</pre><h4>⚠️ Critical: ETCD Encryption Key</h4><p>If your cluster uses <strong>encryption at rest</strong> for Secrets, you <strong>MUST</strong> have a backup of the encryption key. Without it, all your Secrets will be encrypted garbage in the restored cluster.</p><p><strong>Check if encryption is enabled:</strong></p><pre># On existing control plane<br>kubectl get pod kube-apiserver-* -n kube-system -o yaml | grep encryption-provider-config<br># If you see output, encryption is enabled</pre><p><strong>Backup the encryption key:</strong></p><pre># The encryption configuration file<br>sudo cp /etc/kubernetes/pki/etcd/encryption-config.yaml /backup/<br><br># Example content:<br>apiVersion: apiserver.config.k8s.io/v1<br>kind: EncryptionConfiguration<br>resources:<br>  - resources:<br>      - secrets<br>    providers:<br>      - aescbc:<br>          keys:<br>            - name: key1<br>              secret: &lt;BASE64_ENCODED_KEY&gt;  # ← THIS IS CRITICAL!<br>      - identity: {}</pre><p><strong>Store it securely:</strong></p><pre># Encrypt and store with your ETCD backup<br>openssl enc -aes-256-cbc -salt -in encryption-config.yaml \<br>  -out encryption-config.yaml.enc -pass pass:your-secure-password<br><br># Upload alongside ETCD backup<br>aws s3 cp encryption-config.yaml.enc s3://backups/</pre><h3>🚀 The Complete Restoration Workflow</h3><p><strong>Architecture Overview</strong></p><pre>┌─────────────────────────────────────────────────────┐<br>│         Management Cluster (CAPI)                   │<br>│  ┌──────────────────────────────────────────────┐   │<br>│  │  clusterctl creates workload cluster         │   │<br>│  └──────────────────────────────────────────────┘   │<br>└─────────────────────────────────────────────────────┘<br>                         │<br>                         ↓<br>┌─────────────────────────────────────────────────────┐<br>│       New Workload Cluster (empty)                  │<br>│  ┌──────────────────────────────────────────────┐   │<br>│  │  1️⃣  Bootstrap control plane nodes            │  │<br>│  │  2️⃣  Inject restoration script                │  │<br>│  │  3️⃣  Restore ETCD BEFORE kubeadm init         │  │<br>│  │  4️⃣  Cluster restored with all resources      │  │<br>│  │  5️⃣  Clean up old workers &amp; update LB         │  │<br>│  └──────────────────────────────────────────────┘   │<br>└─────────────────────────────────────────────────────┘</pre><h3>📝 Step 1: The Restoration Script</h3><p>This script handles everything: detection, download, validation, and restoration.</p><pre>#!/bin/bash<br>set -euo pipefail<br><br># Script: etcd_restore.sh<br># Purpose: Restore ETCD from backup before kubeadm init<br>readonly KUBEADM_FILE=&quot;/run/kubeadm/kubeadm.yaml&quot;<br>readonly BACKUP_URL=&quot;https://github.com/your-org/backups/raw/main&quot;<br>readonly DEFAULT_SNAPSHOT=&quot;etcd-snapshot.db.tar.gz&quot;<br>log() { echo -e &quot;\033[0;32m[INFO]\033[0m $1&quot;; }<br>err() { echo -e &quot;\033[0;31m[ERROR]\033[0m $1&quot; &gt;&amp;2; exit &quot;${2:-1}&quot;; }<br># Only run on control plane nodes<br>if [ ! -f &quot;$KUBEADM_FILE&quot; ]; then<br>  log &quot;Not a control plane - skipping&quot;<br>  exit 0<br>else<br>  log &quot;✅ First control plane detected - restoring ETCD&quot;<br>fi<br># Get snapshot name (argument or default)<br>SNAPSHOT_NAME=&quot;${1:-$DEFAULT_SNAPSHOT}&quot;<br>SNAPSHOT_URL=&quot;${BACKUP_URL}/${SNAPSHOT_NAME}&quot;<br>TEMP_TAR=&quot;/tmp/${SNAPSHOT_NAME}&quot;<br>TEMP_SNAPSHOT=&quot;/tmp/etcd-snapshot.db&quot;<br># Download backup<br>log &quot;📥 Downloading snapshot: ${SNAPSHOT_NAME}&quot;<br>if ! curl -fsSL --retry 3 --retry-delay 5 --max-time 300 \<br>     -o &quot;$TEMP_TAR&quot; &quot;$SNAPSHOT_URL&quot;; then<br>  err &quot;Download failed&quot;<br>fi<br>[ -f &quot;$TEMP_TAR&quot; ] || err &quot;Snapshot not found after download&quot;<br>log &quot;✅ Downloaded successfully ($(du -h $TEMP_TAR | awk &#39;{print $1}&#39;))&quot;<br># Extract<br>log &quot;📦 Extracting snapshot...&quot;<br>tar -xzf &quot;$TEMP_TAR&quot; -C /tmp/ || { rm -f &quot;$TEMP_TAR&quot;; err &quot;Extraction failed&quot;; }<br>[ -f &quot;$TEMP_SNAPSHOT&quot; ] || err &quot;snapshot.db not found after extraction&quot;<br># Verify integrity<br>log &quot;🔍 Verifying snapshot integrity...&quot;<br>ETCDCTL_API=3 etcdctl snapshot status &quot;$TEMP_SNAPSHOT&quot; --write-out=table || {<br>  rm -f &quot;$TEMP_SNAPSHOT&quot; &quot;$TEMP_TAR&quot;<br>  err &quot;Snapshot corrupted or invalid&quot;<br>}<br># Get node information<br>HOSTNAME=$(hostname)<br>HOSTIP=$(hostname -I | awk &#39;{print $1}&#39;)<br>log &quot;Node: ${HOSTNAME} (${HOSTIP})&quot;<br># Restore snapshot<br>log &quot;🔄 Restoring ETCD snapshot...&quot;<br>ETCDCTL_API=3 etcdctl snapshot restore &quot;$TEMP_SNAPSHOT&quot; \<br>  --name &quot;${HOSTNAME}&quot; \<br>  --data-dir /var/lib/etcd \<br>  --initial-cluster &quot;${HOSTNAME}=https://${HOSTIP}:2380&quot; \<br>  --initial-advertise-peer-urls &quot;https://${HOSTIP}:2380&quot; \<br>  --initial-cluster-token &quot;restored-${HOSTNAME}&quot; || err &quot;Restoration failed&quot;<br># Verify<br>[ -d /var/lib/etcd ] || err &quot;ETCD directory not created&quot;<br>log &quot;ETCD data dir: /var/lib/etcd ($(du -sh /var/lib/etcd | awk &#39;{print $1}&#39;))&quot;<br># Cleanup<br>rm -f &quot;$TEMP_SNAPSHOT&quot; &quot;$TEMP_TAR&quot;<br>log &quot;✅ ETCD restoration complete - ready for kubeadm init&quot;🔑 Key Points</pre><h4>🔑 Key Points</h4><ol><li><strong>Smart Detection</strong>: Checks /run/kubeadm/kubeadm.yaml to determine if it should run</li><li><strong>First Control Plane Only</strong>: Uses grep &quot;kind: ClusterConfiguration&quot; to detect primary node</li><li><strong>Idempotent</strong>: Can be run multiple times safely</li><li><strong>Error Handling</strong>: Fails fast with clear error messages</li><li><strong>Verification</strong>: Validates snapshot integrity before restoration</li></ol><h3>🎭 Step 2: CAPI Manifest Configuration</h3><h4>⚠️ Critical Configuration Points</h4><p>Here’s the complete CAPI manifest with <strong>three critical elements</strong> you must not forget:</p><pre>apiVersion: controlplane.cluster.x-k8s.io/v1beta1<br>kind: KubeadmControlPlane<br>metadata:<br>  name: restored-cluster-control-plane<br>  namespace: default<br>spec:<br>  kubeadmConfigSpec:<br>    # ⚠️ CRITICAL #1: Inject script as a file<br>    files:<br>      # The restoration script MUST be injected as a file<br>      - path: /etc/kubernetes/etcd_restore.sh<br>        owner: root:root<br>        permissions: &quot;0755&quot;<br>        content: |<br>          #!/bin/bash<br>          set -euo pipefail<br>          # Complete script content from above<br>          # ... (full script here) ...<br>      <br>      # ⚠️ CRITICAL: ETCD Encryption Key (if encryption-at-rest enabled)<br>      - path: /etc/kubernetes/pki/encryption-config.yaml<br>        owner: root:root<br>        permissions: &quot;0600&quot;<br>        content: |<br>          apiVersion: apiserver.config.k8s.io/v1<br>          kind: EncryptionConfiguration<br>          resources:<br>            - resources:<br>                - secrets<br>              providers:<br>                - aescbc:<br>                    keys:<br>                      - name: key1<br>                        secret: YOUR_BASE64_ENCODED_KEY_HERE<br>                - identity: {}<br>    <br>    # ⚠️ CRITICAL #2: Execute BEFORE kubeadm init<br>    preKubeadmCommands:<br>      # This runs BEFORE kubeadm init - critical timing!<br>      - /etc/kubernetes/etcd_restore.sh etcd-backup-20241130.tar.gz<br>    <br>    # Standard kubeadm configuration<br>    initConfiguration:<br>      nodeRegistration:<br>        kubeletExtraArgs:<br>          cloud-provider: external<br>        # ⚠️ CRITICAL #3: Ignore preflight error<br>        # Without this, kubeadm init will fail because /var/lib/etcd already exists<br>        ignorePreflightErrors:<br>          - DirAvailable--var-lib-etcd<br>    <br>    clusterConfiguration:<br>      clusterName: restored-cluster<br>      apiServer:<br>        extraArgs:<br>          enable-admission-plugins: NodeRestriction,PodSecurityPolicy<br>          # ⚠️ CRITICAL: Enable encryption if it was enabled in original cluster<br>          encryption-provider-config: /etc/kubernetes/pki/encryption-config.yaml<br>        extraVolumes:<br>          - name: encryption-config<br>            hostPath: /etc/kubernetes/pki/encryption-config.yaml<br>            mountPath: /etc/kubernetes/pki/encryption-config.yaml<br>            readOnly: true<br>            pathType: File<br>      etcd:<br>        local:<br>          dataDir: /var/lib/etcd<br>          <br>  replicas: 3<br>  version: v1.28.0</pre><h3>⚠️ Three Critical Points Explained</h3><h4>1. Script MUST be injected as a file</h4><p>❌ <strong>WRONG APPROACH</strong>:</p><pre>preKubeadmCommands:<br>  - curl -o /tmp/script.sh https://... &amp;&amp; chmod +x /tmp/script.sh &amp;&amp; /tmp/script.sh</pre><p><strong>Why it fails:</strong></p><ul><li>Network may not be stable during bootstrap</li><li>No content verification possible</li><li>Harder to debug</li><li>Not versioned with your infrastructure</li></ul><p>✅ <strong>CORRECT APPROACH</strong>:</p><pre>files:<br>  - path: /etc/kubernetes/etcd_restore.sh<br>    permissions: &quot;0755&quot;<br>    content: |<br>      #!/bin/bash<br>      # Complete script</pre><pre>preKubeadmCommands:<br>  - /etc/kubernetes/etcd_restore.sh</pre><p><strong>Why it works:</strong></p><ul><li>Script is embedded in the manifest</li><li>Guaranteed to be present before execution</li><li>Version controlled with your cluster config</li><li>Easy to audit and debug</li></ul><h4>2. preKubeadmCommands execution timing</h4><p>The order of operations is <strong>critical</strong>:</p><pre>1️⃣ CAPI creates machine<br>2️⃣ Cloud-init bootstrap<br>3️⃣ files: written to disk<br>4️⃣ preKubeadmCommands: executed  ← Our script runs HERE<br>   └─ Script restores ETCD to /var/lib/etcd<br>5️⃣ kubeadm init (with ignorePreflightErrors)<br>6️⃣ Cluster operational with restored data</pre><p>If you put it in postKubeadmCommands, it&#39;s <strong>too late</strong> — kubeadm will have already initialized an empty ETCD.</p><h4>3. ignorePreflightErrors is MANDATORY</h4><p>Without this, you’ll get this error:</p><pre>[preflight] Running pre-flight checks<br>error execution phase preflight: [preflight] Some fatal errors occurred:<br>  [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty</pre><p><strong>Why it happens:</strong></p><ul><li>kubeadm checks that /var/lib/etcd is empty before initializing</li><li>Our script has already restored data there</li><li>We need to tell kubeadm “yes, we know, this is intentional”</li></ul><p><strong>Solution:</strong></p><pre>initConfiguration:<br>  nodeRegistration:<br>    ignorePreflightErrors:<br>      - DirAvailable--var-lib-etcd</pre><h3>🎬 Step 3: Execute the Restoration</h3><p><em>Timeline: 0 to 30 Minutes</em></p><p><strong>T+0min: Deploy the Cluster</strong></p><pre># Apply the CAPI manifest<br>kubectl apply -f restored-cluster.yaml</pre><pre># Watch cluster creation<br>clusterctl describe cluster restored-cluster --show-conditions all</pre><p><strong>T+5min: Control Planes Bootstrap</strong></p><pre># Watch machines come up<br>kubectl get machines -w</pre><pre>NAME                                  PHASE      AGE<br>restored-cluster-control-plane-abc   Running    2m<br>restored-cluster-control-plane-def   Running    1m<br>restored-cluster-control-plane-ghi   Pending    30s</pre><p><strong>T+10min: ETCD Restoration (Automatic)</strong></p><p>The restoration happens automatically on the first control plane:</p><pre># Logs from the first control plane (via cloud-init)<br>[INFO] ✅ First control plane detected - restoring ETCD<br>[INFO] 📥 Downloading snapshot: etcd-backup-20241130.tar.gz<br>[INFO] ✅ Downloaded successfully (48M)<br>[INFO] 📦 Extracting snapshot...<br>[INFO] 🔍 Verifying snapshot integrity...<br>[INFO] Node: ip-10-0-1-100 (10.0.1.100)<br>[INFO] 🔄 Restoring ETCD snapshot...<br>[INFO] ETCD data dir: /var/lib/etcd (1.2G)<br>[INFO] ✅ ETCD restoration complete - ready for kubeadm init</pre><p><strong>T+15min: Cluster Operational</strong></p><pre># Get kubeconfig<br>clusterctl get kubeconfig restored-cluster &gt; restored.kubeconfig<br><br># Verify<br>export KUBECONFIG=restored.kubeconfig<br>kubectl get nodes<br>kubectl get pods -A<br><br># 🎉 All your workloads are back!<br>kubectl get deployments -A<br>kubectl get secrets -A<br>kubectl get configmaps -A</pre><h4>What Gets Restored Automatically</h4><p>✅ All namespaces<br>✅ All Deployments, StatefulSets, DaemonSets<br>✅ All Services and Ingresses<br>✅ All Secrets and ConfigMaps<br>✅ All RBAC (Roles, ClusterRoles, Bindings)<br>✅ All PersistentVolumeClaims (PVs reconnect automatically)<br>✅ All CustomResourceDefinitions and instances<br>✅ All admission webhooks and configurations</p><h3>🧹 Step 4: Post-Restoration Cleanup</h3><p>After restoration, you need to clean up old infrastructure that no longer exists.</p><h4>1. Remove Old Worker Nodes</h4><p>The restored cluster state references old worker nodes that no longer exist:</p><pre># List all nodes<br>kubectl get nodes<br><br>NAME                    STATUS     ROLES    AGE   VERSION<br>restored-cp-1           Ready      master   15m   v1.28.0<br>restored-cp-2           Ready      master   12m   v1.28.0<br>restored-cp-3           Ready      master   10m   v1.28.0<br>old-worker-1           NotReady    worker   5d    v1.28.0  ← Old node<br>old-worker-2           NotReady    worker   5d    v1.28.0  ← Old node<br>old-worker-3           NotReady    worker   5d    v1.28.0  ← Old node<br><br># Remove old worker nodes<br>kubectl delete node old-worker-1 old-worker-2 old-worker-3<br><br># Or use a pattern<br>kubectl get nodes --no-headers | grep NotReady | awk &#39;{print $1}&#39; | xargs kubectl delete node<br><br># Verify cleanup<br>kubectl get nodes<br>NAME                    STATUS   ROLES    AGE   VERSION<br>restored-cp-1           Ready    master   15m   v1.28.0<br>restored-cp-2           Ready    master   12m   v1.28.0<br>restored-cp-3           Ready    master   10m   v1.28.0</pre><h4>2. Update Load Balancer Backend Pool</h4><p>Your load balancer still points to the old control plane IPs. Update it:</p><p><strong><em>AWS (ELB/ALB/NLB)</em></strong></p><pre># Get new control plane IPs<br>NEW_CP_IPS=$(kubectl get nodes -l node-role.kubernetes.io/control-plane \<br>  -o jsonpath=&#39;{.items[*].status.addresses[?(@.type==&quot;InternalIP&quot;)].address}&#39;)<br><br>echo &quot;New control plane IPs: $NEW_CP_IPS&quot;<br><br># Update target group (replace with your target group ARN)<br>TARGET_GROUP_ARN=&quot;arn:aws:elasticloadbalancing:region:account:targetgroup/...&quot;<br><br># Deregister old targets<br>aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN \<br>  --query &#39;TargetHealthDescriptions[*].Target.Id&#39; --output text | \<br>  xargs -n1 | while read target; do<br>    aws elbv2 deregister-targets --target-group-arn $TARGET_GROUP_ARN \<br>      --targets Id=$target<br>  done<br><br># Register new targets<br>for ip in $NEW_CP_IPS; do<br>  aws elbv2 register-targets --target-group-arn $TARGET_GROUP_ARN \<br>    --targets Id=$ip,Port=6443<br>done<br><br># Verify<br>aws elbv2 describe-target-health --target-group-arn $TARGET_GROUP_ARN</pre><p><strong><em>Azure (Load Balancer)</em></strong></p><pre># Get resource group and LB name<br>RESOURCE_GROUP=&quot;my-rg&quot;<br>LB_NAME=&quot;k8s-lb&quot;<br>BACKEND_POOL_NAME=&quot;k8s-control-plane&quot;<br><br># Get new IPs<br>NEW_CP_IPS=$(kubectl get nodes -l node-role.kubernetes.io/control-plane \<br>  -o jsonpath=&#39;{.items[*].status.addresses[?(@.type==&quot;InternalIP&quot;)].address}&#39;)<br><br># Update backend pool<br>for ip in $NEW_CP_IPS; do<br>  az network lb address-pool address add \<br>    --resource-group $RESOURCE_GROUP \<br>    --lb-name $LB_NAME \<br>    --pool-name $BACKEND_POOL_NAME \<br>    --name &quot;cp-${ip}&quot; \<br>    --ip-address $ip<br>done<br><br># Remove old addresses (if needed)<br>az network lb address-pool address list \<br>  --resource-group $RESOURCE_GROUP \<br>  --lb-name $LB_NAME \<br>  --pool-name $BACKEND_POOL_NAME</pre><p><strong><em>GCP (Load Balancer)</em></strong></p><pre># Get instance group<br>INSTANCE_GROUP=&quot;k8s-control-plane-ig&quot;<br>REGION=&quot;us-central1&quot;<br><br># List instances<br>gcloud compute instance-groups managed list-instances $INSTANCE_GROUP \<br>  --region $REGION<br><br># The new control planes should auto-register if using managed instance groups<br># Manual verification:<br>gcloud compute backend-services get-health k8s-apiserver-backend --global</pre><p><strong><em>Manual / HAProxy</em></strong></p><p>Update your HAProxy configuration:</p><pre># Edit haproxy.cfg<br>sudo vi /etc/haproxy/haproxy.cfg<br><br># Update backend section<br>backend k8s_apiserver<br>    balance roundrobin<br>    option httpchk GET /healthz<br>    http-check expect status 200<br>    # Replace old IPs with new ones:<br>    server cp1 10.0.1.100:6443 check<br>    server cp2 10.0.1.101:6443 check<br>    server cp3 10.0.1.102:6443 check<br><br># Reload HAProxy<br>sudo systemctl reload haproxy<br><br># Verify<br>curl -k https://your-lb-endpoint:6443/healthz</pre><h4>3. Automated Cleanup Script</h4><p>Here’s a complete cleanup script you can run after restoration:</p><pre>#!/bin/bash<br>set -euo pipefail<br><br>echo &quot;🧹 Post-Restoration Cleanup&quot;<br><br># 1. Remove old nodes<br>echo &quot;📋 Removing old NotReady nodes...&quot;<br>OLD_NODES=$(kubectl get nodes --no-headers | grep NotReady | awk &#39;{print $1}&#39;)<br><br>if [ -n &quot;$OLD_NODES&quot; ]; then<br>  echo &quot;Found old nodes to remove:&quot;<br>  echo &quot;$OLD_NODES&quot;<br>  echo &quot;$OLD_NODES&quot; | xargs kubectl delete node<br>  echo &quot;✅ Old nodes removed&quot;<br>else<br>  echo &quot;✅ No old nodes to remove&quot;<br>fi<br><br># 2. Get new control plane IPs<br>echo &quot;📝 New control plane IPs:&quot;<br>kubectl get nodes -l node-role.kubernetes.io/control-plane \<br>  -o custom-columns=NAME:.metadata.name,IP:.status.addresses[0].address<br><br># 3. Reminder to update load balancer<br>echo &quot;&quot;<br>echo &quot;⚠️  MANUAL ACTION REQUIRED:&quot;<br>echo &quot;Update your load balancer backend pool with the new IPs above&quot;<br>echo &quot;&quot;<br><br># 4. Verify cluster health<br>echo &quot;🔍 Cluster health check:&quot;<br>kubectl get nodes<br>kubectl get pods -A | grep -v Running | grep -v Completed || echo &quot;✅ All pods Running&quot;<br>kubectl get componentstatuses<br><br>echo &quot;✅ Cleanup complete&quot;</pre><h3>🔍 Troubleshooting</h3><h4>Problem 1: “Snapshot restore failed”</h4><pre># Symptom<br>Error: snapshot file has wrong CRC<br><br># Solution<br># Verify snapshot integrity<br>etcdctl snapshot status backup.db --write-out=table<br><br># Try an older backup<br>./etcd_restore.sh etcd-backup-previous.tar.gz</pre><h4>Problem 2: “kubeadm init still fails on DirAvailable”</h4><pre># Symptom<br>[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty<br><br># Cause<br>Missing ignorePreflightErrors in manifest<br><br># Solution<br># Verify in your KubeadmControlPlane:<br>initConfiguration:<br>  nodeRegistration:<br>    ignorePreflightErrors:<br>      - DirAvailable--var-lib-etcd  # ← Must be present!</pre><h4>Problem 3: “Pods stuck in CrashLoopBackOff”</h4><pre># Cause<br>PersistentVolumes not reconnected<br><br># Solution<br>kubectl get pv,pvc -A<br><br># Force pod recreation<br>kubectl delete pod &lt;pod-name&gt; --grace-period=0 --force</pre><h4>Problem 4: “Old nodes show as NotReady forever”</h4><pre># Expected behavior - these are old nodes from the backup<br><br># Solution<br>kubectl delete node &lt;old-node-name&gt;<br><br># Or batch delete<br>kubectl get nodes | grep NotReady | awk &#39;{print $1}&#39; | xargs kubectl delete node</pre><h4>Problem 5: “Secrets are unreadable/corrupted after restore”</h4><pre># Symptom<br>kubectl get secret my-secret -o yaml<br># Returns encrypted garbage or error: &quot;unable to decrypt&quot;<br><br># Cause<br>Missing or incorrect ETCD encryption key<br><br># Solution<br># 1. Verify encryption was enabled in original cluster<br>kubectl get pod kube-apiserver-* -n kube-system -o yaml | grep encryption-provider-config<br><br># 2. Restore the encryption-config.yaml<br># Must be EXACTLY the same key as original cluster<br>kubectl create secret generic encryption-key \<br>  --from-file=encryption-config.yaml=/backup/encryption-config.yaml \<br>  -n kube-system<br><br># 3. Update kube-apiserver manifest to use encryption<br># Add to /etc/kubernetes/manifests/kube-apiserver.yaml:<br>#   --encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml<br><br># 4. Restart kube-apiserver<br>kubectl -n kube-system delete pod kube-apiserver-&lt;node-name&gt;<br><br># 5. Verify Secrets are readable<br>kubectl get secret my-secret -o jsonpath=&#39;{.data.password}&#39; | base64 -d</pre><p><strong>Prevention</strong>: Always backup encryption-config.yaml alongside ETCD snapshots!</p><h3>📊 Success Metrics</h3><h4><a href="http://localhost:3000/capi/etcd-restore.html#recovery-time-objective-rto">Recovery Time Objective (RTO)</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*C3lwyYNrqJFClMpKcEI0DQ.png" /></figure><h4>Have you used this procedure? Questions or improvements?</h4><p><strong>Let’s connect</strong>:</p><ul><li>🐦 Twitter: <a href="https://twitter.com/jfpucheu">@jfpucheu</a></li><li>💼 LinkedIn: <a href="https://linkedin.com/in/jfpucheu">jfpucheu</a></li><li>📧 Email: jfpucheu@gmail.com</li></ul><p><strong>Tags</strong>: #Kubernetes #ETCD #ClusterAPI #DisasterRecovery #DevOps #SRE #K8s</p><blockquote>💡 <strong>Pro Tip</strong>: Bookmark this article! When you’re in crisis mode at 3 AM with a dead cluster, you’ll be glad you did. 😉</blockquote><p><em>If this article saved your cluster, give it a ⭐ and share it with your team!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1f5228ec6b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Securely Bootstrapping Secrets in a ClusterAPI Cluster with ExternalSecrets & PushSecret]]></title>
            <link>https://medium.com/@jfpucheu/securely-bootstrapping-secrets-in-a-clusterapi-cluster-with-externalsecrets-pushsecret-bb52a9385055?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb52a9385055</guid>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[external-secrets-operator]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Fri, 15 Aug 2025 12:13:47 GMT</pubDate>
            <atom:updated>2025-08-15T12:13:47.503Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGFV4ozM2EmEwYcFHtHfNw.png" /></figure><p>When I started automating Kubernetes cluster creation using <a href="https://cluster-api.sigs.k8s.io/">ClusterAPI</a>, one problem kept coming back:</p><p><strong>How can I securely inject secrets into a brand new cluster right at bootstrap — without storing them in Git or relying on fragile post-install hacks?</strong></p><p>After a lot of searching, I found a hidden gem in <a href="https://external-secrets.io/">External Secrets Operator (ESO)</a>: the powerful but lesser-known <strong>PushSecret</strong> feature.</p><h3>🌍 The Problem: Secure Secret Bootstrap</h3><p>In a typical ClusterAPI setup, you have a <strong>management cluster</strong> that provisions <strong>workload clusters</strong>. That works great, but raises a tricky issue:</p><blockquote><strong><em>How can you get critical secrets (certs, tokens, credentials) into a cluster at creation time — securely, and without manual steps?</em></strong></blockquote><p>Most solutions I found fell short:</p><ul><li>Putting secrets in Git (security nightmare),</li><li>Writing brittle post-creation scripts,</li><li>Waiting for the cluster to be ready before installing ESO/Vault (too late for early-stage secrets).</li></ul><h3>🔑 The Solution: ExternalSecrets + PushSecret</h3><p>If you’ve used External Secrets Operator before, you likely know it for syncing secrets from Vault, AWS Secrets Manager, GCP Secret Manager, etc. to Kubernetes.</p><p>But what many people don’t know is that ESO has a very useful feature called <strong>PushSecret</strong>.</p><h3>🧪 What Is PushSecret?</h3><p><strong>PushSecret</strong> lets you sync a Kubernetes secret <em>to another cluster</em> — <strong>without installing ESO on the target cluster</strong>.</p><p>That means you can:</p><ul><li>Define a secret source (Vault, AWS, or even a local Secret)</li><li>Push it from the management cluster to a remote child cluster</li><li>Do this <strong>without Git</strong>, <strong>without ESO installed on the target</strong>, and <strong>without manual work</strong></li></ul><h3>⚙️ How Does It Work?</h3><p>Here’s the basic idea:</p><ol><li>Install ESO only on the <strong>management cluster</strong>.</li><li>Define a ClusterSecretStore or SecretStore (with kubeconfig) to point to the child cluster.</li><li>Create a PushSecret that selects a local secret and sends it to the target cluster.</li><li>ESO does the sync as soon as the child cluster API is available.</li></ol><h3>Example: PushSecret</h3><p><strong>Define a </strong><strong>ClusterSecretStore:</strong></p><pre>apiVersion: external-secrets.io/v1<br>kind: ClusterSecretStore<br>metadata:<br>  name: mycapicluster-secretstore<br>  namespace: default<br>spec:<br>  provider:<br>    kubernetes:<br>      remoteNamespace: default<br>      authRef:<br>        name: mycapicluster-kubeconfig<br>        key: value<br>        namespace: default</pre><p>💡 <em>These kubeconfigs are automatically generated by ClusterAPI or its bootstrap provider (CAPBK).</em></p><p><strong>Create a </strong><strong>PushSecret:</strong></p><pre>apiVersion: external-secrets.io/v1alpha1<br>kind: PushSecret<br>metadata:<br>  name: pushsecret-mysecret<br>spec:<br>  refreshInterval: 60s<br>  selector:<br>    secret:<br>      name: mysecret<br>  secretStoreRefs:<br>  - kind: ClusterSecretStore<br>    name: mycapicluster-secretstore<br>  data:<br>    - match:<br>        remoteRef:<br>          remoteKey: mysecret</pre><h3>🎯 Real-World Use Cases</h3><ul><li>🔐 Injecting CSI driver credentials (AWS EBS, AzureDisk, Vault CSI)</li><li>🔑 Sharing a global TLS certificate across clusters</li><li>📦 Distributing container registry tokens (e.g., GitHub Container Registry)</li><li>⚙️ Bootstrapping cert-manager or external-dns with initial secrets</li></ul><h3>❤️ Why ClusterAPI + ExternalSecrets Works So Well</h3><p>ClusterAPI + ExternalSecrets is a powerful combo because:</p><ul><li>The management cluster <strong>generates kubeconfig and CA</strong> for new clusters automatically</li><li><strong>Cluster names are predictable</strong> (based on the cluster object)</li><li>You can <strong>predefine PushSecrets</strong> pointing to future clusters<br> → ESO pushes secrets as soon as the API is live — no manual steps</li></ul><h3>🔍 Concrete Example</h3><p>Provisioning a cluster named devcluster creates a secret named devcluster-kubeconfig in the namespace.<br> You can reference it like this:</p><pre>apiVersion: external-secrets.io/v1<br>kind: ClusterSecretStore<br>metadata:<br>  name: devcluster-secretstore<br>  namespace: kube-system<br>spec:<br>  provider:<br>    kubernetes:<br>      remoteNamespace: kube-system<br>      authRef:<br>        name: devcluster-kubeconfig<br>        key: value<br>        namespace: devcluster<br>---<br>apiVersion: external-secrets.io/v1alpha1<br>kind: PushSecret<br>metadata:<br>  name: registry-token-push<br>  namespace: kube-system<br>spec:<br>  refreshInterval: 60s<br>  selector:<br>    secret:<br>      name: registry-token<br>      namespace: kube-system<br>  secretStoreRefs:<br>  - kind: ClusterSecretStore<br>    name: devcluster-secretstore<br>  data:<br>    - match:<br>        remoteRef:<br>          remoteKey: registry-token</pre><p><strong>Check status:</strong></p><pre>$ kubectl get pushsecrets -n  kube-system<br>NAME                    STATUS      AGE<br>registry-token-push     Synchronized   2m</pre><h3>✅ Benefits</h3><ul><li>🔐 <strong>Secure:</strong> No secrets in Git</li><li>⚙️ <strong>Automated:</strong> Push when API is ready</li><li>🧼 <strong>Minimal:</strong> ESO only on management cluster</li><li>🌐 <strong>Flexible:</strong> Works with Vault, AWS, GCP, Azure, and native secrets</li></ul><h3>🧵 Summary</h3><p>If you use ClusterAPI and want a <strong>secure, automated</strong> way to inject secrets from day one — try <strong>PushSecret</strong> in ExternalSecrets Operator.</p><p>It solved a long-standing pain point for me, and it might save you hours of scripting too.</p><p>📘 <strong>Docs:</strong></p><ul><li><a href="https://external-secrets.io/latest/guides/pushsecrets/">External Secrets PushSecret</a></li><li><a href="https://cluster-api.sigs.k8s.io/">ClusterAPI</a></li><li><a href="https://devops-notes.com/capi/pushsecret.html">Devops-notes.com</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb52a9385055" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[--goaway-chance : The Parameter You Should Know to Keep Your Kubernetes API Server Stable ]]></title>
            <link>https://medium.com/@jfpucheu/goaway-chance-the-parameter-you-should-know-to-keep-your-kubernetes-api-server-stable-eca6c2ccd851?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/eca6c2ccd851</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kube-api-server]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[performance-tunning]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Sat, 19 Jul 2025 11:03:43 GMT</pubDate>
            <atom:updated>2025-07-19T11:05:40.844Z</atom:updated>
            <content:encoded><![CDATA[<h3>--goaway-chance : The Parameter You Should Know to Keep Your Kubernetes API Server Stable 🚀</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*VOOBwbBK5_8_oQsLutwIrg.png" /></figure><p>In many Kubernetes clusters where a <strong>load balancer</strong> sits in front of multiple kube-apiserver instances, it&#39;s common to observe <strong>uneven load distribution</strong> among API servers. This is an issue I’ve encountered several times and spent a lot of time investigating.</p><h3>⚠️ Symptoms</h3><ul><li>The most recently started kube-apiserver receives very little traffic.</li><li>The number of active <strong>goroutines</strong> on that server is significantly lower than on the others.</li><li>During upgrades or restarts, one server can become <strong>overloaded</strong>, taking over all the connections from a previously stopped instance — sometimes resulting in <strong>OOMKilled</strong> errors. 💥</li></ul><h3>🧠 Root Cause</h3><p>The kube-apiserver uses <strong>HTTP/2</strong> and maintains <strong>persistent connections</strong>. Once established, these connections are <strong>never rotated</strong> or rebalanced, regardless of the load balancing algorithm being used (round-robin, IP hash, etc.).</p><p>As a result, if you have three API servers and one of them restarts, it will not receive any new connections automatically. The already established client connections remain bound to the older servers, leaving the newly restarted one <strong>idle</strong>. 💤</p><h3>🛠️ The Solution: --goaway-chance</h3><p>The --goaway-chance parameter in kube-apiserver is a <strong>powerful but often overlooked</strong> option that can dramatically improve <strong>control plane stability</strong> in large clusters. 💡</p><h4>📘 Official Description</h4><p><em>To prevent HTTP/2 clients from getting stuck on a single API server, this flag allows the server to randomly close a connection by sending a </em><em>GOAWAY signal.<br> Ongoing requests will not be interrupted, and the client will reconnect — likely hitting a different API server through the load balancer.</em></p><ul><li><strong>Range</strong>: from 0 (disabled) to 0.02 (2% of requests)</li><li><strong>Recommended starting point</strong>: 0.001 (0.1%, or 1 in every 1000 requests)</li><li><strong>Warning</strong>: Do <strong>not</strong> enable this if you have a <strong>single API server</strong> or <strong>no load balancer</strong> in front ⚠️</li></ul><h4>🧪 Configuration Example</h4><p>Just add this flag to your kube-apiserver startup arguments:</p><pre>--goaway-chance=0.001</pre><h3>📊 Expected Result</h3><p>Once enabled, you’ll likely observe a <strong>balanced load</strong> across all your API servers. Here’s a simplified example of what this might look like:</p><p><strong>Before:</strong></p><ul><li>server 1: 70% load</li><li>server 2: 30%</li><li>server 3: 0%</li></ul><p><strong>After enabling </strong><strong>--goaway-chance:</strong></p><ul><li>server 1: 33%</li><li>server 2: 33%</li><li>server 3: 33%</li></ul><p>This helps prevent overloads and ensures better <strong>resource utilization</strong> across your control plane. ✅</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kauFaa8Yux34u9qD3Xk4A.png" /><figcaption>On the graph below, we can see the difference before and after applying the parameter</figcaption></figure><h3>🎯 Conclusion</h3><p>If you’re running multiple kube-apiserver instances behind a load balancer, the --goaway-chance parameter is an essential tool to ensure <strong>load distribution and cluster stability</strong>. 🔐</p><p>It’s easy to configure, has minimal impact, and can <strong>prevent costly downtimes</strong> or <strong>crashes during maintenance</strong>. ⏱️</p><p><strong>Don’t overlook it. 💪</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eca6c2ccd851" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Node Stability and Performance: Tuning Kubelet for Better Resource Management]]></title>
            <link>https://medium.com/@jfpucheu/kubernetes-node-stability-and-performance-tuning-kubelet-for-better-resource-management-e0f95ccfefe9?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0f95ccfefe9</guid>
            <category><![CDATA[kubeadm]]></category>
            <category><![CDATA[tuning]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubelet]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Fri, 30 May 2025 07:37:47 GMT</pubDate>
            <atom:updated>2025-05-30T07:37:47.469Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a practical post for Kubernetes operators and administrators looking to improve cluster resilience and performance by fine-tuning kubelet parameters. The focus is on preventing node crashes and optimizing resource usage, especially for clusters running in production environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*CeDRb9IUV-tPfz9F1waKOA.png" /></figure><h3>🚨 Why Tuning Matters</h3><p>In a Kubernetes cluster, <strong>nodes are the foundation</strong> — if a node goes down, all the workloads (pods) running on it are impacted. One common cause of node instability is poor resource management at the kubelet level. Without proper reservations and eviction policies, pods can consume all the system memory or CPU, leading to <strong>Out of Memory (OOM)</strong> errors or even system crashes.</p><p>This article covers two key areas of kubelet tuning:</p><ol><li><strong>Resource Reservations and Evictions</strong></li><li><strong>Graceful Node Shutdown Settings</strong></li></ol><p>The goal is to help you configure your nodes so they <strong>remain stable under load</strong>, <strong>avoid system-level OOMs</strong>, and <strong>terminate gracefully when needed</strong>, particularly on platforms like <strong>OpenStack</strong>.</p><h3>⚙️ 1. Reserve Resources to Protect the Node</h3><h4>❗ Problem</h4><p>By default, if you don’t reserve any resources for system or kubelet processes, pods can consume <strong>100% of a node’s memory or CPU</strong>. This can starve the system, cause critical services to crash, and render the node temporarily unusable.</p><h4>✅ Solution: Use kubeReserved, systemReserved, and evictionHard</h4><ul><li><strong>systemReserved</strong>: Resources set aside for system-level processes (e.g., systemd, journald).</li><li><strong>kubeReserved</strong>: Resources reserved for Kubernetes components like kubelet, container runtime, etc.</li><li><strong>evictionHard</strong>: Memory and storage thresholds at which kubelet starts evicting pods before the system runs out of resources completely.</li></ul><h4>🔧 Example Configuration</h4><pre>apiVersion: kubelet.config.k8s.io/v1beta1<br>kind: KubeletConfiguration<br>kubeReserved:<br>  cpu: 420m<br>  memory: 9Gi<br>systemReserved:<br>  cpu: 100m<br>  memory: 1Gi<br>evictionHard:<br>  memory.available: 100Mi<br>  nodefs.available: 10%<br>  imagefs.available: 15%<br>  nodefs.inodesFree: 5%</pre><blockquote>💡 These values can be adjusted based on your node specs (CPU cores, total memory). Below is a basic recommendation logic for automation tools like Ansible.</blockquote><pre>KubeletKubeReservedMemory: &gt;-<br>  {% if ansible_memtotal_mb &gt;= 256000 %}13Gi<br>  {% elif ansible_memtotal_mb &gt;= 128000 %}9Gi<br>  {% elif ansible_memtotal_mb &gt;= 64000 %}6Gi<br>  {% elif ansible_memtotal_mb &gt;= 31900 %}4Gi<br>  {% elif ansible_memtotal_mb &gt;= 16000 %}3Gi<br>  {% elif ansible_memtotal_mb &gt;= 8000 %}2Gi<br>  {% elif ansible_memtotal_mb &gt;= 4000 %}1Gi<br>  {% else %}255Mi<br>  {% endif %}<br>KubeletKubeReservedCpu: &gt;-<br>  {% if ansible_processor_vcpus &gt;= 64 %}740m<br>  {% elif ansible_processor_vcpus &gt;= 32 %}420m<br>  {% elif ansible_processor_vcpus &gt;= 16 %}260m<br>  {% elif ansible_processor_vcpus &gt;= 8 %}180m<br>  {% elif ansible_processor_vcpus &gt;= 4 %}140m<br>  {% elif ansible_processor_vcpus &gt;= 2 %}100m<br>  {% elif ansible_processor_vcpus &gt;= 1 %}60m<br>  {% else %}10m<br>  {% endif %</pre><pre>apiVersion: kubelet.config.k8s.io/v1beta1<br>kind: KubeletConfiguration<br>kubeReserved:<br>  cpu: {{ KubeletKubeReservedCpu }}<br>  memory: {{ KubeletKubeReservedMemory }}<br>systemReserved:<br>  cpu: 100m<br>  memory: 1Gi<br>evictionHard:<br>  memory.available: 100Mi<br>  nodefs.available: 10%<br>  imagefs.available: 15%<br>  nodefs.inodesFree: 5%</pre><blockquote>📘 Official Docs:<br> <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/">Kubernetes Resource Reservations</a><br> <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/">Reserve-compute-resources</a></blockquote><p>Part of this tuning could be enabled by default on images built with Image Builder. You can find the script <a href="https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/ansible/roles/kubernetes/files/usr/libexec/kubernetes/kubelet-resource-sizing.sh">here</a>.</p><h3>⏱️ 2. Configure Graceful Shutdown for Your Nodes</h3><h4>❗ Problem</h4><p>During system shutdown or reboot (planned or unplanned), nodes can terminate without properly shutting down running pods. This can result in <strong>data loss</strong>, <strong>application errors</strong>, and <strong>inconsistent states</strong>.</p><h4>✅ Solution: Enable and Tune shutdownGracePeriod</h4><p>Kubelet uses <strong>systemd inhibitor locks</strong> to delay the node shutdown and give time for pods to terminate gracefully. This feature improves application reliability, especially for stateful or critical services.</p><h4>🔧 Recommended Settings:</h4><pre>shutdownGracePeriod=60s<br>shutdownGracePeriodCriticalPods=30s</pre><p>This configuration:</p><ul><li>Reserves <strong>60 seconds</strong> to complete the node shutdown.</li><li>Gives <strong>30 seconds</strong> to gracefully shut down regular pods.</li><li>Keeps the last <strong>30 seconds</strong> for critical system pods.</li></ul><blockquote>📘 Official Docs:<br> <a href="https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/">Graceful Node Shutdown (K8s Docs)</a><br> <a href="https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/">Kubernetes Blog Post</a></blockquote><h3>✅ Final Thoughts</h3><p>Tuning your kubelet settings is a <strong>low-effort, high-impact</strong> improvement that can drastically increase the <strong>resilience</strong> and <strong>performance</strong> of your Kubernetes nodes. Especially in production environments or cloud platforms like <strong>OpenStack</strong>, it’s crucial to:</p><ul><li>Reserve resources for essential system components</li><li>Define eviction thresholds to avoid OOM errors</li><li>Gracefully shut down workloads to avoid data corruption</li></ul><p>By applying these best practices, you ensure that your nodes stay healthy and your applications remain available — even under heavy load or system shutdown events.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0f95ccfefe9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating a kubeadm Live Kubernetes Cluster to Cluster-api on Openstack]]></title>
            <link>https://medium.com/@jfpucheu/migrating-a-kubeadm-live-kubernetes-cluster-to-cluster-api-on-openstack-c6323014c133?source=rss-ca7365ab90b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6323014c133</guid>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[kubeadm]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openstack]]></category>
            <dc:creator><![CDATA[Jean-François PUCHEU]]></dc:creator>
            <pubDate>Sat, 17 May 2025 09:43:03 GMT</pubDate>
            <atom:updated>2025-08-15T12:18:48.074Z</atom:updated>
            <content:encoded><![CDATA[<p>Managing Kubernetes clusters manually or with custom scripts can become complex and time-consuming, especially as your infrastructure grows. Cluster API (CAPI) is a powerful open-source tool that helps simplify and standardize how Kubernetes clusters are created, updated, and managed — using Kubernetes-style APIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fv-JZ5oTET59M0JNMCPJFg.jpeg" /></figure><h3><strong>Important Notes :</strong></h3><p>Currently, this procedure is in an experimental stage and should be thoroughly tested before being used in a production environment. <strong>It is only compatible with an external ETCD and an external LBAAS.</strong></p><p>It is designed create cluster.x-k8s.io/secret bundle to migrate a cluster created with kubeadm to a kubeadm-based cluster managed by Cluster API. At this stage, the script has been developed <strong>specifically for the Cluster API Provider OpenStack (CAPO ) </strong>but can be inspiring for other providers.</p><p>My goal is to improve the process to transition from an external ETCD to a local ETCD on the control-plane nodes, and also to migrate from a local ETCD on legacy control-planes to a local ETCD on CAPI control-planes.</p><p>The key to this, based on my analysis, would be to force CAPI to add the first CAPI control-plane node using a kubeadm join instead of a kubeadm init.</p><p>In the case of an external ETCD, this works because the secrets and ETCD are already initialized. The kubeadm init command does not pose any issues, as the kubelet simply joins an already existing API endpoint.</p><p>Feel free to share any suggestions or ideas for improvements or future developments.</p><h3>Migration Process Overview</h3><p>The procedure is based on having a hybrid legacy/CAPI cluster during the migration.</p><p>It is carried out in five main steps:</p><ol><li><strong>Retrieving the necessary secrets and configurations</strong> from the existing cluster.</li><li><strong>Preparing the Cluster API (CAPI) configuration</strong>.</li><li><strong>Importing the secrets into CAPI</strong>.</li><li><strong>Create the CAPI control-plane and CAPI worker nodes</strong> on the existing cluster.</li><li><strong>Removing the nodes the old cluster nodes</strong>.</li></ol><h3>Step 1: Retrieving the necessary secrets and configurations.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*GCTw7mXdd30VS5OOo8RZyA.png" /></figure><p>First, run the <a href="https://github.com/jfpucheu/devops-notes/blob/main/src/capi/prepare_secrets.sh">prepare_secrets.sh</a> script on a legacy control plane node, passing the name of the cluster you want to migrate as an argument. This name should match the cluster_name defined in CAPI. You can find the script <a href="https://github.com/jfpucheu/devops-notes/blob/main/src/capi/prepare_secrets.sh">Here</a>.</p><p>The script will generate a file named ${CLUSTER_NAME}-secret-bundle.yaml.</p><pre>./prepare_secrets.sh ${CLUSTER_NAME}</pre><p>and get the file: ${CLUSTER_NAME}-secret-bundle.yaml</p><h3>Step 2: Preparing the Cluster API (CAPI) configuration.</h3><p><strong>Manual step: secret cloud.yaml</strong></p><p>Based on the file cloud.yaml , create your encoded secret cloud.yaml in base64.</p><pre>export OPENSTACK_CLOUD_YAML_B64=$(cat cloud.yaml | base64)<br><br># apply the secret in your Cluster-api cluster (run once )<br>envsubst &lt; secret.yaml | kubectl apply -f -</pre><p>based on env_example file create vars file for your cluster and source it:</p><pre>source env_example</pre><p>Now, based on the ‘cluster-template-migration.yaml’, generate your cluster configuration, and pay <strong>close attention to the following parameters</strong>:</p><p><em>For the example, I hardcoded the parameters directly in the code, but it’s recommended to pass these values </em><strong><em>as cluster environment variables instead</em></strong><em>.</em></p><ul><li><strong>External etcd endpoints in KubeadmControlPlane section :</strong></li></ul><pre>...<br>    clusterConfiguration:<br>      etcd:<br>        external:<br>          caFile: /etc/kubernetes/pki/etcd/ca.crt<br>          certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt<br>          keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key<br>          endpoints:<br>          - https://192.168.10.10:2379<br>          - https://192.168.10.11:2379<br>          - https://192.168.10.12:2379<br>    ...</pre><ul><li><strong>External Api endpoints in OpenStackCluster section if you use CAPO:</strong></li></ul><pre>...<br>    controlPlaneEndpoint:<br>    host: api.mydevcluster.com<br>    port: 443<br>    ...</pre><ul><li><strong>Add a secutitygroup rule in OpenStackCluster section if you use CAPO to allow traffic beween old and new clusters:</strong></li></ul><pre>...<br>  managedSecurityGroups:<br>    allNodesSecurityGroupRules:<br>    - direction: ingress<br>      etherType: IPv4<br>      name: Allow old secutirygroup<br>      description: &quot;Allow all between old and new control plane and workers&quot;<br>      remoteGroupID: &quot;old-secutity-group-id&quot;<br>  ...</pre><h3>Step 3: Importing the secrets into CAPI.</h3><p>Apply your ${CLUSTER_NAME}-secret-bundle.yaml into you CAPI Controller Cluster:</p><pre>kubectl apply -f ${CLUSTER_NAME}-secret-bundle.yaml</pre><p>CAPI will now detect that the Certificate Authority (CA) and related secrets already exist, and will skip generating new ones.”</p><h3>Step 4: Create the CAPI control-plane and CAPI worker nodes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*TIsfz5QG_rOSpUyFARFhrw.png" /></figure><pre># create capi  ${CLUSTER_NAME} cluster<br>envsubst &lt;  cluster-template-migration.yaml | kubectl apply -f</pre><p>Since the etcd database is shared between the old and new cluster, and the PKI secrets (such as TLS certificates and private keys) are identical, creating this new Kubernetes cluster will actually result in the addition of new control plane and CAPI worker nodes to the existing cluster, rather than forming a separate cluster.</p><p><strong>Get your cluster state:</strong></p><p>You will now see both your old control plane and worker nodes, as well as the new ones, in your cluster using kubectl get nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*l7Sa6F6pfFPZc4eLPBG_0w.png" /></figure><p>In your CAPI cluster, running the clusterctl command will show only the new nodes that are managed by CAPI:</p><pre><br># Get  ${CLUSTER_NAME} cluster<br>clusterctl get cluster ${CLUSTER_NAME}<br><br>NAME                                                    READY  SEVERITY  REASON  SINCE  MESSAGE<br>Cluster/dev                                             True                     22h<br>├─ClusterInfrastructure - OpenStackCluster/dev<br>├─ControlPlane - KubeadmControlPlane/dev-control-plane  True                     22h<br>│ └─3 Machines...                                       True                     22h    See dev-control-plane-2xjv4, dev-control-plane-lrt8m, ...<br>└─Workers<br>  ├─MachineDeployment/dev-az1                           True                     22h<br>  │ └─Machine/dev-az1-z6zr4-9dldr                       True                     22h<br>  ├─MachineDeployment/dev-az2                           True                     22h<br>  │ └─Machine/dev-az2-nx55k-s265c                       True                     22h<br>  └─MachineDeployment/dev-az3                           True                     22h<br>    └─Machine/dev-az3-95fng-hsqfv                       True                     22h</pre><h3>Step 5: Remove the old cluster nodes</h3><p>Make sure to update your load balancer (HAProxy or MetalLB) to include the new control plane nodes and remove the old ones.</p><p>Delete old nodes using:</p><pre># Delete old nodes<br>kubectl delete old-node</pre><p>Don’t forget to do the same for the old control plane nodes: stop containerd and kubelet to ensure that static pods are no longer running.” “And that’s it! You can now enjoy managing your cluster with CAPI 😊.</p><h3>Resources</h3><p><a href="https://cluster-api.sigs.k8s.io/">https://cluster-api.sigs.k8s.io/</a></p><p><a href="https://cluster-api-openstack.sigs.k8s.io/">https://cluster-api-openstack.sigs.k8s.io/</a></p><p><a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/">https://kubernetes.io/docs/reference/setup-tools/kubeadm/</a></p><p><a href="https://devops-notes.com/">https://github.com/jfpucheu/devops-notes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6323014c133" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>