<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nagarjuna Reddy on Medium]]></title>
        <description><![CDATA[Stories by Nagarjuna Reddy on Medium]]></description>
        <link>https://medium.com/@arjun0451?source=rss-6893836c0a25------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fzPplpIVZm0PBHtr6b8dQQ.png</url>
            <title>Stories by Nagarjuna Reddy on Medium</title>
            <link>https://medium.com/@arjun0451?source=rss-6893836c0a25------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:19:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arjun0451/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MetalLB Layer 2 Monitoring Dashboard with Grafana and Prometheus]]></title>
            <link>https://medium.com/@arjun0451/metallb-layer-2-monitoring-dashboard-with-grafana-and-prometheus-da42cbf0dcdf?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/da42cbf0dcdf</guid>
            <category><![CDATA[metallb]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[l2-mode]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 13:46:01 GMT</pubDate>
            <atom:updated>2026-06-30T13:46:01.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern OpenShift environments rely heavily on LoadBalancer services to expose applications outside the cluster. In cloud platforms, this functionality is usually provided by the cloud provider. In on-premises, edge, or bare-metal environments, MetalLB fills that gap by assigning external IP addresses and advertising them on the network.</p><p>This article focuses on <strong>MetalLB Layer 2 mode</strong>, one of the most widely used deployment models in OpenShift. While MetalLB is relatively easy to deploy, troubleshooting network issues can be difficult without the right visibility. Problems such as IP pool exhaustion, failed ARP responses, uneven VIP ownership, or repeated failovers often remain unnoticed until users begin reporting connectivity issues.</p><p>The Grafana dashboard presented in this article was designed to provide operational visibility into MetalLB rather than simply displaying metrics. It combines controller, speaker, and networking metrics into a single view, making it easier to understand how MetalLB is behaving in real time and quickly identify potential issues.</p><h3>Why Monitor MetalLB Layer 2?</h3><p>Unlike BGP mode, where multiple nodes can advertise routes simultaneously, Layer 2 mode assigns each Virtual IP (VIP) to a single node at a time. That node is responsible for answering ARP requests (IPv4) or NDP requests (IPv6) from the network.</p><p>When the active node becomes unavailable, another MetalLB speaker takes ownership of the VIP and sends Gratuitous ARP or unsolicited Neighbor Advertisements so that switches and clients update their MAC address tables without waiting for cache expiration.</p><p>Although this process is automatic, administrators often need answers to questions such as:</p><ul><li>Is every speaker running the latest configuration?</li><li>Which node currently owns a VIP?</li><li>Are ARP requests being answered successfully?</li><li>Is a failover expected or is the cluster unstable?</li><li>Is the IP pool close to running out of addresses?</li></ul><p>These operational questions cannot be answered by checking pod status alone.</p><h3>Understanding the Layer 2 Architecture</h3><p><strong>MetalLB Layer 2 Architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BhSKE27kFi6BzwE-YhsSPQ.png" /></figure><p>In a Layer 2 deployment, the <strong>MetalLB Controller</strong> manages IP address allocation and watches Kubernetes resources such as IPAddressPool and Service objects. It decides which external IP should be assigned but does not participate in network advertisement.</p><p>The <strong>MetalLB Speaker</strong>, running on each eligible node, is responsible for advertising VIPs on the network. Only one speaker owns a VIP at any given time and responds to ARP or NDP requests from external clients.</p><p>When leadership changes because of maintenance or a node failure, the new speaker immediately sends Gratuitous ARP or Neighbor Advertisements. This updates the network’s MAC address tables and redirects traffic to the new owner with minimal interruption.</p><h3>Common Operational Challenges</h3><p>Even in healthy clusters, several conditions can affect application availability if they are not detected early.</p><h3>IP Pool Exhaustion</h3><p>Every LoadBalancer service consumes an IP address from a configured pool. Once the pool is exhausted, new services remain in a Pending state until additional addresses are made available.</p><p>Monitoring pool utilization helps identify capacity issues before deployments are affected.</p><h3>Uneven VIP Distribution</h3><p>Ideally, VIP ownership should be reasonably balanced across available nodes. If one node owns significantly more VIPs than others, it becomes a larger failure domain. A single node outage could trigger multiple simultaneous failovers, increasing recovery time and network activity.</p><h3>Configuration Synchronization</h3><p>All MetalLB speakers should operate with the same configuration. If one or more speakers fail to load the latest configuration, VIP ownership and failover behavior may become inconsistent.</p><h3>Layer 2 Communication</h3><p>Every external connection begins with an ARP or NDP request. If those requests are not answered, applications may become unreachable even though Kubernetes reports all pods and services as healthy.</p><p>Monitoring Layer 2 traffic provides an early indication of these issues.</p><h3>Failover Stability</h3><p>Occasional Gratuitous ARP messages are expected during upgrades or planned maintenance. However, frequent or continuous announcements may indicate repeated leader changes, unstable nodes, or network problems that require investigation.</p><h3>Dashboard Design Philosophy</h3><p>This dashboard was built around real operational scenarios rather than simply exposing Prometheus metrics.</p><p>Instead of asking, “What metric should be displayed?”, the focus was on questions administrators ask during incidents.</p><ul><li>Are all speakers synchronized?</li><li>Is the IP pool healthy?</li><li>Which node currently owns each VIP?</li><li>Is MetalLB responding to every ARP request?</li><li>Are failovers occurring as expected?</li><li>Is one node handling significantly more services than others?</li></ul><p>Each panel answers one of these questions, allowing operators to move quickly from detection to diagnosis without switching between multiple tools.</p><h3>Dashboard Overview</h3><p><strong>MetalLB Layer 2 Dashboard</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3a3gvU3Yzfx_T96UHWo2g.png" /></figure><p>The dashboard is organized into logical sections that follow the lifecycle of MetalLB operations:</p><ul><li><strong>Cluster Status</strong> validates speaker configuration and overall allocator health.</li><li><strong>Pool Health</strong> tracks IP usage, utilization, and remaining capacity.</li><li><strong>Leadership Distribution</strong> shows how VIP ownership is spread across nodes.</li><li><strong>VIP Mapping</strong> provides a live view of which node and speaker currently advertise each service.</li><li><strong>Layer 2 Networking</strong> monitors ARP/NDP requests, responses, and dropped requests.</li><li><strong>Failover Analysis</strong> tracks Gratuitous ARP activity and identifies unstable VIPs or excessive leadership changes.</li></ul><p>By combining these views into a single dashboard, administrators can monitor capacity, networking behavior, and failover activity without manually correlating multiple metrics.</p><h3>Dashboard Walkthrough</h3><p>The dashboard is organized to follow the operational flow of MetalLB. It starts with overall cluster health, moves into IP address management, shows how VIP ownership is distributed, and finally provides visibility into Layer 2 network activity and failover events.</p><p>Rather than looking at individual metrics in isolation, the dashboard brings together related information so that potential issues can be identified quickly.</p><h3>Cluster Status</h3><p><strong>Cluster Status</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x6DSbXqz6KkPQHuSvXRQ6Q.png" /></figure><p>The first section provides a quick health check of the MetalLB deployment. It focuses on speaker configuration and IP pool utilization, making it the first place to look during routine health checks or incident investigations.</p><h3>Speaker Configuration Status</h3><p>The first two panels monitor whether every MetalLB speaker is running the latest configuration.</p><h3>Speakers Config Loaded</h3><p><strong>Metric</strong></p><pre>sum(metallb_k8s_client_config_loaded_bool)</pre><p>This panel shows how many speaker instances have successfully loaded the current MetalLB configuration.</p><p>A healthy cluster should have every speaker reporting a loaded configuration. If the value is lower than the number of speaker pods, one or more nodes may not be advertising services correctly.</p><p>Possible causes include:</p><ul><li>Invalid MetalLB configuration</li><li>Failed configuration reload</li><li>Communication issues between the controller and speaker</li></ul><h3>Speakers Config Stale</h3><p><strong>Metric</strong></p><pre>sum(metallb_k8s_client_config_stale_bool)</pre><p>This panel highlights speakers still using an outdated configuration.</p><p>The expected value is always <strong>0</strong>.</p><p>A non-zero value indicates that at least one node has not applied the latest configuration changes. Although the cluster may continue serving traffic, future VIP assignments or failovers can behave unexpectedly until the configuration is synchronized.</p><h3>IP Pool Capacity</h3><p>The remaining panels provide an overview of IP address allocation and available capacity.</p><h3>IPs In Use</h3><p><strong>Metric</strong></p><pre>sum(metallb_allocator_addresses_in_use_total)</pre><p>Displays the number of addresses currently assigned to LoadBalancer services.</p><p>This metric helps track growth over time and provides a quick indication of how much of the available address space is being consumed.</p><h3>Total Pool IPs</h3><p><strong>Metric</strong></p><pre>sum(metallb_allocator_addresses_total)</pre><p>Shows the total number of IP addresses available across all configured address pools.</p><p>This value changes only when IP pools are expanded or modified.</p><h3>Pool Utilization</h3><p><strong>Metric</strong></p><pre>100 * sum(metallb_allocator_addresses_in_use_total)<br>/<br>sum(metallb_allocator_addresses_total)</pre><p>This panel calculates overall pool utilization as a percentage.</p><p>Recommended thresholds:</p><p>UtilizationStatusBelow 70%Healthy70–90%Monitor capacityAbove 90%Expand IP pools</p><p>High utilization doesn’t immediately impact running services, but it prevents new LoadBalancer services from receiving external IP addresses once the pool is exhausted.</p><h3>Free IPs Remaining</h3><p>This panel shows the number of available addresses left in the configured pools.</p><p>Monitoring this value is often more useful than watching utilization percentages because it provides a clear indication of remaining capacity.</p><p>For example:</p><ul><li>200 free IPs indicate plenty of capacity.</li><li>10 free IPs suggest planning for expansion.</li><li>0 free IPs mean new LoadBalancer services will remain in the Pending state.</li></ul><h3>Leadership Distribution</h3><p><strong>VIP Leadership Distribution</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZnw6lNyZeAXBReUzcRPOQ.png" /></figure><p>One of the most useful sections of the dashboard shows how MetalLB distributes VIP ownership across cluster nodes.</p><p>Only one speaker advertises a VIP in Layer 2 mode, making leadership distribution an important operational metric.</p><h3>Active Services Per Node</h3><p><strong>Metric</strong></p><pre>sort_desc(sum by (node) (metallb_speaker_announced))</pre><p>This bar chart displays the number of services currently announced by each node.</p><p>A balanced environment should show a relatively even distribution of VIP ownership.</p><p>This indicates that leadership is spread evenly across the cluster.</p><p>An imbalanced distribution might look like this:</p><p>Node VIPs</p><p>worker-1 42</p><p>worker-2 6</p><p>worker-3 5</p><p>Although applications continue to function normally, a failure of <strong>worker-1</strong> would require dozens of VIPs to move simultaneously, increasing network convergence time and creating a larger failure domain.</p><p>This panel makes those situations immediately visible.</p><h3>Live Node-to-Service Mapping</h3><p><strong>Live VIP Ownership</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s2ubtB8UGFSvYRVSjb-uYA.png" /></figure><p>During troubleshooting, one of the most common questions is:</p><blockquote>Which node currently owns this LoadBalancer IP?</blockquote><p>The <strong>Node → Speaker Pod → VIP → Service</strong> table answers this instantly.</p><p>Each row shows:</p><ul><li>Physical node</li><li>MetalLB speaker pod</li><li>Advertised VIP</li><li>Kubernetes Service</li><li>Advertisement protocol</li></ul><p>Instead of searching through Kubernetes resources or MetalLB logs, administrators can quickly identify the current owner of any VIP.</p><p>This view is especially useful during:</p><ul><li>Planned maintenance</li><li>Node failures</li><li>Failover verification</li><li>Service migration</li><li>Network troubleshooting</li></ul><p>It also helps confirm that VIP ownership matches expectations after upgrades or cluster changes.</p><h3>Why These Panels Matter Together</h3><p>Although each panel provides valuable information on its own, their real strength comes from being viewed together.</p><p>For example, imagine a new application cannot obtain an external IP address.</p><p>The dashboard immediately provides a logical troubleshooting path:</p><ol><li>Check <strong>Pool Utilization</strong> and <strong>Free IPs Remaining</strong> to verify address availability.</li><li>Confirm all speakers have successfully loaded the latest configuration.</li><li>Ensure no speakers are reporting stale configurations.</li><li>Verify VIP ownership using the live mapping table.</li><li>Confirm leadership is balanced across the cluster.</li></ol><p>Within a few minutes, operators can determine whether the issue is related to capacity, configuration, or service advertisement without manually querying multiple Kubernetes resources.</p><h3>ARP/NDP Request Monitoring</h3><p><strong>ARP/NDP Requests Received</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oD5GO6yEkKH7s9CO1JbBzg.png" /></figure><p>Whenever a client connects to a LoadBalancer service, it first needs to determine which MAC address owns the Virtual IP. For IPv4, this is done using the <strong>Address Resolution Protocol (ARP)</strong>, while IPv6 uses the <strong>Neighbor Discovery Protocol (NDP)</strong>.</p><p>The <strong>ARP/NDP Requests Received</strong> panel displays the rate at which MetalLB receives these requests for each advertised VIP.</p><p><strong>Metric</strong></p><pre>rate(metallb_layer2_requests_received[$__rate_interval])</pre><h3>How to Interpret the Panel</h3><p>A low and steady request rate is expected in most environments because clients and switches cache MAC address information. Once cached, they do not need to send repeated ARP or NDP requests.</p><p>Higher request rates may occur during:</p><ul><li>Client reconnects</li><li>Switch or router cache expiration</li><li>Application deployments</li><li>Network topology changes</li></ul><p>A sudden and sustained increase across multiple VIPs may indicate excessive ARP traffic or instability in the surrounding network.</p><h3>ARP/NDP Response Monitoring</h3><p><strong>ARP/NDP Responses Sent</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXVwNB4H_bgsgBvbOLSerQ.png" /></figure><p>Receiving ARP or NDP requests is only half the picture. MetalLB must also respond to those requests so clients can resolve the VIP and establish connections.</p><p>This panel tracks the response rate for every advertised VIP.</p><p><strong>Metric</strong></p><pre>rate(metallb_layer2_responses_sent[$__rate_interval])</pre><h3>What Healthy Looks Like</h3><p>Under normal conditions, the request and response graphs should closely match.</p><p>For every ARP request received, there should be a corresponding response from the active MetalLB speaker.</p><p>If response rates consistently fall below request rates, external clients may fail to reach the service even though Kubernetes reports everything as healthy.</p><p>Possible causes include:</p><ul><li>Speaker failures</li><li>Network interruptions</li><li>Incorrect Layer 2 advertisements</li><li>Configuration issues</li></ul><p>Comparing the request and response panels side by side provides a quick way to verify that MetalLB is actively responding to network discovery traffic.</p><h3>Dropped Request Detection</h3><p><strong>Dropped Layer 2 Requests</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeHBrZSFamLIqo3EqioKCQ.png" /></figure><p>One of the most valuable panels in the dashboard measures the difference between incoming requests and outgoing responses.</p><p><strong>Metric</strong></p><pre>sum(rate(metallb_layer2_requests_received[$__rate_interval))<br>-<br>sum(rate(metallb_layer2_responses_sent[$__rate_interval))</pre><p>The expected value is always <strong>zero</strong>.</p><p>A value above zero means MetalLB is receiving discovery requests but is not responding to all of them.</p><p>Even a small sustained increase deserves investigation because unanswered ARP or NDP requests can prevent external clients from reaching applications.</p><p>This panel serves as an excellent candidate for alerting, helping identify network issues before they become widespread.</p><h3>Gratuitous ARP/NDP Monitoring</h3><p><strong>Gratuitous ARP Activity</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvMc-MDZoZcLTr6hp5LE6g.png" /></figure><p>When VIP ownership changes, MetalLB sends <strong>Gratuitous ARP</strong> (IPv4) or <strong>unsolicited Neighbor Advertisements</strong> (IPv6). These packets notify switches and connected devices that the VIP is now owned by a different node.</p><p>The dashboard tracks these announcements for every VIP.</p><p><strong>Metric</strong></p><pre>rate(metallb_layer2_gratuitous_sent[$__rate_interval])</pre><h3>Expected Behavior</h3><p>Small background activity is normal and helps refresh network forwarding tables.</p><p>Short spikes are also expected during:</p><ul><li>Planned maintenance</li><li>Node reboots</li><li>Cluster upgrades</li><li>VIP migration</li></ul><p>These events typically settle within a few seconds.</p><p>Continuous or repetitive spikes, however, may indicate repeated leadership changes or an unstable environment that requires further investigation.</p><h3>Leadership Failover Analysis</h3><p><strong>Gratuitous ARP by Node</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DoRdF3Xl4GE9NlO3ksuHgw.png" /></figure><p>Understanding which node is generating Gratuitous ARP announcements makes it much easier to verify failover behavior.</p><p>This panel joins MetalLB metrics with Kubernetes node information to display Gratuitous ARP activity per node.</p><p><strong>Metric</strong></p><pre>sum by (node)(<br>rate(metallb_layer2_gratuitous_sent[$__rate_interval])<br>* on(pod) group_left(node)<br>kube_pod_info{namespace=&quot;metallb-system&quot;}<br>)</pre><h3>Typical Scenarios</h3><p><strong>Normal operation</strong></p><p>All nodes show minimal background activity.</p><p><strong>Planned maintenance</strong></p><p>The original owner stops advertising the VIP, while another node briefly spikes as it announces the new ownership.</p><p><strong>Unexpected failovers</strong></p><p>Frequent spikes moving between multiple nodes often indicate repeated leader elections or node instability.</p><p>Viewing this panel alongside the VIP ownership table provides a clear picture of how leadership changes are occurring within the cluster.</p><h3>Measuring Failover Stability</h3><p>Not every Gratuitous ARP packet indicates a problem. During quiet periods, there may be very little client traffic, making simple percentage calculations misleading.</p><p>To avoid false alarms, the dashboard calculates the <strong>Gratuitous-to-Request ratio</strong> only when there is sufficient network activity.</p><p><strong>Metric</strong></p><pre>(100 *<br>rate(metallb_layer2_gratuitous_sent[$__rate_interval])<br>/<br>rate(metallb_layer2_requests_received[$__rate_interval]))<br>and<br>(rate(metallb_layer2_requests_received[$__rate_interval]) &gt; 0.01)</pre><p>This approach prevents inflated percentages during periods of minimal traffic while still highlighting excessive Gratuitous ARP activity when clients are actively accessing services.</p><p>Sustained high ratios under normal traffic conditions may indicate unstable VIP ownership or repeated failovers.</p><h3>Identifying Unstable VIPs</h3><p><strong>Failover Severity Ranking</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1Cgp79q7ZDyueqPHse02w.png" /></figure><p>The final panel ranks VIPs based on the cumulative number of Gratuitous ARP announcements.</p><p><strong>Metric</strong></p><pre>sort_desc(metallb_layer2_gratuitous_sent)</pre><p>This view makes it easy to identify services experiencing significantly more failover activity than others.</p><p>Interpreting the data requires considering the observation period.</p><ul><li>A high value accumulated over several weeks is generally expected.</li><li>A rapidly increasing counter over minutes or hours may indicate repeated failovers, unstable nodes, or networking issues.</li></ul><p>Combined with the previous panels, this ranking helps narrow investigations to the services experiencing the greatest amount of Layer 2 activity.</p><h3>Bringing the Metrics Together</h3><p>Each networking panel provides a different view of MetalLB’s behavior, but together they form a complete operational picture.</p><p>A healthy environment typically exhibits the following characteristics:</p><ul><li>ARP and NDP request rates remain stable.</li><li>Responses closely match incoming requests.</li><li>Dropped requests stay at zero.</li><li>Gratuitous ARP activity is minimal during normal operation.</li><li>Brief spikes occur only during planned failovers.</li><li>VIP ownership remains stable across nodes.</li></ul><p>When one or more of these patterns changes, the dashboard helps determine whether the issue is related to client discovery, Layer 2 communication, VIP ownership, or failover activity.</p><p>By combining these metrics into a single view, administrators can quickly validate network health and troubleshoot MetalLB Layer 2 behavior without relying solely on logs or packet captures.</p><p>Get the dashboard manifest from the GitHub : <a href="https://raw.githubusercontent.com/arjun0451/ocp-sysops-kit/refs/heads/main/Monitoring/metallb-monitoring-l2-mode/metallb-l2-dashboard.json"><strong>https://raw.githubusercontent.com/arjun0451/ocp-sysops-kit/refs/heads/main/Monitoring/metallb-monitoring-l2-mode/metallb-l2-dashboard.json</strong></a></p><h3>Conclusion</h3><p>Monitoring MetalLB extends beyond checking whether pods are running or IP addresses have been allocated. Reliable Layer 2 networking depends on healthy ARP and NDP communication, balanced VIP ownership, synchronized speaker configuration, and predictable failover behavior.</p><p>This dashboard combines these operational insights into a single Grafana view, enabling faster troubleshooting, better capacity planning, and improved visibility into how MetalLB behaves in production. Whether investigating connectivity issues, validating failovers during maintenance, or monitoring long-term network stability, these metrics provide the context needed to keep LoadBalancer services reliable and available.</p><blockquote><strong><em>Disclaimer</em></strong><em>:</em></blockquote><blockquote><em>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners</em></blockquote><p>LinkedIn: <a href="https://www.linkedin.com/in/nnreddy51/">https://www.linkedin.com/in/nnreddy51/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da42cbf0dcdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When the Network Remembers the Wrong Node]]></title>
            <link>https://medium.com/@arjun0451/when-the-network-remembers-the-wrong-node-04ed7db914b7?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/04ed7db914b7</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[ovn]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[metallb]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 10:56:01 GMT</pubDate>
            <atom:updated>2026-04-11T10:56:01.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_AmHGytZ_WNCDBZo" /><figcaption>Photo by <a href="https://unsplash.com/@2hmedia?utm_source=medium&amp;utm_medium=referral">2H Media</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Everything Worked. And Yet, It Did Not</strong></p><p>On the day we upgraded the MetalLB Operator on our OpenShift cluster, something unexpected happened. The upgrade completed without errors. The VIP ownership transferred between worker nodes exactly as designed. The MetalLB controller logs showed clean failover. But users were still hitting errors. Traffic was being dropped. Applications were unreachable randomly</p><p>We Observed the issue with two services and the following chain will explain more details of the problem statement step by step.</p><p>These are two Virtual IPs (VIPs) were involved: 192.115.97.141 and 192.115.97.131. Both are advertised via MetalLB in L2 mode — meaning one worker node at a time claims ownership of each VIP by responding to ARP requests with its MAC address. When ownership moves, the new node must make that change known to the network immediately.</p><blockquote>L2 mode MetalLB works by making one node ‘own’ the VIP at the ARP level. Every device on the network that wants to reach the VIP sends traffic to the MAC address that last responded to an ARP request for that IP. If the MAC changes and the network does not update — traffic goes to the wrong place.</blockquote><h4>The failure chain in plain terms</h4><p>During the MetalLB speaker restart that is part of any operator upgrade:</p><ul><li>Node A<strong> </strong>held ownership of both VIPs before the upgrade.</li><li>The speaker pod on Node A restarted. VIP ownership transferred to Node B.</li><li>Node B sent a Gratuitous ARP (GARP) — a broadcast announcement saying ‘I now own this IP, update your records’.</li><li>The switch did not honour the GARP on the VIP VLAN.</li><li>The switch ARP cache still pointed to Node A’s MAC. For up to 30 minutes, all traffic for the VIPs was forwarded to a node that no longer owned them.</li><li>Node A’s speaker was no longer listening. Traffic was silently dropped.</li></ul><h4><strong>How MetalLB L2 Mode Actually Works</strong></h4><p>Before walking through more investigation workflow, it helps to understand what MetalLB is actually doing at the wire level in L2 mode — because the failure only makes sense once you understand the mechanism.</p><p>When you create a Kubernetes Service of type LoadBalancer and MetalLB assigns an external IP from a pool, one of the MetalLB speaker pods — running on one of your worker nodes — becomes the ‘owner’ of that IP. Ownership here has a very specific, physical meaning: that node will respond to ARP requests for the VIP.</p><p>ARP (Address Resolution Protocol) is how devices on a local network translate an IP address into a MAC address — the hardware identifier of a network interface. When any device on your network wants to send traffic to the VIP, it first sends an ARP broadcast: ‘Who has 192.115.97.141? Tell me your MAC.’ The MetalLB speaker node replies: ‘That is me, here is my MAC address.’</p><p>From that point, the switch (and any other device on the segment) caches this IP-to-MAC mapping. All traffic for the VIP is forwarded directly to that node’s MAC address.</p><h3>What happens during a failover</h3><p>When the MetalLB speaker on Node A restarts — as happens during an operator upgrade — it temporarily stops responding to ARP requests for the VIPs it owned. Another speaker node detects the absence and takes ownership. The new owner sends a Gratuitous ARP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Umk4FYlLJ6-a13Vk-wd_6w.png" /></figure><p>A Gratuitous ARP is an unsolicited ARP reply — a broadcast that says ‘this IP now belongs to this MAC, update your caches.’ It is the standard mechanism for fast failover in L2 environments. Every switch and host that receives it should immediately update their ARP table.</p><p><strong>The problem: </strong>‘Should’ is not ‘will’. Whether a device accepts a GARP depends on its configuration. In our case, the switch had GARP processing disabled on the VIP VLAN.</p><p><strong>Investigation: Building the Evidence Chain</strong></p><p><em>You Cannot Debug What You Cannot Measure</em></p><p>The first principle we established before touching anything: we needed continuous, timestamped evidence from multiple vantage points simultaneously. A single observation from a single location is not enough to prove causation — especially when the window of failure could be seconds wide.</p><p>We identified three layers to monitor in parallel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5C4nKYL9IZjxILIFJCD4A.png" /></figure><h4>Why the bastion and not the client for ARP?</h4><p>This distinction matters enormously and is worth dwelling on. Our application clients live on a different subnet than the VIPs. This means:</p><ul><li>Clients never ARP for the VIP directly.<strong> </strong>They ARP for their default gateway instead. The gateway then routes traffic toward the VIP subnet.</li><li>Client ARP tables tell you nothing about VIP ownership. They only tell you about the gateway’s MAC.</li></ul><p>The bastion host sits on the same subnet as the VIPs<strong>. </strong>It ARPs for the VIP directly. It sees the real MAC of whichever node currently owns the VIP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UkwV42-ikBOrn5354m4_Fw.png" /></figure><h4>Script 1: ARP Cache Monitor</h4><p>Run from the bastion host. Reads the kernel ARP table every second and logs the MAC and state for each VIP. This shows when the cache updates — or fails to update — after a failover.</p><pre>#!/usr/bin/env bash<br># arp_watch.sh — monitors kernel ARP cache for VIP entries<br># Usage: ./arp_watch.sh 192.115.97.141 192.115.97.131<br><br>VIPS=(&quot;$@&quot;)<br>if [ ${#VIPS[@]} -eq 0 ]; then<br>  echo &quot;Usage: $0 &lt;vip1&gt; &lt;vip2&gt; ...&quot;<br>  exit 1<br>fi<br><br>while true; do<br>  TS=$(date +&quot;%Y-%m-%d %H:%M:%S.%3N&quot;)<br>  for VIP in &quot;${VIPS[@]}&quot;; do<br>    ENTRY=$(ip neigh show &quot;$VIP&quot; 2&gt;/dev/null)<br>    if [[ -z &quot;$ENTRY&quot; ]]; then<br>      echo &quot;$TS | ARP | $VIP | NO_ENTRY&quot;<br>    else<br>      MAC=$(echo &quot;$ENTRY&quot; | awk &#39;{print $5}&#39;)<br>      STATE=$(echo &quot;$ENTRY&quot; | awk &#39;{print $6}&#39;)<br>      echo &quot;$TS | ARP | $VIP | MAC=$MAC | STATE=$STATE&quot;<br>    fi<br>  done<br>  sleep 1<br>done</pre><p>The STATE field from ip neigh is critical — it tells you whether the entry is REACHABLE (recently confirmed), STALE (not confirmed but still being used), DELAY (about to be re-probed), or FAILED.</p><p><strong>Script 2: Live ARP Prober (arping)</strong></p><p>Run from the bastion host. Actively sends ARP requests for each VIP every second and records the MAC that replies. This is the ground truth — whatever MAC responds here is the actual current owner of the VIP at the wire level.</p><pre>#!/usr/bin/env bash<br># arping_watch.sh — active ARP probe for real-time VIP owner detection<br># Usage: INTERFACE=eth0 ./arping_watch.sh 192.115.97.42 192.115.97.38<br><br>INTERFACE=&quot;${INTERFACE:-ens3}&quot;<br>VIPS=(&quot;$@&quot;)<br>if [ ${#VIPS[@]} -eq 0 ]; then<br>  echo &quot;Usage: INTERFACE=eth0 $0 &lt;vip1&gt; &lt;vip2&gt; ...&quot;<br>  exit 1<br>fi<br><br>while true; do<br>  TS=$(date +&quot;%Y-%m-%d %H:%M:%S.%3N&quot;)<br>  for VIP in &quot;${VIPS[@]}&quot;; do<br>    OUTPUT=$(arping -c 1 -I &quot;$INTERFACE&quot; &quot;$VIP&quot; 2&gt;/dev/null)<br>    if echo &quot;$OUTPUT&quot; | grep -q &quot;Unicast reply&quot;; then<br>      MAC=$(echo &quot;$OUTPUT&quot; | awk -F&#39;[][]&#39; &#39;/reply/ {print $2}&#39;)<br>      echo &quot;$TS | ARPING | $VIP | MAC=$MAC&quot;<br>    else<br>      echo &quot;$TS | ARPING | $VIP | NO_REPLY&quot;<br>    fi<br>  done<br>  sleep 1<br>done</pre><p><strong>Script 3: Continuous Application Traffic Test</strong></p><p>Run from the application client on a different subnet. Fires parallel HTTP requests continuously and logs the response code and latency with millisecond timestamps. This tells you exactly when users would have experienced failures.</p><pre>#!/usr/bin/env bash<br># curl_test.sh — continuous HTTP traffic test with timestamped output<br># Usage: ./curl_test.sh &lt;VIP&gt; &lt;PORT&gt; [PARALLEL] [INTERVAL]<br><br>VIP=&quot;${1}&quot;<br>PORT=&quot;${2:-80}&quot;<br>PARALLEL=&quot;${3:-3}&quot;<br>INTERVAL=&quot;${4:-0.2}&quot;<br><br>if [ -z &quot;$VIP&quot; ]; then<br>  echo &quot;Usage: $0 &lt;VIP&gt; &lt;PORT&gt; [PARALLEL] [INTERVAL]&quot;<br>  exit 1<br>fi<br><br>URL=&quot;http://${VIP}:${PORT}&quot;<br><br>run_request() {<br>  TS=$(date +&quot;%Y-%m-%d %H:%M:%S.%3N&quot;)<br>  RESULT=$(curl -s -o /dev/null -w &quot;%{http_code} %{time_total}&quot; --max-time 5 &quot;$URL&quot;)<br>  CODE=$(echo &quot;$RESULT&quot; | awk &#39;{print $1}&#39;)<br>  TIME=$(echo &quot;$RESULT&quot; | awk &#39;{print $2}&#39;)<br>  echo &quot;$TS | CURL | $VIP | HTTP=$CODE | TIME=${TIME}s&quot;<br>}<br><br>while true; do<br>  for ((i=1; i&lt;=PARALLEL; i++)); do<br>    run_request &amp;<br>  done<br>  wait<br>  sleep &quot;$INTERVAL&quot;<br>done</pre><p>Script-4: Take the Backup of the servicel2status to check the VIP and MAC mapping in the OCP cluster and compare the mapping matching with switch level as reference.</p><pre>#!/usr/bin/env bash<br># ============================================================<br>#  MetalLB L2 Service → Node → MAC Mapping<br>#  Maps every servicel2statuses entry to its advertising node<br>#  and the node&#39;s physical MAC (from k8s.ovn.org/l3-gateway-config)<br>#<br>#  Usage: ./metallb_l2_map.sh<br>#  Prerequisites: oc CLI logged in with cluster-admin<br># ============================================================<br><br>RED=&#39;\033[0;31m&#39;<br>GREEN=&#39;\033[0;32m&#39;<br>CYAN=&#39;\033[0;36m&#39;<br>YELLOW=&#39;\033[1;33m&#39;<br>RESET=&#39;\033[0m&#39;<br><br>TMP_L2=/tmp/metallb_l2status_$$.json<br>TMP_NODES=/tmp/metallb_nodes_$$.json<br>TMP_SVC=/tmp/metallb_svc_$$.json<br>TMP_PY=/tmp/metallb_map_$$.py<br>trap &quot;rm -f $TMP_L2 $TMP_NODES $TMP_SVC $TMP_PY&quot; EXIT<br><br># ── Fetch servicel2statuses ────────────────────────────────<br>echo -e &quot;${CYAN}Fetching servicel2statuses...${RESET}&quot;<br>oc get servicel2statuses.metallb.io -n metallb-system -o json &gt; &quot;$TMP_L2&quot; 2&gt;/dev/null || {<br>  echo -e &quot;${RED}ERROR: Could not fetch servicel2statuses${RESET}&quot;<br>  exit 1<br>}<br><br>COUNT=$(python3 -c &quot;import json; print(len(json.load(open(&#39;$TMP_L2&#39;))[&#39;items&#39;]))&quot;)<br>if [[ &quot;$COUNT&quot; -eq 0 ]]; then<br>  echo -e &quot;${YELLOW}No servicel2statuses entries found${RESET}&quot;<br>  exit 0<br>fi<br>echo -e &quot;${GREEN}Found $COUNT L2 service(s)${RESET}&quot;<br><br># ── Fetch all LoadBalancer services across all namespaces ─<br>echo -e &quot;${CYAN}Fetching LoadBalancer services...${RESET}&quot;<br>oc get svc -A -o json 2&gt;/dev/null | python3 -c &quot;<br>import sys, json<br>data = json.load(sys.stdin)<br>out = {}<br>for s in data.get(&#39;items&#39;, []):<br>    if s[&#39;spec&#39;].get(&#39;type&#39;) != &#39;LoadBalancer&#39;:<br>        continue<br>    ns  = s[&#39;metadata&#39;][&#39;namespace&#39;]<br>    nm  = s[&#39;metadata&#39;][&#39;name&#39;]<br>    ing = s.get(&#39;status&#39;, {}).get(&#39;loadBalancer&#39;, {}).get(&#39;ingress&#39;, [])<br>    eip = ing[0].get(&#39;ip&#39;, &#39;Pending&#39;) if ing else &#39;Pending&#39;<br>    out[f&#39;{ns}/{nm}&#39;] = eip<br>import json as j; print(j.dumps(out))<br>&quot; &gt; &quot;$TMP_SVC&quot;<br><br># ── Fetch nodes ────────────────────────────────────────────<br>echo -e &quot;${CYAN}Fetching node annotations...${RESET}&quot;<br>oc get nodes -o json &gt; &quot;$TMP_NODES&quot; 2&gt;/dev/null || {<br>  echo -e &quot;${RED}ERROR: Could not fetch nodes${RESET}&quot;<br>  exit 1<br>}<br><br># ── Write python script to tempfile — avoids heredoc JSON quoting issues ──<br>cat &gt; &quot;$TMP_PY&quot; &lt;&lt; &#39;PYEOF&#39;<br>import json, sys<br><br>l2data   = json.load(open(sys.argv[1]))<br>nodedata = json.load(open(sys.argv[2]))<br>svc_data = json.load(open(sys.argv[3]))<br><br>node_mac  = {}<br>node_ip   = {}<br>node_role = {}<br><br>for node in nodedata.get(&#39;items&#39;, []):<br>    name = node[&#39;metadata&#39;][&#39;name&#39;]<br>    anns = node[&#39;metadata&#39;].get(&#39;annotations&#39;, {})<br>    lbls = node[&#39;metadata&#39;].get(&#39;labels&#39;, {})<br><br>    l3gw = anns.get(&#39;k8s.ovn.org/l3-gateway-config&#39;, &#39;{}&#39;)<br>    try:<br>        mac = json.loads(l3gw).get(&#39;default&#39;, {}).get(&#39;mac-address&#39;, &#39;N/A&#39;)<br>    except Exception:<br>        mac = &#39;N/A&#39;<br><br>    addrs = node.get(&#39;status&#39;, {}).get(&#39;addresses&#39;, [])<br>    ip    = next((a[&#39;address&#39;] for a in addrs if a[&#39;type&#39;] == &#39;InternalIP&#39;), &#39;N/A&#39;)<br>    roles = [k.split(&#39;/&#39;)[-1] for k in lbls if &#39;node-role.kubernetes.io&#39; in k]<br><br>    node_mac[name]  = mac<br>    node_ip[name]   = ip<br>    node_role[name] = &#39;,&#39;.join(roles) if roles else &#39;worker&#39;<br><br>B = &#39;\033[1m&#39;<br>D = &#39;\033[2m&#39;<br>G = &#39;\033[0;32m&#39;<br>R = &#39;\033[0m&#39;<br><br>C = [32, 22, 36, 17, 19, 17, 12]<br>H = [&#39;SERVICE&#39;, &#39;NAMESPACE&#39;, &#39;ADVERTISING NODE&#39;, &#39;NODE IP&#39;, &#39;MAC ADDRESS&#39;, &#39;EXTERNAL IP&#39;, &#39;ROLE&#39;]<br><br>def fmt(cols, pre=&#39;&#39;, post=&#39;&#39;):<br>    return pre + &#39;  &#39;.join(f&quot;{str(v):&lt;{C[i]}}&quot; for i,v in enumerate(cols)) + post<br><br>print()<br>print(fmt(H, B, R))<br>print(fmt([&#39;-&#39;*c for c in C], D, R))<br><br>items = l2data.get(&#39;items&#39;, [])<br>for item in items:<br>    lbl  = item[&#39;metadata&#39;].get(&#39;labels&#39;, {})<br>    st   = item.get(&#39;status&#39;, {})<br>    svc  = lbl.get(&#39;metallb.io/service-name&#39;,      &#39;unknown&#39;)<br>    ns   = lbl.get(&#39;metallb.io/service-namespace&#39;, &#39;unknown&#39;)<br>    node = st.get(&#39;node&#39;, &#39;N/A&#39;)<br>    eip = svc_data.get(f&#39;{ns}/{svc}&#39;, &#39;N/A&#39;)<br>    print(fmt([svc, ns, node, node_ip.get(node,&#39;N/A&#39;), node_mac.get(node,&#39;N/A&#39;), eip, node_role.get(node,&#39;N/A&#39;)]))<br><br>print()<br>print(f&quot;{D}  Total: {len(items)} service(s){R}&quot;)<br>print()<br>PYEOF<br><br>python3 &quot;$TMP_PY&quot; &quot;$TMP_L2&quot; &quot;$TMP_NODES&quot; &quot;$TMP_SVC&quot;</pre><p><strong>Three Terminals. Three Vantage Points. One Timeline.</strong></p><p>Before initiating the MetalLB operator upgrade, we opened three terminal sessions simultaneously and started all three scripts. The upgrade itself was kicked off only after all monitoring was confirmed running and producing output.</p><p>With all three logging, initiated the MetalLB operator upgrade through the OpenShift console. The entire upgrade window lasted approximately 5 minutes. All three log files captured the complete event with millisecond precision.</p><p>Correlating Three Log Streams Into One Story</p><p>After the upgrade window, aligned all three log files on a common timeline. The pattern was unambiguous.</p><p><strong>arping_live.log — the MAC change</strong></p><p>The arping log gave us the exact timestamp when VIP ownership moved — and crucially, the two-second window where no node was responding at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-ERPDKWHhYf3rGxSm1wNA.png" /></figure><p><strong>arp_cache.log — the stale cache</strong></p><p>The ARP cache on the bastion updated within a few minutes once <strong>arping</strong> traffic refreshed it. But the switch itself — which we later confirmed with the network team — retained the stale MAC for the full 30-minute ARP cache timeout</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5GDIRNZY6lJpytRD8rR_GQ.png" /></figure><p><strong>curl_traffic.log — the user impact</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BStNNP6_WGgGk8ZsQVohmg.png" /></figure><p>The arping log showed VIP ownership transferred correctly within 30 seconds. The curl log showed users were still failing 30 minutes later. The gap between these two events is the switch ARP cache — the network’s memory of a state that no longer existed.</p><p>Investigate the network events at switch level for the same timestamp, intresting the found the VIP is flaged as rough due to high number of swapping of the VIP between the nodes.</p><p><strong>Root Cause: Two Network Misconfigurations</strong></p><p>With the evidence collected, the root cause analysis pointed unambiguously at two switch-level configuration issues — neither of which was visible from the Kubernetes layer.</p><h4><strong>Finding 1: GARP not enabled on the VIP VLAN</strong></h4><p>When Node B took ownership of the VIPs and sent Gratuitous ARP broadcasts to announce the new MAC mapping, the switch on the VIP VLAN did not process them. GARP was disabled on that VLAN — a common security hardening measure that prevents rogue devices from poisoning ARP tables, but which also prevents legitimate fast-failover from working.</p><p>The consequence: even though Node B was correctly advertising the VIPs, the switch continued forwarding all VIP traffic to Node A’s MAC address. Node A’s speaker was down. The traffic arrived at a node that could not handle it, and was dropped.</p><h4>Finding 2: Switch ARP cache timeout of 30 minutes</h4><p>The switch was configured with a 30-minute ARP cache timeout — meaning it would not re-probe or refresh a cached MAC entry for half an hour unless explicitly triggered. In most enterprise environments, this is set between 4 and 10 minutes. At 30 minutes, even a brief GARP-related delay compounds into a significant outage window.</p><p>Combined with GARP being disabled, the switch had no mechanism to learn the new MAC except by waiting for the cache to expire and allowing a new ARP request-response cycle to occur naturally. That took 30 minutes.</p><p><strong>What This Incident Taught Us</strong></p><ul><li><strong>Do not assume the cluster is the whole story. </strong>MetalLB completed its failover correctly. The failure lived entirely in the network layer — invisible from kubectl or the operator logs.</li><li><strong>ARP from the client tells you nothing about VIP ownership. </strong>Always use arping from a host on the same subnet as the VIPs for ground truth.</li><li><strong>Timestamps across layers are non-negotiable. </strong>Without millisecond-precision logs from arping, arp, and curl running simultaneously, we would have had no way to prove the 30-minute gap between VIP transfer and traffic recovery.</li><li><strong>GARP is not optional in L2 load balancing. </strong>Any environment using MetalLB in L2 mode must ensure GARP is accepted on the VIP VLAN. This is as fundamental as the load balancer itself.</li><li><strong>Switch ARP cache timeout must match your failover SLA. </strong>A 30-minute cache timeout is incompatible with sub-minute failover objectives. Review and align these values before the next maintenance window.</li></ul><blockquote><strong><em>Disclaimer</em></strong><em>:</em></blockquote><blockquote><em>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners</em></blockquote><p>LinkedIn: <a href="https://www.linkedin.com/in/nnreddy51/">https://www.linkedin.com/in/nnreddy51/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04ed7db914b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenShift Role-Based Access Control (RBAC) Implementation and Management]]></title>
            <link>https://medium.com/@arjun0451/openshift-role-based-access-control-rbac-implementation-and-management-c72876bfab0e?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/c72876bfab0e</guid>
            <category><![CDATA[openshift-4]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes-rbac]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Sun, 29 Mar 2026 10:29:53 GMT</pubDate>
            <atom:updated>2026-03-29T10:29:53.838Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5jRRlYUcirzpKTvr" /><figcaption>Photo by <a href="https://unsplash.com/@pconrad?utm_source=medium&amp;utm_medium=referral">Peter Conrad</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>1. Objective</h3><p>This document defines the standard operating procedure for implementing and managing <strong>Role-Based Access Control (RBAC)</strong> within OpenShift clusters.</p><p>The goal is to ensure that all access to cluster resources is <strong>secure, auditable, and aligned with the principle of least privilege</strong>, while supporting both <strong>infrastructure operations</strong> and <strong>application development</strong> needs.</p><h3>2. Scope</h3><p>This SOP applies to all <strong>OpenShift clusters</strong> managed by the organization — including <strong>production, staging, and development</strong> environments.</p><p>It covers:</p><ul><li>User and group access management</li><li>Service account creation and token usage for automation (CI/CD)</li><li>Role assignment and review process</li><li>Ongoing auditing and governance of RBAC configurations</li></ul><h3>3. Core Concepts of RBAC</h3><p>Role-Based Access Control (RBAC) in Kubernetes functions by granting permissions to users or service accounts based on the roles they are assigned.<br> The Kubernetes API server inherently supports RBAC, which is configured through specific API resources under rbac.authorization.k8s.io.</p><h3>Here’s how the key components work:</h3><ul><li><strong>Role:</strong> Defines a set of permissions (e.g., get, create, delete) for specific resource types (e.g., Pods, Deployments) within a namespace.</li><li><strong>ClusterRole:</strong> Similar to a Role, but non-namespaced. Used to manage cluster-level resources or to grant permissions to namespaced resources across all namespaces.</li><li><strong>RoleBinding:</strong> Links a Role to one or more subjects (users or service accounts), granting the permissions defined in the Role within its namespace.</li><li><strong>ClusterRoleBinding:</strong> Links a ClusterRole to one or more subjects, granting them the permissions defined in the ClusterRole across the entire cluster.</li></ul><p>The following representation the rolebinding and clusterrolebinding mapping with users,groups and serviceaccounts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*39icKDOcegdcYLlv4YyqKg.png" /><figcaption>RoleBinding</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hQpiFEXQ8vvMEouPGm9rg.png" /><figcaption>ClusterRolebinding</figcaption></figure><h3>3.1 How RBAC Works — Step-by-Step Flow</h3><p><strong>1. Authentication:</strong><br> A user, group, or service account first authenticates to the Kubernetes API (using tokens, certificates, or OAuth).</p><p><strong>2. Authorization:</strong><br>The Kubernetes API server checks whether the user’s identity has the required permissions to perform the requested action.</p><p><strong>3. Role Evaluation:</strong></p><p>If the request targets a namespace, the API server looks for RoleBindings in that namespace.</p><p>If no namespace is specified or the action is cluster-scoped, it checks ClusterRoleBindings.</p><p><strong>4. Decision:</strong><br>The API server grants or denies access based on the combination of Roles/ClusterRoles and their associated bindings.</p><h3>4. Role Definitions and Access Matrix</h3><p>Every organization follows its own <strong>User Access Management</strong> standards based on internal policies, compliance requirements, and operational maturity.<br>The following approach represents <strong>one such best practice</strong> for implementing and managing access in OpenShift using RBAC principles.</p><p>However, it’s important to note that <strong>the exact implementation may vary</strong> depending on individual organizational practices, team structures, and security governance models.</p><blockquote>As a good security practice, it is strongly recommended to onboard all users through centralized identity management systems such as LDAP.<br>Additionally, enable password rotation policies to avoid static credentials and minimize the risk of compromised accounts.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-PQ6TjYPzJ1kXcxCTVARQ.png" /></figure><h3>5. RBAC Implementation Guidelines and Best Practices</h3><h3>5.1 Principle of Least Privilege</h3><p>Grant the minimum permissions necessary for each user or service account to perform their function. Avoid giving broad or cluster-wide access unless explicitly justified.</p><h3>5.2 Use Default Roles When Possible</h3><p>OpenShift provides default, well-tested roles (admin, edit, view, and cluster-admin). Use these whenever they meet the requirement instead of creating redundant custom roles.</p><h3>5.3 Custom Roles for Specific Needs</h3><p>When default roles are insufficient, create custom roles with precise permissions. Document the purpose and permissions of all custom roles thoroughly.</p><h3>5.4 Group-Based Access</h3><p>Whenever possible, assign roles to OpenShift groups (e.g., LDAP/AD groups synced to OpenShift) rather than individual users. This simplifies management and provides a single point of control for user access.</p><h3>5.5 Service Accounts for Automation</h3><p>Use dedicated service accounts for CI/CD pipelines, monitoring agents, and other automated processes. Assign only the necessary permissions to these service accounts and manage their tokens securely.</p><h3>5.6 Regular Review and Audit</h3><p>Periodically review RBAC configurations, especially for cluster-admin and admin roles, to ensure they remain appropriate. Conduct regular audits to identify and rectify any unauthorized or excessive permissions.</p><h3>5.7 Separate Responsibilities</h3><p>Implement segregation of duties by ensuring no single individual has excessive control over the entire OpenShift environment. For instance, separate responsibilities for infrastructure management from application deployment.</p><h3>5.8 Namespace-Scoped Access</h3><p>For application teams, always prefer namespace-scoped roles (e.g., edit or view within a specific project) over cluster-scoped roles. This isolates application access and prevents unintended impact on other projects.</p><h3>5.9 Environment-Specific RBAC</h3><p>Tailor RBAC policies to the specific environment (production, staging, development). Production environments should have the strictest controls, with more lenient policies potentially applied to development environments (while still adhering to security best practices).</p><h3>5.10 Document All RBAC Decisions</h3><p>Maintain comprehensive documentation of all RBAC policies, custom roles, group assignments, and justifications for elevated privileges. This aids in auditing, troubleshooting, and onboarding new team members.</p><h3>6. Implementation Steps</h3><h3>6.1 Create Groups and Add Users</h3><pre># Create OpenShift groups for various roles<br>oc adm groups new ocp-infra-admins   # Group for Platform Administrators<br>oc adm groups new ocp-infra-ops      # Group for Platform Operations<br>oc adm groups new app-ops            # Group for Application Operations<br>oc adm groups new app-devs           # Group for Application Developers</pre><pre># Add users to their respective OpenShift groups<br>oc adm groups add-users ocp-infra-admins user1 user2  <br>oc adm groups add-users ocp-infra-ops user3           <br>oc adm groups add-users app-ops user4                 <br>oc adm groups add-users app-devs user5</pre><h3>6.2 Assign Roles to Groups</h3><h4>Cluster-Level Assignments</h4><pre>oc adm policy add-cluster-role-to-group cluster-admin ocp-infra-admins  <br>oc adm policy add-cluster-role-to-group cluster-reader ocp-infra-ops</pre><h4>Namespace-Level Assignments (example: team1-apps)</h4><pre>oc adm policy add-role-to-group view app-ops -n team1-apps  <br>oc adm policy add-role-to-group edit app-devs -n team1-apps</pre><h3>6.3 Create Service Account for CI/CD</h3><pre>oc create sa &lt;namespace&gt;-sa -n team1-apps<br>oc adm policy add-role-to-user edit -z cdci-sa -n team1-apps<br>oc create token &lt;namespace&gt;-sa -n team1-apps --duration=12h</pre><p>For OpenShift 4.10 and earlier:</p><pre>oc get secret namespace-sa-token-&lt;hash&gt; -n team1-apps -o jsonpath=&#39;{.data.token}&#39; | base64 --decode</pre><h3>6.4 Validate Role Bindings</h3><pre># Validate if user1 (Platform Admin) can create pods in team1-apps namespace<br>oc auth can-i create pods --as=user1 -n team1-apps</pre><pre># Review namespace-level role bindings for team1-apps<br>oc describe rolebinding -n team1-apps</pre><pre># Review cluster-level role bindings<br>oc describe clusterrolebinding</pre><h3>7. References</h3><ol><li><a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/authentication_and_authorization/using-rbac?utm_source=chatgpt.com">Red Hat OpenShift RBAC Documentation (v4.18)</a></li><li><a href="https://kubernetes.io/docs/concepts/security/rbac-good-practices?utm_source=chatgpt.com">Kubernetes RBAC Good Practices</a></li><li><a href="https://www.cisecurity.org/benchmark/kubernetes">CIS Kubernetes Benchmark — RBAC and Authorization</a></li></ol><blockquote><strong>Disclaimer</strong>:</blockquote><blockquote>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c72876bfab0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CoreDNS in OpenShift]]></title>
            <link>https://medium.com/@arjun0451/coredns-in-openshift-01f3142bde25?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/01f3142bde25</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[openshift-4]]></category>
            <category><![CDATA[coredns]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Fri, 21 Nov 2025 19:26:39 GMT</pubDate>
            <atom:updated>2026-04-11T15:13:01.370Z</atom:updated>
            <content:encoded><![CDATA[<p>Understanding CoreDNS, Forwarders, ndots, and Name Resolution Flow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ikYi8-rSWR9SzAzK" /></figure><blockquote>When it comes to OpenShift (or Kubernetes in general), DNS isn’t just a background service — it’s one of the most critical components keeping service discovery alive inside your cluster.<br> Every Pod, Service, and Route depends on DNS resolution to communicate reliably.</blockquote><blockquote>In this article, we’ll take a deep technical walk through <strong>how DNS works in OpenShift</strong>, how <strong>CoreDNS</strong> handles internal and external lookups, what <strong>forward plugins</strong>, <strong>ndots</strong>, and <strong>search domains</strong> actually mean, and how to <strong>troubleshoot</strong> DNS performance issues like timeouts or failures</blockquote><h4>1. DNS Architecture in OpenShift</h4><p>In OpenShift, the <strong>DNS resolution for Pods and Services</strong> is handled by <strong>CoreDNS</strong> — a scalable, cloud-native DNS server that understands Kubernetes resources.</p><p>Every cluster typically includes:</p><ul><li>A <strong>Service</strong> named dns-default (ClusterIP, e.g. 172.30.0.10)</li><li>Bascially the 10th IP from the cluster service network is reserved for the CoreDNS by default. (ex. 172.30.0.<strong>10)</strong></li><li>One or more <strong>CoreDNS Pods</strong> (dns-default-*) running in the openshift-dns namespace</li><li>A <strong>DaemonSet</strong> called node-resolver, which adds local host entries for certain metadata like image registry entries in the node /etc/hosts</li></ul><h3>Internal DNS domain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0iX9F5Gq0t06vB6AL89mg.png" /></figure><p>By default, all internal names end with the cluster domain:</p><pre>*.svc.cluster.local (# svcname.namespace.svc.cluster.local)<br>*.pod.cluster.local. (# podip.namespace.pod.cluster.local)</pre><p>For example:</p><ul><li>backend.myapp.svc.cluster.local</li><li>10-128-3-45.myapp.pod.cluster.local</li></ul><p>These names are resolved <strong>by CoreDNS</strong>, not by the node’s operating system.</p><h4>2. How CoreDNS Resolves Internal Names</h4><p>When a Pod performs a lookup like:</p><pre>backend.myapp.svc</pre><p>the query first reaches the <strong>CoreDNS Service ClusterIP (e.g. 172.30.0.10)</strong>.</p><p>CoreDNS checks:</p><ol><li>If the name belongs to cluster.local (the internal domain)</li><li>If a matching <strong>Service</strong>, <strong>Pod</strong>, or <strong>Endpoint</strong> exists</li><li>If found → returns the corresponding ClusterIP or Pod IP</li></ol><p>Example: Internal resolution</p><p>Login to the test pod and execute the following command will resolve to the service clusterIP, This means the backend Service in the myapp namespace maps to ClusterIP 172.30.52.19.</p><pre>$ dig backend.myapp.svc.cluster.local +short<br>172.30.52.19</pre><h4>3. The Forward Plugin &amp; External DNS Resolution</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWw9Jl4ROx00iPzp13ziJQ.png" /></figure><p>When a Pod queries an <strong>external domain</strong> like redhat.com,or any other enterprise zone CoreDNS doesn’t know that internally — so it uses the <strong>forward plugin</strong> to send the query upstream.</p><p>Openshift provide capabilities to configure the forward plugin for the external DNS resolution via configmaps in the Openshift-dns namespace.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: dns-default<br>  namespace: openshift-dns<br>data:<br>  Corefile: |<br>    .:5353 {<br>        errors<br>        health<br>        kubernetes cluster.local in-addr.arpa ip6.arpa {<br>            pods insecure<br>            fallthrough in-addr.arpa ip6.arpa<br>        }<br>        forward . /etc/resolv.conf {<br>            policy sequential<br>        }<br>        cache 900<br>        reload<br>    }</pre><p>forward . /etc/resolv.conf → means forward all non-cluster queries to the DNS servers listed in the <strong>CoreDNS Pod’s </strong><strong>/etc/resolv.conf</strong>, usually the node’s upstream resolver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*jdOQWPA0wVf8O06sp48teg.png" /></figure><p>This allows <strong>external DNS lookups</strong> (ex: internet or corporate domains).</p><p>Openshift CoreDNS also supports configuring the Enterprise DNS resolution configuring additional forward lookup nameservers as follows.</p><p>All queries for example.com will be sent to the specific external nameservers (192.168.1.10 and 192.168.1.11).</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: dns-default<br>  namespace: openshift-dns<br>data:<br>  Corefile: |<br>    .:5353 {<br>        errors<br>        health<br>        kubernetes cluster.local in-addr.arpa ip6.arpa {<br>            pods insecure<br>            fallthrough in-addr.arpa ip6.arpa<br>        }<br>        <br>        # 1. Forwarding for Enterprise Domain (e.g., example.com)<br>        # All queries for example.com will be sent to the specific external nameservers (192.168.1.10 and 192.168.1.11).<br>        # This takes precedence over the general &#39;.&#39; forward below.<br>        example.com:5353 {<br>            errors<br>            cache 300<br>            forward . 192.168.1.10 192.168.1.11<br>        }<br>        <br>        # 2. General Forwarding to Node&#39;s /etc/resolv.conf<br>        # Queries that are NOT for Kubernetes (cluster.local) and NOT for the specific domain (example.com) <br>        # will be forwarded to the nameservers listed in the node&#39;s /etc/resolv.conf.<br>        forward . /etc/resolv.conf {<br>            policy sequential<br>        }<br>        <br>        cache 900<br>        reload<br>    }</pre><p>One More important thing to know is in case of Static pods — like the control plane components (kube-apiserver, etcd, etc.) — are managed directly by <strong>kubelet</strong>, not the scheduler.</p><p>They use the node’s /etc/resolv.conf by default, since the cluster DNS might not be ready when they start.</p><h4>4. DNS inside the Pod</h4><p>Each pod inherits a DNS configuration file at /etc/resolv.conf, which defines:</p><ul><li>The <strong>nameserver</strong> (CoreDNS ClusterIP)</li><li>The <strong>search domains</strong></li><li>The <strong>ndots</strong> setting controlling how lookups are expanded</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3mibAKZKTy1_u6PP_uffVQ.png" /><figcaption>pods resolv.conf</figcaption></figure><p>It is important to understand the what details are part of the pods /etc/resolv.conf file</p><p><strong>nameserver 172.30.0.10</strong> → Points to the CoreDNS service inside the cluster.</p><p><strong>search domains</strong> → Suffixes automatically appended to names with fewer dots. if any external forwards configured in the coredns it will be also be part of the search domain like example.com</p><pre>search myapp.svc.cluster.local svc.cluster.local cluster.local example.com<br>nameserver 172.30.0.10<br>options ndots:5</pre><p><strong>options ndots:5</strong> → Any query with fewer than 5 dots will go through multiple search domain expansions before being sent to external DNS.</p><h3>Real DNS Resolution Scenarios</h3><p>Let’s explore how DNS behaves in practice, using two common scenarios.</p><h3>Scenario 1 — Resolving an External Domain (redhat.com)</h3><p>The application Pod is trying to resolve redhat.com as part of this testing , if we can inspect the the pod&#39;s /etc/resolv.conf as discussed earlier we can able to find the all the search domains and nameserver which is ClusterIP of coreDNS and ndots default values.</p><pre>search myapp.svc.cluster.local svc.cluster.local cluster.local example.com<br>nameserver 172.30.0.10<br>options ndots:5<br><br>##testing to resolve the redhat.com from the test pod in myapp namespace<br>#oc exec -it -n myapp dns-test -- ping redhat.com</pre><p><strong>Resolution flow:</strong></p><ol><li>CoreDNS first attempts:</li></ol><ul><li>redhat.com.default.svc.cluster.local</li><li>redhat.com.svc.cluster.local</li><li>redhat.com.cluster.local</li><li>redhat.com.example.com</li></ul><p>All fail (NXDOMAIN) Finally, redhat.com is queried externally through the <strong>forward plugin</strong></p><p>Example CoreDNS logs ( ensure the CoreDNS log levels are configured to <strong>all</strong> capture the lookup logs)</p><pre>[INFO] 10.128.3.25:49243 - 49753 &quot;A IN redhat.com.default.svc.cluster.local. udp 58 false 512&quot; NXDOMAIN qr,aa,rd 151 0.0185s<br>[INFO] 10.128.3.25:51179 - 1194 &quot;A IN redhat.com.svc.cluster.local. udp 50 false 512&quot; NXDOMAIN qr,aa,rd 143 0.0003s<br>[INFO] 10.128.3.25:39281 - 47418 &quot;A IN redhat.com.cluster.local. udp 46 false 512&quot; NXDOMAIN qr,aa,rd 139 0.0008s<br>[INFO] 10.128.3.25:56661 - 17114 &quot;A IN redhat.com.example.com. udp 43 false 512&quot; NXDOMAIN qr,rd,ra 43 0.0082s<br>[INFO] 10.128.3.25:46659 - 26128 &quot;A IN redhat.com. udp 32 false 512&quot; NOERROR qr,rd,ra 62 0.0839s</pre><p>Key Observation to note down here is One query from the pod resulted in <strong>five DNS lookups</strong> due to ndots:5</p><p>The ndots configuration determines how many dots (&quot;.&quot;) must appear in a domain name before the resolver treats it as a fully qualified domain name (FQDN). If ndots is set to 5, the resolver will only consider a domain as fully qualified if it contains <strong>five or more dots</strong>. Otherwise, it will <strong>append the search domains</strong> from the search list one by one, trying each in order until the query resolves successfully.</p><blockquote>Note: If the communication is happening within the same namespace, the application pod doesn’t need to use the full FQDN of the service — just the service name is enough for DNS resolution. However, if the application needs to connect to a service in a <strong>different namespace</strong>, it should use the <strong>fully qualified domain name (FQDN)</strong> of that service to ensure faster and more reliable name resolution.</blockquote><h4>Scenario 2 — Resolving a Fully Qualified Domain Name (FQDN)</h4><pre>oc exec -it -n myapp dns-test -- ping test-app.myapp.svc.cluster.local.<br>oc exec -it -n myapp dns-test -- dig redhat.com.</pre><p>Notice the <strong>trailing dot</strong> (local.)— this makes it a <strong>fully qualified domain name</strong>.CoreDNS immediately queries the exact name <strong>once</strong>, without appending any search domains.</p><pre>[INFO] 10.128.3.25:48320 - 6962 &quot;A IN test-app.myapp.svc.cluster.local. udp 58 false 512&quot; NOERROR qr,aa,rd 114 0.0026s</pre><h3>Why ndots:5 Matters</h3><p>The ndots setting determines when a query is treated as “absolute” vs “relative.”</p><p><strong>5 (default) → </strong>Try up to 5 search domain expansions before direct lookup</p><p><strong>1–3 (tuned) → </strong>Fewer retries, faster lookups for external domains</p><p><strong>0 → </strong>Always query exact name directly</p><p>If your applications frequently connect to external domains (e.g. APIs, SaaS endpoints), you can <strong>reduce unnecessary DNS lookups</strong> by lowering ndots to 3 or using FQDNs in configuration.</p><h4>Can You Override ndots per Pod or per Application?</h4><p>you can control ndots and other DNS options at the pod level using the <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config">dnsConfig</a> field in your Deployment, Pod, or StatefulSet manifest.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: sample-web<br>  namespace: test<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: sample-web<br>  template:<br>    metadata:<br>      labels:<br>        app: sample-web<br>    spec:<br>      containers:<br>      - name: web<br>        image: registry.access.redhat.com/ubi9/ubi-minimal<br>        command: [&quot;sleep&quot;, &quot;3600&quot;]<br>      dnsPolicy: ClusterFirst<br>      dnsConfig:<br>        options:<br>        - name: ndots<br>          value: &quot;3&quot;<br>        searches:<br>        - test.svc.cluster.local<br>        - svc.cluster.local<br>        - cluster.local<br>        - example.com</pre><p>With ndots:3, the pod treats as <strong>more likely an external domain</strong>, so it skips the first few internal lookups and reaches the external resolver faster.</p><h3>Common Issues and Solutions</h3><p>Now lets look into some of the common issues and troubleshooting steps.</p><h4>1. DNS Query Timeouts</h4><pre># CoreDNS pod logs<br>[ERROR] plugin/errors: 2 backend.myapp.svc.cluster.local. A: read udp 10.128.3.12:54121-&gt;172.30.0.10:53: i/o timeout</pre><p>Troubleshooting steps</p><pre>1.Make sure to check all the coredns pods are healthy in openshift-dns namespace<br><br>#oc get pods -n openshift-dns<br><br>2. Check if the POD can able to resolv the coreDNS clusterIP address<br><br>NAMESPACE=&lt;your-namespace&gt;<br><br>for POD in $(oc get pods -n $NAMESPACE -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>  echo &quot;------ $POD ------&quot;<br>  oc exec -n $NAMESPACE $POD -- cat /etc/resolv.conf<br>  echo<br>done<br><br>## Test <br>NAMESPACE=myapp<br>[test@upi-0 ~]$ for POD in $(oc get pods -n $NAMESPACE -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>&gt;   echo &quot;------ $POD ------&quot;<br>&gt;   oc exec -n $NAMESPACE $POD -- cat /etc/resolv.conf<br>&gt;   echo<br>&gt; done<br>------ test-app-788cf69d54-c6drg ------<br>search myapp.svc.cluster.local svc.cluster.local cluster.local <br>nameserver 172.30.0.10<br>options ndots:5<br><br>------ test-app-788cf69d54-m7pv8 ------<br>search myapp.svc.cluster.local svc.cluster.local cluster.local<br>nameserver 172.30.0.10<br>options ndots:5<br><br>------ test-app-788cf69d54-pmkcs ------<br>search myapp.svc.cluster.local svc.cluster.local cluster.local <br>nameserver 172.30.0.10<br>options ndots:5<br><br>In this senerio the pod is able to pod resolver is pointing to DNS Service IP:</pre><p>Check if the pods can be able to resolve the internal services endpoint of the pod</p><pre>###<br>HOST_ADDR=test-app-service.myapp.svc.cluster.local<br><br>for dnspod in $(oc get pods -n openshift-dns -l dns.operator.openshift.io/daemonset-dns=default -o name --no-headers); do<br>  dnsip=$(oc get pod -n openshift-dns ${dnspod#pod/} -o jsonpath=&#39;{.status.podIP}&#39;)<br>  echo -ne &quot;$dnspod querying $HOST_ADDR via $dnsip -&gt;\t&quot;<br>  oc exec -n openshift-dns $dnspod -- dig @$dnsip $HOST_ADDR -p 5353 +short 2&gt;/dev/null<br>done<br><br>pod/dns-default-8s2hb querying test-app-service.myapp.svc.cluster.local via 10.130.0.12 -&gt; 172.30.132.168<br>pod/dns-default-jc6gt querying test-app-service.myapp.svc.cluster.local via 10.131.0.6 -&gt; 172.30.132.168<br>pod/dns-default-p7hpr querying test-app-service.myapp.svc.cluster.local via 10.129.2.6 -&gt; 172.30.132.168<br>pod/dns-default-qwk2l querying test-app-service.myapp.svc.cluster.local via 10.128.0.38 -&gt; 172.30.132.168<br>pod/dns-default-xkdbl querying test-app-service.myapp.svc.cluster.local via 10.129.0.10 -&gt; 172.30.132.168<br>pod/dns-default-z5gn9 querying test-app-service.myapp.svc.cluster.local via 10.128.2.5 -&gt; 172.30.132.168</pre><p>This helps confirm if the <strong>CoreDNS service and the kubelet DNS forwarding</strong> are behaving consistently</p><p>In case of Upstream DNS Error resolutions check if the Upstream servers are access and no firewall issues and validate the ports are opened to communicate on both TCP and UDP</p><pre><br># Define upstream DNS target<br>UPSTREAM_DNS_IP=&quot;110.68.5.26&quot;<br>UPSTREAM_DNS_PORT=&quot;53&quot;<br>TARGET_DOMAIN=&quot;redhat.com&quot;<br><br>echo -e &quot;\n=== Testing DNS Resolution from Each CoreDNS Pod ===&quot;<br>echo &quot;Target Domain: $TARGET_DOMAIN&quot;<br>echo &quot;Upstream DNS:  ${UPSTREAM_DNS_IP}:${UPSTREAM_DNS_PORT}&quot;<br>echo &quot;Protocol: TCP and UDP\n&quot;<br><br># Iterate once through each DNS pod<br>for dnspod in $(oc get pods -n openshift-dns -l dns.operator.openshift.io/daemonset-dns=default -o name); do<br>  echo &quot;----- Pod: $dnspod -----&quot;<br><br>  # TCP lookup<br>  echo -e &quot;TCP lookup:&quot;<br>  oc exec -n openshift-dns -c dns $dnspod -- \<br>    dig @$UPSTREAM_DNS_IP $TARGET_DOMAIN -p $UPSTREAM_DNS_PORT +tcp +short 2&gt;/dev/null || echo &quot;TCP lookup failed&quot;<br><br>  # UDP lookup<br>  echo -e &quot;\nUDP lookup:&quot;<br>  oc exec -n openshift-dns -c dns $dnspod -- \<br>    dig @$UPSTREAM_DNS_IP $TARGET_DOMAIN -p $UPSTREAM_DNS_PORT +notcp +short 2&gt;/dev/null || echo &quot;UDP lookup failed&quot;<br><br>  echo -e &quot;---------------------------------------------\n&quot;<br>done<br><br><br>[test@upi-0 ~]$ ./test.sh <br><br>=== Testing DNS Resolution from Each CoreDNS Pod ===<br>Target Domain: redhat.com<br>Upstream DNS:  110.68.5.26:53<br>Protocol: TCP and UDP\n<br>----- Pod: pod/dns-default-8s2hb -----<br>TCP lookup:<br>152.200.142.250<br>134.235.198.240<br><br>UDP lookup:<br>134.235.198.240<br>152.200.142.250<br>---------------------------------------------<br><br>----- Pod: pod/dns-default-jc6gt -----<br>TCP lookup:<br>152.200.142.250<br>134.235.198.240<br><br>UDP lookup:<br>134.235.198.240<br>152.200.142.250<br>---------------------------------------------</pre><ol><li>Outbound DNS connectivity from each CoreDNS pod to the specified upstream DNS server.</li><li>Protocol consistency — whether both TCP and UDP DNS queries succeed.</li><li>Network path validation — if certain nodes have egress or firewall restrictions affecting DNS traffic.</li><li>Pod-level verification — confirms that each CoreDNS pod can forward queries correctly.</li></ol><pre><br>ns=&quot;openshift-dns&quot;<br><br>echo -e &quot;\n=== Counting &#39;i/o timeout&#39; occurrences in CoreDNS pods ===&quot;<br>echo -e &quot;Namespace: $ns\n&quot;<br><br># Collect counts of i/o timeouts per CoreDNS pod, sort by descending count<br>for pod in $(oc -n $ns get pods -l dns.operator.openshift.io/daemonset-dns=default -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>  count=$(oc -n $ns logs -c dns $pod | grep -c &#39;i/o timeout&#39;)<br>  printf &quot;%-40s %5d\n&quot; &quot;$pod&quot; &quot;$count&quot;<br>done | sort -k2 -nr<br><br><br>=== Counting &#39;i/o timeout&#39; occurrences in CoreDNS pods ===<br>Namespace: openshift-dns<br><br>dns-default-4fx2d                           132<br>dns-default-nxk8r                            87<br>dns-default-p6g5t                            22<br>dns-default-bmv7l                             0</pre><p>This command helps you quickly identify:</p><ol><li><strong>Which CoreDNS pods are facing the most DNS forwarding timeouts</strong>, possibly due to:</li></ol><ul><li>Node-level network latency or packet loss.</li><li>Unreachable upstream DNS servers.</li><li>Pod scheduling location on nodes with poor egress connectivity.</li><li>Resource starvation (CPU throttling) causing delayed responses.</li></ul><p><strong>If the issue is cluster-wide or node-specific</strong> — if only one or two pods consistently show more timeouts.</p><pre>ns=&quot;openshift-dns&quot;<br>pod=&quot;dns-default-pkp7m&quot;   # Replace with the pod that had the most timeouts from previous test<br><br>echo -e &quot;\n=== Analyzing &#39;i/o timeout&#39; events in $pod ===\n&quot;<br><br>oc -n $ns logs -c dns $pod \<br>  | awk &#39;$0 ~ &quot;i/o timeout&quot;{print $5, $9}&#39; \<br>  | sed &#39;s/100.*-&gt;//&#39; \<br>  | sort | uniq -c | sort -rn<br><br><br>Output<br>   82  10.10.10.10 redhat.com.<br>   47  10.10.10.11 example.com.<br>   12  10.10.10.10 updates.redhat.com.<br>    3  10.10.10.12 registry.access.redhat.com.</pre><p>This tells you:</p><ul><li>Which upstream DNS server (10.10.10.10, 10.10.10.11, etc.) CoreDNS was forwarding queries to.</li><li>Which domains were being queried when the timeouts occurred.</li><li>The count of such timeout events per upstream or domain combination.</li></ul><p>From the example above:</p><ul><li>Most timeouts occurred while CoreDNS tried to reach 10.10.10.10 to resolve redhat.com — likely meaning network latency or unreachability toward that upstream.</li><li>You can now focus your troubleshooting on connectivity from that node (where this CoreDNS pod runs) to 10.10.10.10:53 (UDP/TCP).</li></ul><pre>ns=&quot;openshift-dns&quot;<br><br>echo -e &quot;\n=== Aggregating &#39;i/o timeout&#39; instances across all CoreDNS pods ===\n&quot;<br><br>for pod in $(oc -n $ns get pods -l dns.operator.openshift.io/daemonset-dns=default \<br>  -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>  oc -n $ns logs -c dns $pod | awk &#39;$0 ~ &quot;i/o timeout&quot;{print $5, $9}&#39;<br>done | sed &#39;s/100.*-&gt;//&#39; | sort | uniq -c | sort -rn<br><br>  155 redhat.com. 10.10.10.10<br>   87 example.com. 10.10.10.11<br>   26 updates.redhat.com. 10.10.10.10<br>    5 registry.access.redhat.com. 10.10.10.12</pre><p>This output gives you <strong>cluster-wide visibility</strong> into where DNS queries are failing due to network or forwarding issues.</p><ul><li><strong>Column 1:</strong> Count of timeout instances.</li><li><strong>Column 2:</strong> FQDN that CoreDNS was attempting to resolve.</li><li><strong>Column 3:</strong> Upstream DNS server it tried to reach.</li></ul><p>From the example above:</p><ul><li>Most timeouts occurred when querying <strong>redhat.com</strong> via <strong>upstream 10.10.10.10</strong>.</li><li>Fewer failures occurred for <strong>example.com</strong> via <strong>10.10.10.11</strong>.</li><li>This pattern immediately highlights that <strong>10.10.10.10</strong> might be slow, unreachable, or dropping UDP packets.</li></ul><pre><br>oc exec -n myapp test-app-788cf69d54-c6drg -- bash -c &#39;<br>while true; do<br>  echo -n &quot;$(date)  &quot;;<br>  curl -s -o /dev/null -w &quot;%{time_namelookup} %{time_total} %{http_code}\n&quot; https://www.redhat.com -k;<br>  sleep 25;<br>done&#39;<br><br>Wed Oct 15 11:03:00 SGT  0.015 0.482 200<br>Wed Oct 15 11:03:25 SGT  2.054 2.112 000<br>Wed Oct 15 11:03:50 SGT  0.012 0.496 200<br><br>time_namelookup = 2.054 (DNS lookup took more than 2 seconds (timeout threshold)).<br><br>time_total = 2.112 (Total request time was slightly longer — meaning the bottleneck was DNS resolution itself.<br><br>http_code = 000 No HTTP response (DNS lookup likely failed or timed out).</pre><p><strong>Measure query time</strong> (latency) for both <strong>A (IPv4)</strong> and <strong>AAAA (IPv6)</strong> DNS lookups directly from within a pod</p><pre>oc exec -n myapp test-app-788cf69d54-c6drg -- bash -c &#39;<br>while true; do<br>  echo -n &quot;$(date)  &quot;;<br>  dig A www.redhat.com +stats | grep &quot;Query time&quot;;<br>  dig AAAA www.redhat.com +stats | grep &quot;Query time&quot;;<br>  sleep 10;<br>done&#39;</pre><p>This test we can able to measure any issue with A lookup or AAAA lookup and query time.</p><p><strong>Useful Queries</strong></p><pre># Test with dig (comprehensive)<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com +trace          # Full resolution trace<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com +short +stats   # Quick with timing<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com +tcp            # Force TCP<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com +timeout=2      # Set timeout<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com @172.30.0.10    # Query specific server<br><br># Test SOA record (Start of Authority)<br>oc exec -it &lt;pod-name&gt; -- dig google.com SOA +short<br><br># Test with specific query types<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com A +short        # IPv4<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com AAAA +short     # IPv6<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com MX +short       # Mail servers<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com TXT +short      # Text records<br>oc exec -it &lt;pod-name&gt; -- dig redhat.com NS +short       # Nameservers<br></pre><p><strong>Official Documentation</strong></p><ul><li><a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/"><strong>Kubernetes DNS for Services and Pods</strong></a></li><li><a href="https://docs.openshift.com/container-platform/latest/networking/dns/dns-default.html"><strong>OpenShift DNS Operator </strong></a><strong>(CoreDNS)</strong></li><li><a href="https://docs.openshift.com/container-platform/latest/networking/dns/dns-operator.html#dns-operator-configuring_forwarders_dns-operator"><strong>OpenShift CoreDNS ConfigMap &amp; Forwarding</strong></a></li></ul><p><strong>Disclaimer</strong>:</p><p>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners.</p><p><a href="https://www.linkedin.com/in/nnreddy51/">https://www.linkedin.com/in/nnreddy51/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=01f3142bde25" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenShift Alert manager Configuration for SMTP Alerts]]></title>
            <link>https://medium.com/@arjun0451/openshift-alert-manager-configuration-for-smtp-alerts-8f8a8aa5c202?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f8a8aa5c202</guid>
            <category><![CDATA[redhat-ocp]]></category>
            <category><![CDATA[openshift-4]]></category>
            <category><![CDATA[alertmanager]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Tue, 25 Mar 2025 04:01:48 GMT</pubDate>
            <atom:updated>2026-04-11T15:14:48.979Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview</h3><p>Alertmanager handles alerts from Prometheus, processes them, and sends notifications based on defined rules.<br>This article provides a full Overview and configuration of Alertmanager with SMTP email setup for different alert levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mzs31pkuuoet2cO2Qf7QsQ.png" /><figcaption>AlertManager Logical Diagram</figcaption></figure><h3>OpenShift AlertManager Architecture</h3><ol><li><strong>Alert Generation</strong>:</li></ol><ul><li><strong>Prometheus</strong> monitors your OpenShift cluster by collecting metrics</li><li><strong>Alert Rules</strong> define when conditions should trigger alerts</li><li>When thresholds are exceeded, Prometheus fires alerts to AlertManager</li></ul><p><strong>2. Alert Processing (AlertManager Core)</strong>:</p><ul><li>Centralizes alert handling</li><li>Groups similar alerts to reduce noise</li><li>Deduplicates identical alerts</li><li>Applies routing rules to determine where alerts should be sent.</li><li>Applies routing rules to determine where alerts should be sent</li></ul><p><strong>3. Alert Control Mechanisms</strong>:</p><ul><li><strong>Silence Manager</strong> allows operators to temporarily mute specific alerts</li><li><strong>Inhibition Rules</strong> automatically suppress less critical alerts when more important ones are firing.</li></ul><p><strong>4. Alert Distribution</strong>:</p><ul><li><strong>Routing Tree</strong> determines which alerts go to which receivers based on labels and rules</li><li><strong>Receivers</strong> are the notification channels (Email, Slack, PagerDuty, Webhooks)</li></ul><p><strong>5. Management Interface</strong>:</p><ul><li><strong>Web UI</strong> provides a central place to view and manage the entire alerting system</li><li>Allows configuration of silences, inhibition rules, and receivers (with clear “Configures” and “Manages” labels)</li></ul><h3>Full alertmanager.yaml Configuration</h3><pre>global:<br>  # SMTP server settings for sending email alerts<br>  smtp_smarthost: &#39;smtp.example.com:587&#39;   # SMTP server address<br>  smtp_from: &#39;alertmanager@example.com&#39;    # Sender email address<br>  smtp_auth_username: &#39;your-smtp-user&#39;     # SMTP authentication username<br>  smtp_auth_password: &#39;your-smtp-password&#39; # SMTP authentication password<br>  smtp_require_tls: true                   # Enforce TLS encryption<br>inhibit_rules:<br>  - equal:<br>      - namespace<br>      - alertname<br>    source_matchers:<br>      - severity = critical<br>    target_matchers:<br>      - severity =~ warning|info<br>  - equal:<br>      - namespace<br>      - alertname<br>    source_matchers:<br>      - severity = warning<br>    target_matchers:<br>      - severity = info<br>  - equal:<br>      - namespace<br>    source_matchers:<br>      - alertname = InfoInhibitor<br>    target_matchers:<br>      - severity = info<br>route:<br>  # Default receiver if no specific route matches<br>  receiver: &quot;default-email&quot;<br><br>  # Group alerts with the same label (e.g., alertname, severity) together<br>  group_by: [&#39;alertname&#39;, &#39;severity&#39;]<br><br>  # Wait time before sending an alert group (to batch multiple alerts)<br>  group_wait: 30s<br><br>  # Interval before sending new alerts in the same group<br>  group_interval: 5m<br><br>  # Time before resending an alert if it remains active<br>  repeat_interval: 3h<br><br>  # Define specific routing for different alert severities<br>  routes:<br>    - matchers:<br>        - severity=&quot;critical&quot;<br>      receiver: &quot;critical-email&quot;<br>      repeat_interval: 30m  # Critical alerts resend every 30 minutes<br><br>    - matchers:<br>        - severity=&quot;warning&quot;<br>      receiver: &quot;warning-email&quot;<br>      repeat_interval: 3h  # Warning alerts resend every 3 hours<br><br>receivers:<br>  - name: &quot;critical-email&quot;<br>    email_configs:<br>      - to: &quot;critical@example.com&quot;  # Send critical alerts here<br><br>  - name: &quot;warning-email&quot;<br>    email_configs:<br>      - to: &quot;warning@example.com&quot;  # Send warning alerts here<br><br>  - name: &quot;default-email&quot;<br>    email_configs:<br>      - to: &quot;default@example.com&quot;  # Default email recipient</pre><p><strong>Key Components of the AlertManager configuration File</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*rHtWKMlcGXsJFSqtobyrwA.png" /><figcaption>Alertmanager components</figcaption></figure><h4><strong>Alert Routing Breakdown</strong></h4><p>The following table explains how the grouping and processing section works in the alert manager core module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*6aqqbmrdnpwQAw1KJMpOjg.png" /><figcaption>Alert manager Route Module</figcaption></figure><h4><strong>Let us Go through this concept with an use case</strong></h4><h3>Alert Rule : High Pod CPU Usage</h3><p>This alert fires when a pod’s CPU usage exceeds 70% for 2 minutes</p><pre>## Sample Alerting Rule for monitoring the High CPU usage of the pods<br>groups:<br>  - name: pod-alerts<br>    rules:<br>      - alert: HighPodCPUUsage<br>        expr: (sum by (pod) (rate(container_cpu_usage_seconds_total{container!=&quot;&quot;,pod!=&quot;&quot;}[2m])) / sum by (pod) (kube_pod_container_resource_requests{resource=&quot;cpu&quot;})) * 100 &gt; 70<br>        for: 2m<br>        labels:<br>          severity: critical<br>        annotations:<br>          summary: &quot;High CPU Usage Detected&quot;<br>          description: &quot;Pod {{ $labels.pod }} has been using more than 70% CPU for over 2 minutes.&quot;</pre><h4>Scenario</h4><blockquote>The HighPodCPUUsage alert fires if a pod’s CPU is above 70% for 2 minutes.</blockquote><blockquote>The alert is labeled <strong>severity: critical</strong> and routed to the critical-email receiver.</blockquote><blockquote>Alertmanager evaluates the alert, <strong>groups </strong>it, and sends an email notification.</blockquote><blockquote>If the alert remains active, it will resend every 30 minutes, because the repeat_interval of the critical alerts is 30 min.</blockquote><blockquote>Once CPU usage drops below <strong>70%</strong>, the alert is resolved, and no further emails are sent.</blockquote><h3>Email Notification Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*TV8RdVnj2QT3cbPP5wDkbA.png" /></figure><h4><strong>How to Customize Alert Intervals</strong></h4><p>in case of customization, OpenShift administrator are allowed to configure the repeat interval values as per the project requirements , we can control the repeat_intervals per severity wise or can set globally for all the alert levels.</p><p>To modify alert resend timing, adjust the repeat_interval:</p><pre>routes:<br>  - matchers:<br>      - severity=&quot;critical&quot;<br>    receiver: &quot;critical-email&quot;<br>    repeat_interval: 15m  # Resend every 15 minutes<br><br>  - matchers:<br>      - severity=&quot;warning&quot;<br>    receiver: &quot;warning-email&quot;<br>    repeat_interval: 2h  # Resend every 2 hours</pre><p><strong>Using Different SMTP Servers for Different Alert Levels</strong></p><p>To override the default SMTP settings(global configurations), define custom SMTP configurations inside a receiver.</p><pre>receivers:<br>  - name: &quot;critical-email&quot;<br>    email_configs:<br>      - to: &quot;critical@example.com&quot;<br>        smarthost: &quot;smtp.critical-alerts.com:587&quot;<br>        auth_username: &quot;critical-user&quot;<br>        auth_password: &quot;critical-pass&quot;<br>        require_tls: true</pre><p>This ensures <strong>critical alerts use a separate SMTP server</strong>, while others use the global SMTP settings.</p><h4><strong>Inhibit Rules in Alertmanager</strong></h4><p><strong>Inhibit rules</strong> in Alertmanager are used to <strong>silence lower-severity alerts</strong> when a higher-severity alert is already active. This prevents duplicate notifications and reduces alert noise.</p><p>How It Works</p><ul><li>If a <strong>critical</strong> alert is firing, <strong>warning</strong> alerts for the same service/pod <strong>will be suppressed</strong>.</li><li>This ensures <strong>only the most severe issue</strong> is reported, avoiding redundant alerts.</li></ul><pre>inhibit_rules:<br>  - source_matchers:<br>      - severity=&quot;critical&quot;<br>    target_matchers:<br>      - severity=&quot;warning&quot;<br>    equal: [&#39;alertname&#39;, &#39;namespace&#39;, &#39;pod&#39;]</pre><h4>Use Case Example</h4><p>Without Inhibit Rules</p><ul><li>Critical Alert:<strong> </strong>“Pod is down” (fires)</li><li>Warning Alert: “Pod CPU usage high” (fires at the same time)</li></ul><p>Result: You receive <strong>both alerts</strong>, even though the critical one already tells you the pod is down.</p><p>With Inhibit Rules</p><ul><li>Critical Alert:<strong> </strong>“Pod is down” (fires)</li><li>️ Warning Alert:<strong> </strong>“Pod CPU usage high” (suppressed)</li></ul><p>Result<strong>:</strong> You only get the <strong>critical alert</strong>, avoiding unnecessary warnings.</p><p><strong>Validating the SMTP connectivity test from Node level before configuring the receiver.</strong></p><pre>## Test Alert Manager Configuration <br>##Login to the infra node in the openshift cluster<br>## Update the following variable values as per your requirement and apply.<br><br><br>$# FROM=&quot;emaildress&quot;<br>$# TO=&quot;emailaddress&quot;<br>$# SMARTHOST=&quot;email-smtp.ap-southeast-1.amazonaws.com&quot; ## email server details<br>$# AUTH_USERNAME=&quot;AKIAW3dsdE4QUPdsdsB&quot; ## UserName<br>$# AUTH_PASSWORD=&quot;BGLtD/aasdadadaatargagaf&quot; ## Password<br><br>## create a sample attachment file<br>$# cat &lt;&lt; EOF &gt; email.txt<br>From: $FROM<br>To: $TO<br>Subject: Test email with curl<br>Test email<br>EOF<br><br>##encrypted authenticated SMTP with TLS (port 587/TCP):<br><br>$# curl -sv smtp://$SMARTHOST:587 --ssl --mail-from $FROM --mail-rcpt $TO --upload-file email.txt --user &quot;$AUTH_USERNAME:$AUTH_PASSWORD&quot; --anyauth<br><br>## unencrypted SMTP (port 25/TCP):<br><br>$# curl -sv smtp://$SMARTHOST --mail-from $FROM --mail-rcpt $TO --upload-file email.txt</pre><h4>How to Apply &amp; Verify Configuration in Openshift</h4><p>Extract the Alert manager config file from the cluster</p><pre>oc -n openshift-monitoring get secret alertmanager-main --template=&#39;{{ index .data &quot;alertmanager.yaml&quot; }}&#39; |base64 -d &gt; alertmanager.yaml</pre><p>Update the alert manager configuration file as per your requirement and replace the new definition.</p><pre>oc -n openshift-monitoring create secret generic alertmanager-main --from-file=altermanager.yaml --dry-run -o=yaml |  oc -n openshift-monitoring replace secret --filename=-</pre><ul><li>Wait for Alert Manager to reload it’s configuration.</li></ul><pre>oc logs alertmanager-main-0 -c alertmanager -f -n openshift-monitoring<br>ts=2024-12-18T04:52:06.955Z caller=cluster.go:705 level=info component=cluster msg=&quot;gossip not settled&quot; polls=0 before=0 now=2 elapsed=2.000358988s<br>ts=2024-12-18T04:52:08.596Z caller=coordinator.go:113 level=info component=configuration msg=&quot;Loading configuration file&quot; file=/etc/alertmanager/config_out/alertmanager.env.yaml<br>ts=2024-12-18T04:52:08.596Z caller=coordinator.go:126 level=info component=configuration msg=&quot;Completed loading of configuration file&quot; file=/etc/alertmanager/config_out/alertmanager.env.yaml</pre><p>Simulate the alert generation ,and you are done! Alert Manager should now redirect <strong>critical</strong> and other severity errors to your Emails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*a7i_v9iFiNuBquRi2piSNw.png" /><figcaption>Email Alert</figcaption></figure><p><strong>Disclaimer</strong>:</p><p>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners.</p><p>LinkedIn: https://www.linkedin.com/in/nnreddy51/</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f8a8aa5c202" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hunting the Memory Monster: How Large Machine Configs Crashed My OpenShift ClusteThe Curious Case…]]></title>
            <link>https://medium.com/@arjun0451/hunting-the-memory-monster-how-large-machine-configs-crashed-my-openshift-clustethe-curious-case-feb0a3070d2d?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/feb0a3070d2d</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Sun, 02 Mar 2025 07:52:07 GMT</pubDate>
            <atom:updated>2025-10-29T07:32:36.564Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Large Machine Configs Crashed My OpenShift Cluster</h3><blockquote>It All Started With a Cluster Outage…..</blockquote><p>One fine day, I got an alert—OpenShift was completely unresponsive. No console access, no API access, and worst of all, I couldn’t even SSH into the master nodes. Something was seriously wrong.</p><h4>First Signs of Observations</h4><ol><li><strong>Master nodes were getting hung</strong> even after rebooting, they became unresponsive again.</li><li>The OpenShift API and web console were completely inaccessible.</li><li><strong>CPU usage on the master nodes was stuck at 100% for a long time and did not come down to normal again</strong>, making them practically unusable.</li></ol><p><em>At this point, I had no idea what was causing the issue, but I knew I had to act fast</em>. Let’s systematically approach the case</p><p>Step 1: Since I couldn’t SSH, the first thing I did was <strong>reboot a master node manually</strong>.</p><p><strong>After rebooting</strong>, I quickly logged in before the node could hang again. I immediately ran top command to capture the top CPU and memory-consuming process details.</p><pre>top -o %MEM<br>top -o %CPU</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*G3gz8Yluy1peFlwG6zcR4w.png" /><figcaption>CPU usage pattern</figcaption></figure><p>And what I saw <strong>shocked me. kube-apiserver was consuming an insane amount of memory!! machine-config process followed by kube-apiserver on the table top in terms of memory utilization.</strong></p><p>This explained why the master nodes were getting hung. The API server was overloading the system, causing a full lockup. But why?</p><p>Step 2: Digging into kube-apiserver</p><p>Next, I needed to find out <strong>what was making kube-apiserver consume so much memory</strong>.</p><p>Checking API Server Resource Usage.</p><pre>oc adm top pods -n openshift-kube-apiserver<br><br>kube-apiserver-master-1: 8GB memory usage<br>kube-apiserver-master-2: 6GB memory usage<br>kube-apiserver-master-3: 4GB memory usage</pre><p>One thing was clear: <strong>Something inside the API server cache was eating all the memory</strong>.</p><p>Step 3: Finding the Culprit (Too Many Large Objects in etcd)</p><p>Since the API server loads objects from <strong>etcd</strong>, I checked which objects were <strong>large in size</strong>.</p><pre>export ETCD_POD_NAME=$(oc get pods -n openshift-etcd -l app=etcd --field-selector=&quot;status.phase==Running&quot; -o jsonpath=&quot;{.items[0].metadata.name}&quot;)<br><br>oc exec -n openshift-etcd -c etcdctl ${ETCD_POD_NAME} -- sh -c &quot;etcdctl get / --prefix --keys-only  | grep -oE &#39;^/[a-z|.]+/[a-z|.|8]&#39; | sort | uniq -c | sort -rn&quot; | while read KEY; do printf &quot;$KEY\t&quot; &amp;&amp; oc exec -n openshift-etcd ${ETCD_POD_NAME} -c etcdctl -- etcdctl get ${KEY## } --prefix --write-out=json | jq &#39;[.kvs[].value | length] | add &#39; | numfmt --to=iec ; done | sort -k3 -hr | column -t<br><br>1200   /kubernetes.io/machineconfiguration.openshift.io    1300M<br>450   /kubernetes.io/operators.coreos.com                 33M<br>561  /kubernetes.io/secrets                              27M<br>992   /kubernetes.io/configmaps                           20M<br>304   /kubernetes.io/apiserver.openshift.io               11M<br>191   /kubernetes.io/apiextensions.k8s.io                 7.4M</pre><p><strong>Result:</strong> <strong>Over 1200 MachineConfig objects! and the overall size of MC objects is above 1GB.</strong></p><pre>for mc in $(oc get mc -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>    size=$(oc get mc &quot;$mc&quot; -o json | wc -c)<br>    echo &quot;$mc: $(awk &quot;BEGIN {print $size/1024}&quot;) KB&quot;<br><br>rendered-master-1abcdd: 900.588 KB<br>rendered-master-2caddd: 901.104 KB<br>rendered-worker-3wdsds: 996.236 KB<br>rendered-worker-423dsf: 842.146 KB</pre><p><strong>Why is this a problem?</strong></p><ul><li>The API server <strong>loads MachineConfig objects into memory</strong>, so <strong>1GB+ objects quickly overload RAM</strong>.</li><li>Every MachineConfigPool render was <strong>creating a new object (~1MB each)</strong>, leading to unnecessary duplication.</li><li>The more MachineConfig objects, the bigger the API server’s cache , causing <strong>100% CPU spikes</strong> as it struggled to keep up.</li></ul><p>This was <strong>the root cause</strong> — <strong>huge MachineConfig objects overloading kube-apiserver</strong> and <strong>crashing master nodes</strong>.</p><p>Step 4: Fixing the Issue</p><p>Once I knew the problem, the fix became clear: <strong>clean up old MachineConfig objects and optimize their size.</strong></p><p>Why are stale Machineconfigs objects not getting cleared automatically? Openshift won’t perform the auto GC of the older machine configs,and as per <a href="https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/machine_configuration/machine-configs-garbage-collection">offical documentation</a></p><blockquote>The Machine Config Operator (MCO) does not perform any garbage collection activities. This means that all rendered machine configs remain in the cluster. Each time a user or controller applies a new machine config, the MCO creates new rendered configs for each affected machine config pool. Over time, this can lead to a large number of rendered machine configs, which can make working with machine configs confusing. Having too many rendered machine configs can also contribute to disk space issues and performance issues with etcd.</blockquote><p>However this oc adm prune approach works above OpenShift 4.16+ . So for OCP version below 4.16, the unused MCP clean up should be through manual approach.</p><ol><li>Find the unused machineconfig objects</li></ol><pre>oc get mcp<br># check if any mcp is in degrade or updating status<br># if all the mcp are in normal status the current mcp version is the latest mc version rendered on mcp pool.<br></pre><p>2. Take the backup of the unused machineconfig objects</p><pre>#!/bin/bash<br>for mc in $(oc get mc -o jsonpath=&#39;{.items[*].metadata.name}&#39;); do<br>  oc get mc $mc -o yaml &gt; $mc.yaml<br>done</pre><p>3. Check the Etcd Database size before Housekeeping, the total size of the DB is around 1.3 GB</p><pre>oc exec -n openshift-etcd -c etcdctl ${ETCD_POD_NAME} -- sh -c &quot;etcdctl endpoint status -w table&quot;<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>| https://10.20.1.1:2379 | aasdasdea79bc20aa |   3.5.6 |  1.5 GB |     false |      false |       761|  990115878 |          990114878 |        |<br>| https://10.20.2.2:2379 | a5sadd3ed5a1b1dar |   3.5.6 |  1.5 GB|      true |      false |       761 |  990115878 |          990114878|        |<br>| https://10.20.3.3:2379 | asd6bff66043de9ad |   3.5.6 |  1.5 GB|     false |      false |       761|  990115878 |          990114878|        |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+</pre><p>4. Deleting Unused MachineConfig Objects</p><pre>### Master mc clean up older than 7 days<br>for MC in $(oc get mc -o json | jq -r &quot;<br>  .items[] |<br>  select(.metadata.creationTimestamp | . != null and (fromdateiso8601 &lt;= \&quot;$(date --date=&#39;-7day&#39; +%Y-%m-%dT%H:%M:%SZ)\&quot;) ) |<br>  select(.metadata.name | startswith(\&quot;rendered\&quot;)) |<br>  select(.metadata.ownerReferences[].name == \&quot;master\&quot;) |<br>  .metadata.name&quot;); do<br>  echo $MC;<br>  oc delete mc $MC;<br>done</pre><pre>### worker mc clean up older than 7 days<br>for MC in $(oc get mc -o json | jq -r &quot;<br>  .items[] |<br>  select(.metadata.creationTimestamp | . != null and (fromdateiso8601 &lt;= \&quot;$(date --date=&#39;-7day&#39; +%Y-%m-%dT%H:%M:%SZ)\&quot;) ) |<br>  select(.metadata.name | startswith(\&quot;rendered\&quot;)) |<br>  select(.metadata.ownerReferences[].name == \&quot;worker\&quot;) |<br>  .metadata.name&quot;); do<br>  echo $MC;<br>  oc delete mc $MC;<br>done</pre><p>4. Check the ETCD database size post housekeeping of unused MC objects</p><pre>oc exec -n openshift-etcd -c etcdctl ${ETCD_POD_NAME} -- sh -c &quot;etcdctl endpoint status -w table&quot;<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>|         ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+<br>| https://10.20.1.1:2379 | aasdasdea79bc20aa |   3.5.6 |  310 MB |     false |      false |       762|  990115898 |          990114879 |        |<br>| https://10.20.2.2:2379 | a5sadd3ed5a1b1dar |   3.5.6 |  311 MB|      true |      false |       762 |  990115898 |          990114879|        |<br>| https://10.20.3.3:2379 | asd6bff66043de9ad |   3.5.6 |  310 MB|     false |      false |       762|  990115879 |          990114879|        |<br>+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+</pre><p>Check the etcd pods if the compaction process triggered and marked stale object for delection and monitor the dbsize and dbinsize of etcd to figure out the etcd auto defrag task kick in, in general etcd compaction will trigger every 5 min to deleting old version objects.</p><pre>oc exec $ETCD_POD_NAME -c etcd -n openshift-etcd -- etcdctl endpoint status --cluster -w json | jq &#39;.[].Status|&quot;dbSize: &quot; + (.dbSize|tostring) + &quot;, dbSizeInUse: &quot; + (.dbSizeInUse|tostring) + &quot;, (dbSize-dbSizeInUse)/dbSize =&gt; &quot; + ((.dbSize - .dbSizeInUse)/.dbSize*100|tostring)+&quot;%&quot;&#39;<br>&quot;dbSize: 1571761152, dbSizeInUse: 1252084224, (dbSize-dbSizeInUse)/dbSize =&gt; 33.191886472312206%&quot;<br>&quot;dbSize: 15376242176, dbSizeInUse: 1252080128, (dbSize-dbSizeInUse)/dbSize =&gt; 33.00023661034663%&quot;<br>&quot;dbSize: 1569250304, dbSizeInUse: 1252076032, (dbSize-dbSizeInUse)/dbSize =&gt; 32.733019778366927%&quot;</pre><pre>(dbSize-dbSizeInUse)/dbSize =&gt; 45%<br><br>If the above ratio is above 45%, the ETCD operator will auto trigger the defragmentation task and reduce the DB size.</pre><p>After applying these fixes, I monitored the cluster.</p><ol><li><strong>Master node CPU usage dropped from 100% to a stable 30–40%.</strong></li><li><strong>Memory usage on kube-apiserver reduced from 8GB to ~2GB.</strong></li><li><strong>Cluster API and Console became accessible again.</strong></li><li><strong>No more SSH lockups!</strong></li></ol><h3>Final Thoughts</h3><p>This issue was a <strong>great reminder</strong> that <strong>small misconfigurations can snowball into major outages</strong>. Keeping API objects optimized and avoiding unnecessary cache loads is <strong>critical</strong> for OpenShift stability.</p><p>Have you faced similar API server performance issues? Let’s discuss this in the comments!</p><p><strong>Disclaimer</strong>:</p><p>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=feb0a3070d2d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Storage Deep Dive: Persistent Volumes and CSI Drivers Explained]]></title>
            <link>https://medium.com/@arjun0451/kubernetes-storage-deep-dive-persistent-volumes-and-csi-drivers-explained-6ba9f36c7ad7?source=rss-6893836c0a25------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ba9f36c7ad7</guid>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[csi-driver]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[pvc]]></category>
            <category><![CDATA[volumeattachment]]></category>
            <dc:creator><![CDATA[Nagarjuna Reddy]]></dc:creator>
            <pubDate>Sun, 29 Dec 2024 08:22:02 GMT</pubDate>
            <atom:updated>2024-12-29T08:22:02.361Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Kubernetes has transformed the way we build, deploy, and manage applications, and storage is a vital component of this ecosystem. Persistent Volumes (PVs) allow stateful workloads to persist data beyond the lifecycle of a Pod, while Container Storage Interface (CSI) drivers ensure seamless interaction between Kubernetes and storage backends.</p><p>This article offers a practical, step-by-step guide to understanding the lifecycle of Kubernetes Persistent Volumes and their interaction with CSI drivers. It also addresses real-world scenarios like error handling and rescheduling, making it a must-read for Kubernetes practitioners and enthusiasts.</p><h3>Core Concepts</h3><h3>Persistent Volumes (PVs)</h3><ul><li>Kubernetes abstractions for storage, decoupling the storage lifecycle from Pods.</li><li>Can be dynamically or statically provisioned.</li></ul><h3>Persistent Volume Claims (PVCs)</h3><ul><li>Requests for storage by Pods. PVCs bind to PVs based on the requested capacity and access modes.</li></ul><h3>CSI Drivers</h3><ul><li>Plugins enable Kubernetes to provision, attach, detach, and manage storage from external backends.</li></ul><h3>StorageClass</h3><ul><li>A blueprint for creating PVs dynamically. Defines parameters such as storage type, size, and access modes.</li></ul><h3>Detailed Workflow</h3><p>This section outlines the step-by-step interactions in the Kubernetes storage lifecycle, enhanced with real-world insights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_NWKHlvnT42o0qXaGC1Sig.png" /></figure><h3>1. Volume Provisioning</h3><ul><li>A Pod creates a PVC specifying its storage requirements.</li><li>Kubernetes uses the associated StorageClass to request volume provisioning via the CSI driver.</li><li>The CSI driver communicates with the storage backend, provisions the volume, and returns its metadata.</li><li>The PV is created and bound to the PVC.</li></ul><h3>2. Volume Attachment</h3><ul><li>When a Pod is scheduled, the CSI Node Plugin attaches the volume to the node.</li><li>The volume is mounted, making it accessible to the application.</li></ul><h3>3. Volume Usage</h3><ul><li>Application Pods seamlessly perform I/O operations on the volume via the CSI Node Plugin.</li><li>These operations are passed to the storage backend for execution.</li></ul><h3>4. Volume Detachment</h3><ul><li>Upon Pod termination or rescheduling, the volume is detached from the node.</li><li>The volume remains intact and can be re-attached to another node(by CSI-Driver).</li></ul><h3>5. Rescheduling Scenario</h3><ul><li>In case a Pod is rescheduled to a new node, the volume is first detached from the old node and then attached to the new one(CSI driver).</li></ul><h3>Key Real-World Insights</h3><h3>1. Pod Access Despite Issues with CSI Driver Daemon Set</h3><ul><li>If a CSI Node Plugin DaemonSet faces issues like ImagePullBackOff, the Pod can continue to use the attached volume as long as it remains on the same node.</li><li>This is because the CSI driver primarily involves volume provisioning, attachment, and detachment not runtime I/O operations.</li></ul><h3>2. Error Handling</h3><ul><li>In storage provisioning failure (e.g., backend issues), the PVC remains Pending.</li><li>Application Pods depending on this PVC , It won’t start until the PVC issue is resolved.</li></ul><h3>Practical Use Case</h3><p>Scenario: Rescheduling a Pod using a dynamically provisioned volume.</p><ol><li>The Pod is terminated on its original node.</li><li>The volume is detached from the node by the CSI Node Plugin.</li><li>The Pod is rescheduled to a new node.</li><li>The volume is attached to the new node and mounted for the Pod.</li></ol><p>Result: Seamless access to the same data on the new node.</p><h3>Frequently Asked Questions (FAQ)</h3><h3>1. Does the CSI driver cause slowness in storage operations?</h3><ul><li>CSI drivers are not typically the cause of <strong>I/O slowness</strong>. The slowness is often due to:</li><li>Storage backend issues (e.g., overloaded disks or insufficient throughput).</li><li>Misconfigured StorageClass (e.g., inappropriate performance tiers or access modes).</li><li>Network bottlenecks (e.g., high latency between Kubernetes nodes and the storage backend).</li></ul><h3>2. What happens if CSI driver Pods are not running?</h3><ul><li>Existing Pods: Continue to operate normally.</li><li>New Pods: Fail to start due to volume operations reliance on the CSI driver.</li></ul><h3>3. What happens if the volume file or data is missing?</h3><p>Pods encounter read/write errors or fail to start.</p><p>Check the events in the application namespace and describe the PV and PVC for more details to identify the issue.</p><blockquote>kubectl get events -n &lt;namespace&gt;</blockquote><blockquote>kubectl describe pvc -n &lt;namespace&gt;</blockquote><blockquote>kubectl describe pv</blockquote><h3>4. Can I delete a Persistent Volume manually?</h3><p>Deleting a PV directly is not recommended</p><blockquote>use kubectl delete pvc.</blockquote><h3>5. <strong>Can i Dynamically resize a PV ?</strong></h3><p>Yes, If your storage backend and CSI driver support volume expansion.</p><p>Check the storage clause definition ifallowVolumeExpansion: true then dynamic resizing support is configured in the storage clause level.</p><h3>6. What is the effect of force-deleting a pod using a mounted volume?</h3><p>Force deleting a pod(kubectl delete pod — force) might leave the volume in an attached state.</p><p>This can prevent other pods from using the volume until it is detached.</p><p>Always allow graceful pod termination or detach the volume manually if needed.</p><h3>7. What happens if a PVC is deleted while the pod is running?</h3><p>Deleting a PVC with a running Pod may lead to undefined behavior, especially if the reclaim policy is Delete.</p><p>The Pod might continue to use the volume temporarily, but future operations could fail.</p><p>Best practice: Always delete the Pod before deleting the PVC.</p><p><strong>Disclaimer</strong>:</p><p>The views and insights expressed in this piece are solely those of the author and do not represent the beliefs of their employer. These reflections are firmly based on personal experience. While every attempt has been made to ensure the information is accurate, the author cannot guarantee its correctness; it is ultimately the reader’s responsibility to verify and apply the content wisely. All trademarks mentioned are the property of their respective owners.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ba9f36c7ad7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>