<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tobias Massoth on Medium]]></title>
        <description><![CDATA[Stories by Tobias Massoth on Medium]]></description>
        <link>https://medium.com/@tobias_80378?source=rss-8ad957956480------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SB3julADjiPguKNnx3qfnQ.jpeg</url>
            <title>Stories by Tobias Massoth on Medium</title>
            <link>https://medium.com/@tobias_80378?source=rss-8ad957956480------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:53:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tobias_80378/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AKS Platform-Initiated Upgrades: What You Need to Know Before June 2027]]></title>
            <link>https://medium.com/@tobias_80378/aks-platform-initiated-upgrades-what-every-kubernetes-administrator-needs-to-know-before-june-2027-65f92fda9e54?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/65f92fda9e54</guid>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 08:15:48 GMT</pubDate>
            <atom:updated>2026-11-16T08:17:32.787Z</atom:updated>
            <content:encoded><![CDATA[<p>You might think: “<em>It is November 2026 and this guy is talking about June 2027… MONTHS TO GO!”</em></p><p>True. But time flies and depending on how many AKS clusters you maintain, what the grade of automation is, potentially already some unsupported AKS versions which are in use and you are permanently to keep up to date, June 2027 will be coming soon. Once it is there and you run unsupported AKS versions, you will remember this guy who published a blog post in November 2026 ;-)</p><p>This post is about Microsoft’s new upgrade policy (called “platform-initiated upgrades”) for AKS which enforces to run a supported AKS version and what you can do to always stay in control of your AKS clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eb3mEbpdWry3pIYjD3fftw.png" /></figure><h3>Introduction</h3><p>Managing Kubernetes clusters comes with an ongoing responsibility: Keeping them secure, supported, and up to date.</p><p>For organizations running Azure Kubernetes Service (AKS), Kubernetes upgrades have traditionally been managed through manual upgrades, automatic upgrade channels, or Azure Kubernetes Fleet Manager.</p><p>But what happens when a cluster remains on an unsupported Kubernetes version for too long?</p><p><strong>Starting June 1, 2027, AKS will begin automatically upgrading clusters that remain on unsupported Kubernetes (minor) versions after the applicable platform support grace period.</strong></p><p>While this policy strengthens security and platform reliability, it also introduces operational considerations for teams managing production workloads.</p><h3>What Are AKS Platform-Initiated Upgrades?</h3><p>A platform-initiated upgrade is an automatic Kubernetes version upgrade triggered by Azure Kubernetes Service when a cluster remains on an unsupported version beyond its grace period.</p><p>Unlike customer-managed upgrades, which are controlled through maintenance windows and upgrade policies, platform-initiated upgrades are triggered by the AKS platform itself without control when the upgrade shall happen.</p><p>The primary objective is simple:</p><p><strong>Ensure that AKS clusters are eventually brought back to supported Kubernetes versions, reducing the security and operational risks associated with outdated software.</strong></p><p>When a Kubernetes version reaches end of life, it no longer receives community security patches, bug fixes, or vulnerability remediation.</p><p>Continuing to operate on such versions can expose organizations to security vulnerabilities, compatibility issues, and reduced support coverage.</p><p>Platform-initiated upgrades address this problem by preventing clusters from remaining permanently on unsupported versions.</p><h3>Initial rollout: June 2027</h3><p>Microsoft has announced that the initial rollout begins on <strong>June 1, 2027</strong>. After this date, the <strong>60-day platform support grace period </strong>applies to unsupported clusters. In this period you get notified that you run an unsupported AKS version. If there is no action to upgrade to a supported AKS version, a platform-initiated upgrade will be enforced after the grace period of 60 days.</p><h3>How Platform-Initiated Upgrades Differ from Regular AKS Upgrades</h3><p>Platform-initiated upgrades behave differently from customer-controlled upgrades in several important ways. These differences can have significant implications for production environments.</p><h4>Maintenance windows are not guaranteed</h4><p>Normally, AKS administrators configure maintenance windows to schedule upgrades during periods of low application traffic.</p><p>However, platform-initiated upgrades can occur outside these windows.</p><p>For example, even if a production cluster has a maintenance window configured for Sunday morning, a platform-initiated upgrade could begin during a busy weekday.</p><p>This makes application resilience and upgrade readiness particularly important.</p><h4>Deprecated API checks are bypassed</h4><p>Kubernetes regularly deprecates and removes APIs as new versions are released. During customer-managed upgrades, AKS can detect breaking API changes that could affect workloads.</p><p>Platform-initiated upgrades do not stop because of deprecated API usage. If an application, controller, or custom resource depends on an API removed in the target Kubernetes version, it may stop functioning after the upgrade.</p><p>This makes proactive compatibility testing essential.</p><h4>Pod Disruption Budgets may be overridden</h4><p>Pod Disruption Budgets (PDBs) help protect application availability by controlling how many pods can be voluntarily disrupted at once.</p><p>During a platform-initiated upgrade, AKS first attempts a normal node drain. However, if the drain fails or times out because of restrictive PDBs, AKS can override those restrictions to continue the upgrade.</p><p>As a result, applications may experience disruption if they are not designed to tolerate node replacements.</p><h4>Upgrade capacity settings are best effort</h4><p>Node surge settings normally control how much additional node capacity AKS creates during an upgrade. Platform-initiated upgrades attempt to honor these configurations.However, if the required surge capacity is unavailable, AKS can fall back to upgrading nodes in place, one at a time, using the equivalent of maxSurge=0 and maxUnavailable=1.</p><p>This can temporarily reduce available workload capacity.</p><h3>Which Kubernetes Version Will AKS Choose?</h3><p>AKS selects the <strong>lowest supported Kubernetes version within the cluster’s current support tier</strong>, using the latest available patch release for that minor version.</p><p>For example:</p><ul><li>A cluster on the community-support track is upgraded toward the lowest currently supported community minor version.</li><li>A cluster enrolled in LTS is upgraded to the applicable supported LTS version.</li></ul><p>If a cluster is several minor versions behind (let’s assume 1.28 while the lowest supported AKS version is 1.33), AKS upgrades it incrementally because Kubernetes control-plane upgrades cannot skip minor versions.</p><p>The control plane is upgraded first, followed by the node pools. During node upgrades, nodes are reimaged with the appropriate image for the target Kubernetes version. The underlying node OS SKU is not automatically changed. AKS also preserves the cluster’s billing tier and LTS enrollment.</p><h3>How to Identify AKS Clusters at Risk</h3><p>Azure Resource Graph provides a convenient way to identify affected AKS clusters. You can run the following query in Azure Resource Graph Explorer:</p><pre>resources<br>| where type =~ &#39;microsoft.containerservice/managedclusters&#39;<br>| project<br>    clusterName = name,<br>    resourceGroup,<br>    subscriptionId,<br>    location,<br>    kubernetesVersion = properties.kubernetesVersion,<br>    tier = properties.sku.tier,<br>    supportPlan = properties.supportPlan,<br>    upgradeChannel = properties.autoUpgradeProfile.upgradeChannel,<br>    powerState = properties.powerState.code<br>| order by kubernetesVersion asc, clusterName asc</pre><p>This query retrieves the cluster name, resource group, subscription, region, Kubernetes version, support plan, configured upgrade channel and power state. The results can be mapped against the <a href="https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions">official AKS Kubernetes release calendar</a> to identify versions approaching or exceeding their end-of-life dates.</p><h3>How to Avoid Platform-Initiated Upgrades</h3><p>The best way to avoid a platform-initiated upgrade is straightforward:</p><p><strong>Keep the AKS clusters on supported Kubernetes versions and manage upgrades proactively.</strong></p><p>There are several way to achieve it:</p><h4>Use a supported automatic upgrade channel</h4><p>For AKS Standard clusters, consider configuring the stable automatic upgrade channel.</p><p>The stable channel automatically keeps clusters on a supported Kubernetes minor version.</p><p>This method might best fit to non-production environments to quickly identify issues coming with the AKS version upgrade.</p><p>AKS Automatic clusters already use the stable channel by default.</p><h4>Monitor upgrade notifications</h4><p>Microsoft provides several mechanisms for identifying clusters affected by the policy, including Azure Resource Health, Azure Advisor, portal notifications, and AKS Communication Manager.</p><p>AKS Communication Manager can send notifications about upcoming and completed platform-initiated upgrades. Organizations should integrate these signals into their existing operational monitoring and change-management processes.</p><h4>Consider Azure Kubernetes Fleet Manager</h4><p>For organizations managing multiple AKS clusters, Azure Kubernetes Fleet Manager can simplify upgrade orchestration.</p><p>It allows teams to define update groups, stage upgrades across environments, and validate changes before moving into production.</p><p>For example, upgrades can progress from development to staging and finally production.</p><h3>What Happens If a Platform-Initiated Upgrade Fails?</h3><p>Platform-initiated upgrades can encounter operational blockers.</p><p>Examples include insufficient subnet IP capacity, unavailable VM SKUs, incompatible add-ons, resource locks or Azure Policy restrictions.</p><p>If an upgrade fails, AKS automatically retries it according to a platform-determined schedule. The cluster remains unsupported until a successful upgrade completes. Administrators can investigate failure details through Azure Resource Health and the AKS diagnostic tools.</p><p>Platform-initiated upgrades cannot be canceled after they start, and AKS does not support downgrading the cluster back to its previous unsupported Kubernetes version.</p><h3>What This Means for Responsible Teams</h3><p>The introduction of platform-initiated upgrades reinforces an important change in how organizations should think about managed Kubernetes operations.</p><p>A managed Kubernetes service reduces the burden of running Kubernetes infrastructure, but it does not eliminate responsibility for application compatibility, upgrade testing, and lifecycle governance.</p><p>For the responsible team, this means Kubernetes version management should become part of regular operational planning. Teams should maintain a current cluster inventory, monitor support deadlines, establish repeatable upgrade procedures, and continuously test application compatibility.</p><p>There are some industries that do have high compliance standards and requirements or in other situations there is only low budget to perform a tested upgrade every 2–3 months. In these cases, the best way forward is to make use of AKS Long-Term Support (LTS). It costs money but it gives more freedom and time to upgrade in a controlled and tested way.</p><h3>Final Thoughts</h3><p>Think and plan ahead to always upgrade and stay on a supported version even if it adds effort and maybe headache. If ignored, I promise, the platform-initiated upgrade will make you headache and add even more effort and stress when you figure out that something broke and the app does not come up anymore or there are any other issues.</p><p><strong>My message for AKS administrators is clear: Upgrade on your terms before the platform has to upgrade for you.</strong></p><h3>References</h3><ul><li><a href="https://learn.microsoft.com/en-us/azure/aks/platform-initiated-upgrades">Microsoft Learn — Platform-initiated upgrades in AKS</a></li><li><a href="https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions">Microsoft Learn — Supported Kubernetes versions in AKS</a></li><li><a href="https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-cluster">Microsoft Learn — Automatically upgrade an AKS cluster</a></li><li><a href="https://learn.microsoft.com/en-us/azure/aks/planned-maintenance">Microsoft Learn — Planned maintenance for AKS</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65f92fda9e54" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure SRE Agent — A Closer Look at Network Integration]]></title>
            <link>https://medium.com/@tobias_80378/azure-sre-agent-a-closer-look-at-network-integration-c181c9609386?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/c181c9609386</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[azuresreagent]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 18:55:24 GMT</pubDate>
            <atom:updated>2026-07-06T18:55:24.055Z</atom:updated>
            <content:encoded><![CDATA[<p>When writing this blog post, Azure SRE Agent has already been become GA, however, there are plenty of features still being in public preview. One very important and most awaited feature is called “<strong>Network Integration</strong>”.</p><p>This feature enables the Azure SRE Agent with <strong>various network settings</strong>. One of these capabilities supports network restrictions by whitelisting specific public domains. Another one supports virtual network integration which allows connectivity to workloads only being reachable via private IP addresses, such as virtual machines but also private AKS cluster using private endpoints.</p><p>Specifically the latter capability is the reason why you need to make use of the <strong>Azure SRE Agent Network Integration</strong> feature. In many customer environments, there are strict security requirements to only allow access to workloads via its private IP address since public endpoint are not allowed.</p><p>In this blog post, we will focus on enabling the network integration feature, followed by some insights how Azure SRE Agent is integrated.</p><p>Before the Azure SRE Network Integration feature can be used, it is important to toggle “<strong>early access to features</strong>” in the <strong>Basics </strong>section of the Azure SRE Agent — this is required as long as Network Integration feature is still on <strong>public preview</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W8XHAg7AW30M1Lc9tIxsEg.jpeg" /></figure><p>The registration process may take several minutes until “<strong>Workplace Configuration</strong>” becomes available. Once it is available, there are three different egress modes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaU8AVWN1rF0w-AbYL0J-w.jpeg" /></figure><p>The <strong>three Networking Egress modes</strong> are the following.</p><ul><li><strong>Unrestricted (default)</strong>: Allows all outbound traffic from the agent to any publicly accessible service or endpoint without restrictions. This is the default mode.</li><li><strong>Limited</strong>: This mode denies outbound traffic by default.<br>Only explicitly whitelisted domains via public Internet are reachable.</li><li><strong>Azure VNet</strong>: The agent integrates into a delegated subnet of an Azure virtual network. Any VNet configurations such as DNS, routing firewalling, get automatically applied.<br>This mode is most suitable if private and enterprise-controlled egress is required.</li></ul><h3>Limited Mode</h3><p>Before we jump to the most restrictive Network Integration mode, let’s first have a look at the “<strong>limited</strong>” mode.<br>There are some preconfigured whitelisting items and collections ready to use such as <strong>package registry </strong>and <strong>code repositories </strong>which just can be toggled…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*D-KiwDMVFCZq7nzCDma5BA.jpeg" /></figure><p>… and in addition several Azure public endpoints which can be allowed as well. This is just a small subset of the sheer number of Azure services which are available. For those (most Azure services) custom hosts/domains can be added if required.</p><p>All preconfigured whitelisting items make use of the Azure SRE infra network which is the default network route the agent uses to communicate with publicly facing services as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*2X01aVAiJyAqrLjGQbYg7A.jpeg" /></figure><h3>Azure VNet Mode</h3><p>If there is a need to let route all traffic through a custom VNet, the “<strong>Azure VNet”</strong> mode is the right choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*VUgGHp5jhsl-aKqnPqYASQ.png" /></figure><p>In order to integrate Azure SRE Agent into a subnet, the following <strong>three requirements</strong> must be met:</p><ul><li>VNet must be in same region where the Azure SRE Agent has been deployed</li><li>Subnet must be of size /28 or larger</li><li>Subnet must be delegated to Microsoft.App/environments</li></ul><p><strong>Personal remark</strong>: The agent will only consume a single IP address from the selected subnet. Hence, the agent will not benefit from a larger subnet as mentioned in the Microsoft requirements.</p><p>Once the required information have been provided, the agent is integrated quite quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*6ys4gkYoJ35IXPcSbkitmw.png" /></figure><p>When I completed this process, I immediately sent a prompt which took just little more time (few more seconds than usual) to start thinking. This is because the agent was still restarting / reloading since it is running inside a container with Azure Linux 3.</p><p>When I checked the “connected devices” setting in the VNet, no network device appeared. So, I was not sure whether the integration worked fine. The consumed IP address by the agent is not displayed anywhere.</p><p>So, I set up a totally different VNet in a different region (West Europe) and deployed a Windows-based virtual machine with IIS installed. The VNet into which the Azure SRE Agent was integrated had been deployed to the Sweden Central region previously. Both networks got peered.</p><p>Then, I submitted a prompt to sent an HTTP request to the private IP address of the VM and drum-roll … IT WORKED!</p><p>The agent was able to establish a connection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ltVD286iEFjGpDVjyJfwtQ.jpeg" /></figure><p>Now, the more interesting question is, what is the IP address the agent uses to connect to the VM?</p><p>To find out, I submitted another prompt and executed the following command to see what the source IP address is and it is 10.20.0.14.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*q_7odD9IrcilWs2_EXvnbw.jpeg" /></figure><p>This <strong>IP address will always be the same</strong> no matter if the agent is used directly using an interactive chat by a user, scheduled task or automatic response to an incident as part of the incident response plan.</p><p>I wanted to figure out if an agent shutdown will reset the IP address, however, the same IP address was kept. Once I disconnected the agent from the VNet and started the vnet integration process from scratch, a different IP address was selected. However, there is no math behind (or at least acknowledged by me) — a randomly selected IP address from the subnet was selected.</p><p>I also wanted to know if the IP address has been directly assigned to the agent itself or somewhere used as a kind of a gateway or something similar. The agent consumes an IP address from within the range 169.254.0.0 and there are multiple hops between the agent and the VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*DuPj2txASTpLESEmPNPr_A.jpeg" /></figure><p>For any services which are only accessible via its private IP address, the Azure SRE Agent makes use of the same tools which have been enabled. So, as an example, the agent would be able to connect to a private AKS cluster to list the running pods or to connect to the database and query individual tables inside a database.</p><p>However, there is still some work to do by the Microsoft product team and room for improvement, e.g. listing all running pods in a private AKS cluster requires command invocation which on the other side is not recommended to have this AKS functionality enabled. This is one example where Microsoft security recommendations hit Azure SRE Agent in accessing private AKS cluster.</p><h3>Conclusion</h3><p>The Azure SRE Agent consumes a random IP address from the selected subnet. Hence, if there are strict firewall requirements in place which is typically the case in a privately routed network, it is best to use the subnet’s IP range to allow connectivity to workloads which are available only via their private IP addresses.<br>The VNet configuration (such as DNS) is used by the Azure SRE Agent, so domains and other DNS records can be resolved.<br>The Azure SRE Agent can establish a connection to services outside of Azure, such as on-premises if an Azure ExpressRoute or Azure S2S VPN has been set up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c181c9609386" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AKS Upgrade Channels NodeImage vs. node-image — What’s the Difference?]]></title>
            <link>https://medium.com/@tobias_80378/aks-upgrade-channels-nodeimage-vs-node-image-whats-the-difference-dfc98010a449?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfc98010a449</guid>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Thu, 14 May 2026 20:15:42 GMT</pubDate>
            <atom:updated>2026-05-14T20:15:42.324Z</atom:updated>
            <content:encoded><![CDATA[<p>This topic can be a kind of confusing and this is what I experience sometimes in conversations when talking about Terraform and AKS autoupgrade capabilities. Both values ‘NodeImage’ and ‘node-image’ are basically used to manage how the operating system is patched however both belong to different autoupgrade capabilities which will be covered in this post.</p><p>The feature <strong>cluster autoupgrade</strong> in AKS is not new and can be used to ensure that AKS performs Kubernetes version upgrades and OS patches as desired. There are various upgrade channels available from which one can be configured and made use of. The channels are the following:</p><ul><li><strong>rapid</strong>: The cluster will be automatically upgraded to the latest supported patch release on the latest supported minor version.</li><li><strong>stable</strong>: The upgrade will be automatically upgraded to the latest supported patch release on the second latest minor version available for AKS.</li><li><strong>patch</strong>: The cluster will be automatically upgraded to the latest supported patch version while keeping the current minor version always the same.</li><li><strong>node-image (legacy)</strong>: The Kubernetes version will <strong>NOT </strong>be upgraded. Instead, only the node image will be automatically upgraded to the latest image version available keeping the operating system patched.<br>This ‘node-image’ channel has already been flagged as legacy. More information below.</li><li><strong>none</strong>: The cluster will never be upgraded — neither AKS version nor node images. Upgrades must be handled separetely, e.g. by using Azure Kubernetes Fleet Manager, Terraform or manual action</li></ul><p>As you can see, the big problem is the following: <br>When AKS has been configured to get auto upgraded, its nodes will only receive the latest node image when the upgrade is performed. In case the next AKS version (no matter if will be just a patch or minor version upgrade) is weeks away and there may be newer node images available, the OS of the cluster nodes will not be patched in the meanwhile.</p><p>To make sure that the operating system of the AKS cluster nodes are always running the latest node image version by using the channel “node-image”, AKS patch versions will never be upgraded.</p><p>That said, the cluster autoupgrade feature does not support both, upgrade of AKS patches/minor versions AND node images configured at the same time.</p><p>So, we now covered ‘node-image’ as part of the AKS <strong>cluster autoupgrade</strong> feature. However, where is ‘NodeImage’ joining the party?</p><p>Microsoft introduced the so-called <strong>OS autoupgrade</strong> feature to specifically cover patches for operating systems (no matter if it is Ubuntu or Azure Linux) while cluster autoupgrade covers upgrades of the AKS version in parallel. Similar to cluster autoupgrade, the OS autoupgrade feature does also have different channels as described next:</p><ul><li><strong>NodeImage</strong>: Nodes will be automatically updated with a newly patched VHD containing security fixes and bug fixes on a weekly cadence. Node image upgrades are supported as long as cluster Kubernetes minor version is still in support.</li><li><strong>SecurityPatch</strong>: Nodes will be automatically updated with security patches only. AKS patches nodes live without draining pods if node reimaging is not necessary. However, sometimes it might be necessary to upgrade the node’s virtual hard disk (VHD) with patches. In both scenarios, the configured schedule will be considered.</li><li><strong>Unmanaged</strong>: The OS built-in patching mechanism automatically applies OS updates without reboot even if it would be required. Newly allocated machines are initially unpatched and will be patched based on the OS&#39;s configuration once per day around 06:00 UTC. Windows doesn&#39;t automatically apply security patches.<br>An additional tool such as kured is required to manage the reboot.</li><li><strong>none</strong>: Nodes don&#39;t have security updates applied automatically.</li></ul><p>Both cluster autoupgrade and OS autoupgrade have been quickly introduced with its upgrade channels. However, it is worth to mention that the older ‘node-image’ channel has been flagged as legacy and it is not recommended anymore to make use of it . Instead, the channel ‘NodeImage’ as part of OS autoupgrade should be used.</p><p>In addition, Microsoft clearly recommends to make use of both, cluster autoupgrade for AKS version upgrades (primarily ‘stable’) and OS autoupgrade for OS upgrades (primarily ‘NodeImage’). For each, distinct schedules can be configured.</p><p>It is important to mention that there might be some technical limitations which have not been highlighted in this post since the focus is specifically on getting the understanding right on ‘NodeImage’ and ‘node-image’. <br>Links to both features with additional information and limitations are provided below.</p><p>Link to cluster autoupgrade:<br><a href="https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-cluster">https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-cluster</a></p><p>Link to OS autoupgrade:<br><a href="https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-os-image">https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-os-image</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfc98010a449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure SRE Agent in Action — A First Experience]]></title>
            <link>https://medium.com/@tobias_80378/azure-sre-agent-in-action-a-first-experience-066453f7a5ea?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/066453f7a5ea</guid>
            <category><![CDATA[azuresreagent]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[sre]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Tue, 03 Mar 2026 21:35:15 GMT</pubDate>
            <atom:updated>2026-03-12T20:30:43.558Z</atom:updated>
            <content:encoded><![CDATA[<p>The focus of Site Reliability Engineering (SRE) is on creating reliable, scalable and highly automated systems. <strong>Azure SRE Agent</strong> supports this discipline by providing additional features through connected systems and products. Various tools and services such as Azure Kubernetes Service, GitHub, Azure SQL Database among many others are supported.</p><p>Due to this rich set of integrations, Azure SRE Agent is capable to run end-to-end workflows in order to react to certain incidents. It is also capable to address potential issues which have not been arise as incidents yet in a proactive way.</p><p>In this blog post, we will cover some basics but will also scratch at the surface of sub-agents and some real world experiences gathered so far what it means to make use of Azure SRE Agent from a cost perspective but also other views.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jtkuVKVsbQJcDwgIsnaPXg.png" /></figure><h3>The Power of Azure SRE Agent</h3><p>As already introduced in the intro, Azure SRE Agent consists of many capabilities and integrations due to various supported services and products. Therefore, it becomes essential part of an entire ecosystem and supports the following:</p><ul><li><strong>Monitoring and observability services</strong> such as Azure Monitor (specifically alerts, workbooks, metrics), DataDog, Azure Log Analytics, Applications Insights, Grafana</li><li><strong>Incident management </strong>such as ServiceNow and PagerDuty</li><li><strong>Source control and CI/CD</strong> such as GitHub and Azure DevOps</li><li><strong>Data sources</strong> such as Azure Data Explorer and MCP servers</li></ul><p>Azure SRE Agent is capable to automate the analysis and incidents resolution. When an issue has already been logged in ServiceNow, it is also possible to trigger the agent automatically.</p><p>In addition to that, Azure SRE Agent can run on a schedule if workflows have been configured properly. Those <strong>workflows </strong>(also called <strong>scheduled tasks</strong>) can be described in more details what the agent shall explicitly do. Similar to classic prompts. The more precise the prompt is, the better the result will be. To limit the resources which the agent shall analyze, it is possible to provide Azure Resource IDs and other specific information.</p><p>Azure SRE Agent does also have <strong>memory system</strong> built-in. It means, whenever a task or basically a prompt has been completed, the agent remembers what the steps were and is also able to reduce the amount of steps from the initial run when a similar or the same issue appears. In addition, the memory can also be preloaded by providing runbooks, knowledge base articles and other files which can be easily uploaded.</p><p>The Agent can be deployed in two different permission levels to keep control what the agent is allowed to do:</p><ul><li><strong>Reader</strong>: Read-only permissions on the resources it manages</li><li><strong>Privileged</strong>: Take approved actions on resources it has access to</li></ul><p>While you might want to make use of the privileged permission level, there might be certain prompts which shall not be applied autonomously on Azure resources (such a database). Hence, you want first to review the analysis before the action gets executed. The scheduled task offers the capability to configure the agent autonomy level. The autonomy level can be configured in two ways. The <strong>autonomous level (default)</strong> executes the prompt directly while the <strong>review level</strong> first asks for approval by a human before modifying or applying the next step (such as restarting a service).</p><p>Based on my experience, I highly recommend only to integrate and use tools which are really useful and beneficial. It does not make sense to integrate any kind of tool and let run Azure SRE Agent against anything you have in your environment without additional value. The reason is that Azure SRE Agent is not cheap and you are billed per second the Azure SRE Agent runs. The more information are accessible and used, the longer the task might run and your bill goes up. We will come back to the cost topic later again.</p><h3>Initial Deployment of Azure SRE Agent</h3><p>The deployment of an Azure SRE Agent is quite straight forward and guided very well. Similar to other Microsoft services (Foundry, Copilot, etc.), there is a separate portal to manage all agents. As of writing this blog post, only three Azure regions are supported:</p><ul><li>Sweden Central</li><li>East US 2</li><li>Australia East</li></ul><p>Other Azure regions will be added sometime later in the future, but definitely when this Azure service will become GA.</p><p>When opening the agent set up, there is few information which need to be provided as usual. At the bottom, you also have the choice to create a new or use an existing Application Insights resource.</p><p>Application Insights will automatically deploy an Azure Log Analyticss Workspace if a new one shall be created. This is required because Application Insights stores all LLM interactions and tool executions for logging and auditing reasons in the Azure Log Analytics workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*Ou7I685saVH9XbxV3s3aPA.png" /></figure><p>The next steps ask for supported Azure resources to onboard. This step can be skipped as well in case the resources do not exist. Otherwise, the relevant resource group can be selected immediately.</p><p><strong>Be cautious:</strong> If multiple supported Azure resources (such as AKS and Azure SQL Database) reside in the same resource group, all of them will be onboarded at the same time. There is no way to exclude them for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcHJ-BVifQ9u2DdyFBO1Gg.png" /></figure><p>Now, the most important step is to decide if the agent shall be running in read-only (reader) or privileged mode. Depending on the choice, there are different permissions being added to the managed identity which will be assigned to the previous selected and supported resources. In case there are other supported services selected, the list of permissions will vary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kA-WOx8KE83DrYo9Bl3Wpw.png" /></figure><p>Finally, the resource groups will be displayed one more time before …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dIW9W6cWc0wXQgYrZXkyLg.png" /></figure><p>… the agent gets deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DF57v2F-kw1q-SmpWMdXAA.png" /></figure><p>At the end of the deployment, the Azure SRE Agent is ready to be configured as desired. From this time onwards, the Azure SRE Agent will be billed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*72mQjvv8MZlM3vt2YFpGSw.png" /></figure><p>As soon as the agent has been created, it monitors all resources which have been selected during the deployment steps. The agent now examines system data and learns about resource’s behavior. Depending on the tasks and configuration, the agent is ready to respond to problems, no matter how the application scales or what the complexity level is. This behavior and everything related is called always-on flow.</p><h3>Azure SRE Default Agent and Sub-agents</h3><p>The Azure SRE Agent can be used to configure scheduled tasks. The so-called default agent uses built-in functionalities. In this case it runs the prompt as described in the task details. On the screenshot, it is clearly described, if there are any issues which can be solved by restarting a service or single cluster node, the default agent shall continue.</p><p>However, the agent autonomy level has been set to “<strong>review</strong>”<strong> </strong>which means before anything will be restarted, the action first needs to be approved by a human. A simple “yes” after reviewing what exactly is to be restarted is absolutely sufficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X1yMjyAZlC0q0rWjsi4jHQ.png" /></figure><p>This is a quite low-effort scheduled task. However, much more complex prompts but also services and tools can be integrated as well which are not part of the built-in functionalities of the default agent.</p><p>For this purpose, Azure SRE Sub-agents can be used to define those scenarios. Azure CLI and MCP servers can be connected to cover and execute custom actions.</p><p>Sub-agents is a much more complex topic which would blow up this post. Therefore, a separate one is planned.</p><h3>Cost Contol</h3><p>During the Azure SRE Agent deployment, we already recognized that several Azure services must be deployed in order to make use of the SRE functionalities. Starting from Application Insights along with Azure Log Analytics workspace, alert rule up to the actual Azure SRE Agent. These are the basic services if you just want to onboard an Azure Kubernetes Service cluster.</p><p>Most likely you already know that Azure Log Analytics workspace can become cost intensive the more logs and events are pumped into. Gladly, any task or interaction which has been executed so far, will be stored in Azure Log Analytics workspace. Hence, no pod logs nor other information will be stored. Nevertheless, depending on how many tasks and interactions have been run, the bill already goes up.</p><p>From my perspective, the billing driver is not Azure Log Analytics workspace but the Azure SRE Agent itself. There are two different action types which lead to costs:</p><ul><li>Always-on flow (base fee, billed per hour)</li><li>Active flow (billed per second everytime a task runs or the agent is prompted)</li></ul><p>The base fee (always-on flow) occurs for any Azure SRE Agent no matter if the agent runs or is stopped.</p><p>In addition to the cost for always-on flow, there is an additional cost per task and per second it runs — active flow. Let’s assume there is 1 task which runs 4 times each 300 seconds. These 4 task executions already issue a billing of ~28 EUR when Azure SRE has been deployed in the Azure region Sweden Central.</p><p>The good thing is, if the agent is waiting for your input/response, the wait time is not billed.</p><p>Some final words to costs: As already mentioned earlier, it is wise not to integrate any tools you have in order to keep the execution time of tasks as low as possible and make only use where appropriate.</p><p>This basically leads to the next section.</p><h3>Azure SRE Agent as a Shared Service</h3><p>Would it be a good or bad decision to use Azure SRE Agent as a shared service for multiple applications?</p><p>From a cost perspective it might be a good reason to have a common one for multiple applications because the cost for always-on flow only needs to be payed once.</p><p>However, there are several factors why it is NOT a good decision to use it as a shared service.</p><h4>Scenario #1 — limited permissions in a multi app environment</h4><p>First of all, imagine there is one team responsible for several applications. All applications have strong dependencies to each other. However, there are few people or maybe freelancers who are brought in with access to only a single application due to confidential data stored in the other ones. Let’s assume, her name is Tina and she shall also use Azure SRE Agent to act with the app environment and services she has access to. <br>She gets the SRE Agent Standard User assigned, so she is able to run prompts and she has ONLY access to the AKS cluster on which the application runs which she is allowed to access.</p><p>So, Tina can access the Azure SRE Agent and run any prompt she would like. Let’s imagine she prompts the following: <br>“list all AKS clusters to which the Azure SRE managed identity does have access to”</p><p>So, the Azure SRE Agent does not care nor evaluate Tina’s permissions but uses its own context and permissions to check the environment and resources to which the Azure SRE Agent has access to. <br>This leads to the following output. Tina does only have access to aks-tina, however, the agent returns information about the AKS cluster aks-rob to which she does not have any access. <br>I want to highlight that this behavior is expected and works as designed since the permissions of the SRE managed identity have a higher precedence to the permissions of Tina.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GOVoUSzwOkj3mmNsgVv8FA.png" /></figure><p>Since the Azure SRE Agent runs in privileged mode, it is allowed to run the following command “kubectl get pods -A” against AKS cluster named aks-rob.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdqiiJJSbi1kQFZG3rP17A.png" /></figure><p><strong>Conclusion</strong>: The Azure SRE Agent’s permissions execute prompts without checking the user’s permissions if the user is allowed or not. This works by design and is documented on MSDocs. <br>In those kind of scenarios where team members have different permissions, it must be strongly evaluated if team members should retrieve permissions on Azure SRE Agent or not. <br>If All team members shall get permissions and work with Azure SRE Agent, it is required to deploy multiple Azure SRE Agents to have clear boundaries in place but costs will increase.</p><h4>Scenario #2 — Memory system</h4><p>Azure SRE Agent memory system improves knowledge handling and responses by learning from each session. The memory system can be built up by adding runbooks, team standards, knowledge base articles such as service-specific context and more. Each time an issue occurs, the agent makes use of the memory and is able to provide better resolutions to repeating issues, similar ones or just as described in KB articles.</p><p>Let’s assume multiple applications had been onboarded to the same Azure SRE Agent and applications share the same error messages or even error codes (because of laziness of some developers), the response from the agent might be misleading or not applicable to all application because same error code might be handled differently for multiple applications.</p><p>One way to fix this problem would be to make the prompt very precise, e.g. if this error code 123 occurs in App A (with Azure Resource ID DEF), the following service needs to be restarted. However, if the same error code 123 occurs in App C (with Azure Resource ID XYZ), the cache must be cleared.</p><p>If it is not possible to have such a separation in place (and to be honest, earlier or later somebody will forget to make a prompt much more precise), the only way is to have dedicated Azure SRE Agents deployed for each application.</p><p><strong>Some final words:</strong> Azure SRE Agent is a very powerful service and there are many opportunities and ways to automate analysis and incident resolution. However, it must be planned very well and I hope that some of my previous experiences help to get started with Azure SRE Agent in a smoother way.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=066453f7a5ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replacement Strategy for OpenSource NGINX on Azure Kubernetes Service]]></title>
            <link>https://medium.com/@tobias_80378/replacement-strategy-for-opensource-nginx-on-azure-kubernetes-service-8e325040b5c3?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e325040b5c3</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Fri, 06 Feb 2026 15:53:41 GMT</pubDate>
            <atom:updated>2026-02-06T15:53:41.957Z</atom:updated>
            <content:encoded><![CDATA[<p>In November 2025, there was an announcement of the Open Source NGINX retirement in March 2026 and only little time left to act accordingly. Any NGINX replacement by a different tool can take a lot of effort and time to get it into production. I have already seen companies struggling because the roadmap has not considered the retirement and therefore needs some repriotization or even the release process for new tools might consume a lot of time. <br>In order to get more time for the replacement, there are couple of ways and I would like to highlight one (AKS Managed NGINX or also called application routing add-on) which requires only little effort and gives you much more time to select the right tool for the final replacement while extended support is offered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Abwyaaz84gY196auNhNhAw.png" /></figure><h3>The First Move</h3><p>As described above, there is a low-effort activity to get Open Source NGINX replaced. The replacement is again NGINX, however, it is provided by Microsoft and available as Managed NGINX for Azure Kubernetes Service, also called application routing add-on.</p><p>Microsoft already committed to provide extended support and security fixes (based on criticality) for NGINX until November 2026 if the AKS application routing add-on is being used.</p><p>In order to make use of the AKS application routing add-on, the Open Source NGINX needs to be removed. Before it can be removed, it is required to first identify what the ingress class name is which needs to be reused later. The Ingresses can stay alive because it is compatible to the new IngressController, hence only the existing IngressController needs to be replaced. For this purpose, the application routing add-on base deployment but also the YAML Manifest for the new IngressController need to be prepared — in this scenario, an internal Azure Load Balancer will be deployed.</p><p>Before running any of the next commands, it is wise to run and test in a non-production environment first.</p><h4>1. Prepare Application Routing Add-on Base Deployment</h4><p>The following command will enable the AKS Managed NGINX add-on but will not deploy anything else. This command can be run right away on existing AKS clusters.</p><pre>RESOURCE_GROUP=aks-rg<br>CLUSTER_NAME=aks01<br><br>az aks approuting update \<br>  --resource-group $RESOURCE_GROUP \<br>  --name $CLUSTER_NAME \<br>  --nginx None</pre><h4>2. Prepare YAML Manifest for the IngressController Deployment</h4><p>For the IngressController configuration, there are two lines which need to be adjusted in the YAML manifest.<br><strong>1. ingressClassName:</strong> The same IngressClass as identified earlier must be referenced here. This will ensure that existing Ingresses use the new IngressController since the old IngressClass name is identical to the new IngressClass name of the application routing add-on.<br><strong>2. azure-load-balancer-internal-subnet:</strong> Here, the name of the VNet subnet from which an IP address shall be allocated must be referenced. This must be any subnet name of the same VNet in which the AKS cluster nodes have already been deployed.</p><pre>apiVersion: approuting.kubernetes.azure.com/v1alpha1<br>kind: NginxIngressController<br>metadata:<br>  name: nginx<br>spec:<br>  ingressClassName: &lt;name of previous/existing ingressClassName&gt;<br>  controllerNamePrefix: nginx<br>  loadBalancerAnnotations: <br>    service.beta.kubernetes.io/azure-load-balancer-internal: &quot;true&quot;<br>    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: &quot;&lt;name of vnet subnet&gt;&quot;</pre><h3>Apply the Magic</h3><p>Since the add-on has already been enabled and the switch over shall now be applied, the following steps need to be completed sequentially.</p><ol><li>The Open Source NGINX needs to be deleted from the AKS cluster. Once this has been completed successfully, any applications making use of the NGINX Ingress will not be reachable anymore. This depends how the OSS NGINX Ingress has been deployed originally.</li><li>By running the YAML manifest, a new internal Azure Load Balancer will be deployed and get available as LoadBalancer in the AKS cluster. The External-IP of the IngressController should reflect an IP address from the selected VNet subnet.</li><li>Once the steps 1 and 2 have been completed, the applications should be available again.</li></ol><h3>What’s Next?</h3><p>Once the previous commands have been executed, the replacement of the Open Source NGINX by the application routing add-on has been completed successfully.</p><p>Now, you have won a little bit more time to evaluate the right tool for your AKS clusters before the next deadline in November 2026 ends.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e325040b5c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Kubernetes Service OIDC Authentication using Self-Hosted Keycloak Instead of Entra ID]]></title>
            <link>https://medium.com/microsoft/azure-kubernetes-service-oidc-authentication-using-self-hosted-keycloak-instead-of-entra-id-33e90285c2ba?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/33e90285c2ba</guid>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[oidc]]></category>
            <category><![CDATA[keycloak]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 15:47:48 GMT</pubDate>
            <atom:updated>2026-02-02T15:47:48.976Z</atom:updated>
            <content:encoded><![CDATA[<p>The Microsoft Product Group for Azure Kubernetes Service recently released a new feature to support <strong>authentication other than Entra ID</strong> in order to access and maintain Azure Kubernetes Service clusters. For this purpose <strong>external identity providers</strong> can be configured and used for individual AKS clusters.</p><p><strong>Any external identity provider using the industry-standard OpenID Connect (OIDC) is supported</strong>. GitHub and Google Identity are officially documented on Microsoft Docs. However, other OIDC identity providers such as <strong>Keycloak </strong>are not. While GitHub and Google Identity are both SaaS products, there might be also some reasons to make use of <strong>self-hosted OIDC identity providers </strong>like <strong>Keycloak</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yI56BM4Y0AjbGF6tx2nm6Q.png" /></figure><p><strong>Use cases for such as self-hosted OIDC identity provider </strong>are the following but not limited:</p><ul><li><strong>Multiple identity providers shall be used</strong> simultaneously on a single cluster (from same company or in a mixed set up with external suppliers)</li><li><strong>Identities must not be synchronized</strong> to a SaaS ID provider (such as Google Identity, Entra ID, etc.)</li><li>There is an <strong>existing Keycloak instance available on-premises which shall be used across any Kubernetes cluster</strong> for authentication, no matter where it is hosted</li><li><strong>Custom claim validation and user mapping rules</strong> are absolutely required to further validate and restrict access</li><li><strong>On-premises CI/CD pipelines</strong> must be able to access Azure Kubernetes Service the<strong> same way users authenticate and access</strong></li></ul><p>In this blog post, we will first go through some basics and prerequisites followed by the deployment of a fresh AKS cluster and the OIDC identity provider configuration for Keycloak. After, we will configure Keycloak and AKS in a way we can make use of OIDC authentication using standard flow.</p><h3>Supported AKS Configurations</h3><p>Before we jump into the details, it is worth to highlight what the considerations are in order to make use of external OIDC authentication:</p><p>First, the self-hosted Keycloak instance <strong>MUST be publicly available</strong>. The reason is that the AKS ApiServer needs to reach the instance.</p><p>Second, it does not matter whether the AKS cluster has been set up in <strong>local accounts with Kubernetes RBAC</strong> or <strong>Microsoft Entra ID authentication with Azure RBAC </strong>— all types of AKS authentication are supported and external OIDC identity authentication can be put on top.</p><h3>Supporting Documents</h3><p>Before we start to deploy anything, I would like to highlight the following two Microsoft Docs pages, which first explain the feature and prerequisites and secondly the technical implementation for GitHub and Google Identity.</p><p>Some steps which will be shown below make use of such samples from Microsoft Docs, specifically the use of the technical implementation for Google Identity.</p><ul><li><strong>Explanation and Prerequisites: </strong><br><a href="https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-overview">https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-overview</a></li><li><strong>Technical implementation: </strong><br><a href="https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-configure?pivots=google-identity">https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-configure?pivots=google-identity</a></li></ul><p>That being said, let’s start now.</p><h3>Initial Set up and Preparation</h3><p>First of all, we need to make sure that the feature (at the time of writing this blog post, it is still in public preview) is available.</p><p>So, we need to register the feature as shown below. The third command might not be executed from your point of view because it might already be registered. However, this command propagates the previous configuration.</p><pre>az feature register --name JWTAuthenticatorPreview --namespace Microsoft.ContainerService<br>az feature show --name JWTAuthenticatorPreview --namespace Microsoft.ContainerService<br>az provider register --namespace Microsoft.ContainerService</pre><p>Once this is done, we can start deploying the Azure Kubernetes Service cluster. Both variables shown below will be used several times. Therefore, update them once.</p><pre>RESOURCE_GROUP=rg-keycloak<br>NAME=aks-keycloak01</pre><p>After, the Azure Resource Group will be deployed…</p><pre>az group create \<br>    --name $RESOURCE_GROUP \<br>    --location &quot;westeurope&quot;</pre><p>… and the AKS cluster afterwards.</p><pre>az aks create \<br>    --name $NAME \<br>    --resource-group $RESOURCE_GROUP \<br>    --network-plugin azure \<br>    --network-plugin-mode overlay \<br>    --pod-cidr 192.168.0.0/16 \<br>    --network-dataplane cilium \<br>    --node-count 1 \<br>    --enable-aad \<br>    --enable-azure-rbac \<br>    --generate-ssh-keys</pre><p>Once it has been deployed, we also need an ingress controller to publish the Keycloak instance. For this purpose, we make use of the application routing add-on.</p><pre>az aks approuting enable --resource-group $RESOURCE_GROUP --name $NAME</pre><p>And finally, we require the credentials to initially deploy Keycloak.</p><pre>az aks get-credentials \<br>    --resource-group $RESOURCE_GROUP \<br>    --name $NAME \<br>    --overwrite-existing</pre><h3>Keycloak Basic Deployment</h3><p>There is a quite good guide to get started. However, only the very basics are covered. For a more comprehensive and hardened deployment, you should read the documentation and follow additional instructions.</p><p>Link: <a href="https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start">https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start</a></p><p>As per the documentation, the following Kubernetes YAML file contains all configs for a basic Keycloak deployment.</p><pre>wget &quot;https://raw.githubusercontent.com/keycloak/keycloak-quickstarts/refs/heads/main/kubernetes/keycloak.yaml&quot;</pre><p>For the deployment itself, we create a namespace called <strong>keycloak </strong>first and deploy the resources.</p><pre>kubectl create ns keycloak<br>kubectl -n keycloak apply -f keycloak.yaml</pre><p>Once Keycloak has been deployed successfully and all Pods are up and running, the next required step is to login with the admin account. For this purpose we will also update the password because the Keycloak instance will be accessible from the public Internet. Until now it is not publicly available, so let’s use <strong>kubectl port-foward</strong>.</p><pre>kubectl -n keycloak port-forward keycloak-0 8080:8080</pre><p>As the default login credentials, we use <strong>admin:admin</strong> and update the password afterwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J5KjPSFQVb5eUvCddFG5YA.png" /></figure><p>When accessing the Keycloak instance via the web browser, we also see that the website is not trusted. The reason is that there is no signed certificate from a public CA (does not matter if this is localhost or an FQDN we will use later). Therefore, we need to create a certificate.<br>This certificate will also be required by Azure to communicate with Keycloak via Port 443 in a secure manner.</p><h3>Create and Let Sign a Certificate by Public CA</h3><p>Ideally would be to have a signed certificate from a public CA. Therefore, I decided to request one from my favorite certificate issuing provider which is Namecheap.</p><p>I used the following command to create a certificate signing request (CSR) and the respective private key. The resulting certificate and private key will be required later.</p><pre>openssl req -new -newkey rsa:3072 -nodes -keyout keycloak.test-massoth.de.key -out keycloak.test-massoth.de.csr</pre><p>I am not detailing the steps to buy a certificate or to complete the request.<br>After the CSR process was successfully completed, I had the following files in the same directory in my WSL instance. Both files ending with <strong>crt </strong>and <strong>bundle </strong>are new and both are encoded in the required PEM format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*WX-JcVWYKKD6lVd2f7YcOg.png" /></figure><p>I encountered problems later when applying the certificate only. The reason was that the intermediate and root certificates from my chosen public CA are not well-known everywhere. Azure Kubernetes Service reported an <strong>502 Bad Gateway </strong>error when trying to reach the Keycloak instance. Therefore, I had to combine both <strong>bundle </strong>and <strong>crt </strong>files in one file. I used the following command.</p><pre>cat keycloak.test-massoth.de.crt keycloak.test-massoth.de.bundle &gt; keycloak.test-massoth.de.crt-bundle</pre><p>After, this new-bundle file can be used to create the Secret resource.</p><pre>kubectl -n keycloak create secret tls keycloak-cert --cert=keycloak.test-massoth.de.crt-bundle --key=keycloak.test-massoth.de.key</pre><p>All what we need now is an Ingress resource to make the Keycloak instance available from public Internet. This YAML references the new namespace called <strong>keycloak </strong>but also contains <strong>keycloak.test-massoth.de</strong> as the FQDN to make Keycloak available if accessed e.g. in web browser.</p><p>For this purpose I created another subdomain under <strong>test-massoth.de </strong>earlier and referenced the public IP which is used by the nginx controller for incoming traffic (can be find easily using <strong>kubectl get services -A</strong>).</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: keycloak<br>  namespace: keycloak<br>spec:<br>  ingressClassName: webapprouting.kubernetes.azure.com<br>  tls:<br>  - hosts:<br>    - keycloak.test-massoth.de<br>    secretName: keycloak-cert<br>  rules:<br>  - host: keycloak.test-massoth.de<br>    http:<br>      paths:<br>      - backend:<br>          service:<br>            name: keycloak<br>            port:<br>              number: 8080<br>        path: /<br>        pathType: Prefix</pre><p>The Ingress resource was applied wit the following command…</p><pre>kubectl apply -f ingress-keycloak.yaml</pre><p>… and Keycloak is accessible via web browser. We are able to login with the admin user and the updated password from the previous step.</p><h3>Implement Keycloak Configuration</h3><p>Since this is an entirely new instance, we first need to create a new realm which will be referenced in our configuration later. This realm is called <strong>aks</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JbR8tzLinVY_lJ-7h-iL6A.png" /></figure><p>Once this is done, we need to switch to this realm. Afterwards, we need to create a client. This client is used in our configuration later as well — similar to a Service Principal — which will be used to authenticate against the realm and to legitimate users to perform the authentication.</p><p>This client is called <strong>aks-keycloak </strong>with client type <strong>OpenID Connect</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6D-z73csWkHur129PIaew.jpeg" /></figure><p>In the next step, we need to enable <strong>Standard Flow </strong>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ma2ILBjNiwplIaxTH-rTZw.png" /></figure><p>… and finally enter redirect URIs and a web origin address. First, we will test whether Keycloak does work in general. The values below will be replaced in a future step to make AKS authentication work.</p><p>The values are:</p><ul><li>Valid redirect URIs: <a href="https://www.keycloak.org/app/*">https://www.keycloak.org/app/*</a></li><li>Web origins: <a href="https://www.keycloak.org/app/*">https://www.keycloak.org</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zy9bUvPBQ3gNVuYRm8TnZw.png" /></figure><p>The client configuration has now been completed. Before we can check the client configuration, we first need a user we can use for authentication.</p><p>For this purpose, we need to switch to the user set up and create a new user called <strong>aks-tobias</strong>, including First name and Last name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7HwzkLHADVgfmCQHo2E57Q.png" /></figure><p>After the user has been created, let’s open the user properties and set the credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEk3zMKqndOyGM8NOuetww.png" /></figure><p>Now, we have everything created to check redirection but also authentication. We need to go to this website …</p><p><a href="https://www.keycloak.org/app/">https://www.keycloak.org/app/</a></p><p>… and enter the following values.</p><ul><li><strong>Keycloak URL: https://keycloak.test-massoth.de</strong></li><li><strong>Realm: aks</strong></li><li><strong>Client: aks-keycloak</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VU_-Ve0tVu8Bgchcj6AFEA.png" /></figure><p>After clicking on <strong>Save</strong>, we are redirected to our Keycloak instance. We will be asked to enter a valid username and password. We use the user account that has been created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*RMuzXEa0UfA4YarhtaAB1Q.png" /></figure><p>The authentication is working fine and we are being welcomed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqQX65r8ERSVQbcgiaGkGg.png" /></figure><p>In order to improve security, we will be configuring <strong>client authentication </strong>in the client <strong>aks-keycloak</strong>. Once this is configured, we will not be able to repeat previous steps because the Keycloak website does not support client authentication.</p><p>We need to go back to our Keycloak instance and enable <strong>client authentication</strong> in the client context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*hT8F3tI3WZbwBHo9AMd5wA.jpeg" /></figure><p>On the <strong>Credentials </strong>tab, we need to select <strong>Client Id and Secret </strong>for <strong>Client Authenticator </strong>and <strong>generate </strong>a <strong>client secret.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8asvkcV42QOQiSD7ifQjDg.png" /></figure><p>In addition to the previous configuration, we also need to update the redirect URIs. This is required to perform local tests for Kubernetes client (kubectl) authentication but also for AKS authentication later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lrYmBkJrqC1iJMdHTmiuoQ.png" /></figure><p>Once this has been completed, we are ready to run our first Kubernetes related authentication test.</p><h3>Configure OIDC Authentication and Perform Tests</h3><p>In order to make kubectl work with OIDC authentication, best is to first need to install the <strong>OIDC plugin oidc-login</strong>. For this purpose we first install krew as plugin manager. The installation process is described quite well here: <a href="https://krew.sigs.k8s.io/docs/user-guide/setup/install/">https://krew.sigs.k8s.io/docs/user-guide/setup/install/</a></p><p>Once this is done, we install <strong>oidc-login</strong> as plugin with the following command.</p><pre>kubectl krew install oidc-login</pre><p>Finally, we are ready to run the next command which will perform the OIDC authentication against our Keycloak instance.</p><pre>kubectl oidc-login get-token \<br>--oidc-issuer-url=https://keycloak.test-massoth.de/realms/aks \<br>--oidc-client-id=aks-keycloak \<br>--oidc-client-secret=&lt;SECRET FROM ABOVE&gt;</pre><p>If you receive the following error, you know that your certificate is not trusted.</p><blockquote>error: get-token: authentication error: oidc error: oidc discovery error: Get “<a href="https://keycloak.test-massoth.de/realms/aks/.well-known/openid-configuration">https://keycloak.test-massoth.de/realms/aks/.well-known/openid-configuration</a>&quot;: <strong>tls: failed to verify certificate: x509: certificate signed by unknown authority</strong></blockquote><p>There are two reasons:</p><ul><li>You used a self-signed certificate</li><li>The intermediate/root certificate is unknown and cannot be verified (as mentioned above, you need the certificate bundle)</li></ul><p>You could add an additional parameter (insecure-skip-tls-verify) to the previous command to ignore the certificate error. <br>I clearly recommend to check the created Kubernetes Secret resource instead of using the additional parameter.</p><p>However, let’s assume the command ran successfully and as a result, we retrieve a token. The token exists of three parts separated by a dot.</p><blockquote>{“kind”:”ExecCredential”,”apiVersion”:”client.authentication.k8s.io/v1beta1&quot;,”spec”:{“interactive”:false},”status”:{“expirationTimestamp”:”2026–01–15T12:31:12Z”,”token”:”<strong>&lt;part #1&gt;.&lt;part #2&gt;.&lt;part #3&gt;</strong>”}}</blockquote><p>In order to better understand what this token contains, we use the web tool <a href="https://jwt.ms/">jwt.ms</a> from Microsoft. Very interesting is the middle part of the token which reflects information about the OIDC instance but also claims.</p><blockquote>[..]<br>.{ “exp”: 1768480272, <br>“iat”: 1768479972, <br>“auth_time”: 1768479687, <br>“jti”: “2c128702-ec81–7465-dddb-7c61c457056b”, <br>“iss”: “https://keycloak.test-massoth.de/realms/aks&quot;, <br>“aud”: “aks-keycloak”, <br>“sub”: “d280dfc5-fd59–4621-bfee-7de83247b9cd”, <br>“typ”: “ID”,<br>“azp”: “aks-keycloak”,<br>“nonce”: “lx7goy4Z9dbzYP32YQKe6jM_vVg9jKi-tZrzMRDkPx4”, <br>“sid”: “GXDWIC-uCIFcFavJhdVQJ_J6”, <br>“at_hash”: “lJEqSyUYCPR0yQrX6u1YNg”, <br>“acr”: “0”, <br>“email_verified”: false, <br>“name”: “Tobias Massoth”,<br>“preferred_username”: “aks-tobias”, <br>“given_name”: “Tobias”, <br>“family_name”: “Massoth”, <br>“email”: “test@test-massoth.de” }.<br>[..]</blockquote><p>For the next steps we need to consider the following properties:</p><ul><li>iss = <a href="https://keycloak.test-massoth.de/realms/aks">https://keycloak.test-massoth.de/realms/aks</a></li><li>aud = aks-keycloak</li></ul><p>Both properties are required to make our OIDC identity provider instance approachable from the AKS cluster. For this purpose we need to prepare the configuration file which will be uploaded to our AKS instance.</p><p>The following JSON file <strong>jwt-config.json</strong> has been reused from the <a href="https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-configure?pivots=google-identity#create-jwt-authenticator-configuration">Microsoft website (link)</a> and adjusted. The value mentioned in <strong>url</strong> has been replaced by the one from <strong>iss</strong> and <strong>audiences</strong> by <strong>aud</strong>.</p><p>The other information such as validation rules and claim mappings have been adjusted to reference <strong>keycloak </strong>instead of <strong>google</strong>. The prefix <strong>aks:jwt:</strong> is absolutely required and we will cover this part later.</p><pre>{<br>    &quot;issuer&quot;: {<br>        &quot;url&quot;: &quot;https://keycloak.test-massoth.de/realms/aks&quot;,<br>        &quot;audiences&quot;: [<br>            &quot;aks-keycloak&quot;<br>        ]<br>    },<br>    &quot;claimValidationRules&quot;: [<br>        {<br>            &quot;expression&quot;: &quot;has(claims.sub)&quot;,<br>            &quot;message&quot;: &quot;must have sub claim&quot;<br>        }<br>    ],<br>    &quot;claimMappings&quot;: {<br>        &quot;username&quot;: {<br>            &quot;expression&quot;: &quot;&#39;aks:jwt:keycloak:&#39; + claims.sub&quot;<br>        }<br>    },<br>    &quot;userValidationRules&quot;: [<br>        {<br>            &quot;expression&quot;: &quot;has(user.username)&quot;,<br>            &quot;message&quot;: &quot;must have username&quot;<br>        },<br>        {<br>            &quot;expression&quot;: &quot;!user.username.startsWith(&#39;aks:jwt:keycloak:system&#39;)&quot;,<br>            &quot;message&quot;: &quot;username must not start with &#39;aks:jwt:keycloak:system&#39;&quot;<br>        }<br>    ]<br>}</pre><p>Once the file has been adjusted and saved, we can run and upload the file with the following command. The execution of the command takes round about 5 minutes.</p><pre>az aks jwtauthenticator add \<br>--resource-group $RESOURCE_GROUP \<br>--cluster-name $NAME \<br>--name external-auth \<br>--config-file jwt-config.json</pre><p>Any adjustments to the JSON file can be easily applied by just repeating the previous command with <strong>update </strong>instead of <strong>add.</strong></p><p>Once it has been successfully applied, it will be returned on command line. We are now ready to authenticate against the AKS cluster.</p><p>For this purpose, we need to prepare our kubeconfig file.<br>It is easier to first run the command <strong>az aks get-credentials </strong>and to have the config file in place. Then the objects <strong>contexts </strong>and <strong>users</strong> can be adjusted easier.<br>The object <strong>clusters </strong>has not been changed.</p><pre>apiVersion: v1<br>clusters:<br>- cluster:<br>    certificate-authority-data: &lt;omitted data&gt;<br>    server: https://aks-kc01-dns-718s1opp.hcp.westeurope.azmk8s.io:443<br>  name: aks-keycloak01<br>contexts:<br>- context:<br>    cluster: aks-keycloak01<br>    user: aks-user<br>  name: aks-keycloak01<br>current-context: aks-keycloak01<br>kind: Config<br>users:<br>- name: aks-user<br>  user:<br>    exec:<br>      apiVersion: client.authentication.k8s.io/v1beta1<br>      command: kubectl<br>      args:<br>      - oidc-login<br>      - get-token<br>      - --oidc-issuer-url=https://keycloak.test-massoth.de/realms/aks<br>      - --oidc-client-id=aks-keycloak<br>      - --oidc-client-secret=0rH7d0d3vwllIKiWSALmeGNg1Fi3wkll</pre><p>Once the <strong>config </strong>file has been updated, we run the next command to see if we can reach the AKS cluster and get authenticated.</p><pre>kubectl get nodes</pre><p><strong>Good news: Authentication process works</strong>, however we are denied to access the AKS cluster. This is basically a good message because our user does not have any permissions inside the cluster. A role nor clusterrole has been created and associated with our user so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWnksXWiD2Hx2vZZt5ER1A.png" /></figure><p>On the other side, the referenced user does not seem to be <strong>aks-tobias. </strong>Trust me, it is. However, the required claim property <strong>sub </strong>as mentioned in the <strong>file jwt-config.json </strong>does not have a clear name as value but the ID of the user object in Keycloak. This is default behavior while Google and GitHub reference the login username of the user in the <strong>sub</strong> value. Therefore, we only see the ID which is correct. If we check again the middle part of the previous token, we see this value:</p><blockquote>[..]<br>“iss”: “https://keycloak.test-massoth.de/realms/aks&quot;, <br>“aud”: “aks-keycloak”, <br><strong>“sub”: “d280dfc5-fd59–4621-bfee-7de83247b9cd”, </strong><br>[..]</blockquote><p>We need to reference the value in the sub parameter in the (cluster-)rolebinding to provide permissions to the user. <br>If we just would continue, we would not understand which user is exactly meant behind the referenced ID. Therefore, we will apply a mapping and overwrite the value of the parameter with the login username to achieve the same behavior as with Google or GitHub.</p><p>I also tried to replace the property in the <strong>file jwt-config.json</strong>, however, rerunning the command will result in an error that the claim property other than <strong>sub</strong> is unknown.</p><p>To apply the remapping, we need to go to the <strong>Client scopes</strong> in the realm <strong>aks </strong>(make sure it is selected — see left above on the screenshot) and open the <strong>Profile</strong> client scope.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRR4awP61a_wasSqUyXy7Q.png" /></figure><p>In the <strong>Profile </strong>client scope, there is a <strong>User </strong>attribute in which the <strong>Token Claim Name</strong> must be changed from <strong>username </strong>to <strong>sub</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MDio2ey9OZv1b0Fs90XmJQ.png" /></figure><p>After saving and rerunning the previous kubectl command, we finally see that the same error message now contains the clear username aks-tobias instead of the user id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*yE2EFCYQfOpuH4_idbbMiw.png" /></figure><p>The reason is that the <strong>pub </strong>claim does now have a different value if we run the oidc-login command again</p><pre>kubectl oidc-login get-token \<br>--oidc-issuer-url=https://keycloak.test-massoth.de/realms/aks \<br>--oidc-client-id=aks-keycloak \<br>--oidc-client-secret=&lt;SECRET FROM ABOVE&gt;</pre><p>Output:</p><blockquote>[..]<br> “aud”: “aks-keycloak”,<br> “sub”: “aks-tobias”,<br> “typ”: “ID”,<br> “azp”: “aks-keycloak”,<br> “name”: “Tobias Massoth”,<br>[..]</blockquote><h3>Deploy Kubernetes ClusterRole and ClusterRoleBindung</h3><p>In the previous section we already checked that the Keycloak user aks-tobias does have access to the cluster, however the user does not have any permissions. Hence, we need to create a suitable Kubernetes ClusterRole and ClusterRoleBinding in order to access resources inside the cluster.</p><p>For this purpose, the following definition has been created. <br>The <strong>ClusterRoleBinding </strong>references the user <strong>aks-tobias</strong>. It is worth to note that the user is not just aks-tobias but contains <strong>aks:jwt:keycloak:</strong> as well. The reason is that AKS needs to know how the user shall be authenticated and provided with access. By using the prefix <strong>aks:jwt:keycloak:</strong> AKS knows that it must reach out to the Keycloak instance instead of Entra ID.</p><p>Keep in mind that you can configure multiple OIDC identity providers in the same clusters, so it is required to distinguish between them — using this prefix which we already defined in the <strong>file jwt-config.json.</strong></p><pre>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRole<br>metadata:<br>  name: external-user-role<br>rules:<br>- apiGroups: [&quot;&quot;]<br>  resources: [&quot;pods&quot;, &quot;services&quot;, &quot;nodes&quot;]<br>  verbs: [&quot;get&quot;, &quot;list&quot;]<br>- apiGroups: [&quot;apps&quot;]<br>  resources: [&quot;deployments&quot;, &quot;replicasets&quot;]<br>  verbs: [&quot;get&quot;, &quot;list&quot;]<br>---<br>apiVersion: rbac.authorization.k8s.io/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: external-user-binding<br>subjects:<br>- kind: User<br>  name: aks:jwt:keycloak:aks-tobias<br>  apiGroup: rbac.authorization.k8s.io<br>roleRef:<br>  kind: ClusterRole<br>  name: external-user-role<br>  apiGroup: rbac.authorization.k8s.io</pre><p>After applying the ClusterRole and ClusterRoleBinding …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*95ZZQZ5HW_u9bXvdTlto7Q.png" /></figure><p>… the user aks-tobias does now have permissions to list the AKS nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*NcrkV9uGtaJSbuU61NfH5Q.png" /></figure><h3>Troubleshooting</h3><p>As already highlighted above, I had issues with the certificate. Therefore, I would like to recommend and summarize the following.</p><p>If you run into the following error message, the reason might be that the certificate is not trusted. It appeared when I configured the AKS cluster and tried to access it using kubectl.</p><blockquote>“Unhandled Error” err=”couldn’t get current server API group list: the server has asked for the client to provide credentials”</blockquote><p>In this special case, the certificate was trusted locally and in my WSL, however, due to reasons I do not understand, it was not trusted by AKS itself.</p><p>I configured Diagnostics Settings and added an Azure Log Analytics workspace to check the kube-apiserver events. I encountered the next error message which was reported minutely using the KQL query.</p><pre>AKSControlPlane<br>| where Category == &quot;kube-apiserver&quot;<br>| where Level == &quot;ERROR&quot;</pre><p>The error message was saying that it cannot reach the Keycloak instance.</p><blockquote>oidc authenticator: initializing plugin: 502 Bad Gateway: Request to ‘<a href="https://keycloak.test-massoth.de/realms/aks/.well-known/openid-configuration&#39;">https://keycloak.test-massoth.de/realms/aks/.well-known/openid-configuration&#39;</a> cannot be made. Please make sure the OIDC identity provider is publicly accessible, and it is reachable from your cluster node(s).</blockquote><p>This error message disappeared itself after a while. I assume that the certificate (not bundle) which I created as a Kubernetes Secret was somehow cached. After I replaced the certificate by the bundle it was working.</p><h3>Useful Links</h3><p><strong>Microsoft Docs #1 — OIDC Authentication Overview:<br></strong><a href="https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-overview">https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-overview</a></p><p><strong>Microsft Docs #2 — OIDC Idenity Provider Configuration:<br></strong><a href="https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-configure">https://learn.microsoft.com/en-us/azure/aks/external-identity-provider-authentication-configure</a></p><p><strong>Keycloak Docs:</strong><br><a href="https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start">https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start</a></p><p><strong>Krew — installation:<br></strong><a href="https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start">https://www.keycloak.org/getting-started/getting-started-kube#_before_you_start</a></p><p><strong>JWT.MS — Decode OIDC token:<br></strong><a href="https://jwt.ms/">https://jwt.ms/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33e90285c2ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/azure-kubernetes-service-oidc-authentication-using-self-hosted-keycloak-instead-of-entra-id-33e90285c2ba">Azure Kubernetes Service OIDC Authentication using Self-Hosted Keycloak Instead of Entra ID</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Kubernetes Service Deep Dive Into Azure CNI Pod Subnet]]></title>
            <link>https://medium.com/microsoft/azure-kubernetes-service-deep-dive-into-azure-cni-pod-subnet-47c8f77cab68?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/47c8f77cab68</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cilium]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 05 Jan 2026 18:44:04 GMT</pubDate>
            <atom:updated>2026-01-05T18:44:04.921Z</atom:updated>
            <content:encoded><![CDATA[<p>There are several Azure CNI features available. From classic Azure CNI Node Subnet over Azure CNI Overlay to Bring Your Own CNI. Each CNI does have its reason for existence. Azure CNI Node Subnet is used to integrate a Pod natively into an Azure VNet and to consume IP addresses from a specific subnet which is also used by the AKS Worker Nodes.</p><p>The <strong>successor </strong>of Azure CNI Node Subnet <strong>is called Azure CNI Pod Subnet</strong> which is recommended by Microsoft whenever Pods need to be integrated directly into the VNet — similar to Azure CNI Node Subnet. However, AKS Worker Nodes and Pods are now clearly separated which offers additional benefits but also have some specifics. Some of them are:</p><ul><li>Node and Pod scaling do not interfere with each other due to separate subnets</li><li>Pods consume IP addresses from a dedicated, delegated subnet</li><li>Pods use the IP addresses to communicate AKS internally to establish connectivity between services or other Pods.</li><li>Pods use the IP addresses to communicate with other services outside of AKS (e.g. on-premises). The IP addresses will not be NATed. Hence, the destination sees the Pods origin IP address and not the one from the AKS Worker Node.</li><li>Support for Cilium, Calico and other products</li><li>An Azure Log Analytics Workspace is a hard requirements and can lead to additional costs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dy1EOaoDMe6FRsJDx8AoEw.png" /></figure><p>Microsoft clearly recommends to make use of Azure CNI Powered by Cilium. Hence, this blog post will cover Cilium.</p><h3>IP Allocation Modes</h3><p>Azure CNI Pod Subnet offers two different IP allocation modes how available IP addresses of the VNet subnet are managed and assigned to the respective AKS Worker Nodes.</p><ul><li>Dynamic IP Allocation (DIPA)</li><li>Static Block Allocation (SBA)</li></ul><p>Either DIPA or SBA need to be configured during AKS cluster creation time. If a node pool is added to an existing AKS cluster, the same or a different allocation mode can be selected. <br>If none have been explicitly specified, DIPA will be used as default.</p><p>Both IP Allocation Modes will be described in more details.</p><h4><strong>Dynamic IP Allocation (DIPA)</strong></h4><p>There are some hard facts that need to be considered when making use of the default allocation mode:</p><ul><li>Dynamic IP Allocation (DIPA) is the default allocation mode.</li><li>Better IP address usage: DIPA assigns blocks of 16 IP addresses from the Pod Subnet to a specific node. This block consists of several individual IP addresses such as 10.200.16.20, 10.200.16.35–10.200.16.51 (as I encountered in my lab)</li><li>When 8 IP addresses (50%) of this block are consumed by Pods, a new block of 16 IP addresses will be allocated and assigned to the same AKS Worker Node.</li><li>Following this approach, DIPA runs into some limitations which means that a DIPA-based AKS cluster node pool supports a maximum of 65k Pods.</li><li>DIPA does NOT support Cilium Network Policies (if this is a requirements, check Static Block Allocation covered in the next section)</li></ul><p>Let’s deploy an AKS cluster with Azure CNI Pod Subnet using the following command (should be mentioned here, that the monitoring add-on is a hard requirement. In this case, an AKS-managed Log Analytics Workspace will be created):</p><pre><br>RG=&lt;Name of the existing Resource Group&gt;<br>AKS=&lt;Name of the AKS which shall be provisioned&gt;<br>NODE_SUBNET_ID=&lt;Resource ID of the Node subnet for the AKS Worker Nodes&gt;<br>POD_SUBNET_ID=&lt;Resource ID of the Pod subnet&gt;<br><br>az aks create -g $RG -n $AKS \<br>  --network-plugin=azure \<br>  --network-dataplane=cilium \<br>  --vnet-subnet-id=$NODE_SUBNET_ID \<br>  --pod-subnet-id=$POD_SUBNET_ID \<br>  --enable-addons=monitoring \<br>  --generate-ssh-keys</pre><p>Once the AKS cluster has been provisioned successfully, the most interesting question is, how can we check the current status of DIPA?</p><pre>kubectl get nodenetworkconfigs -n kube-system -o wide</pre><p>This command provides some insights. First of all, we see the Subnet CIDR on the screenshot which has been specified during the AKS deployment with parameter <strong>pod-subnet-id</strong>.</p><p>In addition, we also see the <strong>REQUESTED IPS</strong> and <strong>ALLOCATED IPS</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xqugydpTdWT5-z0HCDi9eA.png" /></figure><p>Since we only have few pods running which utilize the pod subnet, only the first block of allocated IP addresses (meaning less than 8 IP addresses currently utilized) are in use.</p><p>However, we can check what the IP addresses are which have been assigned to the first AKS Worker Node. For this purpose, we run the following command:</p><pre>kubectl get nodenetworkconfigs aks-nodepool1-42477691-vmss000000 -n kube-system -o yaml</pre><p>The result provides a status section which highlights the following information — due to the amount of data and explanations, it is cut into multiple pieces.</p><p>The first piece focuses on the assigned IP addresses.</p><pre>status:<br>  assignedIPCount: 16<br>  networkContainers:<br>  - [..]<br>    ipAssignments:<br>    - ip: 10.200.16.36<br>      name: 1b5595fb-e2d6-4c4b-bd94-1b5e25f3eebf<br>    - ip: 10.200.16.37<br>      name: 22ff4bb2-24f5-493a-8b4f-a8d35720c79f<br>    - ip: 10.200.16.25<br>      name: 9c2fa065-4b8b-4002-b239-623887bff32d<br>    - [..]<br>    primaryIP: 10.200.16.21<br>    subnetAddressSpace: 10.200.16.0/20<br>    type: vnet<br>    vnetID: aks-network<br>    subnetID: pods<br>    subnetName: pods</pre><p>All allocated IP addresses are listed here. The second line indicates that as of running the command, only 16 IP addresses have been allocated by the AKS Worker Node so far. <br>The first IP address of the first block is used by the node itself. This IP address is also called ‘primaryIP’ and is used for internal use.</p><p>The status block underneath beginning with ‘subnetAddressSpace’ indicates information about the VNet and the respective subnet from which the IP addresses are allocated.<br>For an overview which IP addresses have been allocated by the AKS Worker Nodes and are currently in use by Pods, you must use other kubectl commands (such as kubectl get pods -o wide) or Azure Log Analytics Workspace.</p><p>Let’s go to the second piece.</p><pre>status:<br>  assignedIPCount: 16<br>  networkContainers:<br>  - assignmentMode: dynamic<br>    nodeIP: 10.200.0.5<br>    version: 0<br>    defaultGateway: 10.200.16.1<br>    id: ccc07261-d15f-48ce-a6bc-2053fd6dcb35<br>    [..]<br>  scaler:<br>    batchSize: 16<br>    maxIPCount: 250<br>    releaseThresholdPercent: 150<br>    requestThresholdPercent: 50</pre><p>This one is about DIPA (“dynamic”) as <strong><em>assigmentMode </em></strong>as highlighted underneath <strong><em>networkContainers</em></strong>. There are also some other generic information. One important one is <strong><em>version</em></strong>. Whenever a new block of IP addresses is going to be allocated, this integer is incremented by one.</p><p>So, <strong><em>version </em></strong>and <strong><em>assignedIPCount </em></strong>perfectly matches since it is still the first block of IP addresses which is reflected by <strong><em>version </em></strong>with value of “0” and by <strong><em>assignedIPCount </em></strong>with value of “16”.</p><p>The status block <strong><em>scaler</em></strong> reflects the configuration how many IP addresses are going to be allocated at once but also the maximum IP addresses to be consumed by the AKS Worker Node. 250 is the maximum Pod number an AKS Worker Node can operate.</p><p>So, the overall output looks like this (some information filtered).</p><pre>apiVersion: acn.azure.com/v1alpha<br>kind: NodeNetworkConfig<br>metadata:<br>  creationTimestamp: &quot;2025-12-29T17:32:09Z&quot;<br>  finalizers:<br>  - finalizers.acn.azure.com/dnc-operations<br>  generation: 1<br>  labels:<br>    kubernetes.azure.com/podnetwork-delegationguid: &lt;GUID&gt;<br>    kubernetes.azure.com/podnetwork-subnet: pods<br>    kubernetes.azure.com/podnetwork-type: vnet<br>    managed: &quot;true&quot;<br>    owner: aks-nodepool1-42477691-vmss000000<br>  name: aks-nodepool1-42477691-vmss000000<br>  namespace: kube-system<br>  ownerReferences:<br>  - apiVersion: v1<br>    blockOwnerDeletion: true<br>    controller: true<br>    kind: Node<br>    name: aks-nodepool1-42477691-vmss000000<br>    uid: 3113e13f-295f-40a1-afa1-e23c169199ad<br>  resourceVersion: &quot;16992&quot;<br>  uid: 3d781686-fdb5-48ba-b219-704d1ee5a43c<br>spec:<br>  requestedIPCount: 16<br>status:<br>  assignedIPCount: 16<br>  networkContainers:<br>  - assignmentMode: dynamic<br>    defaultGateway: 10.200.16.1<br>    id: ccc07261-d15f-48ce-a6bc-2053fd6dcb35<br>    ipAssignments:<br>    - ip: 10.200.16.36<br>      name: 1b5595fb-e2d6-4c4b-bd94-1b5e25f3eebf<br>    - ip: 10.200.16.37<br>      name: 22ff4bb2-24f5-493a-8b4f-a8d35720c79f<br>    - ip: 10.200.16.25<br>      name: 9c2fa065-4b8b-4002-b239-623887bff32d<br>    - [..]<br>    nodeIP: 10.200.0.5<br>    primaryIP: 10.200.16.21<br>    resourceGroupID: aks-rg<br>    subcriptionID: &lt;GUID&gt;<br>    subnetAddressSpace: 10.200.16.0/20<br>    subnetID: pods<br>    subnetName: pods<br>    type: vnet<br>    version: 0<br>    vnetID: aks-network<br>  scaler:<br>    batchSize: 16<br>    maxIPCount: 250<br>    releaseThresholdPercent: 150<br>    requestThresholdPercent: 50</pre><p>After deploying some more Pods, the previous command now shows that the version has been incremented but also the number of assigned and allocated IPs increased.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoqepYhq6__e0bBZixppPQ.png" /></figure><p>Now, since we understand how DIPA works and how to check status of the network configuration, let’s continue with <strong>Static Block Allocation (SBA)</strong>.</p><h3><strong>Static Block Allocation (SBA)</strong></h3><p>The Static Block Allocation (SBA) mode is more powerful and as described earlier, it is able to operate up to 1 million Pods. We could simply run SBA in the same cluster where DIPA runs — just in a different node pool. However, to have a clear separation, a new AKS cluster with Static Block Allocation will be provisioned. All what we need to do is to reuse the previous deployment command and add one more parameter called <strong><em>pod-ip-allocation-mode</em></strong> as you can see below.</p><pre>RG=&lt;Name of the existing Resource Group&gt;<br>AKS=&lt;Name of the AKS which shall be provisioned&gt;<br>NODE_SUBNET_ID=&lt;Resource ID of the Node subnet for the AKS Worker Nodes&gt;<br>POD_SUBNET_ID=&lt;Resource ID of the Pod subnet&gt;<br><br>az aks create -g $RG -n $AKS \<br>  --network-plugin=azure \<br>  --network-dataplane=cilium \<br>  --vnet-subnet-id=$NODE_SUBNET_ID \<br>  --pod-subnet-id=$POD_SUBNET_ID \<br>  --enable-addons=monitoring \<br>  --pod-ip-allocation-mode=StaticBlock \<br>  --generate-ssh-keys</pre><p>While the AKS cluster is currently provisioning, some additional information and facts:</p><ul><li>Static Block Allocation assigns a block of IP addresses until the end of lifetime of the respective AKS Worker Node, e.g. if a node is configured to host 250 Pods, 16 x /28 address blocks will be assigned to each of the AKS Worker Nodes. Hence, this allocation mode requires that the VNet has sufficient IP addresses available from the very beginning.<br>As a result of larger address blocks, cluster limitations increase leading to an overall supported number of 1 million Pods.</li><li>SBA supports Cilium Network Policies if Cilium as Network Data Plane has been deployed.</li><li>Migration from Azure CNI Pod Subnet to Azure CNI Overlay is supported at any time. This option provides you additional flexibility if the solution and direct accessibility from/to Pods in VNet subnet is not clear from the beginning and you might want to have the option to switch later.</li><li>SBA does not support Private Link Service to expose an application at this time.</li></ul><p>The following command now shows that allocation mode is static and number of IP addresses have been allocated for the maximum of Pods that can be running on AKS Worker Nodes.</p><pre>kubectl get nodenetworkconfigs  -n kube-system -o wide</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tNylZbu0R-e4T-F2UJjrYA.png" /></figure><p>If we now check how the NodeNetworkConfig looks like, we reuse the previous command from DIPA section…</p><pre>kubectl get nodenetworkconfigs aks-nodepool1-31564631-vmss000000 -n kube-system -o yaml</pre><p>… and retrieve a similar output layout with some different specifics. Let’s split the output into multiple pieces again.</p><pre>apiVersion: acn.azure.com/v1alpha<br>kind: NodeNetworkConfig<br>metadata:<br>  [..]<br>  requestedIPCount: 250<br>status:<br>  assignedIPCount: 255<br>  networkContainers:<br>  - [..]<br>    ipAssignments:<br>    - ip: 10.200.31.144/28<br>      name: 3b5d54ae-9ea2-46bf-9b11-a63c7fe61ff4<br>    - ip: 10.200.16.144/28<br>      name: b261dc7c-e729-4252-b7bc-9f979d6e3376<br>    - ip: 10.200.31.224/28<br>      name: c43474c6-08f6-49bc-99ad-4a0289def868<br>    - [..]<br>    primaryIP: 10.200.16.16/28<br>    subnetAddressSpace: 10.200.16.0/20<br>    subnetID: pods<br>    subnetName: pods<br>    type: vnetblock<br>    vnetID: aks-network</pre><p>What is quite interesting here is, <strong><em>requestedIPCount </em>(=250) </strong>is equal to the maximum amount of Pods which can be deployed and run on the AKS Worker Node. In order to serve all potentially running Pods with an IP address, all required IP addresses are allocated immediately as highlighted by <strong><em>assignedIPCount </em>(=255)</strong>. There is not an entire /24 block assigned, but multiple smaller chunks of /28 address ranges — see the <strong><em>ipAssignments </em></strong>section above. AKS does ignore the network address and broadcast address of each block as shown on the next screenshot. The IP address 10.200.16.144 is listed above in the <strong><em>ipAssignments </em>(</strong>10.200.16.144/28) section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*NBXjOijcpOUSbKIkdkzm4w.png" /></figure><p><strong>To be very clear here: </strong><br>No matter how many Pods are really running on the AKS Worker Node, 256 IP addresses are reserved and potentially wasted if not used in this example. <br>If you follow Kubernetes Best Practices, you would have a minimum of three Worker Nodes. It would mean that 768 IP addresses (3x 256) are reserved from the beginning. To avoid this, you could also configure the maximum number of Pods which could be hosted on an AKS Worker Node from 250 down to 63 (64 -1 because 1 IP address is reserved for internal purposes) if you know that there will not be so many Pods running. See IP Address Planning following this section later.</p><p>Coming to the next piece:</p><pre>[..]<br>status:<br>  assignedIPCount: 255<br>  networkContainers:<br>  - assignmentMode: static<br>    primaryIP: 10.200.16.16/28<br>    resourceGroupID: aks-rg<br>    subnetAddressSpace: 10.200.16.0/20<br>    subnetID: pods<br>    subnetName: pods<br>    type: vnetblock<br>    version: 0<br>    vnetID: aks-network</pre><p>The <strong><em>scaler </em></strong>section in the <strong><em>status </em></strong>section is missing. The reason is, it is not required anymore compared to DIPA since no batches will be assigned but all IP addresses at once. <br>Hence, the <strong><em>assignmentMode </em></strong>is set to “static<strong><em>”</em></strong> and the <strong><em>type </em></strong>is configured with “vnetblock”. Honestly, I do not really know nor understand why it is called “vnetblock”<strong><em> </em></strong>since there is not ONE big block assigned nor multiple /28 ranges directly followed by each other (there can be /28 spaces in between which will be then assigned to other AKS Worker Nodes). <br>Lastly, <strong><em>version </em></strong>which shows the value “0” will not change anymore. So, it does not provide any additional value.</p><p>So, the overall output looks like this (some information filtered).</p><pre>apiVersion: acn.azure.com/v1alpha<br>kind: NodeNetworkConfig<br>metadata:<br>  creationTimestamp: &quot;2025-12-29T22:49:29Z&quot;<br>  finalizers:<br>  - finalizers.acn.azure.com/dnc-operations<br>  generation: 1<br>  labels:<br>    kubernetes.azure.com/podnetwork-delegationguid: &lt;GUID&gt;<br>    kubernetes.azure.com/podnetwork-subnet: pods<br>    kubernetes.azure.com/podnetwork-type: vnetblock<br>    managed: &quot;true&quot;<br>    owner: aks-nodepool1-31564631-vmss000000<br>  name: aks-nodepool1-31564631-vmss000000<br>  namespace: kube-system<br>  ownerReferences:<br>  - apiVersion: v1<br>    blockOwnerDeletion: true<br>    controller: true<br>    kind: Node<br>    name: aks-nodepool1-31564631-vmss000000<br>    uid: 9854d9b7-f6e5-4361-9578-c86555593af8<br>  resourceVersion: &quot;1034&quot;<br>  uid: 83adf2a5-255a-4edf-811f-e49a63d52ffd<br>spec:<br>  requestedIPCount: 250<br>status:<br>  assignedIPCount: 255<br>  networkContainers:<br>  - assignmentMode: static<br>    defaultGateway: 10.200.16.1<br>    id: d872ffe0-a97a-4635-9fd3-64c2acd3918a<br>    ipAssignments:<br>    - ip: 10.200.31.144/28<br>      name: 3b5d54ae-9ea2-46bf-9b11-a63c7fe61ff4<br>    - ip: 10.200.16.144/28<br>      name: b261dc7c-e729-4252-b7bc-9f979d6e3376<br>    - ip: 10.200.31.224/28<br>      name: c43474c6-08f6-49bc-99ad-4a0289def868<br>    - [..]<br>    nodeIP: 10.200.0.6<br>    primaryIP: 10.200.16.16/28<br>    resourceGroupID: aks-rg<br>    subcriptionID: &lt;GUID&gt;<br>    subnetAddressSpace: 10.200.16.0/20<br>    subnetID: pods<br>    subnetName: pods<br>    type: vnetblock<br>    version: 0<br>    vnetID: aks-network</pre><p>As already highlighted in this section, SBA allocates all IP addresses for the entire lifetime of an AKS Worker Node. Therefore, <strong>proper IP address planning and calculation needs to be considered </strong>upfront.</p><h3><strong>SBA — IP address planning</strong></h3><p>Assuming the default max value of 250 Pods in AKS version 1.34 will be used. It means, that the IP address block must be large enough for 250 Pods which finally means 250 IP addresses — for one AKS Worker Node. <br>There is a calculation which needs to be followed in order to do proper IP address planning for the entire AKS cluster: <br>(16 IP addresses * N) — 1 = Number of available Pod IPs, where N is any integer ≥ 1.<br>This means, (16 * 16) — 1 = 255 IPs available. A maximum of 250 Pods can be deployed on an AKS Worker Node, hence there is a waste of 5 IP addresses. That multiplied by the number of Worker Nodes which are expected to scale, let’s say 100, leads to an overall waste of 500 IP addresses. Consider that AKS Worker Nodes might be fully utilized from a CPU/memory perspective. There might be additional IP addresses available which cannot be assigned to Pods anymore. <br>The question is if this is acceptable from an IP address wastage — if not, maybe (16 * 15) — 1 = 239 Pods or even less per node would work as well.</p><p>Therefore IP address planning needs to be considered and carefully planned for the Azure VNet. <br>Too less IP addresses lead to scaling issues/limits. <br>Too many planned IP addresses lead to potential IP waste.</p><h3>Container Network Service — How IP Address Assignment Works Under the Hood</h3><p>Let’s now dive into the details and how both allocation modes really work and are working behind the scenes.<br>Most probably, you have already recognized that there is a DaemonSet deployed called <strong>azure-cns</strong>. The abbreviation <strong>CNS </strong>stands for <strong>Container Network Service </strong>and handles all API requests between the CNI and Azure VNet (the responsible component to do so is <strong>azure-ipam</strong> which is also part of CNS) in order to serve pods with an available IP address.</p><p>The following command returns all Pods running in the kube-system namespace which also lists the azure-cns related Pods.</p><pre>kubectl get pods -n kube-system</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*NxcTgs-y3BrSzsXjZC3MMg.png" /></figure><p>Since Cilium is running as CNI data plane and we make use of Azure CNI Pod Subnet, the azure-cns Pods are absolutely required by the Cilium Pods to provide new Pods with an available IP address. There is a tight integration and hard dependency.</p><p>The azure-cns related Pods are running as part of a DaemonSet. When manipulating the DaemonSet and its Pods to better understand what the impact is when CNS does not work properly anymore, I experienced, that changes made to the DaemonSet are not permanent. Any changes are reverted within few seconds. Even if the DaemonSet is deleted, it will be recreated. <br>The reason is, the Azure Kubernetes Service makes use of <strong>ENO</strong>. If you check the labels of the DaemonSet azure-cns , you see that this one is managed by ENO.<br><strong>ENO is a dynamic configuration management tool </strong>which runs in the background, somewhere on the other side where the AKS Master Nodes run as well.</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  annotations:<br>    deprecated.daemonset.template.generation: &quot;1&quot;<br>  generation: 1<br>  labels:<br>    app: azure-cns<br>    app.kubernetes.io/managed-by: Eno<br>    kubernetes.azure.com/managedby: aks<br>[..]</pre><p>So, it means, we need to find a different way to manipulate the DaemonSet and the quickest and easiest way is to run a bash script.</p><pre>while [ true ]<br>do<br>   kubectl delete ds azure-cns -n kube-system<br>   kubectl delete pods -l k8s-app=azure-cns -n kube-system --force --grace-period=0<br>done</pre><p>This script ensures that the CNS Pods will not be ready anymore since they will be killed immediately after they have been created via the DaemonSet.</p><p>Now, it is time to deploy a new Pod…</p><pre>kubectl run nginx091 --image=nginx:latest </pre><p>… and realize, that this Pod will stuck in pending. If you run a describe against the Pod…</p><pre><br>kubectl describe pod nginx091</pre><p>… you figure out that a recurring error message appears. This error message highlights, that Cilium is incapable to retrieve an IP address from CNS. This is because there is no service responding on <strong>http://localhost:10090</strong>.</p><blockquote>Warning FailedCreatePodSandBox 102s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “4ac4fef3791ec07d7510962e8c21bfbf9b595a8d9704a8e9bdb2fe8b5fbd17df”: plugin type=”cilium-cni” failed (add): http request failed: Post “<strong>http://localhost:10090/network/requestipconfigs</strong>&quot;: dial tcp 127.0.0.1:10090: connect: connection refused; <strong>failed to request IP address from CNS</strong></blockquote><p>As soon as the bash script has been terminated, the azure-cns Pods are deployed as well and the previously deployed nginx Pod will retrieve an IP address from Cilium (and lastly from CNS).</p><p>This proofs that Azure CNI Pod Subnet functionality is highly depending on the azure-cns Pods.</p><p>Now that everything is working again, let’s run a curl command directly on one of the AKS Worker Nodes. The path “/debug/ipaddresses” used in this command has been found in the respective <a href="https://github.com/Azure/azure-container-networking">azure-container-networking GitHub repo</a>. The following curl command requests to get an overview of all IP addresses assigned to Pods.</p><pre>curl localhost:10090/debug/ipaddresses -d &#39;{&quot;IPConfigStateFilter&quot;:[&quot;Assigned&quot;]}&#39;</pre><p>The output is a JSON object as well and contains a lot of information. Based on the previously deployed nginx Pod, you find the Pod specific information.</p><blockquote>{“IPConfigurationStatus”:[<br>……<strong>{“state”:”Assigned”,”ID”:”10.200.31.65&quot;,”IPAddress”:”10.200.31.65&quot;,”LastStateTransition”:”2025–12–31T22:17:21.641818362Z”,”NCID”:”4e838347–3639–43bc-aebf-48f988e50d8f”,”PodInfo”:{“PodName”:”nginx091&quot;,”PodNamespace”:”default”,”PodInfraContainerID”:”8ac857eba8ad585ccea2d267506b83facea326da66aaa52472111fd1686c6908&quot;,”PodInterfaceID”:”8ac857eba8ad585ccea2d267506b83facea326da66aaa52472111fd1686c6908&quot;,”Version”:2,”SecondaryInterfaceSet”:false}},</strong>{“state”:”Assigned”,”ID”:”10.200.31.118&quot;,”IPAddress”:”10.200.31.118&quot;,”LastStateTransition”:”2025–12–31T22:17:21.542926077Z”,”NCID”:”4e838347–3639–43bc-aebf-48f988e50d8f”,”PodInfo”:{“PodName”:”coredns-5b5b89b58c-ltjwk”,”PodNamespace”:”kube-system”,”PodInfraContainerID”:”931a2333db94a38990e88b7489547461976b13213872088c4697b2d889a9ed8e”,”PodInterfaceID”:”931a2333db94a38990e88b7489547461976b13213872088c4697b2d889a9ed8e”,”Version”:2,”SecondaryInterfaceSet”:false}},<strong><br></strong>……<br>],”Response”:{“ReturnCode”:0,”Message”:””}}</blockquote><p>This is the end of the blog post and I really hope that you find this blog post useful and gained a better understanding what and how Azure CNI Pod Subnet works but also what the implications are if CNS does not work properly.</p><p>At the end, I would like to share some more important links:</p><ul><li>Azure CNI Pod Subnet overview and descriptions: <br><a href="https://learn.microsoft.com/en-us/azure/aks/concepts-network-azure-cni-pod-subnet">https://learn.microsoft.com/en-us/azure/aks/concepts-network-azure-cni-pod-subnet</a></li><li>ENO — Dynamic Configuration Management Tool:<br><a href="https://github.com/Azure/eno">https://github.com/Azure/eno</a></li><li>GitHub repo to Container Network Service (CNS): <br><a href="https://github.com/Azure/azure-container-networking/">https://github.com/Azure/azure-container-networking/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47c8f77cab68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/azure-kubernetes-service-deep-dive-into-azure-cni-pod-subnet-47c8f77cab68">Azure Kubernetes Service Deep Dive Into Azure CNI Pod Subnet</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AKS Private Cluster with Private Endpoint vs. API Server VNet Integration]]></title>
            <link>https://medium.com/@tobias_80378/aks-private-cluster-with-private-endpoint-vs-api-server-vnet-integration-d1dc2918cb2b?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1dc2918cb2b</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 21:58:06 GMT</pubDate>
            <atom:updated>2025-12-22T21:58:06.276Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post I will detail out what both Azure Kubernetes Service Private Cluster with Private Endpoint but also API Server VNet Integration are. Although both can be used in conjunction with AKS Private Cluster, each feature does have specialties with regards to network integration and visibility, network routing and firewalling.</p><p>While API Server VNet Integration has been released recently (Summer 2025), AKS Private Cluster with Private Endpoint does exist since few years. The latter feature is the de-facto standard when AKS clusters must be fully reachable and accessible in private networks. This feature ensures that both AKS Worker Nodes and control-plane must be available privately in the network. API Server VNet Integration does the same in a slightly different way and there are good reasons to use API Server VNet Integration over Private Endpoint.</p><h3>Specialties of AKS Private Cluster with Private Endpoint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kz9jOcy4CEgMkoPD98TzvA.png" /></figure><h4>Network Integration and Visibility</h4><ul><li>The AKS Worker Nodes are deployed into a VNet subnet in the customer’s subscription. On the other side, the AKS control-plane is operated in an AKS-managed subscription outside of the tenant. In order to make the AKS control-plane reachable from the AKS Worker Nodes but also to administer the AKS Cluster from a client, it is required to deploy a Private Endpoint. This Private Endpoint in the customer’s subscription is directly linked via Private Link Service with an Azure Load Balancer in the AKS-managed subscription of the external tenant. The Azure Load Balancer contains a backend which consists of the AKS control-plane. Everything starting from the Azure Load Balancer is totally invisible to the customer and the way between the customer’s Private Endpoint and AKS-managed Load Balancer in the external tenant is tunneled.</li><li>Some industries and security/network departments have strict standards and compliance requirements when it comes to tunneling between environments because you must measure what the risk level in conjunction to the other’s side network and security configuration is. Gladly, Private Endpoint is a one-way communication which means, AKS Worker Nodes and clients can communicate with the AKS control-plane but not vice-versa which eliminates accessing the customer’s environment from AKS control-plane, hence from the external tenant. <br>This kind of tunneling should somewhere be documented in the solution, so a compliance review does not mark it as a potential finding.</li></ul><h4>Kubelet Configuration</h4><ul><li>The bootstrap-config file located under /<strong>var/lib/kubelet/bootstrap-kubeconfig</strong> on each AKS Worker Node contains a line called ‘server’ highlighting the API Server address (could be anything such as standalone server or a cluster behind a load balancer). <br>In this case the FQDN of the Private Link is specified</li></ul><pre>apiVersion: v1<br>kind: Config<br>clusters:<br>- name: localcluster<br>  cluster:<br>    certificate-authority: /etc/kubernetes/certs/ca.crt<br>    server: https://massoth.privatelink.westeurope.azmk8s.io:443<br>[..]</pre><h4>Network Routing</h4><ul><li>One of the biggest (security) issues with Private Endpoint in a secured network environment is the automatically propagated /32 system route on VNet level between VNet Peerings. <br>Usually, a User Defined Route (UDR) is configured with the VNet subnet’s address range to route traffic directly to the Azure Firewall or an NVA-based third-party firewall. This works as long as there is no other system route in place which is more specific than the user-defined route in UDR. Since the Private Endpoint system route (/32) is more specific than the underlying subnet (let’s assume a /24 subnet size), the Private Endpoint will ignore the UDR and bypass the firewall.</li><li>If there is an additional NSG associated to the Private Endpoint’s subnet, above scenario does still apply, however, the NSG ruleset controls and allows/blocks incoming traffic.</li></ul><h4>Firewall and Dynamic IP Address Assignment</h4><ul><li>The AKS API Server is accessible via the Private Endpoint in a non-delegated subnet. This Private Endpoint must be reachable via TCP/443.</li><li>The IP address assigned to the Private Endpoint is dynamically allocated and cannot be set to static (current limitation). Therefore, if IP address is used in Firewall ruleset, the IP address could change at any time and the AKS control-plane might not be accessible anymore. Hence, the FQDN of the API Server should be used in Firewall rulesets. However, if an NSG is used for the subnet in which the Private Endpoint resides, it does not support FQDN filtering. So, this needs to be an IP address which potentially can lead to an outage if the dynamic IP address would change.</li></ul><h4>Costs</h4><ul><li>As long as only little data are transferred via Private Endpoint, costs will be very low.</li></ul><h3>Specialties of AKS Private Cluster with API Server VNet Integration</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKmYuKD_-FF0sIb12WC5YQ.png" /></figure><h4>Network Integration and Visibility</h4><ul><li>Equivalent to the set up with Private Endpoint, the AKS Worker Nodes are deployed into a VNet subnet in the customer’s subscription and the AKS control-plane is operated in an AKS-managed subscription outside of the tenant. <br>In addition to the subnet in which the AKS Worker Nodes reside, an additional delegated subnet (minimum size /28) is required to place the relevant main component for the API Server VNet Integration — an Azure Load Balancer. Among the Azure Load Balancer which is just one resource in this newly delegated subnet, the IP addresses of the AKS Master Nodes (from the AKS-managed subscription in the external tenant) will be projected as well. The IP addresses belong to the backend configuration part of the Azure Load Balancer and can be seen there. However, there are no dedicated network interfaces associated to the subnet or so.</li></ul><h4>Kubelet Configuration</h4><ul><li>The bootstrap-config file used for Private Endpoint looks quite identical compared with the API Server VNet Integration. However, it does not specify an FQDN in the /<strong>var/lib/kubelet/bootstrap-kubeconfig</strong> file on each AKS Worker Node but a static IP address — this is also referenced in the Frontend of the Azure Load Balancer.</li></ul><pre>apiVersion: v1<br>kind: Config<br>clusters:<br>- name: localcluster<br>  cluster:<br>    certificate-authority: /etc/kubernetes/certs/ca.crt<br>    server: https://10.100.1.4:443<br>[..]</pre><h4>Network Routing</h4><ul><li>Compared to any other solution containing an Azure Load Balancer, there are no differences how network routing works.</li></ul><h4>Firewall and Static IP Address Assignment</h4><ul><li>The Azure Load Balancer sits in front of the control-plane. The Load Balancing rule listens on TCP/443. However, the backend is reachable via TCP/4443 and the Health Probe checks on TCP/9988. Both Azure Load Balancer Frontend IP and Backend Pool IPs all reside in the same subnet.<br>Depending on the network design, all beforementioned ports must be considered in Azure Firewall / NVA-based 3rd Party Firewall and/or NSG. Otherwise traffic will be blocked somewhere and the AKS cluster will not work properly.</li><li>The IP address of the Azure Load Balancer is set statically, so it will not change and the AKS API Server FQDN directly points to this one. Honestly, I do not know why the FQDN has not been specified in the bootstrap-kubeconfig.</li></ul><h4>Costs</h4><ul><li>As long as only little data are transferred via Azure Load Balancer (Standard SKU), costs will be very low.</li></ul><h3>Conclusion</h3><p>From an overall point of view, both Private Endpoint and API Sever VNet Integration achieve the same goal: <br>Make all communication between AKS Worker Nodes and control-plane private and integrate all necessary components into a private network.</p><p>The costs are similar as well. <br>In case there are no compliance or security constraints or requirements, the overall guiding question is: <br>How does the landing zone design and integration look like?</p><p>There is no right or wrong in decision-making in favor to one of both features. At the end, the — I called it — specialties as highlighted above need to be evaluated and considered what the impact would be in your environment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1dc2918cb2b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing Resiliency in Azure Kubernetes Service with Chaos Studio]]></title>
            <link>https://medium.com/microsoft/testing-resiliency-in-azure-kubernetes-service-with-chaos-studio-948e6fefe6c7?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/948e6fefe6c7</guid>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Thu, 27 Nov 2025 23:13:15 GMT</pubDate>
            <atom:updated>2025-11-27T23:13:15.691Z</atom:updated>
            <content:encoded><![CDATA[<p>In my previous blog post, I was writing about Vibe Coding by using Copilot to develop a Python-based web app. This web app is now used as part of this blog post to demonstrate how faults inside the app architecture can be tested in order to get a good feeling about resiliency when an app component fails.</p><p>Especially in microservice and distributed architectures including databases, storage, third-party tools and services and much more, it can become quite complex to understand all the different dependencies and what the impact to the entire architecture and application availability will be if any of these components fails.</p><p>Azure Chaos Studio perfectly integrates into Azure Kubernetes Service and offers many actions to simulate faults by default and therefore to test resiliency of the microservice architecture within Kubernetes. Starting from network and connectivity issues including DNS resolution, simulating Pod faults and permanent unavailability up to high pressure on CPU and RAM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RewX-bVz5rZb96RoEkUxJg.png" /></figure><p>In this blog post, we will cover the following:</p><ul><li>Deployment of open-source chaos mesh</li><li>Deployment of Azure Chaos Studio and experiments</li><li>Run actions against Azure Kubernetes Service to issue faults</li></ul><p>To get started, we simply deployed a standard Azure Kubernetes Cluster which is publicly available. Azure Chaos Studio and Chaos Mesh does also work with AKS Private Clusters, however, it requires additional configuration which will not be covered here.</p><p>Once the AKS had been deployed, I was deploying the Python-based Web Application as described here:</p><p><a href="https://tobias-massoth.de/running-a-copilot-developed-web-app-on-azure-kubernetes-service-e533cdcacc91">Running a Copilot Developed Web App on Azure Kubernetes Service</a></p><p>Now, we have a fully functional app running which is available directly in the Web Browser via the Azure Load Balancer’s Public IP Address.</p><p>Starting from here, let’s now complete the first prerequisite. In order to run Chaos Mesh experiments, we first need to deploy the open-source Chaos Mesh deployment.</p><pre>helm repo add chaos-mesh https://charts.chaos-mesh.org<br>helm repo update<br>kubectl create ns chaos-testing<br>helm install chaos-mesh chaos-mesh/chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock</pre><p>The deployment does only take a couple of seconds to get set up successfully.</p><p>The controller pods are responsible to retrieve the chaos instructions and apply them in the cluster accordingly, later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*aVWAJNILv4RlCV3fZxbw_w.png" /></figure><p>Once this is done, we can continue to onboard our Azure Kubernetes Service cluster in Azure Chaos Studio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GAC2tNF9TJCoFEY20I5NA.png" /></figure><p>The onboarding of AKS takes a couple of minutes to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*VGEhXktGEYSiESSWKEMetg.png" /></figure><p>Once the onboarding has been completed, we are ready to create our first experiment. The experiment will further describe what we want to test. There are various chaos experiments available by default. More to come later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*5kETfjuvCJYdF2VobGAfNw.png" /></figure><p>The experiment will be created as a resource and is editable at any time later. Since we want to have this experiment specifically run only one action, we provide the name aks-chaos-experiment01-pod-chaos. This experiment will later make the backend running on AKS unavailable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5XM9h1d3JQf3Eo_DhITJ_A.png" /></figure><p>When the experiment starts running, the selected identity will access AKS via the API and tell the Chaos Mesh controller what to do.</p><p>The <strong>Azure Kubernetes Service Cluster Admin RBAC</strong> role will automatically assigned to the user assigned identity if you are allowed (if you are owner or have a similar role).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PN3VAdOTb0nHkuJhOqCdWg.png" /></figure><p>An experiment consists of one or multiple steps which will be completed sequentially. Each step consists of at least one branch. A branch covers an action which runs against a specific AKS cluster. Branches run in parallel if multiple ones are defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m8xk2Yyho-SEY1RCz3RlCA.png" /></figure><p>As fault, we select the <strong>AKS Chaos Mesh Pod Chaos </strong>fault and a template is automatically filled in. Since our frontend and backend run in different namespaces, we need to adjust the jsonSpec to reference the calc-backend namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*gWU3aPM1aLr_vR-21fcQ9A.png" /></figure><p>As target resource, we select the previously deployed AKS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*cxcS2xKrLLchPQ3XqdU0Ww.png" /></figure><p>Once the fault description has been saved, we are ready to create the experiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9wNpgnaTb4aJFHjwfbbGug.png" /></figure><p>If you make any adjustments to the user assigned identity, e.g. replacing by a different one, do not forget to assign the Azure Kubernetes Service Admin Cluster RBAC role to this identity again. Otherwise, you will receive an <strong>Access Denied</strong>.</p><p>Now, let’s run the experiment and see if it works. We start the experiment and it first takes 1–2 minutes to get started. The reason is that the Chaos Mesh controller first needs to get contacted and the jsonSpec must be transmitted and interpreted. After, Chaos Mesh controller starts scheduling and running the experiment inside the AKS cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VvFwZHO0BT4T3eMywDRLA.png" /></figure><p>We immediately see that the backend Pod crashed but restarted again. However, the backend is not available anymore. An operation in the frontend will be entered to get processed in the backend and result to be returned to the frontend again. However, we see that the backend is unreachable.</p><p>This error message is an indicator that this kind of issue has not been properly caught in application code since it is a little bit cryptical and not user-friendly. As a follow-up, we would implement an code change to reflect a better and user-friendly message saying that the function to perform the operation is currently not available and somebody is already working on the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJToybCoUBcnOb5D97rwyg.png" /></figure><p>What happened behind the scenes in fact?</p><p>By digging deeper, we see that Chaos Mesh crashed only once the Pod and modified the manifest by replacing the backend container image by <strong>Google Pause </strong>image. This is the reason that the Pod is again “up and running”, however without any logic. Hence, it does not respond anymore.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUXUyJ1TVhi83DkVjpWFzw.png" /></figure><p>This configuration and behavior stays active for 5 minutes, since this is the duration we selected in the experiment config. After 5 minutes, the manifest is manipulated back to make use of the backend container image again and the application is fully available again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YrRvortMMuCzC26qxLTVhA.png" /></figure><p>When we simply click submit again, the result has been delivered on time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2705MTOypCH3IJhbB8w9wQ.png" /></figure><p>In combination with test automation, Azure Chaos Studio is a powerful service to test resiliency of application architecture and its functionality whenever a new feature or other changes have been made. Additionally, I can check whether I have considered any faults and how those get handled inside the application when they appear in fact.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=948e6fefe6c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft/testing-resiliency-in-azure-kubernetes-service-with-chaos-studio-948e6fefe6c7">Testing Resiliency in Azure Kubernetes Service with Chaos Studio</a> was originally published in <a href="https://medium.com/microsoft">Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a Copilot Developed Web App on Azure Kubernetes Service]]></title>
            <link>https://medium.com/@tobias_80378/running-a-copilot-developed-web-app-on-azure-kubernetes-service-e533cdcacc91?source=rss-8ad957956480------2</link>
            <guid isPermaLink="false">https://medium.com/p/e533cdcacc91</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure-kubernetes-service]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[co-pilot]]></category>
            <dc:creator><![CDATA[Tobias Massoth]]></dc:creator>
            <pubDate>Mon, 20 Oct 2025 20:13:37 GMT</pubDate>
            <atom:updated>2025-10-20T20:13:37.786Z</atom:updated>
            <content:encoded><![CDATA[<p>For one of my upcoming blog posts about Chaos Mesh, I was brain storming a suitable scenario. Blog posts take a lot of time from drafting a first rough one including some demos and writing down instructions finally leading to the polished story. Any additional contents or development work take further time and effort. To reduce the overall effort, I decided to use Copilot for this one and it was the right decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMukfnCsKg2YR7YJoR21YQ.png" /></figure><p>In the next couple of minutes, I will show how Copilot created a 99% working microservice architecture. Here is what I wanted to achieve:</p><ul><li>A python web app</li><li>Two microservices (frontend and backend)</li><li>Running Docker containers</li><li>YAML manifests ready to deploy on Azure Kubernetes Service</li></ul><p>The exact prompt I used:</p><pre>Create a python web app which consists of two microservices. <br>The first microservice is just a nice frontend which has an input textbox to enter a mathmatic instruction. <br>After clicking the submit button, the input will be sent to the second microservice which will perform mathmatic calculation as provided.<br>Both microservices shall run as docker containers. Create also Kubernetes manifests to deploy the microservices on Kubernetes.</pre><p>The prompt might not be 100% precise, however, it was impressive what the result was at the end.</p><p>After submitting the prompt, Copilot started to draft a todo list and created the backend but also frontend folders. Finally, all required files were created and added to the respective folders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*x1D2-QbbFEAY0RqYqMoBDg.png" /></figure><p>After all python and Docker files had been created, Copilot also recommended to create a unit test if I agree. So, I let Copilot do and had to execute the following command.</p><pre>/bin/python3.11 -c &quot;import sys; sys.path.insert(0, &#39;backend&#39;); import test_app; test_app.test_calculate_ok(); test_app.test_calculate_bad(); print(&#39;TESTS PASSED&#39;)&quot;</pre><p>The command ran successfully and returned “TESTS PASSED”. Afterwards, Copilot immediately started to create a Docker Compose file which I have not asked for but for local test purposes and if Docker containers run fine, it is OK. I did not make use of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*9eb_ar2g3pWnbe1ZymJQ8g.png" /></figure><p>So, Docker Compose file was created quite quickly and continued with the next task. The Kubernetes manifests were structured quite well. There were finally four files:</p><ul><li>backend-deployment.yaml</li><li>frontend-deployment.yaml</li><li>namespace.yaml</li><li>ingress.yaml</li></ul><p>Each file of the *-deployment.yaml’s also contained a service next to the respective deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*p4LnjsnLPmiCzmFvC9qdvQ.png" /></figure><p>In the last step, Copilot also provided the commands to run Docker containers locally with Docker Compose but also docker build and push commands. Additionally, the kubectl commands were listed as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*MSajKC1WdXTCdd1ZRktlcg.png" /></figure><p>Now, the most important question is: Does it all work and run without issues or did Copilot do some mistakes?</p><p>There were some mistakes and definitions Copilot did not consider in one of the previous steps it performed. However, the Python web app most importantly worked without issues as you can see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*l0vqqG5xbV26sY3HheZvlQ.png" /></figure><p>(the word “nice” in the prompt has been somehow ignored or interpreted as something “basic”. I should have used beautiful or a more powerful word)</p><p>Now, let’s have a look what I recognized when I went through the application code but also all other files. <strong>Here’s the result what worked fine but also what did not and had to be adjusted</strong>:</p><ol><li>The Dockerfiles used a slim Python3.11 base image and the absolut minimum of Docker commands, however, the application runs as root inside the container.</li><li>The namespace.yaml created a calc-app namespace in Kubernetes.</li><li>The frontend and backend deployments contained the absolute minimum a K8S manifest requires. Both ran successfully AFTER some modifications:<br>A) The <strong>created namespace was not referenced</strong> in both deployment nor in the service YAMLs. However, the namespace was referenced in the command instruction (kubectl … -n calc-app) as given by Copilot. If ignored or not read carefully, all resources would have been deployed into the default namespace if the kubectl command had been written down and executed manually (not copied from Copilot output). I personally prefer to have the namespace defined in YAML where possible.<br>B) The container image just referenced a frontend:latest and backend:latest without providing further details. Both are dependent on the previous Docker build and push commands referencing the registry. As mentioned by Copilot, this was a task the developer (in this case it was me) had to complete.</li><li>The ingress controller was deployed in the default namespace as well and did not have an explicit host parameter. I know it is optional, however, I prefer one. Since it has not been explicitly mentioned in the prompt, it is fine. All what I had to do was to add a DNS Name to the Azure Public IP linked to the Azure Load Balancer in front of the Azure Kubernetes Service cluster. The ingress worked on port 80 and the application was reachable.</li><li>Python web app for both frontend and backend worked without issues. Frontend sends data to backend, retrieves the results and outputs to the user.</li><li>Python web app does also contained data validation which means if I enter non-mathematical data such a string, it is validated and returned successfully that it is not a legal math calculation.</li><li>Python code is not 100% accurate as assumed. <br>Fun part is that Copilot added an commented URL to the *.py file highlighting from where it got some food for thought (namely from stackoverflow).</li></ol><p><strong>Final remark:<br></strong>It is again very impressive what Copilot does and how powerful it is. The entire app, including testing and modifications, <strong>took less than one hour</strong> of my time.</p><p>So, I will now continue with the actual story.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e533cdcacc91" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>