<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Umesh Kaul on Medium]]></title>
        <description><![CDATA[Stories by Umesh Kaul on Medium]]></description>
        <link>https://medium.com/@umeshkaul_39077?source=rss-64e3f5728d09------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Qmf9rATWr2F0Z5VM</url>
            <title>Stories by Umesh Kaul on Medium</title>
            <link>https://medium.com/@umeshkaul_39077?source=rss-64e3f5728d09------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:56:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@umeshkaul_39077/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Securing Kubernetes MCP Server with Pomerium and Google OAuth 2.0]]></title>
            <link>https://medium.com/@umeshkaul_39077/securing-kubernetes-mcp-server-with-pomerium-and-google-oauth-2-0-7a186adc0d7d?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a186adc0d7d</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[pomerium]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[mcp-client]]></category>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Sat, 11 Oct 2025 16:08:42 GMT</pubDate>
            <atom:updated>2025-10-12T23:19:00.783Z</atom:updated>
            <content:encoded><![CDATA[<p>MCP has rapidly transformed the AI landscape in less than a year. While it has standardized access to tools for LLMs, it has also created security challenges. In this post, we’ll explore how to add authentication and authorization to the <a href="https://github.com/containers/kubernetes-mcp-server">Kubernetes MCP server</a>, which exposes tools like helm_list, pods_list, pods_log, and pods_get etc</p><p>We’ll use <a href="https://www.pomerium.com/">Pomerium</a>, an open-source identity-aware proxy that has recently added <a href="https://www.pomerium.com/docs/capabilities/mcp">MCP support</a> to provide secure access to Model Context Protocol (MCP) servers. The demonstration will show a user authenticating to Pomerium via Google OAuth and being authorized to run only an allowed list of commands based on Pomerium configuration.</p><h3>Overview of the Demo</h3><ul><li>A user accesses the MCP Server via <a href="https://github.com/modelcontextprotocol/inspector">MCP Inspector</a> (GUI-based) or <a href="https://github.com/tileshq/mcp-cli">MCP-CLI</a> (CLI-based) testing tools</li><li>Requests are routed through Pomerium, which handles Google OAuth 2.0 authentication</li><li>Once authenticated, Pomerium allows access to only an allowed list of <a href="https://github.com/containers/kubernetes-mcp-server">Kubernetes MCP server</a> tools (see configuration below). Example: based on the policy in pomerium.yaml, helm_list will work whereas events_list will not</li></ul><pre>policy:<br>      allow:<br>        and:<br>          - email:<br>              in: [youremail@gmail.com]<br>          - mcp_tool:<br>              in: [helm_list, pods_list, events_list]  ## allowed list of tools to be allowed</pre><blockquote><em>Pomerium offers </em><a href="https://file+.vscode-resource.vscode-cdn.net/Users/umeshkaul/Projects/using-pomerium-to-secure-kubernetes-mcp-server/README-MEDIUM.md#more-about-pomerium-mcp-support"><em>three ways</em></a><em> to protect MCP servers, but we will demonstrate only one of them in this blog post.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zNnBMObJomKpor3ZxHpEiw.png" /></figure><h3>Configuration &amp; Setup</h3><h3>1. Configure Google OAuth Client</h3><p>Create OAuth 2.0 credentials in <a href="https://console.cloud.google.com/">Google Cloud Console</a> with redirect URI e.g https://www.ukaul.net:8443/oauth2/callback and note the Client ID and Secret.</p><h3>2. Launch Pomerium and Kubernetes MCP Server</h3><ul><li>Make sure you have access to a working Kubernetes cluster</li><li>Configure P<strong>omerium</strong>, as per example configuration below.</li></ul><pre>shared_secret: ozGghKH6NPBjjih/fqRhPTIgSNX4R2D5zXfQH+ZUKzc=    # you can generate this using command : head -c32 /dev/urandom | base64<br>cookie_secret: K6eNnqNk6i0TQqihJoprwdeIBrPtvjBZgqYx2tT3SU0=    # you can generate this using command : head -c32 /dev/urandom | base64<br>idp_client_id: &quot;your-google-client-id-here&quot;      # your google oauth2 client id goes here<br>idp_client_secret: &quot;your-google-client-secret-here&quot;  # your google oauth2 client secret goes here<br>authenticate_service_url: https://www.ukaul.net:8443      # your public URL<br>authorize_service_url: https://www.ukaul.net:8443          <br>forward_auth_url: https://www.ukaul.net:8443<br>idp_provider: google<br>idp_scopes: &quot;openid email profile&quot;<br>certificate_file: &quot;/root/certs&quot;<br>certificate_key_file: &quot;/root/key&quot;<br>log_level: info   # options: debug, info, warn, error<br>runtime_flags:<br>  mcp: true<br>routes:<br>  - from: https://www.ukaul.net:8443<br>    path: /mcp<br>    to: http://192.168.15.5:8080/mcp  # this is ip address where your kubernetes-mcp-server listens <br>    pass_identity_headers: false<br>    set_request_headers:<br>      authorization: &quot;&quot;<br>    mcp:<br>      server:<br>        enabled: true<br>    policy:<br>      allow:<br>        and:<br>          - email:<br>              in: [youremail@gmail.com]  # change this <br>          - mcp_tool:<br>              in: [helm_list, pods_list, events_list]  # list of commands allowed</pre><ul><li>Specify the configuration files in the docker-composefile below. Customize the file paths to your setup</li></ul><pre>version: &#39;3.9&#39;<br>services:<br>  pomerium:<br>    image: pomerium/pomerium:latest<br>    container_name: pomerium<br>    network_mode: &quot;host&quot;<br>    volumes:<br>      - /tmp/pomerium:/root/<br>    command: --config=/root/pomerium.yaml<br>    depends_on:<br>      - kubernetes-mcp-server<br>  kubernetes-mcp-server:<br>    image: quay.io/manusa/kubernetes_mcp_server:latest-linux-amd64<br>    container_name: kubernetes-mcp-server<br>    network_mode: &quot;host&quot;<br>    environment:<br>      KUBECONFIG: &quot;/root/kubeconfig.yaml&quot;<br>    volumes:<br>      - /tmp/pomerium:/root/<br>    command: [&quot;--port&quot;, &quot;8080&quot;, &quot;--log-level&quot;, &quot;7&quot;]</pre><h3>3a. Use <a href="https://github.com/modelcontextprotocol/inspector">MCP Inspector</a> to test</h3><ul><li>Follow <a href="https://github.com/modelcontextprotocol/inspector?tab=readme-ov-file#running-the-inspector">these</a> instructions to download and install mcp-inspector.</li><li>📟 Launch mcp-inspector (see my example below)</li></ul><pre>~ % mcp-inspector<br><br>Starting MCP inspector...<br>⚙️ Proxy server listening on localhost:6277<br>🔑 Session token: 9e595de6f9033a71e4db82dbfa6222d596b99698cd0138c44407b8d51a0c28ae<br>   Use this token to authenticate requests or set DANGEROUSLY_OMIT_AUTH=true to disable auth<br>🚀 MCP Inspector is up and running at:<br>   http://localhost:6274/?MCP_PROXY_AUTH_TOKEN=9e595de6f9033a71e4db82dbfa6222d596b99698cd0138c44407b8d51a0c28ae<br>🌐 Opening browser...<br>New StreamableHttp connection request<br>Query parameters: {&quot;url&quot;:&quot;https://www.ukaul.net:8443/mcp&quot;,&quot;transportType&quot;:&quot;streamable-http&quot;}<br>Created StreamableHttp server transport<br>Created StreamableHttp client transport<br>Client &lt;-&gt; Proxy  sessionId: f4fe9bb0-b954-4a81-9e93-8344ab1cc81d<br>Error from MCP server: Error: Error POSTing to endpoint (HTTP 401): {&quot;error&quot;:&quot;Unauthenticated&quot;,&quot;request_id&quot;:&quot;083d7427-9176-4d72-81ed-b8b45d2c8b50&quot;}<br>    at StreamableHTTPClientTransport.send (file:///Users/umeshkaul/Projects/inspector/node_modules/@modelcontextprotocol/sdk/dist/esm/client/streamableHttp.js:284:23)<br>    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)<br>New StreamableHttp connection request<br>Query parameters: {&quot;url&quot;:&quot;https://www.ukaul.net:8443/mcp&quot;,&quot;transportType&quot;:&quot;streamable-http&quot;}<br>Created StreamableHttp server transport<br>Created StreamableHttp client transport<br>Client &lt;-&gt; Proxy  sessionId: be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Received GET message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Error from MCP server: Error: SSE stream disconnected: TypeError: terminated<br>    at processStream (file:///Users/umeshkaul/Projects/inspector/node_modules/@modelcontextprotocol/sdk/dist/esm/client/streamableHttp.js:203:88)<br>    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e<br>Error from MCP server: Error: Error POSTing to endpoint (HTTP 401): {&quot;error&quot;:&quot;Unauthorized&quot;,&quot;request_id&quot;:&quot;e9e63bda-4f70-4f16-bbc3-e35852873cca&quot;}<br>    at StreamableHTTPClientTransport.send (file:///Users/umeshkaul/Projects/inspector/node_modules/@modelcontextprotocol/sdk/dist/esm/client/streamableHttp.js:284:23)<br>    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)<br>Received POST message for sessionId be9eb6e4-646a-43c1-8218-7b16f92d4f6e</pre><ul><li><em>The following demonstration shows how only whitelisted tools are authorized while others are denied</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GLjajmmwfqdbjvOL4op2Tg.gif" /></figure><h3>3b. Use <a href="https://github.com/tileshq/mcp-cli">mcp-cli</a> to test</h3><ul><li>Use npm install -g mcp-cli to download and install mcp-cli. See <a href="https://github.com/tileshq/mcp-cli?tab=readme-ov-file#installation">this</a> link for detailed instructions.</li><li>💻 Launch the cli (see my example below)</li></ul><pre>mcp-cli % npm start https://www.ukaul.net:8443/mcp<br><br>&gt; mcp-cli@1.0.0 start<br>&gt; node dist/cli/cli.js https://www.ukaul.net:8443/mcp<br>🚀 MCP CLI - Interactive Client<br>Connecting to: https://www.ukaul.net:8443/mcp<br>🔗 Attempting to connect to https://www.ukaul.net:8443/mcp with OAuth...<br>🚢 Trying Streamable HTTP transport with OAuth...<br>Could not load OAuth Protected Resource metadata, falling back to /.well-known/oauth-authorization-server Error: Resource server does not implement OAuth 2.0 Protected Resource Metadata.<br>📌 OAuth redirect handler called - opening browser<br>🌐 Opening browser for authorization: https://www.ukaul.net:8443/.pomerium/mcp/authorize?response_type=code&amp;client_id=db0c4b07-8981-41c5-a845-6e9029da35de&amp;code_challenge=P43mK15rVhrz0q-boNx5QR99PSMz4eAlXqEqFhTKKGQ&amp;code_challenge_method=S256&amp;redirect_uri=http%3A%2F%2Flocalhost%3A8090%2Fcallback&amp;scope=mcp%3Atools<br>Client error: UnauthorizedError: Unauthorized<br>OAuth callback server started on http://localhost:8090<br>📥 Received callback: /callback?code=yPT0BOxo9M7uf8%2Ft%2FTkimHfawyEwgQXeXv1C5pP%2F6g%2Ffc48VhSYw8aTDKLSkYsvn1abNgzfI9inA%2F%2BmeiICjlkxFo1IbreNeXKOMsmKwxtoiWNQ9H8gfLP3D56755w%3D%3D&amp;state=<br>✅ Authorization code received: yPT0BOxo9M...<br>✅ Authorization code received: yPT0BOxo9M...<br>🔐 Authorization completed, reconnecting...<br>Could not load OAuth Protected Resource metadata, falling back to /.well-known/oauth-authorization-server Error: Resource server does not implement OAuth 2.0 Protected Resource Metadata.<br>🎯 Successfully connected using streamable-http transport<br>🎯 Interactive MCP Client<br>Connected via: streamable-http<br>Type &quot;help&quot; for available commands or &quot;quit&quot; to exit<br>mcp&gt; 📥 Received callback: /apple-touch-icon-precomposed.png<br>❌ No authorization code or error in callback<br>📥 Received callback: /apple-touch-icon.png<br>❌ No authorization code or error in callback<br>mcp&gt;<br>mcp&gt; list<br>📋 Available tools:<br>1. configuration_view<br>   Description: Get the current Kubernetes configuration content as a kubeconfig YAML<br>2. events_list<br>   Description: List all the Kubernetes events in the current cluster from all namespaces</pre><ul><li><em>The following demonstration shows how only whitelisted tools are authorized while others are denied</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQ3P5E6AvNRk_K_vIdxn5g.gif" /></figure><h3>More about Pomerium MCP Support</h3><p>Pomerium offers three methods to protect MCP servers — we only explored the first one in this post.</p><p><strong>1. External AI Access to Internal MCP Servers</strong></p><p>Pomerium acts as a secure gateway that verifies both user and AI authorization before allowing access to internal MCP servers. This prevents direct external AI access to sensitive internal data while applying granular role-based permissions. For example, a marketing manager using Claude to generate reports from the company’s CRM database would have Pomerium verify both the manager’s identity and authorization before allowing access to the internal MCP server e.g they may get read-only access to some tools while getting full access to some other tools exposed by the MCP server. This is the one which we will demonstrate in this blog post.</p><p><strong>2. OAuth Management for External Services</strong></p><p>Pomerium handles authentication flows for external services like GitHub that MCP servers need to connect to, providing secure temporary tokens instead of storing credentials directly. This eliminates credential exposure to AI agents while maintaining secure access. For instance, when an internal AI agent needs to search private GitHub repositories, Pomerium manages the OAuth flow with GitHub and provides the MCP server with a secure token, ensuring the AI never directly handles or sees sensitive developer credentials.</p><p><strong>3. Secure Token Provision for Internal Applications</strong></p><p>Pomerium issues user-specific, short-lived tokens to internal applications, enabling AI features to authenticate on behalf of specific employees while ensuring all AI actions are auditable and user-attributable. For example, when an employee uses a project management application’s AI feature to create tasks, Pomerium issues a secure token that the application passes to the AI, allowing the AI to prove it’s acting on the employee’s behalf rather than authenticating itself directly.</p><h3>References</h3><ul><li><a href="https://www.pomerium.com/docs/capabilities/mcp">Pomerium MCP Documentation</a></li><li><a href="https://console.cloud.google.com/auth/overview?project=pomerium-playground">Google Cloud Console</a></li><li><a href="https://github.com/containers/kubernetes-mcp-server">Kubernetes MCP Server</a></li><li><a href="https://modelcontextprotocol.io/legacy/tools/inspector">MCP Inspector</a></li><li><a href="https://github.com/tileshq/mcp-cli">MCP-CLI</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a186adc0d7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ztunnel Under the Hood: A Deep Dive into Istio’s Ambient Mode Networking]]></title>
            <link>https://medium.com/@umeshkaul_39077/ztunnel-under-the-hood-a-deep-dive-into-istios-ambient-mode-networking-960b6de10ee6?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/960b6de10ee6</guid>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Fri, 26 Sep 2025 04:31:24 GMT</pubDate>
            <atom:updated>2025-09-29T03:48:47.530Z</atom:updated>
            <content:encoded><![CDATA[<h3>What You’ll Learn</h3><ul><li>How Istio eliminates sidecar containers while maintaining secure networking</li><li>The Linux kernel magic that allows processes to “teleport” between namespaces</li><li>Step-by-step implementation of a real-world networking concept</li><li>Why this approach is more efficient than traditional service mesh architectures</li><li>Refer to this <a href="https://github.com/umeshkaul/ztunnel-under-the-hood">repo</a> for full source code used below</li></ul><h3>Background</h3><p>This post along with this <a href="https://github.com/umeshkaul/ztunnel-under-the-hood">repo</a> explores the core networking technology behind Istio’s ztunnel, a key component of its ambient mesh. Unlike the traditional sidecar model that injects a separate container into each pod, ztunnel operates as a single DaemonSet on each node. This approach allows it to handle the networking for all pods on that node, simplifying the architecture and improving efficiency.</p><p>The unique part of ztunnel’s design is how it intercepts and manages traffic without using the common veth pairs, some kind of vpn tunnel or running a full sidecar proxy. Instead, it leverages powerful, low-level Linux kernel features to directly manage sockets in the network namespace of each application pod.</p><h3>Ztunnel’s Kernel Magic Explained</h3><p>At the heart of this capability are two key Linux features:</p><p><strong>Network Namespaces:</strong> The Linux kernel isolates the network stack of each pod into a separate network namespace. These namespaces are treated as kernel objects and can be referenced.</p><p><strong>setns() System Call:</strong> A privileged process, like ztunnel&#39;s node agent, can use the setns() system call to temporarily enter another process&#39;s namespace.</p><p>The ztunnel process combines these features to “teleport” into a pod’s namespace, create the necessary listening sockets, and then return to its own namespace. Because the sockets are “pinned” to the pod’s namespace, they remain active and functional there. This allows ztunnel to listen for and redirect traffic to the pod’s workload without the need for a sidecar container. Traffic redirection is then handled by other kernel features like iptables TPROXY or eBPF.</p><p>This repository serves as a practical demonstration of these exact concepts, providing a concrete example of how a process can create and manage sockets in a separate network namespace. The code in the repository is a great way to see how these advanced kernel features work in a simplified, real-world context.</p><h3>The Istio Solution</h3><p>As explained in <a href="https://blog.howardjohn.info/posts/ztunnel-compute-traffic-view/">Howard John’s excellent blog post on ztunnel’s architecture</a> and <a href="https://istio.io/latest/docs/ambient/architecture/traffic-redirection/#:~:text=Once%20the%20istio,node%2Dlocal%20ztunnel">Istio documentation</a>, Istio solves this through a clever handoff mechanism:</p><ol><li><strong>CNI Node Agent</strong> (istio-cni) opens the pod&#39;s network namespace file</li><li><strong>CNI Node Agent</strong> sends the network namespace file descriptor to ztunnel via Unix Domain Socket using SCM_RIGHTS</li><li><strong>Ztunnel</strong> receives the file descriptor and uses setns() to enter the pod&#39;s network namespace</li><li><strong>Ztunnel</strong> creates listening sockets inside the pod’s namespace</li><li><strong>Ztunnel</strong> returns to its original namespace while the sockets remain bound to the pod’s namespace</li></ol><p>The key insight is that while sockets are created in the target namespace, they remain valid and accessible from the host namespace.</p><h3>Emulating the Istio Solution for fun :)</h3><h4>What This Demo Shows</h4><p><strong>Goal:</strong> Make a process (“ztunnel”) create <strong>listeners inside a pod’s netns</strong> after a <strong>CNI-like sender</strong> provides a <strong>netns file descriptor (FD)</strong> over a <strong>Unix Domain Socket (UDS)</strong> — just like Istio ambient mode describes.</p><p><strong>Mapping:</strong></p><ul><li><strong>cni-emulator</strong> ⇢ the <strong>istio-cni node agent</strong>: opens the pod&#39;s netns file (e.g., /var/run/netns/ns2) and <strong>sends that FD</strong> to ztunnel via UDS using SCM_RIGHTS.</li><li><strong>ztunnel-emulator</strong> ⇢ the <strong>ztunnel proxy</strong>: <strong>receives the netns FD</strong>, temporarily <strong>setns() into that netns</strong>, <strong>binds listeners</strong> on 127.0.0.1:{15008,15006,15001} inside the pod netns, then returns to its own netns. The sockets remain attached to the <strong>pod netns</strong>.</li></ul><h4>High level Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMglRJIOKabNvL-JfoYpRA.png" /></figure><p><strong>KEY RESULT :</strong> Ztunnel process (running in host namespace) now has active listeners inside the pod’s namespace (ns2). These listeners can accept connections from applications running inside the pod, enabling sidecar-less networking.</p><h4>CNI Emulator (cni-emulator/cni-emulator.go)</h4><p>Simulates the istio-cni node agent:</p><ul><li><strong>Connects</strong> to ztunnel’s Unix Domain Socket</li><li><strong>Opens</strong> the pod’s network namespace file</li><li><strong>Sends</strong> the file descriptor to ztunnel using SCM_RIGHTS</li></ul><pre>// Key code snippet<br>f, err := os.Open(nsPath)  // Open pod&#39;s netns file<br>rights := unix.UnixRights(int(f.Fd()))  // Prepare FD for transfer<br>c.WriteMsgUnix([]byte{1}, rights, nil)  // Send FD via UDS</pre><h4>Ztunnel Emulator (ztunnel-emulator/ztunnel-emulator.go)</h4><p>Simulates the ztunnel proxy:</p><ul><li><strong>Listens</strong> on Unix Domain Socket for CNI connections</li><li><strong>Receives</strong> the network namespace file descriptor</li><li><strong>Enters</strong> the pod’s namespace using setns()</li><li><strong>Creates</strong> listening sockets on ports 15008, 15006, 15001</li><li><strong>Returns</strong> to original namespace (sockets remain in pod namespace)</li></ul><pre>// Key code snippet<br>nsfd, err := recvFD(conn)  // Receive netns FD from CNI<br>unix.Setns(nsfd, unix.CLONE_NEWNET)  // Enter pod&#39;s netns<br>// Create listeners on 15008, 15006, 15001<br>net.Listen(&quot;tcp4&quot;, &quot;127.0.0.1:&quot;+port)</pre><h3>Demo Walk-thru</h3><h4>Prerequisites</h4><ul><li>Linux system with network namespace support</li><li>Go 1.24+</li><li>Root privileges or appropriate capabilities</li></ul><h4>1. Build the Components from <a href="https://github.com/umeshkaul/ztunnel-under-the-hood">this repo</a></h4><pre># Build CNI emulator<br>cd cni-emulator<br>go build -o cni-emulator cni-emulator.go</pre><pre># Build ztunnel emulator<br>cd ../ztunnel-emulator  <br>go build -o ztunnel-emulator ztunnel-emulator.go</pre><h4>2. Set Capabilities (if not running as root)</h4><pre>sudo setcap cap_sys_admin+ep ./ztunnel-emulator</pre><h4>3. Create Test Network Namespace</h4><pre>$&gt; sudo ip netns add ns2</pre><pre>$&gt; sudo ip -n ns2 link set lo up</pre><pre>$&gt; sudo ip -n ns2 link show<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</pre><h4>4. Start Ztunnel Emulator</h4><pre><br>$&gt; sudo ./ztunnel-emulator/ztunnel-emulator /tmp/zt.sock &amp;<br>[1] 3074196</pre><pre>$&gt; ps -ef | egrep -i ztunnel-emulator<br>root     3074196 3031603  0 22:01 pts/20   00:00:00 sudo ./ztunnel-emulator/ztunnel-emulator /tmp/zt.sock<br>root     3074205 3074196  0 22:01 pts/25   00:00:00 sudo ./ztunnel-emulator/ztunnel-emulator /tmp/zt.sock<br>root     3074206 3074205  0 22:01 pts/25   00:00:00 ./ztunnel-emulator/ztunnel-emulator /tmp/zt.sock<br>ukaul    3074493 3031603  0 22:01 pts/20   00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox --exclude-dir=.venv --exclude-dir=venv -E -i ztunnel-emulator</pre><pre>5. Send Network Namespace (CNI Step)</pre><pre>$&gt; sudo ip netns exec ns2 ss -ntlp<br>State       Recv-Q       Send-Q             Local Address:Port             Peer Address:Port      Process       </pre><pre>$&gt; sudo ./cni-emulator/cni-emulator /tmp/zt.sock /var/run/netns/ns2<br>listeners ready</pre><pre>$&gt; sudo ip netns exec ns2 ss -ntlp<br>[sudo] password for ukaul: <br>State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port Process                                        <br>LISTEN  0       4096         127.0.0.1:15008        0.0.0.0:*     users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=9))  <br>LISTEN  0       4096         127.0.0.1:15006        0.0.0.0:*     users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=10)) <br>LISTEN  0       4096         127.0.0.1:15001        0.0.0.0:*     users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=11))</pre><h4>6. Verify Listeners in Pod Namespace</h4><pre># Launch a shell into the namesapce </pre><pre>$&gt; sudo ip netns exec ns2 bash<br>    <br>root$ ss -ntlp<br>State       Recv-Q      Send-Q           Local Address:Port            Peer Address:Port     Process                                            <br>LISTEN      0           4096                 127.0.0.1:15008                0.0.0.0:*         users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=9))      <br>LISTEN      0           4096                 127.0.0.1:15006                0.0.0.0:*         users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=10))     <br>LISTEN      0           4096                 127.0.0.1:15001                0.0.0.0:*         users:((&quot;ztunnel-emulato&quot;,pid=3074206,fd=11))     </pre><pre># Test connectivity</pre><pre>root$ nc -v -z localhost 15008<br>Connection to localhost (127.0.0.1) 15008 port [tcp/*] succeeded!<br> <br>root$ nc -v -z localhost 15006<br>Connection to localhost (127.0.0.1) 15006 port [tcp/*] succeeded!<br> <br>root$ nc -v -z localhost 15001<br>Connection to localhost (127.0.0.1) 15001 port [tcp/*] succeeded!<br> <br>root$ nc -v -z localhost 15002<br>nc: connect to localhost (127.0.0.1) port 15002 (tcp) failed: Connection refused</pre><h4>7. Cleanup</h4><pre># Remove test namespace<br>sudo ip netns delete ns2</pre><pre># Stop ztunnel emulator<br>sudo pkill ztunnel-emulator</pre><h3>References</h3><p>Basics :</p><ul><li><a href="https://man7.org/linux/man-pages/man7/unix.7.html">Unix Domain Sockets and File Descriptor Passing</a> — Linux manual page explaining UDS and SCM_RIGHTS for FD transfer</li><li><a href="https://man7.org/linux/man-pages/man7/namespaces.7.html">Network Namespaces</a> — Linux manual page covering network namespace concepts and setns() system call</li></ul><p>Istio Specific :</p><ul><li><a href="https://istio.io/latest/docs/ambient/architecture/traffic-redirection/">Istio Ambient Traffic Redirection</a> — Official Istio documentation on ambient mode traffic redirection mechanisms</li><li><a href="https://istio.io/latest/docs/ambient/architecture/traffic-redirection/#:~:text=Once%20the%20istio,node%2Dlocal%20ztunnel">Istio CNI-Ztunnel Communication</a> — Detailed explanation of how CNI and ztunnel communicate via file descriptors</li><li><a href="https://blog.howardjohn.info/posts/ztunnel-compute-traffic-view/">Howard John’s Ztunnel Architecture Blog</a> — Deep dive into ztunnel’s compute traffic view and namespace management</li><li><a href="https://www.solo.io/blog/understanding-istio-ambient-ztunnel-and-secure-overlay">Understanding Istio Ambient Ztunnel and Secure Overlay</a> — Comprehensive overview of ambient mode architecture and security model</li><li><a href="https://istio.io/latest/blog/2023/rust-based-ztunnel/">Istio’s Rust-based Ztunnel</a> — Announcement and technical details of Istio’s migration to Rust-based ztunnel implementation</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=960b6de10ee6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[curl walk-through of updated K8sGPT MCP Server]]></title>
            <link>https://medium.com/@umeshkaul_39077/curl-walk-through-of-updated-k8sgpt-mcp-server-d1ba1fad79e0?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1ba1fad79e0</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[k8sgpt]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Sun, 10 Aug 2025 13:41:57 GMT</pubDate>
            <atom:updated>2025-08-25T09:24:43.792Z</atom:updated>
            <content:encoded><![CDATA[<p><em>IMP NOTE: </em>This is an update to my <a href="https://medium.com/@umeshkaul_39077/introducing-http-support-for-k8sgpt-mcp-server-with-curl-walk-through-20549903a876"><em>previous post</em></a> — we’ve enhanced MCP support for k8sgpt by moving to a more up-to-date Go library. See <a href="https://github.com/k8sgpt-ai/k8sgpt/pull/1557"><em>this PR</em></a> for details.</p><h3>Introduction</h3><p>The<a href="https://k8sgpt.ai/"> K8sGPT</a> Model Context Protocol (MCP) server provides a powerful interface for analyzing Kubernetes clusters and getting AI-powered insights into cluster issues. This guide is an update to <a href="https://medium.com/@umeshkaul_39077/introducing-http-support-for-k8sgpt-mcp-server-with-curl-walk-through-20549903a876">my previous post</a> and will walk you through testing the MCP server using curl commands, from basic initialization to advanced AI analysis.</p><h3>Prerequisites</h3><p>Before you begin, ensure you have:</p><ul><li><strong>K8sGPT installed</strong> and configured</li><li><strong>curl</strong> installed on your system</li><li><strong>Access to a Kubernetes cluster</strong> with some resources to analyze</li><li><strong>AI provider configured</strong> (OpenAI, Azure, etc.) for AI-powered analysis</li></ul><h3><strong>Pre-requisite</strong></h3><p>Data Flow</p><pre>curl----&gt;K8sGPT_MCP_Server------&gt;Kubernetes_Cluster</pre><ol><li>Install the latest version of K8sGPT cli using <a href="https://github.com/k8sgpt-ai/k8sgpt?tab=readme-ov-file#cli-installation">these</a> instructions (you need at least <a href="https://github.com/k8sgpt-ai/k8sgpt/releases/tag/v0.4.24">v0.4.24</a>)</li><li>Make sure you have access to some Kubernetes cluster i.e your KUBECONFIG is pointing to a working Kubernetes cluster and working.</li></ol><h3>Starting the MCP Server</h3><p>First, start the K8sGPT MCP server with the following command:</p><pre>k8sgpt serve --mcp --mcp-http -v</pre><p>This command:</p><ul><li>Starts the MCP server on port 8089</li><li>Enables HTTP mode for MCP communication</li><li>Enables verbose logging for debugging</li><li>Starts the main K8sGPT server on port 8080</li><li>Starts the metrics server on port 8081</li></ul><p><strong>Expected startup output:</strong></p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1754011569.2079282,&quot;caller&quot;:&quot;server/server.go:147&quot;,&quot;msg&quot;:&quot;binding metrics to 8081&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1754011569.2081358,&quot;caller&quot;:&quot;server/mcp.go:76&quot;,&quot;msg&quot;:&quot;Starting MCP HTTP server&quot;,&quot;port&quot;:&quot;8089&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1754011569.208404,&quot;caller&quot;:&quot;server/server.go:106&quot;,&quot;msg&quot;:&quot;binding api to 8080&quot;}</pre><p>The server will initialize various analyzers and be ready to accept MCP requests.</p><h3>Understanding the MCP Protocol</h3><p>The K8sGPT Model Context Protocol (MCP) server uses <strong>JSON-RPC 2.0 over HTTP POST</strong>. All requests below :</p><ul><li>Use <strong>POST method</strong></li><li>Include Content-Type: application/json header</li><li>Follow <strong>JSON-RPC 2.0 format</strong></li><li>Include session management via Mcp-Session-Id header (required , although it does not get validated)</li></ul><h3>Step-by-Step Testing Guide</h3><h3>Step 1: Initialize the MCP Server</h3><p>First, establish a session with the MCP server:</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 1,<br>    &quot;method&quot;: &quot;initialize&quot;,<br>    &quot;params&quot;: {<br>      &quot;protocolVersion&quot;: &quot;2025-03-26&quot;,<br>      &quot;capabilities&quot;: {<br>        &quot;tools&quot;: {},<br>        &quot;resources&quot;: {},<br>        &quot;prompts&quot;: {}<br>      },<br>      &quot;clientInfo&quot;: {<br>        &quot;name&quot;: &quot;test-client&quot;,<br>        &quot;version&quot;: &quot;1.0.0&quot;<br>      }<br>    }<br>  }&#39;</pre><p><strong>Expected Response:</strong></p><pre>{<br>  &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>  &quot;id&quot;: 1,<br>  &quot;result&quot;: {<br>    &quot;protocolVersion&quot;: &quot;2025-03-26&quot;,<br>    &quot;capabilities&quot;: {<br>      &quot;prompts&quot;: {},<br>      &quot;resources&quot;: {<br>        &quot;subscribe&quot;: true<br>      },<br>      &quot;tools&quot;: {<br>        &quot;listChanged&quot;: true<br>      }<br>    },<br>    &quot;serverInfo&quot;: {<br>      &quot;name&quot;: &quot;k8sgpt&quot;,<br>      &quot;version&quot;: &quot;1.0.0&quot;<br>    }<br>  }<br>}</pre><p><strong>Important:</strong> Extract the session ID from the response headers:</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;method&quot;:&quot;initialize&quot;,&quot;params&quot;:{&quot;protocolVersion&quot;:&quot;2025-03-26&quot;,&quot;capabilities&quot;:{&quot;tools&quot;:{},&quot;resources&quot;:{},&quot;prompts&quot;:{}},&quot;clientInfo&quot;:{&quot;name&quot;:&quot;test-client&quot;,&quot;version&quot;:&quot;1.0.0&quot;}}}&#39; \<br>  -v 2&gt;&amp;1 | grep -i &quot;mcp-session-id&quot;</pre><h3>Step 2: List Available Tools</h3><p>List all available tools using the session ID:</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 2,<br>    &quot;method&quot;: &quot;tools/list&quot;,<br>    &quot;params&quot;: {}<br>  }&#39;</pre><p><strong>Available Tools:</strong></p><ul><li><strong>analyze</strong>: Analyze Kubernetes resources for issues and problems</li><li><strong>cluster-info</strong>: Get Kubernetes cluster information and version</li><li><strong>config</strong>: Configure K8sGPT settings including custom analyzers and cache</li></ul><p>The response will include detailed schema information for each tool, including input parameters and descriptions.</p><h3>Step 3: Analyze Cluster Problems (Basic)</h3><p>Get a list of all problems in the cluster:</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 3,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;explain&quot;: false<br>      }<br>    }<br>  }&#39;</pre><p><strong>What you’ll get:</strong></p><ul><li><strong>Total Problems Found</strong>: Number of issues detected</li><li><strong>Problem Types</strong>: Different categories of issues (Pods, ConfigMaps, etc.)</li><li><strong>Status</strong>: ProblemDetected, NoProblem, or Error</li><li><strong>Detailed Results</strong>: JSON structure with specific error messages</li></ul><p><strong>Example Response Analysis:</strong></p><ul><li><strong>Total Problems Found:</strong> 13</li><li><strong>Problem Types:</strong> 1 Pod with image pull error, 12 unused ConfigMaps</li><li><strong>Status:</strong> ProblemDetected</li><li><strong>Provider:</strong> Empty (no AI provider specified)</li></ul><h3>Step 4: Filter Analysis by Resource Type</h3><p>Analyze only specific resource types (e.g., Pods):</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 4,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;filters&quot;: [&quot;Pod&quot;],<br>        &quot;explain&quot;: false<br>      }<br>    }<br>  }&#39;</pre><p><strong>Available Filters:</strong></p><ul><li>Pod, Deployment, Service</li><li>ConfigMap, Secret, Ingress</li><li>PersistentVolumeClaim, Job, CronJob</li><li>And many more Kubernetes resource types</li></ul><p><strong>Response Analysis:</strong></p><ul><li><strong>Total Problems Found:</strong> 1 (filtered from 13)</li><li><strong>Problem Type:</strong> Pod with image pull error</li><li><strong>Status:</strong> ProblemDetected</li><li><strong>Filter Applied:</strong> Pod only</li></ul><h3>Step 5: Get AI-Powered Analysis with Explanations</h3><p>Enable AI analysis with detailed explanations (this will eventually be migrated to use <a href="https://modelcontextprotocol.io/specification/2025-06-18/client/sampling">MCP sampling</a> which provides a standardized way for MCP servers to request LLM sampling (“completions” or “generations”) from language models via clients)</p><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 5,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;filters&quot;: [&quot;Pod&quot;],<br>        &quot;explain&quot;: true<br>      }<br>    }<br>  }&#39;</pre><p><strong>AI Analysis Output:</strong></p><pre>AI Provider: openai</pre><pre>0: Pod default/bad-nginx()<br>- Error: Back-off pulling image &quot;nginx:bad&quot;: ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image &quot;docker.io/library/nginx:bad&quot;: docker.io/library/nginx:bad: not found<br>Error: The image &quot;nginx:bad&quot; cannot be found in the Docker registry, leading to a failure in pulling the image.</pre><pre>Solution: <br>1. Check the image name for typos.<br>2. Verify the image tag &quot;bad&quot; exists in the Docker Hub.<br>3. If not, use a valid tag (e.g., &quot;nginx:latest&quot;).<br>4. Update your Kubernetes deployment with the correct image.<br>5. Redeploy the application.</pre><p><strong>What AI Analysis Provides:</strong></p><ul><li><strong>Error Explanation</strong>: Human-readable description of the problem</li><li><strong>Root Cause Analysis</strong>: Why the issue occurred</li><li><strong>Step-by-Step Solution</strong>: Actionable steps to resolve the issue</li><li><strong>Best Practices</strong>: Recommendations for prevention</li></ul><h3>Advanced Analysis Options</h3><h3>Analyze Specific Namespace</h3><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 6,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;namespace&quot;: &quot;default&quot;,<br>        &quot;explain&quot;: true<br>      }<br>    }<br>  }&#39;</pre><h3>Use Specific AI Backend</h3><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 7,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;backend&quot;: &quot;openai&quot;,<br>        &quot;explain&quot;: true<br>      }<br>    }<br>  }&#39;</pre><h3>Get Cluster Information</h3><pre>curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: YOUR_SESSION_ID_HERE&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 8,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;cluster-info&quot;,<br>      &quot;arguments&quot;: {}<br>    }<br>  }&#39;</pre><p><strong>Expected Response:</strong></p><pre>{<br>  &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>  &quot;id&quot;: 8,<br>  &quot;result&quot;: {<br>    &quot;content&quot;: [<br>      {<br>        &quot;type&quot;: &quot;text&quot;,<br>        &quot;text&quot;: &quot;Kubernetes v1.28.0&quot;<br>      }<br>    ]<br>  }<br>}</pre><h3>Complete Test Script</h3><p>Here’s a complete script that performs all the testing steps:</p><pre>#!/bin/bash</pre><pre># MCP Server URL<br>MCP_URL=&quot;http://localhost:8089/mcp&quot;</pre><pre>echo &quot;Step 1: Initializing MCP server...&quot;<br>INIT_RESPONSE=$(curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 1,<br>    &quot;method&quot;: &quot;initialize&quot;,<br>    &quot;params&quot;: {<br>      &quot;protocolVersion&quot;: &quot;2025-03-26&quot;,<br>      &quot;capabilities&quot;: {<br>        &quot;tools&quot;: {},<br>        &quot;resources&quot;: {},<br>        &quot;prompts&quot;: {}<br>      },<br>      &quot;clientInfo&quot;: {<br>        &quot;name&quot;: &quot;test-client&quot;,<br>        &quot;version&quot;: &quot;1.0.0&quot;<br>      }<br>    }<br>  }&#39;)</pre><pre>echo &quot;Initialization response: $INIT_RESPONSE&quot;</pre><pre># Extract session ID<br>SESSION_ID=$(curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:1,&quot;method&quot;:&quot;initialize&quot;,&quot;params&quot;:{&quot;protocolVersion&quot;:&quot;2025-03-26&quot;,&quot;capabilities&quot;:{&quot;tools&quot;:{},&quot;resources&quot;:{},&quot;prompts&quot;:{}},&quot;clientInfo&quot;:{&quot;name&quot;:&quot;test-client&quot;,&quot;version&quot;:&quot;1.0.0&quot;}}}&#39; \<br>  -v 2&gt;&amp;1 | grep -i &quot;mcp-session-id&quot; | cut -d&#39; &#39; -f3)</pre><pre>echo &quot;Session ID: $SESSION_ID&quot;</pre><pre>echo -e &quot;\nStep 2: Listing tools...&quot;<br>curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: $SESSION_ID&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:2,&quot;method&quot;:&quot;tools/list&quot;,&quot;params&quot;:{}}&#39; | jq &#39;.&#39;</pre><pre>echo -e &quot;\nStep 3: Analyzing cluster problems...&quot;<br>curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: $SESSION_ID&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:3,&quot;method&quot;:&quot;tools/call&quot;,&quot;params&quot;:{&quot;name&quot;:&quot;analyze&quot;,&quot;arguments&quot;:{&quot;explain&quot;:false}}}&#39; | jq &#39;.&#39;</pre><pre>echo -e &quot;\nStep 4: Analyzing Pods only...&quot;<br>curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: $SESSION_ID&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:4,&quot;method&quot;:&quot;tools/call&quot;,&quot;params&quot;:{&quot;name&quot;:&quot;analyze&quot;,&quot;arguments&quot;:{&quot;filters&quot;:[&quot;Pod&quot;],&quot;explain&quot;:false}}}&#39; | jq &#39;.&#39;</pre><pre>echo -e &quot;\nStep 5: Getting AI analysis with explanations...&quot;<br>curl -s -X POST $MCP_URL \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Accept: application/json,text/event-stream&quot; \<br>  -H &quot;Mcp-Session-Id: $SESSION_ID&quot; \<br>  -d &#39;{&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;id&quot;:5,&quot;method&quot;:&quot;tools/call&quot;,&quot;params&quot;:{&quot;name&quot;:&quot;analyze&quot;,&quot;arguments&quot;:{&quot;filters&quot;:[&quot;Pod&quot;],&quot;explain&quot;:true}}}&#39; | jq &#39;.&#39;</pre><pre>echo -e &quot;\nTesting completed!&quot;</pre><h3>Conclusion</h3><p>The K8sGPT Model Context Protocol server provides a powerful interface for Kubernetes cluster analysis. By following this guide, you can:</p><ul><li>✅ Initialize and establish sessions with the MCP server</li><li>✅ List available tools and their capabilities</li><li>✅ Perform basic cluster analysis</li><li>✅ Filter analysis by resource types</li><li>✅ Get AI-powered explanations and solutions</li><li>✅ Use advanced features like namespace filtering and custom AI backends</li></ul><p>The MCP protocol enables seamless integration with other tools and provides a standardized way to interact with K8sGPT’s analysis capabilities.</p><h3>References</h3><ul><li><a href="https://docs.k8sgpt.ai/">K8sGPT Documentation</a></li><li><a href="https://modelcontextprotocol.io/">Model Context Protocol Specification</a></li><li><a href="https://www.jsonrpc.org/specification">JSON-RPC 2.0 Specification</a></li></ul><p><em>Happy testing! 🚀</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1ba1fad79e0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing HTTP Support for K8sGPT MCP Server— with curl walk-through]]></title>
            <link>https://medium.com/@umeshkaul_39077/introducing-http-support-for-k8sgpt-mcp-server-with-curl-walk-through-20549903a876?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/20549903a876</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mcp-inspector]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Sun, 20 Jul 2025 18:07:37 GMT</pubDate>
            <atom:updated>2025-08-10T13:45:24.827Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Important Note:</strong> Given the rapid pace of change in Agentic AI — especially with MCP — the instructions below apply only to k8sgpt MCP server version <strong>v0.4.22</strong>. See my <a href="https://medium.com/@umeshkaul_39077/curl-walk-through-of-updated-k8sgpt-mcp-server-d1ba1fad79e0">latest post</a> for testing <strong>v0.4.23</strong> and above.</p><p>K8sGPT MCP server now also <a href="https://github.com/k8sgpt-ai/k8sgpt/pull/1546">supports HTTP </a>(besides stdio). This post explores this interface using curl and MCP <a href="https://github.com/modelcontextprotocol/inspector">inspector</a>.</p><blockquote><a href="https://k8sgpt.ai/">K8sGPT</a> is an AI-powered tool that helps diagnose and fix Kubernetes issues with intelligent insights and automated troubleshooting. Prior to this change, k8sgpt’s MCP server lived only over stdio — making it hard to run remotely, integrate with proxies, or serve multiple clients.</blockquote><p>This update brings k8sgpt in line with the evolving <a href="https://modelcontextprotocol.io/introduction#http-transport">Model Context Protocol (MCP)</a>, allowing it to serve context over plain HTTP, opening the door for:</p><ul><li>Remote usage without needing subprocess management</li><li>Simpler integration into web-native AI agent stacks</li><li>Compatibility with HTTP-based proxies, auth layers, and service meshes</li></ul><h3><strong>Test using curl</strong></h3><p>Data Flow</p><pre>curl----&gt;K8sGPT_MCP_Server------&gt;Kubernetes_Cluster</pre><ol><li>Install the latest version of K8sGPT cli using <a href="https://github.com/k8sgpt-ai/k8sgpt?tab=readme-ov-file#cli-installation">these</a> instructions (you need at least <a href="https://github.com/k8sgpt-ai/k8sgpt/releases/tag/v0.4.22">v0.4.22</a>)</li><li>Make sure you have access to some Kubernetes cluster i.e your KUBECONFIG is pointing to a working Kubernetes cluster e.g.</li></ol><pre># Verify you have connectivity to a working kubernetes cluster<br>$ k cluster-info<br>Kubernetes control plane is running at https://127.0.0.1:56708<br>CoreDNS is running at https://127.0.0.1:56708/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy<br><br>To further debug and diagnose cluster problems, use &#39;kubectl cluster-info dump&#39;.</pre><ol><li>Start the mcp server using the command <em>k8sgpt serve — mcp — mcp-http</em>. By default it uses tcp port 8089 for mcp http. See example below.</li></ol><pre># Make sure you have the right minimum version<br>$ k8sgpt version                                                                                                                                                              1 ↵<br>k8sgpt: 0.4.22 (fe46087), built at: unknown<br><br># Test to make sure k8sgpt command is working before we start mcp server<br>$ k8sgpt analyze<br>AI Provider: AI not used; --explain not set<br><br>No problems detected<br><br># Start the server, by default it listens on <br>$ k8sgpt serve --mcp --mcp-http                   <br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1752858865.4233768,&quot;caller&quot;:&quot;server/mcp.go:51&quot;,&quot;msg&quot;:&quot;starting MCP server with http transport on port&quot;,&quot;port&quot;:&quot;8089&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1752858865.423517,&quot;caller&quot;:&quot;server/server.go:147&quot;,&quot;msg&quot;:&quot;binding metrics to 8081&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1752858865.424039,&quot;caller&quot;:&quot;server/server.go:106&quot;,&quot;msg&quot;:&quot;binding api to 8080&quot;}<br>[Send] Attempting to send response with key: 0<br>W0718 13:15:21.664733   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 0<br>W0718 13:17:12.229500   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 0<br>W0718 13:17:27.916235   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 0<br>W0718 13:17:34.446426   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 0<br>W0718 13:17:44.495824   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 0<br>[Send] Attempting to send response with key: 0<br>[Send] Attempting to send response with key: 2<br>[Send] Attempting to send response with key: 2<br>[Send] Attempting to send response with key: 2<br>W0718 13:20:22.364491   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 2<br>[Send] Attempting to send response with key: 5<br>W0718 13:22:54.128367   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 5<br>W0718 13:23:15.111462   31017 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5<br>[Send] Attempting to send response with key: 5</pre><p>2. Use curl to test mcp interface e.g. initialize the mcp server</p><pre>$ curl -X POST http://localhost:8089/mcp \     <br> -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 1,<br>    &quot;method&quot;: &quot;initialize&quot;<br>  }&#39;<br><br>{&quot;id&quot;:1,&quot;jsonrpc&quot;:&quot;2.0&quot;,&quot;result&quot;:{&quot;capabilities&quot;:{&quot;prompts&quot;:{&quot;listChanged&quot;:false},&quot;resources&quot;:{&quot;listChanged&quot;:false},&quot;tools&quot;:{&quot;listChanged&quot;:false}},&quot;protocolVersion&quot;:&quot;2024-11-05&quot;,&quot;serverInfo&quot;:{&quot;name&quot;:&quot;k8sgpt&quot;,&quot;version&quot;:&quot;1.0.0&quot;}}}%</pre><p>3. Use curl to get list of tools</p><pre>$ curl -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 1,<br>    &quot;method&quot;: &quot;tools/list&quot;<br>  }&#39;<br><br>{<br>  &quot;id&quot;: 1,<br>  &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>  &quot;result&quot;: {<br>    &quot;tools&quot;: [<br>      {<br>        &quot;description&quot;: &quot;Analyze Kubernetes resources&quot;,<br>        &quot;inputSchema&quot;: {<br>          &quot;$schema&quot;: &quot;https://json-schema.org/draft/2020-12/schema&quot;,<br>          &quot;properties&quot;: {<br>            &quot;namespace&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;backend&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;language&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;filters&quot;: {<br>              &quot;items&quot;: {<br>                &quot;type&quot;: &quot;string&quot;<br>              },<br>              &quot;type&quot;: &quot;array&quot;<br>            },<br>            &quot;labelSelector&quot;: {<br>              &quot;type&quot;: &quot;string&quot;<br>            },<br>            &quot;noCache&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;<br>            },<br>            &quot;explain&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;<br>            },<br>            &quot;maxConcurrency&quot;: {<br>              &quot;type&quot;: &quot;integer&quot;<br>            },<br>            &quot;withDoc&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;<br>            },<br>            &quot;interactiveMode&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;<br>            },<br>            &quot;customHeaders&quot;: {<br>              &quot;items&quot;: {<br>                &quot;type&quot;: &quot;string&quot;<br>              },<br>              &quot;type&quot;: &quot;array&quot;<br>            },<br>            &quot;withStats&quot;: {<br>              &quot;type&quot;: &quot;boolean&quot;<br>            }<br>          },<br>          &quot;type&quot;: &quot;object&quot;<br>        },<br>        &quot;name&quot;: &quot;analyze&quot;<br>      },<br>      {<br>        &quot;description&quot;: &quot;Get Kubernetes cluster information&quot;,<br>        &quot;inputSchema&quot;: {<br>          &quot;$schema&quot;: &quot;https://json-schema.org/draft/2020-12/schema&quot;,<br>          &quot;properties&quot;: {},<br>          &quot;type&quot;: &quot;object&quot;<br>        },<br>        &quot;name&quot;: &quot;cluster-info&quot;<br>      },<br>      {<br>        &quot;description&quot;: &quot;Configure K8sGPT settings&quot;,<br>        &quot;inputSchema&quot;: {<br>          &quot;$schema&quot;: &quot;https://json-schema.org/draft/2020-12/schema&quot;,<br>          &quot;properties&quot;: {<br>            &quot;customAnalyzers&quot;: {<br>              &quot;items&quot;: {<br>                &quot;properties&quot;: {<br>                  &quot;name&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;<br>                  },<br>                  &quot;connection&quot;: {<br>                    &quot;properties&quot;: {<br>                      &quot;url&quot;: {<br>                        &quot;type&quot;: &quot;string&quot;<br>                      },<br>                      &quot;port&quot;: {<br>                        &quot;type&quot;: &quot;integer&quot;<br>                      }<br>                    },<br>                    &quot;type&quot;: &quot;object&quot;<br>                  }<br>                },<br>                &quot;type&quot;: &quot;object&quot;<br>              },<br>              &quot;type&quot;: &quot;array&quot;<br>            },<br>            &quot;cache&quot;: {<br>              &quot;properties&quot;: {<br>                &quot;type&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;bucketName&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;region&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;endpoint&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;insecure&quot;: {<br>                  &quot;type&quot;: &quot;boolean&quot;<br>                },<br>                &quot;storageAccount&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;containerName&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                },<br>                &quot;projectId&quot;: {<br>                  &quot;type&quot;: &quot;string&quot;<br>                }<br>              },<br>              &quot;type&quot;: &quot;object&quot;<br>            }<br>          },<br>          &quot;type&quot;: &quot;object&quot;<br>        },<br>        &quot;name&quot;: &quot;config&quot;<br>      }<br>    ]<br>  }<br>}</pre><p>4. Use curl to see if there are problems with any Pod in default namespace.</p><p>&gt; <em>Note that I am not using LLM queries here since `explain: false`. You can set it to “true” with </em><a href="https://github.com/k8sgpt-ai/k8sgpt?tab=readme-ov-file#llm-ai-backends"><em>proper LLM backed configured</em></a><em> and get LLM analyzed results.</em></p><pre><br># Example with no LLM configuration i.e just analyze <br>$ curl -s -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 3,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;namespace&quot;: &quot;default&quot;,<br>        &quot;labelSelector&quot;: &quot;&quot;, &quot;filters&quot;: [&quot;Pod&quot;],<br>        &quot;maxConcurrency&quot;: 1<br>      }<br>    }<br>  }&#39;| jq -c -r &#39;.result.content[0].text&#39;<br>{<br>  &quot;provider&quot;: &quot;localai&quot;,<br>  &quot;errors&quot;: null,<br>  &quot;status&quot;: &quot;ProblemDetected&quot;,<br>  &quot;problems&quot;: 1,<br>  &quot;results&quot;: [<br>    {<br>      &quot;kind&quot;: &quot;Pod&quot;,<br>      &quot;name&quot;: &quot;default/bad-nginx&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;Back-off pulling image \&quot;nginx:bad-nginx\&quot;: ErrImagePull: failed to pull and unpack image \&quot;docker.io/library/nginx:bad-nginx\&quot;: failed to resolve reference \&quot;docker.io/library/nginx:bad-nginx\&quot;: failed to do request: Head \&quot;https://registry-1.docker.io/v2/library/nginx/manifests/bad-nginx\&quot;: proxyconnect tcp: dial tcp: lookup devproxy.bloomberg.com on 192.168.65.254:53: no such host&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    }<br>  ]<br>}<br><br><br># If you need LLM analysis you would need below curl parameters after you configure LLM backend<br>$ curl -s -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 3,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;namespace&quot;: &quot;default&quot;,<br>        &quot;backend&quot;: &quot;openai&quot;,<br>        &quot;language&quot;: &quot;english&quot;,<br>        &quot;labelSelector&quot;: &quot;&quot;, &quot;filters&quot;: [&quot;Pod&quot;],<br>        &quot;explain&quot;: true, <br>        &quot;maxConcurrency&quot;: 1<br>      }<br>    }<br>  }&#39;  </pre><p>5. Use curl to check for cluster wide problems</p><pre>$ n_proxy curl -s -X POST http://localhost:8089/mcp \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;jsonrpc&quot;: &quot;2.0&quot;,<br>    &quot;id&quot;: 3,<br>    &quot;method&quot;: &quot;tools/call&quot;,<br>    &quot;params&quot;: {<br>      &quot;name&quot;: &quot;analyze&quot;,<br>      &quot;arguments&quot;: {<br>        &quot;namespace&quot;: &quot;kube-system&quot;,<br>        &quot;backend&quot;: &quot;openai&quot;,<br>        &quot;language&quot;: &quot;english&quot;,<br>        &quot;labelSelector&quot;: &quot;&quot;, &quot;filters&quot;: [&quot;ConfigMap&quot;],<br>        &quot;explain&quot;: false,<br>        &quot;maxConcurrency&quot;: 1<br>      }<br>    }<br>  }&#39; | jq -c -r &#39;.result.content[0].text&#39;<br><br>{<br>  &quot;provider&quot;: &quot;openai&quot;,<br>  &quot;errors&quot;: null,<br>  &quot;status&quot;: &quot;ProblemDetected&quot;,<br>  &quot;problems&quot;: 5,<br>  &quot;results&quot;: [<br>    {<br>      &quot;kind&quot;: &quot;ConfigMap&quot;,<br>      &quot;name&quot;: &quot;kube-system/extension-apiserver-authentication&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;ConfigMap extension-apiserver-authentication is not used by any pods in the namespace&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    },<br>    {<br>      &quot;kind&quot;: &quot;ConfigMap&quot;,<br>      &quot;name&quot;: &quot;kube-system/kube-apiserver-legacy-service-account-token-tracking&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;ConfigMap kube-apiserver-legacy-service-account-token-tracking is not used by any pods in the namespace&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    },<br>    {<br>      &quot;kind&quot;: &quot;ConfigMap&quot;,<br>      &quot;name&quot;: &quot;kube-system/kube-root-ca.crt&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;ConfigMap kube-root-ca.crt is not used by any pods in the namespace&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    },<br>    {<br>      &quot;kind&quot;: &quot;ConfigMap&quot;,<br>      &quot;name&quot;: &quot;kube-system/kubeadm-config&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;ConfigMap kubeadm-config is not used by any pods in the namespace&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    },<br>    {<br>      &quot;kind&quot;: &quot;ConfigMap&quot;,<br>      &quot;name&quot;: &quot;kube-system/kubelet-config&quot;,<br>      &quot;error&quot;: [<br>        {<br>          &quot;Text&quot;: &quot;ConfigMap kubelet-config is not used by any pods in the namespace&quot;,<br>          &quot;KubernetesDoc&quot;: &quot;&quot;,<br>          &quot;Sensitive&quot;: []<br>        }<br>      ],<br>      &quot;details&quot;: &quot;&quot;,<br>      &quot;parentObject&quot;: &quot;&quot;<br>    }<br>  ]<br>}</pre><h3><strong>Test using MCP Inspector.</strong></h3><p>Data Flow</p><pre>Mcp_Inspector_UI----&gt;K8sGPT_MCP_Server</pre><p>The <a href="https://github.com/modelcontextprotocol/inspector">MCP Inspector</a> is an interactive developer tool for testing and debugging MCP servers. Install it using <a href="https://modelcontextprotocol.io/docs/tools/inspector#getting-started">these</a> instructions.</p><ol><li>Launching the inspector will launch a web browser on your local machine with access to inspector UI</li><li>Configure the inspector with your mcp http endpoint (see screenshot below) and click Connect</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNFLxPCphtmQeQ4lJty4UA.png" /></figure><p>3. If everything is working , you will see list of Tools under Tools, Run the tool and you should see the results. See screenshot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nq0nJSEYMSQWyPtr12beUg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20549903a876" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI in Action: Fixing Kubernetes with Go and MCP]]></title>
            <link>https://medium.com/@umeshkaul_39077/building-an-ai-powered-kubernetes-troubleshooting-assistant-with-mcp-89f786682be9?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/89f786682be9</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[k8sgpt]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Wed, 04 Jun 2025 02:34:19 GMT</pubDate>
            <atom:updated>2025-07-23T09:47:16.931Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Building an AI-Powered Kubernetes Troubleshooting Assistant with MCP and Golang</em></p><p>This is the continuation of the series about using AI for troubleshooting Kubernetes clusters, part-1 of which you can find <a href="https://medium.com/@umeshkaul_39077/using-ai-for-kubernetes-troubleshooting-ea83fde2c1fd">here</a></p><blockquote><em>NOTE: See this </em><a href="https://github.com/umeshkaul/using-ai-mcp-tools-for-troubleshooting-kubernetes"><em>repository</em></a><em> for source code and step-by-step process on how to try it out against your Kubernetes cluster.</em></blockquote><p>This <a href="https://github.com/umeshkaul/using-ai-mcp-tools-for-troubleshooting-kubernetes">project</a> showcases the development of an AI-powered Kubernetes troubleshooting assistant using the Model-Context-Protocol (MCP) protocol. Implemented in Go using <a href="https://github.com/mark3labs/mcp-go">mark3labs/mcp-go</a> SDK, the prototype integrates large language models (LLMs) with standard Kubernetes tools like kubectl and k8sgpt to deliver a natural language, conversational interface for managing and troubleshooting clusters.</p><h3>System Architecture</h3><h4>MCP Protocol Overview</h4><p>The Model Context Protocol (MCP) is an open standard that lets language models securely interact with external tools and data via a unified JSON-RPC interface. It standardizes how models discover, call, and receive responses from APIs — like a USB-C port for AI. MCP makes it easier to build modular, interoperable AI systems without custom integrations.</p><h4>System Components</h4><p>In this project, MCP is used to:</p><ul><li>Expose kubectl and k8sgpt as callable tools</li><li>Handle tool discovery and execution</li><li>Manage communication between the AI client/agent and Kubernetes tools using SSE. SSE enables lightweight, real-time streaming from server to client — ideal for delivering tool outputs and logs in MCP-based AI workflows.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l225O6IxQ5zW5RSCe6V_fw.png" /></figure><p>Data Flow :</p><pre>1. User Input → MCP Client (natural language query)<br>2. MCP Client → OpenAI LLM (query interpretation)<br>3. OpenAI LLM → MCP Client (tool selection and arguments)<br>4. MCP Client → MCP Server (tool execution request)<br>5. MCP Server → MCP Tools (kubectl/k8sgpt execution)<br>6. MCP Tools → Kubernetes Cluster (command execution)<br>7. Results flow back up through the same path<br>8. MCP Client → User (formatted response)</pre><p>Example Flow:</p><pre>1. User: &quot;check to see if there are problems with any pods&quot;<br>2. LLM interprets and selects k8sgpt analyze<br>3. MCP Server executes k8sgpt analyze<br>4. Results flow back to user<br>5. If issues found, LLM selects kubectl commands<br>6. MCP Server executes kubectl commands<br>7. Results flow back to user</pre><h3>Code Overview</h3><p>The <a href="https://github.com/umeshkaul/using-ai-mcp-tools-for-troubleshooting-kubernetes">project</a> consists of two main components:</p><h4>Server Component</h4><pre>// Create MCP server with basic capabilities<br>mcpServer := server.NewMCPServer(<br>    &quot;kubernetes-troubleshooter&quot;,<br>    &quot;1.0.0&quot;,<br>)<br><br>// Create and add the k8sgpt tool<br>k8sGptTool := mcp.NewTool(<br>    &quot;k8sgpt&quot;,<br>    mcp.WithDescription(<br>        &quot;Execute &#39;k8sgpt&#39; command to interact with a Kubernetes cluster...&quot;,<br>    ),<br>)<br><br>// Create and add the kubectl tool<br>kubectlTool := mcp.NewTool(<br>    &quot;kubectl&quot;,<br>    mcp.WithDescription(<br>        &quot;Use &#39;kubectl&#39; command to check if there are any issues...&quot;,<br>    ),<br>)</pre><h4>Client Component</h4><pre>// Create a new MCP client using HTTP transport<br>mcpClient, err := client.NewSSEMCPClient(MCPEndpoint)<br>if err != nil {<br>    log.Fatalf(&quot;failed to create MCP client: %v&quot;, err)<br>}<br><br>// Initialize OpenAI client<br>client := openai.NewClient()</pre><h3>Example: Fixing an Image Pull Error</h3><p>When a user asks to check for pod issues:</p><pre>Enter your question about the Kubernetes cluster (type &#39;quit&#39; to exit):<br>check to see if there are problems with any pods running in my cluster and fix them, wait for 10 seconds and confirm that the problem is resolved before completing the task</pre><p>The system automatically:</p><ol><li>Deletes the problematic pod</li><li>Updates the deployment with the correct image</li><li>Monitors the rollout</li><li>Verifies the fix</li></ol><pre><br>&gt; check to see if there are problems with any pods running in my cluster and fix them, wait for 10 seconds and confirm that the problem is resolved before completing the task<br>checking for tool calls<br>Iteration 1 - Tool call: k8sgpt<br>Tool call arguments: {&quot;arguments&quot;:&quot;analyze&quot;}<br>Tool result content: [{{&lt;nil&gt;} text AI Provider: AI not used; --explain not set<br><br>0: Pod default/nginx-5545cbc86d-n2xgh(Deployment/nginx)<br>- Error: failed to pull and unpack image &quot;docker.io/library/nginx007:latest&quot;: failed to resolve reference &quot;docker.io/library/nginx007:latest&quot;: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed<br><br>1: Service default/nginx()<br>- Error: Service has not ready endpoints, pods: [Pod/nginx-5545cbc86d-n2xgh], expected 1<br>}]<br>checking for tool calls<br>Iteration 2 - Tool call: kubectl<br>Tool call arguments: {&quot;arguments&quot;: &quot;set image deployment/nginx nginx=docker.io/library/nginx:latest&quot;}<br>Tool result content: [{{&lt;nil&gt;} text deployment.apps/nginx image updated<br>}]<br>Iteration 2 - Tool call: kubectl<br>Tool call arguments: {&quot;arguments&quot;: &quot;rollout status deployment/nginx&quot;}<br>Tool result content: [{{&lt;nil&gt;} text Waiting for deployment &quot;nginx&quot; rollout to finish: 1 old replicas are pending termination...<br>Waiting for deployment &quot;nginx&quot; rollout to finish: 1 old replicas are pending termination...<br>Waiting for deployment &quot;nginx&quot; rollout to finish: 1 old replicas are pending termination...<br>deployment &quot;nginx&quot; successfully rolled out<br>}]<br><br>Got LLM response: The issues with the pods in your cluster have been addressed. The following fixes were implemented:<br><br>- The image for the NGINX deployment was updated to a valid reference.<br><br>After waiting, the deployment has successfully rolled out.<br>Task completed after 2 iterations</pre><h3>Try it yourself</h3><p>Feel free to clone this <a href="https://github.com/umeshkaul/using-ai-mcp-tools-for-troubleshooting-kubernetes?tab=readme-ov-file#4-using-the-mcp-kubernetes-troubleshooter">repository</a> and try it out yourself and let me know what you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89f786682be9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AI for Kubernetes Troubleshooting]]></title>
            <link>https://medium.com/@umeshkaul_39077/using-ai-for-kubernetes-troubleshooting-ea83fde2c1fd?source=rss-64e3f5728d09------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea83fde2c1fd</guid>
            <dc:creator><![CDATA[Umesh Kaul]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 20:58:48 GMT</pubDate>
            <atom:updated>2025-04-25T02:00:25.662Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Summary</strong></h3><p>Artificial intelligence continues to transform our lives. This post explores how we can use it to help us troubleshoot problems in a Kubernetes cluster.</p><blockquote>NOTE: See this <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/tree/main">repository</a> for step-by-step process and manifests referred below</blockquote><p><strong>AI Needs Access to Your Cluster</strong></p><p>If you ask ChatGPT questions about Kubernetes, it surely knows about it. But in order for it to help you troubleshoot issues in a Kubernetes cluster, it must have control over it.</p><p>Basically, an LLM (Large Language Model) should be able to mimic what a typical Kubernetes administrator would do — i.e., run commands, look at the output, and then decide the next step until the problem is solved.</p><p><strong>Tools / Function Calling</strong></p><p>This is exactly what the <strong>function/tool calling</strong> feature is designed to address. You can read more about it <a href="https://platform.openai.com/docs/guides/function-calling?api-mode=chat#overview">here</a>. Here is an <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/tree/main/python-function-calling-example">example</a> of how to use this feature in python.</p><p>This feature is used by AI agents to accomplish complex, multi-step operations, for example:</p><p><em>Example interaction with an AI Agent with access to run commands against your cluster:</em></p><blockquote><em>You say:<br> “A pod in my Kubernetes cluster isn’t starting.”</em></blockquote><blockquote><em>The AI agent uses function calling to run:<br> </em><em>kubectl get pods -A</em></blockquote><blockquote><em>It sees the pod status is </em><em>ErrImagePull, then runs:<br> </em><em>kubectl describe pod</em></blockquote><blockquote><em>It finds the reason:<br> </em><em>&quot;Failed to pull image &#39;myregistry/payments-api:latest&#39; – image not found.&quot;</em></blockquote><blockquote><em>It replies:<br> </em>“The pod can’t start because the image myregistry/payments-api:latest doesn’t exist or is inaccessible—check the image name or your registry credentials.”</blockquote><blockquote><em>The AI agent can now also </em><strong><em>fix the problem</em></strong><em> if you ask it to.</em></blockquote><p>In this <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/tree/main">repository</a>, we will use function/tool calling with the following tools to troubleshoot three problem scenarios in a local <a href="https://kind.sigs.k8s.io/">kind</a> cluster:</p><ul><li><a href="https://github.com/gptscript-ai/gptscript"><strong>GPTScript</strong></a> — for OpenAI’s function calling. Think of this as a cli based AI agent in this case which enables users to combine natural language prompts with traditional scripting elements like Bash, Python, and more.</li><li><a href="https://k8sgpt.ai/"><strong>K8sGPT</strong></a> is an AI-powered tool designed to scan your Kubernetes clusters and identify issues, translating them into easy-to-understand explanations.</li><li><strong>kubectl</strong> — for interacting with the cluster</li></ul><p><strong>Problem Scenarios</strong></p><ol><li><strong>mis-configured nginx image tag : </strong>In this scenario, we will apply <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/blob/main/misconfigured-nginx-image-scenario/misconfigured-nginx-image.yaml">this deployment</a>, which has a wrong label resulting in the pod not being able to start.</li><li><strong>mis-configured root volume for nginx : </strong>In this scenario, we will apply <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/blob/main/misconfigured-nginx-ro-volume-scenario/misconfigured-nginx-ro-root-volume.yaml">this manifest</a>, which has an nginx deployment with a misconfigured root volume, preventing the pod from starting.</li><li><strong>mis-configured service label : </strong>In this scenario, we will apply <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/blob/main/misconfigured-nginx-service-label/misconfigured-label.yaml">this manifest</a>, which has an nginx deployment with a misconfigured service label. This results in the service having no endpoints.</li></ol><p><strong>System Overview</strong></p><p>Below is a schematic that shows the <strong>data flow between different components</strong> of the system we’ll use to troubleshoot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHnEQr16rWI2ISlkGC3VQQ.png" /><figcaption>Data flow between gptscript , k8sgpt , kubectl and LLM</figcaption></figure><p><strong>Flow Description:</strong></p><ol><li>The user provides a natural language instruction to gptscript CLI tool asking it to solve the problem. The LLM is told that it can run kubectl and k8sgpt commands if needed. (Note: The LLM cannot run commands directly but can ask gptscript to run them using OpenAI’s function calling.)</li><li>gptscript (runs on the user’s system) sends the input to the LLM using function/tool calling.</li><li>The LLM returns structured instructions (e.g., “run kubectl get pods”).</li><li>gptscript executes those commands (like kubectl or k8sgpt) in the terminal.</li><li>Results can optionally be sent back to the LLM for follow-up actions.</li></ol><blockquote><strong>NOTE </strong>:If you’ve been following the AI space, you may have heard of <a href="https://www.anthropic.com/news/model-context-protocol">Anthropic’s MCP</a> (Model Context Protocol) and the broader concept of AI agents. While we won’t be using MCP specifically, the core idea is the same: leveraging the tool-calling capabilities of large language models (LLMs) to solve complex problems through intelligent, multi-step reasoning. Anthropic’s MCP is a protocol that standardizes how AI, tools, and agents communicate, making it easier for them to work together. For example, if an AI agent needs to diagnose a server issue, it can use MCP to ask a monitoring tool for logs, analyze them using a language model, and respond with a clear solution — all in a smooth, coordinated flow. k8sGpt is also working on adding <a href="https://k8sgpt.ai/roadmap">MCP support</a> which when available will remove the need for gptscript used in this case</blockquote><h3>Try it yourself</h3><p>Feel free to clone this <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/tree/main">repository</a> and try it out yourself and let me know what you think.</p><p>One thing which is very clear is that LLMs are inherently<strong> non-deterministic </strong>which means clear and <strong>precise prompts</strong> are critical to getting the desired result, especially when specifying what <strong>NOT</strong> to do (e.g. avoid changing security policies in manifests). To gain better insight into LLM behavior specifically in the context of troubleshooting Kubernetes issues, I recommend walking through the examples below — especially <a href="https://github.com/umeshkaul/troubleshooting-kubernetes-using-ai/tree/main?tab=readme-ov-file#scenario-2-troubleshooting-mis-configured-root-volume">Troubleshooting misconfigured root volume</a>.</p><h3>Conclusion</h3><ul><li>AI tools like <em>k8sgpt</em> can accurately diagnose common Kubernetes issues and provide helpful explanations. The approach reduces manual debugging effort but still requires human oversight for complex or novel issues.</li><li>Integrating LLM function/tool calling (using <em>gptscript</em> for now but MCP in future) enables automation of troubleshooting and fixes via <em>kubectl</em>.</li><li>This AI-driven workflow is effective in local/test clusters for quick issue resolution but I would not call it ready for production use yet.</li><li>Non-determinism is a fundamental characteristic of modern artificial intelligence i.e. AI systems often yield different results even when presented with identical input. This is challenge since you are never sure what approach it will take to troubleshoot the problem and can be partially overcome with better <strong>prompts</strong></li></ul><p>I hope you will find this useful and try different Kubernetes troubleshooting scenarios and see how AI deals with it. I would love to hear about that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea83fde2c1fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>