<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Param Dave on Medium]]></title>
        <description><![CDATA[Stories by Param Dave on Medium]]></description>
        <link>https://medium.com/@paramdave890?source=rss-913dead08eff------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZaamPy4k2x1s8V8olWYMtA.jpeg</url>
            <title>Stories by Param Dave on Medium</title>
            <link>https://medium.com/@paramdave890?source=rss-913dead08eff------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:05:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@paramdave890/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Rancher 2.13: Why Your Imported Cluster Stays Disconnected (and the One-Line Fix)]]></title>
            <link>https://medium.com/@paramdave890/rancher-2-13-why-your-imported-cluster-stays-disconnected-and-the-one-line-fix-e698e6ffae97?source=rss-913dead08eff------2</link>
            <guid isPermaLink="false">https://medium.com/p/e698e6ffae97</guid>
            <category><![CDATA[rancher-clusters]]></category>
            <category><![CDATA[rancher]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Param Dave]]></dc:creator>
            <pubDate>Mon, 01 Jun 2026 17:44:06 GMT</pubDate>
            <atom:updated>2026-06-01T17:44:40.131Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A three-hour debugging story that ends with a single missing environment variable Rancher v2.13 forgot to put in its own import manifest — and a stuck cluster that came alive the moment we added it back.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWZfHfn1XLNAoB2i7pwGDA.png" /></figure><h3>TL;DR</h3><p>If you import a fresh EKS (or any Kubernetes) cluster into a Rancher v2.13 management server and the cluster sits forever as “Cluster agent is not connected” — even though:</p><ul><li>cattle-cluster-agent pod is Running 1/1</li><li>Logs say Connected to proxy at wss://rancher.example.com/v3/connect/register</li><li>The WebSocket is actively exchanging keepalive traffic</li><li>Rancher’s pod logs show Handling backend connection request [c-xxxxx]</li><li><em>Other</em> clusters on the same Rancher are perfectly happy</li></ul><p>…you are almost certainly hitting <a href="https://github.com/rancher/rancher/issues/39911">rancher/rancher#39911</a>. The fix is one line: add CATTLE_FEATURES to the agent deployment.</p><h3>The Scenario</h3><p>Imagine you run a Rancher v2.13 management cluster and use it to govern ~10 downstream Kubernetes clusters across a couple of cloud accounts. Connectivity is set up via VPC peering, security-group ingress on port 443, the works. The same recipe has happily onboarded clusters for the last 6 months.</p><p>Today you onboard one more cluster — let’s call it team-eks-prod-new — using the bog-standard Rancher import flow:</p><ol><li>In the Rancher UI: Cluster Management → Import Existing Cluster → Generic.</li><li>Give it a name.</li><li>Copy the kubectl apply -f https://rancher.example.com/v3/import/&lt;token&gt;_c-xxxxx.yaml command.</li><li>Run it on the new cluster.</li><li>Wait for the UI to flip the cluster to Active.</li></ol><p>Everything from step 1 to step 4 works exactly as expected. The manifest deploys cleanly, the agent pod comes up. And then…</p><pre>Cluster: team-eks-prod-new<br>State:   Unavailable<br>Message: Cluster agent is not connectedy</pre><p>It just sits there. For <em>hours</em>.</p><p>You restart the agent. Nothing. You delete and re-import the cluster. Nothing. You apply the manifest with a fresh registration token. Nothing.</p><p>Every other cluster on this same Rancher install — including ones in the <em>same downstream account</em> with the <em>same network topology</em> — works fine. Only the new one is stuck. Welcome to the rabbit hole.</p><h3>What the Symptoms Actually Look Like</h3><p>This is what you see on the downstream cluster (team-eks-prod-new):</p><pre>$ kubectl -n cattle-system get pods<br>NAME                                    READY   STATUS    RESTARTS   AGE<br>cattle-cluster-agent-69bc8ddc54-tslq7   1/1     Running   0          29m</pre><p>Pod is healthy. Agent log says:</p><pre>INFO: https://rancher.example.com/ping is accessible<br>INFO: rancher.example.com resolves to 10.x.x.x<br>time=&quot;...&quot; level=info msg=&quot;Rancher agent version v2.13.0 is starting&quot;<br>time=&quot;...&quot; level=info msg=&quot;Connecting to wss://rancher.example.com/v3/connect/register<br>                          with token starting with abc123xyz...&quot;<br>time=&quot;...&quot; level=info msg=&quot;Connecting to proxy&quot; url=&quot;wss://...&quot;<br>time=&quot;...&quot; level=info msg=&quot;Connected to proxy&quot; url=&quot;wss://...&quot;</pre><p>And then… silence. Forever. No errors, no further messages. The connection is just sitting there.</p><p>On the Rancher management cluster side:</p><pre>$ kubectl -n cattle-system logs -l app=rancher | grep c-xxxxx<br>... [INFO] Handling backend connection request [c-xxxxx]<br>... [INFO] Starting cluster controllers for c-xxxxx<br>... [INFO] Starting cluster agent for c-xxxxx [owner=true]<br>... [ERROR] error syncing &#39;c-xxxxx&#39;: handler cluster-deploy:<br>            cannot connect to the cluster&#39;s Kubernetes API, requeuing<br>... [ERROR] error syncing &#39;c-xxxxx&#39;: handler cluster-deploy:<br>            cannot connect to the cluster&#39;s Kubernetes API, requeuing<br>... [ERROR] error syncing &#39;c-xxxxx&#39;: handler cluster-deploy:<br>            cannot connect to the cluster&#39;s Kubernetes API, requeuing</pre><p>That cluster-deploy: cannot connect to the cluster&#39;s Kubernetes API is the smoking gun, but it&#39;s also the world&#39;s least helpful error message. Rancher <em>received the WebSocket</em>, accepted it, opened cluster controllers… and then can&#39;t talk to the downstream API. Through what? Why? It&#39;s not telling.</p><p>The cluster’s management.cattle.io/v3 status shows:</p><pre>conditions:<br>  - type: Connected<br>    status: &quot;False&quot;<br>  - type: AgentDeployed<br>    status: &quot;True&quot;<br>  - type: Ready<br>    status: &quot;False&quot;<br>    reason: Disconnected<br>    message: Cluster agent is not connected</pre><p>That last condition — Disconnected: Cluster agent is not connected — is what the UI is reading from. And it&#39;s persistently False even though the WebSocket is demonstrably alive.</p><h3>Dead Ends</h3><p>Before I get to the actual fix, let me list everything I tried that did not help, so you don’t waste your time on them:</p><h3>1. “Just restart the agent”</h3><pre>kubectl -n cattle-system rollout restart deploy/cattle-cluster-agent</pre><p>Result: new pod, brand-new TCP connection, brand-new Connected to proxy log line, same wedge.</p><h3>2. Delete and re-import the cluster in Rancher</h3><p>Got a fresh c-yyyyy cluster ID with a fresh registration token. Applied it. The cattle-credentials-XXXXXXXX secret had a brand new hash suffix (cattle-credentials-aaaaaaaaaa vs the previous cattle-credentials-bbbbbbbbbb), confirming a <em>fresh</em> token had really been issued by Rancher. Same wedge.</p><h3>3. Run the official rancher-cleanup job on the downstream</h3><p><a href="https://github.com/rancher/rancher-cleanup">rancher/rancher-cleanup</a> does a thorough job of removing Rancher-installed CRDs, ClusterRoles, namespaces, secrets — everything. After the cleanup completed, I re-imported into a totally clean cluster. Same wedge.</p><h3>4. Bounce a Rancher mgmt pod to clear in-memory tunnel state</h3><p>Rancher caches per-cluster session state in memory. Killing the pod that currently holds the leader lease forces a re-election and rebuilds state. I tried this, watched the new leader pick up c-xxxxx, and… same wedge.</p><h3>5. Stare at the network path</h3><p>The conventional wisdom for “Connected to proxy, then nothing” is some middlebox in the WebSocket path. I checked everything:</p><ul><li>VPC peering routes: present and correct in both directions</li><li>Security groups: ALB allows ingress 443 from the downstream pod CIDR</li><li>NACLs: open both ways</li><li>curl https://rancher.example.com/ping from inside the agent pod → 200 OK, 23 ms</li><li>ss -tnpi inside the agent pod showed the WebSocket connection actively exchanging traffic: 700 KB sent, 150 KB received, keepalives flowing, lastsnd: 90ms</li></ul><p>The network was fine. The data plane was fine. <em>Something else</em> was broken.</p><h3>The Real Diagnostic</h3><p>I started comparing my broken cluster to a working cluster. Looked at the management.cattle.io/v3 CRs side by side. Identical specs. Identical agentFeatures. Identical appliedAgentEnvVars. Identical conditions… except for the obvious Connected=False vs Connected=True.</p><p>Then I had a hunch and ran this on the Rancher mgmt pod logs:</p><pre>$ kubectl -n cattle-system logs -l app=rancher --tail=50000 \<br>  | grep -oE &#39;stv-cluster-c-[a-z0-9]+&#39; | sort -u<br>stv-cluster-c-aaa1a<br>stv-cluster-c-aaa2b<br>stv-cluster-c-aaa3c<br>stv-cluster-c-aaa4d<br>... (all working clusters)</pre><p>stv-cluster-c-xxxxx was missing.</p><p>Rancher v2.13 actually opens two WebSocket channels for each managed cluster:</p><ol><li>c-&lt;id&gt; — the legacy norman API tunnel (this is what shows up first; it&#39;s the one we&#39;ve been watching)</li><li>stv-cluster-c-&lt;id&gt; — the Steve v3 aggregation tunnel (the modern one)</li></ol><p>The Connected=True cluster condition only flips when <em>both</em> are healthy. The cluster-deploy handler that posts cluster manifests to your downstream uses the Steve path, not the legacy one. Without stv-cluster-c-xxxxx, every cluster-deploy reconcile attempt fails with that lovely opaque cannot connect to the cluster&#39;s Kubernetes API error.</p><p>So the question morphed from <em>“why isn’t the cluster connecting?”</em> to <em>“why isn’t the agent opening the second WebSocket channel?”</em></p><h3>The Root Cause</h3><p>Time to Google. Searching for the exact symptoms turned up <a href="https://github.com/rancher/rancher/issues/39911">rancher/rancher#39911 — “Re-registering Imported Cluster Does Not Succeed”</a>, which describes the same situation almost word-for-word.</p><p>The fix buried in the thread:</p><blockquote><em>I fixed by adding the </em><em>CATTLE_FEATURES env to the Rancher Agent deployment:</em></blockquote><blockquote><em>CATTLE_FEATURES=embedded-cluster-api=false,fleet=false,monitoringv1=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningv2=false,rke2=false</em></blockquote><p>The Rancher v2.13 import manifest template does not include the CATTLE_FEATURES env variable in the agent Deployment. Without it, the agent doesn&#39;t know that the multi-cluster-management-agent feature is on, and so it never opens the second (Steve aggregation) WebSocket channel.</p><p>Here is the actual env block from the broken agent deployment:</p><pre>env:<br>  - name: CATTLE_SERVER<br>    value: https://rancher.example.com<br>  - name: CATTLE_CA_CHECKSUM<br>  - name: CATTLE_CLUSTER<br>    value: &quot;true&quot;<br>  - name: CATTLE_K8S_MANAGED<br>    value: &quot;true&quot;<br>  - name: CATTLE_CLUSTER_REGISTRY<br>  - name: CATTLE_CREDENTIAL_NAME<br>    value: cattle-credentials-aaaaaaaaaa<br>  - name: CATTLE_SUC_APP_NAME_OVERRIDE<br>  - name: CATTLE_SERVER_VERSION<br>    value: v2.13.0<br>  - name: CATTLE_INSTALL_UUID<br>    value: 00000000-0000-0000-0000-000000000000<br>  - name: CATTLE_INGRESS_IP_DOMAIN<br>    value: sslip.io<br>  - name: STRICT_VERIFY<br>    value: &quot;false&quot;<br># notice: NO CATTLE_FEATURES</pre><p>And on a working cluster that was imported on an older Rancher version (which Rancher then upgraded in place to v2.13):</p><pre>env:<br>  - name: CATTLE_FEATURES<br>    value: &quot;embedded-cluster-api=false,fleet=false,managed-system-upgrade-controller=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningprebootstrap=false,provisioningv2=false,rke2=false,turtles=false,ui-sql-cache=true&quot;<br>  - name: CATTLE_SERVER<br>    value: https://rancher.example.com<br>  ... (everything else same)</pre><p>The working cluster has CATTLE_FEATURES. It was inherited from the older import manifest and carried through the Rancher upgrade. Any <em>new</em> cluster imported on v2.13 misses it entirely.</p><p>That’s why this bug only bites new imports, not existing clusters. Existing clusters were imported before the regression. They just keep working until you re-import them.</p><h3>The Fix</h3><p>Patch the cattle-cluster-agent deployment on the broken downstream cluster to add the missing env var. Use the same value Rancher reports in cluster.management.cattle.io/&lt;id&gt;.status.agentFeatures:</p><pre>kubectl -n cattle-system patch deploy cattle-cluster-agent --type=json -p=&#39;[<br>  {<br>    &quot;op&quot;: &quot;add&quot;,<br>    &quot;path&quot;: &quot;/spec/template/spec/containers/0/env/-&quot;,<br>    &quot;value&quot;: {<br>      &quot;name&quot;: &quot;CATTLE_FEATURES&quot;,<br>      &quot;value&quot;: &quot;embedded-cluster-api=false,fleet=false,managed-system-upgrade-controller=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningprebootstrap=false,provisioningv2=false,rke2=false,turtles=false,ui-sql-cache=true&quot;<br>    }<br>  }<br>]&#39;</pre><p>Wait ~30 seconds for the rollout. On the management cluster, you should immediately see:</p><pre>[INFO] Handling backend connection request [c-xxxxx]<br>[INFO] Handling backend connection request [stv-cluster-c-xxxxx]   ← the missing one!</pre><p>And the cluster condition flips:</p><pre>$ kubectl get cluster.management.cattle.io c-xxxxx \<br>    -o jsonpath=&#39;Connected={.status.conditions[?(@.type==&quot;Connected&quot;)].status} Ready={.status.conditions[?(@.type==&quot;Ready&quot;)].status}&#39;<br>Connected=True Ready=True</pre><p>That’s it.</p><h3>Useful Commands</h3><p>A cheat-sheet of the diagnostics that actually told me something useful:</p><pre># 1. List which clusters have BOTH tunnels active right now<br>kubectl -n cattle-system logs -l app=rancher --tail=20000 \<br>  | grep -oE &#39;\[(stv-cluster-)?c-[a-z0-9]+\]&#39; \<br>  | sort | uniq -c | sort -rn<br><br># 2. Compare a broken cluster&#39;s conditions to a healthy one&#39;s<br>kubectl get cluster.management.cattle.io &lt;broken&gt; -o jsonpath=&#39;{range .status.conditions[*]}{.type}={.status}{&quot;\n&quot;}{end}&#39; | sort<br>kubectl get cluster.management.cattle.io &lt;healthy&gt; -o jsonpath=&#39;{range .status.conditions[*]}{.type}={.status}{&quot;\n&quot;}{end}&#39; | sort<br><br># 3. Check the agent deployment&#39;s env vars<br>kubectl -n cattle-system get deploy cattle-cluster-agent \<br>  -o jsonpath=&#39;{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{&quot;\n&quot;}{end}&#39;<br><br># 4. Confirm the WS connection from the pod&#39;s POV<br>AGENT=$(kubectl -n cattle-system get pod -l app=cattle-cluster-agent -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br>kubectl -n cattle-system exec $AGENT -c cluster-register -- ss -tnpi \<br>  | grep -A 1 &#39;&lt;rancher_ALB_IP&gt;&#39;</pre><p>If Connected is False and you can&#39;t see a stv-cluster-c-&lt;id&gt; line in the Rancher pod logs anywhere, you&#39;ve found this bug. Patch the env var and move on with your day.</p><h3>A Side-Effect Worth Knowing About</h3><p>While debugging this, I bounced a couple of Rancher management pods to clear in-memory state. That dropped every downstream cluster’s existing WebSocket connections. Most reconnected fine — <em>but</em> a handful of clusters whose stv-cluster-c-&lt;id&gt; channel had been alive for months also exhibited the dead-but-alive WebSocket symptom afterwards.</p><p>That is: their agent’s TCP connection was still established (you could see it active in ss -tnpi), keepalives were flowing, but Rancher had lost the corresponding session in its in-memory registry and the agent had not noticed it was effectively orphaned. The cluster CR stayed Connected=True (stale value), but anything Rancher tried to <em>do</em> with that tunnel failed.</p><p>The fix in that case was the simplest possible thing:</p><pre>kubectl -n cattle-system rollout restart deploy/cattle-cluster-agent</pre><p>That breaks the orphaned TCP socket and forces a fresh dial, after which both c-&lt;id&gt; and stv-cluster-c-&lt;id&gt; re-register cleanly. Worth knowing if you ever do a Rancher restart and notice some downstream clusters get sad afterwards.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e698e6ffae97" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>