<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by sentraorb on Medium]]></title>
        <description><![CDATA[Stories by sentraorb on Medium]]></description>
        <link>https://medium.com/@sentraorb?source=rss-89669a6506c5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by sentraorb on Medium</title>
            <link>https://medium.com/@sentraorb?source=rss-89669a6506c5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:07:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sentraorb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One Gateway to Rule All Your LLMs: Building a Production-Ready AI Stack with LiteLLM]]></title>
            <link>https://aws.plainenglish.io/one-gateway-to-rule-all-your-llms-building-a-production-ready-ai-stack-with-litellm-1ffcb29a7733?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ffcb29a7733</guid>
            <category><![CDATA[ai-engineer]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[litellm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[genrative-ai]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 10:55:37 GMT</pubDate>
            <atom:updated>2026-07-14T10:57:12.830Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>How a single OpenAI-compatible proxy can tame multi-provider chaos, control costs, and give you full observability over your AI workloads.</blockquote><p>If you’re building AI products today, you’re probably juggling more than one model provider. OpenAI for general tasks, Anthropic’s Claude for long-context reasoning, Azure OpenAI for enterprise compliance, and maybe a self-hosted vLLM cluster running Llama, DeepSeek, or Qwen to keep costs down. Toss in Ollama for local development, and suddenly your codebase is a spaghetti bowl of SDKs, API keys, retry logic, and billing dashboards.</p><p>This is exactly the problem <strong>LiteLLM Gateway</strong> solves. It sits between your applications and every LLM provider you use, exposing a single OpenAI-compatible API while handling the messy operational realities — routing, failover, rate limiting, cost tracking, and security — in one place.</p><p>In this post, I’ll walk through a reference architecture for running LiteLLM in production, layer by layer.</p><h4>The Big Picture</h4><p>At a high level, the architecture has five layers:</p><ol><li><strong>Client Applications</strong> — everything that consumes LLMs</li><li><strong>LiteLLM Gateway</strong> — the central proxy and control plane</li><li><strong>Model Providers</strong> — cloud APIs and self-hosted inference</li><li><strong>Config &amp; Secrets</strong> — how the gateway is configured securely</li><li><strong>Observability &amp; Data Stores</strong> — where metrics, logs, and state live</li></ol><p>Let’s break each one down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zFxMRtOqOFpetEH240RL0g.png" /></figure><h4>Layer 1: Client Applications</h4><p>Your consumers can be anything — a <strong>web app</strong>, a <strong>mobile client</strong>, a <strong>RAG pipeline</strong>, autonomous <strong>AI agents</strong>, or other internal <strong>APIs</strong>. The beauty of the gateway pattern is that none of them need to know (or care) which model actually serves their request.</p><p>Every client speaks one dialect: the OpenAI API format. That means you can swap gpt-4o for claude-sonnet or a self-hosted Llama model with a config change — zero code changes on the client side.</p><pre>import openai<br>client = openai.OpenAI(<br>    base_url=&quot;https://your-litellm-gateway.com&quot;,<br>    api_key=&quot;sk-your-gateway-key&quot;<br>)<br>response = client.chat.completions.create(<br>    model=&quot;claude-sonnet&quot;,  # routed by LiteLLM, not OpenAI<br>    messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Hello!&quot;}]<br>)</pre><h4>Layer 2: The LiteLLM Gateway — The Heart of the System</h4><p>This is where all the operational intelligence lives. The gateway handles nine critical concerns:</p><h4>API Authentication</h4><p>Instead of scattering provider API keys across every service, clients authenticate to the gateway with <strong>virtual keys</strong>. You can issue keys per team, per app, or per environment — each with its own permissions and budgets.</p><h4>Request Routing</h4><p>Requests are routed to the right provider based on the model name, deployment health, latency, or custom rules. Want all gpt-4 traffic to hit Azure in production but OpenAI in dev? One config entry.</p><h4>Load Balancing</h4><p>Multiple deployments of the same model (say, three Azure regions plus OpenAI) can be load-balanced automatically — round-robin, least-busy, or latency-based.</p><h4>Retry &amp; Failover</h4><p>When a provider throws a 429 or an outage hits, LiteLLM automatically retries and fails over to a healthy deployment. Your users never see the hiccup.</p><h4>Rate Limiting</h4><p>Set TPM (tokens per minute) and RPM (requests per minute) limits per key, per user, or per team — protecting both your upstream quotas and your wallet.</p><h4>Budget Enforcement</h4><p>Assign hard spending caps to teams or projects. When a budget is exhausted, requests are blocked — no more surprise five-figure bills at month end.</p><h4>Cost Tracking</h4><p>Every request is metered with token counts and dollar costs, attributed to the key that made it. Finance will love you.</p><h4>Logging</h4><p>Full request/response logging can be shipped to your existing logging stack for debugging, auditing, and compliance.</p><h4>Metrics</h4><p>Latency, error rates, token throughput, and spend are exposed as metrics for real-time dashboards and alerting.</p><h4>Layer 3: Model Providers — Cloud and Self-Hosted, Side by Side</h4><p>The gateway fans out to whatever backends you configure:</p><ul><li><strong>Azure OpenAI</strong> — for enterprises that need data residency, private networking, and Microsoft compliance guarantees.</li><li><strong>Anthropic</strong> — Claude models for strong reasoning, long context windows, and coding tasks.</li><li><strong>OpenAI</strong> — the default choice for many workloads, with the broadest model lineup.</li><li><strong>vLLM Cluster</strong> — self-hosted, high-throughput inference for open-weight models like <strong>Llama, DeepSeek, and Qwen</strong>. This is where serious cost savings happen at scale: vLLM’s paged attention and continuous batching squeeze maximum tokens-per-second out of your GPUs.</li><li><strong>Ollama</strong> — perfect for local development and edge deployments. Developers can test against the same gateway API they’ll use in production.</li></ul><p>The key insight: <strong>cloud and self-hosted models become interchangeable</strong>. Start on OpenAI, migrate high-volume workloads to your vLLM cluster, and keep the cloud providers as failover — all behind one API.</p><h4>Layer 4: Config &amp; Secrets</h4><p>Two things feed the gateway:</p><ul><li><strong>Gateway Config (YAML / Env)</strong> — model definitions, routing rules, rate limits, and budgets live in a declarative YAML file. It’s version-controllable, reviewable, and easy to promote across environments.</li><li><strong>Secrets Manager</strong> — provider API keys and tokens should never live in the config file itself. LiteLLM integrates with secrets managers so credentials are injected at runtime, rotated centrally, and never leaked into Git.</li></ul><p>A minimal config looks like this:</p><pre>model_list:<br>  - model_name: gpt-4o<br>    litellm_params:<br>      model: azure/gpt-4o-deployment<br>      api_base: os.environ/AZURE_API_BASE<br>      api_key: os.environ/AZURE_API_KEY<br>- model_name: claude-sonnet<br>    litellm_params:<br>      model: anthropic/claude-sonnet-4-5<br>      api_key: os.environ/ANTHROPIC_API_KEY<br>  - model_name: llama-70b<br>    litellm_params:<br>      model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct<br>      api_base: http://vllm-cluster:8000/v1<br>router_settings:<br>  routing_strategy: least-busy<br>  num_retries: 3</pre><h4>Layer 5: Observability &amp; Data Stores</h4><p>A gateway without visibility is a black box. The reference stack includes:</p><ul><li><strong>Metrics → Prometheus + Grafana</strong>: Scrape the gateway’s /metrics endpoint and build dashboards for latency percentiles, error rates, token throughput, and spend per team.</li><li><strong>Logging → ELK / Loki / Datadog</strong>: Ship structured request logs to whichever aggregation stack you already run.</li><li><strong>Database → PostgreSQL</strong>: LiteLLM persists virtual keys, users, teams, budgets, and spend records here. This is the source of truth for governance.</li><li><strong>Cache / Rate Limit Store → Redis</strong>: Redis powers distributed rate limiting across gateway replicas and can cache identical responses — cutting both latency and cost for repeated queries.</li></ul><h4>Why This Architecture Wins</h4><p>Pulling it all together, this design delivers five concrete benefits:</p><ol><li><strong>Single OpenAI-compatible API</strong> — one SDK, one integration, every model.</li><li><strong>Multiple providers &amp; self-hosted models</strong> — mix cloud APIs with your own GPU clusters freely.</li><li><strong>High availability with retry &amp; failover</strong> — provider outages become non-events.</li><li><strong>Cost, usage &amp; performance visibility</strong> — every token is tracked, attributed, and dashboarded.</li><li><strong>Centralized security &amp; governance</strong> — one place for keys, budgets, rate limits, and audit logs.</li></ol><h4>When Should You Adopt This?</h4><p>You probably need an LLM gateway when any of these become true:</p><ul><li>You use (or plan to use) <strong>more than one model provider</strong></li><li>Multiple <strong>teams or apps</strong> share LLM access and you need cost attribution</li><li>You’ve been burned by a <strong>provider outage or rate limit</strong> in production</li><li>You’re moving high-volume workloads to <strong>self-hosted inference</strong> but want cloud fallback</li><li>Compliance requires <strong>centralized logging and access control</strong> for AI usage</li></ul><p>If you’re a single developer calling one API, this is overkill. But the moment LLMs become shared infrastructure in your organization, a gateway stops being optional.</p><h4>Getting Started</h4><p>Spinning up a proof of concept takes minutes:</p><pre>pip install &#39;litellm[proxy]&#39;<br>litellm --config config.yaml</pre><p>Point your OpenAI SDK at http://localhost:4000, and you&#39;re routing through the gateway. From there, layer in PostgreSQL for key management, Redis for distributed rate limiting, and Prometheus for metrics as you head toward production.</p><h4>Final Thoughts</h4><p>The LLM ecosystem is fragmenting, not consolidating — new providers and open-weight models ship every month. Betting your architecture on a single vendor’s SDK is a liability. A gateway like LiteLLM turns that fragmentation into an advantage: you get pricing leverage, resilience, and the freedom to always use the best model for the job.</p><p>Build the abstraction once. Swap models forever.</p><p><em>If you found this useful, drop a clap 👏 and follow for more posts on production AI infrastructure. Questions about your own LLM stack? Let’s discuss in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ffcb29a7733" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/one-gateway-to-rule-all-your-llms-building-a-production-ready-ai-stack-with-litellm-1ffcb29a7733">One Gateway to Rule All Your LLMs: Building a Production-Ready AI Stack with LiteLLM</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes 1.35: What Platform Engineers Should Actually Care About]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://faun.pub/kubernetes-1-35-what-platform-engineers-should-actually-care-about-a194c768f04d?source=rss-89669a6506c5------2"><img src="https://cdn-images-1.medium.com/max/1248/1*wufKkcr7z44_KeGE7WU6TA.png" width="1248"></a></p><p class="medium-feed-snippet">Kubernetes releases move fast. Every few months a new version arrives with a long list of features, fixes, and deprecations. But in&#x2026;</p><p class="medium-feed-link"><a href="https://faun.pub/kubernetes-1-35-what-platform-engineers-should-actually-care-about-a194c768f04d?source=rss-89669a6506c5------2">Continue reading on FAUN.dev()  »</a></p></div>]]></description>
            <link>https://faun.pub/kubernetes-1-35-what-platform-engineers-should-actually-care-about-a194c768f04d?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a194c768f04d</guid>
            <category><![CDATA[k8s-series]]></category>
            <category><![CDATA[k8s-cluster-management]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Sat, 07 Mar 2026 17:41:11 GMT</pubDate>
            <atom:updated>2026-03-17T17:47:09.982Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Progressive Canary Deployments on Kubernetes with Argo Rollouts and Istio]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://faun.pub/progressive-canary-deployments-on-kubernetes-with-argo-rollouts-and-istio-0e6513f7645e?source=rss-89669a6506c5------2"><img src="https://cdn-images-1.medium.com/max/1024/1*Q-KiACyXrmCysAMPZx4-3w.png" width="1024"></a></p><p class="medium-feed-snippet">Overview</p><p class="medium-feed-link"><a href="https://faun.pub/progressive-canary-deployments-on-kubernetes-with-argo-rollouts-and-istio-0e6513f7645e?source=rss-89669a6506c5------2">Continue reading on FAUN.dev()  »</a></p></div>]]></description>
            <link>https://faun.pub/progressive-canary-deployments-on-kubernetes-with-argo-rollouts-and-istio-0e6513f7645e?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e6513f7645e</guid>
            <category><![CDATA[argo-rollouts]]></category>
            <category><![CDATA[canary-release]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[canary-deployments]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Mon, 29 Dec 2025 17:16:17 GMT</pubDate>
            <atom:updated>2026-01-25T21:16:52.370Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Vertical Pod Autoscaler (VPA) with InPlaceOrRecreate in Kubernetes 1.33.x]]></title>
            <link>https://aws.plainenglish.io/vertical-pod-autoscaler-vpa-with-inplaceorrecreate-in-kubernetes-1-33-x-5dd8968c9fa7?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dd8968c9fa7</guid>
            <category><![CDATA[vpa]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cost]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Mon, 01 Dec 2025 05:02:19 GMT</pubDate>
            <atom:updated>2025-12-15T06:30:00.145Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview</h3><p>In the dynamic world of Kubernetes, defining static CPU and memory requests for workloads is often a guessing game. Set them too low, and you face performance throttling or OOMKilled errors. Set them too high, and you waste valuable infrastructure capacity.</p><p>The Vertical Pod Autoscaler (VPA) has long been the solution to this problem, automatically adjusting resource requests to “right-size” your workloads. However, until recently, applying these adjustments usually meant restarting the pods — a disruptive process for production applications.</p><p>With the release of <strong>VPA v1.5.0</strong> and its stabilized integration with <strong>Kubernetes 1.33.x</strong>, this changes fundamentally. The introduction of the InPlaceOrRecreate update mode allows VPA to modify resource requests on running containers dynamically, without disrupting the pod.</p><p>This post will explore how this feature works, set up a practical example, and dive deep into analyzing how VPA makes its decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cd_LwJ1Y5AUe-8fI7IsBMg.png" /></figure><h4>The Evolution of VPA Update Modes</h4><p>VPA relies on three core components: the <strong>Recommender</strong> (analyzes historical metrics from Prometheus/Metrics Server), the <strong>Updater</strong> (evicts pods that need updates), and the <strong>Admission Controller</strong> (applies correct resources to new pods).</p><p>Traditionally, VPA managed updates using modes like Recreate (killing and restarting pods) or the now-deprecated Auto mode.</p><p><strong>Enter: </strong>InPlaceOrRecreate</p><p>Introduced in VPA v1.5.0, the InPlaceOrRecreate mode is the new standard for production-ready autoscaling. It intelligently attempts to update resources with minimal disruption:</p><ul><li><strong>In-Place Update (Preferred):</strong> If the underlying container runtime and Kubernetes API support live request updates, VPA modifies the resource requests on the running container. <strong>Zero downtime.</strong></li><li><strong>Recreate (Fallback):</strong> If an in-place update isn’t possible (e.g., due to specific scheduling constraints or older cluster versions), it falls back to terminating and recreating the pod.</li></ul><h4>The Strategy: Start Small, Scale Dynamic</h4><p>This new mode enables a highly efficient cost-saving strategy: deploy workloads with very low initial resource requests and let VPA dynamically resize them up to their actual needs without restarting them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Kh3ejriEiNf3JuQrQWi8Q.png" /></figure><h4>How VPA Helps with Low Initial Requests</h4><p>Many workloads are deployed with <strong>very low resource requests</strong> (e.g., cpu: 5m, memory: 64Mi) to save costs or due to lack of data. This often leads to:</p><ul><li><strong>CPU throttling</strong> → low performance.</li><li><strong>OOMKilled errors</strong> → insufficient memory allocation.</li></ul><p><strong>Before vs After Scenario: Low Requests → VPA Adjustments<br>Initial Deployment (Low Requests)</strong></p><pre>resources:<br>  requests:<br>    cpu: 5m<br>    memory: 64Mi<br>  limits:<br>    cpu: 10m<br>    memory: 128Mi</pre><p><strong>Problems:</strong></p><ul><li>Frequent throttling.</li><li>OOMKilled errors.</li><li>Application performance unstable.</li></ul><p><strong>After VPA with </strong><strong>InPlaceOrRecreate</strong></p><p>With <strong>VPA v1.5.0</strong> and InPlaceOrRecreate:</p><ul><li>VPA observes actual usage and increases requests dynamically.</li><li>Pods that start with low resources are “right-sized” automatically.</li><li>In-place updates ensure these adjustments happen with <strong>minimal disruption</strong>.</li></ul><pre>resources:<br>  requests:<br>    cpu: 250m<br>    memory: 256Mi<br>  limits:<br>    cpu: 500m<br>    memory: 512Mi</pre><p><strong>Improvements:</strong></p><ul><li>Right-sized CPU/memory requests.</li><li>Stable pod performance.</li><li>In-place update prevents unnecessary restarts.</li></ul><h4>Comparison Table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3H1Zuilr1rpCD2seAo1Iqw.png" /></figure><h4>Hands-on Tutorial: Implementing In-Place VPA</h4><p>Let’s demonstrate this capability. We will deploy a CPU-intensive application with intentionally low resource requests and watch VPA fix it in-place.</p><p><strong>Prerequisites</strong></p><ul><li>Kubernetes cluster version 1.33.x or later (crucial for stable in-place updates).</li><li>VPA version 1.5.0 or later installed.</li></ul><p><strong>Step 1: Installation</strong></p><p>If VPA is not already installed, use the official Helm chart. Ensure you enable the Admission Controller, Recommender, and Updater, pointing them to your metrics source (e.g., Prometheus).</p><pre>wget https://github.com/cowboysysop/charts/releases/download/vertical-pod-autoscaler-11.1.0/vertical-pod-autoscaler-11.1.0.tgz<br>tar -xvf vertical-pod-autoscaler-11.1.0.tgz<br>helm upgrade --install vpa ./vertical-pod-autoscaler -n kube-system</pre><p><strong>Step 2: Deploy the Workload and VPA</strong></p><p>We will create a deployment running stress-ng to artificially generate high CPU load. We will configure it with starved resources: only <strong>50m CPU</strong> and <strong>100Mi Memory</strong>.</p><p>We will then apply a VPA configuration with updateMode: InPlaceOrRecreate. We also set a <strong>Resource Policy</strong> to define safety guardrails (min/max allowed resources).</p><p>Create a file named vpa-demo.yaml:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: high-cpu-utilization-deployment<br>  namespace: test<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: cpu-utilization-app<br>  template:<br>    metadata:<br>      labels:<br>        app: cpu-utilization-app<br>    spec:<br>      containers:<br>      - name: cpu-utilization-container<br>        image: ubuntu<br>        command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;apt-get update &amp;&amp; apt-get install -y stress-ng &amp;&amp; while true; do stress-ng --cpu 1; done&quot;]<br>        resources:<br>          # Intentionally low initial requests<br>          requests:<br>            cpu: &quot;0.05&quot; # 50m<br>            memory: &quot;100Mi&quot;<br>          limits:<br>            cpu: &quot;100m&quot;<br>            memory: &quot;200Mi&quot;<br>---<br>apiVersion: &quot;autoscaling.k8s.io/v1&quot;<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: stress-vpa<br>  namespace: test<br>spec:<br>  targetRef:<br>    apiVersion: &quot;apps/v1&quot;<br>    kind: Deployment<br>    name: high-cpu-utilization-deployment<br>  updatePolicy:<br>    # The critical setting for in-place updates<br>    updateMode: InPlaceOrRecreate<br>  resourcePolicy:<br>    containerPolicies:<br>      - containerName: &#39;*&#39;<br>        minAllowed:<br>          cpu: 50m<br>          memory: 50Mi<br>        # VPA will not recommend more than these values<br>        maxAllowed:<br>          cpu: 200m<br>          memory: 500Mi<br>        controlledResources: [&quot;cpu&quot;, &quot;memory&quot;]</pre><p>Apply the manifest:</p><pre>kubectl apply -f vpa-demo.yaml</pre><p>Initially, the pods will be heavily CPU throttled. As the VPA Recommender gathers metrics, it will recognize the starvation. The Updater will then trigger an in-place update to increase the CPU request on the running containers to meet the demand, up to the defined maxAllowed policy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-77qTDIjO4UT-e7kmbwoFA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glMz_IAN4Fvp6RAYi38cIg.png" /></figure><h4>Analyzing VPA Behavior: Deep Dive</h4><p>Understanding how VPA arrives at its recommendations is vital for production confidence. Let’s examine the status of our VPA after it has run for a while.</p><p>Run: kubectl describe vpa stress-vpa -n test</p><pre>...<br>Spec:<br>  Resource Policy:<br>      Max Allowed:<br>        Cpu:     200m<br>        Memory:  500Mi<br>...<br>  Update Policy:<br>    Update Mode:  InPlaceOrRecreate<br>Status:<br>  Recommendation:<br>    Container Recommendations:<br>      Container Name:  cpu-utilization-container<br>      Lower Bound:<br>        Cpu:     200m<br>        Memory:  250Mi<br>      Target:<br>        Cpu:     200m<br>        Memory:  351198544  # ~335Mi<br>      Uncapped Target:<br>        Cpu:     476m<br>        Memory:  351198544  # ~335Mi<br>      Upper Bound:<br>        Cpu:     200m<br>        Memory:  500Mi</pre><p>Let’s decode this output, which reveals distinct behaviors for CPU and Memory.</p><p><strong>The CPU Analysis: Policy Capping</strong></p><p>You will notice something interesting in the CPU recommendations:</p><ul><li><strong>Uncapped Target: 476m:</strong> Based on actual usage data, the VPA’s algorithm determined the pod <em>wants</em> 476m of CPU to run optimally.</li><li><strong>Target: 200m:</strong> However, the actual recommendation is clamped down to 200m.</li></ul><p><strong>Why?</strong> This is your resourcePolicy in action. You set a maxAllowed.cpu of 200m. VPA respected this limit. The fact that the Lower Bound is also 200m indicates that even the most conservative safe estimate from the recommender was higher than your policy limit.</p><p><strong>Result:</strong> Your pods will be updated in-place to 200m CPU, but they will likely remain CPU-throttled because they actually need more. To fix this, you would need to increase the maxAllowed CPU in the VPA spec.</p><p><strong>The Memory Analysis: Safety Buffering</strong></p><p>Memory recommendations often confuse users because they seem higher than instantaneous usage. For example, your pod might currently use 120Mi, but VPA recommends a <strong>Target of ~335Mi</strong>.</p><p><strong>Why does VPA recommend so much memory?</strong> VPA is designed to prevent OOMKills, which kill the application entirely. Unlike CPU, memory cannot be throttled. Therefore, VPA is intentionally conservative:</p><ol><li><strong>Historical Trends:</strong> It looks at long-term data (days), not just the last 5 minutes. If the pod spiked to 300Mi yesterday, VPA remembers.</li><li><strong>Smoothing &amp; Percentiles:</strong> VPA aims for high percentiles (e.g., 95th percentile of usage) and adds a safety margin to ensure enough headroom for sudden spikes.</li></ol><p><strong>Result:</strong> The memory recommendation of ~335Mi is within your policy limits (50Mi — 500Mi), so VPA will apply this target to ensure stability.</p><p>After the recommendation, without any pod restart or recreation, the existing pods automatically received the new recommended values, and any newly launched pods also started with the updated values.</p><h4>Benefits of InPlaceOrRecreate</h4><ol><li><strong>Reduced Downtime</strong> — Adjust requests without restarts (if supported).</li><li><strong>Smarter Updates</strong> — Automatically falls back to recreate only when necessary.</li><li><strong>Right-Sizing</strong> — Fixes workloads with <strong>low initial requests</strong> by adjusting dynamically.</li><li><strong>Cost Efficiency</strong> — Avoids overprovisioning while preventing under-provisioning.</li></ol><h4>Best Practices</h4><ul><li>Start with <strong>Initial</strong> in production, then move to InPlaceOrRecreate.</li><li>Always monitor recommendations before enabling auto-updates.</li><li>Deploy workloads with <strong>safe minimums</strong>; let VPA optimize them.</li></ul><h4>Management and Troubleshooting Strategies</h4><p>Managing VPA in production requires a few key tactics.</p><p><strong>1. Essential Diagnostic Commands</strong></p><p>When VPA isn’t behaving as expected, gather data in this order:</p><pre># 1. Check the VPA config and current recommendations<br>kubectl describe vpa stress-vpa -n test<br># 2. Check the target pod&#39;s current requests vs actual usage<br>kubectl describe pod &lt;your-pod-name&gt; -n test<br>kubectl top pod &lt;your-pod-name&gt; -n test<br># 3. Check VPA logs for errors (adjust namespace if needed)<br>kubectl -n kube-system logs deployment/vpa-recommender<br>kubectl -n kube-system logs deployment/vpa-updater<br># 4. Look for OOMKills or recent events<br>kubectl get events -n test --sort-by=.lastTimestamp | tail -n 50</pre><p><strong>2. Safe Operations Strategy</strong></p><p>Before turning on automated updates in production:</p><ul><li><strong>Start in “Off” Mode:</strong> Set updateMode: &quot;Off&quot;. Let VPA run for at least 24 hours to gather data and generate recommendations without applying them. Compare these recommendations against your current settings.</li><li><strong>Use Pod Disruption Budgets (PDB):</strong> Even with in-place updates, fallback recreation might occur. Always ensure a PDB is active for your deployment to prevent VPA from taking down too many replicas simultaneously.</li><li><strong>Resetting History:</strong> If you make a radical change to your application and past historical metrics are no longer relevant, you can force VPA to re-learn by deleting and recreating the VPA object.</li></ul><pre>kubectl delete vpa stress-vpa -n test<br>kubectl apply -f vpa-demo.yaml</pre><h3>Conclusion</h3><p>With VPA v1.5.0 running on Kubernetes 1.33.x, the InPlaceOrRecreate mode offers the optimal balance of stability and cost-efficiency. It allows operations teams to move away from static, padded resource estimates and toward a dynamic infrastructure that adjusts to actual workload demands without constant disruptions.</p><p>By starting workloads with minimal requests and allowing VPA to handle the vertical scaling in-place, you can achieve significant infrastructure savings while ensuring application performance.</p><p>This results in:</p><ul><li><strong>Stable applications</strong> (no more OOMKills or throttling).</li><li><strong>Better resource utilization</strong> across the cluster.</li><li><strong>Minimal downtime</strong> with in-place adjustments.</li><li><strong>Future-ready scaling</strong> aligned with Kubernetes 1.33+.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dd8968c9fa7" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/vertical-pod-autoscaler-vpa-with-inplaceorrecreate-in-kubernetes-1-33-x-5dd8968c9fa7">Vertical Pod Autoscaler (VPA) with InPlaceOrRecreate in Kubernetes 1.33.x</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Spikes to Stability: How We Cut Latency & 5xxs with a Smarter Kubernetes Ingress Design]]></title>
            <link>https://aws.plainenglish.io/from-spikes-to-stability-how-we-cut-latency-5xxs-with-a-smarter-kubernetes-ingress-design-d8074171ffb6?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8074171ffb6</guid>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Mon, 11 Aug 2025 05:40:04 GMT</pubDate>
            <atom:updated>2025-08-11T14:32:33.559Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:-</h3><blockquote>“The engineering deep dive into how we re-architected Kubernetes ingress for faster, more reliable traffic routing.”</blockquote><p>When you run <strong>latency-sensitive workloads</strong> in Kubernetes at scale, every unnecessary network hop, uneven pod spread, or misconfigured load balancer can directly impact <strong>p99 latency</strong> and <strong>application stability</strong>.</p><p>We were facing:</p><ul><li>Random <strong>5xx errors and latency </strong>during Registration of nodes/deregistration of nodes.</li><li>Extra hops count adding<strong> High p99 latency</strong>.</li><li>Traffic imbalances across Availability Zones (AZs).</li></ul><p>After weeks of investigation, we implemented three key changes that <strong>reduced latency by ~35%</strong> and cut 5xx errors by over <strong>90%</strong>.</p><h4>1. Stop Relying on K8s-Managed Load Balancers — Use NodePort + ASG Integration</h4><p><strong>The problem:</strong><br> Kubernetes Service of type LoadBalancer is convenient but:</p><ul><li>Registers <strong>all cluster nodes</strong> into the target group — even if they don’t have pods for that service.</li><li>Routes traffic to random nodes, causing <strong>extra internal hops</strong>.</li><li>Delay in registering new pods or removing old ones, leading to intermittent <strong>5xx errors</strong> — sometimes during node deregistration while still receiving in-flight traffic.</li></ul><p><strong>The fix — NodePort + ASG Integrated LB:</strong><br>We exposed our Ingress/application via a <strong>NodePort</strong> service and managed the Load Balancer ourselves using <strong>Terraform</strong> or AWS Console. The LB was directly integrated with the <strong>infrastructure ASG</strong> that runs ingress or application pods.</p><p><strong>Implementation:</strong></p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: ingress-service<br>spec:<br>  type: NodePort<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      nodePort: 30080<br>  selector:<br>    app: ingress-app</pre><p><strong>Create LB and Target Group Manually (Terraform)</strong></p><pre>resource &quot;aws_lb&quot; &quot;ingress_nlb&quot; {<br>  name               = &quot;ingress-nlb&quot;<br>  internal           = true<br>  load_balancer_type = &quot;network&quot;<br>  subnets            = [&quot;subnet-123&quot;, &quot;subnet-456&quot;, &quot;subnet-789&quot;]<br>}<br><br>resource &quot;aws_lb_target_group&quot; &quot;ingress_tg&quot; {<br>  name         = &quot;ingress-tg&quot;<br>  port         = 30080<br>  protocol     = &quot;TCP&quot;<br>  vpc_id       = &quot;vpc-xxxxxx&quot;<br>  target_type  = &quot;instance&quot;<br>}</pre><p><strong>Integrate Target Group with Infra ASG</strong></p><pre>resource &quot;aws_autoscaling_attachment&quot; &quot;asg_tg&quot; {<br>  autoscaling_group_name = aws_autoscaling_group.infra_asg.name<br>  alb_target_group_arn   = aws_lb_target_group.ingress_tg.arn<br>}</pre><p><strong>Traffic Flow After Integration</strong></p><ul><li><strong>Old Path:</strong> LB → random node → forwarded to correct node → pod</li><li><strong>New Path:</strong> LB → ASG node running pod → pod (via NodePort)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BmCh9z52QWTU8JLRmcNjaQ.png" /></figure><p><strong>Impact:</strong></p><ul><li>Instances in the Target Group are now registered automatically via the ASG.</li><li>No missing targets after scale-up events.</li><li>Removed cross-node hops, reducing latency.</li></ul><h4>2. Enforce Traffic Locality with externalTrafficPolicy: Local</h4><p>By default, when you create a Kubernetes Service of type NodePort or LoadBalancer, Kubernetes sets:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: ingress-service<br>spec:<br>  externalTrafficPolicy: Cluster<br>  type: NodePort<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      nodePort: 30080<br>  selector:<br>    app: ingress-app</pre><p><strong>What </strong><strong>externalTrafficPolicy: Cluster Means</strong></p><ul><li><strong>Traffic routing behavior</strong>: Incoming traffic to any node’s Service port can be routed to <strong>any backend pod in the cluster</strong>, regardless of which node it’s on.</li><li><strong>Cross-node forwarding</strong>: If the receiving node doesn’t have the pod, kube-proxy forwards the packet to another node that does.</li><li><strong>Source IP loss</strong>: The packet’s original source IP is replaced by the node’s IP, which can break IP-based security rules or make logs less useful.</li><li><strong>Extra hop = extra latency</strong>: That extra internal network hop increases request latency, especially for large payloads or busy clusters.</li></ul><p><strong>Switching to </strong><strong>externalTrafficPolicy: Local</strong></p><p>We explicitly set:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: ingress-service<br>spec:<br>  externalTrafficPolicy: Local<br>  type: NodePort<br>  ports:<br>    - port: 80<br>      targetPort: 8080<br>      nodePort: 30080<br>  selector:<br>    app: ingress-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-aD_HDJmFCV1SrI4IGo9g.png" /></figure><p><strong>What changes?</strong></p><ul><li><strong>Only nodes with matching pods</strong> handle the traffic.</li><li><strong>Preserves the client’s source IP</strong>, which is critical for:</li><li>WAF / IP allowlists or blocklists</li><li>Geo-based routing or compliance rules</li><li>Debugging client-specific issues</li><li><strong>No cross-node forwarding</strong> → traffic comes in and is served directly by the node’s local pod.</li></ul><p><strong>Benefits We Observed</strong></p><ul><li>Eliminated the internal network hop.</li><li>Reduced p99 latency by <strong>~20%</strong> for ingress requests.</li><li>Improved security logging fidelity.</li><li>Lower CPU/network usage on nodes that were just acting as “traffic forwarders” before.</li><li>Reduced extra cross-zone forwarding costs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmCKwehQ0sP3H92hXZanug.png" /></figure><h4>3. Distribute Pods Evenly with topologySpreadConstraints and Pod Anti-Affinity</h4><p>After switching to externalTrafficPolicy: Local, we saw a new challenge: only nodes running ingress pods were marked healthy in the target group. Because Kubernetes wasn’t actively balancing pods across nodes or zones, we ended up with “pod clumping” — some nodes had multiple ingress pods while others had none.</p><p>This caused:</p><ul><li><strong>Hotspot nodes</strong> with high CPU utilization</li><li><strong>Uneven traffic distribution</strong> across the healthy nodes</li><li><strong>Increased latency</strong> during peak load</li></ul><p>To fix this, we applied <strong>topologySpreadConstraints</strong> and <strong>pod anti-affinity</strong>:</p><pre>topologySpreadConstraints:<br>  - maxSkew: 1<br>    topologyKey: kubernetes.io/hostname<br>    whenUnsatisfiable: DoNotSchedule<br>    labelSelector:<br>      matchLabels:<br>        app: ingress-app<br><br>affinity:<br>  podAntiAffinity:<br>    requiredDuringSchedulingIgnoredDuringExecution:<br>      - labelSelector:<br>          matchLabels:<br>            app: ingress-app<br>        topologyKey: kubernetes.io/hostname</pre><p><strong>Why this works</strong></p><ul><li>topologySpreadConstraints ensures pods are evenly distributed across nodes.</li><li>Pod anti-affinity guarantees that two pods of the same app never run on the same node.</li><li>Together, they eliminate hotspots, maximize per-node utilization, and keep latency consistent even under heavy load.</li></ul><p>If you want to understand more about “topologySpreadConstraints” read my <a href="https://aws.plainenglish.io/misconfigured-topologyspreadconstraints-almost-took-down-my-app-heres-what-i-learned-c216e5377f1d">blog</a></p><p><strong>Impact:</strong></p><ul><li>Healthy nodes evenly populated across the cluster</li><li>Higher overall system stability</li></ul><h4>Final Results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sCPii2KUb9YN0SgaNObkzg.png" /></figure><h4>Key Takeaways</h4><ul><li><strong>Control your LB provisioning</strong> when you need predictable, high-performance routing.</li><li><strong>Traffic locality</strong> is essential for low latency.</li><li><strong>Pod topology constraints</strong> help maintain balanced performance across the cluster.</li></ul><p>By combining <strong>NodePort + ASG-integrated LB</strong>, externalTrafficPolicy: Local, and topologySpreadConstraints, we achieved consistent low-latency performance and eliminated almost all random 5xx spikes.</p><p>Have questions?<br>Feel free to share your experiences or connect with me on <a href="https://www.linkedin.com/in/anup-dubey/">LinkedIn</a> or in the comments!</p><h3>A message from our Founder</h3><p><strong>Hey, </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>Sunil</strong></a><strong> here.</strong> I wanted to take a moment to thank you for reading until the end and for being a part of this community.</p><p>Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? <strong>We don’t receive any funding, we do this to support the community. ❤️</strong></p><p>If you want to show some love, please take a moment to <strong>follow me on </strong><a href="https://linkedin.com/in/sunilsandhu"><strong>LinkedIn</strong></a><strong>, </strong><a href="https://tiktok.com/@messyfounder"><strong>TikTok</strong></a>, <a href="https://instagram.com/sunilsandhu"><strong>Instagram</strong></a>. You can also subscribe to our <a href="https://newsletter.plainenglish.io/"><strong>weekly newsletter</strong></a>.</p><p>And before you go, don’t forget to <strong>clap</strong> and <strong>follow</strong> the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8074171ffb6" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/from-spikes-to-stability-how-we-cut-latency-5xxs-with-a-smarter-kubernetes-ingress-design-d8074171ffb6">From Spikes to Stability: How We Cut Latency &amp; 5xxs with a Smarter Kubernetes Ingress Design</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Misconfigured topologySpreadConstraints Almost Took Down My App — Here's What I Learned]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aws.plainenglish.io/misconfigured-topologyspreadconstraints-almost-took-down-my-app-heres-what-i-learned-c216e5377f1d?source=rss-89669a6506c5------2"><img src="https://cdn-images-1.medium.com/max/1576/1*T3CMFr8pNbu76kwHQtp7IA.png" width="1576"></a></p><p class="medium-feed-snippet">Introduction</p><p class="medium-feed-link"><a href="https://aws.plainenglish.io/misconfigured-topologyspreadconstraints-almost-took-down-my-app-heres-what-i-learned-c216e5377f1d?source=rss-89669a6506c5------2">Continue reading on AWS in Plain English »</a></p></div>]]></description>
            <link>https://aws.plainenglish.io/misconfigured-topologyspreadconstraints-almost-took-down-my-app-heres-what-i-learned-c216e5377f1d?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c216e5377f1d</guid>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[topologyspreadconstraint]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[topology]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Thu, 24 Jul 2025 06:09:16 GMT</pubDate>
            <atom:updated>2025-08-06T08:48:38.596Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Implementing Karpenter on AWS EKS: Fast, Cost-Efficient Autoscaling & Production-Ready…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://aws.plainenglish.io/implementing-karpenter-on-aws-eks-fast-cost-efficient-autoscaling-production-ready-1ba3d107ced6?source=rss-89669a6506c5------2"><img src="https://cdn-images-1.medium.com/max/1028/1*FJ1Z9xaqwxFPtJZf48Mp6g.png" width="1028"></a></p><p class="medium-feed-snippet">Introduction</p><p class="medium-feed-link"><a href="https://aws.plainenglish.io/implementing-karpenter-on-aws-eks-fast-cost-efficient-autoscaling-production-ready-1ba3d107ced6?source=rss-89669a6506c5------2">Continue reading on AWS in Plain English »</a></p></div>]]></description>
            <link>https://aws.plainenglish.io/implementing-karpenter-on-aws-eks-fast-cost-efficient-autoscaling-production-ready-1ba3d107ced6?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ba3d107ced6</guid>
            <category><![CDATA[bottle-rocket]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[karpenter]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Mon, 21 Jul 2025 18:13:04 GMT</pubDate>
            <atom:updated>2025-07-22T23:59:12.474Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Migrating Amazon EKS to Bottlerocket AMI: Architecture, Real-World Issues & Fixes]]></title>
            <link>https://faun.pub/migrating-amazon-eks-to-bottlerocket-ami-architecture-real-world-issues-fixes-7594f898a972?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/7594f898a972</guid>
            <category><![CDATA[bottle-rocket]]></category>
            <category><![CDATA[karpenter]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 18:06:17 GMT</pubDate>
            <atom:updated>2025-07-14T19:12:01.866Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A deep-dive journey into how we transitioned to Bottlerocket OS for Kubernetes on Amazon EKS, addressed real-world production issues, and unlocked better security, performance, and manageability.</blockquote><h4><strong>Introduction:-</strong></h4><p>Modern Kubernetes workloads demand leaner, faster, and more secure operating systems. While Amazon Linux 2/2023 has served its purpose, it includes extra components we don’t need for container workloads. That’s why we migrated to <strong>Bottlerocket</strong>, AWS’s purpose-built container host OS.</p><p>In this blog, I’ll walk through:</p><ul><li>Why Bottlerocket for EKS</li><li>Architecture &amp; provisioning strategy</li><li>Real challenges we faced — and exactly how we solved them</li><li>Security, logging, and customization tips</li></ul><p><strong>🧊 What is Bottlerocket?</strong></p><p><strong>Bottlerocket</strong> is an open-source Linux-based OS created by AWS, designed <strong>specifically for container hosting</strong>. With a <strong>read-only root filesystem</strong>, <strong>no SSH</strong>, <strong>atomic updates</strong>, and <strong>SELinux enforcement</strong>, it aligns perfectly with EKS and security-first infrastructure.</p><p><strong>🔄 Bottlerocket vs Amazon Linux 2/2023</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2FAEd8Hp8cCDAtgvkw4tYw.png" /></figure><p><strong>🏗️ Architecture: Bottlerocket</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BScDjLHzmPgHwd9c.png" /></figure><p>We used <strong>Karpenter</strong> to launch Bottlerocket-based worker nodes dynamically. Bottlerocket settings are passed via TOML-based user data — including bootstrap logic, labels, taints, DNS settings, and admin containers.</p><blockquote><strong><em>Tip:</em></strong><em> Bottlerocket doesn’t support shell-based EC2 user data like </em><em>#!/bin/bash. All customization must go through TOML.</em></blockquote><p><strong>⚙️ Node Group Creation with Karpenter</strong></p><p>Instead of traditional node groups, we used <strong>Karpenter NodePools</strong> with a custom Bottlerocket-based launch template.</p><p>Example provisioning YAML:</p><pre>provider:<br>  amiFamily: Bottlerocket</pre><p>and we used the following <strong>TOML settings</strong> for user-data</p><pre>  userData: |<br>    [settings.host-containers.admin]<br>    enabled = true<br>    user-data = &quot;&lt;base64-ssh-key&gt;&quot;        #for ssh ec2-user<br>    [settings.host-containers.control]<br>    enabled = true<br>    [settings.kubernetes.node-labels]<br>    &#39;nodegroup&#39; = &#39;bottlerocket&#39;<br>    [settings.kubernetes]<br>    cluster-dns-ip = &quot;10.10.0.2&quot;<br>    [settings.dns]<br>    name-servers = [&quot;10.10.0.2&quot;]<br>    search-list = [&quot;abc.local&quot;]<br>    [settings.host-containers.bootstraps]<br>    source = &quot;&lt;private-ecr&gt;/infra:bottle-rocket&quot;<br>    superpowered = true<br>    enabled = true</pre><h4>💥 Real-World Challenges (and How We Solved Them)</h4><h4>❌ Problem 1: SSH Not Available</h4><p><strong>Why it happened:</strong> Bottlerocket disables root SSH by default.<br> <strong>Solution:</strong> Use the admincontainer and SSM through the command line or console:</p><pre>aws ssm start-session --target &lt;instance-id&gt;</pre><p>By default, the admin container’s local user will be “ec2-user”. Users can add their own keys by populating the admin container’s user data with a base64-encoded JSON block.</p><pre>{<br>  &quot;ssh&quot;: {<br>    &quot;authorized-keys...&quot;,<br>  }<br>}</pre><p>And if you would like to change this, you can set the user value like so:</p><pre>{<br>  &quot;user&quot;: &quot;bottlerocket&quot;,<br>  &quot;ssh&quot;: {<br>    &quot;authorized-keys...&quot;,<br>  }<br>}</pre><p><strong>Why it’s needed</strong>: Bottlerocket doesn’t support SSH directly on the host. You connect to the <strong>admin container</strong> (via ssm or SSH), which has access to system tools and logs.</p><h4>❌<strong>Problem 2: Volume Mount Fails at </strong><strong>/data</strong></h4><p><strong>Error:</strong><br> mkdir /data: read-only file system</p><p><strong>Why it happened:</strong> Bottlerocket’s root filesystem is <strong>read-only</strong>.<br> <strong>Fix:</strong> Use writable paths like this/var/data:</p><pre>volumes:<br>  - name: log-volume<br>    hostPath:<br>      path: /var/data/app/&lt;app-name&gt;/&lt;logs&gt;<br>      type: DirectoryOrCreate<br>volumeMounts:<br>  - name: log-volume<br>    mountPath: /data/app/&lt;app-name&gt;/&lt;logs&gt;</pre><h4>❌Problem 3: Host Container Cannot Access Mounted Logs</h4><p><strong>Scenario:</strong> You want a host container (e.g., bootstraps) to upload logs to S3, but it fails because it can&#39;t see./var/data/...</p><p><strong>Fix:</strong> Access the host filesystem from inside the container using:</p><pre>tar czf logs.tar.gz /proc/1/root/var/data/log/</pre><p>This path exposes the actual host root to the container in its mount namespace.</p><h4>❌Problem 4: Assigning Labels &amp; Taints</h4><p><strong>Fix:</strong> Inject labels and taints at boot:</p><pre>[settings.kubernetes.node-labels]<br>&quot;nodegroup&quot; = &quot;bottlerocket&quot;</pre><pre>[settings.kubernetes.node-taints]<br>&quot;nodegroup&quot; = &quot;bottlerocket:NoExecute&quot;</pre><p>This ensures Karpenter correctly tags and filters nodes during scheduling.</p><h4>❌Problem 5: Logging Integration</h4><p>Since Bottlerocket doesn’t support native agents, we ran <strong>custom log forwarders</strong> as <strong>host containers</strong> and pointed them /proc/1/root/var/data/log/to read logs.</p><h4>🔄 Bottlerocket Updates</h4><p>Automatic, image-based updates are part of Bottlerocket’s design.</p><p>Manual updates:</p><pre>apiclient update check<br>apiclient update apply</pre><h4>🧠 Final Thoughts</h4><p>Migrating to <strong>Bottlerocket</strong> for EKS isn’t just about minimizing the OS footprint — it’s a strategy for increasing cluster security, automating patching, and reducing node lifecycle complexity. Combined with <strong>Karpenter</strong>, it becomes a powerful model for elastic Kubernetes compute.</p><h4>🔗 References</h4><ul><li><a href="https://bottlerocket.dev/">Bottlerocket Official Site</a></li><li><a href="https://karpenter.sh/">Karpenter</a></li><li><a href="https://bottlerocket.dev/en/os/1.39.x/api/settings/">Bottlerocket Settings API</a></li><li><a href="https://bottlerocket.dev/en/os/1.36.x/concepts/restricted-filesystem/">Bottlerocket Host Filesystem Restrictions</a></li></ul><p><strong>Have questions? Tried something similar?</strong><br>Feel free to share your experiences or connect with me on <a href="https://www.linkedin.com/in/anup-dubey/">LinkedIn</a> or in the comments!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Q5y4yWTJLmHyIQSC.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7594f898a972" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/migrating-amazon-eks-to-bottlerocket-ami-architecture-real-world-issues-fixes-7594f898a972">Migrating Amazon EKS to Bottlerocket AMI: Architecture, Real-World Issues &amp; Fixes</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Reduced System Latency: DNS Caching + Smarter CPU Configuration in Kubernetes]]></title>
            <link>https://aws.plainenglish.io/how-we-reduced-system-latency-dns-caching-smarter-cpu-configuration-in-kubernetes-2dacc9a68904?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dacc9a68904</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[elastic-kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[latency]]></category>
            <category><![CDATA[dns]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 17:56:30 GMT</pubDate>
            <atom:updated>2025-06-29T06:17:57.190Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s microservices-driven architectures, performance bottlenecks often emerge from places we least expect—like DNS resolution and CPU misconfiguration. At [Your Company], we recently faced issues with elevated latency and intermittent request failures. After a deep investigation, we uncovered two key contributors:</p><ol><li>Unreliable DNS resolution under high traffic</li><li>Misuse of CPU limits in Kubernetes pods</li></ol><p>This blog outlines how we tackled both problems and significantly improved system responsiveness and reliability.</p><p><strong>🔍 Symptom: Intermittent Latency Spikes and Request Failures</strong></p><p>Our alerting systems started flagging spikes in request latencies across services. These were hard to reproduce and occurred seemingly at random. The symptoms included:</p><ul><li>Increased <strong>DNS resolution failures</strong></li><li>Higher <strong>latency during peak traffic</strong></li><li><strong>CPU throttling</strong> even when nodes had available capacity</li></ul><p>After tracing and debugging through logs, traces, and metrics, two root causes emerged.</p><p><strong>⚠️ Problem #1: DNS Resolution Failures in Kubernetes</strong></p><p>Like most Kubernetes users, we relied on kube-dns (or CoreDNS) for internal DNS resolution. During peak traffic, we noticed a surge in DNS-related timeouts. The cause? A combination of:</p><ul><li>Network congestion between nodes and the DNS pod</li><li>UDP packet loss (DNS runs over UDP by default)</li><li>Retry storms from failed lookups</li></ul><p>The default CoreDNS setup became a bottleneck—especially since every pod made external and internal DNS queries directly through a shared service.</p><p><strong>✅ Solution: Deploying NodeLocal DNSCache</strong></p><p>We adopted NodeLocal DNSCache, which runs a lightweight DNS cache on every node. This localized DNS resolution brought immediate benefits:</p><ul><li>Reduced DNS lookup latency by avoiding cross-node traffic</li><li>Minimized UDP packet loss by keeping DNS traffic local</li><li>Increased resilience by isolating DNS failures to a single node</li></ul><p>🔄 <strong>Before vs After DNSCache</strong>:</p><ul><li><strong>Average DNS resolution time</strong> dropped from ~100ms to &lt;5ms</li><li>DNS errors reduced by over 90%</li><li>API call latency improved by 15–25% during high load</li></ul><p><strong>⚠️ Problem #2: Misconfigured Kubernetes CPU Limits</strong></p><p>At the same time, we noticed some pods were <strong>throttled</strong> despite plenty of node-level CPU availability. Upon inspection, we discovered:</p><ul><li>Many pods had <strong>low CPU limits</strong> set for safety</li><li>Once these limits were hit, Linux cgroups throttled CPU time</li><li>Even though CPU <em>requests</em> were modest, setting <em>limits</em> tightly created artificial bottlenecks</li></ul><p>This mismatch caused uneven CPU utilization across the cluster and created high tail latencies under load.</p><p><strong>✅ Solution: Rethinking CPU Limits — A Case for Limit-Free Pods</strong></p><p>Following research and experimentation, we adopted a more nuanced strategy:</p><ul><li>For critical or latency-sensitive services, we <strong>removed CPU limits</strong> while retaining conservative CPU <strong>requests</strong></li><li>We relied on <strong>HPA (Horizontal Pod Autoscaler)</strong> to scale under load</li><li>Cluster Autoscaler ensured that node provisioning kept up with demand</li></ul><p>This allowed the kernel and Kubernetes to make more efficient scheduling decisions without triggering cgroup throttling.</p><p>📉 <strong>Impact</strong>:</p><ul><li>Tail latency (p95/p99) reduced by ~30%</li><li>Smoother CPU usage without artificial throttling</li><li>More predictable autoscaling behavior</li></ul><h3>🎯 Results &amp; Learnings</h3><p>By combining <strong>NodeLocal DNS caching</strong> and <strong>removing restrictive CPU limits</strong>, we achieved measurable improvements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*erhJNgC6fJzVAyb0Uq3T1Q.png" /></figure><h3>💡 Key Takeaways</h3><ul><li>DNS, often overlooked, can be a <strong>hidden latency culprit</strong> — especially in containerized environments. Local caching is a game changer.</li><li>CPU limits, when misused, act as <strong>anti-patterns</strong> for performance-sensitive workloads. Sometimes less control brings better results.</li><li>Observability and tracing were critical in identifying these bottlenecks. Metrics alone wouldn’t have revealed the full picture.</li></ul><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dacc9a68904" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-we-reduced-system-latency-dns-caching-smarter-cpu-configuration-in-kubernetes-2dacc9a68904">How We Reduced System Latency: DNS Caching + Smarter CPU Configuration in Kubernetes</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating to AWS Graviton in EKS: Challenges, Multi-Arch Setup, and Our Learnings]]></title>
            <link>https://faun.pub/migrating-to-aws-graviton-in-eks-challenges-multi-arch-setup-and-our-learnings-a1d324a5b6af?source=rss-89669a6506c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1d324a5b6af</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[multiarch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[graviton]]></category>
            <dc:creator><![CDATA[sentraorb]]></dc:creator>
            <pubDate>Tue, 10 Jun 2025 19:48:46 GMT</pubDate>
            <atom:updated>2025-06-16T07:56:28.297Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Writing this blog a bit late, but the migration taught us a lot and helped us streamline the system efficiently.</blockquote><h3>Why Graviton?</h3><p>Graviton instances are built on ARM architecture and offer better price-performance than traditional AMD/Intel-based (x86_64) instances in many workloads. Naturally, like many modern DevOps teams, we wanted to take advantage of this in our Kubernetes (EKS) clusters.</p><p>But as we began the migration, we hit an early blocker — <strong>Graviton machines (ARM-based) were not readily available in all regions</strong>, and we feared potential shortages or instance launch failures in future production scenarios. This made us rethink our approach.</p><h3>The Initial Challenge: Graviton Instance Shortage</h3><p>When we started testing with Graviton EC2 instances in EKS, we noticed something critical:</p><blockquote><strong><em>ARM machines were fewer in availability compared to x86_64.</em></strong></blockquote><p>This meant that if we went fully ARM-based and AWS didn’t have available capacity in our region or AZs, we could face cluster outages. And that was unacceptable. We needed <strong>fallback flexibility</strong>.</p><h3>Our Strategy: Multi-Architecture Support (ARM + AMD)</h3><p>Instead of going “all in” on Graviton, we thought:</p><blockquote><strong><em>What if we ran a multi-architecture EKS node group with both ARM and AMD EC2 machines?</em></strong></blockquote><p>Our plan was:</p><ol><li>Configure an ASG (Auto Scaling Group) with a <strong>preference for Graviton (ARM)</strong> machines.</li><li>If ARM was unavailable, fall back to AMD (x86) machines.</li><li>Automatically pull the <strong>right Docker image</strong> based on the node’s architecture (ARM64 or AMD64).</li></ol><p>Simple in theory. Complex in execution.</p><h3>ASG Blocker: UI Doesn’t Support Mixed Architectures</h3><p>When we approached this with standard AWS tooling, we hit a wall.</p><blockquote><strong><em>AWS Console UI doesn’t support mixing architectures in a single ASG.</em></strong></blockquote><p>We reached out to the AWS team, and they confirmed — this setup is only possible via <strong>CLI or API</strong>, not the UI.</p><p>So, we turned to <strong>Terraform</strong> to design a robust solution</p><h3>Our Terraform Trick: Two Launch Templates, One ASG</h3><p>We designed a Terraform setup where:</p><ul><li>We created <strong>two separate launch templates</strong>:</li><li>One for <strong>ARM64</strong> Graviton instances.</li><li>One for <strong>AMD64</strong> x86 instances.</li><li>We used launch template<strong> overrides</strong> within a <strong>mixed-instance policy ASG</strong>.</li><li>The ASG was configured with <strong>priority-based instance overrides</strong>:</li><li><strong>First preference</strong>: ARM64 (Graviton)</li><li><strong>Second fallback</strong>: AMD64</li></ul><p>This way, the ASG would <strong>try to launch ARM instances first</strong>. If ARM machines weren’t available, it would automatically fall back to AMD instances, ensuring availability.</p><p><strong>✅ Problem 1 Solved: Hybrid node group with ARM preference.</strong></p><p>But we weren’t done yet. Even if we solved the instance architecture problem, what about the Docker images?</p><h3>Next Challenge: Docker Image Compatibility</h3><p>By default, Docker images are tied to an architecture (e.g.,amd64). If you run an amd64image on an arm64 node, the container might crash due to incompatibility.</p><p>So we needed to build <strong>multi-arch Docker images</strong> that could:</p><ul><li>Automatically detect the underlying EC2 architecture.</li><li>Pull and run the correct image variant (arm64 or amd64).</li></ul><h3>Building Multi-Arch Docker Images with buildx</h3><p>Here’s how we solved it:</p><h3>Step 1: Enable buildx in Docker</h3><p>We used Docker’s buildxtool to build multi-architecture images:</p><pre>docker buildx create --use<br>docker buildx build --platform linux/amd64,linux/arm64 -t my-app:v1 --push .</pre><p>This command builds the image for both AMD and ARM and pushes it to the registry with a <strong>manifest index</strong> that maps the correct architecture to each image variant.</p><h3>Step 2: Push to ECR with Image Index</h3><p>In AWS ECR, we pushed both architectures under the same tag:</p><ul><li>my-app:v1 now includes:</li><li>my-app:arm64</li><li>my-app:amd64</li></ul><p>The Docker client on the EC2 instance will <strong>automatically pull the matching image</strong> for the node’s architecture.</p><p><strong>✅ Problem 2 Solved: Architecture-aware container deployments.</strong></p><h3>Putting It All Together</h3><p>With both ASG setup and multi-arch Docker images ready, we had a robust hybrid setup:</p><ul><li>Kubernetes pods are scheduled on <strong>Graviton nodes</strong> by default.</li><li>If ARM nodes are unavailable, workloads fall back to <strong>x86 nodes</strong>.</li><li>Docker images always match the underlying node architecture.</li></ul><h3>Learnings and Takeaways</h3><p>Here are a few things we learnt that might help others:</p><ol><li><strong>Multi-arch is not just about Docker</strong> — you need your entire infra (ASG, launch templates, metrics, dashboards) to support and understand both architectures.</li><li><strong>Terraform is your best friend</strong> when the AWS Console can’t do what you want.</li><li><strong>Test all third-party tools and sidecars</strong> — many popular containers (e.g., Fluentd, Prometheus exporters) don’t always publish multi-arch images.</li><li>Monitor carefully — use labels and node selectors in Kubernetes to control where critical workloads are scheduled.</li></ol><h3>Final Thoughts</h3><p>This wasn’t an easy migration, but it made our infrastructure more resilient and future-ready.</p><p>And yes, this blog is late. But our system is now faster, cheaper, and smarter because of it. 💪</p><p><strong>Have questions? Tried something similar?</strong><br>Feel free to share your experiences or connect with me on <a href="https://www.linkedin.com/in/anup-dubey/">LinkedIn</a> or in the comments!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*NRs5QmNvMF55sjj6.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1d324a5b6af" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/migrating-to-aws-graviton-in-eks-challenges-multi-arch-setup-and-our-learnings-a1d324a5b6af">Migrating to AWS Graviton in EKS: Challenges, Multi-Arch Setup, and Our Learnings</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>