<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Pepuho on Medium]]></title>
        <description><![CDATA[Stories by Daniel Pepuho on Medium]]></description>
        <link>https://medium.com/@danielpepuho?source=rss-5c346eb781b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Hgl-7-qrRlLRq5NJtY5sgg.jpeg</url>
            <title>Stories by Daniel Pepuho on Medium</title>
            <link>https://medium.com/@danielpepuho?source=rss-5c346eb781b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:42:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielpepuho/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM on EKS: Training Pipelines with Kubeflow]]></title>
            <link>https://danielpepuho.medium.com/llm-on-eks-training-pipelines-with-kubeflow-f648868173fa?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f648868173fa</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 00:34:45 GMT</pubDate>
            <atom:updated>2026-10-27T00:40:51.765Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published on: <a href="https://danielcristho.site/blog/llm-on-eks-pipeline-kubeflow">danielcristho.site</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*kINhwDPGvoDYG9K2jdHG6A.png" /></figure><p>Hi there! 👋🏽 Let’s continue this series. In previous posts, we got a Llama-3.1–8B-Instruct serving on EKS, then I made it autoscale with Ray and Karpenter. This time it’s about fine-tuning the model, quantizing it, and pushing it back into the same serving stack, all orchestrated as a repeatable pipeline instead of a one-off script.</p><h3>TL;DR</h3><ul><li>Kubeflow Pipelines standalone, running on the same EKS cluster as the earlier posts, orchestrates a 3-step fine-tuning pipeline: prep data, LoRA fine-tune, AWQ quantize, then push straight to S3.</li><li>KFP’s default in-cluster MySQL and MinIO are swapped for Amazon RDS and S3, no PersistentVolumeClaims, no EBS CSI driver needed.</li><li>Ray stays dedicated to serving (autoscaling vLLM); training runs as plain pods through Argo Workflows, not Ray jobs (more on why below).</li><li>GPU steps land on the same Karpenter-provisioned g4dn.xlarge NodePool from the previous post.</li><li>The pipeline writes to the same S3 bucket vllm_stack.py already reads from; actually serving this pipeline&#39;s output still means pointing vllm_stack.py&#39;s hardcoded model_id at it (more on that below).</li></ul><h3>The Stack</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e8decb606481a1beeab001118d2b7ca/href">https://medium.com/media/8e8decb606481a1beeab001118d2b7ca/href</a></iframe><h3>Architecture</h3><p>Here’s how the pieces fit together. The diagram covers the whole flow: deploying the infrastructure, running the training pipeline in Kubeflow, storing artifacts in S3 and RDS, serving the model with Ray and vLLM, and Karpenter adding GPU nodes when more capacity is needed.</p><p>Everything runs on the same EKS cluster as before. Karpenter still creates GPU nodes on demand using the same g4dn.xlarge NodePool for vLLM autoscaling.</p><p>The new part is Kubeflow Pipelines, installed without the full Kubeflow platform. It uses RDS for metadata and S3 for artifact storage instead of the default in-cluster MySQL and MinIO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ylj_tv34UCbZ8D-aUQmd7Q.png" /><figcaption>Kubeflow Pipelines on EKS.</figcaption></figure><p>Following the numbers in the diagram:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/283cbb1be91ed5eb23ad0c537906275c/href">https://medium.com/media/283cbb1be91ed5eb23ad0c537906275c/href</a></iframe><h3>Why Kubeflow over Ray Tune + Ray Train?</h3><p>Ray is already part of the stack (ray_stack.py), so why not use Ray Tune or Ray Train for training instead of adding Kubeflow? Ray and Kubeflow solve different problems. Ray Tune or Ray Train is about <strong>execution</strong>: running training or hyperparameter search across multiple GPUs or nodes. Kubeflow Pipelines is about <strong>orchestrating</strong> the process: managing a multi-step workflow from data prep to training to quantization to upload as a tracked DAG. Each run keeps its history, artifact lineage is clear (which dataset produced which model), there&#39;s a UI to see which step failed, and pipelines can be reused and versioned.</p><p>For fine-tuning with LoRA on a single g4dn.xlarge GPU, we don&#39;t really need Ray Train/Tune because we&#39;re only using one GPU. Ray becomes more useful when training needs multiple nodes. Also, the current Ray cluster is used for serving LLMs with vLLM autoscaling, not for training. The finetune_pipeline.py script runs as a regular pod through Argo Workflows, not as a Ray job. If we need distributed fine-tuning in the future, we can combine both. A Kubeflow step could submit a Ray Train job to the same cluster, so they work together rather than replacing each other.</p><h3>Why RDS and S3?</h3><p>Kubeflow Pipelines <strong>ships with in-cluster MySQL and MinIO by default</strong>, which is great for getting started locally. Since this series focuses on Amazon EKS, those get replaced with managed AWS services instead. <strong>Amazon RDS stores pipeline metadata and ML Metadata</strong>, while <strong>Amazon S3 stores pipeline artifacts</strong>. This keeps state outside the Kubernetes cluster, making the platform easier to operate and more resilient. It also sidesteps a real constraint of this cluster: no EBS CSI driver is installed (model weights already live in S3, Ray’s model-cache uses emptyDir), so the default MySQL/MinIO PersistentVolumeClaims would just sit Pending forever.</p><h3>How Is the Kubeflow Pipelines Manifest Integrated into the Project?</h3><p>Kubeflow Pipelines standalone ships as Kustomize, not Helm, so cluster.add_helm_chart() doesn&#39;t apply here. Instead, the upstream manifest is rendered once with kustomize build and vendored into pipeline_llm/kubeflow/manifests/, then patched at synth time (in kubeflow_stack.py) to swap the default in-cluster MySQL/MinIO for RDS + S3:</p><pre>kustomize build &quot;https://github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.2.0&quot; \<br>  &gt; pipeline_llm/kubeflow/manifests/kfp-cluster-scoped.yaml<br><br>kustomize build &quot;https://github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-emissary?ref=2.2.0&quot; \<br>  &gt; pipeline_llm/kubeflow/manifests/kfp-pipelines-base.yaml</pre><h3>What Does the Pipeline Actually Do?</h3><p>finetune_pipeline.py compiles down to three steps:</p><ol><li><strong>prepare_dataset</strong>: pulls a small instruction-tuning subset (mlabonne/guanaco-llama2-1k) and writes it out as JSONL. CPU-only, no GPU node needed.</li><li><strong>finetune_lora</strong>: LoRA fine-tunes the base model for a single epoch, then merges the adapter into full-precision weights. Runs on a GPU node.</li><li><strong>quantize_and_upload</strong>: AWQ-quantizes the merged model back to INT4, then uploads it to s3://&lt;bucket&gt;/pipeline-output/latest/, inside the same bucket vllm_stack.py reads from. Also GPU.</li></ol><p>The pipeline definition itself is short. Each step is a @dsl.component, and the two GPU steps get pinned to Karpenter&#39;s GPU NodePool with a small helper that requests one GPU, selects the workload=gpu label, and tolerates the nvidia.com/gpu taint:</p><pre>def _on_gpu_node(task: dsl.PipelineTask) -&gt; dsl.PipelineTask:<br>    task.set_accelerator_type(&quot;nvidia.com/gpu&quot;).set_accelerator_limit(1)<br>    kubernetes.add_node_selector(task, label_key=&quot;workload&quot;, label_value=&quot;gpu&quot;)<br>    kubernetes.add_toleration(task, key=&quot;nvidia.com/gpu&quot;, operator=&quot;Equal&quot;, value=&quot;true&quot;, effect=&quot;NoSchedule&quot;)<br>    return task<br>@dsl.pipeline(name=&quot;qwen-lora-finetune&quot;, description=&quot;Prep data -&gt; LoRA fine-tune -&gt; AWQ quantize -&gt; push to S3&quot;)<br>def finetune_pipeline(<br>    bucket_name: str,<br>    base_model: str = &quot;Qwen/Qwen2.5-1.5B-Instruct&quot;,<br>    num_examples: int = 200,<br>    epochs: int = 1,<br>    s3_prefix: str = &quot;pipeline-output/latest&quot;,<br>):<br>    prep = prepare_dataset(num_examples=num_examples)<br>    train = finetune_lora(dataset_in=prep.outputs[&quot;dataset_out&quot;], bucket_name=bucket_name, base_model=base_model, epochs=epochs)<br>    _on_gpu_node(train)<br>    quant = quantize_and_upload(model_in=train.outputs[&quot;model_out&quot;], bucket_name=bucket_name, s3_prefix=s3_prefix)<br>    _on_gpu_node(quant)<br><br>if __name__ == &quot;__main__&quot;:<br>    compiler.Compiler().compile(finetune_pipeline, __file__.replace(&quot;.py&quot;, &quot;.yaml&quot;))</pre><p>Passing prep.outputs[&quot;dataset_out&quot;] into finetune_lora is what gives KFP the dependency graph: no explicit ordering, the artifact handoff alone decides that finetune_lora waits for prepare_dataset, and quantize_and_upload waits for finetune_lora.</p><p>The goal isn’t to achieve high accuracy. We use a small dataset and one epoch to test whether the entire pipeline works, from GPU scheduling with Karpenter and artifact sharing between steps to deploying the trained model for serving without manual steps.</p><h3>Why a Different Base Model for Training?</h3><p>The earlier posts use <strong>hugging-quants/Meta-Llama-3.1–8B-Instruct-AWQ-INT4</strong> for serving. However, <strong>fine-tuning requires full-precision weights</strong>, so the pipeline starts from a full-precision model, applies LoRA, merges the adapter, and quantizes the result to AWQ INT4.</p><p>Initially, we tried Llama-3.1–8B, but the g4dn.xlarge GPU (T4, 14.74 GB VRAM) could handle QLoRA training but not merging the adapter into full-precision weights. Reloading in 4-bit or 8-bit also caused errors during AWQ quantization.</p><p><strong>To avoid adding a larger GPU, we switched to Qwen/Qwen2.5–1.5B-Instruct</strong>, which can be trained and merged on the same GPU..</p><p>The trade-off is that the output model cannot directly replace the Llama model currently served by vllm_stack.py. Using a matching base model or a larger GPU would be needed to make them compatible.</p><h3>Deploywhich can be trained and merged on the same GPU.</h3><pre>cdk bootstrap<br>cdk deploy --all</pre><p>Getting kubeflow-stack to a healthy state took a few real rounds of debugging, worth being honest about rather than pretending the first deploy just worked:</p><ul><li>An EC2 security group rule description containing -&gt;, rejected by AWS&#39;s character allowlist for rule descriptions.</li><li>CloudFormation doesn’t resolve {{resolve:secretsmanager:...}} dynamic references inside Custom Resource properties, and eks.KubernetesManifest is one, so the RDS password needed a small Lambda that reads Secrets Manager directly instead of the usual secretValueFromJson() trick.</li><li>Three separate container images upstream had quietly moved off gcr.io since this manifest was pinned to 2.2.0 (ml-pipeline-ui&#39;s frontend, argoexec, and a transient registry blip on nvidia-device-plugin), each needing its own fixed reference.</li><li>mlpipeline-minio-artifact&#39;s default demo credentials (minio/minio123) short-circuited KFP&#39;s object store client straight into a static-credential path, authenticating against real S3 with garbage keys instead of ever attempting IRSA. Blanking that Secret out was the actual fix, not the IRSA role debugging it took to find that.</li></ul><p>None of these are infrastructure decisions worth re-litigating, they’re the drift that shows up whenever you vendor a manifest instead of tracking a moving target. cdk destroy --all cleanly tears everything down afterward, with the same two Karpenter-related teardown gotchas from the last two posts (an EC2NodeClass finalizer and an orphaned IAM instance profile), fixed the same way as before.</p><p>Redeploying from scratch weeks later (after the base model swap below) surfaced two more:</p><ul><li>The S3 model bucket has RemovalPolicy.RETAIN, so it survived the teardown with real data in it. Redeploying eks_stack.py tried to s3.Bucket(...) a bucket name that already existed and CloudFormation rejected the changeset outright:</li></ul><pre>Early validation failed for change set cdk-deploy-change-set:<br>eks-stack/ModelBucket/Resource  (AWS::S3::Bucket ModelBucketB33D855B)<br>  Resource of type &#39;AWS::S3::Bucket&#39; with identifier &#39;eks-llm-pipeline-model-bucket&#39; already exists.</pre><p>Fixed by referencing the existing bucket instead of recreating it: s3.Bucket.from_bucket_name(self, &quot;ModelBucket&quot;, bucket_name).</p><ul><li>cluster.add_helm_chart()/add_manifest(), called from karpenter_stack.py, ray_stack.py, vllm_stack.py, and kubeflow_stack.py, all parent to the Cluster construct that eks_stack.py owns. CDK resolves which CloudFormation stack a resource belongs to by walking up the construct tree, not by which file the code lives in, so every one of those Helm charts and manifests physically deploys as part of eks-stack, regardless of which Python stack class wrote the call. Running cdk deploy eks-stack alone deployed the entire app, Karpenter, Ray, vLLM, and Kubeflow Pipelines included; the later cdk deploy karpenter-stack/ray-stack/vllm-stack/kubeflow-stack commands just confirmed to CDK that nothing had changed:</li></ul><pre>eks-stack: deploying... [1/3]<br>✅  eks-stack (no changes)<br>✨  Deployment time: 0s<br><br>karpenter-stack: deploying... [2/3]<br>✅  karpenter-stack<br>✨  Deployment time: 12.93s</pre><h3>Running the Pipeline</h3><p>Compiled finetune_pipeline.py locally (python finetune_pipeline.py, producing finetune_pipeline.yaml), then uploaded it through the KFP UI (reachable via kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80, since there&#39;s no public load balancer for it):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yOWVS-DL1d_uU0_p.png" /><figcaption>Uploading finetune_pipeline.yaml in the KFP UI.</figcaption></figure><p>Once uploaded, KFP renders the compiled DAG: three components, with the dataset_out and model_out artifacts as the handoff between them, exactly the dependency graph the artifact wiring in the code implies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JqgAhGjGRdQgpwT4.png" /><figcaption>finetune_pipeline graph in the KFP UI.</figcaption></figure><p>Hit <strong>Create run</strong>, set bucket_name, and the first step, prepare-dataset, needs no GPU and runs in seconds:</p><pre>[KFP Executor 2026-08-28 11:41:26,465 INFO]: Loading KFP component &quot;prepare_dataset&quot; from /tmp/tmp.pfCjK0T5YM/ephemeral_component.py<br>...<br>Downloading data: 100%|██████████| 967k/967k [00:00&lt;00:00, 2.06MB/s]<br>Generating train split: 100%|██████████| 1000/1000 [00:00&lt;00:00, 86752.38 examples/s]<br>Creating json from Arrow format: 100%|██████████| 1/1 [00:00&lt;00:00, 95.20ba/s]<br>[KFP Executor 2026-08-28 11:41:28,539 INFO]: Wrote executor output file to /tmp/kfp_outputs/output_metadata.json.<br>I0828 11:41:28.988313 24 object_store.go:175] uploadFile(localFilePath=&quot;...&quot;, blobFilePath=&quot;prepare-dataset/dataset_out&quot;)<br>I0828 11:41:29.152326 24 launcher_v2.go:151] publish success.</pre><p>Dataset pulled from Hugging Face, subset generated, uploaded to S3. Then it’s finetune-lora&#39;s turn, which needs a GPU node Karpenter doesn&#39;t have sitting around idle. The first two attempts after the base model swap both died here, not from pipeline code, from Karpenter deleting the node it had just provisioned:</p><pre>{&quot;message&quot;:&quot;disrupting node(s)&quot;,&quot;reason&quot;:&quot;empty&quot;,&quot;decision&quot;:&quot;delete&quot;,&quot;disrupted-node-count&quot;:1,<br> &quot;disrupted-nodes&quot;:[{&quot;Node&quot;:{&quot;name&quot;:&quot;ip-10-0-204-81.ec2.internal&quot;},&quot;capacity-type&quot;:&quot;on-demand&quot;,&quot;instance-type&quot;:&quot;g4dn.xlarge&quot;}]}</pre><p>The GPU NodePool had consolidationPolicy: WhenEmpty, consolidateAfter: &quot;30s&quot;. A brand-new node needs time to join, have the NVIDIA device plugin register nvidia.com/gpu, then get a pod actually bound to it, routinely more than 30 seconds. Karpenter&#39;s disruption loop ran its &quot;empty&quot; check in that window, saw zero pods yet, and deleted the node out from under finetune-lora mid-schedule, twice in a row on two separate runs. Bumping consolidateAfter to &quot;5m&quot; fixed it; the third run&#39;s node survived long enough to actually get used.</p><p>The next retry failed due to a packaging conflict, not an infrastructure issue.</p><pre>ImportError: pyarrow requires NumPy 2.0 or newer, found 1.26.4</pre><p>pytorch/pytorch:2.4.0-cuda12.1-cudnn9-runtime ships numpy==1.26.4 via conda; datasets==2.21.0 pulls in an unpinned (so: latest) pyarrow built against a NumPy 2.0 ABI. Forcing numpy&gt;=2.0 directly seemed like the fix, until pip&#39;s resolver pointed out why that numpy was there in the first place:</p><pre>ERROR: Cannot install accelerate==0.33.0, numpy&gt;=2.0, peft==0.12.0 and transformers==4.44.2 because<br>these package versions have conflicting dependencies.<br>ERROR: ResolutionImpossible</pre><p>transformers==4.44.2, peft==0.12.0, and accelerate==0.33.0 all still pin numpy&lt;2. The actual fix was pinning pyarrow&lt;17 instead, so pip resolves a pyarrow build compatible with the numpy the other packages already require, rather than fighting them over numpy itself.</p><p>With both of those fixed, the run that finally went through pulled the base model from S3 (cached there by an earlier attempt, behind the VPC gateway endpoint, no NAT charge) instead of Hugging Face:</p><pre>Fetching 10 files: 100%|██████████| 10/10 [00:09&lt;00:00,  1.01it/s]<br>Generating train split: 200 examples [00:00, 476.92 examples/s]<br>Map: 100%|██████████| 200/200 [00:00&lt;00:00, 710.15 examples/s]<br>  0%|          | 0/25 [00:00&lt;?, ?it/s]...100%|██████████| 25/25 [00:43&lt;00:00,  1.73s/it]<br>{&#39;loss&#39;: 3.9413, &#39;grad_norm&#39;: 5.671746730804443, &#39;learning_rate&#39;: 3.2000000000000005e-05, &#39;epoch&#39;: 0.4}<br>{&#39;loss&#39;: 3.3486, &#39;grad_norm&#39;: 5.183830261230469, &#39;learning_rate&#39;: 1.2e-05, &#39;epoch&#39;: 0.8}<br>{&#39;train_runtime&#39;: 43.2565, &#39;train_samples_per_second&#39;: 4.624, &#39;train_steps_per_second&#39;: 0.578, &#39;train_loss&#39;: 3.5962399291992186, &#39;epoch&#39;: 1.0}</pre><p>Nine seconds to fetch the cached base model, 43 seconds to train, no download-shards wait, no checkpoint-shard loading wait, no peft/bnb merge warnings either, since there&#39;s no quantized base to merge back into anymore. Compare that to the original Llama attempt&#39;s multi-minute shard downloads before training even started: this is what dropping the whole 4-bit/8-bit merge path actually bought.</p><p>quantize_and_upload picked the merged model up straight from finetune-lora&#39;s model_out artifact in S3, visible in the step&#39;s executor input in the KFP run view:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sS9-k0QoUVz1nZb3.png" /><figcaption>quantize-and-upload executor input, <strong>model_in</strong> pointing at finetune-lora’s <strong>model_out</strong> in S3.</figcaption></figure><p>Before quantizing, AutoAWQ pulls its own default calibration dataset (~214k examples, separate from the fine-tuning data) to compute activation scales. That download isn’t cached to S3 like the base model, so it goes through NAT on every run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DX4trmeJ_zhBxzb2.png" /><figcaption><strong>AutoAWQ</strong> generating its calibration split in the quantize-and-upload logs.</figcaption></figure><p>Then it AWQ-quantized all 28 layers, about 10 minutes:</p><p>and pushed the result to S3:</p><pre>$ aws s3 ls s3://eks-llm-pipeline-model-bucket/pipeline-output/latest/ --human-readable<br>2026-10-22 22:18:51    1.5 GiB model.safetensors<br>2026-10-22 22:18:51      986 Bytes config.json<br>2026-10-22 22:18:54      7.3 KiB tokenizer_config.json<br>2026-10-22 22:18:54      6.7 MiB tokenizer.json</pre><pre>$ kubectl get workflow qwen-lora-finetune-lhrv5 -n kubeflow<br>NAME                       STATUS      AGE<br>qwen-lora-finetune-lhrv5   Succeeded   19m</pre><p>The entire pipeline works, and the trained model is stored in S3.</p><h3>Conclusion</h3><p>Kubeflow Pipelines now runs on the same EKS cluster as the earlier posts, using RDS for metadata and S3 for artifacts. It completes the workflow by turning fine-tuning, quantization, and model storage into a repeatable pipeline instead of a one-off script.</p><p>One major change was switching the base model. The original plan used Llama-3.1–8B throughout the series, but merging the fine-tuned adapter on a g4dn.xlarge GPU proved difficult. Rather than adding a larger GPU node pool just for this step, the pipeline switched to Qwen2.5-1.5B-Instruct, which avoids the memory issue.</p><p>There is still one limitation: the fine-tuned model does not automatically replace the model currently served by vllm_stack.py. Connecting them requires changing model_id, so the final integration is not yet automatic.</p><p>The process also involved several practical issues, including outdated image references, CloudFormation limitations, IRSA credential conflicts, leftover S3 resources, CDK configuration quirks, Python dependency conflicts, and Karpenter removing GPU nodes too early. None was particularly unusual, but each required proper debugging. The pipeline now runs end to end because these issues were investigated and fixed rather than simply worked around.</p><p>By the way, the diagram in this post also shows the chatbot (a Streamlit web app), which we haven’t covered here. We’ll get to it in the next post on hyperparameter tuning. Aight. Thanks for reading this one too, hope you found something useful 🚀</p><p>References:</p><ul><li><a href="https://www.kubeflow.org/">Kubeflow</a></li><li><a href="https://aws.amazon.com/blogs/storage/machine-learning-with-kubeflow-on-amazon-eks-with-amazon-efs">Machine Learning with Kubeflow on Amazon EKS with Amazon EFS</a></li><li><a href="https://awslabs.github.io/kubeflow-manifests/docs/deployment">AWS Labs: Deploy Kubeflow on AWS</a></li><li><a href="https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize/env/platform-agnostic-emissary">Kubeflow Pipelines manifests (kustomize/env/platform-agnostic-emissary)</a></li><li><a href="https://github.com/kubeflow/pipelines/tree/2.2.0/manifests/kustomize/cluster-scoped-resources">Kubeflow Pipelines manifests (kustomize/cluster-scoped-resources)</a></li><li><a href="https://github.com/awslabs/kubeflow-manifests/tree/main/awsconfigs/apps/pipeline">AWS Labs Kubeflow manifests (RDS + S3 overlay)</a></li><li><a href="https://www.flaticon.com/free-icon/user_6175043">Flaticon: User Icon</a></li></ul><p>Wanna see the full code? See the repo here:</p><p><a href="https://github.com/danielcristho/eks-cdk-llm">GitHub - danielcristho/eks-cdk-llm: A repository for exploring LLM Serving and MLOps practices on AWS Kubernetes (EKS) using Infrastructure as Code (AWS CDK).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f648868173fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM on EKS: Scaling with Ray & Karpenter]]></title>
            <link>https://danielpepuho.medium.com/llm-on-eks-scaling-with-ray-karpenter-fc71c30ad6f2?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc71c30ad6f2</guid>
            <category><![CDATA[karpenter]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[ray-framework]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 15:49:26 GMT</pubDate>
            <atom:updated>2026-07-10T16:17:55.181Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published on: <a href="https://danielcristho.site/blog/llm-on-eks-scale-llm">danielcristho.site</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*kINhwDPGvoDYG9K2jdHG6A.png" /></figure><p>Hi there, it’s been 2 months. Back in May, we deployed an <a href="https://danielpepuho.medium.com/llm-on-eks-serving-with-vllm-b97aeef5b804">LLM on Amazon EKS using vLLM</a>. It gave me a much better understanding of what it takes to serve an LLM in production.</p><p>But after getting it up and running, one question kept coming to mind: <em>What happens when a single GPU isn’t enough?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*65r3UX7gcSSxoiu8.gif" /><figcaption>Lindalee Cat GIF. Source: <a href="https://tenor.com/view/lindalee-cat-whathappened-gif-18419219"><em>tenor.com</em></a></figcaption></figure><blockquote>Sure, we could add more replicas. But those replicas still need GPU nodes, and keeping expensive GPU instances running 24/7 just in case traffic spikes doesn’t sound like a great idea.</blockquote><p>So in this post, we’ll extend the previous setup by introducing <a href="https://github.com/ray-project/ray">Ray</a> for distributed inference and <a href="https://github.com/aws/karpenter-provider-aws">Karpenter for AWS</a> for dynamic GPU node provisioning, allowing the cluster to scale up (and back down) as demand changes.</p><h3>TL;DR</h3><ul><li>Running distributed LLM inference with Ray</li><li>Autoscaling worker pods using KubeRay</li><li>Provisioning GPU nodes dynamically with Karpenter</li><li>Extending the previous vLLM deployment to scale on demand</li></ul><h3>Why Ray?</h3><p>Kubernetes can certainly scale pods, but serving LLMs comes with a few extra challenges. A single GPU can only handle so much, and once traffic grows, we need a way to distribute inference requests across multiple workers instead of relying on a single pod.</p><p><a href="https://github.com/ray-project/ray">Ray</a> is built for distributed AI workloads, making it natural fit here. I mainly chose it because:</p><ul><li><strong>Distributed inference</strong> — spreads requests across multiple workers instead of relying on a single replica.</li><li><strong>Built-in autoscaler</strong> — adjusts the number of Ray workers based on workload.</li><li><strong>Kubernetes-native</strong> — integrates nicely with EKS through <a href="https://github.com/ray-project/kuberay">KubeRay: Ray on Kubernetes</a>.</li></ul><p>Rather than treating each pod as an isolated deployment, Ray lets them work together as a single inference cluster.</p><h3>Why Karpenter?</h3><p>Ray can create additional worker pods, but it can’t magically create GPU node. If every GPU node is already busy, those worker pods will remain in the ‘Pending’ state until Kubernetes finds available capacity.</p><p>That’s where <a href="https://github.com/aws/karpenter-provider-aws">Karpenter</a> comes in:</p><ul><li><strong>Provision nodes on demand</strong> — launches new GPU instances when the cluster runs out of capacity.</li><li><strong>No fixed node groups</strong> — avoids keeping idle GPU instances running all day.</li><li><strong>Scale back down</strong> — automatically removes unused nodes to reduce infrastructure costs.</li></ul><p>Together, <a href="https://github.com/ray-project/ray">Ray</a> and <a href="https://github.com/aws/karpenter-provider-aws">Karpenter</a> solve different parts of the scaling problem. Ray decides <em>when</em> more workers are needed, while Karpenter makes sure there’s actually somewhere for those workers to run.</p><h3>What We Tryna Build</h3><p>This time, the architecture is a little different.</p><p>Instead of running a single vLLM pod, we’ll build a small Ray cluster on Amazon EKS. Ray will distribute inference requests across multiple workers, while Karpenter automatically provisions GPU nodes whenever the cluster needs more capacity.</p><p>The flow looks like this:</p><ul><li>User sends an inference request</li><li>Ray routes the request to an available worker</li><li>Workers run vLLM to generate responses</li><li>Ray creates additional workers as traffic increases</li><li>Karpenter provisions new GPU nodes when the cluster runs out of capacity</li><li>Unused workers and nodes are removed once traffic drops</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dlg8cwWXS1lr2gsH50Adyg.png" /><figcaption>Project Architecture</figcaption></figure><h3>The Stack</h3><p>We’ll reuse most of the infrastructure from the previous post. This time, we’re adding a few more components to make the deployment capable of scaling automatically.</p><ul><li><strong>Amazon EKS</strong> — runs the Kubernetes workloads, including Ray and vLLM.</li><li><strong>Amazon S3</strong> — stores artifacts such as model files used by the inference service.</li><li><strong>AWS CDK</strong> — provisions the infrastructure as code.</li><li><strong>Ray</strong> — distributes inference requests across multiple workers.</li><li><strong>Karpenter</strong> — provisions GPU nodes on demand.</li><li><strong>vLLM</strong> — serves the LLM through an OpenAI-compatible API.</li><li><strong>Meta Llama 3.1</strong> — the language model used for inference.</li><li><strong>NVIDIA Device Plugin</strong> — exposes GPU resources so workloads like vLLM can request and use NVIDIA GPUs.</li></ul><p>You can see the stack below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVc0xScKSt2RsXjFCSUgcg.png" /><figcaption>Stack Overview</figcaption></figure><h3>The Code</h3><p>We’ll reuse most of the infrastructure from the previous post and build on top of it. This project is organized into separate CDK stacks, with each stack responsible for a specific part of the deployment.</p><pre>scale_llm/<br>├── eks/<br>├── karpenter/<br>├── ray/<br>└── vllm/</pre><p>This separation keeps each component independent and easier to maintain. Let’s go through each stack and see what it does.</p><h3>EKS Stack</h3><p>The EksStack reuses the same cluster setup from the previous post, but this time there&#39;s only one node group instead of two.</p><pre># CPU nodes, runs system pods (CoreDNS, kube-proxy) and the Ray head<br>self.cluster.add_nodegroup_capacity(<br>    &quot;ManagedNodeGroup&quot;,<br>    desired_size=3,<br>    min_size=3,<br>    max_size=4,<br>    disk_size=50,<br>    instance_types=[ec2.InstanceType(&quot;t3.medium&quot;)],<br>    ami_type=eks.NodegroupAmiType.AL2023_X86_64_STANDARD,<br>    labels={&quot;role&quot;: &quot;system&quot;},<br>)</pre><p>No GPU node group this time. GPU capacity is fully dynamic now, Karpenter provisions and terminates GPU nodes on its own, so there’s nothing static to define here.</p><p>desired_size went from 2 to 3, and disk_size from the default 20GB to 50GB. The Ray head pod ended up sharing this node group with CoreDNS and kube-proxy, and once you add KubeRay&#39;s autoscaler sidecar (which rides along with the head pod), 2 nodes just didn&#39;t leave enough CPU headroom. The bigger disk is there because pulling Ray&#39;s CUDA image on a 20GB root volume was triggering disk-pressure evictions.</p><p>Since Karpenter creates its own EC2 instances and needs to register them with the cluster, it needs an IAM role of its own:</p><pre>self.karpenter_node_role = iam.Role(<br>    self, &quot;KarpenterNodeRole&quot;,<br>    assumed_by=iam.ServicePrincipal(&quot;ec2.amazonaws.com&quot;),<br>    managed_policies=[<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEKSWorkerNodePolicy&quot;),<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEC2ContainerRegistryReadOnly&quot;),<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEKS_CNI_Policy&quot;),<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonSSMManagedInstanceCore&quot;),<br>    ],<br>)</pre><p>The Karpenter controller itself also needs a role, one it assumes via IRSA so it can call the EC2 and EKS APIs:</p><pre>oidc_issuer = self.cluster.cluster_open_id_connect_issuer<br>irsa_condition = CfnJson(self, &quot;KarpenterIrsaCondition&quot;, value={<br>    f&quot;{oidc_issuer}:sub&quot;: &quot;system:serviceaccount:karpenter:karpenter&quot;,<br>    f&quot;{oidc_issuer}:aud&quot;: &quot;sts.amazonaws.com&quot;,<br>})</pre><pre>self.karpenter_controller_role = iam.Role(<br>    self, &quot;KarpenterControllerRole&quot;,<br>    assumed_by=iam.FederatedPrincipal(<br>        federated=self.cluster.open_id_connect_provider.open_id_connect_provider_arn,<br>        assume_role_action=&quot;sts:AssumeRoleWithWebIdentity&quot;,<br>        conditions={&quot;StringEquals&quot;: irsa_condition},<br>    ),<br>)</pre><pre># Karpenter talks to the K8s API server as this IAM role, same as `aws eks get-token` does,<br># not through its pod&#39;s ServiceAccount token — so it also needs an aws-auth mapping.<br>self.cluster.aws_auth.add_role_mapping(self.karpenter_controller_role, groups=[&quot;system:masters&quot;])</pre><p>Last, we tag the private subnets so Karpenter knows where it’s allowed to launch nodes:</p><pre>for subnet in self.vpc.private_subnets:<br>    Tags.of(subnet).add(&quot;karpenter.sh/discovery&quot;, cluster_name)</pre><p>A few things worth noting:</p><ul><li>IRSA alone isn’t enough for the controller, it also needs the aws-auth role mapping, otherwise it gets stuck on &quot;the server has asked for the client to provide credentials&quot;</li><li>the cluster’s security group is passed to the Karpenter stack by ID instead of by tag — Tags.of(cluster.cluster_security_group) is a silent no-op in CDK since it&#39;s an imported reference, not a resource this stack owns</li><li>karpenter_node_role also gets a matching aws-auth mapping (system:bootstrappers / system:nodes), same as any other node group&#39;s role</li></ul><p>The system node group is now:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31e1665766673128ddf0bd6f9b6230ea/href">https://medium.com/media/31e1665766673128ddf0bd6f9b6230ea/href</a></iframe><blockquote>GPU nodes are launched by Karpenter, so there’s no fixed count, they only exist while there’s demand for them.</blockquote><h3>Karpenter Stack</h3><p>The KarpenterStack installs the Karpenter controller via Helm, then defines the two CRDs that tell it what to provision: an EC2NodeClass (AMI, subnets, security group, IAM role) and a NodePool (instance types, limits, disruption behavior).</p><pre>karpenter_chart = cluster.add_helm_chart(<br>    &quot;Karpenter&quot;,<br>    chart=&quot;karpenter&quot;,<br>    repository=&quot;oci://public.ecr.aws/karpenter/karpenter&quot;,<br>    version=&quot;1.4.0&quot;,<br>    namespace=&quot;karpenter&quot;,<br>    create_namespace=True,<br>    values={<br>        &quot;serviceAccount&quot;: {<br>            &quot;name&quot;: &quot;karpenter&quot;,<br>            &quot;annotations&quot;: {&quot;eks.amazonaws.com/role-arn&quot;: karpenter_controller_role.role_arn},<br>        },<br>        &quot;settings&quot;: {&quot;clusterName&quot;: cluster_name},<br>    },<br>)</pre><p>serviceAccount.name is pinned to &quot;karpenter&quot; instead of letting Helm auto-generate one from the release name. The IRSA trust policy back in EksStack hardcodes system:serviceaccount:karpenter:karpenter as the expected subject, so if the actual SA name doesn&#39;t match that exactly, the pod fails sts:AssumeRoleWithWebIdentity with an access denied.</p><p>Next, the EC2NodeClass:</p><pre>ec2_node_class = cluster.add_manifest(&quot;GpuEC2NodeClass&quot;, {<br>    &quot;apiVersion&quot;: &quot;karpenter.k8s.aws/v1&quot;,<br>    &quot;kind&quot;: &quot;EC2NodeClass&quot;,<br>    &quot;metadata&quot;: {&quot;name&quot;: &quot;gpu&quot;},<br>    &quot;spec&quot;: {<br>        &quot;amiSelectorTerms&quot;: [{&quot;alias&quot;: &quot;al2023@latest&quot;}],<br>        &quot;subnetSelectorTerms&quot;: [{&quot;tags&quot;: {&quot;karpenter.sh/discovery&quot;: cluster_name}}],<br>        &quot;securityGroupSelectorTerms&quot;: [{&quot;id&quot;: cluster_security_group_id}],<br>        &quot;role&quot;: karpenter_node_role.role_name,<br>        &quot;blockDeviceMappings&quot;: [{<br>            &quot;deviceName&quot;: &quot;/dev/xvda&quot;,<br>            &quot;ebs&quot;: {&quot;volumeSize&quot;: &quot;100Gi&quot;, &quot;volumeType&quot;: &quot;gp3&quot;, &quot;encrypted&quot;: True, &quot;deleteOnTermination&quot;: True},<br>        }],<br>    },<br>})</pre><p>And the NodePool:</p><pre>gpu_node_pool = cluster.add_manifest(&quot;GpuNodePool&quot;, {<br>    &quot;apiVersion&quot;: &quot;karpenter.sh/v1&quot;,<br>    &quot;kind&quot;: &quot;NodePool&quot;,<br>    &quot;metadata&quot;: {&quot;name&quot;: &quot;gpu&quot;},<br>    &quot;spec&quot;: {<br>        &quot;template&quot;: {<br>            &quot;metadata&quot;: {&quot;labels&quot;: {&quot;workload&quot;: &quot;gpu&quot;}},<br>            &quot;spec&quot;: {<br>                &quot;nodeClassRef&quot;: {&quot;group&quot;: &quot;karpenter.k8s.aws&quot;, &quot;kind&quot;: &quot;EC2NodeClass&quot;, &quot;name&quot;: &quot;gpu&quot;},<br>                &quot;requirements&quot;: [<br>                    {&quot;key&quot;: &quot;node.kubernetes.io/instance-type&quot;, &quot;operator&quot;: &quot;In&quot;, &quot;values&quot;: [&quot;g4dn.xlarge&quot;]},<br>                    {&quot;key&quot;: &quot;karpenter.sh/capacity-type&quot;, &quot;operator&quot;: &quot;In&quot;, &quot;values&quot;: [&quot;on-demand&quot;]},<br>                ],<br>                &quot;taints&quot;: [{&quot;key&quot;: &quot;nvidia.com/gpu&quot;, &quot;value&quot;: &quot;true&quot;, &quot;effect&quot;: &quot;NoSchedule&quot;}],<br>            },<br>        },<br>        &quot;limits&quot;: {&quot;nvidia.com/gpu&quot;: &quot;2&quot;},<br>        &quot;disruption&quot;: {&quot;consolidationPolicy&quot;: &quot;WhenEmpty&quot;, &quot;consolidateAfter&quot;: &quot;30s&quot;},<br>    },<br>})</pre><p>A few things worth noting:</p><ul><li>spec.role (not spec.instanceProfile) tells Karpenter to create and manage its own IAM instance profile at runtime, that&#39;s why EksStack grants the controller role extra iam:CreateInstanceProfile / iam:GetInstanceProfile permissions</li><li>the nvidia.com/gpu=true:NoSchedule taint keeps everything except GPU workloads off these nodes, it&#39;s matched by a toleration on the Ray GPU worker pods in VllmStack</li><li>limits.nvidia.com/gpu: &quot;2&quot; caps Karpenter at 2 GPU nodes at a time, matching this account&#39;s GPU instance quota</li><li>consolidateAfter: &quot;30s&quot; terminates idle GPU nodes fast, g4dn.xlarge isn&#39;t something you want sitting idle</li></ul><h3>Ray Stack</h3><p>The RayStack is small, it only installs the KubeRay operator:</p><pre>self.kuberay_operator = cluster.add_helm_chart(<br>    &quot;KubeRayOperator&quot;,<br>    chart=&quot;kuberay-operator&quot;,<br>    repository=&quot;https://ray-project.github.io/kuberay-helm/&quot;,<br>    version=&quot;1.3.0&quot;,<br>    namespace=&quot;kuberay&quot;,<br>    create_namespace=True,<br>    values={<br>        &quot;resources&quot;: {<br>            &quot;requests&quot;: {&quot;cpu&quot;: &quot;100m&quot;, &quot;memory&quot;: &quot;128Mi&quot;},<br>            &quot;limits&quot;: {&quot;cpu&quot;: &quot;500m&quot;, &quot;memory&quot;: &quot;256Mi&quot;},<br>        }<br>    },<br>)</pre><p>The operator itself doesn’t run a Ray cluster, it watches for RayCluster / RayService custom resources and reconciles whatever it finds. The actual head + worker cluster is defined in VllmStack, as part of the RayService spec below.</p><p>self.kuberay_operator is exposed from this stack so VllmStack can declare an explicit dependency on it (node.add_dependency(kuberay_operator)). Without that, CDK has no way of knowing the kuberay namespace and its CRDs need to exist before the manifests in VllmStack are applied, and they&#39;d occasionally get applied out of order.</p><h3>vLLM Stack</h3><p>The VllmStack ties everything together. It installs the NVIDIA device plugin, ships the Ray Serve entrypoint script as a ConfigMap, and defines a RayService that runs vLLM as an autoscaling Ray Serve deployment.</p><pre>nvidia_plugin = cluster.add_helm_chart(<br>    &quot;NvidiaDevicePlugin&quot;,<br>    chart=&quot;nvidia-device-plugin&quot;,<br>    repository=&quot;https://nvidia.github.io/k8s-device-plugin&quot;,<br>    version=&quot;0.17.1&quot;,<br>    namespace=&quot;kube-system&quot;,<br>    values={<br>        &quot;nodeSelector&quot;: {&quot;workload&quot;: &quot;gpu&quot;},<br>        &quot;tolerations&quot;: [{&quot;key&quot;: &quot;nvidia.com/gpu&quot;, &quot;operator&quot;: &quot;Exists&quot;, &quot;effect&quot;: &quot;NoSchedule&quot;}],<br>    },<br>)</pre><p>Same as the previous post, this is what makes nvidia.com/gpu requestable as a resource on whatever GPU nodes Karpenter launches.</p><p>The Ray Serve app script (serve_vllm.py) gets mounted into the containers through a ConfigMap:</p><pre>serve_configmap = cluster.add_manifest(&quot;VllmServeConfigMap&quot;, {<br>    &quot;apiVersion&quot;: &quot;v1&quot;,<br>    &quot;kind&quot;: &quot;ConfigMap&quot;,<br>    &quot;metadata&quot;: {&quot;name&quot;: &quot;vllm-serve-script&quot;, &quot;namespace&quot;: &quot;kuberay&quot;},<br>    &quot;data&quot;: {&quot;serve_vllm.py&quot;: _load_serve_script()},<br>})<br>serve_configmap.node.add_dependency(kuberay_operator)</pre><p>Then the RayService itself, this is the bulk of the stack:</p><pre>ray_service = cluster.add_manifest(&quot;VllmRayService&quot;, {<br>    &quot;apiVersion&quot;: &quot;ray.io/v1&quot;,<br>    &quot;kind&quot;: &quot;RayService&quot;,<br>    &quot;metadata&quot;: {&quot;name&quot;: &quot;vllm&quot;, &quot;namespace&quot;: &quot;kuberay&quot;},<br>    &quot;spec&quot;: {<br>        &quot;serveConfigV2&quot;: (<br>            &quot;applications:\n&quot;<br>            &quot;  - name: vllm\n&quot;<br>            &quot;    import_path: serve_vllm:build_app\n&quot;<br>            &quot;    route_prefix: /\n&quot;<br>            &quot;    runtime_env:\n&quot;<br>            &quot;      working_dir: /serve-scripts\n&quot;<br>            &quot;      pip:\n&quot;<br>            &quot;        - vllm==0.8.5\n&quot;<br>            f&quot;    args:\n&quot;<br>            f&quot;      model: \&quot;{model_id}\&quot;\n&quot;<br>            &quot;      dtype: half\n&quot;<br>            &quot;      quantization: awq\n&quot;<br>            &quot;      max_model_len: 4096\n&quot;<br>            &quot;      download_dir: /model-cache\n&quot;<br>            &quot;    deployments:\n&quot;<br>            &quot;      - name: VLLMDeployment\n&quot;<br>            &quot;        num_replicas: 1\n&quot;<br>            &quot;        max_ongoing_requests: 5\n&quot;<br>            &quot;        autoscaling_config:\n&quot;<br>            &quot;          min_replicas: 0\n&quot;<br>            &quot;          max_replicas: 2\n&quot;<br>            &quot;          target_ongoing_requests: 2\n&quot;<br>            &quot;        ray_actor_options:\n&quot;<br>            &quot;          num_gpus: 1\n&quot;<br>        ),<br>        &quot;rayClusterConfig&quot;: {<br>            &quot;rayVersion&quot;: &quot;2.44.1&quot;,<br>            &quot;enableInTreeAutoscaling&quot;: True,<br>            &quot;headGroupSpec&quot;: {<br>                &quot;rayStartParams&quot;: {&quot;dashboard-host&quot;: &quot;0.0.0.0&quot;, &quot;num-gpus&quot;: &quot;0&quot;},<br>                &quot;template&quot;: {<br>                    &quot;spec&quot;: {<br>                        &quot;nodeSelector&quot;: {&quot;role&quot;: &quot;system&quot;},<br>                        &quot;containers&quot;: [{<br>                            &quot;name&quot;: &quot;ray-head&quot;,<br>                            &quot;image&quot;: &quot;rayproject/ray:2.44.1-gpu&quot;,<br>                            &quot;resources&quot;: {&quot;requests&quot;: {&quot;cpu&quot;: &quot;1&quot;, &quot;memory&quot;: &quot;2Gi&quot;}},<br>                        }],<br>                    },<br>                },<br>            },<br>            &quot;workerGroupSpecs&quot;: [{<br>                &quot;groupName&quot;: &quot;gpu-workers&quot;,<br>                &quot;replicas&quot;: 0,<br>                &quot;minReplicas&quot;: 0,<br>                &quot;maxReplicas&quot;: 2,<br>                &quot;rayStartParams&quot;: {&quot;num-gpus&quot;: &quot;1&quot;},<br>                &quot;template&quot;: {<br>                    &quot;spec&quot;: {<br>                        &quot;tolerations&quot;: [{&quot;key&quot;: &quot;nvidia.com/gpu&quot;, &quot;operator&quot;: &quot;Exists&quot;, &quot;effect&quot;: &quot;NoSchedule&quot;}],<br>                        &quot;nodeSelector&quot;: {&quot;workload&quot;: &quot;gpu&quot;},<br>                        &quot;containers&quot;: [{<br>                            &quot;name&quot;: &quot;ray-worker&quot;,<br>                            &quot;image&quot;: &quot;rayproject/ray:2.44.1-gpu&quot;,<br>                            &quot;resources&quot;: {<br>                                &quot;requests&quot;: {&quot;cpu&quot;: &quot;2&quot;, &quot;memory&quot;: &quot;12Gi&quot;},<br>                                &quot;limits&quot;: {&quot;cpu&quot;: &quot;4&quot;, &quot;memory&quot;: &quot;14Gi&quot;, &quot;nvidia.com/gpu&quot;: &quot;1&quot;},<br>                            },<br>                        }],<br>                    },<br>                },<br>            }],<br>        },<br>    },<br>})</pre><p>A few things worth noting:</p><ul><li>min_replicas: 0 / max_replicas: 2 on the deployment&#39;s autoscaling_config is what drives everything else, Ray Serve scales vLLM replicas based on target_ongoing_requests, and each new replica needs a GPU worker pod, which needs a GPU node, which is where Karpenter comes in</li><li>enableInTreeAutoscaling: true turns on KubeRay&#39;s autoscaler sidecar in the head pod, that&#39;s the thing actually creating/removing worker pods, separate from Ray Serve&#39;s own replica autoscaling above it</li><li>worker replicas start at 0, so no GPU node exists at deploy time, Karpenter only provisions one once a worker pod goes Pending</li><li>the head pod runs on the system node group (nodeSelector: role: system) with num-gpus: 0, it only routes and manages the cluster, it doesn&#39;t need a GPU itself</li><li>image is rayproject/ray:2.44.1-gpu, not ray-ml:2.44.1-gpu, that ray-ml tag turned out to be deprecated on Docker Hub for this version</li></ul><p>There’s a catch with scaling GPU workers to zero: every time Karpenter provisions a fresh node, download_dir: /model-cache starts out empty, so vLLM re-downloads the ~5.7GB model from HuggingFace on every scale-up. Fine occasionally, annoying if traffic is spiky and nodes keep cycling. So the worker pod also gets an initContainer that pulls a cached copy from S3 first:</p><pre>&quot;initContainers&quot;: [<br>    {<br>        &quot;name&quot;: &quot;model-cache-sync&quot;,<br>        &quot;image&quot;: &quot;amazon/aws-cli:2.17.62&quot;,<br>        &quot;command&quot;: [<br>            &quot;sh&quot;, &quot;-c&quot;,<br>            &quot;aws s3 sync s3://$MODEL_BUCKET/model-cache /model-cache || true&quot;,<br>        ],<br>        &quot;env&quot;: [<br>            {&quot;name&quot;: &quot;AWS_DEFAULT_REGION&quot;, &quot;value&quot;: self.region},<br>            {&quot;name&quot;: &quot;MODEL_BUCKET&quot;,       &quot;value&quot;: model_bucket_name},<br>        ],<br>        &quot;volumeMounts&quot;: [{&quot;name&quot;: &quot;model-cache&quot;, &quot;mountPath&quot;: &quot;/model-cache&quot;}],<br>    }<br>],</pre><p>The || true matters, it makes this best-effort: an empty or missing bucket just falls through to the normal HuggingFace download, nothing breaks on first run. The GPU node&#39;s IAM role (karpenter_node_role) already has grant_read on the model bucket from EksStack, so no extra permissions are needed here.</p><p>This only helps once the bucket actually has something in it, though, S3 doesn’t populate itself. After the first worker pod finishes downloading the model, seed the cache once:</p><pre>kubectl cp kuberay/&lt;worker-pod&gt;:/model-cache ./model-cache<br>aws s3 sync ./model-cache s3://&lt;your-bucket&gt;/model-cache</pre><p>After that, every future scale-up pulls from S3 (fast, same-region, effectively free) instead of hitting HuggingFace over the internet.</p><p>Last, the same NLB pattern as before, just pointed at the Ray head’s Serve port instead of a plain vLLM pod:</p><pre>nlb_svc = cluster.add_manifest(&quot;VllmNlbService&quot;, {<br>    &quot;apiVersion&quot;: &quot;v1&quot;,<br>    &quot;kind&quot;: &quot;Service&quot;,<br>    &quot;metadata&quot;: {<br>        &quot;name&quot;: &quot;vllm-nlb&quot;,<br>        &quot;namespace&quot;: &quot;kuberay&quot;,<br>        &quot;annotations&quot;: {<br>            &quot;service.beta.kubernetes.io/aws-load-balancer-type&quot;: &quot;nlb&quot;,<br>            &quot;service.beta.kubernetes.io/aws-load-balancer-scheme&quot;: &quot;internet-facing&quot;,<br>        },<br>    },<br>    &quot;spec&quot;: {<br>        &quot;type&quot;: &quot;LoadBalancer&quot;,<br>        &quot;selector&quot;: {&quot;role&quot;: &quot;ray-head&quot;},<br>        &quot;ports&quot;: [{&quot;port&quot;: 80, &quot;targetPort&quot;: 8000}],<br>    },<br>})</pre><h3>Deploy</h3><pre>cdk bootstrap   # first time only<br>cdk deploy --all</pre><p>This deploys the four stacks in order: eks-stack → karpenter-stack → ray-stack → vllm-stack. The cluster itself takes the longest, 15-20 minutes, so grab a coffee.</p><p>Once eks-stack is up, the system node group is there:</p><pre>$ kubectl get nodes<br>NAME                          STATUS   ROLES    AGE   VERSION<br>ip-10-0-13-44.ec2.internal    Ready    &lt;none&gt;   4m    v1.34.7-eks-40737a8<br>ip-10-0-31-9.ec2.internal     Ready    &lt;none&gt;   4m    v1.34.7-eks-40737a8<br>ip-10-0-52-201.ec2.internal   Ready    &lt;none&gt;   4m    v1.34.7-eks-40737a8</pre><p>Three CPU nodes, no GPU nodes. Nothing’s asking for a GPU yet, so Karpenter hasn’t provisioned anything.</p><p>Once karpenter-stack is up, the controller should be running:</p><pre>$ kubectl get pods -n karpenter<br>NAME                                                         READY   STATUS    RESTARTS   AGE<br>eksstackeksllmscalechartkarpenter29d18760-7b8d7cbf68-bhtfd   1/1     Running   0          11m<br>eksstackeksllmscalechartkarpenter29d18760-7b8d7cbf68-nmh8m   1/1     Running   0          11m</pre><p>And once ray-stack + vllm-stack finish, the Ray head comes up on the system nodes:</p><pre>$ kubectl get pods -n kuberay<br>NAME                        READY   STATUS    RESTARTS   AGE<br>kuberay-operator-...         1/1    Running   0          5m<br>vllm-raycluster-...-head     2/2    Running   0          3m</pre><p>No worker pod yet, RayService was deployed with replicas: 0. This is also where things stopped being straightforward. On paper, the head comes up, Serve builds the vllm app, and you&#39;re done — in practice, it took five redeploys to get there, none of them infrastructure problems, all of them Ray Serve / Helm gotchas that only show up once something actually tries to run:</p><pre>$ kubectl get rayservice vllm -n kuberay<br>NAME   SERVICE STATUS   NUM SERVE ENDPOINTS<br>vllm</pre><p>Empty SERVICE STATUS isn&#39;t a good sign. The real status lives behind the head&#39;s dashboard API, not in this column:</p><pre>kubectl run -n kuberay checkstatus --image=curlimages/curl:8.10.1 --restart=Never -i --rm --timeout=20s -- \<br>  curl -s http://&lt;raycluster&gt;-head-svc.kuberay.svc.cluster.local:8265/api/serve/applications/<br># look at .applications.vllm.status and .message</pre><blockquote>What that turned up, in the order they surfaced</blockquote><blockquote><strong>working_dir: /serve-scripts rejected.</strong> Ray Serve&#39;s runtime_env.working_dir only accepts remote URIs (file://, s3://, …), not a bare local path — even though that&#39;s exactly where the ConfigMap mounts the script. Fix: drop working_dir entirely, set PYTHONPATH: /serve-scripts under runtime_env.env_vars instead, so import_path: serve_vllm:build_app still resolves.</blockquote><blockquote><strong>num_replicas + </strong><strong>autoscaling_config together.</strong> Ray Serve won&#39;t accept a fixed replica count alongside an autoscaling config — drop num_replicas, autoscaling owns replica count now.</blockquote><blockquote><strong>KubeRay caches </strong><strong>serveConfigV2 by its own text, not by what&#39;s mounted.</strong> Editing serve_vllm.py alone (the ConfigMap) didn&#39;t get picked up — operator logs showed &quot;shouldUpdate&quot;: false, reason: &quot;Current V2 Serve config matches cached Serve config.&quot; on every reconcile. It only re-submits when the YAML string itself changes. Fix: fold a hash of the script into an (unused) arg in args:, so any script edit changes the YAML and busts the cache automatically.</blockquote><blockquote><strong>The NVIDIA device plugin never scheduled at all.</strong> With the app finally RUNNING, the GPU worker pod sat Pending — Insufficient nvidia.com/gpu. The nvidia-device-plugin Helm chart ships a default nodeAffinity requiring an NFD/GPU-Operator label (nvidia.com/gpu.present, among others) that nothing in this cluster sets, so it never matched our Karpenter-provisioned node. Passing affinity: {} to override it turned out to be a no-op (Helm doesn&#39;t let an empty map clear a chart&#39;s default nested value) — the actual fix was adding nvidia.com/gpu.present and nvidia.com/mps.capable labels directly on the Karpenter GPU NodePool&#39;s node template.</blockquote><p>None of this was visible from kubectl get pods — the head and worker pods looked healthy the entire time. The only way to see any of it was polling /api/serve/applications/ directly.</p><p>Once all of the above landed, a fresh cdk deploy eks-stack finally gets a clean result:</p><pre>$ curl -s http://&lt;raycluster&gt;-head-svc.kuberay.svc.cluster.local:8265/api/serve/applications/ | jq &#39;.applications.vllm.status&#39;<br>&quot;RUNNING&quot;</pre><p>Now send it a request (see <a href="https://alanhe421.github.io/markdown-to-medium-tool/#inference">Inference</a> below) and watch the autoscaler react. The worker pod shows up first, Pending, since there&#39;s no GPU node to put it on:</p><pre>$ kubectl get pods -n kuberay -w<br>vllm-raycluster-...-worker-gpu-xxxx   0/1   Pending   0   3s</pre><p>Karpenter picks that up within a few seconds and launches a g4dn.xlarge:</p><pre>$ kubectl get nodes -l workload=gpu -w<br>NAME                          STATUS     ROLES    AGE   VERSION<br>ip-10-0-44-171.ec2.internal   NotReady   &lt;none&gt;   8s    v1.34.7-eks-40737a8<br>ip-10-0-44-171.ec2.internal   Ready      &lt;none&gt;   38s   v1.34.7-eks-40737a8</pre><p>Once the node is Ready, the worker pod schedules and vLLM starts loading the model on it, same download-and-load as the previous post, just happening on a node that didn&#39;t exist a minute ago:</p><pre>$ kubectl logs -f -n kuberay -l ray.io/group=gpu-workers -c ray-worker<br>...<br>INFO 07-08 ... non-default args: {&#39;model&#39;: &#39;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&#39;, &#39;dtype&#39;: &#39;half&#39;, &#39;quantization&#39;: &#39;awq&#39;, &#39;max_model_len&#39;: 4096}<br>...</pre><p>Stop sending traffic, and once Ray Serve scales the deployment back down to 0 replicas, the worker pod goes away, and 30 seconds later (consolidateAfter: &quot;30s&quot;) so does the GPU node:</p><pre>$ kubectl get nodes -l workload=gpu<br>No resources found</pre><p>That’s the whole loop: no GPU node running until there’s real demand for one, and nothing left running once demand disappears.</p><blockquote>Run cdk destroy --all when you&#39;re done. Karpenter-provisioned nodes aren&#39;t tracked by CloudFormation the same way a static node group is, so double check with kubectl get nodes and the EC2 console that nothing GPU-shaped got left behind.</blockquote><h3>Inference</h3><p>Nothing fancy here yet — just a couple of plain curl requests straight to the NLB, one replica, one request at a time. The point right now is just proving the whole chain (NLB → Ray Serve → vLLM → GPU worker) actually answers. Scaling multiple replicas under concurrent load is its own section further down.</p><p>Grab the NLB endpoint:</p><pre>$ kubectl get svc vllm-nlb -n kuberay -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;<br>a8ad896440d644814b07e7628475888d-0452cb7d4da839df.elb.us-east-1.amazonaws.com</pre><p>Check the model’s loaded:</p><pre>$ curl http://&lt;nlb-endpoint&gt;/v1/models<br>{&quot;object&quot;:&quot;list&quot;,&quot;data&quot;:[{&quot;id&quot;:&quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,&quot;object&quot;:&quot;model&quot;,&quot;owned_by&quot;:&quot;vllm&quot;}]}</pre><p>That one worked on the first try. Sending an actual prompt didn’t:</p><pre>curl http://&lt;nlb-endpoint&gt;/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,<br>    &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is CAP theorem?&quot;}],<br>    &quot;max_tokens&quot;: 150<br>  }&#39;</pre><p>/v1/models only reads model_config, but /v1/chat/completions builds vLLM&#39;s OpenAIServingChat on first use — and that&#39;s where it broke, twice, both API changes in this vLLM version that nothing catches until a real chat request hits it:</p><ul><li>AsyncEngineArgs(worker_use_ray=False) — worker_use_ray no longer exists; dropped it (single GPU, vLLM was never going to spin up its own Ray workers anyway).</li><li>OpenAIServingChat(..., served_model_names=[...]) — that argument&#39;s gone too, replaced by a required OpenAIServingModels object built from a BaseModelPath.</li></ul><p>With both fixed, the same request goes through. Time for a slightly more fun prompt than usual:</p><pre>curl http://&lt;nlb-endpoint&gt;/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,<br>    &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Three boxes are labeled: &#39;\&#39;&#39;APPLES&#39;\&#39;&#39;, &#39;\&#39;&#39;ORANGES&#39;\&#39;&#39;, and &#39;\&#39;&#39;APPLES AND ORANGES&#39;\&#39;&#39;. All labels are incorrect. If you can only pull one fruit from one box, how do you correctly label all boxes? Show me the steps&quot;}],<br>    &quot;max_tokens&quot;: 512<br>  }&#39;</pre><pre>{<br>  &quot;id&quot;: &quot;chatcmpl-89ac1890-c23e-4bb7-85a1-0fcef1f405b8&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;choices&quot;: [{<br>    &quot;message&quot;: {<br>      &quot;role&quot;: &quot;assistant&quot;,<br>      &quot;content&quot;: &quot;To correctly label all the boxes, we need to find a fruit that is unique to one box and can be used to determine the correct labels.\n\nStep 1: \nPull a fruit from the &#39;APPLES AND ORANGES&#39; box. Since it&#39;s labeled incorrectly, it might contain either an apple or an orange. Let&#39;s assume we pull an orange from this box.\n\nStep 2: \nNow, we have an orange from the &#39;APPLES AND ORANGES&#39; box. We know that the &#39;ORANGES&#39; box cannot be the correct label for this box since we already have an orange from it...\n\n[...]\n\nHere are the correct labels for the boxes:\n\n- &#39;APPLES AND ORANGES&#39; box: APPLES AND ORANGES\n- &#39;ORANGES&#39; box: APPLES\n- &#39;APPLES&#39; box: ORANGES AND APPLES&quot;<br>    },<br>    &quot;finish_reason&quot;: &quot;stop&quot;<br>  }],<br>  &quot;usage&quot;: {&quot;prompt_tokens&quot;: 84, &quot;completion_tokens&quot;: 509, &quot;total_tokens&quot;: 593}<br>}</pre><p>A real round trip — request in, answer out, routed through Ray Serve and served off a GPU node that didn’t exist a couple minutes earlier. Worth noting: the reasoning here is actually wrong (it ends up relabeling boxes with labels that were never on any box to begin with, and the trick only ever requires pulling from the “APPLES AND ORANGES” box) — that’s the 8B AWQ-quantized model being a bit small for multi-step logic puzzles, not an infra problem. The pipe worked; the model just isn’t that smart.</p><h3>Does It Actually Scale?</h3><p>Same idea as the previous post — a Streamlit chatbot on top of the OpenAI-compatible endpoint, chatting away just fine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3jacmUY2qNelscZt.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8oVTZVOfDPOB3xEI.png" /></figure><p>But that’s also exactly why it’s the wrong tool for checking whether autoscaling actually works. A chat UI sends one message, waits for the full response, then sends the next — always one request in flight, never more. target_ongoing_requests: 2 only kicks in once more requests are <em>in flight simultaneously</em> than a single replica can comfortably handle, and no amount of typing quickly into Streamlit will ever produce that. So to actually see the autoscaler react, 8 requests went out at once via curl instead of one at a time through the UI:</p><pre>for i in 1 2 3 4 5 6 7 8; do<br>  curl -s -o resp$i.json -w &quot;req$i: %{http_code} %{time_total}s\n&quot; \<br>    -X POST http://&lt;nlb-endpoint&gt;/v1/chat/completions \<br>    -H &quot;Content-Type: application/json&quot; -d @q$i.json &amp;<br>done<br>wait</pre><pre>req4: 200 16.97s<br>req7: 200 17.14s<br>req6: 200 17.13s<br>req5: 200 17.14s<br>req2: 200 17.31s<br>req1: 200 31.27s<br>req3: 200 32.33s<br>req8: 200 32.31s</pre><p>Five came back in ~17s — that’s max_ongoing_requests: 5 per replica, so the first five just ran. The other three queued behind them and took almost double, ~32s, waiting for a slot to free up. Meanwhile, this showed up in another terminal:</p><pre>$ kubectl get pods -n kuberay<br>vllm-raycluster-...-gpu-workers-worker-4kdlf   0/1   Pending   0   1s</pre><p>A second worker pod, right on cue:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*sPmXUbcP9yh_OsCJ.png" /></figure><p>Watched through Pending -&gt; Init -&gt; PodInitializing-&gt; Running, while a second GPU node shows up right alongside it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*YXXO3emt7gXhbCWB.png" /></figure><p>Timing it out end to end, from Ray Serve first asking for a 2nd replica to that replica actually serving:</p><p>Stage Time Elapsed Ray Serve requests a 2nd replica (worker pod Pending) 11:02:29 — Karpenter&#39;s new GPU node Ready 11:03:17 ~48s Worker pod container Running (scheduled + image pulled) 11:06:19 ~3m vLLM finishes loading and the 2nd replica goes RUNNING 11:11:56 ~5m37s <strong>Total, request to serving</strong> <strong>~9m27s</strong></p><p>Provisioning the node itself was the fast part. The two things that actually ate the clock were pulling the 6GB rayproject/ray:2.43.0-gpu image on a brand new node (3 minutes) and vLLM&#39;s own model load + CUDA graph capture (~5.5 minutes) — the same cold-start cost from <a href="https://alanhe421.github.io/markdown-to-medium-tool/#deploy">Deploy</a>, just paid a second time for the second replica. This is roughly the real cost of &quot;scale on demand&quot; here: cheap to provision, ~9-10 minutes before a new GPU replica is actually useful. Fine for a slow ramp in traffic, not something you&#39;d want to lean on for a sudden spike.</p><p>If that ~9m27s ever needed to come down, none of it’s Karpenter’s fault — the node itself was ready in under a minute. The obvious next levers:</p><ul><li><strong>Bake a custom AMI with the image pre-pulled.</strong> Most of the ~3 minutes is containerd pulling a ~6GB image on a brand new instance. An AMI built with rayproject/ray:2.43.0-gpu already cached (EC2 Image Builder, or just a snapshot) turns that into a no-op.</li><li><strong>Skip CUDA graph capture.</strong> enforce_eager=True on the AsyncEngineArgs trades a bit of per-request latency for skipping the ~37s graph capture step entirely — worth it if replicas come and go often.</li><li><strong>Actually seed the S3 model cache.</strong> The initContainer from <a href="https://alanhe421.github.io/markdown-to-medium-tool/#vllm-stack">vLLM Stack</a> is already wired up, it just never got seeded this session — a populated bucket turns the ~37s HuggingFace download into a same-region S3 pull instead.</li><li><strong>Keep one warm replica instead of scaling from zero.</strong> min_replicas: 1 means one GPU node running 24/7 (some idle cost) in exchange for the first request never paying this ~9 minute tax — a reasonable trade if traffic is bursty but rarely fully idle.</li></ul><p>None of these were worth doing for this post, they only pay off under real, sustained traffic, but they’re the obvious next lever if ~9m27s to scale up ever becomes the actual bottleneck.</p><h3>Conclusion</h3><p>The opening question was: what happens when a single GPU isn’t enough? Now there’s an actual answer: Ray Serve notices, KubeRay asks for another node, Karpenter provisions a g4dn.xlarge, and ~9 minutes later there&#39;s a second replica taking traffic — no capacity sitting around unused, no manual intervention.</p><p>That ~9 minutes is also the honest part of this post. Autoscaling GPU inference isn’t free of cold starts, it just moves them from “always paid, all day” to “paid once, on demand.” Whether that trade is worth it depends entirely on how bursty the traffic actually is — fine for a slow ramp, not something to lean on for a sudden spike.</p><p>The bigger lesson, though, was everywhere in <a href="https://alanhe421.github.io/markdown-to-medium-tool/#deploy">Deploy</a>: almost none of the real bugs were infrastructure problems. CDK deployed cleanly every time. It was Ray Serve config quirks, a Helm chart’s assumptions about a cluster we don’t run, and vLLM API signatures that moved between versions — all invisible from kubectl get pods, all only surfacing once something actually tried to run. Autoscaling on Kubernetes is very achievable; getting there means budgeting time for exactly this kind of debugging, not just writing the YAML.</p><blockquote>Run cdk destroy --all when you&#39;re done. Karpenter-provisioned nodes aren&#39;t tracked by CloudFormation the same way a static node group is — double check kubectl get nodes and the EC2 console after destroying to make sure nothing GPU-shaped got left behind.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3dtGFjcsm55QryZB.png" /></figure><p>What’s next? I’m really excited about <a href="https://github.com/kubeflow/kubeflow">Kubeflow</a> — let’s see what we can do with it.</p><p>Aight. Thanks for reading this one too, hope you found something useful🚀</p><p>References:</p><ul><li><a href="https://github.com/awslabs/cdk-eks-blueprints/blob/main/docs/addons/kuberay-operator.md">AWS Labs: KubeRay Operator Add-on</a></li><li><a href="https://docs.ray.io/en/latest/cluster/kubernetes/index.html">KubeRay: Ray on Kubernetes</a></li><li><a href="https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4">hugging-quants/Meta-Llama-3.1–8B-Instruct-AWQ-INT4</a></li><li><a href="https://www.flaticon.com/free-icon/user_6175043">Flaticon: User Icon</a></li><li><a href="https://custom.typingmind.com/tools/model-icons/llama">TypingMind: LLaMA Model Icon</a></li></ul><p>Wanna see the full code? See the repo here:</p><p><a href="https://github.com/danielcristho/eks-cdk-llm">GitHub - danielcristho/eks-cdk-llm: A repository for exploring LLM Serving and MLOps practices on AWS Kubernetes (EKS) using Infrastructure as Code (AWS CDK).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc71c30ad6f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM on EKS: Serving with vLLM]]></title>
            <link>https://danielpepuho.medium.com/llm-on-eks-serving-with-vllm-b97aeef5b804?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b97aeef5b804</guid>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Tue, 05 May 2026 00:33:56 GMT</pubDate>
            <atom:updated>2026-07-10T15:50:51.847Z</atom:updated>
            <content:encoded><![CDATA[<p>Originally published at: <a href="https://danielcristho.site/blog/llm-on-eks-vllm">danielcristho.site</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*kINhwDPGvoDYG9K2jdHG6A.png" /></figure><p>Last year, I mentioned that I’m interested in learning how to serve LLMs in production. At first it was just curiosity, but over time I wanted to actually try building something — not just reading about it.</p><p>This post is a small step in that direction: serving an LLM using <a href="https://github.com/vllm-project/vllm">vLLM</a>, deployed on <a href="https://aws.amazon.com/eks">Amazon EKS</a>, provisioned the infra using <a href="https://github.com/aws/aws-cdk">AWS CDK</a>, and wrapped into a simple chatbot using <a href="https://github.com/streamlit/streamlit">Streamlit</a>.</p><h3>TL;DR</h3><ul><li>Exploring LLM serving on a Kubernetes cluster (EKS)</li><li>Using vLLM as the inference engine</li><li>Provisioning the infrastructure with AWS CDK (IaC)</li><li>Building a simple chatbot to interact with the LLM using Streamlit</li></ul><h3>What We Tryna Build</h3><p>The idea is simple: build a small chatbot powered by an LLM and run the model on Kubernetes.</p><p>I’m not focusing on training models here. I would like to understand how to properly serve an LLM.</p><p>The flow looks like this:</p><ul><li>User interacts with a chatbot (running locally)</li><li>The chatbot sends a request to a vLLM API</li><li>The model processes the request and returns a response</li><li>The vLLM service runs on Amazon EKS</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mps-ZWuEIzgu_C3fbPe7bg.png" /><figcaption>Project Architecture</figcaption></figure><h3>Prerequisites</h3><p>Before we dive in, you’ll need:</p><ul><li><strong>AWS Account &amp; IAM</strong>: An AWS Account ID and an IAM User with Administrator access (IAM to manage EKS). We’ll need the IAM username to map kubectl (admin) permissions to the EKS cluster.</li><li><strong>AWS CLI</strong> installed and configured (aws configure) using your IAM user credentials.</li><li><strong>AWS CDK</strong> installed (npm install -g aws-cdk).</li></ul><blockquote><em>AWS usually limits new accounts to 0 vCPUs for “Running On-Demand G and VT instances”. You’ll need to go to the AWS Service Quotas console and request an increase to at least 4 vCPUs to run the </em><em>g4dn.xlarge node.</em></blockquote><h3>The Stack</h3><ul><li><strong>vLLM</strong> — inference engine for the LLM. Fast, supports streaming, and exposes an OpenAI-compatible API out of the box.</li><li><strong>Amazon EKS</strong> — The Kubernetes service on AWS to run the vLLM workload.</li><li><strong>AWS CDK</strong> — infrastructure as code to manage AWS infra, at this time I’ll using Python. One cdk deploy and everything is provisioned.</li><li><strong>Streamlit</strong> — simple chatbot UI that talks to the vLLM endpoint.</li></ul><h3>Why vLLM?</h3><p>There are a few ways to serve an LLM — you could use TGI, Triton, or just raw HuggingFace transformers. I went with vLLM for a few reasons:</p><ul><li><strong>PagedAttention</strong> — manages GPU memory more efficiently, which matters a lot on a single g4dn.xlarge</li><li><strong>OpenAI-compatible API</strong> — the chatbot can use the openai Python SDK without any changes</li><li><strong>Streaming support</strong> — responses stream token by token, which makes the chatbot feel more responsive</li></ul><h3>Why EKS?</h3><p>I could’ve just spun up an EC2 instance and SSH’d in. But that’s not really building reliable infrastructure — that’s just running a script on a server.</p><p>EKS gives us a proper environment to run GPU workloads: node groups, taints, and tolerations to make sure only the vLLM pod lands on the GPU node, and a LoadBalancer service to expose the endpoint.</p><h3>Environment Setup</h3><p>Before getting into the code, let’s set up a .env file at the root of the project. We&#39;ll use this to manage our AWS configurations so we don&#39;t hardcode them into the repository.</p><pre># AWS Config<br>AWS_DEFAULT_ACCOUNT=123456789012<br>AWS_DEFAULT_REGION=us-east-1<br>AWS_ADMIN_USER=your_aws_username<br>AWS_BUCKET=eks-llm-model-bucket<br><br># EKS Config<br>CLUSTER_NAME=eks-llm<br><br># VLLM Config<br># VLLM_URL will be added later after the deployment is live<br># VLLM_URL=http://&lt;nlb-endpoint&gt;.elb.us-east-1.amazonaws.com</pre><h3>The Code</h3><h3>EKS Stack</h3><p>The EksStack provisions everything at the infrastructure level: VPC, EKS cluster, node groups, and an S3 bucket for model storage.</p><pre>vpc = ec2.Vpc(self, &quot;EksVpc&quot;, max_azs=2)<br>cluster = eks.Cluster(<br>    self, &quot;EksCluster&quot;,<br>    version=eks.KubernetesVersion.V1_34,<br>    vpc=vpc,<br>    default_capacity=0,<br>    kubectl_layer=kubectl_layer,<br>)</pre><p>default_capacity=0 means no default node group — we define our own below.</p><p>We have two node groups:</p><pre># 1. CPU, runs system pods (CoreDNS, kube-proxy, etc.)<br>self.cluster.add_nodegroup_capacity(<br>    &quot;ManagedNodeGroup&quot;,<br>    desired_size=1,<br>    min_size=1,<br>    max_size=1,<br>    instance_types=[ec2.InstanceType(&quot;t3.medium&quot;)],<br>    ami_type=eks.NodegroupAmiType.AL2023_X86_64_STANDARD,<br>)<br><br># 2. GPU, for running vLLM<br>gpu_node_role = iam.Role(<br>    self,<br>    &quot;GpuNodeRole&quot;,<br>    assumed_by=iam.ServicePrincipal(&quot;ec2.amazonaws.com&quot;),<br>    managed_policies=[<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEKSWorkerNodePolicy&quot;),<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEC2ContainerRegistryReadOnly&quot;),<br>        iam.ManagedPolicy.from_aws_managed_policy_name(&quot;AmazonEKS_CNI_Policy&quot;),<br>    ],<br>)<br>self.cluster.add_nodegroup_capacity(<br>    &quot;GpuNodeGroup&quot;,<br>    desired_size=1,<br>    min_size=0,<br>    max_size=2,<br>    disk_size=100,<br>    instance_types=[ec2.InstanceType(&quot;g4dn.xlarge&quot;)],<br>    node_role=gpu_node_role,<br>    ami_type=eks.NodegroupAmiType.AL2023_X86_64_NVIDIA,<br>    labels={&quot;workload&quot;: &quot;gpu&quot;},<br>    taints=[<br>        eks.TaintSpec(<br>            key=&quot;nvidia.com/gpu&quot;,<br>            value=&quot;true&quot;,<br>            effect=eks.TaintEffect.NO_SCHEDULE,<br>        )<br>    ],<br>)<br>self.cluster.aws_auth.add_user_mapping(<br>    iam.User.from_user_name(self, &quot;AdminUser&quot;, os.environ[&quot;AWS_ADMIN_USER&quot;]),<br>    groups=[&quot;system:masters&quot;],<br>)<br># Allow GPU nodes to read from the model bucket<br>self.model_bucket.grant_read(gpu_node_role)</pre><p>The disk_size=100 ensures we don&#39;t get pod eviction issues, as the default 20GB is too small for the vLLM container image and the model cache. The taint nvidia.com/gpu=true:NoSchedule on the GPU node group means no pod will be scheduled there unless it explicitly tolerates it. This keeps system pods off the GPU node.</p><p>The S3 bucket is for model weights, and the GPU node role gets read access to it:</p><pre># S3 bucket for model weights<br>self.model_bucket = s3.Bucket(<br>    self,<br>    &quot;ModelBucket&quot;,<br>    bucket_name=os.environ.get(&quot;AWS_BUCKET&quot;),<br>    removal_policy=RemovalPolicy.RETAIN,<br>    block_public_access=s3.BlockPublicAccess.BLOCK_ALL,<br>)</pre><p>We will create those instances:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad44a9c1aec8137bc82e17a112ae6e8f/href">https://medium.com/media/ad44a9c1aec8137bc82e17a112ae6e8f/href</a></iframe><h3>vLLM Stack</h3><p>The VllmStack takes the cluster from EksStack and deploys vLLM on top of it.</p><p>First, we install the NVIDIA device plugin via Helm. This is what makes EKS aware of the GPU on the node — without it, you can’t request nvidia.com/gpu as a resource in your pod spec.</p><pre>cluster.add_helm_chart(<br>    &quot;NvidiaDevicePlugin&quot;,<br>    chart=&quot;nvidia-device-plugin&quot;,<br>    repository=&quot;https://nvidia.github.io/k8s-device-plugin&quot;,<br>    namespace=&quot;kube-system&quot;,<br>    values={<br>        &quot;nodeSelector&quot;: {&quot;workload&quot;: &quot;gpu&quot;},<br>        &quot;tolerations&quot;: [{&quot;key&quot;: &quot;nvidia.com/gpu&quot;, &quot;operator&quot;: &quot;Exists&quot;, &quot;effect&quot;: &quot;NoSchedule&quot;}],<br>    },<br></pre><p>Note the toleration on the plugin itself — it needs to run on the GPU node to expose the GPU, so it has to tolerate the taint we set earlier.</p><p>Then the vLLM Deployment:</p><pre>cluster.add_manifest(&quot;VllmDeployment&quot;, {<br>            &quot;apiVersion&quot;: &quot;apps/v1&quot;,<br>            &quot;kind&quot;: &quot;Deployment&quot;,<br>            &quot;metadata&quot;: {&quot;name&quot;: &quot;vllm&quot;, &quot;namespace&quot;: &quot;default&quot;},<br>            &quot;spec&quot;: {<br>                &quot;replicas&quot;: 1,<br>                &quot;selector&quot;: {&quot;matchLabels&quot;: {&quot;app&quot;: &quot;vllm&quot;}},<br>                &quot;template&quot;: {<br>                    &quot;metadata&quot;: {&quot;labels&quot;: {&quot;app&quot;: &quot;vllm&quot;}},<br>                    &quot;spec&quot;: {<br>                        &quot;tolerations&quot;: [{&quot;key&quot;: &quot;nvidia.com/gpu&quot;, &quot;operator&quot;: &quot;Exists&quot;, &quot;effect&quot;: &quot;NoSchedule&quot;}],<br>                        &quot;nodeSelector&quot;: {&quot;workload&quot;: &quot;gpu&quot;},<br>                        &quot;containers&quot;: [{<br>                            &quot;name&quot;: &quot;vllm&quot;,<br>                            &quot;image&quot;: &quot;vllm/vllm-openai:latest&quot;,<br>                            &quot;args&quot;: [<br>                                &quot;--model&quot;, model_id,<br>                                &quot;--download-dir&quot;, &quot;/model-cache&quot;,<br>                                &quot;--dtype&quot;, &quot;half&quot;,<br>                                &quot;--quantization&quot;, &quot;awq&quot;,<br>                                &quot;--max-model-len&quot;, &quot;4096&quot;,<br>                            ],<br>                            &quot;env&quot;: [<br>                                {&quot;name&quot;: &quot;AWS_DEFAULT_REGION&quot;, &quot;value&quot;: self.region},<br>                                {&quot;name&quot;: &quot;MODEL_BUCKET&quot;, &quot;value&quot;: model_bucket_name},<br>                                {&quot;name&quot;: &quot;VLLM_PORT&quot;, &quot;value&quot;: &quot;8000&quot;},<br>                            ],<br>                            &quot;ports&quot;: [{&quot;containerPort&quot;: 8000}],<br>                            &quot;resources&quot;: {<br>                                &quot;limits&quot;: {&quot;nvidia.com/gpu&quot;: &quot;1&quot;},<br>                                &quot;requests&quot;: {&quot;memory&quot;: &quot;12Gi&quot;, &quot;cpu&quot;: &quot;2&quot;},<br>                            },<br>                            &quot;volumeMounts&quot;: [{&quot;name&quot;: &quot;model-cache&quot;, &quot;mountPath&quot;: &quot;/model-cache&quot;}],<br>                            &quot;readinessProbe&quot;: {<br>                                &quot;httpGet&quot;: {&quot;path&quot;: &quot;/health&quot;, &quot;port&quot;: 8000},<br>                                &quot;initialDelaySeconds&quot;: 120,<br>                                &quot;periodSeconds&quot;: 15,<br>                            },<br>                        }],<br>                        &quot;volumes&quot;: [{&quot;name&quot;: &quot;model-cache&quot;, &quot;emptyDir&quot;: {}}],<br>                    },<br>                },<br>            },<br>        })</pre><p>A few things worth noting:</p><ul><li>nodeSelector: workload=gpu pins the pod to the GPU node group</li><li>nvidia.com/gpu: 1 requests exactly one GPU</li><li>dtype: half and quantization: awq drops the model size to ~5.7GB so it comfortably fits in the 16GB VRAM of g4dn.xlarge without OOM</li><li>max-model-len: 4096 caps the context window to avoid OOM</li></ul><p>Finally, a LoadBalancer service to expose the endpoint publicly:</p><pre>cluster.add_manifest(&quot;VllmService&quot;, {<br>    &quot;kind&quot;: &quot;Service&quot;,<br>    &quot;metadata&quot;: {<br>        &quot;annotations&quot;: {&quot;service.beta.kubernetes.io/aws-load-balancer-type&quot;: &quot;nlb&quot;},<br>    },<br>    &quot;spec&quot;: {<br>        &quot;type&quot;: &quot;LoadBalancer&quot;,<br>        &quot;ports&quot;: [{&quot;port&quot;: 80, &quot;targetPort&quot;: 8000}],<br>    },<br>})</pre><h3>Deploy</h3><pre>cdk bootstrap   # first time only<br>cdk deploy --all</pre><p>After the deployment is success you’ll see the node on this cluster:</p><pre>$ kubectl get nodes<br><br>NAME                          STATUS   ROLES    AGE     VERSION<br>ip-10-0-xx-yy.ec2.internal   Ready    &lt;none&gt;   9m18s   v1.34.7-eks-40737a8</pre><pre>$ kubectl get nodes --show-labels | grep gpu<br><br>ip-10-0-xx-yy.ec2.internal   Ready    &lt;none&gt;   4m23s   v1.34.7-eks-40737a8   beta.kubernetes.io/arch=amd64,<br>...<br>workload=gpu</pre><p>Wait for the vLLM pod to be ready (~5–10 minutes, model is downloaded from HuggingFace on first start):</p><pre>$ kubectl get pods<br><br>NAME                   READY   STATUS    RESTARTS   AGE<br>vllm-64c858884-pz4gz   0/1     Running   0          2m24s</pre><pre>$ kubectl logs -f deployment/vllm<br><br>WARNING 05-01 12:48:00 [argparse_utils.py:257] With `vllm serve`, you should provide the model as a positional argument or in a config file instead of via the `--model` o<br>ption. The `--model` option will be removed in a future version.<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]        █     █     █▄   ▄█<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.0<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]   █▄█▀ █     █     █     █  model   hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:299]<br>(APIServer pid=1) INFO 05-01 12:48:00 [utils.py:233] non-default args: {&#39;model_tag&#39;: &#39;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&#39;, &#39;model&#39;: &#39;hugging-quants/Meta-L<br>lama-3.1-8B-Instruct-AWQ-INT4&#39;, &#39;dtype&#39;: &#39;half&#39;, &#39;max_model_len&#39;: 4096, &#39;quantization&#39;: &#39;awq&#39;, &#39;download_dir&#39;: &#39;/model-cache&#39;}<br>...</pre><h3>Inference</h3><p>Once the pod is running, grab the NLB endpoint:</p><pre>kubectl get svc vllm -o jsonpath=&#39;{.status.loadBalancer.ingress[0].hostname}&#39;</pre><p>Then, check that the model is loaded:</p><pre>curl http://&lt;nlb-endpoint&gt;/v1/models</pre><p>You should see something like:</p><pre>{<br>  &quot;object&quot;: &quot;list&quot;,<br>  &quot;data&quot;: [{<br>    &quot;id&quot;: &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,<br>    &quot;object&quot;: &quot;model&quot;,<br>    &quot;owned_by&quot;: &quot;vllm&quot;<br>  }]<br>}</pre><p>Next, send it a prompt:</p><pre>curl http://&lt;nlb-endpoint&gt;/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,<br>    &quot;messages&quot;: [<br>      {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful AI assistant.&quot;},<br>      {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is CAP theorem?&quot;}<br>    ],<br>    &quot;max_tokens&quot;: 150<br>  }&#39;</pre><p>Response:</p><pre>{<br>  &quot;id&quot;: &quot;chatcmpl-8609921b347e2718&quot;,<br>  &quot;object&quot;: &quot;chat.completion&quot;,<br>  &quot;created&quot;: 1777640350,<br>  &quot;model&quot;: &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;,<br>  &quot;choices&quot;: [<br>    {<br>      &quot;index&quot;: 0,<br>      &quot;message&quot;: {<br>        &quot;role&quot;: &quot;assistant&quot;,<br>        &quot;content&quot;: &quot;The CAP theorem, also known as the Brewer&#39;s CAP theorem, is a fundamental concept in distributed systems. It was first proposed by Eric Brewer in 2000.\n\n**CAP stands for:**\n\n1. **Consistency**: This refers to the ability of a system to ensure that all nodes in the system have the same view of the data. In other words, all nodes see the same data, and any updates are reflected uniformly across the system.\n2. **Availability**: This refers to the ability of a system to ensure that every request receives a (non-error) response, without guarantee that it contains the most recent version of the information. In other words, the system is always available, even if some nodes are down or&quot;<br>      },<br>}<br>...</pre><p>The vLLM logs:</p><pre>(APIServer pid=1) INFO:     10.0.253.68:42022 - &quot;GET /health HTTP/1.1&quot; 200 OK<br>(APIServer pid=1) INFO 05-01 12:59:19 [loggers.py:271] Engine 000: Avg prompt throughput: 1.4 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 28.8%<br>(APIServer pid=1) INFO:     10.0.253.68:33400 - &quot;GET /health HTTP/1.1&quot; 200 OK<br>(APIServer pid=1) INFO 05-01 12:59:29 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 28.8%<br>(APIServer pid=1) INFO 05-01 12:59:39 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 28.8%<br>(APIServer pid=1) INFO:     10.0.253.68:34814 - &quot;GET /health HTTP/1.1&quot; 200 OK<br>(APIServer pid=1) INFO:     10.0.228.56:20496 - &quot;POST /v1/chat/completions HTTP/1.1&quot; 200 OK</pre><p>Finally, if you get a response back, congrats, the model is live. 🎉</p><p>Working with the API endpoint is great, but typing curl commands is not exactly a great user experience. Let&#39;s build a chatbot UI on top of this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*ToRR5rZ_iRjIEH94.gif" /><figcaption>Source: Tenor GIF</figcaption></figure><h3>Chatbot with Streamlit</h3><p>So let’s build a simple chatbot using <a href="https://github.com/streamlit/streamlit">Streamlit</a> that talks directly to the vLLM.</p><p>The nice part? Since vLLM exposes an OpenAI-compatible API, we can just use the OpenAI Python SDK without any effort.</p><h3>Setup</h3><p>Install the dependencies:</p><pre>pip install streamlit openai</pre><p>Let’s create a simple UI:</p><pre>mkdir src<br>touch src/app.py</pre><pre>import os<br>import streamlit as st<br>from openai import OpenAI<br>from dotenv import load_dotenv<br><br>load_dotenv()<br># Using the URL from AWS load balancer<br>VLLM_URL = os.getenv(&quot;VLLM_URL&quot;, &quot;http://xx-yy.elb.us-east-1.amazonaws.com&quot;)<br>MODEL_ID = &quot;hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4&quot;<br>client = OpenAI(base_url=f&quot;{VLLM_URL}/v1&quot;, api_key=&quot;none&quot;)<br>st.set_page_config(page_title=&quot;Llama 3 Chatbot&quot;, page_icon=&quot;🦙&quot;)<br>st.title(&quot;🦙 Llama 3 Chatbot&quot;)<br>st.caption(&quot;Powered by vLLM on EKS&quot;)<br>if &quot;messages&quot; not in st.session_state:<br>    st.session_state.messages = []<br>for msg in st.session_state.messages:<br>    st.chat_message(msg[&quot;role&quot;]).write(msg[&quot;content&quot;])<br>if prompt := st.chat_input(&quot;How is you day? Say something...&quot;):<br>    st.session_state.messages.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt})<br>    st.chat_message(&quot;user&quot;).write(prompt)<br>    with st.chat_message(&quot;assistant&quot;):<br>        stream = client.chat.completions.create(<br>            model=MODEL_ID,<br>            messages=st.session_state.messages,<br>            stream=True,<br>        )<br>        response = st.write_stream(chunk.choices[0].delta.content or &quot;&quot; for chunk in stream)<br>    st.session_state.messages.append({&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: response})</pre><p>Run the UI:</p><pre>streamlit run src/app.py</pre><pre>You can now view your Streamlit app in your browser.<br><br>Local URL: http://localhost:8501<br>Network URL: http://ww.xx.yy.zz:8501</pre><p>Open the URL in your browser, and you should see a simple chatbot interface. Type in a message, and watch the response stream back token by token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5yqruibVnFSq0maVuzbqFQ.png" /><figcaption>Interacting with the LLM via Streamlit UI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jIOduou05hGKCBu7DRDuBw.png" /><figcaption>Interacting with the LLM via Streamlit UI</figcaption></figure><h3>Conclusion</h3><p>Serving an LLM is a bit different than deploying a typical web app. Memory constraints are real — we had to use an AWQ quantized model just to make it fit inside a single g4dn.xlarge instance without hitting OOM. But combining vLLM for inference and AWS CDK to spin up the EKS infrastructure makes the whole setup pretty straightforward.</p><blockquote>Don’t forget to run cdk destroy --all when you&#39;re done! Leaving an EKS cluster and a g4dn.xlarge node running 24/7 will result in a very hefty AWS bill😄😄.</blockquote><p>Aight. Thanks for reading this post, hope you found something useful 🚀</p><p>References:</p><ul><li><a href="https://aws.amazon.com/blogs/machine-learning/serving-llms-using-vllm-and-amazon-ec2-instances-with-aws-ai-chips">AWS Blog: Serving LLMs using vLLM and Amazon EC2 instances with AWS AI chips</a></li><li><a href="https://aws.amazon.com/blogs/machine-learning/deploy-llms-on-amazon-eks-using-vllm-deep-learning-containers">AWS Blog: Deploy LLMs on Amazon EKS using vLLM Deep Learning Containers</a></li></ul><p>Wanna see the full code? See the repo here:</p><p><a href="https://github.com/danielcristho/eks-cdk-llm">GitHub - danielcristho/eks-cdk-llm: A repository for exploring LLM Serving and MLOps practices on AWS Kubernetes (EKS) using Infrastructure as Code (AWS CDK).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b97aeef5b804" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to DHI]]></title>
            <link>https://danielpepuho.medium.com/introduction-to-dhi-0213b33fe8f1?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/0213b33fe8f1</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 06:29:08 GMT</pubDate>
            <atom:updated>2026-04-04T06:29:08.718Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*6ms1-Uom0v8YVZYa63pFyA.png" /></figure><p>Containers have become the core of modern application delivery. But as adoption grows, so does the attack surface. From vulnerable base images to supply chain risks, security is no longer optional — it’s foundational.</p><p>This is where <a href="https://github.com/docker-hardened-images">Docker Hardened Images (DHI)</a> come into play.</p><p>Docker Hardened Images are minimal, secure, and production-ready container images maintained directly by Docker. They are designed to reduce vulnerabilities from the start while simplifying compliance and integration into existing workflows. <a href="https://docs.docker.com/dhi/features">Docker Hardened Images features</a></p><p>Instead of relying on generic base images and fixing issues later with scanners, DHI focuses on building a secure foundation from the beginning.</p><h3>TL;DR</h3><ul><li>Secure, minimal, production-ready container images by Docker</li><li>Near-zero CVEs with continuous patching</li><li>Built-in SBOM, provenance (SLSA L3), and signed metadata</li><li>Drop-in replacement for existing Docker workflow</li><li>Available in Community, Select, and Enterprise tiers</li></ul><h3>What is DHI?</h3><p>Docker Hardened Images (DHI) are curated container images built with a strong focus on security, minimalism, and maintainability. They are designed to reduce the number of known vulnerabilities by limiting unnecessary packages and continuously applying patches.</p><h3>Why does it matter?</h3><p>Most container vulnerabilities originate from base images. If your foundation is already risky, everything built on top inherits that risk. DHI minimizes this exposure early in the lifecycle.</p><h3>When should you use it?</h3><ul><li>When building production-grade applications</li><li>When compliance and security are priorities</li><li>When you want to reduce dependency on reactive scanning</li></ul><h3>How does it work?</h3><p>DHI combines multiple security layers:</p><h3>Security by Default</h3><ul><li>Near-zero known vulnerabilities (CVEs)</li><li>Non-root containers by default</li><li>Minimal packages → smaller attack surface</li></ul><h3>Full Transparency</h3><ul><li>Signed SBOM (Software Bill of Materials)</li><li>SLSA Level 3 build provenance</li><li>VEX (context about exploitability)</li><li>Cryptographic signatures for integrity</li></ul><h3>Hardened Supply Chain</h3><ul><li>Packages built from source by Docker</li><li>Signed and verified artifacts</li><li>Reduced risk from compromised public registries</li></ul><h3>Developer Friendly</h3><ul><li>Built on Alpine and Debian, requiring minimal changes to adopt. Full support glibc and musl available in both variants for broad application compatibility</li><li>Compatible with existing CI/CD</li><li>No major workflow changes required</li></ul><h3>How to Get Started?</h3><p>Using DHI doesn’t require additional software installation because it’s integrated directly into the Docker Engine. However, since it uses a specialized registry, you must first authenticate.</p><h3>1. Authenticate to the DHI Registry</h3><p>Open your terminal and run the login command for the official dhi.io registry:</p><pre>docker login dhi.io</pre><p>You will be prompted for your <strong>Docker ID</strong> and <strong>Personal Access Token (PAT)</strong>. Ensure your account has access to a Docker tier that supports DHI (Select or Enterprise tiers).</p><h3>2. Update your Dockerfile</h3><p>Implementation is as simple as updating a single line in your Dockerfile. Just replace your standard image prefix with dhi.io/.</p><p><strong>Example: Standard Node.js Image</strong></p><pre>FROM node:22-alpine<br># ... rest of your build</pre><p><strong>Example: Hardened Node.js Image</strong></p><pre>FROM dhi.io/node:22<br># ... rest of your build</pre><h3>Best Practices: Implementation Strategies</h3><p>Since DHI images are designed to be extremely minimal — often lacking a shell (like /bin/sh) or a package manager (like apt/apk)—you should adopt the following strategies:</p><h3>Adopt Multi-Stage Builds</h3><p>DHI is a <strong>runtime-optimized</strong> image, not a build-time environment. Perform your compilation and dependency installation in a standard “builder” image, then copy only the necessary artifacts into the final DHI layer.</p><pre># Stage 1: Build (Standard Image)<br>FROM node:22 AS builder<br>WORKDIR /app<br>COPY . .<br>RUN npm install &amp;&amp; npm run build<br><br># Stage 2: Runtime (Hardened Image)<br>FROM dhi.io/node:22<br>WORKDIR /app<br>COPY --from=builder /app/dist ./dist<br># DHI runs as a non-root user by default<br>CMD [&quot;node&quot;, &quot;dist/index.js&quot;]</pre><h3>Deep Debugging with “docker debug”</h3><p>Because DHI images lack an interactive shell, docker exec -it will not work by default. Instead, use the new <a href="https://docs.docker.com/reference/cli/docker/debug/">Docker Debug</a> capability to inspect your running containers:</p><pre>docker debug &lt;container_id&gt;</pre><h3>Conclusion</h3><p><a href="https://github.com/docker-hardened-images">DHI</a> represents a shift in how we think about container security.</p><p>Instead of reacting to vulnerabilities after scanning, DHI helps prevent them from being introduced in the first place — starting from the base image.</p><p>It doesn’t replace tools like Trivy or other scanners, but it significantly reduces the noise and risk those tools have to deal with.</p><p>If you’re already building containers, adopting hardened images is one of the lowest-effort, highest-impact improvements you can make.</p><p>Aight, thanks for taking the time read this post. Happy containerazation 🚀</p><p>References:</p><ul><li><a href="https://docs.docker.com/dhi">Docker Hardened Images</a></li><li><a href="https://docs.docker.com/dhi/features">Docker Hardened Images Features</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0213b33fe8f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS EKS: Running Your First Workload Using AWS CDK]]></title>
            <link>https://aws.plainenglish.io/aws-eks-running-your-first-workload-using-aws-cdk-34e2fd6235b0?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/34e2fd6235b0</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws-cdk]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Sat, 07 Feb 2026 12:43:17 GMT</pubDate>
            <atom:updated>2026-02-15T22:15:59.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FB5G-AucDCcy4kuVqnWGcg.png" /></figure><h3>TL;DR</h3><p><strong>-</strong> Created an EKS cluster and deployed a workload using AWS CDK</p><p><strong>-</strong> Separated stack between infrastructure and application</p><p><strong>-</strong> Defined Kubernetes resources (ConfigMap, Deployment, Service) in code</p><p><strong>-</strong> Learned why cdk deploy --all is often required to apply updates</p><p>Now that we have a running EKS cluster created with AWS CDK, it’s time to deploy our first workload. In this post, we’ll define Kubernetes resources using AWS CDK, deploy a simple web workload to the cluster, and expose it so it can be accessed externally.</p><p>This will give us a practical foundation before moving on to ingress, scaling, and Karpenter in later posts.</p><h3>Defining Kubernetes manifests in CDK</h3><p>By default, Kubernetes workloads are defined using <strong>YAML manifests and applied manually using </strong><strong>kubectl</strong>. While this approach works, it often leads to duplicated configuration, limited reusability, and infrastructure logic being scattered across multiple files.</p><p>With AWS CDK, we can define Kubernetes resources directly in code and let CDK handle applying them to the cluster. Under the hood, CDK still talks to the Kubernetes API, but we gain the benefits of using a real programming language — such as structure, composition, and version control.</p><p>BTW, our project structure should be like this one:</p><pre>running-first-workloads/<br>├── app.py<br>├── cdk.json<br>├── requirements.txt<br>├── eks<br>│   ├── __init__.py<br>│   └── eks_stack.py # 1. Create EKS cluster <br>├── workloads<br>│   ├── __init__.py<br>│   └── running_first_workloads_stack.py # 2. Running workload<br>└── tests</pre><p><strong>-</strong> eks/eks_stack.py</p><p>Defines the core EKS infrastructure, including the VPC, EKS control plane, managed node group, and IAM access configuration. This stack is responsible only for the cluster itself.</p><p><strong>-</strong> workloads/running_first_workloads_stack.py</p><p>Defines Kubernetes resources that run on top of the existing cluster. In this post, this includes the ConfigMap, Deployment, and Service for the Caddy web server.</p><p><strong>-</strong> app.py</p><p>Acts as the entry point that wires the two stacks together. The EKS cluster created in eks_stack.py is passed into the workload stack, allowing workloads to be deployed without recreating the cluster.</p><p>Now let’s do some work and start defining our first Kubernetes workload using AWS CDK inside running_first_workloads_stack.py.</p><h3>Stack Definition and Constructor</h3><p>We start by defining a new CDK stack class that accepts an existing EKS cluster as a parameter.</p><pre>class RunningFirstWorkloadsStack(Stack):<br><br>def __init__(<br>        self,<br>        scope: Construct,<br>        construct_id: str,<br>        cluster: eks.Cluster,<br>        **kwargs<br>    ) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs</pre><p>Unlike the cluster stack, this stack does not create any AWS infrastructure. Instead, it receives a reference to an already running EKS cluster. This reference allows CDK to know <strong>where Kubernetes resources should be applied</strong>.</p><p>For our first workload, we’ll deploy Caddy, a lightweight and modern web server. Caddy is a good fit for this example because it behaves like a real-world application component while remaining simple enough to focus on Kubernetes and CDK concepts rather than application logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_VZwlL2HWzRt4uMg.png" /><figcaption>Caddy Logo. <em>Image Source: Caddy Community</em></figcaption></figure><p><strong>-</strong> A ConfigMap to store the Caddy configuration</p><p><strong>-</strong> A Deployment to run the Caddy containers</p><p><strong>-</strong> A Service to expose the application externally</p><p>All of these resources are defined using cluster.add_manifest() in the workload stack. CDK takes care of applying the manifests to the cluster in the correct order, allowing us to focus on how the workload is structured rather than how it is manually deployed.</p><p>Next, we’ll start by defining a ConfigMap to store the Caddy configuration.</p><h3>ConfigMap: Storing the Caddy Config</h3><p>Before running any containers, we need to define how Caddy should behave. Instead of embedding configuration directly into the container image, we store the configuration in a ConfigMap. This is a common Kubernetes pattern that keeps application configuration separate from the runtime environment.</p><p>For this example, the configuration is intentionally minimal. We configure Caddy to listen on port 80 and return a simple HTTP response. This keeps the focus on how the workload is deployed, not on application logic. This ConfigMap will create an object named caddy-config that contains a Caddyfile. Caddy automatically reads its configuration from this file when it starts.</p><pre># ConfigMap for Caddyfile<br>        cluster.add_manifest(<br>            &quot;CaddyConfig&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;v1&quot;,<br>                &quot;kind&quot;: &quot;ConfigMap&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy-config&quot;<br>                },<br>                &quot;data&quot;: {<br>                    &quot;Caddyfile&quot;: &quot;&quot;&quot;<br>:80 {<br>    respond &quot;Hello from Caddy v1&quot;<br>}<br>&quot;&quot;&quot;<br>                }<br>            }<br>        )</pre><h3>Deployment: Running the Containers</h3><p>The next step is to run the application containers. In Kubernetes, this is done using a Deployment, which manages how pods are created, updated, and kept running.</p><p>In this example, the Deployment is responsible for:</p><p><strong>-</strong> Running the Caddy container image</p><p><strong>-</strong> Mounting the ConfigMap as a volume</p><p><strong>-</strong> Managing multiple replicas for availability</p><pre># Caddy Deployment<br>        cluster.add_manifest(<br>            &quot;CaddyDeployment&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;apps/v1&quot;,<br>                &quot;kind&quot;: &quot;Deployment&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy&quot;<br>                },<br>                &quot;spec&quot;: {<br>                    &quot;replicas&quot;: 2,<br>                    &quot;selector&quot;: {<br>                        &quot;matchLabels&quot;: {<br>                            &quot;app&quot;: &quot;caddy&quot;<br>                        }<br>                    },<br>                    &quot;template&quot;: {<br>                    &quot;metadata&quot;: {<br>                        &quot;labels&quot;: { &quot;app&quot;: &quot;caddy&quot; },<br>                        &quot;annotations&quot;: {<br>                            &quot;configmap-reload-ts&quot;: str(int(time.time()))<br>                        }<br>                    },<br>                        &quot;spec&quot;: {<br>                            &quot;containers&quot;: [<br>                                {<br>                                    &quot;name&quot;: &quot;caddy&quot;,<br>                                    &quot;image&quot;: &quot;caddy:2&quot;,<br>                                    &quot;ports&quot;: [<br>                                        {<br>                                            &quot;containerPort&quot;: 80<br>                                        }<br>                                    ],<br>                                    &quot;volumeMounts&quot;: [<br>                                        {<br>                                            &quot;name&quot;: &quot;caddy-config&quot;,<br>                                            &quot;mountPath&quot;: &quot;/etc/caddy&quot;<br>                                        }<br>                                    ]<br>                                }<br>                            ],<br>                            &quot;volumes&quot;: [<br>                                {<br>                                    &quot;name&quot;: &quot;caddy-config&quot;,<br>                                    &quot;configMap&quot;: {<br>                                        &quot;name&quot;: &quot;caddy-config&quot;<br>                                    }<br>                                }<br>                            ]<br>                        }<br>                    }<br>                }<br>            }<br>        )</pre><p>Let’s do some break down:</p><p><strong>-</strong> replicas: 2: Runs two instances of the Caddy container, providing basic availability and allowing us to observe how Kubernetes distributes traffic.</p><p><strong>-</strong> container image: Uses the official caddy:2 image</p><p><strong>-</strong> volumeMounts and volumes: The ConfigMap created earlier is mounted into the container at /etc/caddy, which is where Caddy expects its configuration file by default.</p><h3>Service: Exposing the Application</h3><p>At this point, the Caddy pods are running perfectly fine inside the cluster — buuut from the outside world, they might as well not exist. You can kubectl get pods all day long, but your browser still won’t load anything.</p><p>To make the application actually usable, we define a <strong>Service</strong>, which gives our pods a stable identity and a proper way to receive traffic.</p><p>In this example, we use a Service of type <strong>LoadBalancer</strong>, which is the simplest way to expose an application externally on AWS.</p><pre># Service to expose Caddy<br>        cluster.add_manifest(<br>            &quot;CaddyService&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;v1&quot;,<br>                &quot;kind&quot;: &quot;Service&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy-service&quot;<br>                },<br>                &quot;spec&quot;: {<br>                    &quot;type&quot;: &quot;LoadBalancer&quot;,<br>                    &quot;selector&quot;: {<br>                        &quot;app&quot;: &quot;caddy&quot;<br>                    },<br>                    &quot;ports&quot;: [<br>                        {<br>                            &quot;port&quot;: 80,<br>                            &quot;targetPort&quot;: 80<br>                        }<br>                    ]<br>                }<br>            }<br>        )</pre><p>Once the Service is created, Kubernetes and AWS work together to expose the application. After a short delay, the Service will receive an external IP or DNS name that can be used to access Caddy from a browser or via curl</p><h3>Deploying the Stack (EKS Stack and Workload Stack)</h3><p>Before we dive deeper, here is the full code for eks_stack.py</p><pre>from aws_cdk import (<br>Stack,<br>aws_eks as eks,<br>aws_ec2 as ec2,<br>)<br><br>from aws_cdk.lambda_layer_kubectl_v32 import KubectlV32Layer<br><br>from aws_cdk import aws_iam as iam<br><br>from constructs import Construct<br><br>class CdkEksStack(Stack):<br>    def __init__(self, scope: Construct, construct_id: str, **kwargs) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br>        kubectl_layer = KubectlV32Layer(self, &quot;KubectlLayer&quot;)<br>        vpc = ec2.Vpc(<br>            self,<br>            &quot;EksVpc&quot;,<br>            max_azs=2 # AZ<br>        )<br>        self.cluster = eks.Cluster(<br>            self,<br>            &quot;EksCluster&quot;,<br>            version=eks.KubernetesVersion.V1_32,<br>            vpc=vpc,<br>            default_capacity=0,<br>            kubectl_layer=kubectl_layer<br>        )<br>        self.cluster.add_nodegroup_capacity(<br>            &quot;ManagedNodeGroup&quot;,<br>            desired_size=2,<br>            min_size=1,<br>            max_size=3,<br>            instance_types=[ec2.InstanceType(&quot;t3.medium&quot;)], # 2vcpu/4gi<br>        )<br>        self.cluster.aws_auth.add_user_mapping(<br>            iam.User.from_user_name(<br>            self,<br>            &quot;AdminUser&quot;,<br>            &quot;&lt;YOUR_IAM&gt;&quot;<br>            ),<br>            groups=[&quot;system:masters&quot;],<br>        )</pre><p>Also, the running_first_workloads_stack.py</p><pre>from aws_cdk import (<br>Stack,<br>aws_eks as eks,<br>)<br><br>from constructs import Construct<br><br>import time <br><br>class RunningFirstWorkloadsStack(Stack):<br>    def __init__(<br>        self,<br>        scope: Construct,<br>        construct_id: str,<br>        cluster: eks.Cluster,<br>        **kwargs<br>    ) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br>class RunningFirstWorkloadsStack(Stack):<br>    def __init__(<br>        self,<br>        scope: Construct,<br>        construct_id: str,<br>        cluster: eks.Cluster,<br>        **kwargs<br>    ) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br>        # ConfigMap for Caddyfile<br>        cluster.add_manifest(<br>            &quot;CaddyConfig&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;v1&quot;,<br>                &quot;kind&quot;: &quot;ConfigMap&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy-config&quot;<br>                },<br>                &quot;data&quot;: {<br>                    &quot;Caddyfile&quot;: &quot;&quot;&quot;<br>:80 {<br>    respond &quot;Hello from Caddy v2&quot;<br>}<br>&quot;&quot;&quot;<br>                }<br>            }<br>        )<br>        # Caddy Deployment<br>        cluster.add_manifest(<br>            &quot;CaddyDeployment&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;apps/v1&quot;,<br>                &quot;kind&quot;: &quot;Deployment&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy&quot;<br>                },<br>                &quot;spec&quot;: {<br>                    &quot;replicas&quot;: 2,<br>                    &quot;selector&quot;: {<br>                        &quot;matchLabels&quot;: {<br>                            &quot;app&quot;: &quot;caddy&quot;<br>                        }<br>                    },<br>                    &quot;template&quot;: {<br>                    &quot;metadata&quot;: {<br>                        &quot;labels&quot;: { &quot;app&quot;: &quot;caddy&quot; },<br>                        &quot;annotations&quot;: {<br>                            &quot;configmap-reload-ts&quot;: str(int(time.time()))<br>                        }<br>                    },<br>                        &quot;spec&quot;: {<br>                            &quot;containers&quot;: [<br>                                {<br>                                    &quot;name&quot;: &quot;caddy&quot;,<br>                                    &quot;image&quot;: &quot;caddy:2&quot;,<br>                                    &quot;ports&quot;: [<br>                                        {<br>                                            &quot;containerPort&quot;: 80<br>                                        }<br>                                    ],<br>                                    &quot;volumeMounts&quot;: [<br>                                        {<br>                                            &quot;name&quot;: &quot;caddy-config&quot;,<br>                                            &quot;mountPath&quot;: &quot;/etc/caddy&quot;<br>                                        }<br>                                    ]<br>                                }<br>                            ],<br>                            &quot;volumes&quot;: [<br>                                {<br>                                    &quot;name&quot;: &quot;caddy-config&quot;,<br>                                    &quot;configMap&quot;: {<br>                                        &quot;name&quot;: &quot;caddy-config&quot;<br>                                    }<br>                                }<br>                            ]<br>                        }<br>                    }<br>                }<br>            }<br>        )<br>        # Service to expose Caddy<br>        cluster.add_manifest(<br>            &quot;CaddyService&quot;,<br>            {<br>                &quot;apiVersion&quot;: &quot;v1&quot;,<br>                &quot;kind&quot;: &quot;Service&quot;,<br>                &quot;metadata&quot;: {<br>                    &quot;name&quot;: &quot;caddy-service&quot;<br>                },<br>                &quot;spec&quot;: {<br>                    &quot;type&quot;: &quot;LoadBalancer&quot;,<br>                    &quot;selector&quot;: {<br>                        &quot;app&quot;: &quot;caddy&quot;<br>                    },<br>                    &quot;ports&quot;: [<br>                        {<br>                            &quot;port&quot;: 80,<br>                            &quot;targetPort&quot;: 80<br>                        }<br>                    ]<br>                }<br>            }<br>        )</pre><p>And the last one, app.py:</p><pre>#!/usr/bin/env python3<br>import aws_cdk as cdk<br><br>from eks.eks_stack import CdkEksStack<br><br>from workloads.running_first_workloads_stack import RunningFirstWorkloadsStack<br><br>app = cdk.App()<br># 1. Create EKS cluster stack<br>eks_stack = CdkEksStack(<br>    app,<br>    &quot;EksStack&quot;,<br>)<br># 2. Deploy workloads on top of the cluster<br>RunningFirstWorkloadsStack(<br>    app,<br>    &quot;RunningFirstWorkloadsStack&quot;,<br>    cluster=eks_stack.cluster,<br>)<br>app.synth()</pre><h3>Listing Available Stacks</h3><p>Before deploying, it’s often useful to see which stacks are defined in the application. You can list them using:</p><pre>$ cdk list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/0*2H4KEuRYFdNpz215.png" /></figure><h3>Deploying All Stacks</h3><p>Since the workload stack depends on the EKS stack, the simplest approach is to deploy all stacks at once:</p><pre>$ cdk deploy --all</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/0*3KuGw4Ax3kUQ1a0C.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S5hIodRiIGmEtgVZ.png" /></figure><p>CDK will:</p><p><strong>-</strong> Deploy the EKS infrastructure first</p><p><strong>-</strong> Wait for the cluster to become available</p><p><strong>-</strong> Apply the Kubernetes manifests defined in the workload stack</p><p>If you want to deploy only a specific stack, you can target it by name:</p><pre>$ cdk deploy EksStack<br><br>or <br><br>$ cdk deploy RunningFirstWorkloadsStack</pre><h3>Verify Deployment</h3><p>Once the deployment is complete, let’s see that the workload is running correctly using Kubernetes commands.</p><p>First, let’s check the pods:</p><pre>$ kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/0*7tKmiYXIFYEWW4Ju.png" /></figure><p>Both pods are in the Running state and ready, which means the Caddy containers started successfully and are using the configuration provided via the ConfigMap.</p><p>Next, let’s check the Service:</p><pre>$ kubectl get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cRZpJmJ9g-36X2Tf.png" /></figure><p>Here we can see that the caddy-service has been assigned an external endpoint by AWS. This means the load balancer is ready and traffic can now reach the application from outside the cluster.</p><h3>Accessing the Application</h3><p>Copy the value from the <strong>EXTERNAL-IP</strong> column and open it in your browser, or use curl:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/0*uZZodKpXJkOyPhWB.png" /></figure><p>At this point, the workload is fully deployed and accessible.</p><h3>Make Change</h3><p>To see how changes are applied, let’s update the Caddy configuration. We’ll modify the response in the Caddyfile from v1 to v2:</p><pre>respond &quot;Hello from Caddy v1&quot; -&gt; respond &quot;Hello from Caddy v2&quot;</pre><p>First, try deploying only the workload stack:</p><pre>$ cdk deploy RunningFirstWorkloadsStack</pre><p>At first glance, this looks like the correct approach — after all, only the workload code has changed. However, after the deployment completes, nothing appears to have changed:</p><p><strong>-</strong> The running pods are still the same</p><p><strong>-</strong> No rolling update is triggered</p><p><strong>-</strong> The application response still returns Hello from Caddy v1</p><p>In other words, from the cluster’s point of view, the workload has not been updated.</p><h4>Why Didn’t This Work?</h4><p>Although the ConfigMap content changed in code, Kubernetes manifests defined using cluster.add_manifest() are applied through a custom resource backed by the EKS stack. When deploying only the workload stack, CDK may determine that there are no CloudFormation-level changes that require reapplying the manifests.</p><p>As a result, the Kubernetes provider is not re-invoked, and the updated configuration is never applied to the cluster.</p><p>Next, we deploy all stacks together:</p><pre>$ cdk deploy --all</pre><p>This time, the behavior is different:</p><p><strong>-</strong> The EKS stack is re-evaluated</p><p><strong>-</strong> The Kubernetes provider is refreshed</p><p><strong>-</strong> The updated manifests are applied to the cluster</p><p><strong>-</strong> A rolling update is triggered on the Deployment</p><p><strong>-</strong> New pods are created with the updated configuration</p><p>After the rollout completes, the application now responds with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*0r1qZwUUW3PUqgL8.png" /></figure><h3>What’s next</h3><p>At this point, we’ve successfully deployed a workload to EKS using AWS CDK. We defined Kubernetes resources in code, deployed them through CDK, exposed the application externally, and verified that updates can be rolled out predictably.</p><p>This setup it’s still just the beginning.</p><p>In the next posts, we’ll continue building on top of this foundation by exploring more production-oriented patterns, including:</p><p><strong>-</strong> CDK for more</p><p><strong>-</strong> Ingress</p><p><strong>-</strong> Autoscaling with Karpenter</p><p>Each of these topics will build directly on the cluster and workload we’ve created, moving the setup closer to a production-ready environment.</p><p>Aight, thanks for taking the time to read this post. Don’t forget to clean up the stack. Here is the full code of this post:</p><p><a href="https://github.com/danielcristho/that-i-write/tree/main/eks-cdk/running-first-workloads">danielcristho/that-i-write/running-first-workloads</a></p><p>Originally posted at <a href="https://danielcristho.site/blog/cdk-running-first-workload">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34e2fd6235b0" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/aws-eks-running-your-first-workload-using-aws-cdk-34e2fd6235b0">AWS EKS: Running Your First Workload Using AWS CDK</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS EKS: Create Your First Cluster Using AWS CDK]]></title>
            <link>https://aws.plainenglish.io/aws-eks-create-your-first-cluster-using-aws-cdk-0e31804350ff?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e31804350ff</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 08:54:47 GMT</pubDate>
            <atom:updated>2026-02-15T22:16:34.994Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FB5G-AucDCcy4kuVqnWGcg.png" /></figure><h3>TL;DR</h3><p><strong>-</strong> Create an EKS cluster using AWS CDK Python</p><p><strong>-</strong> Provision infrastructure via AWS CloudFormation</p><p><strong>-</strong> The setup includes a VPC, EKS control plane, and a managed EC2 node group</p><p><strong>-</strong> Configure IAM access to use kubectl</p><p><strong>-</strong> Verify the cluster and clean up resources when done</p><h3>Introduction</h3><p>About 2 years ago, I’ve been using AWS CDK to manage infrastructure, mostly for smaller workloads like deploying APIs on AWS Lambda, you can see the repo here: <a href="https://github.com/danielcristho/cdk-go-simple-restapi">cdk-go-simple-restapi</a>. This year, I want to take it a step further by using AWS CDK to create and manage an AWS EKS cluster.</p><p>I’ll be writing a short series of posts around this topic, starting with the basics. In this first post, we’ll focus on building a minimal EKS cluster using AWS CDK as the foundation for the next parts of the series.</p><h3>Why AWS CDK for EKS?</h3><p>Amazon EKS already provides multiple ways to create and manage clusters, from the AWS Console to CloudFormation and Terraform. However, when infrastructure starts to grow in complexity, the way it’s defined and maintained becomes just as important as the resources themselves.</p><p>CDK allows you to define infrastructure using familiar programming languages. Instead of managing large YAML or JSON templates, you work with code — loops, conditions, and abstractions included. This makes infrastructure easier to reason about, review, and evolve over time. According to the documentation, AWS CDK supports multiple languages such as Python, Go, TypeScript, JavaScript and C#</p><p>For EKS specifically, the kit provides higher-level constructs that abstract away a lot of the boilerplate required to get a cluster running. Networking, IAM roles, and node groups can be defined in a few lines of code while still allowing you to customize the parts that matter.</p><h3>How AWS CDK Works with Amazon EKS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NKIVSA3Z9LVUyKJi.png" /><figcaption>CDK interaction with AWS Services</figcaption></figure><p>The diagram above shows us how CDK interacts with AWS services when creating an Amazon EKS cluster.</p><p>Everything starts from your CDK application (stack), where infrastructure is defined using a programming language such as Go, TypeScript, or Python. At this stage, no AWS resources are created yet. When you run cdk synth, AWS CDK translates your code into a standard AWS CloudFormation template.</p><p>This CloudFormation template is then deployed using cdk deploy. At this point, AWS CloudFormation becomes responsible for provisioning the infrastructure. It creates all required AWS resources, including the VPC, IAM roles, the EKS control plane, and the managed node group. <strong>AWS CDK itself does not bypass CloudFormation, it simply acts as a higher-level abstraction on top of it</strong>.</p><p>Once CloudFormation finishes, the Amazon EKS cluster is fully provisioned and exposes a Kubernetes control plane backed by AWS-managed infrastructure. From this point forward, the cluster behaves like a standard Kubernetes cluster.</p><p>Optionally, the CDK can also interact with the Kubernetes API to manage workloads, such as applying Kubernetes manifests or deploying Helm charts. This step usually happens after the cluster is ready and is covered in later parts of this series.</p><h3>Setup</h3><p>In this series, all AWS CDK examples will be written in Python. While AWS CDK supports multiple languages such as TypeScript and Go, the underlying concepts and constructs remain the same regardless of the language used.</p><h3>Prerequisites</h3><ul><li><strong>AWS account and configured AWS CLI</strong></li></ul><p>An active AWS account with permissions to create EKS, VPC, IAM, and CloudFormation resources. Installed and configured with valid credentials (aws configure).</p><ul><li><strong>Python 3.10+</strong></li></ul><p>Used for writing the AWS CDK application in this series.</p><ul><li><strong>AWS CDK installed</strong></li></ul><p>Installed globally via npm:</p><pre>$ npm install -g aws-cdk</pre><ul><li><strong>kubectl installed</strong></li></ul><p>Used to interact with the EKS cluster once it is created.</p><p>Notes</p><ul><li>This post focuses on creating the EKS cluster. No Kubernetes workloads are deployed yet.</li><li>Make sure your AWS credentials have sufficient permissions before running cdk deploy, as EKS provisioning may take several minutes.</li></ul><h3>Project Initialization</h3><p>We’ll start by creating a new AWS CDK project using Python. AWS CDK provides a project template that sets up the basic structure, dependencies, and configuration needed to get started.</p><p>First, create a new directory and initialize a CDK app:</p><pre>$ mkdir cdk-eks<br>$ cd cdk-eks<br>$ cdk init app --language python</pre><p>This command generates a basic project structure for a Python-based CDK application. The output after you run cdk init</p><pre>Applying project template app for python</pre><pre># Welcome to your CDK Python project!<br>This is a blank project for CDK development with Python.<br>The `cdk.json` file tells the CDK Toolkit how to execute your app.<br>...<br>Enjoy!<br>Executing Creating virtualenv...<br>Executing Installing dependencies...<br>✅ All done!</pre><p>After initialization, you should see a structure similar to this:</p><pre>.<br>├── app.py<br>├── cdk_eks<br>│   ├── cdk_eks_stack.py<br>│   └── __init__.py<br>├── cdk.json<br>├── README.md<br>├── requirements-dev.txt<br>├── requirements.txt<br>├── source.bat<br>└── tests<br>    ├── __init__.py<br>    └── unit</pre><p><strong>-</strong> app.py</p><p>The entry point of the CDK application. This is where the stack is instantiated.</p><p><strong>-</strong> cdk_eks_stack.py</p><p>Contains the definition of the CDK stack where we’ll define the EKS cluster.</p><p><strong>-</strong> requirements.txt</p><p>Python dependencies for the CDK app.</p><p><strong>-</strong> cdk.json</p><p>CDK configuration file, including the command used to run the app.</p><p><strong>-</strong> tests/</p><p>A placeholder for unit tests, which we’ll use in a later post.</p><p><strong>1. Install Dependencies</strong></p><p>Before writing any code, activate the Python virtual environment and install the dependencies. BTW, you need to add this one to your requirements.txt: aws-cdk.lambda-layer-kubectl-v34.</p><pre>$ source .venv/bin/activate<br>$ pip install -r requirements.txt</pre><p>Keeping dependencies isolated using a virtual environment helps avoid version conflicts and keeps the project reproducible.</p><p><strong>2. Run Bootstrap</strong></p><p>If this is your first time using AWS CDK in the selected AWS account and region, you’ll need to bootstrap the environment:</p><pre>$ cdk bootstrap</pre><p>This command creates the necessary resources in your AWS account that CDK requires to deploy stacks.</p><p>At this point, the project is ready, and we can start defining the EKS cluster using AWS CDK.</p><h3>Creating the EKS Cluster</h3><p>In this section, we’ll define a minimal EKS cluster using AWS CDK. The goal is not to build a production-ready cluster yet, but to create a foundation that we can extend in later posts.</p><p>All changes will be made inside the CDK stack file</p><p><strong>1.</strong> Import Required Modules</p><p>Open cdk_eks/cdk_eks_stack.py and start by importing the required CDK modules:</p><pre>from aws_cdk import (<br>Stack,<br>aws_eks as eks,<br>aws_ec2 as ec2,<br>)</pre><pre>from aws_cdk.lambda_layer_kubectl_v32 import KubectlV32Layer</pre><pre>from aws_cdk import aws_iam as iam</pre><pre>from constructs import Construct</pre><p>We’ll use:</p><ul><li>ec2 for networking (VPC)</li><li>eks for creating the EKS cluster and its node groups</li></ul><p><strong>2.</strong> Define the VPC</p><p>EKS requires a VPC. For simplicity, we’ll let CDK create one for us:</p><pre>class CdkEksStack(Stack):</pre><pre>def __init__(self, scope: Construct, construct_id: str, **kwargs) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br>        kubectl_layer = KubectlV32Layer(self, &quot;KubectlLayer&quot;)<br>        vpc = ec2.Vpc(<br>            self,<br>            &quot;EksVpc&quot;,<br>            max_azs=2 # AZ<br>        )</pre><p>This creates a basic VPC across two Availability Zones, which is sufficient for a minimal cluster.</p><p><strong>3.</strong> Create the EKS Cluster</p><p>Next, we define the EKS cluster itself:</p><pre>cluster = eks.Cluster(<br>    self,<br>    &quot;EksCluster&quot;,<br>    version=eks.KubernetesVersion.V1_32,<br>    vpc=vpc,<br>    default_capacity=0,<br>    kubectl_layer=kubectl_layer<br>)</pre><p>A few important points:</p><ul><li>Kubernetes version is explicitly set to avoid unexpected upgrades.</li><li>default_capacity=0 disables the default node group so we can define our own.</li></ul><p><strong>4.</strong> Add a Managed Node Group</p><p>Now we add a managed node group to run workloads:</p><pre>cluster.add_nodegroup_capacity(<br>    &quot;ManagedNodeGroup&quot;,<br>    desired_size=2,<br>    min_size=1,<br>    max_size=3,<br>    instance_types=[ec2.InstanceType(&quot;t3.medium&quot;)],<br>)</pre><p>This creates:</p><ul><li>A managed EC2-based node group</li><li>Autoscaling between 1 and 3 nodes</li><li>Instance type for testing</li></ul><p><strong>5.</strong> Add IAM</p><p>By default, creating an EKS cluster with AWS CDK does not automatically grant Kubernetes access to the IAM identity used to deploy the stack. While the control plane and node group roles are configured, additional IAM-to-Kubernetes RBAC mapping is required to access the cluster using kubectl.</p><pre>cluster.aws_auth.add_user_mapping(<br>    iam.User.from_user_name(<br>    self,<br>    &quot;AdminUser&quot;,<br>    &quot;&lt;REPLACE_WITH_YOUR_IAM&gt;&quot;<br>    ),<br>    groups=[&quot;system:masters&quot;],<br>)</pre><p>Or using Role</p><pre>cluster.aws_auth.add_role_mapping(<br>    iam.Role.from_role_name(<br>    self,<br>    &quot;EksAdminRole&quot;,<br>    &quot;&lt;REPLACE_WITH_YOUR_ROLE&gt;&quot;<br>    ),<br>    groups=[&quot;system:masters&quot;],<br>)</pre><p>Here is the full code:</p><pre>from aws_cdk import (<br>Stack,<br>aws_eks as eks,<br>aws_ec2 as ec2,<br>)</pre><pre>from aws_cdk.lambda_layer_kubectl_v32 import KubectlV32Layer</pre><pre>from aws_cdk import aws_iam as iam</pre><pre>from constructs import Construct</pre><pre>class CdkEksStack(Stack):<br>    def __init__(self, scope: Construct, construct_id: str, **kwargs) -&gt; None:<br>        super().__init__(scope, construct_id, **kwargs)<br>        kubectl_layer = KubectlV32Layer(self, &quot;KubectlLayer&quot;)<br>        vpc = ec2.Vpc(<br>            self,<br>            &quot;EksVpc&quot;,<br>            max_azs=2<br>        )<br>        cluster = eks.Cluster(<br>            self,<br>            &quot;EksCluster&quot;,<br>            version=eks.KubernetesVersion.V1_32,<br>            vpc=vpc,<br>            default_capacity=0,<br>            kubectl_layer=kubectl_layer<br>        )<br>        cluster.add_nodegroup_capacity(<br>            &quot;ManagedNodeGroup&quot;,<br>            desired_size=2,<br>            min_size=1,<br>            max_size=3,<br>            instance_types=[ec2.InstanceType(&quot;t3.medium&quot;)],<br>        )<br>        cluster.aws_auth.add_user_mapping(<br>            iam.User.from_user_name(<br>            self,<br>            &quot;AdminUser&quot;,<br>            &quot;&lt;YOUR_IAM&gt;&quot;<br>            ),<br>            groups=[&quot;system:masters&quot;],<br>        )</pre><ul><li>A VPC</li><li>An EKS control plane</li><li>A managed node group for worker nodes</li></ul><p>All of this is defined as code and will be provisioned via CloudFormation.</p><p><strong>5</strong> Synthesize the Stack</p><p>Before deploying, it’s a good idea to check what CDK will generate:</p><pre>$ cdk synth</pre><pre>cdk synth 2&gt;&amp;1 | head -50</pre><pre>Resources:<br>  KubectlLayer600207B5:<br>    Type: AWS::Lambda::LayerVersion<br>    Properties:<br>      Content:<br>        S3Bucket:<br>          Fn::Sub: cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}<br>        S3Key: cc5bb5a423d0f1ccbfa20b3016434049b477f393e38ad2be0e8cba029f2a2373.zip<br>      Description: /opt/kubectl/kubectl 1.32.3; /opt/helm/helm 3.17.2<br>      LicenseInfo: Apache-2.0<br>...</pre><p>This command outputs the CloudFormation template that will be used to create the EKS cluster.</p><h3>Deploying the Stack</h3><p>With the stack definition in place, we can now deploy the EKS cluster using AWS CDK. This step will trigger AWS CloudFormation to provision all required resources.</p><p><strong>1.</strong> Deploy the Stack</p><p>Run the following command from the project root:</p><pre>$ cdk deploy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KhnYk-AhFymDPVY5.png" /></figure><p>During deployment, CDK will:</p><ul><li>Package and upload assets (if any)</li><li>Create or update the CloudFormation stack</li><li>Provision AWS resources such as the VPC, IAM roles, EKS control plane, and managed node group</li></ul><p>You’ll be prompted to confirm the deployment, as the stack creates IAM roles and other security-related resources. Review the changes and approve the deployment when prompted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Clj2DUFEtxNdOMtW.png" /></figure><h3>Verifying the Cluster</h3><p>After the deployment completes, the next step is to verify that the EKS cluster has been created successfully and is in a healthy state.</p><p>At this point, you should see a CloudFormation stack created by AWS CDK in the AWS Console. This stack represents all resources defined in the CDK application, including the VPC, IAM roles, EKS control plane, and the managed node group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rLAgoXDts76Owtb1.png" /></figure><p>The stack status should be <strong>CREATE_COMPLETE</strong>, indicating that all resources were provisioned without errors. This also reinforces that AWS CDK relies on CloudFormation as the underlying provisioning engine.</p><p>Next, navigate to the Amazon EKS console. You should see the newly created cluster listed and marked as Active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eO7SXX8YcYgVxobB.png" /></figure><p><strong>1.</strong> Accessing the Cluster with kubectl</p><p>To interact with the cluster, update your kubeconfig:</p><pre>$ aws eks update-kubeconfig --name &lt;cluster-name&gt; --region &lt;region&gt;</pre><p>Once configured, verify that the nodes are registered:</p><pre>$ kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*qN2CJdqfJcRlQGzx.png" /></figure><h3>Cleaning Up Resources</h3><p>This command deletes the CloudFormation stack and all resources created by AWS CDK, including the EKS cluster, node groups, and networking components. Since EKS is not covered by the AWS free tier, it’s recommended to clean up resources when they are no longer needed.</p><pre>$ cdk destroy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*khGkVYGJBBKHMMLK.png" /></figure><h3>What’s Next?</h3><p>At this point, we have a fully functional Amazon EKS cluster created and managed using AWS CDK. The cluster is accessible via kubectl, the node group is running, and IAM access has been configured correctly. This gives us a foundation, but so far, the cluster is still empty.</p><p>In the next post, we’ll move beyond infrastructure and start running actual workloads on this cluster. Instead of applying raw Kubernetes YAML files manually, we’ll use AWS CDK to deploy workloads in a more structured and repeatable way.</p><p>Aight, thanks for taking the time read this post. Here is the full code of this post:</p><p><a href="https://github.com/danielcristho/that-i-write/tree/main/eks-cdk/create-kube-cluster">danielcristho/that-i-write/create-kube-cluster</a></p><p>Originally posted at <a href="https://danielcristho.site/blog/cdk-create-kube-cluster">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e31804350ff" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/aws-eks-create-your-first-cluster-using-aws-cdk-0e31804350ff">AWS EKS: Create Your First Cluster Using AWS CDK</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Logging in Caddy]]></title>
            <link>https://danielpepuho.medium.com/exploring-logging-in-caddy-9188d44d4794?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9188d44d4794</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[caddy]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 18:59:13 GMT</pubDate>
            <atom:updated>2026-02-15T22:17:33.585Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQe7vPDr52UNOTVDTfeidA.png" /></figure><p>At some point, I realized that <strong><em>‘it works’</em></strong> is not enough. I need to know <strong><em>how it works</em></strong> too.”</p><p>When something slows down or fails, the question is no longer <strong><em>“is the service running?”</em></strong> but:</p><ul><li>Which component handled this request?</li><li>How long did it take?</li><li>Did the proxy retry or switch upstreams?</li></ul><p><strong>Without clear answers, even a simple issue can turn into a long-time debugging😆. This is where logging stops being a checkbox feature and starts becoming a core part of system design.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*KYM3Nz3lmwNmhBmr.jpg" /></figure><h3>Why Default Logs Are Not Enough?</h3><p>By default, Caddy already gives you access logs. They show incoming requests, response status codes, and basic metadata. For simple setups, that might be sufficient.</p><p>But once a system grows even slightly those logs start to fall short.</p><p>The first issue is noise. Default logs tend to mix everything together: health checks, internal probes, real user traffic. Important signals get buried under routine requests.</p><p>The second issue is missing context. When a request goes through a reverse proxy with load balancing, the most interesting part is often not the request itself, but what happens behind the scenes:</p><ul><li>How long did the upstream take to respond?</li><li>Was there a retry or a fallback?</li></ul><p>Without this information, debugging becomes guesswork. A 502 response tells you that something failed, but not where or why.</p><p><strong>This post explores how to make Caddy logs actually useful especially in a Dockerized setup with reverse proxying and load-balanced Go APIs.</strong></p><h3>Understanding Caddy Logging Model</h3><p>Like most web servers, Caddy separates access logs from error logs. <strong>Access logs tell you what happened to a request</strong>. <strong>Error logs tell you why something went wrong</strong>.</p><p><strong>1.</strong> Access Logs</p><p>Access logs describe the lifecycle of an incoming request:</p><ul><li>who made the request</li><li>what was requested</li><li>how the server responded</li><li>how long it took</li></ul><p>This is where most operational insights come from. When properly configured, access logs can tell you:</p><ul><li>which upstream handled a request</li><li>how traffic is distributed</li><li>where latency is introduced</li></ul><p><strong>2.</strong> Error Logs</p><p>Error logs focus on failures and exceptional cases:</p><ul><li>upstream connection errors</li><li>timeouts</li><li>TLS or protocol issues</li></ul><p>These logs are usually quieter, but they become critical when things break. They complement access logs by explaining why a request failed, not just that it failed.</p><h3>Enabling JSON Access Logs</h3><p>For this setup, I’m using a simple Go application as the backend worker. Each instance responds with its hostname, making it easy to see how requests are distributed by the load balancer.</p><pre>func handler(w http.ResponseWriter, r *http.Request) {<br>    hostname, _ := os.Hostname()<br>    fmt.Fprintf(w, &quot;Hello from %s\n&quot;, hostname)<br>}</pre><blockquote><em>The full load-balancing setup, Caddy in front of three Go workers running in Docker has already been covered in a previous post. If you need the full context, you can refer to here:</em></blockquote><blockquote><a href="https://danielcristho.site/blog/go-caddy-load-balancer#the-project-setup"><em>https://danielcristho.site/blog/go-caddy-load-balancer#the-project-setup</em></a></blockquote><h3>Logging in a Docker-Based Load Balancer</h3><p>When running Caddy inside Docker, the most practical logging strategy is to write logs to stdout and let Docker handle log collection.</p><p>When you first run the stack:</p><pre>$ docker compose up -d --build </pre><pre>$ docker logs load_balancer</pre><p>You’ll mostly see logs like:</p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5864136,&quot;msg&quot;:&quot;maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.586685,&quot;msg&quot;:&quot;GOMEMLIMIT is updated&quot;,&quot;package&quot;:&quot;github.com/KimMachineGun/automemlimit/memlimit&quot;,&quot;GOMEMLIMIT&quot;:16911289958,&quot;previous&quot;:9223372036854775807}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.586744,&quot;msg&quot;:&quot;using config from file&quot;,&quot;file&quot;:&quot;/etc/caddy/Caddyfile&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5883265,&quot;msg&quot;:&quot;adapted config to JSON&quot;,&quot;adapter&quot;:&quot;caddyfile&quot;}<br>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:1765883971.5883417,&quot;msg&quot;:&quot;Caddyfile input is not formatted; run &#39;caddy fmt --overwrite&#39; to fix inconsistencies&quot;,&quot;adapter&quot;:&quot;caddyfile&quot;,&quot;file&quot;:&quot;/etc/caddy/Caddyfile&quot;,&quot;line&quot;:2}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5897346,&quot;logger&quot;:&quot;admin&quot;,&quot;msg&quot;:&quot;admin endpoint started&quot;,&quot;address&quot;:&quot;localhost:2019&quot;,&quot;enforce_origin&quot;:false,&quot;origins&quot;:[&quot;//localhost:2019&quot;,&quot;//[::1]:2019&quot;,&quot;//127.0.0.1:2019&quot;]}<br>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:1765883971.5899262,&quot;logger&quot;:&quot;http.auto_https&quot;,&quot;msg&quot;:&quot;server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server&quot;,&quot;server_name&quot;:&quot;srv0&quot;,&quot;http_port&quot;:80}<br>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:1765883971.5901978,&quot;logger&quot;:&quot;http&quot;,&quot;msg&quot;:&quot;HTTP/2 skipped because it requires TLS&quot;,&quot;network&quot;:&quot;tcp&quot;,&quot;addr&quot;:&quot;:80&quot;}<br>{&quot;level&quot;:&quot;warn&quot;,&quot;ts&quot;:1765883971.590206,&quot;logger&quot;:&quot;http&quot;,&quot;msg&quot;:&quot;HTTP/3 skipped because it requires TLS&quot;,&quot;network&quot;:&quot;tcp&quot;,&quot;addr&quot;:&quot;:80&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5902092,&quot;logger&quot;:&quot;http.log&quot;,&quot;msg&quot;:&quot;server running&quot;,&quot;name&quot;:&quot;srv0&quot;,&quot;protocols&quot;:[&quot;h1&quot;,&quot;h2&quot;,&quot;h3&quot;]}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5905173,&quot;logger&quot;:&quot;tls.cache.maintenance&quot;,&quot;msg&quot;:&quot;started background certificate maintenance&quot;,&quot;cache&quot;:&quot;0xc00070c200&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5906706,&quot;msg&quot;:&quot;autosaved config (load with --resume flag)&quot;,&quot;file&quot;:&quot;/config/caddy/autosave.json&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.591785,&quot;msg&quot;:&quot;serving initial configuration&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5926607,&quot;logger&quot;:&quot;tls&quot;,&quot;msg&quot;:&quot;cleaning storage unit&quot;,&quot;storage&quot;:&quot;FileStorage:/data/caddy&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765883971.5949264,&quot;logger&quot;:&quot;tls&quot;,&quot;msg&quot;:&quot;finished cleaning storage units&quot;}</pre><blockquote><em>These are startup and internal runtime logs, not HTTP access logs. Access logs only appear after real HTTP traffic flows through Caddy.</em></blockquote><p>To enable JSON logging globally in the Caddyfile, you can add output stdout as shown below.</p><pre>{<br>    log {<br>        level INFO<br>        format json<br>    }<br>}<br>:80 {<br>    log {<br>        output stdout<br>        format json<br>    }<br>reverse_proxy worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy random<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p>At this point:</p><ul><li>logs are emitted to stdout</li><li>Docker captures them automatically</li></ul><p>Once the stack is running, send a few requests:</p><pre>curl http://localhost:8082<br>curl http://localhost:8082<br>curl http://localhost:8082</pre><p>You should see responses like:</p><pre>Hello from worker_1<br>Hello from worker_2<br>Hello from worker_3</pre><p>Now check the logs again:</p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884414.2170205,&quot;logger&quot;:&quot;tls&quot;,&quot;msg&quot;:&quot;cleaning storage unit&quot;,&quot;storage&quot;:&quot;FileStorage:/data/caddy&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884414.2187994,&quot;logger&quot;:&quot;tls&quot;,&quot;msg&quot;:&quot;finished cleaning storage units&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884423.2440372,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;51346&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Sec-Fetch-Mode&quot;:[&quot;navigate&quot;],&quot;Sec-Gpc&quot;:[&quot;1&quot;],&quot;Connection&quot;:[&quot;keep-alive&quot;],&quot;Priority&quot;:[&quot;u=0, i&quot;],&quot;Accept&quot;:[&quot;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&quot;],&quot;Sec-Fetch-Dest&quot;:[&quot;document&quot;],&quot;Sec-Fetch-Site&quot;:[&quot;none&quot;],&quot;User-Agent&quot;:[&quot;Mozilla/5.0 (X11; Linux x86_64; rv:145.0) Gecko/20100101 Firefox/145.0&quot;],&quot;Sec-Fetch-User&quot;:[&quot;?1&quot;],&quot;Accept-Language&quot;:[&quot;en-US,en;q=0.5&quot;],&quot;Accept-Encoding&quot;:[&quot;gzip, deflate, br, zstd&quot;],&quot;Upgrade-Insecure-Requests&quot;:[&quot;1&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000733253,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:03 GMT&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884423.7748682,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;51346&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;User-Agent&quot;:[&quot;Mozilla/5.0 (X11; Linux x86_64; rv:145.0) Gecko/20100101 Firefox/145.0&quot;],&quot;Sec-Fetch-User&quot;:[&quot;?1&quot;],&quot;Sec-Gpc&quot;:[&quot;1&quot;],&quot;Upgrade-Insecure-Requests&quot;:[&quot;1&quot;],&quot;Sec-Fetch-Dest&quot;:[&quot;document&quot;],&quot;Sec-Fetch-Site&quot;:[&quot;none&quot;],&quot;Accept-Language&quot;:[&quot;en-US,en;q=0.5&quot;],&quot;Accept-Encoding&quot;:[&quot;gzip, deflate, br, zstd&quot;],&quot;Priority&quot;:[&quot;u=0, i&quot;],&quot;Accept&quot;:[&quot;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&quot;],&quot;Connection&quot;:[&quot;keep-alive&quot;],&quot;Sec-Fetch-Mode&quot;:[&quot;navigate&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000783659,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:03 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884424.1060758,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;51346&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Upgrade-Insecure-Requests&quot;:[&quot;1&quot;],&quot;Sec-Fetch-Site&quot;:[&quot;none&quot;],&quot;Accept-Language&quot;:[&quot;en-US,en;q=0.5&quot;],&quot;Sec-Gpc&quot;:[&quot;1&quot;],&quot;Sec-Fetch-Dest&quot;:[&quot;document&quot;],&quot;Sec-Fetch-User&quot;:[&quot;?1&quot;],&quot;User-Agent&quot;:[&quot;Mozilla/5.0 (X11; Linux x86_64; rv:145.0) Gecko/20100101 Firefox/145.0&quot;],&quot;Accept&quot;:[&quot;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&quot;],&quot;Accept-Encoding&quot;:[&quot;gzip, deflate, br, zstd&quot;],&quot;Sec-Fetch-Mode&quot;:[&quot;navigate&quot;],&quot;Priority&quot;:[&quot;u=0, i&quot;],&quot;Connection&quot;:[&quot;keep-alive&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.002117571,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:04 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884424.456112,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;51346&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Connection&quot;:[&quot;keep-alive&quot;],&quot;Upgrade-Insecure-Requests&quot;:[&quot;1&quot;],&quot;Sec-Fetch-Site&quot;:[&quot;none&quot;],&quot;Sec-Fetch-User&quot;:[&quot;?1&quot;],&quot;Priority&quot;:[&quot;u=0, i&quot;],&quot;Accept-Language&quot;:[&quot;en-US,en;q=0.5&quot;],&quot;Sec-Fetch-Mode&quot;:[&quot;navigate&quot;],&quot;Accept&quot;:[&quot;text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8&quot;],&quot;Accept-Encoding&quot;:[&quot;gzip, deflate, br, zstd&quot;],&quot;Sec-Fetch-Dest&quot;:[&quot;document&quot;],&quot;User-Agent&quot;:[&quot;Mozilla/5.0 (X11; Linux x86_64; rv:145.0) Gecko/20100101 Firefox/145.0&quot;],&quot;Sec-Gpc&quot;:[&quot;1&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000959646,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:04 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884440.3606126,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;49074&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;],&quot;Accept&quot;:[&quot;*/*&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000425884,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:20 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884442.148407,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;49090&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Accept&quot;:[&quot;*/*&quot;],&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000592974,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:22 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884442.6677225,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;49102&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;],&quot;Accept&quot;:[&quot;*/*&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000432397,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:22 GMT&quot;]}}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765884443.2411807,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;49106&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Accept&quot;:[&quot;*/*&quot;],&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.000496529,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Date&quot;:[&quot;Tue, 16 Dec 2025 11:27:23 GMT&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Content-Length&quot;:[&quot;20&quot;]}}</pre><p>To make the logs more readable you can use jq:</p><pre>docker logs -f load_balancer 2&gt;&amp;1 | jq -R &#39;fromjson? | {level, logger, msg, request: {method: .request.method, uri: .request.uri}, status, duration}&#39;</pre><pre>...<br>{<br>  &quot;level&quot;: &quot;info&quot;,<br>  &quot;logger&quot;: &quot;tls&quot;,<br>  &quot;msg&quot;: &quot;finished cleaning storage units&quot;,<br>  &quot;request&quot;: {<br>    &quot;method&quot;: null,<br>    &quot;uri&quot;: null<br>  },<br>  &quot;status&quot;: null,<br>  &quot;duration&quot;: null<br>}<br>{<br>  &quot;level&quot;: &quot;info&quot;,<br>  &quot;logger&quot;: &quot;http.log.access.log0&quot;,<br>  &quot;msg&quot;: &quot;handled request&quot;,<br>  &quot;request&quot;: {<br>    &quot;method&quot;: &quot;GET&quot;,<br>    &quot;uri&quot;: &quot;/&quot;<br>  },<br>  &quot;status&quot;: 200,<br>  &quot;duration&quot;: 0.000733253<br>}<br>...</pre><h3>Which Upstream Handled This Request?</h3><p>Using Debug Logs to Expose Load Balancer Decisions</p><p>Earlier, we saw that standard access logs don’t explicitly show which backend handled a request. Caddy can expose upstream selection details, just not in access logs. The information lives in debug-level logs, specifically inside the reverse proxy handler.</p><p>By enabling debug mode, Caddy reveals how routing decisions are made internally.</p><pre>{<br>    log {<br>        level DEBUG<br>        format json<br>    }<br>}</pre><p>In this mode, a single request produces logs like this, you can see which upstream that handled request &quot;dial&quot;:&quot;worker_3:8081&quot;:</p><pre>{&quot;level&quot;:&quot;debug&quot;,&quot;logger&quot;:&quot;http.handlers.reverse_proxy&quot;,&quot;msg&quot;:&quot;selected upstream&quot;,&quot;dial&quot;:&quot;worker_3:8081&quot;,&quot;total_upstreams&quot;:3}</pre><h3>Simulating Failure (Error Logs)</h3><p>So far, everything looks fine. Requests succees and access logs show no anomalies.</p><p>Real systems rarely fail all at once. They fail partially one backend goes down, a health check starts failing, or a connection becomes unstable. In this setup, stopping one worker is enough:</p><pre>$ docker stop worker_2</pre><p>Try to hit the URL again:</p><pre>$ curl http://localhost:8082</pre><pre>Hello from worker_3<br>Hello from worker_1<br>Hello from worker_3</pre><p>From the client’s perspective, nothing breaks. Requests are still served, and responses keep coming back from the remaining healthy workers. This is exactly what we expect from a load balancer.</p><p>Now if you see the logs again, there is an unhealthy upstream:</p><pre>{: dial tcp: lookup worker_2 on 127.0.0.11:53: no such host&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765992098.8407533,&quot;logger&quot;:&quot;http.handlers.reverse_proxy.health_checker.active&quot;,&quot;msg&quot;:&quot;HTTP request failed&quot;,&quot;host&quot;:&quot;worker_2:8081&quot;,&quot;error&quot;:&quot;Get \&quot;http://worker_2:8081/\&quot;: dial tcp: lookup worker_2 on 127.0.0.11:53: no such host&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765992101.6597724,&quot;logger&quot;:&quot;http.handlers.reverse_proxy.health_checker.active&quot;,&quot;msg&quot;:&quot;HTTP request failed&quot;,&quot;host&quot;:&quot;worker_2:8081&quot;,&quot;error&quot;:&quot;Get \&quot;http://worker_2:8081/\&quot;: dial tcp: lookup worker_2 on 127.0.0.11:53: no such host&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765992104.640808,&quot;logger&quot;:&quot;http.handlers.reverse_proxy.health_checker.active&quot;,&quot;msg&quot;:&quot;HTTP request failed&quot;,&quot;host&quot;:&quot;worker_2:8081&quot;,&quot;error&quot;:&quot;Get \&quot;http://worker_2:8081/\&quot;: dial tcp: lookup worker_2 on 127.0.0.11:53: no such host&quot;}</pre><p>This message is emitted by Caddy’s active health checker. It indicates that:</p><ul><li>worker_2 is no longer reachable</li><li>the failure happens during health check probes</li></ul><h3>Customizing Log Output</h3><p>Up to this point, all examples use JSON access logs. They are structured, machine-friendly, and easy to filter.</p><p>Caddy allows you to control how logs are encoded, not just what gets logged. This is useful when the goal shifts from processing logs to reading them.</p><h3>Console Logs</h3><p>Besides json, Caddy also provides a console encoder. It formats logs in a more friendly way while still preserving structure.</p><pre>{<br>    log {<br>        level INFO<br>        format json<br>    }<br>}</pre><pre>:80 {<br>    log {<br>        output stdout<br>        format console<br>    }</pre><pre>    reverse_proxy worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy random<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><pre>$ docker restart load_balancer</pre><p>After restarting the container, access logs will look like this:</p><pre>$ docker logs load_balancer</pre><pre>2025/12/17 17:31:44.788 INFO    http.log.access.log0    handled request {&quot;request&quot;: {&quot;remote_ip&quot;: &quot;172.21.0.1&quot;, &quot;remote_port&quot;: &quot;36598&quot;, &quot;client_ip&quot;: &quot;172.21.0.1&quot;, &quot;proto&quot;: &quot;HTTP/1.1&quot;, &quot;method&quot;: &quot;GET&quot;, &quot;host&quot;: &quot;localhost:8082&quot;, &quot;uri&quot;: &quot;/&quot;, &quot;headers&quot;: {&quot;User-Agent&quot;: [&quot;curl/7.81.0&quot;], &quot;Accept&quot;: [&quot;*/*&quot;]}}, &quot;bytes_read&quot;: 0, &quot;user_id&quot;: &quot;&quot;, &quot;duration&quot;: 0.001321876, &quot;size&quot;: 20, &quot;status&quot;: 200, &quot;resp_headers&quot;: {&quot;Date&quot;: [&quot;Wed, 17 Dec 2025 17:31:44 GMT&quot;], &quot;Content-Length&quot;: [&quot;20&quot;], &quot;Content-Type&quot;: [&quot;text/plain; charset=utf-8&quot;], &quot;Via&quot;: [&quot;1.1 Caddy&quot;]}}</pre><h4>Choosing the Right Format</h4><p>There is no universally “correct” log format.</p><ul><li>JSON works best when logs are consumed by tools or pipelines.</li><li>Console works better when you are actively watching logs and debugging in real time.</li></ul><h3>Shaping Logs Without Changing Their Meaning</h3><p>Caddy also allows fine-grained control over how log fields are named and formatted. This is not about adding new information, but about making existing information easier to work with.</p><p>For example, you can:</p><ul><li>rename common fields</li><li>control timestamp formats</li><li>standardize duration units</li><li>normalize log levels</li></ul><pre>... <br>{<br>    log {<br>        output stdout<br>        format json {<br>            time_format     &quot;2006-01-02 15:04:05 MST&quot;<br>            time_local<br>            duration_format &quot;ms&quot;<br>            level_format    &quot;upper&quot;<br>        }<br>    }<br>}</pre><p>Here’s the updated access log after applying the custom JSON format:</p><pre>{sp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Date&quot;:[&quot;Wed, 17 Dec 2025 17:36:47 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;]}}<br>{&quot;level&quot;:&quot;debug&quot;,&quot;ts&quot;:1765993007.632117,&quot;logger&quot;:&quot;http.handlers.reverse_proxy&quot;,&quot;msg&quot;:&quot;selected upstream&quot;,&quot;dial&quot;:&quot;worker_1:8081&quot;,&quot;total_upstreams&quot;:3}<br>{&quot;level&quot;:&quot;debug&quot;,&quot;ts&quot;:1765993007.6339025,&quot;logger&quot;:&quot;http.handlers.reverse_proxy&quot;,&quot;msg&quot;:&quot;upstream roundtrip&quot;,&quot;upstream&quot;:&quot;worker_1:8081&quot;,&quot;duration&quot;:0.001633852,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;36378&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;X-Forwarded-For&quot;:[&quot;172.21.0.1&quot;],&quot;X-Forwarded-Proto&quot;:[&quot;http&quot;],&quot;X-Forwarded-Host&quot;:[&quot;localhost:8082&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;],&quot;Accept&quot;:[&quot;*/*&quot;]}},&quot;headers&quot;:{&quot;Date&quot;:[&quot;Wed, 17 Dec 2025 17:36:47 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;]},&quot;status&quot;:200}<br>{&quot;level&quot;:&quot;INFO&quot;,&quot;ts&quot;:&quot;2025-12-17 17:36:47 UTC&quot;,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;36378&quot;,&quot;client_ip&quot;:&quot;172.21.0.1&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;],&quot;Accept&quot;:[&quot;*/*&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:2,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Date&quot;:[&quot;Wed, 17 Dec 2025 17:36:47 GMT&quot;],&quot;Content-Length&quot;:[&quot;20&quot;],&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;]}}</pre><p>What Changed?</p><p>The structure of the log is still JSON, but key fields are now normalized:</p><ul><li>level is uppercase (INFO)</li><li>readable timestamps and localized</li><li>duration is expressed in milliseconds instead of fractional seconds</li></ul><p>This makes the log easier to read during debugging.</p><h3>Reducing Noise and Risk in Logs</h3><p>Once logs are readable and structured, the next concern is volume and exposure. Not every field in an access log is equally useful, and some of them are better left out entirely.</p><p>Caddy allows selective filtering at the log level, making it possible to reduce noise without changing how requests are handled.</p><h4>Reducing Noise by Removing Fields</h4><p>Access logs often include fields that are technically correct but not relevant, like request headers.</p><p>If headers are not part of your debugging workflow, you can remove them:</p><pre>...<br>log {<br>    format filter {<br>        request delete<br>        wrap json<br>    }<br>}</pre><p>With the request field removed entirely, access logs become significantly smaller:</p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;duration&quot;:0.001026852,&quot;size&quot;:20,&quot;status&quot;:200}</pre><p>In other cases, it’s enough to remove only specific nested fields:</p><pre>...<br>log {<br>    format filter {<br>        request&gt;headers&gt;Authorization delete<br>        resp_headers&gt;Server delete<br>        wrap json<br>    }<br>}</pre><p>Instead of deleting the entire request, selectively removing fields gives a better balance.</p><pre>&quot;headers&quot;: {&quot;User-Agent&quot;: [&quot;curl/7.81.0&quot;], &quot;Accept&quot;: [&quot;*/*&quot;]}</pre><h3>Obscuring Sensitive Information</h3><p>Some fields should not be stored in plain text at all. Rather than relying on downstream redaction, Caddy can obscure sensitive data at the source:</p><pre>log {<br>    format filter {<br>        request&gt;client_ip ip_mask {<br>            ipv4 24<br>            ipv6 56<br>        }<br>        wrap json<br>    }<br>}</pre><p>With IP masking enabled, the change is subtle but important:</p><pre>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1765994419.4105134,&quot;logger&quot;:&quot;http.log.access.log0&quot;,&quot;msg&quot;:&quot;handled request&quot;,&quot;request&quot;:{&quot;remote_ip&quot;:&quot;172.21.0.1&quot;,&quot;remote_port&quot;:&quot;58562&quot;,&quot;client_ip&quot;:&quot;172.21.0.0&quot;,&quot;proto&quot;:&quot;HTTP/1.1&quot;,&quot;method&quot;:&quot;GET&quot;,&quot;host&quot;:&quot;localhost:8082&quot;,&quot;uri&quot;:&quot;/&quot;,&quot;headers&quot;:{&quot;Accept&quot;:[&quot;*/*&quot;],&quot;User-Agent&quot;:[&quot;curl/7.81.0&quot;]}},&quot;bytes_read&quot;:0,&quot;user_id&quot;:&quot;&quot;,&quot;duration&quot;:0.00146917,&quot;size&quot;:20,&quot;status&quot;:200,&quot;resp_headers&quot;:{&quot;Content-Type&quot;:[&quot;text/plain; charset=utf-8&quot;],&quot;Date&quot;:[&quot;Wed, 17 Dec 2025 18:00:19 GMT&quot;],&quot;Via&quot;:[&quot;1.1 Caddy&quot;],&quot;Content-Length&quot;:[&quot;20&quot;]}}</pre><pre>```sh <br>&quot;client_ip&quot;: &quot;172.21.0.0&quot;</pre><p>The request remains traceable across logs, but the exact client address is no longer exposed.</p><blockquote><em>Filtering is not about producing minimal logs. It’s about producing </em><em>intentional logs.</em></blockquote><blockquote><em>I only apply filters after understanding:</em></blockquote><blockquote><strong><em>-</em></strong><em> which fields I actually use</em></blockquote><blockquote><strong><em>-</em></strong><em> which ones are never read</em></blockquote><h3>TL;DR</h3><ul><li>Access logs show what happened to a request.</li><li>Debug logs show how requests are routed.</li><li>Error logs show what fails behind the scenes.</li><li>Filtering and formatting reduce noise without hiding behavior.</li></ul><p>Aight, that’s all. Thank you for taking your time to read this post.</p><p>Feel free to give me feedback, tips, or a different perspective. I’d love to hear yours and continue the discussion.</p><p>Happy logging!🧐</p><p>Originally posted at <a href="https://danielcristho.site/blog/caddy-log-comprehensive">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9188d44d4794" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Image Compression: gzip vs zstd]]></title>
            <link>https://danielpepuho.medium.com/docker-image-compression-gzip-vs-zstd-1b342222d7e7?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b342222d7e7</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[discussion]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gzip]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Tue, 02 Dec 2025 14:57:22 GMT</pubDate>
            <atom:updated>2026-02-15T22:18:10.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gS9uvZn-Nsqlz_H1ddDICw.png" /></figure><p>Docker images are already compressed when you push them to registries like <a href="https://hub.docker.com/">Docker Hub</a>, <a href="https://markdown-to-medium.surge.sh/ghcr.io">GHCR</a>, <a href="https://aws.amazon.com/ecr">AWS ECR</a>, etc.</p><p>So why would anyone compress an image again by using <a href="https://www.gzip.org/">gzip</a> or <a href="https://github.com/facebook/zstd">zstd</a>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*GCRt6YFy_J32HYkE.png" /></figure><p>Because in the real world, engineers often need to:</p><ul><li>move images between servers (air-gapped networks),</li><li>migrates registries,</li><li>backup image to object storage,</li><li>load or save images repeatedly in Continuous Integration (CI) pipelines.</li></ul><p>In these cases, docker save produces a raw .tar file often huge. Compressing that tarball can cut transfer and storage time by 50-80%.</p><p><strong>But what’s the best compression tool? So we will test gzip vs zstd</strong>.</p><h3>🤔 When Do We Need Image Compression?</h3><p>Like I said before, you need to compress your image to:</p><ul><li>transfer image between SSH or local network (LAN),</li><li>work with offline or air-gapped servers,</li><li>backup images to object storage,</li><li>migrate to new registry.</li></ul><blockquote><strong><em>Skip compression when images are pushed directly to registry like DockerHub, registries are handled that</em></strong><em>.</em></blockquote><h3>Test Up</h3><p>To get a realistic number, we will test three images:</p><ul><li><a href="https://hub.docker.com/layers/library/alpine/latest/images/sha256-9eec16c5eada75150a82666ba0ad6df76b164a6f8582ba5cb964c0813fa56625">alpine:latest</a> -&gt; ~8MB</li><li><a href="https://hub.docker.com/layers/library/mariadb/10.6.18/images/sha256-23492dcccccd10285fd946f2d07956aed1dbd56f34d153d7de367840fc0afa88">maridb:10.6.18</a> -&gt; ~396MB</li><li><a href="https://hub.docker.com/repository/docker/danielcristh0/datascience-notebook/tags/python-3.10.11/sha256-7d1c9ee5ee5d2ad2051f5a2e56d0861e28ced6b8a76ce0928075426b3762b79d">Custom Jupyterlab image</a> -&gt; ~5.4GB</li></ul><h3>Environment</h3><h4>1. Local Computer</h4><ul><li>CPU: 4 cores / 8 threads</li><li>Storage: NVMe SSD</li><li>OS: Ubuntu 22.04</li><li>Docker Version: 27x</li><li>Tools: gzip, zstd, time, scp</li></ul><h4>2. VPS</h4><ul><li>CPU: 4 cores / 8 threads</li><li>Storage: VirtIO-backed SSD</li><li>OS: Ubuntu 22.04</li><li>Docker Version: 27x</li><li>Tools: gzip, zstd, time</li></ul><h3>Command used</h3><p>Below are the commands used to save, compress, transfer, decompress, and load the Docker images during testing.</p><h4>1. Save the image (local machine)</h4><pre>docker pull &lt;image_name&gt;<br>docker save &lt;image_name&gt; -o &lt;image_name&gt;.tar</pre><pre>$ docker pull alpine:latest<br><br>latest: Pulling from library/alpine<br>Digest: sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412<br>...</pre><pre>$ docker save alpine:latest -o alpine_latest.tar<br><br>$ docker save mariadb:10.6.18 -o mariadb_10.tar<br><br>$ docker save danielcristh0/datascience-notebook:python-3.10.11 -o jupyter_notebook.tar</pre><pre>$ ls -lh<br><br>total 5,6G<br>-rw------- 1 user group 8,3M Nov 28 19:51 alpine_latest.tar<br>-rw------- 1 user group 5,2G Nov 28 20:18 jupyter_notebook.tar<br>-rw------- 1 user group 384M Nov 28 20:14 mariadb_10.tar</pre><h4>2. Compress the image (local machine)</h4><p><strong>-</strong> gzip</p><pre>time gzip -k &lt;image&gt;.tar</pre><pre>$ time gzip -k alpine_latest.tar<br>gzip -k alpine_latest.tar  0,44s user 0,01s system 99% cpu 0,452 total<br><br>$ time gzip -k mariadb_10.tar<br>gzip -k mariadb_10.tar  17,21s user 0,62s system 99% cpu 17,979 total<br><br>$ time gzip -k jupyter_notebook.tar<br>gzip -k jupyter_notebook.tar  238,83s user 3,56s system 99% cpu 4:03,16 total</pre><ul><li>-k → keep original file</li><li>gzip uses a single CPU thread at its default level (≈ level 6)</li></ul><p><strong>-</strong>zstd</p><pre>time zstd -T0 -19 &lt;image&gt;.tar</pre><pre>$  time zstd -T0 -19 alpine_latest.tar<br>alpine_latest.tar    : 37.01%   (8617984 =&gt; 3189867 bytes, alpine_latest.tar.zst)<br>zstd -T0 -19 alpine_latest.tar  3,64s user 0,10s system 100% cpu 3,734 total<br><br>$  time zstd -T0 -19 mariadb_10.tar<br>mariadb_10.tar       : 16.95%   (402636288 =&gt; 68258055 bytes, mariadb_10.tar.zst)<br>zstd -T0 -19 mariadb_10.tar  172,89s user 0,66s system 191% cpu 1:30,81 total<br><br>$  time zstd -T0 -22 jupyter_notebook.tar<br>Warning : compression level higher than max, reduced to 19<br>zstd: jupyter_notebook.tar.zst already exists; overwrite (y/n) ? y<br>jupyter_notebook.tar : 24.79%   (5560227328 =&gt; 1378450873 bytes, jupyter_notebook.tar.zst)<br>zstd -T0 -22 jupyter_notebook.tar  4759,54s user 19,32s system 188% cpu 42:11,68 total</pre><ul><li>-T0 → use all CPU threads</li><li>-22 → request maximum compression (automatically reduced to -19 by zstd)</li></ul><h4>3. Transfer to VPS</h4><p><strong>-</strong> gzip</p><pre>time scp &lt;image_name&gt;.tar.gz user@vps:/tmp/</pre><pre>$ time scp alpine_latest.tar.gz onomi@myserver:/tmp<br>alpine_latest.tar.gz    100% 3588KB 174.8KB/s   00:20<br>scp alpine_latest.tar.gz onomi@myserver:/tmp  0,11s user 0,29s system 1% cpu 23,208 total<br><br>$ time scp mariadb_10.tar.gz onomi@myserver:/tmp/<br>mariadb_10.tar.gz   100%  114MB   2.2MB/s   00:50<br>scp mariadb_10.tar.gz onomi@myserver:/tmp/  0,46s user 0,84s system 2% cpu 52,457 total<br><br>$ time scp jupyter_notebook.tar.gz onomi@myserver:/tmp/<br>jupyter_notebook.tar.gz  100% 1765MB   3.4MB/s   08:35<br>scp jupyter_notebook.tar.gz onomi@myserver:/tmp/  5,03s user 10,42s system 2% cpu 8:38,50 total</pre><p><strong>-</strong> zstd</p><pre>time scp &lt;image_name&gt;.tar.zst user@vps:/tmp/</pre><pre>$ time scp alpine_latest.tar.zst onomi@myserver:/tmp<br>alpine_latest.tar.zst   100% 3115KB 343.4KB/s   00:09<br>scp alpine_latest.tar.zst onomi@myserver:/tmp  0,10s user 0,18s system 1% cpu 22,728 total<br><br>$ time scp mariadb_10.tar.zst onomi@myserver:/tmp/<br>mariadb_10.tar.zst  100%   65MB   3.0MB/s   00:21<br>scp mariadb_10.tar.zst onomi@myserver:/tmp/  0,29s user 0,59s system 3% cpu 23,285 total<br><br>$ time scp jupyter_notebook.tar.zst onomi@myserver:/tmp/<br>jupyter_notebook.tar.zst    100% 1315MB   2.3MB/s   09:44<br>scp jupyter_notebook.tar.zst onomi@myserver:/tmp/  3,94s user 7,64s system 1% cpu 9:46,33 tot</pre><h4>4. Load the Image on the Server (VPS)</h4><p>Now that the compressed images are transferred to the VPS, the next step is to decompress them and load the Docker image into the remote server.</p><p><strong>-</strong> gzip</p><pre>time gzip -dk &lt;image&gt;.tar.gz</pre><pre>$ time gzip -dk alpine_latest.tar.gz<br>real    0m0.189s<br>user    0m0.116s<br>sys     0m0.039s<br><br>$ time gzip -dk mariadb_10.tar.gz<br><br>real    0m5.108s<br>user    0m3.813s<br>sys     0m1.129s<br><br>$ time gzip -dk jupyter_notebook.tar.gz<br>real    1m8.344s<br>user    0m48.466s<br>sys     0m13.408s</pre><p><strong>-</strong> zstd</p><pre>time zstd -d &lt;image&gt;.tar.zst</pre><pre>$ time zstd -d alpine_latest.tar.zst<br>alpine_latest.tar.zst: 8617984 bytes<br><br>real    0m4.121s<br>user    0m0.041s<br>sys     0m0.043s<br><br>$ time zstd -d mariadb_10.tar.zst<br>mariadb_10.tar.zst  : 402636288 bytes<br>real    0m3.455s<br>user    0m0.983s<br>sys     0m0.927s<br><br>$ time zstd -d jupyter_notebook.tar.zst<br>jupyter_notebook.tar.zst: 5560227328 bytes<br>real    0m31.810s<br>user    0m14.599s<br>sys     0m13.600s</pre><blockquote><em>Decompression in </em><em>zstdis extremely fast, 5-10x faster than compression, even for large files.</em></blockquote><h4>5. Loading the Image Into Docker</h4><p>Once decompressed, load the .tar file:</p><pre>docker load -i &lt;image&gt;.tar</pre><pre>$ docker load -i jupyter_notebook.tar<br><br>Loaded image: danielcristh0/datascience-notebook:python-3.10.11</pre><pre>$ docker images<br><br>IMAGE                                               ID             DISK USAGE   CONTENT SIZE   EXTRA<br>danielcristh0/datascience-notebook:python-3.10.11   9b38bf7c570f       11.4GB         5.56GB</pre><h4>6. Analysis: gzip vs zstd</h4><p>After running all compression, transfer, decompression, and loading tests across three different Docker images, let’s compare gzip and zstd.</p><p>Size Comparison</p><blockquote><em>zstd consistently produces much smaller output files than </em><em>gzip, especially on medium and large images.</em></blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa0ed12e12f9ad23de7e10c693d5b819/href">https://medium.com/media/fa0ed12e12f9ad23de7e10c693d5b819/href</a></iframe><blockquote><em>zstd gives around 20-50% better compression than gzip.</em></blockquote><p>Speed</p><p>gzip is faster than zstd at compression.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d1a0596a5875dd944ffe49d0f3bf235/href">https://medium.com/media/7d1a0596a5875dd944ffe49d0f3bf235/href</a></iframe><blockquote><em>zstd gives better compression but requires significantly more CPU.</em></blockquote><p>Transfer Speed (via SCP)</p><blockquote><em>Because </em><em>zstd produces smaller files, transfer times are 2x faster. But on larger files, </em><em>zstd can still lose to </em><em>gzip depending on </em><strong><em>CPU and disk performance</em></strong><em>.</em></blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a13cfeb6bd0ddb77cf03b9af4ffee868/href">https://medium.com/media/a13cfeb6bd0ddb77cf03b9af4ffee868/href</a></iframe><h3>When Should You Use gzip or zstd?</h3><h3>Use zstd when you want</h3><ul><li>The smallest compressed Docker images</li><li>Fast decompression</li><li>Faster transfers across networks</li><li>Long-term backups</li></ul><h3>Use gzip when you want:</h3><ul><li>Fast compression</li><li>Low CPU usage</li><li>Simplicity</li></ul><h3>Conclusion</h3><p>If you need to compress Docker images, here’s the quick answer:</p><p>Use zstd when you want:</p><ul><li>Smaller archive sizes (around 20–50% smaller than gzip)</li><li>Faster decompression</li><li>Faster network transfers</li></ul><p>Use gzip when you want:</p><ul><li>Fast compression</li><li>Low CPU usage</li><li>Simplicity</li></ul><p>Aight, that’s all. Feel free to give me feedback, tips, or different benchmarks. I’d love to hear your feedback and continue the discussion.</p><p>Happy containerization! 🐳</p><p>Originally posted at <a href="https://danielcristho.site/blog/docker-image-compression">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b342222d7e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Firefox in Docker? Yes, with a GUI and noVNC]]></title>
            <link>https://danielpepuho.medium.com/running-firefox-in-docker-yes-with-a-gui-and-novnc-8f5f9ca9dbdb?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f5f9ca9dbdb</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Fri, 21 Nov 2025 14:48:31 GMT</pubDate>
            <atom:updated>2026-02-15T22:18:48.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cIkAlphrpA_uk2p-gBn0fQ.png" /></figure><p>Docker isn’t just for serve your code, appliactions. you can actually run a full desktop app inside it. In this project, I containerized Firefox with a virtual desktop and made it accessible through a browser using noVNC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Z5d0yVchr6tzT4KzTZaTHw.jpeg" /></figure><h3>What this project does?</h3><p>It creates a lightweight container that:</p><ul><li>Runs a minimal desktop environment (Fluxbox)</li><li>Launches Firefox <strong>-</strong> Serves a VNC display using x11vnc</li><li>Exposes that desktop through noVNC (so you can open it in your web browser)</li></ul><p>You can literally open Firefox running inside Docker, from your browser tab. All using a single docker compose up.</p><h3>How it works?</h3><p>Here’s a quick breakdown of what happens inside the container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*3zNTjmsrfK7wuMyQ.png" /><figcaption>Process breakdown of the components running inside the container</figcaption></figure><p>Everything runs headlessly, there’s no physical display, but the combo of Xvfb + Fluxbox gives Firefox a virtual desktop.</p><h3>🐋 Dockerfile Overview</h3><pre>FROM alpine:edge<br><br>RUN apk add --no-cache \<br>    faenza-icon-theme \<br>    firefox \<br>    fluxbox \<br>    xfce4 \<br>    xvfb \<br>    x11vnc \<br>    novnc \<br>    supervisor \<br>    bash \<br>    net-tools<br><br># Install noVNC &amp; websockify from source code<br>RUN apk add --no-cache git python3 py3-pip \<br>    &amp;&amp; rm -rf /usr/share/novnc \<br>    &amp;&amp; git clone https://github.com/novnc/noVNC.git /usr/share/novnc \<br>    &amp;&amp; git clone https://github.com/novnc/websockify.git /usr/share/novnc/utils/websockify \<br>    &amp;&amp; ln -sf /usr/share/novnc/vnc.html /usr/share/novnc/index.html<br><br>ENV DISPLAY=:1<br>ENV RESOLUTION=1920x1080x24<br><br># Set vnc password<br>ARG VNC_PASS=dummypass<br><br># Create vnc password file<br>RUN mkdir -p /root/.vnc &amp;&amp; \<br>    x11vnc -storepasswd &quot;$VNC_PASS&quot; /root/.vnc/passwd<br><br>COPY supervisord.conf /etc/supervisord.conf<br><br>EXPOSE 5900 6080<br><br>CMD [&quot;supervisord&quot;, &quot;-c&quot;, &quot;/etc/supervisord.conf&quot;, &quot;-n&quot;]</pre><h3>📦 Docker Compose Setup</h3><pre>version: &#39;3.8&#39;<br>services:<br>  vnc_firefox:<br>    build: .<br>    container_name: vnc_firefox<br>    ports:<br>      - &quot;5901:5900&quot; # VNC<br>      - &quot;6080:6080&quot; # noVNC web UI<br>     healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;netstat -tln | grep -q 6080 || exit 1&quot;]<br>      interval: 1m30s<br>      timeout: 30s<br>      retries: 5<br>      start_period: 30s</pre><h3>🔧 Under the Hood (process supervision)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*jIlqSnknvFbAgpod.jpg" /></figure><p>Everything is managed by supervisord, which runs:</p><ul><li>Xvfb - virtual framebuffer display</li><li>x11vnc - provides VNC access</li><li>fluxbox - lightweight window manager (WM)</li><li>firefox - your GUI browser</li><li>novnc_proxy - web socket bridge</li></ul><p>Example supervisord.conf:</p><pre># Supervisor main config<br>[supervisord]<br>nodaemon=true<br>logfile=/var/log/supervisord.log<br>pidfile=/var/run/supervisord.pid<br>childlogdir=/var/log<br><br># XVirtual Framebuffer (Xvfb)<br># Creates a virtual display environment (:1)<br>[program:xvfb]<br>command=/usr/bin/Xvfb :1 -screen 0 1920x1080x24<br>autostart=true<br>autorestart=true<br>priority=10<br># x11vnc, VNC server<br>[program:x11vnc]<br>command=/usr/bin/x11vnc -display :1 -rfbauth /root/.vnc/passwd -forever -shared -rfbport 5900<br>autostart=true<br>autorestart=true<br>priority=20<br># fluxbox, lightweight window manager<br>[program:fluxbox]<br>command=/usr/bin/fluxbox<br>environment=DISPLAY=&quot;:1&quot;<br>autostart=true<br>autorestart=true<br>priority=30<br># Runs firefox inside the Xvfb + Fluxbox environment<br>[program:firefox]<br>command=/usr/bin/firefox<br>environment=DISPLAY=&quot;:1&quot;<br>autostart=true<br>autorestart=true<br>priority=40<br># noVNC, WebSocket VNC Proxy<br># Bridges VNC (port 5900) to a web interface (port 6080)<br>[program:novnc]<br>command=/usr/share/novnc/utils/novnc_proxy --vnc localhost:5900 --listen 6080<br>autostart=true<br>autorestart=true<br>priority=50</pre><p>Our project structure should look likes this:</p><pre>.<br>├── docker-compose.yml<br>├── Dockerfile<br>└── supervisord.conf</pre><h3>How to Run It?</h3><ol><li>Build &amp; start the container:</li></ol><pre>$ docker compose up --build -d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/0*WZmzaUeAV2InbUxp.gif" /><figcaption>Running the project</figcaption></figure><p>2. Access it by using your browser or VNC client. Using <strong>dummypass</strong> as password:</p><ul><li>Using noVNC Web: http://localhost:6080</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*MP6nAjMNj9oMN07Q.gif" /><figcaption>Acess using your browser</figcaption></figure><ul><li>Open any VNC viewer (e.g., RealVNC, TigerVNC, Remmina), then connect to: localhost:5901</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*_VvWbLYqXx2-eI0h.gif" /><figcaption>Acess using your VNC client</figcaption></figure><h3>What’s Next?</h3><p>While this is mostly a fun experiment, it can be used for:</p><ul><li>Headless browser testing environments</li><li>Remote browsing (isolated Firefox)</li><li>Demonstrating GUI automation setups</li></ul><p><strong>NB</strong>: 😎 This post is part of my <strong>“Container Stuffs”</strong> open-source project on GitHub. Upcoming labs include:</p><ul><li>PostgreSQL master–replica setup</li><li>Redis Sentinel cluster</li><li>Multi-node Docker Swarm simulations</li><li>and more to come!</li></ul><p><a href="https://github.com/danielcristho/container-stuffs">GitHub - danielcristho/container-stuffs: A hands on Docker lab for learning and experimenting with container setups, from single-node to multi-node environments.</a></p><h3>Conclusion</h3><p>Docker is more than backend services you can literally containerize <strong>entire user experiences</strong>.</p><p>Projects like this prove that containers aren’t limited to APIs, databases, and background workers. With a bit of creativity, you can run full desktops, interactive UIs, browsers, automation toolchains, and even full development environments inside isolated, reproducible containers.</p><p>Running Firefox inside Docker with noVNC shows how:</p><ul><li>system-level components (Xvfb, window managers, VNC)</li><li>web technologies (WebSockets, noVNC)</li><li>and container orchestration (Docker + Compose) can blend together to create something both useful and fun.</li></ul><p>Cheers, feel free to give me a feedback, and happy containerizing! 🐳</p><p>Originally posted at <a href="https://danielcristho.site/blog/run-firefox-on-docker">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f5f9ca9dbdb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Load Balancing in Caddy Using Docker]]></title>
            <link>https://danielpepuho.medium.com/exploring-load-balancing-in-caddy-using-docker-26e396a2570b?source=rss-5c346eb781b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/26e396a2570b</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[caddy]]></category>
            <dc:creator><![CDATA[Daniel Pepuho]]></dc:creator>
            <pubDate>Wed, 27 Aug 2025 15:51:59 GMT</pubDate>
            <atom:updated>2026-02-15T22:19:44.293Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQe7vPDr52UNOTVDTfeidA.png" /></figure><p>Hello there! In this post, we’ll dive into the world of Caddy, a modern and powerful web server. Built using Go, Caddy offers a range of built-in features, including <strong>reverse proxy</strong> and <strong>load balancing</strong>.</p><p>By the way, in this project, I’ll use <strong>Caddy 2</strong>, the latest version, which comes with a completely rewritten architecture and enhanced features. To make our setup and testing as smooth as possible, we’ll leverage Docker. Docker allows us to create a reproducible environment with all the necessary components easily.</p><p>The main focus of this post will be on understanding and experimenting with Caddy’s <strong>load balancing algorithms</strong>. We’ll explore how Caddy intelligently distributes incoming traffic, the different strategies it offers (such as <strong>Round Robin</strong>, <strong>Least Connection</strong>, <strong>Random</strong>, and more), and how you can configure the for various use cases.</p><p>By using Docker, we’ll quickly spin up multiple backend services (which are called “workers”) and route requests through a single Caddy instance acting as our load balancer.</p><h3>The Project Setup</h3><p>To get started, let’s look at the project structure. Our setup is straightforward, allowing us to focus on the core concepts of load balancing.</p><p>Here’s the project structure:</p><pre>├── caddy<br>│   └── Caddyfile<br>├── docker-compose.yml<br>└── src<br>    ├── Dockerfile<br>    ├── go.mod<br>    └── main.go</pre><p>Our setup is composed of three main parts:</p><p><strong>1.</strong> <strong>Backend Workers (</strong><strong>src)</strong>: These are simple Go applications that will handle the requests. Each worker will simply return a “Hello from [hostname]” message, allowing us to see which server is handling the request. This is the perfect way to visualize how Caddy distributes the load.</p><p><strong>2.</strong> <strong>Caddy Load Balancer (</strong><strong>caddy)</strong>: Our Caddy instance will act as the reverse proxy, forwarding requests to the worker services. We’ll use the Caddyfile to define our load balancing rules.</p><p><strong>3.</strong> <strong>Docker Compose (</strong><strong>docker.compose.yml)</strong>: This file orchestrates everything, defining and running our multi-container application with a single command.</p><h3>Understanding the Components</h3><p>Let’s break down the files in our project.</p><p>src/main.go</p><p>This is a simple Go HTTP server. It listens on port 8081 and responds to any request by printing its hostname. This is our primary tool for observing the load balancing behavior.</p><pre>package main<br><br>import (<br>    &quot;fmt&quot;<br>    &quot;net/http&quot;<br>    &quot;os&quot;<br>)<br>func handler(w http.ResponseWriter, r *http.Request) {<br>    hostname, _ := os.Hostname()<br>    fmt.Fprintf(w, &quot;Hello from %s\n&quot;, hostname)<br>}<br>func main() {<br>    http.HandleFunc(&quot;/&quot;, handler)<br>    http.ListenAndServe(&quot;:8081&quot;, nil)<br>}</pre><p>src/Dockerfile</p><p>This Dockerfile builds our Go application into a lightweight, self-contained Docker image.</p><pre>FROM golang:1.24 AS builder<br><br>WORKDIR /go/src/app<br>COPY . .<br><br>RUN go mod download<br><br>RUN CGO_ENABLED=0 go build -o /go/bin/app<br><br>FROM golang:1.24-alpine<br><br>COPY --from=builder /go/bin/app /<br><br>EXPOSE 8081<br>ENV PORT 8081<br><br>CMD [&quot;/app&quot;]</pre><p>docker-compose.yml</p><p>This file ties everything together. We define a service for our Caddy load balancer and three worker services.</p><pre>version: &quot;3.8&quot;<br><br>services:<br>  load_balancer:<br>    image: caddy:2.10-alpine<br>    container_name: load_balancer<br>    ports:<br>      - &quot;8082:80&quot;<br>    volumes:<br>      - ./caddy/Caddyfile:/etc/caddy/Caddyfile<br><br>  worker_1:<br>    build: ./src<br>    container_name: worker_1<br>    hostname: worker_1<br>    expose:<br>      - &quot;8081&quot;<br><br>  worker_2:<br>    build: ./src<br>    container_name: worker_2<br>    hostname: worker_2<br>    expose:<br>      - &quot;8081&quot;<br><br>  worker_3:<br>    build: ./src<br>    container_name: worker_3<br>    hostname: worker_3<br>    expose:<br>      - &quot;8081&quot;</pre><p>caddy/Caddyfile</p><p>This is where the magic happens! This Caddyfile is Caddy’s configuration file. We’ll define a <strong>reverse proxy</strong> that routes to our workers. The lb_policy directive is where we’ll specify our load balancing algorithms.</p><pre>:80 {<br>    reverse_proxy worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy round_robin<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><h3>Experimenting with Load Balancing Algorithms</h3><p>Now that our project is set up, we can start experimenting. To run the project, simply execute docker-compose up in your terminal. You can then send requests to http://127.0.0.1:8082 and observe which worker responds.</p><p><strong>1.</strong> Round Robin (lb_policy round_robin)</p><p>This is the most common and simplest load balancing algorithm. Caddy distributes incoming requests to the backend servers in a sequential, rotating manner. It’s a fair and predictable method, assuming all servers are equally capable of handling the load.</p><p><strong>How to Configure</strong>:</p><p>Modify your caddy/Caddyfile to use the round_robin policy.</p><pre>:80 {<br>    reverse_proxy worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy round_robin<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p><strong>How to Test</strong>:</p><p>After runningdocker-compose up -d --build, open your terminal and send a few requests using curl. You should see that Caddy distributes the traffic evenly among the three workers.</p><pre>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_1<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_2<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_3<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_1</pre><p>Now, let’s test Caddy’s fault tolerance. In a real-world scenario, a server might crash or become unresponsive. We’ll simulate this by manually stopping one of the workers.</p><p>Run the following command in your terminal to stop worker_1:</p><pre>$ docker stop worker_1</pre><p>After a few moments (the health_interval you set, e.g., 3 seconds), Caddy will perform its next health check, detect that worker_1 is unresponsive, and automatically mark it as unhealthy.</p><p>Now, send a few more requests. What do you expect to happen? With worker_1 down, Caddy should intelligently stop routing traffic to it and redirect all requests to the remaining healthy servers (worker_2 and worker_3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*EZNfgL8QtnxGmUYO.gif" /></figure><p><strong>2.</strong> Weighted Round Robin (lb_policy weighted_round_robin)</p><p>This algorithm is a more advanced version of Round Robin. It allows you to assign a “weight” to each backend server, which determines its share of the requests. <strong>Servers with a higher weight will receive more traffic than those with a lower weight</strong>. <strong>This is ideal when you have servers with varying capacities, for example, a new, more powerful server and an older, less powerful one</strong>.</p><p>You can also use this policy to gradually drain traffic from an old server or ramp up traffic to a new one during deployments, making it a very useful strategy.</p><p><strong>How to Configure</strong>:</p><p>To use this policy, you need to add the weight to each server’s address in the Caddyfile. For our example, let’s give worker_1 a higher weight of 3, while worker_2 and worker_3 each have a weight of 1. This means worker_1 should handle three out of every five requests.</p><pre>:80 {<br>    reverse_proxy  worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy weighted_round_robin 3 1 1<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p>After updating the Caddyfile, make sure to reload or restart your Caddy container to apply the changes. You can do this with docker-compose up -d --build.</p><p><strong>How to Test</strong>:</p><p>Now, let’s send a few requests to our load balancer and see how Caddy distributes the traffic according to the assigned weights. Send a few requests using curl and observe the responses.</p><pre>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_1<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_1<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_2<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_3<br><br>$ curl http://127.0.0.1:8082<br># Output: Hello from worker_1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*cs65v0L-_3zEHR2w.gif" /></figure><p><strong>3.</strong> Least Connection (lb_policy ip_hash)</p><p>Aight, let’s dive into Least Connection. Unlike Round Robin, which is a simple, sequential algorithm, Least Connection is a dynamic and more intelligent load balancing policy. It chooses the backend server with the fewest number of currently active requests. This policy is excellent for situations where your requests have a highly variable processing time.</p><p>For example, if one of your servers gets a handful of complex, long-running requests while the others are handling many small, quick ones, this algorithm will automatically route new traffic to the servers that are less burdened, preventing a single server from becoming a bottleneck. If there’s a tie, meaning two or more servers have the same lowest number of connections, Caddy will randomly choose one of them.</p><p><strong>How to Configure</strong>:</p><p>Configuring this policy is simple. You just need to change the lb_policy directive in your Caddyfile.</p><pre>:80 {<br>    reverse_proxy worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy least_conn<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p>After updating yourCaddyfile, make sure to restart your Caddy container with docker-compose up -d --build to apply the changes.</p><p><strong>How to Test</strong>:</p><p>To demonstrate the Least Connection algorithm, you’ll need to modify your Go code to simulate a long-running request. This will allow you to see how Caddy intelligently routes traffic away from the busy worker.</p><p><strong>-</strong> Update Your Go Code</p><p>Open your src/main.go file and add a new handler that will simulate a task with a significant delay. This will act as our “long-running request.”</p><pre>package main<br><br>import (<br>    &quot;fmt&quot;<br>    &quot;net/http&quot;<br>    &quot;os&quot;<br>    &quot;time&quot;<br>)<br><br>func handler(w http.ResponseWriter, r *http.Request) {<br>    hostname, _ := os.Hostname()<br>    fmt.Fprintf(w, &quot;Hello from %s\n&quot;, hostname)<br>}<br><br>func longHandler(w http.ResponseWriter, r *http.Request) {<br>    hostname, _ := os.Hostname()<br>    time.Sleep(10 * time.Second)<br>    fmt.Fprintf(w, &quot;Hello! long running request finished from %s\n&quot;, hostname)<br>}<br><br>func main() {<br>    http.HandleFunc(&quot;/&quot;, handler)<br>    http.HandleFunc(&quot;/long&quot;, longHandler)<br>    http.ListenAndServe(&quot;:8081&quot;, nil)<br>}</pre><p><strong>-</strong> Rebuild and Run Docker Compose</p><p>After updating your code, you must rebuild and run your containers to apply the changes.</p><pre>$ docker-compose up -d --build</pre><p><strong>-</strong> Test the Scenario</p><p>Now, you can test the Least Connection algorithm using two separate terminals.</p><p>Terminal 1 (Long-Running Request):</p><p>Start a request to the /long endpoint. This will open a connection to one of the workers and hold it for 10 seconds. Caddy will detect that this worker has an active, ongoing connection.</p><pre>$ curl http://127.0.0.1:8082/long</pre><p>Terminal 2 (Normal Requests):</p><p>Immediately after running the command in Terminal 1, switch to Terminal 2 and send several quick requests to the root endpoint (/).</p><pre>$ curl http://127.0.0.1:8082/<br># Output: Hello from worker_2<br><br>$ curl http://127.0.0.1:8082/<br># Output: Hello from worker_3<br><br>$ curl http://127.0.0.1:8082/<br># Output: Hello from worker_2</pre><p>You will observe that Caddy will not send requests to the worker that is currently busy with the long-running request. Instead, all new requests will be routed to the other two workers, demonstrating how the least_conn algorithm effectively balances load dynamically. Once the long-running request is complete, that worker will once again be available to handle new requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*0L_3JRtSAV4NQMdJ.gif" /></figure><p><strong>4.</strong> IP Hash (lb_policy ip_hash)</p><p>The IP Hash load balancing algorithm is different from the previous ones because it’s focused on session persistence. Instead of distributing requests based on a sequential or random order, it creates a hash from the client’s IP address and uses that hash to consistently route all requests from that same client to the <strong>same backend server</strong>.</p><p><strong>How to Configure</strong>:</p><p>Configuring the IP Hash policy is straightforward. You simply need to replace the lb_policy directive in your Caddyfile with ip_hash.</p><pre>:80 {<br>    reverse_proxy  worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy ip_hash<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p>After updating your Caddyfile, make sure to restart your Caddy container with docker-compose up -d --build to apply the changes.</p><p><strong>How to Test</strong>:</p><p>To test this algorithm, you’ll need to send requests from different “clients” (i.e., different IP addresses) and observe where they are routed. The easiest way to simulate this is by sending requests from your local machine and then using a proxy or a different network to see if the requests are routed to a different server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*c17QR1OyR4LQBWpE.gif" /></figure><p>No matter how many times you run curl from the same machine, the requests will always be routed to the same worker. <strong>This is because Caddy is hashing your local IP address (127.0.0.1 or the container’s internal IP) and consistently mapping it to that specific worker</strong>.</p><p>This demonstrates how IP Hash ensures <strong>session stickiness</strong> without needing to share session data across all servers. It’s a powerful tool for maintaining a consistent user experience.</p><p><strong>5.</strong> Random (lb_policy random)</p><p>The Random load balancing policy is the simplest and most unpredictable of all the algorithms. As its name suggests, it selects a backend server at random for each new request. There is no sequential pattern or special logic; every request has an equal chance of being routed to any of the available servers.</p><p>While it may seem less sophisticated than other algorithms, the Random policy is surprisingly effective in many scenarios. It’s fast, has a very low overhead, and can be a great choice for distributing traffic evenly across a large pool of homogenous servers. It naturally avoids the <strong>“thundering herd”</strong> problem that can sometimes occur with Round Robin on <strong>first-come-first-served</strong> requests, as it prevents all clients from hitting the same server at the same time.</p><p><strong>How to Configure</strong>:</p><p>Configuring this policy is the easiest. Simply replace the lb_policy directive in your Caddyfile with random.</p><pre>:80 {<br>    reverse_proxy  worker_1:8081 worker_2:8081 worker_3:8081 {<br>        lb_policy random<br>        health_uri /<br>        health_interval 3s<br>    }<br>}</pre><p>After updating your Caddyfile, make sure to restart your Caddy container with docker-compose up -d --build to apply the changes.</p><p><strong>How to Test</strong>:</p><p>To test the Random policy, send a series of quick requests and observe the output. Unlike the predictable pattern of Round Robin or the consistent output of IP Hash, the responses will come from different workers in an unpredictable order.</p><pre>curl http://127.0.0.1:8082<br># Output: Hello from worker_3<br><br>curl http://127.0.0.1:8082<br># Output: Hello from worker_1<br>curl http://127.0.0.1:8082<br><br># Output: Hello from worker_2<br>curl http://127.0.0.1:8082<br># Output: Hello from worker_1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*zTKpRquu1pnI9-i8.gif" /></figure><h3>Conclusion</h3><p>Throughout this post, we’ve explored the core capabilities of Caddy as a powerful web server and a flexible load balancer. Using a simple Docker setup, we were able to quickly demonstrate five different load balancing algorithms, each with its own unique advantages:</p><p><strong>-</strong> <strong>Round Robin</strong>: The classic, simple approach for evenly distributing traffic in a predictable sequence.</p><p><strong>-</strong> <strong>Weighted Round Robin</strong>: A smarter version of Round Robin that allows you to prioritize traffic to more powerful servers.</p><p><strong>-</strong> <strong>Least Connection</strong>: A dynamic algorithm that routes traffic based on real-time load, preventing a single server from becoming a bottleneck.</p><p><strong>-</strong> <strong>IP Hash</strong>: The ideal choice for session stickiness, ensuring a consistent user experience by always routing a client to the same backend server.</p><p><strong>-</strong> <strong>Random</strong>: A straightforward and fast algorithm for scattering traffic across servers, effective for a large pool of homogenous workers.</p><p>This hands-on experience proved that Caddy is not just a simple web server but a robust tool for building scalable and reliable applications. Caddy’s elegant syntax and powerful features make it an excellent choice for anyone looking to simplify their server configurations without sacrificing control or performance. Whether you’re a seasoned developer or just starting, Caddy offers a smooth and intuitive experience that can handle everything from a single website to a complex, distributed application.</p><p>References:</p><p><strong>-</strong> <a href="https://caddyserver.com/docs/caddyfile/directives/reverse_proxy">Caddy Documentation: Reverse Proxy</a></p><ul><li><a href="https://caddy.community/">Caddy Community Wiki</a></li></ul><p>Originally published at <a href="https://danielcristho.site/blog/go-caddy-load-balancer">danielcristho.site</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26e396a2570b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>