<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akriotis Kyriakos on Medium]]></title>
        <description><![CDATA[Stories by Akriotis Kyriakos on Medium]]></description>
        <link>https://medium.com/@akyriako?source=rss-c6fda6c1c195------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sXbxbYLxtfGqB-uhlwdj1A.jpeg</url>
            <title>Stories by Akriotis Kyriakos on Medium</title>
            <link>https://medium.com/@akyriako?source=rss-c6fda6c1c195------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:59:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akyriako/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Deploy a Production-Grade vLLM Stack on T Cloud Public CCE]]></title>
            <link>https://itnext.io/how-to-deploy-a-production-grade-vllm-stack-on-t-cloud-public-cce-b79894043f87?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/b79894043f87</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[open-telekom-cloud]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 06:14:04 GMT</pubDate>
            <atom:updated>2026-07-03T07:29:16.762Z</atom:updated>
            <content:encoded><![CDATA[<p>Build and scale a production-grade LLM inference platform on T Cloud Public CCE with the vLLM Production Stack, Kubernetes, and modern cloud-native practices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2QNasvt7O5n_9rP1txcgw.png" /></figure><h3>TL;DR</h3><p>In this guide, we’ll deploy the <strong>vLLM Production Stack</strong> on <strong>T Cloud Public CCE</strong> and walk through the most common production deployment patterns. We’ll start with a <strong>single-GPU deployment</strong> as the quickest way to get a production-ready inference service running. From there, we’ll explore <strong>multi-node distributed serving</strong> for large-scale models, <strong>single-node tensor parallelism</strong> to leverage multiple GPUs within a single machine, and <strong>disaggregated prefill/decode serving</strong> for advanced, high-throughput inference workloads. By the end of this article, you’ll understand when to use each scenario and have the foundation to deploy and scale vLLM on Kubernetes with confidence.</p><h3>Introduction</h3><p><a href="https://docs.vllm.ai/en/stable/">vLLM</a> is an open-source inference and serving engine for large language models. It is designed to improve serving throughput and GPU memory efficiency, mainly through <a href="https://huggingface.co/docs/text-generation-inference/en/conceptual/paged_attention">PagedAttention</a>, continuous batching, prefix caching, and an OpenAI-compatible serving interface.</p><p>The <a href="https://docs.vllm.ai/projects/production-stack/en/latest/">vLLM Production Stack</a> is a separate upstream project from the vLLM ecosystem. It provides a Kubernetes-native, cluster-wide reference implementation for deploying inference services on top of vLLM. Its purpose is to help users move from a single vLLM instance to a distributed deployment without changing application code. It also includes deployment options such as Helm chart deployment, CRD-based deployment, and Gateway API integration, together with documented capabilities such as metrics dashboards, request routing, KV cache-aware routing, prefix-aware routing, distributed tracing, KEDA-based autoscaling, and other production-oriented patterns.</p><h3>Prerequisites</h3><p>The prerequisites for this guide are identical to those described in the blueprint <a href="https://arch.otc-service.com/docs/blueprints/by-use-case/ai/litellm/litellm">Build an Unified LLM Gateway with LiteLLM on CCE</a>. Before proceeding, ensure that all required services, infrastructure components, access permissions, and configuration prerequisites outlined in the blueprint have been completed.</p><ol><li>a <strong>Cloud Container Engine (CCE)</strong> cluster, with at least one GPU node</li><li>A domain name with DNS management delegated to the T Cloud Public DNS service</li><li>a bastion host in <strong>Elastic Cloud Service (ECS)</strong>; <em>optional but strongly recommended</em></li><li>a <strong>Distributed Cache Service (DCS)</strong> Redis instance</li><li>a <strong>Relational Database Service (RDS)</strong> PostgreSQL server</li><li>an <strong>Object Storage Service</strong> bucket</li><li>an <strong>SFS Turbo</strong> file system</li><li>A Hugging Face read-only token</li></ol><p>Before proceeding to any deployment and configuration ensure that the necessary namespace is created:</p><pre>kubectl create namespace vllm</pre><h3>Creating the Secret</h3><p>Before deploying LiteLLM, a Kubernetes Secret must be created; create a file named <strong>vllm-prodstack-secrets.yaml</strong> to provide the required runtime configuration and credentials:</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: vllm-prodstack-secrets<br>type: Opaque<br>stringData:<br>  VLLM_MASTER_KEY: &lt;VLLM_MASTER_KEY&gt;<br>  HF_TOKEN: &lt;HF_TOKEN&gt;</pre><blockquote>Each key in this secret serves a specific purpose:</blockquote><blockquote>VLLM_MASTER_KEY: is used in this deployment as a shared secret for authenticating internal communication between vLLM Production Stack components. The value is stored as a Kubernetes Secret and injected into the workloads as an environment variable. All participating services must use the same key value to establish trusted communication. For production environments, the key should be generated as a strong random secret and managed securely through Kubernetes Secrets or external secret management solutions.</blockquote><blockquote>HF_TOKEN: s used to authenticate against <a href="https://huggingface.co/">Hugging Face</a> services. The deployment may require access to Hugging Face-hosted model repositories, tokenizer assets, or embedding models during model initialization and runtime operations. The token is primarily required when using gated or private repositories, while publicly accessible models can generally be downloaded without authentication.</blockquote><blockquote>In this blueprint, the Secret separates sensitive runtime credentials from the public Helm configuration. This makes the configuration easier to maintain and reduces the risk of exposing API keys, session-signing secrets, or third-party access tokens.</blockquote><pre>kubectl apply -f vllm-prodstack-secrets.yaml -n vllm</pre><h3>Creating the Persistence Volume Claim</h3><p>vLLM persists model artifacts locally once they are pulled. In a single-replica setup, a node-local or ReadWriteOnce volume is sufficient. However, when scaling to multiple replicas, each pod would otherwise maintain its own copy of the models. This leads to redundant storage consumption, increased network usage during model downloads, and longer initialization times.</p><p>To address this, a shared storage backend is required. SFS Turbo provides a managed, POSIX-compliant file system with ReadWriteMany semantics, allowing multiple pods across nodes to access the same data concurrently. By mounting SFS Turbo as the persistent volume for Ollama, all replicas can reuse a single set of model files. This approach improves operational efficiency by reducing duplication and ensuring consistency across replicas. It also simplifies scaling, as new pods can immediately access preloaded models without requiring additional initialization steps.</p><p>First we need to create a PersistentVolumeClaim, namely pvc-vllm-models.yaml and mount the SFS Turbo file system we created in the previous step:</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: pvc-vllm-models<br>  labels:<br>    model: &quot;vllm-models&quot;<br>  annotations:<br>    everest.io/volume-as: absolute-path                <br>    everest.io/sfsturbo-share-id: &lt;SFSTURBO_SHARE_ID&gt;        <br>    everest.io/path: /vllm-models                                <br>    everest.io/reclaim-policy: retain-volume-only      <br>    everest.io/csi.enable-sfsturbo-dir-quota: &quot;true&quot;   <br>spec:<br>  accessModes:<br>    - ReadWriteMany      <br>  resources:<br>    requests:<br>      storage: 500Gi      <br>  storageClassName: csi-sfsturbo     </pre><blockquote>Replace the value of everest.io/sfsturbo-share-id with the one matching your environment.</blockquote><pre>kubectl create namespace vllm<br>kubectl apply -f pvc-vllm-models</pre><h3>Installing KubeRay Operator (Optional)</h3><p><a href="https://github.com/ray-project/kuberay">KubeRay</a> is the Kubernetes integration project for <a href="https://github.com/ray-project/ray">Ray</a>. It allows Ray clusters and Ray applications to run natively on Kubernetes by introducing Kubernetes-aware management and automation capabilities for Ray workloads.</p><p>Ray itself is a distributed computing framework used primarily for AI and machine learning workloads. It provides mechanisms for distributing Python-based execution across multiple processes, GPUs, and nodes. Ray is commonly used for distributed model training, batch processing, distributed inference, and scalable serving applications. More information about the framework can be found in its <a href="https://www.ray.io/">official site</a>.</p><p>The KubeRay Operator is the Kubernetes Operator component of the KubeRay project. It extends Kubernetes through Custom Resource Definitions (CRDs) and controllers that automate the deployment and lifecycle management of Ray clusters. Instead of manually creating and coordinating Ray head nodes, worker nodes, services, and scaling logic, the operator manages these resources declaratively through Kubernetes manifests.</p><blockquote><strong>Kuberay is necessary only for distributed vLLM serving patterns that require multi-node orchestration</strong>, such as pipeline parallelism or distributed prefill/decode deployments. For single-node, single-GPU deployments, KubeRay is not required and the vLLM Production Stack can operate with standard Kubernetes resources alone.</blockquote><p>In this blueprint, the KubeRay Operator is installed to provide the Kubernetes control layer required for distributed Ray-based inference workloads. The vLLM Production Stack uses RayCluster and RayService resources to deploy, scale, and manage distributed vLLM serving instances across the Kubernetes cluster. This step prepares the CCE environment with the necessary Kubernetes extensions and controllers required for distributed inference orchestration.</p><p>We are going to deploy the KubeRay Operator using the official Helm chart:</p><pre>helm repo add kuberay https://ray-project.github.io/kuberay-helm/<br>helm repo update<br><br>helm install kuberay-operator kuberay/kuberay-operator \<br>  -n kuberay-system \<br>  --create-namespace</pre><h3>Choosing the Right Deployment Scenario</h3><p>Before deploying the vLLM Production Stack, it’s worth taking a moment to choose the deployment model that best matches your workload and infrastructure. Different scenarios are optimized for different goals; whether that’s serving a model on a single GPU, scaling across multiple nodes, maximizing the performance of a multi-GPU server, or separating prefill and decode workloads for advanced production deployments.</p><p>The following table summarizes the supported deployment scenarios, their intended use cases, key prerequisites, and important trade-offs. Throughout this guide, we’ll focus on the deployment model that best fits our target environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*lkr4RFdW2TBLC6wjPFXYfw.png" /></figure><h3>Single-Node, Standalone Serving</h3><blockquote>We will need 1 node with specs: <strong>GPU-accelerated | pi5e.2xlarge.4 | 8 vCPUs | 32 GiB | GPU: nvidia-l4 x 1</strong></blockquote><p>In this scenario, we deploy the <a href="https://huggingface.co/openai/gpt-oss-20b">openai/gpt-oss-20b</a> model as a standalone inference service on CCE using a single NVIDIA L4 GPU. The objective is to establish a minimal production-ready deployment pattern that exposes the model through the OpenAI-compatible vLLM API without introducing distributed execution or multi-node orchestration complexity.</p><blockquote>openai/gpt-oss-20b is an open-weight large Mixture of Experts (MoE) model from OpenAI designed for general-purpose text generation, reasoning, code generation and agentic workloads. With approximately 20 billion parameters (3.6B active), the model provides substantially higher reasoning and language generation capability than smaller instruction-tuned models while still remaining deployable on modern enterprise GPU hardware with careful memory utilization tuning and quantization-aware serving strategies.</blockquote><p>Within this deployment, vLLM acts as the inference runtime responsible for loading the model, optimizing GPU memory usage, managing request batching, and exposing the serving endpoint through an OpenAI-compatible API interface. This allows existing AI applications, SDKs, and automation frameworks to consume the model using familiar OpenAI API semantics without requiring model-specific integrations.</p><blockquote><strong>Can GPT-OSS 20B run on NVIDIA L4 24GB?</strong></blockquote><blockquote>The gpt-oss-20b model can operate on a single NVIDIA L4 GPU with 24 GB VRAM, as the model requires approximately 18.6 GB of GPU memory in this configuration. When deployed with Q4_K_M quantization, inference throughput can reach roughly 35 tokens per second on NVIDIA L4 hardware.</blockquote><blockquote>For a more detailed report check <a href="https://willitrunai.com/can-run/gpt-oss-20b-on-l4-24gb">here</a>.</blockquote><p>Persistent model caching is enabled through a mounted Kubernetes persistent volume so that downloaded Hugging Face artifacts survive pod restarts and redeployments. This significantly reduces model initialization times and avoids repeated downloads during operational lifecycle events.</p><pre>servingEngineSpec:<br>  vllmApiKey: <br>    secretName: vllm-prodstack-secrets<br>    secretKey: VLLM_MASTER_KEY<br><br>  tolerations:<br>    - key: &quot;nvidia.com/gpu&quot;<br>      operator: &quot;Equal&quot;<br>      value: &quot;true&quot;<br>      effect: &quot;NoSchedule&quot;<br><br>  modelSpec:<br>    - name: gpt-oss-20b<br>      repository: vllm/vllm-openai<br>      tag: latest<br>      modelURL: openai/gpt-oss-20b<br><br>      hf_token: <br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      replicaCount: 1<br><br>      requestCPU: 4<br>      requestMemory: &quot;16Gi&quot;<br>      requestGPU: 1<br><br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br><br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-l4<br><br>      vllmConfig:<br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;gpt-oss-20b&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.90&quot;<br>          - &quot;--max-model-len&quot;<br>          - &quot;8192&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;</pre><blockquote>requestCPU &amp; requestMemory define the host resources reserved for the vLLM serving pod. The GPU remains the primary execution resource for model inference, while CPU and system memory support request processing, tokenization, model initialization, cache handling, and the OpenAI-compatible API process. For this single-GPU baseline, the deployment reserves 4 vCPUs and 16 GiB of memory per serving replica.</blockquote><p>We can now deploy the vLLM Production Stack with the Helm chart:</p><pre>helm repo add vllm https://vllm-project.github.io/production-stack<br>helm repo update<br><br>helm upgrade --install vllm vllm/vllm-stack \<br>  -n vllm --create-namespace \<br>  -f vllm-values-single-node.yaml \<br>  --reset-values</pre><h3>Single-Node, Tensor Parallelism on NVIDIA T4/L4</h3><blockquote>We will need 1 node with specs: <strong>GPU-accelerated | pi2.4xlarge.4 | 16 vCPUs | 64 GiB | GPU: nvidia-t4 x 2</strong></blockquote><p>This scenario demonstrates how vLLM can use tensor parallelism to deploy larger instruction-tuned models on compact GPU configurations commonly used for edge inference, development environments, or cost-sensitive AI platforms. The deployment uses a CCE worker node equipped with two NVIDIA Tesla T4 GPUs and serves <a href="https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ">Qwen/Qwen2.5-32B-Instruct-AWQ</a>, a 32-billion-parameter model provided in AWQ INT4 format.</p><blockquote><strong>Can Qwen2.5–32B-Instruct-AWQ run on NVIDIA T4 16GB?</strong></blockquote><blockquote>The AWQ INT4 checkpoint requires approximately 18–22 GB of effective GPU memory during inference, before additional memory is reserved for KV cache and runtime overhead. This exceeds the capacity of a single 15 GB Tesla T4 GPU, so the deployment uses tensor parallelism to distribute the model across both local GPUs.</blockquote><p>A model of this size exceeds the practical capacity of a single 15 GB T4 accelerator once runtime overhead, KV cache allocation, and inference buffers are taken into account. By enabling tensor parallelism, vLLM distributes the model weights across both local GPUs and exposes the deployment through a single OpenAI-compatible inference endpoint.</p><blockquote>Qwen/Qwen2.5-32B-Instruct-AWQ is a 32-billion-parameter instruction-tuned language model from Alibaba Cloud optimized for conversational AI, reasoning, code generation, and multilingual inference workloads. The AWQ INT4 quantized variant significantly reduces GPU memory consumption while maintaining strong inference performance and instruction-following quality. This allows the model to be deployed efficiently on smaller accelerator classes such as dual NVIDIA Tesla T4 GPUs using tensor parallelism through vLLM.</blockquote><p>The T4 platform is particularly well suited for this type of deployment because it supports AWQ quantization acceleration directly. This allows the model to run efficiently in its native AWQ INT4 format while maintaining a relatively small memory footprint compared to full-precision deployments. In practice, the quantized model requires approximately 18–22 GB of effective GPU memory during inference, making a dual-T4 configuration a realistic target for this deployment topology.</p><blockquote>Since both GPUs are attached to the same Kubernetes worker node, <strong>no Ray or KubeRay components are required for this scenario</strong>.</blockquote><pre>servingEngineSpec:<br>  vllmApiKey:<br>    secretName: vllm-prodstack-secrets<br>    secretKey: VLLM_MASTER_KEY<br><br>  tolerations:<br>    - key: &quot;nvidia.com/gpu&quot;<br>      operator: &quot;Equal&quot;<br>      value: &quot;true&quot;<br>      effect: &quot;NoSchedule&quot;<br><br>  modelSpec:<br>    - name: qwen2-5-32b-instruct-awq<br>      repository: vllm/vllm-openai<br>      tag: v0.8.5<br>      modelURL: Qwen/Qwen2.5-32B-Instruct-AWQ<br><br>      hf_token:<br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      replicaCount: 1<br><br>      requestCPU: 8<br>      requestMemory: &quot;48Gi&quot;<br>      requestGPU: 2<br><br>      extraEnv:<br>        - name: HF_HUB_DISABLE_XET<br>          value: &quot;1&quot;<br><br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br><br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-t4<br><br>      vllmConfig:<br>        tensorParallelSize: 2<br>        pipelineParallelSize: 1<br><br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;qwen2-5-32b-instruct-awq&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.80&quot;<br>          - &quot;--max-model-len&quot;<br>          - &quot;2048&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;<br>          - &quot;--quantization&quot;<br>          - &quot;awq&quot;<br><br>      shmSize: &quot;32Gi&quot;</pre><p>We can now deploy the vLLM Production Stack with the Helm chart:</p><pre>helm repo add vllm https://vllm-project.github.io/production-stack<br>helm repo update<br><br>helm upgrade --install vllm vllm/vllm-stack \<br>  -n vllm --create-namespace \<br>  -f vllm-values-single-node-tp-t4.yaml \<br>  --reset-values</pre><p>After deployment, we can confirm from the vLLM logs that tensor parallelism initialized successfully with two tensor-parallel ranks, and via <strong>nvidia-smi</strong> we can see that model memory allocated across both V100 devices while serving inference requests through the OpenAI-compatible API endpoint. Open a shell on the pod that host the model and execute the following command, after replacing the value of VLLM_MASTER_KEY:</p><pre>curl http://127.0.0.1:8000/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer &lt;VLLM_MASTER_KEY&gt;&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;qwen2-5-32b-instruct-awq&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Explain tensor parallelism in one paragraph.&quot;<br>      }<br>    ]<br>  }&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*Ifdx_iYdYaGnnAPp.png" /></figure><h3>Single-Node, Tensor Parallelism on NVIDIA V100</h3><blockquote>We will need 1 node with specs: <strong>GPU-accelerated | p2s.4xlarge.8 | 16 vCPUs | 128 GiB | GPU: nvidia-v100-pcie-32gb x 2</strong></blockquote><p>This scenario demonstrates how vLLM can serve a large language model that does not fit into a single GPU by using tensor parallelism across multiple GPUs on the same CCE worker node. The deployment uses a node equipped with 2x NVIDIA Tesla V100 32 GB accelerators and deploys the <a href="https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct">meta-llama/Llama-3.1-70B-Instruct</a> model in a GPTQ INT4 quantized format.</p><blockquote>hugging-quants/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4 is a GPTQ INT4 quantized variant of Meta&#39;s Llama 3.1 70B Instruct model designed for high-quality conversational AI, reasoning, summarization, and advanced instruction-following workloads. With approximately 70 billion parameters, the original FP16 model exceeds the memory capacity of most standard enterprise GPU deployments. The GPTQ INT4 quantized checkpoint substantially reduces the memory footprint while preserving most of the model&#39;s inference quality, making it suitable for tensor-parallel deployments on multi-GPU Kubernetes nodes using vLLM.</blockquote><blockquote><strong>Can LLama-3.1–70B-Instruct run on NVIDIA V100 32GB?</strong></blockquote><blockquote>The selected GPTQ INT4 model still requires approximately 35–40 GB of effective GPU memory during inference once model weights, runtime overhead, and KV cache allocation are taken into account. While this is significantly smaller than the original FP16 variant of Llama 3.1 70B, it still exceeds the practical capacity of a single V100 32 GB GPU, which is why tensor parallelism across both local GPUs is required for this deployment scenario.</blockquote><p>A standard FP16 deployment of a 70B model would exceed the available memory of a single V100 GPU. To make the deployment possible on this hardware class, the model is loaded using GPTQ quantization and distributed across both GPUs through:</p><pre>tensorParallelSize: 2</pre><p>With this configuration, vLLM splits the model tensors between the two local GPUs while exposing a single inference endpoint to the client application.</p><p>Several configuration adjustments were required to make the deployment compatible with V100 GPUs:</p><ul><li>The AWQ quantized model variant cannot be used and has to be replaced with a GPTQ INT4 checkpoint because AWQ kernels require newer GPU architectures than Volta-based V100 cards.</li><li>The blueprint also uses vllm/vllm-openai:v0.8.5, which proved more stable for Volta-class hardware than newer runtime combinations.</li><li>To reduce GPU memory pressure during model initialization and KV cache allocation, we lower the GPU memory utilization threshold and limits the maximum context length:</li></ul><pre>--gpu-memory-utilization 0.75<br>--max-model-len 2048</pre><ul><li>We additionally disable the Hugging Face Xet transfer backend, in order to avoid interrupted large-model downloads in environments where long-running Xet/CAS connections may be unstable or affected by enterprise network controls.</li></ul><pre>HF_HUB_DISABLE_XET=1</pre><blockquote>Since both GPUs are attached to the same Kubernetes worker node, <strong>no Ray or KubeRay components are required for this scenario</strong>.</blockquote><p>The final configuration should look like this:</p><pre>servingEngineSpec:<br>  vllmApiKey:<br>    secretName: vllm-prodstack-secrets<br>    secretKey: VLLM_MASTER_KEY<br><br>  tolerations:<br>    - key: &quot;nvidia.com/gpu&quot;<br>      operator: &quot;Equal&quot;<br>      value: &quot;true&quot;<br>      effect: &quot;NoSchedule&quot;<br><br>  modelSpec:<br>    - name: llama3-1-70b-instruct-gptq-int4<br>      repository: vllm/vllm-openai<br>      tag: v0.8.5<br>      modelURL: hugging-quants/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4<br><br>      hf_token:<br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      replicaCount: 1<br><br>      requestCPU: 8<br>      requestMemory: &quot;96Gi&quot;<br>      requestGPU: 2<br><br>      extraEnv:<br>        - name: HF_HUB_DISABLE_XET<br>          value: &quot;1&quot;<br><br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br><br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-v100-pcie-32gb<br><br>      vllmConfig:<br>        tensorParallelSize: 2<br>        pipelineParallelSize: 1<br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;llama3-1-70b-instruct-gptq-int4&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.75&quot;<br>          - &quot;--max-model-len&quot;<br>          - &quot;2048&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;<br>          - &quot;--quantization&quot;<br>          - &quot;gptq&quot;<br><br>      shmSize: &quot;32Gi&quot;</pre><p>We can now deploy the vLLM Production Stack with the Helm chart:</p><pre>helm repo add vllm https://vllm-project.github.io/production-stack<br>helm repo update<br><br>helm upgrade --install vllm vllm/vllm-stack \<br>  -n vllm --create-namespace \<br>  -f vllm-values-single-node-tp-v100.yaml \<br>  --reset-values</pre><p>After deployment, we can confirm from the vLLM logs that tensor parallelism initialized successfully with two tensor-parallel ranks, and via <strong>nvidia-smi</strong> we can see that model memory allocated across both V100 devices while serving inference requests through the OpenAI-compatible API endpoint. Open a shell on the pod that host the model and execute the following command, after replacing the value of VLLM_MASTER_KEY:</p><pre>curl http://127.0.0.1:8000/v1/chat/completions \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -H &quot;Authorization: Bearer &lt;VLLM_MASTER_KEY&gt;&quot; \<br>  -d &#39;{<br>    &quot;model&quot;: &quot;llama3-1-70b-instruct-gptq-int4&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;Explain tensor parallelism in one paragraph.&quot;<br>      }<br>    ]<br>  }&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8Ib1ovUND_6C_6iN.png" /></figure><h3>Multi-Node, Distributed Serving</h3><blockquote>We will need 2 nodes with specs: <strong>GPU-accelerated | pi5e.4xlarge.4 | 16 vCPUs | 64 GiB | GPU: nvidia-l4 x 1</strong></blockquote><p>This scenario demonstrates how large language models can be served across multiple Kubernetes worker nodes when a single GPU is no longer sufficient to host the model independently. Instead of relying on one large multi-GPU server, the deployment combines multiple smaller GPU nodes into a distributed inference topology using Ray, KubeRay, and vLLM. This approach is particularly relevant for enterprise environments where GPU resources are fragmented across several worker nodes or where high-memory accelerator platforms are either unavailable or prohibitively expensive.</p><p>The Qwen2.5-32B-Instruct-AWQ model is distributed across two single-GPU NVIDIA L4 worker nodes through pipeline parallelism. Each node contributes one GPU to the inference pipeline, while Ray orchestrates the distributed execution and communication between the participating workers. From the client perspective, however, the deployment still behaves as a single OpenAI-compatible inference endpoint. The complexity of model distribution, stage coordination, and GPU communication remains fully abstracted behind the vLLM serving layer.</p><p>This architecture addresses one of the most common operational challenges in enterprise AI adoption: <strong>deploying models that exceed the practical memory capacity of individual accelerators without requiring specialized high-end GPU servers</strong>. By distributing the model across multiple commodity GPU nodes, organizations can increase the usable capacity of their existing Kubernetes GPU infrastructure while maintaining centralized API access, Kubernetes-native orchestration, and scalable distributed inference management through KubeRay.</p><blockquote><strong>Can Qwen2.5–32B-Instruct-AWQ run on a single NVIDIA L4?</strong></blockquote><blockquote>The selected AWQ INT4 model requires approximately 18–22 GB of effective GPU memory during inference once model weights, runtime overhead, and KV cache allocation are taken into account. Although this is significantly smaller than a full-precision 32B model, it leaves little practical headroom on a single NVIDIA L4 with 24 GB VRAM. For this scenario, the model is therefore distributed across two single-GPU L4 nodes with Ray and pipeline parallelism, allowing vLLM to serve the model through one endpoint while spreading the memory load across both GPUs.</blockquote><p>The final configuration should look like this:</p><pre>servingEngineSpec:<br>  vllmApiKey:<br>    secretName: vllm-prodstack-secrets<br>    secretKey: VLLM_MASTER_KEY<br><br>  tolerations:<br>    - key: &quot;nvidia.com/gpu&quot;<br>      operator: &quot;Equal&quot;<br>      value: &quot;true&quot;<br>      effect: &quot;NoSchedule&quot;<br><br>  modelSpec:<br>    - name: qwen2-5-32b-instruct-awq<br>      repository: vllm/vllm-openai<br>      tag: v0.8.5<br>      modelURL: Qwen/Qwen2.5-32B-Instruct-AWQ<br><br>      hf_token:<br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      replicaCount: 1<br><br>      requestCPU: 4<br>      requestMemory: &quot;20Gi&quot;<br>      requestGPU: 1<br><br>      extraEnv:<br>        - name: HF_HUB_DISABLE_XET<br>          value: &quot;1&quot;<br>        - name: VLLM_ATTENTION_BACKEND<br>          value: &quot;FLASHINFER&quot;<br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-l4<br><br>      vllmConfig:<br>        tensorParallelSize: 1<br>        pipelineParallelSize: 2<br>        dtype: &quot;float16&quot;<br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;qwen2-5-32b-instruct-awq&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.90&quot;<br>          - &quot;--max-model-len&quot;<br>          - &quot;2048&quot;<br>          - &quot;--kv-cache-dtype&quot;<br>          - &quot;fp8&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;<br>          - &quot;--quantization&quot;<br>          - &quot;awq&quot;<br><br>      shmSize: &quot;12Gi&quot;<br><br>      raySpec:<br>        enabled: true<br>        headNode:<br>          requestCPU: 4<br>          requestMemory: &quot;20Gi&quot;<br>          requestGPU: 1<br><br>          nodeSelectorTerms:<br>            - matchExpressions:<br>                - key: accelerator<br>                  operator: In<br>                  values:<br>                    - nvidia-l4</pre><p>We can now deploy the vLLM Production Stack with the Helm chart:</p><pre>helm repo add vllm https://vllm-project.github.io/production-stack<br>helm repo update<br><br>helm upgrade --install vllm vllm/vllm-stack \<br>  -n vllm --create-namespace \<br>  -f vllm-values-multi-node-distributed.yaml \<br>  --reset-values</pre><p>After successful deployment, use the Ray head pod to validate the distributed cluster state. The head pod is the cluster coordinator and exposes the complete Ray runtime view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*66C_XBWIBC7g4CwC.png" /></figure><blockquote>The ray status command validates the distributed execution layer itself. It confirms that the Ray cluster was formed successfully, that both Ray nodes joined the cluster correctly, and that the distributed resources required by vLLM are available. In this scenario, the output should report two active nodes and two available GPUs, proving that the pipeline-parallel deployment spans both Kubernetes worker nodes rather than running entirely on a single host.</blockquote><p>To validate GPU utilization directly, open shells on <strong>both</strong> the Ray head and Ray worker pods and execute:</p><pre>nvidia-smi</pre><p>Both nvidia-smi outputs should show active CUDA processes and allocated GPU memory from the vLLM worker runtime. This confirms that the model weights and inference pipeline stages were successfully distributed across both Kubernetes worker nodes rather than executing entirely on a single GPU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jInF_bYRxe5HQPG8.png" /></figure><blockquote>The nvidia-smi command validates GPU participation on each node. While Ray confirms the logical distributed cluster, nvidia-smi confirms that the vLLM worker processes are actively consuming GPU memory on both accelerators. This is particularly important in multi-node inference scenarios because a successful API response alone does not guarantee that the model was actually distributed across both GPUs. Observing active memory allocation and running CUDA processes on both nodes confirms that the model weights and inference workload were successfully placed across the distributed pipeline stages.</blockquote><h3>Disaggregated Prefill/Decode Serving</h3><blockquote>We will need 2 nodes with specs: <strong>GPU-accelerated | pi5e.4xlarge.4 | 16 vCPUs | 64 GiB | GPU: nvidia-l4 x 1</strong></blockquote><p>This addresses a different problem than the previous deployment patterns. The objective is no longer simply making a model fit into GPU memory, but improving inference efficiency and reducing latency under high request concurrency.</p><p>Large language model inference consists of two fundamentally different phases. During the <strong>prefill phase</strong>, the model processes the entire input prompt and builds the KV cache. This stage is highly compute-intensive and benefits from large parallel processing capacity. During the <strong>decode phase</strong>, the model generates tokens iteratively using the previously generated KV cache. Decode workloads are typically more latency-sensitive and often become bottlenecked by memory bandwidth and cache access patterns rather than raw compute throughput.</p><p>In a traditional deployment, both phases execute on the same GPU resources. This means prompt ingestion and token generation compete for the same accelerator capacity. Under heavy concurrent load, long prompts can therefore negatively impact token generation latency for active inference sessions.</p><p><strong>Disaggregated prefill/decode serving separates these workloads into dedicated execution paths</strong>. One group of vLLM instances is optimized for prompt ingestion and KV cache generation, while another group is optimized for token decoding. KV cache data is transferred between both stages through a coordinated serving layer. This allows the infrastructure to scale and tune both workloads independently depending on traffic patterns and application behavior.</p><p>This architecture becomes particularly valuable for enterprise AI platforms serving many simultaneous users, agentic workflows, retrieval-augmented generation pipelines, or applications with large prompt contexts. By separating prompt processing from token generation, organizations can improve GPU utilization efficiency, reduce tail latency, and optimize infrastructure allocation for different inference characteristics rather than treating all inference workloads identically.</p><p>For the prefill/decode scenario, the deployment is no longer built around one vLLM engine. It defines two separate serving engines for the same model: one engine handles the prefill phase, and the other handles the decode phase. Both engines, for the economy of the blueprint, use the same Qwen/Qwen2.5-14B-Instruct-AWQ checkpoint and expose the same served model name, but they are assigned different roles in the inference flow.</p><p>The prefill engine is configured as the KV cache producer. It processes the incoming prompt, builds the initial KV cache, and sends that cache to the decode engine. This is why its LMCache configuration uses kvRole: kv_producer, nixlRole: sender, and points to the decode engine service through nixlPeerHost. The decode engine is configured as the KV cache consumer. It receives the transferred cache and continues token generation, so its LMCache configuration uses kvRole: kv_consumer, nixlRole: receiver, and listens on 0.0.0.0:55555.</p><p><a href="https://github.com/LMCache/LMCache">LMCache</a> is the distributed KV cache layer used by vLLM to transfer and manage inference state between the prefill and decode engines. In a traditional deployment, the KV cache remains local to the same GPU that processed the prompt. In a disaggregated architecture, however, the prompt is processed by one engine while token generation continues on another engine running on a different GPU or node. The KV cache therefore needs to be transferred between both stages efficiently and with minimal overhead.</p><blockquote>In this scenario, LMCache acts as the coordination and transport layer for that cache exchange. The prefill engine generates the KV cache and publishes it through the LMCache producer configuration, while the decode engine receives the cache through the LMCache consumer configuration. This allows the decode engine to continue token generation without reprocessing the original prompt from scratch.</blockquote><blockquote>The deployment uses LMCache together with NIXL-enabled GPU transfer buffers. This enables the KV cache to be exchanged directly through GPU-accessible memory instead of repeatedly serializing and rebuilding inference state between requests. From an architectural perspective, LMCache is the component that makes disaggregated serving practical, because it decouples prompt processing from token generation while preserving the inference context required for continuous generation.</blockquote><blockquote>Without LMCache, both phases would need to execute on the same vLLM engine, which would eliminate the operational benefits of separating prompt ingestion and decode workloads across independent GPU resources.</blockquote><p>Both engines enable prefix caching and LMCache because cache transfer is the core mechanism that makes disaggregated prefill/decode serving work. NIXL is enabled as the transport layer for moving KV cache data between the producer and consumer, and a CUDA-backed transfer buffer is configured so the cache exchange can use GPU memory.</p><p>The router is also configured differently from the previous scenarios. Instead of simple request distribution, it uses disaggregated_prefill routing logic. This allows the router to distinguish between the prefill and decode backends through their model labels and send each request through the correct two-stage inference path.</p><blockquote><a href="https://github.com/ai-dynamo/nixl">NIXL</a> is the high-performance transport layer used by LMCache to move KV cache data between the prefill and decode engines. In a disaggregated serving architecture, the KV cache generated during prompt ingestion must be transferred efficiently from one GPU-backed vLLM instance to another. NIXL provides the communication mechanism that enables this exchange without forcing the cache data to be repeatedly serialized through slower CPU-bound workflows.</blockquote><blockquote>In this deployment, the prefill engine acts as the NIXL sender while the decode engine acts as the NIXL receiver. Both engines allocate dedicated GPU-backed transfer buffers, allowing KV cache blocks to be exchanged directly between the participating inference workers. This significantly reduces the overhead associated with transferring large prompt contexts and makes continuous token generation practical even when the inference stages are separated across different Kubernetes nodes.</blockquote><blockquote>From an architectural perspective, NIXL is what enables LMCache to operate as a distributed GPU-aware KV cache system rather than simply a software-level cache manager. It provides the low-latency transport path required for real-time inference serving, especially for long-context prompts and highly concurrent workloads where KV cache movement becomes a critical performance factor.</blockquote><p>The final configuration should look like this:</p><pre>servingEngineSpec:<br>  enableEngine: true<br>  runtimeClassName: &quot;&quot;<br>  containerPort: 8000<br><br>  modelSpec:<br>    - name: qwen2-5-14b-prefill<br>      repository: lmcache/vllm-openai<br>      tag: 2025-05-27-v1<br>      modelURL: Qwen/Qwen2.5-14B-Instruct-AWQ<br><br>      replicaCount: 1<br><br>      requestCPU: 4<br>      requestMemory: &quot;20Gi&quot;<br>      requestGPU: 1<br><br>      hf_token:<br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-l4<br><br>      extraEnv:<br>        - name: HF_HUB_DISABLE_XET<br>          value: &quot;1&quot;<br><br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br><br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      vllmConfig:<br>        enablePrefixCaching: true<br>        maxModelLen: 4096<br>        v1: 1<br><br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;qwen2-5-14b-instruct-awq&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.85&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;<br>          - &quot;--quantization&quot;<br>          - &quot;awq&quot;<br><br>      lmcacheConfig:<br>        enabled: true<br>        cudaVisibleDevices: &quot;0&quot;<br>        kvRole: &quot;kv_producer&quot;<br>        enableNixl: true<br>        nixlRole: &quot;sender&quot;<br>        nixlPeerHost: &quot;vllm-qwen2-5-14b-decode-engine-service&quot;<br>        nixlPeerPort: &quot;55555&quot;<br>        nixlBufferSize: &quot;1073741824&quot;<br>        nixlBufferDevice: &quot;cuda&quot;<br>        nixlEnableGc: true<br>        enablePD: true<br>        cpuOffloadingBufferSize: 0<br><br>      labels:<br>        model: qwen2-5-14b-prefill<br><br>    - name: qwen2-5-14b-decode<br>      repository: lmcache/vllm-openai<br>      tag: 2025-05-27-v1<br>      modelURL: Qwen/Qwen2.5-14B-Instruct-AWQ<br><br>      replicaCount: 1<br><br>      requestCPU: 4<br>      requestMemory: &quot;20Gi&quot;<br>      requestGPU: 1<br><br>      hf_token:<br>        secretName: vllm-prodstack-secrets<br>        secretKey: HF_TOKEN<br><br>      nodeSelectorTerms:<br>        - matchExpressions:<br>            - key: accelerator<br>              operator: In<br>              values:<br>                - nvidia-l4<br><br>      extraEnv:<br>        - name: HF_HUB_DISABLE_XET<br>          value: &quot;1&quot;<br><br>      extraVolumes:<br>        - name: vllm-model-cache<br>          persistentVolumeClaim:<br>            claimName: pvc-vllm-models<br><br>      extraVolumeMounts:<br>        - name: vllm-model-cache<br>          mountPath: /models<br><br>      vllmConfig:<br>        enablePrefixCaching: true<br>        maxModelLen: 4096<br>        v1: 1<br><br>        extraArgs:<br>          - &quot;--served-model-name&quot;<br>          - &quot;qwen2-5-14b-instruct-awq&quot;<br>          - &quot;--gpu-memory-utilization&quot;<br>          - &quot;0.85&quot;<br>          - &quot;--download-dir&quot;<br>          - &quot;/models/vllm-downloads&quot;<br>          - &quot;--quantization&quot;<br>          - &quot;awq&quot;<br><br>      lmcacheConfig:<br>        enabled: true<br>        cudaVisibleDevices: &quot;0&quot;<br>        kvRole: &quot;kv_consumer&quot;<br>        enableNixl: true<br>        nixlRole: &quot;receiver&quot;<br>        nixlPeerHost: &quot;0.0.0.0&quot;<br>        nixlPeerPort: &quot;55555&quot;<br>        nixlBufferSize: &quot;1073741824&quot;<br>        nixlBufferDevice: &quot;cuda&quot;<br>        nixlEnableGc: true<br>        enablePD: true<br><br>      labels:<br>        model: qwen2-5-14b-decode<br><br>  containerSecurityContext:<br>    capabilities:<br>      add:<br>        - SYS_PTRACE<br><br>routerSpec:<br>  enableRouter: true<br>  repository: lmcache/lmstack-router<br>  tag: latest<br><br>  replicaCount: 1<br><br>  containerPort: 8000<br>  servicePort: 80<br><br>  routingLogic: disaggregated_prefill<br>  enablePD: true<br>  engineScrapeInterval: 15<br>  requestStatsWindow: 60<br><br>  resources:<br>    requests:<br>      cpu: &quot;2&quot;<br>      memory: &quot;4Gi&quot;<br>    limits:<br>      cpu: &quot;4&quot;<br>      memory: &quot;8Gi&quot;<br><br>  labels:<br>    environment: router<br>    release: router<br><br>  extraArgs:<br>    - &quot;--prefill-model-labels&quot;<br>    - &quot;qwen2-5-14b-prefill&quot;<br>    - &quot;--decode-model-labels&quot;<br>    - &quot;qwen2-5-14b-decode&quot;</pre><blockquote>SYS_PTRACE adds Linux process tracing capabilities to the container runtime. This is typically only required for low-level debugging, profiling, or runtime inspection tools such as CUDA/NCCL diagnostics and advanced GPU performance analysis.</blockquote><blockquote>Standard vLLM inference deployments generally do not require this capability. In security-conscious Kubernetes environments, container capabilities should remain minimized unless explicitly needed by the workload or troubleshooting process, otherwise it can/should be omitted.</blockquote><p>We can now deploy the vLLM Production Stack with the Helm chart:</p><pre>helm repo add vllm https://vllm-project.github.io/production-stack<br>helm repo update<br><br>helm upgrade --install vllm vllm/vllm-stack \<br>  -n vllm --create-namespace \<br>  -f vllm-values-multi-node-disaggregated.yaml \<br>  --reset-values</pre><p>After deployment, we can confirm from the vLLM logs that both the prefill and decode engines initialized successfully with the correct KV roles and NIXL configurations. The router logs should also confirm that the disaggregated prefill routing logic is active and that both engines are registered correctly with their respective model labels and LMCache roles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GLdlGuOxOeCE1_lL.png" /></figure><p>Finally, we can observe active GPU memory allocation on both engines through nvidia-smi in their respective pods, confirming that both the prefill and decode workloads are executing on their assigned GPU resources according to the disaggregated serving.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eWwx1ULJKGLig_QC.png" /></figure><h3>Closing Thoughts</h3><p>Deploying large language models in production is about much more than simply starting an inference server. Reliability, scalability, observability, and operational simplicity are what separate a proof of concept from a platform that can support real-world AI applications.</p><p>In this article, we’ve explored how the <strong>vLLM Production Stack</strong> and <strong>T Cloud Public CCE</strong> complement each other to provide a robust foundation for production inference. Starting from a straightforward single-GPU deployment, we gradually expanded to distributed serving, tensor parallelism, and disaggregated prefill/decode architectures, each addressing a different set of performance and scalability requirements.</p><p>One of the biggest advantages of this approach is that the underlying Kubernetes platform remains consistent as your workloads evolve. Whether you’re serving a single model for internal use or operating a fleet of GPU-backed inference services, you can adopt more advanced deployment patterns without redesigning your entire infrastructure.</p><p>As the open-source AI ecosystem continues to evolve, having a cloud-native deployment strategy becomes increasingly important. With vLLM’s high-performance inference engine and the operational capabilities of T Cloud Public CCE, you have a production-ready platform capable of supporting everything from experimentation to enterprise-scale AI services.</p><p>I hope this guide helps you accelerate your own deployments and provides a solid starting point for building scalable, reliable LLM inference platforms.</p><p><em>This article was originally posted as a blueprint in </em><strong><em>T Cloud Public Architecture Center</em></strong><em>: </em><a href="https://arch.otc-service.com/docs/blueprints/by-use-case/ai/deploy-vllm-production-stack-on-cce">Deploy vLLM Production Stack on CCE</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fz4-o4JFJa0k%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dz4-o4JFJa0k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fz4-o4JFJa0k%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27ae287e166b48ab982633aedd4cb659/href">https://medium.com/media/27ae287e166b48ab982633aedd4cb659/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b79894043f87" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-to-deploy-a-production-grade-vllm-stack-on-t-cloud-public-cce-b79894043f87">How to Deploy a Production-Grade vLLM Stack on T Cloud Public CCE</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy OpenStack on Kubernetes in under 60 minutes]]></title>
            <link>https://itnext.io/deploy-openstack-on-kubernetes-in-under-60-minutes-f4ff01e2b424?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4ff01e2b424</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[digital-sovereignty]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[openstack]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Wed, 10 Jun 2026 12:22:12 GMT</pubDate>
            <atom:updated>2026-06-16T23:34:30.251Z</atom:updated>
            <content:encoded><![CDATA[<p>Build a Kubernetes-native OpenStack cloud in under an hour using Proxmox, K3s, Ceph and OpenStack-Helm.</p><figure><img alt="Illustrated container port scene featuring cargo cranes and shipping containers above the OpenStack and Kubernetes logos. The OpenStack logo and the Kubernetes helm-wheel logo are displayed side by side on a dock, symbolizing the convergence of OpenStack infrastructure services and Kubernetes orchestration in modern cloud platforms." src="https://cdn-images-1.medium.com/max/1024/1*dlPV7Yy_txRmY5TtO2b5EA.png" /><figcaption><em>The relationship between OpenStack and Kubernetes has evolved from coexistence to collaboration, with projects such as OpenStack-Helm bringing the two platforms closer together.</em></figcaption></figure><h3>TL;DR</h3><p>Instead of manually assembling every <a href="https://www.openstack.org/">OpenStack</a> moving part, we use a bunch of automation scripts to built around <a href="https://k3s.io/">K3s</a>, <a href="https://rook.io/">Rook-Ceph</a>, and <a href="https://github.com/openstack/openstack-helm">OpenStack-Helm</a> in order to deploy a fully functional OpenStack environment on a three-node Kubernetes cluster. Starting from a handful of virtual machines, we will bootstrap Kubernetes, provision distributed storage, prepare the networking stack, and deploy the OpenStack control plane and compute services with a series of repeatable, idempotent, scripted steps.</p><p>The result is not a toy deployment (not a production deployment though) or a collection of isolated services. By the end of the process, you will have a working OpenStack cloud with Keystone, Glance, Nova, Neutron, Cinder, Heat, Horizon, and Skyline running on Kubernetes, backed by Ceph storage and ready to launch networks &amp; virtual machines for your homelab or your OpenStack development environment.</p><h3>Prerequisites</h3><p>Before we begin, it is worth noting that <strong>this deployment is intentionally and inevitably opinionated, due to nature and complexity of the objective</strong>. Deploying OpenStack on Kubernetes involves a considerable number of moving parts, and the choices made throughout this guide reflect a specific approach that prioritizes simplicity, repeatability, and automation. That said, <strong>the underlying concepts are not tied to a particular environment</strong>. While the examples use Proxmox and OPNsense, the same deployment can be adapted to other virtualization and networking platforms, provided that the required infrastructure, networking, and storage and load-balancing capabilities are provided.</p><p>Throughout this article, it’s assumed that a virtualization and networking foundation is already in place. For that matter, <a href="https://www.proxmox.com/">Proxmox</a> provides the compute layer used to host the Kubernetes nodes, while <a href="https://opnsense.org/">OPNsense</a> is responsible for network services such as DNS, load balancing, and virtual IP management. These components are treated as existing infrastructure and their installation and configuration are therefore <strong>outside the scope</strong> of this article.</p><p>So this is what you are going to need:</p><ul><li>a <strong>Proxmox</strong> cluster (&gt;= 8.4.2) with enough capacity to host 3 additional VMs (6 vCPUs, 16GB RAM, 2 NICs, 2 iSCSI disks) that will serve as Kubernetes nodes</li><li>an <strong>OPNSense</strong> instance (≥= 25.1.12) supporting the SDN/Zone/VNet functionality of the Proxmox cluster and its VMs or LXCs</li><li>an SDN VXLAN <strong>Zone</strong> and a tagged VLAN-Aware <strong>VNet</strong> in Proxmox. Throughout the remainder of this article, these resources will be referred to as ixzone and ixvnet, respectively.</li></ul><figure><img alt="Creating the Proxmox SDN VXLAN zone (ixzone). The VXLAN zone is configured with an MTU of 1450 to accommodate encapsulation overhead and serves as the network foundation for tenant connectivity within the OpenStack environment." src="https://cdn-images-1.medium.com/max/1024/1*wkZyMh5eGVdGJQ_gwBm_lg.png" /><figcaption>Creating the Proxmox SDN VXLAN zone (<strong>ixzone</strong>). The VXLAN zone is configured with an MTU of <strong>1450</strong> to accommodate encapsulation overhead and serves as the network foundation for tenant connectivity within the OpenStack environment.</figcaption></figure><ul><li><strong>two interfaces</strong> are configured in OPNsense. The first is attached to the untagged network and serves as the Kubernetes management network. The second is attached to VLAN 70 and provides connectivity for OpenStack external networks, including floating IP traffic. The configuration is shown in the screenshots below:</li></ul><figure><img alt="OPNsense interface assignments used by the OpenStack deployment. The untagged physical interface vtnet1 (1) is assigned as the Kubernetes management network, while the VLAN interface vlan0.70 (2), created on top of vtnet1 and configured with VLAN tag 70 (3), is dedicated to OpenStack external networking and floating IP traffic." src="https://cdn-images-1.medium.com/max/1024/1*OOB--Pmb11bw_wSKsmQS6A.png" /><figcaption>OPNsense interface assignments used by the OpenStack deployment. The untagged physical interface <strong>vtnet1</strong> (1) is assigned as the Kubernetes management network, while the VLAN interface <strong>vlan0.70</strong> (2), created on top of <strong>vtnet1</strong> and configured with VLAN tag <strong>70</strong> (3), is dedicated to OpenStack external networking and floating IP traffic.</figcaption></figure><figure><img alt="VLAN configuration in OPNsense showing the vlan0.70 interface attached to the vtnet1 parent interface and configured with VLAN tag 70. This VLAN provides the network segment used by OpenStack external networking, enabling connectivity for provider networks and floating IP traffic." src="https://cdn-images-1.medium.com/max/1024/1*ob1RDwJvjNgSNoYWky0JeQ.png" /><figcaption>VLAN configuration in OPNsense showing the <strong>vlan0.70</strong> interface attached to the <strong>vtnet1</strong> parent interface and configured with VLAN tag <strong>70</strong>. This VLAN provides the network segment used by OpenStack external networking, enabling connectivity for provider networks and floating IP traffic.</figcaption></figure><p>Before proceeding, clone the deployment repository to the node (the one destined to be the master node of your K3s cluster) that will be used to orchestrate the installation. All automation scripts, configuration templates, and deployment assets referenced throughout this article are contained in the repository:</p><pre>git clone https://github.com/akyriako/osh.git<br>cd osh</pre><p><a href="https://github.com/akyriako/osh.git">GitHub - akyriako/osh</a></p><blockquote>Unless stated otherwise, all commands shown in the remainder of this guide are executed from within the cloned repository directory.</blockquote><h3>Architecture</h3><p>At the end of this deployment, we will have a fully functional OpenStack cloud running entirely on top of a three-node <a href="https://k3s.io/">K3s</a> cluster.</p><p>The Kubernetes layer forms the foundation of the platform. A single control-plane node and two worker nodes host all OpenStack services as Kubernetes workloads, deployed through OpenStack-Helm. Rather than dedicating individual virtual or physical machines to specific OpenStack roles, the control plane, networking services, storage services, and compute components are distributed across the cluster and managed through Kubernetes scheduling.</p><figure><img alt="**Alt text:** Architecture diagram showing a three-node K3s cluster hosting OpenStack-Helm services. OPNsense with HAProxy sits at the network edge and distributes traffic to NGINX and Traefik ingress controllers running on the Kubernetes nodes. The cluster hosts OpenStack services including Keystone, Glance, Horizon, Skyline, Heat, Nova, Neutron, and Placement, together with supporting infrastructure such as MariaDB, RabbitMQ, Memcached, Open vSwitch, and libvirt. Persistent storage is provide" src="https://cdn-images-1.medium.com/max/1024/1*JroexT64wpSbyIHyocD5Ew.png" /><figcaption><strong>Target architecture</strong>: a three-node K3s cluster running containerized OpenStack services, backed by a distributed Ceph storage cluster and exposed through OPNsense-managed ingress, DNS, and load balancing.</figcaption></figure><p>Persistent storage is provided by a Rook-managed Ceph cluster running on the same nodes. Each virtual machine contributes local storage devices to Ceph, creating a shared storage backend that is consumed both by Kubernetes and OpenStack services such as Cinder and Glance. This removes the need for an external storage appliance while still providing a distributed and resilient storage layer.</p><p>At the edge of the environment, OPNsense acts as the networking entry point. <a href="https://github.com/haproxy/haproxy">HAProxy</a> distributes incoming traffic across the Kubernetes cluster, while DNS records resolve OpenStack service endpoints to virtual IP addresses managed by <a href="https://docs.opnsense.org/manual/unbound.html">OPNsense Unbound DNS</a> service.</p><p>Two ingress paths are used within the cluster. NGINX Ingress exposes public OpenStack endpoints such as Horizon, Skyline, and Keystone, while Traefik (ingress controller that comes bundled with K3s) is retained for management interfaces and supporting services. This separation keeps administrative traffic distinct from the OpenStack APIs consumed by users and applications.</p><p>The OpenStack control plane itself consists of <a href="https://www.openstack.org/software/releases/dalmatian/components/keystone">Keystone</a> for identity management, <a href="https://www.openstack.org/software/releases/dalmatian/components/glance">Glance</a> for image services, <a href="https://www.openstack.org/software/releases/dalmatian/components/cinder">Cinder</a> for block storage, <a href="https://www.openstack.org/software/releases/dalmatian/components/heat">Heat</a> for orchestration, <a href="https://www.openstack.org/software/releases/dalmatian/components/horizon">Horizon</a> and <a href="https://www.openstack.org/software/releases/dalmatian/components/skyline">Skyline</a> as UI layer, and the <a href="https://www.openstack.org/software/releases/dalmatian/components/nova">Nova</a>, <a href="https://www.openstack.org/software/releases/dalmatian/components/neutron">Neutron</a>, and <a href="https://www.openstack.org/software/releases/dalmatian/components/placement">Placement</a> services required to provision and operate virtual machines. Open vSwitch provides the networking foundation for tenant connectivity, while libvirt and KVM enable Nova virtual machine instances to run directly on the Kubernetes nodes.</p><h3>Prepare the Proxmox Virtual Machines</h3><p>Each node has to be provisioned with <strong>two virtual network interfaces</strong>. The first interface is attached to the VXLAN-backed management network (ixvnet) and is used for Kubernetes, Ceph, and OpenStack internal communication. The second interface is connected to the same virtual network but carries VLAN tag 70, providing access to the external provider network that Neutron later exposes as the public physical network.</p><figure><img alt="Proxmox Virtual Environment hardware configuration screen showing a Kubernetes worker node. The virtual machine is configured with 14 GB of memory, 6 vCPUs, two virtual disks, and two network interfaces. One network interface is connected to the management network, while the second is attached to VLAN 70 for external provider network connectivity. The secondary disk is reserved for Ceph OSD storage." src="https://cdn-images-1.medium.com/max/1024/1*0wclyD77fqhOMhFLMI6cHA.png" /><figcaption>Proxmox virtual machine configuration used for the Kubernetes nodes that host OpenStack.</figcaption></figure><p>The storage configuration includes <strong>two virtual disks</strong>. The primary disk hosts the operating system and Kubernetes workloads, while the secondary disk is dedicated exclusively to Ceph. During the Ceph installation phase, this disk is discovered as /dev/sdb and consumed by Rook to create an Object Storage Daemon (OSD), contributing storage capacity to the distributed Ceph cluster.</p><h3>Install K3s Control Plane and Workers</h3><p>The first step is to establish the Kubernetes cluster that will host the entire OpenStack deployment. Since <a href="https://github.com/openstack/openstack-helm">OpenStack-Helm</a> packages OpenStack services as Helm charts, a functioning Kubernetes environment must be in place before any OpenStack components can be deployed.</p><p>The install-k3s.sh script bootstraps the cluster by installing K3s on the control-plane node and preparing it for the addition of worker nodes. During this phase, it installs and configures K3s, prepares cluster access, installs the tooling required throughout the remainder of the deployment, and exposes the default Traefik ingress controller (ingress for non-OpenStack workloads) on HTTP/8080 and HTTPS/8443.</p><blockquote>K3s was chosen because it provides a fully conformant Kubernetes distribution while keeping the operational footprint small, making it well suited for lab environments and proof-of-concept deployments.</blockquote><pre>sudo ./scripts/install-k3s.sh</pre><p>Once the control-plane installation is complete, the script generates a ready-to-use worker join-command. This command contains the information required for additional <strong>worker</strong> nodes to securely join the cluster and <strong>must be executed on each worker node</strong>.</p><blockquote>After running the join-command on each worker node and confirming that all nodes are part of the cluster, only then you are clear to move on to the next phase of the deployment.</blockquote><h3>Post-Installation Steps for K3s Nodes</h3><p>With the Kubernetes cluster in place, there is one more preparatory step before we start deploying storage and OpenStack services. While K3s can run perfectly well with its default operating system settings, OpenStack and Ceph place considerably higher demands on the underlying nodes. As the number of pods, files, watches, and network connections grows, the default kernel and system limits that are perfectly acceptable for a small Kubernetes cluster can become restrictive for a cloud platform.</p><p>The post-k3s.sh script prepares each node for the workload that will follow. It applies a number of operating system and Kubernetes tuning parameters, including disabling swap and increasing file descriptor and inotify limits. These changes help ensure that Kubernetes, containerd, Ceph, and the OpenStack services have sufficient resources available as the environment grows.</p><pre>sudo ./scripts/post-k3s.sh</pre><blockquote>Run this script <strong>on every node</strong> in the cluster, including both the control-plane and worker nodes. Although the changes are relatively small, they establish a more suitable baseline for the storage and virtualization workloads that will be introduced in the next phases of the deployment.</blockquote><h3>Prepare OPNsense HAProxy and Unbound DNS</h3><p>Before deploying storage and OpenStack services, we need to establish how users and applications will reach them. In a public cloud providers, this responsibility is typically handled by managed load balancers and DNS services. In our environment, those capabilities are provided by OPNsense through a combination of <a href="https://docs.opnsense.org/manual/firewall_vip.html">Virtual IP addresses</a>, <a href="https://docs.opnsense.org/manual/unbound.html#overrides">DNS records</a>, and HAProxy listeners.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmk-JSAqSsflG1j3pyJ_gQ.png" /></figure><p>1️⃣ The first step is to create two Virtual IP addresses that will act as stable entry points into the Kubernetes cluster. One is reserved for the NGINX ingress controller that will later expose the public OpenStack endpoints, while the other is used by Traefik, which provides access to cluster management and supporting services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNpxXDEiqqc9VQL3pxaZxA.png" /></figure><p>2️⃣ Unbound DNS overrides are then configured so that requests for the OpenStack and management domains resolve to the appropriate Virtual IPs. This gives us predictable hostnames from the very beginning of the deployment and avoids having to track individual Kubernetes node addresses as services move around the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDggcXEEUmPNisr4OfGG2Q.png" /></figure><p>3️⃣ Finally, HAProxy has to be configured to distribute incoming traffic across all Kubernetes nodes. Rather than directing requests to a specific machine, HAProxy forwards them to whichever node is currently capable of serving the requested ingress traffic. This creates a single, consistent access layer in front of the cluster and allows the ingress controllers to remain highly available even if individual nodes become unavailable.</p><p>First we need to create the <strong>Real Servers</strong>:</p><ul><li>Navigate to <strong>Services → HAProxy</strong>.</li><li>Open the <strong>Real Servers</strong> configuration page.</li><li>Create a total of <strong>12 server definitions</strong>, 4 for each Kubernetes node.</li><li>The first group represents the NGINX ingress endpoints (80 and 443).</li><li>The second group represents the Traefik ingress endpoints (8080 and 8443).</li><li>Apply the configuration before proceeding with the creation of backend pools and public listeners.</li></ul><figure><img alt="OPNsense HAProxy configuration screen showing the definition of real servers for a three-node K3s cluster. Each Kubernetes node is registered four times to represent the NGINX ingress endpoints on ports 80 and 443 and the Traefik ingress endpoints on ports 8080 and 8443. These server entries are later used to build backend pools and public listeners for OpenStack and cluster management traffic." src="https://cdn-images-1.medium.com/max/1024/1*4SHYk11sigSowbxXqyGX_g.png" /><figcaption>Creating the HAProxy real server definitions used to forward ingress traffic to the Kubernetes cluster. Each node is registered multiple times, once for every ingress endpoint exposed by the cluster: HTTP (80) and HTTPS (443) for the NGINX ingress controller, and HTTP (8080) and HTTPS (8443) for Traefik. These backend definitions are later grouped into HAProxy pools and exposed through dedicated virtual IP addresses.</figcaption></figure><p>Next we need to create the <strong>Backend Pools</strong>:</p><ul><li>Navigate to <strong>Services → HAProxy</strong>.</li><li>Open <strong>Virtual Services → Backend Pools</strong>.</li><li>Create a total of <strong>4 backend pools</strong>.</li><li>Add the corresponding server entries for all three Kubernetes nodes to each pool (the ones we created in the previous step).</li><li>The resulting pools represent the NGINX (80, 443) and Traefik (8080, 8443) ingress endpoints.</li><li>Apply the configuration before proceeding with the creation of public services.</li></ul><figure><img alt="OPNsense HAProxy configuration screen showing the creation of backend pools. Four pools are configured to group the NGINX ingress endpoints on ports 80 and 443 and the Traefik ingress endpoints on ports 8080 and 8443. Each pool contains the corresponding ingress servers from all three Kubernetes nodes, enabling HAProxy to load-balance traffic across the cluster." src="https://cdn-images-1.medium.com/max/1024/1*V_tmDn8XeZMJ5YNGIjMDLA.png" /><figcaption>Creating the HAProxy backend pools that aggregate the ingress endpoints of all Kubernetes nodes. Four pools are defined: two for the NGINX ingress controller (80 and 443) and two for Traefik (8080 and 8443). Each pool contains the corresponding server definitions from every cluster node, allowing HAProxy to distribute traffic across the cluster rather than targeting a single host.</figcaption></figure><p>Last we need to create the <strong>Public Services</strong>:</p><ul><li>Navigate to <strong>Services → HAProxy</strong>.</li><li>Open <strong>Virtual Services → Public Services</strong>.</li><li>Configure the listener to bind to the appropriate virtual IP address and port.</li><li>Associate the listener with the corresponding backend pool.</li><li>Create a total of <strong>4 public services</strong>: two for the NGINX ingress VIP and two for the Traefik ingress VIP.</li><li>Apply the configuration to activate the listeners.</li></ul><figure><img alt="OPNsense HAProxy configuration screen showing the creation of public services. Each listener is bound to a virtual IP address and forwards traffic to a corresponding backend pool. Four listeners are configured in total, exposing the NGINX ingress controller on ports 80 and 443 and the Traefik ingress controller on ports 80 and 443 through separate virtual IP addresses." src="https://cdn-images-1.medium.com/max/1024/1*P-LKfwASAXGv1hXmQ5zkXA.png" /><figcaption>Creating the HAProxy public services that expose the Kubernetes ingress controllers through dedicated virtual IP addresses. Each public service binds a virtual IP and port combination to one of the backend pools created previously, providing a stable entry point for traffic entering the cluster. Two listeners are configured for the NGINX ingress controller (80 and 443) and two for Traefik (80 and 443), each mapped to their respective backend pools.</figcaption></figure><h3>Install Ceph</h3><p>With Kubernetes deployed and external access to the cluster established through HAProxy, DNS, and ingress controllers, the next component we need is storage. OpenStack services generate and consume persistent data from the very beginning of the deployment, whether it is virtual machine images, block volumes, databases, or service metadata. Rather than relying on local node storage, we will build a distributed storage layer that is shared across the entire cluster.</p><p>To achieve this, the install-ceph.sh script deploys a Rook-managed Ceph cluster on Kubernetes. Each node (Proxmox virtual machine) contributes a dedicated storage device, exposed to the virtual machine as a raw disk and attached as /dev/sdb. These disks are used <strong>exclusively</strong> by Ceph to create Object Storage Daemons (OSDs), which form the foundation of the distributed storage cluster.</p><pre>sudo ./scripts/install-ceph.sh</pre><p>That will create:</p><p>✅ a Ceph deployment (1 manager, 3 monitors, 3 OSDs on /dev/sdb on each VM)<br>✅ aCephCluster named ceph, and a CephBlockPool named rdb<br>✅ aStorageClass named general based on provisioner rook-ceph.rbd.csi.ceph.com<br>✅ the Ceph management web interface (via Traefik ingress)</p><p>Rook takes care of deploying and managing the Ceph components inside Kubernetes, while Ceph itself provides the resilient block storage backend consumed by both Kubernetes and OpenStack. As part of the installation, the script creates the Ceph cluster, provisions the OSDs on the dedicated disks, creates an RBD pool, and exposes a Kubernetes StorageClass that will later be used by OpenStack-Helm itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVV2gdpMoyGXUaiI-k83YQ.png" /></figure><blockquote>At this stage, we are not yet deploying any OpenStack services. Instead, we are establishing one of the most critical building blocks of the platform: <strong>a shared storage layer that spans all nodes in the cluster</strong>. By the time the installation completes, the Kubernetes environment has gained a distributed storage backend capable of supporting Glance image storage, Cinder volumes, and the persistent data requirements of the OpenStack control plane.</blockquote><p>The Ceph dashboard is also exposed through the management ingress, allowing you to verify the health of the cluster before proceeding with the OpenStack deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVrpBhmDPTwKjp6XHX8JHw.png" /></figure><h3>Install OpenStack-Helm Dependencies</h3><p>At this point, we have a functioning Kubernetes cluster, a distributed Ceph storage backend, and a network layer capable of exposing services outside the cluster. What we do not have yet is an environment that OpenStack can actually run on.</p><p>OpenStack-Helm is a bit more than a collection of Helm charts. It expects Kubernetes to be organized in a particular way, with dedicated ingress paths, storage integrations, scheduling rules, and supporting components already in place. Before we can deploy a single OpenStack service, we need to prepare the cluster for that matter.</p><p>The install-osh-deps.sh script prepares the environment expected by the OpenStack-Helm charts. This step creates the required namespaces, pulls the OpenStack-Helm repositories, and installs the Rook-Ceph adapter so that OpenStack-Helm can consume the Ceph cluster created in the previous step. It also deploys thededicated NGINX ingress controller for the public OpenStack endpoints. This keeps OpenStack API and dashboard traffic separate from the management services exposed through Traefik . Later, services such as Keystone, Horizon, and Skyline will use this ingress path.</p><pre>sudo ./scripts/install-osh-deps.sh</pre><blockquote><em>K3s ships with </em><a href="https://docs.k3s.io/networking/networking-services#traefik-ingress-controller"><em>Traefik</em></a><em> as its default ingress controller. Earlier in the deployment, we reconfigured Traefik to listen on ports </em><em>8080 and </em><em>8443, reserving the standard </em><em>80 and </em><em>443 ports for the dedicated NGINX ingress controller that will expose the OpenStack APIs and dashboards.</em></blockquote><p>Another important part of this step is node preparation. OpenStack-Helm relies on specific node labels and scheduling expectations, so the script adjusts labels and taints to make the K3s nodes suitable for OpenStack workloads. Without this alignment, charts may install successfully but fail to schedule the required pods.</p><p>Additionally it will deploy <a href="https://github.com/kite-org/kite">Kite</a>, a modern lightweight Kubernetes web dashboard, and expose it via Traefik.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6YIno3hjFQVZGkK7TJm_A.png" /></figure><h3>Install OpenStack Infrastructure Components</h3><p>Now that Kubernetes has been prepared for OpenStack-Helm, we can start deploying the first OpenStack layer. This is not yet the cloud that users will interact with. There is no dashboard, no identity service, no images, and no virtual machines. What we are deploying here is the internal machinery that the rest of OpenStack depends on.</p><p>The infrastructure phase brings up the shared services used by the OpenStack control plane. <a href="https://mariadb.org/">MariaDB</a> provides the database backend, <a href="https://www.rabbitmq.com/">RabbitMQ</a> handles service-to-service messaging, <a href="https://www.memcached.org/">Memcached</a> supports caching, and <a href="https://www.openvswitch.org/">Open vSwitch</a> lays the groundwork for the networking stack that Neutron will use later.</p><pre>sudo ./scripts/install-osh.sh --infra</pre><p>In the next phase we will focus on the actual control plane: identity, image management, dashboards, and the services users will recognize as OpenStack.</p><h3>Install OpenStack Core Components</h3><p>With the underlying infrastructure now in place, we can begin deploying the OpenStack control plane.</p><p>1️⃣ This phase introduces the first services that users will directly interact with and marks the transition from platform preparation to cloud deployment. While the previous steps focused on Kubernetes, storage, networking, and supporting infrastructure, this stage establishes the foundations of an operational OpenStack environment.</p><pre>sudo ./scripts/install-osh.sh --core</pre><p>The deployment begins with <strong>Keystone</strong>, which serves as the identity service for the entire platform. Every authentication request, service registration, and API interaction ultimately passes through Keystone, making it the central point of trust within the cloud.</p><p>Next comes <strong>Glance</strong>, the image service responsible for storing and distributing virtual machine images. Whether launching a Linux distribution, a network appliance, or a custom workload, Glance becomes the catalog from which compute instances obtain their boot images.</p><p>To provide a user-facing experience, <strong>Horizon</strong> is deployed as the traditional OpenStack dashboard. Alongside it, we can optionally deploy <strong>Skyline</strong> which offers a more modern web interface built around OpenStack APIs, giving users an alternative way to interact with cloud resources. Both dashboards are exposed through the dedicated NGINX ingress layer established earlier, making them accessible through the public OpenStack endpoints.</p><figure><img alt="OpenStack Horizon login page displayed in a web browser. The interface contains fields for username, password, and domain selection, along with the OpenStack logo and sign-in button. Horizon serves as the traditional web dashboard for managing OpenStack resources and authenticates users through Keystone." src="https://cdn-images-1.medium.com/max/1024/1*u3Gv2qfMu2_cCJEMRYPoDQ.png" /><figcaption>Horizon is the traditional OpenStack dashboard and provides a web-based interface for managing cloud resources, including instances, networks, images, volumes, and projects.</figcaption></figure><blockquote>At this stage, our cloud is <strong>not yet</strong> capable of launching virtual machines. The compute and networking services that make this possible are introduced later. What we laid here is the operational foundations of OpenStack: identity management, image management, and user access.</blockquote><p>The remaining phases will build upon this foundation by adding storage, orchestration, networking, and ultimately the ability to provision and operate virtual machines.</p><p>2️⃣ Although the Kubernetes platform already has access to distributed storage through Ceph, OpenStack still needs a way to expose that storage to its users. This is the role of Cinder.</p><p><strong>Cinder</strong> introduces block storage as a first-class OpenStack service, allowing users and applications to provision persistent volumes independently of the virtual machines that consume them. These volumes can be attached, detached, resized, and managed throughout their lifecycle without affecting the underlying instances.</p><pre>sudo ./scripts/install-osh.sh --cinder</pre><blockquote>In this deployment, Cinder is backed by the Ceph cluster that was created earlier. This means that block volumes are not tied to a particular Kubernetes node or compute host. Instead, they are stored on the distributed Ceph backend, providing the resilience and flexibility expected from a real cloud storage service.</blockquote><blockquote>While users may not immediately interact with Cinder when exploring the dashboards, many workloads depend on it. Databases, stateful applications, and long-lived virtual machines often require storage that persists beyond the lifetime of a single instance. Cinder provides that capability and completes another essential building block of the cloud platform.</blockquote><p>3️⃣ The final component introduced during this phase is Heat, OpenStack’s native orchestration service.</p><p><strong>Heat</strong> allows collections of OpenStack resources to be defined as templates and deployed as a single stack. Rather than creating networks, security groups, volumes, floating IPs, and compute instances individually, Heat coordinates the creation of these resources by interacting with the corresponding OpenStack APIs in the correct order.</p><pre>sudo ./scripts/install-osh.sh --heat</pre><p>A Heat template can describe both the resources themselves and the relationships between them. For example, a template can specify that a volume must be attached to a server, or that a floating IP must be associated with a particular instance. Heat then orchestrates the deployment and lifecycle of the entire stack as a single unit.</p><p>Although Heat is not required to operate the cloud, it is an integral part of the OpenStack ecosystem and provides the orchestration capabilities expected from a complete OpenStack deployment. By installing it at this stage, we ensure that the environment includes OpenStack’s native mechanism for template-driven resource provisioning and lifecycle management.</p><p>4️⃣ The final service deployed, which is optional as we already have installed Horizon, is Skyline, OpenStack’s next-generation web dashboard.</p><p>For many years, Horizon has been the primary graphical interface for OpenStack. <strong>Skyline</strong> was introduced to provide a more modern user experience, built around OpenStack APIs and contemporary web technologies while exposing many of the same cloud management capabilities.</p><pre>sudo ./scripts/install-osh.sh --skyline</pre><blockquote>Skyline does not replace any core OpenStack functionality. Instead, it provides an alternative way for administrators and users to interact with the platform, complementing the traditional Horizon dashboard already deployed in the previous step.</blockquote><figure><img alt="Skyline login page displayed through a web browser. The interface shows the OpenStack region selector, domain selection, username and password fields, and a custom cloud branding theme. The page is served through the OpenStack ingress layer and authenticates users against Keystone." src="https://cdn-images-1.medium.com/max/1024/1*MxSaIJvAW_auD6y53zP0hw.png" /><figcaption>Skyline provides a modern web interface for interacting with OpenStack services and complements the traditional Horizon dashboard deployed earlier in the guide.</figcaption></figure><p>By exposing both Horizon and Skyline, the environment offers access to both the established OpenStack interface and its newer counterpart, allowing users to choose the experience that best fits their operational preferences.</p><h3>Install Libvirt</h3><p>Up to this point, every component we have deployed has been focused on supporting the OpenStack control plane. This step shifts our attention to the compute layer.</p><p>OpenStack ultimately exists to provision and manage virtual machines, and those virtual machines need a hypervisor. In this deployment, that role is provided by <a href="https://linux-kvm.org/page/Main_Page">KVM</a> and <a href="https://libvirt.org/">libvirt</a> running directly on the Kubernetes nodes.</p><p>The purpose of this step is to prepare the cluster for virtualization workloads <strong>before</strong> the Nova compute services are introduced. Because the deployment runs on Kubernetes, libvirt is not installed in the traditional manner on dedicated hosts. Instead, it is deployed as a Kubernetes workload with the permissions and host access required to interact with the underlying KVM capabilities exposed by each node.</p><pre>sudo ./scripts/install-osh.sh --libvirt</pre><blockquote>It is important to note at this point, that t<strong>he libvirt pods do not immediately transition to a </strong>Running<strong> state</strong>. <em>This is expected</em>. At this stage, part of the networking infrastructure required by the compute stack is not yet available. Once Neutron and the remaining compute services are deployed in the next phase, libvirt completes its initialization and becomes fully operational.</blockquote><p>By the end of this step, the cluster has everything required to host virtual machines from a hypervisor perspective. The final piece missing is the OpenStack compute stack itself, which will connect Nova, Neutron, Placement, and libvirt into a complete virtualization platform.</p><h3>Install OpenStack Compute Kit</h3><p>This is the final major step of the deployment and the point at which our OpenStack cloud becomes fully operational.</p><p>Up to now, we have assembled all the supporting layers required by the platform: Kubernetes provides orchestration, Ceph provides storage, the control plane services handle identity and image management, and libvirt has prepared the nodes for virtualization. What remains is to connect these pieces together and introduce the services responsible for scheduling, networking, and running virtual machines.</p><p>The compute kit deploys three key OpenStack services: Placement, Nova, and Neutron.</p><pre>sudo ./scripts/install-osh.sh --compute</pre><p><strong>Placement</strong> is responsible for resource inventory and allocation. It maintains an accurate view of the compute resources available across the environment and allows OpenStack services to determine where workloads can be scheduled.</p><p><strong>Nova</strong> provides the compute service itself. It is responsible for managing the lifecycle of virtual machine instances, coordinating with the hypervisor layer, consuming images from Glance, and interacting with the rest of the OpenStack control plane whenever an instance is created, modified, or removed.</p><p><strong>Neutron</strong> delivers networking services to the cloud. It provides the abstractions used to create tenant networks, routers, ports, security groups, and floating IPs, while integrating with the Open vSwitch infrastructure deployed earlier in the process.</p><p>This step configures Nova and Neutron to operate with KVM virtualization, Open vSwitch networking, and the Ceph storage backend established in the previous steps. Together, these services transform the environment from a collection of infrastructure components into a functioning Infrastructure-as-a-Service platform. Once this phase completes, our cloud is capable of launching virtual machines. Users can authenticate through Keystone, select images stored in Glance, provision storage through Cinder, consume networking services through Neutron, and have Nova schedule and run workloads across the cluster.</p><p>At this point, our OpenStack deployment is complete and ready to host its first instances.</p><h3>Validation and Testing</h3><p>1️⃣ Open your Horizon dashboard at https://horizon.osh.proxmox.home (or the equivalent FQDN in your environment) and sign in using the default administrative credentials:</p><ul><li><strong>Username:</strong> admin</li><li><strong>Password:</strong> password</li><li><strong>Domain:</strong> Default</li></ul><p>2️⃣ Open <strong>Project</strong> <strong>→ API Access</strong> where you should be able to see all OpenStack services API endpoints we deployed so far:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t-wwUCeaTsdeyyqdpLCo2w.png" /></figure><p>3️⃣ Open <strong>Admin</strong> <strong>→ Compute → Hypervisors → Compute Host </strong>where you should be able to see all KVM hosts available. Make sure their <strong>Status</strong> is Enabled and their <strong>State</strong> is Up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrcoQ2gxTLEYX8sq0qMJBQ.png" /></figure><p>4️⃣ First, we are going to create a shared VPC. Open <strong>Admin</strong> <strong>→ Network → Networks</strong> and click <strong>Create Network. </strong>Choose the following parameter values<strong>:</strong></p><ul><li><strong>Name:</strong> test_vpc</li><li><strong>Project:</strong> admin</li><li><strong>Provider Network Type:</strong> VXLAN</li><li><strong>Segmentation ID</strong>: 959</li><li><strong>Enable Admin State</strong>: true</li><li><strong>Shared</strong>: true</li><li><strong>MTU</strong>: 1450</li><li><strong>Subnet Name</strong>: test_subnet</li><li><strong>Network Address</strong>: 192.168.111.0/24</li></ul><p>The values used for <strong>Segmentation ID</strong> (959) and <strong>Network Address</strong> (192.168.111.0/24) are provided as examples only. The VXLAN segmentation ID can be any unused identifier within your environment, and the subnet CIDR can be replaced with any non-overlapping private network range that fits your addressing scheme.</p><p>These are going to be the private addresses assigned to the virtual machines, valid only within the VPC itself. The only requirement is that both values are unique and do not conflict with existing OpenStack networks.</p><p>5️⃣ Next, we are going to create an External Network. Open <strong>Admin</strong> <strong>→ Network → Networks</strong> and click <strong>Create Network. </strong>Choose the following parameter values<strong>:</strong></p><ul><li><strong>Name:</strong> ext</li><li><strong>Project:</strong> admin</li><li><strong>Provider Network Type:</strong> flat</li><li><strong>Physical Network</strong>: public</li><li><strong>Enable Admin State</strong>: true</li><li><strong>Shared</strong>: true</li><li><strong>External Network</strong>: true</li><li><strong>Subnet Name</strong>: ext_subnet</li><li><strong>Network Address</strong>: 10.70.0.0/16</li><li><strong>Gateway IP</strong>: 10.70.168.1</li><li><strong>Enable DHCP</strong>: false</li><li><strong>Allocation Pools</strong>: 10.70.250.1,10.70.250.254</li><li><strong>DNS Name Servers</strong>: 10.70.168.1</li></ul><p>Unlike the tenant network created in the previous step, <strong>these values are not arbitrary</strong> and must reflect the characteristics of your environment. The <strong>Physical Network</strong> value (public) refers to the Neutron provider network configured by the deployment and mapped to the secondary network interface (eth1) of the Kubernetes nodes. The subnet configuration (10.70.0.0/16), gateway (10.70.168.1), allocation pool, and DNS server originate from the VLAN-backed network attached to that interface and managed by OPNsense. This network serves as the external connectivity layer for OpenStack and is used by services such as floating IPs and provider networks. If your environment uses a different VLAN, subnet, gateway, or address allocation scheme, adjust these values accordingly to match your own network configuration.</p><figure><img alt="OPNsense DHCPv4 configuration screen for the VLAN-backed external network. The interface is configured with a 10.70.0.0/16 subnet, a DHCP allocation range from 10.70.168.2 to 10.70.168.254, and a DNS server at 10.70.168.1. These values are later referenced when configuring the OpenStack external network and floating IP address pool." src="https://cdn-images-1.medium.com/max/1024/1*BSNe2Jg5113h0Rhvz4Zthg.png" /><figcaption>The subnet allocated to VLAN 70, the DHCP address pool used for dynamic address assignment, and the DNS server advertised to clients. These values form the basis of the external network configuration created in Neutron.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgGWEYuBRqde0oraWJ0QQA.png" /></figure><p>6️⃣ Next, we are going to create a Router and Floating IPs.</p><p>Go to <strong>Admin</strong> <strong>→ Network → Routers </strong>and click<strong> Create Router </strong>using as<strong> Name: </strong>router1<strong> </strong>and<strong> External Network: </strong>ext. Then, open <strong>router1</strong> and go to <strong>Interfaces → Add Interface </strong>and select as private subnet test_vpc/192.168.111.0/24.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uh3u_E9OAS87faeAquE1ww.png" /></figure><p>Go to <strong>Admin</strong> <strong>→ Network → Floating IPs </strong>click <strong>Allocate IP to Project</strong>, choose <strong>Pool</strong>: ext 10.70.0.0/16 and click <strong>Allocate Floating IP</strong> to exit.</p><p>7️⃣ With the networking stack in place, it is time to validate that the cloud can perform its primary function: launching virtual machines and providing network connectivity between them.</p><p>Create 3 instances using the CirrOS image and the m1.tiny flavor. Attach all three instances to the test_vpc network created earlier. For one of the instances, also associate the floating IP allocated from the ext external network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ps4rZZWaxi3zuWiOqDhZ1w.png" /><figcaption>Our 3 CirrOS instances. Two instances use only tenant networking, while the third is associated with a floating IP from the external network, providing a path for validating external connectivity and SSH access.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a30qMd5MeWNGIE3Qo12hnQ.png" /><figcaption>Our network topology after the deployment of the tenant network, router, floating IP, and test instances.</figcaption></figure><p>Once the instances have reached the <strong>Active</strong> state, open their VNC consoles from Horizon and perform a series of connectivity checks:</p><p>↔️ Start by verifying <strong>east-west connectivity within the tenant network</strong>. From each instance, ping the private IP addresses of the other two instances to confirm that Neutron networking, Open vSwitch, and the VXLAN overlay are operating correctly.</p><p>↕️ Next,<strong> verify north-south connectivity by testing outbound access from the instances to an external destination</strong>. A successful response confirms that routing and external network access are functioning as expected.</p><figure><img alt="VNC console of a CirrOS virtual machine running in OpenStack. The terminal shows successful ICMP communication with another instance on the tenant network (192.168.111.202) and successful connectivity to the external address 8.8.8.8. The output demonstrates both internal tenant-network communication and outbound internet access through the OpenStack networking stack." src="https://cdn-images-1.medium.com/max/1024/1*CpQDe6-IaKg6Z00DZuHj_g.png" /><figcaption>Connectivity validation from a virtual machine running inside the test_vpc tenant network. The instance successfully communicates with another instance on the same VXLAN-backed network and reaches an external destination through the Neutron router and external provider network, confirming that both east-west and north-south networking are functioning correctly.</figcaption></figure><p>As we can see in the image above:</p><ul><li><strong>East-west connectivity test</strong>: The instance (192.168.111.195) successfully pings another virtual machine (192.168.111.202) on the same tenant network, validating Neutron networking, Open vSwitch forwarding, and VXLAN encapsulation between instances.</li><li><strong>North-south connectivity test:</strong>The instance successfully reaches an external destination (8.8.8.8), confirming that routing, source NAT, and connectivity through the external provider network are operating as expected.</li></ul><p>🏁 As last step we need to validate floating IP connectivity. From your workstation, establish an SSH connection to the instance that was assigned the floating IP. If the connection succeeds, traffic is successfully traversing the complete network path: OPNsense, the provider network, Neutron, Open vSwitch, and the target virtual machine.</p><h3>Closing Thoughts</h3><p>Running OpenStack on Kubernetes is no longer an experiment or a proof of concept. Projects such as OpenStack-Helm, Rook and Kubernetes have matured to the point where complex cloud platforms can be deployed, operated, and upgraded using the same cloud-native practices that have become commonplace the last years.</p><p>At a time when digital sovereignty, technological self-determination, and control over critical infrastructure are rapidly becoming matters of national security, understanding how cloud platforms are built is valuable in its own right. The ability to operate and evolve strategic infrastructure without dependency on foreign providers, proprietary ecosystems, or external political decisions is no longer merely a technical consideration; it is increasingly becoming a question of resilience, autonomy and long-term competitiveness.</p><p>If you found this post helpful, give it a 👏 and follow me for more insights on Kubernetes and cloud-native tech. More great content coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4ff01e2b424" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/deploy-openstack-on-kubernetes-in-under-60-minutes-f4ff01e2b424">Deploy OpenStack on Kubernetes in under 60 minutes</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-Cluster Leader Election for Kubernetes Workloads]]></title>
            <link>https://itnext.io/cross-cluster-leader-election-for-kubernetes-workloads-3548f77b0531?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/3548f77b0531</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[leader-election]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 09:46:46 GMT</pubDate>
            <atom:updated>2026-04-13T19:32:47.151Z</atom:updated>
            <content:encoded><![CDATA[<p>Kubernetes-native leader election across multiple Kubernetes clusters using a shared PostgreSQL database and the client-go leader election module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/0*kCOyWtmxSUFB2E6j.jpg" /><figcaption>Leader Election, Across the Cluster-Verse</figcaption></figure><h3>TL;DR</h3><p>This article explores how to achieve cross-cluster leader election for Kubernetes workloads using PostgreSQL as a backend instead of the Kubernetes API. By plugging a custom database-backed lock into the <a href="https://github.com/kubernetes/client-go/tree/master/tools/leaderelection">client-go leader election module</a>, multiple instances even across different clusters, can coordinate to ensure that only one acts as the leader at any given time.</p><h3>Overview</h3><p>In distributed systems, especially in Kubernetes environments, it’s common to run multiple replicas of the same workload for availability and scalability. However, not all workloads are designed to run concurrently across all available replicas. Some require a <strong>single active instance</strong> responsible for coordinating actions such as scheduling jobs, processing queues, managing shared state etc. This is where leader election becomes necessary.</p><h3>Leader Election in Kubernetes</h3><p>Kubernetes ensures that multiple instances of a Pod can run reliably, <strong>but it does not inherently guarantee that only one of those instances performs a specific task; aka act a a leader</strong>. Without coordination, this can lead to duplicated work, race conditions, or inconsistent system state.</p><p><em>Leader election</em> introduces a coordination mechanism where one instance is elected as the leader and is responsible for executing critical or even all sections of the workload, while the others remain passive or on standby. If the leader fails or becomes unavailable, another instance takes over. This pattern is commonly used in controllers, operators, and background workers. In a typical Kubernetes setup, leader election is scoped to a single cluster because it relies on the Kubernetes API as the coordination backend.</p><p><strong>This becomes a limitation when workloads span multiple clusters and require a single global leader.</strong></p><p>Kubernetes provides a resource called Lease, as part of the coordination API group, which is specifically designed for lightweight distributed coordination. A Lease object contains metadata such as the identity of the current leader, the duration of the lease and timestamps for renewals.</p><p>Leader election works by having participants attempt to acquire or renew a lease. The pod that successfully holds and continuously renews the lease is considered the leader of the replica set. If the lease is not renewed within a defined time window, it is assumed that the leader has failed, and other participants can attempt to acquire it.</p><p>This mechanism is efficient because it avoids heavy resource contention and relies on simple update operations against the Kubernetes API server. However, it is inherently limited to a single cluster. <strong>When workloads span multiple clusters, this model no longer applies.</strong></p><p>In such cases, one option is to introduce a shared “control-plane” cluster and have all workloads coordinate through its Kubernetes API, which quickly becomes complex and difficult to manage. An alternative approach, which this article focuses on, is to use a PostgreSQL database as a shared backend for storing and updating the leader election state.</p><p>This solution relies on the Kubernetes leader election module, specifically k8s.io/client-go/tools/leaderelection/resourcelock which implements the core leader election algorithm including the lease acquisition and renewal logic, the leader transition handling and the respective timing configuration (lease duration, renew deadline, retry period etc.)</p><p>Instead of using the default Kubernetes-backed lock (such as Lease objects), the implementation provides a custom lock that satisfies the expected resourcelock interface.</p><blockquote>This allows the leader election logic from <a href="https://github.com/kubernetes/client-go/tree/master/tools/leaderelection">client-go</a> to remain unchanged, while replacing the storage backend with PostgreSQL. As a result, the same leader election semantics are preserved, <strong>but coordination is no longer tied to a single Kubernetes control plane.</strong></blockquote><h3>Database setup and migration execution</h3><p>To enable leader election across clusters, we need a shared and reliable coordination backend. In this implementation, PostgreSQL is used to store the leader election state. Every participating instance (each pod from each replica set, from every cluster) connects to the same database and reads or updates a single record that represents the current leader.</p><p>The schema is minimal but you could extend it as you wish in order to hold the information you need. It is just a single table named leader_election_locks that holds <strong>one row per logical election</strong>. Each row is identified by a combination of namespace and name, and stores the serialized leader election record in JSON format. A version field is used to implement optimistic concurrency control, ensuring that updates only succeed if no other instance has modified the record in the meantime. An updated_at timestamp is also maintained for observability and debugging.</p><blockquote>The article assumes you have already a PostgreSQL instance in place. If not, you can easily spin up one using the <a href="https://awesome-docker-compose.com/postgresql-official">official docker-compose manifest</a> or via <a href="https://autobase.tech/">Autobase for PostgreSQL</a>.</blockquote><p>To apply the migration, the repository provides a small helper under cmd/migration, that collects and executes the SQL files located in the migrations directory:</p><pre>CREATE TABLE leader_election_locks (<br>    namespace   text        NOT NULL,<br>    name        text        NOT NULL,<br>    record_json jsonb       NOT NULL,<br>    version     bigint      NOT NULL DEFAULT 1,<br>    updated_at  timestamptz NOT NULL DEFAULT now(),<br>    PRIMARY KEY (namespace, name)<br>);<br><br>CREATE INDEX idx_leader_election_locks_updated_at<br>    ON leader_election_locks (updated_at);</pre><p>Running it is straightforward:</p><pre>go run ./cmd/migration</pre><p><a href="https://github.com/akyriako/pgleele/blob/9f809dc3c13c9858d12cbd93cba740e0974f5873/cmd/migrate/main.go">pgleele/cmd/migrate/main.go at 9f809dc3c13c9858d12cbd93cba740e0974f5873 · akyriako/pgleele</a></p><p>Once executed, the required table is created and ready to be used by the leader election logic. From that point on, all instances participating in the election will rely on this table as their shared coordination point.</p><h3>Implementing the PostgreSQL resource lock</h3><p>At the core of this solution is pglock/lock.go , where we replaces the default Kubernetes resource lock with a PostgreSQL-backed alternative. Instead of relying on Lease objects stored in the Kubernetes API, the custom Lock component persists and manages the leader election state directly in the database.</p><p>The PostgreSQL lock works because it satisfies the interface expected by the Kubernetes leader election library, namely the resourcelock.Interface from <a href="https://github.com/kubernetes/client-go/tree/master/tools/leaderelection">client-go</a>, and delegates all state operations to PostgreSQL.</p><p>This interface defines how the leader election module interacts with the underlying coordination mechanism. It does not care whether the state is stored in a Kubernetes Lease, a ConfigMap, or, in this case, a PostgreSQL table. As long as the interface is implemented correctly, the leader election logic remains unchanged.</p><p>The interface requires a small set of methods:</p><ul><li>Get() retrieves the current leader election record</li><li>Create() initializes the record if it does not exist</li><li>Update() attempts to modify the record</li><li>Identity() returns the unique identity of the participant</li><li>Describe() provides a human-readable identifier for logging</li></ul><p>By implementing these methods in the Lock component in pglock/lock.go, the PostgreSQL-backed lock becomes a drop-in replacement for the standard Kubernetes resource locks. The Lock component identifies the specific election using a namespace and name, and keeps track of a local version. This version is important, as it enables optimistic concurrency control when multiple instances attempt to update the same record.</p><p>1️⃣ The leader election module periodically calls Get() method which tries to retrieves the current election state, resourcelock.LeaderElectionRecord from the database and allows each instance to understand who the current leader is and whether the lock is already held.</p><pre>func (l *Lock) Get(ctx context.Context) (*resourcelock.LeaderElectionRecord, []byte, error) {<br> const q = `<br>SELECT record_json, version<br>FROM leader_election_locks<br>WHERE namespace = $1 AND name = $2<br>`<br><br> var raw []byte<br> var version int64<br><br> err := l.pool.QueryRow(ctx, q, l.namespace, l.name).Scan(&amp;raw, &amp;version)<br> if err != nil {<br>  if errors.Is(err, pgx.ErrNoRows) {<br>   return nil, nil, ErrNotFound<br>  }<br>  return nil, nil, fmt.Errorf(&quot;getting leader lock failed: %w&quot;, err)<br> }<br><br> var rec resourcelock.LeaderElectionRecord<br> if err := json.Unmarshal(raw, &amp;rec); err != nil {<br>  return nil, nil, fmt.Errorf(&quot;unmarshalling leader record failed: %w&quot;, err)<br> }<br><br> l.mu.Lock()<br> l.lastVersion = version<br> l.mu.Unlock()<br><br> return &amp;rec, raw, nil<br>}</pre><p>2️⃣ If no record exists yet, the Create() method is used to insert the a row. Since multiple instances may attempt this at the same time, the implementation relies on PostgreSQL’s unique constraint on (namespace, name) to ensure only one succeeds. Any duplicate insert attempts are treated as a normal primary key violation, guaranteeing that way deterministic strong consistency.</p><pre>func (l *Lock) Create(ctx context.Context, ler resourcelock.LeaderElectionRecord) error {<br> raw, err := json.Marshal(ler)<br> if err != nil {<br>  return fmt.Errorf(&quot;marshalling leader record failed: %w&quot;, err)<br> }<br><br> const q = `<br>INSERT INTO leader_election_locks(namespace, name, record_json, version, updated_at)<br>VALUES ($1, $2, $3, 1, now())<br>`<br> _, err = l.pool.Exec(ctx, q, l.namespace, l.name, raw)<br> if err != nil {<br>  var pgErr *pgconn.PgError<br>  if errors.As(err, &amp;pgErr) &amp;&amp; pgErr.Code == &quot;23505&quot; {<br>   return ErrAlreadyExists<br>  }<br>  return fmt.Errorf(&quot;creating leader lock failed: %w&quot;, err)<br> }<br><br> l.mu.Lock()<br> l.lastVersion = 1<br> l.mu.Unlock()<br><br> return nil<br>}</pre><p>3️⃣ The most critical part is the Update() method. This is where leadership is renewed or transferred. Instead of blindly updating the row, the query includes the previously observed version. If the version still matches, the update succeeds and increments the version. If not, it means another instance has already modified the record, and the update is rejected.</p><p>This approach ensures that only one instance can successfully update the leader record at a time, which is enough to maintain consistent leader election semantics. Combined with the retry and timing logic from <a href="https://github.com/kubernetes/client-go/tree/master/tools/leaderelection">client-go</a>, it provides a reliable coordination mechanism without requiring any Kubernetes API interaction.</p><pre>func (l *Lock) Update(ctx context.Context, ler resourcelock.LeaderElectionRecord) error {<br> raw, err := json.Marshal(ler)<br> if err != nil {<br>  return fmt.Errorf(&quot;marshalling leader record failed: %w&quot;, err)<br> }<br><br> l.mu.Lock()<br> expectedVersion := l.lastVersion<br> l.mu.Unlock()<br><br> if expectedVersion == 0 {<br>  return fmt.Errorf(&quot;updating called before get; no cached version&quot;)<br> }<br><br> const q = `<br>UPDATE leader_election_locks<br>SET record_json = $3,<br>    version = version + 1,<br>    updated_at = now()<br>WHERE namespace = $1<br>  AND name = $2<br>  AND version = $4<br>RETURNING version<br>`<br> var newVersion int64<br> err = l.pool.QueryRow(ctx, q, l.namespace, l.name, raw, expectedVersion).Scan(&amp;newVersion)<br> if err != nil {<br>  if errors.Is(err, pgx.ErrNoRows) {<br>   return ErrVersionConflict<br>  }<br>  return fmt.Errorf(&quot;updating leader lock failed: %w&quot;, err)<br> }<br><br> l.mu.Lock()<br> l.lastVersion = newVersion<br> l.mu.Unlock()<br><br> return nil<br>}</pre><blockquote>This separation is what makes the approach robust. The proven leader election behavior from Kubernetes is preserved, while the storage backend is replaced with PostgreSQL, enabling coordination across clusters without changing the core algorithm.</blockquote><h3>Implementing a sample workload</h3><p>To understand how everything fits together, we will build a simple client that requires a leader and consequently a leader election process. This sample will not demonstrate <em>how</em> leader election works (that is already handled by <a href="https://github.com/kubernetes/client-go/tree/master/tools/leaderelection">client-go</a>) but <em>where</em> the coordination state lives and <em>how</em> the application integrates with it.</p><p>1️⃣ We need at first, build a unique identity that includes cluster, namespace, and pod information. In a single-cluster setup, a pod name would be usually enough. In a cross-cluster scenario on the other hand it isn’t, because without including the cluster dimension, two different clusters could produce identical identities, which would break the election semantics.</p><pre>identity := fmt.Sprintf(&quot;%s/%s/%s&quot;, cfg.ClusterName, cfg.Namespace, cfg.PodName)<br>lock := pglock.New(pool, cfg.ElectionNamespace, cfg.ElectionName, identity)</pre><p>Once identity is defined, the application initializes a new Lock. This is where the PostgreSQL-backed implementation comes in. As we already discussed before, the Lock component exposes a small set of methods Get, Create, and Update, which represent the full lifecycle of the election state.</p><p>2️⃣ The leader election loop itself is driven by the configuration defined in a structure called leaderelection.LeaderElectionConfig. This is where the behavior is defined:</p><ul><li>LeaseDuration defines how long leadership is considered valid without renewal (defaults to 15s)</li><li>RenewDeadline defines how long the current leader keeps trying to renew before giving up (defaults to 10s)</li><li>RetryPeriod defines how frequently non-leaders attempt to acquire the lock (default to 2s)</li></ul><blockquote>These three parameters are not arbitrary. They enforce a timing model where a leader must continuously prove liveness, while others compete to take over when that liveness is lost.</blockquote><blockquote>All three parameters default values as well are not arbitrary. They’re set to the default values that are used internally in Kubernetes API and is really not advised to change them.</blockquote><pre>lec := leaderelection.LeaderElectionConfig{<br>  Lock:            lock,<br>  ReleaseOnCancel: true,<br>  LeaseDuration:   cfg.LeaseDuration,<br>  RenewDeadline:   cfg.RenewDeadline,<br>  RetryPeriod:     cfg.RetryPeriod,<br>  Name:            cfg.ElectionName,<br>  Callbacks: leaderelection.LeaderCallbacks{<br>   OnStartedLeading: func(ctx context.Context) {<br>    klog.Infof(&quot;elected as leader: identity=&#39;%s&#39;&quot;, identity)<br><br>    // Do what you have to do or call a function like leaderRun<br>    leaderRun(ctx, identity)<br>   },<br>   OnStoppedLeading: func() {<br>    klog.Infof(&quot;stepping down...&quot;)<br>    //os.Exit(0)<br><br>    // Do what you have to do or call a function to perform clean-up<br>   },<br>   OnNewLeader: func(id string) {<br>    if id == identity {<br>     return<br>    }<br>    klog.Infof(&quot;current leader: identity=&#39;%s&#39;&quot;, id)<br>   },<br>  },<br> }</pre><p>3️⃣ Once a new leader elector, leaderelection.NewLeaderElector, is created and Run() is called, the flow is consistent regardless of the backend:</p><pre> elector, err := leaderelection.NewLeaderElector(lec)<br> if err != nil {<br>  klog.Error(&quot;creating new leader elector failed: %v&quot;, err)<br>  os.Exit(1)<br> }<br><br> elector.Run(ctx)</pre><p>Run() binds the current process to the provided context. As long as the context is active, the instance participates in the election. When the context is cancelled (for example on SIGTERM or SIGINT), Run() ensures that participation stops cleanly and leadership is released.</p><p>Internally, Run() starts a loop that continuously attempts (on <strong>every</strong> pod of the replica set of the workload) to acquire or renew leadership. This loop is not exposed directly but is encapsulated inside the elector. It drives calls to the resource lock (via the methods we discussed already Get, Create, Update) based on the configured timing parameters.</p><p>The sample application, or any application you will implement using pglock or any similar implementation for that matter, never calls these methods directly during election. Run() is the component that decides <em>when</em> and <em>how often</em> they are invoked based on the configuration leaderelection.LeaderElectionConfig we provided to the elector.</p><p>Run() is also responsible for invoking the registered callbacks (see above OnStartedLeading, OnStoppedLeading, OnNewLeader). It guarantees, regardless of resource lock implementation, that:</p><ul><li>OnStartedLeading runs only after leadership is successfully acquired</li><li>OnStoppedLeading is triggered when leadership is lost or the context ends</li><li>OnNewLeader is triggered when a different leader is observed</li></ul><p>4️⃣ The leader election loop as we discussed above, is driven by Run(), but the application logic is defined through <strong>callbacks</strong>. The two that matter the most, are OnStartedLeading and OnStoppedLeading. These are the points where your application pods react to changes in leadership.</p><p>OnStartedLeading is invoked <strong>when the instance has successfully acquired the lock and is now considered the leader</strong>. At this point, the election mechanism has already done the heavy lifting and there is no ambiguity about ownership. This callback is where you attach any logic that must run exclusively on that single instance that now acts as a leader.</p><p>In the example, the callback calls leaderRun(ctx, identity). That function represents the <em>leader-only</em> workload. It’s deliberately demonstrating a simple thing: once leadership is obtained, the process enters a controlled execution loop that continues as long as leadership is maintained or the parent context is cancelled:</p><pre>func leaderRun(ctx context.Context, identity string) {<br> ticker := time.NewTicker(5 * time.Second)<br> defer ticker.Stop()<br><br> klog.Infof(&quot;leader workload started: identity=&#39;%s&#39;&quot;, identity)<br><br> for {<br>  select {<br>  case &lt;-ctx.Done():<br>   klog.Infof(&quot;leader workload stopping: identity=&#39;%s&#39;&quot;, identity)<br>   return<br>  case &lt;-ticker.C:<br>   // do what you have to do here. here is your application/business logic<br>   klog.Infof(&quot;leader workload tick: identity=&#39;%s&#39;&quot;, identity)<br>  }<br> }<br>}</pre><p>OnStoppedLeading is triggered when the instance loses leadership. This can happen for several reasons: the process failed to renew the lease in time, another instance successfully updated the record, or the application context was cancelled. The important part is that this callback marks the end of exclusive ownership and you can either inform your logs or proceed to clean-up tasks depending on your application logic.</p><blockquote>This whole dance described above is the key takeaway of the election pattern. <strong>The workload is not aware of leader election directly. It simply runs while the context is active and stops when it is cancelled</strong>. Leader election controls <em>when</em> that context is valid.</blockquote><blockquote>In a real system, leaderRun() or however you want to call that method for that matter, would be replaced with something meaningful: a controller loop, a job scheduler, or a queue processor. The structure would remain the same. The only requirement is that the workload respects context cancellation so that leadership transitions remain clean and predictable.</blockquote><h3>Simulating leader election across multiple instances</h3><p>To see the behavior in practice we will let multiple instances of the sample application run simultaneously and compete to become the leader.</p><p>1️⃣ First, build the application:</p><pre>go build -o demo ./cmd/app</pre><p>2️⃣ Before starting the instances, ensure that:</p><ul><li>PostgreSQL is running and reachable</li><li>the migration has been applied</li><li>all instances use the same POSTGRES_DSN, ELECTION_NAMESPACE, and ELECTION_NAME</li></ul><p>ELECTION_NAMESPACE and ELECTION_NAME define <strong>the scope and identity of a leader election group</strong>. Together, they uniquely identify a single election within the shared PostgreSQL backend.</p><p><strong>They are not related to Kubernetes namespaces</strong> in a strict sense. In this implementation, they are simply logical identifiers used to group participants that should compete for the same leadership.</p><p>At the database level, <strong>these two values map directly to the primary key</strong> of the leader_election_locks table. This means:</p><ul><li>every unique (namespace, name) pair represents one independent election</li><li>all instances using the same pair will compete for the same leadership</li><li>instances using different values will not see or affect each other</li></ul><p>ELECTION_NAME defines <em>what</em> is being elected. It typically represents a specific workload or responsibility, such as a scheduler, controller, or background processor.</p><p>ELECTION_NAMESPACE is used to group or isolate elections. It can be used to separate environments (for example dev, staging, prod) or logical domains within a system.</p><p>A useful way to think about it is:</p><ul><li>ELECTION_NAME → the role</li><li>ELECTION_NAMESPACE → the scope</li></ul><p>For example:</p><pre>ELECTION_NAMESPACE=prod<br>ELECTION_NAME=job-runner</pre><p>All instances configured with these values will compete to become the single leader responsible for running the job runner in the production environment. If you change either value, you create a completely separate election. <strong>This allows multiple independent leader election processes to coexist in the same PostgreSQL database without interfering with each other.</strong></p><p>3️⃣ Each instance must have a unique identity. In a Kubernetes environment, this comes naturally from pod metadata. For local testing, you can simulate this by setting different pod name and namespace values.</p><p>Start <strong>three instances</strong> in separate terminals (we will leave the default value for ELECTION_NAMESPACE and ELECTION_NAME):</p><pre>POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \<br>K8S_CLUSTER_NAME=cluster-a \<br>K8S_NAMESPACE=default \<br>K8S_POD_NAME=pod-1 \<br>./demo</pre><pre>POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \<br>K8S_CLUSTER_NAME=cluster-b \<br>K8S_NAMESPACE=default \<br>K8S_POD_NAME=pod-2 \<br>./demo</pre><pre>POSTGRES_DSN=&quot;postgres://user:pass@localhost:5432/db&quot; \<br>K8S_CLUSTER_NAME=cluster-c \<br>K8S_NAMESPACE=default \<br>K8S_POD_NAME=pod-3 \<br>./demo</pre><blockquote>Replace the POSTGRES_DSN value with your own database connection string, matching your PostgreSQL user, password, host, port, and database name.</blockquote><p>4️⃣ Once all instances are running, you will observe that:</p><ul><li><strong>only one instance</strong> logs that it has become leader</li><li>the other instances report the current leader through OnNewLeader</li><li><strong>only the leader</strong> executes the periodic workload</li></ul><p>To observe a leadership change, terminate the leader process (for example with <em>Ctrl+C</em>). After the lease expires, one of the remaining instances will acquire leadership. You will observe:</p><ul><li>the previous leader stops its workload</li><li>a new instance logs that it has started leading</li><li>the workload resumes on the new leader</li></ul><p>This demonstrates that all instances are coordinating exclusively through PostgreSQL. There is no Kubernetes API involved, yet the behavior matches the standard Kubernetes coordination API leader election semantics: a single active leader, automatic fail-over, and continuous renewal of ownership.</p><h3>Conclusion</h3><p>This approach demonstrates that leader election for Kubernetes workloads does not have to be tied to the Kubernetes API. By implementing the resourcelock.Interface and using PostgreSQL as the coordination backend, <strong>we retain the proven leader election semantics</strong> from client-go while <strong>extending them beyond a single cluster boundary</strong>.</p><p>The main advantage of this design is flexibility. It enables coordination across clusters, regions, or even hybrid environments where a shared Kubernetes control plane is not available or not desirable. It also keeps the application structure unchanged. Workloads still rely on the same leader election model and callbacks, which makes the transition straightforward.</p><p>There are, however, trade-offs. PostgreSQL becomes a critical dependency for coordination. Its availability and latency directly affect leader election behavior, especially lease renewal and failover timing. This also introduces a different operational concern compared to relying on the Kubernetes API, which is already part of the cluster control plane. Additionally, this model does not introduce stronger guarantees than the original leader election mechanism. It should still be treated as a coordination tool, not as strict distributed locking with <strong>fencing</strong>.</p><blockquote><strong>Fencing</strong> is a mechanism used in distributed systems to prevent a previous leader from continuing to act after it has lost leadership.</blockquote><blockquote>In simple terms, even if a node <em>thinks</em> it is still the leader (due to network delays or pauses), fencing ensures that its actions are rejected. This is usually done by attaching a <strong>monotonically increasing token (fencing token)</strong> to each leadership term. Only the node with the latest token is allowed to perform operations.</blockquote><blockquote>Without fencing, two instances might briefly act as leader at the same time, leading to conflicting updates.</blockquote><p>PostgreSQL itself, is not a hard requirement. The same approach can be applied to <strong>any</strong> backend that can provide consistent read and conditional update semantics. The only requirement is to implement the expected interface correctly. This opens the door to using other systems such as Redis, etcd, or any data-store that can support similar concurrency patterns. The full implementation, including the PostgreSQL-backed lock and the sample application, is available here:</p><p><a href="https://github.com/akyriako/pgleele">GitHub - akyriako/pgleele</a></p><p>This repository can be used as a starting point or adapted to fit different environments and backend choices, depending on the coordination requirements.</p><p>If you found this post helpful, give it a 👏 and follow me for more insights on Kubernetes and cloud-native tech. More great content coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3548f77b0531" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/cross-cluster-leader-election-for-kubernetes-workloads-3548f77b0531">Cross-Cluster Leader Election for Kubernetes Workloads</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure a Shared Træfik Gateway on K3s]]></title>
            <link>https://itnext.io/configure-a-shared-tr%C3%A6fik-gateway-on-k3s-b09fb0d32c6e?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/b09fb0d32c6e</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes-ingress]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[traefik]]></category>
            <category><![CDATA[kubernetes-gateway]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Tue, 17 Feb 2026 10:12:58 GMT</pubDate>
            <atom:updated>2026-08-14T07:06:51.590Z</atom:updated>
            <content:encoded><![CDATA[<p>Træfik Gateway API on K3s: Shared Gateway, TLS and Helm caveats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OI3NjHJmIsc5XHJz" /></figure><h3>TL;DR</h3><p>Ingress NGINX Controller for Kubernetes is retired on March 2026 and Kubernetes Ingress API will receive no future development<strong>. </strong>Configure Traefik as a shared Gateway in <a href="https://k3s.io/">K3s</a>.</p><h3>Introduction</h3><p>The Kubernetes <a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/"><strong>Ingress API</strong></a> is feature-frozen and will not developed any further. New traffic management capabilities are being developed in the Kubernetes <a href="https://kubernetes.io/docs/concepts/services-networking/gateway/"><strong>Gateway API</strong></a>, which provides a more expressive and extensible model for service networking, but is still lacking the feature parity with the Ingress API. As stated in Kubernetes official documentation:</p><blockquote><strong>“The Kubernetes project recommends using </strong><a href="https://gateway-api.sigs.k8s.io/"><strong>Gateway</strong></a><strong> instead of </strong><a href="https://kubernetes.io/docs/concepts/services-networking/ingress/"><strong>Ingress</strong></a>. The Ingress API has been frozen. This means that:</blockquote><blockquote>- The Ingress API is generally available, and is subject to the <a href="https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api">stability guarantees</a> for generally available APIs. The Kubernetes project has no plans to remove Ingress from Kubernetes.</blockquote><blockquote>- The Ingress API is no longer being developed, and will have no further changes or updates made to it.”</blockquote><p>In the meantime, the <a href="https://github.com/kubernetes/ingress-nginx/"><strong>ingress-nginx</strong></a> project (Ingress NGINX Controller for Kubernetes) <a href="https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/">retires on March 2026</a>, forcing half of the Kubernetes universe racing against time to migrate to a viable alternative.</p><p>Ingress and Gateway APIs <strong>can coexist</strong> in the same cluster. Migration does not need to be immediate or global. Many environments adopt Gateway API incrementally by introducing <strong>shared Gateways</strong> for new workloads while continuing to run existing services behind Ingress.</p><p>Using a <strong>shared Gateway</strong> has several advantages. It allows a single, centrally managed entry point to serve multiple applications, reduces duplication of configuration, and limits the permissions required by application teams. Applications only need to create route resources in their own namespaces, while certificates, listeners, and exposure policies remain under the control of the Gateway owner.</p><h3>Gateway API Components</h3><p>The Gateway API defines a set of resources that separate <strong>infrastructure concerns</strong> from <strong>application routing concerns</strong>. Each resource (4 stable at the time of writing) has a clear responsibility and scope:</p><h4>GatewayClass</h4><p>A GatewayClass represents a type of Gateway implementation (exactly as the IngressClass for the Ingress API). It is defined by the infrastructure or platform layer and is typically installed by a Gateway controller (for example, Traefik, Envoy Gateway, Kong and <a href="https://gateway-api.sigs.k8s.io/implementations/#gateway-controller-implementation-status">many others</a>).</p><blockquote>Application teams do not interact with GatewayClass directly.</blockquote><h4>Gateway</h4><p>A Gateway represents a network entry point into the cluster. It defines where and how traffic enters the system (per cluster if shared or per application if dedicated).</p><blockquote>Gateways are usually managed by platform or infrastructure teams and are often shared across multiple applications and namespaces.</blockquote><h4>Routes</h4><p>Routes define <strong>application-level routing</strong> to backend services and attach to a Gateway. HTTPRoute is the most common route type and is used for good ole’ HTTP and HTTPS traffic. GRPCRoute on the other hand, defines gRPC-specific rules for routing traffic from a Gateway to backend network endpoints.</p><blockquote>Routes are typically created and managed by application teams in their own namespaces.</blockquote><h3>Configure K3s Helm Charts Manifests</h3><p><a href="https://k3s.io/">K3s</a>, in its default configuration, comes pre-configured with Traefik and this where we are going to focus on. Traefik Gateway CRDs are installed but no GatewayClass or Gateway are enabled or deployed. K3s is bootstrapped with a Traefik Ingress.</p><p>K3s uses a built-in mechanism to deploy and manage core cluster components using Helm charts. This mechanism is implemented through two custom resources: HelmChart and HelmChartConfig. Together, they allow K3s to install, configure, and upgrade packaged applications during cluster startup and runtime.</p><p>This system is primarily used to bootstrap components such as Traefik, CoreDNS, metrics-server, and other optional add-ons. The presence of a HelmChart resource signals the K3s Helm controller to install or upgrade the corresponding Helm release and a HelmChartConfig resource provides user-defined overrides for a corresponding HelmChart. K3s supports placing manifests under:</p><pre>/var/lib/rancher/k3s/server/manifests/</pre><p>This mechanism allows cluster configuration and enables automatic upfront bootstrapping and (re)configuration without the need of manual Helm commands. To configure Traefik, we need to update the traefik-config.yaml as following:</p><ul><li>Enable the kubernetesGateway under providers.</li><li>Add a new stanza gateway.listeners. We need to add two listeners, one for HTTP and one HTTPS traffic and point them to the internal ports of the Traefik container, namely 8000 and 8443 respectively.</li><li>Each listener should explicitly set namespacePolicy.from to All. Without this configuration, the value defaults to Same, restricting Gateway access to workloads within the same namespace only.</li><li>For the HTTPS listener set mode to Terminate. With this knob TLS connections are terminated at the Gateway. TLS handling is centralized at the Gateway and like this we avoid the need for individual applications to manage certificates or serve HTTPS themselves.</li><li>When using protocol: HTTPS with mode: Terminate, the Gateway <strong>must</strong> reference <strong>at least one TLS certificate</strong>. Without a certificate, the Gateway controller cannot program the listener. Certificates are referenced using the stanzacertificateRefs, which point to Kubernetes Secret objects of type kubernetes.io/tls. The Secret must exist in the same namespace as the Gateway, contain a valid certificate and private key and of course match the hostnames served by the listener.</li></ul><blockquote>In a shared Gateway setup, certificate management is typically handled by the platform or infrastructure layer.</blockquote><pre>apiVersion: helm.cattle.io/v1<br>kind: HelmChartConfig<br>metadata:<br>  name: traefik<br>  namespace: kube-system<br>spec:<br>  valuesContent: |-<br>    providers:<br>      kubernetesIngress:<br>        enabled: true<br>      kubernetesGateway:<br>        enabled: true<br>    gateway:<br>      listeners:<br>        web:<br>          port: 8000<br>          protocol: HTTP<br>          namespacePolicy:<br>            from: All<br>        websecure:<br>          port: 8443<br>          protocol: HTTPS<br>          mode: Terminate<br>          certificateRefs:<br>            - kind: Secret<br>              name: wildcard-cert-tls<br>          namespacePolicy:<br>            from: All</pre><p>Create the wildcard-cert-tls (see next paragraph) and <strong>then</strong> restart the K3s service to pick up the changes. It should provision a GatewayClass and a Gateway in namespace <em>kube-system</em>.</p><pre>sudo systemctl restart k3s</pre><blockquote>Assuming you are running a Debian-family operating system; otherwise, adjust the command above accordingly.</blockquote><h3>Create a Wildcard Certificate</h3><p>A wildcard certificate allows a single certificate to cover multiple hostnames under the same domain. For example, a certificate for *.apps.mylab.home can be used for serving multiple applications e.g.:</p><ul><li>whoami.apps.mylab.home</li><li>grafana.apps.mylab.home</li><li>any other subdomain under apps.mylab.home</li></ul><p>This is useful in shared Gateway deployments, where multiple applications attach routes to the same HTTPS listener.</p><p>Let’s lay the create a wildcard certificate for our private published services</p><p>1️⃣ Create a file <strong>wildcard.cnf</strong> and paste the following:</p><pre>[req]<br>default_bits       = 4096<br>prompt             = no<br>default_md         = sha256<br>distinguished_name = dn<br>x509_extensions    = v3_req<br><br>[dn]<br>CN = apps.mylab.home<br><br>[v3_req]<br>subjectAltName = @alt_names<br>keyUsage = digitalSignature, keyEncipherment<br>extendedKeyUsage = serverAuth<br><br>[alt_names]<br>DNS.1 = apps.mylab.home<br>DNS.2 = *.apps.mylab.home</pre><blockquote><em>Replace </em><em>CN,</em><em>DNS.1 and </em><em>DNS.2 with the values specific to your environment.</em></blockquote><p>2️⃣ Create the Certificate</p><pre>openssl req -x509 -nodes -days 825 \<br>  -newkey rsa:4096 \<br>  -keyout tls.key \<br>  -out tls.crt \<br>  -config wildcard.cnf</pre><p>3️⃣ Create the Kubernetes Secret</p><pre>kubectl -n kube-system create secret tls wildcard-cert-tls \<br>  --cert=tls.crt \<br>  --key=tls.key</pre><p>4️⃣ Create DNS Records</p><p>You have to make sure that the external IPs that are assigned to your Gateway/Ingress (the node IPs in case of a vanilla K3s) are pointing, with or without a floating IP, to *.apps.mylab.home. Adjust your DNS records accordingly.</p><p>Once both Gateway and Secret are configured, the Gateway can serve HTTPS for any route whose hostname matches the certificate.</p><blockquote>If on the other hand you are planning to expose services to the public, TLS certificates should be issued by a publicly trusted certificate authority (like Let’s Encrypt for instance). In Kubernetes environments, this is commonly handled using <a href="https://cert-manager.io/"><strong>cert-manager</strong></a> with a DNS-based ACME challenge (<a href="https://cert-manager.io/docs/configuration/acme/dns01/">DNS-01</a>). DNS validation is required for wildcard certificates and/or for clusters that are not directly reachable over HTTP.</blockquote><blockquote>In this setup, cert-manager is responsible for obtaining and renewing the certificate, while the resulting TLS Secret is referenced by the Gateway listener. The DNS provider configuration (DNS solver) must be able to create and remove validation records for the target domain.</blockquote><blockquote>This approach allows certificate management to remain automated and independent of application routing configuration.</blockquote><h3>Create a Workload</h3><p>We are going to test the whole setting now with deploying and exposing <a href="https://github.com/traefik/whoami">traefik/whoami</a> internally.</p><p>1️⃣ Create and deploy the manifests for the application itself</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: whoami<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: whoami<br>  namespace: whoami<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: whoami<br>  template:<br>    metadata:<br>      labels:<br>        app: whoami<br>    spec:<br>      containers:<br>        - name: whoami<br>          image: traefik/whoami:v1.11<br>          ports:<br>            - name: http<br>              containerPort: 80<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: whoami<br>  namespace: whoami<br>spec:<br>  selector:<br>    app: whoami<br>  ports:<br>    - name: http<br>      port: 80<br>      targetPort: http</pre><p>2️⃣ Create and deploy the manifests for the Route</p><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: whoami<br>  namespace: whoami<br>spec:<br>  parentRefs:<br>    - name: traefik-gateway<br>      namespace: kube-system<br>  hostnames:<br>    - whoami.apps.mylab.home<br>  rules:<br>    - matches:<br>        - path:<br>            type: PathPrefix<br>            value: /<br>      backendRefs:<br>        - name: whoami<br>          port: 80</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o7Z5JS3Me1EjpgxND6lwNQ.png" /><figcaption>Of course the Certificate is not trusted as it is a self-signed one.</figcaption></figure><h3>Closing Thoughts</h3><p>Retiring <a href="https://github.com/kubernetes/ingress-nginx"><strong>kubernetes/ingress-nginx</strong></a> sets a genuinely bad precedent for the ecosystem because it shows that even <em>foundational</em> “it’s everywhere” components can become unsustainable and then get shut down without a fully community-owned succession plan.</p><p>There isn’t a single definitive estimate of the blast radius of this event, but the Kubernetes Steering Committee itself in a statement calls it “<em>the piece of critical infrastructure for about </em><strong><em>half</em></strong><em> of cloud native environments.</em>”. You get the picture right?</p><p><strong>So act now, before you need to (re)act under pressure after March 2026</strong>.</p><p>As long as Ingress API is getting no new features, sooner or later you’ll need considering moving to Gateway API (unless you have a strong technical reason to do otherewise).</p><blockquote>⚠<em>️</em> <strong>This article does not advocate replacing the Kubernetes Ingress NGINX Controller with Traefik Gateway. It simply explains how to configure a K3s cluster to use a shared Gateway, leaving the choice of controller to your specific needs.</strong></blockquote><p>If you found this post helpful, give it a 👏 and follow me for more insights on Kubernetes and cloud-native tech. More great content coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b09fb0d32c6e" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/configure-a-shared-tr%C3%A6fik-gateway-on-k3s-b09fb0d32c6e">Configure a Shared Træfik Gateway on K3s</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own Managed Kubernetes Service on Proxmox with CAPI and Kamaji]]></title>
            <link>https://itnext.io/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-and-kamaji-7858ad41cb09?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/7858ad41cb09</guid>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[kamaji]]></category>
            <category><![CDATA[proxmox]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Thu, 18 Dec 2025 10:48:30 GMT</pubDate>
            <atom:updated>2025-12-22T09:18:26.318Z</atom:updated>
            <content:encoded><![CDATA[<p>Abstract Kubernetes control-plane complexity with Kamaji by managing master nodes as Pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IXE3_BEejXu4zP0PNxdNgg.png" /></figure><blockquote>If you are new to <a href="https://github.com/kubernetes-sigs/cluster-api">Cluster API (CAPI)</a>, please refer to my previous article, <a href="https://medium.com/itnext/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-8d9786644818">Build your own Managed Kubernetes Service on Proxmox with CAPI</a>, for a detailed introduction.</blockquote><h3>Prerequisites</h3><p>For this lab you need to bring with:</p><ol><li>a single-node or even better a multi-node <a href="https://www.proxmox.com/">Proxmox</a> cluster. <em>Installing and configuring the Proxmox cluster itself is </em><strong><em>out</em></strong><em> of the scope of this article</em>.</li><li>Relevant hands-on experience on Proxmox and Kubernetes.</li><li>Relevant hands-on experience with CAPI.</li><li>A workstation with homebrew, kubectl, <a href="https://cluster-api.sigs.k8s.io/clusterctl/overview.html">clusterctl</a>.</li><li>A virtual machine with <a href="https://k3s.io/">k3s</a> (4 vCPUs, 6GB RAM no ballooning) for implementing the Kamaji part of the lab (In this case, the management cluster cannot be illustrated using <a href="https://kind.sigs.k8s.io/">kind</a>, unlike in the previous article.</li></ol><blockquote><em>⚠️</em> This lab requires Proxmox version <em>8.4.12</em>. Earlier versions exhibit issues with the integration with the <a href="https://github.com/ionos-cloud/cluster-api-provider-proxmox">CAPMOX</a> provider, and later versions have not been tested by this guide.</blockquote><p>In this lab we are going to demonstrate how you could create your own Managed Kubernetes Service using CAPI with Proxmox as infrastructure provider, <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/">kubeadm</a> as bootstrap provider and for the role of control plane provider we are going to use <a href="https://kamaji.clastix.io/">Kamaji</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*NVKLUogNeKy8SOT2Hx7EBg.png" /></figure><h3>What is Kamaji?</h3><p><a href="https://github.com/clastix/kamaji">Kamaji</a> is an open-source project developed by <a href="https://clastix.io/">Clastix</a> that implements the Kubernetes control plane as a hosted service running entirely on Kubernetes itself. Instead of provisioning dedicated virtual machines or bare-metal servers for etcd, kube-apiserver, the controller manager, and the scheduler, Kamaji runs these components as Pods within a management cluster and exposes them to tenant clusters through a lightweight API. This hosted control plane model decouples control plane lifecycle management from the underlying infrastructure, significantly reducing operational overhead and resource duplication.</p><p>Under the hood, Kamaji introduces two new API groups. The TenantControlPlane resource, which defines the desired configuration for a control plane: Kubernetes version, certificate settings, replica count, scheduling constraints and exposure method (ClusterIP, LoadBalancer or NodePort). Separately, a DataStore resource manages the backing store for one or more tenant control planes, which is by default <a href="https://etcd.io/">etcd</a>, but optionally other better scaling alternatives such as MySQL, PostgreSQL or NATS (via Kine) can used as a replacement.</p><h3>Preparing your Proxmox Users and API Tokens</h3><blockquote><em>❗️</em>You can skip this section if you have already completed the setup by following the steps outlined in the previous article.</blockquote><p>Before we can build images, the image-builder needs permission to interact with your Proxmox environment through its API. This access allows it to launch a virtual machine, perform the build process, and then turn that VM into a reusable template. Execute the following command in one of your Proxmox nodes:</p><pre>pveum user add image-builder@pve<br>pveum aclmod / -user image-builder@pve -role PVEAdmin<br>pveum user token add image-builder@pve capi -privsep 0<br>pveum user add capmox@pve<br>pveum aclmod / -user capmox@pve -role PVEAdmin<br>pveum user token add capmox@pve capi -privsep 0</pre><h3>Building Images for Proxmox with image-builder</h3><blockquote><em>‼️</em>Given that this is a quite lengthy configuration step, and to avoid cluttering the article or making it unnecessarily long, please refer to the corresponding section in the previous article, if you have not already completed it. Otherwise, you can safely skip this step.</blockquote><h3>Installing Prerequisites</h3><blockquote>To maintain a single guide for both macOS and Linux, we’ll use <a href="https://brew.sh/">Homebrew</a> to install the binaries required for managing the cluster.</blockquote><p>1️⃣ Install Homebrew</p><pre>/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&quot;</pre><p>Follow the instructions of the installer, to complete the installation.</p><p>2️⃣ Install kubectl</p><pre>brew install kubectl</pre><p>3️⃣ Install clusterctl</p><pre>brew install clusterctl</pre><blockquote><em>⚠️</em> This lab requires clusterctl version v1.11.3 or later.</blockquote><p>The clusterctl CLI delivers a streamlined “day one” experience for CAPI by automatically retrieving and installing the necessary provider manifests. It embeds recommended best practices for provider management, helping you avoid mis-configurations and simplifying tasks like upgrades.</p><p>Built on top of clusterctl, the Cluster API Operator is a Kubernetes operator that brings a fully declarative workflow for managing provider lifecycles within your management cluster. It enhances the deployment and day-to-day operation of Cluster API, making routine tasks and GitOps-driven automation effortless.</p><h3>Create Configuration for Proxmox</h3><p>Create a a file named capmox.env, and fill it in with the following content:</p><pre>export PROXMOX_URL=&quot;https://&lt;PROXMOX_NODE_IP&gt;:8006&quot;<br>export PROXMOX_TOKEN=&#39;capmox@pve!capi&#39;<br>export PROXMOX_SECRET=&quot;&lt;CAPMOX_API_TOKEN_SECRET&gt;&quot;                          <br>export PROXMOX_SKIP_TLS_VERIFY=&quot;true&quot;</pre><blockquote>Replace every occurrence of &lt;PROXMOX_NODE_IP&gt; with the IP address and CIDR corresponding to your bridge interface (vmbr0, for this lab).</blockquote><blockquote>Do the same for &lt;CAPMOX_API_TOKEN_SECRET&gt;, using the appropriate value for your environment.</blockquote><h3>Setting up a Management Cluster</h3><p>This is the cluster that runs one or more Infrastructure Providers and holds resources (for example, Machines), when provisioning multiple workload clusters.</p><p>⏩ <strong>SSH into the virtual machine that will host the K3s cluster.</strong></p><p>1️⃣ Create the cluster</p><pre>curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;--disable traefik --disable servicelb&quot; sh -<br><br>sudo chmod 644 /etc/rancher/k3s/k3s.yaml<br>sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config<br>sudo chown &lt;USER&gt;:&lt;USER&gt; ~/.kube/config</pre><blockquote>Replace &lt;USER&gt; with the appropriate user and group for your environment.</blockquote><p>If the management cluster is running on a different virtual machine, copy ~/.kube/config to your bootstrap machine where the required prerequisites (Homebrew, kubectl, and clusterctl) are already installed.</p><blockquote>Make sure you change the server value from <a href="https://127.0.0.1:6443/">https://127.0.0.1:6443</a> to the IP address assigned on the network device that uses the bridge vmbr0.</blockquote><p>⏩ <strong>Switch now to your bootstrap machine.</strong></p><p>2️⃣ Install Cert-Manager</p><p>Kamaji relies on Kubernetes dynamic admission control mechanisms, including validating and mutating webhook configurations. These webhooks communicate over TLS, with certificates managed by <a href="https://github.com/cert-manager/cert-manager">cert-manager</a>, which therefore must be installed as a prerequisite.</p><pre>helm repo add jetstack https://charts.jetstack.io<br>helm repo update<br><br>helm install \<br>  cert-manager jetstack/cert-manager \<br>  --namespace cert-manager \<br>  --create-namespace \<br>  --set installCRDs=true</pre><p>3️⃣ Install the Kamaji Operator</p><pre>helm repo add clastix https://clastix.github.io/charts<br>helm repo update<br><br>helm install kamaji clastix/kamaji \<br>    --version 0.0.0+latest \<br>    --namespace kamaji-system \<br>    --create-namespace \<br>    --set image.tag=latest</pre><p>4️⃣ Install MetalLB</p><blockquote><em>⚠️ </em>Kamaji provisions the control plane as a set of Pods exposed through a Service. This Service must be assigned an IP address within the network defined by the vmbr0 interface so that it is reachable by worker nodes and by consumers of the cluster. For this reason, a solution such as <a href="https://github.com/metallb/metallb">MetalLB</a> is required. Other alternatives may also be suitable; e.g., <a href="https://github.com/kube-vip/kube-vip">kube-vip</a> or <a href="https://github.com/k3s-io/klipper-lb">klipper</a> the embedded K3s service load balancer which in our case would mean you need to reinstall the cluster, since it was provisioned <strong>without</strong> Traefik and Klipper.</blockquote><pre>kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.3/config/manifests/metallb-native.yaml<br><br>kubectl apply -f - &lt;&lt;EOF<br>apiVersion: metallb.io/v1beta1<br>kind: IPAddressPool<br>metadata:<br>  name: default-pool<br>  namespace: metallb-system<br>spec:<br>  addresses:<br>  - 192.168.1.240-192.168.1.250<br>---<br>apiVersion: metallb.io/v1beta1<br>kind: L2Advertisement<br>metadata:<br>  name: default-advertisement<br>  namespace: metallb-system<br>spec:<br>  ipAddressPools:<br>    - default-pool<br>EOF</pre><blockquote>️Replace the IP address pool range, in this case 192.168.1.240–192.168.1.250, so it matches your environment. Ensure that the selected IP range belongs to the vmbr0 interface network.</blockquote><p>5️⃣ Transform the cluster to a Management Cluster</p><p>With clusterctl installed and prerequisites satisfied, run clusterctl init to convert your Kubernetes cluster into a management cluster.</p><pre>source capmox.env<br><br>clusterctl init \<br>  --infrastructure proxmox \ <br>  --ipam in-cluster \ <br>  --control-plane kamaji</pre><blockquote>The command above will install <a href="https://github.com/ionos-cloud/cluster-api-provider-proxmox/">Cluster API Provider for Proxmox VE</a> (CAPMOX) as infrastructure provider, Kubeadm as bootstrap provider and Kamaji as control-plane provider.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_UHLimBONSml08hPYZ9ooA.png" /></figure><h3>Creating a Cluster with the Kamaji CP Provider</h3><p>When you use a KamajiControlPlane alongside your Cluster and Machine objects, the Kamaji control-plane provider watches for that resource and <strong>spins up the Kubernetes control-plane components as Pods in your management cluster</strong>. It provisions etcd, generates and rotates certificates, and configures the API server, controller-manager and scheduler according to the specs in the KamajiControlPlane. As Pod replicas become ready, Kamaji wires them into a highly available endpoint and exposes it (ClusterIP, LoadBalancer or NodePort), while your worker nodes simply run kubeadm join against that hosted control plane, giving you <strong>a fully managed, multi-tenant control plane without requiring dedicated VMs</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b28xcAXaDkqqrExisy9B1w.png" /><figcaption>Multiple Control Planes <strong>are running as Kubernetes Pods</strong> while their Worker Nodes <strong>are Proxmox VMs</strong>.</figcaption></figure><p>1️⃣ Create Configuration for Kamaji</p><p>Before creating a tenant Kubernetes cluster, some configuration is required on the management cluster for Kamaji. Create a a file named kamaji.env, and fill it in with the following content:</p><pre># Cluster Configuration<br>export CLUSTER_NAME=&quot;capi-kamaji-quickstart&quot;<br>export CLUSTER_NAMESPACE=&quot;default&quot;<br>export CONTROL_PLANE_REPLICAS=3<br>export KUBERNETES_VERSION=&quot;v1.33.1&quot;<br>export CLUSTER_DATASTORE=&quot;default&quot;<br><br># Networking Configuration<br>export IP_RANGE=&#39;[&quot;192.168.1.230-192.168.1.239&quot;]&#39;<br>export IP_PREFIX=24<br>export GATEWAY=&quot;192.168.1.1&quot;<br>export DNS_SERVERS=&#39;[&quot;192.168.1.1&quot;,&quot;8.8.8.8&quot;]&#39;<br>export NETWORK_BRIDGE=&quot;vmbr0&quot;<br>export NETWORK_MODEL=&quot;virtio&quot;<br><br># Node Configuration<br>export SSH_USER=&quot;&lt;USER&gt;&quot;<br>export SSH_AUTHORIZED_KEY=&quot;ssh-rsa AAAAB...&quot;<br>export NODE_REPLICAS=2<br>export NODE_LABELS=<br>export NODE_TAINTS=<br><br># Resource Configuration<br>export SOURCE_NODE=&quot;pve-01&quot;<br>export TEMPLATE_ID=101<br>export ALLOWED_NODES=&#39;[&quot;pve-01&quot;, &quot;pve-02&quot;]&#39;<br>export MEMORY_MIB=&quot;2048&quot;                                            <br>export NUM_CORES=2<br>export NUM_SOCKETS=1<br>export BOOT_VOLUME_DEVICE=&quot;scsi0&quot;<br>export BOOT_VOLUME_SIZE=20<br>export FILE_STORAGE_FORMAT=&quot;raw&quot;<br>export STORAGE_NODE=&quot;local-lvm&quot;<br><br>export CLOUD_INIT_CONFIG=&quot;#cloud-config package_update=true packages=- net-tools&quot;</pre><blockquote>Replace IP_RANGE, GATEWAY, DNS_SERVERS, NETWORK_BRIDGE, SSH_USER, SSH_AUTHORIZED_KEY, SOURCE_NODE, TEMPLATE_ID, ALLOWED_NODES values and any other variable for that matter, so they match <strong>your</strong> environment.</blockquote><p>2️⃣ Generate the manifests</p><p>Download the <a href="https://raw.githubusercontent.com/clastix/cluster-api-control-plane-provider-kamaji/master/templates/proxmox/capi-kamaji-proxmox-template.yaml">capi-kamaji-proxmox-template.yaml</a> template file and save it in your working directory, then use the following commands to generate a cluster manifest from this template.</p><pre>source kamaji.dev <br><br>clusterctl generate cluster $CLUSTER_NAME \<br>    --from capi-kamaji-proxmox-template.yaml \<br>    &gt; capi-kamaji-proxmox-cluster.yaml</pre><p>3️⃣ Allow CAPMOX to over-commit memory ⚠️</p><p>When the infrastructure controller provisions a new VM on Proxmox, it first sums the RAM already allocated on the target host to determine if your MachineTemplate will fit. If you haven’t explicitly enabled memory over-commit, this check will refuse to schedule any VM once the sum of allocated RAM reaches the node’s nominal capacity, causing new VMs to hang even though Proxmox could safely oversubscribe memory in practice.</p><p>For that matter, add the following properties to the ProxmoxCluster resource:</p><pre>schedulerHints:<br>    memoryAdjustment: 0</pre><p>For example:</p><pre>apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1<br>kind: ProxmoxCluster<br>metadata:<br>  labels:<br>    cluster.x-k8s.io/cluster-name: tenant-01-kamaji<br>  name: tenant-01-kamaji<br>  namespace: default<br>spec:<br>  ...<br>  dnsServers:<br>    - 192.168.1.1<br>    - 8.8.8.8<br>  ...<br>  schedulerHints:<br>    memoryAdjustment: 0</pre><p>4️⃣ Create the cluster</p><p>Execute the command:</p><pre>kubectl apply -f capi-kamaji-proxmox-cluster.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQUdKh2e0SMnMAxKEmbYqw.png" /><figcaption>The worker nodes are being provisioned in Proxmox.</figcaption></figure><p>This process will take a few minutes. Once it completes, retrieve the <strong>kubeconfig</strong> file for the newly created cluster by running:</p><pre>clusterctl get kubeconfig capi-kamaji-quickstart &gt; capi-kamaji-quickstart.kubeconfig</pre><p>5️⃣ Deploy a CNI</p><p>For demonstration purposes, <a href="https://docs.tigera.io/calico/latest/about/">Calico</a> will be deployed as the CNI for the newly created cluster. In your lab, you may choose a different CNI if preferred. Execute the following command:</p><pre>kubectl --kubeconfig=./capi-kamaji-quickstart.kubeconfig \<br>  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml</pre><blockquote><em>When issues arise, you can access any node directly by opening an SSH session as the </em>root<em> user. This allows you to inspect system logs, review configuration files, and run diagnostic commands on each machine.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YqIoiM7m9Jr8_ZJgcreIA.png" /><figcaption>The workload cluster is now ready for use!</figcaption></figure><p>6️⃣ Install Kamaji Console (Optional)</p><p>The <a href="https://kamaji.clastix.io/guides/console/"><strong>Kamaji Console</strong></a> is a web-based user interface provided by the Kamaji project to observe and manage hosted Kubernetes control planes managed by Kamaji. There is limited functionality available in the free version, but it is included here for completeness.</p><p>Before installing the console, you must provision the required Secret:</p><pre>cat &lt;&lt;EOF | kubectl apply -f -<br>apiVersion: v1<br>kind: Secret<br>type: Opaque<br>metadata:<br>  name: kamaji-console<br>  namespace: kamaji-system<br>data:<br>  ADMIN_EMAIL: &lt;ADMIN_EMAIL&gt;<br>  ADMIN_PASSWORD: &lt;ADMIN_PASSWORD&gt;<br>  JWT_SECRET: &lt;JWT_SECRET&gt;<br>  NEXTAUTH_URL: &lt;NEXTAUTH_URL&gt;<br>EOF</pre><blockquote>Replace all values of the variables under the data stanza so they match <strong>your</strong> environment.</blockquote><p>And then install the console via its Helm Chart:</p><pre>helm repo add clastix https://clastix.github.io/charts<br>helm repo update<br><br>helm install console clastix/kamaji-console -n kamaji-system</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjBmXp31pTxo5EM90KGYlQ.png" /></figure><h3>Next Steps</h3><p>In the next article, we will explore how to use Kamaji together with tools such as <a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> and <a href="https://sveltos.projectsveltos.io/">Sveltos</a> to take advantage of the automation capabilities provided by GitOps.</p><p>If you found this post helpful, give it a 👏 and follow me for more insights on Kubernetes and cloud-native tech. More great content coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7858ad41cb09" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-and-kamaji-7858ad41cb09">Build your own Managed Kubernetes Service on Proxmox with CAPI and Kamaji</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aggregate Open Telekom Cloud CCE Logs with Fluent Bit & Grafana Loki]]></title>
            <link>https://itnext.io/aggregate-open-telekom-cloud-cce-logs-with-fluent-bit-grafana-loki-6aa98c8792da?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa98c8792da</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[fluentbit]]></category>
            <category><![CDATA[open-telekom-cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[logs]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 11:11:24 GMT</pubDate>
            <atom:updated>2026-06-29T12:49:38.267Z</atom:updated>
            <content:encoded><![CDATA[<p>Consolidate all your Open Telekom Cloud Container Engine logs in a intuitive Grafana dashboard using Loki and Fluent-Bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTwEnIpWtT9Tspy8lN4lWA.png" /></figure><p>This blueprint explains how to collect and centralize logs from Cloud Container Engine (CCE) workloads using <a href="https://fluentbit.io/">Fluent Bit</a> and <a href="https://grafana.com/oss/loki/">Grafana Loki</a>. It covers the end-to-end setup for forwarding logs from CCE nodes and containers to a centralized Loki instance, enabling efficient querying, visualization, and troubleshooting through Grafana dashboards. The goal is to provide a consistent and scalable logging approach for Kubernetes environments running on Open Telekom Cloud.</p><p><a href="https://fluentbit.io/">Fluent Bit</a> is a lightweight, high-performance log and metrics processor designed for cloud and containerized environments. It collects data from various sources, processes and filters it, and then forwards it to multiple destinations such as Loki, Elasticsearch, or cloud logging services. Built with minimal resource consumption in mind, Fluent Bit integrates seamlessly with Kubernetes through DaemonSets, making it an efficient choice for aggregating logs from all nodes and containers in a cluster.</p><h3>Installing Grafana Loki</h3><p>If you don’t already have a Grafana Loki instance running, you can set it up first before proceeding with log aggregation. The installation process is covered in detail in the companion blueprint <a href="https://arch.otc-service.com/docs/blueprints/by-use-case/observability/deploy-grafana-loki-on-cce">Deploy Grafana Loki on CCE</a>, which explains how to deploy Loki in microservices mode on Cloud Container Engine (CCE) with Open Telekom Cloud Object Storage (OBS) as the backend. Once Loki is up and running, you can continue here to install and configure Fluent-Bit and start collecting and centralizing logs from your CCE workloads.</p><h3>Installing Fluent Bit as Log Forwarder</h3><p>Fluent Bit, as already mentioned, acts as the central log collector in the cluster, running as a DaemonSet, collecting logs from every node and container in the CCE cluster, parsing and enriching them with Kubernetes metadata, and finally forwarding them to Grafana Loki.</p><p>Create a values file called <strong>values-fluentbit.yaml</strong>:</p><pre>elasticsearch:<br>  enabled: false<br><br>opensearch:<br>  enabled: false<br><br>kind: DaemonSet<br><br>config:<br>  service: |<br>    [SERVICE]<br>        Daemon Off<br>        Flush 1<br>        Log_Level info<br>        Parsers_File /fluent-bit/etc/parsers.conf<br>        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf<br>        HTTP_Server On<br>        HTTP_Listen 0.0.0.0<br>        HTTP_Port 2020<br>        Health_Check On<br><br>  inputs: |<br>    [INPUT]<br>        Name tail<br>        Path /var/log/containers/*.log<br>        Exclude_Path /var/log/containers/*fluent-bit*.log<br>        multiline.parser docker, cri<br>        Tag kube.*<br>        Mem_Buf_Limit 50MB<br>        Skip_Long_Lines On<br>        Refresh_Interval 10<br><br>  filters: |<br>    [FILTER]<br>        Name kubernetes<br>        Match kube.*<br>        Kube_URL https://kubernetes.default.svc:443<br>        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt<br>        Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token<br>        Kube_Tag_Prefix kube.var.log.containers.<br>        Merge_Log On<br>        Keep_Log On<br>        K8S-Logging.Parser On<br>        K8S-Logging.Exclude On<br>        Labels Off<br>        Annotations Off<br><br>    [FILTER]<br>        Name modify<br>        Match kube.*<br>        Remove stream<br>        Remove time<br>        Remove kubernetes.container_hash<br>        Remove kubernetes.pod_id<br><br>  outputs: |<br>    [OUTPUT]<br>        Name loki<br>        Match kube.*<br>        Host loki-gateway.monitoring.svc.cluster.local<br>        Port 80<br>        uri /loki/api/v1/push<br>        labels job=fluent-bit, namespace=$kubernetes[&#39;namespace_name&#39;], pod=$kubernetes[&#39;pod_name&#39;], container=$kubernetes[&#39;container_name&#39;]<br>        remove_keys kubernetes,host,container_id,_p<br>        line_format key_value<br>        drop_single_key raw<br>        auto_kubernetes_labels off<br><br>  customParsers: |<br>    [PARSER]<br>        Name docker_no_time<br>        Format json<br>        Time_Keep Off<br>        Time_Key time<br>        Time_Format %Y-%m-%dT%H:%M:%S.%L<br><br>resources:<br>  requests:<br>    cpu: 100m<br>    memory: 128Mi<br>  limits:<br>    cpu: 500m<br>    memory: 256Mi<br><br>tolerations:<br>  - operator: Exists<br>    effect: NoSchedule<br>  - operator: Exists<br>    effect: NoExecute<br><br>securityContext:<br>  runAsNonRoot: true<br>  runAsUser: 1000<br>  readOnlyRootFilesystem: true<br>  allowPrivilegeEscalation: false<br>  capabilities:<br>    drop:<br>      - ALL<br><br>serviceMonitor:<br>  enabled: false<br><br>hostNetwork: false<br>dnsPolicy: ClusterFirst</pre><h3>Installing Grafana (Optional)</h3><p>For verification, we’ll install Grafana with Helm and include a simple demo dashboard to confirm that logs are reaching Loki. This lightweight setup is only for validation and smoke testing. For most production clusters, the recommended approach is to deploy the <a href="https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md">kube-prometheus-stack</a> bundle, which provides Grafana together with Prometheus, Alertmanager, and curated dashboards, offering tighter integration and easier ongoing operations.</p><p>Create a values file called <strong>values-grafana.yaml</strong>:</p><pre>adminUser: admin<br><br>persistence:<br>  enabled: true<br>  size: 5Gi<br>  accessModes:<br>    - ReadWriteOnce<br>  storageClassName: &quot;csi-disk&quot;<br><br>service:<br>  enabled: true<br>  type: ClusterIP<br>  port: 80<br><br>resources:<br>  requests:<br>    cpu: 100m<br>    memory: 256Mi<br>  limits:<br>    cpu: 500m<br>    memory: 512Mi<br><br>datasources:<br>  datasources.yaml:<br>    apiVersion: 1<br>    datasources:<br>      - name: Loki<br>        type: loki<br>        access: proxy<br>        url: http://loki-gateway.monitoring.svc.cluster.local<br>        jsonData:<br>          maxLines: 1000<br>          derivedFields:<br>            - datasourceUid: Loki<br>              matcherRegex: &quot;traceID=(\\w+)&quot;<br>              name: TraceID<br>              url: &quot;$${__value.raw}&quot;<br>        isDefault: true<br>        editable: true<br><br>dashboardProviders:<br>  dashboardproviders.yaml:<br>    apiVersion: 1<br>    providers:<br>      - name: &#39;default&#39;<br>        orgId: 1<br>        folder: &#39;&#39;<br>        type: file<br>        disableDeletion: false<br>        editable: true<br>        options:<br>          path: /var/lib/grafana/dashboards/default<br><br>dashboards:<br>  default:<br>    loki-logs:<br>      gnetId: 18494<br>      revision: 1<br>      datasource: Loki<br><br>grafana.ini:<br>  server:<br>    root_url: &quot;%(protocol)s://%(domain)s:%(http_port)s/&quot;<br>  analytics:<br>    check_for_updates: true<br>    check_for_plugin_updates: true<br>  log:<br>    mode: console<br>  paths:<br>    data: /var/lib/grafana/<br>    logs: /var/log/grafana<br>    plugins: /var/lib/grafana/plugins<br>    provisioning: /etc/grafana/provisioning<br><br>securityContext:<br>  runAsNonRoot: true<br>  runAsUser: 472<br>  runAsGroup: 472<br>  fsGroup: 472<br>  allowPrivilegeEscalation: false<br>  capabilities:<br>    drop:<br>      - ALL<br><br>serviceMonitor:<br>  enabled: false<br><br>rbac:<br>  create: true<br>  pspEnabled: false<br><br>serviceAccount:<br>  create: true<br>  autoMount: true</pre><p>and deploy via Helm:</p><pre>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br><br>helm upgrade --install grafana grafana/grafana \<br>-f values-grafana.yaml \<br>-n monitoring --create-namespace \<br>--set &quot;adminPassword=$(openssl rand -base64 24)&quot; \<br>--reset values</pre><p>Go to <em>Grafana</em> -&gt; <em>Dashboards</em> and click the dashboard we provisioned as bundle with the Helm Chart <em>Loki Kubernetes Logs</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hhgyYoFw2mpbST9DR1kGDg.png" /></figure><blockquote><em>💡 </em>The Grafana admin password can be found in grafana secret in monitoring namespace.<br><em>⚠️</em> This Grafana installation and the provided <a href="https://grafana.com/grafana/dashboards/18494-kubernetes-logs-from-loki/">dashboard</a> are intended for demonstration purposes only.</blockquote><p><em>This article was originally posted as a blueprint in </em><strong><em>Open Telekom Cloud Architecture Center</em></strong><em>: </em><a href="https://arch.otc-service.com/docs/blueprints/by-use-case/observability/aggregate-cce-logs-with-fluentbit-and-grafana-loki"><em>Aggregate CCE Logs with Fluent Bit &amp; Grafana Loki</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*hEyzsH-_uPEX2D3J.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa98c8792da" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/aggregate-open-telekom-cloud-cce-logs-with-fluent-bit-grafana-loki-6aa98c8792da">Aggregate Open Telekom Cloud CCE Logs with Fluent Bit &amp; Grafana Loki</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Grafana Loki on Open Telekom Cloud CCE]]></title>
            <link>https://akyriako.medium.com/deploy-grafana-loki-on-open-telekom-cloud-cce-434d54a1337e?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/434d54a1337e</guid>
            <category><![CDATA[logs]]></category>
            <category><![CDATA[open-telekom-cloud]]></category>
            <category><![CDATA[cloud-container-engine]]></category>
            <category><![CDATA[grafana-loki]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 10:25:59 GMT</pubDate>
            <atom:updated>2026-06-29T12:53:03.350Z</atom:updated>
            <content:encoded><![CDATA[<p>Deploy Grafana Loki on Open Telekom Cloud Container Engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oS5om_wSe1smvQHmpRcZHA.png" /></figure><p>Deploying <a href="https://grafana.com/docs/loki/latest/">Grafana Loki</a> on Cloud Container Engine (CCE) enables a scalable, cloud-native log aggregation and monitoring solution within the Open Telekom Cloud environment. Loki integrates seamlessly with Grafana to collect, index, and visualize log data without the high storage costs associated with traditional log management systems. By running Loki on CCE, users can take advantage of Kubernetes-native deployments, simplified scaling, and secure, managed infrastructure. This approach provides an efficient foundation for observability, supporting modern DevOps practices and improving operational insights across distributed applications.</p><h3>Architecture</h3><p>Grafana Loki is built around a modular architecture that separates log ingestion, storage, and querying to achieve scalability and cost efficiency. When logs are collected from applications or system components, they are first received by the <strong>distributor</strong>, which authenticates and validates the data before <em>routing</em> it to the <strong>ingesters</strong>. The ingesters <em>temporarily hold log streams in memory</em>, compress them into chunks, and then <em>push these chunks to long-term storage</em> such as S3 or in our case, Open Telekom Cloud Object Storage (OBS). At the same time, they write small index files that map log labels to their corresponding chunks, stored either in a key-value database or alongside the chunks in object storage.</p><p>When a user runs a query from Grafana or directly through the Loki API, the querier reads the index to locate the relevant chunks, retrieves them from storage, and returns the combined results. In larger or production setups, a <strong>query frontend</strong> can be added to optimize performance by caching and parallelizing requests. This division of responsibilities — distributors for routing, ingesters for writing, and queriers for reading, ensures Loki remains efficient even at scale. On Cloud Container Engine (CCE), this architecture integrates naturally with Kubernetes, providing a reliable and cost-effective platform for log aggregation and analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMvzM99PM-9rLAq79az5ug.png" /><figcaption>For more information, consult the official <a href="https://grafana.com/docs/loki/latest/get-started/architecture/">Loki architecture</a> documentation.</figcaption></figure><h3>Installing Grafana Loki</h3><p>Installing Grafana Loki on Cloud Container Engine (CCE) using Helm is the most straightforward and flexible deployment approach. The Helm chart provided by Grafana supports several deployment modes, allowing users to tailor the setup to their operational needs and cluster scale.</p><ol><li>The <a href="https://grafana.com/docs/loki/latest/get-started/deployment-modes/#monolithic-mode">monolithic</a> mode deploys Loki as a single process that combines all core components — distributor, ingester, querier, and compactor — into one service. This variant is ideal for small environments, test setups, or single-tenant use cases where simplicity and ease of management are more important than high scalability.</li><li>The <a href="https://grafana.com/docs/loki/latest/get-started/deployment-modes/#simple-scalable">simple scalable</a> variation provides a middle ground, offering some degree of component separation and scalability without the full complexity of the distributed deployment. It’s often used in medium-sized environments or early production stages.</li><li>And finally, the <a href="https://grafana.com/docs/loki/latest/get-started/deployment-modes/#microservices-mode">microservices</a> mode (also referred in the past as the distributed mode) separates Loki’s components into individual services. Each component can scale independently, offering higher performance and fault tolerance. This setup suits production workloads and larger clusters where log volume and query traffic vary significantly.</li></ol><p>In this blueprint, Loki will be deployed on Cloud Container Engine (CCE) in microservices mode as it is the only one that provides the flexibility and performance needed for large production-grade environments where reliability and throughput are critical. For backend storage, the deployment will use Open Telekom Cloud Object Storage (OBS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jLUJeYj8mDRO_4v4i8dT6Q.png" /></figure><p>Go to <em>Open Telekom Cloud Console</em> -&gt; <em>Object Storage Service</em> and click <em>Create Bucket</em> to create an OBS bucket for Loki.</p><blockquote>A single bucket is sufficient for <a href="https://grafana.com/docs/loki/latest/operations/storage/boltdb-shipper/">boltdb-shipper</a> or <a href="https://grafana.com/docs/loki/latest/operations/storage/tsdb/">TSDB</a> modes.</blockquote><blockquote>📌 Enable versioning to protect against accidental deletes during compaction.<br>📌 Keep the bucket private.</blockquote><p><strong>Optionally</strong>, if no access key pair exists, open the <em>Open Telekom Cloud Console</em> -&gt; <em>Identity &amp; Access Management (IAM)</em>. In the IAM dashboard, select <em>Users</em>, choose the user you intend to work with, and open the <em>Security Settings</em> tab. Scroll down to the <em>Access Keys</em> section and select <em>Create Access Key</em> to generate a new Access Key and Secret Key pair.</p><ol><li>Create a namespace and provision the Access Key and Secret Key pair as a Kubernetes Secret:</li></ol><pre>kubectl create namespace monitoring<br><br>kubectl -n monitoring create secret generic loki-obs-credentials \<br>--from-literal=S3_ACCESS_KEY_ID=&#39;&lt;ACCESS_KEY&gt;&#39; \<br>--from-literal=S3_SECRET_ACCESS_KEY=&#39;&lt;SECRET_KEY&gt;&#39; \<br>--from-literal=S3_REGION=&#39;&lt;REGION&gt;&#39;</pre><p>or alternatively create a manifest called, <strong>loki-obs-credentials.yaml</strong>, and apply it with <strong>kubectl</strong>:</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>    name: loki-obs-credentials<br>    namespace: monitoring<br>type: Opaque<br>stringData:<br>    S3_ACCESS_KEY_ID: &quot;&lt;ACCESS_KEY&gt;&quot;<br>    S3_SECRET_ACCESS_KEY: &quot;&lt;SECRET_KEY&gt;&quot;<br>    S3_REGION: &quot;&lt;REGION&gt;&quot;</pre><blockquote>Loki’s S3 client supports the use of static credentials. To prevent exposing sensitive information in the Helm values file, these credentials are stored in a Kubernetes Secret. The environment variables use the S3_ prefix, following the naming conventions of the S3 SDK. Replace the placeholder values with your own access key (AK) and secret key (SK). In this blueprint, all resources will be deployed in the eu-de region, which will be used as the default region throughout the setup.</blockquote><p>Next, we need to prepare the Helm chart override values, namely <strong>values-loki-obs.yaml</strong>:</p><pre>deploymentMode: Distributed<br><br>global:<br>    dnsService: &quot;coredns&quot;<br><br>loki:<br>    auth_enabled: false<br><br>    schemaConfig:<br>        configs:<br>        - from: &quot;2024-01-01&quot;<br>            store: tsdb<br>            object_store: s3<br>            schema: v13<br>            index:<br>                prefix: loki_index_<br>                period: 24h<br><br>    storage:<br>        type: s3<br>        bucketNames:<br>            chunks: loki<br>            ruler: loki<br>            admin: loki<br>        s3:<br>            endpoint: obs.eu-de.otc.t-systems.com<br>            region: ${S3_REGION}<br>            accessKeyId: ${S3_ACCESS_KEY_ID}<br>            secretAccessKey: ${S3_SECRET_ACCESS_KEY}<br>            s3ForcePathStyle: true<br><br>    limits_config:<br>        ingestion_rate_mb: 8<br>        ingestion_burst_size_mb: 16<br>        max_query_series: 200000<br>        max_global_streams_per_user: 5000<br>        reject_old_samples: true<br>        reject_old_samples_max_age: 72h<br>        retention_period: 30d<br><br>    compactor:<br>        retention_enabled: true<br>        delete_request_store: s3<br>        compaction_interval: 5m<br>        delete_request_cancel_period: 24h<br><br>backend: { replicas: 0 }<br>read:    { replicas: 0 }<br>write:   { replicas: 0 }<br>singleBinary: { replicas: 0 }<br><br>gateway:<br>    enabled: true<br>    replicas: 1<br>    service:<br>        type: ClusterIP<br>        port: 80<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 50m,  memory: 128Mi }<br>        limits:   { cpu: 200m, memory: 256Mi }<br><br>distributor:<br>    replicas: 2<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true  <br>    resources:<br>        requests: { cpu: 100m, memory: 256Mi }<br>        limits:   { cpu: &quot;1&quot;,  memory: 512Mi }<br><br>ingester:<br>    replicas: 3<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>        - -ingester.wal-enabled=true<br>    resources:<br>        requests: { cpu: 500m, memory: 2Gi }<br>        limits:   { cpu: &quot;2&quot;,  memory: 4Gi }<br>    persistence:<br>        enabled: true<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        size: 20Gi    <br>        storageClass: csi-disk  <br><br>indexGateway:<br>    replicas: 2<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 200m, memory: 1Gi }<br>        limits:   { cpu: &quot;1&quot;,  memory: 2Gi }<br>    persistence:<br>        enabled: true<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        size: 10Gi<br>        storageClass: csi-disk  <br><br>querier:<br>    replicas: 2<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 200m, memory: 1Gi }<br>        limits:   { cpu: &quot;1&quot;,  memory: 2Gi }<br><br>queryFrontend:<br>    replicas: 2<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 100m, memory: 256Mi }<br>        limits:   { cpu: 500m, memory: 512Mi }<br><br>queryScheduler:<br>    replicas: 2<br>    maxUnavailable: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 100m, memory: 256Mi }<br>        limits:   { cpu: 500m, memory: 512Mi }<br><br>ruler:<br>    enabled: true<br>    replicas: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 100m, memory: 512Mi }<br>        limits:   { cpu: 500m, memory: 1Gi }<br>    persistence:<br>        enabled: true<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        size: 2Gi<br>        storageClass: csi-disk  <br><br>compactor:<br>    replicas: 1<br>    extraEnvFrom:<br>        - secretRef: { name: loki-obs-credentials }<br>    extraArgs:<br>        - -config.expand-env=true<br>    resources:<br>        requests: { cpu: 200m, memory: 1Gi }<br>        limits:   { cpu: &quot;1&quot;,  memory: 2Gi }<br>    persistence:<br>        enabled: true<br>        accessModes: [&quot;ReadWriteOnce&quot;]<br>        size: 10Gi<br>        storageClass: csi-disk  <br><br>memberlist:<br>    service:<br>        publishNotReadyAddresses: true<br><br>monitoring:<br>    serviceMonitor: { enabled: false }<br>    selfMonitoring: { enabled: false }<br><br>podSecurityContext: { fsGroup: 10001 }<br>securityContext:<br>    runAsNonRoot: true<br>    runAsUser: 10001<br>    runAsGroup: 10001<br>    allowPrivilegeEscalation: false</pre><blockquote>The key points to note from this configuration are:</blockquote><blockquote>deploymentMode: Distributed, dictates Loki to run in a fully distributed architecture, where each core component—such as the distributor, ingester, and querier—operates as a separate service. This enables independent scaling, better performance, and higher resilience, making it suitable for production environments.</blockquote><blockquote>schemaConfig.configs.store: tsdb, tells Loki to use the <a href="https://grafana.com/docs/loki/latest/operations/storage/tsdb/">TSDB</a> (Time Series Database) format for storing log data. This modern storage engine improves query performance, compaction, and index management compared to older backends, and is the recommended option for recent Loki versions.</blockquote><blockquote>The storage.type: s3 setting defines S3-compatible object storage as Loki’s backend for storing log chunks. The storage.s3 section provides the connection details—such as endpoint, bucket name, and credentials—needed for Loki to write and read data from that storage. Make sure you adjust the storage.s3.endpoint and storage.bucketNames.* values according to the properties of the OBS bucket we provisioned in the previous step.</blockquote><blockquote>global.dnsService: coredns, defines the DNS service Loki uses for internal name resolution within the Kubernetes cluster. Setting it explicitly to coredns ensures compatibility with the default DNS setup in most modern Kubernetes distributions, one of them CCE. If this value is not specified, Loki defaults to using kube-dns.</blockquote><blockquote>For deeper understanding of the configuration values of the Helm chart for this deployment mode refer to the official guide: <a href="https://grafana.com/docs/loki/latest/setup/install/helm/install-microservices/">Install the microservice Helm chart</a>.</blockquote><p>And finally, we need to deploy the Helm chart:</p><pre>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br><br>helm upgrade --install loki grafana/loki \<br>-f values-loki-obs.yaml \<br>-n monitoring --create-namespace \<br>--reset-values</pre><blockquote>Ensure that you select the grafana/loki chart from the Helm repository, as this is the current version that supports deployment in microservices mode. <strong>Do not</strong> use grafana/loki-distributed, which is deprecated and no longer maintained and replaced by grafana/loki.</blockquote><h3>Verification</h3><p>If the deployment was successful, you should see multiple pods running in the monitoring namespace:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UGdBXVQ5RCuJeptdemZazA.png" /></figure><p>along with the <strong>loki_cluster_seed.json</strong> file created in your OBS bucket:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eF10n07pw5GbYpSiEdXYEg.png" /></figure><blockquote>The <strong>loki_cluster_seed.json</strong> file is a metadata file generated by Loki to help manage and identify the state of the cluster. It contains information such as the cluster’s unique identifier, deployment configuration, and references to index and storage metadata. This file acts as a seed record that allows Loki components to coordinate and recognize each other during startup or recovery. In setups using object storage, such as Open Telekom Cloud OBS, the file is stored in the bucket defined for Loki’s backend and ensures consistent initialization of the cluster after restarts or scaling events.</blockquote><p><em>This article was originally posted as a blueprint in </em><strong><em>Open Telekom Cloud Architecture Center</em></strong><em>: </em><a href="https://arch.otc-service.com/docs/blueprints/by-use-case/observability/deploy-grafana-loki-on-cce"><em>Deploy Grafana Loki on CCE</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*ClLMbIMh2LI4_Ktk.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=434d54a1337e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build your own Managed Kubernetes Service on Proxmox with CAPI]]></title>
            <link>https://itnext.io/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-8d9786644818?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d9786644818</guid>
            <category><![CDATA[proxmox]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[cluster-api]]></category>
            <category><![CDATA[kubeadm]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Sat, 18 Oct 2025 12:32:32 GMT</pubDate>
            <atom:updated>2025-12-22T08:48:51.139Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Build your own KaaS platform on Proxmox with Cluster API.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hfRSFSk-fdoVTzBs" /></figure><h3>What is CAPI?</h3><p><a href="https://github.com/kubernetes-sigs/cluster-api">Cluster API</a> (CAPI) is an open-source Kubernetes sub-project. Its goal is to bring Kubernetes-style, declarative APIs and controllers to the problem of bootstrapping, configuring, upgrading and operating entire Kubernetes clusters, treating clusters themselves (and the machines that compose them) as first-class Kubernetes resources rather than as external, manually-provisioned infrastructure.</p><p>CAPI is deployed to a Kubernetes cluster, it leverages your familiar toolchain (kubectl, CRDs, manifests) to manage clusters at scale. When you request a new cluster, via a new custom resource named Cluster, a <strong>provider-specific controller</strong> provisions the necessary resources in the target infrastructure. As cluster demands change or when upgrades are required, simply updating the Kubernetes manifests and reapplying them triggers rolling updates or scale-out operations, all managed by the same reconciliation loops that govern standard Kubernetes workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*LrU2nPVyEN0Ir_4o_cKLqQ.png" /></figure><p>Adopting CAPI brings consistency and repeatability to cluster operations. Cluster operators can define golden-image references, networking topologies and scaling policies in version-controlled manifests. The reconciliation-driven approach ensures that drift is corrected automatically, while upgrade strategies (for example, in-place control-plane upgrades followed by a rolling reboot of worker nodes) can be codified and reused. This makes Cluster API especially valuable for organizations managing hundreds of clusters across multiple teams, regions or infrastructure providers, including public clouds and on-premises environments.</p><h3>How is CAPI working?</h3><p>CAPI is built around a few key abstractions and components that work together to model and manage Kubernetes clusters declaratively. At a high level, you have:</p><h4>Management &amp; Workload Clusters</h4><p>A <em>management cluster</em> is a regular Kubernetes control plane where the CAPI controllers run. It holds all of the Cluster API Custom Resources (CRs) and drives the lifecycle of one or more <em>workload clusters; </em>the actual Kubernetes clusters you are going to use for running your applications.</p><h4>Custom Resource Definitions (CRDs)</h4><p>CAPI introduces its own CRDs to represent clusters and the machines that comprise them:</p><ul><li>Cluster<strong>: </strong>The top-level object defining a Kubernetes cluster (networking, Kubernetes version, etc.).</li><li>Machine, MachineSet and MachineDeployment<strong>: </strong>Machines map to individual VMs or bare-metal hosts. A MachineSet ensures a stable number of Machines, while a MachineDeployment provides rolling-update semantics over MachineSets (much like a Deployment for Pods).</li><li>ControlPlane: Encapsulates control-plane-specific settings (certificate management, high-availability, version upgrades) and drives creation or scaling of control-plane Machines (the default ControlPlane provider is KubeadmControlPlane).</li><li>MachineHealthCheck: Defines criteria for identifying unhealthy Nodes and triggers automatic replacement of the corresponding Machines.</li></ul><h4>Provider Components</h4><p>The reconciliation logic is split, abstracted and delegated into three <strong>pluggable</strong> provider types, each implemented by its very own controller manager:</p><ul><li><strong>Infrastructure provider: </strong>Talks to the target cloud or virtualization API (AWS, Azure, GCP, OpenStack, Proxmox, VMware etc.) to create and delete VMs, networks, load balancers and related resources.</li></ul><blockquote>You can find an indicative, but not extensive list, of the most common providers here: <a href="https://image-builder.sigs.k8s.io/capi/capi#providers">https://image-builder.sigs.k8s.io/capi/capi#providers</a></blockquote><ul><li><strong>Bootstrap provider: </strong>Prepares a newly-provisioned VM to become a Kubernetes Node: generating certs, initializing the control plane on the first Machine, and running kubeadm join on subsequent control-plane or worker Machines.</li><li><strong>ControlPlane provider: </strong>Manages the lifecycle of control-plane Machines themselves (implementing rolling upgrades of API servers via e.g.KubeadmControlPlane).</li></ul><h4>Controllers &amp; Reconciliation</h4><p>For each CRD, a corresponding controller watches the .spec you declare, compares it to the actual state in the cloud and Kubernetes, and takes actions to converge them. If you increase the replica count of aMachineDeployment, the MachineDeployment controller will create a new MachineSet (or update an existing one), which in turn lets the Infrastructure and Bootstrap controllers provision and join new VMs. If someone manually deletes a VM, the next reconciliation cycle notices the missing Machine object and reprovisions it automatically. Finalizers ensure that when you delete a Cluster resource, machines and cloud resources are torn down in the proper order before the Cluster object is removed.</p><h3>Prerequisites</h3><p>For this lab you need to bring with:</p><ol><li>a <a href="https://www.proxmox.com/">Proxmox</a> installation (single-node or even better a multi-node cluster). <em>Installing and configuring Proxmox is </em><strong><em>out</em></strong><em> of the scope of this article</em>.</li><li>Relevant hands-on experience on Proxmox and Kubernetes.</li><li>A workstation with homebrew, kubectl, clusterctl and kind.</li><li>A virtual machine with <a href="https://k3s.io/">k3s</a> for implementing the management cluster.</li></ol><p>In this lab we are going to demonstrate how you could create your own Managed Kubernetes Service using CAPI with Proxmox as infrastructure provider, kubeadm as bootstrap provider and for the role of control plane provider we are going to use <strong>kubeadm</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*p4hZ3Op9AByKF2LqEjGCTg.png" /></figure><h3>Preparing your Proxmox Users and API Tokens</h3><p>Before we can build images, the image-builder needs permission to interact with your Proxmox environment through its API. This access allows it to launch a virtual machine, perform the build process, and then turn that VM into a reusable template. Execute the following command in one of your Proxmox nodes:</p><pre>pveum user add image-builder@pve<br>pveum aclmod / -user image-builder@pve -role PVEAdmin<br>pveum user token add image-builder@pve capi -privsep 0<br><br>pveum user add capmox@pve<br>pveum aclmod / -user capmox@pve -role PVEAdmin<br>pveum user token add capmox@pve capi -privsep 0</pre><h3>Building Images for Proxmox with image-builder</h3><p>With <a href="https://github.com/kubernetes-sigs/image-builder">image-builder</a> we try to standardize and automate the creation of virtual machine images that are pre-configured for Kubernetes and Cluster API, ensuring consistency, security, and speed during cluster provisioning. Rather than relying on untested or manually assembled OS snapshots, image-builder uses declarative “recipes” (built with tools like <a href="https://developer.hashicorp.com/packer">Packer</a> and <a href="https://www.redhat.com/en/ansible-collaborative">Ansible</a>) to bake in all the required components as kernel settings, container runtime, kubeadm binaries, networking plugins, and any custom security hardening into one baseline image. These images then serve as the immutable base for all control-plane and worker nodes, eliminating variability and drift across environments.</p><p>By producing provider-specific artifacts, image-builder decouples the image creation lifecycle from cluster creation. The result is faster cluster bootstraps, reproducible upgrades aligned to Kubernetes versions, and a GitOps-friendly workflow where image updates propagate through your pipelines just like application code.</p><p>So let’s perform the following steps to install, configure and create or base line with the image-builder:</p><p>1️⃣ Clone (or fork and clone) the repo in your workstation:</p><pre>git clone git@github.com:kubernetes-sigs/image-builder.git<br>cd image-builder/images/capi</pre><p>2️⃣ Install the dependencies:</p><p>In order to install the dependencies automatically execute the following command:</p><pre>make deps</pre><p>This will install dependencies in the folderimage-builder/images/capi/.bin (if they are not already on your system). Then you need to add that to your PATH environment variable before they can be used:</p><pre>export PATH=$PWD/.bin:$PATH</pre><p>3️⃣ Configure for Proxmox</p><p>Add a new file atimage-builder/images/capi/packer/proxmox/.env and provide the required environment variables, replacing with your own values where necessary:</p><pre>export PROXMOX_URL=&quot;https://&lt;PROXMOX_NODE_IP&gt;:8006/api2/json&quot;<br>export PROXMOX_USERNAME=image-builder@pve!capi<br>export PROXMOX_TOKEN=&lt;IMAGE_BUILD_API_TOKEN&gt;<br>export PROXMOX_NODE=&quot;&lt;PROXMOX_NODE_NAME&gt;&quot;<br>export PROXMOX_ISO_POOL=&quot;local&quot;<br>export PROXMOX_STORAGE_POOL=&quot;&lt;PROXMOX_STORAGE_POOL&gt;&quot;<br>export DISK_FORMAT=&quot;&lt;DISK_FORMAT&gt;&quot;</pre><pre>export PROXMOX_BRIDGE=&quot;vmbr0&quot;<br>export PROXMOX_NIC_MODEL=&quot;virtio&quot;</pre><pre># If you are working with OPNSense, image-builder is not <br># assigning a Proxmox MAC address to the VM, and this seems <br># to be an issue with DHCP in VXLANs controlled by OPNSense!<br># In theory you can configure your MTU and VLAN tag with the <br># following variables. For the time being leave them commented.</pre><pre># export PROXMOX_MTU=&quot;1450&quot;<br># export PROXMOX_VLAN=&quot;23&quot;<br></pre><pre>export PACKER_FLAGS=&quot;--var memory=2048 --var &#39;kubernetes_rpm_version=1.32.5&#39; --var &#39;kubernetes_semver=v1.32.5&#39; --var &#39;kubernetes_series=v1.32&#39; --var &#39;kubernetes_deb_version=1.32.5-1.1&#39;&quot;</pre><blockquote>Make sure you add the <em>.env</em> file in your <em>.gitignore</em> in order to avoid leaking API tokens when you commit the changes to your repo (if you have forked it in the first place).</blockquote><blockquote>The environment variables <em>PROXMOX_ISO_POOL</em>, <em>PROXMOX_BRIDGE</em>,<em>PROXMOX_STORAGE_POOL</em> here are optional, just to demonstrate how to override the default values, because we might need to adjust some of them later on for a multi-node Proxmox cluster scenario.</blockquote><p>If you want to bake an image for a specific Kubernetes version (or pass specific variables to parameterize the baseline image) use the PACKER_FLAGS variable. You can omit altogether but I don’t recommend it until you fully grasp how image-builder utilizes Packer behind the scenes.</p><p>4️⃣ Configure PROXMOX_STORAGE_POOL and DISK_FORMAT</p><p>⚠️ <em>Very Important:</em></p><p>If you are running on a <strong>single-node Proxmox</strong> installation you need to set PROXMOX_STORAGE_POOL value to local-lvm. Consequently, because local-lvm is not accepting qcow2 disk formats, you need to set DISK_FORMAT value to raw.</p><p>If, on the other hand, you are running a <strong>multi-node Proxmox cluster</strong>, you need to set PROXMOX_STORAGE_POOL value as the name of a storage <strong>that can be shared among nodes, </strong>otherwise you will not be able to spin control-plane and worker nodes on other nodes than the one you uploaded your baseline image. In addition to that, you have to configure that very storage to accept qcow2 format. Set DISK_FORMAT value to qcow2.</p><p>📓 <em>A short side-note:</em></p><p>In my case, I run a <a href="https://www.truenas.com/">TrueNAS</a> server as a virtual machine in one of the Proxmox nodes and I expose an NFS share which I add as storage in Proxmox (funny circle I know; performance is not great so I do recommend building an external NAS if you have the budget).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xXshgre7U0Z3-kHZixXTSQ.png" /></figure><p>Installing a separate TrueNAS, is an effort worth investing in, as you could externalize <strong>all</strong> the storage demands of <strong>all</strong> your Kubernetes clusters to TrueNAS by using <a href="https://github.com/democratic-csi/democratic-csi">democratic-csi</a> as CSI (but this is way out of the scope of this article, maybe I will write it up in a separate article in the future).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lm_eu81SxxzF8STx5x_Yhg.png" /></figure><p>5️⃣ Adjust Packer Template</p><p>We are going to make some adjustments in the file that Packer is using to create the baseline image, as it is missing some details that come in very handy when working with Proxmox.</p><p>Some of them are just missing, others just regulate the hardware specs that we are going to demand for our base line image template. So open the file below image-builder/images/capi/packer/proxmox/packer.json.tmpl:</p><p>Add in network_adapters and new property called model, as shown in the snippet below:</p><pre>&quot;network_adapters&quot;: [<br>        {<br>          &quot;model&quot;: &quot;{{user `nic_model`}}&quot;,<br>          &quot;bridge&quot;: &quot;{{user `bridge`}}&quot;,<br>          &quot;mtu&quot;: &quot;{{ user `mtu` }}&quot;,<br>          &quot;vlan_tag&quot;: &quot;{{user `vlan_tag`}}&quot;<br>        }<br> ],</pre><p>Under variables, add:</p><pre>&quot;nic_model&quot;: &quot;{{env `PROXMOX_NIC_MODEL`}}&quot;,</pre><p>change (remove or append according to -/+ prefixes):</p><pre>- &quot;disk_format&quot;: &quot;qcow2&quot;,<br>+ &quot;disk_format&quot;: &quot;{{env `DISK_FORMAT`}}&quot;,<br>- &quot;sockets&quot;: &quot;2&quot;,<br>+ &quot;sockets&quot;: &quot;1&quot;,</pre><p>6️⃣ Build Image</p><p>Execute the following commands:</p><pre>source packer/proxmox/.env<br>make build-proxmox-ubuntu-2204</pre><p>That will build a base image for your control-plane and worker nodes, based on Ubuntu 22.04; the process takes quite some time (~25–35mins depending on the specs of your workstation, your internal &amp; external networks speed etc) so sit back and wait for it. In the meantime you can open your Proxmox console and observe how a new virtual machine is being created, gets its network configuration, package updates, and eventually having installed (via Ansible) all the necessary binaries that will make it suitable to serve as a Kubernetes node. When all fine tuning is complete, it will be automatically converted to a template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQfexqBboDstr-wZoWJxzA.png" /></figure><h3>Installing Prerequisites</h3><p>To maintain a single guide for both macOS and Linux, we’ll use <a href="https://brew.sh/">Homebrew</a> to install the binaries required for managing the cluster.</p><p>1️⃣ Install Homebrew</p><pre>/bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&quot;</pre><p>Follow the instructions of the installer, to complete the installation.</p><p>2️⃣ Install KiND</p><pre>brew install kind</pre><p>3️⃣ Install kubectl</p><pre>brew install kubectl</pre><p>4️⃣ Install clusterctl</p><pre>brew install clusterctl</pre><p>The clusterctl CLI delivers a streamlined “day one” experience for CAPI by automatically retrieving and installing the necessary provider manifests. It embeds recommended best practices for provider management, helping you avoid mis-configurations and simplifying tasks like upgrades.</p><p>Built on top of clusterctl, the Cluster API Operator is a Kubernetes operator that brings a fully declarative workflow for managing provider lifecycles within your management cluster. It enhances the deployment and day-to-day operation of Cluster API, making routine tasks and GitOps-driven automation effortless.</p><h3>Create Configuration for Proxmox</h3><p>Before creating a workload cluster, certain configurations required on your management cluster. This is done by setting up environment variables for the <a href="https://github.com/ionos-cloud/cluster-api-provider-proxmox/">Cluster API Provider for Proxmox VE</a> (CAPMOX) and then generating a cluster manifest.</p><p>Create a a file named capmox.env, and fill it in with the following content:</p><pre>## -- Controller settings -- ##<br>export PROXMOX_URL=&quot;https://&lt;PROXMOX_NODE_IP&gt;:8006&quot;<br>export PROXMOX_TOKEN=&#39;capmox@pve!capi&#39;<br>export PROXMOX_SECRET=&quot;&lt;CAPMOX_API_TOKEN_SECRET&gt;&quot;                          </pre><pre>## -- Required workload cluster default settings -- ##<br>export PROXMOX_SOURCENODE=&quot;&lt;PROXMOX_SOURCENODE_NAME&gt;&quot;                                     <br>export TEMPLATE_VMID=&quot;&lt;PROXMOX_BASE_IMAGE_TEMPLATE_ID&gt;&quot;                                          <br>export ALLOWED_NODES=&quot;[&lt;PROXMOX_NODE1_NAME&gt;,&lt;PROXMOX_NODE2_NAME&gt;...]&quot;<br>export VM_SSH_KEYS=&quot;ssh-rsa AAAAB...&quot;</pre><pre>## -- networking configuration-- ##<br>export CONTROL_PLANE_ENDPOINT_IP=&quot;192.168.1.230&quot;                    <br>export NODE_IP_RANGES=&quot;[192.168.1.220-192.168.1.229]&quot;               <br>export GATEWAY=&quot;192.168.1.1&quot;                                        <br>export IP_PREFIX=&quot;24&quot;                                               <br>export DNS_SERVERS=&quot;[192.168.1.1]&quot;                                  <br>export BRIDGE=&quot;vmbr0&quot;<br>export NETWORK_MODEL=&quot;virtio&quot;                                       </pre><pre>## -- xl nodes -- ##<br>export BOOT_VOLUME_DEVICE=&quot;scsi0&quot;                                   <br>export BOOT_VOLUME_SIZE=&quot;20&quot;                                       <br>export NUM_SOCKETS=&quot;1&quot;                                              <br>export NUM_CORES=&quot;2&quot;                                                <br>export MEMORY_MIB=&quot;2048&quot;                                            <br>export FILE_STORAGE_FORMAT=&quot;raw&quot;<br>export STORAGE_NODE=&quot;local-lvm&quot;<br>export NODE_REPLICAS=1</pre><pre>export CLOUD_INIT_CONFIG=&quot;#cloud-config package_update=true packages=- net-tools&quot;## -- Controller settings -- ##</pre><blockquote>Replace all the <em>192.168.1.0/24</em> IP address with the CIDR that matches your the bridge interface<em>vmbr0</em>, or whichever bridge you chose to use for that matter.</blockquote><blockquote>Replace all the values in placeholders with the ones corresponding in your lab.</blockquote><h3>Setting up a Management Cluster</h3><p>This is the cluster that runs one or more Infrastructure Providers and holds resources (for example, Machines), when provisioning multiple workload clusters.</p><p>1️⃣ Create the cluster</p><p>Create a kind config file named kind-management-cluster.yaml, with extra mounts to allow the Docker provider to access Docker on the host:</p><pre>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>networking:<br>  ipFamily: dual<br>nodes:<br>- role: control-plane<br>  extraMounts:<br>    - hostPath: /var/run/docker.sock<br>      containerPath: /var/run/docker.sock</pre><p>and execute the following command:</p><pre>kind create cluster --name=mgmt-capmox --config=kind-management-cluster.yaml</pre><p>the kubeconfig of the newly generated cluster is already merged with your KUBECONFIG and your kubectl is automatically set to use its context.</p><p>If you are planning to try Kamaji as control-plane provider, install the management cluster, using <a href="https://k3s.io/">K3s</a>, in a brand new virtual machine (that either uses the same bridge, vmbr0 in our lab, as the one you used above or one that can communicate with vmbr0) , by executing the following command:</p><pre>curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;--disable=traefik --disable=servicelb&quot; sh -</pre><p>You can find the kubeconfig of the newly created cluster under /etc/rancher/k3s/k3s.yaml. Copy it somewhere in your workstation (preferably under ~/.kube/k3s.yaml) and run:</p><pre>export KUBECONFIG=~/.kube/k3s.yaml</pre><blockquote>Make sure you change the server value from <a href="https://127.0.0.1:6443/"><em>https://127.0.0.1:6443</em></a> to the IP address assigned on the network device that uses the bridge <em>vmbr0</em>.</blockquote><p>2️⃣ Transform the cluster to a Management Cluster</p><p>With clusterctl installed and prerequisites satisfied, run clusterctl init to convert your Kubernetes cluster into a management cluster.</p><pre>source capmox.env<br>clusterctl init --infrastructure proxmox --ipam in-cluster</pre><blockquote>The command above will install <a href="https://github.com/ionos-cloud/cluster-api-provider-proxmox/">Cluster API Provider for Proxmox VE</a> (CAPMOX) as infrastructure provider and Kubeadm as control-plane &amp; bootstrap provider.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GcT7VutCjT_8kyZ1Vp9wpQ.png" /><figcaption>1. Infrastructure Provider (Proxmox) 2. Control-Plane Provider (Kubeadm) 3. Bootstrap Provider (Kubeadm)</figcaption></figure><h3>Creating a Cluster with Kubeadm CP Provider</h3><p>The kubeadm control plane provider watches KubeadmControlPlane resources in your management cluster and drives the lifecycle of control-plane nodes. When you create or update a KubeadmControlPlane, the provider generates a corresponding set of Machine objects (using your MachineTemplate) and orchestrates kubeadm init on the first node, then kubeadm join for additional replicas.</p><p>It manages certificate generation and rotation, handles rolling upgrades by replacing one control-plane node at a time, and ensures the Kubernetes API servers remain available throughout. All configuration (Kubernetes version, component settings, high-availability topology) is defined declaratively in the KubeadmControlPlane spec, and the provider’s reconciliation loop converges the actual state to match.</p><p>When using the kubeadm control plane provider with the Proxmox infrastructure provider (CAPMOX), the control plane controller creates Machine objects <strong>whose underlying VMs are provisioned on Proxmox</strong> according to your MachineTemplate. As each VM becomes available, the kubeadm provider runs kubeadm init on the first node and kubeadm join on subsequent replicas, while Proxmox handles the VM lifecycle, boot, networking, and storage. This integration lets you declaratively manage high-availability control planes on your Proxmox cluster using the familiar CAPI workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O0f8sYszBAis1KjYARITrQ.png" /><figcaption>Multiple Control Planes and Worker Nodes <strong>are running as VMs </strong>in Proxmox</figcaption></figure><p>1️⃣ Generate the manifests</p><p>The clusterctl generate cluster command returns a YAML manifest for creating a workload cluster.</p><pre>source capmox.env<br>clusterctl generate cluster capi-quickstart \ <br>  --kubernetes-version v1.33.0 \<br>  --control-plane-machine-count=1 \ <br>  --worker-machine-count=1  &gt; capi-quickstart.yaml</pre><p>This creates a manifest calledcapi-quickstart.yaml with a predefined list of CAPI resources as Cluster, Machines and MachineDeployments, etc. The generated file has some issues when it comes to play with Proxmox as target infrastructure and we have to make some manual changes.</p><p>2️⃣ Change the CIDR block of the Cluster object ⚠️</p><p>You need to change the cidrBlock value to something not overlapping with the CIDR block of your vmbr0 bridge (in this case, yours might differ) otherwise your Pods will starts stealing IP addresses from that network segment. So go and replace the value, I choose in my case a CIDR I know that I am not using somewhere else either in a physical or virtual LAN, 10.44.0.0/16.</p><pre>apiVersion: cluster.x-k8s.io/v1beta1<br>kind: Cluster<br>metadata:<br>  name: capi-quickstart<br>  namespace: default<br>spec:<br>  clusterNetwork:<br>    pods:<br>      cidrBlocks:<br>      - 10.44.0.0/16<br>  controlPlaneRef:<br>    apiVersion: controlplane.cluster.x-k8s.io/v1beta1<br>    kind: KubeadmControlPlane<br>    name: capi-quickstart-control-plane<br>  infrastructureRef:<br>    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1<br>    kind: ProxmoxCluster<br>    name: capi-quickstart</pre><p>3️⃣ Allow CAPMOX to overcommit memory ⚠️</p><p>When the infrastructure controller provisions a new VM on Proxmox, whether it’s a control-plane or worker node , it first sums the RAM already allocated on the target host to determine if your MachineTemplate will fit. If you haven’t explicitly enabled memory over-commit, this check will refuse to schedule any VM once the sum of allocated RAM reaches the node’s nominal capacity, causing new VMs to hang even though Proxmox could safely oversubscribe memory in practice.</p><p>For that matter, add the following properties to the ProxmoxCluster resource:</p><pre>schedulerHints:<br>    memoryAdjustment: 0</pre><p>It should eventually look like this:</p><pre>apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1<br>kind: ProxmoxCluster<br>metadata:<br>  name: capi-quickstart<br>  namespace: default<br>spec:<br>  allowedNodes:<br>  - pve-am06<br>  controlPlaneEndpoint:<br>    host: 192.168.1.230<br>    port: 6443<br>  dnsServers:<br>  - 192.168.1.1<br>  ipv4Config:<br>    addresses:<br>    - 192.168.1.220-192.168.1.229<br>    gateway: 192.168.1.1<br>    prefix: 24<br>  schedulerHints:<br>    memoryAdjustment: 0</pre><p>4️⃣ Create the cluster</p><p>Execute the command:</p><pre>kubectl apply -f capi-quickstart.yaml</pre><p>Then sit back and watch Proxmox clone the template and begin provisioning control-plane and worker nodes; a process that will take a few minutes to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yS2cRL2wRCrCYvy-G6vYRQ.png" /></figure><p>When is done, get the <strong>kubeconfig</strong> file of your new cluster by executing:</p><pre>clusterctl get kubeconfig capi-quickstart &gt; capi-quickstart.kubeconfig</pre><p>5️⃣ Deploy a CNI</p><p>For demonstration purposes, we are going to deploy <a href="https://docs.tigera.io/calico/latest/about/">Calico</a> as the CNI solution of our newly created cluster. In your lab you can choose the CNI of your liking. Execute the following command:</p><pre>kubectl --kubeconfig=./capi-quickstart.kubeconfig \<br>  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSQ4n9Qh6_lAsz14Cf-73w.png" /></figure><blockquote>When issues arise, you can access any node directly by opening an SSH session as the <em>root</em> user. This allows you to inspect system logs, review configuration files, and run diagnostic commands on each machine.</blockquote><h3>Next Steps</h3><p>In the <a href="https://akyriako.medium.com/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-and-kamaji-7858ad41cb09">next article</a>, we’ll explore how to leverage <strong>Kamaji</strong> to containerize the control plane of tenant clusters within the management cluster, leaving only the worker nodes running as virtual machines in Proxmox.</p><p>Kamaji is an open-source operator developed by <a href="https://clastix.io/">Clastix</a>, which imagines the Kubernetes control plane as a hosted service <strong>running entirely on Kubernetes</strong> itself. Rather than provisioning dedicated virtual machines or bare-metal servers for etcd, kube-apiserver, controller manager and scheduler, Kamaji deploys those components as <strong>Pods</strong> inside the management cluster and presents them to tenant clusters via a lightweight API. This Hosted Control Plane pattern decouples control plane lifecycle from underlying infrastructure and dramatically reduces both operational overhead and resource duplication.</p><p>If you found this post helpful, give it a 👏 and follow me for more insights on Kubernetes and cloud-native tech. More great content coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d9786644818" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/build-your-own-managed-kubernetes-service-on-proxmox-with-capi-8d9786644818">Build your own Managed Kubernetes Service on Proxmox with CAPI</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Touch SSL: Automating Let’s Encrypt on Open Telekom Cloud CCE]]></title>
            <link>https://akyriako.medium.com/zero-touch-ssl-automating-lets-encrypt-on-open-telekom-cloud-cce-cc42ef2bf5a6?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc42ef2bf5a6</guid>
            <category><![CDATA[lets-encrypt]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ssl-certificate]]></category>
            <category><![CDATA[cert-manager]]></category>
            <category><![CDATA[open-telekom-cloud]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Thu, 03 Jul 2025 07:19:05 GMT</pubDate>
            <atom:updated>2026-06-29T12:55:58.480Z</atom:updated>
            <content:encoded><![CDATA[<p>Seamless Let’s Encrypt automation for Open Telekom Cloud Container Engine with a DNS01 solver.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzUmwPM4j629sgS_BgUYPA.png" /></figure><p>A DNS01 challenge is a type of challenge used in the Domain Name System (DNS) to verify ownership of a domain during the process of obtaining an SSL/TLS certificate, often through services like <a href="https://letsencrypt.org/">Let’s Encrypt</a>.</p><p>When you request a certificate, the Certificate Authority (CA) needs to ensure that you have control over the domain for which you’re requesting the certificate. The DNS01 challenge is one of the methods used to prove this control. Here’s how it generally works:</p><ol><li><strong>Challenge Issuance</strong>: The CA provides you with a unique token (a random string of characters) that needs to be added to your domain’s DNS records.</li><li><strong>DNS Record Addition</strong>: You must create a specific DNS TXT record for your domain that includes the token provided by the CA. This record usually follows a format like _acme-challenge.example.com with a value corresponding to the token.</li><li><strong>Verification</strong>: Once you’ve added the TXT record to your domain’s DNS configuration, the CA will query your domain’s DNS records to look for the TXT record. If it finds the correct token, it confirms that you control the domain.</li><li><strong>Certificate Issuance</strong>: After successful verification, the CA will issue the SSL/TLS certificate.</li></ol><p>The DNS01 challenge is commonly used because it allows for domain validation without needing to serve files over HTTP, which can be advantageous in certain situations, such as when you don’t have a web server configured or when you’re managing multiple subdomains.</p><p>One of the tools, that can be employed in the context of Kubernetes, to secure certificates from a Certificate Authority (CA) via the ACME protocol using the the DNS01 challenge, is <a href="https://cert-manager.io/">cert-manager</a>. Specifically for Open Telekom Cloud, we can use an additional webhook that acts as an ACME DNS01 solver for Open Telekom Cloud’s Domain Name Service, <a href="https://github.com/akyriako/cert-manager-webhook-opentelekomcloud">cert-manager-webhook-opentelekomcloud</a>.</p><h3>Prerequisites</h3><p>Single prerequisite is cert-manager. If you don’t have it already installed on your CCE Cluster, this can be very easily done using a Helm Chart:</p><pre>helm repo add jetstack https://charts.jetstack.io <br>helm repo update <br><br>helm install \<br>  cert-manager jetstack/cert-manager \<br>  --namespace cert-manager \<br>  --create-namespace \<br>  --version v1.15.3 \<br>  --set crds.enabled=true</pre><h3>Installing the ACME DNS01 Solver</h3><p><strong>cert-manager-webhook-opentelekomcloud</strong> is an ACME DNS01 solver webhook for Open Telekom Cloud DNS written in Golang, and requires <strong>cert-manager</strong> to be installed first.</p><h4>Acquiring Access/Secret Keys</h4><p>In the <em>Open Telekom Cloud Console</em>, go to <em>My Credentials</em> ➡️ <em>Access Keys</em> and either pick up an existing pair or create a new one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzHjhOEqCryyVCCFGIWAdA.png" /></figure><p>Export this pair as environment variables:</p><pre>export OS_ACCESS_KEY={value}<br>export OS_SECRET_KEY={value}</pre><h4>Installing the Helm Chart</h4><pre>helm repo add cert-manager-webhook-opentelekomcloud https://akyriako.github.io/cert-manager-webhook-opentelekomcloud/<br>helm repo update<br><br>helm upgrade --install \<br>    acme-dns cert-manager-webhook-opentelekomcloud/cert-manager-webhook-opentelekomcloud \<br>  --set opentelekomcloud.accessKey=$OS_ACCESS_KEY \<br>  --set opentelekomcloud.secretKey=$OS_SECRET_KEY \<br>  --namespace cert-manager</pre><h3>Installing Cluster Issuers</h3><p>You are going to need one ClusterIssuer for the <em>production</em> and one for the <em>staging</em> Let&#39;s Encrypt endpoint.</p><blockquote><strong>cert-manager</strong> has a known bug, that <strong>prevents</strong> custom webhooks to work with anIssuer. For that reason you need to install your issuer as ClusterIssuer.</blockquote><h4>For Staging</h4><p>Create and deploy the following manifest (<strong>opentelekomcloud-letsencrypt-staging.yaml</strong>):</p><pre>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: opentelekomcloud-letsencrypt-staging<br>  namespace: cert-manager<br>spec:<br>  acme:<br>    email: user@company.com<br>    server: https://acme-staging-v02.api.letsencrypt.org/directory<br>    privateKeySecretRef:<br>      name: opentelekomcloud-letsencrypt-staging-tls-key<br>    solvers:<br>    - dns01:<br>        webhook:<br>          groupName: acme.opentelekomcloud.com<br>          solverName: opentelekomcloud<br>          config:<br>            region: &quot;eu-de&quot;<br>            accessKeySecretRef:<br>              name: cert-manager-webhook-opentelekomcloud-creds<br>              key: accessKey<br>            secretKeySecretRef:<br>              name: cert-manager-webhook-opentelekomcloud-creds<br>              key: secretKey</pre><blockquote>Replace placeholder <strong>email</strong> value, user@company.com, with the email that will be used for requesting certificates from Let&#39;s Encrypt.</blockquote><pre>kubectl apply -f opentelekomcloud-letsencrypt-staging.yaml</pre><h4>For Production</h4><p>Create and deploy the following manifest (<strong>opentelekomcloud-letsencrypt.yaml</strong>):</p><pre>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: opentelekomcloud-letsencrypt<br>  namespace: cert-manager<br>spec:<br>  acme:<br>    email: user@company.com<br>    server: https://acme-v02.api.letsencrypt.org/directory<br>    privateKeySecretRef:<br>      name: opentelekomcloud-letsencrypt-tls-key<br>    solvers:<br>    - dns01:<br>        webhook:<br>          groupName: acme.opentelekomcloud.com<br>          solverName: opentelekomcloud<br>          config:<br>            region: &quot;eu-de&quot;<br>            accessKeySecretRef:<br>              name: cert-manager-webhook-opentelekomcloud-creds<br>              key: accessKey<br>            secretKeySecretRef:<br>              name: cert-manager-webhook-opentelekomcloud-creds<br>              key: secretKey</pre><blockquote>Replace placeholder <strong>email</strong> value, <em>user@company.com</em>, with the email that will be used for requesting certificates from Let&#39;s Encrypt.</blockquote><pre>kubectl apply -f opentelekomcloud-letsencrypt.yaml</pre><h3>Requesting a Certificate</h3><p>Create and deploy the following manifest (<strong>certificate-subdomain-example-com.yaml</strong>):</p><pre>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: certificate-subdomain-example-com<br>spec:<br>  dnsNames:<br>  - &#39;*.subdomain.example.com&#39;<br>  issuerRef:<br>    kind: ClusterIssuer<br>    name: opentelekomcloud-letsencrypt-staging<br>  secretName: certificate-subdomain-example-com-tls</pre><blockquote>Replace placeholder DNS name *.subdomain.example.com, with one that you own and will be used to request a certificate from Let&#39;s Encrypt.</blockquote><pre>kubectl apply -f certificate-subdomain-example-com</pre><blockquote>Using the <em>staging</em> endpoint of Let’s Encrypt before moving to the <em>production</em> endpoint is a best practice. Let’s Encrypt imposes rate limits on the number of certificates you can request in a given period to prevent abuse. By testing with the staging environment, you avoid hitting these limits during your development and testing phases.</blockquote><h3>Exposing a workload with an Ingress</h3><p>Create and deploy the following manifest (<strong>workload-ingress.yaml</strong>):</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: workload-ingress<br>  labels:<br>    app: workload<br>  annotations:<br>    kubernetes.io/elb.class: union<br>    kubernetes.io/elb.id: &quot;{value}&quot;<br>    kubernetes.io/elb.port: 443<br>spec:<br>  ingressClassName: cce<br>  tls:<br>  - hosts:<br>      - subdomain.example.com<br>    secretName: certificate-subdomain-example-com-tls<br>  rules:<br>  - host: subdomain.example.com<br>    http:<br>      paths:<br>      - backend:<br>          service:<br>            name: workload-svc<br>            port:<br>              number: 80<br>        path: /<br>        pathType: ImplementationSpecific</pre><p>You need to have:</p><ul><li>a <em>workload</em> installed in your CCE Cluster (you can experiment with <strong>traefik/whoami</strong>)</li><li>this workload exposed with a Service, <em>workload-svc</em>, of type NodePort</li><li>a Shared Elastic Load Balancer, replace {value} with its ID.</li></ul><pre>kubectl apply -f workload-ingress.yaml</pre><p>If you visit in your browser the address <a href="https://subdomain.example.com/">https://subdomain.example.com</a> you will notice that the endpoint is served in HTTPS and is secured by a valid certificate.</p><p><em>This article was originally posted as a best practice in </em><strong><em>Open Telekom Cloud Architecture Center</em></strong><em>: </em><a href="https://arch.otc-service.com/docs/best-practices/containers/cloud-container-engine/issue-an-acme-certificate-with-dns01-solver-in-cce"><em>Issue an ACME Certificate with DNS01 Solver in CCE</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ljpIF0cQcI4yZjnTdTSl1w.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc42ef2bf5a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Typesense on Kubernetes]]></title>
            <link>https://itnext.io/deploy-typesense-on-kubernetes-893fe452c36a?source=rss-c6fda6c1c195------2</link>
            <guid isPermaLink="false">https://medium.com/p/893fe452c36a</guid>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[typesense]]></category>
            <category><![CDATA[kubernetes-operator]]></category>
            <category><![CDATA[raft]]></category>
            <dc:creator><![CDATA[Akriotis Kyriakos]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 14:23:52 GMT</pubDate>
            <atom:updated>2025-07-07T05:25:14.280Z</atom:updated>
            <content:encoded><![CDATA[<p>Deploy a highly available, self-healing Typesense cluster on Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSJnPu8ZXG46GPpHv_8OWA.png" /></figure><h3>What is Typesense?</h3><p><a href="https://typesense.org/">Typesense</a> is an open-source search engine designed for speed and simplicity. Its core purpose is to provide instant, relevant search results for applications and websites, making it a popular choice for developers who want powerful search functionality without the complexity of traditional search engines like <a href="https://www.elastic.co/elasticsearch">Elasticsearch</a>.</p><p>It supports features such as <strong>full-text search</strong>, <strong>real-time indexing</strong>, <strong>faceted filtering</strong>, and <strong>dynamic sorting</strong>. Typesense also includes advanced capabilities like <strong>vector and semantic search</strong>, allowing it to go beyond simple keyword matching and understand the intent behind queries.</p><h3>What are the deployment options?</h3><p>Typesense offers two main deployment options: the so-called <strong>Typesense Cloud</strong> and <strong>self-hosting</strong>. Typesense Cloud is a fully managed service where the infrastructure, scaling, high availability, and updates are handled for you by Typesense Inc., provided you’re willing to invest a not negligible sum per month for infrastructure and network traffic costs. On the self-hosting frontier, there is official support for Docker, Docker Compose, and native binaries for Linux and macOS — but <strong>not</strong> for Kubernetes.</p><p>Deploying Typesense on Kubernetes clusters presents several notable challenges, primarily due to the way Typesense manages cluster membership and consensus using <a href="https://github.com/baidu/braft">braft</a>, an implementation of the <a href="https://raft.github.io/raft.pdf">Raft</a> algorithm. One of the most persistent issues is <strong>quorum management</strong>. In Kubernetes, pods are ephemeral and their IP addresses can change, which conflicts with Raft’s expectation of stable node identities. This, if not done correct, can lead to situations where the cluster loses quorum or retains outdated information about nodes, <strong>resulting in downtime or the need for manual intervention to restore the cluster</strong>.</p><p><strong>All</strong> the community hacks and workaround so far, for installing Typesense on Kubernetes — such as using plain StatefulSets, Helm charts, custom scripts, sidecars or they just deploy a single node cluster that is not eventually free of the problems mentioned above — fall short as they cannot tackle the actual core problem: that Typesense relies on a stable and consistently updated quorum configuration for its Raft consensus.</p><h3>How is Raft working?</h3><p><a href="https://raft.github.io/raft.pdf">Raft</a> nodes operate in one of three possible states: <em>follower</em>, <em>candidate</em>, or <em>leader</em>. Every new node always joins the quorum as a follower. Followers can receive log entries from the leader and participate in voting for electing a leader. If no log entries are received for a specified period of time, a follower transitions to the candidate state. As a candidate, the node can accept votes from its peers nodes. Upon receiving a majority of votes, the candidate is becoming the leader of the quorum. The leader’s responsibilities include handling new log entries and replicating them to other nodes.</p><p>Another thing to consider is what happens when the node set changes, when nodes join or leave the cluster. If a quorum of nodes is available, raft can dynamically modify the node set without any issue (this happens every 30sec). But if the cluster cannot form a quorum, then problems start to appear or better to pile up. A cluster with N nodes can tolerate a failure of at most (N-1)/2 nodes without losing its quorum. If the available nodes go below this threshold then two events are taking place:</p><ul><li><strong>Raft declares the whole cluster as unavailable!</strong> (no leader can be elected, no more log entries can be processed)</li><li><strong>the remaining nodes are restarted in bootstrap mode!</strong></li></ul><p>In a Kubernetes environment, the nodes are actually Pods which are rather <strong>volatile by nature and their lifetime is quite ephemeral</strong> and subject to potential restarts, and that puts the whole concept of raft protocol consensus under a tough spot. As we can read in the official documentation of Typesense when it comes to <a href="https://typesense.org/docs/guide/high-availability.html#recovering-a-cluster-that-has-lost-quorum">recovering a cluster that has lost quorum</a>, it is explicitly stated:</p><p><em>“If a Typesense cluster loses more than </em><em>(N-1)/2 nodes at the same time, the cluster becomes unstable because it loses quorum and the remaining node(s) cannot safely build consensus on which node is the leader. To avoid a potential split brain issue, Typesense then stops accepting writes and reads </em><strong><em>until some manual verification and intervention is done</em></strong><em>.”</em></p><p>In production environments, manual intervention is sometimes impossible or undesirable, and downtime for a service like Typesense may be unacceptable. So what is the solution?</p><h3>Meet Typesense Kubernetes Operator (TyKO)</h3><p><a href="https://akyriako.github.io/typesense-operator-docs/">Typesense Kubernetes Operator</a> (TyKO) simplifies deploying, scaling, and managing Typesense clusters in Kubernetes by fulfilling the following objectives:</p><ul><li>Provides a Kubernetes-native interface to define and manage Typesense cluster configurations using a CRD named TypesenseCluster.</li><li>Simplifies the life-cycle automation of a HA Typesense cluster by handling aspects like: bootstrapping admin API Keys as Secret, deploying Typesense as a StatefulSet, provisioning Typesense services (headless, discovery and health-check Services), actively discovering and updating Typesense’s nodes list and mounting it in aConfigMap, managing the underlying storage with PersistentVolumeClaims, exposing Typesense API endpoint via an Ingress, exposing Typesense metrics as Prometheus targets and finally provisioning <a href="https://github.com/typesense/typesense-docsearch-scraper">DocSearch Scrapers</a> instances for content-crawling as CronJobs.</li><li>Actively re-discovers the quorum configuration reacting to changes in ReplicaSet <strong>without the need of an additional sidecar container.</strong></li><li>Recovers automatically a cluster that has lost quorum <strong>without the need of manual intervention</strong>,<strong> </strong>as long as the encountered error is not hardware related.</li></ul><h3>Talking is cheap, show me some code</h3><p>The operator’s installation is done with a simple Helm Chart:</p><pre>helm repo add typesense-operator https://akyriako.github.io/typesense-operator/<br>helm repo update<br><br>helm upgrade --install typesense-operator typesense-operator/typesense-operator -n typesense-system --create-namespace</pre><p>No complicated wiring, no complex tuning, batteries included — ready to go!</p><p>Let’s deploy a cluster now.</p><p>The cluster provisioning, as we mentioned before, relies on a new Custom Resource Definition that comes with the operator, called TypesenseCluster. The <a href="https://akyriako.github.io/typesense-operator-docs/docs/crds/">configuration options of the CRD</a> are plenty, but you can start here with a very simple manifest before delving more in it.</p><pre>apiVersion: ts.opentelekomcloud.com/v1alpha1<br>kind: TypesenseCluster<br>metadata:<br>  name: demo-cluster<br>spec:<br>  image: typesense/typesense:29.0<br>  replicas: 3<br>  storage:<br>    size: 100Mi<br>    storageClassName: &lt;STORAGE_CLASS_NAME&gt;</pre><p>Make sure you replace &lt;STORAGE_CLASS_NAME&gt; with the respective value supported from your underlying Kubernetes distribution and within minutes you will get a 3-nodes, high-available, self-healing cluster regardless if you are running on a local development machine with KiND, on bare-metal, on a cloud offered managed Kubernetes service or even on OpenShift.</p><p>Of course the cluster can be upgraded to a newer or downgraded to an earlier version of Typesense simply by changing the spec.image property. Same counts for storage.size as long as your CSI Provider permits to do so.</p><p>For more information concerning the internals of the controller and its configuration you can refer to the <a href="https://akyriako.github.io/typesense-operator-docs/">documentation</a>. Make sure, to leave a ⭐️ in the <a href="https://github.com/akyriako/typesense-operator">GitHub repository</a>, and if you are already using TyKO in your organisation would really make a difference if you would consider the option to enlist yourselves as <a href="https://akyriako.github.io/typesense-operator-docs/docs/adopters">Adopters</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=893fe452c36a" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/deploy-typesense-on-kubernetes-893fe452c36a">Deploy Typesense on Kubernetes</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>