<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Seralahthan on Medium]]></title>
        <description><![CDATA[Stories by Seralahthan on Medium]]></description>
        <link>https://medium.com/@techexpertise?source=rss-5cc3faae5c14------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Ljtb0Hw6ciD-En9gaJ2c_w.jpeg</url>
            <title>Stories by Seralahthan on Medium</title>
            <link>https://medium.com/@techexpertise?source=rss-5cc3faae5c14------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 22:02:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@techexpertise/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[One Pod per Agent Doesn’t Scale. Here’s How Agent Substrate Fixes It]]></title>
            <link>https://medium.com/swlh/one-pod-per-agent-doesnt-scale-here-s-how-agent-substrate-fixes-it-f5b0a0eb8d31?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5b0a0eb8d31</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[agent-substrate]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 06:31:19 GMT</pubDate>
            <atom:updated>2026-10-02T06:31:19.574Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZYYM_-VXrsG1NpaTN8604Q.png" /></figure><h3>What it really costs to run thousands of AI agents on Kubernetes, and how a shared pool of snapshot-driven workers changes both the economics and the security model</h3><p>Most AI agents spend their lives waiting.</p><p>Some of that waiting is short. While working on a task, an agent calls a model and waits a few seconds for the reply, then calls a tool and waits for the result. But most of it is much longer, and it happens between tasks: waiting for the user to read the answer and type the next message, for a human to approve a step, or for a webhook or a scheduled trigger that might not arrive until tomorrow.</p><p>A coding or support agent might be busy for a few minutes in an hour and idle for the rest. In that sense agents look much more like chat sessions than like batch jobs. Those long gaps between tasks are what this post is about.</p><p>That one observation explains why the way we run agents on Kubernetes is changing. The first generation of agent runtimes gave every agent its own pod. It’s simple, secure and easy to reason about. It’s also expensive once you have thousands of agents, because almost all of those pods are sitting idle.</p><p>Google’s <strong>Agent Substrate</strong> takes a different approach: <strong>keep a small pool of warm worker pods, and move agents on and off them as they wake up and go idle</strong>. This post walks through the full picture. It covers <strong>what the 1:1 model gets right, where it breaks, how Substrate works, and how security has to change once pods are shared over time</strong>.</p><p>This builds on <strong>Sameera Jayasoma’s excellent post</strong>, <a href="https://www.linkedin.com/pulse/agent-sandbox-vs-substrate-sameera-jayasoma-7o3dc/"><strong>Agent Sandbox vs Agent Substrate</strong></a>, which frames the two models using an analogy every backend engineer knows: OS threads versus virtual threads.</p><h3>The threads analogy</h3><p>For decades, server software mapped one request to one OS thread. <br>It worked, but OS threads are heavy. Each one reserves stack memory and costs the kernel scheduling work, so you run out of them long before you run out of CPU. Most of them are blocked on I/O anyway.</p><p><strong>Go’s</strong> <strong>goroutines</strong> and <strong>Java’s virtual threads</strong> fixed this with <strong>M:N scheduling</strong>. Millions of lightweight threads are multiplexed onto a handful of OS threads. When a virtual thread blocks on I/O, the runtime parks it and runs something else on that OS thread.</p><p>Agents on Kubernetes are going through the same shift:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3d3Gkq9ipqfKNHlRFnljvQ.png" /></figure><p>The analogy is useful for scheduling. As we’ll see later, it’s misleading when it comes to memory and security, and that difference matters.</p><h3>The 1:1 model: Agent Sandbox</h3><p><a href="https://github.com/kubernetes-sigs/agent-sandbox"><strong>Agent Sandbox</strong></a> is a Kubernetes SIG project that adds a Sandbox custom resource for running isolated, stateful, single-instance workloads. <br>It&#39;s designed for things like AI agent runtimes that need to execute untrusted, LLM-generated code.</p><p>A Sandbox gives you:</p><ul><li><strong>A stable identity.</strong> Each sandbox has a stable hostname and network identity.</li><li><strong>Persistent storage</strong> that survives restarts.</li><li><strong>Lifecycle management</strong>, including creation, scheduled deletion, pausing and resuming.</li><li><strong>Strong isolation</strong> by delegating to sandbox runtimes like gVisor or Kata Containers.</li></ul><p>It also has extensions for running at scale: SandboxTemplate for reusable definitions, SandboxWarmPool for pre-warmed sandboxes to cut startup time, and SandboxClaim for handing one out from the pool.</p><p>Google offers a managed version on GKE, which became <a href="https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate"><strong>generally available in May 2026</strong></a>. It uses <strong>gVisor</strong> by default, with <strong>Kata Containers</strong> supported through a pluggable interface. According to Google, <strong>its warm pool can hand out 300 sandboxes per second per cluster, with 90% of allocations completing within 200 ms</strong>.</p><p>The mental model is simple. <strong>The pod is the agent.</strong> They’re <strong>created together and they die together</strong>. And that makes security easy, because <strong>every Kubernetes security primitive already works at pod level</strong>:</p><ul><li>The pod’s service account is the agent’s identity.</li><li>A NetworkPolicy that selects the pod by label is the agent&#39;s firewall.</li><li>The pod’s volumes and Secrets are the agent’s files and credentials.</li></ul><p><strong>If you secure the pod, you’ve secured the agent</strong>. <br>That’s a genuinely good property, and it’s why 1:1 is the right default for many teams.</p><h3>Where 1:1 breaks down</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSlK8WCUmZwGI9OqS5J4hw.png" /></figure><blockquote>In the 1:1 model, every agent holds a pod whether it’s working or not. <br>In Substrate, only agents with work in hand hold a worker.</blockquote><p>The trouble starts when you do the arithmetic. Say you run 5,000 agents, and at any moment around 5% of them are actually doing something. <br>(That number is illustrative, but for chat-driven or human-in-the-loop agents it’s a reasonable ballpark.)</p><p><strong>With one pod per agent, you’re running 5,000 pods to serve about 250 agents’ worth of work</strong>. That creates several problems at once.</p><p><strong>1. You pay for idle memory.</strong> A waiting agent still holds its full memory: <br>the runtime, loaded libraries, the conversation context, maybe a browser. <br>CPU can be overcommitted, but memory that’s allocated stays allocated. You end up sizing the cluster for the total number of agents, not for how many are working.</p><p><strong>2. You hit Kubernetes’ own limits.</strong> Nodes have a cap on pods (110 by default on many distributions). Every pod needs an IP address, a scheduling decision, a kubelet sync, and state in etcd. Running tens of thousands of long-lived pods that mostly sleep puts pressure on all of those.</p><p><strong>3. You trade cost for latency.</strong> You can shut idle agents down to save money. But when the user comes back, you’re waiting on a pod to be scheduled, pulled and started, and then the agent has to rebuild its state. Warm pools help with the first part, but a warm pod handed to a returning agent is empty. The agent’s in-memory state is gone unless you’ve written your own save and reload logic.</p><p><strong>4. State lives in the wrong place.</strong> In the 1:1 model, an agent’s working state is tied to a specific pod. Move it, restart it, or scale it to zero, and that state has to go somewhere else first. Each agent team ends up building its own persistence.</p><p>None of this makes 1:1 wrong. It just means the model scales with <strong>the number of agents that exist</strong>, when what you really want is for it to scale with <strong>the number of agents that are busy</strong>.</p><h3>The M:N model: Agent Substrate</h3><p><a href="https://github.com/agent-substrate/substrate"><strong>Agent Substrate</strong></a> is an open-source, Kubernetes-native runtime that <strong>separates the agent’s lifecycle from the pod’s lifecycle</strong>.</p><p>Google <a href="https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate"><strong>introduced it in May 2026</strong></a> as a project for “ultra scale” agents, built on Agent Sandbox’s core capabilities plus a minimal control plane that works around some of Kubernetes’ limits. There’s also a managed version on <a href="https://docs.cloud.google.com/kubernetes-engine/ai-ml/about-agent-substrate"><strong>GKE</strong></a>.</p><p>It has four core concepts:</p><ul><li><strong>Actor:</strong> a single running instance of an agent, with its own state.</li><li><strong>ActorTemplate:</strong> a blueprint for creating actors.</li><li><strong>Worker:</strong> a sandboxed pod that an actor runs in.</li><li><strong>WorkerPool:</strong> a group of pre-started, idle workers, ready to receive actors.</li></ul><p>And a handful of components, all named with an ate prefix:</p><ul><li>ateapi (the control plane for actor lifecycle and routing),</li><li>atelet (a node-level DaemonSet),</li><li>atecontroller (the Kubernetes reconciler),</li><li>atenet (networking), and</li><li>ateom, which runs inside each worker and performs checkpoint and restore. ateom comes in two variants, ateom-gvisor and ateom-microvm, one for each sandbox type.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMBlnIi-FpKpe1fkY7iwEQ.png" /></figure><p>The GKE documentation describes four steps:</p><ol><li><strong>Route.</strong> Each incoming request names the actor it’s for.</li><li><strong>Resume.</strong> If that actor is suspended, Substrate claims a warm worker from the pool and restores the actor’s snapshot onto it.</li><li><strong>Execute.</strong> The request is routed to that worker and the actor handles the task.</li><li><strong>Suspend.</strong> When the actor finishes and goes idle, Substrate takes a fresh snapshot of its memory and files, stores it (in Cloud Storage on GKE), wipes the worker, and returns it to the pool.</li></ol><p>The key technology here is <strong>full-process snapshots</strong>. <br>Substrate uses <strong>gVisor’s native checkpoint</strong> and restore to <strong>capture the whole sandboxed process tree: memory, open files, and filesystem state</strong>. <br>An actor that resumes picks up exactly where it left off, and it doesn’t need to know it was ever frozen. It can also resume <strong>on any worker</strong>, not just the one it was on before.</p><h3>The payoff</h3><p>In <strong>Google’s demo </strong>in the <a href="https://github.com/agent-substrate/substrate"><strong>project’s official repository</strong></a>, <br><strong>Substrate</strong> <strong>multiplexed</strong> <strong>about 250 stateful actors across 8 pods</strong>, which is more than 30x oversubscription. State is fully preserved across hibernation, and the repository quotes <strong>resume operations under 500 ms</strong>.</p><p>Go back to the 5,000-agent example. With Substrate, you size the worker pool for about 250 concurrent actors plus some headroom. The other 4,750 agents cost you only storage for their snapshots.</p><h3>A detail that’s easy to get wrong</h3><p>It’s tempting to read “M:N” and picture many agents running inside one pod at the same time, the way many virtual threads share one heap. <br><strong>That’s not what happens.</strong></p><p>The Substrate architecture docs are clear that a <strong>worker hosts</strong> <strong>one actor at a time</strong>. Each Worker record tracks the actor it’s currently hosting, if any. <br>The “M:N” is <strong>time-multiplexing</strong>: many actors take turns on fewer workers. It isn’t several agents living in the same pod at once.</p><p>One more nuance, and it’s where the two kinds of waiting from the start of this post matter. Substrate suspends an actor when it has <strong>finished handling a request and gone idle</strong>, meaning it’s between requests. <strong>Short waits inside a task don’t count</strong>. If an agent is in the middle of a 30-second tool call, it’s still executing and still holding its worker. This is a real difference from virtual threads, which are parked in the middle of any blocking call.</p><p><strong>The savings come from the long waits between tasks</strong>, which is where most agents spend most of their time. It also means the <strong>way you build an agent affects how much Substrate can save</strong>. An agent that ends its turn and returns while it waits for human approval can be suspended. One that sits in a loop polling for that approval never goes idle, so it keeps its worker the whole time.</p><h3>Security when pods are shared over time</h3><p>In Substrate, only one agent runs in a worker pod at any moment, but the same pod hosts many different agents over its lifetime. That <strong>breaks the usual Kubernetes habit of securing the pod, so Substrate draws the security boundary around each actor instead</strong>. That idea has two parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJ5SU3_pyFSr4A5F7FNOrw.png" /></figure><h4>Security settings belong to the agent, not the pod</h4><p>In the 1:1 model, the pod and the agent are the same thing, so it’s fine to attach identity, network rules and credentials to the pod.</p><p>In Substrate, a worker pod is long-lived and generic. Over a day, the same pod might host dozens of different agents one after another. If identity and policy were attached to the pod, every agent that passed through it would get the same access. That would be a disaster.</p><p>So Substrate attaches them to the actor. When actor A is restored onto a worker, A’s credentials, network rules and files come with it. When A is suspended, they’re removed. The pod has no identity of its own. <br>That’s why the <a href="https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md"><strong>Substrate threat model</strong></a> requires that actor credentials include claims tying them to the specific worker and node the actor is scheduled on. The same agent keeps turning up on different pods, so its credentials need to prove where it is right now.</p><h4>Inside the pod, the agent doesn’t get the whole pod</h4><p>A worker pod has two parts. There’s ateom, <strong>Substrate&#39;s own trusted process</strong> that <strong>performs checkpoint, restore and wipe</strong>. Then there&#39;s the actor itself, running inside a <strong>gVisor or microVM sandbox</strong> nested within the pod.</p><p>The security boundary is drawn around that inner sandbox, not around the edge of the pod. Even if the agent runs malicious code that an LLM generated, it’s contained inside the sandbox. It can’t reach ateom, it can&#39;t interfere with its own wipe, and it can&#39;t read other agents&#39; snapshots. <br>The threat model puts it plainly: the sandbox lifecycle must be controlled from outside the sandbox.</p><h4>What the threat model requires</h4><p>The <a href="https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md"><strong>threat model</strong></a> lays out the rules that make pod sharing over time safe:</p><ul><li>Actors are always sandboxed using a hardened sandbox such as gVisor or a microVM.</li><li>All actor-specific worker state (processes, filesystem, env vars, mounted config, network policy and security policy) must be <strong>completely reset</strong> between actors.</li><li>Network policies deny ingress and egress by default, and only open access to specific actors when needed. No workload gets host network access.</li><li>Worker pods and actors have <strong>no direct access to snapshots</strong>, and a snapshot must be checked against a trusted digest or signature before it can be restored.</li></ul><p>The connection-pool analogy fits well here. A pooled database connection has to be reset (session variables, temp tables, the current role) before the next request uses it. A Substrate worker is a pooled sandbox, and the wipe is its reset.</p><h3>New problems Substrate introduces</h3><p>Substrate doesn’t make risk disappear. It moves it somewhere else. <br>In the 1:1 model, the main concern is two neighbours on a node escaping into each other. In Substrate, you also have to worry about:</p><p><strong>Reset correctness.</strong> The whole design depends on the wipe between actors being complete. A leftover file, a cached token or a stale network rule becomes a leak between tenants. The threat model says the suspend and resume cycle should be tested for every supported sandbox type, which is the right instinct. But it’s a requirement you have to verify, not something you get for free.</p><p><strong>Snapshots are sensitive data.</strong> A snapshot is a full memory image. If the agent had an API key, an OAuth token or a user’s private data loaded in memory, that’s in the snapshot. You should treat snapshot storage as having the same sensitivity as a secrets store: encrypt it, restrict access, sign snapshots, and don’t keep them longer than you need to.</p><p><strong>Sandbox escape is still possible.</strong> The threat model lists container breakout and access to the cloud metadata service as remaining risks. Both need layered defences, not a single fix.</p><p><strong>Not all state can be frozen.</strong> A process snapshot captures what’s inside the sandbox. It doesn’t capture the other side of a TCP connection, a session a remote browser service holds for you, or a lock held in an external system. Agents that hold long-lived external connections need to reconnect on resume, or be designed so that nothing important depends on those connections surviving.</p><p><strong>Maturity.</strong> The open-source project is still pre-1.0 (the <a href="https://github.com/agent-substrate/substrate/tags"><strong>latest release</strong></a> at the time of writing is v0.2.0, from September 2026), with no backward-compatibility guarantees yet. AX, the higher-level layer on top, <a href="https://github.com/google/ax"><strong>warns</strong></a> that it’s in heavy development and may introduce breaking changes. On GKE it’s currently offered for evaluation and non-production use, with production support available on an allowlist basis. There are also current limits, such as no GPU passthrough.</p><h3>What Substrate includes, and what it leaves to you</h3><p>Some write-ups describe Substrate as having no security, policy or observability features. That isn’t accurate. Its own documentation covers per-actor gVisor or microVM sandboxing, default-deny network policies, <a href="https://github.com/agent-substrate/substrate/blob/main/docs/network-egress.md"><strong>egress control</strong></a>, <a href="https://github.com/agent-substrate/substrate/blob/main/docs/authentication.md"><strong>authentication</strong></a> with trusted JWT providers, and an <a href="https://github.com/agent-substrate/substrate/blob/main/docs/observability.md"><strong>observability guide</strong></a> for logging, metrics and tracing.</p><p>What <strong>it doesn’t include is a gateway that understands AI traffic</strong>: <strong>policy at the level of individual MCP tools or A2A calls, rate limits per model, or routing between model providers</strong>. Its networking layer, atenet, routes requests to whichever worker an actor is on, using Envoy. Anything that needs to understand the content of agent traffic belongs in a separate layer, such as an AI gateway or a service mesh in front of Substrate.</p><h4>Filling the gap with an AI gateway</h4><p>This is the job of an AI gateway, and it’s where <a href="https://wso2.com/api-platform/ai-gateway/"><strong>WSO2 AI Gateway</strong></a><strong> </strong>fits well alongside Substrate. Substrate decides <strong><em>where</em></strong> an agent runs and keeps it isolated. The gateway governs <strong><em>what the agent says and calls</em></strong> once it’s running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IyuK9L6AD8-dc4W7ujZGeg.png" /></figure><p><em>Substrate handles placement and isolation. The gateway governs inbound agent calls and every outbound model and tool call.</em></p><p>WSO2 AI Gateway covers three kinds of agent traffic from one control plane:</p><ul><li><strong>Calls to models (LLM Gateway).</strong> The gateway routes requests across providers such as OpenAI, Azure, Anthropic, Gemini, Mistral and AWS Bedrock, with failover when a provider is down. It enforces token-based rate limits and cost budgets, and its semantic caching cuts repeated calls. Guardrails like PII masking, prompt validation and JSON Schema checks on outputs run in the same path, and it can plug in Azure Content Safety or AWS Bedrock Guardrails.</li><li><strong>Calls to tools (MCP Gateway).</strong> The gateway secures MCP servers with authentication, tool-level authorization and rate limits. It can also proxy external MCP servers your agents consume, and turn existing REST APIs into MCP tools without writing a new server.</li><li><strong>Calls between agents (A2A).</strong> The <a href="https://github.com/wso2/api-platform/releases/tag/ai-gateway/v2026.09.24"><strong>latest release</strong></a> added Agent-to-Agent protocol support. It publishes governed Agent Cards and applies authentication and rate limiting to traffic between agents.</li></ul><p>It also exports analytics through OpenTelemetry, so model, tool and agent usage lands in the same observability stack you already use for Substrate.</p><p>The two pieces combine neatly. Substrate’s egress control lets you deny all outbound traffic from actors except to the gateway. Every model call and tool call then goes through one governed path, however many actors you run and whichever worker they happen to be on. And because Substrate gives each actor its own credentials, the gateway can apply policy per agent, not per pod.</p><p>WSO2 AI Gateway is part of the open-source <a href="https://github.com/wso2/api-platform"><strong>WSO2 API Platform</strong></a> (Apache 2.0). It’s built on Envoy and runs on Kubernetes, so it sits naturally next to Substrate in the same cluster.</p><p><em>Disclosure: I work as a </em><strong><em>Forward Deployed Engineer at WSO2</em></strong><em>.</em></p><h3>Which one should you use?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hofgMWeon0-0PT6jRb8TZQ.png" /></figure><p>They’re not really rivals. They’re two points on the same spectrum, just like OS threads and virtual threads coexist in any modern runtime.</p><blockquote>Some workloads want a dedicated pod. Many agent workloads, especially ones that wait for people, will be much cheaper on a shared pool.</blockquote><h3>Closing thoughts</h3><p>The 1:1 model gave us a clean, secure way to run agents on Kubernetes, and for a small fleet it’s still the simplest choice. But agents are I/O-bound by nature, and once you have thousands of them, dedicating a pod to each one means paying for a lot of waiting.</p><p>Agent Substrate brings the virtual-thread idea to agents: park the idle ones, keep a small pool of warm workers, and bring agents back in under half a second, on whichever worker is free. The price is that security can no longer just sit on the pod. It has to travel with each agent, and the hand-off between agents has to be airtight.</p><p>The open question for platform teams is where to integrate. You can build directly on Substrate, or go through <a href="https://github.com/google/ax"><strong>AX</strong></a>, which Google describes as its open agentic orchestration runtime and which runs on top of Substrate. Sameera raises exactly this question for <a href="https://openchoreo.dev/"><strong>OpenChoreo</strong></a> at the end of the post, and it’s one every internal developer platform will need to answer soon.</p><h3>References</h3><p><strong>Primary sources (Google and the projects themselves)</strong></p><ol><li>Google Cloud, <a href="https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate"><strong>Bringing you Agent Sandbox on GKE and Agent Substrate</strong></a></li><li><a href="https://github.com/agent-substrate/substrate"><strong>agent-substrate/substrate</strong></a>, GitHub (README, demo and <a href="https://github.com/agent-substrate/substrate/tags">releases</a>)</li><li><a href="https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md"><strong>Substrate architecture</strong></a></li><li><a href="https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md"><strong>Substrate threat model</strong></a></li><li>Google Cloud documentation — <a href="https://docs.cloud.google.com/kubernetes-engine/ai-ml/about-agent-substrate"><strong>About GKE Agent Substrate</strong></a></li><li>Agent Executor — <a href="https://github.com/google/ax"><strong>google/ax</strong></a></li><li><a href="https://github.com/kubernetes-sigs/agent-sandbox"><strong>kubernetes-sigs/agent-sandbox</strong></a></li><li>Google’s application kernel for containers — <a href="https://gvisor.dev/"><strong>gVisor</strong></a></li></ol><p><strong>Further reading</strong></p><ol><li>Sameera Jayasoma, <a href="https://www.linkedin.com/pulse/agent-sandbox-vs-substrate-sameera-jayasoma-7o3dc/"><strong>Agent Sandbox vs Agent Substrate</strong></a>, LinkedIn</li><li><a href="https://www.solo.io/topics/ai-infrastructure/how-google-agent-substrate-works"><strong>How Google Agent Substrate Works: 250 Agents on 8 Pods</strong></a>, Solo.io</li><li><a href="https://www.cloudnativedeepdive.com/agent-substrate-the-agentic-ai-isolation-layer-on-k8s/"><strong>Agent Substrate: The Agentic AI Isolation Layer On K8s</strong></a>, Cloud Native Deep Dive</li><li><a href="https://kagent.dev/blog/kagent-agent-substrate-sandboxes"><strong>Kagent and Agent Substrate: Running AI Agent Sandboxes on Kubernetes</strong></a></li><li>CNCF blog: <a href="https://www.cncf.io/blog/2026/07/07/why-sandboxing-your-agent-is-not-enough/"><strong>Why sandboxing your agent is not enough</strong></a></li></ol><p><strong>WSO2 AI Gateway</strong></p><ol><li><a href="https://wso2.com/api-platform/ai-gateway/">WSO2 AI Gateway</a></li><li><a href="https://github.com/wso2/api-platform">wso2/api-platform</a></li><li><a href="https://github.com/wso2/api-platform/releases/tag/ai-gateway/v2026.09.24">AI Gateway release v2026.09.24</a> (A2A support, OpenTelemetry analytics)</li><li><a href="https://wso2.com/api-platform/learn/ai-gateway-security/">AI Gateway Security: Guardrails for LLM Traffic</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5b0a0eb8d31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/one-pod-per-agent-doesnt-scale-here-s-how-agent-substrate-fixes-it-f5b0a0eb8d31">One Pod per Agent Doesn’t Scale. Here’s How Agent Substrate Fixes It</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Catching Helm Drift Before It Catches You]]></title>
            <link>https://techexpertise.medium.com/catching-helm-drift-before-it-catches-you-4c5c18d8cdc8?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c5c18d8cdc8</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[drift-detection]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Thu, 16 Apr 2026 09:02:05 GMT</pubDate>
            <atom:updated>2026-04-16T09:02:05.665Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BgdCmlFXP_hRfmAO97Kylw.png" /></figure><p>In this blog, we will discuss how to <strong>detect, prevent, and auto-remediate unauthorized changes</strong> in your <strong>Kubernetes cluster</strong> using <strong>GitOps</strong>.</p><h3>The Problem: Helm’s Blind Spot</h3><p>If you’re managing Kubernetes deployments with Helm, you’ve likely encountered this scenario:</p><ol><li>You carefully define your infrastructure in Helm charts</li><li>You deploy with helm installor helm upgrade</li><li>Someone on the team runs a quick kubectl editor kubectl applyto fix an urgent issue</li><li>That manual change is never persisted back to the Helm chart</li><li>The next helm upgradesilently overwrites the manual change — or worse, the cluster silently drifts from your source of truth</li></ol><blockquote><strong>Helm has no built-in mechanism to detect this drift</strong>. It treats the chart as the source of truth and simply applies it. It <strong>doesn’t compare what’s actually running in the cluster against what the chart defines</strong>.</blockquote><p>This is a critical gap in production environments where:</p><ul><li>Multiple team members have kubectl access</li><li>Hotfixes are applied directly to the cluster under pressure</li><li>Compliance requires knowing the exact state of your deployments</li><li>You need an audit trail of who changed what and when</li></ul><h3>The Solution: ArgoCD and GitOps</h3><p><a href="https://argo-cd.readthedocs.io/"><strong>ArgoCD</strong></a> is a CNCF Graduated project that implements the <strong>GitOps</strong> pattern for Kubernetes. It continuously monitors your Git repository (containing your Helm charts) and compares it against the live cluster state. <br>When drift is detected, it:</p><ul><li>Marks resources as <strong>OutOfSync</strong></li><li>Shows you the <strong>exact diff</strong> between desired and live state</li><li>Optionally <strong>auto-heals</strong> by reverting unauthorized changes</li></ul><p>Let’s build this from scratch.</p><h3>Architecture Overview</h3><pre>+------------------+       +------------------+       +------------------+<br>|                  |       |                  |       |                  |<br>|   Git Repo       | &lt;---- |   ArgoCD         | ----&gt; |   K8s Cluster    |<br>|   (Helm Charts)  |       |   (Controller)   |       |   (Live State)   |<br>|                  |       |                  |       |                  |<br>+------------------+       +------------------+       +------------------+<br>     Source of Truth         Continuous Sync          Actual State<br>                             &amp; Drift Detection</pre><p>ArgoCD sits between your Git repository and your Kubernetes cluster. It:</p><ol><li><strong>Watches</strong> your Git repo for changes</li><li><strong>Renders</strong> the Helm chart with your values</li><li><strong>Compares</strong> the rendered manifests against live cluster state</li><li><strong>Reports</strong> any differences (drift)</li><li><strong>Syncs</strong> when you approve (or automatically, if configured)</li></ol><h3>Prerequisites</h3><ul><li>A running Kubernetes cluster (minikube, kind, EKS, GKE, AKS, etc.)</li><li>kubectl configured to access your cluster</li><li>helm CLI installed</li><li>A Git repository (GitHub, GitLab, Bitbucket)</li></ul><h3>Step 1: Install ArgoCD</h3><p>Execute the following commands to install the ArgoCD,</p><pre># Create the argocd namespace<br>kubectl create namespace argocd<br><br># Install ArgoCD using the official manifests<br>kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml<br><br># Wait for all ArgoCD pods to be ready<br>kubectl wait --for=condition=ready pod --all -n argocd --timeout=300s</pre><p>Verify the installation:</p><pre>kubectl get pods -n argocd</pre><p>Expected output:</p><pre>NAME                                               READY   STATUS    RESTARTS   AGE<br>argocd-application-controller-0                    1/1     Running   0          64s<br>argocd-applicationset-controller-f869b9f85-tjc6t   1/1     Running   0          64s<br>argocd-dex-server-656bfff4b5-crzvf                 1/1     Running   0          64s<br>argocd-notifications-controller-9669744c8-lzwn6    1/1     Running   0          64s<br>argocd-redis-7c7678cd79-7x6w7                      1/1     Running   0          64s<br>argocd-repo-server-779879c89d-x4fzt                1/1     Running   0          64s<br>argocd-server-6995588757-hlwnl                     1/1     Running   0          64s</pre><h4>Install the ArgoCD CLI</h4><pre># macOS<br>brew install argocd<br><br># Linux<br>curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64<br>sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd<br>rm argocd-linux-amd64</pre><h4>Access the ArgoCD UI</h4><pre># Port-forward the ArgoCD server<br>kubectl port-forward svc/argocd-server -n argocd 8080:443 &amp;<br><br># Get the initial admin password<br>argocd admin initial-password -n argocd<br><br># Optionally update the admin password<br># argocd admin update-password --current-password &lt;current-password&gt; --new-password &lt;new-password&gt;<br><br># Login via CLI (required for the `argocd app` commands used later in this post)<br>argocd login localhost:8080 --username admin --password &lt;password-from-above&gt; --insecure</pre><p>Access the ArgoCD UI at <a href="https://localhost:8080/">https://localhost:8080</a> (login to the UI again using the admin username and password extracted above)</p><h3>Step 2: Create a Sample Helm Chart</h3><p>Let’s create a simple Helm chart that we’ll use to demonstrate drift detection. If there is an existing Helm chart, you can commit it to Git and use it to integrate with ArgoCD as well.</p><p>Create this structure in your Git repository:</p><h4>Chart.yaml</h4><pre># sample-app/Chart.yaml<br>apiVersion: v2<br>name: sample-app<br>description: A sample application to demonstrate ArgoCD drift detection<br>type: application<br>version: 1.0.0<br>appVersion: &quot;1.0.0&quot;</pre><h4>values.yaml</h4><pre># sample-app/values.yaml<br>replicaCount: 3<br><br>image:<br>  repository: nginx<br>  tag: &quot;1.25&quot;<br>  pullPolicy: IfNotPresent<br><br>service:<br>  type: ClusterIP<br>  port: 80<br><br>resources:<br>  limits:<br>    cpu: 200m<br>    memory: 256Mi<br>  requests:<br>    cpu: 100m<br>    memory: 128Mi<br><br>configMap:<br>  appMode: &quot;production&quot;<br>  logLevel: &quot;info&quot;<br>  maxConnections: &quot;100&quot;</pre><h4>templates/deployment.yaml</h4><pre># sample-app/templates/deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: {{ .Release.Name }}-deployment<br>  labels:<br>    app: {{ .Release.Name }}<br>    managed-by: argocd<br>spec:<br>  replicas: {{ .Values.replicaCount }}<br>  selector:<br>    matchLabels:<br>      app: {{ .Release.Name }}<br>  template:<br>    metadata:<br>      labels:<br>        app: {{ .Release.Name }}<br>    spec:<br>      containers:<br>        - name: {{ .Chart.Name }}<br>          image: &quot;{{ .Values.image.repository }}:{{ .Values.image.tag }}&quot;<br>          imagePullPolicy: {{ .Values.image.pullPolicy }}<br>          ports:<br>            - containerPort: 80<br>              protocol: TCP<br>          resources:<br>            {{- toYaml .Values.resources | nindent 12 }}<br>          envFrom:<br>            - configMapRef:<br>                name: {{ .Release.Name }}-config</pre><h4>templates/service.yaml</h4><pre># sample-app/templates/service.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: {{ .Release.Name }}-service<br>  labels:<br>    app: {{ .Release.Name }}<br>spec:<br>  type: {{ .Values.service.type }}<br>  ports:<br>    - port: {{ .Values.service.port }}<br>      targetPort: 80<br>      protocol: TCP<br>      name: http<br>  selector:<br>    app: {{ .Release.Name }}</pre><h4>templates/configmap.yaml</h4><pre># sample-app/templates/configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: {{ .Release.Name }}-config<br>  labels:<br>    app: {{ .Release.Name }}<br>data:<br>  APP_MODE: {{ .Values.configMap.appMode | quote }}<br>  LOG_LEVEL: {{ .Values.configMap.logLevel | quote }}<br>  MAX_CONNECTIONS: {{ .Values.configMap.maxConnections | quote }}</pre><p>Push this chart to your Git repository:</p><pre>cd your-repo<br>git add sample-app/<br>git commit -m &quot;Add sample Helm chart for ArgoCD demo&quot;<br>git push origin main</pre><p>Sample Helm chart can be found <a href="https://github.com/Seralahthan/ArgoCD-Helm-Drift-Detection">here</a>.</p><h3>Step 3: Register Your Git Repository with ArgoCD</h3><pre># For public repos<br>argocd repo add https://github.com/&lt;your-org&gt;/&lt;your-repo&gt;.git<br><br># For private repos (HTTPS)<br>argocd repo add https://github.com/&lt;your-org&gt;/&lt;your-repo&gt;.git \<br>  --username &lt;git-username&gt; \<br>  --password &lt;git-token&gt;</pre><p>In my case, i have hosted my Helm chart on a public git repository so i will be running the following command:</p><pre>argocd repo add https://github.com/Seralahthan/ArgoCD-Helm-Drift-Detection.git</pre><h3>Step 4: Create an ArgoCD Application</h3><p>We can create the application via CLI or declaratively. In this blog we will be using the CLI way.</p><pre>argocd app create drift-demo \<br>  --repo https://github.com/Seralahthan/ArgoCD-Helm-Drift-Detection.git \<br>  --path sample-app \<br>  --revision main \<br>  --dest-server https://kubernetes.default.svc \<br>  --dest-namespace demo \<br>  --sync-policy automated \<br>  --auto-prune \<br>  --self-heal \<br>  --sync-option CreateNamespace=true</pre><p>Following are some of the key configurations used when creating the ArgoCD application:</p><blockquote>--self-heal=&gt; This is the <strong><em>drift detection fix</em></strong>. <strong><em>ArgoCD will automatically revert any manual changes made to the cluster</em></strong><em>.</em></blockquote><blockquote>--auto-prune=&gt; If a resource is removed from Git, ArgoCD will delete it from the cluster too (otherwise orphaned resources would remain).</blockquote><blockquote>--sync-policy automated =&gt; Automatically sync when ArgoCD detects the Git repo has changed (no manual sync needed)</blockquote><blockquote>--revision main =&gt; Tracks the main branch. Can also pin to a tag or commit SHA.</blockquote><h3>Step 5: Verify the Deployment</h3><pre># Check ArgoCD application status<br>argocd app get drift-demo<br><br># Check the deployed resources<br>kubectl get all -n demo</pre><p>Expected ArgoCD output:</p><pre>Name:               argocd/drift-demo<br>Project:            default<br>Server:             https://kubernetes.default.svc<br>Namespace:          demo<br>URL:                https://localhost:8080/applications/drift-demo<br>Source:<br>- Repo:             https://github.com/Seralahthan/ArgoCD-Helm-Drift-Detection.git<br>  Target:           main<br>  Path:             sample-app<br>SyncWindow:         Sync Allowed<br>Sync Policy:        Automated (Prune)<br>Sync Status:        Synced to main (ec02eb0)<br>Health Status:      Healthy</pre><p>Execute the following command to see the flags set on the SyncPolicy ,</p><pre>argocd app get drift-demo -o yaml | yq &#39;.spec.syncPolicy&#39;<br><br>automated:<br>  prune: true<br>  selfHeal: true<br>syncOptions:<br>  - CreateNamespace=true</pre><p>Now that we have successfully synced our Git Helm chart with the ArgoCD, let’s stimulate some scenarios and check out the Drift Detection.</p><h3>Step 6: Demonstrate Drift Detection</h3><p>Now let’s simulate the exact problem we described — someone makes a manual change directly to the cluster.</p><h4>Scenario 1: Manual Replica Change</h4><p>An engineer scales the deployment directly:</p><pre># Manual change — NOT through Git<br>kubectl scale deployment drift-demo-deployment -n demo --replicas=5</pre><p>What happens next:</p><ol><li>ArgoCD detects the replica count changed from 3 (in Git) to 5 (in live)</li><li>Since selfHeal: true is enabled, <strong>ArgoCD automatically reverts </strong>it back to 3</li><li>The event is logged in ArgoCD’s history</li></ol><p>Check the ArgoCD UI or CLI to see the drift detection:</p><pre># View the diff ArgoCD detected<br>argocd app diff drift-demo<br><br># View sync history (shows the auto-heal event)<br>argocd app history drift-demo<br><br># View the self-heal events with this command<br>kubectl get events -n argocd --field-selector involvedObject.name=drift-demo --sort-by=&#39;.lastTimestamp&#39;</pre><p>Following is an example of events observed:</p><pre>LAST SEEN   TYPE     REASON               OBJECT                   MESSAGE<br>66s         Normal   OperationStarted     application/drift-demo   Initiated automated sync to [ec02eb043ea2f44b42550b87fb7bf57a7fb85856]<br>66s         Normal   ResourceUpdated      application/drift-demo   Updated sync status: Synced -&gt; OutOfSync<br>66s         Normal   ResourceUpdated      application/drift-demo   Updated health status: Healthy -&gt; Progressing<br>65s         Normal   OperationCompleted   application/drift-demo   Partial sync operation to ec02eb043ea2f44b42550b87fb7bf57a7fb85856 succeeded<br>65s         Normal   ResourceUpdated      application/drift-demo   Updated sync status: OutOfSync -&gt; Synced<br>65s         Normal   ResourceUpdated      application/drift-demo   Updated health status: Progressing -&gt; Healthy</pre><h4>Scenario 2: Manual ConfigMap Edit</h4><p>Someone edits the ConfigMap directly:</p><pre># Manual change — NOT through Git<br>kubectl edit configmap drift-demo-config -n demo<br># Change LOG_LEVEL from &quot;info&quot; to &quot;debug&quot;</pre><p>ArgoCD response: Detects the ConfigMap drift and reverts LOG_LEVEL back to &quot;info&quot; within seconds.</p><h4>Scenario 3: Resource Removed from Git (Prune)</h4><p>The prune: true option handles resources that were previously deployed by ArgoCD but then removed from Git.</p><p>For example, if you delete configmap.yaml from your Helm chart templates and push to Git, ArgoCD will automatically delete drift-demo-config from the cluster.</p><blockquote>Note: prune <strong><em>does not delete resources created manually outside of ArgoCD</em></strong> (e.g., via kubectl create). ArgoCD only tracks and manages resources that are part of its rendered Helm output. Manually created &quot;rogue&quot; resources are invisible to ArgoCD. To prevent unauthorized resource creation, use <strong>Kubernetes RBAC</strong> or <strong>admission controllers like OPA Gatekeeper / Kyverno</strong>.</blockquote><h4>Scenario 4: Without Self-Heal (Alert-Only Mode)</h4><p>If you prefer to be <strong><em>notified</em></strong> about drift <strong><em>without automatic remediation</em></strong>, <strong>disable self-heal</strong>:</p><pre>argocd app set drift-demo --self-heal=false --auto-prune=false</pre><p>Now ArgoCD will mark the application as <strong>OutOfSync</strong> and show you the <strong>diff</strong>, but it <strong>won’t make changes</strong>. This is useful for:</p><ul><li>Audit and compliance (detecting unauthorized changes)</li><li>Teams transitioning to GitOps gradually</li><li>Environments where manual overrides are occasionally intentional</li></ul><p>Trigger a manual replica change and check the drift status:</p><pre>kubectl scale deployment drift-demo-deployment -n demo --replicas=5<br><br>argocd app get drift-demo<br><br># Output will show:<br># Sync Status: OutOfSync from main (ec02eb0)<br># Health Status: Healthy</pre><p>View exactly what drifted:</p><pre>argocd app diff drift-demo<br><br>===== apps/Deployment demo/drift-demo-deployment ======<br>124c124<br>&lt;   replicas: 5<br>---<br>&gt;   replicas: 3</pre><blockquote>Note: In most of the real world scenarios, <strong>engineers are allowed to manually modify K8s resources</strong> for various purposes and <strong>often forgot to sync those changes with Git</strong>, with <strong>ArgoCD’s Alert mode</strong> such drifts can be captured and synced to the Git repository without overriding and loosing those changes.</blockquote><p>In ArgoCD’s Alert mode, we can view the details of the drift status and the diff through the <strong>ArgoCD UI</strong> as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T8omImK29-xOLD7DzPIYhg.png" /><figcaption>OutOfSync Detected in Deployment</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ehzQngUjuQNoHqJPUqJEhg.png" /><figcaption>Config Diff which caused the Drift</figcaption></figure><h3>Summary: The Complete GitOps Workflow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzc8V-feS_9-Qzx940HdCg.png" /><figcaption>ArgoCD drift detection flow — from Git commit to auto-revert or manual alert</figcaption></figure><h3>Limitations of ArgoCD’s Drift Detection</h3><p>While ArgoCD is excellent at detecting and remediating drift, it’s important to understand what it <strong>can</strong> and <strong>cannot</strong> track:</p><h4>What ArgoCD Handles Well</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IoSsJe6-2FQ4YVKoAkuYmQ.png" /></figure><h4>What ArgoCD Cannot Track</h4><p><strong>Rogue resource creation</strong> — If someone creates a new resource directly in the cluster that doesn’t exist in Git:</p><pre>kubectl run debug-pod --image=busybox -n demo<br>kubectl create configmap hotfix-config -n demo --from-literal=key=value</pre><p>ArgoCD <strong>will not detect these resources</strong>. It only tracks resources that are defined in your Git repository. These rogue resources will persist silently in the cluster with no visibility in the ArgoCD UI.</p><blockquote>This is a <strong>common real-world scenario — an engineer debugging a production issue creates temporary resources, applies a quick fix directly to the cluster, and forgets to clean up or sync those changes back to Git</strong>.</blockquote><h4>Mitigating Rogue Resource Creation</h4><p>There are a few strategies to address this gap:</p><ul><li><strong>Kubernetes RBAC</strong> — Restrict write access to namespaces so that only the ArgoCD service account can create or modify resources. This prevents the problem at the source but may be too restrictive for teams that need direct cluster access for debugging.</li><li><strong>Admission Controllers (Kyverno/OPA Gatekeeper) in Audit Mode</strong> — Rather than blocking engineers, these tools can log policy violations when unmanaged resources are created, giving you visibility without sacrificing debugging flexibility.</li><li><strong>Periodic Auditing</strong> — Query for resources not managed by ArgoCD using label selectors.</li></ul><p>We will explore <strong>Mitigating Rogue Resource Creation</strong>, <strong>RBAC-based prevention</strong> and <strong>Drift notification strategies</strong> in detail in the next part of this series.</p><p>Stay tuned!</p><p>Hope you enjoyed the blog and got something to take away ✌️</p><p>Thank you for Reading!<br>Cheers 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c5c18d8cdc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Model’s Safety Guardrails Can Be Removed With a Single Math Operation.]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techexpertise.medium.com/your-ai-models-safety-guardrails-can-be-removed-with-a-single-math-operation-096843f41725?source=rss-5cc3faae5c14------2"><img src="https://cdn-images-1.medium.com/max/1500/1*I6o87ITsIHXvA58BpCND7w.png" width="1500"></a></p><p class="medium-feed-snippet">A deep dive into AI alignment, RLHF, abliteration, and the tools that are exposing how fragile current safety mechanisms really are.</p><p class="medium-feed-link"><a href="https://techexpertise.medium.com/your-ai-models-safety-guardrails-can-be-removed-with-a-single-math-operation-096843f41725?source=rss-5cc3faae5c14------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techexpertise.medium.com/your-ai-models-safety-guardrails-can-be-removed-with-a-single-math-operation-096843f41725?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/096843f41725</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rlhf]]></category>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-alignment]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Thu, 12 Mar 2026 06:18:44 GMT</pubDate>
            <atom:updated>2026-03-12T06:18:44.631Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Validation Checks Performed during SSL/TLS handshake — PKIX CertPathValidator]]></title>
            <link>https://techexpertise.medium.com/validation-checks-performed-during-ssl-tls-handshake-pkix-certpathvalidator-24d3f91bd3e2?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/24d3f91bd3e2</guid>
            <category><![CDATA[pkix]]></category>
            <category><![CDATA[certpathvalidator]]></category>
            <category><![CDATA[certificate]]></category>
            <category><![CDATA[tls-handshake]]></category>
            <category><![CDATA[openjdk]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sun, 27 Apr 2025 05:41:46 GMT</pubDate>
            <atom:updated>2025-04-27T05:41:46.026Z</atom:updated>
            <content:encoded><![CDATA[<h3>Validation Checks Performed during SSL/TLS handshake — PKIX CertPathValidator</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XyXZu3_g8vBB8JCXjtsfag.png" /></figure><p>We’ve all been there — setting up HTTPS for a service, only to get smacked with a “certificate validation error”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*Iy116FoGE7F1QytKwqbYkg.gif" /></figure><p>In this post, let’s break down the <strong>critical validation checks</strong> happening during the SSL/TLS handshake — what your browser, API client, or server is secretly doing to make sure that you’re truly connecting to the right entity — and that no malicious actor is trying to fool you.</p><blockquote>Spoiler: It’s way more than just “is the cert valid?”</blockquote><p>There are different validation checks performed by a client when connecting to a backend over HTTPS during the SSL/TLS handshake.</p><p>Following are the <strong>key validation checks</strong> performed during the SSL/TLS handshake to <strong>ensure the</strong> <strong>identity and integrity of the server’s certificate</strong>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57823bb10975751177231818e6229437/href">https://medium.com/media/57823bb10975751177231818e6229437/href</a></iframe><p>Above mentioned validation checks are the main validation checks performed during the SSL/TLS handshake. Other than the Hostname verification check other validations are performed by the <strong>OpenJDK’s PKIX CertPathValidator</strong>.</p><p>Following table gives a <strong>summarised view</strong> of the <strong>different validation checks</strong> and the <strong>phase of the SSL/TLS handshake</strong> they are carried out and the <strong>purpose of the check</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fe94a9a4a53160f07bbb97b0b4e5a2c/href">https://medium.com/media/0fe94a9a4a53160f07bbb97b0b4e5a2c/href</a></iframe><p>Each of these validation checks perform a crucial role completing a successful SSL/TLS handshake. We will discuss some of the validation checks in depth in the upcoming series of blog.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F25270366&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fawkward-handshake-chess-tournament-gif-25270366&amp;image=https%3A%2F%2Fmedia.tenor.com%2FvKD2KFUZIcEAAAAN%2Fawkward-handshake.png&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b891cc66364e0dbed4fb8a377cb830a/href">https://medium.com/media/6b891cc66364e0dbed4fb8a377cb830a/href</a></iframe><p>When two systems try to connect securely, it’s not just a polite handshake — it’s a full-blown interrogation happening under the hood.</p><p>Hope you enjoyed the blog and got something to take away ✌️</p><p>Thank you for Reading!<br>Cheers 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24d3f91bd3e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enabling SSL Debug Logs for WSO2 Products Running on Servers and Containers]]></title>
            <link>https://techexpertise.medium.com/enabling-ssl-debug-logs-for-wso2-products-running-on-servers-and-containers-5ec744cf00a8?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ec744cf00a8</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[ssl-debug]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[servers]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Mon, 24 Mar 2025 05:28:32 GMT</pubDate>
            <atom:updated>2025-03-24T05:28:32.943Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9RyJfET6ioY6l1IyhqzTNA.png" /></figure><p>In this blog we are going to explore,</p><ul><li>The need for enabling SSL debug logs in WSO2 Products</li><li>Enabling SSL debug logs in <strong>Server-based deployments</strong></li><li>Enabling SSL debug logs in <strong>Containerised deployments</strong></li></ul><h3>Need for Enabling SSL Debug Logs in WSO2 Products</h3><p>Enabling SSL debug logs in WSO2 products is crucial for <strong>observing,</strong> <strong>diagnosing and resolving issues related to SSL/TLS communication</strong> between the WSO2 server and external systems.</p><p>SSL debug logs help us <strong>debug ssl handshake failures</strong> caused by,</p><ul><li>Incompatible TLS versions</li><li>Missing or incorrect certificates (expired, revoked, or un-trusted certificates) in the trust store</li><li>Cipher suite mismatches etc.</li></ul><h3>Enabling SSL debug logs in Server-based deployments of WSO2 Products</h3><p>We can easily enable SSL debug logs by leveraging <strong>JSSE-specific dynamic debug tracing support</strong> using the javax.net.debug system property.</p><p>In order to enable SSL debug logs we need to pass the -Djavax.net.debug startup option as a <strong>JVM parameter</strong> along with the <strong>WSO2 Product startup script</strong></p><pre>For Linux Users:<br>Run the following command:<br>sh ${WSO2_SERVER_HOME}/bin/api-manager.sh -Djavax.net.debug=ssl:handshake</pre><pre>For Windows Users:<br>Run the following command: <br><br>${WSO2_SERVER_HOME}/bin/api-manager.bat -Djavax.net.debug=ssl:handshake</pre><blockquote>Note:<br>SSL Debug logs will not be added to the <strong>wso2carbon.log</strong> file in the <strong>&lt;WSO2_SERVER_HOME&gt;/repository/logs</strong> folder.<br>Instead they will be displayed in the <strong>standard output (stdout)</strong>.</blockquote><p>To save these SSL debug logs to a file there are different options available.</p><h4>Using nohup command (for Linux Users)</h4><p>We can use nohup command and direct, <strong>both standard output (</strong>stdout<strong>)</strong> and <strong>standard error (</strong>stderr<strong>)</strong> to nohup.out file,</p><pre>nohup sh ${WSO2_SERVER_HOME}/bin/api-manager.sh -Djavax.net.debug=ssl:handshake &amp;<br># nohup ${command} &amp;<br># no hang-up prevents the process from being terminated when the session ends.<br># The &amp; at the end runs the process in the background.<br># By default, output is redirected to nohup.out if not explicitly specified.</pre><h4>Redirecting stdout and stderr to a file (for Linux and Windows Users)</h4><p>We can redirect the <strong>standard output (stdout)</strong> and <strong>standard error (stderr)</strong> to the <strong>ssl-debug.log</strong> file as following,</p><pre>sh ${WSO2_SERVER_HOME}/bin/api-manager.sh -Djavax.net.debug=ssl:handshake &gt;&gt; ${WSO2_SERVER_HOME}/repository/logs/ssl-debug.log 2&gt;&amp;1<br># This will write the ssl debug logs to the ssl-debug.log file.</pre><h3>Enabling SSL debug logs in Containerised deployments of WSO2 Products</h3><p>In a containerised deployment like Docker/Kubernetes, there are multiple ways we can pass the <strong>-Djavax.net.debug=ssl:handshake”</strong> JVM parameter to the product startup script,</p><ul><li>Adding to the <strong>docker-entrypoint.sh</strong></li><li>As a <strong>Shell Script argument</strong></li><li>As a <strong>“JAVA_OPTS” environment variable</strong></li></ul><h4>Adding to the docker-entrypoint.sh</h4><p>We can add the JVM parameter to the default docker-entrypoint.sh file, but we need to <strong>rebuild the docker image</strong> Or have a <strong>config map</strong> to <strong>mount the updated docker-entrypoint.sh into the APIM pod</strong> if it is a Kubernetes-based deployment.</p><h4>As a Shell Script argument</h4><p>The default docker-entrypoint.sh file is implemented to accept the shell script argument to start the server, it can be used to pass the JVM parameters during the server startup.</p><p>Following is how the docker-entrypoint.sh file of <strong>APIM docker image</strong> looks,</p><pre># start WSO2 Carbon server<br>echo &quot;Start WSO2 Carbon server&quot; &gt;&amp;2<br>if [[ -z &quot;${PROFILE_NAME}&quot; ]]<br>then<br>  # start the server with the provided startup arguments<br>  sh ${WSO2_SERVER_HOME}/bin/api-manager.sh &quot;$@&quot;<br>else<br>  # start the server with the specified profile and provided startup arguments<br>  sh ${WSO2_SERVER_HOME}/bin/api-manager.sh -Dprofile=${PROFILE_NAME} &quot;$@&quot;<br>fi</pre><p><strong>Docker: Enable SSL Debug logs with Shell Script</strong></p><p>So, we can pass the JVM parameter as below,</p><pre>docker run -it --name apim -p 8280:8280 -p 8243:8243 -p 9443:9443 docker.wso2.com/wso2am:4.2.0 &quot;-Djavax.net.debug=ssl:handshake&quot;</pre><p><strong>Kubernetes: Enable SSL Debug logs with Shell Script</strong></p><p>In a Kubernetes-based deployment, we can pass the JVM parameter that needs to inject into the startup command as a shell script argument in the deployment.yamlfile using the args variable as shown below,</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: wso2apim<br>spec:<br>  template:<br>    spec:<br>      containers:<br>        - name: wso2apim<br>          image: docker.wso2.com/wso2am:4.2.0<br>          args: [&quot;-Djavax.net.debug=ssl:handshake&quot;]</pre><h4>As a “JAVA_OPTS” environment variable</h4><p>JAVA_OPTS is an <strong>environment variable</strong> that we can set to pass custom settings to the Java Virtual Machine (JVM).</p><p><strong>Docker: Enable SSL Debug logs with “JAVA_OPTS” environment variable</strong></p><p>We can pass the JVM parameter as below using the JAVA_OPTSenvironment variable,</p><pre>docker run -it --name apim \<br>  -p 8280:8280 -p 8243:8243 -p 9443:9443 \<br>  -e JAVA_OPTS=&quot;-Djavax.net.debug=ssl:handshake&quot; \<br>  docker.wso2.com/wso2am:4.2.0</pre><p><strong>Kubernetes: Enable SSL Debug logs with “JAVA_OPTS” environment variable</strong></p><p>Similarly, in a K8s-based deployment, we can pass the JVM parameter as an environment variable in the deployment.yaml file using the env variable as shown below,</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: wso2apim<br>spec:<br>  template:<br>    spec:<br>      containers:<br>        - name: wso2apim<br>          image: docker.wso2.com/wso2am:4.2.0<br>          env:<br>            - name: JAVA_OPTS<br>              value: &quot;-Djavax.net.debug=ssl:handshake&quot;</pre><blockquote>Note:<br>In both scenarios of shell script and “JAVA_OPTS” environment variable options, ssl debug logs can be found in the <strong>container logs</strong>. In case of K8s the logs will be lost if the Pod is deleted and replaced.</blockquote><p>In order to persist the SSL debug logs we can write it to a log file similar to what we did in the Server-based deployment.</p><blockquote>Note:<br>In a containerised deployment, <strong>we can’t write the SSL debug logs to a file using</strong> <strong>the</strong> nohup <strong>command</strong> as the <strong>main APIM process in the container needs to be run in foreground to keep the container up and running</strong>. Using nohup the APIM process running in the container will run in background and terminate immediately after the entrypoint script completes, causing the container to exit. <br>Instead, the process should be run in the foreground so that Docker keeps the container alive.</blockquote><p>So, <em>to capture SSL debug logs in a containerised deployment, redirect the output of the APIM process directly to a file while keeping it in the foreground</em>.</p><p><strong>Docker/K8s: Redirect “stdout” and “stderr” to a file</strong></p><p>We can <strong>redirect</strong> the standard output (stdout) and standard error (stderr) to the ssl-debug.log file by <strong>modifying</strong> <strong>the</strong> docker-entrypoint.sh script as following in a containerised environment,</p><pre># start WSO2 Carbon server<br>echo &quot;Start WSO2 Carbon server&quot; &gt;&amp;2<br>if [[ -z &quot;${PROFILE_NAME}&quot; ]]<br>then<br>  # start the server with the provided startup arguments<br>  sh ${WSO2_SERVER_HOME}/bin/api-manager.sh &quot;$@&quot; &gt;&gt; ${WSO2_SERVER_HOME}/repository/logs/ssl-debug.log 2&gt;&amp;1<br>else<br>  # start the server with the specified profile and provided startup arguments<br>  sh ${WSO2_SERVER_HOME}/bin/api-manager.sh -Dprofile=${PROFILE_NAME} &quot;$@&quot; &gt;&gt; ${WSO2_SERVER_HOME}/repository/logs/ssl-debug.log 2&gt;&amp;1<br>fi</pre><p>Server logs along with the SSL debug logs will be written to the <strong>${WSO2_SERVER_HOME}/repository/logs/ssl-debug.log </strong>file.</p><p>Since we are a redirecting the stdout and stderr there won’t be any logs in the container logs.</p><p>Hope you enjoyed the blog and got something to take away ✌️</p><p>Thank you for Reading!<br>Cheers 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ec744cf00a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Node Minikube K8s Deployment on M1 Mac with Colima and NFS PV]]></title>
            <link>https://techexpertise.medium.com/multi-node-minikube-k8s-deployment-on-m1-mac-with-colima-and-nfs-pv-762bcf08ac08?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/762bcf08ac08</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[nfs-server]]></category>
            <category><![CDATA[m1-mac]]></category>
            <category><![CDATA[colima]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sat, 05 Aug 2023 09:47:15 GMT</pubDate>
            <atom:updated>2023-08-05T09:47:15.546Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tVl7SquNme2uMLvn" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog we are going to explore how we can run a <strong><em>Multi-Node Minikube K8s deployment on M1 Mac without Docker-Desktop and with NFS PersistentVolume Storage</em></strong>.</p><p>Motivation:<br>I had a requirement where i needed to run a <strong><em>Multi-Node K8s Cluster on M1 Mac without Docker-Desktop which could run a K8s Deployment with a NFS PersistentVolume</em></strong>.</p><h3>What is issue there?</h3><h4>Requirement for running Multi-Node K8s cluster on a Mac host:</h4><blockquote><strong>K8s cluster runs Containers within Pods</strong>. <br>So in order to create a K8s cluster on a Mac host, <strong>first and foremost</strong> we need to have the <strong>ability to run Containers on Mac host</strong>.</blockquote><blockquote><strong>Containers</strong> (any type of container Docker or otherwise) are <strong>natively designed to run on Linux-based systems</strong>. They utilise <strong>Linux kernel features</strong> such as <strong>cgroups</strong> and <strong>namespaces</strong> to provide <strong>isolation</strong> and <strong>resource management</strong> for containers.<br>So we <strong>need to run a Linux VM</strong> on the <strong>Mac host</strong>.</blockquote><blockquote><strong>Why without Docker Desktop?</strong><br><a href="https://www.docker.com/">Docker Inc</a> has updated their <a href="https://www.docker.com/legal/docker-subscription-service-agreement">terms</a> on the usage of <a href="https://www.docker.com/products/docker-desktop"><strong>Docker-Desktop</strong></a> with affective from the 31st of August 2021. So in short, Docker Desktop for Mac/Windows is no longer free for “professional use in larger companies”</blockquote><blockquote><strong>Options</strong> to run a <strong>Multi-Node K8s cluster on Mac</strong>:<br>Let’s explore some of the technologies which support provisioning Multi-Node K8s cluster on Mac:<br>- <strong>Minikube</strong> (<a href="https://minikube.sigs.k8s.io/docs/tutorials/multi_node/">supports multi-node clusters</a>, supports <strong>both VM-based</strong> and <strong>Container-based</strong> approaches to provision the cluster) <em>✅</em><br>- <strong>Docker Desktop for Mac with </strong><a href="https://docs.docker.com/engine/swarm/"><strong>Docker Swarm</strong></a> (<a href="https://docs.docker.com/engine/swarm/swarm-tutorial/">supports multi-node cluster</a>, but we need a solution which doesn’t involve Docker Desktop) <em>❌</em><br>- <strong>Rancher Desktop with K3d</strong> (<a href="https://docs.rancherdesktop.io/how-to-guides/create-multi-node-cluster/">supports multi-node clusters</a>, but <strong>K3d runs K3s</strong> a <strong>minimal K8s distribution</strong>, hence not using this option as well) <em>❌<br>- </em><strong>KinD(Kubernetes in Docker)</strong><em>(</em><a href="https://mcvidanagama.medium.com/set-up-a-multi-node-kubernetes-cluster-locally-using-kind-eafd46dd63e5">supports multi-node clusters</a>, but only supports Container-based approach to provision the cluster) <em>✅</em></blockquote><blockquote>So, we will be using <strong>Minikube</strong> to provision the <strong>Multi-Node K8s cluster</strong> on <strong>Mac</strong>.</blockquote><blockquote>Minikube can be deployed as a <strong>VM</strong>, a <strong>Container</strong>, Or <strong>Bare-Metal</strong>.<br>Depending on the <strong>type of the Host OS</strong>, Minikube supports <a href="https://minikube.sigs.k8s.io/docs/drivers/"><strong>different drivers</strong></a> to achieve <strong>VM-based</strong> and <strong>Container-based</strong> approaches.</blockquote><h4>Issues Here</h4><ul><li>In <strong><em>Apple M1 based on ARM64</em></strong>, Minikube <strong><em>VM-based approach</em></strong> with <strong><em>Hyperkit driver</em></strong> is <a href="https://github.com/moby/hyperkit/issues/310#issuecomment-1003707160"><strong><em>not supported</em></strong></a><strong><em> </em></strong>by the ARM based CPUs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GjCh7ITQ9jL0qvTeshkjuQ.png" /></figure><ul><li>In <strong><em>Apple M1 based on ARM64</em></strong>, Minikube <strong><em>VM-based approach</em></strong> with <strong><em>QEMU2</em></strong> <strong><em>driver</em></strong> <strong><em>is not working</em></strong> (even though it is supported) and <strong><em>fails with an error when joining the worker nod</em>e</strong> <strong><em>to the cluster</em></strong> in a <strong><em>2-node cluster</em></strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EF8GdIdxYHOC7zZ-r_opIg.png" /></figure><ul><li><strong><em>Kubernetes is not yet fully compatible with ARM-based CPUs</em></strong>, so in a <br>VM-based approach to run K8s cluster on Mac host, we need to create <strong><em>VMs with Intel CPUs</em></strong> using <strong><em>hypervisors</em></strong> like <strong><em>Hyperkit</em></strong> Or <strong><em>QEMU2</em></strong>.</li><li><strong><em>Hyperkit hyperviso</em>r</strong> <strong><em>doesn’t have the ability</em></strong> <strong><em>to create VMs with different CPU architecture</em>s</strong>, like <strong><em>Intel</em></strong> and <strong><em>ARM</em></strong>. Thus, Hyperkit hypervisor is <strong><em>not supported on darwin/arm64 </em></strong>to run Minikube cluster.</li><li>But <strong><em>QEMU2 hypervisor</em></strong> has the ability to <strong><em>create VMs with Intel and ARM architectures</em></strong>. <strong><em>QEMU2 hypervisor can create a VM with Intel CPU on a M1 Mac with ARM CPU enabling to run K8s on ARM-based CPUs</em></strong>.</li><li>Since Minikube VM-based approaches resulted in failure, we are left with the Minikube <strong><em>Container-based approach</em> </strong>with <strong><em>Docker</em></strong> driver <strong><em>without using</em></strong> the <strong><em>Docker Desktop for Mac</em></strong>.</li><li>We <strong><em>need a solution</em></strong> which can run <strong><em>dockerd (Docker Daemon)</em></strong> aka <strong><em>Docker Runtime</em></strong> on a Mac host without using <strong><em>Docker Desktop for Mac</em></strong>.</li><li>Bingo!!! <strong>Colima</strong> comes as a saviour.</li></ul><h3>Install Colima to run Docker Containers on macOS</h3><blockquote><a href="https://github.com/abiosoft/colima"><strong>Colima</strong></a> is an open-source project that provides <strong>container runtimes on macOS</strong> <strong>(and Linux)</strong> with minimal setup.</blockquote><blockquote>The name <strong>Colima</strong> means <strong>Containers in </strong><a href="https://github.com/lima-vm/lima"><strong>Lima</strong></a> — <strong>Linux virtual machines on macOS</strong>.</blockquote><blockquote>Note: <br>Even though the <strong>name Colima (bit of a misnomer)</strong> means <strong>Containers in Lima</strong>, <strong>Colima is not a Containerized version of Lima</strong>.</blockquote><blockquote><strong>Colima uses QEMU2 hypervisor</strong> to <strong>create a full Linux VM</strong> with <strong>Intel CPU</strong> on a <strong>M1 Mac (with ARM CPU)</strong> while <strong>Lima uses Docker</strong> to create <strong>containerized Linux VM</strong>.</blockquote><blockquote>Colima was originally intended to be a containerized version of Lima, but the project evolved into a separate tool that uses QEMU2 instead of Docker.</blockquote><h4>Install Colima</h4><p>Easiest way to <strong><em>install Colima</em></strong> is using <strong><em>Homebrew</em></strong>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a080686356bdd58d07aecd1c81a1e081/href">https://medium.com/media/a080686356bdd58d07aecd1c81a1e081/href</a></iframe><p>Check the Colima installation by executing the following command from Mac terminal,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c947f2f255ee043741d03a3986551baf/href">https://medium.com/media/c947f2f255ee043741d03a3986551baf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtOM8_bpAQOZYCq3H0NW7w.png" /></figure><p>Minikube with Docker driver will create a <strong><em>docker containers</em></strong> inside the <strong><em>Colima Linux VM</em></strong>, in order for it to succeed we need <strong><em>Docker CLI</em></strong> to be installed on the Mac host.</p><h4>Install Docker and Docker-Compose</h4><p>Install <a href="https://formulae.brew.sh/formula/docker">Docker</a> and <a href="https://formulae.brew.sh/formula/docker-compose">Docker-Compose</a> via Homebrew,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad620345ed6489afe4a0161b3db3b968/href">https://medium.com/media/ad620345ed6489afe4a0161b3db3b968/href</a></iframe><p>Verify Docker and Docker-Compose installation by executing the following command from Mac terminal,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d554c4e3e7bb1b2a4d10724c4d08c33b/href">https://medium.com/media/d554c4e3e7bb1b2a4d10724c4d08c33b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FV5cEOcTJV2ndwlSHxhbLg.png" /><figcaption>Docker Info</figcaption></figure><p>As you can see from the above image, we have <strong><em>only installed</em></strong> the <strong><em>Docker CLI</em></strong> and <strong><em>dockerd (Docker Daemon)</em></strong> is not yet running.</p><p>We need to <strong><em>create a Colima Linux VM</em></strong> with <strong><em>Docker runtime installed</em></strong> to make the <strong><em>dockerd</em></strong> available on the <strong><em>Mac host</em></strong>.</p><h4>Create Colima Linux VM with a Mount from macOS Host</h4><p>Let’s <strong><em>create</em></strong> a <strong><em>Colima Linux VM</em></strong> with <strong><em>dockerd</em></strong> running with a <strong><em>file moun</em>t</strong> from <strong><em>Mac host</em></strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ee0b98e3301daaa32daa48363dee11f/href">https://medium.com/media/0ee0b98e3301daaa32daa48363dee11f/href</a></iframe><p>In order to <strong><em>create the Colima Linux VM named </em></strong><strong><em>linux-vm while editing the configurations,</em></strong> execute the following command from the Mac terminal,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ff4c6212554abce1b5f973138485671/href">https://medium.com/media/0ff4c6212554abce1b5f973138485671/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7iiBNsMidUU_dkWSR1hRNQ.png" /><figcaption>Colima Linux VM creation with Configurable Options</figcaption></figure><p>We have <strong><em>mount</em></strong> the “<strong><em>/Users/seralahthan/Minikube/nfs-mount”</em></strong> the directory from the <strong><em>Mac host</em></strong> to the <strong><em>Colima Linux VM</em></strong>.</p><p>Colima VM is created successfully, let’s check the <strong><em>status</em></strong> of the <strong><em>Colima VM </em></strong>by executing the following command from the Mac terminal,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e852ce7ac3464ffe9b075f0ef0c45cf7/href">https://medium.com/media/e852ce7ac3464ffe9b075f0ef0c45cf7/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXjL_dFcXWQ1BOOV1XTpRA.png" /><figcaption>Colima list and status of Linux VM</figcaption></figure><p>Now let’s <strong><em>SSH</em></strong> into the linux-vm and <strong><em>check whether our mount is intact</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/384936c96fe95a6a8502dae70567d5f9/href">https://medium.com/media/384936c96fe95a6a8502dae70567d5f9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N8HeHFkF7Ej3AiuiNRx6mg.png" /><figcaption>Check Mount from Linux VM to Mac Host</figcaption></figure><p>Now that we have provisioned a <strong><em>dockerd (Docker runtime</em>)</strong> with Colima Linux VM, we can now create the Minikube cluster with Docker driver.</p><h3>Create Minikube 2-Node Cluster with Docker Driver and Docker Runtime</h3><p>Let’s create a <strong><em>2-node Minikube Cluster</em></strong> named multi-node <strong><em>profile</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f534c0b0915ec628cacdb3cac8237e49/href">https://medium.com/media/f534c0b0915ec628cacdb3cac8237e49/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5enzo2tWefIziM4hNnVVQ.png" /><figcaption>Minikube Cluster Creation</figcaption></figure><p>Set the multi-node cluster profile as the <strong><em>active profile</em></strong>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a89a8ab20080d5a2daf81bc5a8a1c6d6/href">https://medium.com/media/a89a8ab20080d5a2daf81bc5a8a1c6d6/href</a></iframe><blockquote>Note:<br><strong>Each Minikube cluster node</strong> is created as <strong>Docker Container</strong> inside the <strong>Colima Linux VM</strong>.</blockquote><blockquote><strong>Minikube Docker driver</strong> <strong>creates</strong> a <strong>Docker daemon</strong> inside the <strong>Colima Linux VM.<br></strong>Minikube Nodes are created as Docker containers using this Docker daemon.</blockquote><blockquote>We can verify this by <strong><em>SSH</em></strong> into the linux-vm and <strong>listing the Docker containers</strong> with docker ps -a</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*caswBY0q2QBrBeaM1cK55w.png" /><figcaption>Minikube Nodes Run As Docker Containers in Colima Linux VM</figcaption></figure><blockquote>Note:<br><strong>Kubernetes Containers</strong> are <strong>created within</strong> the <strong>Kubernetes Pods</strong> based on the <strong>Container Runtime(Docker) </strong>specified for <strong>Minikube cluster</strong>.</blockquote><blockquote><strong>Kubernetes Pods</strong> run inside the <strong>Docker Containers (Minikube Nodes)</strong> which run in the <strong>Colima Linux VM</strong> which is ultimately <strong>hosted on the Mac host.</strong></blockquote><blockquote>So this can be thought of as <strong>Docker in Docker</strong> Or <strong>K8s in Docker</strong>.</blockquote><h4>Get Minikube IP Range</h4><p>We are <strong><em>going to create a NFS on the Colima Linux VM</em></strong> which should be <strong><em>accessible from the Minikube node</em>s</strong>.</p><p>We need to retrieve the Minikube IP range so that we can export the NFS to be accessible over all the Minikube nodes.</p><p>Execute the following command from the Mac terminal to <strong><em>fetch Minikube Control Plane node IP</em></strong>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d739c56c70120471871e704da90ca42a/href">https://medium.com/media/d739c56c70120471871e704da90ca42a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*daMAGGQQNsIEBEQs5RotAg.png" /></figure><p>In my case the <strong><em>Minikube Control Plane node IP</em></strong> is 192.168.49.2 <br><strong><em>Minikube Cluster Network IP range</em></strong> is <strong><em>first 3 octets</em></strong> of the <strong><em>Minikube Cluster IP</em></strong> fetched, which is 192.168.49.0/24.</p><h3>Create NFS in Colima Linux VM with a Mount to Mac</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c99aeb2e7861c793ca31d8668a34b0ee/href">https://medium.com/media/c99aeb2e7861c793ca31d8668a34b0ee/href</a></iframe><h4>Test Mount NFS to Minikube Node (Optional)</h4><p>Now that we have created a NFS, let’s try to mount it to the Minikube node as a NFS (this is an optional step for testing the mount)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82a3a1784d0656006613eec85fc39138/href">https://medium.com/media/82a3a1784d0656006613eec85fc39138/href</a></iframe><p>This verifies that the NFS can be mount to the Minikube nodes without any errors.</p><h3>Create Kubernetes PV, PVC and Deployment</h3><h4>Create a PersistentVolume with NFS share</h4><p>Now that we have a Minikube cluster created let’s create a <strong><em>PersistentVolume(PV)</em></strong> pointing to the <strong><em>NFS server</em></strong> using the following <strong><em>PV definition yaml</em></strong> file,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ecd297742d90bc438707016ae167f58/href">https://medium.com/media/6ecd297742d90bc438707016ae167f58/href</a></iframe><p>Execute the following command from the Mac terminal to create the PV,</p><pre>kubectl apply -f standard-host-path-nfs-manual-pv.yaml</pre><blockquote>Note that we have used the <em>Minikube Gateway IP address</em> as the n<em>fs server ip </em>along with the nfs mount.</blockquote><blockquote>Above PersistentVolume is created using the standard(default) Minikube Storage Class with k8s.io/minikube-hostpath Storage Provisioner.</blockquote><p>Further details on NFS volume can be found <a href="https://kubernetes.io/docs/concepts/storage/volumes/#nfs">here</a>.</p><p>Let’s now create a PersistentVolumeClaim to request some storage from the NFS PersistentVolume.</p><h4>Create a PersistentVolumeClaim</h4><p>Create a <strong><em>PersistentVolumeClaim</em></strong> in the <strong><em>test</em></strong> namespace using the following yaml definition file,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e34692dda5ad5b0a757789fd0aa720c6/href">https://medium.com/media/e34692dda5ad5b0a757789fd0aa720c6/href</a></iframe><p>Execute the following command to create the PersistentVolumeClaim,</p><pre>kubectl create ns test<br>kubectl apply -f standard-host-path-nfs-manual-pvc.yaml</pre><blockquote>Note that once the PersistentVolumeClaim is created K8s will automatically bind it to a PersistentVolume based on the requested storage of the claim and available storage in the volume.</blockquote><p>We can view the created PersistentVolume and PersistentVolumeClaim using the kubectl get pvc,pv command,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ArZ7EAdl4jL5B71ET4lW8A.png" /></figure><p>Finally, <strong><em>let us create a deployment with 2 pods</em></strong> where<strong><em> each pod is deployed to a separate cluster node</em></strong> (both pods shouldn’t be scheduled to the same node).</p><h4>Create a Deployment with topologySpreadConstraints</h4><p>Since our use-case is to check the functionality of using NFS share in a <br>2-node K8s cluster, we need to make sure the application pods are spread across all the nodes in the cluster (and not scheduled to the same node).</p><blockquote>In order to make sure the pods are spread across all the nodes, we need to use the <em>topologySpreadConstraints</em> with topologyKey: kubernetes.io/hostname and a DoNotSchedule condition based on the pod label.</blockquote><p>Following deployment yaml file can be used to create a deployment of,</p><ul><li>2 busybox pods scheduled in both nodes.</li><li>with a volume mount of /my-test</li><li>and <strong><em>shell process writing the</em></strong> <strong><em>“current time” and the “hostname” of the pod</em></strong> to the /my-test/shared.txt file.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b3113420d6a42682750266e1a159f93/href">https://medium.com/media/5b3113420d6a42682750266e1a159f93/href</a></iframe><p>Execute the following command to create the Deployment,</p><pre>kubectl create -f standard-host-path-nfs-manual-pvc-deployment.yaml</pre><blockquote>Note that we have referenced the <strong><em>persistentVolumeClaim</em></strong> standard-host-path-nfs-manual-pvc as the <strong><em>volume</em></strong> volv and mounted it to the pod under /my-test mount path.</blockquote><p>Pod will execute the following shell script which <strong><em>will write the current time and hostname of th</em>e pod</strong> to the /my-test/shared.txt file within the pod.</p><pre>sleep 60; for i in  $(seq 1 5); do host=$(hostname); now=$(date +&quot;%T&quot;); echo &quot;adding time: $now, from $host&quot; &gt;&gt; /my-test/shared.txt; sleep 10; done; sleep infinity</pre><p>We can verify that the pods are scheduled across both nodes by executing the kubectl get pods -o wide command,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Kr1zKfoYTrImxQyZa7JpA.png" /></figure><p>As you can see from the above image pods are spread across both nodes.</p><blockquote>Each pod will write the data on to its own volume mount<br>/my-test/shared.txt file.<br>But, since the volume mount is backed by the NFS volume both pods should be ideally writing to a common shared file in the Linux VM NFS mount.</blockquote><p>Let’s verify <strong><em>whether we can see data written by both pods to the</em></strong> /Users/seralahthan/Minikube/nfs-mountdirectory on Mac host machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNkCuSFFV-pWO4pJVbK1IA.png" /></figure><p>As you can see, data from both pods have been written to the Mac host at <strong><em>/Users/seralahthan/Minikube/nfs-mount </em></strong>directory.</p><p>This concludes our journey!!!</p><h3>References:</h3><p>[1] <a href="https://www.docker.com/">https://www.docker.com/</a><br>[2] <a href="https://www.docker.com/legal/docker-subscription-service-agreement">https://www.docker.com/legal/docker-subscription-service-agreement</a><br>[3] <a href="https://www.docker.com/products/docker-desktop">https://www.docker.com/products/docker-desktop</a><br>[4] <a href="https://minikube.sigs.k8s.io/docs/tutorials/multi_node/">https://minikube.sigs.k8s.io/docs/tutorials/multi_node</a>/<br>[5] <a href="https://docs.docker.com/engine/swarm/">https://docs.docker.com/engine/swarm/</a><br>[6] <a href="https://docs.docker.com/engine/swarm/swarm-tutorial/">https://docs.docker.com/engine/swarm/swarm-tutorial/</a><br>[7] <a href="https://docs.rancherdesktop.io/how-to-guides/create-multi-node-cluster/">https://docs.rancherdesktop.io/how-to-guides/create-multi-node-cluster/</a><br>[8] <a href="https://mcvidanagama.medium.com/set-up-a-multi-node-kubernetes-cluster-locally-using-kind-eafd46dd63e5">https://mcvidanagama.medium.com/set-up-a-multi-node-kubernetes-cluster-locally-using-kind-eafd46dd63e5</a><br>[9] <a href="https://minikube.sigs.k8s.io/docs/drivers/">https://minikube.sigs.k8s.io/docs/drivers/</a><br>[10] <a href="https://github.com/moby/hyperkit/issues/310#issuecomment-1003707160">https://github.com/moby/hyperkit/issues/310#issuecomment-1003707160</a><br>[11] <a href="https://github.com/abiosoft/colima">https://github.com/abiosoft/colima</a><br>[12] <a href="https://github.com/lima-vm/lima">https://github.com/lima-vm/lima</a><br>[13] <a href="https://formulae.brew.sh/formula/docker">https://formulae.brew.sh/formula/docker</a><br>[14] <a href="https://formulae.brew.sh/formula/docker-compose">https://formulae.brew.sh/formula/docker-compose</a><br>[15] <a href="https://kubernetes.io/docs/concepts/storage/volumes/#nfs">https://kubernetes.io/docs/concepts/storage/volumes/#nfs</a></p><p>Hope you enjoyed the blog and got something to take away ✌️</p><p>Thank you for Reading!<br>Cheers 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=762bcf08ac08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Minikube Multi-Node K8s Cluster with Shared NFS mount PV]]></title>
            <link>https://techexpertise.medium.com/minikube-multi-node-k8s-cluster-with-shared-nfs-mount-pv-cb3105f9a2c7?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb3105f9a2c7</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[nfs-server]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[multinode-cluster]]></category>
            <category><![CDATA[persistent-volume]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 14:39:00 GMT</pubDate>
            <atom:updated>2023-01-22T14:39:00.246Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lu98Fm_K3roVgto1" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog, let us explore <strong><em>how we can create and export a shared NFS mount on Mac</em></strong> and <strong><em>use it as a persistent volume storage in a multi-node K8s cluster</em></strong>.</p><p>Before we dive into the steps, let us try to understand why we would need the shared NFS mount for a multi-node K8s cluster.</p><blockquote>Recently, i came across this requirement, where i had a multi-node Minikube K8s cluster to which i wanted to mount a local directory from the host machine which can be shared and accessed across all the nodes.</blockquote><p>I tried to achieve this requirement using the following approaches,</p><ol><li>Creating a Minikube cluster with --mount-string and --mount options</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e834629a761db274f1e99154c61aa642/href">https://medium.com/media/e834629a761db274f1e99154c61aa642/href</a></iframe><p>2. Creating the Minikube cluster without the --mount options and mount the host directory later</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d9a95cd0e14163768e17bc837fcdc1e/href">https://medium.com/media/4d9a95cd0e14163768e17bc837fcdc1e/href</a></iframe><p>Both of the above approaches <strong><em>only mount the directory from the host file system to the K8s Control Plane node</em></strong>, the mount is not created in other worker nodes in a multi-node cluster.</p><blockquote>This happens as the <strong>default storage provisioner (k8s.io/minikube-hostpath)</strong> is broken for multi-node mode.</blockquote><p>Refer to the <a href="https://stackoverflow.com/questions/75037771/minikube-multi-node-cluster-mounting-host-machine-filesystem-to-all-the-nodes">link</a> for further details on this behaviour and issue.</p><p>So as an alternative, to achieve this requirement, <strong><em>we are going to create NFS share on the host system (Mac in my case)</em></strong> and <strong><em>create a NFS shared Persistent Volume storage</em></strong> to share a mount across all the nodes in a multi-node cluster.</p><p>Let’s have a look at the steps.</p><h4>Create a Minikube Multi-Node K8s Cluster</h4><p>As the first step, <strong><em>let’s create a Minikube 2-node K8s cluster without any mounts</em></strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/824bab64dea86d8f7918ef6b8732ce63/href">https://medium.com/media/824bab64dea86d8f7918ef6b8732ce63/href</a></iframe><p>Fetch the ip of the Minikube cluster using the following command,</p><pre>minikube ip</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*Nx8iVYajr2PJ8M5k6qp86Q.png" /></figure><p>This ip belongs to the Minikube cluster K8s Control Plan node. <br>In my case, all the other nodes will have the ip addresses with <strong>192.168.64.0/24 network CIDR</strong>.</p><h4>Enable NFS server and create a NFS mount on Mac</h4><p>Now that the cluster is up and running, let us <strong><em>enable the NFS server and export a host file system directory as the NFS mount</em></strong>.</p><p>In order to export a file system directory from the host machine as an NFS mount to the Minikube cluster, we need the <strong>NFS daemon to be enabled and running</strong></p><p>Following command can be used to check the status of the NFS daemon and enable and start it on Mac,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a6a335bfe28c886da45089760908350/href">https://medium.com/media/1a6a335bfe28c886da45089760908350/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*Rz-6kGxAzcOwDyQqQWgN9A.png" /></figure><p>Now that we have configured NFS server and created a NFS mount, let us create <strong>K8s PersistentVolume using the NFS share</strong>, <strong>PersistentVolumeClaim</strong> and a <strong>Deployment with 2 pods to write data to the NFS share</strong>.</p><h4>Create a PersistentVolume with NFS share</h4><p>Before we create the PersistentVolume storage, we need to find the <strong>Minikube Gateway IP address</strong>. This IP address will be used to connect to the NFS server in the PersistentVolume configuration file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87bfb93f60cd7f8c6ec1b0d8008d3385/href">https://medium.com/media/87bfb93f60cd7f8c6ec1b0d8008d3385/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NR-goEaBfoZnBer8qY5Uww.png" /></figure><p>Let us now create a PersistentVolume pointing to the NFS server using the following PersistentVolume definition yaml file,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e0d8df0fb66d1c09f9fa6362ec19e9a/href">https://medium.com/media/9e0d8df0fb66d1c09f9fa6362ec19e9a/href</a></iframe><p>Execute the following command to create the PersistentVolume,</p><pre>kubectl apply -f standard-host-path-nfs-pv.yaml</pre><blockquote>Note that we have used the Minikube Gateway IP address as the nfs server ip along with the nfs mount.</blockquote><blockquote>Above PersistentVolume is created using the standard (default) Minikube Storage Class with k8s.io/minikube-hostpath Storage Provisioner.</blockquote><p>Further details on NFS volume can be found <a href="https://kubernetes.io/docs/concepts/storage/volumes/#nfs">here</a>.</p><p>Let’s now create a PersistentVolumeClaim to request some storage from the NFS PersistentVolume.</p><h4>Create a PersistentVolumeClaim</h4><p>Create a <strong><em>PersistentVolumeClaim</em></strong> in the <strong><em>test</em></strong> namespace using the following yaml definition file,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7f035f190ffdafda75d3ca70f040892/href">https://medium.com/media/b7f035f190ffdafda75d3ca70f040892/href</a></iframe><p>Execute the following command to create the PersistentVolumeClaim,</p><pre>kubectl apply -f standard-host-path-nfs-pvc.yaml</pre><blockquote>Note that once the PersistentVolumeClaim is created K8s will automatically bind it to a PersistentVolume based on the requested storage of the claim and available storage in the volume.</blockquote><p>We can view the created PersistentVolume and PersistentVolumeClaim using the kubernetes get pv,pvc command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q1-D8w3Une1fP3vP33WbnA.png" /></figure><p>Finally, <strong><em>let us create a deployment with 2 pods where each pod is deployed in one cluster node (both pods can’t be scheduled to the same node</em>)</strong>.</p><h4>Create a Deployment with topologySpreadConstraints</h4><p>Since our use-case is to check the functionality of using NFS share in a multi-node K8s cluster, we need to make sure the application pods are spread across all the nodes in the cluster (and not scheduled to the same node).</p><blockquote>In order to make sure the pods are spread across all the nodes, we need to use the topologySpreadConstraints with topologyKey: kubernetes.io/hostname and a DoNotSchedule condition based on the pod label.</blockquote><p>Following deployment yaml file can be used to create a deployment of,</p><ul><li>2 busybox pods scheduled in both nodes</li><li>with a volume mount of /my-test</li><li>and <strong><em>shell process writing the “current time” and the “hostname” of the pod</em></strong> to the /my-test/shared.txt file.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb7795bd9fd992cfd2891d970674218c/href">https://medium.com/media/cb7795bd9fd992cfd2891d970674218c/href</a></iframe><p>Execute the following command to create the Deployment,</p><pre>kubectl create -f standard-host-path-nfs-pvc-deployment.yaml</pre><blockquote>Note that we have referenced the <strong>persistentVolumeClaim</strong> nfs-claim as the <strong>volume</strong> volv and mounted it to the pod under /my-test mount path.</blockquote><p>Pod will execute the following shell script which <strong><em>will write the current time and hostname of the</em> <em>pod</em></strong> to the /my-test/shared.txt file within the pod.</p><pre>sleep 60; for i in  $(seq 1 5); do host=$(hostname); now=$(date +&quot;%T&quot;); echo &quot;adding time: $now, from $host&quot; &gt;&gt; /my-test/shared.txt; sleep 10; done; sleep infinity</pre><p>We can verify that the pods are scheduled across both nodes by executing the kubectl get pods -o wide command,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lUpKyecEtp7vKXqAkW6qzA.png" /></figure><p>As you can see the pods are spread across both nodes.</p><blockquote>Each pod will write the data on to its own volume mount <br>/my-test/shared.txt file. <br>But, since the volume mount is backed by the NFS volume both pods should be ideally writing to a common shared file in the NFS mount.</blockquote><p><strong><em>Let’s verify whether we can see data written by both pods in the NFS mount in the host machine</em></strong>, in my case /Users/seran/Minikube/nfs-mount directory in my Mac.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xW4xWmvFvFXg8wP1JQMZQg.png" /></figure><p>As you can see from the above image current time is added to the /Users/seran/Minikube/nfs-mount/shared.txt file from both pods running on separate nodes.</p><blockquote>With NFS Persistent Volumes we don’t need to worry about on which node the pod will get scheduled, since the NFS volume is shared across all the nodes, which ever the node the pod is scheduled the application has access to all its data.</blockquote><h4>References:</h4><p>[1] <a href="https://stackoverflow.com/questions/75037771/minikube-multi-node-cluster-mounting-host-machine-filesystem-to-all-the-nodes">https://stackoverflow.com/questions/75037771/minikube-multi-node-cluster-mounting-host-machine-filesystem-to-all-the-nodes</a><br>[2] <a href="https://kubernetes.io/docs/concepts/storage/volumes/#nfs">https://kubernetes.io/docs/concepts/storage/volumes/#nfs</a><br>[3] <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field">https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#topologyspreadconstraints-field</a></p><p>Hope you enjoyed the blog and got something to take away ✌️</p><p>Thank you for Reading!<br>Cheers 🍻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb3105f9a2c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encrypting Secret data at etcd store on a Minikube K8s Cluster]]></title>
            <link>https://techexpertise.medium.com/encrypting-the-secret-data-at-etcd-store-on-a-minikube-k8s-cluster-2338c68263a5?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/2338c68263a5</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[kubernetes-secret]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[encryption]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sun, 08 Jan 2023 17:55:58 GMT</pubDate>
            <atom:updated>2023-01-08T18:00:12.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cMQaQauSGjFNk3-C" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika Developer Marketing Agency</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this blog let us explore how we can <strong><em>encrypt and store Kubernetes secrets in the etcd key-value store</em></strong></p><p>Before we start, let’s try to understand what is <strong><em>etcd key-value store</em></strong>.</p><blockquote><strong>etcd</strong> is an open source distributed key-value store used to hold and manage the critical information that distributed systems need to keep running.</blockquote><blockquote>It manages the configuration data, state data, and metadata for Kubernetes.</blockquote><p>Generally the data that is stored in etcd key-value store is in <strong><em>plain-text format</em></strong>. So anyone who has access to the etcd key-value store can gain access to some sensitive data like K<strong><em>ubernetes secrets</em></strong>.</p><p>In this blog we will only focus on securely storing Kubernetes secret objects on the etcd key-value store.</p><h4>Prerequisites</h4><ul><li><a href="https://minikube.sigs.k8s.io/docs/start/">Minikube Installation</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/#kubectl">Kubectl Installation</a></li><li>Kubernetes Control Plane running on a Minikube Cluster node</li></ul><p>Let us first explore the default behaviour of how Kubernetes secret objects are stored.</p><h4>Create a Kubernetes Secret object</h4><p>Let’s first create a Kubernetes <strong><em>generic Secret object</em></strong> named <strong><em>my-secret </em></strong>with data <strong><em>key1=supersecret </em></strong>in the <strong><em>test</em></strong> namespace.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/568f8a7bdab2c3be3efd73067b3d7f73/href">https://medium.com/media/568f8a7bdab2c3be3efd73067b3d7f73/href</a></iframe><p>Retrieve the secret we just created in json format,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e343d2adc21727de7ea90f96dfd56a1a/href">https://medium.com/media/e343d2adc21727de7ea90f96dfd56a1a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*RbuuVcUEY5iCfVrUOeNwDA.png" /></figure><p>As you can see from the above output, the <strong><em>data value is stored in a base64 encoded format</em></strong>.</p><p>We can <strong><em>decode</em></strong> and view the <strong><em>data</em></strong> stored in the <strong><em>my-secret</em></strong> object using the following command</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fb9c4bf8851a4febd3e918591a124f5/href">https://medium.com/media/6fb9c4bf8851a4febd3e918591a124f5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0ReJcjEImhMW00_uLeOkw.png" /></figure><p>Now let us check how the <strong><em>my-secret</em></strong> is stored in the <strong><em>etcd key-value store</em></strong>.</p><h4>View the Secrets in the etcd key-value store</h4><p>In order to view the secrets values stored in the etcd-key value store, we need to <strong><em>login to the shell</em></strong> of the <strong><em>etcd pod</em></strong> running on the <strong><em>kube-system</em></strong> namespace.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6cfa211ceb9750ac7c0e84ba86e1f18c/href">https://medium.com/media/6cfa211ceb9750ac7c0e84ba86e1f18c/href</a></iframe><p>After login to the shell of <strong><em>etcd-multi-node </em></strong>pod (etcd key-value store), we need to execute the following command to fetch the <strong><em>my-secret</em></strong> stored in the <strong><em>etcd key-value store </em></strong>using the <strong><em>etcdctl command</em></strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23c33a0d233a7543f63766f092be2d43/href">https://medium.com/media/23c33a0d233a7543f63766f092be2d43/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnhSetxomUz6Zou1wrJLQA.png" /></figure><p>As you can see from the above screenshot the <strong>secret data is stored in plain-text format in the etcd key-value store</strong>.</p><p><strong>Our goal is to encrypt the secret data stored in the etcd key-value store.</strong></p><blockquote>In order to encrypt all the Kubernetes “<strong>Secret”</strong> resources we need to create a <strong>“EncryptionConfiguration”</strong> K8s yaml file (<strong><em>encryption-conf.yam</em>l</strong>) with <strong>Secret</strong> as resource to be encrypted and pass it to the <strong><em>kube-apiserver</em></strong> yaml file under <strong>container.commands</strong> section with “--<strong><em>encryption-provider-config”</em></strong> property.</blockquote><p>Before we dive into the steps of it, let us quickly verify the current configurations of the kube-apiserver process running on the Control Plane.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ada77adc759ac1f1be6fc799ccac4485/href">https://medium.com/media/ada77adc759ac1f1be6fc799ccac4485/href</a></iframe><p>Output of the above command will confirm that the kube-apiserver process is not running with the <strong>encryption-provider-config</strong>, thus Secrets are not encrypted in the etcd key-value store.</p><h4>Create a new EncryptionConfiguration Kubernetes resource file</h4><p>Create a new <strong>&quot;EncryptionConfiguration”</strong> k8s yaml file <br>(<strong>encryption-conf.yaml</strong>) to encrypt all the <strong>k8s “secret” resources</strong>, this yaml file will be passed as the <strong>“--encryption-provider-config”</strong> parameter to the kube-apiserver.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee3db683c3d14f79189f1bee699a69e7/href">https://medium.com/media/ee3db683c3d14f79189f1bee699a69e7/href</a></iframe><p>Note that the <strong><em>resources</em></strong> only contains the <strong><em>secrets </em></strong>as we are only interested in encrypting the secrets stored in the etcd key-value store.</p><p>To generate a new secret for the above config,<br>Generate a 32-byte random key and base64 encode it. <br>If you’re on Linux or macOS, run the following command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e5b1704e3505baffc0c8c7bb0c8ca4d/href">https://medium.com/media/8e5b1704e3505baffc0c8c7bb0c8ca4d/href</a></iframe><p>In order to point the <strong>encryption-conf.yaml</strong> file to the <br><strong>--encryption-provider-config </strong>property, we need to mount the file to the kube-apiserver pod with a <strong>volumeMount</strong> and <strong>volume.</strong></p><p>Copy the <strong>encryption-conf.yaml</strong> file in the host machine to the local directory which is mount to the the Minikube node, in my case i have the local <strong>$HOME/Minikube/mount</strong> directory mount to the <strong>/data</strong> directory of the minikube node.</p><p>Now that the encryption configuration file is available in the Minikube control plane node <strong>“/data/encryption-conf.yaml” </strong>path.</p><blockquote>In order for the <strong>encryption-conf.yaml</strong> file to be correctly picked up by the <br><strong>kube-apiserver</strong> pod it needs to be placed in the <strong>/etc/kubernetes/</strong> path of the <strong>Control Plane</strong> node and <strong>mount as a volume</strong>.</blockquote><p>So, let’s create a subdirectory <strong><em>encryption</em></strong> in the <strong><em>/etc/kubernetes/</em></strong> path and copy the encryption configuration file there.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/278cb9c6ef58c0edecbce62b775259db/href">https://medium.com/media/278cb9c6ef58c0edecbce62b775259db/href</a></iframe><h4>Update the kube-apiserver.yaml</h4><p>Let’s now edit the <strong>“kube-apiserver”</strong> definition file and add,</p><ul><li><strong>“--encryption-provider-config”</strong> property pointing to the configuration file <strong>encryption-conf.yaml</strong> from a <strong>volumeMount</strong> “<strong>/etc/kubernetes/enc</strong>”</li><li><strong>volume</strong> <strong>“enc-vol”</strong> pointing to the path <strong>“/etc/kubernetes/encryption”</strong> where we have the encryption configuration file</li><li><strong>volumeMount</strong> for the <strong>volume</strong> <strong>“enc-vol”</strong> with <strong>mountPath “/etc/kubernetes/enc”</strong></li></ul><p>Navigate to the <strong>“/etc/kubernetes/manifests”</strong> location and edit the <br><strong>“kube-apiserver.yaml”</strong> file with the below changes,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a62ee7c7505a9800c006e9ddd43deb3a/href">https://medium.com/media/a62ee7c7505a9800c006e9ddd43deb3a/href</a></iframe><blockquote>Changes to the <strong>“kube-apiserver.yaml”</strong> file are automatically applied once you save the changes, <strong>existing “kube-apiserver” process</strong> running on the <br><strong>“multi-node” cluster node</strong> will be <strong>destroyed</strong> and a <strong>new process will be created with the “--encryption-provider-config”</strong> parameter we passed.</blockquote><p>Let’s view and verify that the <strong><em>“kube-apiserver”</em></strong> process is running with the <br><strong><em>“--encryption-provider-config”</em></strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ada77adc759ac1f1be6fc799ccac4485/href">https://medium.com/media/ada77adc759ac1f1be6fc799ccac4485/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Jy5bcRo4QoA9CiAzKIwFg.png" /></figure><p>This completes all the configuration steps.</p><p>Now that the encryption configurations are successfully applied, let us <strong><em>create a new secret</em></strong> with the same data value as <strong><em>my-secre</em>t</strong> and view how it is stored in the <strong><em>etcd key-value store</em></strong> after the encryption.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b0cdd7def1c4def53611d65858f2ef7/href">https://medium.com/media/6b0cdd7def1c4def53611d65858f2ef7/href</a></iframe><p>Let’s view the <strong><em>new-secret</em></strong> stored in the <strong><em>etcd key-value store</em></strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f212de2eb6b53d9f5e12da2acd30a8c/href">https://medium.com/media/4f212de2eb6b53d9f5e12da2acd30a8c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmGJuOi10e9aezg7fU0WYg.png" /></figure><p>As you can see the <strong><em>secret</em></strong><em> </em><strong><em>data stored in the “etcd” store is completely encrypted</em></strong><em>.</em></p><p>Now let us try to <strong><em>retrieve this data using the API and check whether we get the correct decrypted data in the base64 encoded forma</em>t</strong>,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c305da598bae4b453ba9b92763e44200/href">https://medium.com/media/c305da598bae4b453ba9b92763e44200/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-sGQIjiuTcTLr1YrFWYig.png" /></figure><p>As you can see there is no change in the behaviour of data retrieval even after the encryption of etcd key-value store, so all working as expected.</p><p>Finally, let us try to <strong><em>view the secrets</em></strong> which are <strong><em>created prior to enabling encryption at the etcd key-value store</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eS1BoGQ1MrasTI7qUfOu9Q.png" /></figure><blockquote>As you can see from the above output, <strong>enabling encryption does not automatically encrypt existing secrets</strong>, it will encrypt any new secrets created after the encryption was enabled.</blockquote><p>In order to <strong><em>encrypt all the existing secrets stored on the etcd key-value store</em></strong>, execute the following command,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0a0bcbec130ae2af19d8104debacc04/href">https://medium.com/media/a0a0bcbec130ae2af19d8104debacc04/href</a></iframe><blockquote>Above command reads all Secrets and then updates them to apply server side encryption. Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.</blockquote><h4>References:</h4><p>[1] <a href="https://etcd.io/">https://etcd.io/</a><br>[2] <a href="https://minikube.sigs.k8s.io/docs/start/">https://minikube.sigs.k8s.io/docs/start/</a><br>[3] <a href="https://kubernetes.io/docs/tasks/tools/#kubectl">https://kubernetes.io/docs/tasks/tools/#kubectl</a><br>[4] <a href="https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/">https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/</a></p><p>Hope you enjoyed the blog and got something to take away.</p><p>Thank you for Reading!<br>Cheers!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2338c68263a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up Basic Authentication for Kubernetes Cluster on Minikube — 1]]></title>
            <link>https://techexpertise.medium.com/setting-up-basic-authentication-for-kubernetes-cluster-on-minikube-1-e84e1b56c64?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/e84e1b56c64</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[cluster]]></category>
            <category><![CDATA[kube-api-server]]></category>
            <category><![CDATA[basic-authentication]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sun, 06 Nov 2022 15:03:03 GMT</pubDate>
            <atom:updated>2022-11-06T15:03:03.684Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up Basic Authentication for Kubernetes Cluster on Minikube — 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WV1mBl7R_KS4e7kK" /><figcaption>Photo by <a href="https://unsplash.com/@growtika_developer_marketing_agency?utm_source=medium&amp;utm_medium=referral">Growtika Developer Marketing</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this post let’s explore how we can setup a <strong>username-password based basic authentication</strong> to access the <strong>kube-apiserver</strong></p><blockquote><strong>Setup basic authentication on Kubernetes (Deprecated in 1.19)</strong></blockquote><blockquote>Note: This is <strong>not recommended in a production environment.</strong> <br>This is only for learning purposes. Also note that this approach is deprecated in Kubernetes version 1.19 and is no longer available in later releases.</blockquote><h4>kube-apiserver authentication mechanisms</h4><p>kube-apiserver authenticates the user before processing the requests.<br>There are different authentication mechanisms that can be configured to be used with kube-apiserver. Few of them are,</p><ul><li><strong>Basic Authentication with Static Password file<br></strong>Plain text username and password fetched from a static password file and used for validation against the credentials passed in the request.</li><li><strong>Bearer Token Authentication with Static Token file<br></strong>Plain text username and token are fetched from a static token file and used for validation against the bearer token passed in the request.</li><li><strong>Certificate based Authentication<br></strong>X509 client certificate is used for authentication. <br>If a client certificate is presented and verified, the common name of the subject is used as the user name for the request</li><li><strong>Third party Identity Services Authentication<br></strong>In this approach kube-apiserver is integrated with third-party identity services like <strong><em>LDAP</em></strong>, <strong><em>Kerberos</em></strong>, <strong><em>External Identity Providers</em></strong>, etc.</li></ul><p>In this blog, our goal is to achieve basic authentication with a static password file.</p><blockquote>In order to enable basic authentication support in kube-apiserver we need to pass the <strong>user details in a csv file</strong> with <strong>“--basic-auth-file”</strong> option.<br><strong>--basic-auth-file=user-details.csv</strong></blockquote><h4>Prerequisites</h4><ul><li><a href="https://minikube.sigs.k8s.io/docs/start/">Minikube installation</a></li><li><a href="https://kubernetes.io/docs/tasks/tools/#kubectl">Kubectl installation</a></li></ul><h4>Create a single node Kubernetes cluster on Minikube</h4><p>First, let’s create a single node k8s <strong>cluster named</strong> <br><strong>“test-node”</strong> with default <strong>“test” namespace</strong> (without any extra configurations) <strong>provisioned with k8s version 1.18.0</strong> (any version lower than v1.19.0) using the following command,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6944ac80aba78c1ea01fb3abbd846c6c/href">https://medium.com/media/6944ac80aba78c1ea01fb3abbd846c6c/href</a></iframe><h4>Set the Active Cluster on Minikube</h4><p>Since we can have multiple k8s clusters running with minikube, we need to set the active cluster to execute the kubectl commands on that cluster</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3ab8990ad671ca7738dc4461209c6c1/href">https://medium.com/media/e3ab8990ad671ca7738dc4461209c6c1/href</a></iframe><h4>Fetch the Kubernetes Control Plane URL</h4><p>Fetch the k8s control plane url by executing the following kubectl command,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e99c691b921d0114201902a7338ac90/href">https://medium.com/media/7e99c691b921d0114201902a7338ac90/href</a></iframe><p>The output will look like the following,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWZ7rMSMOgNkuNPH4ZlRYw.png" /></figure><p>We will be using this url to query the “<strong>kube-apiserver”</strong></p><p><strong>Create the Static Password file in the Minikube node</strong></p><p>In order for the basic authentication to work we need to pass a static password file. This file need to be accessible by the k8s cluster, thus needs to be placed in the Minikube cluster node.</p><p>SSH into the Minikube k8s <strong>cluster node “test-node”</strong> using the following command,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53a3fdc8116f38b158e2cba95093e294/href">https://medium.com/media/53a3fdc8116f38b158e2cba95093e294/href</a></iframe><p>Create a file named “<strong>user-details.csv”</strong> with the user details in the <strong>“/tmp/users”</strong> directory of the cluster node</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9434b8c1bab0676bbcaefa8480565e73/href">https://medium.com/media/9434b8c1bab0676bbcaefa8480565e73/href</a></iframe><blockquote><strong>“user-details.csv”</strong> file has three columns, <strong>password</strong>, <strong>username</strong> and <strong>userid</strong> respectively</blockquote><h4>Create the Namespace, Roles and RoleBindings for the Users</h4><p>In order for the users to perform operations on the k8s cluster we need to create roles (in the specific namespace) with the relevant permissions and bind them to the users using rolebindings.</p><p>Let’s create a <strong>“test” namespace</strong> in the <strong>“test-node” cluster </strong>(“test-node” is already set as the active Minikube cluster)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/795e6100122c08c59a248c1b767b7657/href">https://medium.com/media/795e6100122c08c59a248c1b767b7657/href</a></iframe><p>Create the <strong>role</strong> and <strong>rolebinding</strong> for the <strong>“wso2user”</strong> to fetch the pods, by executing the “<strong>roles-rolebindings-apiserver-basic-auth.yaml”</strong> file,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b9995f3c59195b1fa9892614bfcde76/href">https://medium.com/media/1b9995f3c59195b1fa9892614bfcde76/href</a></iframe><p>Create the <strong>Role</strong> and <strong>RoleBinding</strong> by executing the following kubectl command,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/857ace03d080fbdce598d19a03f6491a/href">https://medium.com/media/857ace03d080fbdce598d19a03f6491a/href</a></iframe><p>Executing the above k8s definition file will,</p><ul><li>Create a role named <strong>“pod-reader”</strong> in the <strong>“test”</strong> namespace with permissions to <strong>“get”</strong> and <strong>“list”</strong> pods.</li><li>Create a rolebinding named <strong>“read-pods”</strong> in the <strong>“test”</strong> namespace which associates the <strong>“pod-reader”</strong> role to the <strong>“wso2user”</strong></li></ul><h4>Configure basic authentication for kube-apiserver</h4><p>Now let’s apply the configuration changes for the <strong>“kube-apiserver”</strong> static pod in the <strong>“kube-system”</strong> namespace configured by the Minikube to support basic authentication by mounting the <strong>“user-details.csv”</strong> file to the pod.</p><p>We will apply the following changes to the “kube-apiserver” pod in the “kube-system” namespace,</p><ul><li>Include the <strong>“--basic-auth-file” startup option</strong> pointing to the <br><strong>“user-details.csv” file</strong> in the <strong>volumeMount “/tmp/auth”</strong></li><li>Configure the <strong>volumes</strong> pointing to the <strong>“/tmp/users” hostpath</strong> where the <strong>“user-details.csv” file</strong> resides in the node</li><li>Configure the <strong>volumeMounts</strong> to mount the <br><strong>“/tmp/users/user-details.csv”</strong> file from the node to the <br><strong>“kube-apiserver” pod</strong> via the mount path <strong>“/tmp/auth/user-details.csv”</strong></li></ul><p>The “kube-apiserver” pod is automatically created by the Minikube cluster in the “kube-system” namespace using the <strong>kube-apiserver.yaml</strong> k8s definition file found in the <strong>/etc/kubernetes/manifests</strong> directory of the cluster node.</p><p>Modifications to the <strong>/etc/kubernetes/manifests/kube-apiserver.yaml</strong> will apply changes to kube-apiserver pod</p><p>Execute the following commands,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ee68a7153609c806157a20b13aae23a/href">https://medium.com/media/8ee68a7153609c806157a20b13aae23a/href</a></iframe><p>Modify the <strong>kube-apiserver.yaml </strong>with the above discussed additional configurations as shown below,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15fc8191641cfbc391e3b36bef93104f/href">https://medium.com/media/15fc8191641cfbc391e3b36bef93104f/href</a></iframe><p>Once the modifications are completed, save the changes to the <br><strong>“kube-apiserver.yaml”</strong>, Minikube will automatically apply the changes and <strong>recreate the “kube-apiserver-test-node” pod</strong> in the <strong>“kube-system” namespace</strong></p><h4>Test API Invocations with Basic Authentication</h4><p>We have completed all the configurations, and the k8s cluster should now support basic authentication with the user details we passed.</p><p>Let’s try to <strong>fetch the pod details</strong> in the <strong>“test”</strong> namespace using the <br><strong>“kube-apiserver” url</strong>, by <strong>passing the “wso2user” credentials in the basic header</strong></p><pre>curl -kv <a href="https://192.168.64.18:8443/api/v1/namespaces/test/pods">https://192.168.64.18:8443/api/v1/namespaces/test/pods</a> \<br>-u &quot;wso2user:abcd&quot;</pre><p>The curl response will look like the following,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*Croqq9nq1knXu-l-bz4NVg.png" /></figure><p><strong>200 OK response</strong> is returned with some <strong>metadata</strong> as shown above.<br>The <strong>“items” array</strong> is empty as there are <strong>no pods running</strong> in the <strong>“test”</strong> namespace.</p><p>Let’s quickly <strong>run a pod</strong> in the <strong>“test” namespace</strong>,</p><pre>kubectl -n test run nginx --image=nginx</pre><p>Now let’s try to fetch the pods again for the <strong>“wso2user”</strong> with the <strong>correct credentials</strong>,</p><pre>curl -kv <a href="https://192.168.64.18:8443/api/v1/namespaces/test/pods">https://192.168.64.18:8443/api/v1/namespaces/test/pods</a> \<br>-u &quot;wso2user:abcd&quot;</pre><p><strong>200 OK response</strong> is returned with a populated <strong>items</strong> array as following,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*eXCcZxMIXMn4WVx7eRSxGg.png" /></figure><p>Now let’s try to fetch the pods for the <strong>“wso2user”</strong> with the <strong>wrong credentials</strong>,</p><pre>curl -kv <a href="https://192.168.64.18:8443/api/v1/namespaces/test/pods">https://192.168.64.18:8443/api/v1/namespaces/test/pods</a> \<br>-u &quot;wso2user:wrongpassword&quot;</pre><p><strong>401 Unauthorized response</strong> is returned with “<strong>status: Failure” </strong>as shown below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*d_AI34601-dxwrdaKlBY7w.png" /></figure><p>Finally, let’s try to fetch the pods with a <strong>non-existing user “wso2user1”</strong></p><pre>curl -kv <a href="https://192.168.64.18:8443/api/v1/namespaces/test/pods">https://192.168.64.18:8443/api/v1/namespaces/test/pods</a> \<br>-u &quot;wso2user1:abcd&quot;</pre><p><strong>401 Unauthorized response</strong> is returned with <strong>“status: Failure” </strong>same as with the wrong credentials.</p><h4>References:</h4><p>[1] <a href="https://minikube.sigs.k8s.io/docs/start/">https://minikube.sigs.k8s.io/docs/start/</a><br>[2] <a href="https://kubernetes.io/docs/tasks/tools/#kubectl">https://kubernetes.io/docs/tasks/tools/#kubectl</a><br>[3] <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/">https://kubernetes.io/docs/reference/access-authn-authz/authentication/</a></p><p>Hope you enjoyed the blog and got something to take away.</p><p>Thank you for Reading!<br>Cheers!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e84e1b56c64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Key Stores and Public Certificates — JKS]]></title>
            <link>https://techexpertise.medium.com/exploring-key-stores-and-public-certificates-jks-723eb6b32a21?source=rss-5cc3faae5c14------2</link>
            <guid isPermaLink="false">https://medium.com/p/723eb6b32a21</guid>
            <category><![CDATA[private-key]]></category>
            <category><![CDATA[java-keystore]]></category>
            <category><![CDATA[pkcs12]]></category>
            <category><![CDATA[openssl]]></category>
            <category><![CDATA[keytool]]></category>
            <dc:creator><![CDATA[Seralahthan]]></dc:creator>
            <pubDate>Sat, 12 Feb 2022 15:47:08 GMT</pubDate>
            <atom:updated>2022-02-12T15:47:08.074Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploring Key Stores and Public Certificates — JKS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*3BB2MxfZ6nIjHhgPtM6c6A.png" /></figure><p>In this post let’s explore the fundamentals of a Java Key Store (JKS) and a Public Key Certificate.</p><h4>!!! Caution: Long Post !!!</h4><p>This post is going to be a bit elaborate and lengthy, so, if you are looking for a quick cheat sheet of commands… Here you go…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d07ec2d7bd7be79701a4a487f8a88d8/href">https://medium.com/media/6d07ec2d7bd7be79701a4a487f8a88d8/href</a></iframe><blockquote>The key store is a storage facility to <strong>store cryptographic keys</strong> and <strong>certificates</strong> used for SSL communications.</blockquote><p>There are different types of key stores available in Java depending on the type of entries the keystore can store and how the keystore can store the entries. <strong><em>Java Key Store (JKS)</em></strong> is one of the widely used key store types available in Java.</p><p><strong><em>Each key store can hold a unique set of keys depending on the type</em></strong>.<br>In General, A key store contains the following types of keys,</p><ul><li><strong>Private Key Entry/Entries</strong> <br>(With a certificate chain for the corresponding Public Key Entry/Entries).</li><li><strong>Trusted Certificates</strong> <br>(Third-party Public keys + Certificate Chains)</li><li><strong>Secret Key Entry/Entries</strong></li></ul><h4><strong><em>Private Key (PrivateKeyEntry)</em></strong></h4><p>It is the key used for <strong><em>asymmetric encryption</em></strong> and <strong><em>signing a digital signature</em></strong>. The private key entry is password protected.</p><p>Generally, a JKS type of key store can have <strong><em>only one private key entry</em></strong> in a key store file. Some key store types, <strong><em>allow having multiple private key entries</em></strong> in a single key store.</p><p>Along with the <strong><em>private key entry</em></strong>, <strong><em>a certificate chain</em></strong> for the corresponding <strong><em>public key entry</em></strong> also will be there.</p><h4><strong><em>Trusted Certificates (TrustedCertificateEntry)</em></strong></h4><p>It holds the third-party public keys used for <strong><em>asymmetric encryption</em></strong> and <strong><em>digital signature</em></strong> <strong><em>validation</em></strong> along with the Trusted Certificate Chains.</p><p>A key store can hold any number of Trusted Certificates.</p><h4>Secret Key (SecretKeyEntry)</h4><p>It is the key used for <strong><em>symmetric encryption</em></strong>.<br>Generally, a <strong><em>JKS doesn’t have SecretKey entries</em></strong>.</p><p>There are different types of Java key stores depending on the entries a key store can store and how these entries are stored in the key store. Some of the popular Java key stores are <strong><em>JKS</em></strong>, <strong><em>JCEKS</em></strong>, <strong><em>PKCS12</em></strong>, <strong><em>DKS</em></strong>, and <strong><em>BKS</em></strong>.</p><p>Let’s explore some of these Java key store types and their differences.</p><h3>Java Key Store (JKS)</h3><blockquote>Is a repository of security certificates — either authorisation certificates or public key certificates — plus corresponding private keys, used for instance in SSL encryption.</blockquote><p>Implementation can be found at <strong><em>java.security.KeyStore</em></strong><br>This key store is Java-specific and usually has an extension of <strong><em>.jks</em></strong> <br>(<strong><em>eg: wso2carbon.jks</em></strong>)</p><p>According to the Java specification, JKS key store manages different types of entries. Each type of entry implements the <strong><em>KeyStore.Entry</em></strong> interface. <br>Three basic <strong><em>KeyStore.Entry</em></strong> implementations are provided:</p><ul><li><strong><em>KeyStore.PrivateKeyEntry</em></strong></li><li><strong><em>KeyStore.SecretKeyEntry</em></strong></li><li><strong><em>KeyStore.TrustedCertificateEntry</em></strong></li></ul><p>Refer to the Java <a href="https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/security/KeyStore.html">documentation</a> for further details.</p><h4>Features of JKS</h4><ul><li>Since this is a Java-specific key store cannot be used with other programming languages.</li><li>It <strong><em>doesn’t support storing Secret Key entries</em></strong>.</li><li>Only a <strong><em>single Private Key entry</em></strong> can be stored in a key store file, <strong><em>multiple Private Key entries are not allowed</em></strong>.</li><li>Private Key entries stored in the JKS key store can’t be extracted.</li></ul><p>Now let’s see how we can implement a key store using Java. <br>Java provides few options to work with key stores :</p><ul><li>Writing a Java code</li><li>Using the inbuilt tool named <strong><em>“keytool</em>”</strong> which comes along with the JDK.</li></ul><blockquote>Keytool is a command-line tool that is shipped along with the JDK.<br>It can be used to create keystore, generate keys, import and export certificates, etc.</blockquote><blockquote>Note that <strong>keytool is only available with Java JDK installations</strong> and <strong>not with Java JRE runtimes</strong>.</blockquote><h4>Create a JKS with Private &amp; Public key pairs using Keytool</h4><pre>keytool -genkeypair -v -alias sample -keyalg RSA -keysize 2048 \<br>-validity 365 -keystore sample_keystore.jks</pre><p>This is achieved by using the keytool <strong><em>‘-genkeypair’</em></strong> option, in earlier version of keytool <strong><em>‘-genkey’</em></strong> option was used which is still supported for backward compatibility.</p><p>The above command will create a JKS key store named <strong><em>sample_keystore.jks</em></strong> which contains, <strong><em>a 2048 bit Private/Public key pair aliased “sample” created using the RSA cryptographic algorithm and a self-signed certificate (SHA256withRSA) with a validity period of 365 days</em></strong>.</p><p>Above command,</p><ol><li><strong><em>Generates a RSA 2048 bit Public and Private key pair</em></strong>,</li><li><strong><em>Wraps the public key into an X.509 v3 self-signed certificate signed by SHA256withRSA with a validity period of 365 days</em></strong>,</li><li>Self-signed certificate is then stored as a <strong><em>single-element certificate chain</em></strong>.</li><li><strong><em>Certificate chain</em></strong> and the <strong><em>Private Key</em></strong> are stored in a new keystore <strong><em>‘sample_keystore.jks’</em></strong> entry identified by <strong><em>alias ‘sample’</em></strong>.</li></ol><p>While executing the above command to generate the key store, you will be prompted for the <strong><em>“CN”</em></strong>, <strong><em>“key store password”</em></strong> and <strong><em>“key password”</em></strong> as below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ZpHFqEWzUx7EI1SyLgiUQ.png" /></figure><p><strong>CN</strong> should generally match the hostname of the server.<br>Generally is good to use the <strong><em>same password</em></strong> for the <strong><em>keystore password</em></strong> and the <strong><em>key password</em></strong> (for the alias “sample”).</p><p>Some servers like <strong><em>Tomcat Catalina Server</em></strong> by default use the “key store password” as the “key password” for the connectors if not explicitly mentioned.</p><p>Not all the parameters are required to be passed in the above keytool command, following are the optional defaults for the keystore creation command,</p><pre>-alias: mykey<br>-keyalg: DSA<br>-keysize: 2048 bit (for both RSA and DSA)<br>-validity: 90 days<br>-storetype: JKS (property value fetched from &#39;java.security.keystore.type&#39; from Java jdk/jre)</pre><p>Refer to the keytool <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html#keytool_option_genkeypair">documentation</a> for further details.</p><h4>Configuring Tomcat Catalina Server with the JKS</h4><p>If you are using a Tomcat Catalina Server and using separate “key password” and “key store password”, you need to explicitly configure it in the <strong><em>server.xml</em></strong> file of the Tomcat Server found in the <strong>&lt;TOMCAT_HOME&gt;/conf/</strong> directory.</p><pre>&lt;!-- Define an SSL Coyote HTTP/1.1 Connector on port 8443 --&gt;<br>&lt;Connector<br>           protocol=&quot;org.apache.coyote.http11.Http11NioProtocol&quot;<br>           port=&quot;8443&quot; maxThreads=&quot;200&quot;<br>           scheme=&quot;https&quot; secure=&quot;true&quot; SSLEnabled=&quot;true&quot;<br>           keystoreFile=&quot;${path_to_keystore}/sample_keystore.jks&quot; <br>           keystorePass=&quot;keystore_password&quot; <br>           keyPass=&quot;key_password&quot;<br>           clientAuth=&quot;false&quot; sslProtocol=&quot;TLS&quot;/&gt;</pre><p>Note that <strong><em>keystorePass</em></strong> is used to specify the “key store password” while <strong><em>keyPass</em></strong> is used to specify the “key password”.</p><p>Refer to the Tomcat <a href="https://tomcat.apache.org/tomcat-7.0-doc/ssl-howto.html#Prepare_the_Certificate_Keystore">documentation</a> for further details.</p><h4>List entries of JKS</h4><p>Now let’s list the key store entries of the <strong><em>“sample_keystore.jks</em>”</strong></p><pre>keytool -list -v -keystore sample_keystore.jks</pre><p>The entries will look like below with a <strong><em>PrivateKeyEntry</em></strong> and a <strong><em>Certificate chain</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hfSltvmjEng4C0JNt6Mg2w.png" /></figure><h4>Converting a JKS to PKCS12 keystore</h4><p>As you would have noticed when we created a JKS key store using the key tool, there was this following Warning message,</p><pre>Warning:<br>The JKS keystore uses a proprietary format. <br>It is recommended to migrate to PKCS12 which is an industry standard format.</pre><p>JKS and JCEKS key stores are in Java proprietary format, and it is recommended to use PKCS12 format key store as it is the standard industry format. Later in this post, we will discover more on JCEKS and PKCS12 key store formats.</p><p>Let’s convert the <strong><em>“sample_keystore.jks”</em></strong> to a PKCS12 key store with <strong><em>.pfx</em></strong> extension.</p><pre>keytool -importkeystore -srckeystore sample_keystore.jks \<br>-destkeystore sample_keystore.pfx -deststoretype pkcs12</pre><p>A successful conversion will result in the following outcome</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pb58TghQOt5qK_erECLv-g.png" /></figure><h4>Export public certificate from JKS in DER format (binary-encoded format)</h4><p>Now, let’s try to export the Public key certificate from the <strong><em>“sample_keystore.jks”</em></strong> with alias <strong><em>“sample</em>”</strong> to <strong><em>“sample.crt”</em></strong> Public certificate file.</p><pre>keytool -exportcert -alias sample -file sample.crt \<br>-keystore sample_keystore.jks</pre><p>Results will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-mqHv2RiZkcJbIgJmN5Bg.png" /></figure><blockquote>If you try to view the content of the “sample.crt” file it would be in a <br>non-human readable format. This is because the Public certificate generated is in a <strong>binary encoded certificate</strong>.</blockquote><p>Public certificates can have content with the following two encodings:</p><ul><li><strong><em>Binary DER-encoded certificates (non-human readable)</em></strong></li><li><strong><em>ASCII (Base64) PEM-encoded certificates (human-readable)</em></strong></li></ul><p><em>DER: Distinguished Encoding Rules (Subset of Basic Encoding Rules of ASN.1)<br>PEM: Privacy-Enhanced Mail</em></p><blockquote>Note: <br>Encoding is a process of transforming the data into a different format using a publicly available schema. It can easily be reversed. The purpose of encoding is not to keep the information secure rather it is to transform the data into a format safe to be consumed by different systems.</blockquote><p>Public certificate can have the following common file extensions:</p><ul><li><strong><em>“.crt”</em></strong> extension: Certificate may be encoded as binary DER Or ASCII PEM.</li><li><strong><em>“.der</em>”</strong> extension: Certificate encoded with binary DER.</li><li><strong><em>“.pem”</em></strong> extension: Certificate encoded with ASCII (Base64).</li></ul><p>So, to get the Public certificate in a human-readable format we can export it either using PEM encoded <strong><em>.crt</em></strong> extension Or PEM encoded <strong><em>.pem</em></strong> extension.</p><pre>keytool -exportcert -alias sample -file sample.pem \<br>-keystore sample_keystore.jks</pre><p>Exporting the Public certificate with <strong><em>.pem</em></strong> extension as above will still result in a binary encoded Public key certificate.</p><p>This is because, <strong><em>regardless of the -file format provided</em></strong>, keytool’s <strong><em>-exportcert</em></strong> command will <strong><em>by default always create the Public key certificate as a binary encoded certificate</em></strong>.</p><h4>Export public certificate from JKS in PEM format <br>(ASCII Base64 human-readble format)</h4><p>In order to export the Public certificate from the JKS key store in a PEM format, we need to pass an additional <strong><em>-rfc</em></strong> argument while using the <br><strong><em>-exportcert</em></strong> command.</p><pre>keytool -exportcert -alias sample -file sample.pem \<br>-keystore sample_keystore.jks -rfc</pre><p>Content of the <strong><em>“sample.pem</em>”</strong> Public certificate will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VSt5MzHR74TVV3va5le9tw.png" /></figure><p>As you can see from above PEM format Public certificate has the <br><strong> — — -BEGIN CERTIFICATE — — —</strong> and <strong>— — -END CERTIFICATE — — —</strong> tags.</p><h4>Import third-party public certificates to the JKS</h4><p>Let’s try to import a third-party’s Public certificate as a <strong><em>Trusted CA Certificate</em></strong>.</p><p>I have extracted the <strong><em>“wso2carbon.pem</em>”</strong> file from the <strong><em>“wso2carbon.jk</em>s”</strong> (default key store shipped with WSO2 Products).</p><p>Let’s try to import “wso2carbon.pem” Public certificate to the “sample_keystore.jks”</p><pre>keytool -import -trustcacerts -alias wso2carbon -file wso2carbon.pem -keystore sample_keystore.jks</pre><p>Results will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4WuftdNa-W1lEdelfsS45Q.png" /></figure><h4>Export private key entry from JKS</h4><p>By default, the JKS key store type doesn’t support exporting the Private key entry using the key tool. But it can be <strong><em>achieved by converting the JKS key store to a PKCS12 key store</em></strong>.</p><p>Let’s explore how we can export the Private key entry from the “sample_keystore.jks”,</p><ul><li>Convert the JKS “sample_keystore.jks” key store to a PKCS12 key store “sample_keystore.pfx”. We have already done it.</li><li>Export the Private key entry from the PKCS12 key store using the <strong><em>openssl</em></strong> into <strong><em>“sample_keystore_private_key.pem”</em></strong> file using PEM encoding</li></ul><pre>openssl pkcs12 -in sample_keystore.pfx -nodes -nocerts \<br>-out sample_keystore_private_key.pem</pre><p><strong><em>pkcs12</em></strong> =&gt; Key store format we are using.<br><strong><em>-nodes</em></strong> =&gt; No DES format so the Private key won’t be encrypted. <br><strong><em>-nocerts</em></strong> =&gt; No certificates at all will be output.</p><p>Exported Private key entry will look like the below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cV4UO9_H1Nu-IrfIGEjAPg.png" /></figure><p>We need to provide the “sample_keystore.pfx” key store password as the <strong><em>Import Password</em></strong>.</p><p>As you can see the Private key entry is with the PEM encoding contains<br><strong> — — -BEGIN PRIVATE KEY — — — </strong>and<strong> — — -END PRIVATE KEY — — — </strong>tags. The Private key is not encrypted.</p><h4>Convert the private key entry from ‘PEM’ encoding to ‘DER’ encoding</h4><p>By default, OpenSSL command saves the Private key entry (encrypted/unencrypted) to the file using the PEM encoding.</p><p>We can convert the PEM format to DER format using the below command,</p><pre>openssl pkey -in sample_keystore_private_key.pem -inform pem \<br>-out sample_keystore_private_key.der -outform der</pre><p>Exported Unencrypted Private key in DER format will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7IsF-cC-D33skfJQTfrviQ.png" /></figure><p>As you can see from above <strong><em>“sample_keystore_private_key.der”</em></strong> contains the <strong><em>unencrypted private key entry encoded in binary DER format</em></strong>.</p><h4>Export the Triple-DES encrypted private key entry from the JKS</h4><p>By default, if we don’t specify the <strong><em>encryption algorithm</em></strong> Or pass <strong><em>-nodes</em></strong> command, OpenSSL will encrypt the Private key entry using a <strong><em>3DES encryption algorith</em>m</strong>.</p><p>Following exports the <strong><em>PEM</em></strong> encoded <strong><em>3DES encrypted</em></strong> private key entry,</p><pre>openssl pkcs12 -in sample_keystore.pfx -nocerts \<br>-out sample_keystore_private_key_default_enc.pem</pre><p>As you can see from the above <strong><em>“sample_keystore_private_key_default_enc.pem”</em></strong> contains the <strong><em>Triple-DES encrypted (3DES) private key entry</em></strong>.</p><p>Please refer to the <a href="https://en.wikipedia.org/wiki/Triple_DES">link</a> for further details on Triple-DES encryption.</p><p>Note the tags with <strong>— — -BEGIN ENCRYPTED PRIVATE KEY — — — </strong>and <br><strong>— — -END ENCRYPTED PRIVATE KEY — — —</strong> which shows us the Private key entry is encrypted.</p><p>Apart from using the default 3DES algorithm, we can export the encrypted Private key entry by specifying one of the following encryption algorithms.</p><ul><li><strong>AES</strong> (aes128, aes192, and aes256) =&gt; <strong>-aes128</strong>, <strong>-aes192</strong>, <strong>-aes256</strong></li><li><strong>DES</strong> =&gt; <strong>-des</strong></li><li><strong>3DES</strong> =&gt; <strong>-des3</strong></li></ul><p>The default is triple DES.</p><h4>Export the ‘AES256’ encrypted private key entry from the JKS</h4><p>Let’s now export the AES256 encrypted Private key entry by, passing the encryption algorithm as below,</p><p>Convert the JKS key store to the PKCS12 key store and then execute the following command,</p><pre>openssl pkcs12 -in sample_keystore.pfx -aes256 -nocerts \<br>-out sample_keystore_private_key_aes256.pem</pre><p>Note that we are passing an additional <strong><em>-aes256 argument</em></strong>.</p><p>Exported AES256 encrypted private key will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1vCwcOlbQPnh7cY6X15PA.png" /></figure><h4>Export the ‘RSAPrivateKey’ from the JKS</h4><p>Convert the JKS key store to the PKCS12 key store.</p><p>As you may have noticed that the Private key entry contains <strong><em>Bag Attributes</em></strong> and <strong><em>Key Attributes</em></strong> apart from the Encrypted/Un-encrypted Private key entry.</p><p>This is because the private key entry we extracted has a <strong><em>PrivateKeyInfo</em></strong> ASN.1 structure as defined in the <strong><em>PKCS#8 (RFC 5208)</em></strong>. PrivateKeyInfo structure allows the Private key entry to contain:</p><ul><li>A Private key</li><li>An OID that identifies the key type</li></ul><p>Newer versions of OpenSSL generate Private key with PKCS#8 format which has — BEGIN PRIVATE KEY —and —END PRIVATE KEY — tags.</p><p>At the start of this blog, we created the “sample_keystore.jks” with an RSA 2048 Private and Public key pair. So we need to export the RSA Private key from the key store. Also, we need to use the RSA Private key for SSL communication.</p><p><strong><em>RSAPrivateKey</em></strong> ASN.1 structure defined in <strong><em>PKCS#1 (RFC 3447)</em></strong> allows the Private key entry to only contain the Private key eliminating the <br>“Bag Attributes” and “Key Attributes”.</p><p>Older versions of OpenSSL generate Private key entry in PKCS#1 format with <br> —BEGIN RSA PRIVATE KEY — and —END RSA PRIVATE KEY —tags.</p><p>Since we are using the newer version of OpenSSL to extract the Private key entry, we got the Private key entry in PKCS#8 format with OID.</p><p>In order to <strong><em>generate the Private key entry with “RSAPrivateKey”</em></strong> let’s use the below command,</p><pre>openssl pkcs12 -in sample_keystore.pfx -nodes -nocerts | \<br>openssl rsa -out sample_keystore_private_key_rsa.pem</pre><p>RSA Private key entry will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5RDAtdTpgVKovsJ0kkPRA.png" /></figure><p>As you can <strong><em>see from the above the Private key entry only contains the private key without other attributes</em></strong>.</p><p>Note the tags with —BEGIN RSA PRIVATE KEY — and <br>—END RSA PRIVATE KEY — which shows us the Private key entry is an <strong><em>RSAPrivateKey</em></strong>. The RSA private key is not encrypted.</p><h4>Extracting ‘RSAPrivateKey’ from the ‘PKCS#8’ private key entry</h4><p>Now let’s consider we already have a Private key entry in PKCS#8 format and we need to extract the RSA Private key from it.</p><p>Let’s use <strong><em>“sample_keystore_private_key.pem”</em></strong>, unencrypted PKCS#8 format private key entry and convert it to RSA Private key.</p><p>Let’s use the below command,</p><pre>openssl rsa -in sample_keystore_private_key.pem \<br>-out sample_keystore_private_key_rsa_converted.pem</pre><p>The converted <strong><em>“sample_keystore_private_key_rsa_converted.pem”</em></strong> will look like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95u84Tmmh17fV_t0jU9yrw.png" /></figure><h4>Extracting Triple DES encrypted ‘RSAPrivateKey’ from the ‘PKCS#8’ private key entry</h4><p>The RSA Private Key we exported Or converted is not encrypted and poses a security threat. Let’s see how we can encrypt the RSA private key while converting.</p><pre>openssl rsa -in sample_keystore_private_key.pem -des3 \<br>-out sample_keystore_private_key_rsa_des3.pem</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-v7iXE25oCU9l8E9lqnNw.png" /></figure><p>As you can see from the above, the RSA Private key is encrypted.</p><p>Note the <strong><em>Proc-Type: 4,ENCRYPTED</em></strong> attribute which shows us the RSA private key entry is encrypted.</p><h4>Import a private key entry to JKS</h4><p>We can’t directly import Private key entry to a JKS type key store.<br>We can import the Private key to a PKCS12 key store and convert it to a JKS key store.</p><p>Following command <strong><em>exports the private key, public certificate and certificate chain as a ‘PKCS12’ keystore</em></strong>,</p><pre>openssl pkcs12 -export -in &lt;public_cert&gt;.crt \<br>-inkey &lt;private_key&gt;.key -chain -CAfile &lt;ca_cert&gt;.crt \<br>-name &quot;sample.com&quot; -out sample_keystore_pkcs12.p12</pre><p>Be sure to <strong><em>set an export password</em></strong> to <strong><em>avoid NullPointerException</em></strong></p><p>Following command converts the ‘PKCS12’ key store to a JKS,</p><pre>keytool -importkeystore -srckeystore sample_keystore_pkcs12.p12 \<br>-srcstoretype PKCS12 \<br>-destkeystore sample_keystore_with_imported_private_key.jks \<br>-deststoretype JKS -deststorepass abcd@1234</pre><h4>Generating JKS with ‘Secret Key’ entry</h4><p>According to the JKS key store specification, <strong><em>we can’t store non-PrivateKeys in a “JKS” type key store</em></strong>.</p><p>Let’s verify the above statement by trying to generate a JKS key store with a Secret key as below,</p><pre>keytool -genseckey -alias secretKey -keypass secretKey \<br>-keyalg AES -keysize 256 -keystore secretKey.jks \<br>-storepass secretKey -storetype JKS -v</pre><p><strong><em>-genseckey</em></strong> option is used to generate the Secret key.<br><strong><em>-keyalg</em></strong> argument is used to pass the Secret key algorithm.<br>Common secret key algorithms include DES, 3DES, and AES.</p><p>The above command will result in a key tool error like below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*42ZILGIGzpPV5w1QVI4Ucg.png" /></figure><p>As you can see even though the <strong><em>256-bit AES secret key is generated it cannot be stored in the JKS type</em> <em>key store</em></strong>.</p><p>Similarly, <strong><em>we can’t import a Secret key entry to a JKS key store</em></strong>.</p><h4>References:</h4><p>[1] <a href="https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/security/KeyStore.html">https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/security/KeyStore.html</a><br>[2] <a href="https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html#keytool_option_genkeypair">https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html#keytool_option_genkeypair</a><br>[3] <a href="https://tomcat.apache.org/tomcat-7.0-doc/ssl-howto.html#Prepare_the_Certificate_Keystore">https://tomcat.apache.org/tomcat-7.0-doc/ssl-howto.html#Prepare_the_Certificate_Keystore</a><br>[4] <a href="https://en.wikipedia.org/wiki/Triple_DES">https://en.wikipedia.org/wiki/Triple_DES</a></p><p>We have explored different aspects of the JKS key store in this blog. <br>We will continue to explore the other types of key stores in future blogs.</p><p>Thank you for reading.<br>Cheers!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=723eb6b32a21" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>