<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by hai huang on Medium]]></title>
        <description><![CDATA[Stories by hai huang on Medium]]></description>
        <link>https://medium.com/@huang195?source=rss-4d3a69661639------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*hmOnUJYItGdUtY-y.jpg</url>
            <title>Stories by hai huang on Medium</title>
            <link>https://medium.com/@huang195?source=rss-4d3a69661639------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:45:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@huang195/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hands-On with MCP Gateway: From Local Setup to Agent Integration in Kagenti]]></title>
            <link>https://medium.com/rossoctl-the-agentic-platform/hands-on-with-mcp-gateway-from-local-setup-to-agent-integration-in-kagenti-f9bd3b7cc334?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9bd3b7cc334</guid>
            <category><![CDATA[enterprise-security]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[model-context-protocol]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 22:05:25 GMT</pubDate>
            <atom:updated>2025-12-05T17:39:25.140Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dT4lgFK6XFsPcCT3FOZChQ.png" /></figure><p><em>by Hai Huang, Evaline Ju, Gloire Rubambiza, Jim Cadden</em></p><p>Last time in <a href="https://medium.com/kagenti-the-agentic-platform/introducing-mcp-gateway-in-kagenti-a-unified-front-door-for-your-mcp-servers-28db5b6ef62d">“Introducing MCP-Gateway in Kagenti”</a>, we explored the <a href="https://github.com/kagenti/mcp-gateway">MCP-Gateway</a>’s architecture and how it stitches multiple MCP servers into a single unified interface. Now it’s time to get hands-on. In part one, we’ll spin up the gateway in a local Kind cluster and plug in some test MCP servers. In part two, we’ll run an end-to-end scenario to see how an agent in <a href="https://github.com/kagenti/kagenti">Kagenti</a> uses tools — through the gateway — to answer real user questions.</p><h3>Part 1: Local Quick Start</h3><h4>Prerequisites</h4><ul><li>Docker/Podman installed on your machine.</li><li>npm + kubectl + Kind</li></ul><p>Clone the repo:</p><pre>git clone https://github.com/kagenti/mcp-gateway<br>cd mcp-gateway</pre><h4>Installation</h4><p>One command to install everything you will need for this exercise:</p><pre>make local-env-setup</pre><p>This Make target will:</p><ul><li>Create a Kind cluster</li><li>Install <a href="https://istio.io/latest/">Istio</a> (as a Kubernetes <a href="https://gateway-api.sigs.k8s.io/">Gateway API </a>provider)</li><li>Deploy MCP gateway (Broker / Router / Controller)</li><li>Deploy the test MCP servers and example configurations</li></ul><h4>Testing</h4><p>To launch <a href="https://modelcontextprotocol.io/docs/tools/inspector">MCP Inspector </a>and test the installed MCP servers by pointing it to the MCP gateway:</p><pre>make inspect-gateway</pre><p>This starts MCP Inspector, an interface to give real-time visibility into MCP tool calls, and opens it with the correct <em>serverUrl</em> (the gateway’s MCP endpoint). In case the browser does not open the MCP Inspector web UI properly the first time, just hit refresh.</p><p>Once the Inspector connects, expand the tool list and run a couple of test tools that are aggregated via the gateway. Because the gateway prefixes tools by backend to prevent collisions, you’ll see them grouped by tool prefix (e.g., demo_, example_). The quick-start loads example configs to connect test servers to the gateway so you can experiment immediately. In the next section, we will show how these configs need to be constructed.</p><p>In the MCP Inspector, going to Tools → List Tools, you should see something similar below with a list of tools, aggregated from multiple MCP servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-kBA-vLbZEkL3BvvICvyw.png" /></figure><p>Clicking on one of these tools, you can run it, e.g., <em>test2_hello_world</em> would give the following response:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1G-Ndg5fXLxkfyjbUREypg.png" /></figure><p>This exercise shows how the gateway is able to aggregate tools from multiple MCP servers and route tool calls to their corresponding server.</p><p>Before proceeding to Part 2, where Kagenti will stand up its own Kind cluster, it is best to clean up the Kind cluster that MCP gateway had created using the following command:</p><pre>make local-env-teardown</pre><h3>Part 2: Using MCP-Gateway in Kagenti with a weather agent</h3><p>Now that you were able to get the gateway running locally, let’s plug it into an agent platform to see it in action. You can follow the introduction blog <a href="https://medium.com/kagenti-the-agentic-platform/hands-on-introduction-to-agentic-ai-concepts-with-kagenti-d70f15aa9051">“Hands-On Introduction to Agentic AI Concepts with Kagenti”</a> to set up the basic weather example. Check that you can already interact with the agent.</p><p>Here we expand on the basic weather example by leveraging MCP gateway to call the tool. There is still only a single tool, but we can imagine the same backend tool server serving more tools, or different tool servers being available that can be connected with the same pattern.</p><p>First, check on the status of MCP gateway components in the <em>mcp-system</em> and <em>gateway-system</em> namespaces. The broker, router, and discovery controller are in the <em>mcp-system</em> namespace, and the <a href="https://www.envoyproxy.io/">Envoy proxy</a> is in the <em>gateway-system</em> namespace. Register the weather tool that contacts the <a href="https://open-meteo.com/">Open-meteo weather API</a> with the gateway. To do this, create an <a href="https://gateway-api.sigs.k8s.io/api-types/httproute/">HTTPRoute</a>. HTTPRoutes are Kubernetes resources defined by the <a href="https://gateway-api.sigs.k8s.io/">Gateway API </a>to describe how HTTP traffic should be routed from a gateway (where traffic enters a cluster) to backend services. Paste the following into a file and apply via <em>kubectl apply -f [file-name]</em>, replacing namespace as needed with the one that your tool is available in.</p><pre>apiVersion: gateway.networking.k8s.io/v1<br>kind: HTTPRoute<br>metadata:<br>  name: weather-tool-route<br>  namespace: task1<br>  labels:<br>    mcp-server: &quot;true&quot;<br>spec:<br>  parentRefs:<br>  - name: mcp-gateway<br>    namespace: gateway-system<br>  hostnames:<br>  - &quot;weather-tool.mcp.local&quot; #note this is matching the gateway listener. It is purely for internal routing by envoy<br>  rules:<br>  - matches:<br>    - path:<br>        type: PathPrefix<br>        value: /<br>    backendRefs:<br>    - name: weather-tool<br>      port: 8000</pre><p>Then create a <em>MCPServer</em> custom resource (CR) to represent the backend server, noting the namespace where the HTTPRoute is present. Again, paste the following into a file and apply via <em>kubectl apply -f [file-name]</em>. The <em>toolPrefix</em> represents the prefix added to all the tools from this particular backend server, and the <em>targetRef</em> references the HTTPRoute for the backend.</p><pre>apiVersion: mcp.kagenti.com/v1alpha1<br>kind: MCPServer<br>metadata:<br>  name: weather-tool-servers<br>  namespace: task1<br>spec:<br>  toolPrefix: weather_<br>  targetRef:<br>    group: gateway.networking.k8s.io<br>    kind: HTTPRoute<br>    name: weather-tool-route<br>    namespace: task1</pre><p>The weather agent has to be told to leverage the MCP gateway instead of the weather tool directly. This would allow an agent to access all the tools that would be managed by the gateway.</p><p>If you look at the weather agent deployment now through <em>kubectl get deployment/weather-service -n task1 -o yaml </em>you can see something like:</p><p><em>MCP_URL: </em><a href="http://weather-tool:8000/mcp"><em>http://weather-tool:8000/mcp</em></a>. This indicates that the agent is talking to the tool directly. Update the <em>MCP_URL</em> to point to the gateway envoy proxy — namely set <em>MCP_URL</em> in the deployment to <a href="http://mcp-gateway-istio.gateway-system.svc.cluster.local:8080/mcp"><em>http://mcp-gateway-istio.gateway-system.svc.cluster.local:8080/mcp</em></a><em>.</em></p><p>You can edit the deployment manually or patch it via:</p><pre>kubectl set env deployment/weather-service -n task1 MCP_URL=&quot;http://mcp-gateway-istio.gateway-system.svc.cluster.local:8080/mcp&quot; </pre><p>For reference, you can review all the <a href="https://github.com/kagenti/kagenti/blob/main/docs/gateway.md#weather-agent--tools-no-auth">steps here</a>.</p><p>After the deployment is successfully running again, you can try an agent call. If the setup is <strong>not</strong> done correctly, the agent may cite a knowledge cutoff and give historical weather data, say something about the weather in general, or potentially hallucinate completely. If done correctly, the agent will give similar style real-time responses to before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7s4kFRqxJpZsrGzj14JFA.png" /></figure><h3>Next Steps</h3><p>In this walkthrough, we explored how to get started with an MCP gateway locally, then how it integrates with the Kagenti agentic platform with the weather tool example. In future posts, we will dive deeper into the technical aspects of the MCP gateway and provide the agent developer perspective for leveraging Kagenti and the gateway.</p><p>💡 Explore the open-sourced Github projects: <a href="https://github.com/kagenti/kagenti">Kagenti </a>and <a href="https://github.com/kagenti/mcp-gateway">MCP gateway</a><br>💬 Join the community at <a href="https://discord.gg/4SHAQ3Hgm4">https://discord.gg/4SHAQ3Hgm4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9bd3b7cc334" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rossoctl-the-agentic-platform/hands-on-with-mcp-gateway-from-local-setup-to-agent-integration-in-kagenti-f9bd3b7cc334">Hands-On with MCP Gateway: From Local Setup to Agent Integration in Kagenti</a> was originally published in <a href="https://medium.com/rossoctl-the-agentic-platform">Rossoctl — The Cloud Native Agentic Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing MCP-Gateway in Kagenti: A Unified Front Door for Your MCP Servers]]></title>
            <link>https://medium.com/rossoctl-the-agentic-platform/introducing-mcp-gateway-in-kagenti-a-unified-front-door-for-your-mcp-servers-28db5b6ef62d?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/28db5b6ef62d</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[model-context-protocol]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Thu, 06 Nov 2025 00:33:42 GMT</pubDate>
            <atom:updated>2025-11-06T00:33:42.803Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dCVaGyXmbykeSrt8ioKyZw.png" /></figure><p><em>By Hai Huang, Maia Iyer, Evaline Ju, Mariusz Sabath, Paolo Dettori</em></p><h3>The Challenge: Tool Sprawl in Agentic AI</h3><p>As AI agents become more sophisticated, they increasingly depend on external tools and services — whether those are data lookups, external APIs, or specialize business logic. The Model Context Protocol (MCP) emerged to standardize these integrations: agents (MCP clients) can discover and invoke tools exposed by MCP servers.</p><p>But once you deploy multiple MCP servers — perhaps one for weather data, another for calendar/CRM tools, another for file-store operations — you’ll hit these real-world pain points:</p><ul><li>Each agent must know every server’s endpoint and hostname.</li><li>Tool name collisions become common (e.g., two different servers define <em>getEvents</em> tool).</li><li>Routing, versioning, authentication, monitoring get reimplemented for each service.</li><li>Without a central layer, security controls and observability are inconsistent, especially as the number of tools scales.</li></ul><p>This is where MCP-Gateway comes in.</p><h3>What Is MCP-Gateway in Kagenti?</h3><p><a href="https://github.com/kagenti/mcp-gateway">MCP-Gateway</a> is an <a href="https://www.envoyproxy.io/">Envoy</a>-based gateway and routing layer that sits between your agents and internal and external MCP servers. It provides a <strong>single unified endpoint</strong> into your tool ecosystem, and handles discovery, routing, metadata, authentication, and policy enforcement.</p><p>With MCP-Gateway, agents connect to one endpoint, and the gateway handles the rest.</p><h3>Why Use MCP-Gateway?</h3><h4>Unified Endpoint for Agents</h4><p>All agents are configured to talk to one endpoint, not dozens. The gateway abstracts the backend topology so changes don’t break agent code.</p><h4>Simplified Discovery &amp; Routing</h4><p>Backends can be added or removed behind the gateway without needing to update every agent — you simply update the gateway config or custom resource definitions (CRDs). Tool prefixes prevent name clashes.</p><h4>Centralized Policy &amp; Observability</h4><p>Security controls, rate limiting, auditing, tracing happen in one place. Your backend servers don’t each have to implement their own. You get a central observability plane across all tool usage.</p><h4>Flexible Deployment Options</h4><p>Run in standalone (YAML) or Kubernetes (CRD) mode, with full integration into Envoy and the Gateway API.</p><h4>Enterprise-Scale Ready</h4><p>As the number of tools and teams grows, the gateway becomes the backbone for governance and reliability. Without it you risk sprawl, shadow tools, inconsistent security, and brittle agent code.</p><h4>Integration with Enterprise Security Tools</h4><p>One of the biggest values of MCP-Gateway is centralizing security and policy. All agent-to-tool traffic passes through one gateway, enabling consistent enforcement and observability.</p><p>The gateway integrates seamlessly with policy engines like <a href="https://github.com/kuadrant"><strong>Kuadrant</strong></a>, which extends Envoy’s external authorization filters to apply fine-grained rules — RBAC, rate limits, OpenID scopes, and more. Kuadrant is optional; you can also use OPA, Keycloak Authorization Services, or a custom WASM module.</p><p>This plug-and-play design makes MCP-Gateway adaptable for a wide range of production environments — from simple setups with static rules to enterprise deployments with dynamic, policy-as-code enforcement.</p><h3>How it works: High-Level Architecture</h3><p>Here’s a diagram of the core components of the MCP Gateway:</p><figure><img alt="Fig 1. Key components of the MCP gateway architecture" src="https://cdn-images-1.medium.com/max/1024/1*Ock1uC5XWB_i5vHNutNZuA.png" /><figcaption>Fig 1. Key components of the MCP gateway architecture</figcaption></figure><p>The core components include the Broker, the Router, and the Discovery Controller which configures the gateway based on present MCPServer Custom Resources. All of the traffic routing is facilitated by Envoy.</p><h4>Broker</h4><p>The <strong>Broker</strong> acts as the <strong>control hub</strong> of the MCP-Gateway. It keeps track of what tools exist, where they live, how they should be used, and who can use them.</p><p>Key responsibilities:</p><ul><li><strong>Discovery &amp; Registration</strong> — Tracks available MCP servers via config files (standalone) or CRDs and HTTPRoutes (Kubernetes).</li><li><strong>Virtual MCP Server</strong> — Exposes a filtered view tailored to each agent’s needs.</li><li><strong>Tool Prefix Management</strong> — Prevents naming collision by isolating tools with prefixes.</li><li><strong>Policy Catalog</strong> –Stores auth scopes, allowed metadata, and routing constraints.</li><li><strong>Declarative Security </strong>— Defines OAuth, metadata tagging, and policies via YAML and CRDs.</li></ul><h4>Router</h4><p>The <strong>Router</strong> is the runtime data path. This is where requests meet policy. It processes incoming MCP traffic, applies routing logic, injects metadata, and integrates with Envoy filters for security and observability.</p><p>If the Broker is the brain, the Router is the spinal cord of the system — fast, reactive, and policy-driven.</p><p>Key Responsibilities</p><ul><li><strong>Routing &amp; Dispatch</strong> — Examines incoming MCP requests, looks up the right server from the Broker’s routing table, and forwards traffic accordingly.</li><li><strong>Metadata Injection</strong> — Adds useful headers such as <em>x-mcp-tool</em>, <em>x-mcp-server,</em> and session identifiers for downstream observability or auditing.</li><li><strong>Authentication &amp; Authorization</strong> — Validates tokens and enforces resource-level access before forwarding traffic (based on Broker’s policy catalog).</li><li><strong>Streaming &amp; Session Handling</strong> — Supports both HTTP and streaming transports (SSE or chunked responses) so tools that produce long-running outputs can still operate efficiently.</li></ul><h4>Discovery Controller</h4><p>In Kubernetes, MCP-Gateway supports a declarative model: you define MCPServer CRs and HTTPRoute resources from the Gateway API, and the controller watches them and updates the broker and router dynamically. No manual config required.</p><p>Example CRD:</p><pre>apiVersion: mcp.kagenti.com/v1alpha1<br>kind: MCPServer<br>metadata:<br>  name: calendar-tools<br>spec:<br>  targetRef:<br>    group: gateway.networking.k8s.io<br>    kind: HTTPRoute<br>    name: calendar-route<br>  toolPrefix: cal_</pre><p>This makes adding and removing backend services an operation carried out via Kubernetes objects, rather than manually editing YAML configs.</p><h4>Envoy Data Plane</h4><p>MCP-Gateway uses Envoy as its data-plane proxy, inheriting all the strengths of Envoy — TLS termination, observability, rate limiting, and authorization filters. Envoy manages routing, transport (HTTP/gRPC), and metadata injection, while MCP-Gateway adds tool-aware logic and policies on top.</p><h3>Summary</h3><p>If you’re building an agent-tool architecture with multiple MCP servers, you’re juggling more than just making tool calls — you’re managing discovery, routing, security, monitoring, versioning, namespace collisions, and scale. The MCP-Gateway gives you a <strong>centralized, scalable, policy-first front door</strong> into your tool ecosystem.</p><p>Key features:</p><ul><li>Aggregation of multiple MCP servers behind one gateway.</li><li>Tool-prefixing and namespace isolation (so you avoid collisions).</li><li>Support for both standalone (file-based config) mode and Kubernetes controller mode.</li><li>Integration with the Kubernetes Gateway API (HTTPRoute) and CRDs for MCPServer.</li><li>Metadata tagging, routing logic, and ability to plug in Envoy filters for auth/rate-limit/etc.</li></ul><p>You’ll reduce complexity for agents, enforce governance consistently, monitor tool usage across the board, and retain flexibility to grow. In short — less tool-sprawl, more control!</p><p>Stay tuned for upcoming posts on how MCP-Gateway integrates with identity systems (SPIFFE/SPIRE) and policy engines like Kuadrant for Zero Trust enforcement.</p><h3>Next Steps</h3><p>💡 Explore the open-sourced Github projects: <a href="https://github.com/kagenti/kagenti">Kagenti </a>and <a href="https://github.com/kagenti/mcp-gateway">MCP gateway</a><br>💬 Join the community at <a href="https://discord.gg/4SHAQ3Hgm4">https://discord.gg/4SHAQ3Hgm4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28db5b6ef62d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rossoctl-the-agentic-platform/introducing-mcp-gateway-in-kagenti-a-unified-front-door-for-your-mcp-servers-28db5b6ef62d">Introducing MCP-Gateway in Kagenti: A Unified Front Door for Your MCP Servers</a> was originally published in <a href="https://medium.com/rossoctl-the-agentic-platform">Rossoctl — The Cloud Native Agentic Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 5. Container Storage Interface (CSI)]]></title>
            <link>https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/6119770cdfea</guid>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[spire]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[workload-identity]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Mon, 19 May 2025 13:03:17 GMT</pubDate>
            <atom:updated>2025-05-19T20:42:44.561Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkqX5fXeSAiBnBLSERVTjA.png" /></figure><h3>Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 5. Container Storage Interface (CSI)</h3><p>By Hai Huang, Maia Iyer, Mariusz Sabath, James Cadden (IBM Research)</p><h3>Blog Series</h3><p>Part 1: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e">Introduction</a><br>Part 2: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336">SPIFFE-helper</a><br>Part 3: <a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033">Seccomp Unotify</a><strong><br></strong>Part 4: <a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a">Container Lifecycle Hooks</a><br><strong>Part 5: Container Storage Interface (this article)</strong></p><p>In earlier posts, we explored a range of prototype solutions aimed at making SPIRE more accessible to application developers — each offering unique advantages but also introducing its own limitations. In this concluding entry of the series, we present a solution built on the Container Storage Interface (CSI), which addresses the shortcomings of our previous approaches and offers a practical, robust path forward.</p><h3><strong>Container Storage Interface</strong></h3><p>The Container Storage Interface (CSI)[<a href="https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/">1</a>] is a standardized API that enables container orchestration platforms like Kubernetes to expose arbitrary storage systems to containers in a consistent and portable way. Before CSI, integrating storage into Kubernetes clusters typically required built-in support for specific storage providers, making it difficult to extend or customize. CSI decouples storage logic from the core orchestrator, enabling vendors and platform engineers to develop and deploy storage plugins independently.</p><p>Although originally designed to manage block and file storage volumes, CSI’s plugin model can be creatively repurposed to deliver other types of data into a container’s filesystem. By leveraging this extensibility, we can treat workload identity artifacts — such as SPIFFE SVIDs — as “volumes” that are automatically mounted at runtime. This is particularly powerful because the CSI driver runs with elevated privileges outside the workload’s namespace, giving it the ability to provision identity material <em>on behalf of</em> containers, without requiring any cooperation from the workload itself.</p><p>This approach provides several key benefits. It sidesteps the limitations of lifecycle hooks like PostStart (i.e., race condition and restarting container runtime), avoids the need for custom sidecars or intrusive application changes, and enables fine-grained control over identity distribution based on pod metadata. By piggybacking on a well-supported and battle-tested interface like CSI, we gain a robust foundation for integrating SPIRE into containerized environments in a clean, modular, and platform-native way.</p><h3><strong>The Existing SPIFFE CSI driver</strong></h3><p>As discussed in earlier posts, SPIRE already includes a CSI driver known as the SPIFFE CSI Driver. This driver is deployed as a DaemonSet, ensuring it runs on every worker node in the cluster. Its sole responsibility is to bind-mount the SPIRE agent’s Unix Domain Socket into workload pods. While this makes the agent’s Workload API accessible from within the container, it places the burden of integration on the application. Specifically, workload processes must actively communicate with the SPIRE agent over the socket, adhering to the SPIFFE Workload API. This can be done either directly — by implementing the API logic themselves — or indirectly using an auxiliary tool such as SPIFFE-helper to automate SVID retrieval and renewal.</p><p>While functional, this approach introduces several challenges. First, it requires application-level awareness of SPIFFE, which can be intrusive and burdensome — especially for legacy workloads or third-party software. Second, managing helper processes like SPIFFE-helper adds operational complexity and resource overhead. Finally, because the SPIRE Workload API socket is mounted <em>into</em> the workload’s namespace, the workload must initiate the attestation workflow itself, limiting opportunities for platform operators to enforce or automate identity provisioning in a uniform way. These constraints motivated us to explore an alternative design — one that enables the platform, rather than the workload, to manage SPIFFE identity delivery in a transparent and secure manner.</p><h3><strong>Our New SPIFFE CSI driver</strong></h3><p>At its core, our new CSI driver builds on the same fundamental concept explored in earlier solutions based on Seccomp Unotify and Container Lifecycle Hooks: enabling the platform to provision SPIFFE identities on behalf of workloads, without requiring direct application involvement. CSI simply offers a cleaner and more maintainable mechanism for delivering these identity artifacts. Rather than relying on low-level syscall interception or lifecycle race conditions, the CSI framework provides a stable, orchestrator-native channel for injecting data into containers. The following diagram outlines the key steps involved in our custom CSI driver workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*gtnLeyv9Z3iSKxJdP72iag.png" /></figure><ol><li><strong>Pod Creation</strong>: A developer deploys a workload pod that references the custom SPIFFE CSI driver as a volume source.</li></ol><p>2. <strong>CSI Invocation</strong>: As the container runtime begins setting up the pod sandbox — including creating the necessary cgroups — it detects the CSI volume and invokes our driver.</p><p>3. <strong>Cgroup Impersonation</strong>: Just like in our earlier prototypes, the CSI driver temporarily “impersonates” the workload by attaching to its cgroup. This allows the driver to perform actions on behalf of the workload, as if it were the container process itself.</p><p>4. <strong>Attestation with SPIRE</strong>: Within the workload’s cgroup context, the CSI driver communicates with the SPIRE agent via its Unix Domain Socket, performing the attestation flow and securely retrieving the workload’s SPIFFE identity files (SVIDs, keys, and bundles).</p><p>5. <strong>Identity Injection</strong>: After successful attestation, the driver writes the identity material directly into the CSI-mounted volume, which is already wired into the workload container’s filesystem.</p><p>6. <strong>Cgroup Restoration</strong>: Finally, the CSI driver exits the impersonation context and returns to its original cgroup, completing the operation cleanly and without leaving behind any privileged state.</p><h3><strong>How are the CSI drivers different?</strong></h3><p>Although both are CSI drivers, the interface they expose to workloads differs significantly. The existing SPIFFE CSI driver simply mounts a Unix Domain Socket into the container, requiring workloads to speak the SPIFFE Workload API to retrieve identity documents. In contrast, our new CSI driver mounts the identity artifacts directly — as read-only files such as X.509 SVIDs, private keys, CA bundles, or JWT-SVIDs — into the container’s filesystem. This approach eliminates the need for additional helper programs or API integration, enabling applications to consume SPIFFE identities natively through familiar file-based mechanisms (e.g., TLS libraries, curl, or sidecar-less service meshes).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*K0V-YU1ki6F7FEMWGSPc3w.png" /></figure><p>An important security benefit of this design is that it removes the need to expose the SPIRE agent’s Unix Domain Socket directly to untrusted workloads. This eliminates a critical attack surface: when arbitrary user workloads can communicate with the SPIRE agent over its socket, any vulnerabilities — whether in the kernel, socket handling code, or third-party libraries used by the agent — could potentially be exploited. By acting as an isolated intermediary, the CSI driver decouples workload access from the SPIRE agent entirely, reducing the risk of privilege escalation or lateral movement. In essence, our approach not only simplifies identity delivery, but also hardens the trust boundary between the SPIRE agent and the workloads it serves. This comparison between the CSI drivers is depicted in the diagram above.</p><h3><strong>Developer’s Perspective</strong></h3><p>CSI drivers are usually deployed to an environment by the infrastructure management team. For a developer to make use of a CSI driver, he needs to specify the following lines in his deployment YAML file:</p><pre>volumeMounts:<br> — name: csi-identity<br>   mountPath: /var/run/secrets/spire.io<br>   readOnly: true</pre><p>This tells the CSI driver to place X.509 certificate and key, along with a CA bundle file, into the <em>/var/run/secrets/spire.io/</em> directory at the container startup time. This is similar to how Kubernetes places service account tokens into the <em>/var/run/secrets/kubernetes.io</em> directory.</p><p><strong>Mutating Admission Webhook</strong></p><p>To further streamline adoption, we introduced a lightweight mutating admission webhook that allows developers to enable the new CSI driver with a simple pod annotation:</p><pre>spiffe.io/inject-cert: “true”</pre><p>Behind the scenes, the webhook automatically modifies the pod spec to inject the necessary volume and volumeMount blocks for the CSI driver — removing the need for developers to manually edit deployment manifests. In its current form, the webhook provisions X.509 SVIDs, but the annotation-based interface makes it easy to extend support for JWT-SVIDs or other identity formats in the future. This annotation-driven model also provides a flexible way to configure driver behavior per workload, enabling fine-grained control without introducing complexity for application teams.</p><h3><strong>Challenges and Next Steps</strong></h3><p>We are fairly satisfied with how our new SPIFFE CSI driver works. It addresses the usability and security issues of the current CSI driver, while avoids all the pitfalls of the two other prototypes that we had explored so far. However, in its current state, it does have its own caveats:</p><p><strong>Pod-level attestation</strong></p><p>Since a CSI driver is invoked very early in a pod’s lifecycle, it faces the same challenge as our solution based on Container Lifecycle Hooks[<a href="https://github.com/opencontainers/runtime-spec/blob/main/runtime.md">2</a>] — specifically, the timing issue we previously mitigated using the PostStart hook. However, with a CSI driver, we don’t even have the option to delay code execution until the SPIRE agent is ready to properly attest workloads. The root cause of this problem is that when the SPIRE agent performs workload attestation, it expects the attesting process to be running inside of a container in a pod. To verify this, it checks the status.containerStatuses field of the pod’s metadata. However, since none of the containers, including the init containers, have started yet, the agent finds no containers and always return an error.</p><p>This behavior can be suppressed by setting the option <em>spire-agent.workloadAttestors.k8s.disableContainerSelectors</em> to true when installing SPIRE. As containers of the same pod already share many resources, e.g., network namespace, service account, hostname, etc., there is not a lot of practical value in differentiating the containers within the same pod. However, even after setting the above configuration option, attestation was still failing.</p><p>Digging deeper into the attestation code in the SPIRE agent, it turns out that the Kubernetes attestor expects the attesting process is residing in a specific cgroup hierarchy, such as:</p><pre>/sys/fs/cgroup/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod[podID].slice/cri-containerd-[containerID].scope</pre><p>This is the standard cgroup layout used by Kubernetes and distributions like OpenShift. The first-level cgroup directory represents the pod, while the second level contains the cgroups for the containers within that pod. When our CSI driver is invoked, only the pod-level cgroup has been created by the container runtime; the container-level cgroups do not yet exist. Attempting to enter the pod-level cgroup and perform attestation with the SPIRE Agent at this point will result in a failure.</p><p>To solve this, we modified our CSI driver to first create a temporary container-level cgroup within the existing pod-level cgroup directory, and this is used exclusively for workload attestation. This solution cleanly resolves the issue without exposing any implementation details to the workload pods.</p><p><strong>Next Steps</strong></p><ol><li><strong>Token Refresh</strong><br> Some attentive readers may have noticed that all the proof-of-concept implementations we presented only handle the initial acquisition of workload identities when a workload starts. However, under Zero Trust principles, security tokens should be short-lived and regularly rotated. In the CSI driver implementation, this could be achieved by starting a per-pod background thread that periodically acquires new identities on behalf of the workload, following the same process used during initial startup.</li><li><strong>Configurable Options</strong><br> Different workloads may have varying configuration requirements — for example, choosing between X.509 certificates and JWT tokens, setting the token refresh interval, or specifying the names of identity files. These options could be exposed as pod annotations, read by the mutating webhook, and then passed as parameters to the CSI driver.</li><li><strong>Resilient Failure Recovery</strong><br> As with any software system, failures are inevitable. If the CSI driver crashes, the local runtime system (e.g., Kubelet) will eventually restart it. Ideally, the restarted driver should resume from where it left off, allowing workloads that rely on periodic token refreshes to continue running without interruption. To enable this, relevant state information could be stored as metadata associated with the workloads. Upon restart, the CSI driver could scan all local pods and spawn background threads to resume token management for each affected pod.</li></ol><h3><strong>Conclusion</strong></h3><p>This is the final part of this blog series. It was very educational for us to explore various options to make SPIRE more friendly to use for developers. We are happy with our final solution based on the CSI driver, and we are currently working on the next steps we outlined above to make the initial proof-of-concept more production-ready, and in a format that we think can work with the SPIFFE/SPIRE community to be upstreamed. We have open sourced both the new CSI driver and its mutation webhook. Feel free to try these out yourself!</p><ul><li>Our SPIFFE CSI Driver: <a href="https://github.com/huang195/spire-csi-external">https://github.com/huang195/spire-csi-external</a></li><li>Mutating webhook: <a href="https://github.com/huang195/spire-admission-controller-external">https://github.com/huang195/spire-admission-controller-external</a></li></ul><p>We look forward to working with the SPIFFE/SPIRE community, and more broadly, the CNCF community, to learn more ways to make it even more developer-friendly, so workload identities can become an integral part of any workload with minimal, or even zero, effort from developers. If you have any questions about this work, please feel free to reach out to any of the authors on the SPIRE Slack channel!</p><h3><strong>About Us</strong></h3><p>Our team at IBM Research is dedicated to advancing technologies that enhance modern applications by integrating Zero Trust principles. You can find our previous posts covering SPIFFE/SPIRE and workload identity here: (<a href="https://medium.com/universal-workload-identity">our blog</a>).</p><h3><strong>Citations</strong></h3><ol><li><a href="https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/">https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/</a></li><li><a href="https://github.com/opencontainers/runtime-spec/blob/main/runtime.md">https://github.com/opencontainers/runtime-spec/blob/main/runtime.md</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6119770cdfea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea">Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 5. Container Storage Interface (CSI)</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 4. Container Lifecycle Hooks]]></title>
            <link>https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/0375881bd88a</guid>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[spire]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 16:50:53 GMT</pubDate>
            <atom:updated>2025-05-19T13:07:21.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkqX5fXeSAiBnBLSERVTjA.png" /></figure><h3>Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 4. Container Lifecycle Hooks</h3><p>By Hai Huang, Maia Iyer, Mariusz Sabath, James Cadden (IBM Research)</p><h3>Blog Series</h3><p>Part 1: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e">Introduction</a><br>Part 2: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336">SPIFFE-helper</a><br>Part 3: <a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033">Seccomp Unotify</a><strong><br>Part 4: Container Lifecycle Hooks (this article)</strong><br>Part 5: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea">Container Storage Interface</a></p><p>In our previous blog post, we explored how Seccomp Unotify can be used to allow developers to leverage SPIRE-issued identities within their workloads — without requiring any code changes. However, the performance overhead caused by context switching on every <em>open()</em> and <em>openat()</em> call indiscriminately can pose challenges for certain workload types. In this article, we present an alternative approach that significantly reduces this overhead while maintaining the same seamless, developer-friendly experience.</p><h3>Container Lifecycle Hooks</h3><p>In the Open Container Initiative (OCI) runtime specification [<a href="https://github.com/opencontainers/runtime-spec/blob/e935f995dd67c853bd93d6937bab898d36eb1747/config.md">1</a>], a container always goes through the following list of lifecycle events:</p><ul><li><strong>CreateRuntime</strong>: Sets up the runtime environment needed to manage containers (e.g., creating necessary resources and configurations).</li><li><strong>CreateContainer</strong>: Creates the container instance, including setting up namespaces, cgroups, and the root filesystem.</li><li><strong>StartContainer</strong>: Launches the container’s init process, transitioning it from created to running.</li><li><strong>PostStart</strong>: Executes any lifecycle hooks defined to run immediately after the container starts.</li><li><strong>PreStop</strong>: Executes lifecycle hooks just before the container is stopped, allowing for graceful shutdown tasks.</li></ul><p>Container Lifecycle Hooks are optional commands or scripts configured to run at these specific lifecycle events. These hooks allow custom behavior to be injected during key phases of container execution, and they give operators and platform developers greater control over container behavior without modifying the containerized application itself.</p><p>In this context, it’s easy to envision leveraging one of these hooks to run a lightweight script that communicates with a local SPIRE agent, retrieves the container’s identities, and places them in a location accessible to the container’s processes. To ensure the identity files are available from the outset, this hook should be triggered early in the container’s lifecycle — ideally before any processes have started.</p><p>We initially considered the <em>CreateRuntime</em> hook, but it occurs too early in the container’s lifecycle — before the runtime has set up any cgroups or started container processes. At this stage, attestation with the SPIRE agent isn’t feasible. Conversely, the <em>CreateContainer</em> and <em>StartContainer</em> hooks present a different challenge: their code executes within the container’s namespace, limiting our ability to use techniques like cgroup hopping to impersonate the workload and acquire identity files on its behalf. The <em>PreStop</em> hook, being triggered only at the end of a container’s lifecycle, is not applicable for our purposes. This leaves <em>PostStart</em> as the most viable option. Since <em>PostStart</em> runs in the runtime’s namespace, it allows us to perform cgroup hopping freely. However, this hook point also comes with certain limitations, which we will address in the following sections.</p><h3>Create and Install Hooks</h3><p>The structure of the hook code is similar to the Seccomp Unotify version we described in our previous blog. Instead of invoked by syscalls, i.e., <em>open()</em> and <em>openat()</em>, this code is triggered by <em>PostStart</em> event. As a result, it is only invoked once per container lifetime instead of continuously by every <em>open()</em> and <em>openat()</em> invocation. This results in a much lower runtime performance impact.</p><p>The hook code is consisted of three main parts: i) cgroup hopping, ii) acquiring identity files, and iii) placing them into the workload’s filesystem.</p><p><strong>1. Cgroup Hopping</strong> <br>This is the function to enter a cgroup. It places the process <em>sourcePID</em> into the same set of cgroups as that of the process <em>targetPID</em>.</p><pre>enterCgroup() { <br> sourcePID=$1 <br> targetPID=$2 <br> <br> while IFS= read -r line; do <br>   IFS=’:’ read -ra items &lt;&lt;&lt; “$line” <br>   if [ ${#items[@]} -eq 3 ]; then <br>     path=”/sys/fs/cgroup/${items[1]}${items[2]}/cgroup.procs” <br>     echo “path: $path” <br> <br>     # enter cgroup <br>     echo “$sourcePID” &gt; $path <br> <br>   else <br>     echo “Error: cannot parse cgroup file” <br>     exit 1 <br>   fi <br>   done &lt; “/proc/$targetPID/cgroup” <br> }</pre><p><strong>2. Acquiring identity files</strong></p><p>By leveraging <em>cgroup hopping</em>, we can impersonate the workload and securely acquire its identity files from a local SPIRE agent, as shown below. In this context, <em>$BASHPID</em> refers to the process ID of the hook script itself, while <em>$CONTAINER_PID</em> represents the init process ID of the containerized workload. Using the <em>enterCgroup()</em> function defined earlier, we temporarily “hop” into the workload’s cgroups and then back out. While operating within the workload’s cgroups, we perform attestation with the SPIRE agent.</p><pre>enterCgroup $BASHPID $CONTAINER_PID<br>/root/spire-1.10.3/bin/spire-agent api fetch -socketPath /var/run/spire/agent-sockets/spire-agent.sock -write /tmp<br>enterCgroup $BASHPID $$</pre><p><strong>3. Placing identity files</strong></p><p>Finally, we place the identity files to the container’s file system where its processes can use.</p><pre>nsenter -t $CONTAINER_PID — mount sh -c “mkdir -p /var/run/secrets/spire.io/”<br>cat /tmp/bundle.0.pem | nsenter -t $CONTAINER_PID — mount sh -c “cat &gt; /var/run/secrets/spire.io/bundle.0.pem”<br>cat /tmp/svid.0.key | nsenter -t $CONTAINER_PID — mount sh -c “cat &gt; /var/run/secrets/spire.io/svid.0.key”<br>cat /tmp/svid.0.pem | nsenter -t $CONTAINER_PID — mount sh -c “cat &gt; /var/run/secrets/spire.io/svid.0.pem”</pre><p>The full shell script can be found <a href="https://github.com/huang195/spire-lifecycle-hook/blob/main/postStart.sh">here</a>. This needs to be placed at <em>/usr/local/bin/postStart.sh</em> on every worker node in the cluster. To specify this code should be invoked for <em>PostStart</em> events, we need to specify it in the configuration file of the container runtime. For <em>containerd</em>, its configuration file is located at <em>/etc/containerd/cri-base.json</em>, in which, we need to add the following lines:</p><pre>“hooks”: { <br> “postStart”: [ <br>  { <br>   “path”: “/usr/local/bin/postStart.sh” <br>  } <br> ]<br>}</pre><p>For this change to take into effect, we need to restart the container runtime, i.e.,</p><pre># systemctl restart containerd</pre><h3>A developer’s perspective</h3><p>All of the steps described above are handled entirely by the infrastructure team and are performed only once. Developers do not need to be aware of the underlying mechanics. From their perspective, when their workload pods start, the necessary identity files will simply appear in the <em>/var/run/secrets/spire.io</em> directory. This experience mirrors how Kubernetes automatically mounts service account tokens into the <em>/var/run/secrets/kubernetes.io</em> directory at pod startup — providing a familiar, seamless integration for developers.</p><p>As mentioned earlier, using the <em>PostStart</em> hook comes with certain limitations. Specifically, code executed at this hook point starts up <strong>concurrently</strong> with the workload’s processes, and there is no strict ordering guarantee about which will start first. This introduces a race condition: we cannot ensure that SPIRE-issued identity files will be available at the expected file system location before the workload begins trying to access them.</p><p>To address this, we require developers to make a small modification to their workload’s deployment YAML. This ensures proper coordination between the identity provisioning logic and the application startup. Below is an example deployment manifest:</p><pre>apiVersion: v1 <br>kind: Pod <br>metadata: <br> name: openssl-server <br> labels: <br>  app.kubernetes.io/name: openssl-server <br>spec: <br> restartPolicy: Never <br> containers: <br> — name: openssl-server <br>   image: finalgene/openssh <br>   command: [“sh”] <br>   args: [“-c”, “openssl s_server -accept 3000 -CAfile /var/run/secrets/spire/bundle.0.pem -cert /var/run/secrets/spire/svid.0.pem -key /var/run/secrets/spire/svid.0.key -state -www &amp;&amp; break”]</pre><p>This pod launches an SSL server using the openssl command. With the hook code correctly installed, deploying the pod may appear to work — but in practice, the pod will only start successfully some of the time due to the aforementioned race condition. The issue arises because the workload may begin execution before the identity files are fully provisioned.</p><p>To ensure a reliable startup every time, the args field in the pod specification needs to be adjusted slightly, as shown below:</p><pre>args: [“-c”, “while true; do [ -f /var/run/secrets/spire.io/svid.0.pem ] &amp;&amp; openssl s_server -accept 3000 -CAfile /var/run/secrets/spire/bundle.0.pem -cert /var/run/secrets/spire/svid.0.pem -key /var/run/secrets/spire/svid.0.key -state -www &amp;&amp; break; sleep 1; done”]</pre><p>We added a while true; … done loop around the main command line to test if the identity file /var/run/secrets/spire.io/svid.0.pem has been created before proceeding.</p><h3>Challenges and Next Steps</h3><p>To summarize, here are the benefits of the solution based on Container Lifecycle Hooks:</p><p>· <strong>Less performance overhead</strong>: Only invoked once per container lifecycle instead of every <em>open()</em> and <em>openat()</em> syscall. This removes all the runtime overhead that we have seen with our previous Seccomp-based solution.</p><p>· <strong>More resource efficient</strong>: No longer requires a per-node agent</p><p>From a user experience perspective, this approach is on par with the Seccomp-based solution — both require only a minimal change to the workload’s deployment YAML. However, the main drawback is that the installed hook code applies globally to <em>all</em> containers on a given worker node, including those that don’t require identity files. To avoid unnecessary overhead, we need to add logic within the hook code to selectively provision identity files only for relevant pods. This can be achieved, for example, by checking for specific pod annotations.</p><p>Additionally, installing or upgrading the hook code can be disruptive. For changes to take effect, worker nodes typically need to be properly drained, and the container runtime must be restarted. If this process is not handled carefully, it can destabilize affected nodes and potentially render them unusable — introducing operational risk during hook lifecycle management.</p><p>In the next article, we describe our third, and final, solution that solves all the shortcomings of the first two solutions we explored so far. It is based on Container Storage interface (CSI). Stay tuned!</p><h3>About Us</h3><p>Our team at IBM Research is dedicated to advancing technologies that enhance modern applications by integrating Zero Trust principles. You can find our previous posts covering SPIFFE/SPIRE and workload identity here: (<a href="https://medium.com/universal-workload-identity">our blog</a>).</p><h3>Citations</h3><ol><li><a href="https://github.com/opencontainers/runtime-spec/blob/e935f995dd67c853bd93d6937bab898d36eb1747/config.md">https://github.com/opencontainers/runtime-spec/blob/e935f995dd67c853bd93d6937bab898d36eb1747/config.md</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0375881bd88a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a">Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 4. Container Lifecycle Hooks</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 3. Seccomp Unotify]]></title>
            <link>https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cdf3da86033</guid>
            <category><![CDATA[workload-identity]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[spire]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 13:21:36 GMT</pubDate>
            <atom:updated>2025-05-19T13:07:00.512Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkqX5fXeSAiBnBLSERVTjA.png" /></figure><h3>Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 3. Seccomp Unotify</h3><p>By Hai Huang, Maia Iyer, Mariusz Sabath, James Cadden (IBM Research)</p><h3>Blog Series</h3><p>Part 1: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e">Introduction</a><br>Part 2: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336">SPIFFE-helper</a><br><strong>Part 3: Seccomp Unotify (this article)<br></strong>Part 4: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a">Container Lifecycle Hooks</a><br>Part 5: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea">Container Storage Interface</a></p><p>In our previous articles, we described why interacting with the SPIRE agent through a Unix Domain Socket puts unnecessary burdens on application developers. Even with the use of SPIFFE-helper, there is still a lot of manual work (e.g., to rebuild container images, customize per-workload SPIFFE-helper configuration files, and modify various deployment artifacts) and it is especially true for larger and more complex workloads. Ideally, application developers should focus on application logic and yet can still benefit from SPIRE without having to know a whole lot about it. In this article, we describe our first attempt<em> </em>to make SPIRE easier to use by building an agent that uses Seccomp Unotify [<a href="https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html">1</a>]. Its advantages over SPIFFE-helper along with its weaknesses will also be fully explored.</p><h3>Seccomp Agent: A High-Level Overview</h3><p>Our key insight to this solution is that when a typical workload accesses an identity file — whether a JWT token or an X.509 certificate — it must first read the file into memory. Since user-level processes can only read files through system calls (syscalls), intercepting these events when a process attempts to open an identity file presents an opportunity. By capturing these events automatically, we could request identity files on behalf of the intercepted processes, like how SPIFFE-helper works. Moreover, by implementing this mechanism at a lower level in the stack, applications could seamlessly leverage SPIRE-provided services without requiring explicit configuration or modifications.</p><p>At a high level, our solution provides SPIRE identities to applications through the following process:</p><ol><li>The target application issues an <em>open()</em> syscall to access an identity file.</li><li>We intercept this syscall and temporarily pause the target application.</li><li>We connect to the local SPIRE agent and request identity files on behalf of the target application.</li><li>The acquired identity files are placed in the expected filesystem location for the target application to access.</li><li>We resume the target application.</li><li>The target application proceeds to open and use the identity file as intended.</li></ol><p>In the final step, the target application remains completely unaware of the work done on its behalf. The user experience should be seamless, much like how Kubernetes automatically provides service account tokens (JWT token files) to pods at a fixed location in the container’s filesystem. This allows processes within the container to authenticate to the API server and access various resources without requiring explicit configuration or intervention. We believe it is possible to provide this type of user experience to workloads using SPIRE-issued identities.</p><h3>Seccomp Unotify</h3><p>There are several technologies available for intercepting system calls, including Ptrace, eBPF, and Seccomp, each with its own advantages and trade-offs. After evaluating these options, we chose Seccomp as our primary approach, given our focus on containerized environments such as Kubernetes and OpenShift. Many mainstream container runtimes, including Containerd, already offer mature support for Seccomp, making it a natural fit. Additionally, Seccomp provides a built-in mechanism to keep the target process in a suspended state while we retrieve its identity from the backend SPIRE agent, ensuring minimal disruption to application execution.</p><p>Typically, Seccomp filters in containerized environments are used to enforce security policies by making binary decisions — either allowing or denying specific syscalls invoked by a process. However, our use case requires more than simple syscall filtering; we need to intercept syscalls and execute additional logic to provide the application with SPIRE-issued identity files dynamically.</p><p>Fortunately, Linux kernel 5.9 introduced the Seccomp Unotify feature, which allows for greater flexibility in syscall interception. Instead of outright blocking or allowing a syscall, Seccomp Unotify enables the kernel to send a notification to a user-space handler whenever a syscall is intercepted. This user-space component can then perform additional operations — such as fetching identity files from SPIRE — before resuming the syscall execution. This capability perfectly aligns with our needs, allowing us to transparently provide applications with dynamic identities while maintaining security and performance.</p><h3>Impersonation</h3><p>When building a solution to interact with local SPIRE agents on behalf of workloads, a critical decision is how to impersonate the workload to acquire its identity files. There are multiple approaches to achieve this.</p><p>One common method is using SPIFFE-helper, which forks workloads as child processes. In most cases, this allows direct communication with the local SPIRE agent without requiring explicit impersonation. Another option is SPIRE’s Delegated Identity API, a feature designed specifically for agents and proxies to assume the identity of other processes. However, using this API requires configuring special privileges for the agents and proxies. Unfortunately, at the time of our implementation, the Delegated Identity API was still under active discussion and hadn’t been merged. As a result, we adopted an alternative approach: cgroup hopping.</p><p>In a containerized environment, all processes within a pod run inside cgroups, which are created by the container runtime at startup. When a process connects to the SPIRE agent, the agent performs workload attestation by inspecting the process’s PID. From there, it determines the corresponding cgroup and retrieves pod-related metadata — such as labels, namespace, and service account — to assign identities.</p><p>Cgroup hopping leverages this mechanism by temporarily placing our agent process into the same cgroup as the target workload. Once inside, we connect to the SPIRE agent, acquire the necessary identity files, and then revert to our original cgroup. This technique allowed us to securely impersonate workloads at the time, but as the Delegated Identity API becomes more mature, it may offer a more standardized and robust solution in the future.</p><h3>Implementation</h3><p>The overall architecture of our Seccomp agent is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*6aj4hers5yImQMkSE_K-nQ.png" /></figure><ol><li><strong>Intercepting Syscalls</strong><br>When a process inside the workload container attempts to open an identity file (e.g., an X.509 certificate), it makes an open syscall. The Seccomp filter triggers at this point, pausing the process and handing control to a user-space Seccomp agent.</li><li><strong>Validating the Request and Impersonate as the Workload</strong><br>The Seccomp agent inspects the syscall parameters, verifying that the requested file is an identity file and that SPIRE should handle its issuance. To properly request the file on behalf of the workload, the agent temporarily enters the same cgroup as the workload container.</li><li><strong>Requesting the Identity File</strong><br>Now acting as the workload, the Seccomp agent communicates with the local SPIRE agent via a Unix Domain Socket, requesting the necessary identity file.</li><li><strong>Restoring the Original Cgroup</strong><br>Once the identity file is acquired, the Seccomp agent exits the workload’s cgroup and returns to its original cgroup, ensuring minimal interference with the workload’s execution environment.</li><li><strong>Delivering the Identity File Transparently</strong><br>To complete the process, the Seccomp agent enters the mount namespace of the workload container and copies the identity file to the appropriate location. Finally, control is handed back to the kernel, which resumes the previously paused workload process. From the workload’s perspective, the open syscall now succeeds as expected, with the requested identity file seamlessly in place — making the entire process completely transparent.</li></ol><p>This approach ensures secure, automated identity management for containerized workloads while leveraging Seccomp, cgroup hopping, and SPIRE for seamless authentication.</p><h3>Create and Install Seccomp Filter</h3><p>By default, container runtime’s Seccomp filters do not use the Seccomp Unotify mechanism, meaning workloads cannot directly leverage it for syscall interception. To enable this capability, a system administrator must create a custom Seccomp filter and deploy it to worker nodes at a container runtime-specific location.</p><p>For Containerd, the filter must be placed at:<br> 📂 /var/lib/kubelet/seccomp</p><p><strong>Example Seccomp Filter</strong></p><p>The following JSON filter file configures Seccomp to intercept <em>open()</em> and <em>openat()</em> syscalls and forward them to a Seccomp agent listening on a Unix Domain Socket at <em>/run/seccomp-agent.socket</em>:</p><pre>{<br>  “architectures” : [<br>    “SCMP_ARCH_X86”,<br>    “SCMP_ARCH_X32”<br>  ],<br>  “defaultAction” : “SCMP_ACT_ALLOW”,<br>  “listenerPath”: “/run/seccomp-agent.socket”,<br>  “listenerMetadata”: “LZT=1\nSPIRE=1”,<br>  “syscalls” : [<br>    {<br>      “action” : “SCMP_ACT_NOTIFY”,<br>      “names” : [<br>        “openat”,<br>        “open”<br>      ]<br>    }<br>  ]<br>}</pre><h3>A developer’s perspective</h3><p>Everything discussed so far, i.e., deploying the Seccomp agent and installing Seccomp filter, is done by the infrastructure team. The only thing the dev team needs to know is the name of the new filter. If the above example filter was placed at <em>/var/lib/kubelet/seccomp/profiles/notify.json</em>, then the name of the new filter is <em>profiles/notify.json</em>. This information should be provided to the dev team by the infrastructure team.</p><p>An example pod deployment file might look like the following:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br> name: openssl-server<br> labels:<br>  app.kubernetes.io/name: openssl-server<br>spec:<br> securityContext:<br>  seccompProfile:<br>   type: Localhost<br>   localhostProfile: profiles/notify.json<br>  restartPolicy: Never<br>  containers:<br>  - name: openssl-server<br>    image: finalgene/openssh<br>    command: [“sh”]<br>    args: [“-c”, “openssl s_server -accept 3000 -CAfile /tmp/bundle.0.pem -cert /tmp/svid.0.pem -key /tmp/svid.0.key -state -www”]</pre><p>From the perspective of an app developer, the <strong>ONLY</strong> thing that he/she needs to change is to specify the new Seccomp filter in the <em>seccompProfile</em> field when creating workloads. Everything else is taken cared by the underlying infrastructure.</p><p>This pod deploys a TLS server that listens for TLS connections from clients. The TLS handshake is done using the key and certificate files in the<em> /tmp</em> directory, which the container image <em>finalgene/openssh</em> does not provide. Instead, it relies on the Seccomp agent to intercept the <em>open</em> and <em>openat</em> calls whenever the openssl command tries to open the key and certificate files, and then it populates SPIRE issued files at the location that the workload process expects.</p><h3>Challenges and Next Steps</h3><p>To summarize, here are the benefits of using the Seccomp agent from the perspective of application developers:</p><ul><li><strong>More developer friendly</strong>: Instead of having to deal with external programs like SPIFFE-helper or an interface such as a Unix Domain Socket and/or the need to either rebuild container images or run with sidecar containers, workloads can consume identity files directly without any modification to the application code.</li><li><strong>More secure</strong>: A nice side effect of using the Seccomp agent is that it does workload attestation on the behalf of the workload, so there is no need to expose the Unix Domain Socket that SPIRE agent listens to directly to untrusted domains, thus, reducing the attack surface. Due to unforeseeable kernel or library bugs, the socket interface could potentially be exploited to gain access to the SPIRE agent and/or SPIRE server, causing severe leak of sensitive information.</li></ul><p>So, what are the drawbacks of this approach?</p><ul><li><strong>Consume more resources</strong>: The Seccomp agent runs as a daemonset on every worker node. It will consume more CPU and memory when intercepting syscalls and interacting with the local SPIRE agent. However, compared to having a SPIFFE-helper running alongside of each workload pod, a per-node agent potentially is more resource efficient.</li><li><strong>Performance overhead</strong>: Due to Seccomp Unotify filter indiscriminately capturing all specified syscalls, e.g., open() and openat(), even those calls that are unrelated to the workload opening an identity file are intercepted, e.g., opening a network socket or a video/audio file. This system-wide capturing of these syscalls could cause slow down as each captured event will trigger additional kernel-to-user and context switching overheads. Our rudimentary experiments show that:<br>- <strong>Microbenchmark</strong>: Running 100k open() syscalls in a tight loop, we saw 6 times slow down. This is the worst-case scenario.<br>- <strong>Macrobenchmark</strong>: We enabled the Seccomp agent while compiling the latest version of the Linux kernel, and it resulted in about 10% slower build time. This is more reasonable, but still non-negligible.</li></ul><p>In this article, we described our experience building an agent based on Seccomp Unotify to unburden application developers from all the manual work they would need to do to leverage SPIRE-issued identities in their applications. Despite the benefits of this solution, there are some drawbacks as well, i.e., higher resource usage and some performance impact. In the next article, we will describe another solution based on Container Lifecycle Hooks, with its own pros and cons. So, stay tuned!</p><h3>About Us</h3><p>Our team at IBM Research is dedicated to advancing technologies that enhance modern applications by integrating Zero Trust principles. You can find our previous posts covering SPIFFE/SPIRE and workload identity here: (<a href="https://medium.com/universal-workload-identity">our blog</a>).</p><h3>Citations</h3><p>1. <a href="https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html">https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html</a><br>2. <a href="https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api">https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cdf3da86033" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033">Developer-Friendly Zero Trust using SPIFFE/SPIRE — part 3. Seccomp Unotify</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 2. SPIFFE-helper]]></title>
            <link>https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b5495159336</guid>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[workload-identity]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[spire]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 12:59:16 GMT</pubDate>
            <atom:updated>2025-05-19T13:07:48.572Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkqX5fXeSAiBnBLSERVTjA.png" /></figure><h3>Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 2. SPIFFE-helper</h3><p>By Hai Huang, Maia Iyer, Mariusz Sabath, James Cadden (IBM Research)</p><h3>Blog Series</h3><p>Part 1: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e">Introduction</a><br><strong>Part 2: SPIFFE-helper (this article)<br></strong>Part 3: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033">Seccomp Unotify</a><br>Part 4: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a">Container Lifecycle Hooks</a><br>Part 5: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea">Container Storage Interface</a></p><p>In this second installment of our blog series, we share our experience deploying an application that mirrors real-world behaviors and complexities. Initially, the application relied solely on static certificates and tokens. Therefore, by incorporating SPIFFE-helper [<a href="https://github.com/spiffe/spiffe-helper">1</a>] and making targeted modifications to the deployment artifacts, we enabled the use of dynamically issued certificates and tokens provided by SPIRE. In this post, we’ll dive into the required changes and discuss why SPIFFE-helper is well-suited for simpler applications but can become cumbersome in more complex deployment scenarios.</p><h3>Medical Diagnosis Application</h3><p>Red Hat Validated Patterns [<a href="https://validatedpatterns.io/">2</a>] proved to be an ideal solution for our needs. These application deployment patterns, derived from real customer engagements, can be easily replicated and deployed on a local OpenShift cluster using GitOps methodology. At the time of writing, more than 20 such patterns were available, and we chose the Medical Diagnosis [<a href="https://validatedpatterns.io/patterns/medical-diagnosis/">3</a>] pattern for this work.</p><p>This application pattern simulates a distributed medical care environment where patients visit clinics for chest X-ray imaging. These images, along with patient information, are securely transmitted to a central medical hub, where an AI model analyzes them to detect signs of pneumonia. The results are then stored in a backend database, and a Grafana dashboard provides a visual representation of the entire data pipeline, as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*c6ndBtwLFKYHkLomSviGKg.png" /></figure><p>The Medical Diagnosis application consists of multiple smaller components that work together and communicate with one another. Given that sensitive personal information is exchanged between these components, ensuring secure communication is essential. Initially, static keys and certificates were generated at provisioning time to facilitate secure communication. However, since these credentials are never rotated, this approach contradicts the principles of Zero Trust.</p><p>Our first task was to deploy SPIRE, enabling various application components to be attested and assigned appropriate identity tokens (JWT) or certificates (X.509). These credentials are short-lived and automatically rotated, enhancing security and aligning with Zero Trust principles.</p><h3>Steps to integrate SPIRE</h3><p>In the following sections, we describe what are all the things an application developer would need to do so his application is able to leverage dynamically issued identities from SPIRE.</p><p><strong>1. Install SPIRE and SPIFFE CSI driver (<em>done once by sysadmins</em>)</strong></p><p>We installed SPIRE on an OpenShift cluster using Helm, following these instructions [<a href="https://spiffe.io/docs/latest/spire-helm-charts-hardened-about/installation/">4</a>]. Once the SPIRE server and its agents were deployed, workloads could leverage the SPIFFE CSI driver to communicate with the local SPIRE agent. The SPIFFE CSI driver facilitates this by bind-mounting the Unix Domain Socket (UDS) that the SPIRE agent listens on, making it accessible to the workload. Workloads are then responsible for acquiring their own identities via the Workload API through the UDS.</p><p><strong>2. Use of SPIFFE-helper</strong></p><p>As understanding SPIFFE Workload APIs and making use of them via a UDS is likely not easy for most app developers, SPIFFE-helper was introduced to shield developers from having to deal with these complexities. The helper program is essentially a wrapper around an application or application component, i.e., a parent process, that can interact with the SPIRE agent’s Workload APIs via a UDS on behalf of the workload. However, the app developer does need to specify a few things in a configuration file for the helper program to shepherd its child process appropriately. An example configuration file is shown below:</p><pre>agent_address = “/tmp/spire-agent/public/api.sock”<br>cmd = “ghostunnel”<br>cmd_args = “server — listen localhost:8002 — target localhost:8001 — keystore certs/svid_key.pem — cacert certs/svid_bundle.pem — allow-uri-san spiffe://example.org/Database”<br>cert_dir = “certs”<br>renew_signal = “SIGUSR1”<br>svid_file_name = “svid.pem”<br>svid_key_file_name = “svid_key.pem”<br>svid_bundle_file_name = “svid_bundle.pem”<br>cert_file_mode = 0444<br>key_file_mode = 0444</pre><p>The configuration file tells the SPIFFE-helper where to find the UDS (<em>/tmp/spire-agent/pubic/api.sock</em>), what is the application it is attesting on behalf of (i.e., child process — <em>ghostunnel</em>), the directory to place identity files (<em>certs</em>), name of the identity files, etc. As identity files are rotated periodically, the helper program can also optionally send a Unix signal (i.e., <em>SIGUSR1</em>) to the child process whenever rotation occurs, so that the application could reload the new identity files.</p><p>There are at least several parameters that a developer will need to customize for each application, e.g., <em>cmd</em>, <em>cmd_args</em>, <em>cert_dir</em>, etc. More commonly, other items such as the frequency of identity refresh, the type of identity file to obtain, etc. also need to be specified per application.</p><p><strong>3. Rebuild Application Images</strong></p><p>In a containerized environment, where workloads are deployed as container images, leveraging SPIFFE-helper requires either rebuilding the images to include the SPIFFE-helper binary and its configuration file or adding an init container and a sidecar container with a shared volume for the main application container. Since the latter approach demands additional resources and introduces more changes to deployment artifacts, we opted for the former.</p><p><strong>4. Modify Application Deployment Artifacts</strong></p><p>Finally, each deployment YAML file must be modified to ensure the entrypoint now directs to SPIFFE-helper instead of the actual application binary. Furthermore, the deployment YAML file also needs to be updated to include the use of SPIFFE CSI driver, allowing the UDS to be accessible by the SPIFFE-helper within the container at runtime. Below is an example deployment YAML file:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br> name: image-generator<br> namespace: xraylab-1<br>spec:<br> replicas: 1<br> selector:<br>  matchLabels:<br>   name: image-generator<br> template:<br>  metadata:<br>   labels:<br>    app: image-generator<br>    name: image-generator<br>  spec:<br>   serviceAccountName: demo<br>   containers:<br>   - env:<br>    - name: SECONDS_WAIT<br>      value: “1”<br>    image: medical-diagnosis/generator:rebuilt<br>    command: [“/usr/src/app/spiffe-helper”]<br>    args: [“-config”, “/usr/src/app/helper.conf”]<br>    imagePullPolicy: Always<br>    name: image-generator<br>    resources:<br>      limits:<br>       cpu: “2”<br>       memory: 1G<br>      requests:<br>       cpu: 400m<br>       memory: 100M<br>    volumeMounts:<br>    - name: spiffe-workload-api<br>      mountPath: /spiffe-workload-api<br>      readOnly: true<br>   volumes:<br>   - name: spiffe-workload-api<br>     csi:<br>      driver: “csi.spiffe.io”<br>      readOnly: true</pre><p>There are several items in this deployment YAML file that a developer would need to add or modify, e.g., <em>image</em>, <em>command</em>, <em>args</em>, and <em>volumes</em>.</p><h3>Challenges and Next Steps</h3><p>If only a few container images needed rebuilding, a few SPIFFE-helper configuration files required customization, and a few deployment files had to be modified, performing these steps manually would be manageable. However, given the many components in our application, repeating this process manually for each one quickly became tedious. This challenge led us to explore alternative — and potentially better — solutions to SPIFFE-helper that would simplify the process for application developers.</p><p>We devised multiple solutions, each built on different technical foundations, with its own set of advantages and trade-offs. In the next blog article, we will explore one such solution based on Seccomp Unotify [<a href="https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html">5</a>].</p><h3>About Us</h3><p>Our team at IBM Research is dedicated to advancing technologies that enhance modern applications by integrating Zero Trust principles. You can find our previous posts covering SPIFFE/SPIRE and workload identity here: (<a href="https://medium.com/universal-workload-identity">our blog</a>).</p><h3>Citations</h3><p>1. <a href="https://github.com/spiffe/spiffe-helper">https://github.com/spiffe/spiffe-helper</a><br>2. <a href="https://validatedpatterns.io/">https://validatedpatterns.io/</a><br>3. <a href="https://validatedpatterns.io/patterns/medical-diagnosis/">https://validatedpatterns.io/patterns/medical-diagnosis/</a><br>4. <a href="https://spiffe.io/docs/latest/spire-helm-charts-hardened-about/installation/">https://spiffe.io/docs/latest/spire-helm-charts-hardened-about/installation/</a><br>5. <a href="https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html">https://man7.org/linux/man-pages/man2/seccomp_unotify.2.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b5495159336" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336">Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 2. SPIFFE-helper</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 1. Introduction]]></title>
            <link>https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e?source=rss-4d3a69661639------2</link>
            <guid isPermaLink="false">https://medium.com/p/a184cbdaf67e</guid>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[spire]]></category>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[hai huang]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 13:38:33 GMT</pubDate>
            <atom:updated>2025-05-19T13:08:12.111Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkqX5fXeSAiBnBLSERVTjA.png" /></figure><h3>Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 1. Introduction</h3><p>By Hai Huang, Maia Iyer, Mariusz Sabath, James Cadden (IBM Research)</p><h3>Blog Series</h3><p><strong>Part 1: Introduction (this article)<br></strong>Part 2: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-2-spiffe-helper-0b5495159336">SPIFFE-helper</a><br>Part 3: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-3-seccomp-unotify-2cdf3da86033">Seccomp Unotify</a><br>Part 4: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-4-container-lifecycle-hooks-0375881bd88a">Container Lifecycle Hooks</a><br>Part 5: <a href="https://medium.com/@huang195/developer-friendly-zero-trust-using-spiffe-spire-part-5-container-storage-interface-csi-6119770cdfea">Container Storage Interface</a></p><h3>Introduction</h3><p>The traditional perimeter-based security model, that relies on establishing a strong boundary and trusting everything inside it, is ill-suited for modern, complex environments. In contrast, the Zero Trust security model [<a href="https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=930420">1</a>], which assumes that no entity should be trusted by default but requires that every request to be authenticated and authorized regardless of its origin, is much more suitable for today’s distributed and dynamic cloud applications.</p><p>At the heart of Zero Trust lies the effective management of workload identities — such as certificates or tokens — which are essential for enabling secure, authenticated, and authorized access to resources and services. Zero Trust’s stringent requirements demand that every user, device, process, and service possess an identity that is highly secure, frequently rotated, and consistently enforced. While Zero Trust’s fine-grained control over authentication and authorization — determining who or what can connect, when, and from where — is highly beneficial, its implementation introduces a major burden for application developers.</p><p>In this blog series, we will share our journey with SPIRE for managing workload identities, covering everything from deployment and usage to the challenges we faced from a developer’s perspective. We’ll explore three techniques that, when applied, can greatly simplify SPIRE adoption for developers. Across the series, we will evaluate each of these techniques and discuss their strengths and limitations.</p><h3>Developer-Friendly Identity Management</h3><p>Application developers’ focus is on building and optimizing core application logic. Consequently, solutions designed to manage identities for applications should aim to minimize the burden on developers while ensuring seamless integration into existing workflows. To achieve this, the ideal identity management solution should embody the following characteristics:</p><ul><li><strong>Minimal Disruption to Applications</strong>: The solution should require little-to-no changes to the application code, its deployment and configuration files, or any associated build artifacts, such as container images.</li><li><strong>Transparent Runtime Identity Provisioning</strong>: Workload identities should be provided automatically and transparently at runtime. This approach is akin to how Kubernetes delivers service account tokens to pods by placing them at a predetermined location before the main application process begins.</li></ul><p>By adhering to these principles, we can empower developers to focus more on building innovative solutions while seamlessly embedding secure, Zero Trust-aligned identity management into their applications.</p><h3>Why SPIFFE/SPIRE?</h3><p>In a landscape filled with identity management solutions, SPIFFE/SPIRE [<a href="https://spiffe.io/">2</a>] has emerged as a de facto standard for managing workload identities — whether in private data centers or across multiple public clouds.</p><ul><li><strong>SPIFFE</strong> (<strong>Secure Production Identity Framework for Everyone</strong>) defines a framework and a set of standards for securely identifying and authenticating communication between services.</li><li><strong>SPIRE</strong> (<strong>SPIFFE Runtime Environment</strong>) implements SPIFFE by providing a robust toolchain of APIs to establish trust between software systems running in diverse hosting environments.</li></ul><p>If your application already operates across multiple clouds, each with its own identity management system, or you’re running on a single cloud but want to avoid vendor lock-in, SPIRE offers a standardized, cloud-agnostic approach to workload identity management.</p><h3>SPIRE Workload Identity Approaches</h3><p>There are already numerous blog posts [<a href="https://blog.gitguardian.com/getting-started-with-spiffe/">3</a>,<a href="https://aembit.io/blog/spiffe-and-spire-your-questions-answered/">4</a>] covering SPIRE architecture and quickstart guides for integrating SPIRE into a simple application. Rather than repeating those, this article assumes readers have a basic understanding of SPIRE and are seriously considering adopting it — either for a new application or to migrate an existing application that currently uses another method for managing workload identities.</p><p>SPIRE provides three primary ways for workloads to attest and obtain their identities at runtime:</p><ol><li><strong>Native SPIFFE Support</strong>: Requires modifying application code to interact directly with the Workload API over a Unix Domain Socket or via a language-specific library such as Go-SPIFFE or Java-SPIFFE.</li><li><strong>SPIFFE-Aware Proxies</strong>: Proxies like Envoy and Ghostunnel can sit in front of applications, interacting with the SPIFFE Workload API on their behalf.</li><li><strong>Helper Programs</strong>: Tools such as SPIFFE-helper act as a wrapper around applications, retrieving workload identities via the SPIFFE Workload API and writing identity files to disk, where applications can use them.</li></ol><h3>Comparing the Three Approaches</h3><p>Each approach has its trade-offs in terms of control, complexity, and resource requirements:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8XBVGEFUxL3vn2WfD6yaUw.png" /></figure><p>Choosing the Right Approach</p><ul><li>For security-sensitive applications with skilled engineers, a custom SPIRE SDK (like Uber’s approach) provides maximum control but requires significant development effort.</li><li>For organizations hesitant to modify application code, SPIFFE-aware proxies enable easy adoption but come with higher resource consumption and operational overhead.</li><li>For most applications, helper programs like SPIFFE-helper offer the best balance — minimal changes, no extra resource requirements, and sufficient security for managing workload identities.</li></ul><p>As the first two approaches have significant higher entry barriers (due to either engineering skills or compute resources, or both), for the remainder of this blog series, we will focus on the third approach, sharing our experience with using SPIFFE-helper — what we liked, what we didn’t, and how the developer experience can be improved.</p><h3>Next Steps</h3><p>In the next article of this series, we will dive deeper into our experience with SPIFFE-helper, exploring its benefits and challenges in real-world scenarios. Specifically, we’ll walk through how we integrated SPIFFE-helper into an existing microservices application to replace manually managed certificates with auto-rotated SPIFFE-based identities. We’ll also discuss key lessons learned and practical optimizations using techniques such as <strong>Seccomp Unotify</strong>, <strong>Container Lifecycle Hooks</strong>, and <strong>Container Storage Interface </strong>to enhance the developer experience when using SPIRE for workload identity management. Stay tuned!</p><h3>About Us</h3><p>Our team at IBM Research is dedicated to advancing technologies that enhance modern applications by integrating Zero Trust principles. You can find our previous posts covering SPIFFE/SPIRE and workload identity here: (<a href="https://medium.com/universal-workload-identity">our blog</a>).</p><h3>Citations</h3><p>1. <a href="https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=930420">https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=930420</a><br>2. <a href="https://spiffe.io/">https://spiffe.io/</a><br>3. <a href="https://blog.gitguardian.com/getting-started-with-spiffe/">https://blog.gitguardian.com/getting-started-with-spiffe/</a><br>4. <a href="https://aembit.io/blog/spiffe-and-spire-your-questions-answered/">https://aembit.io/blog/spiffe-and-spire-your-questions-answered/</a><br>5. <a href="https://www.uber.com/blog/our-journey-adopting-spiffe-spire/">https://www.uber.com/blog/our-journey-adopting-spiffe-spire/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a184cbdaf67e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/developer-friendly-zero-trust-using-spiffe-spire-part-1-introduction-a184cbdaf67e">Developer-Friendly Zero Trust using SPIFFE/SPIRE — Part 1. Introduction</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>