<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mariusz Sabath on Medium]]></title>
        <description><![CDATA[Stories by Mariusz Sabath on Medium]]></description>
        <link>https://medium.com/@mrsabath?source=rss-a352fae2c28b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*HrHi-f3dXRuSbMsoEle5BQ.jpeg</url>
            <title>Stories by Mariusz Sabath on Medium</title>
            <link>https://medium.com/@mrsabath?source=rss-a352fae2c28b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:57:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mrsabath/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Identity in Agentic Platforms: Enabling Secure, Least-Privilege Access]]></title>
            <link>https://medium.com/rossoctl-the-agentic-platform/identity-in-agentic-platforms-enabling-secure-least-privilege-access-996527f1c983?source=rss-a352fae2c28b------2</link>
            <guid isPermaLink="false">https://medium.com/p/996527f1c983</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[access-management]]></category>
            <category><![CDATA[access-control]]></category>
            <dc:creator><![CDATA[Mariusz Sabath]]></dc:creator>
            <pubDate>Mon, 16 Jun 2025 22:21:58 GMT</pubDate>
            <atom:updated>2025-06-17T17:11:42.164Z</atom:updated>
            <content:encoded><![CDATA[<p>By <a href="https://www.linkedin.com/in/mariusz-sabath-b36b0b20/">Mariusz Sabath</a>, <a href="https://www.linkedin.com/in/pdettori/">Paolo Dettori</a>, <a href="https://medium.com/@maia.raj.iyer">Maia Iyer</a>, <a href="https://medium.com/@alan.cha">Alan Cha</a>, <a href="https://medium.com/@huang195">Hai Huang</a>, <a href="https://medium.com/@cwiklikj">Jarek Cwiklik</a></p><figure><img alt="Image generated with Microsoft Image Creator" src="https://cdn-images-1.medium.com/max/1024/1*FbqKjRDnSpaA6ISrqYxs8Q.png" /><figcaption>Image generated with Microsoft Image Creator</figcaption></figure><p>In modern software systems, controlling who can do what is one of the most critical decisions we make. This is especially true in emerging agentic platforms, where autonomous agents, tools, and orchestrators continuously interact to fulfill user goals. Without careful access control, these components can become attack vectors, compliance risk, and source of unintended behavior.</p><p>This raises a key question:</p><p><strong>How do we manage secure access in a world where components are dynamic, distributed, and intelligent?</strong></p><p>The answer starts with understanding two key concepts:</p><ul><li>Identity — knowing <em>who </em>is acting, and</li><li>Zero Trust — never assuming that identity alone is enough.</li></ul><h3>What is Zero Trust?</h3><p>Zero Trust is a security model that assumes no implicit trust, not in users, not in devices, and not even in internal workloads. Instead, it enforces explicit, per-request access decisions based on strong identity verification for every actor (human and machine), least-privilege access, and continuous validation.</p><p>In dynamic agentic platforms, this means no static, long-lived secrets, no blanket permissions, and every actor must prove who it is and why it should be allowed, every time.</p><p>This post explores our approach using industry-proven standards like <a href="https://oauth.net/2/">OAuth2.0</a>, <a href="https://oauth.net/2/token-exchange/">Token Exchange</a>, and <a href="https://spiffe.io/docs/latest/spiffe-about/overview/">SPIFFE</a> to solve identity propagation across complex agentic workflows.</p><h3>What Is an Agentic Platform?</h3><p>Before diving into identity mechanics, let’s clarify the foundational concepts behind an agentic platform:</p><ul><li><strong>Agentic Applications</strong>: These are applications built from autonomous, decision-making components that can plan, reason, and act in pursuit of user-defined goals.</li><li><strong>Agents:</strong> These are the individual software components that perform reasoning. Agents often rely on large language models (LLMs) to break down complex, multi-step problems, plan task, and call tools. They are the intelligent actors in agentic system.</li><li><strong>Tools:</strong> These are APIs, services, or systems, both internal and external, that agents invoke to accomplish specific steps, such as querying database, retrieving data from a third party systems, etc.</li><li><strong>Agent Orchestrator:</strong> The central coordinator responsible for launching agents, managing their execution, and routing requests and results between components. It handles lifecycle, identity and policy enforcement.</li></ul><p>In short, Agents are the decision-makers, while <em>agentic</em> refers to the full system and its ability to reason, act, and coordinate autonomously.</p><p>Examples of agentic applications include chatbots that solve real-time customer issues, automated assistants handling IT support tickets, or multi-agents system collaborating on complex workloads. These systems are inherently dynamic: users initiate requests, agents call tools (or other agents), and results propagate across multiple components, forming paths that require strong identity and access control.</p><figure><img alt="Picture 1. Dynamic flow in Agentic environment" src="https://cdn-images-1.medium.com/max/1024/1*I7zI0rdOKZ17n0hWlqEBMQ.png" /><figcaption><em>Picture 1. Dynamic flow in Agentic environment</em></figcaption></figure><p>To secure these paths, we adopt Zero Trust principles: every request must be authenticated, authorized, and evaluated using the least privilege model.</p><p>For a more detailed overview of the Agentic Platform, check our previous blog <a href="https://medium.com/kagenti-the-agentic-platform/toward-a-cloud-native-platform-for-ai-agents-70081f15316d">Toward a Cloud-Native Platform for AI Agents</a>.</p><h3>The Problem: Access Propagation in a Dynamic World</h3><p>Let’s walk through a basic use case:</p><ol><li>The Agent Orchestrator deploys an agent.</li><li>A user initiates a request to the Agent Orchestrator.</li><li>The Agent Orchestrator redirects the request to an appropriate Agent</li><li>The Agent needs to access a third-party tool (like an API) on the user’s behalf.</li></ol><figure><img alt="Picture 2. A basic agentic flow" src="https://cdn-images-1.medium.com/max/1024/1*sNEuMfj5d8hPkjWFoLtdEA.png" /><figcaption><em>Picture 2. A basic agentic flow</em></figcaption></figure><p>The question is:</p><p><strong>How do we allow the Agent to access the Tool securely, without violating zero-trust principles?</strong></p><h3>What Not to Do</h3><h3>1. Use a Super-User Credential</h3><figure><img alt="Picture 3. Super-User Credentials" src="https://cdn-images-1.medium.com/max/1024/1*-QJd1QMegpeZ3r4xtsvpOA.png" /><figcaption><em>Picture 3. Super-User Credentials</em></figcaption></figure><p>Give the agent a powerful, long-lived API key. While simple, this has serious flaws:</p><ul><li>No user-specific access control — the agent can access anything, anytime.</li><li>Credential management complexity — distributing and rotating secrets is risky.</li><li>High blast radius — leaked credentials give unrestricted access.</li><li>Violates zero-trust — access should be evaluated per request, not baked into the agent.</li></ul><h3>2. Pass User Credentials End-to-End</h3><figure><img alt="Picture 4. User passed credentials" src="https://cdn-images-1.medium.com/max/1024/1*yCXjYovKHdMQTfnudZ4Z6A.png" /><figcaption><em>Picture 4. User passed credentials</em></figcaption></figure><p>Let the user authenticate once and pass their credentials down the chain.</p><p>Better, but still problematic:</p><ul><li>Credentials may reach untrusted components.</li><li>Not dynamic — we can’t know in advance which tools will be used.</li><li>Still violates per-request evaluation and leads to potential leaks or misuse.</li></ul><h3>The Better Way: OAuth2.0 and Token Exchange</h3><p>To solve these problems, we use OAuth2.0 — a well-established framework for delegated authorization.</p><figure><img alt="Picture 5. Agentic platform with Keycloak for Access Management" src="https://cdn-images-1.medium.com/max/1024/1*UmmKJi87kpZ2eQFbMaWb7Q.png" /><figcaption><em>Picture 5. Agentic platform with Keycloak for Access Management</em></figcaption></figure><h3>Authenticate the User with Keycloak</h3><p>We use <a href="https://www.keycloak.org/">Keycloak</a>, an open-source identity and access management system, as our OAuth authorization server to handle user authentication and token issuance. It manages login flows, token generation, and access policies for both users and workloads.</p><p>Here’s how it works:</p><ol><li>Users authenticate via a login component.</li><li>This login component redirects user to Keycloak.</li><li>User credentials are never shared with any part of the Agentic system outside of Keycloak.</li><li>Keycloak verifies the user and issues a short-lived access token, which includes:<br>— Signature<br>— Expiry (exp)<br>— Claims (roles, scopes, audience)</li></ol><p>This token can now be used by the Agent Orchestrator, but not passed all the way downstream. Why?</p><h3>Token Propagation Pitfall: Why Passing Tokens Downstream Breaks Security</h3><figure><img alt="Picture 6. The same token passed across all the components" src="https://cdn-images-1.medium.com/max/1024/1*9_tMGN-ON7CUKKs9e9Mypw.png" /><figcaption><em>Picture 6. The same token passed across all the components</em></figcaption></figure><p>A common pattern, passing the same user token across all components creates serious security risk:</p><ul><li>Sensitive user claims might be exposed to downstream services.</li><li>Downstream tools might reuse the token in unintended or unsafe ways</li><li>Violates least-privilege principle — tools can act on behalf of users or upstream components they should not have access to.</li></ul><p>We need a better method to scope access and limit token usage to a single intended component.</p><h3>The Right Way: Token Exchange</h3><p>OAuth2.0 supports Token Exchange: a mechanism where one access token can be exchanged for a new one with a different audience and reduced scope.</p><p><strong>Flow:</strong></p><ol><li>The Agent Orchestrator receives the user’s access token.</li><li>Before invoking the Agent, it calls Keycloak’s token exchange endpoint, presenting:<br> — The current token (as proof of delegation)<br> — The audience (e.g., the Agent)</li><li>Keycloak returns a new access token scoped only for the Agent.</li></ol><p>This pattern can repeat:</p><ul><li>The Agent exchanges its token to call a Tool.</li><li>The Tool’s token is scoped for its own use only. It cannot be reused to impersonate or call upstream components (like the Agent or Orchestrator).</li><li>If the Tool needs to access another service (e.g., a database or external API), it must perform its own token exchange, ensuring that access remains explicit and limited.</li></ul><p><strong>Benefits:</strong></p><ul><li>Agent tokens are scoped and restricted to a specific audience.</li><li>Each hop in the chain gets only the access it needs.</li><li>No component can misuse upstream tokens.</li></ul><h3>Authenticated Workloads: Enter SPIFFE and SPIRE</h3><p>Now, we have almost a failproof solution, but how do non-human components like Agents, Tools, and Orchestrator prove who they are to Keycloak without embedding secrets or using long-live credentials? Keycloak clients must be authenticated to communicate with Keycloak, however we don’t want to hand out secrets to every workload.</p><p>This is where <a href="https://spiffe.io/docs/latest/spiffe-about/overview/">SPIFFE</a> (Secure Production Identity Framework for Everyone) and its reference implementation <a href="https://spiffe.io/docs/latest/spire-about/spire-concepts/">SPIRE</a> come in. SPIFFE provides a standard for cryptographically verifiable identities for worklods, and SPIRE automates their issuance and attestation.</p><h3>SPIFFE + SPIRE:</h3><ul><li>SPIRE Server securely attests workloads (e.g. Agents, Tools) using platform tooling or capabilities.</li><li>It issues SVID (SPIFFE Verifiable Identity Documents) — short-lived, cryptographically signed identities.</li><li>SVIDs are typically x.509 certificates or JWTs.</li><li>These identities are tied to workload attributes (e.g., namespace, pod labels).</li><li>SPIRE also exposes JWKS endpoint (OIDC-compatible) that Keycloak uses to validate incoming SVIDs.</li></ul><h3>Authentication Flow using SPIRE:</h3><ol><li>Workloads request and receive SVIDs from SPIRE.</li><li>These workloads use SVID JWTs to authenticate to Keycloak.</li><li>Keycloak validates the identity using SPIRE’s OIDC endpoint.</li><li>Once verified, the workload is authorized to perform token exchange.</li></ol><h3>Final Architecture: Identity Flow with SPIFFE + Keycloak</h3><p>Now we have all the components needed. Let’s create a final architecture that enforces Zero Trust, least privilege, and short-lived credentials through the combination of OAuth2.0, Token Exchange, and SPIFFE/SPIRE workload identity.</p><figure><img alt="Picture 7. The final architecture: OAuth2.0, Token Exchange, and SPIRE" src="https://cdn-images-1.medium.com/max/1024/1*qFcClJ0H8NGjVAPr3GFBnQ.png" /><figcaption><em>Picture 7. The final architecture: OAuth2.0, Token Exchange, and SPIRE</em></figcaption></figure><h3>Here’s how it works, step by step:</h3><ol><li><strong>User Authentication</strong><br> — The user logs in via the Login component, which redirects to Keycloak<br> — Upon successful authentication, the user receives an OAuth2 access token, scoped only to the initial client (e.g., the Agent Orchestrator).</li><li><strong>Client Self-Registration</strong><br> — New agentic workloads (e.g., Agents, Tools) are dynamically provisioned by the platform. These workloads self-register as clients in Keycloak, presenting their SPIFFE identity.<br> — A human operator (or automated policy engine) then configures Token Exchange permissions in Keycloak, allowing specific clients to exchange tokens for specific downstream audiences. This enables decentralized provisioning with centralized policy enforcement.</li><li><strong>Workload Attestation</strong><br> — The SPIRE Server attests the workloads: Agent Orchestrator, Agents, and Tools.<br> — Each workload receives a SPIFFE Verifiable Identity Document (SVID), either a JWT or x.509 certificate, used to prove their identity to Keycloak and other services.</li><li><strong>Workload Authentication</strong><br> — The Agent Orchestrator authenticates to Keycloak using its SVID.<br> — This replaces the need for static application secrets or long-lived credentials.</li><li><strong>Token Exchange: Agent Orchestrator ➝ Agent</strong><br>The Agent Orchestrator uses the user’s token + its own identity (SVID) to request a Token Exchange from Keycloak.<br>The requested audience is the Agent.<br>Keycloak verifies that:<br> — The Orchestrator is a trusted client via SPIFFE ID.<br> — The token exchange for the Agent audience is permitted.<br>A new token, scoped only for the Agent, is issued.</li><li><strong>Token Exchange: Agent ➝ Tool</strong><br> — The Agent repeats the process: uses the received token + its own SVID to exchange it for a new token intended for the Tool.<br> — This new token is restricted in scope (audience = Tool) and has a short expiry.</li><li><strong>Tool Enforces Access</strong><br>The Tool validates the token:<br> — Verifies Keycloak’s signature.<br> — Confirms audience and expiry.<br>If needed, the Tool may request another token exchange for calling external APIs (e.g., HR systems, databases), using the same delegation and restriction mechanism.</li></ol><p>For more detailed flow, with code examples, check our <a href="https://github.com/kagenti/kagenti/tree/main/examples/identity/keycloak_token_exchange#demonstrating-keycloak-token-exchange">Kagenti project</a>.</p><h3>Benefits Recap:</h3><ul><li>User credentials are never passed to agentic components.</li><li>Short-lived tokens scoped to the specific audience and flow stage.</li><li>Agents and Tools are verified workloads using SPIRE, and don’t share credentials.</li><li>Each token exchange is explicit, auditable, and least-privilege scoped.</li><li>End-to-end auditability and traceability of each access decision and token use, enabling security reviews, incident response, and compliance validation.</li><li>Dynamic self-registration of workloads with Keycloak using SPIFFE identity, enabling automated provisioning while keeping policy control centralized.</li></ul><p>This model makes your Agentic Platform secure, composable, and ready for complex, dynamic workflows — without violating Zero Trust assumptions.</p><h3>Conclusion</h3><p>Dynamic agentic platforms require equally dynamic identity solutions. By combining OAuth2.0, Token Exchange, and SPIRE, we build secure, scalable systems that adhere to zero-trust principles without compromising developer velocity.</p><p>This approach enables secure, per-request delegation, granular access control, and workload attestation, all while avoiding the pitfalls of static credentials and over-permissioned systems.</p><p>We’ll be sharing more details about our implementation and lessons learned in future posts. Stay tuned, and if you are working on similar challenges, we’d love to collaborate!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996527f1c983" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rossoctl-the-agentic-platform/identity-in-agentic-platforms-enabling-secure-least-privilege-access-996527f1c983">Identity in Agentic Platforms: Enabling Secure, Least-Privilege Access</a> was originally published in <a href="https://medium.com/rossoctl-the-agentic-platform">Rossoctl — The Cloud Native Agentic Platform</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Tornjak with Helm Charts]]></title>
            <link>https://medium.com/universal-workload-identity/deploying-tornjak-with-helm-charts-e51fc21b962c?source=rss-a352fae2c28b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e51fc21b962c</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[spiffe]]></category>
            <category><![CDATA[hybrid-cloud]]></category>
            <dc:creator><![CDATA[Mariusz Sabath]]></dc:creator>
            <pubDate>Fri, 10 Sep 2021 15:09:40 GMT</pubDate>
            <atom:updated>2021-09-10T15:09:40.344Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous blogs, <a href="https://medium.com/@lumjjb">Brandon Lum</a> provided an overview of the <a href="https://medium.com/universal-workload-identity/untangling-the-multi-cloud-identity-and-access-control-problem-ba4a51ec0e4a">problems that multi-cloud deployments</a> are facing, and he presented a specific solution called Universal Workload Identity, followed by a deep-dive of the open-sourced <a href="https://medium.com/universal-workload-identity/shepherding-your-cloud-native-cattle-with-tornjak-eb0b9a7c96bc">project Tornjak</a> — a tool for managing workload identities in the cloud.</p><p>Brandon’s examples show a quick start and demonstrate a few basic setup scenarios.</p><p>In this tutorial, we will show you new examples of the deployment that provide a reference architecture of workloads that are using Tornjak and SPIRE in the Kubernetes environment.</p><h3>Deployment of Tornjak Server vs SPIRE Agents</h3><p>Every deployment scenario provided in this blogpost is made of two steps:</p><p>The first step deploys components needed for a centralized organization identity management server, mainly Tornjak UI service bundled together with the SPIRE server, typically installed in the “tornjak” namespace. The deployment is performed by the “tornjak” helm chart, which sets up all components, including the proxy for communicating with the SPIRE server, server configuration, roles, service account, etc. Additionally, this chart contains a plugin for deploying the <strong>OIDC</strong> extension that is used in the <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-oidc-tutorial.md">OIDC Tutorial</a>, and the optional multi-cluster configuration setup with various <em>nodeAttestors</em>, as described later.</p><p>The second step deploys the components needed for managing identity on the workload hosting side. The “spire” helm chart sets up a Kubernetes <em>daemonSet</em> for deploying SPIRE agents on every worker node, typically in the “spire” namespace. It also deploys an optional instance of <strong>The Workload Registrar,</strong> that dynamically registers workload entries in the SPIRE server, using a format provided by the Identity Template. More about the workload registrar<a href="https://github.com/IBM/trusted-service-identity/blob/spire-helm/docs/spire-workload-registrar.md"> here</a>.</p><p>Once everything is deployed, the <strong>Chief Information Security Officer (</strong>CISO) can access the Tornjak UI Dashboard interface, to view and manage all the SPIRE cluster configuration and metadata, securely manage agents, and workload identity entries.</p><h3>Assumptions</h3><p>These assumptions below support the specific deployment outlined by this tutorial. There are several ways to customize the components, however, following these assumptions makes the process easier.</p><h4><strong>Ingress Access</strong></h4><p>To work properly, the agents need to be able to access the SPIRE server. When everything is deployed in one cluster, the communication can be done internally, using services, as described later in the tutorial. But when the SPIRE server is in a different cluster, it requires opening an Ingress that allows for public access.</p><h4><strong>SPIFFE Trust Domain</strong></h4><p><strong>SPIFFE Trust Domain </strong>corresponds to the trust root of a SPIFFE identity provider. A trust domain could represent an individual, organization, environment, or department running their own independent SPIFFE infrastructure.</p><p>During all the deployments we will be using a single <em>SPIFFE Trust Domain.</em></p><p>All workloads identified in the same trust domain are issued identity documents that can be verified against the root keys of the trust domain.</p><p>Each SPIRE server is associated with a single <em>SPIFFE Trust Domain</em> that must be unique within that organization. Without federation, only agents that belong to the same Trust Domain can communicate with the SPIRE Server.</p><p>The trust domain takes the same form as a DNS name (for example, prod.acme.com), however, it does not need to correspond to any DNS infrastructure.</p><p>More about <a href="https://spiffe.io/docs/latest/planning/scaling_spire/#single-trust-domain">SPIFFE Trust Domains</a>.</p><h4>Persisting Server Data</h4><p>Tornjak and SPIRE servers are deployed as a Kubernetes <em>StatefulSet</em>. To persist the SPIRE data, the server keeps its artifacts in persistent storage. This can be done using <a href="https://kubernetes.io/docs/concepts/storage/volumes/#hostpath"><em>HostPath</em></a> — directly on the host system, or through <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">persistent volumes</a>.</p><h3>Deployment Scenarios</h3><p>This section describes various deployment scenarios, sorted by complexity, starting with the simple ones. We suggest running them in the specified order.</p><p>Instructions are based on helm charts and scripts available in our open-sourced <a href="https://github.com/IBM/trusted-service-identity/">Trusted Service Identity</a> repository, so go ahead and clone it now, as described in <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-helm.md#step-0-get-code">this documentation</a>.</p><p>Deployment Scenario List:</p><ul><li>Tornjak and SPIRE agents in one cluster</li><li>Tornjak and SPIRE agents on OpenShift</li><li>A single Tornjak and multiple SPIRE agent sets in multiple clusters, in multiple clouds</li></ul><h3>Deploy Tornjak and SPIRE agents in one cluster</h3><p>This is the simplest scenario that we are using in this tutorial. All the components are deployed in one cluster, separated by the namespaces. For this scenario, we don’t need any external communication setup (e.g. Ingress). SPIRE agents can communicate directly with the SPIRE Server via Services.</p><p>CISO can access and manage Tornjak UI directly using HTTP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uVeqGoiA3lrR8fNvHFGotg.png" /><figcaption>Single Cluster on local minikube or kind</figcaption></figure><p>This scenario requires a local Kubernetes runtime like <em>minikube</em> or <em>kind</em>. The detailed instructions to follow, including the Prerequisites, are provided <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-helm.md#prerequisites">here</a>.</p><p>First, we deploy the SPIRE server using the helm chart, then we skip the Ingress setup, and go directly to accessing the Tornjak Server as <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-helm.md#accessing-tornjak-server">documented here</a>.</p><p>After successfully testing the access to Tornjak, we can move on to the next stage — deploying the SPIRE agents. The agents need certificates to securely communicate with the SPIRE server via TLS. Certificates are kept in “spire-bundle” <em>configMap</em> in the “tornjak” namespace. Because we will be deploying the agents in the “spire” namespace, we need to copy the certificates there as well.</p><p>This command does the trick:</p><pre>kubectl -n tornjak get configmap spire-bundle -oyaml | kubectl patch --type json --patch &#39;[{&quot;op&quot;: &quot;replace&quot;, &quot;path&quot;: &quot;/metadata/namespace&quot;, &quot;value&quot;:&quot;spire&quot;}]&#39; -f - --dry-run=client -oyaml &gt; spire-bundle.yaml</pre><p>It creates the “spire-bundle.yaml” file that can be used for creating the certificates in other places. This is a useful command, and we will use it in all the scenarios.</p><p>So, let’s deploy it:</p><pre>kubectl -n spire apply -f spire-bundle.yaml</pre><p>Now, let’s just set up access to the SPIRE server. We can use the <em>“ExternalName Service”</em> method because everything is deployed in the same cluster, and then we can run the deployment using “spire” helm charts.</p><p>When deploying SPIRE agents, we also deploy <strong><em>The Workload Registrar</em>,</strong> which dynamically registers workload entries in the SPIRE server, as Kubernetes pods are created. This last step requires manual registration of the <em>Registrar</em> and then we are ready to deploy the test workload. These last steps are described in <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-workload-registrar.md#register-workload-registrar-with-the-spire-server">the following document</a>.</p><p>Once the <em>Registrar</em> is properly registered, you will see new entries showing up on the Tornjak interface.</p><p>Now we can deploy a sample workload, view it on the Tornak Entry List, and then test the process for obtaining its identity, as described <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-workload-registrar.md#create-sample-deployment">here</a>.</p><h3>Deploy Tornjak and SPIRE agents on OpenShift</h3><p>The next scenario explores a deployment on <a href="https://www.redhat.com/en/technologies/cloud-computing/openshift/container-platform"><em>OpenShift</em></a><em>.</em> Since OpenShift is a cloud-based Kubernetes container platform, the main benefit of deploying in there, is its ability to be Cloud provider agnostic. This means there is the same interface and behavior no matter the underlying cloud provider. Additionally, it introduces elevated security and integrated Continuous Integration/Continuous Delivery (CI/CD) Solutions, as well as being highly scalable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hd_SmQY-N9lal5jPnFSKLQ.png" /><figcaption>Single Cluster on OpenShift</figcaption></figure><p>For this example, we will use <strong><em>Red Hat OpenShift on Kubernetes</em></strong> (ROKS) in IBM Cloud. To get a test cluster on ROKS, follow the steps outlined here:<a href="https://www.ibm.com/cloud/openshift"> https://www.ibm.com/cloud/openshift</a></p><p>Tornjak deployment is similar to the local cluster, but since OpenShift introduces additional security constraints, there are a few, additional steps outlined in <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-on-openshift.md">this document</a>.</p><p>The main benefit of this exercise is that now Tornjak server is publicly available, so we can easily access it from other locations, other clusters, and even other clouds.</p><h3>Deploy a single Tornjak and multiple SPIRE agent sets in multiple clusters, in multiple clouds</h3><p>Lastly, we will set up a multi-cloud solution where Tornjak and SPIRE server are hosted in one cluster, and workloads with the SPIRE agents are deployed in different clusters hosted in different clouds (IBM Cloud and Amazon EKS).</p><p>The benefit of keeping the Tornjak and the SPIRE server separate from the SPIRE agents and workloads is the threat model. The blast radius attacks initiated from inside the workload would be limited to the agent side, and they should not affect the SPIRE server configuration. Another advantage is the ability to use one the Tornjak instance to manage multiple clusters of agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVYo5qNDDwvNQpho_SxXnQ.png" /><figcaption>Multi-cluster deployment</figcaption></figure><p>When we deploy workloads in clusters that are different from the one that is hosting Tornjak, the SPIRE agents must be able to communicate with the SPIRE server using a public interface. This is typically done by setting up <em>Ingress</em> or <em>External Routes</em> that expose external communication with the cluster. Ingress setup depends on the Cloud Service Provider.</p><p>Another challenge is that SPIRE Server must be able to trust the remote agents. This is done through the node attestation. The default <em>nodeAttestor</em> is the Kubernetes attestor (<em>“k8s_psat”</em>). The idea is as follows — we need to capture the portable KUBECONFIG files for every remote cluster and pass them to the SPIRE server, so it can make a callback to each remote cluster, and pull Kubernetes level information about each worker node. This was not needed when the SPIRE server was running in the same cluster as the agents.</p><p>In addition to the Kubernetes attestor, there are also cloud-provider-specific attestors that we are experimenting with. At the moment of writing this blog, we were able to successfully use AWS <em>nodeAttestor</em>.</p><p>We suggest using the OpenShift cluster created in the previous exercise to host the central Tornjak server. We can either reconfigure the existing Tornjak deployment or reinstall it using the multi-cloud setup.</p><p>The required steps are described in details in the <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-multi-cluster.md">mulit-cluster document</a> .</p><h3>What’s Next?</h3><p>Hopefully, you found this set of examples useful, and now you are ready to deploy actual applications that take advantage of the Universal Workload Identity approach, to better control access to services or applications hosted in multiple clouds.</p><p>Our next tutorial will discuss Templates for Custom Identity format. This functionality comes in handy when we are using the multi-cluster solution. It is a good idea to pass the cluster-specific information like the cluster-name, region, or data-center location together with the identity. All this additional information can be configured on the cluster level during the SPIRE agents’ deployment. For now, see more info <a href="https://github.com/IBM/trusted-service-identity/blob/main/docs/spire-workload-registrar.md#configure-workload-registrar">here</a>.</p><p>We will also focus on providing instructions for setting up OpenID Connection (OIDC) as a secure solution for multi-cloud remote access to AWS S3 storage, and secrets managed by <em>HashiCorp Vault</em>, using a single Tornjak server that provides better control over security and identity auditability for your organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e51fc21b962c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/universal-workload-identity/deploying-tornjak-with-helm-charts-e51fc21b962c">Deploying Tornjak with Helm Charts</a> was originally published in <a href="https://medium.com/universal-workload-identity">Universal Workload Identity</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secret Management with Trusted Service Identity for Sensitive Data Protection]]></title>
            <link>https://medium.com/@mrsabath/secret-management-with-trusted-service-identity-for-sensitive-data-protection-e511eb66d87f?source=rss-a352fae2c28b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e511eb66d87f</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Mariusz Sabath]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 17:49:34 GMT</pubDate>
            <atom:updated>2020-02-13T17:49:34.287Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, we will explore the problems with current methods of storing secrets in the Kubernetes cluster and give an overview of the solution called Trusted Service Identity, recently open-sourced by IBM Research. This project ties Secret Management technologies with workload identity via host provenance and integrity. For developers who like to play with new technologies, we provide a step by step installation and demo.</p><h4><strong>What is the problem with secrets?</strong></h4><p>Today, if a micro-service needs to retrieve or exchange data with some other external or internal service, it must authenticate itself, and that requires secrets (password, API key, certificate, etc.) that are typically stored as the Kubernetes secret.</p><p>Even though Kubernetes secrets were design to be stored securely (e.g. encrypted at rest) by the orchestration, they can be simply mounted to an arbitrary container and read by anyone who has access to this namespace including Cloud Operators. Those knowing the secret, can also access the sensitive data that need to be protected.</p><p>The problem is, that these operators or admins might be third-party employees and may not be certified to access this data or they could lack security clearance or legal requirements e.g. specific citizenship. On top of that, there is an increasing need for an accurate audit trail, to track who (and what process) accessed what and when.</p><h4>Typical customer implementation</h4><p>Let’s take a closer look at the typical customer implementation today. The application is deployed in the Kubernetes cluster and needs to access some DB (e.g. IBM Cloudant). In order to authenticate itself to this DB, an application must present some authentication token or a user-id and password, API key or certificate. Since this token should not be hard-coded into the image used for instantiating this application, the container must be able to obtain this information during run-time or initialization. Today, this is typically done via Kubernetes secrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDcmnbrK3xcPaEWd88ZTHQ.png" /><figcaption>Typical secret deployment</figcaption></figure><p>The problem with this solution is that the secrets are static, meaning all the containers that belong to the same deployment will get the same secret. To make matters worse, anyone with access to this namespace will be able to read it.</p><p>We could go a bit further in fixing it and move the secret into a secure Vault. Then all we keep in the cluster is a secret allowing us to read from Vault, but this will not stop a malicious administrator from eventually getting access to the sensitive data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0z0hPbtFl_pHPgnrsjzCg.png" /><figcaption>Typical secret deployment with Vault</figcaption></figure><p>With this model, the secret will remain as-is during the deployment and there is no easy way to control the lifetime of the secret, its expiration or rotation. In addition, all the secrets are created and managed by humans.</p><p>If the deployment spans multiple locations or regions, the management and governance over who has access to what, becomes a nightmare. The controls are not unified and tracking it without global visibility becomes very difficult.</p><h4>Trusted Service Identity by IBM Research</h4><p>Our IBM Research team together with IBM Cloud and Cognitive Software group has been working for a while on addressing this problem. The solution is called Trusted Service Identity (TSI).</p><p>As a result, secrets are stored securely in Vault, which is an industry-standard tool for managing secrets, and they are injected to the application by Trusted Service Identity based on the workload identity of the application. The release of secrets is governed by very fine-grained access control policies managed by the Chief Information Security Officer (CISO).</p><p>Policies, as well as the secrets, can be dynamically modified, keys revoked or updated, without changing anything in deployed applications, while TSI retains detailed audit trail tracking who, meaning what process, got what and when.</p><h4>How is it done?</h4><p>Every container that requires a secret for accessing the sensitive data, gets assigned a short-lived measured identity, in form of the JSON Web Token (JWT) token, that is signed with the root of trust. This measured identity is a form of <em>a digital biometric</em> that contains run-time measurements of the application, like image name, namespace, cluster name, location, data center, etc.</p><p>The actual secret is stored securely in Vault and it is protected by policies that define what application can retrieve it. The Vault auth plugin validates the token, checks the policy, then TSI injects this secret to the running container, the same way as regular Kubernetes secrets are done, but without being stored anywhere in Kubernetes. The policies can be dynamically modified, keys revoked, without touching the application.</p><p>This solution allows for very fine-grained access control and uniformed tracking of who got what and when.</p><h4>Interesting use cases</h4><p>TSI addresses several different use cases. Here are a few interesting examples:</p><ul><li>Permissions can be enforced based on the workload identity or a geographical location. <em>“Data about Europeans can be only accessed within European geographical boundaries, by an application that was well tested, validated, approved and continuously monitored for compliance”</em> (example of GDPR)</li><li>Selected images must be executed only in specific data centers under specific accounts and clusters</li><li>Provide short-lived access to data, using quick expiration dates</li><li>Automate the frequent key rotation without modifying the application</li><li>Common deployment code across multi-clusters, multi-geographical distribution with a location-based data restriction</li><li>Different data feeds for different environments (development, staging, production)</li><li>Fine controlled, the dynamic release of secrets and configurations at runtime or specific time</li></ul><h4>Quick demo</h4><p>Here is a quick demo of Trusted Service Identity running on Minikube. If you like to follow the installation and demo instructions, please visit our GitHub project: <a href="https://github.com/IBM/trusted-service-identity">https://github.com/IBM/trusted-service-identity</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSgTDyfmQJ9w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSgTDyfmQJ9w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSgTDyfmQJ9w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eba7db02eb13cfe9db034d472af57eb2/href">https://medium.com/media/eba7db02eb13cfe9db034d472af57eb2/href</a></iframe><h4>OK, tell me more</h4><p>Detailed TSI Architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8vehp8HS9PF57WtdS5g1g.png" /><figcaption>Trusted Service Identity</figcaption></figure><p>In the pictured example, we want to protect access, to the sensitive data in IBM Cloudant.</p><p>We also have a Kubernetes cluster, extended with TSI, where each node gets intermediate CA, signed by the root CA (Certificate Authority) during the secure bootstrapping of the cluster. Each node has also a JWT Signing Service, that contains singing authority. This could be either software or hardware TPM.</p><p>Root CA is securely stored in Vault and Vault is extended with our authentication plugin, <em>Trusted Service Identity Authentication Vault plugin</em>.</p><p>Before we begin, Chief Information Security Officer (CISO), or some other secure process, deploy in Vault secret keys for accessing the Cloudant DB. Then creates policies, that define what are the measurements of the application that can get access to these secrets. These measurements represent the identity of the application.</p><p>Now, we can deploy an application in this cluster. This application wants to access the sensitive data stored in IBM Cloudant. Cluster Operator annotates the deployment of this application with the names and types of the secrets to be injected and their location.</p><p>Here is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*43z8os-NzyEkxPdvDCWJPQ.png" /><figcaption>Pod annotation for retrieving secrets</figcaption></figure><p>In this example, there are 2 secrets requested: “mysecret1” and “mysecret2”. The first one is using role <em>demo</em> and the policy type <em>ti-demo-all</em> and the second one type <em>ti-demo-r</em>.</p><p><em>admission.trusted.identity/inject: “true</em>” — this means the TSI will be activated for this deployment and the modified TSI Admission Controller would start the sidecar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtQgRvbVyNepCshAUR3Q3A.png" /><figcaption>Trusted Service Identity component interaction</figcaption></figure><p>The sidecar collects the measurements called <em>claims</em> that define the identity of this application and sends them securely to the Signing Service. The Signing Service signs these claims using the intermediate CA that was installed during the secure bootstrapping of the cluster and returns the JWT token back to the sidecar. The sidecar requests the annotated secrets from Vault by passing the JWT token along with the request.</p><p>Our Authentication Plugin in Vault intercepts the request call, validates the signature and the expiration date on the JWT token, and if everything is OK, it uses the provided claims against the policies to retrieve the secret. If the measurements match the policy, the secret is released to the application. The sidecar injects the secret to the running container mounting it at a specified location, e.g. <em>mysecrets/myubuntu-mysecret1</em> in our example.</p><p>As a result, the secret has been delivered to the container run-time memory, without being ever stored anywhere in Kubernetes, but from the point of view of the application, there were no additional changes needed.</p><p>Here is a sample JWT token created by TSI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SwFX_DNa_ydOfntKtIsM6g.png" /><figcaption>Sample JWT Token</figcaption></figure><p>It is composed of three parts, the header, payload containing the actual claims and signature for validation. The claims included here represent the measured identity of the application. They contain <em>cluster-region</em> ( e.g. Germany, eu-de), <em>cluster-name</em>, individual <em>machineid</em> which is a unique worker node id, unique pod id, and the list of images making up the pod, including the image signature, so the image can be validated and guarantee that the application is running the code that you want to be running, and it was not tampered with.</p><p>There is also a token expiration timestamp, typically set to 30 seconds, to make these tokens quite ephemeral and short-lived, protecting the security from leaking.</p><p>These are the run-time measurements that represent the identity of the application, signed with the root of trust and used for evaluation against policies controlling the secrets.</p><h4>Sample policies and keys</h4><p>The secrets stored in Vault are protected by policies. Policies are composed of the policy-type and the attributes, the same that are used for building claims, and they represent the path to the secret. If the claims provided in the request match the policy attribute path, the secret will be released to the application.</p><p>Here are sample policies created in Vault:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1ozMTZuT62vidq8gQJy8g.png" /><figcaption>Sample Vault policies</figcaption></figure><p>The top policy, `ti-demo-all` uses all the main attributes from the claims: cluster-region, cluster-name, namespace and image to build a path for the secret. In this example it allows `read` access to the secret.</p><p>The second policy defines a path using only two measurements, <em>cluster-region</em> and <em>image</em>. Simply put, it allows read access to the secret as long as the container is running from a specific region and it was instantiated from a specific image.</p><p>Now, let’s try to create a few sample keys, that are using these policies. First, we want to create a key with the path matching the top policy, using <em>trusted-identity</em> default namespace. The format would be a following:</p><pre>vault kv put secret/ti-demo-all/${REGION}/${CLUSTER}/trusted-identity/${IMGSHA}/mysecret1 secret=${SECRET}</pre><p>Here is the actual command for a key to be used by a container running in Germany (eu-de) in EUCluster using <em>ubuntu 18.04</em> image with a specific signature <em>250cc6f3f3ffc5cdaa9d8f4946ac79821aafb4d3afc93928f0de9336eba21aa4</em> for the container.</p><pre>REGION=”eu-de”<br>CLUSTER=”EUCluster”<br>SECRET=”very5ecret!value”</pre><pre># First we need to convert the image into the encoded imageSHA:<br>export UIMG=”ubuntu:18.04@sha256:250cc6f3f3ffc5cdaa9d8f4946ac79821aafb4d3afc93928f0de9336eba21aa4”<br># For Mac OSX<br>UIMGSHA=$(echo -n “$UIMG” | shasum -a 256 | awk ‘{print $1}’)<br></pre><pre>vault kv put secret/ti-demo-all/${REGION}/${CLUSTER}/trusted-identity/${UIMGSHA}/mysecret1 secret=${SECRET}</pre><p>And here is an example of creating a key for the second policy format:</p><pre>SECRET2=”ev3nBetter5ecre#”<br>vault kv put secret/ti-demo-ri/${REGION}/${UIMGSHA}/mysecret2 secret=${SECRET2}</pre><p>Once the keys are stored in the Vault, the container annotated as in the example above, running from <em>eu-de</em>, in cluster <em>EUCluster</em> with an image <em>ubuntu.18.04</em> matching the image signature, will get the <em>mysecret1</em> and <em>mysecret2</em> injected in specified locations.</p><h4>Separation of roles</h4><p>TSI nicely separates functional roles for different personas. Here is the list of all individual functions corresponding to personas used by TSI:</p><ul><li><strong>Cluster Installer</strong> — responsible for bootstrapping TSI before on-boarding of the applications. After the cluster is created, private keys need to be installed on every worker node, then each node needs to be registered with Vault to obtain <em>intermediate CA</em>. This persona requires sysadmin access role to Kubernetes cluster and Vault admin access to obtain certificates for JWT Signing Service (JSS). Eventually, this role will be replaced by CI/CD bootstrapping pipeline that is deploying the cluster.</li><li><strong>Application Operator</strong> — to deploy, operate and manage the application that is using TSI and as such, this persona requires access to the Kubernetes cluster namespace. The application must be annotated with secret names that will be injected to the container.</li><li><strong>Policy Maker </strong>— Chief Information Security Officer (CISO). This persona manages or approves policies in Vault . It defines what policy type and what are the measurements of the application that is able to obtain the secret. Requires read/write access to Vault.</li><li><strong>Secret Maker </strong>— this persona, currently similar to Policy Maker, manages the secret life-cycle (create, delete, revoke, rotate, etc). Requires read/write access to Vault. This might be replaced by the automation process managing the secrets.</li></ul><h4>Detailed Overview of TSI</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FrmdzVJc37cg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DrmdzVJc37cg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrmdzVJc37cg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8c90b20b35100cbc4c337452879790b/href">https://medium.com/media/a8c90b20b35100cbc4c337452879790b/href</a></iframe><h4>Ready to play with TSI?</h4><p>Trusted Service Identity has been open-sourced and publicly available in GitHub: <a href="https://github.com/IBM/trusted-service-identity">https://github.com/IBM/trusted-service-identity</a></p><p>If you are ready to give TSI a try, all you need is the Kubernetes cluster. We have been testing TSI on <a href="https://www.ibm.com/cloud/kubernetes-service/">IBM Cloud Kubernetes Service</a>, <a href="https://minikube.sigs.k8s.io/">Minikube</a> and OpenShift(e.g. <a href="https://github.com/code-ready/crc">Red Hat CodeReady Containers</a>).</p><p>To install, please follow the simple steps outlined <a href="https://github.com/IBM/trusted-service-identity#installation">here</a> then follow our <a href="https://github.com/IBM/trusted-service-identity/blob/master/examples/vault-plugin/README.md">Vault sample demo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e511eb66d87f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>