<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Isuru Cumaranathunga on Medium]]></title>
        <description><![CDATA[Stories by Isuru Cumaranathunga on Medium]]></description>
        <link>https://medium.com/@isurucuma?source=rss-71c96e854bde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CDkSeJ0pX3bYZhgIVo-p7g.jpeg</url>
            <title>Stories by Isuru Cumaranathunga on Medium</title>
            <link>https://medium.com/@isurucuma?source=rss-71c96e854bde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:25:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@isurucuma/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Saving Too Often Becomes Dangerous]]></title>
            <link>https://medium.com/codex/when-saving-too-often-becomes-dangerous-87aad4ba2809?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/87aad4ba2809</guid>
            <category><![CDATA[autosave-systems]]></category>
            <category><![CDATA[adaptive-rate-limiting]]></category>
            <category><![CDATA[aimd]]></category>
            <category><![CDATA[database-performance]]></category>
            <category><![CDATA[backpressure]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 19:31:39 GMT</pubDate>
            <atom:updated>2026-10-02T08:19:14.570Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Canva Used Backpressure to Protect Its Database</h4><h3>When Canva became difficult to use</h3><p>People opened Canva to get something done a presentation, a poster, a design they needed to finish. But during periods described by Canva’s engineers in May 2015, the service suffered severe slowdowns that affected everyone using it.</p><p>Put yourself at that screen. You have work to finish, and the application is struggling. The surprising contributor was autosave: the feature protecting your progress was putting pressure on the wider product. Saving less frequently could help, but it would leave more recent changes exposed if your browser crashed.</p><p>This decade-old case still matters whenever clients send work faster than a backend can handle it. Through Canva’s solution, you’ll learn how <strong>backpressure</strong> lets the backend ask clients to slow down, and how <strong>adaptive autosave intervals</strong> reduce demand quickly and restore it carefully. You’ll see when that approach can help your own systems and what it costs the people using them.</p><p>We’ll follow your birthday-card changes into Canva’s saving mechanism, working through those decisions together.</p><blockquote>Canva published its account on May 28, 2015, without individual incident dates, durations, affected-user counts, or a breakdown of failed actions. Our birthday card example illustrates the design decisions; it isn’t a reported incident.</blockquote><h3>Your latest changes need somewhere to go</h3><p>You’re making a birthday card. You’ve replaced the heading, moved a photograph, and spent a few minutes getting the colours right. The card now looks the way you want it to.</p><p>If your browser closed unexpectedly now, how much would you have to do again? That depends on which changes had reached the server before the interruption. Seeing your new heading on screen doesn’t, by itself, tell you that it has been saved remotely.</p><p>This is why Canva originally attempted an autosave every five seconds while a design kept changing. Frequent saves reduced the amount of recent work exposed to a crash or an accidentally closed tab.</p><p>From your chair, that is a sensible promise. You can concentrate on the card while the editor handles saving. You shouldn’t need to understand the machinery behind it.</p><p>But each attempt asks that machinery to do some work. To understand how a feature helping you could affect everyone else, follow one of those saves beyond your screen.</p><h3>Your save is one of many</h3><p>When you change the heading, the client has a difference to send. Engineers call these differences <em>deltas</em>. Canva periodically sent them through an operation called update-content. Its document service handled the save path, with design information stored in MongoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVaIRGwAYdIhsMlTHh2ILw.png" /></figure><p>Your changing design sends deltas through update-content and the document service to MongoDB, many active editors multiply the same save path into a shared workload.</p><p><em>Your save joins the saves from other active editors, all drawing on shared database write capacity.</em></p><p>Your card creates only a small stream of requests. Add 1,000 continuously active editors, each saving every five seconds, and the combined rate is roughly 200 requests a second. At 10,000 such editors, it is roughly 2,000.</p><p>These are illustrative numbers. The calculation is:</p><pre>approximate autosave QPS = continuously active editors ÷ autosave interval</pre><p>QPS means queries per second; here, we’re counting autosave operations. It tells you how much save traffic the backend handles each second. An idle editor is not assumed to keep writing every five seconds.</p><p>Sending differences keeps the client from needing to send the whole design each time, but those saves still create database work. Canva identified autosave as its largest contributor to write load.</p><p>Your five-second schedule now has company. Each editor is making the same reasonable request, and together those requests can exceed what the shared system can handle.</p><h3>You have to keep this working during maintenance too</h3><p>Now take the engineering seat for a moment. You need to keep those card changes safe, but thousands of other editors need the same thing.</p><p>Your database has a limited ability to process writes while maintaining acceptable performance. That is its available <em>write capacity</em>. During an activity spike, incoming saves can consume that capacity faster than the system can comfortably serve them.</p><p>There is another complication. You may need to migrate data or perform maintenance. That work can temporarily reduce the capacity available for saves even if the number of people editing hasn’t changed. Canva described both situations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYaQ-VB94D6Tp8oncebPCA.png" /></figure><p>A rise in design activity increases demand, while maintenance reduces available capacity; either can push autosave writes beyond the safe operating boundary.</p><blockquote><em>Keeping the product responsive requires matching save demand to the capacity available now.</em></blockquote><p>So a traffic level that was fine yesterday may be too much during today’s maintenance. Meanwhile, the client (browser or IOS application) editing your birthday card still follows its five-second schedule.</p><p>The engineering problem is becoming clearer: the saving policy needs a way to respond when conditions change.</p><h3>Giving the backend breathing room</h3><p>You could buy enough database capacity to handle exceptional peaks. Canva considered that, but much of the expensive extra capacity would sit unused under ordinary load.</p><p>You could also lengthen every autosave interval permanently. Think back to the heading and photograph on your card: you would be accepting a larger gap between remote saves even when the system had plenty of room.</p><p>A more useful arrangement lets clients save frequently under ordinary conditions and wait longer when the backend needs relief.</p><p>To do that, the backend must communicate with the clients producing the work. It needs to return information that influences when the next save happens.</p><p>That feedback is <strong>backpressure</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ImpNolxZCDbbU89LCzOwbg.png" /></figure><p><em>Backpressure gives your client information it can use to reduce future save traffic.</em></p><p>Your browser or iOS app is the <em>producer,</em> it creates save requests. The document service and database form the path consuming that work. Canva added feedback from that path so clients could change their behavior.</p><p>For your engineering decision, the useful property is where the reduction happens. A longer client interval means fewer future requests enter the backend at all. Across many editors, that can relieve the workload responsible for the wider slowdown.</p><p>Now the backend needs a reliable basis for requesting that change.</p><h3>Deciding when to ask clients to slow down</h3><p>From the engineering seat, you need to observe both how much save traffic is arriving and how well the service is handling it.</p><p>Canva monitored update-content using QPS and latency. Latency is how long an operation takes to complete. A threshold sets the point beyond which a measurement should trigger a slowdown.</p><p>QPS helps you detect rising activity, more people editing means more saves. Latency gives you another view. During maintenance, the same request rate may take longer to process because less capacity is available.</p><p>Canva compared both measurements with configured thresholds. If either exceeded its limit, the response included:</p><pre>{ &quot;throttle&quot;: true }</pre><p>That boolean true or false gave the client enough information to adjust its next interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNmR2UU96lzqOH1rvMUZcg.png" /></figure><p>The client sends a delta; Canva measures update-content QPS and latency, compares them with thresholds, returns a throttle signal, and the client adjusts its interval to change future load.</p><blockquote>Measurements become a response signal, the signal changes your client’s schedule, and the new schedule changes the measured workload.</blockquote><p>This keeps the responsibility manageable. Your editor doesn’t need to understand database maintenance or the service’s deployment. The backend decides whether its thresholds have been crossed; the client responds to the result.</p><p>Those thresholds could be adjusted at runtime. That gives operators a way to lower the target during planned maintenance. Canva also factored in the number of running service instances: a total traffic target must be interpreted in light of how that traffic is distributed across the service.</p><p>The published monitor used a five-minute request rate and median latency the middle value among sampled durations. <strong>It excluded failures from its timing statistics because failures tended to return quickly and could make the measurements look misleadingly healthy.</strong></p><p>That is a useful detail when you’re choosing what to measure. A quick failure doesn’t mean your card was saved quickly.</p><p>The feedback path is now complete on the backend side. The next decision belongs to the client: how much should it change its schedule?</p><h3>Your client receives the signal</h3><p>Return to the birthday card. You’re still making changes, and your editor receives a save response containing throttle: true.</p><p>Canva’s client doubles its autosave interval. Five seconds becomes ten. If subsequent responses continue requesting throttling, ten becomes twenty, then forty, subject to configured limits.</p><p>That produces substantial relief. Ignoring request duration for this simple comparison, a five-second interval means twelve saves per minute; ten seconds means six; twenty seconds means three. Each doubling roughly halves the request rate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfQBPNCdNWqhV0MYD9r38g.png" /></figure><p>During pressure, autosave intervals rise from 5 to 10 to 20 to 40 seconds; during recovery they fall by one second per healthy response, while request rate moves in the opposite direction.</p><p><em>Your client reduces demand quickly, then restores frequent saving in smaller steps.</em></p><p>Once the backend returns throttle: false, the published iOS code subtracts one second from the interval: forty becomes thirty-nine, then thirty-eight on the next healthy response, continuing toward the minimum.</p><p>Why return so carefully? If every client jumped straight back to five seconds, the demand you just removed could arrive again before you knew the database had room for it. Smaller adjustments let the next round of measurements inform the next decision.</p><p>Canva called this <strong>Additive Increase / Multiplicative Decrease</strong>, or AIMD, borrowing the approach from TCP congestion control. The idea is to reduce demand decisively and restore it cautiously.</p><blockquote>Minimum and maximum intervals bound the adjustment. Canva did not publish their production values.</blockquote><p>For your birthday card, the client is now changing how long it waits between saves. That interval is different from request latency, which measures how long an individual save takes to process. Shortening the interval increases how often you save.</p><h3>Checking whether the feedback works</h3><p>With this design in place, you would want to see two behaviors: demand falling when the threshold is lowered, and returning gradually when the restriction is lifted.</p><p>Canva demonstrated both on three selected production document-service instances. Together, they were handling about 14 update-content requests per second. Engineers then applied a 6 QPS threshold across those instances, about two per instance.</p><p>As clients received the throttle signal, they reduced their save rate. Throughput fell and settled around the target. When engineers removed the threshold, clients gradually returned toward their preferred frequency.</p><p>For the engineering problem you’ve been working through, the demonstration closes an important gap: the clients actually changed the workload reaching the service.</p><h3>What if everyone’s next save lands together?</h3><p>Keep your birthday card’s browser client in mind for one related consideration beyond Canva’s published implementation.</p><p>Your editor now waits twenty seconds between saves. If many other clients happen to follow the same schedule from the same starting moment, their requests can still arrive in clusters. The average rate may have fallen while individual bursts remain large.</p><p>That synchronized rush is a form of the <strong>thundering-herd problem</strong>. Equal intervals alone don’t guarantee synchronization, but clients acting together can preserve it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wgppTtJGQoCcnzFrJiVhMA.png" /></figure><p><em>Related enhancement, not documented by Canva: jitter spreads client actions across time.</em></p><p>You can reduce that alignment by adding <strong>jitter</strong>, a randomized adjustment to scheduling. Your client’s wait and another client’s wait differ slightly, spreading future saves within a chosen window. The adjustment must respect the intended save-interval limits.</p><p>AWS illustrates this principle in its analysis of retries: deterministic backoff can leave clusters of calls, and jitter helps spread them out. Applying the idea to autosave is our architectural extension. Canva’s account does not report using jitter or experiencing a thundering-herd incident.</p><h3>Back to the card you were making</h3><p>You’re still trying to finish the same birthday card. From your side of the screen, the goal hasn’t changed: keep the application usable and protect the changes you’ve made.</p><p>The engineering response now makes sense in those terms. The client saves frequently when the backend’s measurements permit it. When demand grows or available capacity falls, feedback lengthens the interval and reduces write pressure. When conditions improve, the interval shrinks gradually.</p><p>There is still a cost you can feel. If your latest heading change hasn’t reached the server, a longer interval leaves it exposed for longer. Canva’s account doesn’t describe enough client recovery behavior to promise what would survive a crash.</p><p>The controller has limits too. It depends on clients receiving and following its signal. Smoothed measurements take time to react, and a healthy median can conceal slow requests. Adaptive saving manages the trade-off; it cannot guarantee every save under every condition.</p><p>What Canva demonstrated was a way to adjust an expensive shared workload while preserving its purpose. You started with a card to finish. Following your changes from the editor to the database showed why the next save needed to take the rest of the system into account.</p><p>That is the lesson to take into your own systems: when a useful feature starts overwhelming shared resources, examine how its producers decide when to create work. Giving them feedback may let you preserve the feature while reducing the pressure it creates.</p><h3>Sources</h3><ol><li>Brendan Humphreys, Canva Engineering, <a href="https://www.canva.dev/blog/engineering/knowing-when-to-back-off/">“Knowing when to back off”</a>, May 28, 2015.︎</li><li>Marc Brooker, AWS Architecture Blog, <a href="https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/">“Exponential Backoff and Jitter”</a>, March 4, 2015; updated May 2023.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87aad4ba2809" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/when-saving-too-often-becomes-dangerous-87aad4ba2809">When Saving Too Often Becomes Dangerous</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Pod Has an Identity. Use It.]]></title>
            <link>https://medium.com/@isurucuma/your-pod-has-an-identity-use-it-a95b9d96d486?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/a95b9d96d486</guid>
            <category><![CDATA[zero-trust]]></category>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[azure-active-directory]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[workload-identity]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Sat, 23 May 2026 20:35:18 GMT</pubDate>
            <atom:updated>2026-05-23T20:35:18.096Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Azure Workload Identity: From Kubernetes Service Accounts to Zero-Secret Cloud Access</em></h4><h3>Where This Started</h3><p>In my recent work, I joined a team running almost all of their workloads on Azure Kubernetes Service, spread across multiple clusters. They had taken a zero-trust approach to security, which meant that even services within the same cluster and the same namespace had to authenticate before talking to each other or to any Azure resource. Nothing was implicitly trusted just because it was “inside.”</p><p>Coming into that environment, I had to learn Azure Workload Identity properly, not just follow a tutorial and copy-paste some YAML. I needed to understand what was actually happening at each step. This article is the way I eventually logged it into my mind, and I am writing it down here hoping it helps anyone else trying to build the same mental model.</p><h3>The Problem: Where Does the Password Live?</h3><p>When a service running inside Kubernetes needs to read a secret from Azure Key Vault, write a message to Azure Service Bus, or query an Azure SQL database, it needs to prove its identity to Azure first. That proof has traditionally been a client secret: a password the service sends alongside its requests.</p><p>The problem is simple: a password has to live somewhere.</p><p>In a Kubernetes setup, that “somewhere” is usually a Kubernetes Secret object, an environment variable in a pod spec, or a config file baked into a container image. All of these share the same underlying problem. You created a credential, you stored it, and now you have to:</p><ul><li>Rotate it before it expires or is compromised</li><li>Make sure no one accidentally logs it or exposes it in a CI pipeline</li><li>Decide which pods get access to it</li><li>Repeat this for every service that needs to talk to Azure</li></ul><p>Azure Workload Identity solves this by eliminating the credential entirely. Instead of sharing a password, your Kubernetes cluster and Azure AD establish a cryptographic trust. They agree to accept each other’s identity proofs without ever exchanging a secret.</p><p>To understand how this works, we need to understand the mechanism it is built on: Kubernetes service account tokens.</p><h3>Part 1: Kubernetes Service Accounts and How Tokens Work</h3><h3>What Is a Service Account?</h3><p>In Kubernetes, every process running inside the cluster needs an identity. For application pods, Kubernetes provides <strong>service accounts</strong>, a namespaced resource that represents the identity of a pod.</p><p>When you create a pod, Kubernetes assigns it a service account (the default one unless you specify otherwise). That service account, combined with <strong>Roles</strong>, <strong>ClusterRoles</strong>, and <strong>RoleBindings</strong>, forms Kubernetes&#39; built-in access control (RBAC). If you want a pod to be able to list Secrets in a namespace or read ConfigMaps, you grant that permission to the service account.</p><p>This RBAC system only controls access to Kubernetes resources. It has nothing to do with Azure, yet. That distinction is important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BESfx0KtwZIVHoq9eQM-Mg.png" /><figcaption>Illustration on how pods, service accounts, role bindings and roles are linked</figcaption></figure><h3>How Kubernetes Issues Tokens</h3><p>When a pod is scheduled, it needs a way to prove its service account identity when making requests to the Kubernetes API. That proof is a <strong>JWT</strong> (JSON Web Token), a cryptographically signed string that says “this token was issued for service account X in namespace Y by this cluster.”</p><p>Modern Kubernetes generates these tokens using the <strong>TokenRequest API</strong>, which is part of the kube-apiserver. The process:</p><ol><li><strong>Kubelet</strong> (the agent running on each node) requests a token from the TokenRequest API on behalf of the new pod.</li><li><strong>kube-apiserver</strong> issues a JWT, signed with the cluster’s private key. The token encodes: which service account it is for, which specific pod it is bound to, when it expires, and what audience it is valid for.</li><li><strong>Kubelet</strong> mounts the token into the pod’s filesystem as a <strong>Projected Volume</strong>, typically at /var/run/secrets/kubernetes.io/serviceaccount/token.</li></ol><p>Three properties make these tokens different from a stored password:</p><ul><li><strong>Pod-bound</strong>: the token is tied to a specific pod. Even if someone intercepts it after that pod terminates, it is useless.</li><li><strong>Time-limited</strong>: tokens expire (default: 1 hour). Kubelet automatically refreshes the token when it reaches 80% of its lifetime. The application never has to handle rotation.</li><li><strong>Audience-scoped</strong>: a token issued for the Kubernetes API server will not be accepted by a different system unless you specifically configure it to.</li></ul><p>When an application inside a pod makes a Kubernetes API call, the SDK reads this token from the file, attaches it as a Bearer token in the request header, and sends it. The kube-apiserver verifies the signature using its own public key, checks the expiry, and evaluates RBAC to decide whether the request is allowed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dM89GLLGX5HxCwuK0sVEXg.png" /><figcaption>Illustration on how k8s cluster tokens are handled</figcaption></figure><h3>The OIDC Issuer: The Cluster’s Public Identity</h3><p>Here is a piece that is easy to overlook but is the foundation of everything that follows.</p><p>Every Kubernetes cluster has an <strong>OIDC issuer URL</strong>. At this URL, the cluster publishes two things:</p><ul><li>An <strong>OIDC discovery document</strong> at /.well-known/openid-configuration</li><li>A <strong>JWKS endpoint</strong> (JSON Web Key Set) that contains the cluster’s <strong>public key</strong></li></ul><p>Because tokens are signed with the cluster’s private key, anyone who has the corresponding public key can verify that a token was genuinely issued by this cluster, without any shared password and without contacting the cluster directly.</p><p>Think of it like a government that issues passports. The government is the issuer, the authority behind every document. Each passport carries a cryptographic seal, a signature stamped on the document using the government’s private key. The government also publishes the specification of that seal publicly, so that border control officers anywhere in the world can verify a passport by checking the seal against the published spec. They do not need to call the government for every passport. In this analogy, the OIDC issuer is the government, the token’s signature is the seal, and the JWKS endpoint is the publicly available specification that lets anyone verify it.</p><p>This is what <strong>federation</strong> means in practice: two separate systems agree to trust each other’s identity proofs without sharing a password. One system says “I will accept any token signed by that cluster, because I can verify the signature using its public key.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RUwqGfukN2BXCREOvd1lA.png" /><figcaption>Illustration on how the k8s issued token is exchanged to the Azure AD token</figcaption></figure><h3>Part 2: Azure Workload Identity</h3><h3>The Core Idea</h3><p>Kubernetes already issues cryptographically signed, pod-bound, auto-rotating tokens. Azure Workload Identity takes those tokens and uses them as the credential for accessing Azure resources, no stored secrets needed.</p><p>The flow in one sentence: your pod uses its Kubernetes token to prove its identity to Azure AD, and Azure AD exchanges that for an Azure access token the pod can use to call Azure services.</p><h3>Prerequisites</h3><p>Two flags need to be enabled when creating your AKS cluster:</p><pre>az aks create \<br>  --enable-oidc-issuer \<br>  --enable-workload-identity \<br>  ...</pre><p>--enable-oidc-issuer makes the cluster publish its OIDC discovery endpoint so Azure AD can reach it.<br>--enable-workload-identity installs a component called the mutating admission webhook (explained below).</p><p>You also need a <strong>User-Assigned Managed Identity (UMID)</strong> in Azure, an identity resource in Azure AD that your workload will act as. Think of it as an “account” in Azure that belongs to your service.</p><h3>Setting It Up: The Three Pieces</h3><p>Three things need to be created and connected before a pod can use Workload Identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LupCV8TEF0DAZ5RfAMz6pw.png" /><figcaption>Illustration on how service account, UMID and federated identity credential linked</figcaption></figure><p><strong>1. Create a User-Assigned Managed Identity</strong></p><pre>az identity create --name my-workload-identity --resource-group my-rg</pre><p><strong>2. Create a Kubernetes Service Account, annotated with the UMID’s client ID</strong></p><pre>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: my-service-account<br>  namespace: my-namespace<br>  annotations:<br>    azure.workload.identity/client-id: &quot;&lt;UMID client ID&gt;&quot;</pre><p><strong>3. Create a Federated Identity Credential on the UMID</strong></p><p>This is the trust declaration. You are telling Azure AD: “Trust tokens from this cluster, issued for this service account, and exchange them for Azure AD tokens.”</p><p>It has exactly three fields:</p><p>FieldValueWhat it means<strong>Issuer</strong>AKS OIDC issuer URLWhich cluster’s tokens to accept<strong>Subject</strong>system:serviceaccount:&lt;namespace&gt;:&lt;sa-name&gt;Which service account specifically<strong>Audience</strong>api://AzureADTokenExchangeMarks the token as intended for exchange</p><pre>az identity federated-credential create \<br>  --name my-federated-cred \<br>  --identity-name my-workload-identity \<br>  --resource-group my-rg \<br>  --issuer &quot;&lt;AKS OIDC issuer URL&gt;&quot; \<br>  --subject &quot;system:serviceaccount:my-namespace:my-service-account&quot; \<br>  --audience &quot;api://AzureADTokenExchange&quot;</pre><p><strong>4. Assign Azure RBAC roles to the UMID</strong></p><p>The UMID needs permission to access specific Azure resources. Roles are assigned at the resource level, for example granting “Key Vault Secrets User” on a specific Key Vault:</p><pre>az role assignment create \<br>  --assignee &quot;&lt;UMID client ID&gt;&quot; \<br>  --role &quot;Key Vault Secrets User&quot; \<br>  --scope &quot;/subscriptions/.../resourceGroups/.../providers/Microsoft.KeyVault/vaults/my-vault&quot;</pre><p>When a workload exchanges its Kubernetes token for an Azure AD access token, that token carries the UMID’s identity. Every time the workload makes a request to an Azure resource, the resource checks the access token against its RBAC assignments for the UMID. If the role is there, the request is allowed. If a role is added or removed, Azure RBAC can take up to 30 minutes to propagate the change, and the workload will only see it once it acquires a fresh Azure AD token (tokens have a randomized lifetime between 60 and 90 minutes).</p><p><strong>5. Deploy your pod with the workload identity label</strong></p><pre>spec:<br>  serviceAccountName: my-service-account<br>  labels:<br>    azure.workload.identity/use: &quot;true&quot;</pre><h3>The Webhook: How the Right Token Gets Into the Pod</h3><p>You might wonder: if you are using a regular Kubernetes service account token, how does Azure AD end up with a token it can exchange? The default service account token is scoped to the Kubernetes API server, not to api://AzureADTokenExchange.</p><p>This is where the <strong>Mutating Admission Webhook</strong> comes in. When a pod is created with the azure.workload.identity/use: &quot;true&quot; label and a service account annotated with a client-id, the webhook intercepts the pod creation request and automatically injects:</p><ul><li>A <strong>projected volume</strong> containing a service account token issued with audience api://AzureADTokenExchange</li><li>Three <strong>environment variables</strong>: AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_FEDERATED_TOKEN_FILE (the path to that token file)</li></ul><p>The Azure SDK reads these environment variables automatically when you use DefaultAzureCredential. You do not write any of this yourself; the webhook handles it transparently.</p><h3>Token Exchange: The Runtime Flow</h3><p>This is what happens every time your application needs to access an Azure resource:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lBViQX75AGOu81NPoJqCWA.png" /><figcaption>Illustration on how token exchange flow happens</figcaption></figure><ol><li><strong>Kubelet</strong> requests a projected token from the TokenRequest API with audience api://AzureADTokenExchange, and mounts it into the pod at the path referenced by AZURE_FEDERATED_TOKEN_FILE.</li><li>The <strong>application</strong> calls an Azure SDK function, for example “get this secret from Key Vault.”</li><li>The <strong>Azure SDK</strong> reads the Kubernetes token from AZURE_FEDERATED_TOKEN_FILE.</li><li>The <strong>SDK</strong> sends the Kubernetes token to Azure AD’s token endpoint as a client_assertion, essentially saying &quot;here is my cryptographic proof of identity, please give me an Azure access token.&quot;</li><li><strong>Azure AD</strong> looks up the federated identity credential for this client ID and retrieves the OIDC issuer URL of the cluster.</li><li><strong>Azure AD</strong> fetches the public key (JWKS) from the cluster’s OIDC endpoint.</li><li><strong>Azure AD</strong> cryptographically verifies the Kubernetes token’s signature using that public key. It also checks that the issuer, subject, and audience in the token match the federated identity credential exactly.</li><li><strong>Azure AD</strong> issues a short-lived Azure access token scoped to the requested resource.</li><li>The <strong>SDK</strong> uses that Azure access token to call Key Vault. Key Vault verifies the token and checks that the UMID has the required role. If yes, it returns the secret.</li></ol><p>No stored secret was involved at any point. The Kubernetes token was created fresh when the pod started, and Azure AD verified it using publicly available cryptographic information.</p><h3>A Note on Service-to-Service Calls</h3><p>The flow above describes accessing an Azure-managed resource like Key Vault. If Service A wants to call your own Service B, whether B is inside the cluster or elsewhere, the same mechanism works as long as Service B is registered as an Azure AD application and Service A requests a token scoped to B’s App ID URI. The exchange flow is identical; only the target resource changes.</p><h3>Part 3: Why Not Client Credentials?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_6mLGsVg9fRnJA9IW9LLWw.png" /><figcaption>Illustration on the comparison between client credentials and workload identity</figcaption></figure><p>The <strong>Client Credentials Grant</strong> is the traditional approach. A service has a client_idand a client_secret (or a certificate). It sends both to Azure AD and gets an access token back.</p><p>It works. But the secret has to live somewhere, and in a Kubernetes environment, that usually means a Kubernetes Secret object. That Secret needs to be created safely, rotated periodically, restricted to the right pods, and audited for accidental exposure. Each of those steps is operational overhead, and each is an opportunity for something to go wrong.</p><p>Here is how the two approaches compare:</p><p>Client CredentialsWorkload IdentityCredential to manageclient_secret or certNoneSecret rotationManual or automatedAutomatic (kubelet)Identity granularityApp-levelPod-level (per service account)Blast radius on credential leakEntire applicationSingle service accountWorks outside AKSYesYes (any k8s with OIDC)</p><p>With client credentials, you own the full lifecycle of the secret: creating it, distributing it securely, rotating it on schedule, and making sure it never leaks through a log file or a misconfigured CI variable. With Workload Identity, there is no secret. The trust is cryptographic, each service account maps to exactly one UMID with its own role assignments, and every token exchange is logged in Azure AD’s sign-in logs against a specific service account identity. When something goes wrong, you have a real audit trail.</p><h3>Conclusion</h3><p>Coming into that zero-trust AKS environment, the thing that made Workload Identity click for me was realising it is not a new concept bolted on top of Kubernetes. It is Kubernetes’ own token infrastructure extended to talk to Azure. The cluster already knew how to issue signed, short-lived, pod-bound tokens. Azure Workload Identity just wired up a way for Azure AD to trust those tokens using the same OIDC federation that already powers browser-based single sign-on.</p><p>This pattern is not unique to Azure, AWS has the same thing under the name IRSA (IAM Roles for Service Accounts) and GCP calls it Workload Identity Federation, but the underlying mechanism is identical across all three: a Kubernetes-issued OIDC token exchanged for a cloud provider token, no stored secrets, same trust-through-cryptography model.</p><p><strong>Further reading</strong>: <a href="https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster">AKS Workload Identity quickstart</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a95b9d96d486" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Backend Is Lying to You: The Hidden Cost of Missing Request Timeouts]]></title>
            <link>https://medium.com/@isurucuma/your-backend-is-lying-to-you-the-hidden-cost-of-missing-request-timeouts-cfdd0b0ff66d?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfdd0b0ff66d</guid>
            <category><![CDATA[failure]]></category>
            <category><![CDATA[resilience]]></category>
            <category><![CDATA[time-out]]></category>
            <category><![CDATA[client-server-connection]]></category>
            <category><![CDATA[backend-engineering]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 22:58:46 GMT</pubDate>
            <atom:updated>2026-04-04T22:58:46.094Z</atom:updated>
            <content:encoded><![CDATA[<h4>How one slow downstream turns silent waiting into backlog, restarts, and whole-service failure</h4><h3>It’s 2 AM. Your Phone Is Ringing.</h3><p>You’re half asleep. You grab your phone. It’s a PagerDuty alert. Checkout is down. You open your laptop, squint at the dashboards, and start doing the usual triage.</p><blockquote>CPU? Fine. Memory? Fine. Database? Looks normal.</blockquote><p>But requests are just… not coming back. Users are staring at spinning loaders. Your logs show incoming traffic. Your dashboards show requests entering the system. But responses? Missing. No crash. No panic. No obvious 5xx spike. Just silence.</p><p>You keep digging and trace it back to a downstream dependency. Maybe it’s a payment gateway. Maybe it’s an internal profile service. Maybe it’s a third-party API. It started getting slow hours ago. Slow became very slow. Very slow became effectively never.</p><p>And your service? It just kept waiting. Patiently. Forever. Until it ran out of the ability to do useful work for anyone else.</p><p>The fix, once you find it, is almost insulting in how small it is: a timeout you never set.</p><p>This article is about that timeout. What it actually means, what breaks when you skip it, and how to think about it clearly in production systems.</p><p><a href="https://github.com/isurucuma/production-article-series-timeouts">This repository holds the code relevant to this demo, feel free to explore it if you are interested</a></p><h3>What Even Is a Request Timeout?</h3><p>Whenever one service calls another over HTTP, there is a simple contract:</p><ol><li>open a connection,</li><li>send the request,</li><li>wait for the response.</li></ol><p>A <strong>request timeout</strong> is the boundary that says: <em>“I am willing to wait, but not indefinitely.”</em></p><p>The trap is that many engineers think of “timeout” as one single number. In practice, a request has multiple phases, and each phase can fail differently.</p><h3>1. Connection Timeout</h3><p>This is how long you’re willing to wait to establish the network connection in the first place.</p><p>If DNS is wrong, a firewall is blocking traffic, the host is unreachable, or the network path is unhealthy, this is where you get stuck.</p><p><strong>Think of it as:</strong> dialing someone’s number and waiting for it to ring.</p><p>Typical starting range: <strong>250ms to 1s</strong>.</p><h3>2. Response Timeout</h3><p>This is how long you’re willing to wait for the downstream system to actually start responding after your request has been sent.</p><p>This is the <strong>dangerous</strong> one, because the connection already worked. The other service “picked up the phone.” But now it is not saying anything back. Maybe it’s stuck on a slow query. Maybe it’s waiting on its own dependency. Maybe it’s overloaded and barely making progress.</p><p>From your side, all you see is a request that is still open and not making forward progress.</p><p><strong>Think of it as:</strong> the other person answered, but all you hear is silence.</p><p>Typical starting range: <strong>based on the downstream’s real latency</strong>, not guesswork.</p><h3>3. Write Timeout</h3><p>This is how long you’re willing to spend sending the request body. Mostly for POST, PUT, and PATCH type requests.</p><p>For small API calls, this usually is not the main problem. But if you upload files, send large payloads, or stream data to another service, write timeouts matter.</p><p><strong>Think of it as:</strong> you’re trying to talk, but the other side stopped listening.</p><h3>4. Overall Deadline</h3><p>This is the outer boundary for the full request lifecycle.</p><p>Even if you configure the individual phases, you still usually want one total budget for the whole operation.</p><p>This is the line that says:</p><blockquote><em>“No matter what happens underneath, this call is not allowed to exceed X.”</em></blockquote><p>Here’s the request timeouts in one picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvSLuci1RJ-xavsrrL3WgQ.png" /><figcaption>request timeout illustration</figcaption></figure><p>Different stacks expose these knobs differently, but the underlying problem is the same everywhere.</p><h3>What Actually Happens When You Skip Timeouts?</h3><p>This is where most explanations stop too early. They say “set a timeout” and move on. But if you want to build reliable systems, you need to understand the blast radius of not doing it.</p><h3>Requests Stay Alive Longer Than You Think</h3><p>When your service handles a request and calls something downstream, it has to hold on to resources while it waits:</p><ul><li>Memory,</li><li>A request handler,</li><li>A worker thread or coroutine or goroutine,</li><li>A connection from a pool,</li><li>Buffers,</li><li>Queue slots,</li><li>Sometimes locks or other in-process state.</li></ul><p>The implementation details differ by language and framework, but the pattern is universal: <strong>a request that never finishes keeps consuming capacity.</strong></p><blockquote>One stuck request is annoying.</blockquote><blockquote>One hundred stuck requests is an incident.</blockquote><blockquote>One thousand stuck requests is how an otherwise healthy service starts looking dead from the outside.</blockquote><h3>The Damage Spreads Sideways</h3><p>The really nasty part is that the endpoint making the slow downstream call usually isn’t the only thing that suffers.</p><p>As capacity gets tied up:</p><ul><li>Unrelated endpoints become slow,</li><li>Health checks get delayed,</li><li>Background jobs stop making progress,</li><li>Connection pools fill up,</li><li>Queue depths rise,</li><li>Latency spikes across the whole service.</li></ul><p>From the outside, it looks like <em>your entire service</em> is broken.</p><p>But the real problem may be a single downstream dependency that is simply taking too long to respond.</p><h3>Cascading Failure Starts Small</h3><p>Imagine a chain like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zn5CKYrvUz2fulMNxn9yFg.png" /><figcaption>failure cascading illustration</figcaption></figure><p>If the external gateway slows down:</p><ul><li>Payments starts accumulating stuck requests</li><li>Orders starts waiting on Payments</li><li>API starts waiting on Orders</li><li>User-facing latency explodes</li><li>Retries kick in and make everything worse</li></ul><p>One bad dependency can poison every service upstream of it.</p><p>This is a <strong>cascading failure</strong>, and missing or overly generous timeouts are one of the easiest ways to create one.</p><h3>Silent Failures Are the Worst Failures</h3><p>This is what makes timeout bugs so deceptive.</p><p>If your service is crashing, you get stack traces.</p><p>If your service is returning 500s, your error dashboards light up.</p><p>But if your service is simply waiting forever, you may get almost nothing:</p><ul><li>No explicit application error,</li><li>No panic,</li><li>No crash,</li><li>Just open requests that never finish.</li></ul><p>Your system can be deeply unhealthy while the usual “error rate” signals still look strangely calm.</p><p>A real timeout changes that. It turns an invisible hang into a visible failure:</p><ul><li>Clear error in logs</li><li>Timeout metric increment</li><li>Request that ends instead of lingering forever</li><li>Downstream dependency you can now identify and investigate</li></ul><blockquote>Loud failures are fixable.</blockquote><blockquote>Silent hangs are how you lose an hour in the middle of the night.</blockquote><h3>The Demo: Let’s Break Something</h3><p>Theory matters, but this becomes much easier to understand when you watch it happen.</p><p>For the demo, I deployed three things:</p><ul><li><strong>Service A</strong>, the caller service receiving client traffic</li><li><strong>Service B</strong>, the downstream dependency that Service A calls on every request</li><li>An observability stack with Prometheus and Grafana</li></ul><p>I then drove the system with k6 under sustained load and ran the same failure scenario twice:</p><ul><li>Once with <strong>no timeout</strong> in Service A</li><li>Once with a <strong>3 second outbound timeout</strong> in Service A</li></ul><p>The architecture is simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aR-G0mHu6CbAsPJ7rM0gBQ.png" /><figcaption>demo setup illustration</figcaption></figure><p>The point of the experiment is simple too: compare what happens when a caller waits forever versus when it fails fast and preserves its own capacity.</p><h3>Without a Timeout, One Bad Endpoint Becomes a Whole-Service Problem</h3><p>In the unbounded run, Service A keeps waiting on Service B no matter how long Service B takes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QtCcPAZ7vp6YHi549u3HRg.png" /><figcaption>unbounded scenario grafana dashbaord</figcaption></figure><p>The Grafana dashboard tells the story immediately:</p><ul><li>Service A success rate fell to about The 31.2%</li><li>request throughput collapsed to about 144 req/s</li><li>In-flight requests climbed to about The 2.12k</li><li>average latency rose into roughly the 5s to 9s range</li><li>Goroutines climbed into roughly 13k to 15k</li><li>Heap usage rose sharply</li></ul><p>The most important detail is in the goroutine and heap panels: the service-a metrics disappear in the middle and then come back again.</p><p>That is a strong sign that Service A was restarted repeatedly under memory pressure during the run. In other words, this is no longer just a slow endpoint. The service itself is falling over.</p><p>If this were a real backend with many routes, the impact would not stay isolated to the one endpoint calling the bad dependency. While Service A is down or restarting, unrelated endpoints fail too. Health checks fail. Lightweight routes fail. Everything behind that process is affected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*M_WBlPBdZfYEH0tgR91KxQ.png" /><figcaption>k6 load testing tool (client) results for unbounded scenario</figcaption></figure><p>The k6 result shows the same failure from the client side. Only about 43% of requests got any response back at all. The rest failed somewhere in the middle because the service became unavailable or unresponsive before a response could be returned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fpSG1o9ixlqPbUnxzrrq9g.png" /><figcaption>service A got out of memory and restarted in the middle</figcaption></figure><p><em>Unbounded run: rising in-flight requests, rising latency, much higher goroutine and heap growth, and missing </em><em>service-a metric segments that indicate restarts under pressure.</em></p><p><em>Unbounded </em><em>k6 result: only about </em><em>43% of requests received any response at all.</em></p><h3>With a Timeout, The Endpoint Still Fails but the Service Holds Together</h3><p>In the bounded run, the downstream is still unhealthy. The difference is that Service A refuses to wait forever.</p><p>That changes the shape of the failure completely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XiqgQCLJIZdstyRhW-bxcw.png" /><figcaption>bounded scenario grafana dashbaord</figcaption></figure><p>The Grafana dashboard shows:</p><ul><li>Service A success rate at about 23.0%</li><li>Request throughput around The 668 req/s</li><li>average latency staying around 2.68s</li><li>Goroutines staying much flatter, around roughly 8k to 9k instead of spiking toward 15k</li><li>Lower and more stable heap usage than the unbounded run</li><li>Continuous service-a metric lines instead of disappearing and reappearing</li></ul><p>That does <strong>not</strong> mean the endpoint is healthy. In fact, this one endpoint can fail more often once you add the timeout, because some slow downstream responses that would have eventually limped through are now cut off at the timeout boundary.</p><p>That is the right trade.</p><p>The goal is not to squeeze a few extra successes out of one poisoned endpoint. The goal is to stop that endpoint from consuming so much memory and concurrency that the whole service becomes unstable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*Pyyk0VQx1HUpBmdvHk--CQ.png" /><figcaption>k6 load testing tool (client) results for bounded scenario</figcaption></figure><p>The k6 result makes that difference very clear. In the bounded run, <strong>all</strong> requests reached the client. Many of them were failures, but they were explicit failures. The service stayed responsive enough to return timeouts instead of disappearing mid-request.</p><p>That is what resilience looks like in practice:</p><ul><li>One endpoint may return timeout responses</li><li>The process stays up</li><li>Unrelated endpoints have a far better chance of staying healthy</li><li>The failure stays contained instead of spreading across the whole service</li></ul><p><em>Bounded run: near-zero in-flight buildup, continuous </em><em>service-a metrics, and much lower resource growth even while the endpoint is failing.</em></p><p><em>Bounded </em><em>k6 result: all requests reached the client, but many were explicit timeout failures rather than silent hangs or mid-flight collapses.</em></p><h3>What the Comparison Actually Proves</h3><p>Side by side, the difference is not subtle.</p><ul><li><strong>Unbounded:</strong> fewer client-visible completions, backlog growth, restart behavior, and a real risk of whole-service failure</li><li><strong>Bounded:</strong> more explicit failures on the affected endpoint, but lower internal buildup, a stable process, and much better service-level resilience</li></ul><p>That is the real job of a timeout.</p><blockquote>It does not guarantee success for the failing endpoint.</blockquote><blockquote>It prevents one failing dependency from taking the whole service with it.</blockquote><h3>How To Choose a Timeout</h3><p>There is no universal magic number, but there is a practical way to reason about one.</p><p>Start with real latency data from the dependency:</p><ul><li>Look at p95 and p99 under load</li><li>Choose a timeout that is informed by those numbers, not by vibes</li><li>A useful starting point is often around 2x the downstream p99, adjusted by your own budget</li></ul><p>Then work backward from your own SLA.</p><p>If your service is supposed to answer in 1s, you cannot casually hand one downstream call a 3s timeout and still claim you have a 1s API. Every downstream timeout has to fit inside the total time budget for the request.</p><p>Finally, tune with production feedback.</p><p>If healthy traffic is timing out, your timeout may be too aggressive.</p><p>If the system still spends far too long waiting before giving up, your timeout may be too generous.</p><h3>A Few Rules That Matter in Production</h3><p><strong>Timeouts and retries are different tools.</strong> A timeout tells you when to stop waiting. A retry answers whether you should try again. Retries without care can turn a struggling dependency into a worse outage, so use backoff, jitter, and limits.</p><p><strong>Timeouts on writes need idempotency.</strong> If a write times out, you may not know whether the downstream actually completed it. That is how duplicate charges, duplicate emails, and duplicate orders happen.</p><p><strong>Defaults are often dangerous.</strong> Never assume your HTTP client ships with safe production defaults. Set timeouts explicitly.</p><p><strong>Cancellation should propagate.</strong> If the upstream request has already timed out or the caller is gone, the system should stop spending resources on work that no longer matters.</p><h3>Quick Reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Pd2pUt7gw46mHijcOSGoQ.png" /><figcaption>quick reference for setting timeouts</figcaption></figure><h3>The Point</h3><blockquote>Distributed systems rarely fail in a clean, obvious way.</blockquote><blockquote>They get slow. They stall. They hang. They keep just enough life in them to tempt upstream services into waiting far too long.</blockquote><blockquote>Timeouts turn that silence into a boundary.</blockquote><blockquote>They turn indefinite waiting into a controlled failure.</blockquote><blockquote>They do not make bad dependencies healthy.</blockquote><blockquote>They stop bad dependencies from making your service unhealthy too.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfdd0b0ff66d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Containers Are Just Linux Processes With Better Boundaries]]></title>
            <link>https://medium.com/codex/containers-are-just-linux-processes-with-better-boundaries-465661f89d58?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/465661f89d58</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[containerization]]></category>
            <category><![CDATA[chroot]]></category>
            <category><![CDATA[namespaces]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Mon, 09 Feb 2026 19:44:30 GMT</pubDate>
            <atom:updated>2026-02-19T06:21:55.823Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building a mini-Docker from scratch with Linux namespaces and chroot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W8FtE0c4IvbQgkBBZqmJ9A.png" /><figcaption>header image how containers work</figcaption></figure><p>When I first started using Docker, I thought containers were some kind of lightweight virtual machines. Tiny computers inside my computer. Magic boxes that somehow ran my code in isolation.</p><p>I was wrong.</p><p>And if you think the same way, you’re in good company. Most developers start with docker run commands, not Linux internals. That&#39;s why containers can feel like magic or, worse, like something too complex to truly understand.</p><p>But here’s the truth that changed everything for me:</p><blockquote>A container is just a regular Linux process.</blockquote><p>That’s it. Not a mini-VM. Not a separate operating system. Just a process like any other program running on your computer but with some clever boundaries drawn around it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2X4jTQAFd79oy59N8ChP3A.png" /><figcaption>Image explaining how a linux process with access limited looks like</figcaption></figure><p>In this article, I’ll show you exactly how this works by building three tiny programs in Go. Each one will peel back another layer of the container mystery:</p><ol><li>First, we’ll launch a process with no isolation at all</li><li>Then, we’ll add namespace isolation (where things start to feel “container-like”)</li><li>Finally, we’ll add filesystem isolation using Alpine Linux file system</li></ol><p>If you have a Linux machine, Go installed, and sudo access, you can run every single line of code yourself. No Docker required.</p><h3>The Lightbulb Moment: Containers vs Virtual Machines</h3><p>Before we dive into code, let’s clear up the biggest misconception.</p><p>Here’s what makes virtual machines and containers fundamentally different:</p><p><strong>Virtual Machines:</strong></p><ul><li>Emulate an entire computer, including fake hardware</li><li>Run a complete guest operating system with its own kernel</li><li>Heavy they take time to boot and consume significant memory</li><li>Isolated through a hypervisor (a strong, thick boundary)</li></ul><p><strong>Containers:</strong></p><ul><li>Reuse the host’s Linux kernel</li><li>No separate kernel or operating system</li><li>Lightweight they start in milliseconds and use minimal memory</li><li>Isolated through Linux kernel features (a clever, thin boundary)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDeMpSuanlpYPQ-G3HtR5Q.png" /><figcaption>vms vs containers</figcaption></figure><p>Think of it this way: A virtual machine is like building a house inside a warehouse. A container is like drawing rooms on the warehouse floor with tape. Both give you separated spaces, but one is far lighter than the other.</p><p>The key insight:</p><blockquote>A container is not a special machine type. It’s a process tree with controlled visibility and privileges.</blockquote><p>That single sentence is worth reading twice.</p><h3>What You’ll Need</h3><p>To follow along(better if you can, but not a must), you’ll need:</p><ul><li>A Linux machine (Ubuntu, Debian, Fedora, or similar)</li><li>Go installed (any recent version works)</li><li>sudo access (we&#39;ll need it for namespace and chroot operations)</li></ul><p>Please clone this repo for all the code files, scripts: <a href="https://github.com/isurucuma/linux-containerization">https://github.com/isurucuma/linux-containerization</a></p><p>Don’t worry if you’ve never written Go before. The code is simple, and I’ll explain every important line.</p><h3>Step 1: Running a Process With No Isolation</h3><p>Let’s start with the simplest possible version: launching a process without any container features at all.</p><p>Here’s our first program:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;os/exec&quot;<br>)<br><br>func main() {<br> if len(os.Args) &lt; 2 {<br>  panic(&quot;usage: run &lt;cmd&gt; [args...] | child &lt;cmd&gt; [args...]&quot;)<br> }<br><br> switch os.Args[1] {<br> case &quot;run&quot;:<br>  run()<br> case &quot;child&quot;:<br>  child()<br> default:<br>  panic(&quot;unknown command&quot;)<br> }<br>}<br><br>func run() {<br> if len(os.Args) &lt; 3 {<br>  panic(&quot;usage: run &lt;cmd&gt; [args...]&quot;)<br> }<br><br> fmt.Printf(&quot;[parent] cmd=%v pid=%d\n&quot;, os.Args[2:], os.Getpid())<br><br> args := append([]string{&quot;child&quot;}, os.Args[2:]...)<br> cmd := exec.Command(&quot;/proc/self/exe&quot;, args...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}<br><br>func child() {<br> if len(os.Args) &lt; 3 {<br>  panic(&quot;usage: child &lt;cmd&gt; [args...]&quot;)<br> }<br><br> command := os.Args[2]<br> commandArgs := os.Args[3:]<br><br> fmt.Printf(&quot;[child] cmd=%v pid=%d\n&quot;, os.Args[2:], os.Getpid())<br><br> cmd := exec.Command(command, commandArgs...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}</pre><p>Save this as without-constraints/main.go and run it:</p><pre>go build -o bin/without-constraints ./cmd/without-constraints<br>./bin/without-contraints run /bin/sh</pre><p>You’ll see output like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*EUi843sRyhBi6s-wAqH-Mw.png" /><figcaption>without-constraints program output</figcaption></figure><h3>What’s Actually Happening Here?</h3><p>This program does something that might seem odd at first: it runs itself twice.</p><ol><li>You call it with run mode</li><li>It immediately launches itself again in child mode</li><li>The child mode then runs your actual command (/bin/sh)</li></ol><p>“Why the two-step dance?” you might ask. Great question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCguLDwK8EABkIK70iLbgA.png" /><figcaption>image illustrating how the parent process runs itself as another chile process</figcaption></figure><p>This re-execution pattern is the foundation of how container runtimes work. When we add isolation features in the next steps, those features need to be applied when <strong>creating a new process</strong>, not when calling a function. The parent process sets up the isolation flags, then spawns a child process that lives inside those boundaries.</p><h3>The Magic of /proc/self/exe</h3><p>Look at this line closely:</p><pre>args := append([]string{&quot;child&quot;}, os.Args[2:]...)<br>cmd := exec.Command(&quot;/proc/self/exe&quot;, args...)</pre><p>/proc/self/exe is a special file that points to the currently running program. It&#39;s like a mirror: when you look at it, you see yourself.</p><p>So when we do exec.Command(&quot;/proc/self/exe&quot;, ...), we&#39;re telling Linux: &quot;Run this same program again, but with different arguments.&quot;</p><p>Why not just call child() directly as a function? Because namespace isolation (which we&#39;ll add next) requires a brand new process. You can&#39;t &quot;become isolated&quot; mid-execution you have to be born isolated.</p><h3>What to Notice</h3><p>Right now, there’s <strong>zero isolation</strong>:</p><ul><li>Both PIDs are visible on the host</li><li>Both processes share the same hostname</li><li>Both see the same filesystem</li><li>Both see the same list of running processes</li></ul><p>This is just regular process launching. But it’s the skeleton we’ll build on.</p><h3>Understanding Namespaces: The Core Container Trick</h3><p>Before we write more code, you need to understand what namespaces actually are.</p><p>Imagine you’re in a busy office building. You can see everyone, hear every conversation, and access every room. Now imagine someone gives you special glasses that only show you people on your floor. And earplugs that only let through sounds from your department. And a key card that only opens certain doors.</p><p>You’re still in the same building, using the same electricity and plumbing. But your <strong>view</strong> of the building is now scoped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77KeYfIkz_QRSMF4VmOFGg.png" /></figure><p>That’s what Linux namespaces do.</p><p>A namespace gives a process a <strong>private view</strong> of some system resource. The kernel is shared (the building), but each process can have its own perspective (the filtered view).</p><p>Here are the three namespaces we’ll use:</p><ul><li><strong>UTS</strong> Hostname and domain name. Your container can be named “web-server” even if the host is named “laptop-001”.</li><li><strong>PID</strong> Process ID numbering Inside the container, your app appears to be PID 1 (like it’s the only thing running).</li><li><strong>Mount</strong> Filesystem mount table You can mount/unmount filesystems without affecting the host</li></ul><p><strong>Why this matters:</strong></p><ul><li>Without UTS isolation, your container would show the host’s hostname (information leakage)</li><li>Without PID isolation, running ps would show every process on the host (no privacy)</li><li>Without mount isolation, your mount operations could mess with the host filesystem (dangerous!)</li></ul><p>These boundaries aren’t absolute security barriers they’re visibility controls. But they provide most of the separation people expect from containers.</p><h3>Step 2: Adding Namespace Isolation</h3><p>Now let’s add the actual “container” part. We’ll modify our program to create isolated namespaces.</p><p>Here’s the updated code:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;os/exec&quot;<br> &quot;syscall&quot;<br>)<br><br>func main() {<br> if len(os.Args) &lt; 2 {<br>  panic(&quot;usage: run &lt;cmd&gt; [args...] | child &lt;cmd&gt; [args...]&quot;)<br> }<br><br> switch os.Args[1] {<br> case &quot;run&quot;:<br>  run()<br> case &quot;child&quot;:<br>  child()<br> default:<br>  panic(&quot;unknown command&quot;)<br> }<br>}<br><br>func run() {<br> if len(os.Args) &lt; 3 {<br>  panic(&quot;usage: run &lt;cmd&gt; [args...]&quot;)<br> }<br><br> fmt.Printf(&quot;[parent] cmd=%v pid=%d\n&quot;, os.Args[2:], os.Getpid())<br><br> args := append([]string{&quot;child&quot;}, os.Args[2:]...)<br> cmd := exec.Command(&quot;/proc/self/exe&quot;, args...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br> cmd.SysProcAttr = &amp;syscall.SysProcAttr{<br>  Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS,<br> }<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}<br><br>func child() {<br> if len(os.Args) &lt; 3 {<br>  panic(&quot;usage: child &lt;cmd&gt; [args...]&quot;)<br> }<br><br> command := os.Args[2]<br> commandArgs := os.Args[3:]<br><br> fmt.Printf(&quot;[child] cmd=%v pid=%d\n&quot;, os.Args[2:], os.Getpid())<br><br> cmd := exec.Command(command, commandArgs...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}</pre><p>Save this as namespaces/main.go and run it:</p><pre>go build -o bin/namespaces ./cmd/namespaces<br>sudo ./bin/namespaces run /bin/sh</pre><p>(Note: You need sudo because creating new namespaces requires elevated privileges.)</p><p>I will change the hostname inside the container</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*N4wBHSa3CliYzDc4aYQfnw.png" /><figcaption>changing hostname inside the container</figcaption></figure><p>But lets spawn a new terminal and see the host’s hostname</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*Y0dzOHQHGGO8bCAhb0Ivog.png" /><figcaption>check the hotname from the host</figcaption></figure><p>As you can see the host’s hostname is not changed, that itself explains how namespaces work</p><h3>The One Line That Changes Everything</h3><p>The magic happens right here:</p><pre>cmd.SysProcAttr = &amp;syscall.SysProcAttr{<br>	Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS,<br>}</pre><p>This tells the Linux kernel: “When you create this new process, put it in brand new UTS, PID, and mount namespaces.”</p><p>Those flags are:</p><ul><li>CLONE_NEWUTS: Give this process its own hostname view</li><li>CLONE_NEWPID: Give this process its own PID numbering (it becomes PID 1 in its own universe)</li><li>CLONE_NEWNS: Give this process its own mount table view</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOgpVzb5NOkaaNZrdmKNFw.png" /></figure><h3>What You’ll See</h3><p>When you run the command, you might see something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*N4wBHSa3CliYzDc4aYQfnw.png" /></figure><p><strong>Notice anything wild?</strong> The child process thinks it’s PID 1!</p><p>From the child’s perspective, it’s the first process in the entire system (like the init process that starts when Linux boots). But from the host&#39;s perspective, it&#39;s just process 218xx among thousands.</p><p>This is the container illusion in action.</p><h3>What’s Still Missing?</h3><p>Even with namespaces, the process is still using the host’s root filesystem. If you run ls /, you&#39;ll see the host&#39;s directories. If you run cat /etc/hostname, you&#39;ll see the host&#39;s hostname.</p><p>Namespaces gave us isolation, but we haven’t given our container its own filesystem yet.</p><p>That’s the next step.</p><h3>Step 3: Adding Filesystem Isolation</h3><p>Now we’re going to do something that will make this feel like a “real” container: give it its own root filesystem.</p><p>We’ll use Alpine Linux rootfs(root file system) a tiny Linux distribution that’s only about 5MB that contains everything needed for a minimal Linux environment.</p><h3>Getting Alpine Minirootfs</h3><p>First, use the following shell script file to download the Alpine’s mini root filesystem: <a href="https://raw.githubusercontent.com/isurucuma/linux-containerization/refs/heads/main/setup-chroot.sh">https://raw.githubusercontent.com/isurucuma/linux-containerization/refs/heads/main/setup-chroot.sh</a></p><pre>./setup-chroot.sh</pre><p>You now have a complete (minimal) Linux filesystem sitting in a folder. It has /bin, /etc, /lib, and all the usual directories just without your host&#39;s specific files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OAFVOo4MmdGt4x0AhcA4A.png" /></figure><h3>The Code</h3><p>Here’s our final version:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;os/exec&quot;<br> &quot;syscall&quot;<br>)<br><br>func main() {<br> if len(os.Args) &lt; 2 {<br>  panic(&quot;usage: run &lt;rootfs&gt; &lt;cmd&gt; [args...] | child &lt;rootfs&gt; &lt;cmd&gt; [args...]&quot;)<br> }<br><br> switch os.Args[1] {<br> case &quot;run&quot;:<br>  run()<br> case &quot;child&quot;:<br>  child()<br> default:<br>  panic(&quot;unknown command&quot;)<br> }<br>}<br><br>func run() {<br> if len(os.Args) &lt; 4 {<br>  panic(&quot;usage: run &lt;rootfs&gt; &lt;cmd&gt; [args...]&quot;)<br> }<br><br> rootfs := os.Args[2]<br> fmt.Printf(&quot;[parent] rootfs=%s cmd=%v pid=%d\n&quot;, rootfs, os.Args[3:], os.Getpid())<br><br> args := append([]string{&quot;child&quot;, rootfs}, os.Args[3:]...)<br> cmd := exec.Command(&quot;/proc/self/exe&quot;, args...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br> cmd.SysProcAttr = &amp;syscall.SysProcAttr{<br>  Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS,<br> }<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}<br><br>func child() {<br> if len(os.Args) &lt; 4 {<br>  panic(&quot;usage: child &lt;rootfs&gt; &lt;cmd&gt; [args...]&quot;)<br> }<br><br> rootfs := os.Args[2]<br> command := os.Args[3]<br> commandArgs := os.Args[4:]<br><br> fmt.Printf(&quot;[child] rootfs=%s cmd=%v pid=%d\n&quot;, rootfs, os.Args[3:], os.Getpid())<br><br> if err := syscall.Sethostname([]byte(&quot;alpine-container&quot;)); err != nil {<br>  panic(err)<br> }<br><br> if err := syscall.Mount(&quot;&quot;, &quot;/&quot;, &quot;&quot;, uintptr(syscall.MS_PRIVATE|syscall.MS_REC), &quot;&quot;); err != nil {<br>  panic(err)<br> }<br><br> if err := syscall.Chroot(rootfs); err != nil {<br>  panic(err)<br> }<br><br> if err := os.Chdir(&quot;/&quot;); err != nil {<br>  panic(err)<br> }<br><br> if err := os.MkdirAll(&quot;/proc&quot;, 0o755); err != nil {<br>  panic(err)<br> }<br><br> if err := syscall.Mount(&quot;proc&quot;, &quot;/proc&quot;, &quot;proc&quot;, 0, &quot;&quot;); err != nil {<br>  panic(err)<br> }<br> defer syscall.Unmount(&quot;/proc&quot;, 0)<br><br> cmd := exec.Command(command, commandArgs...)<br> cmd.Stdin = os.Stdin<br> cmd.Stdout = os.Stdout<br> cmd.Stderr = os.Stderr<br><br> if err := cmd.Run(); err != nil {<br>  panic(err)<br> }<br>}</pre><p>Save this as chroot/main.go.</p><h3>Running Your First “Real” Container</h3><p>Try this:</p><pre>go build -o bin/chroot ./cmd/chroot<br>sudo go run chroot/main.go run ./chroot /bin/sh</pre><p>You’ll get a shell prompt. Now type:</p><pre>hostname</pre><p>You’ll see alpine-containernot your host&#39;s hostname.</p><p>Try this:</p><pre>ls /</pre><p>You’ll see Alpine’s filesystem, not your host’s.</p><p>Try this:</p><pre>ps</pre><p>You’ll only see processes in your container’s PID namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgwzvE_PebcUXoL22l62iA.png" /></figure><blockquote><strong>You just built a container from scratch.</strong></blockquote><h3>Breaking Down the Critical Lines</h3><p>Let’s walk through what’s happening in the child() function:</p><p><strong>1. Set a custom hostname:</strong></p><pre>syscall.Sethostname([]byte(&quot;alpine-container&quot;))</pre><p>This proves our UTS namespace is working. We can change the hostname without affecting the host.</p><p><strong>2. Prevent mount propagation:</strong></p><pre>syscall.Mount(&quot;&quot;, &quot;/&quot;, &quot;&quot;, uintptr(syscall.MS_PRIVATE|syscall.MS_REC), &quot;&quot;)</pre><p>This tells Linux: “Any mounts I do from now on should stay private to me.” Without this, mount operations could leak to the host system.</p><p><strong>3. Change the root filesystem:</strong></p><pre>syscall.Chroot(rootfs)</pre><p>chroot means &quot;change root.&quot; After this call, / no longer points to the host&#39;s root directory it points to our Alpine directory. It&#39;s like swapping out the entire filesystem while the process is running.</p><p><strong>4. Move into the new root:</strong></p><pre>os.Chdir(&quot;/&quot;)</pre><p>Even after changing the root, our current working directory might still reference the old filesystem. This moves us fully into the new context.</p><p><strong>5. Mount </strong><strong>/proc:</strong></p><pre>os.MkdirAll(&quot;/proc&quot;, 0o755)<br>syscall.Mount(&quot;proc&quot;, &quot;/proc&quot;, &quot;proc&quot;, 0, &quot;&quot;)</pre><p>/proc is a special virtual filesystem that shows information about processes. We need to mount it inside our new root so commands like ps work correctly.</p><p>The defer statement ensures we clean up the mount when the container exits.</p><h3>The Full Picture: What We’ve Built</h3><p>Let’s recap what our tiny container runtime does:</p><ol><li><strong>Creates a new process</strong> with isolated namespaces (UTS, PID, mount)</li><li><strong>Changes the root filesystem</strong> to a minimal Alpine Linux environment</li><li><strong>Sets up a custom hostname</strong> to prove isolation</li><li><strong>Mounts </strong><strong>/proc</strong> so process inspection works</li><li><strong>Runs your command</strong> inside this isolated context</li></ol><p>This is genuinely how containers work under the hood. The core ideas are identical to Docker, just without all the production features.</p><h3>So What Does Docker Actually Add?</h3><p>You might be wondering: “If I just built a container in 100 lines of Go, what is Docker doing with its millions of lines of code?”</p><p>Great question. Docker adds all the stuff you need to run containers in production:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4AVKjkvPky9WM0uoJUp5Q.png" /></figure><p>Think of it this way:</p><ul><li><strong>Linux kernel</strong> provides the raw primitives (namespaces, cgroups, chroot)</li><li><strong>OCI runtimes</strong> (like runc) turn those primitives into a standard container format</li><li><strong>Docker</strong> wraps everything in beautiful developer UX and production features</li></ul><p>But the core truth remains: <strong>containers are just processes with boundaries.</strong></p><h3>Why This Matters</h3><p>Understanding this changes how you work with containers:</p><p><strong>When debugging:</strong></p><ul><li>Instead of thinking “the container is broken,” you can think “which isolation boundary is causing this issue?”</li><li>You can use tools like nsenter to peek into namespaces</li><li>You understand why mounting volumes is just mount namespace manipulation</li></ul><p><strong>When securing:</strong></p><ul><li>You know containers share the kernel (so a kernel vulnerability affects all containers)</li><li>You understand why rootless containers need user namespaces</li><li>You can reason about what capabilities and seccomp profiles actually control</li></ul><p><strong>When optimizing:</strong></p><ul><li>You understand why containers start fast (no OS boot, just process fork)</li><li>You know why they’re light (shared kernel, no hardware emulation)</li><li>You can make informed decisions about when to use containers vs VMs</li></ul><h3>The One Thing to Remember</h3><p>If you take away only one thing from this article, let it be this:</p><blockquote><em>Containers are not mini virtual machines. They are Linux processes with constrained visibility and privilege.</em></blockquote><p>When you run docker run, you are not booting a new operating system. You&#39;re not starting a virtual machine. You&#39;re launching a process that the Linux kernel has placed inside carefully drawn boundaries.</p><p>The magic isn’t in the container technology itself it’s in how cleverly Linux lets processes have their own view of the system while sharing the same kernel.</p><h3>Going Further</h3><p>If this sparked your curiosity and you want to keep learning, here are some natural next steps:</p><ol><li><strong>Add cgroups</strong> to limit memory, CPU, and PID usage</li><li><strong>Explore user namespaces</strong> for rootless containers (running containers without sudo)</li><li><strong>Add network namespaces</strong> and virtual ethernet pairs for isolated networking</li><li><strong>Replace chroot with pivot_root</strong> (the production-grade way to change roots)</li><li><strong>Read the OCI runtime specification</strong> to see how the industry standardized all of this</li></ol><p>But honestly, if you’ve made it this far and run the code yourself, you already understand containers at a deeper level than most developers who use them daily.</p><p>And that understanding will serve you well.</p><p><strong>Ready to try it yourself?</strong> All the code from this article is available to run. Fire up a Linux terminal and start with Step 1. By Step 3, you’ll have built your own container runtime from scratch.</p><p>Use this repo: <a href="https://github.com/isurucuma/linux-containerization">https://github.com/isurucuma/linux-containerization</a></p><p>No magic required. Just Linux, doing what it does best.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=465661f89d58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/containers-are-just-linux-processes-with-better-boundaries-465661f89d58">Containers Are Just Linux Processes With Better Boundaries</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Build Dynamic Authorization Systems by Embracing Google’s Zanzibar]]></title>
            <link>https://medium.com/codex/lets-build-dynamic-authorization-systems-by-embracing-google-s-zanzibar-4a4aa9e1581f?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a4aa9e1581f</guid>
            <category><![CDATA[zanzibar]]></category>
            <category><![CDATA[openfga]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[fine-grained]]></category>
            <category><![CDATA[access-control]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 16:18:28 GMT</pubDate>
            <atom:updated>2025-08-11T09:36:11.003Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>“There has to be a better way…”</em> Every developer who’s ever built permissions</h4><h3>The Story Begins: Meet Sarah and Her Authorization Nightmare</h3><p>Sarah stares at her screen, coffee growing cold. As a senior developer at TechCorp, she’s been tasked with building document permissions for their new collaboration platform. Simple enough, right?</p><p>“Let’s just use roles,” she thinks. “Editor, Viewer, Admin. Done.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1eFYAH6RjsEHOjyJTSxBg.png" /><figcaption>Sarah planning the roles</figcaption></figure><p>But then reality hits.</p><p>Alice from Marketing needs to edit Document A, but only view Document B. Bob from Engineering should access his team’s docs, but not HR files. The simple three-role system becomes docA_editor, docB_editor, docC_viewer…</p><p>Fast forward two weeks: 1000 documents have exploded into 2000+ roles. User assignments are a nightmare. Sarah’s elegant solution has become an unmaintainable mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aaSEAEINEHEW9qyPCnh8EA.png" /><figcaption>Sarah’s desktop cluttered with roles</figcaption></figure><p>Sound familiar? You’re not alone.</p><h3>The Problem with Traditional Role-Based Access Control (RBAC)</h3><h3>Why RBAC Falls Short in Modern Applications</h3><p>RBAC worked great when applications were simpler. But today’s dynamic, resource-heavy applications expose its fundamental flaws:</p><p><strong>The Static Problem</strong>: Roles are fixed, but resources are created constantly. Every new document, project, or team potentially needs new roles.</p><p><strong>The Granularity Trap</strong>: Make roles too broad (“editor for everything”) and you compromise security. Make them too specific and you get role explosion.</p><p><strong>The Scalability Nightmare</strong>: 10 teams × 100 documents × 3 permission levels = 3000 potential roles. Good luck managing that.</p><h3>The Role Explosion Problem</h3><p>Here’s what Sarah faced:</p><ul><li>Initial plan: 3 roles</li><li>Reality check: 2000+ roles</li><li>Management overhead: Who has what role? Which roles are orphaned?</li><li>Performance impact: Checking hundreds of roles per request</li><li>Developer sanity: Rapidly approaching zero</li></ul><p>The real kicker? Should a Marketing team member edit HR documents just because they’re both “editors”? RBAC can’t distinguish between resources naturally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmOM3GtmCjBxKYWhlZVdXg.png" /></figure><p>Sarah realized she needed something fundamentally different. Something that understands relationships, not just roles.</p><h3>Enter Dynamic Authorization: The Hero We Need</h3><h3>What Makes Authorization “Dynamic”?</h3><p>Instead of asking “What role does Alice have?”, dynamic authorization asks “What’s Alice’s relationship to this document?”</p><p>This shift changes everything:</p><p><strong>Relationship-based</strong>: Permissions flow through natural connections. Alice can edit documents she owns. Bob can edit documents his team owns.</p><p><strong>Resource-aware</strong>: Each document, project, or file can have its own access rules without creating new roles.</p><p><strong>Hierarchical</strong>: Your organizational structure drives permissions naturally. If Engineering belongs to Product department, permissions can flow upward.</p><p><strong>Contextual</strong>: “Alice can edit documents she created” is fundamentally different from “Alice can edit all documents.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W-GyElzIbZY4X0bLpgZyDQ.png" /></figure><h3>The Zanzibar Approach: Google’s Solution</h3><p>Google faced the ultimate authorization challenge:</p><ul><li>YouTube: Billions of videos, millions of creators</li><li>Drive: Trillions of files, complex sharing</li><li>Calendar: Intricate meeting permissions</li></ul><p>Their insight? <strong>Model permissions as relationships between entities.</strong></p><p>Instead of assigning roles, you define relationships:</p><ul><li>Alice <strong>owns</strong> Document-A</li><li>Bob <strong>belongs to</strong> Engineering-Team</li><li>Engineering-Team <strong>can edit</strong> Document-A</li><li>Therefore: Bob <strong>can edit</strong> Document-A</li></ul><p>The magic? Complex permissions emerge from simple relationships. No role explosion. No manual assignments. Just natural, scalable authorization.</p><p>This is why it’s called “fine-grained”: you get per-resource permissions without the management overhead.</p><h3>Our Story: Building Document Management Authorization</h3><p>Let’s make this concrete with a story that unfolds step by step.</p><h3>Meet Our Characters</h3><ul><li><strong>Alice</strong>: A product manager who creates documents</li><li><strong>Bob</strong>: A developer who joins the Engineering team</li><li><strong>Engineering Team</strong>: Bob’s team that needs access to certain documents</li><li><strong>Product Department</strong>: The department that Engineering belongs to</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*B8Tl217vRv-tqUbL4dlI2Q.png" /></figure><h3>The Challenge</h3><p>We want to build a system where:</p><ul><li>Users create documents and become their owners</li><li>Teams can be granted access to specific documents</li><li>New team members automatically get team permissions</li><li>No manual role assignments needed</li></ul><h3>The Story Unfolds: Real Scenarios</h3><p>Let’s watch our authorization system come to life through five key moments:</p><blockquote><strong>Scene 1</strong>: Alice creates doc-001 and becomes its owner</blockquote><blockquote><strong>Scene 2</strong>: Alice grants the Engineering team edit access to doc-001</blockquote><blockquote><strong>Scene 3</strong>: Bob joins the Engineering team</blockquote><blockquote><strong>Scene 4</strong>: Engineering team becomes part of the Product department</blockquote><blockquote><strong>Scene 5</strong>: Product department members gets edit access to doc-002</blockquote><p>Now here’s the magic question: After these events, can Bob edit doc-001? Can Bob edit doc-002?</p><p>The answer should be yes to both — but how does the system know this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SfqH-bYb9lL31lUlKpEuWw.png" /></figure><h3>The Magic Table: How Relationships Are Stored</h3><p>Every relationship in Zanzibar can be imagined as a single row in what’s essentially a giant table. Think of it as the “phonebook” of permissions.</p><h4>Tuple Examples</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q5hBTYPyDyND1aDkwyvUqg.png" /></figure><h4>Here’s what the table columns mean</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*OWXq3PD44Y_gnA3pmDnJjA.png" /></figure><blockquote><strong>Important note</strong>: This is a simplified view for understanding. The actual Zanzibar implementation is more complex, but this table captures the core concept.</blockquote><h3>Walking Through the Examples</h3><p><strong>Row 1: Direct Ownership</strong></p><pre>user:alice → document:doc-001#owner</pre><p><strong>Translation</strong>: “Alice owns document doc-001”, an owner relationship</p><p><strong>Row 2: Team Permission</strong></p><pre>team:engineering#member → document:doc-001#editor</pre><p><strong>Translation</strong>: “Members of the team engineering can edit document doc-001”</p><p><strong>Row 3: Team Membership</strong></p><pre>user:bob → team:engineering#member</pre><p><strong>Translation</strong>: “Bob is a member of the team engineering”</p><h3>🎯 The Magic Happens Here!</h3><p>Now, when we ask “Can Bob edit doc-001?”, Zanzibar follows the chain:</p><ol><li>Is Bob directly allowed? No direct relationship found</li><li>What groups is Bob in? Bob is a member of team:engineering</li><li>What can team:engineering#member do? They can edit document:doc-001</li><li><strong>Result</strong>: Bob can edit doc-001! ✅</li></ol><p><strong>Row 4: Team-Department Relationship</strong></p><pre>team:engineering → department:product#team</pre><p><strong>Translation</strong>: “Team engineering belongs to the product department.” In other words, the engineering team has a team relationship with the product department</p><p><strong>Row 5: Department-Level Permission</strong></p><pre>department:product#member → document:doc-002#editor</pre><p><strong>Translation</strong>: “Members of the department product can edit document doc-002”. Members of the product department have an editor relationship with the doc-002.</p><h3>🎯 Cascading Permissions!</h3><p>Now, when we ask “Can Bob edit doc-002?”, the chain becomes:</p><ol><li>Is Bob directly allowed? No</li><li>What groups is Bob in? Bob → team:engineering#member</li><li>What department does his team belong to? team:engineering → department:product#team</li><li>What can department:product#member do? They can edit document:doc-002</li><li><strong>But wait!</strong> Bob is a team:engineering#member, not a department:product#member</li><li><strong>The computed relationship</strong>: Since Bob is a member of a team that belongs to the department, Bob becomes a department:product#member automatically</li><li><strong>Result</strong>: Bob can edit doc-002! ✅</li></ol><h3>The Fine-Grained Authorization Model: Defining the Rules</h3><p>Before we can store relationships, we need to define what relationships are possible. Here’s our authorization model:</p><pre>type user<br><br>type team<br>  relations<br>    define member: [user]<br>    define department: [department]<br><br>type department<br>  relations<br>    define member: member from team<br>    define team: [team]<br><br>type document<br>  relations<br>    define owner: [user]<br>    define editor: [user, team#member, department#member] or owner</pre><h4>This model defines:</h4><ul><li><strong>Users can be team members</strong>: Simple, direct relationship</li><li><strong>Teams belong to departments</strong>: Teams can have a department relationship</li><li><strong>Department members are inherited from team members</strong>: The magic member from team creates computed relationships</li><li><strong>Documents can have owners and editors</strong>: Including team/department members through the or owner union</li></ul><p>Here’s the key insight: When we say define member: member from team in the department type, we&#39;re telling OpenFGA that anyone who is a member of a team that belongs to this department automatically becomes a department member too.</p><p>This inheritance is what makes the “computed relationship” in step 6 above possible. Bob becomes a department:product#member not through direct assignment, but because he&#39;s a member of a team that belongs to the department.</p><p>The model is the blueprint that makes complex permission inheritance feel natural and automatic.</p><h3>The Graph Emerges: Visualizing Relationships</h3><h3>From Table to Graph</h3><p>While the table shows us the raw data, the real magic becomes clear when we visualize these relationships as a graph:</p><ul><li><strong>Nodes</strong>: Our entities (users, teams, departments, documents)</li><li><strong>Edges</strong>: The relationships between them</li><li><strong>Paths</strong>: How permissions flow through the organization</li></ul><p>Here’s what our story looks like as a graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFRVRhI_kYieNQVcKvWYkg.png" /><figcaption>Relationship diagram</figcaption></figure><h3>The DAG Property: Why This Works</h3><p>This graph has a special property: it’s a <strong>Directed Acyclic Graph (DAG)</strong>:</p><ul><li><strong>Directed</strong>: Relationships have direction (Bob is a member of Engineering, not the other way around)</li><li><strong>Acyclic</strong>: No circular dependencies (prevents infinite loops during permission checks)</li></ul><p>This DAG structure is crucial because:</p><ol><li><strong>Performance</strong>: We can traverse paths efficiently without getting stuck in loops</li><li><strong>Reasoning</strong>: We can clearly trace how permissions flow</li><li><strong>Consistency</strong>: No contradictory permission chains</li></ol><h3>Permission Inheritance in Action</h3><p>Let’s trace how permissions flow through our graph:</p><p><strong>Direct Permissions</strong>:</p><ul><li>Alice → doc-001 (owner)</li><li>Bob → Engineering (member)</li><li>Engineering → Product (team)</li></ul><p><strong>Inherited Permissions</strong>:</p><ul><li>Bob → doc-001 (through Engineering membership)</li><li>Bob → doc-002 (through Engineering → Product chain)</li></ul><p><strong>The Beautiful Result</strong>: Bob gets access to both documents without any direct assignment. The organizational structure drives the permissions naturally.</p><h3>Graph Traversal: How Zanzibar Thinks</h3><p>When asked “Can Bob edit doc-001?”, Zanzibar performs a graph traversal:</p><ol><li><strong>Start</strong>: Look for direct Bob → doc-001 relationship</li><li><strong>Expand</strong>: Find all groups Bob belongs to (Engineering)</li><li><strong>Check</strong>: Can Engineering members edit doc-001? Yes!</li><li><strong>Result</strong>: Permission granted ✅</li></ol><p>For “Can Bob edit doc-002?”:</p><ol><li><strong>Start</strong>: Look for direct Bob → doc-002 relationship</li><li><strong>Expand</strong>: Find Bob’s groups (Engineering)</li><li><strong>Expand further</strong>: Find Engineering’s parent groups (Product)</li><li><strong>Check</strong>: Can Product members edit doc-002? Yes!</li><li><strong>Result</strong>: Permission granted ✅</li></ol><p>The graph structure makes complex permission inheritance feel natural and predictable.</p><h3>OpenFGA: Making Zanzibar Accessible</h3><h3>What is OpenFGA?</h3><p>Google’s Zanzibar paper was brilliant, but implementing it from scratch? That’s a massive undertaking. But don’t need to worry since we have the OpenFGA, the open-source implementation that makes Zanzibar’s power accessible to everyone.</p><h3>From Zanzibar Concepts to OpenFGA Implementation</h3><p>Let’s connect the dots between what we’ve learned and how OpenFGA actually works:</p><p><strong>Our Table Rows</strong> → <strong>OpenFGA Tuples</strong></p><pre>// Row 3 from our table becomes:<br>client.ClientTupleKey{<br>    User:     &quot;user:bob&quot;,<br>    Relation: &quot;member&quot;,<br>    Object:   &quot;team:engineering&quot;,<br>}</pre><p><strong>Our Authorization Model</strong> → <strong>OpenFGA Schema</strong></p><p><strong>Our Graph Traversal</strong> → <strong>OpenFGA APIs</strong></p><pre>// &quot;Can Bob edit doc-001?&quot; becomes:<br>response, err := fgaClient.Check(context.Background()).<br>    Body(client.ClientCheckRequest{<br>        User:     &quot;user:bob&quot;,<br>        Relation: &quot;editor&quot;,<br>        Object:   &quot;document:doc-001&quot;,<br>    }).Execute()</pre><h3>The Two Key APIs You Need to Know</h3><p><strong>1. Check API</strong>: “Can user X do Y on resource Z?”</p><pre>// Is Bob allowed to edit doc-001?<br>response, err := fgaClient.Check(context.Background()).<br>    Body(client.ClientCheckRequest{<br>        User:     &quot;user:bob&quot;,<br>        Relation: &quot;editor&quot;,<br>        Object:   &quot;document:doc-001&quot;,<br>    }).Execute()<br>if err != nil {<br>    // Handle error<br>}<br>// Returns: response.Allowed = true</pre><p><strong>2. ListObjects API</strong>: “What can user X access?”</p><pre>// What documents can Bob edit?<br>response, err := fgaClient.ListObjects(context.Background()).<br>    Body(client.ClientListObjectsRequest{<br>        User:     &quot;user:bob&quot;,<br>        Relation: &quot;editor&quot;,<br>        Type:     &quot;document&quot;,<br>    }).Execute()<br>if err != nil {<br>    // Handle error<br>}<br>// Returns: response.Objects = [&quot;document:doc-001&quot;, &quot;document:doc-002&quot;]</pre><h3>Real Implementation: Code Walkthrough</h3><h3>Setting Up OpenFGA</h3><p>Let’s get our hands dirty with a real implementation. The quickest way to get OpenFGA running locally is by using Docker Compose.</p><p><strong>Quick Setup with Docker Compose</strong>:</p><pre># Download the official docker-compose.yaml<br>curl -O https://raw.githubusercontent.com/openfga/openfga/main/docker-compose.yaml<br><br># Start OpenFGA with PostgreSQL<br>docker-compose up -d</pre><p>This will start OpenFGA on http://localhost:8080 with a PostgreSQL database backend.</p><p>For more installation options and detailed setup instructions, check the <a href="https://github.com/openfga/openfga/blob/main/README.md">official OpenFGA documentation</a>. <strong>Initialize the Go Client</strong>:</p><p><strong>💡 Want to follow along?</strong> <a href="https://github.com/isurucuma/fga-tutorial">Explore this comprehensive tutorial repository, featuring a working Docker Compose setup and Go code examples that demonstrate every concept discussed in this article.</a></p><pre>package main<br><br>import (<br> &quot;context&quot;<br> &quot;encoding/json&quot;<br> &quot;fmt&quot;<br> openfga &quot;github.com/openfga/go-sdk&quot;<br><br> . &quot;github.com/openfga/go-sdk/client&quot;<br>)<br><br>// for the sake of simplicity, I have not handled errors in this example.<br>func main() {<br> // Initialize OpenFGA client<br> fgaClient, err := NewSdkClient(&amp;ClientConfiguration{<br>  ApiUrl: &quot;http://localhost:8080&quot;,<br> })<br> if err != nil {<br>  panic(fmt.Sprintf(&quot;Error initializing OpenFGA client: %v&quot;, err))<br> }<br><br> // Create a new store<br> resp, err := fgaClient.CreateStore(context.Background()).<br>  Body(ClientCreateStoreRequest{<br>   Name: &quot;Document Management System&quot;,<br>  }).Execute()<br><br> // Store the store ID for future use<br> storeId := resp.Id<br><br> // here we need to update the client configuration with the new store ID<br> err = fgaClient.SetStoreId(storeId)</pre><h3>Loading Our Authorization Model</h3><p>Let’s create our authorization model that supports our document management scenario:</p><pre>// Configure our authorization model<br> var authModelString = `{<br>  &quot;schema_version&quot;: &quot;1.1&quot;,<br>  &quot;type_definitions&quot;: [<br>    {<br>      &quot;type&quot;: &quot;user&quot;,<br>      &quot;relations&quot;: {},<br>      &quot;metadata&quot;: null<br>    },<br>    {<br>      &quot;type&quot;: &quot;team&quot;,<br>      &quot;relations&quot;: {<br>        &quot;member&quot;: {<br>          &quot;this&quot;: {}<br>        },<br>        &quot;department&quot;: {<br>          &quot;this&quot;: {}<br>        }<br>      },<br>      &quot;metadata&quot;: {<br>        &quot;relations&quot;: {<br>          &quot;member&quot;: {<br>            &quot;directly_related_user_types&quot;: [<br>              {<br>                &quot;type&quot;: &quot;user&quot;<br>              }<br>            ]<br>          },<br>          &quot;department&quot;: {<br>            &quot;directly_related_user_types&quot;: [<br>              {<br>                &quot;type&quot;: &quot;department&quot;<br>              }<br>            ]<br>          }<br>        }<br>      }<br>    },<br>    {<br>      &quot;type&quot;: &quot;department&quot;,<br>      &quot;relations&quot;: {<br>        &quot;member&quot;: {<br>          &quot;tupleToUserset&quot;: {<br>            &quot;computedUserset&quot;: {<br>              &quot;relation&quot;: &quot;member&quot;<br>            },<br>            &quot;tupleset&quot;: {<br>              &quot;relation&quot;: &quot;team&quot;<br>            }<br>          }<br>        },<br>        &quot;team&quot;: {<br>          &quot;this&quot;: {}<br>        }<br>      },<br>      &quot;metadata&quot;: {<br>        &quot;relations&quot;: {<br>          &quot;member&quot;: {<br>            &quot;directly_related_user_types&quot;: []<br>          },<br>          &quot;team&quot;: {<br>            &quot;directly_related_user_types&quot;: [<br>              {<br>                &quot;type&quot;: &quot;team&quot;<br>              }<br>            ]<br>          }<br>        }<br>      }<br>    },<br>    {<br>      &quot;type&quot;: &quot;document&quot;,<br>      &quot;relations&quot;: {<br>        &quot;owner&quot;: {<br>          &quot;this&quot;: {}<br>        },<br>        &quot;editor&quot;: {<br>          &quot;union&quot;: {<br>            &quot;child&quot;: [<br>              {<br>                &quot;this&quot;: {}<br>              },<br>              {<br>                &quot;computedUserset&quot;: {<br>                  &quot;relation&quot;: &quot;owner&quot;<br>                }<br>              }<br>            ]<br>          }<br>        }<br>      },<br>      &quot;metadata&quot;: {<br>        &quot;relations&quot;: {<br>          &quot;owner&quot;: {<br>            &quot;directly_related_user_types&quot;: [<br>              {<br>                &quot;type&quot;: &quot;user&quot;<br>              }<br>            ]<br>          },<br>          &quot;editor&quot;: {<br>            &quot;directly_related_user_types&quot;: [<br>              {<br>                &quot;type&quot;: &quot;user&quot;<br>              },<br>              {<br>                &quot;type&quot;: &quot;team&quot;,<br>                &quot;relation&quot;: &quot;member&quot;<br>              },<br>              {<br>                &quot;type&quot;: &quot;department&quot;,<br>                &quot;relation&quot;: &quot;member&quot;<br>              }<br>            ]<br>          }<br>        }<br>      }<br>    }<br>  ]<br>}`<br><br> var body openfga.WriteAuthorizationModelRequest<br> err = json.Unmarshal([]byte(authModelString), &amp;body)<br><br> data, err := fgaClient.WriteAuthorizationModel(context.Background()).<br>  Body(body).<br>  Execute()<br><br> // Save the model ID for future use<br> modelId := data.AuthorizationModelId<br><br> // Update the client configuration with the new model ID<br> err = fgaClient.SetAuthorizationModelId(modelId)</pre><h3>Creating Our Story with Code</h3><p>Now let’s recreate our five scenes using actual API calls:</p><p><strong>Scene 1: Alice creates doc-001</strong></p><pre>scenarioTuples := ClientWriteRequest{<br>  Writes: []ClientTupleKey{<br>   {<br>    User:     &quot;user:alice&quot;,<br>    Relation: &quot;owner&quot;,<br>    Object:   &quot;document:doc-001&quot;,<br>   },<br>  },<br> }<br> _, err = fgaClient.Write(context.Background()).<br>  Body(scenarioTuples).<br>  Options(options).<br>  Execute()</pre><p><strong>Scene 2 &amp; 3: Engineering team gets access, Bob joins</strong></p><pre>scenarioTuples := ClientWriteRequest{<br>  Writes: []ClientTupleKey{<br>   {<br>    User:     &quot;team:engineering#member&quot;,<br>    Relation: &quot;editor&quot;,<br>    Object:   &quot;document:doc-001&quot;,<br>   },<br>   {<br>    User:     &quot;user:bob&quot;,<br>    Relation: &quot;member&quot;,<br>    Object:   &quot;team:engineering&quot;,<br>   },<br>  },<br> }<br> _, err = fgaClient.Write(context.Background()).<br>  Body(scenarioTuples).<br>  Options(options).<br>  Execute()</pre><p><strong>Scenes 4 &amp; 5: Department structure and permissions</strong></p><pre>scenarioTuples := ClientWriteRequest{<br>  Writes: []ClientTupleKey{<br>   {<br>    User:     &quot;team:engineering&quot;,<br>    Relation: &quot;team&quot;,<br>    Object:   &quot;department:product&quot;,<br>   },<br>   {<br>    User:     &quot;department:product#member&quot;,<br>    Relation: &quot;editor&quot;,<br>    Object:   &quot;document:doc-002&quot;,<br>   },<br>  },<br> }<br> _, err = fgaClient.Write(context.Background()).<br>  Body(scenarioTuples).<br>  Options(options).<br>  Execute()</pre><h3>The Magic in Action</h3><p>Now let’s test our authorization logic:</p><pre>// Can Bob edit doc-001?<br> checkOptions := ClientCheckOptions{<br>  AuthorizationModelId: &amp;modelId,<br> }<br><br> checkBody := ClientCheckRequest{<br>  User:     &quot;user:bob&quot;,<br>  Relation: &quot;editor&quot;,<br>  Object:   &quot;document:doc-001&quot;,<br> }<br><br> checkData, err := fgaClient.Check(context.Background()).<br>  Body(checkBody).<br>  Options(checkOptions).<br>  Execute()<br> fmt.Printf(&quot;Can Bob edit doc-001? %v\n&quot;, *checkData.Allowed) // true<br><br> // Can Bob edit doc-002?<br> checkBody.Object = &quot;document:doc-002&quot;<br> checkData, err = fgaClient.Check(context.Background()).<br>  Body(checkBody).<br>  Options(checkOptions).<br>  Execute()<br> fmt.Printf(&quot;Can Bob edit doc-002? %v\n&quot;, *checkData.Allowed) // true</pre><p>Here is an illustration on how OpenFGA evaluates a permission check request</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*0fCZSuVzfdNeT_c-yf-D4g.png" /><figcaption>Illustration on how OpenFGA traverse through knowledge graph</figcaption></figure><pre>// Can Alice edit doc-001?<br>checkBody.User = &quot;user:alice&quot;<br>checkBody.Object = &quot;document:doc-001&quot;<br>checkData, err = fgaClient.Check(context.Background()).<br>    Body(checkBody).<br>    Options(checkOptions).<br>    Execute()<br>fmt.Printf(&quot;Can Alice edit doc-001? %v\n&quot;, *checkData.Allowed) // true</pre><p><strong>🎯 The Owner Magic</strong>: Notice something interesting? Although we never explicitly granted Alice editor permissions on doc-001, she can still edit it. This is because our authorization model defines editor: [user, team#member, department#member] or owner - meaning all owners automatically inherit editor permissions. No additional tuples needed!</p><h3>The End of Our Story: From Nightmare to Dream</h3><h3>Sarah’s Transformation</h3><p>Remember Sarah from our opening? Let’s check in on her now.</p><p>The 2000+ roles cluttering her database are gone. The nested if-statements scattered throughout her codebase are gone. The sleepless nights wondering if the new intern accidentally gave Marketing access to HR documents are gone.</p><p>Sarah’s authorization system now handles complex scenarios with elegant simplicity:</p><ul><li><strong>New document created?</strong> One relationship tuple: user:user001 → document:doc-004#owner</li><li><strong>New team member joins?</strong> One relationship tuple: user:user001 → team:engineering#member</li><li><strong>Complex permission check?</strong> One API call: fgaClient.Check()</li></ul><h3>What We’ve Accomplished</h3><p>In this journey, we’ve seen how:</p><ul><li><strong>Role explosion</strong> transforms into <strong>relationship elegance</strong></li><li><strong>Scattered authorization logic</strong> becomes <strong>centralized and declarative</strong></li><li><strong>Manual permission management</strong> evolves into <strong>automatic inheritance</strong></li><li><strong>Complex permission trees</strong> simplify into <strong>single API calls</strong></li></ul><h3>The Beautiful Result</h3><p>Sarah’s system now scales effortlessly. A thousand documents? A million? The authorization model remains the same. New organizational structures? Just update the relationships. Complex permission requirements? The graph handles it naturally.</p><p>Most importantly, Sarah can focus on what she loves, building amazing features instead of wrestling with authorization plumbing.</p><h3>Choosing the Right Path</h3><p>Not every application needs the complexity of relationship-based authorization. Like any powerful tool, it shines in the right context.</p><p><strong>Stick with traditional RBAC when:</strong></p><ul><li>You have simple, stable permission structures</li><li>Resources don’t have individual access controls</li><li>Your organizational hierarchy rarely changes</li><li>A handful of roles cover most use cases</li></ul><p><strong>Consider relationship-based authorization when you’re facing:</strong></p><ul><li><strong>Role explosion</strong>: More than a handful of roles per resource type</li><li><strong>Complex organizational structures</strong>: Teams, departments, nested hierarchies</li><li><strong>Dynamic permissions</strong>: Frequently changing access patterns</li><li><strong>Resource-level control</strong>: Different permissions per document, project, or file</li><li><strong>Scaling challenges</strong>: Traditional RBAC hitting performance or management walls</li></ul><h3>Your Implementation Options</h3><p>If relationship-based authorization fits your needs, you don’t have to build it from scratch:</p><ul><li><strong>Open-source implementations</strong>: Community-driven, customizable solutions</li><li><strong>Managed services</strong>: Cloud-hosted, enterprise-ready platforms</li><li><strong>Self-hosted solutions</strong>: Full control over your authorization infrastructure</li></ul><p><strong>Ready to experiment?</strong> This <a href="https://github.com/isurucuma/fga-tutorial">tutorial repository</a> contains everything you need to get started with Docker Compose setup, complete Go implementation, and all the examples from this article running in practice.</p><h3>The Takeaway</h3><p>Sarah’s story isn’t about everyone needing to abandon RBAC; it’s about recognizing when you’ve outgrown it.</p><p>For simple applications, basic roles work beautifully. For complex, dynamic systems with resource-specific permissions, relationship-based patterns can transform your authorization from a maintenance nightmare into an elegant solution.</p><p>The key is knowing which tool fits your problem.</p><p><strong>Choose wisely. Build confidently.</strong></p><p>More into:</p><ul><li><a href="https://openfga.dev/">Fine-Grained Authorization | OpenFGA</a></li><li><a href="https://github.com/isurucuma/fga-tutorial">GitHub - isurucuma/fga-tutorial: This is a repo created on explaining the Fine grained authorization concepts with OpenFGA, Zanzibar related</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a4aa9e1581f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/lets-build-dynamic-authorization-systems-by-embracing-google-s-zanzibar-4a4aa9e1581f">Let’s Build Dynamic Authorization Systems by Embracing Google’s Zanzibar</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Our Journey to Centralized Identity and Access Management]]></title>
            <link>https://medium.com/pickme-engineering-blog/our-journey-to-centralized-identity-management-707d16f06fbf?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/707d16f06fbf</guid>
            <category><![CDATA[pickme]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[identity-management]]></category>
            <category><![CDATA[user-management-system]]></category>
            <category><![CDATA[access-management]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Sat, 10 May 2025 10:59:36 GMT</pubDate>
            <atom:updated>2025-05-12T11:02:18.307Z</atom:updated>
            <content:encoded><![CDATA[<h4>Lessons from PickMe’s IAM Migration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mq9-ThJ06jCwSVxI" /><figcaption>Performance, reliability and maintainability</figcaption></figure><p>In the heart of Sri Lanka’s bustling digital transformation stands PickMe, the country’s homegrown ride-hailing platform connecting drivers, passengers, and merchants across the island nation. As with many rapidly scaling technology companies, our journey has been marked by continuous evolution by building systems that can grow with our user base while maintaining the security and performance our stakeholders expect.</p><p>Identity and access management (IAM) sits at the crossroads of security, user experience, and system architecture. For years, PickMe operated with domain-specific authentication systems, each tailored to the unique needs of our various user types. Drivers authenticated differently from passengers, portal administrators had their own authentication flows, and merchant partners accessed our system through yet another custom implementation. This approach served us well during our initial growth phase, allowing each domain team to move quickly and independently.</p><p>However, as our platform matured and our user base expanded, the fragmentation of our identity infrastructure began to show its limitations. What started as a pragmatic solution had evolved into a complex web of authentication services, varied token implementations, and scattered user databases. Each new feature or security enhancement required changes across multiple systems, increasing both development time and the risk of inconsistencies.</p><p>This article chronicles our technical journey from this distributed identity landscape to a centralized IAM solution built on Keycloak. It’s a story that will resonate with engineering teams facing similar challenges in growing systems and those who’ve experienced the tension between domain-specific customization and platform-wide consistency.</p><h3>The Legacy System Landscape</h3><p>At PickMe, our system evolved organically as we expanded from a simple ride-hailing service to a comprehensive platform encompassing multiple transportation options, food delivery, and merchant services. Each domain drivers, passengers, merchants, and internal operations developed its own authentication mechanisms to meet immediate business needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4my3PfEXijFk8vxs2ydPgA.png" /><figcaption>Domain specific auth services</figcaption></figure><p>This domain-centric approach meant that our user management was highly fragmented. Driver information lived in the driver domain’s database, passenger accounts in another database, and merchant users in yet another. Some domains shared common databases for users, but with different schemas and access patterns. Authentication services were equally scattered, with custom implementations handling login flows, password management, and token generation for each user type.</p><blockquote>The challenges of this fragmented approach became increasingly apparent as we scaled:</blockquote><p>First, our engineering teams had to maintain numerous custom authentication services, each with its own codebase, dependencies, and potential security vulnerabilities. When security standards evolved or vulnerabilities were discovered, we needed to update multiple systems independently.</p><p>Second, our backend services had to work with multiple token formats and validation mechanisms. Some services even needed to validate different token types depending on which domain the request originated from, leading to complex and error-prone code paths.</p><p>Finally, user management became increasingly difficult. Simple operations like finding all accounts associated with a phone number or email address required queries across multiple databases. Implementing consistent password policies or multi-factor authentication flows across all user types meant duplicating work across teams.</p><p>As our engineering organization matured, these inefficiencies became increasingly difficult to justify. The fragmented identity landscape not only slowed development but also created potential security gaps and inconsistent user experiences. We needed a more sustainable approach.</p><h3>Why Centralized IAM? The Keycloak Decision</h3><p>The decision to move to a centralized IAM solution wasn’t made lightly. Our engineering team carefully evaluated the tradeoffs between continuing with our domain-specific implementations and investing in a platform-wide solution.</p><p>Three key benefits ultimately made centralization through Keycloak the clear choice:</p><p>First, centralized user and permission management would dramatically simplify administration. Rather than navigating multiple systems to manage users, roles, and permissions, our operations team could use a single interface. This would not only save time but also reduce the risk of configuration errors and inconsistencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1bG2Llf2fN2CIfycuIBkA.png" /><figcaption>Keycloak as a central IAM</figcaption></figure><p>Second, standardizing on a single token type would simplify our backend services. Instead of handling multiple authentication mechanisms, our services could focus on a consistent approach to validation and authorization. By moving token validation to our API gateway using Keycloak’s public key, we could even remove authentication logic from many services entirely.</p><p>Third, Keycloak offered capabilities we would otherwise need to build ourselves: federated authentication, single sign-on, user self-service, and comprehensive audit logging. Leveraging these built-in features would allow us to focus our engineering resources on our core business rather than continually reinventing identity management wheels.</p><p>The open-source nature of Keycloak also aligned with our technology values. We could inspect the code, contribute back to the community, and avoid vendor lock-in. The active community around Keycloak gave us confidence that the platform would continue to evolve with best practices in identity management.</p><p>With these benefits in mind, we began planning our migration to Keycloak. Little did we know that the most interesting challenges were still ahead of us, particularly how to transition our existing user base without disrupting their experience or requiring them to create new accounts.</p><h3>The Migration Challenges</h3><p>While selecting Keycloak as our centralized IAM solution was relatively straightforward, implementing the migration presented complex technical challenges that tested our engineering team’s ingenuity. The migration needed to balance thorough architectural transformation with absolutely minimal disruption to our users, a delicate equilibrium to maintain.</p><h3>Mapping a Complex User Ecosystem</h3><p>Our first significant challenge was conceptual:</p><blockquote>how do we map our existing user taxonomy into Keycloak’s domain model?</blockquote><p>Our legacy system had evolved organically, with different domains implementing users, roles, and permissions in ways that made sense within their isolated contexts.</p><p>Keycloak’s model revolves around realms, users, roles, groups, clients, and scopes, a powerful but structured approach to identity management. We needed to carefully analyze our existing authentication services to understand how each domain defined and used identity concepts. This analysis revealed surprising inconsistencies in how similar concepts were implemented across domains.</p><p>For example, in some domains, permissions were directly assigned to users, while in others, they were assigned to roles that users inherited. Some domains had complex hierarchical role structures, while others used flat permission lists. Each of these patterns needed a thoughtful mapping to Keycloak’s capabilities.</p><p>This mapping exercise wasn’t merely a technical challenge but also an organizational one. We needed to build consensus across multiple teams about how identity would be structured going forward a process that required careful negotiation and occasional compromise.</p><h3>The Password Migration Conundrum</h3><p>Perhaps the most fundamental challenge in any authentication system migration is handling user passwords. Following security best practices, our existing systems stored only hashed versions of passwords, not the original values. This created an immediate problem: while we could migrate basic user information into Keycloak, we couldn’t transfer passwords because we didn’t have them in their original form.</p><p>We faced a difficult choice.</p><blockquote>We could force all users to reset their passwords.</blockquote><p>An approach that would be technically straightforward but would create significant friction. Alternatively, we needed to develop a mechanism that would allow users to continue using their existing credentials without knowing a migration had occurred.</p><p>This challenge was particularly significant given our business requirement that</p><blockquote>Existing users must not be logged out during the migration.</blockquote><p>A seamless transition was paramount to maintain trust and avoid disruption, especially for our drivers whose livelihoods depend on continuous platform access.</p><h3>Maintaining Session Continuity</h3><p>The business directive was clear and challenging:</p><blockquote>The migration to Keycloak could not result in users being logged out.</blockquote><p>This requirement transformed what might have been a “big bang” migration into something far more nuanced.</p><p>Consider the complexity: at any given moment, we had thousands of active users across our ecosystem, drivers navigating to their next passenger, customers tracking their rides, merchants fulfilling orders. Each of these users had active sessions with tokens issued by our legacy authentication systems. If we simply switched over to Keycloak, all these tokens would become invalid, forcing everyone to log in again.</p><p>For our passengers, this might be an inconvenience. For our drivers on the road, it could mean lost income. For merchants in the middle of preparing orders, it could result in confusion and delays. The business impact of a forced logout would be substantial and unacceptable.</p><p>We needed to develop a sophisticated transition mechanism that could honor existing sessions while gradually moving authentication responsibilities to Keycloak, essentially running both systems in parallel during a transition period.</p><h3>Extending Keycloak for OTP Authentication</h3><p>Our final major challenge involved one of our most important authentication flows: one-time password (OTP) login. Both our driver and passenger applications relied heavily on OTP-based authentication, allowing users to receive verification codes via SMS rather than maintaining traditional passwords.</p><p>While Keycloak offers impressive flexibility, its built-in capabilities weren’t sufficient to support our specific OTP scenarios. We needed customized flows that would:</p><ol><li>Initiate OTP generation and delivery through our existing SMS gateway</li><li>Validate submitted OTPs according to our security policies (including rate limiting and expiry)</li><li>Handle edge cases specific to our user base and regional context</li><li>Maintain compatibility with our mobile applications’ existing authentication UX</li></ol><p>This gap between Keycloak’s out-of-the-box capabilities and our requirements meant we needed to extend the platform significantly. We would need to develop custom authentication flows while still leveraging Keycloak’s core identity infrastructure a challenge that required a deep understanding of both Keycloak’s internals and our specific business requirements.</p><p>These challenges collectively represented significant technical hurdles, but they also offered an opportunity to reevaluate and improve our identity architecture holistically. In the next section, we’ll explore the solutions we engineered to address each of these challenges, turning potential roadblocks into opportunities for innovation.</p><h3>Engineering Solutions</h3><p>After thoroughly analyzing our challenges, we engineered solutions that not only addressed our immediate migration needs but also established a foundation for more robust identity management. Here’s how we tackled each of the major challenges we faced.</p><h3>Mapping Users, Roles, and Permissions to Keycloak’s Domain Model</h3><p>Our first task was to translate our existing identity structures into Keycloak’s conceptual framework. We conducted extensive analysis of our existing systems, documenting all user types, roles, and permission structures across domains.</p><p>We ultimately settled on a mapping that leveraged Keycloak’s full feature set:</p><ul><li><strong>Users</strong>: Migrated directly to Keycloak’s user entities, with custom attributes added to store domain-specific information.</li><li><strong>Roles</strong>: Mapped to Keycloak roles, with hierarchical relationships preserved through the use of composite roles.</li><li><strong>Groups</strong>: Employed to organize users by domains and business units, providing an additional organizational layer.</li><li><strong>Clients</strong>: Created for each application (both frontend and API) that needed to interact with Keycloak.</li><li><strong>Scopes</strong>: Used to represent granular permissions, allowing for fine-grained access control.</li></ul><p>For authorization scenarios requiring complex permission checks, we configured Keycloak’s Authorization Services. This allowed us to define resources and permissions within Keycloak, then use its authorization endpoint to perform access checks, effectively removing authorization code from our services.</p><p>The most significant achievement of this mapping was eliminating the need for authentication and authorization logic in our individual services. Previously, each service maintained its own code for token validation and access control. With Keycloak, these responsibilities shifted to the centralized IAM system and our API gateway, simplifying our services considerably.</p><h3>The Token Exchange Mechanism: Enabling Seamless Transitions</h3><p>Perhaps our most innovative solution was the token exchange mechanism we developed to handle the <strong><em>“no logout”</em> </strong>requirement. This elegant approach allowed existing users to continue using the platform uninterrupted while we gradually transitioned to Keycloak.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tolhka_h0iYHTNV6YYsVcA.png" /><figcaption>Highlevel view of token exchange mechanism</figcaption></figure><p>Here’s how our token exchange process worked:</p><p>1. When a request with an old-format token reached our API gateway, we intercepted it before forwarding it to the backend services.</p><p>2. The gateway called our newly developed “Token Exchanger Service” with the user’s legacy token.</p><p>3. The Token Exchanger first validated the legacy token to ensure it was legitimate.</p><p>4. It then checked if this user already existed in Keycloak.</p><p>5. If the user didn’t exist, it performed an “on-the-fly” migration:</p><ul><li>Retrieved the user’s basic information from the appropriate legacy database</li><li>Created a user account in Keycloak via the Admin API</li><li>Configured appropriate roles and group memberships</li></ul><p>6. Using Keycloak’s impersonation capabilities, the Token Exchanger obtained a new Keycloak token for this user.</p><p>7. The API gateway replaced the legacy token in the request with the new Keycloak token and forwarded the request to the backend services.</p><p>The entire process happened transparently, with the user completely unaware of the behind-the-scenes token transformation. This approach offered several advantages:</p><ul><li>Users remained logged in throughout the migration process</li><li>User migration happened incrementally as users interacted with the system</li><li>We avoided the massive database migration that would have been required to move all users at once</li><li>Backend services gradually transitioned to working exclusively with Keycloak tokens</li></ul><p>The Token Exchanger Service contained sophisticated caching mechanisms to avoid repeated calls to Keycloak for users who had already been migrated, improving performance during the transition period.</p><h3>Handling Password Migration</h3><p>For users with password-based authentication, we implemented a clever staged migration strategy:</p><ol><li>Initially, users continued using their existing tokens via the token exchange mechanism described above.</li><li>When these tokens expired and users attempted to log in with their password, we:</li></ol><ul><li>Verified the password against our legacy authentication system</li><li>If valid, prompt the user to set a new password (presenting this as a security enhancement rather than a migration requirement)</li><li>Stored the new password in Keycloak</li><li>Completed the user’s migration to the new system</li></ul><p>This approach avoided the need to send mass password reset emails or force users to take immediate action. Instead, users naturally migrated as they interacted with the system, creating a gradual transition that minimized support requests and user friction.</p><p>For users who didn’t log in frequently, we eventually implemented a scheduled migration process that created placeholder accounts in Keycloak and sent password reset emails, but only after the majority of active users had already migrated organically.</p><h3>Custom OTP Authentication Flow</h3><p>Our OTP-based login flow required significant customization beyond Keycloak’s standard capabilities. We leveraged Keycloak’s Service Provider Interface (SPI) to develop a custom authenticator that implemented our specific business rules for OTP authentication.</p><p>Our custom authenticator handled:</p><ul><li>OTP generation with configurable complexity rules</li><li>Customizable timeout periods for OTP validity</li><li>Retry count limitations and rate limiting to prevent brute force attacks</li><li>Domain-specific validation rules</li><li>Special handling for new versus returning users</li></ul><p>The flow worked by having Keycloak generate and return the OTP through its custom authentication flow. Once the OTP was generated by Keycloak, our application services would handle the actual delivery of these OTPs to users. This separation of concerns allowed us to maintain our existing delivery mechanisms while benefiting from centralized authentication management.</p><p>We structured our authenticator as a series of discrete steps that could be arranged into different authentication flows, giving us flexibility to create domain-specific experiences while maintaining centralized control.</p><p>The development of this custom authenticator required deep understanding of Keycloak’s internals, but it allowed us to maintain the familiar authentication experience our users expected while gaining all the benefits of centralized identity management.</p><p>By designing our custom authenticator to be configurable through Keycloak’s admin interface, we ensured that future adjustments to timeouts, retry counts, and other parameters could be made without code changes, significantly improving our operational efficiency.</p><h3>API Gateway Integration</h3><p>A final critical piece of our solution was integrating Keycloak with our API gateway. We configured the gateway to validate Keycloak tokens using Keycloak’s public key, enabling token verification without making network calls to Keycloak for every request.</p><p>This approach allowed us to:</p><ol><li>Remove token validation code from individual services</li><li>Implement consistent token handling across all APIs</li><li>Centralize token-related policies like refresh handling and expiration</li><li>Reduce latency by avoiding unnecessary validation calls</li></ol><p>The combination of our token exchange mechanism and API gateway integration created a robust transition architecture that maintained security while enabling a seamless migration experience.</p><p>Through these four engineering solutions, we successfully migrated to a centralized IAM platform without disrupting our users or operations. The migration required significant technical creativity, especially in developing the token exchange mechanism and custom OTP authenticator, but the resulting improvements in security, manageability, and developer experience have justified the investment many times over.</p><h3>Outcomes and Lessons Learned</h3><p>The transition to Keycloak as our centralized IAM solution has delivered significant benefits across our organization. Our engineering teams now work with a consistent authentication framework, eliminating the need to maintain multiple custom implementations. This has reduced our technical debt and freed up developer resources to focus on core business features rather than identity infrastructure.</p><p>From an operational perspective, user management has become dramatically simpler. Our administrators can manage users, roles, and permissions through a single interface, improving efficiency and reducing the risk of configuration errors. The centralized audit logging has also enhanced our security posture, giving us visibility into authentication events across all domains.</p><p>Perhaps most importantly, we’ve established a scalable foundation for future growth. As we add new services and user types, we can leverage our existing IAM infrastructure rather than building new authentication systems from scratch. The federated authentication capabilities of Keycloak have also opened possibilities for partnerships that would have been technically challenging under our previous approach.</p><p>Throughout this journey, we learned several valuable lessons:</p><ol><li>Incremental migration is preferable to “big bang” approaches, especially for critical infrastructure like authentication.</li><li>Creative solutions like our token exchange mechanism can enable smooth transitions without disrupting users.</li><li>While off-the-shelf solutions provide tremendous value, be prepared to extend them for your specific requirements.</li><li>Cross-team collaboration is essential when refactoring systems that span multiple domains.</li></ol><h3>Conclusion</h3><p>Our migration to Keycloak represents more than just a technical upgrade — it’s a fundamental shift in how we approach identity management at PickMe. By centralizing our IAM infrastructure, we’ve improved security, simplified operations, and established a foundation for future growth.</p><p>The journey wasn’t without challenges, but each obstacle provided an opportunity to rethink our approach to authentication and authorization. The solutions we developed, particularly our token exchange mechanism and custom OTP authenticator, demonstrate how technical creativity can enable business continuity during significant architectural changes.</p><p>As an engineer who had the opportunity to engage in the R&amp;D efforts for many of the solutions described in this article, I found this project particularly rewarding. Developing the token exchange mechanism and designing the custom OTP authenticator presented complex technical challenges that expanded my knowledge of both identity management principles and Keycloak’s internals. The experience of migrating a production system with thousands of active users without service disruption has been invaluable for my professional growth, teaching me how to balance technical excellence with pragmatic business requirements.</p><p>For teams considering similar migrations, I encourage you to approach the challenge with both technical rigor and creativity. The path to centralized identity management may not be straightforward, but the benefits in security, maintainability, and developer experience make it well worth the journey.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=707d16f06fbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pickme-engineering-blog/our-journey-to-centralized-identity-management-707d16f06fbf">Our Journey to Centralized Identity and Access Management</a> was originally published in <a href="https://medium.com/pickme-engineering-blog">The PickMe Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go’s Escape Analysis Demystified: When Variables Move to the Heap (and Why It Matters)]]></title>
            <link>https://medium.com/codex/gos-escape-analysis-demystified-when-variables-move-to-the-heap-and-why-it-matters-91200e34729d?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/91200e34729d</guid>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Thu, 27 Mar 2025 17:14:08 GMT</pubDate>
            <atom:updated>2025-03-28T15:06:15.377Z</atom:updated>
            <content:encoded><![CDATA[<h4>Go: Stack vs Heap Allocation</h4><p>Go’s compiler is pretty smart. It decides where variables should be stored either on the stack (faster, limited size) or the heap (slower, but more flexible). The process behind this decision is called <strong>escape analysis</strong>, and understanding it can help you write more efficient Go code.</p><h3>When Does a Variable Escape to the Heap?</h3><p>A variable <strong>escapes</strong> to the heap when the compiler determines that it <strong>outlives the function’s stack frame</strong> or requires <strong>dynamic memory allocation</strong>. Here are common cases where this happens:</p><ol><li>Returning a pointer to a local variable</li><li>Storing a value inside an interface</li><li>Using a slice that grows dynamically</li><li>Passing a pointer to another function that outlives the caller</li><li>Using a variable inside a goroutine</li></ol><h3>Example: Escape Analysis in Action</h3><p>Let’s analyze a simple Go program:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;time&quot;<br>)<br><br>func main() {<br> fmt.Println(&quot;Understanding Escape Analysis in Go&quot;)<br> value := createValue()<br> fmt.Println(&quot;Created value:&quot;, value)<br><br> counter := 5<br> go decrementCounter(&amp;counter)<br> for counter &gt; 0 {<br>  time.Sleep(500 * time.Millisecond)<br>  fmt.Println(&quot;Counter in main:&quot;, counter)<br> }<br>}<br><br>func createValue() *int {<br> num := 42<br> return &amp;num // num escapes to heap<br>}<br><br>func decrementCounter(counter *int) {<br> for *counter &gt; 0 {<br>  time.Sleep(1 * time.Second)<br>  *counter--<br> }<br>}</pre><h4>What Happens Here?</h4><p>When compiled with:</p><pre>go build -gcflags=&quot;-m&quot; main.go</pre><p>We see output similar to:</p><pre>./main.go:13:2: moved to heap: counter<br>./main.go:10:22: moved to heap: num<br>./main.go:9:13: ... argument does not escape<br>./main.go:9:14: &quot;Understanding Escape Analysis in Go&quot; escapes to heap<br>./main.go:11:13: ... argument does not escape<br>./main.go:11:14: &quot;Created value:&quot; escapes to heap<br>./main.go:17:14: ... argument does not escape<br>./main.go:17:15: &quot;Counter in main:&quot; escapes to heap<br>./main.go:17:35: counter escapes to heap</pre><p>This tells us that:</p><ol><li>num is <strong>moved to the heap</strong> because we are returning its pointer.</li><li>counter is <strong>moved to the heap</strong> because it is shared with a goroutine.</li><li>counter also <strong>escapes</strong> when passed to fmt.Println, since fmt.Println uses an interface (interface{}), which forces heap allocation. Same reason for the string literals passed for the fmt.Println function.</li></ol><h3>Moved to Heap vs. Escapes to Heap</h3><p>These two phrases in escape analysis output might sound similar, but they have different implications:</p><ul><li><strong>Moved to heap</strong>: The compiler proactively moves a variable to the heap because it determines that it cannot be safely allocated on the stack. This often happens when a local variable’s reference is returned or passed to a goroutine.</li><li><strong>Escapes to heap</strong>: This means the variable’s value (not necessarily its reference) must be allocated on the heap due to interactions with interfaces, dynamic allocations, or function calls requiring heap storage.</li></ul><p>For example:</p><pre>func returnPointer() *int {<br> n := 42<br> return &amp;n // &#39;n&#39; is moved to heap<br>}</pre><p>And in an interface scenario:</p><pre>func printValue(i interface{}) {<br> fmt.Println(i)<br>}<br><br>func main() {<br> x := 100<br> printValue(x) // &#39;x&#39; escapes to heap due to interface conversion<br>}</pre><p>Here, x itself is not stored on the heap initially, but its value is copied into an interface, which requires heap allocation.</p><h3>What If We Avoid Returning a Pointer?</h3><p>If we modify createValue to return an int instead:</p><pre>func createValue() int {<br> return 42 // No heap allocation<br>}</pre><p>The escape analysis output changes, and num stays on the stack.</p><p>Similarly, if we remove the go keyword from decrementCounter(&amp;counter), counter will no longer be moved to the heap.</p><h3>Why Minimize Heap Escapes?</h3><p>As software engineers, reducing heap allocations can lead to <strong>better performance</strong> and <strong>lower memory pressure</strong>. Here’s why we should aim to minimize escapes:</p><ul><li><strong>Stack allocation is faster</strong>: Stack memory is much faster to allocate and deallocate compared to heap memory.</li><li><strong>Garbage collection overhead</strong>: Objects on the heap are managed by Go’s garbage collector, which adds runtime overhead.</li><li><strong>Better CPU cache utilization</strong>: The stack is more CPU-cache-friendly, improving execution speed.</li><li><strong>Avoiding unnecessary memory allocations</strong>: Minimizing heap usage reduces the chances of frequent memory allocations and deallocations, leading to more efficient code execution.</li></ul><h3>How to minimize Escapes?</h3><p>To keep variables on the stack, follow these guidelines:</p><ul><li>Be aware when returning pointers to local variables.</li><li>Use value types instead of pointers when possible.</li><li>Be mindful when storing values in interfaces, as they often lead to heap allocation.</li><li>Minimize the sharing of variables between goroutines when possible.</li></ul><p>Next time you’re debugging performance issues, take a peek at your escape analysis, you might be surprised what Go decides for you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91200e34729d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/gos-escape-analysis-demystified-when-variables-move-to-the-heap-and-why-it-matters-91200e34729d">Go’s Escape Analysis Demystified: When Variables Move to the Heap (and Why It Matters)</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup an API-Gateway With Ingress Controller in a Kubernetes Cluster]]></title>
            <link>https://medium.com/codex/setup-an-api-gateway-with-ingress-controller-in-a-kubernetes-cluster-78768281abbb?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/78768281abbb</guid>
            <category><![CDATA[ingresscontroller]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[gateway-service]]></category>
            <category><![CDATA[apache-apisix]]></category>
            <category><![CDATA[ingress]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Tue, 31 Dec 2024 19:55:20 GMT</pubDate>
            <atom:updated>2025-01-06T03:21:08.855Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hands-on with Apache APISIX (Open-source API-Gateway)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*knNssw2MWHZOe2jKeqaCSw.png" /><figcaption>Simple illustration how APISIX gateway is deployed in a kubernetes cluster</figcaption></figure><p><a href="https://medium.com/codex/setup-an-api-gateway-with-ingress-controller-in-a-kubernetes-cluster-78768281abbb?source=friends_link&amp;sk=f108bff95e67497237edb4bed0307869">If you are using free-tier, then you can view this article using this link.</a></p><p>An API gateway is like the front door to your services. It handles incoming requests, routes them to the right backend service, and applies any rules you’ve set up, like rate limiting or authentication. It’s especially useful in microservices architectures where many small services work together.</p><p>Now, Apache APISIX is a fantastic open-source API gateway. Why? It’s fast, super stable, and built for the cloud-native world. Plus, it comes packed with features like dynamic routing, plugin support, and an easy-to-use admin API. In this article, I’ll show you how to set up Apache APISIX as your API gateway in a Kubernetes cluster. Let’s dive in!</p><h3>Prerequisites</h3><p>Before we jump into setting up Apache APISIX, let’s make sure you’re all set. You’ll need:</p><ol><li><strong>A Running Kubernetes Cluster</strong>: If you don’t already have one, you can use tools like Minikube, kind, or a managed Kubernetes service like GKE, EKS, or AKS.</li><li><strong>kubectl Installed</strong>: This is the command-line tool to interact with your cluster.</li><li><strong>Helm Installed</strong>: Helm simplifies the process of deploying applications in Kubernetes.</li></ol><p>That’s it! Once you’ve got these in place, you’re ready to deploy Apache APISIX into your cluster. Let’s move to the installation.</p><h3>Installing Apache APISIX in Kubernetes</h3><p>Now it’s time to deploy Apache APISIX in your Kubernetes cluster. We’ll use Helm to make the process quick and easy. Here’s how you do it:</p><ol><li><strong>Add the Helm Repositories</strong><br>First, add the required Helm repositories for Apache APISIX and Bitnami:</li></ol><pre>helm repo add apisix https://charts.apiseven.com<br>helm repo add bitnami https://charts.bitnami.com/bitnami<br>helm repo update</pre><p>2. <strong>Create a Namespace</strong><br>To keep things organized, create a dedicated namespace for APISIX:</p><pre>kubectl create ns ingress-apisix</pre><p>3. <strong>Install Apache APISIX with the Ingress Controller</strong><br>Use the following command to install Apache APISIX along with its ingress controller:</p><pre>helm install apisix apisix/apisix \<br>    --set gateway.type=LoadBalancer \<br>    --set ingress-controller.enabled=true \<br>    --namespace ingress-apisix \<br>    --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix</pre><blockquote>Here we are telling to set the gateway type to be a LoadBalancer. If you are doing this in any cloud service please make sure that costs are involved. If you are not sure then, Make it as a NodePort or a ClusterIP. Loadbalancer will allow you to call the gateway using an external IP address (but this only works if your cloud provider supports this).</blockquote><p><strong>4. Verify the Deployment</strong><br>Once the installation is complete, check if all the components are running as expected:</p><pre>kubectl get pods -n ingress-apisix<br>kubectl get svc -n ingress-apisix</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXRD6YwG1hleoF44NWrrvA.png" /><figcaption>Result on how pods related to APISIX are deployed</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CU7U3aRSfme1ed5g4i0KLA.png" /><figcaption>Result on how services related to APISIX are configured</figcaption></figure><p>You should see the APISIX gateway, admin service, and ingress controller up and running.</p><p>With APISIX deployed in your cluster, you now have a powerful API-Gateway ready to handle requests. Next, let’s access its admin interface and explore how it works!</p><h3>Accessing Apache APISIX Admin Interface</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*ujNlI5RP-73BnhYP8B88lQ.png" /><figcaption>Illustration on how various components in the APISIX are contributing for traffic handling</figcaption></figure><p>Now that Apache APISIX is installed, let’s connect to its admin interface. The admin interface allows you to manage routes, plugins, and other configurations directly. Here’s how you can access it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*UM5pYoqFMxxYU69-t8GyDg.png" /><figcaption>APISIX gateway has both the control plane and data plane in the same service.</figcaption></figure><ol><li><strong>Port-Forward the Admin Service</strong><br>Use the following command to forward traffic from your local machine to the APISIX admin service running in the Kubernetes cluster:</li></ol><pre>kubectl port-forward svc/apisix-admin 9180:9180 -n ingress-apisix</pre><p>This will make the admin interface accessible on http://localhost:9180 from your local machine.</p><p><strong>2. Check for Existing Routes</strong><br>Let’s see if there are any routes configured in the gateway. Run this command:</p><pre>curl -s http://localhost:9180/apisix/admin/routes -H &#39;X-API-Key: edd1c9f034335f136f87ad84b625c8f1&#39;</pre><p>Since we haven’t added any routes yet, the response will be empty or indicate that no routes are configured.</p><p>At this point, you’ve accessed the APISIX admin interface and confirmed that no routes exist yet. In the next section, we’ll explore how the ingress controller works and set up routes automatically using ingress resources.</p><h3>Understanding Ingress Controllers</h3><p>Let’s talk about ingress controllers for a moment. In Kubernetes, an ingress controller is like a traffic director. It listens for ingress resources, which are configuration files that define how external traffic should be routed to your services and takes care of setting everything up behind the scenes.</p><p>Now, Apache APISIX doesn’t just have an ingress controller — it has a powerful one. When you create an ingress resource, the APISIX ingress controller automatically translates it into routes and configurations in the APISIX gateway. This means you don’t have to manually configure routes using the admin API unless you want to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*i2VGI6ZwzcbmwY2mZU7E9g.png" /><figcaption>How the ingress controller is configuring the gateway based on the ingress resources created in the cluster</figcaption></figure><p>Here’s the beauty of it: by combining the flexibility of APISIX with the simplicity of ingress resources, you get the best of both worlds — dynamic route management and cloud-native ease of use.</p><p>Next, we’ll set up some dummy services in the cluster and see how to configure ingress resources that the APISIX ingress controller will pick up.</p><h3>Deploying Dummy Services</h3><p>To demonstrate how Apache APISIX routes traffic, we’ll deploy two simple dummy services in the Kubernetes cluster. These services will act as backends for our API gateway routes.</p><ol><li><strong>Deploy the First Dummy Service</strong><br>Run the following commands to create a pod and expose it as a service:</li></ol><pre>kubectl run bare-minimum-api-v1 --image jmalloc/echo-server --port 8080 -- 8080<br>kubectl expose pod bare-minimum-api-v1 --port 8080</pre><p>This creates a service named bare-minimum-api-v1 that listens on port 8080.</p><p><strong>2. Deploy the Second Dummy Service</strong><br>Similarly, deploy another dummy service with these commands:</p><pre>kubectl run bare-minimum-api-v2 --image jmalloc/echo-server --port 8080 -- 8080<br>kubectl expose pod bare-minimum-api-v2 --port 8080</pre><p>This creates a service named bare-minimum-api-v2, also listening on port 8080.</p><p><strong>3. Verify the Services</strong><br>Check if the services are up and running by listing them:</p><pre>kubectl get svc -n default</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*kNh-LFpfJQBa5G19464MqQ.png" /><figcaption>How the example services looks like</figcaption></figure><p>You should see both bare-minimum-api-v1 and bare-minimum-api-v2 in the output.</p><p>These dummy services are now ready to receive traffic. In the next step, we’ll configure an ingress resource to route requests to these services using the APISIX gateway.</p><h3>Configuring Ingress Resources</h3><p>Now, let’s set up an ingress resource to define how the APISIX gateway should route traffic to the dummy services we just deployed. In Kubernetes, we define these routes using custom resources for APISIX called ApisixRoute. Here’s how you can do it:</p><ol><li><strong>Create the Ingress Resource Manifest</strong></li></ol><p>Create a new YAML file (let’s call it apisix-routes.yaml) to define the routes for our two dummy services. The manifest will look like this:</p><pre>apiVersion: apisix.apache.org/v2<br>kind: ApisixRoute<br>metadata:<br>  name: api-routes<br>spec:<br>  http:<br>    - name: route-1<br>      match:<br>        hosts:<br>          - local.testhost<br>        paths:<br>          - /v1<br>      backends:<br>        - serviceName: bare-minimum-api-v1<br>          servicePort: 8080<br>    - name: route-2<br>      match:<br>        hosts:<br>          - local.testhost<br>        paths:<br>          - /v2<br>      backends:<br>        - serviceName: bare-minimum-api-v2<br>          servicePort: 8080</pre><p>Here’s what’s happening in the manifest:</p><blockquote>We define two HTTP routes.</blockquote><blockquote>The first route (/v1) will be mapped to the bare-minimum-api-v1 service.</blockquote><blockquote>The second route (/v2) will be mapped to the bare-minimum-api-v2 service.</blockquote><blockquote>Both routes match requests coming to local.testhost</blockquote><p><strong>2. Apply the Ingress Resource</strong></p><p>Once you’ve created the manifest file, apply it to your Kubernetes cluster using the following command:</p><pre>kubectl apply -f apisix-routes.yaml</pre><p>This will create the routes in Apache APISIX, and the ingress controller will automatically configure the gateway to route traffic according to these rules.</p><p><strong>3. Verify the Routes in APISIX</strong></p><p>To make sure the routes were successfully configured, run the same curl command as before:</p><pre>curl -s http://localhost:9180/apisix/admin/routes -H &#39;X-API-Key: edd1c9f034335f136f87ad84b625c8f1&#39;</pre><p>This should now return the two routes you’ve just created, confirming that APISIX is aware of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*zJ_Q-4A8E_hBCs9MBbstwg.png" /><figcaption>APISIX response for the get routes request</figcaption></figure><p>With the ingress resource applied, the APISIX gateway is now ready to route traffic to the appropriate services based on the paths /v1 and /v2. In the next step, we’ll test everything by exposing the APISIX gateway and making some requests to it.</p><h3>Testing the Gateway</h3><p>With the routes configured, it’s time to test the APISIX gateway in action. We’ll expose the APISIX gateway service and use curl to send requests to the defined routes. Here’s how you can do it:</p><ol><li><strong>Expose the APISIX Gateway Service</strong></li></ol><p>To make the APISIX gateway accessible from your local machine, you can either use port forwarding or a load balancer. Let’s start with port forwarding. Run the following command to forward traffic from your local machine to the APISIX gateway service in Kubernetes:</p><pre>kubectl port-forward svc/apisix-gateway 9001:80 -n ingress-apisix</pre><p>This will make the APISIX gateway accessible at <a href="http://localhost:9001.">http://localhost:9001.</a></p><blockquote>If you have already used the type as LoadBalancer when applying the APISIX helm chat, then you don’t need to port forward. You can simply hit the gateway using the External IP given by the cloud provider to apisix-gateway service.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCHHo51UMscflPvJZ_gRqg.png" /><figcaption>Expernal IP will be given by the cloud provider if you set the service type to load balancer</figcaption></figure><p><strong>2. Test the </strong><strong>/v1 Route</strong></p><p>Now that the gateway is exposed, let’s test the first route (/v1) we created. Run the following curl command:</p><pre>curl http://localhost:9001/v1 -H &#39;host:local.testhost&#39;</pre><p>You should see a response like this:</p><blockquote>Request served by bare-minimum-api-v1</blockquote><p>This means that the request was correctly routed to the bare-minimum-api-v1 service.</p><p><strong>3. Test the </strong><strong>/v2 Route</strong></p><p>Next, test the second route (/v2) by running this curl command:</p><pre>curl http://localhost:9001/v2 -H &#39;host:local.testhost&#39;</pre><p>You should see a response like this:</p><blockquote>Request served by bare-minimum-api-v2</blockquote><p>This confirms that the request was routed to the bare-minimum-api-v2 service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*q1rErcVDdk9hzUzn5N634Q.png" /><figcaption>How the result look like served from both the backend services</figcaption></figure><blockquote>In my case the output is v3, v4 it should be v1, v2 in yours. When testing I had to create additional bare-minimum-services that is the case.</blockquote><p>At this point, we’ve successfully verified that the APISIX gateway is properly routing traffic based on the defined paths (/v1 and /v2). The APISIX ingress controller has taken care of setting up the routes dynamically, and the traffic is flowing correctly to the backend services.</p><h3>Comparison with Traditional Ingress</h3><p>At this point, you’ve seen how Apache APISIX works as an API gateway with Kubernetes Ingress. But how does it compare to traditional ingress controllers like Nginx? Let’s take a moment to highlight the key differences and advantages of using APISIX.</p><p><strong>1. Flexibility and Extensibility</strong><br>Traditional ingress controllers like Nginx are good for basic routing, but they’re limited when it comes to more complex use cases like dynamic routing, rate limiting, or custom authentication. APISIX, on the other hand, is highly flexible and extensible. It has built-in support for a wide range of advanced features, such as:</p><ul><li>Rate limiting</li><li>API key authentication</li><li>Custom filters using Lua scripts</li><li>Advanced load-balancing algorithms</li></ul><p><strong>2. Performance and Scalability</strong><br>APISIX is designed to be lightweight and high-performing. It uses NGINX and OpenResty under the hood, combined with a custom routing engine, allowing it to handle millions of requests per second with low latency. Traditional ingress controllers, like Nginx, are generally capable but might not scale as well for very high-throughput environments.</p><p><strong>3. Cloud-Native and Dynamic Configuration</strong><br>While traditional ingress controllers are static and require manual configuration for route changes, APISIX leverages a dynamic configuration model. You can add, update, and remove routes and configurations on the fly without needing to reload or restart the gateway. This is particularly useful in microservices environments where services and routes may change frequently.</p><p><strong>4. Admin API and Customization</strong><br>One of the standout features of APISIX is its Admin API, which allows you to configure routes, plugins, and other gateway features programmatically. Traditional ingress controllers often rely on static configuration files or annotations to define routing rules, but APISIX gives you fine-grained control through its API. You can automate complex configurations or even create custom plugins for your use cases.</p><p><strong>5. Plugin Ecosystem</strong><br>APISIX has a rich ecosystem of built-in and community-driven plugins for things like security (authentication, rate limiting, etc.), monitoring (metrics collection), and traffic management (load balancing). While Nginx has some plugin support, APISIX offers a much broader set of out-of-the-box solutions specifically tailored to the modern microservices and cloud-native landscape.</p><p>In this article, we’ve walked through the process of setting up Apache APISIX as an API gateway in a Kubernetes cluster, leveraging the APISIX Ingress Controller to manage traffic routing to backend services. We covered everything from installing and configuring APISIX to testing routes and comparing it with traditional ingress controllers.</p><p><strong>Next Steps:</strong></p><ol><li><strong>Explore More Advanced Features</strong>: Now that you’ve seen the basics, dive deeper into Apache APISIX’s powerful plugin ecosystem. Explore features like rate limiting, custom authentication, traffic filtering, and more.</li><li><strong>Configure APISIX Using Admin API</strong>: In the next article, we’ll dive into how you can configure and manage APISIX directly through its Admin API, enabling even more complex and automated gateway configurations.</li><li><strong>Scaling and Monitoring</strong>: Learn how to scale your APISIX gateway and monitor its performance, ensuring it can handle high traffic and provide insights into API usage.</li></ol><p>In upcoming articles, we’ll take a closer look at these advanced features, enabling you to unlock the full potential of Apache APISIX in your Kubernetes clusters. Stay tuned for more!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78768281abbb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/setup-an-api-gateway-with-ingress-controller-in-a-kubernetes-cluster-78768281abbb">Setup an API-Gateway With Ingress Controller in a Kubernetes Cluster</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Authenticator Apps Work Behind the Scenes of Your OTPs]]></title>
            <link>https://medium.com/codex/how-authenticator-apps-work-behind-the-scenes-of-your-otps-0777b88075aa?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/0777b88075aa</guid>
            <category><![CDATA[google-authenticator-app]]></category>
            <category><![CDATA[otp-verification]]></category>
            <category><![CDATA[authenticator-apps]]></category>
            <category><![CDATA[hmac]]></category>
            <category><![CDATA[microsoft-authenticator]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 06:09:35 GMT</pubDate>
            <atom:updated>2024-11-21T13:55:39.995Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Google, Microsoft, Authenticators work</h4><p>“Ever wonder what magic happens when you open an authenticator app, type in a code, and instantly gain access? It’s not magic — it’s math! Let’s dive into how authenticator apps work, from the first scan of that QR code to the time-based code generation and validation process.”</p><h3>Registering the Authenticator App with a QR Code</h3><p>When you scan a QR code in your authenticator app, it contains vital information for generating OTPs. This QR code encodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glSk_CHPp8EK4UClfMjJdQ.png" /><figcaption>What the content in the QR code looks like</figcaption></figure><ul><li><strong>Secret Key:</strong> A unique key that ensures each OTP is specific to you.</li><li><strong>OTP Generation Algorithm: </strong>Typically HMAC-SHA1.</li><li><strong>Digits in Each OTP: </strong>Usually six.</li><li><strong>Time Interval: </strong>Typically set to 30 seconds for generating new codes.</li><li><strong>Issuer: </strong>The name of the service.</li></ul><blockquote>Sometimes only the secret key will be there in the QR code or else if you are unable to scan the QR code then, the system displays the secret key and asks you to enter it manually in the authenticator app. In that case, the default values for the other values are used.</blockquote><p>The Authenticator app scans the above QR and stores these values into your profile. These elements allow your app to generate codes that sync with the server without needing constant communication. The secret key is stored secularly so that it will not get compromised.</p><h3>How Does OTP Generation Work?</h3><p>Once the QR code is scanned, your authenticator app uses the TOTP (Time-based One-Time Password) algorithm to generate OTPs.</p><h4>1. TOTP vs. HOTP:</h4><p>TOTP (Time-based One Time Password) builds on HOTP (HMAC-based One-Time Password) but relies on time rather than a counter, generating new codes at regular intervals (usually every 30 seconds).</p><h4>2. Key Parameters:</h4><ul><li><strong>Secret Key:</strong> Shared between your app and the server.</li><li><strong>Time Interval:</strong> The current Unix time divided by 30 seconds forms a “time counter.”</li><li><strong>HMAC Algorithm:</strong> Typically HMAC-SHA1 combines the secret key with the time counter.</li></ul><h4>3. Code Generation Steps:</h4><ul><li><strong>Calculate Time Counter:</strong> Divide current Unix time by 30 seconds to get the time counter.</li><li><strong>Compute HMAC:</strong> Use the secret key and time counter as inputs for HMAC-SHA1.</li><li><strong>Truncate HMAC Result:</strong> Select a specific portion of the hash using dynamic truncation.</li><li><strong>Convert to 6-Digit Code: </strong>Transform the truncated hash into a six-digit integer.</li></ul><p>Each OTP is valid until the next interval resets, allowing both the app and server to produce matching codes independently.</p><p>To get a clearer picture of how time intervals affect OTP generation, let’s walk through a simple example step by step. This will help you understand how a small delay in typing or network issues can influence the validation process.</p><blockquote><strong>Step 1: OTP Generation on the Phone</strong></blockquote><blockquote>Let’s assume that at the moment of OTP generation on your phone, the current Unix time is 1609459200 (which corresponds to a specific date and time). The phone divides this time by a fixed interval, which in most TOTP implementations is 30 seconds.</blockquote><blockquote><strong>Calculation</strong>:<br>1609459200 (Unix time) ÷ 30 (time interval) = 53648640 (time counter)</blockquote><blockquote>The result is 53648640, which becomes the time counter used to generate the OTP. So, at this point, the phone creates a 6-digit OTP based on this time counter.</blockquote><blockquote><strong>Step 2: Typing the Code in the Browser</strong></blockquote><blockquote>Now, you start typing the code you see on the phone into the browser. Let’s say it takes <strong>15 seconds</strong> to type the code and then <strong>1 second</strong> for it to reach the server. By the time the server receives your OTP, the Unix time will have changed slightly.</blockquote><blockquote><strong>New Unix Time on Server</strong>: 1609459216<br>(This is 16 seconds after the initial time when you started generating the OTP.)</blockquote><blockquote><strong>Step 3: OTP Generation on the Server</strong></blockquote><blockquote>The server, which is synchronized with your phone and has the same secret key, also generates an OTP using its current Unix time. When the server calculates the time counter, it divides the new Unix time by 30.</blockquote><blockquote><strong>Calculation</strong>:<br>1609459216 (new Unix time) ÷ 30 = 53648640.533</blockquote><blockquote>The fractional part (.533) is dropped, leaving the server with the same time counter value of 53648640, just like your phone did. As a result, the OTP generated on the server matches the one you entered, and the authentication is successful.</blockquote><blockquote><strong>Assumption: Introducing a Longer Delay</strong></blockquote><blockquote>Now, let’s assume you took <strong>25 seconds</strong> to type the code, and it took an additional <strong>6 seconds</strong> for the code to reach the server due to network latency. The new Unix time on the server is now 1609459231.</blockquote><blockquote><strong>Calculation</strong>:<br>1609459231 (new Unix time) ÷ 30 = 53648641.033</blockquote><blockquote>This time counter is now 53648641 after removing the fractional part. Since this counter is different from the one used by your phone (53648640), the OTP generated on the server will be different from the one you entered.</blockquote><blockquote><strong>Server Responds with “Invalid OTP”</strong></blockquote><blockquote>Because the server generated a different OTP based on the new time counter (53648641), it will determine that the OTP you entered is invalid. The authentication process fails, and the server responds with an error message indicating that the OTP was incorrect.</blockquote><h3>Entering and Validating the Code</h3><p>After receiving your OTP, you enter it on the login page. Here’s how validation occurs without direct communication between the app and server:</p><ol><li><strong>Entering the Code: </strong>You type the 6-digit code into the login form, which is only valid for a short time.</li><li><strong>Server Validation: </strong>The server generates its own OTP using the same secret key and algorithm based on the current time and surrounding intervals. It checks if your entered code matches any generated for those intervals.</li><li><strong>No Communication Required: </strong>This system allows both parties to remain synchronized through shared keys and time intervals, ensuring security while providing flexibility for users.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QzwbppncrGQqs9gr0YAiKQ.png" /><figcaption>OTP verification from the server side</figcaption></figure><p>By understanding these processes, you can appreciate how authenticator apps enhance online security through dynamic OTP generation and validation mechanisms, making unauthorized access significantly more difficult.</p><p>This method of validating the code ensures that the process remains secure while allowing flexibility for users to enter their OTP within a brief window. Whether you’re entering the code from the app on your phone or the server validating it in the background, you both end up at the same destination: <strong>authenticated access to your account</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0777b88075aa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/how-authenticator-apps-work-behind-the-scenes-of-your-otps-0777b88075aa">How Authenticator Apps Work Behind the Scenes of Your OTPs</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[gRPC Tips For Better Efficiency]]></title>
            <link>https://medium.com/@isurucuma/grpc-tips-for-better-efficiency-6c2d5e6f7695?source=rss-71c96e854bde------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c2d5e6f7695</guid>
            <category><![CDATA[remote-procedure-calls]]></category>
            <category><![CDATA[http2]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[protobuf]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Isuru Cumaranathunga]]></dc:creator>
            <pubDate>Sun, 10 Nov 2024 15:12:09 GMT</pubDate>
            <atom:updated>2024-11-10T15:12:09.519Z</atom:updated>
            <content:encoded><![CDATA[<h4>deep dive to the encoding techniques</h4><p>At the end of this article, I will be answering the following questions</p><ol><li><strong>Why it is advised to use field numbers less than 16 for frequently using fields</strong></li><li><strong>Why it is not advised to alter the field numbers after they have been assigned and released to the production</strong></li></ol><p>First of all, let&#39;s understand the realm of gRPC briefly.</p><h3>Why gRPC Shines in Microservices</h3><p>In a world where microservices need to communicate frequently and efficiently, gRPC has emerged as a go-to choice for high-performance, reliable service communication. Its power lies in using Protocol Buffers (protobufs) to serialize data, making it significantly faster and more compact than traditional JSON-based REST APIs. gRPC enables remote procedure calls (RPCs),<strong> letting services communicate as if they were calling local functions</strong> — even if they’re halfway across the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sdNGLVBuud3BO63iAI-0KA.png" /><figcaption>GRPC handles the underline serialization and data transfer</figcaption></figure><p>Let’s see this in action with a simple example: a microservice to create a user.</p><h4>A Simple User Creation Scenario</h4><p>Suppose we want to define a service for creating users. In gRPC, we use a .proto file to define both the structure of the messages we’ll send and the RPC methods available for our service. Here’s how we can define a CreateUser RPC in our proto file:</p><pre>syntax = &quot;proto3&quot;;<br><br>package user;<br><br>service UserService {<br>    // RPC method for creating a user<br>    rpc CreateUser (CreateUserRequest) returns (CreateUserResponse);<br>}<br><br>// Message for the request, containing user&#39;s name and age<br>message CreateUserRequest {<br>    string name = 1;<br>    int32 age = 2;<br>}<br><br>// Message for the response, containing user&#39;s unique ID<br>message CreateUserResponse {<br>    string user_id = 1;<br>}</pre><p>In this example:</p><ul><li><strong>Service Definition</strong>: The UserService defines a single RPC method, CreateUser. This method accepts a CreateUserRequest message and returns a CreateUserResponse message.</li><li><strong>Message Definitions</strong>:</li><li>CreateUserRequest holds the user’s information (their name and age).</li><li>CreateUserResponse includes the user_id, which serves as the unique identifier for the new user.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HGpaKdzzF-UqcU-pgurWgA.png" /><figcaption>Protofile can be compiled to any language specific stub, client and server can be in different languages</figcaption></figure><h4>From Proto File to Language-Specific Stubs</h4><p>To turn this proto file into something we can use in code, we compile it using the Protocol Buffers compiler, protoc. This process generates language-specific files, also known as &quot;stubs,&quot; for both the client and server. These stubs allow the client to call the CreateUser method as if it were a local function, while the server receives and processes this request remotely.</p><h4>Client-Server Interaction</h4><p>Once the stubs are generated, setting up the client and server becomes simple:</p><ol><li><strong>Server Side</strong>: We implement the CreateUser function in our server code, handling the logic to create a new user and return the generated user_id.</li><li><strong>Client Side</strong>: The client invokes the CreateUser RPC just like a regular function call. gRPC handles all the network communication, so the client doesn’t need to manage HTTP methods or headers.</li></ol><blockquote>This feels just like calling a local function, even though gRPC is handling the encoding, transmission, and decoding of messages in the background. It’s this simplicity, combined with gRPC’s efficiency, that makes it perfect for microservices.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7q9QP6GpP2M4xzwjOoD8w.png" /><figcaption>Entire communication flow from client to server in GRPC</figcaption></figure><h3>How Data Gets Transferred in gRPC</h3><p>Once our client calls the CreateUser RPC, gRPC handles the complex process of encoding and transferring the data to the server. This process happens in two main steps:</p><ol><li><strong>Converting the Message to a Byte Array</strong>: gRPC uses Protocol Buffers (protobufs) to serialize data into a compact, efficient binary format.</li><li><strong>Transferring the Data</strong>: gRPC then sends this serialized data over HTTP/2, ensuring a secure and efficient transport layer.</li></ol><p>Let’s take a closer look at each part to understand how gRPC achieves fast and efficient data transfer.</p><h4>From Method Call to Byte Array</h4><p>When the client calls CreateUser, the gRPC client library transforms the CreateUserRequest message (which contains the name and age fields) into a byte array. This binary representation is both smaller and faster to process compared to a typical JSON message, making it ideal for high-performance applications.</p><p>Protocol Buffers (Protobufs) and Serialization</p><p>Protocol Buffers (protobufs) are designed to be lightweight, with minimal encoding overhead. They use a defined structure in the proto file, where each field has a unique field number. These field numbers play a significant role in how the message is serialized.</p><p>The CreateUserRequest message is serialized in two main sections:</p><ol><li><strong>Metadata Section</strong>: Contains information about each field (such as field number and field type) to identify each part of the data.</li><li><strong>Value (data) Section</strong>: Contains the actual data (i.e., the values for name and age), encoded as a byte array.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1aBFt3JjpvmUOGkmAMGakg.png" /><figcaption>Illustration of the meta-data, and data section arranged in a byte array</figcaption></figure><p>The combination of these sections enables efficient transmission and parsing, even across different systems and languages.</p><h4>HTTP/2 and Data Transfer</h4><p>Once encoded, the byte array is transmitted over HTTP/2, which provides several advantages over HTTP/1.1, including multiplexing, header compression, and persistent connections. This enables gRPC to achieve faster response times and lower latency, especially in microservice environments where multiple services need to communicate frequently and efficiently.</p><p>In this process, gRPC abstracts away the complexity, allowing the client to call CreateUser as if it were a simple, local function. The gRPC runtime handles the encoding, transmission, and decoding seamlessly, so developers can focus on business logic without worrying about the underlying data transfer.</p><h3>Understanding gRPC Encoding: Metadata and Value(Data) Sections</h3><p>When gRPC serializes a message, it organizes the data into a highly efficient binary format using Protocol Buffers (protobufs). This serialized data consists of two main sections: the <strong>metadata section</strong> and the <strong>value section</strong>. Together, these sections allow gRPC to maintain structure, ensure fast transmission, and support compatibility across different services.</p><p>Let’s break down each section to see how they work.</p><h4>1. Metadata Section</h4><p>The metadata section is a crucial part of protobuf’s structure. It contains essential information about each field in the message, helping both the client and server interpret the message accurately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JM7O-eeK32zxsHDu5c6BUQ.png" /><figcaption>Illustration of a meta-data section, numbers represent the default sizes allocated to each sub section</figcaption></figure><p>Each field in the metadata section consists of three parts:</p><ul><li><strong>Field Number</strong>: Each field in a protobuf message has a unique field number. This number indicates the field’s identity in a compact way, replacing the need for lengthy field names in the message. Lower field numbers (1–15) are especially important because they take up less space, which we’ll discuss in more detail later.</li><li><strong>Field Type</strong>: The field type specifies the data type (e.g., string, integer) of the field’s value. This is essential because protobufs support various data types, and the type information allows the server to know how to interpret each field’s value.</li><li><strong>Most Significant Bit (MSB)</strong>: The MSB in the metadata helps indicate the end of a byte sequence, signaling to gRPC when it has reached the end of a field’s metadata.</li></ul><p>For example, in our CreateUserRequest message with fields name and age, the metadata section will encode information on the field numbers and types for name and age. This allows gRPC to quickly interpret the structure of the message, even without the original proto file.</p><h4>2. Value(Data) Section</h4><p>The value section holds the actual data for each field, encoded as a compact byte array. Here’s how it’s structured:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZ0_YEhBQ1VCi6AWrMogSQ.png" /><figcaption>Illustratino of a value section</figcaption></figure><ul><li><strong>Byte Array with MSB Control</strong>: Each value is stored as a byte array, where the MSB is used as a continuation flag. This flag allows gRPC to represent variable-length data efficiently. If a value takes up multiple bytes, each byte will set the MSB accordingly to signal whether more bytes are coming.</li><li><strong>Efficient Encoding for Numbers</strong>: For smaller numbers, protobufs use a method called “varint” encoding. This technique minimizes the space used by storing numbers in as few bytes as possible.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I3gh6iTrY3v18kmkQ34nBw.png" /><figcaption>Illustration of an overall representation of data after encoded</figcaption></figure><p>For our CreateUserRequest example, the value section will contain:</p><ul><li>The byte array representing the name string value, with MSBs indicating the start and end of each byte in the string.</li><li>A varint-encoded integer representing the age value, making it compact and efficient.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*c91RAwZ_cJoDERm5g4wVcg.png" /><figcaption>Example of how real data gets encoded by gRPC runtime</figcaption></figure><blockquote>So it is clear that when transferring data in gRPC no field names are getting transferred. Client and the Server only relies on the field numbers for serializing the deserializing. That is one of the key advantage gained by gRPC to reduce the payload of the transferring data.</blockquote><h3>Best Practices with Field Numbers in gRPC</h3><p>Now that we understand the basic structure of gRPC’s encoding, let’s dive into why certain best practices around field numbers can greatly impact performance and compatibility. Specifically, we’ll answer two important questions:</p><ol><li><strong>Why is it advised to use field numbers below 16 for frequently used fields?</strong></li><li><strong>Why should you avoid changing field numbers once a proto file has been released to production?</strong></li></ol><h4>1. Why Use Field Numbers Below 16 for Frequent Fields?</h4><p>In Protocol Buffers, field numbers play a crucial role in message encoding. Field numbers below 16 are considered “compact” because they require only one byte in the metadata section. This efficiency stems from protobuf’s wire format, where lower field numbers use less space due to shorter varint encoding.</p><ul><li><strong>Performance Advantage</strong>: When fields have numbers between 1 and 15, they consume less bandwidth, which reduces latency. This is particularly advantageous for frequently accessed fields since they benefit from reduced encoding overhead.</li><li><strong>Optimized Storage</strong>: By keeping essential fields within this range, the overall message size is minimized, leading to better performance for high-throughput systems.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*7zUQOsldzODpi-PDnEMmfQ.png" /></figure><p>In our CreateUserRequest message, if name and age are frequently accessed fields, assigning them field numbers like 1 and 2 optimizes the message encoding.</p><h4>2. Why Shouldn’t Field Numbers Be Changed After Production Release?</h4><p>Once a proto file is deployed to production, its field numbers should be considered “locked.” Changing field numbers can cause compatibility issues between services because the field numbers are how gRPC knows which data belongs to which field.</p><ul><li><strong>Backward Compatibility</strong>: If a client and server use different field numbers for the same data, they won’t understand each other’s messages. For example, if age was originally field number 2 but was later changed to 3, a server receiving a message with field number 2 wouldn’t know what data it represents.</li><li><strong>Data Integrity</strong>: Changing field numbers can lead to data corruption, with fields mapping to unintended values. If a field’s meaning changes, the server may misinterpret critical information, causing unexpected behavior or even data loss.</li></ul><h4>How to Handle Changes Safely</h4><p>If you need to add or modify fields, consider these guidelines:</p><ul><li><strong>Add New Fields with Unique Numbers</strong>: When extending messages, assign new field numbers that haven’t been used before. This maintains compatibility with older versions of the message.</li><li><strong>Use Reserved Field Numbers</strong>: If a field is no longer in use, mark its number as reserved in the proto file to prevent accidental reuse.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDdl66uysZZds3VesEa1dg.png" /><figcaption>Illustration shows how field names and numbers can be reserved</figcaption></figure><p>In this article, we explored the key encoding techniques in gRPC and how following best practices for field numbers can improve performance and maintain compatibility. Understanding the efficient binary format of gRPC, from metadata and value sections to the role of field numbers, is essential for building fast, reliable, and scalable microservices.</p><p>With these insights, you’re better equipped to optimize your gRPC messages, ensuring that your inter-service communication remains smooth and efficient. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c2d5e6f7695" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>