<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Avinash Babu on Medium]]></title>
        <description><![CDATA[Stories by Avinash Babu on Medium]]></description>
        <link>https://medium.com/@avinashbabu?source=rss-5786ec2b5010------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jMIQU7IrK4Uc395_riFiBg.png</url>
            <title>Stories by Avinash Babu on Medium</title>
            <link>https://medium.com/@avinashbabu?source=rss-5786ec2b5010------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:38:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avinashbabu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[System Design: Generating Time-Ordered Unique IDs with Twitter Snowflake]]></title>
            <link>https://medium.com/@avinashbabu/system-design-generating-time-ordered-unique-ids-with-twitter-snowflake-bae4ac945915?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/bae4ac945915</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 23:42:03 GMT</pubDate>
            <atom:updated>2026-11-17T23:42:03.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Im0OJrJAo7a_v4VuUp1sTA.png" /></figure><p><em>How do distributed systems generate billions of unique IDs while maintaining approximate chronological ordering?</em></p><p>In my <a href="https://medium.com/@avinashbabu/system-design-building-a-scalable-range-assigner-for-unique-id-generation-fb2b76cee13f?postPublishedType=initial">previous article on Range Assigners</a>, we explored how allocating blocks of IDs to individual services can make unique ID generation highly scalable.</p><p>However, there is one limitation.</p><p><strong>What if we want IDs to be generated in chronological order?</strong></p><p>Imagine two events:</p><ul><li>Event A is created at 10:00:01.</li><li>Event B is created at 10:00:02.</li></ul><p>Ideally, we want ID(A) &lt; ID(B).</p><p>A Range Assigner cannot guarantee this across services. Service B might receive a higher range but generate an ID before Service A uses its lower range.</p><p>Let’s explore how to solve this.</p><h3>Approach 1: Timestamp-Based IDs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5t6Y-akp2jT_cwA9ukaeNg.png" /></figure><p>Our first thought might be to use Unix timestamps.</p><p>Instead of incrementing a counter, we generate IDs based on the current timestamp.</p><p>This makes IDs naturally sortable by time, but introduces a few problems.</p><p><strong>Problem 1: Multiple requests at the same time</strong></p><p>Two processes might generate IDs within the same millisecond.</p><p>Both could receive the same timestamp, resulting in duplicate IDs.</p><p><strong>Problem 2: Clock synchronization</strong></p><p>We could introduce a central timestamp service, but that creates an additional dependency and potentially a single point of failure.</p><p>Alternatively, individual machines could use their local clocks.</p><p>However, clocks across different machines are not perfectly synchronized. One machine might be a few milliseconds ahead of another.</p><p>This means an event generated later could receive a smaller timestamp.</p><p><strong>Problem 3: Scalability</strong></p><p>A centralized timestamp or sequence service could become a bottleneck if every request requires coordination.</p><p>We need an approach that combines timestamps with uniqueness, without coordinating every ID.</p><h3>Approach 2: Twitter Snowflake</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pvk-8dLlIP704IH6lDeGZg.png" /></figure><p>Twitter introduced an elegant solution called <strong>Snowflake</strong>.</p><p>Snowflake generates unique, approximately time-ordered IDs using a 64-bit integer.</p><p>Instead of relying entirely on timestamps, it combines three pieces of information.</p><h4>Understanding the 64-Bit Structure</h4><p><strong>0 | 41-bit Timestamp | 10-bit Worker ID | 12-bit Sequence</strong></p><p><strong>1 bit: Sign</strong></p><p>The first bit remains 0 so the ID is positive when represented as a signed 64-bit integer.</p><p><strong>41 bits: Timestamp</strong></p><p>Stores milliseconds elapsed since a custom epoch.</p><p>This allows generated IDs to be approximately ordered by time.</p><p><strong>10 bits: Worker ID</strong></p><p>Identifies the machine or process generating the ID.</p><p>With 10 bits, we can support:</p><p>(2^{10} = 1,024) unique workers.</p><p>Each worker must have a unique identifier to prevent collisions.</p><p><strong>12 bits: Sequence Number</strong></p><p>Handles multiple IDs generated by the same worker within the same millisecond.</p><p>Each worker can generate:</p><p>(2^{12} = 4,096) IDs per millisecond.</p><p>When the sequence is exhausted, the worker waits for the next millisecond.</p><h4>How Snowflake Works</h4><p>Imagine two services generating IDs.</p><p>Service A has Worker ID 1, and Service B has Worker ID 2.</p><p>Both generate an ID at exactly the same millisecond.</p><ul><li>Service A: Timestamp 1000, Worker 1, Sequence 0</li><li>Service B: Timestamp 1000, Worker 2, Sequence 0</li></ul><p>Even though the timestamps are identical, their IDs are different because their Worker IDs are different.</p><p>If Service A generates another ID within the same millisecond, its sequence increases to 1.</p><p>This gives us uniqueness without requiring a network request for every ID.</p><h4>Why Is Snowflake Scalable?</h4><p>Each worker can generate up to 4,096 IDs per millisecond, or approximately 4.1 million IDs per second.</p><p>With 1,024 workers, the theoretical maximum is approximately <strong>4.19 billion IDs per second</strong>, assuming every worker consistently reaches its maximum capacity.</p><p>Actual throughput depends on the implementation and hardware.</p><p>More importantly, individual workers generate IDs locally.</p><p>There is no central database call for every ID, making Snowflake highly scalable.</p><h4>The 69-Year Limitation</h4><p>There is an interesting limitation in Snowflake’s design.</p><p>It allocates only 41 bits to the timestamp.</p><p>Since Snowflake uses 41 bits to store timestamps in milliseconds, the maximum supported duration is:</p><p><strong>2 pow 41 milliseconds ≈ 2.2 trillion milliseconds</strong></p><p>Converting this into years:</p><p><strong>2.2 trillion ÷ 1,000 ÷ 60 ÷ 60 ÷ 24 ÷ 365.25 ≈ 69.7 years</strong></p><p>This means Snowflake can generate IDs for approximately <strong>69.7 years from its custom epoch</strong> before the timestamp field runs out of available values.</p><p>Twitter’s original Snowflake epoch was November 4, 2010, so its 41-bit timestamp range extends into 2080.</p><p>This is not an immediate scalability problem, but a system using this format needs a migration strategy before its timestamp space is exhausted.</p><p>One option is to redesign the bit allocation or adopt a new ID format. Simply resetting the epoch for the same ID namespace risks breaking ordering and uniqueness guarantees.</p><h4>What Are the Remaining Challenges?</h4><p>Snowflake solves many problems, but introduces a few important considerations.</p><p><strong>Clock drift:</strong> If a worker’s clock moves backwards, it may generate timestamps that were already used. Implementations need safeguards, such as detecting backward jumps and waiting or rejecting requests.</p><p><strong>Worker ID allocation:</strong> Two active workers must not use the same Worker ID. This requires reliable coordination during worker registration and recovery.</p><p><strong>Sequence exhaustion:</strong> A worker that generates more than 4,096 IDs within a millisecond must wait before generating additional IDs.</p><p><strong>Global ordering:</strong> Snowflake does not guarantee that all IDs across all workers are generated in strictly increasing order.</p><p>For example, two workers may generate IDs within the same millisecond. Their numerical order depends on the Worker ID, not the exact order in which the events occurred.</p><h4>Does Snowflake Guarantee Causality?</h4><p>Not necessarily.</p><p>This is an important distinction.</p><p>Suppose Event A triggers Event B on another machine.</p><p>We know A happened before B.</p><p>However, if the machine processing B has a clock running behind the machine processing A, Snowflake could assign B a smaller ID.</p><p><strong>Snowflake provides approximate chronological ordering, not guaranteed causal ordering across machines.</strong></p><p>If causal ordering is a strict requirement, we need additional mechanisms.</p><p>Lamport clocks and Hybrid Logical Clocks can help track causal relationships between distributed events.</p><p>Alternatively, if events only need to be ordered within a particular entity, such as a customer or order, a partitioned event log with per-key ordering may be sufficient.</p><h3>Final Thoughts</h3><p>Range Assigners and Snowflake solve different problems.</p><ul><li><strong>Range Assigner:</strong> Generates unique IDs efficiently, but does not guarantee chronological ordering.</li><li><strong>Snowflake:</strong> Generates unique, approximately time-ordered IDs with minimal coordination.</li><li><strong>Logical clocks or ordered logs:</strong> Help when causal or stronger event ordering guarantees are required.</li></ul><p>For our original requirement of generating 1 billion unique 64-bit IDs per day, Snowflake is an excellent alternative when approximate time ordering is also useful.</p><p>The key takeaway is simple:</p><p><strong>Uniqueness, chronological ordering, and causality are three different requirements.</strong></p><p>Understanding which one your system actually needs is essential before choosing an ID generation strategy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bae4ac945915" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design: Building a Scalable Range Assigner for Unique ID Generation]]></title>
            <link>https://medium.com/@avinashbabu/system-design-building-a-scalable-range-assigner-for-unique-id-generation-fb2b76cee13f?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb2b76cee13f</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 01:02:34 GMT</pubDate>
            <atom:updated>2026-11-17T01:02:34.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dg6D2BgL1ntL56ZCFyUysg.png" /></figure><p><em>How do we generate 1 billion unique IDs every day while ensuring scalability, availability, and performance?</em></p><p>Imagine you’re designing a system that needs to generate unique IDs for billions of records. These could be transaction IDs, order IDs, or event IDs.</p><p>At first, generating unique IDs sounds straightforward. But what happens when thousands of services need IDs simultaneously?</p><p>Let’s explore three approaches and understand why a <strong>Range Assigner</strong> can be an effective solution.</p><h3>Understanding the Requirements</h3><p>Before designing the system, let’s define the non-functional requirements.</p><h3>Non-Functional Requirements</h3><ul><li><strong>ID Size:</strong> 64 bits</li><li><strong>Daily Volume:</strong> 1 billion IDs</li><li><strong>Scalability:</strong> Horizontally scalable</li><li><strong>Availability:</strong> Highly available</li><li><strong>Uniqueness:</strong> No duplicate IDs</li></ul><h3>Capacity Estimation</h3><p>A 64-bit unsigned integer supports approximately <strong>18.4 quintillion unique IDs</strong>, which is more than sufficient for our requirement of 1 billion IDs per day.</p><p>Our expected traffic is:</p><ul><li><strong>Daily:</strong> 1 billion IDs</li><li><strong>Yearly:</strong> 365 billion IDs</li><li><strong>Average throughput:</strong> ~11,574 IDs/second</li></ul><p>At this rate, the unsigned 64-bit ID space would theoretically last approximately <strong>50 million years</strong>.</p><p>So, running out of IDs isn’t our concern. The real challenge is generating them efficiently across distributed services.</p><p><em>Note: If we use a signed 64-bit integer and restrict IDs to positive values, the usable space is approximately (2^{63}-1).</em></p><h3>Approach 1: UUID</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NSsO3oEQIEN19dplXIz3dA.png" /></figure><p>The simplest approach is to generate UUIDs.</p><p>Each service can independently generate identifiers without coordinating with a central system.</p><p><strong>Advantages:</strong></p><ul><li>No central dependency</li><li>Highly scalable</li><li>Easy to implement</li><li>Extremely low collision probability with random UUIDs</li></ul><p><strong>Disadvantages:</strong></p><ul><li>Standard UUIDs require <strong>128 bits</strong>, exceeding our 64-bit requirement.</li><li>Larger identifiers consume more storage and index space.</li><li>Random UUIDs, such as UUIDv4, can reduce B-tree index locality and increase insertion overhead.</li></ul><p>Time-ordered UUIDs, such as UUIDv7, improve index locality, but still require 128 bits.</p><p><strong>Verdict:</strong> A good distributed ID generation solution, but it doesn’t satisfy our 64-bit constraint.</p><h3>Approach 2: Database Auto-Increment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDyTVHefXyQAN5DjKD-qew.png" /></figure><p>Another straightforward solution is to use a database sequence or an auto-incrementing column.</p><p>For example:</p><pre>CREATE TABLE orders (<br>    id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,<br>    details TEXT<br>);</pre><p>Every new record receives a unique ID from the database.</p><p><strong>Advantages:</strong></p><ul><li>Simple implementation</li><li>Guaranteed uniqueness within the sequence</li><li>No custom ID generation service required</li></ul><p><strong>Disadvantages:</strong></p><ul><li>A single database instance can become a bottleneck.</li><li>Every ID generation may require a database interaction.</li><li>A standalone database creates a single point of failure.</li><li>Scaling ID generation independently becomes harder.</li></ul><p>Database sequences can be made highly available and can cache IDs, so these limitations mainly apply to a naive centralized implementation.</p><p><strong>Verdict:</strong> Works well for many applications, but direct centralized allocation may introduce unnecessary coordination at high scale.</p><h3>Approach 3: Range Assigner</h3><p>Instead of requesting a new ID from the database every time, what if we allocated a <strong>range of IDs</strong> to each service?</p><p>This is the core idea behind a Range Assigner.</p><p>A Range Assigner is a service responsible for allocating non-overlapping blocks of IDs to different application instances.</p><p>For example, the Range Assigner allocates:</p><ul><li><strong>Service A:</strong> IDs 1 to 1,000,000</li><li><strong>Service B:</strong> IDs 1,000,001 to 2,000,000</li><li><strong>Service C:</strong> IDs 2,000,001 to 3,000,000</li></ul><p>Each service independently generates IDs within its assigned range. Once the range is exhausted, it requests a new one from the Range Assigner.</p><h4>Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YJeDz2MwK04bhvNJH5Taw.png" /></figure><p>The Range Assigner keeps track of which ranges have already been allocated.</p><p>The application services don’t need to coordinate with one another for individual IDs.</p><h4>How It Works</h4><p><strong>Step 1: Request a range</strong></p><p>Service A requests 1 million IDs from the Range Assigner.</p><p><strong>Step 2: Allocate a unique range</strong></p><p>The Range Assigner atomically reserves the range [1, 1,000,000] in durable storage before returning it.</p><p><strong>Step 3: Generate IDs locally</strong></p><p>Service A maintains a local counter.</p><pre>class IDGenerator:<br>    def __init__(self, start, end):<br>        self.current = start<br>        self.end = end<br><br>def next_id(self):<br>        if self.current &gt; self.end:<br>            raise Exception(&quot;Range exhausted&quot;)<br>        value = self.current<br>        self.current += 1<br>        return value</pre><p>This is a simplified single-threaded example. A production implementation needs an atomic counter or appropriate synchronization.</p><p><strong>Step 4: Request another range</strong></p><p>When Service A approaches the end of its assigned range, it asynchronously requests a new one.</p><p>This allows uninterrupted ID generation during normal operation.</p><h4>Why Is This Scalable?</h4><p>The key advantage of a Range Assigner is that <strong>services don’t need to make a network call for every ID they generate</strong>.</p><p>Let’s use our requirements as an example.</p><p>Our system generates <strong>1 billion IDs per day</strong>, and each range contains <strong>1 million IDs</strong>.</p><p>Assuming all allocated IDs are used, the Range Assigner only needs to handle <strong>1,000 allocation requests per day</strong>, compared to 1 billion requests if every ID were allocated individually.</p><p>That’s a <strong>million-fold reduction in allocation requests</strong>.</p><p>Each application service generates IDs locally using a counter, eliminating network overhead for individual ID generation.</p><p>This makes the system scalable because:</p><ul><li><strong>Reduced coordination:</strong> The Range Assigner is contacted only when a new range is needed.</li><li><strong>Horizontal scaling:</strong> Additional application instances can independently generate IDs from their assigned ranges.</li><li><strong>Low latency:</strong> Generating an ID requires only incrementing a local counter.</li><li><strong>Lower database load:</strong> The metadata store is updated once per range rather than once per ID.</li></ul><p>The Range Assigner’s load is primarily determined by the volume of IDs allocated and the chosen range size, not the number of individual ID generation operations.</p><p>In practice, allocation traffic may be bursty, particularly when many application instances start or scale simultaneously.</p><h4>What About High Availability?</h4><p>There’s an important problem we haven’t addressed.</p><p><strong>What happens if the Range Assigner goes down?</strong></p><p>Services with available IDs can continue generating them. However, services that exhaust their ranges cannot request new ones.</p><p>To make the design highly available, we need:</p><p><strong>Multiple Range Assigner instances</strong></p><p>Run several instances behind a load balancer to avoid depending on a single process.</p><p><strong>Durable, strongly consistent storage</strong></p><p>Every range allocation must be persisted atomically. Multiple Range Assigner instances must never allocate overlapping ranges.</p><p>A transactional database or a consensus-backed metadata store can provide this guarantee.</p><p><strong>Prefetching ranges</strong></p><p>Services can request their next range before exhausting the current one, allowing them to tolerate temporary Range Assigner outages.</p><p><strong>Safe recovery</strong></p><p>If an application crashes, it can discard its remaining IDs and request a fresh range after restarting.</p><p>This may create gaps in the ID sequence, but it avoids reusing previously issued IDs.</p><h4>The important trade-off is that <strong>we guarantee uniqueness, not gap-free numbering</strong>.</h4><h4>Trade-offs and Limitations</h4><p>A Range Assigner is not perfect.</p><ul><li><strong>Gaps in IDs:</strong> Unused ranges may be lost when services crash.</li><li><strong>Central allocation dependency:</strong> New ranges cannot be issued if the durable allocator is unavailable.</li><li><strong>Ordering:</strong> IDs are unique but not necessarily globally ordered by creation time. Service B might generate a larger ID before Service A generates a smaller one.</li><li><strong>Range sizing:</strong> Very small ranges create more coordination overhead, while very large ranges can waste more IDs.</li></ul><p>For workloads that only require uniqueness, these are often acceptable trade-offs.</p><h3>Final Thoughts</h3><p>There are several ways to generate unique IDs in a distributed system.</p><p>UUIDs offer simplicity and decentralized generation, but standard UUIDs require 128 bits.</p><p>Database sequences provide strong uniqueness guarantees, but a naive centralized implementation can introduce coordination overhead.</p><p>A <strong>Range Assigner combines centralized coordination for allocating ranges with decentralized ID generation inside application services</strong>.</p><p>For our requirements of 64-bit IDs, 1 billion IDs per day, scalability, and high availability, this is a strong design choice.</p><p>The underlying principle extends beyond ID generation:</p><p><strong>Coordinate infrequently, and let individual services perform most of their work independently.</strong></p><p>That’s one of the key ideas behind designing scalable distributed systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb2b76cee13f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Kubernetes Works Internally: A Deep Dive into Its Architecture and Components]]></title>
            <link>https://medium.com/@avinashbabu/how-kubernetes-works-internally-a-deep-dive-into-its-architecture-and-components-721b775bb8e2?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/721b775bb8e2</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 00:13:35 GMT</pubDate>
            <atom:updated>2026-11-16T00:13:35.799Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hbwtHgSf6wKjUjXFfQrHCw.png" /></figure><p><em>Understanding the Kubernetes control plane, worker nodes, scheduling, networking, storage, controllers, and how they work together to run applications reliably.</em></p><p>Kubernetes is one of the most widely used platforms for managing containerised applications. Most engineers are familiar with commands such as kubectl apply, kubectl get pods, and kubectl describe.</p><p>But what actually happens behind these commands?</p><p>When you create a Deployment with three replicas, which component creates the Pods? How does Kubernetes decide where to schedule them? What happens when a container crashes, a node becomes unavailable, or an application needs to scale?</p><p>To understand Kubernetes properly, we need to look beyond its API objects and explore how its internal components interact.</p><p>In this article, we’ll explore the major Kubernetes components, their responsibilities, and what happens internally when applications are deployed, scaled, updated, and recovered from failures.</p><h3>Understanding Kubernetes Architecture</h3><p>At a high level, a Kubernetes cluster consists of two major parts:</p><ul><li><strong>Control Plane:</strong> Manages cluster state and makes decisions about scheduling, reconciliation, and resource management.</li><li><strong>Worker Nodes:</strong> Run application workloads and manage their containers.</li></ul><p>A simplified architecture looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_yMKxQ9qyBaAJ9vNWySAw.png" /></figure><p>The most important concept is that <strong>Kubernetes is a declarative, reconciliation-based system</strong>.</p><p>Instead of telling Kubernetes exactly how to perform every operation, we describe the desired state, and Kubernetes continuously works to make the actual state match it.</p><p>For example:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: web-app<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: web-app<br>  template:<br>    metadata:<br>      labels:<br>        app: web-app<br>    spec:<br>      containers:<br>        - name: web<br>          image: nginx:1.27<br>          ports:<br>            - containerPort: 80</pre><p>Here, we’re asking Kubernetes to maintain three replicas of an application.</p><p>If a Pod is deleted, Kubernetes eventually creates a replacement. If we change the replica count to five, Kubernetes creates additional Pods.</p><p>How does this happen?</p><p>Let’s understand the components responsible.</p><h3>Part 1: Kubernetes Control Plane Components</h3><p>The control plane is responsible for managing the cluster.</p><p>Its main components are:</p><ol><li>API Server</li><li>etcd</li><li>Scheduler</li><li>Controller Manager</li><li>Cloud Controller Manager</li></ol><h3>API Server: The Entry Point to Kubernetes</h3><p>The Kubernetes API Server (kube-apiserver) is the central interface through which components interact with cluster resources.</p><p>When you execute:</p><pre>kubectl apply -f deployment.yaml</pre><p>The request goes to the API Server.</p><p>The API Server handles:</p><ul><li>Authentication: Who is making the request?</li><li>Authorization: Is the user allowed to perform this action?</li><li>Admission control: Should the request be accepted or modified?</li><li>API validation and resource processing.</li><li>Reading and writing persistent cluster state.</li><li>Providing watch APIs for controllers and other components.</li></ul><p>For example, suppose you create a Deployment.</p><p>The API Server receives the request, processes it, and persists the Deployment object in etcd.</p><p>It does not directly create application containers.</p><p>Instead, controllers observe the new Deployment through the API Server and begin reconciling it.</p><p><strong>Important:</strong> Kubernetes controllers, the scheduler, and kubelets generally communicate through the API Server rather than accessing etcd directly.</p><p>This centralises API access, validation, and authorisation.</p><h4>What Happens If the API Server Goes Down?</h4><p>Existing application containers can continue running because they don’t require a continuous connection to the API Server.</p><p>However:</p><ul><li>New Kubernetes API requests cannot be processed.</li><li>New Pods generally cannot be scheduled.</li><li>Controllers cannot reliably observe or persist changes.</li><li>Cluster recovery and orchestration become impaired.</li></ul><p>This distinction is important: <strong>control-plane availability is not the same as application availability</strong>.</p><h4>etcd: The Source of Truth</h4><p>etcd is a distributed, strongly consistent key-value database that stores Kubernetes cluster state.</p><p>It stores objects such as:</p><ul><li>Deployments, Pods, and Services.</li><li>ConfigMaps and Secrets.</li><li>Nodes and their recorded status.</li><li>Custom Resources.</li><li>Cluster configuration and metadata.</li></ul><p>When a Deployment is created, its specification is persisted through the API Server into etcd.</p><p>Kubernetes can then retrieve that information to maintain the desired state.</p><h4>How Does etcd Maintain Consistency?</h4><p>etcd uses the Raft consensus algorithm.</p><p>In a typical highly available setup, multiple etcd members cooperate to maintain a consistent view of the database.</p><p>For example, with three etcd members, a majority of two is required to make progress on consensus-backed operations.</p><p>This allows the cluster to tolerate one member failure.</p><p>However, losing quorum prevents etcd from committing new writes, significantly affecting Kubernetes control-plane operations.</p><h4>Why Is etcd Critical?</h4><p>Losing etcd data can mean losing the persistent record of your Kubernetes resources.</p><p>That’s why production clusters require:</p><ul><li>Regular etcd snapshots or provider-managed backups.</li><li>Reliable persistent storage.</li><li>Quorum-aware high availability.</li><li>Tested disaster recovery procedures.</li></ul><p>etcd isn’t an application database. It stores Kubernetes control-plane state.</p><h4>Scheduler: Deciding Where Pods Run</h4><p>The Kubernetes Scheduler (kube-scheduler) is responsible for assigning unscheduled Pods to suitable worker nodes.</p><p>Suppose you create a Deployment with three replicas.</p><p>The controllers create the corresponding Pod objects, but those Pods initially have no node assigned.</p><p>The scheduler watches for these unscheduled Pods.</p><p>It evaluates potential nodes using constraints such as:</p><ul><li>CPU and memory requests.</li><li>Node selectors and node affinity.</li><li>Pod affinity and anti-affinity.</li><li>Taints and tolerations.</li><li>Topology spread constraints.</li><li>Persistent volume topology and binding.</li><li>Other scheduler plugin rules.</li></ul><h4>How Scheduling Works</h4><p>Scheduling can be simplified into two main stages.</p><p><strong>Filtering</strong></p><p>The scheduler eliminates nodes that cannot satisfy the Pod’s requirements.</p><p>For example, a Pod requesting 4 CPU cores cannot be scheduled on a node with only 2 unallocated CPU cores available for requests.</p><p><strong>Scoring</strong></p><p>The scheduler scores feasible nodes using configured scheduling plugins.</p><p>It selects a suitable node and records the binding through the API Server.</p><p>The scheduler doesn’t directly create containers.</p><p>That is the kubelet’s responsibility.</p><h4>CPU Requests vs Actual Usage</h4><p>Consider a node with 8 CPU cores.</p><p>Its actual utilisation might be only 20%, but existing Pods could have requested 7 cores.</p><p>A new Pod requesting 4 CPU cores may remain Pending.</p><p>Why?</p><p>Because Kubernetes scheduling is based primarily on resource requests and allocatable capacity, not live CPU utilisation.</p><p>This prevents Kubernetes from placing workloads whose declared resource requirements exceed the available schedulable capacity.</p><h4>Controller Manager: Maintaining Desired State</h4><p>The Kubernetes Controller Manager (kube-controller-manager) runs multiple controllers that reconcile cluster resources.</p><p>A controller observes the current state and compares it with the desired state.</p><p>When there is a difference, it takes action to reduce that difference.</p><p>Examples include:</p><p><strong>Deployment Controller</strong></p><p>Manages Deployments and coordinates ReplicaSets during rollouts and updates.</p><p><strong>ReplicaSet Controller</strong></p><p>Ensures the desired number of Pods exists.</p><p>If a Deployment requires three replicas and one Pod is deleted, the ReplicaSet controller creates a replacement Pod.</p><p><strong>Node Lifecycle Controller</strong></p><p>Monitors node health and participates in handling unavailable nodes.</p><p><strong>Job Controller</strong></p><p>Ensures that Jobs complete their required work.</p><p><strong>ServiceAccount and Namespace Controllers</strong></p><p>Handle aspects of their respective resource lifecycles.</p><h3>How Reconciliation Works</h3><p>Consider a Deployment configured with three replicas:</p><pre>Desired replicas: 3<br>Actual Pods:      2<br>         |<br>         v<br>ReplicaSet Controller<br>         |<br>         v<br>Create one new Pod<br>         |<br>         v<br>Actual Pods:      3</pre><p>The controller doesn’t need someone to manually request the replacement.</p><p>It continuously works toward the desired state.</p><p>Importantly, reconciliation is generally <strong>eventually consistent</strong>, not instantaneous.</p><h4>Cloud Controller Manager: Connecting Kubernetes to Cloud Providers</h4><p>When Kubernetes runs on AWS, Azure, or Google Cloud, it needs to interact with cloud-specific infrastructure.</p><p>The Cloud Controller Manager (cloud-controller-manager) provides a way to separate cloud-provider integration from core Kubernetes components.</p><p>Depending on the provider and configuration, it handles:</p><ul><li>Cloud node information.</li><li>Node addresses and cloud instance identity.</li><li>Cloud-backed load balancers.</li><li>Cloud-related node lifecycle integration.</li><li>Routing integration where supported.</li></ul><p>For example, when you create a Service of type LoadBalancer, the relevant cloud integration can provision a cloud load balancer.</p><p>The precise implementation varies across EKS, AKS, GKE, and other Kubernetes environments.</p><h3>Part 2: Kubernetes Worker Node Components</h3><p>The control plane decides what should happen.</p><p>Worker nodes are responsible for running the workloads.</p><p>The major node components are:</p><ol><li>Kubelet</li><li>Container Runtime</li><li>kube-proxy</li><li>Container Networking Interface (CNI) plugins</li></ol><h4>Kubelet: Managing Pods on a Node</h4><p>The kubelet is an agent running on each worker node.</p><p>Its primary responsibility is to ensure that containers specified in Pods assigned to that node are running.</p><p>The kubelet:</p><ul><li>Watches for Pods assigned to its node.</li><li>Interacts with the container runtime.</li><li>Starts and stops containers.</li><li>Performs configured health probes.</li><li>Reports Pod and node status.</li><li>Manages container restarts.</li><li>Coordinates volume setup and mounting.</li></ul><h4>What Happens When a Container Crashes?</h4><p>Imagine a Deployment with three Pods.</p><p>One container crashes, but its node remains healthy.</p><p>The kubelet detects the failure through the container runtime and restarts the container according to the Pod’s restart policy.</p><p>If it repeatedly crashes, the kubelet introduces restart delays.</p><p>This is commonly observed as:</p><pre>CrashLoopBackOff</pre><p>The important detail is that <strong>the scheduler isn’t involved in restarting a container inside an existing Pod</strong>.</p><p>The Pod is already assigned to a node.</p><p>The kubelet handles its container lifecycle locally.</p><p>However, if the entire Pod is deleted, the ReplicaSet controller creates a replacement Pod, which the scheduler then assigns.</p><h4>Container Runtime: Running Containers</h4><p>The container runtime is responsible for executing containers.</p><p>Common runtimes include containerd and CRI-O.</p><p>Kubernetes communicates with compatible runtimes through the Container Runtime Interface (CRI).</p><p>The runtime handles activities such as:</p><ul><li>Pulling container images.</li><li>Creating and starting containers.</li><li>Stopping containers.</li><li>Reporting container status.</li><li>Managing container execution through lower-level runtime components.</li></ul><p>For example, when a kubelet needs to start an Nginx container, it uses the CRI to instruct the runtime to prepare and run it.</p><h4>The kubelet decides what should run; the runtime performs the container operations.</h4><h4>kube-proxy: Implementing Service Networking</h4><p>Kubernetes Pods have IP addresses, but Pod identities and addresses can change as workloads are replaced.</p><p>Kubernetes Services provide stable virtual networking endpoints.</p><p>For example:</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: web-service<br>spec:<br>  selector:<br>    app: web-app<br>  ports:<br>    - port: 80<br>      targetPort: 80</pre><p>The Service provides a stable way to reach the matching Pods.</p><p>Traditionally, kube-proxy implements Service forwarding rules on nodes using Linux networking mechanisms such as iptables or IPVS.</p><p>The rules allow traffic directed to a Service IP to reach one of its available backends.</p><p>Modern Kubernetes environments may instead use eBPF-based networking solutions that replace kube-proxy’s Service-routing functionality.</p><p>Therefore, kube-proxy is common but isn’t mandatory in every cluster.</p><h3>CNI: How Pods Get Network Connectivity</h3><p>The Container Networking Interface (CNI) defines a standard interface used by networking plugins to configure container networking.</p><p>Common Kubernetes networking solutions include:</p><ul><li>AWS VPC CNI</li><li>Azure CNI</li><li>Calico</li><li>Cilium</li><li>Flannel</li></ul><p>These solutions differ in how they allocate addresses, route packets, and enforce network policies.</p><p>When a Pod is created, its network environment must be configured so it can communicate with other workloads.</p><p>Depending on the networking implementation, this may involve:</p><ul><li>Configuring network interfaces.</li><li>Assigning Pod IP addresses.</li><li>Setting up routes.</li><li>Connecting Pods to the node network.</li><li>Enforcing network policies.</li></ul><h3>How Pod-to-Pod Communication Works</h3><p>Imagine two Pods on different nodes:</p><pre>Pod A<br>  |<br>  v<br>Node A Networking<br>  |<br>  v<br>Cluster / Cloud Network<br>  |<br>  v<br>Node B Networking<br>  |<br>  v<br>Pod B</pre><p>The CNI networking solution provides the connectivity required for this communication.</p><p>Kubernetes expects Pods to communicate using their Pod IP addresses without requiring application-level NAT between Pods.</p><h3>Part 3: Kubernetes Networking and Service Discovery</h3><p>Networking involves more than just the CNI.</p><p>Other important components include Services, EndpointSlices, CoreDNS, Ingress controllers, and Gateway API implementations.</p><h4>Services and EndpointSlices</h4><p>A Service provides a stable endpoint for a changing set of Pods.</p><p>When Pods are created, removed, or become unavailable, Kubernetes updates EndpointSlice resources representing the relevant backend endpoints.</p><p>Service-routing implementations use this information to direct traffic appropriately.</p><p>For example:</p><pre>        Service: web-service<br>                |<br>         +------+------+<br>         |      |      |<br>         v      v      v<br>       Pod A  Pod B  Pod C</pre><p>If Pod B becomes unready, it is normally removed from the ready set of Service endpoints.</p><p>New Service traffic can then be directed toward healthy backends.</p><p>This is why readiness probes are important.</p><h4>CoreDNS: Service Discovery</h4><p>CoreDNS commonly provides cluster DNS.</p><p>For example, an application might connect to:</p><pre>redis.default.svc.cluster.local</pre><p>CoreDNS resolves that name using Kubernetes Service information.</p><p>Without functioning cluster DNS, applications may struggle to discover dependencies even when network connectivity itself works.</p><p>During troubleshooting, it’s useful to distinguish:</p><ul><li>DNS resolution failures.</li><li>Network routing failures.</li><li>Service endpoint problems.</li><li>Application connection failures.</li></ul><p>These can produce similar symptoms while having very different causes.</p><h4>Ingress and Gateway API</h4><p>A Service makes workloads accessible within Kubernetes, or externally depending on the Service type.</p><p>Ingress and Gateway API resources define higher-level traffic-routing requirements.</p><p>They require corresponding controllers or implementations to configure the actual networking components.</p><p>Examples include:</p><ul><li>NGINX Ingress Controller.</li><li>AWS Load Balancer Controller.</li><li>Envoy-based Gateway implementations.</li></ul><p>These components can provide capabilities such as HTTP routing, TLS termination, and traffic management.</p><h3>Part 4: Kubernetes Workload Controllers</h3><p>Kubernetes offers different workload types because not every application has the same operational requirements.</p><h4>Deployments and ReplicaSets</h4><p>Deployments are commonly used for stateless applications.</p><p>They provide:</p><ul><li>Declarative replica management.</li><li>Rolling updates.</li><li>Rollback to previous Deployment revisions.</li><li>Integration with ReplicaSets.</li></ul><p>A Deployment manages ReplicaSets, and ReplicaSets manage the number of Pods.</p><pre>Deployment<br>    |<br>    v<br>ReplicaSet<br>    |<br>    +----&gt; Pod A<br>    +----&gt; Pod B<br>    +----&gt; Pod C</pre><p>When you update an application image, the Deployment controller typically creates a new ReplicaSet and gradually transitions Pods from the old version to the new version.</p><h4>StatefulSets</h4><p>StatefulSets are designed for workloads requiring stable identities and persistent storage.</p><p>Common examples include:</p><ul><li>Redis</li><li>PostgreSQL</li><li>Kafka</li><li>Elasticsearch</li></ul><p>A StatefulSet provides:</p><ul><li>Stable Pod names and ordinal identities.</li><li>Stable network identities when configured with a governing Service.</li><li>Persistent storage associations through volume claim templates.</li><li>Ordered creation, scaling, and rolling updates by default.</li></ul><p>For example:</p><pre>redis-0 -&gt; redis-data-redis-0<br>redis-1 -&gt; redis-data-redis-1<br>redis-2 -&gt; redis-data-redis-2</pre><p>If redis-0 is recreated, it retains the redis-0 identity and normally reconnects to its existing persistent volume claim.</p><p>However, StatefulSets do not automatically provide database replication, primary election, or failover.</p><p>These remain responsibilities of the database system or an operator.</p><h4>DaemonSets</h4><p>A DaemonSet ensures that a Pod runs on each eligible node, or on a selected subset of nodes.</p><p>Common use cases include:</p><ul><li>Log collection agents.</li><li>Monitoring agents.</li><li>CNI-related node agents.</li><li>Security monitoring.</li></ul><p>When a new eligible node joins the cluster, the DaemonSet controller ensures that the necessary Pod is created for it.</p><h4>Jobs and CronJobs</h4><p>Jobs are used for finite workloads that should run to completion.</p><p>Examples include database migrations, batch processing, and data exports.</p><p>CronJobs schedule Jobs based on a recurring time specification.</p><p>For example:</p><pre>apiVersion: batch/v1<br>kind: CronJob<br>metadata:<br>  name: daily-report<br>spec:<br>  schedule: &quot;0 2 * * *&quot;<br>  jobTemplate:<br>    spec:<br>      template:<br>        spec:<br>          restartPolicy: OnFailure<br>          containers:<br>            - name: report<br>              image: example/report:v1</pre><p>This creates a scheduled workload that runs according to the configured schedule.</p><h3>Part 5: Kubernetes Storage and Persistent Volumes</h3><p>Containers have ephemeral writable storage by default.</p><p>If an application needs data to survive Pod replacement, it typically uses persistent storage.</p><p>Kubernetes provides several related abstractions.</p><p><strong>PersistentVolume (PV):</strong> Represents a piece of persistent storage available in the Kubernetes cluster.</p><p><strong>PersistentVolumeClaim (PVC):</strong> Represents a request for storage by a workload. It specifies requirements such as storage size and access modes.</p><p><strong>StorageClass:</strong> Defines how storage should be dynamically provisioned, including the provisioner, storage parameters, and volume binding behaviour.</p><p><strong>Container Storage Interface (CSI) Driver:</strong> Integrates Kubernetes with external storage providers, such as AWS EBS, Azure Disk, and Google Persistent Disk, enabling volume provisioning, attachment, and mounting.</p><h4>How Dynamic Storage Provisioning Works</h4><p>Suppose a Pod references a PVC requesting 100 GiB.</p><p>The typical flow is:</p><ol><li>A PVC is created.</li><li>Kubernetes identifies the appropriate StorageClass.</li><li>The external CSI provisioner arranges for a suitable volume to be created.</li><li>A PV is associated with the claim.</li><li>Kubernetes schedules the Pod with the required storage constraints.</li><li>CSI components arrange attachment and mounting when necessary.</li></ol><p>In AWS, an EBS CSI driver can provision EBS volumes.</p><h4>Availability Zone Constraints</h4><p>An AWS EBS volume is tied to an Availability Zone.</p><p>If the volume exists in zone A, a Pod that uses it generally cannot be scheduled onto a node in zone B.</p><p>Kubernetes accounts for volume topology constraints during scheduling.</p><p>If no suitable node exists in the required zone, the Pod may remain Pending.</p><p>This is particularly important when designing stateful workloads across multiple availability zones.</p><h3>Part 6: Kubernetes Resource Management</h3><p>Kubernetes uses resource requests and limits to manage workloads.</p><p>Consider:</p><pre>resources:<br>  requests:<br>    cpu: &quot;500m&quot;<br>    memory: &quot;512Mi&quot;<br>  limits:<br>    cpu: &quot;2&quot;<br>    memory: &quot;1Gi&quot;</pre><p>Requests and limits serve different purposes.</p><h4>Resource Requests</h4><p>Requests tell the scheduler how much CPU and memory a container needs for placement decisions.</p><p>For example, a Pod requesting 2 CPU cores requires a node with sufficient unallocated CPU request capacity.</p><p>The scheduler doesn’t simply use current CPU utilisation to make this decision.</p><h4>CPU Limits</h4><p>CPU limits are enforced through Linux CPU scheduling mechanisms.</p><p>If an application attempts to consume more CPU than its configured limit, it can be throttled.</p><p>For example:</p><pre>CPU request: 0.5 cores<br>CPU limit:   2 cores<br>Demand:      3 cores<br>Result:      CPU throttling</pre><p>The container continues running, but its performance may be affected.</p><p>Exceeding a CPU limit does not directly cause an OOM kill.</p><h4>Memory Limits</h4><p>Memory limits behave differently.</p><p>If a container consumes memory beyond its limit and memory cannot be reclaimed, the kernel may OOM-kill the container.</p><p>For example:</p><pre>Memory request: 512 MiB<br>Memory limit:   1 GiB<br>Demand:         1.5 GiB<br>Result:         Potential OOM kill</pre><p>If the restart policy permits, kubelet restarts the container.</p><p>Repeated OOM kills can cause CrashLoopBackOff.</p><p>There is also a separate mechanism for node memory pressure, where kubelet may evict Pods to protect node stability.</p><h3>Part 7: Scaling and High Availability</h3><p>Kubernetes provides multiple mechanisms for scaling workloads and infrastructure.</p><h4>Horizontal Pod Autoscaler (HPA)</h4><p>HPA adjusts the number of Pod replicas based on observed metrics.</p><p>For example, it can scale a Deployment based on CPU utilisation.</p><pre>Current replicas: 3<br>CPU utilisation: 85%<br>Target:          60%<br>        |<br>        v<br>HPA calculates desired replicas<br>        |<br>        v<br>Deployment scaled upward</pre><p>The HPA controller changes the target workload’s desired replica count.</p><p>The Deployment and ReplicaSet controllers create the additional Pods, and the scheduler places them.</p><p>HPA does not directly create containers.</p><h4>Vertical Pod Autoscaler (VPA)</h4><p>VPA recommends or, depending on its configuration, adjusts CPU and memory requests for workloads.</p><p>This is useful when applications have poorly sized resource requests.</p><p>Certain VPA update strategies can require Pod recreation, making rollout behaviour important for production workloads.</p><h4>Node Autoscaling</h4><p>HPA scales Pods, but additional Pods may require more node capacity.</p><p>Node autoscalers such as Cluster Autoscaler and Karpenter can provision or manage capacity in supported environments.</p><p>For example:</p><pre>HPA requests more replicas<br>             |<br>             v<br>New Pods remain Pending<br>             |<br>             v<br>Node autoscaler detects capacity demand<br>             |<br>             v<br>Additional node capacity provisioned<br>             |<br>             v<br>Scheduler assigns Pending Pods</pre><p>Pod autoscaling and node autoscaling solve different problems but frequently work together.</p><h4>Pod Disruption Budgets</h4><p>A Pod Disruption Budget (PDB) controls how many healthy Pods can be voluntarily disrupted through the Kubernetes Eviction API.</p><p>For example:</p><pre>apiVersion: policy/v1<br>kind: PodDisruptionBudget<br>metadata:<br>  name: web-pdb<br>spec:<br>  minAvailable: 2<br>  selector:<br>    matchLabels:<br>      app: web-app</pre><p>For a Deployment with three healthy replicas, this allows one voluntary disruption while preserving at least two healthy Pods.</p><p>This is useful during node drains and infrastructure maintenance.</p><p>However, <strong>a PDB does not prevent unexpected node failures</strong>.</p><p>It also doesn’t guarantee zero downtime or prevent every type of application disruption.</p><h3>Part 8: What Happens When a Node Fails?</h3><p>Let’s consider a Deployment with three replicas.</p><p>One worker node suddenly becomes unreachable.</p><p>What happens internally?</p><h4>Node Failure Detection</h4><p>The kubelet normally reports node health through status updates and Node Leases.</p><p>When communication stops, the node lifecycle controller eventually identifies the node as unhealthy.</p><p>The node may transition to NotReady or Unknown.</p><h4>Node Taints and Pod Eviction</h4><p>Kubernetes applies appropriate taints to unavailable nodes.</p><p>Ordinary Pods receive default tolerations for certain node failure conditions, commonly lasting 300 seconds.</p><p>After the applicable toleration expires, Kubernetes can initiate Pod eviction, subject to node lifecycle behaviour.</p><h4>Replica Replacement</h4><p>As affected Pods are removed or marked for deletion, workload controllers can create replacement Pods.</p><p>The scheduler then selects healthy nodes for those new Pods.</p><p>The kubelets on those nodes start the containers.</p><p>One important detail: <strong>Kubernetes cannot necessarily stop containers on an unreachable node</strong>.</p><p>During a network partition, containers might still be running even though the control plane cannot communicate with them.</p><p>This creates special risks for stateful workloads, which need safeguards against duplicate writers and split-brain scenarios.</p><h3>Part 9: Kubernetes Operators and Custom Controllers</h3><p>Built-in Kubernetes controllers manage standard resources such as Deployments and StatefulSets.</p><p>But what if we want Kubernetes to manage a custom application or cloud resource?</p><p>This is where Custom Resource Definitions (CRDs) and operators become useful.</p><h4>Custom Resource Definitions</h4><p>A CRD extends the Kubernetes API with a custom resource type.</p><p>For example, we might define:</p><pre>apiVersion: databases.example.com/v1<br>kind: RedisCluster<br>metadata:<br>  name: production-redis<br>spec:<br>  replicas: 3<br>  version: &quot;7&quot;</pre><p>Kubernetes now stores and exposes this resource through its API.</p><p>However, defining a CRD alone doesn’t create a Redis cluster.</p><p>We need a controller to implement its desired behaviour.</p><h4>Reconciliation Loops</h4><p>A custom controller watches resources and reconciles their desired state with the actual state.</p><p>A simplified reconciliation process looks like this:</p><pre>RedisCluster Custom Resource<br>             |<br>             v<br>       Operator watches<br>             |<br>             v<br>     Read desired state<br>             |<br>             v<br>     Read actual state<br>             |<br>             v<br>        Compare<br>             |<br>      +------+------+<br>      |             |<br>   In Sync      Difference<br>      |             |<br>      v             v<br>   No action    Reconcile<br>                    |<br>                    v<br>             Update status</pre><p>A production-grade operator must consider:</p><ul><li>Idempotency.</li><li>Retries and exponential backoff.</li><li>Eventual consistency.</li><li>Finalizers for deletion cleanup.</li><li>Status conditions and observed generations.</li><li>Recovery from partial failures.</li><li>Avoiding conflicting writers.</li><li>Leader election when appropriate.</li></ul><p>For example, a Redis operator might automate cluster provisioning, configuration, failover, and upgrades.</p><p>This is how Kubernetes can be extended into a platform for managing complex systems.</p><h3>Part 10: Security, Admission, and Access Control</h3><p>Kubernetes security is enforced at several layers.</p><h4>Authentication and Authorization</h4><p>When a user or workload sends a request to the API Server, Kubernetes authenticates the requester and checks whether the action is permitted.</p><p>Role-Based Access Control (RBAC) is a common authorisation mechanism.</p><p>For example, a Role might allow reading Pods in a namespace without allowing their deletion.</p><h4>Admission Controllers</h4><p>After authentication and authorization, admission controllers can validate or modify API requests.</p><p>Examples include:</p><ul><li>Validating and mutating admission webhooks.</li><li>ResourceQuota.</li><li>LimitRanger.</li><li>Pod Security Admission.</li></ul><p>Admission controls help enforce cluster-wide rules before resources are persisted.</p><p>For example, a webhook could reject a Pod that doesn’t meet required security standards.</p><h4>Network Policies</h4><p>Network policies define allowed traffic between Pods and other network endpoints.</p><p>They depend on a networking implementation that supports enforcement.</p><p>For example, a policy might allow application Pods to access Redis on port 6379 while denying that access from unrelated namespaces.</p><p>Network policy support and enforcement behaviour depend on the CNI solution.</p><h3>Part 11: Monitoring and Observability</h3><p>Operating Kubernetes reliably requires visibility into both the control plane and workloads.</p><p>Common components and technologies include:</p><p><strong>Metrics Server:</strong> Provides resource metrics used by mechanisms such as the Horizontal Pod Autoscaler.</p><p><strong>Prometheus:</strong> Collects metrics from applications, nodes, and Kubernetes components.</p><p><strong>Grafana:</strong> Visualises metrics and operational dashboards.</p><p><strong>OpenTelemetry:</strong> Provides a framework for collecting and exporting telemetry such as traces, metrics, and logs.</p><p><strong>Logging agents:</strong> Collect container and node logs for centralised analysis.</p><p>These tools are not all built into Kubernetes by default, but they are commonly used in production environments.</p><h3>Important Metrics to Monitor</h3><p>At the cluster level:</p><ul><li>API Server request latency and error rates.</li><li>etcd health and latency.</li><li>Pending and unschedulable Pods.</li><li>Node readiness.</li><li>CPU and memory pressure.</li><li>Pod restarts and OOM kills.</li><li>Scheduler performance.</li></ul><p>At the application level:</p><ul><li>Request latency.</li><li>Error rates.</li><li>Traffic throughput.</li><li>Saturation.</li><li>Dependency failures.</li></ul><p>For production reliability, monitoring should focus not just on whether Pods are running, but whether applications are actually serving users successfully.</p><h3>Part 12: Putting Everything Together</h3><p>Now let’s revisit our original question:</p><p><strong>What happens internally when we create a Deployment with three replicas?</strong></p><p>Here is the end-to-end flow.</p><h4>Step 1: The User Creates a Deployment</h4><pre>kubectl apply -f deployment.yaml</pre><p>The request reaches the API Server.</p><h4>Step 2: The API Server Processes the Request</h4><p>The API Server authenticates and authorises the request, processes admission requirements, validates the resource, and persists it in etcd.</p><h4>Step 3: The Deployment Controller Reconciles</h4><p>The Deployment controller observes the new Deployment and creates a ReplicaSet representing the desired Pod template.</p><h4>Step 4: The ReplicaSet Controller Creates Pods</h4><p>The ReplicaSet controller observes that three replicas are required.</p><p>It creates three Pod objects.</p><p>At this point, the Pods may still be unscheduled.</p><h4>Step 5: The Scheduler Assigns Nodes</h4><p>The scheduler evaluates node resources and scheduling constraints.</p><p>It selects appropriate nodes and records the Pod bindings.</p><h4>Step 6: Kubelets Start the Containers</h4><p>Each node’s kubelet observes the Pods assigned to it.</p><p>It coordinates volume preparation, networking setup, image pulling, and container startup through the relevant runtime and infrastructure plugins.</p><h4>Step 7: Networking Becomes Available</h4><p>The Pods receive network configuration through the CNI-based networking system.</p><p>If a Service selects these Pods, Kubernetes maintains the relevant EndpointSlice information.</p><p>Once Pods become ready, they can receive Service traffic.</p><h4>Step 8: Controllers Continue Reconciliation</h4><p>Kubernetes doesn’t stop after successful deployment.</p><p>Controllers and kubelets continuously respond to changes.</p><p>If a container crashes, the kubelet can restart it.</p><p>If a Pod is deleted, the ReplicaSet controller creates a replacement.</p><p>If replicas are increased, additional Pods are created and scheduled.</p><p>If a node fails, node lifecycle handling and workload reconciliation work toward restoring the desired state.</p><p>If an HPA changes the replica count, the workload controllers act on the new desired value.</p><p>This continuous reconciliation model is at the heart of Kubernetes.</p><h3>Final Thoughts</h3><p>Kubernetes is not a single orchestrator that performs every action.</p><p>It’s a collection of specialised components that collaborate through a shared API and continuously reconcile the system toward its desired state.</p><p>The API Server exposes and validates cluster operations. etcd stores cluster state. Controllers maintain desired resources. The scheduler assigns Pods to nodes. Kubelets and container runtimes execute workloads. Networking and storage components provide the infrastructure those workloads depend on.</p><p>Together, these components make Kubernetes capable of running and recovering distributed applications at scale.</p><p>The most useful question to ask when debugging Kubernetes isn’t simply:</p><p><em>Why is this Pod failing?</em></p><p>It’s:</p><p><strong>Which component is responsible for the behaviour I’m observing, what state does it see, and what is preventing it from reaching the desired state?</strong></p><p>Once you can answer that, Kubernetes troubleshooting becomes much more systematic.</p><h3>References</h3><ul><li><a href="https://kubernetes.io/docs/concepts/architecture/">Kubernetes: Cluster Architecture</a></li><li><a href="https://kubernetes.io/docs/concepts/overview/components/">Kubernetes: Components</a></li><li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/">Kubernetes: Scheduling, Preemption and Eviction</a></li><li><a href="https://kubernetes.io/docs/concepts/architecture/controller/">Kubernetes: Controllers</a></li><li><a href="https://kubernetes.io/docs/concepts/services-networking/">Kubernetes: Services, Load Balancing and Networking</a></li><li><a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">Kubernetes: Persistent Volumes</a></li><li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">Kubernetes: StatefulSets</a></li><li><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Kubernetes: Custom Resources</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=721b775bb8e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing Production-Ready Infrastructure with Terraform]]></title>
            <link>https://medium.com/@avinashbabu/designing-production-ready-infrastructure-with-terraform-4428a44eec18?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/4428a44eec18</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 22:05:30 GMT</pubDate>
            <atom:updated>2026-11-14T22:05:30.155Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYP03pssTsrHpzIkd_-vtg.png" /></figure><p><em>How to structure Terraform for multiple environments, automate deployments with CI/CD, manage secrets, integrate GitOps, and safely operate cloud infrastructure at scale.</em></p><p>Terraform makes infrastructure provisioning straightforward. You write a configuration, run terraform apply, and resources are created.</p><p>But running Terraform in production is a different challenge.</p><p>What happens when your company operates hundreds of Kubernetes clusters across multiple AWS accounts and regions? How do you prevent an accidental change in development from affecting production? How do you safely deploy infrastructure changes without engineers manually running Terraform?</p><p>This is where <strong>production architecture</strong> becomes important.</p><p>The goal isn’t just to automate infrastructure creation. It’s to build a system that is secure, repeatable, observable, and resilient to failures.</p><h3>Separating Infrastructure Across Environments</h3><p>Imagine a company operating three environments:</p><ul><li>Development</li><li>Staging</li><li>Production</li></ul><p>One approach is to manage everything using a single Terraform configuration and state file.</p><p>While this might work initially, it creates a significant risk: <strong>a single incorrect change can affect multiple environments.</strong></p><p>A better approach is to isolate infrastructure using separate configurations and state files.</p><p>For example:</p><pre>terraform/<br>├── modules/<br>│   ├── networking/<br>│   ├── eks/<br>│   └── database/<br>│<br>└── environments/<br>    ├── dev/<br>    │   ├── main.tf<br>    │   └── backend.tf<br>    ├── staging/<br>    │   ├── main.tf<br>    │   └── backend.tf<br>    └── prod/<br>        ├── main.tf<br>        └── backend.tf</pre><p>Each environment references reusable Terraform modules but maintains its own backend and state.</p><p>This provides three important benefits:</p><p><strong>Isolation:</strong> A Terraform operation targeting development cannot directly modify production resources managed by a different state and properly scoped credentials.</p><p><strong>Security:</strong> Production can use stricter IAM permissions and approval requirements.</p><p><strong>Controlled promotion:</strong> Infrastructure changes can be tested in development and staging before reaching production.</p><p>For larger systems, it’s also worth separating networking, databases, and compute infrastructure into different state files.</p><p>Why?</p><p>A database might change only a few times per year, while compute infrastructure changes daily. Managing them separately reduces the blast radius of deployments.</p><h3>Reusable Modules: Avoiding Infrastructure Duplication</h3><p>Suppose your company needs EKS clusters in 20 AWS regions.</p><p>Writing the same Terraform configuration 20 times would create unnecessary duplication and maintenance overhead.</p><p>Instead, we can build a reusable module.</p><pre>module &quot;eks&quot; {<br>  source = &quot;../../modules/eks&quot;</pre><pre>  cluster_name = &quot;production-eu&quot;<br>  region       = &quot;eu-west-2&quot;<br>  node_count   = 5<br>}</pre><p>The same module can be reused in other regions by changing its inputs.</p><p>The benefits include:</p><ul><li>Consistent infrastructure across environments.</li><li>Centralised security and configuration standards.</li><li>Easier testing and maintenance.</li><li>Faster infrastructure provisioning.</li></ul><p>However, reusability introduces another challenge: changes to shared modules can affect multiple environments.</p><p><strong>Production deployments should use versioned modules rather than automatically consuming the latest changes.</strong></p><p>For example, a module sourced from a Git repository can be pinned to a reviewed release:</p><pre>module &quot;eks&quot; {<br>  source = &quot;git::https://github.com/example/terraform-eks.git?ref=v2.1.0&quot;<br>}</pre><p>This allows infrastructure teams to test new module versions before rolling them out more broadly.</p><p>For critical shared modules, pinning to immutable commit hashes provides stronger reproducibility than relying on mutable tags.</p><h3>Remote State: Protecting Infrastructure Metadata</h3><p>Terraform uses state to track the infrastructure it manages.</p><p>In a team environment, storing state locally is risky. Engineers could have inconsistent state files, and losing a local file could complicate recovery.</p><p>A common production approach is using an S3 backend.</p><pre>terraform {<br>  backend &quot;s3&quot; {<br>    bucket       = &quot;company-terraform-state&quot;<br>    key          = &quot;prod/networking/terraform.tfstate&quot;<br>    region       = &quot;eu-west-2&quot;<br>    encrypt      = true<br>    use_lockfile = true<br>  }<br>}</pre><p>This configuration enables remote state storage and S3-based state locking.</p><p>For production, we should also enable:</p><ul><li>S3 bucket versioning for state recovery.</li><li>Encryption at rest.</li><li>Least-privilege IAM permissions.</li><li>Audit logging for state access.</li><li>Restricted access to production state.</li></ul><p>State locking is especially important because it prevents two Terraform operations from modifying the same state simultaneously.</p><p>However, it doesn’t prevent someone from manually modifying AWS resources.</p><p>That requires additional security controls and drift detection.</p><h3>Building a Safe Terraform CI/CD Pipeline</h3><p>Engineers manually running terraform apply from their laptops isn&#39;t a scalable production workflow.</p><p>A better approach is to manage infrastructure changes through version control and CI/CD.</p><p>A typical deployment process looks like this:</p><pre>Developer opens Pull Request<br>            |<br>            v<br>    Format &amp; Validate<br>            |<br>            v<br>    Security &amp; Policy Checks<br>            |<br>            v<br>      Terraform Plan<br>            |<br>            v<br>       Code Review<br>            |<br>            v<br>      Merge to Main<br>            |<br>            v<br>  Generate Deployable Plan<br>            |<br>            v<br>    Production Approval<br>            |<br>            v<br>      Terraform Apply<br>            |<br>            v<br>  Post-Deployment Validation</pre><p>This pipeline provides traceability and prevents arbitrary infrastructure changes.</p><h3>Planning and Applying Changes</h3><p>A CI/CD pipeline can generate a saved Terraform plan:</p><pre>terraform plan -out=tfplan</pre><p>After approval, the pipeline applies that exact plan:</p><pre>terraform apply tfplan</pre><p>This helps ensure Terraform executes the changes that were reviewed.</p><p>However, saved plans must be managed carefully:</p><ul><li>They can contain sensitive information.</li><li>They can become stale when state changes.</li><li>They must match the configuration and provider versions used during planning.</li></ul><p>In a pull-request workflow, I would generally use speculative plans for review, then generate the deployable plan from the approved commit after merging.</p><p>Production approvals should apply to the actual deployable plan, not only an earlier pull-request preview.</p><h3>Managing Secrets and Credentials Securely</h3><p>Terraform often requires credentials to create infrastructure.</p><p>For example, provisioning cloud resources or databases may involve AWS credentials, API tokens, and database passwords.</p><p>These should never be hardcoded into Terraform files.</p><p>Instead, use short-lived credentials through mechanisms such as AWS IAM roles and CI/CD OpenID Connect (OIDC) federation.</p><p>For runtime secrets, services such as AWS Secrets Manager or HashiCorp Vault can help manage access and rotation.</p><p>There’s also an important Terraform detail:</p><pre>variable &quot;database_password&quot; {<br>  type      = string<br>  sensitive = true<br>}</pre><p>Marking a variable as sensitive hides its value from normal CLI output.</p><p><strong>It doesn’t automatically prevent the value from being stored in Terraform state or plan files.</strong></p><p>Modern Terraform versions also support ephemeral values and provider-specific write-only arguments, which can prevent supported secret values from being persisted.</p><p>A production design must protect state and plan artifacts as carefully as other sensitive infrastructure data.</p><h3>Terraform and GitOps: Where Do They Fit?</h3><p>Terraform and GitOps are often used together, but they solve different problems.</p><p>Terraform is particularly useful for provisioning cloud infrastructure:</p><ul><li>VPCs and networking.</li><li>IAM roles and policies.</li><li>EKS clusters.</li><li>Load balancers.</li><li>Managed cloud databases.</li></ul><p>GitOps tools such as FluxCD and Argo CD are commonly used to manage Kubernetes workloads and configuration continuously.</p><p>A typical architecture might look like this:</p><pre>     Infrastructure Git Repository<br>                    |<br>                    v<br>             Terraform CI/CD<br>                    |<br>                    v<br>          AWS Infrastructure<br>          (VPC, IAM, EKS)<br>                    |<br>                    v<br>           Kubernetes Cluster<br>                    |<br>                    v<br>          FluxCD / Argo CD<br>                    ^<br>                    |<br>        Application Git Repository<br>                    |<br>                    v<br>         Kubernetes Manifests<br>           Helm / Kustomize</pre><p>Terraform provisions the underlying infrastructure.</p><p>GitOps controllers continuously compare the desired Kubernetes configuration with the actual cluster state and reconcile differences.</p><p>For example, suppose someone manually changes a Kubernetes Deployment’s replica count.</p><p>If that field is managed by FluxCD, the controller can detect the difference and restore the desired value.</p><p>Terraform, by contrast, typically reconciles infrastructure during explicitly triggered operations.</p><p>One architectural principle is especially important:</p><p><strong>Avoid having Terraform and GitOps independently manage the same resource fields.</strong></p><p>If both systems attempt to control the same Kubernetes objects, they may repeatedly overwrite each other’s changes.</p><p>Establish clear ownership boundaries between infrastructure provisioning and workload management.</p><h3>Safely Rolling Out Changes Across Hundreds of Clusters</h3><p>Imagine upgrading the networking configuration of 200 Kubernetes clusters.</p><p>Applying the change everywhere simultaneously could introduce a widespread outage.</p><p>A safer approach is progressive rollout.</p><pre> Development<br>     |<br>     v<br> Test Cluster<br>     |<br>     v<br>Canary Production Cluster<br>     |<br>     v<br>Small Production Batch<br>     |<br>     v<br>Remaining Clusters</pre><p>Each stage should have clear success criteria before continuing.</p><p>For example:</p><ul><li>Kubernetes nodes remain healthy.</li><li>Critical workloads are available.</li><li>Network error rates remain within acceptable limits.</li><li>Control-plane latency doesn’t regress.</li><li>Operational alerts remain within expected thresholds.</li></ul><p>If a rollout fails, the pipeline should stop further deployments.</p><p>Recovery may involve reverting a configuration, applying a corrective change, or using a provider-specific recovery procedure.</p><p>Unlike application deployments, infrastructure changes are not always easily reversible. Replacing a database or changing networking resources may require explicit migration and recovery planning.</p><p>That’s why <strong>progressive rollouts and failure detection matter more than simply making Terraform apply faster.</strong></p><h3>Operational Safety: Design for Failure</h3><p>Even a well-designed Terraform pipeline can fail.</p><p>Cloud APIs can time out, permissions can change, quotas can be exhausted, and operations can partially succeed.</p><p>A production architecture should account for these failures.</p><p>Consider a Terraform deployment that successfully creates a VPC and EKS cluster but fails while creating an RDS database.</p><p>Terraform doesn’t automatically roll back the successful resources.</p><p>Recovery requires inspecting the actual infrastructure, confirming the recorded state, fixing the underlying issue, and generating a fresh plan.</p><p>The same principle applies at larger scale.</p><p>A mature infrastructure platform should support:</p><p><strong>Drift detection:</strong> Identify unexpected differences between declared and actual infrastructure.</p><p><strong>Auditability:</strong> Know who approved a change, which pipeline executed it, and what resources were modified.</p><p><strong>Recovery:</strong> Protect state, document failure procedures, and maintain backups for critical infrastructure.</p><p><strong>Observability:</strong> Monitor deployment success rates, failure patterns, infrastructure health, and configuration changes.</p><p><strong>Blast-radius control:</strong> Limit each deployment to a manageable set of resources, accounts, or regions.</p><p>Production reliability comes from designing systems that fail safely, not assuming deployments will always succeed.</p><h3>Final Thoughts</h3><p>Terraform is easy to start with, but production infrastructure requires much more than writing HCL files.</p><p>A well-designed Terraform architecture should:</p><ul><li>Isolate environments and limit deployment blast radius.</li><li>Reuse well-tested, versioned infrastructure modules.</li><li>Protect remote state with encryption, access controls, and locking.</li><li>Deploy through reviewed and auditable CI/CD pipelines.</li><li>Use short-lived credentials and secure secret handling.</li><li>Integrate with GitOps using clear ownership boundaries.</li><li>Roll out infrastructure changes progressively.</li><li>Make failures observable and recovery predictable.</li></ul><p>The real measure of infrastructure automation isn’t how quickly we can create resources.</p><p><strong>It’s how safely and consistently we can change them while real systems depend on them.</strong></p><h3>References</h3><ul><li><a href="https://www.hashicorp.com/en/blog/structuring-hashicorp-terraform-configuration-for-production">HashiCorp: Structuring Terraform Configuration for Production</a></li><li><a href="https://developer.hashicorp.com/terraform/enterprise/workspaces/best-practices">HashiCorp: Workspace Best Practices</a></li><li><a href="https://developer.hashicorp.com/terraform/tutorials/automation/automate-terraform">HashiCorp: Automating Terraform</a></li><li><a href="https://developer.hashicorp.com/terraform/language/backend/s3">HashiCorp: S3 Backend and State Locking</a></li><li><a href="https://developer.hashicorp.com/terraform/language/manage-sensitive-data">HashiCorp: Managing Sensitive Data</a></li><li><a href="https://github.com/open-gitops/documents/blob/main/PRINCIPLES.md">OpenGitOps: GitOps Principles</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4428a44eec18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Terraform Execution Lifecycle: From Init to Destroy]]></title>
            <link>https://medium.com/@avinashbabu/understanding-the-terraform-execution-lifecycle-from-init-to-destroy-3e9ab38d5883?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e9ab38d5883</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Sat, 14 Nov 2026 00:18:53 GMT</pubDate>
            <atom:updated>2026-11-14T00:18:53.573Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Me6I6sawWhjwQSbLXvrUgA.png" /></figure><p><em>An easy-to-follow guide to how Terraform plans, creates, updates, and destroys infrastructure, and what happens behind the scenes.</em></p><p>Infrastructure as Code (IaC) has changed how we provision and manage cloud infrastructure. Instead of manually creating resources through AWS, Azure, or GCP consoles, we define our infrastructure in code and let tools like Terraform manage it.</p><p>But have you ever wondered what actually happens when you run terraform apply?</p><p>Terraform does more than execute API calls. It maintains infrastructure state, identifies dependencies, detects configuration changes, and determines the operations required to reach the desired state.</p><p>Let’s understand the complete Terraform execution lifecycle.</p><h3>How Terraform Works</h3><p>Terraform follows a <strong>declarative approach</strong>. We describe what the infrastructure should look like, rather than specifying every step required to create it.</p><p>For example:</p><pre>resource &quot;aws_s3_bucket&quot; &quot;example&quot; {<br>  bucket = &quot;my-example-terraform-bucket&quot;<br>}</pre><p>We’re telling Terraform that an S3 bucket should exist.</p><p>Terraform determines whether the bucket needs to be created, updated, or left unchanged based on its configuration, state, and information returned by AWS.</p><p>The typical execution lifecycle looks like this:</p><pre>Write Terraform Configuration<br>             |<br>             v<br>      terraform init<br>             |<br>             v<br>       terraform fmt<br>             |<br>             v<br>     terraform validate<br>             |<br>             v<br>       terraform plan<br>             |<br>             v<br>       Review Changes<br>             |<br>             v<br>       terraform apply<br>             |<br>             v<br>      Infrastructure</pre><p>When infrastructure is no longer needed, terraform destroy can remove the resources Terraform manages.</p><p>Let’s explore each stage.</p><h3>Initialize: terraform init</h3><p>The first step is initialising the Terraform working directory.</p><pre>terraform init</pre><p>When we run this command, Terraform:</p><ul><li>Downloads the required provider plugins, such as AWS, Azure, or Kubernetes.</li><li>Downloads modules referenced in the configuration.</li><li>Initialises the configured backend for storing Terraform state.</li><li>Creates the .terraform directory and maintains provider dependency selections in .terraform.lock.hcl.</li></ul><p>Think of terraform init as preparing the environment before Terraform can interact with cloud infrastructure.</p><p>An important distinction is that <strong>terraform init does not create the infrastructure defined by your resources</strong>.</p><p>It can safely be run multiple times, and is also required when certain backend, provider, or module configurations change.</p><h3>Format: terraform fmt</h3><p>Terraform uses HashiCorp Configuration Language (HCL) to describe infrastructure.</p><pre>terraform fmt</pre><p>This command automatically formats Terraform files according to standard conventions.</p><p>For example, it fixes inconsistent indentation and spacing.</p><p>It does not validate whether the infrastructure configuration is correct.</p><p>In a CI/CD pipeline, we can check formatting without modifying files:</p><pre>terraform fmt -check</pre><p>This helps maintain consistent configuration across teams.</p><h3>Validate: terraform validate</h3><p>Once the code is formatted, the next step is validation.</p><pre>terraform validate</pre><p>Terraform checks whether the configuration is syntactically valid and internally consistent.</p><p>For example, it can detect invalid arguments, incorrect resource references, and certain type mismatches.</p><p>However, <strong>validation does not verify whether the cloud provider will successfully create the resources</strong>.</p><p>A configuration might pass validation but still fail during deployment because of missing IAM permissions, resource quotas, or unavailable services.</p><p>The working directory must be initialised before validation.</p><h3>Plan: terraform plan</h3><p>This is one of the most important stages in Terraform’s execution lifecycle.</p><pre>terraform plan</pre><p>Terraform calculates the changes required to make the actual infrastructure match our configuration.</p><h3>What happens behind the scenes?</h3><p>Terraform generally performs these operations:</p><ol><li><strong>Loads the configuration:</strong> Reads the desired infrastructure defined in HCL.</li><li><strong>Reads the state:</strong> Uses Terraform state to identify resources already under management.</li><li><strong>Refreshes resource information:</strong> Queries provider APIs to inspect existing infrastructure and detect drift.</li><li><strong>Builds a dependency graph:</strong> Determines relationships and ordering constraints between resources.</li><li><strong>Generates an execution plan:</strong> Identifies which resources should be created, updated, replaced, or destroyed.</li></ol><p>For example:</p><pre>Terraform will perform the following actions:<br><br>+ aws_s3_bucket.example<br>Plan: 1 to add, 0 to change, 0 to destroy.</pre><p>Terraform uses symbols to represent actions:</p><p>SymbolMeaning+Create a resource~Update an existing resource-Destroy a resource-/+Destroy and recreate a resource+/-Create replacement before destroying the old resource</p><p><strong>A normal </strong><strong>terraform plan does not modify your actual cloud infrastructure.</strong></p><p>It gives you an opportunity to inspect changes before deploying them.</p><p>This is particularly important in production, where an unexpected resource replacement could cause downtime.</p><h3>Apply: terraform apply</h3><p>Once we’ve reviewed the changes, we can apply them.</p><pre>terraform apply</pre><p>By default, Terraform generates a fresh plan, requests confirmation in interactive mode, and then executes the approved operations.</p><p>During execution, Terraform:</p><ul><li>Uses the dependency graph to determine operation ordering.</li><li>Calls provider APIs to create, update, or delete resources.</li><li>Executes independent operations concurrently where possible.</li><li>Updates Terraform state to reflect successfully managed changes.</li></ul><p>For example, if an EC2 instance depends on a subnet, Terraform must create the subnet before creating the instance.</p><p>Terraform also supports <strong>state locking</strong>, where the backend supports it, to prevent concurrent operations from corrupting shared state.</p><h3>What happens if apply fails halfway?</h3><p>Terraform is not a transactional database. It cannot automatically roll back every infrastructure change when an operation fails.</p><p>Some resources may have been created successfully while others failed.</p><p>Terraform records successfully completed operations where possible. After investigating and fixing the error, we can generate another plan to reconcile the remaining differences.</p><p>This is why understanding Terraform state is essential when troubleshooting production deployments.</p><h3>Destroy: terraform destroy</h3><p>When infrastructure is no longer required, Terraform can remove the resources it manages.</p><pre>terraform destroy</pre><p>Terraform generates a destruction plan and, by default, asks for confirmation before proceeding.</p><p>It uses the resource dependency graph to determine a valid destruction order.</p><p>For example, an EC2 instance generally needs to be removed before deleting the subnet on which it depends.</p><p>We can preview destruction without executing it:</p><pre>terraform plan -destroy</pre><p>This is particularly useful for temporary development and testing environments.</p><p><strong>Be careful:</strong> terraform destroy attempts to remove all resources managed by the current Terraform configuration and selected workspace, not just the resource you last changed.</p><h3>Understanding Terraform State</h3><p>One concept connects the entire lifecycle: <strong>Terraform state</strong>.</p><p>Terraform stores information about the resources it manages in a state file, typically called terraform.tfstate.</p><p>Imagine that Terraform creates an EC2 instance:</p><pre>Terraform Configuration<br>      |<br>      v<br>aws_instance.web<br>      |<br>      v<br>Terraform State<br>      |<br>      v<br>EC2 Instance: i-0123456789</pre><p>The state maps the resource address in our configuration to the actual infrastructure object.</p><p>When we run terraform plan again, Terraform uses this mapping to determine what already exists.</p><p>Without reliable state, Terraform cannot safely manage the lifecycle of resources it previously created.</p><p>In production, teams generally use a remote backend, such as Amazon S3 or HCP Terraform, instead of relying on local state files.</p><p>Remote state enables collaboration, centralised access control, backup strategies, and, with supported backends, state locking.</p><h3>How Terraform Fits into CI/CD</h3><p>In production environments, infrastructure changes are typically deployed through automated pipelines rather than manually from an engineer’s laptop.</p><p>A simplified workflow might look like this:</p><pre>Developer Opens Pull Request<br>              |<br>              v<br>        Format Check<br>              |<br>              v<br>          Validate<br>              |<br>              v<br>      Terraform Plan<br>              |<br>              v<br>       Review and Approval<br>              |<br>              v<br>      Terraform Apply<br>              |<br>              v<br>     Infrastructure Updated</pre><p>For controlled deployments, Terraform supports saving an execution plan:</p><pre>terraform plan -out=tfplan</pre><p>Then applying that same reviewed plan:</p><pre>terraform apply tfplan</pre><p>This helps ensure the deployment uses the plan that was approved, rather than silently generating a different one.</p><p>Saved plans must be protected because they may contain sensitive data. They must also remain valid against the state being applied, and infrastructure can still change outside Terraform between planning and execution.</p><p>For production, it’s good practice to use remote state, locking, least-privilege permissions, protected approval workflows, and deployment monitoring.</p><h3>Final Thoughts</h3><p>Terraform’s execution lifecycle is straightforward at the command level:</p><p><strong>Init → Format → Validate → Plan → Apply</strong></p><p>And when infrastructure is no longer needed:</p><p><strong>Destroy</strong></p><p>But what makes Terraform powerful is what happens behind these commands: state management, dependency resolution, drift detection, and declarative reconciliation.</p><p>Understanding these internals helps us design safer infrastructure workflows, troubleshoot deployment failures, and prevent unexpected changes in production.</p><p>The most important habit when using Terraform is simple:</p><p><strong>Always understand what Terraform plans to change before allowing it to change your infrastructure.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e9ab38d5883" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Schema Registry Explained: Managing Data Contracts in Kafka]]></title>
            <link>https://medium.com/@avinashbabu/schema-registry-explained-managing-data-contracts-in-kafka-74b0efebb9a5?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/74b0efebb9a5</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[schemaregistry]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Fri, 13 Nov 2026 00:40:47 GMT</pubDate>
            <atom:updated>2026-11-13T00:40:47.679Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2rOrSEmu-8CXVSumHs5pQ.png" /></figure><p><em>Understanding how Schema Registry helps applications exchange data safely as systems evolve.</em></p><p>Imagine you’re building an e-commerce platform using Apache Kafka.</p><p>Your Order Service publishes events whenever a customer places an order. Several services consume these events:</p><ul><li>Payment Service processes payments.</li><li>Inventory Service updates stock.</li><li>Analytics Service tracks sales.</li><li>Notification Service sends confirmations.</li></ul><p>Everything works perfectly until the Order Service changes the structure of its events.</p><p>Suddenly, one of the consumers starts failing because it expects a different data format.</p><p>This is a common challenge in distributed systems. Applications evolve independently, but they still need to understand each other’s data.</p><p><strong>Schema Registry helps solve this problem by managing data contracts and controlling how they evolve.</strong></p><h3>What Is Schema Registry?</h3><p>A schema defines the structure of data, including field names, data types, and optional fields.</p><p>Consider this Kafka event:</p><pre>{<br>  &quot;order_id&quot;: 101,<br>  &quot;amount&quot;: 250<br>}</pre><p>Now imagine the producer changes amount from a number to a string. Consumers expecting a number might break.</p><p><strong>Schema Registry is a centralized service that stores, versions, and validates schemas shared between producers and consumers.</strong></p><p>It provides three main capabilities:</p><ul><li><strong>Schema management:</strong> Stores schemas centrally so applications can use a shared contract.</li><li><strong>Schema versioning:</strong> Tracks changes to schemas over time.</li><li><strong>Compatibility validation:</strong> Checks whether new schema versions are compatible with existing versions.</li></ul><p>Schema Registry supports Apache Avro, Protocol Buffers (Protobuf), and JSON Schema.</p><h3>How Does Schema Registry Work with Kafka?</h3><p>The communication flow looks like this:</p><p><strong>Producer → Serializer → Kafka → Deserializer → Consumer</strong></p><p>Schema Registry interacts with serializers and deserializers on both sides.</p><p>When a producer publishes an event:</p><ul><li>The serializer registers or looks up the schema in Schema Registry.</li><li>Schema Registry returns a unique schema ID.</li><li>The serializer encodes the message and includes the schema ID.</li><li>Kafka stores the serialized message.</li><li>The consumer’s deserializer extracts the schema ID, retrieves the corresponding schema, and decodes the message.</li></ul><p>Clients typically cache schema information, so they don’t need to contact Schema Registry for every message.</p><p>An important distinction: <strong>Kafka brokers don’t normally communicate with Schema Registry for every event.</strong> Schema validation and serialization are primarily handled by client applications.</p><h3>Understanding Schema Evolution</h3><p>Schema evolution allows data structures to change without unnecessarily breaking existing applications.</p><p>Consider an original order event:</p><pre>{<br>  &quot;order_id&quot;: 101,<br>  &quot;amount&quot;: 250<br>}</pre><p>Later, the business introduces international payments and adds a new field:</p><pre>{<br>  &quot;order_id&quot;: 101,<br>  &quot;amount&quot;: 250,<br>  &quot;currency&quot;: &quot;GBP&quot;<br>}</pre><p>This seems like a simple change, but some consumers might still be running older application versions.</p><p>Additionally, Kafka may contain millions of historical messages using the original schema.</p><p>Schema Registry helps manage these changes through <strong>compatibility rules</strong>.</p><h3>Schema Compatibility Types</h3><p>Schema Registry supports several compatibility modes.</p><p>TypeMeaning<strong>Backward</strong>New consumers can read data written with older schemas.<strong>Forward</strong>Old consumers can read data written with newer schemas.<strong>Full</strong>Both old and new consumers can read data written with either compatible schema.<strong>Transitive</strong>Compatibility is checked against all previous schema versions, rather than only the latest one.</p><p>For example, when using Avro, adding a new field with a default value can be backward compatible because newer consumers can apply the default when reading older events.</p><p>BACKWARD is the default compatibility mode in Confluent Schema Registry.</p><p>For systems that need to replay historical Kafka events, BACKWARD_TRANSITIVE can be particularly useful.</p><p>The precise rules depend on whether you’re using Avro, Protobuf, or JSON Schema.</p><h3>Understanding Subjects, Versions, and Schema IDs</h3><p>Schema Registry uses three important concepts:</p><ul><li><strong>Subject:</strong> A namespace under which schemas are registered, such as orders-value.</li><li><strong>Version:</strong> The evolution of a schema within a subject.</li><li><strong>Schema ID:</strong> A unique identifier for a registered schema definition.</li></ul><p>For example, the orders-value subject might have three versions with different schema IDs.</p><p>When a consumer receives a message, it can use the embedded schema ID to find the correct schema, even if the producer has moved to a newer version.</p><h3>What Happens If Schema Registry Goes Down?</h3><p>Schema Registry is generally not part of Kafka’s message-delivery path.</p><p>Because serializers and deserializers cache schema information, existing applications may continue working during an outage.</p><p>However, new schema registrations and requests for uncached schemas may fail.</p><p>This is why production environments should consider high availability, monitoring, and schema caching.</p><p>In self-managed Confluent Schema Registry, schema metadata is persisted in a compacted Kafka topic called _schemas.</p><h3>Best Practices</h3><p>A few practices help keep schema evolution safe:</p><ul><li><strong>Use compatibility checks:</strong> Validate schema changes before deploying producers.</li><li><strong>Prefer additive changes:</strong> Adding fields with appropriate defaults is often safer than changing existing field types.</li><li><strong>Integrate validation into CI/CD:</strong> Catch incompatible changes before they reach production.</li><li><strong>Use transitive compatibility when needed:</strong> Particularly for applications that replay historical events.</li><li><strong>Treat schemas like APIs:</strong> Changes to shared event structures should be reviewed carefully.</li></ul><p>Remember that schema compatibility doesn’t guarantee business compatibility. Changing the meaning of a field without changing its data type may pass validation but still break applications.</p><h3>Do You Always Need Schema Registry?</h3><p>Not necessarily.</p><p>Kafka can store and deliver messages without Schema Registry.</p><p>For small applications with limited producers and consumers, managing message formats manually might be sufficient.</p><p>However, Schema Registry becomes increasingly valuable as services grow, teams deploy independently, and event formats evolve over time.</p><h3>Final Thoughts</h3><p>Schema Registry solves one of the most important challenges in event-driven systems: <strong>allowing applications to evolve without breaking the consumers that depend on their data.</strong></p><p>Kafka handles the storage and delivery of events, while Schema Registry manages the contracts that describe those events.</p><p>Together, they help teams build reliable, maintainable, and scalable event-driven architectures.</p><blockquote><strong>Kafka ensures messages are delivered. Schema Registry helps ensure those messages can be understood.</strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74b0efebb9a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps Explained: How It Works and Best Practices for Kubernetes]]></title>
            <link>https://medium.com/@avinashbabu/gitops-explained-how-it-works-and-best-practices-for-kubernetes-1787b4e40383?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/1787b4e40383</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Thu, 12 Nov 2026 00:08:59 GMT</pubDate>
            <atom:updated>2026-11-12T00:08:59.473Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wAP7Nm5fBWN9IDvYRC4r6g.png" /></figure><p><em>Understanding GitOps, continuous reconciliation, and how to manage Kubernetes infrastructure reliably at scale.</em></p><p>Imagine managing 100 Kubernetes clusters across AWS, Azure, and Google Cloud. Every cluster has its own applications, configurations, monitoring agents, and infrastructure components.</p><p>Now imagine updating a configuration across all those clusters, only to discover that someone manually changed a few of them. How do you know which configuration is correct? How do you prevent those differences from happening again?</p><p>This is where <strong>GitOps</strong> comes in.</p><h3>What Is GitOps?</h3><p>GitOps is an approach to managing infrastructure and application deployments where <strong>Git acts as the source of truth for the desired state of your system.</strong></p><p>Instead of manually applying changes to Kubernetes, you define your desired configuration in Git. A GitOps controller, such as Argo CD or Flux CD, continuously monitors the repository and ensures that the actual infrastructure matches the configuration stored in Git.</p><p>The idea is simple:</p><blockquote><em>Define what you want in Git, and let automation continuously ensure that your infrastructure matches it.</em></blockquote><p>This makes infrastructure changes more predictable, auditable, and easier to manage.</p><h3>1. How GitOps Works</h3><p>Let’s understand GitOps with a simple Kubernetes example.</p><p>Suppose you have a Kubernetes Deployment running three replicas of a payment service.</p><p>Your deployment configuration contains:</p><p>replicas: 3</p><p>Now your application starts receiving more traffic, and you want to increase the number of replicas to five.</p><h4>Traditional approach</h4><p>You might execute:</p><p>kubectl scale deployment payment-service --replicas=5</p><p>Kubernetes scales the deployment, and everything works.</p><p>But there’s a problem. The configuration in Git might still specify three replicas, while the cluster is running five.</p><p>You’ve now created a difference between your documented configuration and your actual infrastructure.</p><h4>GitOps approach</h4><p>With GitOps, the process is different:</p><ol><li>Update the deployment configuration in Git from replicas: 3 to replicas: 5.</li><li>Create a pull request and get it reviewed.</li><li>Merge the change into the appropriate branch.</li><li>The GitOps controller detects the updated configuration.</li><li>The controller automatically updates Kubernetes to five replicas.</li></ol><p>The deployment happens without anyone manually running kubectl apply.</p><h4>The most important concept: Continuous reconciliation</h4><p>GitOps isn’t just about automatically deploying changes.</p><p>Its most important feature is <strong>continuous reconciliation</strong>.</p><p>A GitOps controller continuously compares two states:</p><ul><li><strong>Desired state:</strong> What Git says the infrastructure should look like.</li><li><strong>Actual state:</strong> What currently exists in the Kubernetes cluster.</li></ul><p>If the states differ, the controller attempts to bring the actual state back to the desired state.</p><p>For example, suppose someone manually changes the deployment back to three replicas while Git specifies five.</p><p>If automated self-healing is enabled, the GitOps controller detects the difference and restores the deployment to five replicas.</p><p>This difference is called <strong>configuration drift</strong>.</p><p>Reconciliation helps ensure that the configuration in Git remains authoritative, even when unexpected changes happen in the cluster.</p><h3>2. The Four Principles of GitOps</h3><p>According to the <a href="https://opengitops.dev/">OpenGitOps specification</a>, GitOps is built on four fundamental principles.</p><h4>Declarative configuration</h4><p>Instead of describing how to perform an operation, describe the final state you want.</p><p>For example, rather than instructing Kubernetes to create five individual pods, declare that your application requires five replicas.</p><p>Kubernetes and the GitOps controller take responsibility for achieving that state.</p><h4>Versioned and immutable</h4><p>Desired configurations should have a complete, versioned history.</p><p>Git makes it possible to track who changed a configuration, what changed, and why it changed.</p><p>This also makes it easier to restore a previously known configuration when something goes wrong.</p><h4>Automatically pulled</h4><p>In GitOps, software agents automatically pull the desired configuration from the source.</p><p>This is different from a traditional deployment pipeline that pushes configuration directly into Kubernetes.</p><p>A pull-based model can reduce the need to distribute cluster credentials across CI/CD pipelines.</p><h4>Continuously reconciled</h4><p>The controller continually observes the actual system and attempts to correct differences from the desired state.</p><p>This is what makes GitOps an ongoing infrastructure management model rather than simply another deployment mechanism.</p><h3>3. GitOps Best Practices for Production</h3><p>GitOps is relatively easy to introduce for a small application. The real challenges appear when managing hundreds of applications and clusters across multiple environments.</p><p>Here are the practices that matter most.</p><h4>Make Git the single source of truth</h4><p>Production configuration changes should normally go through Git.</p><p>Use protected branches, pull request approvals, and automated validation before merging changes.</p><p>Avoid routine manual modifications using kubectl edit, kubectl patch, or direct configuration changes in production.</p><p>If emergency changes are necessary, document them and reconcile the desired configuration afterward.</p><h4>Separate environments and configurations</h4><p>Development, staging, and production should have clearly separated configurations and promotion boundaries.</p><p>Tools such as Helm and Kustomize help reuse shared configuration while maintaining environment-specific values.</p><p>For example, staging might run two replicas while production runs ten.</p><p>Both environments can share the same base deployment, with only the necessary differences maintained separately.</p><p>Prefer promoting tested, immutable container image versions rather than relying on mutable tags such as latest.</p><h4>Never store plaintext secrets in Git</h4><p>Git is an excellent place to store infrastructure configuration, but not plaintext passwords, API keys, or database credentials.</p><p>Instead, use solutions such as:</p><ul><li>External Secrets Operator with AWS Secrets Manager or Azure Key Vault.</li><li>SOPS for encrypted configuration stored in Git.</li><li>Kubernetes RBAC and cloud workload identity to limit access.</li></ul><p>The GitOps controller should have only the permissions it needs to manage its resources.</p><h4>Detect configuration drift and enable self-healing carefully</h4><p>One of GitOps’ greatest strengths is detecting when the actual system differs from the expected configuration.</p><p>However, not every difference is necessarily a problem.</p><p>For example, Kubernetes Horizontal Pod Autoscaler may change the number of replicas automatically.</p><p>If GitOps continuously attempts to restore a fixed replica count while HPA tries to scale the application, the controllers can conflict.</p><p>The solution is to define clear ownership of fields and configure reconciliation rules appropriately.</p><p>Similarly, automated pruning should be enabled carefully because it can delete resources that no longer exist in Git.</p><h4>Validate changes before merging</h4><p>GitOps makes infrastructure changes automatic, which means incorrect configuration can also spread automatically.</p><p>A strong CI validation process should include:</p><ul><li>Kubernetes manifest and schema validation.</li><li>Helm or Kustomize rendering checks.</li><li>Security and policy validation.</li><li>Configuration testing before production promotion.</li></ul><p>Tools such as OPA and Kyverno can enforce rules around resource limits, privileged containers, and approved container images.</p><p>The goal is to catch problems before the GitOps controller attempts to deploy them.</p><h4>Manage dependencies and deployment ordering</h4><p>Not every Kubernetes resource can be deployed independently.</p><p>For example, a Custom Resource Definition (CRD) must exist before Kubernetes can accept custom resources of that type.</p><p>Similarly, infrastructure controllers and operators may need to be ready before their dependent resources are created.</p><p>GitOps tools provide mechanisms for controlling these dependencies.</p><p>Flux CD supports dependsOn relationships between Kustomizations, while Argo CD provides synchronization waves.</p><p>These capabilities become particularly important when bootstrapping infrastructure components across new Kubernetes clusters.</p><h4>Have a safe rollback and recovery strategy</h4><p>When a deployment introduces a problem, GitOps provides a natural way to restore the previous desired configuration.</p><p>You can revert the problematic Git commit or promote a previously tested version.</p><p>The GitOps controller then reconciles the cluster toward that configuration.</p><p>However, changing Git does not automatically undo every effect of a deployment. Database migrations, deleted persistent data, and external infrastructure changes may require separate recovery procedures.</p><p>For critical systems, combine GitOps with tested rollback procedures and staged deployments.</p><h4>Monitor reconciliation health</h4><p>A GitOps controller is another important component of your infrastructure, and it needs monitoring.</p><p>Useful signals include reconciliation failures, configuration drift, deployment health, and time taken for changes to become effective.</p><p>A successful Git merge does not necessarily mean a successful deployment.</p><p>The most important question is whether the infrastructure has reached the desired state and whether the application is healthy.</p><h3>4. GitOps vs Traditional CI/CD</h3><p>Traditional CI/CD and GitOps are closely related, but they solve different parts of the deployment problem.</p><p>In a traditional pipeline, the process might look like:</p><p><strong>Code change → CI builds and tests → CI pushes deployment → Kubernetes</strong></p><p>With GitOps:</p><p><strong>Code change → CI builds and tests → Git configuration updated → GitOps controller reconciles Kubernetes</strong></p><p>The key difference is who initiates the deployment.</p><p>Traditional pipelines commonly push changes into the cluster, while GitOps controllers pull desired configuration and continuously reconcile it.</p><p>GitOps does not replace CI.</p><p>CI is still responsible for building software, running tests, scanning artifacts, and publishing container images.</p><p>GitOps takes responsibility for delivering the approved desired configuration to the environment and maintaining it.</p><h3>5. Argo CD vs Flux CD</h3><p>Two of the most widely used GitOps tools in the Kubernetes ecosystem are Argo CD and Flux CD.</p><h4>Argo CD</h4><p>Argo CD provides an application-oriented approach to GitOps with a built-in web interface.</p><p>It makes it easy to visualize applications, compare desired and actual states, inspect deployment health, and manage synchronization.</p><p>It is particularly useful when development and platform teams want a central interface for understanding application deployments.</p><h4>Flux CD</h4><p>Flux CD takes a Kubernetes-native, controller-based approach.</p><p>It uses Kubernetes custom resources to define Git repositories, Helm releases, Kustomizations, and reconciliation behavior.</p><p>Its modular architecture makes it particularly suitable for declarative platform automation and infrastructure management.</p><p>Both tools support automated reconciliation, Helm, Kustomize, and multi-cluster deployment patterns.</p><p>The choice depends on how your organization wants to structure and operate its delivery platform.</p><h3>6. Real-World Example: Managing Kubernetes Infrastructure at Scale</h3><p>Consider a platform engineering team managing 100 Kubernetes clusters across AWS EKS, Azure AKS, and Google GKE.</p><p>Each cluster runs common infrastructure components such as monitoring agents, ingress controllers, certificate managers, and Kubernetes operators.</p><p>Suppose the team needs to upgrade a shared monitoring agent across all clusters.</p><p>Without GitOps, an engineer might execute a deployment pipeline that directly updates every cluster. The team then needs to verify which clusters succeeded, which failed, and whether any manual changes exist.</p><p>With GitOps, the process becomes more structured.</p><p>The team updates the monitoring agent version in Git and validates the change through CI.</p><p>The new version is first promoted to a small set of staging or canary clusters. GitOps controllers detect the configuration change and reconcile those clusters.</p><p>After confirming that the agent is healthy, the team promotes the same version to the remaining production clusters in controlled batches.</p><p>If a cluster fails to converge, the team can inspect its reconciliation status and identify where the deployment differs from the desired configuration.</p><p>There is also an important distinction when using Kubernetes operators.</p><p>GitOps may deploy an operator or its custom resources, but the operator itself remains responsible for reconciling the resources it manages.</p><p>For example, GitOps can manage a Crossplane resource definition while Crossplane handles the corresponding cloud infrastructure lifecycle.</p><p>This separation of responsibilities allows GitOps to work alongside Kubernetes’ existing reconciliation model.</p><h3>Final Thoughts</h3><p>GitOps is much more than storing Kubernetes YAML files in Git or automatically running deployments after merging a pull request.</p><p>It introduces a different operating model for infrastructure:</p><p><strong>Declare the desired state, version it, automatically retrieve it, and continuously reconcile the system toward it.</strong></p><p>Its real value becomes clear when infrastructure grows across multiple teams, environments, and cloud providers.</p><p>GitOps makes changes easier to review, provides a clear configuration history, helps detect drift, and reduces the dependence on manual operations.</p><p>But GitOps alone does not guarantee reliable infrastructure. Proper validation, access controls, dependency management, monitoring, and recovery procedures are still essential.</p><p>The objective is not simply to automate deployments. It is to make infrastructure <strong>predictable, reproducible, auditable, and easier to operate at scale.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1787b4e40383" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How AI Is Reshaping System Design and Why It Matters More Than Ever]]></title>
            <link>https://medium.com/@avinashbabu/how-ai-is-reshaping-system-design-and-why-it-matters-more-than-ever-9497cf40fbd4?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/9497cf40fbd4</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 00:39:19 GMT</pubDate>
            <atom:updated>2026-11-11T00:39:19.393Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtXlA-Wei6u6Q5gGtw4nIg.png" /></figure><p><em>As AI makes writing code easier, the real engineering challenge is shifting from implementing software to designing reliable, scalable, and intelligent systems.</em></p><p>Software engineering is changing rapidly.</p><p>A few years ago, building an application meant spending considerable time writing APIs, implementing business logic, integrating databases, and testing individual components.</p><p>Today, AI coding tools can generate much of that code in minutes.</p><p>But here’s an interesting question:</p><p><strong>If AI can write most of the code, what becomes the most valuable skill for a software engineer?</strong></p><p>I believe system design is becoming increasingly important.</p><p>And it’s happening for two reasons:</p><ol><li><strong>AI is changing how we build software.</strong> Engineers can spend less time implementing individual components and more time making architectural decisions.</li><li><strong>AI is changing what we build.</strong> Modern applications increasingly include LLMs, semantic search, AI agents, and other components that introduce entirely new design challenges.</li></ol><p>Let’s explore both.</p><h3>1. From Writing Code to Designing Systems</h3><p>Consider a simple application like TinyURL.</p><p>Traditionally, engineers would spend time implementing APIs, designing database schemas, writing caching logic, and building the infrastructure.</p><p>Today, you could ask an AI coding agent to generate much of that implementation.</p><p>But before writing a single line of code, someone still needs to answer important questions.</p><ul><li>How do we generate unique short URLs across multiple servers?</li><li>Should we use SQL or NoSQL?</li><li>What happens when Redis becomes unavailable?</li><li>How do we handle millions of redirects per second?</li><li>Should we prioritise consistency or availability?</li></ul><p>AI can suggest answers to these questions. It can even propose a complete architecture.</p><p>But <strong>understanding the trade-offs and choosing the right architecture for the requirements is still an engineering responsibility.</strong></p><p>This changes the role of a software engineer.</p><p>Instead of focusing primarily on how to write a function, engineers increasingly need to understand how components interact, how systems scale, and what happens when something fails.</p><p>AI accelerates implementation, but poor architectural decisions can still make an application expensive, unreliable, or difficult to maintain.</p><p>In fact, faster implementation can make bad architecture more costly because teams can build the wrong system much faster.</p><h3>2. AI Introduces a New Kind of System Architecture</h3><p>Now let’s look at the second major change.</p><p>Imagine you’re designing a customer support application.</p><p>In a traditional system, a customer sends a message, the backend searches for relevant information, and the application returns a response.</p><p>The architecture might include:</p><p><strong>Client → API Gateway → Application Servers → Database → Response</strong></p><p>Now imagine adding an AI assistant that understands customer questions and generates personalised responses based on company policies.</p><p>Suddenly, the design becomes more complicated.</p><p>The system might need:</p><ul><li><strong>Embedding models</strong> to convert documents and questions into numerical representations.</li><li><strong>Vector databases</strong> to retrieve information based on semantic similarity.</li><li><strong>Retrieval-Augmented Generation (RAG)</strong> to provide relevant company information to the model.</li><li><strong>LLM inference services</strong> to generate responses.</li><li><strong>Guardrails</strong> to prevent unsafe or unauthorised responses.</li><li><strong>Evaluation systems</strong> to measure answer quality and reliability.</li></ul><p>A simplified request flow becomes:</p><p><strong>User → API → Document Retrieval → LLM → Validation → Response</strong></p><p>This is a significant shift.</p><p>Traditional distributed systems are built around components whose behaviour is generally predictable.</p><p>AI systems introduce components whose behaviour can be probabilistic.</p><p>Even when the infrastructure is working perfectly, the AI model might produce an incorrect response.</p><p>That changes what reliability means.</p><h3>3. The Three New Trade-offs: Cost, Latency, and Quality</h3><p>Traditional system design interviews focus heavily on scalability, availability, consistency, and performance.</p><p>These concepts remain essential.</p><p>But AI introduces three closely connected concerns.</p><h4>Cost: Every request has a price</h4><p>In a traditional web application, an API request might involve a few database queries and relatively inexpensive computation.</p><p>With AI, every request can consume thousands of input and output tokens.</p><p>Consider an AI assistant serving 100,000 requests daily.</p><p>If each request costs $0.01 in model inference, that’s approximately <strong>$1,000 per day or $30,000 per 30-day month</strong>, before accounting for other infrastructure expenses.</p><p>At scale, even small improvements matter.</p><p>Engineers now need to think about:</p><ul><li>Using smaller models for simpler tasks.</li><li>Caching frequently requested information.</li><li>Reducing unnecessary context and token usage.</li><li>Routing complex requests to more capable models.</li></ul><p>Model selection becomes an architectural decision, not just an implementation detail.</p><h4>Latency: Not every component responds in milliseconds</h4><p>Traditional databases and caches can often return results within milliseconds.</p><p>LLM inference can take considerably longer, depending on the model, prompt size, generated output, and system load.</p><p>If a customer expects an answer within three seconds, the system must account for retrieval, model inference, validation, and network overhead.</p><p>This introduces new techniques such as streaming responses, asynchronous processing, model routing, and timeouts.</p><p>Engineers also need to distinguish between <strong>time to first token</strong> and the time required to generate a complete response.</p><p>A system that starts streaming within 500 milliseconds can feel responsive even if the full response takes several seconds.</p><h4>Quality: A successful API response isn’t always a correct answer</h4><p>This is perhaps the most interesting change.</p><p>In traditional software, a successful HTTP response often indicates that the application completed its operation.</p><p>With an LLM, receiving 200 OK doesn&#39;t mean the generated information is accurate.</p><p>A model could confidently provide an incorrect refund policy or invent information that doesn’t exist.</p><p>Engineers now need evaluation pipelines, evidence-based retrieval, output validation, and mechanisms to handle uncertain answers.</p><p><strong>A system can have 99.99% availability and still be a terrible AI product if its answers are consistently wrong.</strong></p><p>That’s why quality becomes an explicit nonfunctional requirement.</p><h3>4. Deterministic vs. Probabilistic Components</h3><p>One of the most important principles in modern AI system design is knowing where to use AI and where not to use it.</p><p>Consider an AI banking assistant.</p><p>It might be excellent at understanding a customer’s request:</p><p><em>“Can you transfer £500 from my savings account to my current account?”</em></p><p>But should the LLM itself decide whether the customer has sufficient funds, whether the transfer is authorised, or whether the transaction has succeeded?</p><p>No.</p><p>The model should interpret the user’s intent and help orchestrate the workflow.</p><p>The actual transaction should be handled by deterministic services with explicit validation, authorisation, and transactional guarantees.</p><p>A good architecture separates these responsibilities:</p><p><strong>AI handles interpretation and reasoning.</strong></p><p><strong>Traditional services enforce business rules and execute critical operations.</strong></p><p>This distinction becomes even more important with AI agents that can call APIs, access databases, and perform real-world actions.</p><p>As agents become more capable, engineers must design permission boundaries, idempotency, audit trails, retries, and human approval mechanisms for sensitive operations.</p><p>An incorrect generated sentence is one kind of failure. An incorrect financial transaction is an entirely different one.</p><h3>5. System Design Interviews Are Evolving Too</h3><p>For years, popular system design questions included:</p><ul><li>Design TinyURL.</li><li>Design WhatsApp.</li><li>Design YouTube.</li><li>Design Uber.</li><li>Design a distributed rate limiter.</li></ul><p>These remain excellent problems because they test fundamental distributed systems knowledge.</p><p>But AI is expanding the kinds of architectural questions engineers may face.</p><p>For example:</p><p><strong>Design an AI customer support assistant.</strong></p><p>An interviewer could expect you to explain how documents are indexed, how relevant context is retrieved, how hallucinations are reduced, and what happens when the model is unavailable.</p><p><strong>Design an AI coding assistant.</strong></p><p>You might need to reason about repository indexing, context selection, tool execution, permissions, code validation, and long-running agent workflows.</p><p><strong>Design a scalable LLM inference platform.</strong></p><p>Now the discussion includes GPU utilisation, batching, queueing, tokens per second, model routing, and cost optimisation.</p><p>Even traditional questions can gain an AI dimension.</p><p>A recommendation system might include semantic embeddings and model-based ranking. A search engine might combine keyword matching with vector retrieval and AI-generated summaries.</p><p>The fundamentals haven’t disappeared.</p><p>Instead, <strong>system design interviews can now test two layers of knowledge: distributed systems fundamentals and AI-specific architectural decisions.</strong></p><p>Not every company or engineering role will require deep AI infrastructure knowledge, but understanding these concepts is becoming useful across a growing range of applications.</p><h3>6. How Should Software Engineers Prepare?</h3><p>I wouldn’t recommend abandoning traditional system design concepts to focus entirely on AI.</p><p>Quite the opposite.</p><p>AI systems still depend on databases, networking, caching, load balancing, message queues, distributed coordination, and fault tolerance.</p><p>Without these fundamentals, it’s difficult to design reliable AI applications.</p><p>Instead, I’d approach learning in three stages.</p><p><strong>First, master traditional system design.</strong></p><p>Understand scalability, consistency, availability, partitioning, caching, queues, and database design. These remain the foundation of almost every large-scale application.</p><p><strong>Second, learn AI-specific architecture.</strong></p><p>Understand inference, embeddings, vector search, RAG, model routing, token economics, evaluations, and agent orchestration.</p><p>You don’t necessarily need to know how to train a large language model, but you should understand how to integrate one into a production system.</p><p><strong>Third, build and evaluate a real system.</strong></p><p>Design an AI-powered application, use coding agents to implement it, deploy it, and measure how it behaves.</p><p>What happens when the model times out? How do costs change with longer conversations? What happens when retrieval returns irrelevant documents? How do you detect incorrect answers?</p><p>These questions turn theoretical system design knowledge into practical engineering judgement.</p><h3>Final Thoughts</h3><p>AI is making it easier than ever to turn an idea into working software.</p><p>But working software and a well-designed system are not the same thing.</p><p>The ability to generate code doesn’t automatically solve scalability, reliability, cost optimisation, security, or architectural complexity.</p><p>And as more applications integrate AI, engineers must deal with additional challenges involving probabilistic behaviour, inference latency, output quality, and safety.</p><p>That’s why I believe system design is becoming more valuable.</p><p>The future of software engineering won’t simply be about who can write code the fastest.</p><p>It will increasingly be about <strong>who can design the right system, understand its trade-offs, use AI effectively to build it, and ensure it works reliably in production.</strong></p><p><strong>AI is making code easier to produce. Good engineering judgement is becoming harder to replace.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9497cf40fbd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding People Starts with Understanding Influence]]></title>
            <link>https://medium.com/@avinashbabu/understanding-people-starts-with-understanding-influence-3dd21cb2e50b?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dd21cb2e50b</guid>
            <category><![CDATA[must-read]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[influence]]></category>
            <category><![CDATA[psychology]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 00:20:37 GMT</pubDate>
            <atom:updated>2026-11-10T00:20:37.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/362/0*VRS4dipGJhf4Dj9_" /></figure><p><em>Understanding the psychology behind persuasion, human behaviour, and the decisions we make every day.</em></p><p><em>Influence: The Psychology of Persuasion</em> by Robert Cialdini is one of those must-read books that I believe everyone should read at least once.</p><p>I first read it a long time ago, and recently picked it up again. Rereading it reminded me how relevant its ideas are, so I wanted to share some of the lessons that stood out to me.</p><p>What makes this book interesting is that its principles are everywhere. Whether it’s at work, in relationships, while shopping, negotiating, or simply making everyday decisions, you’ll notice these psychological patterns playing out around you.</p><p>We like to believe our decisions are based on logic and careful thinking. But often, our choices are influenced by psychological shortcuts without us even realising it.</p><p>In <em>Influence</em>, Cialdini explores why people say yes and identifies seven principles of persuasion that help explain human behaviour.</p><p>Let’s explore them.</p><h3>1. Reciprocity: We Feel Obligated to Return Favours</h3><blockquote><em>“When someone does us a favour, we feel an invisible obligation to return it.”</em></blockquote><p>Imagine receiving a Christmas card from someone you don’t know.</p><p>Would you send one back?</p><p>In a famous experiment discussed in the book, a researcher sent Christmas cards to complete strangers. Surprisingly, many recipients sent cards back, even though they had never met the sender.</p><p>This demonstrates <strong>reciprocity</strong>, our tendency to return favours, gifts, or acts of kindness.</p><p>We see this everywhere.</p><ul><li>A restaurant offers a free dessert, making us more inclined to leave a generous tip.</li><li>A colleague helps us with a task, and we naturally want to help them later.</li><li>A company offers a free trial, making us more receptive to becoming a customer.</li></ul><p>The interesting part is that the initial favour doesn’t have to be particularly valuable to create this feeling of obligation.</p><p><strong>The lesson:</strong> Giving something first can encourage others to give back. But genuine generosity builds stronger relationships than favours given with hidden expectations.</p><h3>2. Commitment and Consistency: We Want to Stay True to Our Decisions</h3><blockquote><em>“Once we take a stand, we feel compelled to stay consistent with it, even when circumstances change.”</em></blockquote><p>Once we commit to something, we feel pressure to remain consistent with that decision.</p><p>Imagine someone asking you to run a marathon tomorrow. You would probably refuse.</p><p>But what if they first asked you to walk for ten minutes a day? Then run a kilometre? Then participate in a 5K?</p><p>Each small commitment makes the next step feel more natural.</p><p>This is closely related to the <em>foot-in-the-door technique</em>, where agreeing to a small request increases the likelihood of agreeing to a larger, related request later.</p><p>We also tend to defend previous decisions because changing our minds can feel like admitting we were wrong.</p><p>This explains why people sometimes continue with subscriptions, investments, or commitments even when circumstances have changed.</p><p><strong>The lesson:</strong> Small commitments can create meaningful changes. But consistency shouldn’t prevent us from reconsidering decisions when new information emerges.</p><h3>3. Social Proof: We Look to Others When We’re Uncertain</h3><blockquote><em>“When we don’t know what’s right, we often assume that what everyone else is doing must be right.”</em></blockquote><p>Imagine walking through an unfamiliar city looking for a restaurant.</p><p>Two restaurants are next to each other.</p><p>One is completely empty. The other is packed with customers.</p><p>Which would you choose?</p><p>Most people would be drawn towards the busy restaurant.</p><p>Why?</p><p>Because we often assume that if many people are doing something, there must be a good reason.</p><p>This is <strong>social proof</strong>.</p><p>We see it in product reviews, bestselling books, trending videos, and investment decisions.</p><p>Social proof becomes particularly powerful when we’re uncertain about what to do, especially when the people we’re observing seem similar to us.</p><p>But there’s a problem.</p><p><strong>Popularity doesn’t necessarily mean something is correct or valuable.</strong></p><p>A product with thousands of reviews might still be unsuitable for us. An investment everyone is buying might still be overpriced.</p><p><strong>The lesson:</strong> Other people’s behaviour can provide useful information, but it shouldn’t replace independent judgement.</p><h3>4. Liking: We’re More Easily Persuaded by People We Like</h3><blockquote><em>“The more we like someone, the less convincing they need to do.”</em></blockquote><p>Think about the last time you met someone who shared your interests.</p><p>Perhaps you both enjoyed the same sport, studied at the same university, or grew up in the same city.</p><p>You probably felt more comfortable talking to them.</p><p>Cialdini explains that we’re more likely to agree with people we like.</p><p>Several factors can increase liking, including similarity, genuine compliments, familiarity, and cooperation.</p><p>This is why good salespeople often begin conversations by building rapport rather than immediately trying to sell something.</p><p>The same principle applies to friendships, networking, and professional relationships.</p><p>Finding common ground can make communication easier and create trust.</p><p>But liking someone and evaluating their proposal are two different things.</p><p><strong>The lesson:</strong> Build genuine connections through shared interests and understanding. At the same time, don’t let personal affection become the only reason for making a decision.</p><h3>5. Authority: We Trust People Who Appear to Be Experts</h3><blockquote><em>“We often trust the person behind the title before questioning the knowledge behind their advice.”</em></blockquote><p>Imagine receiving medical advice from two people.</p><p>One is wearing a doctor’s coat and introduces themselves as a specialist. The other is dressed casually and provides no credentials.</p><p>Whose advice would you trust more?</p><p>Most people would instinctively trust the person who appears to be an expert.</p><p>Authority influences our decisions because we learn from an early age to respect teachers, doctors, leaders, and other recognised experts.</p><p>In many situations, this shortcut is useful. We cannot personally become experts in every field.</p><p>However, authority can also be misleading.</p><p>Titles, uniforms, credentials, and confident language can create an impression of expertise even when the person lacks relevant knowledge.</p><p>A successful business leader, for example, might be knowledgeable about running a company but not necessarily about medicine or investing.</p><p><strong>The lesson:</strong> Respect genuine expertise, but evaluate whether someone’s authority is relevant to the decision you’re making.</p><h3>6. Scarcity: We Want What We Might Lose</h3><blockquote><em>“The possibility of losing something often makes it more desirable than the joy of having it.”</em></blockquote><p>Have you ever seen messages like these?</p><p>“Only 2 items remaining.”</p><p>“Offer ends tonight.”</p><p>“Limited seats available.”</p><p>Suddenly, something that wasn’t particularly interesting becomes more attractive.</p><p>This is the <strong>scarcity principle</strong>.</p><p>When something appears rare or difficult to obtain, we often assign it greater value.</p><p>The fear of missing out can sometimes become stronger than our original desire for the product.</p><p>Scarcity isn’t always artificial. Limited availability can be genuine and important.</p><p>But the key question is whether something is valuable because it meets our needs or simply because we might lose the opportunity to get it.</p><p><strong>The lesson:</strong> Don’t confuse urgency with importance. Before making a decision, ask whether you would still want the same thing if it were easily available tomorrow.</p><h3>7. Unity: Shared Identity Makes Influence Stronger</h3><blockquote><em>“When someone feels like one of us, their words carry more weight than those of a stranger.”</em></blockquote><p>The seventh principle, added in the expanded edition of <em>Influence</em>, is <strong>unity</strong>.</p><p>Unity goes beyond liking someone because you share a hobby or interest.</p><p>It’s about feeling that another person belongs to the same group as you.</p><p>Think about the connection between family members, people from the same community, or teammates working towards a common goal.</p><p>When we see someone as part of our group, their opinions and requests can carry greater weight.</p><p>For example, people may be more willing to support an initiative when they feel it represents the values and goals of their community.</p><p>The difference between liking and unity is subtle but important.</p><p><strong>Liking is about someone being similar to us. Unity is about seeing someone as one of us.</strong></p><p><strong>The lesson:</strong> Shared identity creates powerful connections, but belonging to the same group doesn’t automatically make someone’s judgement correct.</p><h3>The Bigger Lesson: Our Brains Love Shortcuts</h3><p>One of the most interesting ideas in <em>Influence</em> is that our brains frequently rely on shortcuts to make decisions.</p><p>We encounter too much information to evaluate everything from scratch.</p><p>So we use signals:</p><ul><li>Someone helped me, so I should help them.</li><li>Everyone seems to like it, so it must be good.</li><li>An expert recommends it, so it must be trustworthy.</li><li>It’s almost sold out, so I should act quickly.</li></ul><p>These shortcuts are not necessarily bad. They help us make decisions efficiently.</p><p>The problem arises when someone deliberately uses them to influence us without providing enough information to make a good decision.</p><p>Understanding these principles gives us two advantages.</p><p>First, we can communicate more effectively, build trust, and persuade others ethically.</p><p>Second, we become better at recognising when our own decisions are being influenced.</p><h3>Final Thoughts</h3><p>The biggest takeaway from <em>Influence</em> isn’t simply learning how to persuade people.</p><p>It’s understanding that <strong>our decisions aren’t always as independent or rational as we think they are.</strong></p><p>The next time you’re about to buy something, agree to a request, or follow someone’s advice, ask yourself:</p><p><em>Am I making this decision because it genuinely makes sense, or because something is influencing me to say yes?</em></p><p>That small moment of awareness can make a significant difference.</p><p>Because understanding influence isn’t just about getting others to say yes.</p><p><strong>It’s also about knowing when you should say no.</strong></p><p><em>Note: The opening quotes for each principle are my own interpretations of the book’s ideas, rather than direct quotations from Robert Cialdini.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dd21cb2e50b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Are Evals in AI? And Why Should Software Engineers Care?]]></title>
            <link>https://medium.com/@avinashbabu/what-are-evals-in-ai-and-why-should-software-engineers-care-ee4e8aa1022c?source=rss-5786ec2b5010------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee4e8aa1022c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai-evals]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Avinash Babu]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 23:32:36 GMT</pubDate>
            <atom:updated>2026-11-08T23:32:36.744Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9Fq5wZbw5oMnI23oTTREg.png" /></figure><p><em>Your AI agent says it fixed the bug. But how do you know it actually did?</em></p><p>Imagine asking Claude Code to fix a bug in your application.</p><p>It reads the code, investigates the issue, makes a few changes, runs some tests, and finally tells you:</p><blockquote><em>I’ve fixed the issue. All tests are passing.</em></blockquote><p>Great! But how do you know it actually did a good job?</p><p>What if it fixed the original bug but introduced another one? What if it modified unrelated files? What if the tests passed, but an important edge case was missed?</p><p>And here’s another interesting question.</p><p><strong>If you gave Claude Code the exact same task ten times, would it successfully solve it every time?</strong></p><p>This is where <strong>AI evals</strong> come in.</p><h3>What Are Evals?</h3><p>Evals, short for evaluations, are a way to measure how well an AI model or agent performs a particular task.</p><p>As software engineers, we’re already familiar with testing.</p><p>We write unit tests, integration tests, and end-to-end tests to make sure our applications behave as expected.</p><p>Evals follow a similar idea, but instead of just testing application code, we’re evaluating the AI system itself.</p><p>For example:</p><ul><li><strong>Unit test:</strong> Does this function correctly handle a nil pointer?</li><li><strong>AI eval:</strong> Can Claude Code identify and fix a nil pointer bug?</li></ul><p>The distinction matters.</p><p>Traditional tests verify whether our code behaves correctly. Evals measure whether an AI system can reliably produce the desired outcome.</p><p>And unlike traditional software, AI models can produce different results for the same input.</p><p>An agent might solve a problem on its first attempt, fail on the second, and find a completely different solution on the third.</p><p>So seeing it succeed once doesn’t necessarily tell us how reliable it is.</p><h3>How Does an Eval Work?</h3><p>Let’s take an example.</p><p>Suppose we have a Kubernetes controller that panics when it encounters a resource without an owner reference.</p><p>We want to evaluate whether Claude Code can fix the issue.</p><p>An eval needs three basic components.</p><h4>1. A Task</h4><p>We give Claude Code a problem to solve.</p><blockquote><em>Fix the panic that occurs when the controller encounters a resource without an owner reference.</em></blockquote><p>The task should have a clear objective and a reproducible starting environment.</p><h4>2. Success Criteria</h4><p>Next, we define what a successful solution should look like.</p><p>For example:</p><ul><li>The controller no longer panics when the owner reference is nil.</li><li>Existing unit tests continue to pass.</li><li>No unrelated files are modified.</li><li>The fix doesn’t introduce new issues.</li></ul><p>These criteria should be defined before running the evaluation.</p><p>Otherwise, we risk deciding what success means based on whatever solution the agent happens to produce.</p><h4>3. A Grader</h4><p>A grader is the mechanism that evaluates the result.</p><p>For coding tasks, this might involve executing tests:</p><pre>go test ./internal/controller/...<br>go test ./...<br>go vet ./...</pre><p>We could also inspect the Git diff to verify that unrelated files weren’t modified.</p><p>If the checks pass, the agent receives a passing result for those criteria.</p><p>It’s important that the evaluation tests are controlled independently of the agent. Otherwise, the agent could accidentally modify the tests it is supposed to satisfy.</p><p>This gives us a simple evaluation process:</p><p><strong>Task → AI Agent → Result → Grader → Score</strong></p><h3>Three Ways to Evaluate an AI Agent</h3><p>Not everything can be measured with a unit test.</p><p>In practice, there are three common approaches to evaluating AI systems.</p><h4>1. Code-Based Evals</h4><p>These use deterministic checks such as unit tests, integration tests, static analysis, or validation scripts.</p><p>For example, after Claude Code fixes a bug, the evaluation framework runs a predefined test suite.</p><p>This is usually the best starting point for software engineering tasks because the checks are automated and reproducible.</p><p>However, passing tests doesn’t necessarily mean the solution is well-designed. Tests can miss edge cases, and a technically correct solution might introduce unnecessary complexity.</p><h4>2. LLM-as-a-Judge</h4><p>What if we want to evaluate something subjective, such as code readability or the quality of an explanation?</p><p>We can use another AI model to evaluate the output against a predefined set of criteria, often called a rubric.</p><p>For example:</p><blockquote><em>Review this code change. Does it introduce unnecessary complexity? Are there simpler alternatives? Does it follow the repository’s existing conventions?</em></blockquote><p>The evaluating model can provide a score and explanation.</p><p>But there’s a catch.</p><p><strong>The AI evaluating the result can also make mistakes.</strong></p><p>So LLM-based grading is useful, but it shouldn’t be treated as an unquestionable source of truth.</p><h4>3. Human Evaluation</h4><p>Sometimes the best evaluator is still a developer.</p><p>For example, deciding whether an architectural change is maintainable or whether a particular trade-off makes sense for a production system may require human judgment.</p><p>In practice, combining automated tests with occasional human review often gives better results than relying on either approach alone.</p><h3>How Can We Use Evals With Claude Code?</h3><p>Let’s say you’ve created a custom CLAUDE.md file with instructions for your Go projects.</p><p>You’ve included guidelines around error handling, concurrency, testing, and coding conventions.</p><p>You feel Claude Code is performing better after adding those instructions.</p><p>But is it actually better?</p><p>Or did you just happen to give it easier tasks recently?</p><p>Instead of guessing, we can use evals.</p><p>Imagine preparing ten coding tasks with known bugs and automated tests.</p><p>We run Claude Code against those tasks under two configurations:</p><p><strong>Configuration A:</strong> Default Claude Code without the additional instructions.</p><p><strong>Configuration B:</strong> Claude Code with your custom CLAUDE.md instructions.</p><p>We reset each repository to its original state before every attempt and use the same model, tools, and evaluation criteria.</p><p>Then we compare the results.</p><p>Here’s what a hypothetical evaluation might look like:</p><p>Here’s what the hypothetical evaluation results looked like:</p><p><strong>With default Claude Code:</strong></p><ul><li>Successfully completed 6 out of 10 tasks.</li><li>Passed existing tests in 8 out of 10 tasks.</li><li>Made unrelated file changes in 3 cases.</li><li>Took an average of 4 minutes per task.</li></ul><p><strong>With custom </strong><strong>CLAUDE.md instructions:</strong></p><ul><li>Successfully completed 8 out of 10 tasks.</li><li>Passed existing tests in 9 out of 10 tasks.</li><li>Made unrelated file changes in only 1 case.</li><li>Took an average of 5 minutes per task.</li></ul><p>These are illustrative numbers, not actual benchmark results.</p><p>In this example, the custom instructions appear to improve task completion and reduce unnecessary changes, although they also increase execution time.</p><p>But ten tasks are still a small sample. We’d want repeated trials before drawing strong conclusions.</p><p>This is the interesting part about evals.</p><p><strong>We can use them to evaluate not just models, but also prompts, agent instructions, tools, skills, and entire agent workflows.</strong></p><p>For example, if you build a Claude Code skill for reviewing Kubernetes controllers, you could create an eval suite containing known concurrency bugs and measure how often the skill correctly identifies them.</p><p>You could even compare different models to determine which provides the best balance of accuracy, speed, and cost for your particular tasks.</p><h3>Building a Simple Eval</h3><p>You don’t need a complicated framework to start experimenting.</p><p>Imagine defining an evaluation task like this:</p><pre>task: fix-controller-panic<br>prompt: &gt;<br>  Fix the panic caused by a nil<br>  owner reference in the controller.<br>checks:<br>  - go test ./internal/controller/...<br>  - go test ./...<br>  - go vet ./...<br>constraints:<br>  - Do not modify unrelated files.<br>  - Do not remove or disable tests.</pre><p>This is just an illustrative YAML definition, not a built-in Claude Code configuration.</p><p>A basic eval runner could:</p><ol><li>Reset the repository to a known buggy commit.</li><li>Start Claude Code with the task.</li><li>Allow Claude to investigate and implement a solution.</li><li>Run predefined tests against the resulting code.</li><li>Record whether the task succeeded.</li></ol><p>Repeat the same evaluation multiple times and calculate the success rate.</p><p>We could also track the number of tool calls, execution time, token usage, and whether the agent modified anything outside the intended scope.</p><p>Over time, these results help us understand whether changes to our AI workflow are actually improving performance.</p><h3>Evals vs. Everyday Testing</h3><p>There’s one distinction worth making.</p><p>When you ask Claude Code to run tests after making changes, you’re verifying the output of a particular coding task.</p><p>That’s good engineering practice.</p><p>But a formal eval goes further.</p><p>It uses a repeatable task, predefined grading criteria, and often multiple trials to measure how reliably the AI system performs.</p><p>For example:</p><p><strong>Everyday verification:</strong></p><blockquote><em>Claude fixed my bug, and the tests passed.</em></blockquote><p><strong>Evaluation:</strong></p><blockquote><em>Claude successfully fixed 17 out of 20 known bugs under the same evaluation setup.</em></blockquote><p>Both are useful, but they answer different questions.</p><p>One tells us whether a particular result appears correct.</p><p>The other helps us measure the reliability of the system producing those results.</p><h3>Why Should Software Engineers Care?</h3><p>As AI coding tools become more capable, it’s tempting to evaluate them based on how impressive their outputs look.</p><p>But a convincing explanation or a clean-looking code change doesn’t necessarily mean the solution is correct.</p><p>Evals help us move from subjective impressions to measurable results.</p><p>They become particularly useful when we’re building custom agents, modifying instructions, creating reusable skills, or automating engineering workflows.</p><p>And we don’t have to start with sophisticated evaluation frameworks.</p><p>Even a small collection of five or ten representative tasks, each with clear success criteria, can teach us a lot about how reliably an AI agent performs.</p><h3>Final Thoughts</h3><p>We’ve become familiar with asking AI to write code, investigate bugs, and automate repetitive tasks.</p><p>The next step is understanding how to measure whether those systems actually work.</p><p>Just as we wouldn’t ship an application without testing it, we shouldn’t assume an AI-powered workflow is reliable simply because it produced a few impressive results.</p><p>Evals give us a way to test our assumptions, compare different approaches, and make informed improvements.</p><p>And perhaps the most important lesson is this:</p><p><strong>Don’t measure an AI agent by how confidently it says it completed a task. Measure how reliably it produces the outcome you actually wanted.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee4e8aa1022c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>