<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by piash.tanjin on Medium]]></title>
        <description><![CDATA[Stories by piash.tanjin on Medium]]></description>
        <link>https://medium.com/@piash.tanjin?source=rss-8508fb16e1c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BFLRe4mbv9GGGXDnROI-YQ.jpeg</url>
            <title>Stories by piash.tanjin on Medium</title>
            <link>https://medium.com/@piash.tanjin?source=rss-8508fb16e1c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:53:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@piash.tanjin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Simple CI/CD to EC2 Using GitHub Actions OIDC + AWS SSM Without SSH or ECR]]></title>
            <link>https://medium.com/@piash.tanjin/simple-ci-cd-to-ec2-using-github-actions-oidc-aws-ssm-without-ssh-or-ecr-ef50c7f6e074?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef50c7f6e074</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Fri, 15 May 2026 04:36:41 GMT</pubDate>
            <atom:updated>2026-05-15T04:36:41.477Z</atom:updated>
            <content:encoded><![CDATA[<p>This setup deploys code directly to an EC2 instance using <strong>GitHub Actions</strong>, <strong>AWS OIDC</strong>, and <strong>AWS Systems Manager</strong>.</p><p>No SSH key.<br>No AWS access key.<br>No ECR.</p><h3>What We Are Building</h3><p>Flow:</p><pre>Push to main<br>→ GitHub Actions starts<br>→ GitHub assumes AWS IAM role using OIDC<br>→ GitHub sends deploy command to EC2 using SSM<br>→ EC2 pulls latest code from GitHub<br>→ App is rebuilt and restarted</pre><h3>Why This Is Better Than SSH-Based CI/CD</h3><p>Traditional deployment usually stores an SSH private key in GitHub Secrets.</p><p>This works, but it has risks:</p><pre>GitHub Actions → SSH into EC2 → git pull → restart app</pre><p>Better approach:</p><pre>GitHub Actions → AWS OIDC → AWS SSM → EC2 deploy command</pre><p>Benefits:</p><ul><li>No SSH key in GitHub</li><li>No AWS access key in GitHub</li><li>No inbound SSH needed for deployment</li><li>Temporary AWS credentials only</li><li>Better audit logs through AWS CloudTrail</li></ul><h3>Step 1: Create GitHub OIDC Identity Provider in AWS</h3><p>Go to:</p><pre>AWS Console<br>→ IAM<br>→ Identity providers<br>→ Add provider</pre><p>Use:</p><pre>Provider type: OpenID Connect<br>Provider URL: https://token.actions.githubusercontent.com<br>Audience: sts.amazonaws.com</pre><p>Click <strong>Add provider</strong>.</p><h3>Step 2: Create IAM Role for GitHub Actions</h3><p>Go to:</p><pre>AWS Console<br>→ IAM<br>→ Roles<br>→ Create role</pre><p>Select:</p><pre>Trusted entity type: Web identity<br>Identity provider: token.actions.githubusercontent.com<br>Audience: sts.amazonaws.com</pre><p>This role is the role GitHub Actions will assume.</p><p>Example role name:</p><pre>github-actions-ssm-deploy-role</pre><h3>Step 3: Restrict IAM Role to Specific GitHub Repo</h3><p>Edit the trust policy of the GitHub Actions role.</p><p>Example for one repository and main branch:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Federated&quot;: &quot;arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;token.actions.githubusercontent.com:aud&quot;: &quot;sts.amazonaws.com&quot;<br>        },<br>        &quot;StringLike&quot;: {<br>          &quot;token.actions.githubusercontent.com:sub&quot;: &quot;repo:YOUR_ORG/YOUR_REPO:ref:refs/heads/main&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p>Replace:</p><pre>123456789012<br>YOUR_ORG<br>YOUR_REPO</pre><h3>Step 4: Add Permissions to GitHub Actions IAM Role</h3><p>The GitHub Actions role needs permission to send commands to EC2 through SSM.</p><p>Attach this inline policy:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;ssm:SendCommand&quot;,<br>        &quot;ssm:GetCommandInvocation&quot;<br>      ],<br>      &quot;Resource&quot;: &quot;*&quot;<br>    }<br>  ]<br>}</pre><p>For production, restrict this to your specific EC2 instance and SSM document.</p><h3>Step 5: Attach IAM Role to EC2 Instance</h3><p>This is very important.</p><p>Your EC2 instance must have an IAM role with:</p><pre>AmazonSSMManagedInstanceCore</pre><p>Without this, the SSM agent cannot register with AWS Systems Manager.</p><p>Go to:</p><pre>AWS Console<br>→ IAM<br>→ Roles<br>→ Create role</pre><p>Choose:</p><pre>Trusted entity: AWS service<br>Use case: EC2</pre><p>Attach policy:</p><pre>AmazonSSMManagedInstanceCore</pre><p>Role name example:</p><pre>ec2-ssm-role</pre><p>Now attach it to your EC2 instance:</p><pre>EC2<br>→ Instances<br>→ Select instance<br>→ Actions<br>→ Security<br>→ Modify IAM role<br>→ Select ec2-ssm-role<br>→ Update IAM role</pre><h3>Step 6: Make Sure SSM Agent Is Running</h3><p>On Ubuntu EC2:</p><pre>sudo systemctl status amazon-ssm-agent</pre><p>If it is not running:</p><pre>sudo systemctl enable amazon-ssm-agent<br>sudo systemctl start amazon-ssm-agent</pre><p>Then check:</p><pre>AWS Systems Manager<br>→ Fleet Manager<br>→ Managed Nodes</pre><p>Your EC2 instance should appear there.</p><h3>Step 7: Add GitHub Secrets</h3><p>Go to:</p><pre>GitHub Repo<br>→ Settings<br>→ Secrets and variables<br>→ Actions</pre><p>Add:</p><pre>AWS_IAM_ROLE = arn:aws:iam::123456789012:role/github-actions-ssm-deploy-role<br>AWS_REGION = ap-south-1<br>EC2_INSTANCE_ID = i-xxxxxxxxxxxxxxxxx</pre><p>AWS_IAM_ROLE is the ARN of the GitHub OIDC role, not AmazonSSMManagedInstanceCore.</p><h3>Step 8: Example GitHub Actions Workflow</h3><p>Create:</p><pre>.github/workflows/deploy.yml</pre><pre>name: Deploy to EC2 using OIDC and SSM</pre><pre>on:<br>  push:<br>    branches:<br>      - main</pre><pre>permissions:<br>  id-token: write<br>  contents: read</pre><pre>env:<br>  AWS_REGION: ${{ secrets.AWS_REGION }}<br>  EC2_INSTANCE_ID: ${{ secrets.EC2_INSTANCE_ID }}<br>  APP_DIR: /home/ubuntu/my-app<br>  PM2_APP_NAME: my-app</pre><pre>jobs:<br>  deploy:<br>    runs-on: ubuntu-latest</pre><pre>    steps:<br>      - name: Configure AWS credentials using OIDC<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          role-to-assume: ${{ secrets.AWS_IAM_ROLE }}<br>          aws-region: ${{ secrets.AWS_REGION }}</pre><pre>      - name: Deploy to EC2 using SSM<br>        run: |<br>          COMMAND_ID=$(aws ssm send-command \<br>            --instance-ids &quot;$EC2_INSTANCE_ID&quot; \<br>            --document-name &quot;AWS-RunShellScript&quot; \<br>            --comment &quot;Deploy application from GitHub Actions&quot; \<br>            --parameters commands=&quot;[<br>              \&quot;cd $APP_DIR\&quot;,<br>              \&quot;git pull origin main\&quot;,<br>              \&quot;npm ci\&quot;,<br>              \&quot;npm run build\&quot;,<br>              \&quot;pm2 restart $PM2_APP_NAME || pm2 start npm --name $PM2_APP_NAME -- start\&quot;,<br>              \&quot;pm2 save\&quot;<br>            ]&quot; \<br>            --query &quot;Command.CommandId&quot; \<br>            --output text)</pre><pre>          echo &quot;Command ID: $COMMAND_ID&quot;</pre><pre>          aws ssm wait command-executed \<br>            --command-id &quot;$COMMAND_ID&quot; \<br>            --instance-id &quot;$EC2_INSTANCE_ID&quot;</pre><pre>          aws ssm get-command-invocation \<br>            --command-id &quot;$COMMAND_ID&quot; \<br>            --instance-id &quot;$EC2_INSTANCE_ID&quot;</pre><h3>Important EC2 Setup</h3><p>Before using this workflow, your EC2 instance should already have:</p><pre>git --version<br>node -v<br>npm -v<br>pm2 -v</pre><p>If PM2 is missing:</p><pre>sudo npm install -g pm2</pre><p>Your app should already be cloned on EC2:</p><pre>cd /home/ubuntu<br>git clone https://github.com/YOUR_ORG/YOUR_REPO.git my-app<br>cd my-app<br>npm install<br>pm2 start npm --name my-app -- start<br>pm2 save</pre><h3>If Your GitHub Repo Is Private</h3><p>The EC2 instance needs permission to run:</p><pre>git pull origin main</pre><p>Use one of these:</p><pre>GitHub deploy key<br>GitHub fine-grained PAT<br>GitHub App installation token</pre><p>For a simple setup, add a deploy key to the repository and configure it on EC2.</p><h3>Common Issue</h3><p>If deployment fails because EC2 is not found by SSM, check this first:</p><pre>Does the EC2 instance have AmazonSSMManagedInstanceCore attached?</pre><p>That was the main issue in my case.</p><p>The EC2 instance had no IAM instance profile, so SSM Agent could not register.</p><p>After attaching the EC2 IAM role with AmazonSSMManagedInstanceCore, the instance appeared in Systems Manager and the deployment worked.</p><h3>Final Thought</h3><p>This is a simple and secure alternative to SSH-based GitHub Actions deployment.</p><p>GitHub Actions does not need SSH keys or AWS access keys.</p><p>It only assumes an AWS IAM role through OIDC and tells EC2 to deploy using Systems Manager.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef50c7f6e074" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Docker Networking A Visual Deep Dive Into What Happens After docker run]]></title>
            <link>https://medium.com/@piash.tanjin/demystifying-docker-networking-a-visual-deep-dive-into-what-happens-after-docker-run-e829ccc8f3fe?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e829ccc8f3fe</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[containerization]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 05:21:48 GMT</pubDate>
            <atom:updated>2026-03-12T14:59:01.426Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How a single </em><em>docker run command creates virtual networks, rewrites packet headers, and tricks your container into thinking it has real internet access.</em></p><p>When I first started using Docker, networking felt like magic. You type docker run -p 80:80 nginx, and suddenly your container is accessible from the internet. But <em>how</em>? What&#39;s actually happening between your container and the outside world?</p><p>I decided to dig deep — really deep — into Docker networking internals, and I built an interactive simulator to visualize every layer of the stack. This post walks through everything I learned: network namespaces, virtual ethernet pairs, Linux bridges, iptables NAT, MASQUERADE, and the complete journey of a packet from container to internet and back.</p><p><strong>Want to play with the simulator yourself?</strong> <a href="https://docker-network.lovable.app/">Try the interactive Docker Networking Simulator here</a> — it lets you step through every operation and see packet headers change in real time.</p><h3>What Happens When You Run docker run -p 80:80 nginx</h3><p>That single command triggers a cascade of Linux kernel operations. Let’s walk through each one.</p><h3>Step 1: Create a Network Namespace</h3><p>Docker calls the kernel to create a <strong>network namespace</strong> — a completely isolated copy of the network stack. Your container gets its own:</p><ul><li>Network interfaces (empty at first)</li><li>Routing table</li><li>iptables rules</li><li>Port space (all ports are free — even if the host is using port 80)</li></ul><pre># The kernel call:<br>unshare(CLONE_NEWNET)</pre><p>This is the foundation of Docker’s network isolation. From the container’s perspective, it’s the only thing on its own little network.</p><h3>Step 2: Create a veth Pair</h3><p>Next, Docker creates a <strong>virtual ethernet pair</strong> — think of it as a virtual network cable with two ends:</p><pre>ip link add vethXXX type veth peer name eth0<br>ip link set eth0 netns &lt;container-pid&gt;</pre><p>One end (eth0) goes inside the container&#39;s namespace. The other end (vethXXX) stays on the host. Anything you send into one end comes out the other. It&#39;s a pipe between two isolated worlds.</p><h3>Step 3: Attach to the docker0 Bridge</h3><p>The host-side end of the veth pair gets plugged into docker0 — a <strong>Linux bridge</strong> that acts as a virtual L2 switch:</p><pre>brctl addif docker0 vethXXX<br>ip link set vethXXX up</pre><p>Every container’s veth host-end plugs into this bridge. This is how containers on the same host can talk to each other — the bridge forwards frames by MAC address, just like a physical network switch.</p><h3>Step 4: Assign an IP Address</h3><p>Docker’s IPAM (IP Address Management) assigns the container an IP from the bridge’s subnet:</p><pre># Inside the container namespace:<br>ip addr add 172.17.0.2/16 dev eth0<br>ip route add default via 172.17.0.1</pre><p>The bridge itself (172.17.0.1) becomes the container’s default gateway. The container now has a complete network stack — it just doesn’t know it’s virtual.</p><h3>Step 5: Configure iptables</h3><p>This is where the real magic happens. Docker adds two critical types of iptables rules:</p><p><strong>Outbound (MASQUERADE):</strong> So containers can reach the internet.</p><pre>iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -j MASQUERADE</pre><p><strong>Inbound (DNAT):</strong> So external traffic reaches the container.</p><pre>iptables -t nat -A DOCKER -p tcp --dport 80 \<br>  -j DNAT --to-destination 172.17.0.2:80</pre><p>And just like that — your container is running with full network connectivity.</p><h3>The Complete Network Stack</h3><p>Here’s the vertical path every packet traverses:</p><pre>📦 Container (nginx)<br>   172.17.0.2<br>       │<br>   eth0 / veth pair<br>   (virtual cable)<br>       │<br>🔀 docker0 Bridge<br>   172.17.0.1<br>       │<br>🔥 iptables NAT<br>   DNAT inbound / MASQUERADE outbound<br>       │<br>   Host NIC (eth0)<br>   192.168.1.50<br>       │<br>🌐 Internet</pre><p>Every single packet — whether it’s a web request, a DNS query, or a simple ping — flows through this entire stack. The container has no idea any of this infrastructure exists.</p><h3>Tracing a Ping: The Full Round Trip</h3><p>Let’s trace what happens when you run ping 8.8.8.8 inside a container. This is where it all comes together.</p><h3>The Outgoing Request</h3><p><strong>1. Container creates the packet</strong></p><pre>SRC: 172.17.0.2  →  DST: 8.8.8.8<br>Type: ICMP Echo Request</pre><p>The container’s kernel builds the packet and routes it via the default gateway (172.17.0.1).</p><p><strong>2. Through the veth pair to docker0</strong></p><p>The packet exits eth0 inside the container, travels through the veth pair, and arrives at the docker0 bridge on the host.</p><p><strong>3. MASQUERADE rewrites the source</strong></p><p>Here’s the critical step. The packet’s source IP is 172.17.0.2 — a private address that doesn&#39;t exist on the internet. Google can&#39;t reply to it.</p><p>Docker’s MASQUERADE rule in the POSTROUTING chain rewrites the source:</p><pre>BEFORE: SRC 172.17.0.2 → DST 8.8.8.8<br>AFTER:  SRC 192.168.1.50 → DST 8.8.8.8<br>              ↑ host&#39;s public IP</pre><p>The kernel’s conntrack module saves this mapping for later.</p><p><strong>4. Packet exits the host</strong></p><p>The packet leaves the host’s physical NIC with the host’s IP as the source. To Google, it looks like the <em>host</em> sent the ping, not a container.</p><h3>The Reply</h3><p><strong>5. Google replies</strong></p><p>Google sends an ICMP Echo Reply to 192.168.1.50 (the host).</p><p><strong>6. conntrack reverses the NAT</strong></p><p>When the reply arrives at the host, conntrack recognizes it as a reply to the earlier MASQUERADE&#39;d packet. It automatically rewrites the destination:</p><pre>BEFORE: SRC 8.8.8.8 → DST 192.168.1.50<br>AFTER:  SRC 8.8.8.8 → DST 172.17.0.2<br>                           ↑ back to container!</pre><p>No explicit iptables rule is needed for this — conntrack handles the reverse automatically.</p><p><strong>7. Back to the container</strong></p><p>The kernel routes the packet to docker0 (because 172.17.0.2 is on that subnet), the bridge forwards it through the veth pair, and the container receives its ping reply.</p><pre>64 bytes from 8.8.8.8: seq=1 ttl=117 time=12.3ms</pre><p>The container never knew its IP was translated, that it crossed a bridge, or that iptables rewrote the packet headers twice. It just sees a normal ping response.</p><h3>The iptables Deep Dive</h3><p>Docker relies heavily on two iptables tables:</p><h3>The nat Table (Address Translation)</h3><p>This is Docker’s primary table. It has two key chains:</p><p><strong>PREROUTING (DNAT)</strong> — For inbound traffic. When someone hits host:80, Docker&#39;s rule rewrites the destination to the container&#39;s IP:</p><pre>-A DOCKER -p tcp --dport 80 \<br>  -j DNAT --to-destination 172.17.0.2:80</pre><p><strong>POSTROUTING (MASQUERADE)</strong> — For outbound traffic. Rewrites the source IP so containers can reach the internet:</p><pre>-A POSTROUTING -s 172.17.0.0/16 \<br>  ! -o docker0 -j MASQUERADE</pre><h3>The filter Table (Accept/Drop)</h3><p>The FORWARD chain controls whether packets are allowed to be forwarded between interfaces:</p><pre>-A FORWARD -o docker0 -j DOCKER<br>-A DOCKER -d 172.17.0.2 -p tcp --dport 80 -j ACCEPT</pre><h3>Processing Order</h3><p>When a packet hits a chain, tables are processed in this order:</p><pre>raw → conntrack → mangle → nat → filter</pre><p>For Docker, only nat (DNAT/SNAT) and filter (ACCEPT/DROP) matter. Everything else passes through untouched.</p><h3>The Docker + UFW Trap</h3><p>If you’re using UFW (Uncomplicated Firewall) on your server, here’s something that catches almost everyone off guard:</p><p><strong>Docker bypasses UFW.</strong></p><p>UFW is just a frontend that generates iptables rules. It inserts rules into the INPUT chain. But Docker writes directly to the nat table&#39;s PREROUTING chain — which executes <em>before</em> UFW&#39;s rules ever see the packet.</p><pre># You set UFW to deny port 8080:<br>$ ufw deny 8080</pre><pre># But Docker&#39;s DNAT rule in PREROUTING runs first:<br>-A DOCKER -p tcp --dport 8080 -j DNAT --to 172.17.0.2:80</pre><pre># Port 8080 is OPEN despite your UFW deny!</pre><h3>The Fix</h3><p>Use the DOCKER-USER chain — Docker checks this chain <em>before</em> its own rules:</p><pre>iptables -I DOCKER-USER -p tcp --dport 80 \<br>  ! -s 10.0.0.0/8 -j DROP</pre><p>Or bind to localhost only:</p><pre>docker run -p 127.0.0.1:8080:80 nginx</pre><h3>Docker Network Types: When to Use Which</h3><h3>Bridge (Default)</h3><p><strong>Best for:</strong> Most applications. Web stacks, microservices on a single host, development.</p><p>Each container gets isolation with its own namespace, IP, and the full NAT stack. Containers on user-defined bridge networks get automatic DNS resolution by container name.</p><pre>docker network create myapp<br>docker run --network myapp --name redis redis<br># Other containers can reach: redis:6379</pre><h3>Host</h3><p><strong>Best for:</strong> Performance-critical apps, network monitoring tools, apps with many dynamic ports.</p><p>The container shares the host’s network namespace directly. No veth, no bridge, no NAT. Zero overhead — but no isolation either, and port conflicts are possible.</p><pre>docker run --network host nginx<br># nginx binds directly to host:80</pre><h3>Overlay (Swarm)</h3><p><strong>Best for:</strong> Multi-host deployments, Docker Swarm clusters, cross-datacenter services.</p><p>Creates a virtual L2 network over the physical L3 network using VXLAN tunneling. Containers on different physical hosts can communicate as if they’re on the same LAN.</p><pre>docker network create -d overlay --opt encrypted prod-network</pre><h3>None</h3><p><strong>Best for:</strong> Security-sensitive processing, batch jobs, custom networking.</p><p>The container gets only a loopback interface. Complete network isolation — zero attack surface.</p><pre>docker run --network none my-crypto-signer<br># Cannot leak data over network</pre><h3>MASQUERADE vs SNAT: What’s the Difference?</h3><p>Both rewrite the source IP of outgoing packets, but:</p><ul><li><strong>MASQUERADE</strong> auto-detects the outgoing interface’s IP address each time. Docker uses this because the host’s IP might change (DHCP, cloud instances).</li><li><strong>SNAT</strong> (--to-source x.x.x.x) uses a hardcoded IP. Slightly faster since it doesn&#39;t need to look up the interface IP, but breaks if the IP changes.</li></ul><p>Docker defaults to MASQUERADE because it’s safer for dynamic environments.</p><h3>Key Takeaways</h3><ol><li><strong>Namespaces provide isolation</strong> — each container gets its own complete network stack.</li><li><strong>veth pairs are virtual cables</strong> — they connect the isolated container namespace to the host’s docker0 bridge.</li><li><strong>docker0 is a virtual switch</strong> — it handles L2 forwarding between containers and the host.</li><li><strong>iptables does all the heavy lifting</strong> — DNAT for inbound port mapping, MASQUERADE for outbound internet access, and filter FORWARD for access control.</li><li><strong>conntrack is the unsung hero</strong> — it tracks connections and automatically reverses NAT on reply packets without any explicit rules.</li><li><strong>UFW doesn’t protect Docker ports</strong> — use the DOCKER-USER chain or bind to localhost.</li><li><strong>The container is completely unaware</strong> — from its perspective, it has a normal network with a normal IP and a normal gateway. All the translation happens transparently.</li></ol><h3>Play With It Yourself</h3><p>I built an interactive simulator that lets you step through every operation, watch packet headers change in real time, and understand each network type visually.</p><p><a href="https://docker-network.lovable.app/"><strong>Try the Docker Networking Simulator</strong></a></p><p>It includes:</p><ul><li>Step-by-step docker run flow with the actual Linux commands</li><li>A full ping simulation showing MASQUERADE and conntrack in action</li><li>iptables chain-by-chain packet tracing</li><li>Network type comparisons with real-world use cases</li><li>UFW vs iptables explanation</li><li>MASQUERADE deep dive</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e829ccc8f3fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DEEP DIVE RabbitMQ Internals Exchanges, Queues, Consumers & How It Ensures Consistency]]></title>
            <link>https://medium.com/@piash.tanjin/deep-dive-rabbitmq-internals-exchanges-queues-consumers-how-it-ensures-consistency-0b35590dd61c?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b35590dd61c</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[queue]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 14:15:14 GMT</pubDate>
            <atom:updated>2026-03-06T14:15:14.258Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comprehensive guide to how RabbitMQ works under the hood — from producers and exchanges to queues, acknowledgements, multiple consumers, and the modern quorum queue architecture that replaced classic mirrored queues. Compared side-by-side with MySQL to make every concept concrete.</em></p><p>For live simulation try this out <a href="https://rabbit-route-pal.lovable.app/">https://rabbit-route-pal.lovable.app</a></p><h3>Why RabbitMQ Matters</h3><p>If Kafka is the highway built for massive, ordered event streams, then RabbitMQ is the intelligent postal service — routing individual messages to the right recipients based on sophisticated rules. Originally developed by Rabbit Technologies and later acquired by Pivotal (now part of Broadcom), RabbitMQ has been the backbone of enterprise messaging for over 15 years.</p><p>RabbitMQ is a message broker. It accepts messages from producers, routes them through exchanges, and delivers them to queues where consumers pick them up. It implements the Advanced Message Queuing Protocol (AMQP) and as of RabbitMQ 4.0, supports both AMQP 0.9.1 and AMQP 1.0 natively. But the real question most developers struggle with is: how does RabbitMQ ensure that messages are not lost, not duplicated, and not consumed by the wrong consumer? That is what this article answers.</p><p>We will use the same order payload from our Kafka deep dive as the running example, and compare RabbitMQ’s behavior to MySQL at every step.</p><h3>The Running Example: An Order Event</h3><pre>{<br>  &quot;orderId&quot;: 1,<br>  &quot;name&quot;: &quot;piash&quot;,<br>  &quot;item&quot;: &quot;Laptop&quot;,<br>  &quot;amount&quot;: 75000,<br>  &quot;timestamp&quot;: &quot;2025-03-06T10:30:00Z&quot;<br>}</pre><p>This order will travel through the entire RabbitMQ pipeline: Producer → Exchange → Queue → Consumer. Let us trace every step and understand how RabbitMQ handles each one differently from both a database and from Kafka.</p><h3>The RabbitMQ Architecture: A Four-Part Journey</h3><p>Unlike Kafka, where producers send directly to topic partitions, RabbitMQ introduces an intermediary layer called an exchange. The message flow is: Producer → Exchange → Binding → Queue → Consumer. Each component has a specific role, and understanding them is key to understanding RabbitMQ’s consistency guarantees.</p><h3>Part 1: The Producer Publishes</h3><p>A producer is any application that sends messages to RabbitMQ. When our e-commerce backend processes Piash’s order, it publishes the message to an exchange — not directly to a queue. This is the first critical difference from Kafka, where producers write directly to topic partitions.</p><pre>// Node.js example using amqplib<br>channel.publish(&quot;order.exchange&quot;, &quot;order.created&quot;, Buffer.from(<br>  JSON.stringify({ orderId: 1, name: &quot;piash&quot;, item: &quot;Laptop&quot; })<br>));</pre><p>The producer specifies two things: the <strong>exchange name</strong> (“order.exchange”) and a <strong>routing key</strong> (“order.created”). The exchange uses the routing key to decide which queue(s) should receive the message.</p><h3>Part 2: The Exchange Routes</h3><p>An exchange is RabbitMQ’s routing engine. It receives messages from producers and routes them to one or more queues based on rules called bindings. There is no equivalent in Kafka or MySQL — it is a concept unique to the AMQP model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*r_ZReWXs83uWq2u8SuYImQ.png" /></figure><pre>Exchange: &quot;order.exchange&quot; (type: direct)<br><br><br>Bindings:<br>  &quot;order.created&quot;  →  queue: &quot;order-processing-queue&quot;<br>  &quot;order.created&quot;  →  queue: &quot;order-analytics-queue&quot;<br>  &quot;order.cancelled&quot; → queue: &quot;cancellation-queue&quot;<br><br><br>Message arrives with routing key &quot;order.created&quot;<br>  → Delivered to: order-processing-queue AND order-analytics-queue<br>  → NOT delivered to: cancellation-queue</pre><p><strong>KEY DIFFERENCE FROM KAFKA</strong></p><p>In Kafka, routing is done by the producer (choosing a partition via key hashing). In RabbitMQ, routing is done by the broker (the exchange applies rules to decide which queues receive the message). This gives RabbitMQ much more flexible routing but adds a component to manage.</p><h3>Part 3: The Queue Stores</h3><p>Once the exchange routes our order, it lands in one or more queues. A queue in RabbitMQ is an ordered buffer of messages waiting to be consumed. This is where RabbitMQ and MySQL share the most conceptual overlap — and also where they differ most fundamentally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*pHGAhlJ7QpR9OvBoFXLyrg.png" /></figure><p><strong>THE CRITICAL DIFFERENCE</strong></p><p>In MySQL, reading a row does not remove it. In RabbitMQ, consuming a message removes it from the queue (after acknowledgement). A message is meant to be processed once by one consumer, then it is gone. This is the “smart broker, dumb consumer” model — the broker decides who gets what and tracks delivery state.</p><h3>Part 4: The Consumer Processes</h3><p>Consumers subscribe to queues and receive messages as they become available. When our order-processing service connects to “order-processing-queue”, RabbitMQ pushes messages to it. This is fundamentally different from Kafka, where consumers pull messages at their own pace.</p><pre>// Consumer subscribes to the queue<br>channel.consume(&quot;order-processing-queue&quot;, (msg) =&gt; {<br>  const order = JSON.parse(msg.content.toString());<br>  // Process order #1 from piash<br>  processOrder(order);<br>  // Acknowledge: tell RabbitMQ to remove this message<br>  channel.ack(msg);<br>});</pre><h3>Multiple Consumers: How RabbitMQ Avoids Double Processing</h3><p>This is the question that confuses most developers coming from a database background. In MySQL, if two services query the same table, they both see the same rows. In RabbitMQ, multiple consumers on the same queue compete for messages. Let us break down exactly how this works.</p><h3>Competing Consumers (Same Queue)</h3><p>When multiple consumers subscribe to the same queue, RabbitMQ distributes messages among them using round-robin dispatch. Each message is delivered to exactly one consumer.</p><pre>Queue: &quot;order-processing-queue&quot;<br>Consumers: C1, C2, C3<br><br><br>Message 1 (orderId: 1) → C1<br>Message 2 (orderId: 2) → C2<br>Message 3 (orderId: 3) → C3<br>Message 4 (orderId: 4) → C1  // round-robin cycles back</pre><p>This is fundamentally different from Kafka’s consumer group model. In Kafka, you assign <strong>partitions</strong> to consumers. In RabbitMQ, you assign <strong>individual messages</strong> to consumers. This means RabbitMQ can distribute work more granularly — no consumer sits idle just because there are fewer partitions than consumers.</p><h3>Fair Dispatch with Prefetch</h3><p>By default, round-robin can be unfair. If C1 is slow and C2 is fast, they still alternate. The solution is <strong>prefetch count</strong> (also called QoS). Setting channel.prefetch(1) tells RabbitMQ: “do not send this consumer a new message until it has acknowledged the previous one.” This ensures busy consumers do not accumulate a backlog while fast consumers sit idle.</p><pre>// Set prefetch to 1: only 1 unacknowledged message at a time<br>channel.prefetch(1);<br><br><br>// Now RabbitMQ distributes based on availability, not rotation<br>Message 1 → C1 (processing...)<br>Message 2 → C2 (C2 is free, gets the next message)<br>Message 3 → C2 (C1 still busy, C2 gets another)<br>Message 4 → C3<br>C1 acks Message 1<br>Message 5 → C1 (now free again)</pre><h3>Independent Consumers (Different Queues)</h3><p>What if you want multiple services to each process every message — like Kafka’s consumer groups? In RabbitMQ, you achieve this by binding multiple queues to the same exchange. Each queue gets its own copy of every message.</p><pre>Exchange: &quot;order.exchange&quot; (type: fanout)<br><br><br>Queue: &quot;order-processing-queue&quot;  →  Processing Service<br>Queue: &quot;order-analytics-queue&quot;   →  Analytics Service<br>Queue: &quot;order-notification-queue&quot; → Notification Service<br><br><br>Producer publishes orderId: 1<br>  → Copy 1 goes to order-processing-queue<br>  → Copy 2 goes to order-analytics-queue<br>  → Copy 3 goes to order-notification-queue</pre><p>Each service consumes from its own queue independently, just like Kafka’s separate consumer groups. The key difference: in Kafka, the broker stores one copy and each group reads from the same log. In RabbitMQ, the broker creates separate copies in separate queues. This uses more memory and disk but gives each service its own buffer, backpressure, and dead-letter handling.</p><p><strong>COMPARISON: KAFKA vs RABBITMQ MULTI-CONSUMER</strong></p><p>Kafka: One topic, multiple consumer groups, each group reads same log independently. RabbitMQ: One exchange, multiple queues (each gets a copy), each queue has its own consumers. Kafka is more storage-efficient; RabbitMQ provides more independent control per consumer.</p><h3>Single Active Consumer</h3><p>RabbitMQ also supports a <strong>single active consumer</strong> pattern. When enabled on a queue (via the x-single-active-consumer argument), only one consumer processes messages at a time, even if multiple are subscribed. The others act as standby. If the active consumer disconnects, RabbitMQ automatically fails over to the next one. This is useful when strict ordering matters and you cannot have messages interleaved across consumers.</p><h3>Acknowledgements: RabbitMQ’s Consistency Backbone</h3><p>This is where RabbitMQ’s reliability story lives. The acknowledgement system is a two-way handshake between producers, the broker, and consumers. It is what prevents message loss and duplication.</p><h3>Publisher Confirms (Producer → Broker)</h3><p>When you publish a message, how do you know the broker actually received it? By default, you do not. The AMQP protocol’s basic.publish is fire-and-forget. To get guarantees, you enable publisher confirms.</p><pre>// Enable publisher confirms<br>await channel.confirmSelect();<br><br><br>// Publish with confirmation<br>channel.publish(&quot;order.exchange&quot;, &quot;order.created&quot;, messageBuffer);<br><br><br>// Wait for broker confirmation<br>await channel.waitForConfirms();<br>// If we reach here, the broker has persisted the message</pre><p>With publisher confirms enabled, RabbitMQ will send a basic.ack back to the producer only after the message has been written to disk (for durable queues) and, for quorum queues, replicated to a majority of nodes. If the broker cannot store the message, it sends a basic.nack, and the producer can retry.</p><h3>Consumer Acknowledgements (Consumer → Broker)</h3><p>On the consumption side, acknowledgements tell RabbitMQ when it is safe to remove a message from the queue. There are three modes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*4-i9KcEOeemoSqzKN7Damw.png" /></figure><pre>1. RabbitMQ delivers message to C1 (marks as &quot;unacknowledged&quot;)<br>2. C1 processes the order: INSERT INTO orders VALUES (1, &quot;piash&quot;)<br>3. C1 sends channel.ack(deliveryTag)<br>4. RabbitMQ removes the message from the queue permanently<br><br><br>If C1 crashes between step 1 and step 3:<br>  → Message stays &quot;unacknowledged&quot;<br>  → RabbitMQ detects C1’s channel/connection is closed<br>  → Message is requeued to the front of the queue<br>  → C2 or C3 picks it up and processes it</pre><p><strong>THIS IS HOW RABBITMQ PREVENTS MESSAGE LOSS</strong></p><p>A message is only removed from the queue after a consumer explicitly acknowledges it. If the consumer dies, the message goes back to the queue. Combined with publisher confirms and durable queues, this gives you an end-to-end durability guarantee.</p><h3>The Duplicate Problem</h3><p>But wait — what if C1 processed the order and inserted it into the database, but then crashed before sending the ack? RabbitMQ will redeliver the message to C2, who processes it again. Now orderId:1 is inserted twice.</p><p>This is the same “at-least-once delivery” problem that Kafka has. The solution is the same: make your consumers idempotent.</p><pre>// Idempotent consumer: use database unique constraint<br>try {<br>  await db.query(<br>    &quot;INSERT INTO orders (order_id, name, item) VALUES ($1, $2, $3)<br>     ON CONFLICT (order_id) DO NOTHING&quot;,<br>    [order.orderId, order.name, order.item]<br>  );<br>  channel.ack(msg);<br>} catch (err) {<br>  channel.nack(msg, false, true); // requeue for retry<br>}</pre><h3>Quorum Queues: RabbitMQ’s Modern Replication Engine</h3><p>For years, RabbitMQ’s high availability story was built on classic mirrored queues — a replication mechanism that was slow, unpredictable under network partitions, and failed Jepsen testing. Starting with RabbitMQ 3.8, a new queue type called quorum queues was introduced as the replacement. With RabbitMQ 4.0, classic queue mirroring was completely removed. Quorum queues are now the only replicated queue type available.</p><h3>How Quorum Queues Work</h3><p>Quorum queues use the Raft consensus algorithm — the same algorithm that Kafka’s KRaft and etcd use. Each quorum queue has a leader and multiple followers distributed across different RabbitMQ nodes:</p><pre>Quorum Queue: &quot;order-processing-queue&quot; (replication factor: 3)<br><br><br>Node 1: Leader   ← handles all reads and writes<br>Node 2: Follower  ← replicates from leader<br>Node 3: Follower  ← replicates from leader<br><br><br>Producer publishes orderId: 1<br>  1. Leader writes to WAL (write-ahead log)<br>  2. Leader replicates to followers IN PARALLEL<br>  3. Once 2/3 nodes confirm (quorum achieved)<br>  4. Leader sends publisher confirm to producer</pre><p>The key insight: publisher confirms are only sent after a <strong>majority</strong> (a quorum) of nodes have persisted the message. This means that even if one node crashes immediately after the confirm, the message survives on the remaining nodes. For a 3-node quorum, any 2 nodes can fail sequentially (not simultaneously) without losing data.</p><h3>Quorum Queues vs Classic Mirrored Queues</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*zGGTQbR3oUtX0julopsBoQ.png" /></figure><h3>When NOT to Use Quorum Queues</h3><p>Quorum queues are not universally better. They are designed for long-lived, mission-critical queues. For temporary queues (like RPC reply queues), exclusive queues, or scenarios where data safety is not a priority, classic (non-mirrored) queues are still available and more appropriate. For high-throughput log-style workloads with message replay, RabbitMQ Streams (an append-only log similar to Kafka topics) may be a better fit.</p><h3>RabbitMQ 4.x: The Modern Era</h3><p>RabbitMQ 4.0, released in September 2024, represents the most significant evolution of the platform since its creation. The latest stable release as of early 2026 is RabbitMQ 4.2.x, with 4.3 under active development.</p><h3>Native AMQP 1.0 Support</h3><p>AMQP 1.0 is now a core protocol in RabbitMQ 4.0, no longer requiring a plugin. The implementation offers significant performance improvements — more than double the throughput of the old 3.13 plugin for some workloads. RabbitMQ now natively supports four protocols: AMQP 0.9.1, AMQP 1.0, MQTT, and Streams.</p><h3>Khepri: The New Metadata Store</h3><p>RabbitMQ 4.x introduces <strong>Khepri</strong>, a Raft-based metadata store designed to replace Mnesia (the Erlang database that historically managed RabbitMQ’s internal metadata like exchanges, queues, bindings, and users). Khepri provides stronger consistency guarantees and better behavior under network partitions. It is fully supported in 4.0+ and is expected to become the default metadata store in future versions.</p><h3>Streams: RabbitMQ’s Answer to Kafka-Style Consumption</h3><p>RabbitMQ Streams are an append-only log data structure (very similar to Kafka topics) that allow consumers to read messages non-destructively, replay from any offset, and support high fan-out. Streams complement queues: use queues for work distribution (competing consumers) and streams for event sourcing and audit logs. In RabbitMQ 4.1+, AMQP 1.0 clients can use SQL-like filter expressions on streams for server-side message filtering.</p><h3>RabbitMQ vs MySQL vs Kafka: The Complete Picture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*34iBuR0jzMe0_mcalJPTpg.png" /></figure><h3>Putting It All Together: The Complete Order Flow</h3><p>Let us trace our order through the entire modern RabbitMQ 4.x stack:</p><p><strong>1. Producer </strong>publishes { orderId: 1, name: “piash” } to “order.exchange” with routing key “order.created” and waits for a publisher confirm.</p><p><strong>2. Exchange </strong>(type: direct) matches routing key “order.created” against its bindings and routes copies to “order-processing-queue” and “order-analytics-queue”.</p><p><strong>3. Quorum queue leader </strong>on Node 1 receives the message, writes to its WAL, replicates in parallel to followers on Node 2 and Node 3. Once 2 of 3 nodes confirm, the leader sends the publisher confirm back to the producer.</p><p><strong>4. Processing service </strong>(C1 and C2 competing on “order-processing-queue” with prefetch=1) — RabbitMQ dispatches orderId:1 to C1 since it is available. C1 processes the order, inserts into the database with an idempotent upsert, and sends channel.ack.</p><p><strong>5. Analytics service </strong>independently consumes the same order from “order-analytics-queue” — its own separate queue, own acknowledgement, own pace. No conflict with the processing service.</p><p><strong>6. If C1 crashes </strong>before acknowledging, RabbitMQ detects the broken connection, requeues the message, and C2 picks it up. Because C2’s consumer is idempotent, even if C1 partially processed it, the result is consistent.</p><p><strong>7. If Node 1 (leader) crashes, </strong>the Raft protocol triggers a leader election among the remaining followers. Node 2 or Node 3 becomes the new leader and continues serving the queue — no messages are lost because they were already replicated to a quorum.</p><h3>Production Best Practices</h3><p><strong>Always use publisher confirms </strong>for any message you cannot afford to lose. Fire-and-forget publishing is only acceptable for metrics or logging where occasional loss is tolerable.</p><p><strong>Always use manual acknowledgements. </strong>Auto-ack mode removes messages before you process them. If your consumer crashes mid-processing, the message is gone forever.</p><p><strong>Set prefetch count appropriately. </strong>A prefetch of 1 gives the fairest distribution but lowest throughput. For most workloads, prefetch of 10–50 provides a good balance between fairness and performance.</p><p><strong>Use quorum queues for anything critical. </strong>Orders, payments, user events — anything where losing a message has business impact should use quorum queues with a replication factor of 3 or 5 (always odd numbers for proper Raft quorum).</p><p><strong>Set delivery limits on quorum queues. </strong>Configure x-delivery-limit to prevent poison messages from being requeued indefinitely. Messages exceeding the limit are dead-lettered for manual inspection.</p><p><strong>Make consumers idempotent. </strong>At-least-once delivery is the default guarantee. Use database unique constraints, deduplication keys, or idempotency tokens to handle redelivery gracefully.</p><p><strong>Consider Streams for replay and audit. </strong>If you need consumers to re-read historical messages, use RabbitMQ Streams instead of queues. Streams provide Kafka-like offset-based consumption without destroying messages.</p><p><strong>Monitor queue depth and consumer utilization. </strong>Use the built-in Prometheus metrics and Grafana dashboards (shipped with RabbitMQ 4.x) to track message rates, queue depth, unacknowledged counts, and Raft health.</p><h3>Final Thoughts</h3><p>RabbitMQ’s design philosophy is fundamentally different from Kafka’s. Where Kafka says “store everything as an ordered log and let consumers figure it out,” RabbitMQ says “route messages intelligently and ensure exactly the right consumer processes each one.” Neither is universally better — they solve different problems.</p><p>RabbitMQ excels at complex routing (direct, topic, fanout, headers exchanges), work distribution among competing consumers with fair dispatch, request-reply patterns and RPC, and scenarios where messages should be consumed exactly once and then removed.</p><p>Kafka excels at massive-scale event streaming, log replay and event sourcing, maintaining strict ordering across partitions, and long-term retention of event history.</p><p>With RabbitMQ 4.x — quorum queues built on Raft, native AMQP 1.0, Khepri metadata store, and Streams for log-style workloads — the gap between the two is narrowing. RabbitMQ can now handle many workloads that previously required Kafka, and vice versa. The best choice depends on your specific consistency, routing, and throughput requirements.</p><p>Understanding these internals — exchanges, bindings, acknowledgements, quorum replication, and the competing consumer pattern — is what separates developers who use RabbitMQ from developers who truly understand it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b35590dd61c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DEEP DIVE Apache Kafka & KRaft: The Architecture Behind Real-Time Data Streaming]]></title>
            <link>https://medium.com/@piash.tanjin/deep-dive-apache-kafka-kraft-the-architecture-behind-real-time-data-streaming-156656a75b4b?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/156656a75b4b</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[kraft]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Fri, 06 Mar 2026 13:22:22 GMT</pubDate>
            <atom:updated>2026-03-06T13:23:45.833Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comprehensive guide to how Kafka works under the hood — from producers and partitions to consumer groups and the new KRaft consensus protocol that replaced ZooKeeper.</em></p><p>You can also explore an interactive, real-world simulation to better understand how Kafka works: <a href="https://kafka-flow-buddy.lovable.app/">https://kafka-flow-buddy.lovable.app/</a></p><h3>Why Kafka Matters</h3><p>If you have ever built a system that needs to process millions of events per second — user clicks, financial transactions, IoT sensor readings, order placements — you have almost certainly encountered Apache Kafka. Originally developed at LinkedIn and later open-sourced through the Apache Software Foundation, Kafka has become the de facto standard for real-time event streaming.</p><p>But here is what most tutorials skip: they teach you how to use Kafka without explaining how it actually works. What happens inside a partition? How do consumer groups avoid stepping on each other? And why did the Kafka community spend years replacing ZooKeeper with KRaft?</p><p>This article walks through the entire Kafka architecture from first principles, using a concrete order payload as our running example. By the end, you will understand not just the what, but the why behind every design decision.</p><h3>The Running Example: An Order Event</h3><p>Throughout this article, we will follow a simple order event through the entire Kafka pipeline:</p><pre>{<br>  &quot;orderId&quot;: 1,<br>  &quot;name&quot;: &quot;piash&quot;,<br>  &quot;item&quot;: &quot;Laptop&quot;,<br>  &quot;amount&quot;: 75000,<br>  &quot;timestamp&quot;: &quot;2025-03-06T10:30:00Z&quot;<br>}</pre><p>This payload gets created by a Producer, routed to a specific Partition within a Topic, stored as an immutable record with an Offset, and eventually consumed by one or more Consumer Groups. Let us trace each step.</p><h3>Step 1: The Producer Creates an Event</h3><p>A Producer is any application that publishes data to Kafka. When our e-commerce backend receives an order from Piash, it serializes the order into a Kafka message and sends it to a topic.</p><h3>What is a Topic?</h3><p>A topic is a named category or feed to which records are published. Think of it as a logical channel. Our topic might be called “orders”. Topics are further divided into partitions for parallelism and scalability.</p><h3>How the Producer Decides Which Partition</h3><p>This is where it gets interesting. The producer does not randomly pick a partition. It uses a partitioning strategy:</p><p><strong>Key-based hashing: </strong>If you provide a message key (say, orderId), Kafka computes hash(key) % numPartitions to determine the target partition. This guarantees that all messages with the same key always land in the same partition, preserving ordering for that key.</p><p><strong>Round-robin: </strong>If no key is provided, Kafka distributes messages evenly across partitions using a round-robin or sticky partitioner approach.</p><p>For our example with 3 partitions:</p><pre>hash(orderId: 1) % 3 = Partition 0<br>hash(orderId: 2) % 3 = Partition 1<br>hash(orderId: 3) % 3 = Partition 2<br>hash(orderId: 4) % 3 = Partition 0  // same key pattern cycles back</pre><p><strong>KEY INSIGHT</strong></p><p>Same key = same partition = guaranteed ordering. This is critical for use cases where event order matters, such as processing all orders from the same customer sequentially.</p><h3>Step 2: Inside the Topic — Partitions and Offsets</h3><p>Once the producer sends our order, it lands inside a specific partition within the “orders” topic. To understand what happens next, we need to understand the core data structure of Kafka.</p><h3>Partitions are Append-Only Logs</h3><p>Each partition is an ordered, immutable sequence of records. New messages are always appended to the end. You cannot update or delete a message once it is written. This is fundamentally different from a database table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*HuIOBNqS0wyT2JRbotY5TA.png" /></figure><h3>Offsets: The Partition’s Primary Key</h3><p>Every message within a partition is assigned a monotonically increasing integer called an offset. The first message gets offset 0, the next gets 1, and so on. This offset uniquely identifies a message within its partition.</p><pre>Partition 0:<br>  Offset 0 → { orderId: 1, name: &quot;piash&quot;, item: &quot;Laptop&quot; }<br>  Offset 1 → { orderId: 4, name: &quot;piash&quot;, item: &quot;Mouse&quot; }<br><br><br>Partition 1:<br>  Offset 0 → { orderId: 2, name: &quot;rahim&quot;, item: &quot;Phone&quot; }<br><br><br>Partition 2:<br>  Offset 0 → { orderId: 3, name: &quot;karim&quot;, item: &quot;Tablet&quot; }</pre><p>Notice that offsets are <strong>local to each partition</strong>. Offset 0 in Partition 0 and Offset 0 in Partition 1 are completely different messages. There is no global primary key across partitions — the identity is always the combination of (Topic, Partition, Offset).</p><h3>Replication: How Kafka Protects Your Data</h3><p>Each partition is replicated across multiple brokers. One replica is the leader (handles all reads and writes), and the others are followers that stay in sync. If a broker fails, one of the followers is promoted to leader. The replication factor (typically 3 in production) determines how many copies exist.</p><h3>Step 3: Consuming Messages — Consumer Groups</h3><p>Now that our order sits in Partition 0 at Offset 0, something needs to read it. This is where consumers and consumer groups come in, and where Kafka’s design truly shines.</p><h3>The Golden Rule of Consumer Groups</h3><p><strong>RULE</strong></p><p>Within a consumer group, each partition is assigned to exactly one consumer at any given time. Different consumer groups consume independently and maintain their own offsets.</p><p>Let us say our “orders” topic has 3 partitions and Consumer Group A has 2 consumers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*usj4STq0cxyl4l3c9RtMnQ.png" /></figure><h3>What If You Add More Consumers Than Partitions?</h3><p>If Consumer Group A had 4 consumers but only 3 partitions, one consumer would sit idle. A partition can never be split across two consumers within the same group. This is why the number of partitions is a critical design decision — it sets the upper bound on parallelism within a single consumer group.</p><h3>Offset Tracking: Where We Left Off</h3><p>Each consumer tracks which offset it has processed for each partition. This offset is committed (saved) to a special internal Kafka topic called __consumer_offsets. If a consumer crashes and restarts, it resumes from the last committed offset — not from the beginning.</p><pre>Consumer C1 processing Partition 0:<br>  Read offset 0 → Process order #1 → Commit offset 1<br>  Read offset 1 → Process order #4 → Commit offset 2<br>  [C1 crashes]<br>  [C1 restarts] → Resume from committed offset 2</pre><h3>Cross-Group Consumption: No Conflict</h3><p>Here is the part that confuses many developers: different consumer groups read the same data completely independently. If Group A (order processing service) and Group B (analytics service) both subscribe to the “orders” topic, they each maintain their own set of offsets. Group A reading offset 5 does not affect Group B’s position at all.</p><p>This is by design. Kafka implements a publish-subscribe model where topics are like broadcast channels. Every subscriber group gets its own independent view of the data.</p><p><strong>ANALOGY</strong></p><p>Think of a Kafka topic as a newspaper printing press. Each consumer group is a separate subscription. The printing press does not care how many subscribers it has — it prints the same pages regardless. Each subscriber reads at their own pace, bookmarks their own page, and never interferes with other subscribers.</p><h3>Step 4: Preventing Duplicate Processing</h3><p>Kafka’s default guarantee is at-least-once delivery. This means a message might be delivered more than once if a consumer crashes after processing but before committing the offset. So how do production systems handle this?</p><h3>Strategy 1: Idempotent Consumers</h3><p>The most common approach. Make your consumer’s processing logic idempotent — meaning processing the same message twice produces the same result. For our order example:</p><pre>// Before inserting an order, check if it already exists<br>INSERT INTO orders (order_id, name, item)<br>VALUES (1, &quot;piash&quot;, &quot;Laptop&quot;)<br>ON CONFLICT (order_id) DO NOTHING;</pre><p>Even if Kafka delivers orderId:1 twice, the second insert is silently ignored.</p><h3>Strategy 2: Kafka Transactions (Exactly-Once)</h3><p>Kafka supports exactly-once semantics (EOS) through its transactional API. Enable enable.idempotence=true on the producer and isolation.level=read_committed on the consumer. This ensures that even in failure scenarios, each message is processed exactly once — but it comes with a throughput cost.</p><h3>Strategy 3: Manual Offset Management</h3><p>Commit the offset and your business logic in a single atomic database transaction. If either fails, both are rolled back, and the message will be reprocessed on the next poll.</p><h3>The KRaft Revolution: Goodbye ZooKeeper</h3><p>Now we come to one of the most significant architectural changes in Kafka’s history. For over a decade, Kafka depended on Apache ZooKeeper for metadata management — tracking which brokers are alive, which partitions exist, who the leaders are, and cluster configuration. With Kafka 4.0, released in March 2025, ZooKeeper has been completely removed and replaced by KRaft.</p><h3>What Was ZooKeeper’s Role?</h3><p>ZooKeeper was a separate distributed coordination service. In the old architecture, it handled controller election (deciding which broker manages partition leadership), cluster membership (tracking which brokers are alive), topic configuration (storing partition counts, replication factors, and ACLs), and consumer group coordination in older Kafka versions.</p><p>The problem? Running two distributed systems (Kafka + ZooKeeper) meant double the operational overhead, double the failure modes, and a hard scalability ceiling around 200,000 partitions.</p><h3>What is KRaft?</h3><p>KRaft stands for <strong>Kafka Raft</strong> — Kafka’s own implementation of the Raft consensus protocol, introduced through KIP-500. Instead of relying on an external system, Kafka now manages its own metadata using a quorum of dedicated controller nodes within the cluster itself.</p><p><strong>HOW KRAFT WORKS</strong></p><p>A set of controller nodes (typically 3 or 5) form a Raft quorum. One is elected the active controller and manages a special metadata partition log. All cluster metadata — topics, partitions, ISRs, configurations — is stored in this internal log. When the active controller fails, the Raft protocol ensures a new leader is elected from the remaining controllers, and because they already have the metadata in memory, failover is nearly instantaneous.</p><h3>ZooKeeper vs KRaft: Side by Side</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*fNtKx17FReNcU2IZStneLQ.png" /></figure><h3>KRaft Deployment Modes</h3><p><strong>Combined mode: </strong>A single process acts as both broker and controller. Suitable for development and small deployments.</p><p><strong>Isolated mode: </strong>Controllers run on dedicated nodes, separate from brokers. This is the recommended approach for production because it isolates metadata management from data plane operations, ensuring that heavy message throughput does not impact cluster coordination.</p><h3>The Migration Path</h3><p>If you are running an existing Kafka cluster with ZooKeeper, you cannot jump directly to Kafka 4.0. The recommended path is to first upgrade to Kafka 3.7 or later (which has production-ready migration tooling), then perform an in-place migration from ZooKeeper to KRaft mode using the official migration guide, then finalize the metadata version, and finally perform a rolling upgrade to Kafka 4.0.</p><p><strong>IMPORTANT</strong></p><p>The migration is a one-way operation. Once finalized, you cannot revert to ZooKeeper mode. Always test the migration thoroughly in a staging environment before touching production.</p><h3>Putting It All Together: The Complete Flow</h3><p>Let us trace our order through the entire system, now including KRaft:</p><p><strong>1. KRaft Controllers </strong>maintain the metadata: topic “orders” exists with 3 partitions, replication factor 3, and partition leaders are Broker 1 (P0), Broker 2 (P1), Broker 3 (P2).</p><p><strong>2. Producer </strong>fetches metadata from any broker (which gets it from the KRaft controllers), learns the partition leaders, and sends { orderId: 1, name: “piash” } directly to Broker 1 (leader of P0) because hash(1) % 3 = 0.</p><p><strong>3. Broker 1 </strong>appends the message to P0 at offset 0. Follower replicas on Broker 2 and Broker 3 pull the data and replicate it.</p><p><strong>4. Consumer Group A </strong>(order processing service) has C1 assigned to P0. C1 reads offset 0, processes the order, and commits offset 1 to __consumer_offsets.</p><p><strong>5. Consumer Group B </strong>(analytics service) independently reads the same message from P0 at offset 0. Its offset tracking is completely separate from Group A.</p><p><strong>6. If Broker 1 crashes, </strong>the KRaft controller detects the failure through the Raft quorum, promotes a follower to leader for P0, and propagates the updated metadata. Producers and consumers discover the new leader and continue operating — all without any ZooKeeper involvement.</p><h3>Production Best Practices</h3><p><strong>Partition count: </strong>Choose your partition count carefully. It determines maximum consumer parallelism and cannot be decreased after creation. A good starting point is 3x to 6x the number of expected consumers.</p><p><strong>Replication factor: </strong>Use 3 for production workloads. Combined with min.insync.replicas=2, this ensures data durability even if one broker is down.</p><p><strong>KRaft controllers: </strong>Deploy an odd number of controllers (3 or 5) for proper quorum. Use dedicated nodes with SSD storage for fast metadata operations.</p><p><strong>Idempotent producers: </strong>Always enable enable.idempotence=true to prevent duplicate messages from producer retries.</p><p><strong>Consumer design: </strong>Make consumers idempotent. Use database unique constraints or deduplication caches to handle at-least-once delivery gracefully.</p><p><strong>Monitoring: </strong>Track consumer lag (difference between latest offset and committed offset), under-replicated partitions, and KRaft controller quorum health.</p><h3>Final Thoughts</h3><p>Kafka’s architecture is elegant in its simplicity: append-only logs partitioned for parallelism, consumer groups for workload distribution, and offset tracking for reliable delivery. The introduction of KRaft takes this further by eliminating the external ZooKeeper dependency, making Kafka a truly self-contained distributed streaming platform.</p><p>Whether you are building real-time analytics pipelines, event-driven microservices, or large-scale data integration systems, understanding these internals helps you make better design decisions — from choosing the right partition count to designing idempotent consumers.</p><p>The shift from ZooKeeper to KRaft is not just a technical upgrade. It represents Kafka’s evolution into a more mature, simpler, and more powerful system. If you have not started your migration planning yet, now is the time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=156656a75b4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Kubernetes Network Policies: From Zero-Trust to Multi-Tier Connectivity]]></title>
            <link>https://medium.com/@piash.tanjin/mastering-kubernetes-network-policies-from-zero-trust-to-multi-tier-connectivity-d167f6258059?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d167f6258059</guid>
            <category><![CDATA[kubernetes-network-policy]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dns]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Mon, 12 Jan 2026 06:49:34 GMT</pubDate>
            <atom:updated>2026-01-12T06:49:34.467Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of Kubernetes, security should never be an afterthought. By default, pods are “non-isolated”; they accept traffic from any source. To build a secure production environment, we must adopt a <strong>Zero-Trust</strong> model using <strong>Network Policies</strong>.</p><p>In this guide, we will build a three-tier architecture (Frontend, Backend, and Cassandra) and secure it step-by-step.</p><h3>🕹️Play with the Analogy</h3><p>Before diving into the code, you can visualize these concepts using our interactive playground: <a href="https://k8s-netpol.lovable.app/"><strong>https://k8s-netpol.lovable.app</strong></a></p><h3>1. The Architecture &amp; Initial Setup</h3><p>First, let’s spin up our components. We will create a frontend and a backend pod, and expose them as services.</p><pre># Create the pods<br>kubectl run frontend --image=nginx --labels=&quot;run=frontend&quot;<br>kubectl run backend --image=nginx --labels=&quot;run=backend&quot;</pre><pre># Expose them as services<br>kubectl expose pod frontend --port=80 --target-port=80<br>kubectl expose pod backend --port=80 --target-port=80</pre><h3>The “Deny All” Baseline</h3><p>Before we allow specific traffic, we lock everything down. This ensures that if a new pod is added to the cluster, it cannot communicate with anything until we explicitly permit it.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: default-deny-all<br>spec:<br>  podSelector: {} # Selects all pods in the namespace<br>  policyTypes:<br>  - Ingress<br>  - Egress</pre><p><em>Result: At this point, if you try to </em><em>curl from frontend to backend, it will hang and fail.</em></p><h3>2. Bridging the Gap: Frontend to Backend</h3><p>Now, let’s create our first set of rules. We need two policies to complete the “handshake”: one for the <strong>Egress</strong> (leaving the frontend) and one for the <strong>Ingress</strong> (entering the backend).</p><h3>network-policy-1: Frontend Egress</h3><p>This allows the frontend to reach out to the backend.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: network-policy-1<br>spec:<br>  podSelector:<br>    matchLabels:<br>      run: frontend<br>  policyTypes:<br>  - Egress<br>  egress:<br>  - to:<br>    - podSelector:<br>        matchLabels:<br>          run: backend</pre><h3>network-policy-2: Backend Ingress</h3><p>This allows the backend to accept requests specifically from the frontend.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: network-policy-2<br>spec:<br>  podSelector:<br>    matchLabels:<br>      run: backend<br>  policyTypes:<br>  - Ingress<br>  ingress:<br>  - from:<br>    - podSelector:<br>        matchLabels:<br>          run: frontend</pre><h3>3. Extending to the Database (Cassandra)</h3><p>Let’s add a Cassandra pod in a specific namespace and allow the backend to communicate with it.</p><pre># Setup the database pod<br>kubectl run cassandra --image=cassandra --labels=&quot;app=cassandra&quot;</pre><h3>The Backend-to-Cassandra Policy</h3><p>We need to update the backend to allow <strong>Egress</strong> to Cassandra and create an <strong>Ingress</strong> policy for Cassandra itself.</p><p><strong>Cassandra Ingress Policy:</strong></p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: cassandra-ingress<br>spec:<br>  podSelector:<br>    matchLabels:<br>      app: cassandra<br>  policyTypes:<br>  - Ingress<br>  ingress:<br>  - from:<br>    - podSelector:<br>        matchLabels:<br>          run: backend</pre><p><strong>Updated Backend Policy (Adding Egress):</strong></p><pre>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: backend-policy-v2<br>spec:<br>  podSelector:<br>    matchLabels:<br>      run: backend<br>  policyTypes:<br>  - Ingress<br>  - Egress<br>  ingress:<br>  - from:<br>    - podSelector:<br>        matchLabels:<br>          run: frontend<br>  egress:<br>  - to:<br>    - podSelector:<br>        matchLabels:<br>          app: cassandra</pre><h3>4. Understanding the Flow: Ingress vs. Egress</h3><p>There is often confusion about directionality. Think of it this way:</p><blockquote><strong><em>The “Apt-Update” Analogy:</em></strong><em> When you run </em><em>apt update on your laptop, you are </em><strong><em>initiating</em></strong><em> a request. From your laptop&#39;s perspective, this is an </em><strong><em>Egress</em></strong><em> request (data leaving your machine). However, the reason you are doing it is to </em><strong><em>pull</em></strong><em> or </em><strong><em>receive</em></strong><em> data from the repository. Even though data eventually flows &quot;into&quot; your machine, the policy governing your machine&#39;s ability to start that conversation is an </em><strong><em>Egress</em></strong><em> policy.</em></blockquote><h3>Traffic Visualization</h3><pre>[ USER ] <br>   |<br>   v (External Traffic)<br>+----------------+       (Egress)       +----------------+       (Egress)       +----------------+<br>|    Frontend    |  ----------------&gt;  |    Backend     |  ----------------&gt;  |   Cassandra    |<br>| (run:frontend) |                     | (run:backend)  |                     | (app:cassandra)|<br>+----------------+  &lt;----------------  +----------------+  &lt;----------------  +----------------+<br>                        (Ingress)                                (Ingress)</pre><ol><li><strong>Frontend to Backend</strong>: Frontend needs <strong>Egress</strong> permission to “talk out.” Backend needs <strong>Ingress</strong> permission to “listen in.”</li><li><strong>Backend to Cassandra</strong>: Backend needs <strong>Egress</strong> permission to “reach out.” Cassandra needs <strong>Ingress</strong> permission to “accept” the connection.</li></ol><h3>5. Testing the Connectivity</h3><p>Once the policies are applied, you can verify using kubectl exec:</p><pre># 1. Test Frontend to Backend (Should Succeed)<br>kubectl exec frontend -- curl -s --connect-timeout 5 backend</pre><pre># 2. Test Backend to Cassandra (Should Succeed on port 9042)<br>kubectl exec backend -- nc -zv cassandra 9042</pre><pre># 3. Test Frontend to Cassandra (Should Fail - No policy allows this)<br>kubectl exec frontend -- nc -zv cassandra 9042</pre><p>By following this pattern, you ensure that even if a pod is compromised, the attacker cannot “move laterally” through your cluster because the network policies strictly limit movement to only what is necessary for the application to function.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d167f6258059" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bare Metal vs Virtual Machines vs Containers: A Deep Dive with Real-World Use Cases]]></title>
            <link>https://medium.com/@piash.tanjin/bare-metal-vs-virtual-machines-vs-containers-a-deep-dive-with-real-world-use-cases-8fdfadfa82b2?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fdfadfa82b2</guid>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[baremetal]]></category>
            <category><![CDATA[isolation]]></category>
            <category><![CDATA[kernal]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 15:32:26 GMT</pubDate>
            <atom:updated>2025-12-14T15:32:26.325Z</atom:updated>
            <content:encoded><![CDATA[<p>Modern infrastructure is built on three fundamental execution models: <strong>bare metal</strong>, <strong>virtual machines (VMs)</strong>, and <strong>containers</strong>. While they often coexist in production, each solves a different class of problems. This article breaks down <strong>how they work internally</strong>, <strong>where EC2 and Docker fit</strong>, and <strong>when you should use which</strong>.</p><h3>1. Bare Metal: Direct Access to Hardware</h3><h3>What Bare Metal Really Means</h3><p>Bare metal refers to running workloads <strong>directly on physical hardware</strong> without any virtualization layer.</p><pre>Physical Server<br>├── CPU<br>├── Memory<br>├── Disk<br>├── Network<br>└── OS (Linux / Windows)<br>    └── Application</pre><p>There is <strong>no hypervisor</strong>, no container runtime between your application and the hardware.</p><h3>Deep Hierarchical Structure</h3><pre>Hardware<br>├── BIOS / Firmware<br>├── CPU (cores, cache, NUMA)<br>├── RAM<br>├── Disk (NVMe / SSD / HDD)<br>├── NIC<br>└── OS Kernel<br>    ├── Scheduler<br>    ├── Memory Manager<br>    ├── Network Stack<br>    └── Filesystem<br>        └── Application Process</pre><h3>Real-World Use Cases</h3><p><strong>Examples</strong></p><ul><li>AWS <strong>EC2 Bare Metal instances</strong> (i3.metal, c5.metal)</li><li>On-premise data centers</li><li>Dedicated servers (Hetzner, Equinix)</li></ul><p><strong>Why teams use it</strong></p><ul><li>Ultra-low latency (no virtualization overhead)</li><li>Direct access to GPUs, FPGAs, or SR-IOV NICs</li><li>Predictable performance</li></ul><p><strong>Common workloads</strong></p><ul><li>High-performance databases (Oracle, SAP HANA)</li><li>Telecom &amp; financial trading systems</li><li>Machine learning training</li><li>Legacy monoliths that can’t be virtualized</li></ul><p><strong>Trade-offs</strong></p><ul><li>Slow provisioning</li><li>Hard to scale</li><li>Expensive to operate</li></ul><h3>2. Virtual Machines: Hardware Virtualization</h3><h3>What a VM Is</h3><p>A VM emulates a <strong>full computer</strong> using a <strong>hypervisor</strong>. Each VM includes its <strong>own OS kernel</strong>.</p><pre>Physical Server<br>└── Hypervisor (KVM / Xen / VMware)<br>    ├── VM1 → OS → App<br>    ├── VM2 → OS → App<br>    └── VM3 → OS → App</pre><h3>Deep Hierarchical Structure</h3><pre>Hardware<br>└── Hypervisor<br>    ├── Virtual CPU<br>    ├── Virtual Memory<br>    ├── Virtual Disk<br>    └── Virtual NIC<br>        └── Guest OS Kernel<br>            ├── Init System (systemd)<br>            ├── System Libraries<br>            └── Application Process</pre><p>Each VM believes it owns the hardware.</p><h3>EC2 as a VM Example</h3><p>When you launch an <strong>EC2 instance</strong>:</p><ul><li>AWS provisions a VM on top of <strong>Nitro Hypervisor</strong></li><li>You get a <strong>full OS</strong> (Ubuntu, Amazon Linux, Windows)</li><li>You manage patches, services, security</li></ul><pre>AWS Data Center<br>└── Physical Host<br>    └── Nitro Hypervisor<br>        └── EC2 Instance<br>            └── Linux OS<br>                └── Your App</pre><h3>Real-World Use Cases</h3><p><strong>Why VMs are popular</strong></p><ul><li>Strong isolation</li><li>Supports different OS types</li><li>Mature security model</li></ul><p><strong>Common workloads</strong></p><ul><li>Legacy enterprise apps</li><li>Lift-and-shift migrations</li><li>Stateful systems (databases, ERP)</li><li>Windows workloads</li></ul><p><strong>Trade-offs</strong></p><ul><li>Heavy resource usage</li><li>Slow startup (minutes)</li><li>OS patching overhead</li></ul><h3>3. Containers: OS-Level Virtualization</h3><h3>What Containers Actually Are</h3><p>Containers are <strong>isolated processes</strong> that share the <strong>host OS kernel</strong>.</p><p>They are <strong>not mini-VMs</strong>.</p><pre>Physical Server / VM<br>└── Host OS (Linux Kernel)<br>    └── Container Runtime (Docker / containerd)<br>        ├── Container A (App + deps)<br>        ├── Container B (App + deps)<br>        └── Container C (App + deps)</pre><h3>Deep Hierarchical Structure</h3><pre>Hardware<br>└── Host OS Kernel<br>    ├── Namespaces (PID, NET, MOUNT, UTS)<br>    ├── cgroups (CPU, Memory, IO)<br>    └── Container Runtime<br>        └── Image Layers (Docker Image)<br>            ├── Base Image<br>            ├── Dependencies<br>            └── Application Binary</pre><p>Key Linux primitives:</p><ul><li><strong>Namespaces</strong> → isolation</li><li><strong>cgroups</strong> → resource limits</li><li><strong>UnionFS</strong> → image layering</li></ul><h3>Docker Images Explained</h3><p>A Docker image is <strong>not an OS</strong>.<br> It’s a <strong>filesystem snapshot + metadata</strong>.</p><pre>Docker Image<br>├── Base layer (alpine / distroless)<br>├── Runtime (node, java, python)<br>├── App dependencies<br>└── Application code</pre><p>When you run it:</p><ul><li>Kernel is shared</li><li>Only the process is isolated</li></ul><h3>Real-World Use Cases</h3><p><strong>Why containers dominate modern systems</strong></p><ul><li>Start in milliseconds</li><li>Extremely portable</li><li>CI/CD friendly</li><li>Perfect for microservices</li></ul><p><strong>Common workloads</strong></p><ul><li>Microservices</li><li>APIs</li><li>Background workers</li><li>CI pipelines</li><li>Kubernetes workloads</li></ul><p><strong>Trade-offs</strong></p><ul><li>Weaker isolation than VMs</li><li>Same kernel requirement</li><li>Requires security hardening</li></ul><h3>4. How They Work Together in Production</h3><p>Most real systems <strong>combine all three</strong>.</p><h3>Typical Cloud-Native Stack</h3><pre>Bare Metal (Cloud Provider)<br>└── VM (EC2 / GCE / Azure VM)<br>    └── Kubernetes Node<br>        └── Pod<br>            ├── Container 1<br>            └── Container 2</pre><ul><li>Cloud providers run <strong>bare metal</strong></li><li>Customers get <strong>VMs</strong></li><li>Kubernetes runs <strong>containers inside VMs</strong></li></ul><h3>5. Performance, Security, and Scaling Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYnYuBTBwmzMFzhMT7T1_Q.png" /></figure><h3>6. Choosing the Right Model</h3><h3>Use Bare Metal when:</h3><ul><li>You need maximum performance</li><li>Hardware access is critical</li><li>Compliance requires physical isolation</li></ul><h3>Use VMs when:</h3><ul><li>You need strong isolation</li><li>You run mixed OS workloads</li><li>You migrate legacy systems</li></ul><h3>Use containers when:</h3><ul><li>You want speed and scale</li><li>You practice DevOps &amp; CI/CD</li><li>You build microservices</li></ul><h3>Final Thoughts</h3><p>Bare metal, VMs, and containers are <strong>not competitors</strong> — they are <strong>layers</strong>.</p><p>Understanding their <strong>internal hierarchy</strong> and <strong>real-world trade-offs</strong> is essential for building reliable, scalable systems in the cloud.</p><blockquote><em>Containers optimize </em><strong><em>delivery</em></strong><em><br> VMs optimize </em><strong><em>isolation</em></strong><em><br> Bare metal optimizes </em><strong><em>performance</em></strong></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fdfadfa82b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Labels and Selectors: The Complete Guide You Actually Need]]></title>
            <link>https://medium.com/@piash.tanjin/kubernetes-labels-and-selectors-the-complete-guide-you-actually-need-6a9d8a0ac7f1?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a9d8a0ac7f1</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[selectors]]></category>
            <category><![CDATA[granularity]]></category>
            <category><![CDATA[filters]]></category>
            <category><![CDATA[labels]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Sun, 14 Dec 2025 06:11:00 GMT</pubDate>
            <atom:updated>2025-12-14T06:11:00.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Problem</h3><p>Last month, I had 50+ pods running in my cluster. Finding specific pods felt like searching for a needle in a haystack. My database randomly landed on slow nodes. Development pods accidentally hit production. It was chaos.</p><p>Then I learned three simple Kubernetes concepts that changed everything: <strong>Labels</strong>, <strong>Field Selectors</strong>, and <strong>Node Selectors</strong>.</p><h3>Labels: Your Organization System</h3><p>Think of labels as sticky notes. You put them on anything in Kubernetes—pods, nodes, services—to describe what they are.</p><pre>labels:<br>  app: webstore<br>  tier: backend<br>  environment: production</pre><p>That’s it. Just key-value pairs. But they’re incredibly powerful.</p><h3>Real Use Case: Finding Things Fast</h3><p>I have a web app with frontend, backend, and database pods. Without labels? I’d have to manually check each pod. With labels? I can find anything instantly:</p><p>bash</p><pre># Show only frontend pods<br>kubectl get pods -l tier=frontend</pre><pre># Show everything in production<br>kubectl get pods -l environment=production</pre><pre># Show all backend pods that are running<br>kubectl get pods -l tier=backend --field-selector status.phase=Running</pre><h3>Managing with Labels</h3><p>bash</p><pre># Add a label<br>kubectl label pods my-pod version=v2.0</pre><pre># Update a label<br>kubectl label pods my-pod version=v2.1 --overwrite</pre><pre># Remove a label<br>kubectl label pods my-pod version-</pre><pre># View labels<br>kubectl get pods --show-labels</pre><h3>Field Selectors: Built-in Filters</h3><p>While labels are tags you create, field selectors filter using Kubernetes’ own fields — like pod status or which node it’s on.</p><h3>Common Field Selectors I Use Daily</h3><p>bash</p><pre># All running pods<br>kubectl get pods --field-selector status.phase=Running</pre><pre># All pending pods (great for troubleshooting)<br>kubectl get pods --field-selector status.phase=Pending</pre><pre># Pods on a specific node<br>kubectl get pods --field-selector spec.nodeName=worker-1</pre><pre># Combine with labels for powerful filtering<br>kubectl get pods -l app=webstore --field-selector status.phase=Running</pre><p><strong>Pro tip:</strong> Field selectors are perfect for operational queries — “show me what’s broken” or “what’s running where?”</p><h3>Node Selectors: Smart Placement</h3><p>Here’s where it gets really useful. Not all nodes are equal — some have SSDs, some have lots of RAM, some are production-grade.</p><p>Node selectors let you say: “This pod needs fast storage” or “This must run in production.”</p><h3>Real Use Case: Hardware Matching</h3><p>I have three types of nodes:</p><ul><li><strong>Fast nodes</strong> with SSDs (expensive)</li><li><strong>Memory nodes</strong> with 64GB RAM (for databases)</li><li><strong>Standard nodes</strong> (everything else)</li></ul><p>First, I labeled my nodes to describe their capabilities:</p><pre>kubectl label nodes worker-1 disktype=ssd<br>kubectl label nodes worker-2 memory=high<br>kubectl label nodes worker-3 environment=production</pre><p>Then I created a pod that needs fast storage:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: database<br>  labels:<br>    app: webstore<br>    tier: database<br>spec:<br>  nodeSelector:<br>    disktype: ssd    # Only run on SSD nodes<br>  containers:<br>  - name: postgres<br>    image: postgres:14</pre><p><strong>Result:</strong> My database automatically lands on nodes with SSDs. Every. Single. Time.</p><h3>The Safety Pattern</h3><p>The most valuable use? Preventing accidents. I label all my nodes by environment:</p><pre># Production nodes<br>kubectl label nodes prod-1 environment=production<br>kubectl label nodes prod-2 environment=production</pre><pre># Development nodes  <br>kubectl label nodes dev-1 environment=development</pre><p>Now every pod gets a node selector:</p><pre>spec:<br>  nodeSelector:<br>    environment: production  # Can&#39;t accidentally hit dev nodes</pre><p><strong>I haven’t had a dev-to-prod accident since.</strong></p><h3>Putting It All Together</h3><p>Here’s my standard workflow for any new application:</p><p><strong>1. Label my nodes</strong> (once per node):</p><p>kubectl label nodes node-1 disktype=ssd environment=production</p><p><strong>2. Create pods with both labels and node selectors</strong>:</p><pre>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: api-server<br>  labels:<br>    app: myapp<br>    tier: backend<br>    environment: production<br>spec:<br>  nodeSelector:<br>    environment: production<br>  containers:<br>  - name: api<br>    image: myapi:v1.0</pre><p><strong>3. Use labels to filter and manage</strong>:</p><pre># View my app<br>kubectl get pods -l app=myapp</pre><pre># Check production workloads<br>kubectl get pods -l environment=production</pre><pre># Find issues<br>kubectl get pods --field-selector status.phase!=Running</pre><pre># See where things are running<br>kubectl get pods -o wide</pre><h3>Common Mistakes (I Made Them All)</h3><p><strong>Mistake #1:</strong> Creating pods with node selectors before labeling nodes.</p><ul><li><strong>Result:</strong> Pods stuck in “Pending” forever</li><li><strong>Fix:</strong> Label nodes first, then create pods</li></ul><p><strong>Mistake #2:</strong> Over-labeling everything</p><ul><li><strong>Bad:</strong> app, application, app-name, application-name (all meaning the same thing)</li><li><strong>Good:</strong> Just app, tier, environment</li></ul><p><strong>Mistake #3:</strong> Using spaces or special characters</p><ul><li><strong>Bad:</strong> app name: my app</li><li><strong>Good:</strong> app-name: my-app</li></ul><h3>Troubleshooting</h3><p><strong>Pod stuck in “Pending”?</strong></p><pre>kubectl describe pod my-pod<br># Look for: &quot;didn&#39;t match Pod&#39;s node affinity/selector&quot;<br># Fix: Add the required label to a node</pre><p><strong>Can’t find your pods?</strong></p><pre># See all labels<br>kubectl get pods --show-labels</pre><p><strong>Pod on the wrong node?</strong></p><pre># Check pod placement<br>kubectl get pods -o wide</pre><pre># Check node labels<br>kubectl get nodes --show-labels</pre><h3>Quick Reference</h3><pre># LABELS - You create these<br>kubectl label nodes NODE key=value<br>kubectl get pods -l key=value<br>kubectl label pods POD key-  # Remove label</pre><pre># FIELD SELECTORS - Built-in Kubernetes fields<br>kubectl get pods --field-selector status.phase=Running<br>kubectl get pods --field-selector spec.nodeName=worker-1</pre><pre># NODE SELECTORS - In pod YAML<br>spec:<br>  nodeSelector:<br>    disktype: ssd<br>    environment: production</pre><h3>The Bottom Line</h3><ul><li><strong>Labels</strong> organize your resources (like name tags)</li><li><strong>Field Selectors</strong> filter using Kubernetes’ built-in fields (status, node name)</li><li><strong>Node Selectors</strong> control where pods run (hardware/environment matching)</li></ul><p>Start with three labels for everything: app, tier, and environment. Add node selectors when you need specific hardware or want to enforce environment boundaries.</p><p>That’s it. These three concepts will solve 90% of your cluster organization problems.</p><h3>Try It Now</h3><p>Pick one pod running in your cluster right now and try:</p><pre># Add labels<br>kubectl label pod YOUR_POD app=test tier=backend</pre><pre># Filter it<br>kubectl get pods -l app=test</pre><pre># Check its status<br>kubectl get pods -l app=test --field-selector status.phase=Running</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a9d8a0ac7f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Virtual Machines vs Containers: A Deep Dive with Real-World Use Cases]]></title>
            <link>https://medium.com/@piash.tanjin/virtual-machines-vs-containers-a-deep-dive-with-real-world-use-cases-c8359d5d9fa3?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8359d5d9fa3</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[virtualization]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Fri, 12 Dec 2025 15:56:46 GMT</pubDate>
            <atom:updated>2025-12-12T15:56:46.711Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of modern IT infrastructure, <strong>Virtual Machines (VMs)</strong> and <strong>Containers</strong> have become the backbone of deploying applications at scale. While both provide isolated environments to run applications, their architecture, performance characteristics, and ideal use cases differ significantly. Understanding these differences is crucial for architects, developers, and DevOps engineers who aim to build scalable, resilient, and cost-efficient systems.</p><h3>What Are Virtual Machines?</h3><p>A <strong>virtual machine</strong> is a software-based emulation of a physical computer. VMs run on a <strong>hypervisor</strong> (such as VMware, KVM, or Hyper-V) that abstracts the underlying hardware and allows multiple VMs to coexist on a single physical host. Each VM contains:</p><ul><li>A <strong>guest operating system</strong> (can differ from the host OS)</li><li>Installed applications</li><li>Necessary binaries and libraries</li></ul><p><strong>Real-World Examples:</strong></p><ul><li><strong>Amazon EC2 instances</strong> are essentially virtual machines running on AWS’s hypervisors.</li><li><strong>VPS (Virtual Private Servers)</strong> offered by providers like DigitalOcean, Linode, and Vultr are classic VM implementations.</li></ul><p><strong>Pros of VMs:</strong></p><ul><li>Strong isolation between workloads</li><li>Ability to run different OSes on the same host</li><li>Mature and well-understood technology with a strong ecosystem</li></ul><p><strong>Cons of VMs:</strong></p><ul><li>Heavy in resource usage (disk, memory, CPU)</li><li>Slow boot times due to full OS startup</li><li>Less efficient when scaling many small applications</li></ul><p><strong>Use Cases:</strong></p><ul><li>Running legacy applications that require full OS control</li><li>Multi-OS testing and development environments</li><li>Hosting multiple isolated services on a single physical server</li></ul><h3>What Are Containers?</h3><p><strong>Containers</strong> are lightweight, portable environments that package an application and its dependencies but <strong>share the host OS kernel</strong>. They provide process-level isolation without the overhead of running a full OS.</p><p><strong>Popular container technologies include:</strong></p><ul><li>Docker</li><li>Podman</li><li>containerd</li></ul><p><strong>Real-World Examples:</strong></p><ul><li>Many cloud-native applications run <strong>Docker containers inside EC2 instances</strong> or other VMs.</li><li>Kubernetes clusters orchestrate containers across multiple VMs or bare-metal servers.</li></ul><p><strong>Pros of Containers:</strong></p><ul><li>Lightweight and fast to start (seconds instead of minutes)</li><li>Efficient resource utilization</li><li>Highly portable — “build once, run anywhere”</li><li>Ideal for microservices architectures</li></ul><p><strong>Cons of Containers:</strong></p><ul><li>Weaker isolation compared to VMs (share OS kernel)</li><li>Cannot run a completely different OS than the host</li><li>Security relies on host configuration and container runtime</li></ul><p><strong>Use Cases:</strong></p><ul><li>Microservices deployment (e.g., API backend, worker processes)</li><li>Continuous Integration/Continuous Deployment (CI/CD) pipelines</li><li>Rapid scaling and dynamic workloads in cloud environments</li><li>Running multiple instances of the same application efficiently</li></ul><h3>How VMs and Containers Work Together</h3><p>In practice, VMs and containers are <strong>often used together</strong>. For example, a typical deployment may look like this:</p><ol><li>An <strong>EC2 instance</strong> (VM) runs a Linux OS.</li><li>Inside the VM, multiple <strong>Docker containers</strong> host microservices.</li><li>Each container packages a specific service with its dependencies, ensuring consistency across environments.</li></ol><p>This hybrid approach leverages the isolation and OS flexibility of VMs while benefiting from the efficiency and portability of containers.</p><p><strong>Visual Analogy:</strong></p><ul><li><strong>VMs</strong> are like fully furnished apartments, each with its own kitchen, bathroom, and utilities.</li><li><strong>Containers</strong> are like rooms within an apartment — lightweight, share common resources, but still private enough to run independently.</li></ul><h3>Pros and Cons Compared</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*skiFSWIlJTG0TaYHOsLCOw.png" /></figure><h3>Choosing the Right Tool</h3><ul><li><strong>Use VMs</strong> when you need strong isolation, want to run multiple OS types, or deal with legacy software.</li><li><strong>Use Containers</strong> for modern microservices, rapid scaling, CI/CD, and efficient resource usage.</li><li><strong>Combine both</strong> for the best of both worlds — containers inside VMs are a standard pattern in cloud deployments, especially with Kubernetes.</li></ul><h3>Conclusion</h3><p>Virtual Machines and Containers are complementary technologies. While VMs provide the heavy-duty isolation and flexibility of running multiple OS types, containers bring lightweight, fast, and portable environments for modern application development. Cloud providers like AWS, Azure, and GCP often blend both, allowing enterprises to maximize efficiency without compromising reliability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8359d5d9fa3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Right KubernetesWorker Node Strategy: EC2 vs Fargate vs Hybrid]]></title>
            <link>https://medium.com/@piash.tanjin/choosing-the-right-kubernetesworker-node-strategy-ec2-vs-fargate-vs-hybrid-740f62877bff?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/740f62877bff</guid>
            <category><![CDATA[cloud-infrastructure]]></category>
            <category><![CDATA[workers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[amazon-eks]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Thu, 20 Nov 2025 16:19:09 GMT</pubDate>
            <atom:updated>2025-11-20T16:19:09.385Z</atom:updated>
            <content:encoded><![CDATA[<p>Designing cloud infrastructure on AWS isn’t just about choosing the right tools—it’s about choosing the right <em>trade-offs</em>. And if you’ve ever had to architect an EKS cluster from scratch, you know exactly how many small (and big) decisions suddenly show up.</p><p>One of the most important? <strong>How your worker nodes will run.</strong></p><p>It sounds simple on the surface—EC2, Fargate, or both—but each one completely changes how your workloads behave, how you scale, how you patch, and even how your DevOps team sleeps at night.</p><p>This article takes a practical, real-world walk through all three options. No marketing fluff. Just what actually matters when you’re making architecture decisions.</p><h3>Ways to Run EKS Worker Nodes</h3><p>Let’s break down the three real choices: EC2, Fargate, and the hybrid setup most teams eventually land on.</p><h3>EKS with EC2 Worker Nodes</h3><p>Running EKS on EC2 is the classic way to operate Kubernetes — the way many teams start because it feels familiar.</p><p>You launch EC2 instances as managed or self-managed node groups. From there, it works like any normal Kubernetes cluster where <strong>you actually control the nodes</strong>.</p><p>This means you get to tweak everything:</p><ul><li>The OS and AMI</li><li>Node-level agents and DaemonSets</li><li>Custom monitoring setups</li><li>GPU, ARM, or specialized instance types</li><li>Node labels, taints, affinity rules—the works</li></ul><p>It’s powerful, flexible, and… yes, a bit heavier to maintain.</p><h3>Pros</h3><ul><li>Full control—you can run anything you need</li><li>Supports GPUs, high-performance storage, custom AMIs</li><li>Perfect for running DaemonSets like log collectors or service mesh agents</li><li>Great for stateful workloads that need predictable nodes</li></ul><h3>Cons</h3><ul><li>You’re responsible for OS updates, patches, AMI version bumps</li><li>Scaling requires some planning (though managed node groups do help)</li><li>More operational overhead compared to serverless options</li></ul><h3>Best For</h3><p>Teams that need maximum control or run workloads that simply <em>cannot</em> run on something abstracted—like databases, GPU jobs, or anything requiring node-level access.</p><h3>EKS with AWS Fargate</h3><p>If EC2 is the traditional approach, <strong>Fargate is the modern “I don’t want to manage servers anymore” option</strong>.</p><p>You don’t see nodes. You don’t patch nodes. You don’t create node groups. You just create pods, and AWS figures out the rest.</p><p>It feels magical—and for stateless workloads, it honestly <em>is</em>.</p><h3>Pros</h3><ul><li>Zero node management (huge win for small teams)</li><li>Autoscaling is instant and hands-off</li><li>Perfect isolation — every pod gets its own micro-VM</li><li>Consistent performance for microservices and API workloads</li></ul><h3>Cons</h3><ul><li>No DaemonSets (since there are no visible nodes)</li><li>Not meant for stateful apps or workloads needing local storage</li><li>Limited OS-level customization</li><li>No GPU support</li></ul><h3>Best For</h3><p>Teams who want to stay focused on the application, not the infrastructure. Stateless microservices and event-driven jobs run beautifully here.</p><h3>Hybrid Setup (EC2 + Fargate)</h3><p>This is, realistically, where most mature EKS clusters end up.</p><p>Why? Because apps in the real world aren’t neatly “all stateless” or “all stateful.” You usually have:</p><ul><li>A bunch of microservices that scale up and down all day</li><li>Monitoring/logging agents that need DaemonSets</li><li>A few persistent or stateful services</li><li>Maybe a GPU workload tucked somewhere</li></ul><p>Hybrid lets you put each workload in its ideal home:</p><ul><li><strong>EC2 nodes →</strong> full control, run DaemonSets, run stateful things</li><li><strong>Fargate →</strong> effortless scaling, reduced ops, clean isolation</li></ul><p>It’s the best of both worlds as long as the scheduling rules are set up cleanly.</p><h3>Pros</h3><ul><li>Flexibility without fully giving up control</li><li>Cost-efficient scaling for stateless workloads</li><li>Ability to run DaemonSets and heavy workloads on EC2</li><li>Clean separation of responsibility and performance isolation</li></ul><h3>Cons</h3><ul><li>More moving pieces than a single-mode setup</li><li>Requires clear node affinity / Fargate profiles</li><li>Monitoring cost and complexity can be higher</li></ul><h3>Best For</h3><p>Real-world production environments with mixed workloads.</p><h3>Diagram: EKS Worker Node Setup Options</h3><pre>AWS Managed Control Plane<br>                 --------------------------<br>                              |<br>        ------------------------------------------------<br>        |                                              |<br>   EC2 Node Group                                AWS Fargate<br>   ----------------                              ----------------<br>   • Any type of Pod                              • Serverless Pods<br>   • DaemonSets: YES                              • DaemonSets: NO<br>   • Full control (AMI, OS, GPU)                  • No node-level access<br>   • Best for stateful / infra agents             • Best for stateless apps</pre><h3>Quick Comparison Table</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*l5pfRnK7RT9nlAXuowWboA.png" /></figure><h3>Important Things to Remember</h3><ul><li><strong>Fargate profiles decide which pods go to Fargate</strong></li><li><strong>DaemonSets only run on EC2</strong>—never Fargate</li><li>Hybrid setups are extremely common because infra agents must run somewhere</li><li>Separate node pools give you cleaner performance isolation</li></ul><h3>Final Thoughts</h3><p>Choosing EC2, Fargate, or Hybrid isn’t just about cost or convenience — it’s about designing an environment where every workload gets the home it deserves.</p><p>Here’s the simple rule of thumb:</p><ul><li><strong>Choose EC2</strong> when you need power and control.</li><li><strong>Choose Fargate</strong> when you want simplicity and hands-off scaling.</li><li><strong>Choose Hybrid</strong> when you’re building something real that needs the reliability of EC2 and the elasticity of Fargate.</li></ul><p>In many teams, the journey looks like this:</p><blockquote><em>Start with EC2 → Move stateless workloads to Fargate → End with a balanced hybrid setup.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=740f62877bff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Observability Tools: All You Need to Know as a Developer, Product User, or Team Lead]]></title>
            <link>https://medium.com/@piash.tanjin/observability-tools-all-you-need-to-know-as-a-developer-product-user-or-team-lead-0ab62b6cd3a8?source=rss-8508fb16e1c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ab62b6cd3a8</guid>
            <category><![CDATA[metrics-and-analytics]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[applicant-tracking-system]]></category>
            <dc:creator><![CDATA[piash.tanjin]]></dc:creator>
            <pubDate>Sat, 15 Nov 2025 10:28:12 GMT</pubDate>
            <atom:updated>2025-11-15T10:28:12.697Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s fast-paced software world, observability isn’t just a buzzword—it’s the lens through which teams understand, monitor, and improve their applications. But with so many tools—Sentry, Prometheus, OpenTelemetry, Microsoft Clarity, and more—it can get confusing. This post breaks down the key observability tools, their use cases, and who benefits from each.</p><h3>What is Observability?</h3><p>Observability is the ability to understand the internal state of a system by examining its outputs. For software, this typically means collecting:</p><ul><li><strong>Metrics</strong>—Numeric indicators like CPU usage, request latency, or error rates.</li><li><strong>Traces</strong>—End-to-end paths of requests in distributed systems.</li><li><strong>Logs</strong>—Detailed event records and errors.</li><li><strong>User Behavior Data</strong>—How users interact with your application or website.</li></ul><p>Different teams—developers, product managers, UX designers, and team leads—care about different types of data.</p><h3>Core Observability Tools and Their Focus</h3><p>Let’s break down the most widely used tools and who benefits most from them.</p><h3>1. Prometheus—Metrics &amp; Alerting</h3><ul><li><strong>Focus:</strong> Time-series metrics like CPU, memory, request counts, or latency.</li><li><strong>Best For:</strong> SREs, DevOps, and infrastructure monitoring.</li><li><strong>Why Use It:</strong> Prometheus provides a pull-based metrics collection system, a powerful query language (PromQL), and alerting capabilities. It’s ideal for monitoring system health and setting up alerts before issues become critical.</li></ul><h3>2. OpenTelemetry—Unified Observability</h3><ul><li><strong>Focus:</strong> Metrics, traces, and logs.</li><li><strong>Best For:</strong> Developers and SREs who need end-to-end observability in microservices or complex applications.</li><li><strong>Why Use It:</strong> OpenTelemetry provides a standardized way to collect telemetry data and export it to multiple backends, including Prometheus, Jaeger, and commercial APM platforms. It helps teams unify their monitoring, tracing, and logging.</li></ul><h3>3. Sentry—Error &amp; Performance Monitoring</h3><ul><li><strong>Focus:</strong> Application-level errors, exceptions, and performance traces.</li><li><strong>Best For:</strong> Developers, QA engineers, and team leads tracking production issues.</li><li><strong>Why Use It:</strong> Sentry captures detailed error reports, stack traces, and performance metrics. It helps developers quickly identify and fix bugs while providing context like user info and environment variables.</li></ul><h3>4. Microsoft Clarity — User Behavior Analytics</h3><ul><li><strong>Focus:</strong> User interactions, session recordings, heatmaps, clicks, and scrolls.</li><li><strong>Best For:</strong> Product managers, UX designers, and marketing teams.</li><li><strong>Why Use It:</strong> Clarity provides insights into how users navigate your site, highlighting pain points, frustration clicks, and areas for optimization. Unlike Prometheus or Sentry, it focuses on <strong>user experience</strong>, not system health.</li></ul><h3>How These Tools Complement Each Other</h3><p>Observability isn’t about choosing just one tool; it’s about using the right combination:</p><ul><li><strong>Prometheus + OpenTelemetry</strong>: Collect metrics via OpenTelemetry and feed them into Prometheus for alerting and dashboards.</li><li><strong>Sentry + OpenTelemetry</strong>: Trace errors in production while correlating them with distributed traces collected via OpenTelemetry.</li><li><strong>Clarity + Sentry</strong>: Understand how user behavior correlates with errors or performance bottlenecks. For example, session replays can show where users struggled before encountering an error.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*rq761Zj6b60tVY_LzRkG0g.png" /></figure><h3>Who Should Care About What</h3><ul><li><strong>Developers:</strong> Sentry, OpenTelemetry—for debugging, tracing, and performance optimization.</li><li><strong>Team Leads / Managers:</strong> Sentry and Prometheus—for dashboards, alerts, and operational visibility.</li><li><strong>Product/UX Teams:</strong> Clarity—for understanding user behavior, friction points, and feature adoption.</li></ul><h3>Conclusion</h3><p>In modern software development, no single tool solves all observability needs. Metrics, logs, traces, and user behavior analytics serve different purposes but complement each other beautifully. Choosing the right mix depends on your role:</p><ul><li><strong>Infrastructure health?</strong> Prometheus.</li><li><strong>Distributed tracing and unified observability?</strong> OpenTelemetry.</li><li><strong>Error monitoring and app performance?</strong> Sentry.</li><li><strong>User behavior insights?</strong> Microsoft Clarity.</li></ul><p>By understanding the strengths of each tool, teams can build an observability stack that supports <strong>developers, product teams, and leadership alike</strong>, improving both system reliability and user experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ab62b6cd3a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>