<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Summit Kumar on Medium]]></title>
        <description><![CDATA[Stories by Summit Kumar on Medium]]></description>
        <link>https://medium.com/@summitkumar?source=rss-f795433473dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7fNRS3czvH5qi6Q1.</url>
            <title>Stories by Summit Kumar on Medium</title>
            <link>https://medium.com/@summitkumar?source=rss-f795433473dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:29:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@summitkumar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Real-Time Personalization: Engineering Decision Systems That Choose the Right Experience Every Time]]></title>
            <link>https://medium.com/@summitkumar/real-time-personalization-engineering-decision-systems-that-choose-the-right-experience-every-time-03a13beddda0?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/03a13beddda0</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 00:01:03 GMT</pubDate>
            <atom:updated>2026-08-11T00:01:03.185Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 21 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“Sending a notification is easy. Sending the right notification, to the right customer, at the right moment, through the right channel — that’s an engineering problem.”</strong></blockquote><p>By the end of Part 20, Atlas had transformed its edge architecture.</p><p>The platform now offered:</p><ul><li><strong><em>API Gateways</em></strong></li><li><strong><em>Backend for Frontend (BFF)</em></strong></li><li><strong><em>Intelligent edge caching</em></strong></li><li><strong><em>Request aggregation</em></strong></li><li><strong><em>Global traffic routing</em></strong></li><li><strong><em>Client-specific APIs</em></strong></li></ul><p>Every request reached Atlas efficiently.</p><p>But another question emerged.</p><p>Not from Engineering.</p><p>From Marketing.</p><blockquote><strong>“We know how to deliver notifications. But how do we decide which one should be delivered?”</strong></blockquote><p>At first glance, this looked like a machine learning problem.</p><p>It wasn’t.</p><p>It was a distributed systems problem.</p><p>Because before any model could make a prediction, Atlas had to answer several questions in less than 200 milliseconds.</p><ul><li><em>Who is this customer?</em></li><li><em>What have they done recently?</em></li><li><em>Which campaigns are currently active?</em></li><li><em>Has this customer already received too many notifications?</em></li><li><em>Which communication channel performs best?</em></li><li><em>Is this customer likely to convert today?</em></li></ul><p>None of these answers existed in one place.</p><p>They were scattered across dozens of services.</p><p>Building the decision engine became one of the most interesting architectural challenges in Atlas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-l7P9tz62tEC0kgt1FnZg.png" /><figcaption>Optimising Edge Computing Architecture</figcaption></figure><h4>Personalization Isn’t a Recommendation Engine</h4><p>Many platforms think personalization means recommending products.</p><p>Atlas viewed it differently.</p><p>Every customer interaction becomes a decision.</p><pre>Customer Opens App<br>↓<br>Should We Send Anything?<br>↓<br>If Yes...<br>↓<br>Which Campaign?<br>↓<br>Which Channel?<br>↓<br>What Time?<br>↓<br>What Message?</pre><p>Every question has multiple possible answers.</p><p>Every answer depends on real-time context.</p><h4>Decisions Need Context</h4><p>Consider a customer who:</p><ul><li><em>Added items to a cart</em></li><li><em>Opened the mobile app five minutes ago</em></li><li><em>Ignored three notifications yesterday</em></li><li><em>Recently upgraded to Premium</em></li><li><em>Lives in a different timezone</em></li></ul><p>Those facts exist across different systems.</p><pre>Orders Service<br>↓<br>Customer Service<br>↓<br>Campaign Service<br>↓<br>Analytics<br>↓<br>Notification History</pre><p>A recommendation model cannot query five services every time a decision is required.</p><p>Latency would become unacceptable.</p><p>Atlas needed another architecture.</p><h4>The Decision Context</h4><p>Instead of requesting information on demand, Atlas builds a unified customer context.</p><pre>Customer Context<br>────────────────────────────<br>Customer Profile<br>Current Session<br>Preferences<br>Campaign History<br>Recent Activity<br>Feature Values</pre><p>Every decision begins with a single object.</p><p>Not dozens of service calls.</p><h4>Building a Feature Store</h4><p>Customer behavior changes continuously.</p><p>Examples include:</p><pre>Last Purchase<br>2 Days Ago</pre><pre>Notifications Opened<br>Last 30 Days<br>18</pre><pre>Average Session Duration<br>11 Minutes</pre><pre>Preferred Channel<br>Push Notification</pre><p>Instead of recalculating these repeatedly, Atlas stores them in a Feature Store.</p><pre>Events<br>↓<br>Feature Pipeline<br>↓<br>Feature Store<br>↓<br>Decision Engine</pre><p>Features become reusable assets shared across the platform.</p><h4>Streaming Feature Updates</h4><p>Static customer information isn’t enough.</p><p>Imagine a customer abandoning their cart.</p><p>Waiting ten minutes to update the profile is too slow.</p><p>Atlas streams feature updates.</p><pre>Cart Updated<br>↓<br>Kafka<br>↓<br>Feature Processor<br>↓<br>Online Feature Store<br>↓<br>Available Immediately</pre><p>Every customer action becomes part of the next decision.</p><h4>Decision Pipeline</h4><p>Every notification request follows the same pipeline.</p><pre>Incoming Event<br>↓<br>Load Customer Context<br>↓<br>Evaluate Policies<br>↓<br>Run Recommendation Models<br>↓<br>Rank Campaigns<br>↓<br>Choose Channel<br>↓<br>Deliver Notification</pre><p>Each stage has a single responsibility.</p><p>Complexity remains manageable.</p><h4>Policy Before Prediction</h4><p>Suppose the recommendation model says:</p><pre>Send Campaign A<br>Confidence<br>96%</pre><p>Business policies still apply.</p><p>Examples include:</p><ul><li><em>Quiet hours</em></li><li><em>Regulatory restrictions</em></li><li><em>Maximum daily notifications</em></li><li><em>Customer opt-out preferences</em></li><li><em>Tenant-specific rules</em></li></ul><p>Atlas always evaluates policies before execution.</p><pre>Recommendation<br>↓<br>Policy Engine<br>↓<br>Approved?<br>↓<br>Delivery</pre><p>The model recommends.</p><p>The policy engine decides.</p><h4>Ranking Instead of Selecting</h4><p>Initially, Atlas selected the first matching campaign.</p><p>As campaigns increased, this stopped working.</p><p>Now every candidate receives a score.</p><pre>Campaign A<br>91<br>────────────<br>Campaign B<br>82<br>────────────<br>Campaign C<br>67</pre><p>The engine ranks opportunities.</p><p>Not rules.</p><p>Marketing gains flexibility without increasing complexity.</p><h4>Multi-Channel Optimization</h4><p>Atlas no longer assumes Push Notifications are always best.</p><p>The platform compares channels.</p><pre>Push<br>Open Rate<br>32%<br>────────────<br>Email<br>Open Rate<br>18%<br>────────────<br>SMS<br>Open Rate<br>61%</pre><p>For one customer, SMS performs best.</p><p>For another, Push dominates.</p><p>The decision engine optimizes channel selection automatically.</p><h4>Experimentation at Scale</h4><p>Every recommendation becomes an opportunity to learn.</p><p>Instead of replacing one model with another…</p><p>Atlas experiments.</p><pre>Traffic<br>↓<br>Model A<br>90%<br>────────────<br>Model B<br>10%</pre><p>Metrics include:</p><ul><li><strong><em>Open Rate</em></strong></li><li><strong><em>Click Rate</em></strong></li><li><strong><em>Conversion Rate</em></strong></li><li><strong><em>Revenue</em></strong></li><li><strong><em>Customer Satisfaction</em></strong></li></ul><p>If the new model performs better…</p><p>Traffic gradually increases.</p><p>Experiments become continuous.</p><h4>Feedback Loops</h4><p>Every decision produces feedback.</p><pre>Recommendation<br>↓<br>Notification Delivered<br>↓<br>Opened?<br>↓<br>Clicked?<br>↓<br>Purchased?</pre><p>Results return to the learning pipeline.</p><p>Tomorrow’s recommendations improve because of today’s outcomes.</p><p>Intelligence emerges through repetition.</p><h4>Keeping Latency Predictable</h4><p>Personalization should never slow customer experiences.</p><p>Atlas defines a strict latency budget.</p><pre>Context Loading<br>30 ms<br>↓<br>Feature Lookup<br>20 ms<br>↓<br>Model Inference<br>45 ms<br>↓<br>Policy Evaluation<br>15 ms<br>↓<br>Response<br>110 ms Total</pre><p>If the decision engine exceeds its budget…</p><p>Fallback strategies activate.</p><h4>Graceful Degradation</h4><p>Suppose the recommendation service becomes unavailable.</p><p>Should Atlas stop sending notifications?</p><p>No.</p><p>Fallback hierarchy:</p><pre>Recommendation Engine<br>↓<br>Unavailable<br>↓<br>Business Rules<br>↓<br>Unavailable<br>↓<br>Default Campaign<br>↓<br>Continue Delivery</pre><p>Personalization improves customer experience.</p><p>Availability protects customer experience.</p><p>Reliability always comes first.</p><h4>Observability for Decisions</h4><p>Traditional monitoring tracks infrastructure.</p><p>Atlas also monitors decisions.</p><p>Examples include:</p><pre>Recommendation Latency</pre><pre>Model Confidence</pre><pre>Campaign Selection Rate</pre><pre>Fallback Usage</pre><pre>Prediction Accuracy</pre><p>Engineering teams monitor both platform health and decision quality.</p><h4>Atlas Personalization Architecture</h4><pre>                Customer Events<br>                        │<br>                        ▼<br>                     Kafka<br>                        │<br>          ┌─────────────┼─────────────┐<br>          ▼             ▼             ▼<br> Feature Pipeline  Customer Data  Analytics<br>          │<br>          ▼<br>    Online Feature Store<br>          │<br>          ▼<br>     Decision Engine<br>          │<br>   ┌──────┼────────┐<br>   ▼      ▼        ▼<br>Policy  Ranking  Channel<br>Engine  Models   Selection<br>   │<br>   ▼<br>Workflow Engine<br>   │<br>   ▼<br>Notification Delivery</pre><p>Every component focuses on one responsibility.</p><p>Together they make thousands of decisions every second.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ee1b3bb1dff2a21b48f83f6653d6e39/href">https://medium.com/media/9ee1b3bb1dff2a21b48f83f6653d6e39/href</a></iframe><p>The objective isn’t building the smartest recommendation engine.</p><p>It’s building the most reliable decision platform.</p><h4>Lessons Learned</h4><p>Atlas discovered that personalization isn’t fundamentally about artificial intelligence.</p><p>It’s about architecture.</p><p>Machine learning contributes only one step in a much larger pipeline.</p><p>Without fresh data…</p><p>Models become stale.</p><p>Without policies…</p><p>Recommendations become unsafe.</p><p>Without observability…</p><p>Performance declines unnoticed.</p><p>Without fallbacks…</p><p>One failed model becomes a customer outage.</p><p>The engineering challenge wasn’t teaching Atlas to make decisions.</p><p>It was ensuring those decisions remained fast, explainable, resilient, and continuously improvable.</p><p>That’s what transforms personalization from a feature into a platform capability.</p><h4>Coming in Part 22</h4><p>Atlas can now make millions of intelligent decisions every minute.</p><p>But another transformation is beginning.</p><p>Engineers are no longer asking systems for information.</p><p>They’re asking systems to complete tasks.</p><p>What happens when autonomous AI agents become first-class components inside a distributed platform?</p><p>How do they coordinate safely?</p><p>How do they access tools securely?</p><p>How do they recover from failures, maintain memory, and collaborate with existing microservices?</p><p>In <strong>Part 22: AI Agents in Production — Building Autonomous Systems That Work Alongside Microservices</strong>, we’ll explore agent orchestration, tool calling, memory architectures, Model Context Protocol (MCP), retrieval-augmented generation (RAG), multi-agent workflows, governance, and the engineering patterns required to deploy AI agents safely in production.</p><p>Because the future of software isn’t replacing microservices with AI.</p><p><strong>It’s teaching AI to become another well-engineered service inside your platform.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03a13beddda0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Tenant SaaS Architecture: Building One Platform for Thousands of Enterprises]]></title>
            <link>https://medium.com/@summitkumar/multi-tenant-saas-architecture-building-one-platform-for-thousands-of-enterprises-75060250c644?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/75060250c644</guid>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Sun, 09 Aug 2026 23:41:01 GMT</pubDate>
            <atom:updated>2026-08-09T23:41:01.417Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 19 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“Building software for one customer is engineering. Building one platform that securely serves ten thousand customers is architecture.”</strong></blockquote><p>By the end of Part 18, Atlas had fundamentally changed how it stored information.</p><p>Events became the source of truth.</p><p>Read models evolved independently.</p><p>Services owned their own data.</p><p>The platform could scale technically.</p><p>Then something changed — not in the architecture, but in the business.</p><p>The sales team landed a second enterprise customer.</p><p>A month later, five more.</p><p>Soon, Atlas wasn’t powering a single organization.</p><p>It was serving banks, insurance companies, retailers, logistics providers, healthcare organizations, and fintech companies simultaneously.</p><p>Every customer wanted the same platform.</p><p>But none of them wanted to share their data.</p><p>That was the day Atlas stopped thinking about <strong>users</strong>…</p><p>…and started thinking about <strong>tenants</strong>.</p><figure><img alt="Scaling Multi-Tenant SaaS Architecture" src="https://cdn-images-1.medium.com/max/1024/1*1LskKtku3O68Ho29wNhshg.png" /><figcaption>Scaling Multi-Tenant SaaS Architecture</figcaption></figure><h4>What Is a Tenant?</h4><p>A tenant isn’t just another customer.</p><p>A tenant is an isolated business operating inside the same platform.</p><p>Imagine Atlas serving three organizations.</p><pre>Atlas<br>├── Acme Bank<br>├── HealthOne<br>└── RetailHub</pre><p>Each organization expects:</p><ul><li><em>Its own users</em></li><li><em>Its own campaigns</em></li><li><em>Its own workflows</em></li><li><em>Its own branding</em></li><li><em>Its own reports</em></li><li><em>Its own security policies</em></li></ul><p>From their perspective…</p><p>They’re the only customer using Atlas.</p><h4>The First Mistake</h4><p>Early Atlas looked like this.</p><pre>Users<br>Campaigns<br>Notifications</pre><p>Every table stored all customers together.</p><p>When the second tenant arrived, engineers added:</p><pre>tenant_id</pre><p>Problem solved?</p><p>Not quite.</p><p>One forgotten WHERE clause later…</p><pre>SELECT *<br>FROM campaigns;</pre><p>Instead of:</p><pre>SELECT *<br>FROM campaigns<br>WHERE tenant_id = ?</pre><p>One customer could accidentally see another customer’s data.</p><p>The platform remained functional.</p><p>Trust disappeared instantly.</p><h4>Tenant Isolation Is More Than Data</h4><p>Isolation exists at multiple layers.</p><pre>Users<br>↓<br>Authentication<br>↓<br>Authorization<br>↓<br>API Layer<br>↓<br>Business Logic<br>↓<br>Database<br>↓<br>Storage<br>↓<br>Monitoring</pre><p>Every request carries tenant identity.</p><p>Every service validates it.</p><p>Every database query enforces it.</p><p>Every log records it.</p><p>Isolation isn’t one feature.</p><p>It’s an architectural principle.</p><h4>Three Multi-Tenant Models</h4><p>Atlas evaluated three approaches.</p><h4>Model 1 — Shared Database</h4><pre>PostgreSQL<br>↓<br>Campaigns<br>tenant_id<br>Users<br>tenant_id</pre><p>Advantages:</p><ul><li><strong><em>Lowest infrastructure cost</em></strong></li><li><strong><em>Easy onboarding</em></strong></li><li><strong><em>Efficient resource usage</em></strong></li></ul><p>Challenges:</p><ul><li><strong><em>Strong isolation required</em></strong></li><li><strong><em>Larger shared tables</em></strong></li><li><strong><em>Noisy neighbor risks</em></strong></li></ul><h4>Model 2 — Separate Schema</h4><pre>Database<br>├── Tenant A Schema<br>├── Tenant B Schema<br>└── Tenant C Schema</pre><p>Advantages:</p><ul><li><strong><em>Better isolation</em></strong></li><li><strong><em>Independent migrations</em></strong></li></ul><p>Challenges:</p><ul><li><strong><em>Operational complexity</em></strong></li><li><strong><em>Schema management</em></strong></li></ul><h4>Model 3 — Dedicated Database</h4><pre>Tenant A<br>↓<br>Database A<br>────────────<br>Tenant B<br>↓<br>Database B<br>────────────<br>Tenant C<br>↓<br>Database C</pre><p>Advantages:</p><ul><li><strong><em>Maximum isolation</em></strong></li><li><strong><em>Independent scaling</em></strong></li><li><strong><em>Easier regulatory compliance</em></strong></li></ul><p>Challenges:</p><ul><li><strong><em>Higher infrastructure cost</em></strong></li><li><strong><em>Operational overhead</em></strong></li></ul><h4>Atlas Chooses a Hybrid Strategy</h4><p>Not every tenant has identical requirements.</p><p>A startup with 50 users doesn’t need a dedicated database.</p><p>A global bank probably does.</p><p>Atlas introduces service tiers.</p><pre>Standard<br>↓<br>Shared Database<br>────────────<br>Enterprise<br>↓<br>Dedicated Database</pre><p>The platform supports both.</p><p>Customers choose according to business requirements.</p><h4>Tenant Resolution</h4><p>Every request begins by identifying the tenant.</p><p>Example:</p><pre>https://acme.atlas.com</pre><p>or</p><pre>X-Tenant-ID<br>acme-bank</pre><p>The gateway resolves:</p><pre>Incoming Request<br>↓<br>Identity<br>↓<br>Tenant<br>↓<br>Routing</pre><p>Every downstream service receives tenant context automatically.</p><p>Applications never guess.</p><h4>Tenant-Aware Authentication</h4><p>Authentication alone isn’t sufficient.</p><p>Suppose the same email exists in two organizations.</p><pre>admin@company.com</pre><p>Atlas authenticates users within tenant boundaries.</p><pre>Identity<br>↓<br>Tenant<br>↓<br>Roles<br>↓<br>Permissions</pre><p>A system administrator in Tenant A has zero authority in Tenant B.</p><p>Identity always includes organizational context.</p><h4>Tenant Configuration</h4><p>Every customer wants customization.</p><p>Examples include:</p><ul><li><em>Notification templates</em></li><li><em>Brand colors</em></li><li><em>Logo</em></li><li><em>Time zone</em></li><li><em>Language</em></li><li><em>Notification limits</em></li><li><em>Compliance policies</em></li></ul><p>Instead of creating custom deployments…</p><p>Atlas stores tenant configuration.</p><pre>Tenant Configuration<br>↓<br>Workflow Engine<br>↓<br>Notification Service<br>↓<br>UI</pre><p>Behavior changes.</p><p>Code doesn’t.</p><h4>Noisy Neighbor Protection</h4><p>Imagine one tenant launches:</p><pre>50 Million Notifications</pre><p>Should everyone else suffer?</p><p>Absolutely not.</p><p>Atlas introduces quotas.</p><p>Examples:</p><pre>API Requests<br>↓<br>Per Tenant<br>────────────<br>Kafka Throughput<br>↓<br>Per Tenant<br>────────────<br>Worker Pools<br>↓<br>Per Tenant</pre><p>Large customers cannot monopolize shared resources.</p><p>Fairness becomes automatic.</p><h4>Tenant-Aware Caching</h4><p>Caching introduces subtle problems.</p><p>Suppose Redis stores:</p><pre>Campaign<br>ID<br>101</pre><p>Another tenant also owns Campaign 101.</p><p>Without tenant awareness…</p><p>The wrong campaign appears.</p><p>Atlas namespaces every cache key.</p><pre>tenant:acme<br>campaign:101</pre><p>Isolation extends beyond databases.</p><h4>Multi-Tenant Observability</h4><p>Production dashboards answer questions like:</p><pre>Which Tenant?<br>↓<br>API Errors<br>────────────<br>Notification Latency<br>────────────<br>Traffic Growth<br>────────────<br>Infrastructure Usage</pre><p>One tenant experiencing problems doesn’t hide inside platform-wide averages.</p><p>Every organization receives visibility into its own workload.</p><h4>Scaling by Tenant</h4><p>Atlas no longer scales only by CPU.</p><p>It scales by customer demand.</p><p>Imagine:</p><pre>RetailHub<br>Holiday Sale<br>↓<br>Autoscale Notification Workers</pre><p>Meanwhile…</p><p>Healthcare tenants remain unchanged.</p><p>Scaling becomes tenant-aware.</p><p>Not platform-wide.</p><h4>The Atlas Multi-Tenant Architecture</h4><pre>                  Global Users<br>                        │<br>                        ▼<br>                  API Gateway<br>                        │<br>              Tenant Resolution<br>                        │<br>        ┌───────────────┼────────────────┐<br>        ▼               ▼                ▼<br> Authentication   Configuration    Routing<br>        │               │                │<br>        ▼               ▼                ▼<br>     Shared      Dedicated      Shared<br>    Services      Services       Cache<br>        │               │<br>        └───────────────┼────────────────┐<br>                        ▼<br>                Event Platform<br>                        │<br>                        ▼<br>             Tenant-Aware Storage</pre><p>Notice something important.</p><p>The application doesn’t ask:</p><blockquote><strong>“Who is the user?”</strong></blockquote><p>It asks:</p><blockquote><strong>“Who is the tenant, and who is the user inside that tenant?”</strong></blockquote><p>Everything else flows from that decision.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc3564d3a89dcda1d59e6a09cbd25142/href">https://medium.com/media/dc3564d3a89dcda1d59e6a09cbd25142/href</a></iframe><p>There is no universal multi-tenant architecture.</p><p>The right choice depends on customer expectations, regulatory requirements, and operational maturity.</p><h4>Lessons Learned</h4><p>One of the biggest misconceptions about SaaS is believing multi-tenancy is simply adding a tenant_id column.</p><p>In reality, tenancy touches every layer of the platform.</p><p><em>Authentication.<br>Authorization.<br>Caching.<br>Messaging.<br>Observability.<br>Scaling.<br>Security.<br>Compliance.<br>Billing.<br>Deployment.</em></p><p>Atlas didn’t become multi-tenant because it shared infrastructure.</p><p>It became multi-tenant because every architectural decision respected organizational boundaries by default.</p><p>The best SaaS platforms don’t make tenants feel like they’re sharing a system.</p><p>They make every tenant feel like the platform was built exclusively for them.</p><h4>Coming in Part 20</h4><p>Atlas now serves thousands of organizations securely from a single platform.</p><p>But another challenge is emerging.</p><p>Different clients consume Atlas in completely different ways.</p><p>Mobile applications need compact responses.</p><p>Web dashboards require aggregated business data.</p><p>Partner APIs demand long-term stability.</p><p>How do you build an edge architecture that serves every client efficiently without coupling them to your internal microservices?</p><p>In <strong>Part 20: The Modern Edge — API Gateways, Backend for Frontend (BFF) &amp; Edge Computing</strong>, we’ll explore API Gateways, request aggregation, Backend for Frontend (BFF), GraphQL, edge caching, intelligent routing, CDN strategies, and the architecture that separates customer experience from backend implementation.</p><p>Because at scale…</p><p><strong>Your customers don’t interact with your microservices. They interact with the edge of your platform.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75060250c644" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event Sourcing & CQRS: Rethinking State in Distributed Systems]]></title>
            <link>https://medium.com/@summitkumar/event-sourcing-cqrs-rethinking-state-in-distributed-systems-823e1d2d6dc1?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/823e1d2d6dc1</guid>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[cqrs-pattern]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Sun, 09 Aug 2026 00:16:01 GMT</pubDate>
            <atom:updated>2026-08-09T00:16:01.315Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 18 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“Traditional systems remember the latest state. Great distributed systems remember every decision that created it.”</strong></blockquote><p>By the end of Part 17, Atlas had mastered one of the hardest engineering challenges: evolving production databases without downtime.</p><p>Schemas could change.</p><p>Services could evolve independently.</p><p>Entire databases could migrate while customers continued using the platform.</p><p>But another challenge quietly emerged.</p><p>Every important action inside Atlas already generated an event.</p><ul><li><em>Campaign Created</em></li><li><em>Campaign Approved</em></li><li><em>Notification Scheduled</em></li><li><em>Notification Delivered</em></li><li><em>Notification Opened</em></li><li><em>Workflow Started</em></li><li><em>Workflow Completed</em></li><li><em>Customer Subscribed</em></li></ul><p>Initially, these events were treated as messages.</p><p>Useful.</p><p>Temporary.</p><p>Disposable.</p><p>Then an engineer asked a deceptively simple question.</p><blockquote><strong>“If every change already exists as an event, why is the database still our source of truth?”</strong></blockquote><p>That question fundamentally changed how Atlas stored information.</p><figure><img alt="Mastering Event Sourcing and CQRS" src="https://cdn-images-1.medium.com/max/1024/1*sfn4IFANPh0oktLQl1Tucw.png" /><figcaption>Mastering Event Sourcing and CQRS</figcaption></figure><h4>The Problem with Current State</h4><p>Most applications only remember the latest value.</p><p>Imagine a campaign record.</p><pre>Campaign<br>Status = ACTIVE</pre><p>Simple.</p><p>Until Product asks:</p><ul><li><em>When was it approved?</em></li><li><em>Who approved it?</em></li><li><em>What was its previous status?</em></li><li><em>How many times was it modified?</em></li><li><em>Can we rebuild yesterday’s state?</em></li></ul><p>The database has one answer.</p><pre>Current State</pre><p>Everything else is gone.</p><p>History disappears every time an UPDATE statement executes.</p><h4>State Is a Consequence of Events</h4><p>Instead of storing the latest status…</p><p>Atlas stores every change.</p><pre>Campaign Created<br>↓<br>Campaign Updated<br>↓<br>Campaign Submitted<br>↓<br>Campaign Approved<br>↓<br>Campaign Activated<br>↓<br>Campaign Paused<br>↓<br>Campaign Resumed</pre><p>The current status is no longer stored directly.</p><p>It is <em>derived</em>.</p><pre>Replay Events<br>↓<br>Current State</pre><p>State becomes a calculation.</p><p>History becomes permanent.</p><h4>The Event Store</h4><p>Atlas introduced a dedicated Event Store.</p><pre>Event Store<br>┌────────────────────────────────────┐<br>1  CampaignCreated<br>2  CampaignUpdated<br>3  CampaignApproved<br>4  CampaignActivated<br>5  NotificationScheduled<br>6  NotificationDelivered<br>└────────────────────────────────────┘</pre><p>Notice something important.</p><p>Events are never modified.</p><p>Never deleted.</p><p>Never updated.</p><p>They are immutable.</p><h4>Why Immutability Matters</h4><p>Traditional systems allow this.</p><pre>UPDATE campaigns<br>SET status=&#39;ACTIVE&#39;;</pre><p>After execution…</p><p>Nobody knows what happened before.</p><p>Event sourcing records intent.</p><pre>{<br>  &quot;eventType&quot;: &quot;CampaignActivated&quot;,<br>  &quot;campaignId&quot;: &quot;cmp-101&quot;,<br>  &quot;activatedBy&quot;: &quot;marketing-admin&quot;,<br>  &quot;timestamp&quot;: &quot;2026-07-31T09:45:18Z&quot;<br>}</pre><p>The system remembers:</p><ul><li><em>What changed</em></li><li><em>Who changed it</em></li><li><em>When it changed</em></li><li><em>Why it changed</em></li></ul><p>Every business decision becomes auditable.</p><h4>Rebuilding State</h4><p>Suppose Campaign 101 has these events.</p><pre>Campaign Created<br>↓<br>Campaign Updated<br>↓<br>Campaign Approved<br>↓<br>Campaign Activated</pre><p>Rebuilding becomes straightforward.</p><pre>Campaign campaign = new Campaign();<br>events.forEach(event -&gt;<br>    campaign.apply(event));</pre><p>Every event transforms the aggregate.</p><p>After replaying the sequence…</p><p>The campaign reaches its latest state.</p><p>No UPDATE statements required.</p><h4>Commands and Queries Have Different Goals</h4><p>Atlas also discovered another problem.</p><p>The system responsible for writing data wasn’t optimized for reading it.</p><p>Marketing dashboards wanted:</p><ul><li><em>Campaign summaries</em></li><li><em>Aggregated metrics</em></li><li><em>Daily reports</em></li></ul><p>The write model wanted:</p><ul><li><em>Consistency</em></li><li><em>Validation</em></li><li><em>Business rules</em></li></ul><p>Trying to optimize both using a single database led to compromises.</p><p>Atlas adopted CQRS.</p><h4>Command Query Responsibility Segregation (CQRS)</h4><p>Instead of one model…</p><p>Atlas separates responsibilities.</p><pre>            Command API<br>                  │<br>                  ▼<br>           Event Store<br>                  │<br>                  ▼<br>         Event Processing<br>                  │<br>        ┌─────────┴─────────┐<br>        ▼                   ▼<br> Search View          Analytics View</pre><p>Commands modify state.</p><p>Queries optimize retrieval.</p><p>Each evolves independently.</p><h4>Read Models</h4><p>When a campaign changes…</p><p>Atlas publishes an event.</p><pre>Campaign Approved<br>↓<br>Kafka<br>↓<br>Projection Service<br>↓<br>Read Database</pre><p>Marketing dashboards query the read database.</p><p>Not the Event Store.</p><p>Read models are optimized for speed.</p><p>The Event Store is optimized for correctness.</p><h4>Multiple Views from One Event Stream</h4><p>One event can build many projections.</p><pre>        Notification Delivered<br>        │<br>        ├──────────────► Customer Timeline<br>        │<br>        ├──────────────► Analytics<br>        │<br>        ├──────────────► Recommendation Engine<br>        │<br>        └──────────────► Audit Platform</pre><p>No duplication at the source.</p><p>Only specialized representations downstream.</p><p>This dramatically reduces coupling between services.</p><h4>Snapshotting</h4><p>Replaying ten events is easy.</p><p>Replaying ten million isn’t.</p><p>Atlas periodically creates snapshots.</p><pre>Events<br>1–10,000<br>↓<br>Snapshot<br>↓<br>Events<br>10,001–10,500</pre><p>To rebuild state:</p><ol><li><em>Load the latest snapshot.</em></li><li><em>Replay only newer events.</em></li></ol><p>Recovery time remains predictable even after years of operation.</p><h4>Versioning Events</h4><p>Business evolves.</p><p>Events must evolve too.</p><p>Version 1:</p><pre>{<br>  &quot;eventType&quot;: &quot;CampaignCreated&quot;,<br>  &quot;name&quot;: &quot;Flash Sale&quot;<br>}</pre><p>Version 2:</p><pre>{<br>  &quot;eventType&quot;: &quot;CampaignCreated&quot;,<br>  &quot;name&quot;: &quot;Flash Sale&quot;,<br>  &quot;channel&quot;: &quot;Push&quot;<br>}</pre><p>Atlas never rewrites historical events.</p><p>Instead, consumers understand multiple versions.</p><p>History remains immutable.</p><p>Applications remain forward compatible.</p><h4>Temporal Debugging</h4><p>One unexpected benefit surprised the engineering team.</p><p>Production debugging became dramatically easier.</p><p>Instead of asking:</p><blockquote><strong>“What is wrong with Campaign 101?”</strong></blockquote><p>Atlas asks:</p><blockquote><strong>“Replay everything that ever happened to Campaign 101.”</strong></blockquote><p>Every decision becomes observable.</p><p>Every bug becomes reproducible.</p><p>Every customer complaint becomes traceable.</p><p>Time itself becomes a debugging tool.</p><h4>Event Replay</h4><p>Suppose Analytics introduces a new KPI.</p><p>Traditional systems require expensive historical backfills.</p><p>Atlas replays events.</p><pre>Event Store<br>↓<br>Replay<br>↓<br>New Projection<br>↓<br>Analytics Ready</pre><p>No production downtime.</p><p>No risky migration.</p><p>History already contains everything required.</p><h4>Atlas Event Architecture</h4><pre>              Command API<br>                    │<br>                    ▼<br>            Business Validation<br>                    │<br>                    ▼<br>              Event Store<br>                    │<br>          Immutable Event Log<br>                    │<br>          ┌─────────┼──────────┐<br>          ▼         ▼          ▼<br>      Kafka      Snapshot   Replay Engine<br>          │<br>          ▼<br>      Projection Services<br>      │        │         │<br>      ▼        ▼         ▼<br>   Search   Analytics  Customer Timeline</pre><p>Notice the architectural shift.</p><p>The Event Store doesn’t replace databases.</p><p>It becomes the authoritative history.</p><p>Databases become optimized views of that history.</p><h4>When Not to Use Event Sourcing</h4><p>Atlas didn’t adopt Event Sourcing everywhere.</p><p>Some workloads remain simpler with CRUD.</p><p>Examples include:</p><ul><li><strong><em>Feature flags</em></strong></li><li><strong><em>Configuration</em></strong></li><li><strong><em>Temporary caches</em></strong></li><li><strong><em>Session storage</em></strong></li><li><strong><em>Small administrative tools</em></strong></li></ul><p>Event Sourcing introduces:</p><ul><li><strong><em>Additional infrastructure</em></strong></li><li><strong><em>More complex debugging</em></strong></li><li><strong><em>Event versioning</em></strong></li><li><strong><em>Projection management</em></strong></li></ul><p>The benefits justify the cost only when history is valuable.</p><p>Architecture should solve business problems — not chase patterns.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a338009b697a2c675e1b57569c7cc588/href">https://medium.com/media/a338009b697a2c675e1b57569c7cc588/href</a></iframe><p>The goal isn’t eliminating relational databases.</p><p>It’s recognizing that <strong>history itself can be the primary data model</strong>.</p><h4>Lessons Learned</h4><p>One of the biggest misconceptions about Event Sourcing is that it’s a database technology.</p><p>It isn’t.</p><p>It’s a way of thinking.</p><p>Traditional systems ask:</p><blockquote><strong>“What does the data look like now?”</strong></blockquote><p>Event-Sourced systems ask:</p><blockquote><strong>“What happened?”</strong></blockquote><p>That subtle shift unlocks capabilities that are nearly impossible with CRUD alone:</p><ul><li><strong><em>Complete audit trails</em></strong></li><li><strong><em>Temporal debugging</em></strong></li><li><strong><em>Rebuilding projections</em></strong></li><li><strong><em>Replay-based migrations</em></strong></li><li><strong><em>Rich business analytics</em></strong></li><li><strong><em>Natural integration with event-driven architectures</em></strong></li></ul><p>Atlas still uses PostgreSQL.</p><p>It still uses Redis.</p><p>It still uses Kafka.</p><p>But the platform no longer treats today’s state as the most valuable information.</p><p>It treats <strong>the journey to that state</strong> as the true system of record.</p><h4>Coming in Part 19</h4><p>Atlas can now preserve every business decision, rebuild state from history, and generate multiple optimized views from a single stream of events.</p><p>Yet another challenge emerges as the platform expands.</p><p>Different enterprises begin using Atlas.</p><p>Each wants complete isolation.</p><p>Their own users.</p><p>Their own campaigns.</p><p>Their own compliance rules.</p><p>Their own branding.</p><p>How do you operate a single platform for thousands of organizations without turning every deployment into a custom implementation?</p><p>In <strong>Part 19: Multi-Tenant SaaS Architecture — Building One Platform for Thousands of Enterprises</strong>, we’ll explore tenant isolation models, shared versus dedicated infrastructure, tenant-aware security, data partitioning, noisy-neighbor protection, per-tenant customization, and the architectural patterns that enable SaaS platforms to scale from one customer to ten thousand.</p><p>Because the hardest part of scaling software isn’t serving more users.</p><p><strong>It’s serving more businesses with the same platform.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=823e1d2d6dc1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Downtime Data Evolution: Migrating Production Databases Without Users Ever Knowing]]></title>
            <link>https://medium.com/@summitkumar/zero-downtime-data-evolution-migrating-production-databases-without-users-ever-knowing-d354f941ad5c?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d354f941ad5c</guid>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Fri, 07 Aug 2026 23:56:01 GMT</pubDate>
            <atom:updated>2026-08-07T23:56:01.595Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 17 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“The most dangerous production deployment isn’t a new application. It’s a database migration.”</strong></blockquote><p>Over the last sixteen articles, Atlas has evolved into a globally distributed, AI-native customer engagement platform.</p><p>It can survive:</p><ul><li><strong><em>Server failures</em></strong></li><li><strong><em>Availability Zone failures</em></strong></li><li><strong><em>Regional outages</em></strong></li><li><strong><em>Massive traffic spikes</em></strong></li><li><strong><em>Security incidents</em></strong></li><li><strong><em>Infrastructure failures</em></strong></li></ul><p>Ironically, one of the highest-risk production changes still wasn’t infrastructure.</p><p>It was changing a single database table.</p><p>Every engineer has experienced it.</p><p>A simple request arrives from Product.</p><blockquote><strong>“Can we add one new field to the Campaign table?”</strong></blockquote><p>Looks harmless.</p><p>Until you realize:</p><ul><li><strong><em>Hundreds of services depend on that schema.</em></strong></li><li><strong><em>Millions of records already exist.</em></strong></li><li><strong><em>Thousands of requests arrive every second.</em></strong></li><li><strong><em>There is no maintenance window.</em></strong></li><li><strong><em>Customers are online across every time zone.</em></strong></li></ul><p>Modern platforms don’t stop serving customers so engineers can migrate databases.</p><p>They evolve while the business keeps running.</p><p>That became Atlas’ next engineering challenge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tamyrjf_drNi3ALoHe0EKg.png" /><figcaption>Zero-Downtime Database Evolution Blueprint</figcaption></figure><h4>The Monolith Database Problem</h4><p>In Atlas’ early architecture, everything lived together.</p><pre>    PostgreSQL<br>   ┌─────────────────────────────────┐<br>   │ Users                           │<br>   │ Campaigns                       │<br>   │ Notifications                   │<br>   │ Analytics                       │<br>   │ Workflow                        │<br>   │ Billing                         │<br>   └─────────────────────────────────┘</pre><p>One schema.</p><p>One deployment.</p><p>One migration.</p><p>One mistake.</p><p>One outage.</p><p>As the platform grew, this became increasingly dangerous.</p><h4>Schema Changes Are API Changes</h4><p>Developers often treat database schemas differently from APIs.</p><p>They shouldn’t.</p><p>Suppose Version 1 looks like this.</p><pre>CREATE TABLE campaigns (<br>    id UUID PRIMARY KEY,<br>    campaign_name VARCHAR(255)<br>);</pre><p>A developer decides to rename the column.</p><pre>ALTER TABLE campaigns<br>RENAME COLUMN campaign_name TO name;</pre><p>The migration succeeds.</p><p>Production doesn’t.</p><p>Every service still executes:</p><pre>SELECT campaign_name<br>FROM campaigns;</pre><p>Applications fail instantly.</p><p>The database didn’t break.</p><p>The contract did.</p><h4>The Expand–Contract Pattern</h4><p>Atlas adopted a simple rule.</p><blockquote><strong>Never remove something until nobody uses it.</strong></blockquote><p>Instead of replacing a column…</p><p>We expand first.</p><pre>ALTER TABLE campaigns<br>ADD COLUMN name VARCHAR(255);</pre><p>Both fields now exist.</p><pre>campaign_name<br>name</pre><p>Applications gradually migrate.</p><p>Only after every service uses the new column…</p><pre>ALTER TABLE campaigns<br>DROP COLUMN campaign_name;</pre><p>No downtime.</p><p>No surprises.</p><p>Just evolution.</p><h4>Dual Writes</h4><p>During migration, Atlas temporarily writes to both schemas.</p><pre>Campaign campaign = new Campaign();<br>campaign.setCampaignName(name);<br>campaign.setName(name);<br>campaignRepository.save(campaign);</pre><p>Now both old and new applications continue functioning.</p><p>Temporary duplication dramatically reduces migration risk.</p><h4>Backfilling Existing Data</h4><p>Adding a column is easy.</p><p>Migrating millions of existing records isn’t.</p><p>The naive approach:</p><pre>UPDATE campaigns<br>SET name = campaign_name;</pre><p>Production immediately experiences:</p><ul><li><em>Table locks</em></li><li><em>Long-running transactions</em></li><li><em>Increased replication lag</em></li><li><em>API latency spikes</em></li></ul><p>Atlas migrates incrementally.</p><pre>Batch 1<br>10,000 Rows<br>↓<br>Batch 2<br>10,000 Rows<br>↓<br>Batch 3<br>10,000 Rows</pre><p>Small batches reduce operational risk.</p><p>Customers never notice.</p><h4>Change Data Capture (CDC)</h4><p>Backfills solve historical data.</p><p>What about new writes happening during migration?</p><p>Atlas introduces Change Data Capture.</p><pre>Application<br>↓<br>Database Write<br>↓<br>Transaction Log<br>↓<br>CDC<br>↓<br>Migration Pipeline</pre><p>Every change automatically propagates.</p><p>Historical data catches up.</p><p>New data never falls behind.</p><h4>Shadow Reads</h4><p>How do you trust a migration before switching production traffic?</p><p>Atlas compares both paths.</p><pre>Application<br>↓<br>Old Column<br>↓<br>Response<br>──────────────<br>Application<br>↓<br>New Column<br>↓<br>Compare</pre><p>Only one response reaches customers.</p><p>The second validates correctness.</p><p>If results differ…</p><p>Traffic never switches.</p><p>Confidence grows before risk.</p><h4>Splitting the Monolith Database</h4><p>Eventually, Atlas outgrew a shared database.</p><p>Instead of:</p><pre>One Database<br>↓<br>Everything</pre><p>The platform evolved into independently owned data stores.</p><pre>Campaign Service<br>↓<br>Campaign Database<br>──────────────<br>Workflow Service<br>↓<br>Workflow Database<br>──────────────<br>Analytics Service<br>↓<br>Analytics Database</pre><p>Each service owns its data.</p><p>No shared schemas.</p><p>No accidental coupling.</p><h4>Data Synchronization Without Shared Tables</h4><p>Previously, services executed joins.</p><pre>SELECT *<br>FROM campaigns c<br>JOIN notifications n</pre><p>Across independent databases…</p><p>This becomes impossible.</p><p>Atlas shifts integration from queries to events.</p><pre>Campaign Updated<br>↓<br>Kafka<br>↓<br>Notification Service<br>↓<br>Local Database</pre><p>Data moves through events.</p><p>Not SQL joins.</p><p>The architecture becomes loosely coupled.</p><h4>Online Index Creation</h4><p>Indexes improve performance.</p><p>Creating them incorrectly destroys production performance.</p><p>Traditional approach:</p><pre>CREATE INDEX idx_campaign_name<br>ON campaigns(name);</pre><p>Large tables become unavailable.</p><p>Atlas creates indexes online.</p><pre>CREATE INDEX CONCURRENTLY<br>idx_campaign_name<br>ON campaigns(name);</pre><p>Reads continue.</p><p>Writes continue.</p><p>Users continue working.</p><p>Database evolution becomes invisible.</p><h4>Versioning Database Migrations</h4><p>Application code already uses Git.</p><p>Database changes deserve the same discipline.</p><pre>V101<br>Create Campaign Table<br>↓<br>V102<br>Add Status Column<br>↓<br>V103<br>Create Index<br>↓<br>V104<br>Remove Legacy Column</pre><p>Every environment evolves identically.</p><p>Developers never manually execute production SQL.</p><p>Automation becomes the source of truth.</p><h4>Blue-Green Database Migrations</h4><p>For critical migrations, Atlas temporarily maintains two environments.</p><pre>    Application<br>        │<br>        ▼<br>   Blue Database<br>  ──────────────<br>   Green Database</pre><p>Traffic gradually shifts.</p><p>Validation occurs continuously.</p><p>Rollback remains immediate.</p><p>Infrastructure costs increase temporarily.</p><p>Deployment confidence increases permanently.</p><h4>Data Quality Monitoring</h4><p>Migration success isn’t measured by completion.</p><p>It’s measured by correctness.</p><p>Atlas continuously validates:</p><ul><li><strong><em>Row counts</em></strong></li><li><strong><em>Checksums</em></strong></li><li><strong><em>Null percentages</em></strong></li><li><strong><em>Duplicate records</em></strong></li><li><strong><em>Foreign key consistency</em></strong></li><li><strong><em>Event ordering</em></strong></li></ul><p>Example:</p><pre>Campaign Records<br>Old Database<br>1,245,889<br>──────────────<br>New Database<br>1,245,889<br>✓ Match</pre><p>Automation catches inconsistencies long before customers do.</p><h4>Atlas Data Evolution Pipeline</h4><pre>            Application<br>                 │<br>         Dual Writes Enabled<br>                 │<br>      ┌──────────┴──────────┐<br>      ▼                     ▼<br>Legacy Schema         Expanded Schema<br>      │                     │<br>      └──────────┬──────────┘<br>                 ▼<br>         Change Data Capture<br>                 │<br>                 ▼<br>         Migration Workers<br>                 │<br>                 ▼<br>      Validation &amp; Shadow Reads<br>                 │<br>                 ▼<br>        Traffic Cutover<br>                 │<br>                 ▼<br>       Legacy Schema Removed</pre><p>Notice the pattern.</p><p>Every migration follows predictable phases.</p><p>Nothing changes all at once.</p><p>Risk is reduced through incremental evolution.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c61e8d0807bd66f2444abb8f35d2fa8/href">https://medium.com/media/7c61e8d0807bd66f2444abb8f35d2fa8/href</a></iframe><p>Every migration balances speed against safety.</p><p>Atlas consistently chooses predictable evolution over risky shortcuts.</p><h4>Lessons Learned</h4><p>Database migrations are rarely dangerous because SQL is difficult.</p><p>They’re dangerous because production never stops.</p><p>Customers don’t care that a schema changed.</p><p>They care that their notification was delivered.</p><p>Their campaign was saved.</p><p>Their workflow continued.</p><p>Atlas treats data evolution as a continuous engineering discipline rather than an occasional operational event.</p><p>Schemas evolve.</p><p>Services evolve.</p><p>Applications evolve.</p><p>But production remains available.</p><p>The biggest shift wasn’t technical.</p><p>It was cultural.</p><p>Engineers stopped asking:</p><blockquote><strong>“When is the maintenance window?”</strong></blockquote><p>They started asking:</p><blockquote><strong>“How can this change happen while customers continue using the platform?”</strong></blockquote><p>That’s the mindset required to operate software at global scale.</p><h4>Coming in Part 18</h4><p>Atlas now evolves applications, infrastructure, and databases without downtime.</p><p>Yet another challenge has quietly emerged.</p><p>Events have become the heartbeat of the platform.</p><p>Every customer action, workflow transition, recommendation, deployment, and audit trail flows through Kafka.</p><p>But what if events aren’t just messages?</p><p>What if they become the <strong>system of record</strong>?</p><p>In <strong>Part 18: Event Sourcing &amp; CQRS — Rethinking State in Distributed Systems</strong>, we’ll explore immutable event logs, rebuilding application state from history, command/query separation, snapshotting, replay strategies, temporal debugging, and why some of the world’s largest distributed systems treat events — not databases — as the ultimate source of truth.</p><p>Because once you realize that <strong>every state change is an event</strong>, you stop designing databases…</p><p><strong>…and start designing history.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d354f941ad5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Planetary Resilience: Designing Multi-Region Systems That Survive the Unthinkable]]></title>
            <link>https://medium.com/@summitkumar/planetary-resilience-designing-multi-region-systems-that-survive-the-unthinkable-b65560dfbd3a?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/b65560dfbd3a</guid>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Thu, 06 Aug 2026 23:51:01 GMT</pubDate>
            <atom:updated>2026-08-06T23:51:01.322Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 16 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“High availability keeps your system running. Resilience keeps your business running.”</strong></blockquote><p>Over the past fifteen articles, Atlas evolved from a simple notification service into an AI-native distributed platform.</p><p>It can:</p><ul><li><strong><em>Process billions of events daily</em></strong></li><li><strong><em>Scale across multiple continents</em></strong></li><li><strong><em>Recover from service failures</em></strong></li><li><strong><em>Secure every request</em></strong></li><li><strong><em>Deploy continuously</em></strong></li><li><strong><em>Optimize cloud infrastructure</em></strong></li><li><strong><em>Personalize customer engagement using AI</em></strong></li></ul><p>On paper, Atlas had achieved everything modern cloud platforms aspire to become.</p><p>Then one ordinary afternoon, an entire cloud region disappeared.</p><p>Not a single service failed.</p><p>An entire region did.</p><p>Load balancers stopped responding.</p><p>Kafka brokers became unreachable.</p><p>Databases were isolated.</p><p>Thousands of Kubernetes Pods continued running…</p><p>With nobody able to reach them.</p><p>Within seconds, millions of mobile applications started retrying.</p><p>Traffic doubled in every remaining region.</p><p>The platform wasn’t facing a service failure.</p><p>It was facing the failure of an entire geography.</p><p>That day taught us an important distinction.</p><blockquote><strong>Availability is about surviving server failures. Resilience is about surviving regional failures.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpL_oymMymhxTCPA8_Bxbw.png" /><figcaption>Planetary Resilience System Architecture Guide</figcaption></figure><h4>The Limits of High Availability</h4><p>Earlier in this series, Atlas adopted Multi-AZ deployments.</p><pre>          Region<br>            │<br>    ┌───────────────┐<br>   AZ-1           AZ-2<br>    │               │<br>    ▼               ▼<br> Application    Application<br>    │               │<br>    └───────┬───────┘<br>            ▼<br>        Database</pre><p>This architecture survives:</p><ul><li><em>Server failures</em></li><li><em>Rack failures</em></li><li><em>Availability Zone failures</em></li></ul><p>But not this.</p><pre>Entire Region<br>↓<br>Unavailable</pre><p>Everything disappears together.</p><h4>Thinking Beyond Regions</h4><p>Atlas expanded into multiple regions.</p><pre>                Global Users<br>                    │<br>        ┌───────────┼───────────┐<br>        ▼           ▼           ▼<br>   Mumbai     Frankfurt     Virginia</pre><p>Now the question changes.</p><p>Instead of asking:</p><blockquote><strong>“How do we recover?”</strong></blockquote><p>We ask:</p><blockquote><strong>“Can users continue without noticing?”</strong></blockquote><p>Recovery becomes secondary. Continuity becomes primary.</p><h4>Active-Passive Architecture</h4><p>The simplest multi-region design looks like this.</p><pre>Primary Region<br>↓<br>Handles Traffic<br>──────────────<br>Secondary Region<br>↓<br>Waiting</pre><p>If the primary region fails:</p><pre>DNS<br>↓<br>Secondary Region<br>↓<br>Traffic Restored</pre><p>Advantages:</p><ul><li><em>Simpler architecture</em></li><li><em>Lower operational complexity</em></li><li><em>Easier data consistency</em></li></ul><p>Disadvantages:</p><ul><li><em>Idle infrastructure</em></li><li><em>Recovery takes minutes</em></li><li><em>Capacity remains unused</em></li></ul><p>For many business systems, this is sufficient.</p><p>Atlas eventually outgrew it.</p><h4>Active-Active Architecture</h4><p>Instead of keeping one region idle, Atlas serves traffic across all regions.</p><pre>                  Global DNS<br>                      │<br>      ┌───────────────┼───────────────┐<br>      ▼               ▼               ▼<br>   Mumbai        Frankfurt       Virginia<br>      │               │               │<br>      ▼               ▼               ▼<br>  Live Traffic    Live Traffic    Live Traffic</pre><p>Every region accepts production traffic.</p><p>If one disappears…</p><p>Remaining regions continue serving users.</p><p>The platform doesn’t switch modes.</p><p>It simply loses capacity.</p><h4>Data Is the Hard Part</h4><p>Stateless services are easy.</p><p>Databases aren’t.</p><p>Imagine a customer updates their notification preferences in Mumbai.</p><p>Milliseconds later…</p><p>Another request reaches Frankfurt.</p><p>Which version is correct?</p><pre>Mumbai<br>Push Enabled = FALSE<br>──────────────<br>Frankfurt<br>Push Enabled = TRUE</pre><p>The network hasn’t failed.</p><p>Replication hasn’t completed yet.</p><p>Distributed systems call this <strong>replication lag</strong>.</p><h4>Synchronous vs Asynchronous Replication</h4><p>Synchronous replication waits.</p><pre>Write<br>↓<br>Mumbai<br>↓<br>Frankfurt<br>↓<br>Success</pre><p>Advantages:</p><ul><li><em>Strong consistency</em></li></ul><p>Disadvantages:</p><ul><li><em>Higher latency</em></li><li><em>Regional dependency</em></li></ul><p>Asynchronous replication behaves differently.</p><pre>Write<br>↓<br>Mumbai<br>↓<br>Success<br>↓<br>Replicate Later<br>↓<br>Frankfurt</pre><p>Advantages:</p><ul><li><em>Low latency</em></li><li><em>Better availability</em></li></ul><p>Trade-off:</p><p><em>Temporary inconsistency.</em></p><p>Atlas chooses based on business requirements — not technical preference.</p><h4>Not Every Piece of Data Is Equal</h4><p>Customer notification history?</p><p>Eventually consistent.</p><p>Authentication tokens?</p><p>Strong consistency.</p><p>Audit logs?</p><p>Immutable and replicated.</p><p>Marketing analytics?</p><p>Asynchronous.</p><p>Atlas classifies every dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82a1fd52a77c1f6e1a99e4c5aaa549be/href">https://medium.com/media/82a1fd52a77c1f6e1a99e4c5aaa549be/href</a></iframe><p>One replication strategy rarely fits every workload.</p><h4>Regional Traffic Management</h4><p>Suppose Frankfurt becomes unavailable.</p><p>Global traffic must shift immediately.</p><pre>               Global Traffic<br>                     │<br>          Frankfurt Unhealthy<br>                     │<br>                     ▼<br>      Mumbai      Virginia<br>        50%          50%</pre><p>Notice something important.</p><p>Traffic isn’t merely redirected.</p><p>Capacity planning already assumed one region could disappear.</p><p>Resilience begins long before failure occurs.</p><h4>Stateless Services Win</h4><p>Early Atlas services stored user sessions locally.</p><pre>Request<br>↓<br>Server A<br>↓<br>Session</pre><p>If traffic moved to another region…</p><p>The session disappeared.</p><p>Atlas centralized session state.</p><pre>User<br>↓<br>JWT<br>↓<br>Any Region<br>↓<br>Validated</pre><p>Requests become portable.</p><p>Regions become interchangeable.</p><h4>Messaging Across Regions</h4><p>Kafka clusters introduce another challenge.</p><p>Suppose Mumbai publishes an event.</p><pre>Notification Created</pre><p>Frankfurt also needs it.</p><p>Atlas introduces cross-region replication.</p><pre>Mumbai Kafka<br>↓<br>Replication<br>↓<br>Frankfurt Kafka<br>↓<br>Consumers</pre><p>Consumers remain local.</p><p>Events remain global.</p><p>The messaging layer survives regional failures without forcing every consumer to communicate across continents.</p><h4>Designing for Failure</h4><p>Traditional disaster recovery begins after failure.</p><p>Atlas prepares before failure.</p><p>Examples include:</p><ul><li><strong><em>Quarterly regional failover exercises</em></strong></li><li><strong><em>Automated backup restoration tests</em></strong></li><li><strong><em>Dependency isolation reviews</em></strong></li><li><strong><em>Simulated network partitions</em></strong></li><li><strong><em>Traffic shift rehearsals</em></strong></li></ul><p>If recovery procedures exist only in documentation…</p><p>They don’t exist.</p><h4>Chaos Engineering</h4><p>One Friday every month, Atlas intentionally breaks production.</p><p>Not recklessly.</p><p>Systematically.</p><p>Examples:</p><pre>Terminate Pods<br>↓<br>Observe Recovery</pre><pre>Block Database Connectivity<br>↓<br>Observe Circuit Breakers</pre><pre>Disable Region<br>↓<br>Observe Traffic Shift</pre><p>Every controlled experiment strengthens confidence.</p><p>Failures discovered during business hours are far cheaper than failures discovered during emergencies.</p><h4>Measuring Recovery</h4><p>Two metrics guide every resilience decision.</p><p><strong>Recovery Time Objective (RTO)</strong></p><p>How quickly must service return?</p><p>Example:</p><pre>API Gateway<br>RTO<br>5 Minutes</pre><p><strong>Recovery Point Objective (RPO)</strong></p><p>How much data can be lost?</p><p>Example:</p><pre>Customer Profiles<br>RPO<br>0 Seconds</pre><pre>Analytics<br>RPO<br>5 Minutes</pre><p>Different systems justify different targets.</p><p>Perfection is expensive.</p><p>Business priorities determine investment.</p><h4>Atlas’ Planetary Architecture</h4><pre>                    Global DNS<br>                         │<br>             Health-Based Routing<br>                         │<br>      ┌──────────────────┼──────────────────┐<br>      ▼                  ▼                  ▼<br>   Mumbai           Frankfurt          Virginia<br>      │                  │                  │<br>      ▼                  ▼                  ▼<br> Kubernetes        Kubernetes        Kubernetes<br>      │                  │                  │<br>      ▼                  ▼                  ▼<br> Local Kafka       Local Kafka       Local Kafka<br>      │                  │                  │<br>      └────────────┬─────┴─────┬────────────┘<br>                   ▼           ▼<br>            Cross-Region Replication<br>                   │<br>                   ▼<br>         Distributed Data Platform</pre><p>Notice a recurring theme throughout the Atlas series.</p><p>Every major architectural evolution reduces dependency on any single component.</p><p>Earlier we removed dependence on individual servers.</p><p>Then databases.</p><p>Then services.</p><p>Now…</p><p>Entire regions.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6e50644031cc6bd217902bec3a07a3f/href">https://medium.com/media/a6e50644031cc6bd217902bec3a07a3f/href</a></iframe><p>Resilience isn’t achieved by eliminating failures.</p><p>It’s achieved by designing systems that continue delivering business value despite them.</p><h4>Lessons Learned</h4><p>One of the biggest misconceptions about disaster recovery is believing it’s a backup strategy.</p><p>It isn’t.</p><p>Backups recover data.</p><p>Resilience preserves business continuity.</p><p>Atlas no longer assumes:</p><ul><li><strong><em>Regions remain available.</em></strong></li><li><strong><em>Networks remain connected.</em></strong></li><li><strong><em>Cloud providers remain healthy.</em></strong></li><li><strong><em>Replication is instantaneous.</em></strong></li></ul><p>Instead, it assumes failures will happen eventually.</p><p>Architectural decisions — from stateless services to cross-region messaging — exist because failure is inevitable, not exceptional.</p><p>The strongest systems aren’t those that never fail.</p><p>They’re the ones whose users barely notice when they do.</p><h4>Coming in Part 17</h4><p>Atlas can now survive service failures, regional outages, infrastructure disruptions, and global traffic shifts.</p><p>Yet one challenge remains.</p><p>Data itself.</p><p>How do companies migrate petabytes of production data without downtime?</p><p>How do they split monolithic databases into independently owned services?</p><p>How do they evolve schemas while processing billions of live requests?</p><p>In <strong>Part 17: Zero-Downtime Data Evolution — Online Migrations, Schema Versioning &amp; Live Database Modernization</strong>, we’ll explore expand-contract migrations, dual writes, change data capture (CDC), online schema evolution, backfills, shadow reads, and the engineering patterns that allow distributed systems to evolve without stopping production.</p><p>Because in long-lived systems…</p><p><strong>The architecture you build today is only valuable if it can evolve tomorrow.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b65560dfbd3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Intelligent Platforms: Building an AI-Native Customer Engagement System]]></title>
            <link>https://medium.com/@summitkumar/engineering-intelligent-platforms-building-an-ai-native-customer-engagement-system-44859d1d2132?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/44859d1d2132</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 23:46:01 GMT</pubDate>
            <atom:updated>2026-08-05T23:46:01.355Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 15 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“The first generation of software automated tasks. The next generation will automate decisions.”</strong></blockquote><p>Over the last fourteen articles, we’ve transformed Atlas from a notification service into a globally distributed customer engagement platform.</p><p>Today Atlas can:</p><ul><li><strong><em>Process billions of events every day</em></strong></li><li><strong><em>Execute complex customer journeys</em></strong></li><li><strong><em>Scale automatically across regions</em></strong></li><li><strong><em>Recover from failures</em></strong></li><li><strong><em>Deploy continuously</em></strong></li><li><strong><em>Secure every service</em></strong></li><li><strong><em>Optimize cloud costs</em></strong></li><li><strong><em>Enable hundreds of engineers through an internal developer platform</em></strong></li></ul><p>From a traditional software engineering perspective, Atlas is complete.</p><p>Then the business asked a question that no amount of infrastructure could answer.</p><blockquote><strong>“Which notification should we send to this customer right now?”</strong></blockquote><p>Not tomorrow.</p><p>Not after a report.</p><p>Not after a data scientist exports a CSV.</p><p>Right now.</p><p>That question marks the transition from <strong>building distributed systems</strong> to <strong>building intelligent distributed systems</strong>.</p><figure><img alt="Building an AI-Native Platform" src="https://cdn-images-1.medium.com/max/1024/1*tr5ket_UHqiWfWeh8Qo8Wg.png" /><figcaption>Building an AI-Native Platform</figcaption></figure><h4>Rules Eventually Stop Scaling</h4><p>Atlas originally relied on business rules.</p><pre>if(user.isPremium()) {<br>    sendPremiumOffer();<br>}<br><br>if(cart.isAbandoned()) {<br>    sendReminder();<br>}<br><br>if(user.hasNotLoggedInFor30Days()) {<br>    sendRetentionCampaign();<br>}</pre><p>It worked well.</p><p>Until Product created hundreds of campaigns.</p><p>Then thousands.</p><p>Different countries required different strategies.</p><p>User behavior changed constantly.</p><p>Marketing wanted experiments every week.</p><p>Soon the rule engine contained thousands of conditions.</p><p>No engineer understood all of them.</p><p>Business logic had become impossible to reason about.</p><h4>From Rules to Predictions</h4><p>Instead of asking:</p><blockquote><strong>“Does this user match a rule?”</strong></blockquote><p>Atlas begins asking:</p><blockquote><strong>“What is the probability this user will engage?”</strong></blockquote><p>Instead of returning:</p><pre>TRUE<br>or<br>FALSE</pre><p>The platform returns a score.</p><pre>Purchase Probability<br>87%</pre><pre>Open Probability<br>62%</pre><pre>Churn Risk<br>11%</pre><p>Software no longer makes binary decisions. It evaluates probabilities.</p><h4>Data Is the Real Product</h4><p>Before introducing machine learning, Atlas already possessed something incredibly valuable.</p><p>Events.</p><p>Every interaction produced data.</p><pre>Notification Delivered<br>↓<br>Notification Opened<br>↓<br>Product Viewed<br>↓<br>Cart Created<br>↓<br>Purchase Completed<br>↓<br>Subscription Renewed</pre><p>Individually, these events seem ordinary.</p><p>Collectively, they describe customer behavior.</p><p>Machine learning doesn’t create intelligence.</p><p>It discovers patterns hidden inside existing data.</p><h4>The Feature Pipeline</h4><p>Raw events aren’t immediately useful for prediction.</p><p>Models need features.</p><p>Examples include:</p><ul><li><em>Number of notifications opened in the last 30 days</em></li><li><em>Average purchase value</em></li><li><em>Preferred engagement hour</em></li><li><em>Device type</em></li><li><em>Country</em></li><li><em>Last active session</em></li><li><em>Campaign click-through rate</em></li></ul><p>Atlas transforms raw events into structured features.</p><pre>    Kafka Events<br>        │<br>        ▼<br>  Feature Pipeline<br>        │<br>        ▼<br>  Feature Store</pre><p>Instead of every model rebuilding identical calculations, Atlas computes them once.</p><p>Every AI service shares the same foundation.</p><h4>Online vs Offline Features</h4><p>Some features change slowly.</p><pre>Country<br><br>Subscription Type<br><br>Registration Date</pre><p>Others change every second.</p><pre>Current Session<br><br>Items in Cart<br><br>Recent Searches<br><br>Current Device</pre><p>Atlas separates them.</p><pre>Offline Feature Store<br>↓<br>Historical Data<br>──────────────<br>Online Feature Store<br>↓<br>Real-Time Features</pre><p>Prediction requests combine both. Accuracy improves without sacrificing latency.</p><h4>Recommendation Service</h4><p>Suppose a customer opens the application.</p><p>Earlier, Atlas asked:</p><pre>Which campaign is active?</pre><p>Now it asks:</p><pre>Which campaign is most relevant?</pre><pre>Customer<br>↓<br>Recommendation Service<br>↓<br>Top Campaign<br>↓<br>Notification Platform</pre><p>The notification service becomes an execution engine.</p><p>The recommendation engine decides <em>what</em> should be executed.</p><p>Responsibilities remain separated.</p><h4>Real-Time Inference</h4><p>Imagine a customer abandoning their shopping cart.</p><p>Atlas doesn’t wait for tomorrow’s batch process.</p><pre>Cart Event<br>↓<br>Kafka<br>↓<br>Feature Update<br>↓<br>Prediction<br>↓<br>Workflow Decision<br>↓<br>Notification</pre><p>Entire decision time:</p><pre>Less Than<br>300 ms</pre><p>Customer engagement becomes contextual.</p><p>Not scheduled.</p><h4>AI Doesn’t Replace Rules</h4><p>One mistake many platforms make is replacing deterministic business rules with machine learning.</p><p>Atlas combines both.</p><pre>Business Policy<br>↓<br>Recommendation Model<br>↓<br>Policy Validation<br>↓<br>Final Decision</pre><p>Example:</p><p>The recommendation model predicts a promotional campaign.</p><p>Business policy notices the customer opted out of marketing notifications.</p><p>Atlas sends nothing.</p><p>AI suggests. Policies decide.</p><h4>Building a Feedback Loop</h4><p>Intelligent systems improve through feedback.</p><p>Every prediction produces an outcome.</p><pre>Prediction<br>↓<br>Notification Sent<br>↓<br>Opened?<br>↓<br>Purchased?<br>↓<br>Ignored?</pre><p>These outcomes return to the training pipeline.</p><p>Tomorrow’s model learns from today’s behavior.</p><p>The platform continuously improves.</p><h4>Model Versioning</h4><p>Earlier we versioned APIs.</p><p>Models require the same discipline.</p><pre>Recommendation Model<br>Version 1<br>↓<br>Version 2<br>↓<br>Version 3</pre><p>New models begin serving a small percentage of traffic.</p><pre>5%↓<br>25%<br>↓<br>50%<br>↓<br>100%<br></pre><p>Performance is compared continuously.</p><p>If engagement decreases…</p><p>Atlas rolls back the model exactly like software.</p><h4>Feature Drift</h4><p>One production incident surprised us.</p><p>Recommendation accuracy suddenly declined.</p><p>Infrastructure looked healthy.</p><p>Models hadn’t changed.</p><p>The problem wasn’t code.</p><p>It was data.</p><p>Customer behavior had evolved.</p><p>The model continued making decisions based on outdated assumptions.</p><p>Atlas continuously monitors:</p><ul><li><em>Feature distributions</em></li><li><em>Prediction confidence</em></li><li><em>Conversion rates</em></li><li><em>Training-serving consistency</em></li></ul><p>Machine learning systems require operational monitoring just like microservices.</p><h4>AI for Operations</h4><p>Atlas doesn’t only apply AI to customers.</p><p>It also helps engineers.</p><p>Imagine Kafka lag begins increasing.</p><p>Earlier alerts simply stated:</p><pre>Consumer Lag<br>High</pre><p>Atlas now correlates events.</p><pre>Deployment<br>↓<br>Latency Increase<br>↓<br>Consumer Lag<br>↓<br>Retry Spike<br>↓<br>Probable Cause<br>Notification Worker Release 2026.07.18</pre><p>Instead of presenting symptoms, Atlas suggests likely root causes.</p><p>Incident response becomes dramatically faster.</p><h4>Predictive Autoscaling</h4><p>Earlier autoscaling reacted to CPU.</p><pre>CPU<br>85%<br>↓<br>Scale</pre><p>Atlas now predicts demand.</p><p>Using historical traffic patterns:</p><pre>Every Monday<br>09:00<br>Traffic Increases<br>220%</pre><p>Infrastructure scales <em>before</em> traffic arrives.</p><p>Customers never experience resource shortages.</p><p>Prediction replaces reaction.</p><h4>Human-in-the-Loop</h4><p>Not every AI decision should execute automatically.</p><p>Atlas categorizes decisions.</p><pre>Low Risk<br>↓<br>Automatic<br>────────────<br>Medium Risk<br>↓<br>Engineer Review<br>────────────<br>High Risk<br>↓<br>Manual Approval</pre><p>Examples:</p><p>Changing notification timing?</p><p>Automatic.</p><p>Deleting campaigns?</p><p>Manual.</p><p>Blocking customer accounts?</p><p>Always reviewed.</p><p>The goal isn’t removing humans.</p><p>It’s helping humans make better decisions.</p><h4>The Atlas AI Architecture</h4><pre>                Customer Events<br>                       │<br>                       ▼<br>                   Kafka Topics<br>                       │<br>         ┌─────────────┼─────────────┐<br>         ▼             ▼             ▼<br> Feature Pipeline  Analytics     Operations<br>         │<br>         ▼<br>    Feature Store<br>  ┌───────────────┐<br>  │ Online        │<br>  │ Offline       │<br>  └───────────────┘<br>         │<br>         ▼<br> Recommendation Models<br>         │<br>         ▼<br> Policy Engine<br>         │<br>         ▼<br> Workflow Engine<br>         │<br>         ▼<br> Notification Platform</pre><p>Notice something important.</p><p>Artificial intelligence isn’t replacing Atlas.</p><p>It’s becoming another service inside the platform.</p><p>It follows the same architectural principles we’ve used throughout this series:</p><ul><li><em>Loose coupling</em></li><li><em>Event-driven communication</em></li><li><em>Independent deployment</em></li><li><em>Observability</em></li><li><em>Versioning</em></li><li><em>Resilience</em></li></ul><p>AI becomes another production workload.</p><p>Not a special system.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fb55ca6f2049e460670483e5efeee77/href">https://medium.com/media/5fb55ca6f2049e460670483e5efeee77/href</a></iframe><p>The objective isn’t making every decision with AI.</p><p>It’s identifying where prediction genuinely improves business outcomes.</p><h4>Lessons Learned</h4><p>One of the biggest misconceptions about AI is believing it’s fundamentally different from software engineering.</p><p>In production, it isn’t.</p><p>Models require:</p><ul><li><strong><em>Versioning</em></strong></li><li><strong><em>Monitoring</em></strong></li><li><strong><em>Rollbacks</em></strong></li><li><strong><em>Testing</em></strong></li><li><strong><em>Observability</em></strong></li><li><strong><em>Security</em></strong></li><li><strong><em>Deployment pipelines</em></strong></li></ul><p>Exactly like every other service we’ve built in Atlas.</p><p>The real innovation isn’t adding machine learning.</p><p>It’s integrating it into a reliable engineering platform.</p><p>Atlas didn’t become intelligent because we deployed a model.</p><p>It became intelligent because we built the surrounding architecture that allows models to evolve safely, fail gracefully, and improve continuously.</p><p>That’s the difference between an AI demo and an AI platform.</p><h4>The Atlas Series So Far</h4><p>Over fifteen articles, Atlas has grown from a simple notification API into a blueprint for modern distributed systems.</p><p>We’ve explored:</p><ul><li><strong><em>Asynchronous messaging</em></strong></li><li><strong><em>Event-driven architecture</em></strong></li><li><strong><em>Distributed workflows</em></strong></li><li><strong><em>Global scalability</em></strong></li><li><strong><em>Real-time analytics</em></strong></li><li><strong><em>Microservice resilience</em></strong></li><li><strong><em>Distributed transactions</em></strong></li><li><strong><em>Polyglot persistence</em></strong></li><li><strong><em>Traffic engineering</em></strong></li><li><strong><em>Continuous delivery</em></strong></li><li><strong><em>Observability</em></strong></li><li><strong><em>Zero Trust security</em></strong></li><li><strong><em>Cost optimization</em></strong></li><li><strong><em>Platform engineering</em></strong></li><li><strong><em>AI-native architecture</em></strong></li></ul><p>Each chapter solved a production problem.</p><p>Each architectural decision built upon the previous one.</p><p>Together, they represent the evolution that many successful engineering organizations experience over years of operating systems at scale.</p><h4>What’s Next?</h4><p>Atlas has reached an important milestone.</p><p>We’ve explored how to build, scale, secure, observe, optimize, and intelligently operate a modern distributed platform.</p><p>The next evolution isn’t another architectural component.</p><p>It’s a new perspective.</p><p>The next series, <strong>Building Atlas: Disaster Recovery &amp; Planetary Resilience</strong>, will explore the challenges that only appear after your platform becomes truly global:</p><ul><li><strong><em>Multi-region active-active architectures</em></strong></li><li><strong><em>Regional failover without downtime</em></strong></li><li><strong><em>Global data replication</em></strong></li><li><strong><em>Recovery Point Objective (RPO) vs Recovery Time Objective (RTO)</em></strong></li><li><strong><em>Chaos engineering</em></strong></li><li><strong><em>Disaster recovery testing</em></strong></li><li><strong><em>Data sovereignty</em></strong></li><li><strong><em>Cross-region messaging</em></strong></li><li><strong><em>Global consistency</em></strong></li><li><strong><em>Black swan engineering</em></strong></li></ul><p>Because the final test of any distributed system isn’t how it behaves on a normal day.</p><p>It’s how it behaves on the day everything goes wrong.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44859d1d2132" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering at Scale: Platform Engineering, Team Topologies & API Evolution]]></title>
            <link>https://medium.com/@summitkumar/engineering-at-scale-platform-engineering-team-topologies-api-evolution-14412ab683ff?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/14412ab683ff</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 23:41:01 GMT</pubDate>
            <atom:updated>2026-08-02T23:41:01.380Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 14 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“Every engineering organization eventually discovers that software doesn’t become difficult because of code. It becomes difficult because hundreds of engineers are changing that code at the same time.”</strong></blockquote><p>By the end of Part 13, Atlas had become a world-class engineering platform.</p><p>It could:</p><ul><li><strong><em>Process billions of events daily</em></strong></li><li><strong><em>Scale across multiple continents</em></strong></li><li><strong><em>Recover automatically from failures</em></strong></li><li><strong><em>Secure every request</em></strong></li><li><strong><em>Optimize cloud costs</em></strong></li><li><strong><em>Deploy hundreds of releases every week</em></strong></li></ul><p>Technically, Atlas was succeeding.</p><p>Organizationally, it wasn’t.</p><p>Engineering grew from:</p><pre>8 Engineers<br> ↓<br>35 Engineers<br>↓<br>120 Engineers<br>↓<br>300+ Engineers</pre><p>The architecture scaled beautifully.</p><p>The engineering organization didn’t.</p><p>Teams began stepping on each other’s deployments.</p><p>APIs changed unexpectedly.</p><p>Duplicate libraries appeared.</p><p>Every squad solved the same infrastructure problems differently.</p><p>The platform wasn’t becoming harder because of technology.</p><p>It was becoming harder because of coordination.</p><p>That was when Atlas stopped asking:</p><blockquote><strong>“How do we scale our systems?”</strong></blockquote><p>And started asking:</p><blockquote><strong>“How do we scale the people building them?”</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2EfBo-zcZUXlJqbMbR2QOw.png" /><figcaption>Scaling Beyond code</figcaption></figure><h4>Conway’s Law Is Real</h4><p>Melvin Conway made an observation decades ago.</p><blockquote><strong>Organizations design systems that mirror their communication structures.</strong></blockquote><p>We experienced it firsthand.</p><p>Initially, Atlas had one engineering team.</p><pre>Engineering Team<br>↓<br>Atlas Platform</pre><p>Communication was simple.</p><p>Everyone understood every service.</p><p>Then the company expanded.</p><pre>Platform Team<br>Campaign Team<br>Workflow Team<br>Analytics Team<br>Mobile Team<br>Security Team</pre><p>Soon the architecture reflected the organization.</p><pre>Campaign Team<br>↓<br>Campaign Service<br>──────────────<br>Workflow Team<br>↓<br>Workflow Service<br>──────────────<br>Analytics Team<br>↓<br>Analytics Platform</pre><p>The architecture wasn’t just technical anymore.</p><p>It had become organizational.</p><h4>The Problem with Shared Ownership</h4><p>In the early days, every engineer modified every service.</p><p>Need a new API?</p><p>Anyone could change it.</p><p>Need a database migration?</p><p>Anyone could deploy it.</p><p>Initially, this felt productive.</p><p>Later it became chaos.</p><p>Imagine three teams deploying the Notification Service simultaneously.</p><pre>Team A<br>↓<br>Feature<br>──────────────<br>Team B<br>↓<br>Bug Fix<br>──────────────<br>Team C<br>↓<br>Performance Improvement</pre><p>Merge conflicts increased.</p><p>Testing became unpredictable.</p><p>Nobody clearly owned production incidents.</p><p>Atlas needed ownership.</p><p>Not more contributors.</p><h4>Team Topologies</h4><p>Atlas reorganized around business capabilities.</p><p>Instead of technology teams:</p><pre>Java Team<br>Database Team<br>Kafka Team</pre><p>We created stream-aligned teams.</p><pre>Campaign Experience<br>↓<br>Owns<br>Campaign APIs<br>Campaign Database<br>Campaign UI<br>Campaign Deployments</pre><p>Another team:</p><pre>Customer Journey<br>↓<br>Workflow Engine<br>Notification Policies<br>Journey Analytics</pre><p>Every team owned an entire business capability.</p><p>Not just code.</p><p>They owned:</p><ul><li><strong><em>Design</em></strong></li><li><strong><em>Development</em></strong></li><li><strong><em>Operations</em></strong></li><li><strong><em>Reliability</em></strong></li><li><strong><em>Production support</em></strong></li></ul><p>Ownership dramatically improved decision-making.</p><h4>Platform Engineering Changes Everything</h4><p>Even with domain ownership, every team repeated the same work.</p><p>Creating Kubernetes deployments.</p><p>Building CI/CD pipelines.</p><p>Configuring monitoring.</p><p>Managing secrets.</p><p>Implementing authentication.</p><p>Ten teams solved identical infrastructure problems.</p><p>Atlas created a Platform Engineering team.</p><p>Not to own applications.</p><p>To build reusable capabilities.</p><pre>        Platform Team<br>      ┌────────┼────────┐<br>      ▼        ▼        ▼<br> CI/CD     Monitoring   Secrets<br>      ▼        ▼        ▼<br> Internal Developer Platform</pre><p>Application teams stopped worrying about infrastructure.</p><p>They focused on business problems instead.</p><h4>The Internal Developer Platform</h4><p>Think of the platform as a product.</p><p>Its customers are engineers.</p><p>Instead of creating infrastructure manually:</p><pre>kubectl apply ...</pre><p>Developers request capabilities.</p><pre>Create New Service<br>↓<br>Platform<br>↓<br>Provision<br>API<br>Database<br>Monitoring<br>CI/CD<br>Secrets<br>Dashboards</pre><p>A new production-ready service can be created in minutes.</p><p>Not days.</p><p>Developer experience becomes a competitive advantage.</p><h4>Standardization Without Restriction</h4><p>One common mistake is forcing every team to use identical technologies.</p><p>Atlas avoided that.</p><p>Instead, we standardized interfaces.</p><p>Example:</p><p>Every service must expose:</p><pre>/health<br>/metrics<br>/readiness</pre><p>Every service emits:</p><ul><li><strong><em>Structured logs</em></strong></li><li><strong><em>Distributed traces</em></strong></li><li><strong><em>Metrics</em></strong></li></ul><p>How teams implement business logic remains their decision.</p><p>Standards create consistency.</p><p>Not rigidity.</p><h4>API Evolution</h4><p>One of Atlas’ biggest operational problems involved APIs.</p><p>Suppose Version 1 looked like this.</p><pre>{<br>  &quot;campaignName&quot;: &quot;Flash Sale&quot;<br>}</pre><p>Six months later…</p><pre>{<br>  &quot;name&quot;: &quot;Flash Sale&quot;<br>}</pre><p>Looks harmless.</p><p>Except hundreds of consumers now fail.</p><p>Atlas established one rule.</p><blockquote><strong>Never break existing consumers without a migration path.</strong></blockquote><h4>API Versioning</h4><p>Instead of replacing APIs:</p><pre>/api/campaigns</pre><p>Atlas versions them.</p><pre>/v1/campaigns<br><br>/v2/campaigns</pre><p>Older clients continue functioning.</p><p>New applications adopt improved contracts gradually.</p><p>Versioning isn’t elegant.</p><p>But production stability rarely is.</p><h4>Consumer-Driven Contracts</h4><p>Earlier, providers defined APIs independently.</p><p>Consumers discovered breaking changes during deployment.</p><p>Atlas adopted Consumer-Driven Contract Testing.</p><pre>Consumer<br>↓<br>Expected Contract<br>↓<br>Provider Validation<br>↓<br>Deployment Allowed</pre><p>If an API change breaks an existing consumer…</p><p>Deployment stops automatically.</p><p>Production becomes the last place problems appear.</p><h4>Shared Libraries Can Become Shared Problems</h4><p>Initially, every team depended on one giant internal library.</p><pre>atlas-common.jar</pre><p>It contained:</p><ul><li><strong><em>Authentication</em></strong></li><li><strong><em>Logging</em></strong></li><li><strong><em>Utilities</em></strong></li><li><strong><em>Kafka</em></strong></li><li><strong><em>HTTP</em></strong></li><li><strong><em>Configuration</em></strong></li></ul><p>Every release required every service to upgrade.</p><p>One bug affected the entire platform.</p><p>Atlas decomposed shared libraries.</p><pre>Logging SDK<br>Authentication SDK<br>Tracing SDK<br>Metrics SDK</pre><p>Smaller libraries evolved independently.</p><p>Dependency management became dramatically easier.</p><h4>Documentation as Code</h4><p>Knowledge trapped inside Confluence quickly became outdated.</p><p>Atlas moved architecture documentation beside source code.</p><pre>Repository<br>├── README.md<br>├── API.md<br>├── Architecture.md<br>└── Runbook.md</pre><p>Every pull request updates code and documentation together.</p><p>The architecture evolves with the platform.</p><h4>Golden Paths</h4><p>One of the most successful platform initiatives wasn’t introducing more technology.</p><p>It was reducing unnecessary decisions.</p><p>A new engineer creating a service follows a Golden Path.</p><pre>Create Service<br>↓<br>Platform Template<br>↓<br>Production Ready<br>↓<br>Deploy</pre><p>The default path includes:</p><ul><li><strong><em>CI/CD</em></strong></li><li><strong><em>Security</em></strong></li><li><strong><em>Monitoring</em></strong></li><li><strong><em>Logging</em></strong></li><li><strong><em>Tracing</em></strong></li><li><strong><em>Health Checks</em></strong></li><li><strong><em>Deployment Pipeline</em></strong></li></ul><p>Engineers begin with best practices instead of discovering them later.</p><h4>Measuring Developer Experience</h4><p>Earlier, Atlas measured infrastructure.</p><p>Now it measures engineering productivity.</p><p>Examples include:</p><pre>Time to Create New Service<br>18 Minutes<br><br>Lead Time<br>Commit<br>↓<br>Production<br>12 Minutes</pre><pre>Average Production Recovery<br>14 Minutes<br><br>Developer Onboarding<br>2 Days</pre><p>The platform team’s success isn’t measured by uptime.</p><p>It’s measured by how effectively other teams build software.</p><h4>The Atlas Organizational Architecture</h4><pre>              Internal Developer Platform<br>                             │<br>      ┌──────────────────────┼──────────────────────┐<br>      ▼                      ▼                      ▼<br>Campaign Team         Workflow Team         Analytics Team<br>      │                      │                      │<br>      ▼                      ▼                      ▼<br>Campaign Service      Workflow Engine       Analytics Platform<br>      │                      │                      │<br>      └──────────────────────┼──────────────────────┘<br>                             ▼<br>                     Shared Platform Services<br>             CI/CD • Security • Observability • Runtime</pre><p>Notice the shift.</p><p>Infrastructure no longer slows teams down.</p><p>It accelerates them.</p><p>Platform Engineering isn’t another operations team.</p><p>It’s an enablement team.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c78b260ad0ba33fde04d53e5d0c0b64/href">https://medium.com/media/5c78b260ad0ba33fde04d53e5d0c0b64/href</a></iframe><p>The goal isn’t making every engineer work the same way.</p><p>It’s making the safest path the easiest path.</p><h4>Lessons Learned</h4><p>The biggest scaling challenge Atlas faced wasn’t Kafka.</p><p>It wasn’t PostgreSQL.</p><p>It wasn’t Kubernetes.</p><p>It was organizational complexity.</p><p>As engineering organizations grow, architecture and team structure become inseparable.</p><p>The best technical decisions fail if ownership is unclear.</p><p>The best engineers slow down if every team reinvents infrastructure.</p><p>The best platform eventually becomes difficult to change if APIs evolve without discipline.</p><p>Atlas solved these problems by treating developer experience as a first-class engineering problem.</p><p>Internal platforms became products.</p><p>Teams owned outcomes instead of components.</p><p>APIs became contracts.</p><p>Documentation lived beside code.</p><p>Engineering velocity increased not because people worked harder, but because the platform removed friction.</p><h4>Coming in Part 15</h4><p>Atlas has now evolved from a simple notification service into a globally distributed engineering platform.</p><p>We’ve explored architecture, reliability, scalability, observability, security, cost optimization, and organizational design.</p><p>But one frontier remains.</p><p>Artificial Intelligence.</p><p>How can Atlas automatically detect incidents before engineers notice them?</p><p>Can it optimize autoscaling based on historical traffic?</p><p>Can it personalize notification delivery using machine learning?</p><p>Can AI assist engineers during production incidents?</p><p>In <strong>Part 15: Engineering Intelligent Platforms — AI, Machine Learning &amp; Autonomous Operations</strong>, we’ll explore online feature stores, recommendation systems, anomaly detection, predictive autoscaling, AIOps, retrieval-augmented incident response, and how modern engineering platforms are beginning to evolve from reactive systems into self-optimizing systems.</p><p>Because the next generation of distributed systems won’t just scale.</p><p><strong>They’ll learn.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14412ab683ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Cost Efficiency: Scaling a Global Platform Without Scaling the Cloud Bill]]></title>
            <link>https://medium.com/@summitkumar/engineering-cost-efficiency-scaling-a-global-platform-without-scaling-the-cloud-bill-bd1df34ccc53?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd1df34ccc53</guid>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 23:41:01 GMT</pubDate>
            <atom:updated>2026-08-01T23:41:01.519Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 13 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“The most expensive distributed system isn’t the one processing the most traffic. It’s the one wasting the most resources.”</strong></blockquote><p>By the end of Part 12, Atlas had become everything we set out to build.</p><p>It was secure.</p><p>It was resilient.</p><p>It was observable.</p><p>It scaled across continents.</p><p>It processed billions of events every day.</p><p>The engineering team celebrated.</p><p>The finance team didn’t.</p><p>The monthly cloud invoice had doubled.</p><p>Traffic had only increased by 20%.</p><p>That was the first time we realized something every engineering organization eventually learns.</p><blockquote><strong>Scaling infrastructure is easy. Scaling efficiently is engineering.</strong></blockquote><p>This article isn’t about reducing costs by shutting down servers.</p><p>It’s about designing systems where every CPU cycle, every database query, every gigabyte of storage, and every network request has a purpose.</p><p>Because at planetary scale, inefficiency becomes your biggest feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JsNKd3c2EYV0GqrMarygOA.png" /><figcaption>Scaling Cloud Engineering Efficiently</figcaption></figure><h4>The Hidden Cost of Success</h4><p>Imagine Atlas processes:</p><pre>1 Billion Notifications<br>Per Day</pre><p>Every notification generates:</p><ul><li><strong><em>Database writes</em></strong></li><li><strong><em>Kafka messages</em></strong></li><li><strong><em>Redis lookups</em></strong></li><li><strong><em>API calls</em></strong></li><li><strong><em>Metrics</em></strong></li><li><strong><em>Logs</em></strong></li><li><strong><em>Analytics events</em></strong></li></ul><p>One notification may cost only a fraction of a cent.</p><p>One billion notifications change the equation.</p><pre>$0.00001<br>×<br>1,000,000,000<br>=<br>$10,000</pre><p>Now imagine five unnecessary operations for every notification.</p><p>Suddenly…</p><p>You’ve doubled your infrastructure cost without adding any customer value.</p><h4>Cost Is an Architectural Metric</h4><p>Traditional dashboards show:</p><pre>CPU<br>Memory<br>Latency<br>Errors</pre><p>Atlas adds another metric.</p><pre>Cost<br>Per Request</pre><p>Instead of asking:</p><blockquote><strong>“Can this architecture scale?”</strong></blockquote><p>We also ask:</p><blockquote><strong>“How much does every request cost?”</strong></blockquote><p>Engineering decisions become measurable.</p><h4>Right-Sizing Compute</h4><p>Early Atlas workers looked like this.</p><pre>Worker<br>8 vCPU<br>32 GB RAM</pre><p>Monitoring showed:</p><pre>Average CPU<br>18%</pre><pre>Average Memory<br>26%</pre><p>Most resources remained unused.</p><p>Atlas analyzed production metrics.</p><p>The same workload comfortably fit on:</p><pre>Worker<br>2 vCPU<br>8 GB RAM</pre><p>Thousands of instances later…</p><p>The savings became enormous.</p><p>Bigger servers rarely make systems faster.</p><p>They usually make invoices larger.</p><h4>Autoscaling Should Scale Down Too</h4><p>Many teams celebrate autoscaling.</p><p>Pods increase during traffic spikes.</p><p>Fantastic.</p><p>Then nobody notices something important.</p><p>Those Pods never disappear.</p><p>Atlas continuously evaluates demand.</p><pre>Traffic<br>↑<br>Scale Out<br>↓<br>Traffic Drops<br>↓<br>Scale In</pre><p>Resources follow demand.</p><p>Not assumptions.</p><p>Autoscaling without scale-down is simply expensive overprovisioning.</p><h4>Spot Instances</h4><p>Not every workload needs guaranteed infrastructure.</p><p>Atlas classifies workloads.</p><p>Critical:</p><ul><li><strong><em>API Gateway</em></strong></li><li><strong><em>Authentication</em></strong></li><li><strong><em>Payment Processing</em></strong></li></ul><p>Flexible:</p><ul><li><strong><em>Analytics</em></strong></li><li><strong><em>Historical Reports</em></strong></li><li><strong><em>Batch Processing</em></strong></li><li><strong><em>Machine Learning</em></strong></li></ul><p>Flexible workloads run on spot capacity.</p><pre>Critical Workloads<br>↓<br>On-Demand<br>Reliable<br>──────────────<br>Batch Jobs<br>↓<br>Spot Instances<br>Lower Cost</pre><p>If spot capacity disappears…</p><p>Batch processing pauses.</p><p>Customers never notice.</p><p>Business-critical systems remain isolated.</p><h4>Storage Costs Grow Forever</h4><p>One lesson surprised us more than any other.</p><p>Compute costs fluctuate.</p><p>Storage only grows.</p><p>Every day Atlas produced:</p><ul><li><strong><em>Delivery logs</em></strong></li><li><strong><em>Audit events</em></strong></li><li><strong><em>Metrics</em></strong></li><li><strong><em>Workflow history</em></strong></li><li><strong><em>Notification payloads</em></strong></li></ul><p>Keeping everything inside PostgreSQL became financially impossible.</p><p>Atlas introduced lifecycle policies.</p><pre>0–30 Days<br>↓<br>PostgreSQL<br>────────────<br>30–180 Days<br>↓<br>Object Storage<br>────────────<br>Older Than 180 Days<br>↓<br>Archive</pre><p>Frequently accessed data remains fast.</p><p>Historical data remains inexpensive.</p><h4>Cache the Right Things</h4><p>Earlier in the series, we introduced Redis.</p><p>Now we optimize it.</p><p>Poor caching:</p><pre>Every Query<br>↓<br>Redis</pre><p>Good caching:</p><pre>Frequently Used<br>↓<br>Redis<br>────────────<br>Rarely Used<br>↓<br>Database</pre><p>Caching everything wastes memory.</p><p>Caching nothing wastes CPU.</p><p>Finding the balance requires production measurements — not intuition.</p><h4>Compress Before You Transmit</h4><p>Atlas sends millions of API responses every minute.</p><p>Suppose each payload averages:</p><pre>25 KB</pre><p>With compression:</p><pre>6 KB</pre><p>Across billions of requests…</p><p>Network costs decrease dramatically.</p><p>Users also benefit.</p><p>Smaller responses mean faster downloads.</p><p>One optimization serves both engineering and customers.</p><h4>Logging Isn’t Free</h4><p>Earlier we emphasized structured logging.</p><p>But logs are expensive.</p><p>Consider this:</p><pre>One Request<br>↓<br>40 Log Entries</pre><p>Now multiply by:</p><pre>300 Million Requests<br>Per Day</pre><p>Atlas began asking a simple question before writing every log.</p><blockquote><strong>Will anyone actually use this information?</strong></blockquote><p>Instead of logging everything:</p><pre>DEBUG<br>INFO<br>TRACE<br>ERROR</pre><p>Production primarily records:</p><ul><li><strong><em>Errors</em></strong></li><li><strong><em>Business events</em></strong></li><li><strong><em>Security events</em></strong></li><li><strong><em>Operational insights</em></strong></li></ul><p>Verbose logs remain available temporarily during investigations.</p><p>Storage costs fall without reducing visibility.</p><h4>Database Queries</h4><p>One inefficient query rarely matters.</p><p>One inefficient query executed a billion times does.</p><p>Earlier code looked like this.</p><pre>for (Notification notification : notifications) {<br>    userRepository.findById(notification.getUserId());<br>}</pre><p>One database call per notification.</p><p>Classic <strong>N+1 Query Problem.</strong></p><p>Atlas batches requests.</p><pre>List&lt;User&gt; users =<br>    userRepository.findAllById(userIds);</pre><p>One query replaces thousands.</p><p>The fastest query is often the one that never executes.</p><h4>Event Filtering</h4><p>Earlier, every consumer received every event.</p><pre>Kafka<br>↓<br>Every Service<br>↓<br>Every Event</pre><p>Analytics ignored payment events.</p><p>Recommendations ignored infrastructure metrics.</p><p>Workers discarded irrelevant messages.</p><p>Atlas introduced topic specialization.</p><pre>Notification Events<br>↓<br>Notification Consumers<br>──────────────<br>Analytics Events<br>↓<br>Analytics Consumers<br>──────────────<br>Audit Events<br>↓<br>Security Platform</pre><p>Less processing.</p><p>Less networking.</p><p>Lower infrastructure costs.</p><h4>Multi-Level Caching</h4><p>Not every cache belongs in Redis.</p><p>Atlas introduced multiple layers.</p><pre>Application Memory<br>↓<br>Redis<br>↓<br>Database</pre><p>The majority of frequently accessed configuration never leaves application memory.</p><p>Redis handles shared state.</p><p>The database becomes the final source of truth.</p><p>Each layer reduces pressure on the next.</p><h4>Measuring Cost Per Feature</h4><p>Engineering often measures infrastructure.</p><p>Atlas measures business capabilities.</p><p>Example:</p><pre>Campaign Delivery<br>Cost<br>₹0.032<br>Per 1,000 Notifications</pre><pre>Workflow Execution<br>₹0.004<br>Per Customer</pre><p>Now optimization becomes targeted.</p><p>Instead of reducing overall cloud costs blindly, teams optimize the most expensive business workflows.</p><h4>The Atlas Cost Architecture</h4><pre>                  Business Services<br>                         │<br>        ┌────────────────┼────────────────┐<br>        ▼                ▼                ▼<br> Application Cache   Redis Cache   PostgreSQL<br>        │                │                │<br>        └────────────────┼────────────────┘<br>                         ▼<br>                    Kafka Events<br>                         │<br>        ┌────────────────┼────────────────┐<br>        ▼                ▼                ▼<br>  Spot Workers     On-Demand APIs   Analytics<br>                         │<br>                         ▼<br>                 Object Storage<br>                         │<br>                         ▼<br>                  Archive Storage</pre><p>Notice the recurring pattern.</p><p>Every expensive resource now has a cheaper alternative for less critical workloads.</p><p>Engineering stops treating infrastructure as binary.</p><p>It becomes tiered.</p><h4>FinOps Is an Engineering Practice</h4><p>Cost optimization isn’t Finance telling Engineering to spend less.</p><p>It’s Engineering designing systems that deliver more value for every dollar spent.</p><p>Atlas reviews every major architectural proposal using four questions.</p><ul><li><strong><em>Does this improve reliability?</em></strong></li><li><strong><em>Does this improve developer productivity?</em></strong></li><li><strong><em>Does this improve customer experience?</em></strong></li><li><strong><em>Is the operational cost justified?</em></strong></li></ul><p>Cost becomes another engineering quality attribute.</p><p>Just like latency.</p><p>Just like availability.</p><p>Just like security.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ca79d3308fd37dd30f2a3e79fe42aa7/href">https://medium.com/media/5ca79d3308fd37dd30f2a3e79fe42aa7/href</a></iframe><p>The cheapest architecture isn’t the goal.</p><p>The most cost-effective architecture is.</p><h4>Lessons Learned</h4><p>One of the biggest myths in distributed systems is believing that scalability and cost efficiency are opposing goals.</p><p>They’re not.</p><p>Poorly designed systems usually cost more <strong>because</strong> they scale inefficiently.</p><p>Atlas became less expensive by becoming better engineered.</p><p>Fewer unnecessary queries.</p><p>Smarter caching.</p><p>Right-sized infrastructure.</p><p>Purpose-built storage.</p><p>Targeted logging.</p><p>Tiered compute.</p><p>Every optimization removed waste rather than functionality.</p><p>Cloud providers charge for consumption.</p><p>Architecture determines what gets consumed.</p><h4>Coming in Part 14</h4><p>Atlas now scales globally, secures every request, observes every workflow, and optimizes every dollar spent.</p><p>Only one major challenge remains.</p><p>People.</p><p>More than one hundred engineers now contribute to Atlas.</p><p>Multiple teams deploy independently.</p><p>Each team owns different services.</p><p>Each service evolves at its own pace.</p><p>How do organizations prevent architectural chaos as engineering organizations grow?</p><p>How do platform teams enable developers without becoming bottlenecks?</p><p>How do APIs evolve without breaking hundreds of consumers?</p><p>In <strong>Part 14: Engineering at Scale — Platform Engineering, Team Topologies &amp; API Evolution</strong>, we’ll explore domain-driven design, platform teams, internal developer platforms, Backstage, API versioning, consumer-driven contracts, engineering governance, and the organizational architecture required to sustain a platform as it grows from a startup product into an enterprise ecosystem.</p><p>Because eventually…</p><p><strong>The biggest scaling challenge isn’t technology. It’s the engineering organization building it.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd1df34ccc53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Security at Scale: Building a Zero Trust Platform for Millions of Users]]></title>
            <link>https://medium.com/@summitkumar/engineering-security-at-scale-building-a-zero-trust-platform-for-millions-of-users-c9dddafd6058?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9dddafd6058</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 23:41:01 GMT</pubDate>
            <atom:updated>2026-07-31T23:41:01.419Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 12 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote><strong>“The biggest security breach rarely begins with sophisticated hacking. It begins with one service trusting another service too much.”</strong></blockquote><p>Over the last eleven articles, Atlas has evolved into a modern cloud-native platform.</p><p>It can:</p><ul><li><strong><em>Process billions of events every day</em></strong></li><li><strong><em>Scale across multiple regions</em></strong></li><li><strong><em>Recover from service failures</em></strong></li><li><strong><em>Deploy continuously</em></strong></li><li><strong><em>Maintain distributed consistency</em></strong></li><li><strong><em>Observe every request from end to end</em></strong></li></ul><p>From an engineering perspective, Atlas looked complete.</p><p>Then one security audit exposed a dangerous assumption.</p><p>Every service inside the cluster trusted every other service.</p><p>If one service was compromised…</p><p>Every downstream service was also at risk.</p><p>Nothing had been breached.</p><p>But the architecture assumed that anything inside the network was trustworthy.</p><p>That assumption no longer holds in modern distributed systems.</p><p>Welcome to the world of <strong>Zero Trust Architecture</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJGN9192BfwyqMNrOZIp_Q.png" /><figcaption>Engineering Zero Trust at Scale</figcaption></figure><h4>The Castle-and-Moat Problem</h4><p>Traditional enterprise applications relied on a simple security model.</p><pre>    Internet<br>      │<br>    Firewall<br>      │<br>  Trusted Network<br>      │<br>  Application</pre><p>If traffic crossed the firewall, it was considered safe.</p><p>That worked when applications lived on a handful of servers.</p><p>Atlas doesn’t.</p><p>Today the platform spans:</p><ul><li><strong><em>Kubernetes clusters</em></strong></li><li><strong><em>Multiple cloud regions</em></strong></li><li><strong><em>Managed databases</em></strong></li><li><strong><em>Message brokers</em></strong></li><li><strong><em>Third-party APIs</em></strong></li><li><strong><em>Mobile applications</em></strong></li><li><strong><em>Internal developer tools</em></strong></li></ul><p>The “inside” of the network no longer exists.</p><p>Every request must prove its identity.</p><p>Every time.</p><h4>Trust Must Be Earned</h4><p>Consider this request.</p><pre>Workflow Engine<br>↓<br>Notification Service</pre><p>Earlier versions of Atlas accepted it immediately.</p><p>The request originated inside the cluster.</p><p>That was enough.</p><p>Modern Atlas asks additional questions.</p><ul><li><strong><em>Who is calling?</em></strong></li><li><strong><em>Which service identity is being used?</em></strong></li><li><strong><em>Is this service allowed to invoke this API?</em></strong></li><li><strong><em>Has its certificate expired?</em></strong></li><li><strong><em>Has the request been modified?</em></strong></li></ul><p>Identity replaces location.</p><h4>Every Service Gets an Identity</h4><p>Users have identities.</p><p>Services should too.</p><p>Instead of relying on IP addresses, every Atlas service receives a cryptographic identity.</p><pre>Workflow Service<br>↓<br>Identity<br><br>workflow-service.atlas.internal</pre><pre>Notification Service<br>↓<br>Identity<br><br>notification-service.atlas.internal</pre><p>Communication now depends on identity rather than network location.</p><h4>Mutual TLS</h4><p>HTTPS protects communication between clients and servers.</p><p>Atlas extends that concept.</p><p>Every service authenticates every other service.</p><pre>    Workflow Service<br>        │<br>        ▼<br>    Certificate<br>        │<br>        ▼<br>    Notification Service<br>        │<br>        ▼<br>    Certificate<br>        │<br>        ▼<br>    Secure Connection</pre><p>Both sides verify each other’s certificates before exchanging data.</p><p>This is <strong>Mutual TLS (mTLS).</strong></p><p>Even if someone gains network access, they cannot impersonate a legitimate service.</p><h4>API Authentication</h4><p>External users authenticate differently.</p><p>Atlas follows OAuth 2.0 and OpenID Connect.</p><pre>Mobile App<br>↓<br>Login<br>↓<br>Identity Provider<br>↓<br>Access Token<br>↓<br>API Gateway<br>↓<br>Business Services</pre><p>The API Gateway validates every token before forwarding requests.</p><p>Backend services don’t perform authentication repeatedly.</p><p>They trust verified identity propagated by the gateway.</p><h4>JWT Isn’t a Session</h4><p>One of the most common misconceptions is treating JWTs like server sessions.</p><p>A JWT isn’t stored on the server.</p><p>It carries signed claims.</p><p>Example:</p><pre>{<br>  &quot;sub&quot;: &quot;USR-10293&quot;,<br>  &quot;role&quot;: &quot;PREMIUM&quot;,<br>  &quot;scope&quot;: [<br>    &quot;notifications:read&quot;,<br>    &quot;campaigns:create&quot;<br>  ],<br>  &quot;exp&quot;: 1784569200<br>}</pre><p>Atlas never stores authorization decisions inside application code.</p><p>Services evaluate claims.</p><p>Policies determine access.</p><h4>Authorization Is More Than Roles</h4><p>Earlier systems often looked like this.</p><pre>if(user.isAdmin()) {<br>    // allow everything<br>}</pre><p>Real businesses aren’t that simple.</p><p>Can a campaign manager:</p><ul><li><strong><em>Create campaigns?</em></strong></li><li><strong><em>Delete campaigns?</em></strong></li><li><strong><em>View analytics?</em></strong></li><li><strong><em>Access payment reports?</em></strong></li></ul><p>Atlas adopts fine-grained authorization.</p><pre>User<br>↓<br>Role<br>↓<br>Permissions<br>↓<br>Policy Evaluation<br>↓<br>Decision</pre><p>Authorization becomes policy-driven rather than code-driven.</p><h4>Secrets Are Not Configuration</h4><p>Early Atlas deployments contained entries like this.</p><pre>database.password=atlas123</pre><p>Convenient.</p><p>Until someone accidentally committed the configuration file.</p><p>Secrets deserve dedicated management.</p><p>Atlas stores:</p><ul><li><strong><em>Database credentials</em></strong></li><li><strong><em>API keys</em></strong></li><li><strong><em>Encryption keys</em></strong></li><li><strong><em>Third-party tokens</em></strong></li><li><strong><em>Signing certificates</em></strong></li></ul><p>Inside a centralized secrets platform.</p><p>Applications request secrets dynamically at runtime.</p><p>Nothing sensitive lives inside source code.</p><h4>Secret Rotation</h4><p>Imagine Firebase rotates its API credentials.</p><p>Should engineers redeploy every service?</p><p>No.</p><p>Atlas periodically refreshes secrets.</p><pre>Secrets Manager<br>↓<br>New Credential<br>↓<br>Notification Service<br>↓<br>Reload<br>↓<br>Continue Processing</pre><p>No downtime.</p><p>No manual intervention.</p><p>No forgotten credentials.</p><h4>Encrypting Data</h4><p>Not every piece of data requires the same level of protection.</p><p>Atlas classifies information.</p><p>DataEncryptionPublic TemplatesOptionalCampaign MetadataStandardCustomer ProfilesRequiredDevice TokensRequiredPayment ReferencesStrongest Available</p><p>Encryption exists in multiple layers.</p><ul><li><strong><em>Data in transit</em></strong></li><li><strong><em>Data at rest</em></strong></li><li><strong><em>Backup storage</em></strong></li><li><strong><em>Object archives</em></strong></li></ul><p>Security isn’t one feature.</p><p>It’s multiple overlapping protections.</p><h4>Protecting Kafka</h4><p>Earlier articles focused on Kafka throughput.</p><p>Security introduces new requirements.</p><pre>Producer<br>↓<br>Authentication<br>↓<br>Authorization<br>↓<br>Kafka Topic</pre><p>Not every producer can publish everywhere.</p><p>Example:</p><pre>Workflow Engine<br>↓<br>notification-events<br>✓ Allowed</pre><pre>Analytics Service<br>↓<br>payment-events<br>✗ Denied</pre><p>Topic-level permissions prevent accidental or malicious event publishing.</p><h4>Database Security</h4><p>Applications should never connect as database administrators.</p><p>Atlas follows least privilege.</p><pre>Notification Service<br>↓<br>Read<br>Templates<br>↓<br>Write<br>Delivery Records</pre><p>It cannot:</p><ul><li><em>Drop tables</em></li><li><em>Modify user accounts</em></li><li><em>Access payment information</em></li></ul><p>Compromising one service doesn’t expose the entire platform.</p><h4>Audit Logging</h4><p>Every sensitive operation produces an immutable audit record.</p><pre>Campaign Deleted<br>↓<br>Who?<br>↓<br>When?<br>↓<br>From Which IP?<br>↓<br>Approved By?</pre><p>Unlike application logs, audit logs are never edited.</p><p>Never deleted.</p><p>Never rewritten.</p><p>Security investigations depend on trustworthy history.</p><h4>Detecting Suspicious Behavior</h4><p>Atlas continuously evaluates unusual activity.</p><p>Examples include:</p><ul><li><em>Thousands of failed logins</em></li><li><em>Unexpected API usage</em></li><li><em>Excessive token creation</em></li><li><em>Service identity changes</em></li><li><em>Access from unfamiliar regions</em></li></ul><p>Instead of reacting after a breach, Atlas attempts to detect abnormal behavior early.</p><p>Security becomes continuous rather than reactive.</p><h4>The Atlas Security Architecture</h4><pre>                Mobile Application<br>                        │<br>                        ▼<br>                 Identity Provider<br>                        │<br>                 OAuth / OIDC Tokens<br>                        │<br>                        ▼<br>                   API Gateway<br>                        │<br>             Authentication &amp; Policies<br>                        │<br>        ┌───────────────┼────────────────┐<br>        ▼               ▼                ▼<br> Workflow Service Notification Service Analytics<br>        │               │                │<br>        └───────────────┼────────────────┘<br>                        ▼<br>                   Mutual TLS<br>                        │<br>                Service Mesh<br>                        │<br>        ┌───────────────┼────────────────┐<br>        ▼               ▼                ▼<br>    Secrets        Databases        Kafka<br>    Manager      (Least Privilege)  ACLs</pre><p>Every layer verifies identity.</p><p>Every request is authenticated.</p><p>Every permission is explicit.</p><p>Trust is never assumed.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fdc653c9e933c9205ba75ef8c91a972e/href">https://medium.com/media/fdc653c9e933c9205ba75ef8c91a972e/href</a></iframe><p>Strong security always introduces operational complexity.</p><p>The objective isn’t eliminating complexity.</p><p>It’s ensuring the complexity exists in the platform instead of the application code.</p><h3>Lessons Learned</h3><p>One of the biggest misconceptions about security is believing it belongs to a dedicated security team.</p><p>In reality, security is an architectural concern.</p><p>Every API.</p><p>Every database connection.</p><p>Every Kafka producer.</p><p>Every deployment.</p><p>Every service-to-service request.</p><p>Every secret.</p><p>Every permission.</p><p>Each becomes another opportunity either to reduce risk — or introduce it.</p><p>Atlas no longer trusts requests simply because they originate from inside the network.</p><p>Identity is verified continuously.</p><p>Authorization is evaluated explicitly.</p><p>Secrets are managed centrally.</p><p>Privileges remain intentionally small.</p><p>The result isn’t a perfectly secure platform.</p><p>No such platform exists.</p><p>Instead, Atlas becomes a platform where compromising one component no longer compromises everything else.</p><p>That is the real objective of Zero Trust.</p><h4>Coming in Part 13</h4><p>Atlas now delivers notifications globally, scales elastically, survives failures, secures every service, and provides complete production observability.</p><p>But one challenge remains largely invisible until cloud bills arrive.</p><p>Cost.</p><p>How do engineering teams continue scaling without infrastructure costs growing faster than the business?</p><p>Why do some distributed systems process ten times more traffic while spending less?</p><p>How do companies optimize compute, storage, networking, and messaging without sacrificing reliability?</p><p>In <strong>Part 13: Engineering Cost Efficiency — Designing Distributed Systems That Scale Without Breaking the Budget</strong>, we’ll explore autoscaling economics, right-sizing infrastructure, spot instances, storage lifecycle policies, intelligent caching, FinOps principles, and the engineering trade-offs that make cloud-native platforms financially sustainable.</p><p>Because at planetary scale…</p><p><strong>Every unnecessary millisecond, every unused CPU core, and every redundant network request eventually appears on your cloud invoice.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9dddafd6058" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering Observability: Logs, Metrics, Traces & Debugging Production at Scale]]></title>
            <link>https://medium.com/@summitkumar/engineering-observability-logs-metrics-traces-debugging-production-at-scale-0a8f1f513299?source=rss-f795433473dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/0a8f1f513299</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[product-design]]></category>
            <dc:creator><![CDATA[Summit Kumar]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 23:41:01 GMT</pubDate>
            <atom:updated>2026-07-30T23:41:01.554Z</atom:updated>
            <content:encoded><![CDATA[<h4>Building Atlas — Part 11 — <em>Published by Summit Kumar • Senior Software Engineer | Mobile Architect | Distributed Systems Enthusiast</em></h4><blockquote>“<strong>If your production system can fail in a thousand different ways, your observability platform must explain exactly which one happened — and why.”</strong></blockquote><p>By the end of Part 10, Atlas had become an engineering platform capable of processing billions of events while deploying hundreds of software releases every week.</p><p>The platform had matured.</p><p>Traffic spikes no longer frightened us.</p><p>Deployments became routine.</p><p>Distributed transactions were reliable.</p><p>Microservices recovered automatically.</p><p>Then one Monday morning, our monitoring dashboard reported something strange.</p><pre>Notification Success Rate<br>99.98%</pre><p>Everything looked healthy.</p><p>Yet customer support tickets kept increasing.</p><p>Users complained that notifications were arriving <strong>8–10 minutes late</strong>.</p><p>Infrastructure looked normal.</p><p>CPU was fine.</p><p>Memory was fine.</p><p>No service was reporting errors.</p><p>So where was the problem?</p><p>That incident taught us one of the most valuable lessons in distributed systems.</p><blockquote><strong>Healthy infrastructure doesn’t necessarily mean a healthy platform.</strong></blockquote><p>You need visibility into the journey of every request — not just the health of individual servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVeHwhKEvMsp951bKNWXqg.png" /><figcaption>Engineering Observability Concepts and Pillars</figcaption></figure><h4>Monitoring Isn’t Observability</h4><p>Many teams use these terms interchangeably.</p><p>They aren’t the same thing.</p><p>Monitoring answers:</p><blockquote><strong>“Is something wrong?”</strong></blockquote><p>Observability answers:</p><blockquote><strong>“Why is it wrong?”</strong></blockquote><p>Imagine looking at this graph.</p><pre>CPU<br>18%</pre><p>Looks healthy.</p><p>Now another.</p><pre>Memory<br>42%</pre><p>Also healthy.</p><p>Database?</p><pre>Latency<br>22 ms</pre><p>Everything appears normal.</p><p>Yet customers still experience delays.</p><p>Traditional monitoring tells us <strong>what</strong> changed.</p><p>Observability explains <strong>why</strong>.</p><h4>The Three Pillars</h4><p>Atlas organizes observability into three complementary signals.</p><pre>            Observability<br>         ┌────────┼────────┐<br>         ▼        ▼        ▼<br>      Metrics    Logs    Traces</pre><p>Each answers different questions.</p><p>Metrics reveal trends.</p><p>Logs explain events.</p><p>Traces connect everything.</p><p>No single signal tells the complete story.</p><h4>Metrics Tell You Something Changed</h4><p>Every service continuously exports measurements.</p><pre>Requests Per Second<br>18,400</pre><pre>P95 Latency<br>240 ms</pre><pre>Error Rate<br>0.2%</pre><pre>Kafka Consumer Lag<br>920,000</pre><p>Metrics are excellent for dashboards.</p><p>They help answer questions like:</p><ul><li><strong><em>Is latency increasing?</em></strong></li><li><strong><em>Is CPU becoming saturated?</em></strong></li><li><strong><em>Is traffic growing?</em></strong></li><li><strong><em>Are retries increasing?</em></strong></li></ul><p>But metrics cannot explain <em>which request</em> failed.</p><p>For that, we need logs.</p><h4>Structured Logging</h4><p>Earlier versions of Atlas generated logs like this.</p><pre>User login failed</pre><p>Useful?</p><p>Not really.</p><p>Which user?</p><p>Which service?</p><p>Which environment?</p><p>Which request?</p><p>Atlas adopted structured logging.</p><pre>{<br>  &quot;timestamp&quot;: &quot;2026-07-24T10:15:21Z&quot;,<br>  &quot;service&quot;: &quot;notification-worker&quot;,<br>  &quot;campaignId&quot;: &quot;CMP-420&quot;,<br>  &quot;userId&quot;: &quot;USR-87342&quot;,<br>  &quot;traceId&quot;: &quot;abc123&quot;,<br>  &quot;status&quot;: &quot;FAILED&quot;,<br>  &quot;provider&quot;: &quot;Firebase&quot;,<br>  &quot;error&quot;: &quot;429 Too Many Requests&quot;<br>}</pre><p>Now every log entry becomes searchable.</p><p>Support engineers no longer search plain text.</p><p>They filter structured data.</p><h4>Logs Should Tell Stories</h4><p>A useful log answers three questions.</p><pre>What happened?<br>Where did it happen?<br>Why did it happen?</pre><p>Bad logging:</p><pre>Notification failed.</pre><p>Good logging:</p><pre>Notification delivery failed.<br>Campaign: CMP-420<br>Provider: Firebase<br>Reason: HTTP 429<br>Retry Attempt: 3<br>Trace ID: abc123</pre><p>The goal isn’t generating more logs.</p><p>The goal is generating useful logs.</p><h4>Why Traces Changed Everything</h4><p>Suppose one customer reports a problem.</p><p>The request travels through:</p><pre>API Gateway<br>↓<br>Workflow Engine<br>↓<br>Template Service<br>↓<br>Notification Worker<br>↓<br>Firebase<br>↓<br>Analytics</pre><p>Each service logs independently.</p><p>Without tracing, engineers manually correlate dozens of log files.</p><p>Atlas assigns every incoming request a unique Trace ID.</p><pre>Trace ID<br>7a1d9ef2...</pre><p>Every downstream service receives the same identifier.</p><p>Now the entire request becomes visible.</p><h4>Distributed Tracing</h4><p>Imagine opening one trace.</p><pre>Request<br>↓<br>Gateway<br>12 ms<br>↓<br>Workflow<br>31 ms<br>↓<br>Template<br>18 ms<br>↓<br>Notification Worker<br>51 ms<br>↓<br>Firebase<br>940 ms</pre><p>The problem becomes obvious.</p><p>Firebase introduced the delay.</p><p>No guessing.</p><p>No searching.</p><p>No assumptions.</p><p>Distributed tracing transforms debugging from detective work into observation.</p><h4>Measuring Percentiles</h4><p>Average latency is misleading.</p><p>Imagine ten requests.</p><pre>9 Requests<br>100 ms<br>1 Request<br>4 Seconds</pre><p>Average:</p><pre>490 ms</pre><p>Does that represent user experience?</p><p>Not really.</p><p>Atlas measures percentiles.</p><pre>P50<br>92 ms<br>P95<br>240 ms<br>P99<br>3.8 Seconds</pre><p>Percentiles reveal tail latency.</p><p>Production systems are often judged by their slowest requests — not their average ones.</p><h4>RED Metrics</h4><p>Every Atlas service exposes three core metrics.</p><pre>Request Rate<br>Errors<br>Duration</pre><p>Together they answer:</p><ul><li><strong><em>How busy is the service?</em></strong></li><li><strong><em>Is it failing?</em></strong></li><li><strong><em>Is it becoming slower?</em></strong></li></ul><p>Every service implements these consistently.</p><p>Uniform dashboards simplify operations dramatically.</p><h4>USE Metrics</h4><p>Infrastructure follows another model.</p><pre>Utilization<br>Saturation<br>Errors</pre><p>Example:</p><pre>CPU<br>Utilization<br>68%</pre><pre>Thread Pool<br>Saturation<br>94%</pre><pre>Disk<br>Errors<br>0</pre><p>Infrastructure health complements application health.</p><p>Neither replaces the other.</p><h4>Observability Across Events</h4><p>Atlas processes events rather than requests.</p><p>A customer journey might span hours.</p><pre>User Registered<br>↓<br>Welcome Notification<br>↓<br>Notification Opened<br>↓<br>Purchase Completed<br>↓<br>Reward Granted</pre><p>Every event shares the same business correlation identifier.</p><p>Now engineers observe entire customer journeys instead of isolated API calls.</p><h4>Alert Fatigue</h4><p>Earlier, Atlas alerted on everything.</p><p>CPU above 70%.</p><p>Memory above 60%.</p><p>Queue depth above 10,000.</p><p>Developers quickly ignored notifications.</p><p>Atlas redesigned alerts around customer impact.</p><p>Examples include:</p><ul><li><em>Notification delivery latency exceeds five minutes.</em></li><li><em>Payment success rate falls below 99%.</em></li><li><em>Kafka consumer lag grows continuously for ten minutes.</em></li><li><em>P99 latency doubles after deployment.</em></li></ul><p>Alerts became fewer.</p><p>But dramatically more useful.</p><h4>Service Level Objectives</h4><p>Every engineering team needs a definition of “healthy.”</p><p>Atlas defines Service Level Objectives (SLOs).</p><pre>Notification Delivery<br>99.95%<br>Within<br>30 Seconds</pre><pre>API Availability<br>99.99%</pre><pre>Workflow Completion<br>99.9%<br>Within<br>2 Minutes</pre><p>Monitoring infrastructure is useful.</p><p>Monitoring business outcomes is essential.</p><h4>Error Budgets</h4><p>Perfection is impossible.</p><p>Atlas embraces this reality.</p><p>Suppose our availability target is:</p><pre>99.9%</pre><p>That allows approximately:</p><pre>43 Minutes<br>Downtime<br>Per Month</pre><p>This becomes the error budget.</p><p>If too much of that budget is consumed:</p><ul><li><strong><em>Feature releases slow down.</em></strong></li><li><strong><em>Reliability improvements become the priority.</em></strong></li><li><strong><em>Risky deployments pause.</em></strong></li></ul><p>Engineering decisions become driven by data instead of intuition.</p><h4>The Atlas Observability Platform</h4><pre>              Business Services<br>                        │<br>        ┌───────────────┼────────────────┐<br>        ▼               ▼                ▼<br>     Metrics         Structured Logs    Traces<br>        │               │                │<br>        └───────────────┼────────────────┘<br>                        ▼<br>             Observability Platform<br>        ┌───────────────┼────────────────┐<br>        ▼               ▼                ▼<br>    Dashboards      Alert Engine     Search<br>                        │<br>                        ▼<br>                 Engineering Teams</pre><p>Notice something important.</p><p>Every service speaks the same observability language.</p><p>Metrics.</p><p>Logs.</p><p>Traces.</p><p>Together they explain both system health and business health.</p><h4>Engineering Trade-offs</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4ef09d13ef48064b989f71280e3e29e/href">https://medium.com/media/f4ef09d13ef48064b989f71280e3e29e/href</a></iframe><p>Observability isn’t free.</p><p>Neither is debugging production without it.</p><h4>Lessons Learned</h4><p>As Atlas matured, we discovered an important truth.</p><p>Failures rarely happen in isolation.</p><p>A slow downstream provider increases queue depth.</p><p>Growing queues increase latency.</p><p>Higher latency triggers retries.</p><p>Retries amplify traffic.</p><p>Traffic overloads databases.</p><p>Databases slow APIs.</p><p>Without observability, engineers see unrelated symptoms.</p><p>With observability, they see one connected story.</p><p>That’s the real value of logs, metrics, and traces.</p><p>They don’t simply collect data.</p><p>They explain cause and effect across an entire distributed platform.</p><p>Production debugging becomes faster.</p><p>Deployments become safer.</p><p>Customer trust improves.</p><p>And engineering teams spend less time guessing.</p><h4>Coming in Part 12</h4><p>Atlas now scales globally, survives failures, deploys continuously, and provides complete visibility into production.</p><p>But another challenge is beginning to dominate engineering discussions.</p><p>Security.</p><p>How do we authenticate millions of users?</p><p>How do services trust one another?</p><p>How do we rotate secrets without downtime?</p><p>How do we prevent unauthorized access across hundreds of microservices?</p><p>How do we build Zero Trust architectures that remain both secure and developer-friendly?</p><p>In <strong>Part 12: Engineering Security at Scale — Identity, Zero Trust &amp; Secrets Management</strong>, we’ll explore OAuth 2.0, OpenID Connect, JWT validation, mTLS, API security, service identities, secret rotation, policy enforcement, and the security architecture that protects Atlas in production.</p><p>Because the strongest distributed system is still only as secure as its weakest service.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0a8f1f513299" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>