<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gyanaa Vaibhav on Medium]]></title>
        <description><![CDATA[Stories by Gyanaa Vaibhav on Medium]]></description>
        <link>https://medium.com/@gynanrudr0?source=rss-1bc35030d6e3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*iUXsx5m7EdnqKLyd</url>
            <title>Stories by Gyanaa Vaibhav on Medium</title>
            <link>https://medium.com/@gynanrudr0?source=rss-1bc35030d6e3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:01:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gynanrudr0/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I Built a Free IP Geo API, So You Don’t Have to Fight With Another API Key Ever Again]]></title>
            <link>https://medium.com/@gynanrudr0/i-built-a-free-ip-geo-api-so-you-dont-have-to-fight-with-another-api-key-ever-again-2c418bf3ff56?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c418bf3ff56</guid>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[api-development]]></category>
            <category><![CDATA[backend-architecture]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Sun, 18 May 2025 05:46:19 GMT</pubDate>
            <atom:updated>2025-09-09T06:11:57.762Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>No tokens. No vendor lock-in. Just country, city, and speed — at scale.</em></blockquote><p>I didn’t build this to compete.<br>I built it because I was tired of the unnecessary friction.</p><p>@You know the feeling:<br>You’re coding. You just need to get the country from an IP.<br>Maybe the city too. Maybe timezone. Just enough to personalise or log something.</p><p>But then you Google <em>“IP Geolocation API”</em> and hit a wall of signup forms, tokens, 1000 request/month free tiers, or unclear docs.</p><p>It kills your flow.</p><p>So I built my own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7MY5_5G9wMcVDGJFYgQcw.png" /></figure><h3>The Idea Was Simple</h3><p>I wanted something that was:</p><ul><li><strong>Instant</strong> — no keys, no auth, no waiting</li><li><strong>Reliable</strong> — fast enough for real-time apps</li><li><strong>Open</strong> — free to use and inspect</li><li><strong>Production-grade</strong> — something <em>I’d</em> use in my own projects</li></ul><p>That became <a href="https://ipwho.org/"><strong>ipwho.org</strong></a></p><p>A free, blazing-fast IP Geolocation API — built with Node.js, TypeScript, Redis, NGINX, and deployed on a VPS I control.</p><p>No serverless black boxes. No vendor lock-in. No credit cards.</p><h3>What You Get (As a Developer)</h3><p>The API gives you all the essentials:</p><ul><li>/me → info about the current IP</li><li>/:ip → info about a specific IP</li><li>/bulk/:ip1,ip2,... → for bulk lookups</li><li>?get=city,country → if you only need specific fields</li></ul><p>Response is fast. Format is clean. There’s even CORS support by default.</p><p>And best of all: <strong>no signup.</strong></p><h3>How It Works Under the Hood</h3><p>I kept the stack lean but solid:</p><ul><li><strong>Node.js + Express + TypeScript</strong> — backend simplicity with strong typing</li><li><strong>Redis</strong> — for rate limiting and optional caching</li><li><strong>NGINX</strong> — reverse proxy with static caching headers</li><li><strong>MaxMind GeoLite2 + IP2Location Lite</strong> — the open-source geolocation DBs</li><li><strong>PM2</strong> — for clustering and process management</li><li><strong>VPS (4 vCPUs, 16 GB RAM)</strong> — hosted like a real service, not a toy</li></ul><p>I wrote everything from scratch — no bloat, no heavy frameworks.</p><h3>Performance? It’s Not Just “Fine” — It’s Fast.</h3><p>This thing can take a punch:</p><ul><li><strong>22K requests/sec</strong> under local load with 1K concurrent users (tested via autocannon)</li><li>Handles 1.3M+<strong> requests/minute</strong> without blinking</li><li>Latency hovers around <strong>50–100ms average</strong>, depending on region</li></ul><p>Most requests are just reads, so with Redis and Express tuned up, it performs reliably even under pressure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z9FYH8IhrIWHVqIZIon3Zw.png" /></figure><h3>What’s Not Perfect (Yet)</h3><p>Let’s be honest — this isn’t AWS Geo service.</p><ul><li>Some <strong>city names may be missing or vague</strong>, depending on IP range</li><li><strong>Proxy detection isn’t amazing</strong>, because we use the <em>Lite</em> version of IP2Location</li><li>Updates are <strong>manual for now</strong> — I sync new DBs periodically</li><li>No analytics or dashboard (yet)</li></ul><p>But that’s fine — because it’s <strong>good enough to be useful</strong>, and if it gains traction, I’ll keep improving it.</p><p>No startup, no VC — just one dev, real-world needs, and a fast tool that works.</p><h3>Who Is This For?</h3><p>Honestly? It’s for:</p><ul><li>Frontend devs who just need to know where the user is</li><li>Indie hackers building dashboards or logging tools</li><li>Backend devs who want to offer personalised responses without calling an external paid API every time</li><li>Anyone tired of creating yet another account just to see if something works</li></ul><p>Use it. Abuse it. Fork it. Break it.</p><p>It’s open-source under MIT:<br><a href="https://github.com/Gyanaa-Vaibhav/ipwho.org">https://github.com/Gyanaa-Vaibhav/ipwho.org</a></p><h3>What’s Next?</h3><p>If enough people use it, I’ll happily:</p><ul><li>Add <strong>account-based analytics</strong> (with login)</li><li>Fix known issues around proxy and accuracy</li><li>Make it even easier to self-host</li><li>Keep updating the GeoLite and IP2Location DBs regularly</li></ul><p>But for now — it just works.<br>No keys. No tokens. Just IP info, fast.</p><h3>Final Thoughts</h3><p>This isn’t the kind of project that goes viral.</p><p>It’s the kind that powers 100 other projects quietly in the background.</p><p>And that’s exactly what I built it for.</p><p>→ <a href="https://ipwho.org/">ipwho.org</a><br>→ <a href="https://github.com/Gyanaa-Vaibhav/ipwho.org">github.com/Gyanaa-Vaibhav/ipwho.org</a></p><h3>License</h3><p>MIT. Free forever.</p><h3>Who Am I?</h3><p>I’m <strong>Gyanaa Vaibhav</strong>, a developer who started from zero.</p><p>No degree. No fancy resume.<br>Just a relentless obsession with learning how real systems work.</p><p>I’ve gone from building React apps to scaling full-stack projects that can handle millions of users — and I’m just getting started.</p><p>Here’s what defines my work:</p><ul><li>I don’t copy-paste tutorials.</li><li>I don’t ship Firebase toy apps.</li><li>I build everything from scratch — backend, infra, scaling, caching, queues, security, observability — the <em>real</em> stuff.</li></ul><p>My stack?<br><strong>Node.js, TypeScript, PostgreSQL, Redis, WebSockets, and System Design that holds under pressure.</strong></p><p>I post blogs to <strong>teach</strong> and to <strong>document how I think</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c418bf3ff56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering OOP & SOLID: From Messy Scripts to Maintainable Systems]]></title>
            <link>https://medium.com/@gynanrudr0/mastering-oop-solid-from-messy-scripts-to-maintainable-systems-21bc216fa627?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/21bc216fa627</guid>
            <category><![CDATA[solid]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[oop]]></category>
            <category><![CDATA[object-oriented]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Sat, 19 Apr 2025 11:01:24 GMT</pubDate>
            <atom:updated>2025-05-05T13:54:29.346Z</atom:updated>
            <content:encoded><![CDATA[<p>Anyone can write code that works. But what happens when you revisit it 3 weeks later and it feels like someone else wrote it? (Spoiler: it <em>was</em> you.)</p><p>This blog is for developers who are sick of:</p><ul><li>Copy-pasting code instead of reusing it.</li><li>Debugging weird behaviour because “someone changed a variable directly.”</li><li>Writing massive classes that try to do 15 things and suck at all of them.</li></ul><p>Let’s fix that.</p><p>With <strong>OOP</strong> and <strong>SOLID</strong>, you write code like an architect — not just a code monkey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SF-nRonBmdXiNOwIlIgUwg.png" /></figure><h3>Part 1: Object-Oriented Programming (OOP)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30HgOU8dB563XG4DJom7Bg.png" /></figure><h3>1. Encapsulation — Protect What Matters</h3><h4>What Is It?</h4><p>Encapsulation means <strong>hiding the internal data</strong> of an object and exposing only the things that are safe to access. You control how data is accessed and changed.</p><h4>Pain Point’s</h4><p><em>1. </em>“Someone changed one variable and suddenly half the app was broken.”<em><br>2. </em>“I had to add the same validation logic in like 5 places.”<em><br>3. </em>“Why can anyone just update this thing directly? Who approved this?”</p><h4>Why It Matters</h4><ul><li>Devs directly change values like user.age = -10 and break your logic.</li><li>You add validation everywhere manually instead of <em>once</em> inside the class.</li><li>One bug in one variable ruins everything.</li></ul><h4>Real-World Analogy</h4><p>Think of a <strong>vending machine</strong>:</p><p>You push a button. It gives you a soda. You don’t mess with the motor or the power supply.</p><h4>Bad Code</h4><pre>class BankAccount {<br>  constructor() {<br>    this.balance = 0;<br>  }<br>}<br>const account = new BankAccount();<br>account.balance = -999999;</pre><h4>Good Code</h4><pre>class BankAccount {<br>  #balance = 0;<br><br>  deposit(amount) {<br>    if (amount &gt; 0) this.#balance += amount;<br>  }<br><br>  getBalance() {<br>    return this.#balance;<br>  }<br>}</pre><blockquote><em>Encapsulation is less about “hiding secrets” and more about </em><strong><em>guarding logic</em></strong><em>. You decide how your object behaves — not the consumer.</em></blockquote><h3>2. Abstraction — Simplicity Over Complexity</h3><h4>What Is It?</h4><p>Abstraction means showing <strong>only what’s necessary</strong>, and hiding how it works internally.</p><h4>Pain Point’s</h4><ol><li>“Why do I have to write these same 5 steps every time I want coffee?”</li><li>“This function is doing too much, and I have to memorise it all.”</li><li>“I don’t care how it works, I just want the result.”</li></ol><h4>Why It Matters</h4><ul><li>Every function has 5+ manual steps.</li><li>You reuse the same logic in different places, but slightly differently.</li><li>Beginners get overwhelmed trying to understand internal guts they don’t need.</li></ul><h4>Real-World Analogy</h4><p>Using <strong>Google Maps</strong>: You just type the destination. It handles traffic, roads, routes — you don’t need to know all that.</p><h4>Bad Code</h4><pre>coffeeMachine.grind();<br>coffeeMachine.heat();<br>coffeeMachine.extract();</pre><h4>Good Code</h4><pre>coffeeMachine.makeEspresso();</pre><blockquote><em>Abstraction makes code </em><strong><em>simpler to use</em></strong><em>, </em><strong><em>easier to test</em></strong><em>, and </em><strong><em>cleaner to change</em></strong><em>. Consumers don’t need the blueprint — just the results.</em></blockquote><h3>3. Inheritance — Don’t Repeat, Reuse</h3><h4>What Is It?</h4><p>Inheritance lets one class <strong>reuse logic</strong> from another — forming a parent → child relationship.</p><h4>Pain Point’s</h4><ol><li>“I copied the same method into 3 different classes. Again.”</li><li>“Made a small change and now 5 files need updates.”</li><li>“This should be reusable — why am I duplicating everything?”</li></ol><h4>Why It Matters</h4><ul><li>You’re copying/pasting methods between similar classes.</li><li>You make updates in 5 places instead of one.</li><li>Minor changes lead to huge code duplication.</li></ul><h4>Real-World Analogy</h4><p>You inherit your <strong>dad’s jawline</strong> and your <strong>mom’s sarcasm</strong> — but you’re still you.</p><h4>Bad Code</h4><pre>class Dog {<br>  bark() {}<br>  eat() {}<br>}<br><br>class Cat {<br>  meow() {}<br>  eat() {}<br>}</pre><h4>Good Code</h4><pre>class Animal {<br>  eat() {}<br>}<br><br>class Dog extends Animal {<br>  bark() {}<br>}<br><br>class Cat extends Animal {<br>  meow() {}<br>}</pre><blockquote><em>Use inheritance for </em><strong><em>shared behaviour</em></strong><em>, not just for hierarchy. Prefer </em><strong><em>composition</em></strong><em> if it’s not a “is-a” relationship.</em></blockquote><h3>4. Polymorphism — One Method, Many Behaviors</h3><h4>What Is It?</h4><p>Polymorphism means <strong>one interface, multiple implementations</strong>. Call a method without knowing the exact class.</p><h4>Pain Point’s</h4><ol><li>“My code has 5 if/else blocks checking object types. Every time I add a new one, I touch 3 places.”</li><li>“I keep rewriting the same switch case. Feels gross.”</li><li>“This thing should just know how to do its job.”</li></ol><h4>Why It Matters</h4><ul><li>Your functions are full of if/else checking .type.</li><li>Every new class needs changes in 4 places.</li><li>The code becomes tightly coupled and fragile.</li></ul><h4>Real-World Analogy</h4><p>You press “Play” on a <strong>remote</strong> — whether it’s a TV, speaker, or console, it just works.</p><h4>Bad Code</h4><pre>function drawShape(shape) {<br>  if (shape.type === &#39;circle&#39;) {<br>    // draw circle<br>  } else if (shape.type === &#39;rectangle&#39;) {<br>    // draw rectangle<br>  } else if (shape.type === &#39;triangle&#39;) {<br>    // draw triangle<br>  }<br>}</pre><h4>Good Code</h4><pre>class Shape {<br>  draw() {<br>    throw error(&quot;Need to over write the main function&quot;)<br>  }<br>}<br><br>class Circle extends Shape {<br>  draw() {<br>    console.log(&quot;Drawing Circle&quot;);<br>  }<br>}<br><br>class Rectangle extends Shape {<br>  draw() {<br>    console.log(&quot;Drawing Rectangle&quot;);<br>  }<br>}<br><br>function render(shape) {<br>  shape.draw(); // doesn&#39;t care if it’s a circle or rectangle<br>}</pre><blockquote><em>Polymorphism isn’t about being fancy — it’s about being </em><strong><em>safe</em></strong><em>, </em><strong><em>scalable</em></strong><em>, and </em><strong><em>clean</em></strong><em>.</em></blockquote><h3>Part 2: SOLID Principles — Build for Growth</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHL9Z78ZXVcSAacGWkFKyw.png" /></figure><h3>S — Single Responsibility Principle</h3><h4>What Is It?</h4><p>Each class should do <strong>one thing only</strong> — and do it well.</p><h4>Pain Point’s</h4><ol><li>“I tried to change one thing, and broke five others.”</li><li>“This class does way too much. I don’t even know what’s safe to touch.”</li><li>“Writing tests for this is like defusing a bomb.”</li></ol><h4>Why It Matters</h4><ul><li>You touch one method, and the whole class breaks.</li><li>It’s impossible to test specific logic.</li><li>You have “god classes” that handle user creation, emails, logging, and payments.</li></ul><h4>Real-World Analogy</h4><p>A <strong>microwave</strong> shouldn’t also do your taxes.</p><h4>Bad Code</h4><pre>class UserService {<br>  createUser() {}<br>  sendEmail() {}<br>  logActivity() {}<br>}</pre><h4>Good Code</h4><pre>class UserService { createUser() {} }<br>class EmailService { sendEmail() {} }<br>class Logger { logActivity() {} }</pre><blockquote><em>This is the most violated principle by </em><strong><em>beginners</em></strong><em>. And it’s usually where the chaos starts.</em></blockquote><h3>O — Open/Closed Principle</h3><h4>What Is It?</h4><p>Code should be <strong>open for extension</strong>, but <strong>closed for modification</strong>.</p><h4>Pain Point’s</h4><ol><li><em>“</em>Every time I add a new feature, I break the working stuff.”</li><li>“I had to dig through a giant function and add another if block.”</li><li>“Adding one case turns into editing four files.”</li></ol><h4>Why It Matters</h4><ul><li>Every new case forces you to open and change working code.</li><li>More changes = more chances to break things.</li></ul><h4>Real-World Analogy</h4><p>Your <strong>OS gets new apps</strong> — but the OS code isn’t rewritten.</p><h4>Bad Code</h4><pre>class InvoicePrinter {<br>  printInvoice(invoice, type) {<br>    if (type === &#39;PDF&#39;) {<br>      // print as PDF<br>    } else if (type === &#39;HTML&#39;) {<br>      // print as HTML<br>    } else if (type === &#39;CSV&#39;) {<br>      // print as CSV<br>    }<br>  }<br>}</pre><h4>Good Code</h4><pre>class InvoiceFormat {<br>  print(invoice) {}<br>}<br><br>class PDFPrinter extends InvoiceFormat {<br>  print(invoice) {<br>    console.log(&quot;Printing invoice as PDF&quot;);<br>  }<br>}<br><br>class HTMLPrinter extends InvoiceFormat {<br>  print(invoice) {<br>    console.log(&quot;Printing invoice as HTML&quot;);<br>  }<br>}<br><br>function printInvoice(printer, invoice) {<br>  printer.print(invoice);<br>}</pre><blockquote><em>Use inheritance or composition to extend behavior. Leave the core untouched.</em></blockquote><h3>L — Liskov Substitution Principle</h3><h4>What Is It?</h4><p>If class B is a subclass of class A, then objects of type A should be replaceable with objects of type B <strong>without breaking the app</strong>.</p><h4>Pain Point’s</h4><ol><li>“I passed in a child class and it broke in the weirdest way.”</li><li>“It looked like the parent, but it didn’t behave like one.”</li><li>“Why does this subclass exist if it can’t even do the basics?”</li></ol><h4>Why It Matters</h4><ul><li>Subclasses override methods and crash things unexpectedly.</li><li>You assume all children behave like the parent… and they don’t.</li></ul><h4>Real-World Analogy</h4><p>If you order a <strong>car</strong>, and they give you a <strong>boat with wheels</strong>, that’s not substitution — that’s deception.</p><h4>Bad Code</h4><pre>class Bird {<br>  fly() {}<br>}<br><br>class Ostrich extends Bird {<br>  fly() {<br>    throw new Error(&quot;I can’t fly!&quot;);<br>  }<br>}</pre><h4>Good Code</h4><pre>class Bird {<br>  move() {}<br>}<br><br>class Eagle extends Bird {<br>  move() { console.log(&quot;Flying&quot;); }<br><br>class Ostrich extends Bird {<br>  move() { console.log(&quot;Running&quot;); }<br>}</pre><blockquote><em>LSP is about </em><strong><em>respecting contracts</em></strong><em>. Never surprise your code consumers.</em></blockquote><h3>I — Interface Segregation Principle</h3><h4>What Is It?</h4><p>Don’t force classes to implement methods they don’t use.</p><h4>Pain Point’s</h4><ol><li>“I’m throwing NotSupported in half my methods — this feels wrong.”</li><li>“I needed one method and had to import an entire service.”</li><li>“Why am I faking support for stuff I’ll never use?”</li></ol><h4>Why It Matters</h4><ul><li>You’re throwing NotSupported in every other method.</li><li>Importing a service just for one function — dragging in all its junk.</li><li>Devs fake compliance with interfaces and get wrecked in production.</li></ul><h4>Real-World Analogy</h4><p>You buy a <strong>basic toaster</strong> and it has fax, scan, email options… but they don’t work. Dumb, right?</p><h4>Bad Code</h4><pre>class Machine {<br>  print() {}<br>  scan() {}<br>  fax() {}<br>}<br><br>class BasicPrinter extends Machine {<br>  print() {}<br>  scan() { throw &quot;Not supported&quot;; }<br>  fax() { throw &quot;Not supported&quot;; }<br>}</pre><h4>Good Code</h4><pre>class Printable { print() {} }<br>class Scannable { scan() {} }<br><br>class BasicPrinter extends Printable {<br>  print() {}<br>}</pre><blockquote><em>Most devs are noobs (thanks to YouTube crash-course culture).<br>ISP sounds overkill — but in large apps, it saves your sanity.<br>Split logic. Compose with precision. </em><strong><em>Less code. Less confusion.</em></strong></blockquote><h3>D — Dependency Inversion Principle</h3><h4>What Is It?</h4><p>High-level code shouldn’t depend on low-level code. Both should depend on abstractions.</p><h4>Pain Point’s</h4><ol><li>“I can’t test this without spinning up a real DB.”</li><li>“Switching from Mongo to Postgres felt like surgery.”</li><li>“This class is glued to one specific tool — and I hate it.”</li></ol><h4>Why It Matters</h4><ul><li>You can’t test your service without spinning up a real database.</li><li>Changing one lib (like switching Mongo to Postgres) breaks everything.</li></ul><h4>Real-World Analogy</h4><p>Your <strong>coffee machine</strong> shouldn’t be hardwired to one brand of beans.</p><h4>Bad Code</h4><pre>class OrderService {<br>  constructor() {<br>    this.db = new MongoDB();<br>  }<br>}</pre><h4>Good Code</h4><pre>class OrderService {<br>  constructor(database) {<br>    this.db = database;<br>  }<br>}</pre><blockquote><em>DI = write code like </em><strong><em>LEGO blocks</em></strong><em>. Everything should plug in, swap out, and run clean.</em></blockquote><h3>Conclusion</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><p>You can write code that works — <strong>but can you write code that lasts?</strong></p><p>That’s what <strong>OOP and SOLID</strong> are really about.</p><p>They’re not just “best practices” from a textbook.</p><p>They’re the tools that help you build systems that are:</p><ul><li>Easy to <strong>understand</strong></li><li>Easy to <strong>change</strong></li><li>Easy to <strong>test</strong></li><li>And hard to <strong>break</strong></li></ul><p>When you follow these principles:</p><ul><li>You stop fighting your own code.</li><li>You make life easier for every other dev (and future-you).</li><li>You think in <strong>systems</strong>, not just functions.</li></ul><p>Whether you’re building your first side project or scaling something in production, the shift from <strong>“just get it done”</strong> to <strong>“do it right”</strong> starts here.</p><p>So next time you’re about to:</p><ul><li>throw everything into one god-class,</li><li>chain 7 if/else blocks,</li><li>or fake methods just to pass a linter…</li></ul><p>Pause.</p><p>Remember what you’ve learned here. Build smarter.</p><p>Because real devs don’t just write code — <strong>they design it.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21bc216fa627" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Designed a Scalable E-Commerce System (Amazon-Style)]]></title>
            <link>https://medium.com/@gynanrudr0/how-i-designed-a-scalable-e-commerce-system-amazon-style-f9536e8e1ea0?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9536e8e1ea0</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[interview-preparation]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[scalability]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Thu, 17 Apr 2025 08:42:49 GMT</pubDate>
            <atom:updated>2025-04-17T08:42:49.230Z</atom:updated>
            <content:encoded><![CDATA[<p>Designing something like <strong>Amazon</strong> isn’t about throwing servers at the problem. It’s about <strong>smart caching</strong>, <strong>asynchronous flows</strong>, and <strong>real-world tradeoffs</strong> that keep infrastructure lean while serving millions.</p><p>This breakdown shows how I approached building an architecture that supports:</p><ul><li><strong>Millions of users</strong></li><li><strong>Search-heavy product discovery</strong></li><li><strong>Massive media delivery</strong></li><li>And <strong>highly dynamic order pipelines</strong></li></ul><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>1. System Scope &amp; BOTE Estimates</h3><p>Before building, you scope the battlefield.</p><h4>User &amp; Load Assumptions</h4><ul><li><strong>Total Users</strong>: 500M</li><li><strong>DAU</strong>: 30M</li><li><strong>Peak Concurrent Users</strong>: ~2M</li><li><strong>Daily Orders</strong>: ~6M</li><li><strong>Peak Order QPS</strong>: ~10K</li></ul><h4>Storage Estimates</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f91252708f0c8ad2d1a29a390246541/href">https://medium.com/media/3f91252708f0c8ad2d1a29a390246541/href</a></iframe><h4>Bandwidth</h4><ul><li><strong>Read</strong>: 10K QPS × 10 products × 10MB = <strong>~100 MBps</strong></li><li><strong>Monthly</strong>: ~300 TB (offloaded to CDN)</li></ul><h4>Servers</h4><ul><li><strong>Assumption</strong>: 1K QPS/server</li><li><strong>Est.</strong>: ~10–15 app servers (excluding queue handlers/CDNs)</li></ul><h3>2. High-Level Architecture</h3><h4>Browsing / Product Discovery</h4><p>Client → Load Balancer → App Server → DB / Cache → CDN</p><ul><li>Media (images/videos) offloaded to <strong>S3 + CDN</strong></li><li>Redis used for <strong>hot product</strong> caching</li><li>SQL for filtered, paginated product queries</li></ul><h4>Ordering / Post-Purchase Workflow</h4><p>Client → LB → App Server → MQ → Seller Service → DB</p><ul><li>Orders trigger <strong>message queue events</strong></li><li>Sellers get <strong>async updates</strong>, process fulfilment</li><li>Buyer notified on stock issues / shipping status</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*YD6NSa1iPdsl24u9Rx1uOA.png" /></figure><h3>3. Caching Strategy — For the Hot 1%</h3><p>We don’t cache everything — just what <em>moves</em>.</p><h4>What’s Cached:</h4><ul><li>Title, rating, <strong>price</strong>, primary image, <strong>short description</strong></li></ul><h4>Where &amp; How:</h4><ul><li><strong>Redis</strong> (global or regional partition)</li><li><strong>Write-through</strong> strategy (always consistent)</li><li>TTL: 24–48 hrs or <strong>evict by LRU</strong></li></ul><blockquote><em>Why? 20% of search hits go to &lt;1% of products. Cache only what’s worth it.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*aLxvdXNqMVFnPiqfhOBeGw.png" /></figure><h3>4. Cold Product Discovery — SQL FTW</h3><p>No NoSQL here. Cold product searches need <strong>structured</strong>, <strong>filterable</strong>, and <strong>predictable</strong> data access.</p><h4>DB Strategy:</h4><ul><li><strong>Relational DB</strong> (PostgreSQL/MySQL)</li><li><strong>Sharded</strong> by product category or range</li></ul><h4>Indexing:</h4><ul><li><strong>Rating</strong>, <strong>brand</strong>, <strong>category</strong>, <strong>shipping ETA</strong></li><li>Avoid price-based sharding — it changes too often</li></ul><h4>Sponsored Products?</h4><p>Handled by a separate override layer on top of search results.</p><h4>If Query is Vague?</h4><p>Use a fallback <strong>RAG search</strong> (Retrieve → Rerank) to improve results.</p><h3>5. Inventory Consistency (Without Locking Everything)</h3><p>Orders don’t go straight to DB. They go through a <strong>message queue</strong>:</p><ol><li>Order placed → event sent to seller</li><li>Seller service verifies inventory</li><li>If item available → reserve &amp; deduct</li><li>Else → notify buyer &amp; trigger refund flow</li></ol><blockquote><em>Async = no DB locks, no collisions, and no bottlenecks.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oy4RLy2cbZ70hMIRHwnhBQ.png" /></figure><h3>6. Tradeoffs That Actually Work</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d57699c9781b9acb4e92a4f5fe54c196/href">https://medium.com/media/d57699c9781b9acb4e92a4f5fe54c196/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dWzKvF09p4usN_1-zkHdxg.png" /></figure><h3>7. What I’d Add in V2 (Without Burning Money on ElasticSearch)</h3><p>Forget ElasticSearch if the budget’s tight or the scale makes it economically insane.</p><p>Here’s what <strong>you’d actually add instead</strong>:</p><ul><li><strong>RAG Search (Retrieve + Rerank)</strong><br>Use Redis + SQL to pull candidates, then re-rank based on behavior/CTR — light, scalable, affordable.</li><li><strong>Keyword Indexing via Postgres FTS</strong><br>Postgres’ full-text search + trigram indexing = 80% of what Elastic does, at 10% of the cost.</li><li><strong>Pre-computed Search Trees</strong><br>Use category-brand hierarchy trees in Redis for fast drill-downs. Instant filtering without querying all rows.</li><li><strong>Geo-Aware Filtering with SQL Partitions</strong><br>You don’t need “search engines” when your filters are structured and region-aware.</li></ul><blockquote><em>TL;DR: </em><strong><em>Search ≠ ElasticSearch.</em></strong><em> You can go far with smart SQL + Redis + RAG.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*F78alMrN36xdqVCUqnhykw.png" /></figure><h3>8. Final Takeaway</h3><p>Amazon-scale isn’t about micro services everywhere. It’s about:</p><ul><li><strong>Async order processing</strong></li><li><strong>Predictable queries</strong></li><li><strong>Hot cache economics</strong></li><li><strong>CDN offload</strong></li><li>And structured access over hype stacks.</li></ul><p>If you get this right, everything else becomes optimisation.</p><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned &amp; What’s Next</h4><ul><li><strong>Product Catalog Storage at Scale</strong></li><li><strong>Smart Caching for E-commerce Browsing</strong></li><li><strong>Price Filtering, Sharding &amp; Indexing</strong></li><li><strong>Asynchronous Order Processing</strong>.</li><li><strong>Search Optimisation Without ElasticSearch</strong></li></ul><h4>Next Topic in System Design</h4><p>With this Amazon breakdown, we’ve reached the end of this system design blog series.</p><p>What’s Next?</p><p>We shift focus to <strong>Design Patterns</strong>, <strong>OOP</strong>, and <strong>SOLID principles</strong> — the foundation of maintainable code.</p><p>Then, each of these 6 systems will be converted into live, functional projects:</p><ul><li><strong>URL Shortener</strong></li><li><strong>Chat App (Real-time)</strong></li><li><strong>Video Streaming Platform</strong></li><li><strong>Ride-Sharing App</strong></li><li><strong>Twitter Feed Generator</strong></li><li><strong>E-commerce Storefront</strong></li></ul><p>We’ve done the architecture — now we build it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9536e8e1ea0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Designed a Scalable Twitter Feed]]></title>
            <link>https://medium.com/@gynanrudr0/how-i-designed-a-scalable-twitter-feed-5fce10bf7c82?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fce10bf7c82</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Wed, 16 Apr 2025 07:09:51 GMT</pubDate>
            <atom:updated>2025-04-16T13:53:00.940Z</atom:updated>
            <content:encoded><![CDATA[<p>In this breakdown, I’ll walk you through how I approached one of the most classic — but toughest — system design questions out there:</p><blockquote><strong><em>“Design the News Feed System for a Twitter-like platform.”</em></strong></blockquote><p>No fluff. <strong>Real constraints. Real trade-offs. Real engineering.</strong></p><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>What Are We Building?</h3><p>We’re building a <strong>Twitter-style feed</strong> system that must support:</p><ul><li><strong>100M+ total users</strong></li><li>Up to <strong>10K followings</strong> per user</li><li><strong>Short tweets</strong> (text + optional media)</li><li>A <strong>home timeline</strong> showing recent posts from followed users</li><li><strong>Sub-200ms latency</strong></li><li>No WebSockets — just <strong>high-performance read APIs</strong></li></ul><h3>Back-of-the-Envelope (BOTE) Estimations</h3><p>Before diving into design, let’s scope the scale.</p><h4>User Activity:</h4><ul><li><strong>DAU</strong>: 20M</li><li><strong>Tweets per user/day</strong>: 5 → <strong>100M tweets/day</strong></li><li><strong>Avg tweet size</strong>: 300 bytes</li><li><strong>20% tweets with media (~5MB)</strong></li></ul><h4>Storage Needs:</h4><ul><li><strong>Text</strong>: 100M × 300B = ~<strong>30GB/day</strong></li><li><strong>Media</strong>: 20M × 5MB = ~<strong>100TB/day</strong></li><li>Rough estimate: <strong>~365PB over 10 years</strong></li></ul><h4>Traffic Estimate:</h4><ul><li><strong>Read:Write Ratio</strong>: 50:1</li><li><strong>Read QPS</strong>: ~60K</li><li><strong>Write QPS</strong>: ~1.2K</li><li><strong>Peak Read QPS</strong>: 180K</li><li><strong>Peak Write QPS</strong>: 6K</li></ul><h4>Bandwidth:</h4><ul><li>Tweet text: ~<strong>20MBps</strong></li><li>Media served via <strong>CDN</strong> (not backend)</li></ul><h4>Server Load:</h4><ul><li><strong>1K QPS/server</strong> → <strong>60–150 app servers</strong></li></ul><h3>High-Level System Architecture</h3><pre>Client<br>   ↓<br>Load Balancer<br>   ↓<br>App Server<br>   ↓<br>Cache (Redis)<br>   ↓<br>SQL DB</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*YLtCHcOv6q-eTvWvJsaRYw.png" /></figure><h4>Why SQL over NoSQL?</h4><ul><li><strong>Time-ordered tweets</strong> need structured queries</li><li>Relational ops like <strong>likes, retweets, replies</strong> = smoother in SQL</li><li>SQL makes <strong>sharding + indexing</strong> more manageable long-term</li></ul><h3>Caching Strategy: Engagement-First</h3><p>We don’t cache everything — only what matters.</p><h4>Cache Criteria:</h4><ul><li><strong>Only cache tweets</strong> with high engagement (not just follower count)</li><li>Score via <strong>likes, retweets, replies</strong></li><li>Use <strong>TTL</strong> of 2–3 days or <strong>LRU eviction</strong></li><li><strong>Redis</strong> = instant access, ultra-fast reads</li></ul><p>This strategy reduces read cost without flooding memory.</p><h3>Feed Generation Models</h3><h4>Fan-out-on-write</h4><ul><li>Push new tweet to followers’ timelines at post time</li><li>Super-fast reads</li><li>Heavy write load (especially for celebs)</li></ul><h4>Fan-out-on-read</h4><ul><li>Pull recent tweets from followed users at read time</li><li>Cheap writes</li><li>Expensive reads if user follows 10K+ accounts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*wQHVoN1aTNjvsA-0M4-E0A.png" /></figure><h3>Hybrid Feed Strategy (Smart Scaling)</h3><p>To scale, we apply a <strong>hybrid approach</strong>:</p><h4>Logic:</h4><ul><li>Users with <strong>&lt;10K followers</strong> → <strong>Fan-out-on-read</strong></li><li><strong>High-engagement users</strong> → <strong>Fan-out-on-write</strong></li><li>This gives us:<br> — <strong>Efficient writes<br> — Low-latency reads<br> — Scalability across follower graphs</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*4L5vt743egH1oNio-MJjYw.png" /></figure><h3>Timeline Storage Design</h3><p>Let’s talk <strong>database schema</strong> for each model.</p><h4>Fan-out-on-write:</h4><pre>user_feed (<br>  user_id,<br>  tweet_id,<br>  created_at,<br>  engagement_score<br>)</pre><h4>Query:</h4><pre>SELECT tweet_id<br>FROM user_feed<br>WHERE user_id = :user<br>ORDER BY created_at DESC<br>LIMIT 100 OFFSET N;</pre><h4>Fan-out-on-read:</h4><pre>tweets (<br>  tweet_id,<br>  user_id,<br>  content,<br>  created_at<br>)</pre><h4><strong>Query Process:</strong></h4><ul><li>Fetch list of followed_user_ids</li><li>Pull recent tweets with (user_id, created_at) index</li><li>Merge + sort at app level</li></ul><h3>User Behaviour &gt; Engineering Assumptions</h3><p>Most users won’t scroll endlessly.</p><ul><li><strong>Tweet read time</strong>: 2–10 seconds</li><li><strong>Max session</strong>: ~2 hours → ~300–700 tweets/day</li><li>Preload <strong>100 tweets</strong>, lazy-load on scroll</li></ul><p>→ No need to cache or generate 10K+ tweets per timeline.</p><h3>Feed Caching Strategy</h3><h4>Don’t cache all feeds. Instead:</h4><ul><li>Cache <strong>high-frequency active users</strong></li><li>Cache <strong>fan-out-on-write</strong> timelines</li><li>Cache only <strong>top 100–200 tweets</strong> per feed</li></ul><p>The rest gets fetched from DB on scroll. Learn More about CAP <a href="https://medium.com/@gynanrudr0/mastering-distributed-systems-cap-theorem-sharding-consistency-models-7fd74772ad45">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8fkxnVZIkxnypO8dwwGpQ.png" /></figure><h3>CAP Tradeoff: Freshness vs Speed</h3><blockquote><strong><em>“Do we want fast but old tweets — or slow but fresh ones?”</em></strong></blockquote><p>For timeline:<br>✅ <strong>Speed &gt; freshness</strong><br>(We aim for ~5–10 min delay)</p><p>For discover tab:<br>Maybe the opposite — prioritising freshness.</p><h3>Pagination &amp; Timeline Scrolling</h3><ul><li>Tweets are <strong>sorted in reverse chronological order</strong></li><li>Use created_at index</li><li>Paginate via:<br> — <strong>Offset-based</strong> for simplicity<br> — <strong>Keyset-based</strong> for performance (especially on infinite scroll)</li><li>Cache top of timeline, lazy fetch below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_TYzMuXXZeAr51UH7sx1w.png" /></figure><h3>Final Design Summary</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2759b3e839ba7d8e579fca218033f54f/href">https://medium.com/media/2759b3e839ba7d8e579fca218033f54f/href</a></iframe><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned &amp; What’s Next</h4><p>We broke down how to scale a <strong>Twitter-style</strong> feed using real data, caching strategies, and fanout models that actually work in <strong>production</strong>.</p><h4>Next Topic in System Design</h4><p>In the next blog, we’ll break down the backend architecture of a scalable <strong>E-commerce system</strong> — covering inventory management, checkout flows, cart caching, pricing logic, and how to keep your DB sane under millions of transactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fce10bf7c82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Scalable Ride-Sharing System — Like Uber or Lyft]]></title>
            <link>https://medium.com/@gynanrudr0/designing-a-scalable-ride-sharing-system-like-uber-or-lyft-067821e15f03?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/067821e15f03</guid>
            <category><![CDATA[scalable]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[full-stack]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Tue, 15 Apr 2025 13:41:30 GMT</pubDate>
            <atom:updated>2025-04-16T13:53:38.160Z</atom:updated>
            <content:encoded><![CDATA[<h3>Designing a Scalable Ride-Sharing System — Like Uber or Lyft</h3><p><strong>Ride-sharing</strong> apps look simple on the surface — one tap and a ride shows up. But behind that tap? A real-time, distributed backend handling:</p><ul><li>Location filtering</li><li>Driver-user matching</li><li>Surge pricing</li><li>Fault recovery</li><li>All while maintaining <strong>sub-second latency</strong> at scale.</li></ul><p>Here’s how I’d architect a backend system that powers millions of daily rides without over engineering.</p><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>1. System Assumptions &amp; BOTE</h3><p>To build for scale, we need hard numbers.</p><h4>Core Assumptions</h4><ul><li><strong>Total Users</strong>: 100M</li><li><strong>DAU</strong>: 10M</li><li><strong>Peak Concurrent</strong>: 500K</li><li><strong>Rides/day</strong>: 10M</li><li><strong>Avg Ride Duration</strong>: 15 mins</li></ul><h4>Storage</h4><ul><li>25KB/ride → 250GB/day</li><li>~1PB over 10 years</li></ul><h4>QPS</h4><ul><li><strong>10M rides/day</strong> ≈ ~115 QPS average</li><li>Peak QPS (5x): ~400</li></ul><h4>Bandwidth</h4><ul><li>400 QPS × 25KB = ~<strong>10MBps</strong></li><li>≈ <strong>25TB/month outbound</strong></li></ul><h4>Server Estimate</h4><ul><li>500 concurrent users/socket/server</li><li>500K / 500 → ~1000 servers</li></ul><h3>2. High-Level System Architecture</h3><pre>Client<br>  ↓<br>Global Load Balancer<br>  ↓<br>Regional Routing (based on user location)<br>  ↓<br>Regional Load Balancer<br>  ↓<br>Request Handler / Matchmaking Service<br>  ↓<br>Real-Time Message Broker (WebSocket, Redis Pub/Sub)<br>  ↓<br>Database (Ride Logs, Metadata, Billing, etc.)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*XqbmXEu5iYpwF9M2G1k7BQ.png" /></figure><p>Each region (<strong>city</strong>) operates as an isolated zone to reduce latency and ensure fault boundaries.</p><h3>3. Real-Time Ride Flow</h3><ol><li>Rider sends request with <strong>coordinates</strong></li><li>Backend filters nearby drivers using <strong>geo-range</strong></li><li>Pushes request to drivers (<strong>WebSocket</strong> or push service)</li><li>First driver to accept → backend locks the ride</li><li>Sends <strong>confirmation</strong> to rider</li><li>Trip is tracked until <strong>completion</strong></li><li>Data written to DB</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Br4GK6_TumrO1aDRQA-s5A.png" /></figure><h3>4. Race Condition &amp; Driver Matching</h3><ul><li>Closest driver gets request first</li><li>5-second timeout fallback to next in range</li><li>On multiple accepts:<br> — Backend holds request in open state<br> — First successful accept locks ride<br> — Others receive “Ride Taken”</li></ul><p>Lock logic handled via <strong>Redis SETNX</strong> or in-memory flag</p><h3>5. Scalable Surge Pricing</h3><ul><li>Surge calculated via <strong>user:driver ratio per zone</strong></li><li>All ratios + multipliers live in Redis</li><li>Examples:<br> — 2:1 → 1.2x<br> — 5:1 → 1.5x<br> — 10:1 → 2.0x</li><li>Updated every few seconds</li><li>No DB queries — ultra-fast reads + <strong>TTL-based refresh</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*FEJ2oaDMwZeBXGgot7YtBw.png" /></figure><h3>6. Driver Location Tracking</h3><ul><li>No constant GPS streaming</li><li>Pings every 10–15 seconds</li><li>Only <strong>send delta</strong> (coordinate changes)</li><li>Active rides get <strong>route tracking</strong></li><li>Zone stats updated in <strong>Redis</strong></li></ul><h3>7. Mid-Ride Failure Handling</h3><p>If a driver <strong>disconnects</strong>:</p><ul><li>Rider notified instantly (<strong>WebSocket</strong> fallback)</li><li>Backend retries reconnection or triggers <strong>reassignment</strong></li><li>Ride state stored in:<br> — <strong>Redis</strong> (ride:{id}) with TTL<br> — Periodic backup to DB <strong>every 60s</strong></li></ul><h3>8. Ride Accept Idempotency</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OhWjhFiYIZPh7PYLREFuFg.png" /></figure><p>When a rider requests a ride, the backend pushes it to nearby drivers simultaneously. Each driver attempts to accept, but only the <strong>first successful lock in Redis using SETNX</strong> is allowed to proceed. This atomic operation guarantees that <strong>only one driver wins the ride</strong>, while all others receive a “Ride Taken” response. It’s fast, distributed-safe, and prevents double booking — even under high load.</p><ul><li>Every ride request tagged with a unique token</li><li>Redis <strong>SETNX</strong> ensures <strong>only one driver</strong> can lock it</li><li>Others auto-rejected</li><li>Prevents duplicate assignments under load</li></ul><h3>9. Analytics Layer (Optional)</h3><p>Optional add-on layer using:</p><ul><li><strong>Kafka</strong> → Snowflake or ClickHouse</li><li>Tracks:<br> — Zone <strong>wait time</strong><br> — Ride success/cancel rates<br> — <strong>Surge triggers</strong><br> — Driver <strong>rejection %</strong></li></ul><p>Batch writes, not real-time. Cheap, scalable, and insightful.</p><h3>Conclusion</h3><p>We built a backend that:</p><ul><li>Handles <strong>10M+ rides/day</strong></li><li>Scales with 1000+ WebSocket servers</li><li>Computes <strong>regional surge pricing</strong> in-memory</li><li>Guarantees atomic ride locking</li><li>Operates per region for speed + resilience</li></ul><p>This isn’t a micro service mess. It’s real-world engineering — simple, fast, and horizontally scalable.</p><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned &amp; What’s Next</h4><ul><li>Real-time ride flow</li><li>Driver matching &amp; race conditions</li><li>Redis-powered surge pricing</li><li>WebSocket + regional infra</li><li>Ride lifecycle + fault handling</li></ul><h4>Next Topic in System Design</h4><p>In the next blog, we’ll design a <strong>social media backend</strong> — covering follower graph modeling, <strong>newsfeed generation</strong> <strong>strategies</strong>, <strong>real-time updates</strong>, write fanout vs read fanout, caching, and scaling Twitter-like systems for millions of users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=067821e15f03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Scalable Video Streaming Platform — YouTube/Netflix Level Architecture]]></title>
            <link>https://medium.com/@gynanrudr0/designing-a-scalable-video-streaming-platform-youtube-netflix-level-architecture-503cd149641f?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/503cd149641f</guid>
            <category><![CDATA[scalable]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 11:01:55 GMT</pubDate>
            <atom:updated>2025-04-16T13:52:18.034Z</atom:updated>
            <content:encoded><![CDATA[<h3>Designing a Scalable Video Streaming Platform — YouTube/Netflix Level Architecture</h3><p>Building a video streaming platform isn’t just about uploading videos and pressing play. It’s about architecting a global, low-latency, fault-tolerant system that can handle:</p><ul><li><strong>Millions</strong> of daily users</li><li>Massive <strong>concurrent views</strong></li><li>Global CDN delivery</li><li>Adaptive streaming (720p, 1080p, etc.)</li><li>Real-time encoding pipelines</li><li>Watch history, metadata, and monetisation</li><li>Over <strong>500+ PB</strong> of video data with 99.99% durability</li></ul><p>This is a deep breakdown of how to design a scalable video platform like YouTube or Netflix — from raw upload to real-time playback, including edge cases, resilience strategies, and scaling decisions.</p><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>1. Traffic &amp; Storage Estimation (10-Year Horizon)</h3><h4><strong>Video Uploads</strong></h4><ul><li><strong>1 million</strong> uploads per day</li><li><strong>Average</strong> video size: 500MB</li><li>Post-encoding (3 resolutions): ~1.5GB/video<br>→ <strong>Storage needs: ~550PB over 10 years</strong></li></ul><h4><strong>User Consumption</strong></h4><ul><li><strong>50 million</strong> daily active users</li><li><strong>Average</strong>: 6 videos watched per user</li><li>Approx. bitrate: 1.5–3 Mbps<br>→ <strong>Monthly bandwidth: ~30PB outbound</strong></li></ul><h4><strong>System Load</strong></h4><ul><li><strong>Average QPS</strong>: ~3500 (based on DAU and video views)</li><li><strong>Peak QPS</strong>: ~18,000</li><li>With 500 QPS/server → ~36 app servers needed (playback load offloaded to CDN)</li></ul><h3>2. Upload-to-Playback Architecture</h3><pre>Client  <br>  ↓  <br>Upload API Gateway  <br>  ↓  <br>Raw File Buffer / Object Store  <br>  ↓  <br>Encoding Job Queue (Kafka, SQS)  <br>  ↓  <br>Encoding Workers (FFmpeg, AWS MediaConvert)  <br>  ↓  <br>Storage Bucket → CDN Distribution  <br>  ↓  <br>Metadata Service (DB)  <br>  ↓  <br>Playback Client</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*MJIrs9KNZ-kCLZuuZNJ1rg.png" /></figure><h3>3. Upload Flow</h3><ol><li><strong>Client </strong>uploads video in chunks</li><li>API stores video temporarily (e.g., <strong>S3 buffer</strong>)</li><li>Encoding is triggered (after upload or progressively)</li><li>A job is pushed to queue (<strong>Kafka, Redis Stream</strong>)</li><li>Worker encodes video into <strong>multiple formats</strong> (e.g., 360p, 720p, 1080p)</li><li>Encoded segments are distributed to <strong>CDN</strong></li><li><strong>Metadata</strong> is updated (status = live)</li><li>User is notified once video is ready</li></ol><blockquote><em>Original video is kept until encoding is successful. Retries can be triggered without re-upload.</em></blockquote><h3>4. Playback Flow</h3><ol><li>Client requests video playback</li><li>Manifest file (<strong>.m3u8/.mpd</strong>) is returned</li><li>Video player streams segments via <strong>adaptive bitrate</strong></li><li><strong>CDN</strong> edge node serves chunked content</li><li><strong>Metadata</strong> (title, likes, progress) is fetched in parallel</li></ol><h3>5. Metadata Schema (RDBMS)</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4616a0e1cc8d2b9abfacf0bfb7f920f/href">https://medium.com/media/b4616a0e1cc8d2b9abfacf0bfb7f920f/href</a></iframe><h3>6. Adaptive Bitrate Streaming</h3><h4>Each video is:</h4><ul><li>Split into<strong> 2–5 second segments</strong></li><li>Encoded into <strong>multiple resolutions</strong></li><li>Linked via a <strong>manifest file</strong> (HLS or DASH)</li></ul><h4>The player:</h4><ul><li>Fetches the <strong>manifest</strong></li><li>Adjusts resolution in real-time based on bandwidth</li><li>Switches <strong>seamlessly</strong> without buffering</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDjLO1hRJ6_Wlm_bugHXew.png" /></figure><h3>7. CDN Strategy and Caching</h3><ul><li>Segments pushed to CDN edge nodes after encoding</li><li>Geo-aware DNS routes users to the nearest edge</li><li>Trending content is <strong>pre-warmed</strong> to avoid cache stampede</li><li>Use signed URLs or token-based access to prevent link leakage and piracy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gV_prI-0Q3hmTmrg2cMbBg.png" /></figure><h3>8. Fault Tolerance &amp; Resilience</h3><p><strong>Chunk Upload Failures</strong><br>→ Retry only the <strong>failed chunks</strong> (not the entire video)</p><p><strong>Encoding Failures</strong><br>→ Retry using the <strong>original copy</strong>; log job <strong>metadata</strong> for traceability</p><p><strong>CDN Propagation Delays</strong><br>→ Delay playback by marking is_ready = false; fallback to origin if urgent</p><p><strong>Trending Video Load</strong><br>→ Pre-warm multiple CDN regions; replicate content ahead of demand</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IEu4fH-tfKiNgngQaf5YdQ.png" /></figure><h3>9. Storage Strategy (Scaling &amp; Cost Control)</h3><ul><li>Frequently watched videos → <strong>hot tier across CDNs</strong></li><li>Older/inactive videos → moved to cold storage (e.g., <strong>Glacier, Coldline</strong>)</li><li>Unpopular content → only encoded in 360p</li><li>All segments are replicated across availability zones for <strong>durability.</strong></li></ul><blockquote><em>This might </em><strong><em>increase</em></strong><em> Storage form </em><strong><em>550PB</em></strong><em> to </em><strong><em>1.5–2EB </em></strong><em>because we might have to save them in multiple CND’s across regions</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*ZV5d7BGjJ4WMWxZfyisXrA.png" /></figure><h3>10. Advanced Features Built-In</h3><ul><li><strong>Watch history:</strong> Track video position per user</li><li><strong>Likes/comments:</strong> Sharded storage by video_id</li><li><strong>Resume playback:</strong> Debounced writes every few seconds</li><li><strong>Monetization:</strong> Manifest-based ad injection via VAST/VPAID</li><li><strong>Content moderation:</strong> Hash checks, ML classifiers before publish</li><li><strong>Live streaming support:</strong> RTMP ingest → real-time HLS push + chat via WebSockets</li></ul><h3>Conclusion</h3><p>Designing a platform like YouTube or Netflix is about more than scale. It’s about:</p><ul><li><strong>Delivering video</strong> with minimal latency</li><li>Building <strong>fault-tolerant</strong> encoding systems</li><li>Managing global distribution</li><li>Handling <strong>unpredictable traffic</strong></li><li><strong>Balancing quality</strong> with cost over hundreds of PB of data</li></ul><p>When you think in <strong>jobs, queues, manifests, and CDN edges</strong> — you’re not building a video site. You’re building the backbone of modern digital media.</p><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned &amp; What’s Next</h4><ul><li>Upload-to-playback flow</li><li>Encoding pipeline w/ fault tolerance</li><li>CDN strategy</li><li>Adaptive bitrate streaming</li><li>Storage scaling &amp; moderation</li><li>Monetization, live streaming, and more</li></ul><h4>Next Topic in System Design</h4><p>In the next blog, we’ll design a <strong>ride-sharing backend</strong> — covering real-time location tracking, driver-rider matching algorithms, surge pricing, geo-distributed databases, and fault-tolerant trip state management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=503cd149641f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Real-Time Chat App That Actually Scales — No Bullsh*t, Just Systems That Work]]></title>
            <link>https://medium.com/@gynanrudr0/designing-a-real-time-chat-app-that-actually-scales-no-bullsh-t-just-systems-that-work-0f3a2f1a35e8?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f3a2f1a35e8</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[scalable-systems]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Sun, 13 Apr 2025 08:22:01 GMT</pubDate>
            <atom:updated>2025-04-16T13:52:18.886Z</atom:updated>
            <content:encoded><![CDATA[<h3>Designing a Real-Time Chat App That Actually Scales — No Bullsh*t, Just Systems That Work</h3><p>Building a chat app isn’t about slapping a textbox on a UI and calling it “real-time.” If you’re building the next <strong>WhatsApp</strong>, <strong>Discord</strong>, or <strong>Slack</strong>, you’re engineering:</p><ul><li>Persistent socket connections across millions of users</li><li>Fast message delivery under <strong>100ms</strong></li><li><strong>Room-based delivery</strong>, presence sync, retries, offline handling</li><li><strong>Media upload</strong>, distribution, and caching</li><li>All while <strong>not</strong> over engineering with 20 micro services you don’t need</li></ul><p>This is a deep dive into how I’d design a real-time chat system that handles <strong>1M DAU</strong>, <strong>300K concurrent connections</strong>, and doesn’t break under pressure.</p><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>1. Core Features</h3><p>Let’s define what the system <em>must</em> support:</p><ul><li>1:1 and Room-based messaging</li><li>Typing indicators, online/offline presence</li><li>Delivery in &lt;100ms</li><li>Media sharing (images, voice notes, short videos)</li><li>Two delivery modes:<br> — <strong>Persistent</strong> (Slack-style)<br> — <strong>Ephemeral</strong> (WhatsApp-style)</li></ul><h4>Constraints:</h4><ul><li>Room-based chats</li><li>Typing indicators</li><li>Online/offline presence</li><li><strong>10M users / 1M DAU</strong></li><li><strong>300K concurrent users</strong></li><li>Real-time delivery under <strong>100ms</strong></li></ul><h3>2. Traffic &amp; Storage Estimation (BOTE)</h3><blockquote><em>Real system design starts with real numbers.</em></blockquote><h4>Messages:</h4><ul><li>1M DAU × 100 messages/day = <strong>100M messages/day</strong></li><li>200 bytes avg → ~<strong>20GB/day</strong></li><li><strong>~73TB over 10 years</strong></li></ul><h4>Media:</h4><ul><li>10% users × 5 media files/day × 10MB avg = ~<strong>10TB/day</strong></li><li><strong>~37PB over 10 years</strong></li></ul><h4>Load &amp; Bandwidth:</h4><ul><li>Peak QPS (reads + writes + presence): ~<strong>18K/sec</strong></li><li>Bandwidth: ~<strong>12MBps sustained</strong>, ~<strong>30TB/month</strong></li></ul><h3>3. High-Level Architecture</h3><pre>Client (Web/Mobile)<br>    ↓<br>Load Balancer<br>    ↓<br>API Gateway<br>    ↓<br>WebSocket Gateway (Sticky Sessions)<br>    ↓                   ↓<br>Message Service     Presence Service<br>    ↓                   ↓<br>Kafka / Redis Streams   ↔ Redis Cluster (User Map)<br>    ↓<br>Sharded Message DB + Object Storage (S3/CDN)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*FJar6_Ts9bttSUFcdr_9Eg.png" /><figcaption>Scalable Real-Time Chat — System Architecture</figcaption></figure><h3>4. Message Flow: Sender → Receiver</h3><ol><li>User A sends message via WebSocket</li><li>WS server forwards to Message Service</li><li>Message Service:<br> — Saves to DB<br> — Pushes to Kafka stream<br> — Checks Redis for User B’s presence</li><li>If online → routed to User B’s socket</li><li>If offline → stored in DB or buffer</li><li>Ack sent back to sender</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vtzVVT_JMCfVqacB-oJBjQ.png" /><figcaption>Real-Time Message Flow — Sender to Receiver (WebSocket)</figcaption></figure><h3>5. Message Schema Design</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64381e7e7263755eed5664e1501685bc/href">https://medium.com/media/64381e7e7263755eed5664e1501685bc/href</a></iframe><h4><strong>Optional Fields:</strong></h4><ul><li>seen_at (for read receipts)</li><li>thread_id (for replies)</li><li>edited_at (for Slack-style edits)</li></ul><h3>6. Presence Tracking at Scale</h3><ul><li>Each WebSocket server maintains an <strong>in-memory map</strong> of connected users.</li><li>On login/logout → presence is <strong>broadcast via Redis Pub/Sub</strong> to all servers</li><li>Redis cluster keeps a <strong>user_id → server_id</strong> map</li><li>Last seen = async updated to DB every 5–10 mins</li><li>Stateless, fault-tolerant, fast</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPPVg9epkQHLqLQ4HdMtew.png" /><figcaption>Presence Tracking in Scalable Chat — Redis Sync Flow</figcaption></figure><h3>7. Real-Time Delivery (&lt;100ms)</h3><h4>Achieved using:</h4><ul><li><strong>Sticky sessions</strong> via gateway → fast routing</li><li><strong>No DB read</strong> on send → direct fanout to WebSocket</li><li>Kafka/RabbitMQ stream ensures cross-server message sync</li><li><strong>Retry on failure</strong> via Kafka DLQ or Redis</li><li><strong>Heartbeats</strong> to detect dropped connections</li></ul><h3>8. Storage &amp; Sharding Strategy</h3><h4>Messages:</h4><ul><li><strong>Sharded DB</strong> (PostgreSQL or ClickHouse)</li><li>Partitioned tables by room_id or created_at</li><li>Soft deletes for moderation</li></ul><h4>Media:</h4><ul><li>Stored in <strong>S3 or similar</strong> (with signed URLs)</li><li>Delivered via <strong>CDN</strong> (Cloudflare, Akamai)</li></ul><h4>Redis:</h4><ul><li>Sharded for socket presence</li><li>TTL for session data</li><li>Optional: Redis Streams for fallback message queue</li></ul><h3>9. Microservices (If Scaling Beyond MVP)</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3897614e44a4aea4121f27fd0b770bbd/href">https://medium.com/media/3897614e44a4aea4121f27fd0b770bbd/href</a></iframe><p>All services can scale independently behind their own load balancers.</p><h3>10. Reliability &amp; Delivery Guarantees</h3><ul><li>Messages = <strong>UUID + Kafka</strong> = repayable, idempotent</li><li>DLQs catch failed deliveries</li><li>Retry logic = exponential backoff</li><li>ACK-based delivery (Slack/WhatsApp behaviour toggle)</li><li><strong>Redis</strong> fallback for undelivered messages (short-term)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYG8T63WBadxo1sC7f0uuw.png" /><figcaption>Message Delivery Guarantees — Retry &amp; Reliability Flow</figcaption></figure><h3>11. Bonus Features Layer (Optional Enhancements)</h3><ul><li>Typing indicators via Pub/Sub</li><li>Push notifications (mobile fallback)</li><li>Reactions schema (emoji, likes)</li><li>E2EE (per-room or per-message)</li><li>API rate limiting &amp; abuse control</li></ul><h3>Conclusion: Real Systems Survive Chaos</h3><p>A chat app at scale is a real-time <strong>distributed system</strong>, not just a WebSocket demo. You’re managing:</p><ul><li>Persistent low-latency delivery</li><li>Offline resilience</li><li>Failovers and retries</li><li>Media storage</li><li>UX features like presence &amp; typing</li></ul><p>You don’t need 10 micro services to build it. Start monolithic. Split only when needed. Redis goes down? Retry. Kafka fails? Rebuffer. User drops offline? Store and resend.</p><p>The mission?<br><strong>Messages delivered in under 100ms — no matter what.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXcYlz7tPAAIVs0TVejN_Q.png" /><figcaption>Slack vs WhatsApp — Message Delivery Model Comparison</figcaption></figure><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned</h4><ul><li>Real-time WebSocket architecture</li><li>Message queues and delivery sync</li><li>Presence handling across servers</li><li>Slack vs WhatsApp model design</li></ul><h4>Next in System Design</h4><p>In the next blog, we’ll design a <strong>video streaming backend</strong> — covering CDN delivery, adaptive bitrate, upload flow, and real-time watch tracking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f3a2f1a35e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Scalable URL Shortener — No Microservices, Just Real Engineering]]></title>
            <link>https://medium.com/@gynanrudr0/designing-a-scalable-url-shortener-no-microservices-just-real-engineering-25ee81447b64?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/25ee81447b64</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 10:54:05 GMT</pubDate>
            <atom:updated>2025-04-16T13:51:34.008Z</atom:updated>
            <content:encoded><![CDATA[<h3>Designing a Scalable URL Shortener — No Microservices, Just Real Engineering</h3><p>System design interviews often become a race to say “microservices,” “Kafka,” and “S3” as fast as possible — even when you’re designing a <strong>simple URL shortener</strong>.</p><p>But here’s the truth:<br>You <strong>don’t need 15 services</strong> to shorten and redirect a URL.</p><p>In this blog, I’ll walk you through how I approached a real interview prompt for building a scalable URL shortener like Bitly — and why I chose a <strong>horizontal monolith + Redis + message queues</strong> setup that balances <strong>performance, cost, and simplicity</strong>.</p><blockquote><em>This is a high-level system design breakdown written as an interview simulation — not a complete implementation. Code version coming soon as part of my real-world project portfolio.</em></blockquote><h3>Step 1: Clarify the Requirements (BOTE Estimation)</h3><p>Before designing anything, let’s ask the right questions.</p><h4>System Goals:</h4><ul><li>Create short URLs (6–8 characters)</li><li>Redirect instantly from short to long URL</li><li>High availability, low latency</li><li>Track clicks and analytics (bonus)</li></ul><h4>Scale Assumptions:</h4><ul><li>500M total users</li><li>50M DAU (Daily Active Users)</li><li>20M short URLs created/day</li><li>~1B redirections/day</li></ul><h3>Step 2: Back-of-the-Envelope Estimation</h3><h4>Storage</h4><ul><li>Each short URL entry ≈ 8 bytes</li><li>20M/day → ~160MB/day → ~60GB/year → ~300GB/5yr</li><li>Storage is <strong>not</strong> a bottleneck</li></ul><h4>Peak Throughput</h4><ul><li>1B redirects/day ≈ ~12K QPS average</li><li>Peak QPS ≈ <strong>60K</strong></li><li>Read:Write = <strong>50:1</strong></li></ul><h4>Server Estimation</h4><ul><li>Assuming 2K QPS/server → Need ~30 servers at peak</li></ul><blockquote><em>Takeaway: This isn’t small — but it’s not FAANG-scale either. Keep it simple unless proven otherwise.</em></blockquote><h3>Step 3: High-Level System Design</h3><pre>Client<br>  ↓<br>Load Balancer<br>  ↓<br>App Server (Monolith)<br>  ↓         ↓<br>Redis Cache        SQL DB<br>       ↕<br>   Message Queue (optional)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*yAOyze3WC4eI7JNPYhj29A.png" /></figure><h3>Step 4: URL Generation Logic</h3><h4>Base62 Encoding (Preferred)</h4><ul><li>Auto-incrementing DB ID → Base62</li><li>ID: 125 → Short Code: gy.in/cb</li></ul><h4>Alternative: Hash(long_url)</h4><ul><li>Stateless</li><li>Must check for collisions in DB</li><li>Add salt to retry</li></ul><h4>Dedupe Option:</h4><ul><li>Either store hash(long_url) to reuse existing</li><li>Or allow same URL to be shortened multiple times (Bitly does this)</li></ul><h3>Step 5: Redis Caching Strategy</h3><h4>Why Redis?</h4><ul><li>90% of the load is reads (GET /short-code)</li><li>Redis offers &lt;10ms latency</li></ul><h4>What to Cache:</h4><ul><li><strong>New links</strong>: Write-through cached for 1 week</li><li><strong>Hot links</strong>: If clicks/day &gt; 1K, keep in Redis permanently</li></ul><h4>Eviction:</h4><ul><li>LRU + TTL</li><li>Redis ~1.25GB = 140M links cached = good buffer</li></ul><blockquote><em>Redis doubles as a “logical shard layer” — we don’t need to shard the DB until it hurts.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgKbNDpK5T3Mnj9A5VXz2A.png" /></figure><h3>Step 6: Solving the Real Scaling Problem</h3><h4>LB Write-Skew Problem:</h4><ul><li>User creates a link on Server A</li><li>LB sends next redirect to Server B — which hasn’t synced yet</li></ul><h4>Solution:</h4><ul><li><strong>Message Queue</strong> to sync all write events across servers</li><li><strong>Write-through cache</strong> ensures recent links are available instantly</li><li><strong>Daily audit</strong> to re-sync all active short codes</li></ul><h4>Bonus Infra:</h4><ul><li>Use VPS clones → DB + Code embedded</li><li>Expand capacity with &lt;1 hour provisioning</li><li>Monolith duplicated across boxes → simple scaling</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7IaNDGWWaUpvGmYazg0m8w.png" /></figure><h3>Step 7: Database &amp; Schema Design</h3><h4>SQL &gt; NoSQL (Why?)</h4><ul><li>Relational schema = cleaner for analytics, reporting, joins</li><li>Structured consistency for high-read workloads</li></ul><h4>Schema</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7e5b77629b7cbcbc9dcc372d6099ec5/href">https://medium.com/media/d7e5b77629b7cbcbc9dcc372d6099ec5/href</a></iframe><blockquote><em>Can extend to user_id, custom_alias, link_type, etc.</em></blockquote><h3>Step 8: Why I Didn’t Use Microservices</h3><blockquote><em>“System design isn’t about breaking stuff into microservices. It’s about not breaking what already works.”</em></blockquote><h4>No Need for Over-Engineering:</h4><ul><li>Monolith handles 100M scale if cloned horizontally</li><li>Redis covers latency and heat</li><li>Message Queue handles sync</li><li>DB scaling is postponed until real pain hits</li></ul><p>You <strong>don’t need 10 services</strong> to shorten a URL. You just need a clean pipe, a fast cache, and awareness of scale.</p><h3>Step 9: What I’d Add in Version 2</h3><ul><li>Vanity URLs (e.g., gy.in/my-product)</li><li>Expiry and TTL-based clean-up</li><li>Rate limiting + abuse detection</li><li>Analytics dashboard</li><li>Multi-region fallback cache (if scale requires it)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gvBoQ2JCvvjFV0nqjiXf_g.png" /></figure><h3>Conclusion</h3><p>This isn’t a tutorial. This is a <strong>production-first, scale-conscious system design </strong>for a real-world use case. You’re not just building a feature — you’re designing the foundation for billions of requests and millions of users.</p><p>Whether it’s for an interview or your next backend challenge, remember:</p><ul><li><strong>Start with numbers</strong></li><li><strong>Keep what works</strong></li><li><strong>Scale only what hurts</strong></li></ul><h3>Up Next in System Design</h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through this blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><blockquote><em>Each system in this series will be converted into live code with backend, database, and infrastructure in the coming weeks. </em><strong><em>Follow</em></strong><em> for updates.</em></blockquote><h4>What We Learned</h4><ul><li>Scale is about <strong>reads, writes, latency, sync</strong></li><li>Redis is a <strong>first-class citizen</strong> in scale systems</li><li>Don’t blindly split things into microservices</li></ul><h4>Next Topic in System Design</h4><p>In the next blog, We’ll build a <strong>room-based chat backend</strong>, handle WebSocket scaling, user presence, Redis sync, and deal with real-time pressure.</p><p>Stay tuned — and if you liked this breakdown, share it with a <strong>real engineer</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25ee81447b64" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logging, Monitoring & Observability — The Missing Pillars of Scalable System Design]]></title>
            <link>https://medium.com/@gynanrudr0/logging-monitoring-observability-in-distributed-systems-e354a047ea51?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e354a047ea51</guid>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[logging-and-monitoring]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Fri, 04 Apr 2025 10:47:26 GMT</pubDate>
            <atom:updated>2025-04-11T16:20:35.720Z</atom:updated>
            <content:encoded><![CDATA[<p>So far, we’ve explored building scalable systems through techniques like <a href="https://medium.com/@gynanrudr0/scaling-databases-indexing-partitioning-and-sharding-560c44b08760"><strong>sharding</strong></a>, <a href="https://medium.com/@gynanrudr0/building-scalable-systems-with-message-queues-event-driven-architecture-56dfd2e8ea7a"><strong>event-driven architectures</strong></a>, <a href="https://medium.com/@gynanrudr0/scaling-systems-load-balancing-explained-6f7a1922b07a"><strong>load balancing</strong></a>, and more. But here’s the hard truth: <strong>none of it matters if you don’t have visibility into what’s actually happening inside your system.</strong></p><p>In distributed systems, your services live across multiple machines, often communicating asynchronously. Without proper visibility, you’ll only discover problems after frustrated users complain — or worse, after silent failures have already impacted your business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1DAMHRokbzPDEbKvNDffQ.png" /></figure><p>This is exactly why Logging, Monitoring, and Observability are essential. Together, they help you answer three crucial questions clearly and effectively:</p><ul><li><strong>What happened?</strong> <em>(Logging)</em></li><li><strong>Is the system healthy right now?</strong> <em>(Monitoring)</em></li><li><strong>Why did something break, and how do I fix it?</strong> <em>(Observability)</em></li></ul><p>Let’s dive into each of these layers and see how they help you build robust, transparent, and easily maintainable distributed systems.</p><h3><strong>Logging — What Happened?</strong></h3><p>Logging is your <strong>first line of defense</strong> in understanding system behavior. It records everything your system does — from user interactions and successful transactions to errors and crashes. Without good logs, troubleshooting is nearly impossible.</p><p><strong>Examples of Effective Logs:</strong></p><ul><li><strong>User Activity</strong>: User #123 logged in at 09:12 AM</li><li><strong>Error Details</strong>: Payment failed: Card declined (User #456)</li><li><strong>System Failures</strong>: Room 32 crashed due to null message exception</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGF1hX96s0zIv6dcXPZorg.png" /><figcaption><strong>Logging Flow in a Distributed System</strong></figcaption></figure><h4><strong>Log Levels Explained</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85021d8d804fe4db022b8baab39bc74b/href">https://medium.com/media/85021d8d804fe4db022b8baab39bc74b/href</a></iframe><blockquote><em>⚠️ </em><strong><em>Risk</em></strong><em>: Too many DEBUG logs can bury critical issues. Use log levels wisely!</em></blockquote><h4><strong>Recommended Logging Tools</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b962b8ef6d3c8383b7d5a06f08092a7/href">https://medium.com/media/9b962b8ef6d3c8383b7d5a06f08092a7/href</a></iframe><h3><strong>Monitoring — Is the System Healthy?</strong></h3><p>While <strong>logging</strong> gives you a record of events, <strong>monitoring</strong> tells you how your system is performing in real-time. It keeps track of important metrics, providing immediate alerts when something goes wrong or performance dips.</p><p>It tracks metrics like:</p><ul><li>🧠 CPU usage</li><li>⏱ Response times</li><li>📉 Drop in messages sent</li><li>🔁 Request counts</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*O4sFzB9wORifLgMKj12pRw.png" /><figcaption><strong>System Monitoring Dashboard</strong></figcaption></figure><h4><strong>Common Metrics You Should Monitor</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/451f0dc63f15be0ff3cd2e639f33ce19/href">https://medium.com/media/451f0dc63f15be0ff3cd2e639f33ce19/href</a></iframe><h4><strong>Recommended Monitoring Tools</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/110362f81dbd57dedb3708fc080b83be/href">https://medium.com/media/110362f81dbd57dedb3708fc080b83be/href</a></iframe><h3><strong>Observability — Why Did It Break?</strong></h3><p>Logging and monitoring help you know <strong>what</strong> happened and <strong>when</strong>, but Observability helps you understand <strong>why </strong>something broke — especially important in complex, distributed environments.</p><p>In distributed systems (like microservices or event-driven architectures), a single user request often spans multiple services. Without Observability, tracking down failures can feel like searching for a needle in a haystack.</p><p><strong>Key Concepts in Observability</strong></p><ul><li><strong>Distributed Tracing</strong>: Track a request as it moves across multiple services, pinpointing exactly where failures occur.</li><li><strong>Correlation IDs</strong>: Assign a unique identifier to each request, enabling easy tracing end-to-end.</li><li><strong>Contextual Logs</strong>: Enhance logs by including additional context (user IDs, session details) rather than generic error messages.</li></ul><h4><strong>Example — Observability in Action (Chat App)</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50d57330b097ea2ae580a86cac98e7f6/href">https://medium.com/media/50d57330b097ea2ae580a86cac98e7f6/href</a></iframe><h4><strong>Recommended Observability Tools</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b23887bfb0750c10d0237dbe22716198/href">https://medium.com/media/b23887bfb0750c10d0237dbe22716198/href</a></iframe><h3><strong>System Architecture Overview</strong></h3><p>Here’s how Logging, Monitoring, and Observability integrate into a typical distributed system architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*olPsvBnbP7un22sFchoKvA.png" /></figure><pre>Client<br>  │<br>  ▼<br>API Gateway<br>  └─ Injects Correlation ID<br>  │<br>  ▼<br>Service A<br>  └─ Logs events<br>  └─ Sends requests to Service B<br>  │<br>  ▼<br>Service B<br>  └─ Logs relevant details<br>  └─ Writes data to Database<br>  │<br>  ▼<br>Observability &amp; Monitoring Tools<br>  └─ Logs → ELK Stack (Centralized logs)<br>  └─ Metrics → Prometheus + Grafana (Dashboards &amp; Alerts)<br>  └─ Traces → Jaeger/OpenTelemetry (End-to-End Request Tracing)</pre><p><strong>Key Points:</strong></p><ul><li><strong>Correlation IDs</strong> injected by the API Gateway make tracing requests easier.</li><li>Logs from each service are collected centrally for fast search and debugging.</li><li>Monitoring and tracing tools provide dashboards and alerts, giving immediate visibility into system health.</li></ul><h3><strong>Conclusion</strong></h3><p>Without visibility, your distributed system becomes a <strong>black box</strong>, leaving you blind to problems until users complain. <strong>Logging, Monitoring, and Observability</strong> aren’t optional — they’re essential for survival.</p><ul><li><strong>Logging</strong> helps you see <strong>what happened</strong> inside your system.<br><em>(Example: “Payment failed: Card declined”)</em></li><li><strong>Monitoring</strong> tells you if your system is <strong>healthy right now</strong>.<br><em>(Example: “API response times increased significantly”)</em></li><li><strong>Observability</strong> helps identify <strong>why</strong> something broke, tracing failures across multiple services.<br><em>(Example: “Database latency caused by inefficient query found through tracing”)</em></li></ul><h3><strong>Up Next in System Design</strong></h3><p>Hey there! I’m Gyanaa Vaibhav , a software engineer deeply passionate about <strong>System Design, Scalability, and High-Performance Architectures.</strong> Through these blogs, I’ll be sharing <strong>deep insights, real-world case studies, and expert breakdowns</strong> of key system design concepts.</p><h4><strong>What We Learned &amp; What’s Next</strong></h4><ul><li><strong>Logs tell you what happened</strong>, but may not show root cause.</li><li><strong>Monitoring gives live signals</strong> about performance.</li><li><strong>Observability connects the dots across services.</strong></li></ul><h4><strong>Next Topic in System Design</strong></h4><p><strong>Congratulations!</strong> You’ve reached the end of the <strong>system design learning phase</strong>. You’ve gained solid foundational knowledge on building robust, scalable, and maintainable distributed systems.</p><p>Now, it’s time to put these concepts into action. Next, we’ll enter the <strong>implementation phase</strong>, starting by building a real-world <strong>URL Shortener</strong> system. You’ll see firsthand how to apply design principles practically — from concept to working code.</p><p>Stay tuned — there’s much more exciting content ahead!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e354a047ea51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Distributed Systems: CAP Theorem, Sharding & Consistency Models]]></title>
            <link>https://medium.com/@gynanrudr0/mastering-distributed-systems-cap-theorem-sharding-consistency-models-7fd74772ad45?source=rss-1bc35030d6e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fd74772ad45</guid>
            <category><![CDATA[scalibility]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Gyanaa Vaibhav]]></dc:creator>
            <pubDate>Thu, 27 Mar 2025 09:44:39 GMT</pubDate>
            <atom:updated>2025-05-01T03:19:01.482Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous blog, we covered <a href="https://medium.com/@gynanrudr0/powering-real-time-systems-with-websockets-ab7f34a1ee89"><strong>WebSockets &amp; Real-Time Systems</strong></a>, enabling live, persistent data flow. But what happens when systems get massive — millions of users, global traffic, and servers spread across continents?</p><p>This is where <strong>Distributed Systems</strong> come in. They allow applications to scale across multiple servers while appearing like one unified system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q341y53tC8jMSFNRZLn8WA.png" /></figure><p>In this blog, we’ll explore:</p><ul><li><strong>Distributed Systems</strong> — What is it and why does it matter?</li><li><strong>CAP Theorem</strong> — Why you can’t have it all</li><li><strong>Consistency Models</strong> — How systems decide what data to show</li><li><strong>Sharding</strong> — How to split data for massive scale</li></ul><h3><strong>What Is a Distributed System?</strong></h3><p>A <strong>Distributed System</strong> is a collection of separate components or services running on multiple machines that collaborate and behave like a single unified system.</p><h4><strong>Simplified Categories</strong></h4><p>Distributed systems can typically be grouped into <strong>three clear types</strong>:</p><h4><strong>1. Client-Server Systems (Monolithic)</strong></h4><ul><li><strong>Structure</strong>: Single, unified backend application deployed across multiple servers.</li><li><strong>Scaling</strong>: Often uses load balancers for horizontal scalability (multiple identical servers).</li></ul><p><strong>Key Characteristics</strong>:<br> — <strong>Monolithic codebase</strong> (one big application).<br> — Not inherently designed for independent scalability of components.<br> — If the application code starts to split significantly, it’s transitioning towards <strong>microservices</strong>, not purely monolithic anymore.</p><p><strong>Example</strong>: Traditional web applications (e.g., a classic PHP/MySQL app running behind a load balancer).</p><h4><strong>2. Peer-to-Peer (P2P) Systems</strong></h4><p><strong>Structure</strong>: Decentralized system with no central server; all nodes (machines) are equally responsible.</p><p><strong>Key Characteristics</strong>:<br> — Nodes directly communicate and share data/resources.<br> — High fault-tolerance, as no single point of failure.<br> — Scalability comes from adding nodes directly.</p><p><strong>Examples</strong>: Torrent networks (BitTorrent), blockchain networks (Bitcoin, Ethereum), earlier versions of Skype.</p><h4><strong>3. Microservices-Based Systems</strong></h4><p><strong>Structure</strong>: Composed of multiple independent, loosely-coupled services.</p><p><strong>Key Characteristics</strong>:<br> — Each service has its own dedicated database, logic, and deployment lifecycle.<br> — Services communicate using well-defined APIs.<br> — Scalable and fault-tolerant — individual services can scale independently based on demand.<br> — Easier to maintain and upgrade individual components.</p><p><strong>Examples</strong>: Netflix, Amazon, Uber.</p><h4><strong>Common Misconceptions</strong></h4><p><strong>Distributed Databases</strong>:</p><ul><li><strong>Not a type of distributed system</strong>, but rather a component within distributed systems (like microservices or even monolithic systems) used to manage data across nodes.</li></ul><p><strong>Modular or Distributed Monolith</strong>:</p><ul><li>Systems with separated logic but a <strong>shared database</strong> or tightly-coupled services are not true microservices.</li><li>These systems are often <strong>in transition</strong>, moving from a monolithic to a fully microservices-based architecture.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6-z8JA8yIFGApaVERXWhA.png" /></figure><h4><strong>Key Takeaway</strong></h4><ul><li><strong>Three main categories</strong>: Client-server (monolithic), P2P, and Microservices.</li><li><strong>Distributed Databases</strong> are building blocks, not distinct system types.</li><li>Systems halfway between monolithic and microservices (shared DB, tight coupling) are transitional, not a separate category.</li></ul><h3><strong>CAP Theorem</strong></h3><p>The <strong>CAP Theorem</strong> is about how a distributed system behaves during a <strong>network partition (P)</strong> — a situation where parts of the system can’t communicate. It explains why a distributed system can never guarantee <strong>Consistency ( C )</strong>, <strong>Availability ( A )</strong>, and <strong>Partition Tolerance ( P )</strong> simultaneously.</p><h4><strong>What Each Letter Means</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4426cf4364be8592cb4e32f3d70e31a/href">https://medium.com/media/f4426cf4364be8592cb4e32f3d70e31a/href</a></iframe><blockquote><strong><em>Note</em></strong><em>: Since networks inevitably fail or get partitioned, </em><strong><em>Partition Tolerance ( P )</em></strong><em> is always required in real-world distributed systems.</em></blockquote><blockquote><em>Therefore, the actual decision is always between </em><strong><em>Consistency ( C )</em></strong><em>and </em><strong><em>Availability ( A )</em></strong><em> when partitions occur.</em></blockquote><h4><strong>CAP Trade-off Explained</strong></h4><p>When <strong>Partition ( P )</strong> occurs, you must choose:</p><ul><li><strong>Consistency ( C )<br>- </strong>Data is always correct, fresh, and synchronized.<br>- The system may reject requests if it cannot guarantee the most recent data.</li><li><strong>Availability ( A )</strong>:<br>- System always responds (<strong>quickly</strong>), but possibly with stale data.<br>- Syncs and corrects data eventually (background syncing or caching).</li></ul><h4><strong>Real-Life Examples of Choosing C vs. A</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bb99e196cdc201ae555eac48ab5f8aa/href">https://medium.com/media/8bb99e196cdc201ae555eac48ab5f8aa/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TB_ZzdoPlP0v4O5MoAQ-6g.png" /></figure><h4><strong>Consistency Models &amp; Use Cases</strong></h4><p>Consistency isn’t just binary — there are various <strong>models</strong> to choose from</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e0d996b84ac1e7f6c2625ec951d8258/href">https://medium.com/media/7e0d996b84ac1e7f6c2625ec951d8258/href</a></iframe><h4><strong>Practical Analogy</strong></h4><ul><li><strong>Strong Consistency</strong>: Like directly querying a database. Always accurate, but may slow down responses.</li><li><strong>Eventual Consistency</strong>: Like reading from Redis Cache — fast, available, but might show slightly older information temporarily.</li><li><strong>Causal Consistency</strong>: Like chat threads — if message A (“Hello”) triggers message B (“How are you?”), users will always see A first.</li></ul><blockquote><strong><em>Important Note on Causal Consistency:</em></strong></blockquote><blockquote><em>The relationships between events (</em><strong><em>causality</em></strong><em>) must be explicitly defined by the developer.</em></blockquote><blockquote><em>Unlike Strong or Eventual Consistency, causal dependencies don’t automatically exist — developers specify what events depend on each other to ensure correct ordering.</em></blockquote><h4><strong>Redis, Databases, and CAP</strong></h4><p>Depending on your <strong>CAP choice</strong>, you’ll select storage mechanisms accordingly:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc0d2749e8e62f75a014b6dce95e17d6/href">https://medium.com/media/dc0d2749e8e62f75a014b6dce95e17d6/href</a></iframe><h4><strong>Key Takeaways</strong></h4><ul><li><strong>Partition Tolerance ( P )</strong> is always assumed — network failures will happen.</li><li>CAP theorem forces you to choose clearly between <strong>Consistency ( C )</strong>and <strong>Availability ( A )</strong> during failures.</li><li>You can’t avoid network partitions; you decide how the system behaves when partitions inevitably happen.</li><li>Tools like Redis, databases, and caching strategies must align clearly with your chosen CAP model.</li></ul><h3><strong>Sharding — Splitting Data for Massive Scale</strong></h3><p><strong>Sharding</strong> is the practice of dividing your database into smaller, manageable pieces (called <strong><em>shards</em></strong>) spread across multiple servers or <strong>databases</strong>. It’s a powerful method to achieve scalability, reduce load, and maintain <strong>high performance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*QDRlAtQaXkniqJboucKOVg.png" /><figcaption>Oracle Help Center</figcaption></figure><h4><strong>How Sharding Works</strong></h4><ul><li>You select a <strong>Shard Key</strong> (like UserID, Region, etc.) to determine where data will be stored.</li><li>Data placement typically uses a hashing or range-based method:</li><li>For example:<br>- UserID 1–1000 → Database 1<br>- UserID 1001–2000 → Database 2</li><li>Each shard independently manages a subset of data.</li></ul><h4><strong>Why Sharding is Beneficial</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18504c22f922a1d56710c24bf908642b/href">https://medium.com/media/18504c22f922a1d56710c24bf908642b/href</a></iframe><h4><strong>Challenges in Sharding :</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbe96ef79b2c467e97d4ce6c875c5cd9/href">https://medium.com/media/cbe96ef79b2c467e97d4ce6c875c5cd9/href</a></iframe><h4><strong>Choosing the Right Shard Key :</strong></h4><ul><li>A great shard key evenly distributes data and queries across shards.</li><li>Poor shard keys create hotspots or data skews, severely limiting performance and scalability.</li><li><strong>Good Shard Keys :<br>- </strong>UserID<br>- UUID<br>- Hash of identifiers</li><li><strong>Poor Shard Keys :<br>- </strong>Country<br>- City<br>- Gender<br>- Status (uneven distribution)</li><li><strong>Tip :</strong> Use hashed identifiers (e.g., hash(UserID)) or UUIDs to ensure a balanced spread of data across shards.</li></ul><h4><strong>Final Takeaway</strong></h4><ul><li><strong>Sharding is powerful</strong>, but introduces complexity and challenges.</li><li>The <strong>shard key choice</strong> is the single most critical decision.</li><li>Avoid pain later by carefully selecting and testing shard strategies upfront.</li></ul><h3><strong>Conclusion</strong></h3><p>Distributed systems enable scalable and resilient architectures — but come with trade-offs:</p><ul><li><strong>CAP Theorem</strong>: Always pick between Consistency and Availability, as partitions are unavoidable.</li><li><strong>Consistency Models</strong>: Decide based on user experience — strong for accuracy, eventual for speed, causal for ordered events.</li><li><strong>Sharding</strong>: Essential for scalability, but carefully choose your shard key to avoid complexity and performance issues.</li></ul><p>Understanding these trade-offs is key to building reliable systems at massive scale.</p><h3>Up Next in System Design</h3><p>Hey there! I’m <strong>Gyanaa Vaibhav</strong>, a software engineer passionate about System Design, Scalability, and High-Performance Architectures. Through these blogs, I’ll be sharing deep insights, real-world case studies, and expert breakdowns of key system design concepts.</p><h4>What We Learned &amp; What’s Next</h4><ul><li><strong>CAP</strong> trade-offs are everywhere — choose wisely for your use case.</li><li><strong>Consistency models</strong> impact speed, accuracy, and UX.</li><li><strong>Sharding</strong> helps scale but brings new challenges.</li></ul><h4>Next Topic in System Design</h4><p>In the next blog, we’ll explore <strong>Logging, Monitoring &amp; Observability</strong> — we’ll learn how to track what’s happening inside distributed systems in real time. Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fd74772ad45" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>