<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praveen Mudalgeri on Medium]]></title>
        <description><![CDATA[Stories by Praveen Mudalgeri on Medium]]></description>
        <link>https://medium.com/@praveenmudalgeri05?source=rss-8fe1c26d01d7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-E4eY3MiYtTUzba3F1lThQ.jpeg</url>
            <title>Stories by Praveen Mudalgeri on Medium</title>
            <link>https://medium.com/@praveenmudalgeri05?source=rss-8fe1c26d01d7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:08:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@praveenmudalgeri05/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[75+ Essential Linux Commands Every User Should Know [with pdf]]]></title>
            <link>https://medium.com/@praveenmudalgeri05/75-essential-linux-commands-every-user-should-know-with-pdf-e19fd6fe7ece?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e19fd6fe7ece</guid>
            <category><![CDATA[linux-commands]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[must-know]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Mon, 26 Jan 2026 15:40:58 GMT</pubDate>
            <atom:updated>2026-01-26T15:40:58.408Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*ENkdUhjP5P3tQ7KBL-WVZg.jpeg" /></figure><p>The command line is where you get real control in Linux. It lets you dig into the system, automate tasks, and solve problems directly.</p><p>This guide includes a hand-picked list of fundamental Linux commands, grouped by category. You’ll find commands for managing files, users, hardware, networking, and more.</p><h3>System-Based Commands</h3><ul><li>uname - Displays system information: kernel version, machine type, and more.</li><li>uname -r - Displays the running Linux kernel&#39;s release version.</li><li>uptime - Shows current time, system uptime, users, and load averages.</li><li>hostname - Shows the system hostname.</li><li>hostname -i - Displays the IP address of the current host.</li><li>last reboot - Shows last reboot times and durations in logs.</li><li>date - Displays the current date and time information.</li><li>timedatectl - Displays detailed system clock and time zone information.</li><li>cal - Displays a simple calendar of the current month.</li><li>w - Shows who is logged on and their activity.</li><li>whoami - Displays the username of the current user.</li><li>finger username - Displays information about a user named username.</li></ul><h3>Hardware-Based Commands</h3><ul><li>dmesg - Displays messages from the kernel&#39;s ring buffer.</li><li>cat /proc/cpuinfo - Displays detailed information about the CPU.</li><li>cat /proc/meminfo - Displays detailed system memory usage information.</li><li>lscpu - Lists information about the CPU.</li><li>lshw - Lists detailed hardware configuration of the system.</li><li>lsblk - Lists information about all available block devices.</li><li>free -m - Shows system memory usage in megabytes.</li><li>lspci -tv - Displays PCI devices in tree format, verbosely.</li><li>lsusb -tv - Shows USB devices as a tree, verbosely.</li><li>dmidecode - Displays hardware information from system BIOS.</li><li>hdparm -i /dev/sda - Displays information of disk /dev/sda.</li><li>badblocks -s /dev/sda - Checks /dev/sda for bad blocks, showing progress.</li></ul><h3>User Management Commands</h3><ul><li>id - Displays the user&#39;s UID, GID, and groups.</li><li>last - Shows list of last logged-in users.</li><li>who - Displays who is currently logged in.</li><li>groupadd admin - Creates a new user group named admin.</li><li>adduser Sam - Creates a new user account named Sam.</li><li>userdel Sam - Deletes the user account named Sam.</li><li>usermod - Modifies properties of an existing user account.</li></ul><h3>File Commands</h3><ul><li>ls -al - Lists all files, with detailed information, in long format.</li><li>pwd - Displays the present working directory&#39;s path.</li><li>mkdir dir1 - Creates a new directory named dir1.</li><li>rm file1 - Deletes the file named file1.</li><li>rm -f file2 - Forcefully deletes the file named file2.</li><li>rm -r dir1 - Recursively removes directory dir1 and its contents.</li><li>rm -rf dir1 - Forcefully deletes directory dir1 and its contents.</li><li>cp file1 file2 - Copies file1, creating or overwriting file2.</li><li>cp -r dir1 dir2 - Copies dir1 to dir2, including subdirectories.</li><li>mv file1 file2 - Renames or moves file1 to file2.</li><li>ln -s /path/to/file_name link_name - Creates symbolic link named link_name to file_name.</li><li>touch file1 - Creates an empty file named file1.</li><li>cat &gt; file1 - Creates or overwrites file1, awaiting standard input.</li><li>more file1 - Displays file1 content, paginating through output.</li><li>head file1 - Displays the first ten lines of file1.</li><li>tail file1 - Displays the last ten lines of file1.</li><li>gpg -c file1 - Encrypts file1 with a symmetric cipher using a passphrase.</li><li>gpg file2.gpg - Decrypts file2.gpg, prompting for the passphrase.</li><li>wc - Counts words, lines, and characters in files.</li><li>xargs - Executes commands with piped or file-provided arguments.</li></ul><h3>Network Commands</h3><ul><li>ip addr show - Displays all network interfaces and their information.</li><li>ip address add 192.168.0.1/24 dev eth0 - Assigns IP address 192.168.0.1 to interface eth0.</li><li>ifconfig - Shows network interfaces and their configuration.</li><li>ping host - Sends ICMP packets, measures round-trip time to host.</li><li>whois domain - Retrieves and displays a domain&#39;s registration information.</li><li>dig domain - Queries DNS, provides domain&#39;s DNS information.</li><li>dig -x host - Resolves IP address to hostname, shows reverse DNS info.</li><li>host google.com - Performs an IP lookup for the domain name.</li><li>wget file_path - Downloads file from the specified path.</li><li>netstat - Displays various network-related information and statistics.</li><li>ss - Displays information about network sockets.</li></ul><h3>Compression / Archive Commands</h3><ul><li>tar -cf backup.tar /home/ubuntu - Creates a tar archive of the /home/ubuntu directory.</li><li>tar -xf backup.tar - Extracts files from the backup.tar archive.</li><li>tar -zcvf backup.tar.gz /home/ubuntu - Creates compressed backup.tar.gz archive of /home/ubuntu.</li><li>gzip file1 - Compresses file1 into file1.gz; original file is removed.</li></ul><h3>Install Packages Commands</h3><ul><li>rpm -i pkg_name.rpm - Installs the package pkg_name.rpm using RPM.</li><li>rpm -e pkg_name - Uninstalls the specified RPM package.</li><li>dnf install pkg_name - Installs the specified package using DNF.</li><li>pacman -S pkg_name - Installs the specified package using Pacman.</li></ul><h3>Install from Source (Compilation)</h3><ul><li>./configure - Checks system compatibility and generates Makefile.</li><li>make - Compiles code using the Makefile.</li><li>make install - Installs compiled code into appropriate system locations.</li></ul><h3>Search Commands</h3><ul><li>grep pattern file - Searches for a given pattern within the specified file.</li><li>grep -r pattern dir1 - Recursively searches for pattern in directory dir1.</li><li>locate file - Finds files named file using a prebuilt index database.</li><li>find /home -name index - Searches /home for files named index.</li><li>find /home -size +10000k - Finds files over 10,000 KB in /home.</li></ul><h3>Login Commands</h3><ul><li>ssh user@hostname - Initiates SSH connection to specified host.</li><li>ssh -p port_number user@hostname - Connects over SSH using a specific port.</li><li>ssh hostname - Connects to hostname using SSH on port 22.</li><li>telnet host - Connects to host via Telnet on port 23.</li></ul><h3>File Transfer Commands</h3><ul><li>scp file.txt remoteuser@remote_host:/remote/directory - Copies file.txt to a remote directory.</li><li>rsync -a /home/ubuntu /backup/ - Synchronizes /home/ubuntu to /backup/, preserving attributes.</li><li>rsync -a /var/www/web/ user@remote_host:/backup/web_backup/ - Syncs local directory to remote backup.</li></ul><h3>Disk Usage Commands</h3><ul><li>df -h - Shows human-readable disk space usage.</li><li>df -i - Shows inode usage for all mounted filesystems.</li><li>fdisk -l - Lists all partitions and information for all drives.</li><li>du -sh /dir1 - Displays total disk usage of /dir1 in human-readable form.</li><li>findmnt - Lists all mounted filesystems and their properties.</li><li>mount device-path mount-point - Mounts a device to the specified mount point.</li></ul><h3>Directory Traversal Commands</h3><ul><li>cd .. - Navigates to the parent directory.</li><li>cd - Changes the current directory to the user&#39;s home.</li><li>cd /mnt - Changes the current directory to /mnt.</li></ul><h3>Cheat Sheet PDF</h3><p>Download your free <a href="https://linuxopsys.com/wp-content/uploads/2022/06/linux-cheat-sheet.pdf">Linux commands cheat sheet in pdf</a> format and print on <a href="https://linuxopsys.com/wp-content/uploads/2022/06/linux-commands-cheat-sheet-A4.pdf">A4 size paper</a>.</p><p>Thanks for reading!</p><p>If you enjoyed this content, don’t forget to leave a like ❤️ and subscribe to get more posts like this every week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e19fd6fe7ece" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design Interview Survival Guide: 5 Patterns to Scale Any App]]></title>
            <link>https://medium.com/@praveenmudalgeri05/system-design-interview-survival-guide-5-patterns-to-scale-any-app-c21510299499?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c21510299499</guid>
            <category><![CDATA[design-thinking]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[interview]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 16:56:08 GMT</pubDate>
            <atom:updated>2026-01-02T16:56:08.022Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcYKP33GfgR_Bp9jAt94xQ.png" /></figure><p>Growth is the ultimate goal of any software project.</p><p>You spend months coding, testing, and refining your application.</p><p>You launch it, and for a while, everything is quiet.</p><p>Then, something incredible happens.</p><p>Your user base starts to grow. You go from 100 users to 1,000, then to 10,000. This is the moment every developer dreams of.</p><p>However, this success brings a specific type of danger.</p><p>The code that worked perfectly for 100 users will often crumble under the weight of 1 million.</p><p>This happens because most initial applications are built to run on a single computer. But physical computers have limits. They only have so much processing power and memory. When you exceed those limits, you have to change your strategy.</p><p>You have to stop thinking about a single computer and start thinking about a system of computers working together.</p><p>This is what we call <strong>Scalability</strong>.</p><p>Scalability is not just about adding more hardware. It is about designing your software so that it can handle more work simply by adding more resources. It is the art of ensuring your application stays fast and reliable, even when the whole world is trying to use it at the same time.</p><p>In this guide, we will look at 5 foundational patterns that enable systems to scale. These are the tools that senior architects use every day.</p><p>By the end of this post, you will understand not just what they are, but exactly how they save your application from crashing.</p><h3>1. Load Balancing</h3><p>Imagine you run a grocery store.</p><p>When you first open, you have very few customers, so you only need one cashier. This works fine until the holiday season hits.</p><p>Suddenly, you have hundreds of people trying to check out at once. The line for that single cashier wraps around the block.</p><p>People get angry and leave.</p><p>To fix this, you hire ten more cashiers.</p><p>But now you have a new problem.</p><p>Customers don’t know which line is the shortest. Everyone rushes to the first register, leaving the others empty. You need someone at the front of the store to direct people to the open registers.</p><p>In system design, that person is the <strong>Load Balancer</strong>.</p><p><strong>What it is:</strong> A Load Balancer is a server (or a piece of hardware) that sits between your users and your application servers. Its only job is to distribute incoming network traffic across multiple servers.</p><p><strong>How it works behind the scenes:</strong> When a user visits your website, they are not connecting directly to your main server. Instead, they connect to the Load Balancer.</p><p>The Load Balancer looks at its list of available servers. It asks, “Who is free right now?” It then forwards the user’s request to a server that isn’t too busy.</p><p>There are different strategies the Load Balancer uses to make this decision:</p><ul><li><strong>Round Robin:</strong> This is the simplest method. It sends requests in a circle. Request 1 goes to Server A. Request 2 goes to Server B. Request 3 goes to Server C, and then it starts over at A.</li><li><strong>Least Connections:</strong> The Load Balancer checks which server is currently handling the fewest active users and sends the new person there.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z6_Nz9nBZe6qo2Ls.jpeg" /></figure><p>Without a Load Balancer, you have a “Single Point of Failure.”</p><p>If your one server crashes, your whole app goes down.</p><p>With a Load Balancer, you can have five servers.</p><p>If one crashes, the Load Balancer notices it is dead (using a “Health Check”) and simply stops sending traffic to it. The users never even know that something went wrong.</p><h3>2. Caching: The Cheat Sheet</h3><p>Let’s say you are taking a very difficult math exam.</p><p>Every time you need to solve a complex equation, you have to open a massive textbook, find the right page, and read the formula. This takes a long time.</p><p>If you are smart, you will write the most common formulas on a piece of paper and keep it right next to you.</p><p>Now, when you need a formula, you look at your paper first.</p><p>It is instant. You only open the heavy textbook if the answer isn’t on your paper.</p><p>In this analogy, the textbook is your <strong>Database</strong>, and the piece of paper is your <strong>Cache</strong>.</p><p><strong>What it is:</strong> Caching is the practice of storing copies of frequently used data in a temporary storage location that is very fast to access. Usually, this storage is the computer’s RAM (Random Access Memory).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lz2FVcA4KeOISdQJ.jpeg" /></figure><p><strong>How it works behind the scenes:</strong> Databases save data to the hard drive (disk). Hard drives are reliable, but they are slow compared to RAM. Reading from a hard drive is like walking to the library. Reading from RAM is like remembering a thought in your brain.</p><p>Here is the flow when you use a cache:</p><ol><li>The user asks for a user profile.</li><li>The application checks the Cache first.</li><li><strong>Cache Hit:</strong> If the data is there, it returns it instantly. This takes milliseconds.</li><li><strong>Cache Miss:</strong> If the data is not there, the application queries the slow Database.</li><li>The application sends the data to the user <em>and</em> saves a copy in the Cache for next time.</li></ol><p>Most applications are “Read-Heavy.” This means people look at data much more often than they change it.</p><p>Think about Twitter.</p><p>You might tweet once a day (write), but you scroll through hundreds of tweets (read).</p><p>If every single scroll required your database to work hard, it would crash.</p><p>Caching takes the pressure off your database. It handles the repetitive requests so your database can focus on the important stuff, like saving new data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c21510299499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design Roadmap for Beginners]]></title>
            <link>https://medium.com/@praveenmudalgeri05/system-design-roadmap-for-beginners-1010a7cbd476?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1010a7cbd476</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[roadmaps]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 16:09:21 GMT</pubDate>
            <atom:updated>2025-12-31T16:09:21.321Z</atom:updated>
            <content:encoded><![CDATA[<p>If you like my work, please do subscribe, for getting such articles regularly:)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*ZPg6EhbC87DObWXJXQ0AKg.jpeg" /></figure><p>If you’re an absolute beginner in System Design, this roadmap will guide you step-by-step through all the key concepts using one consistent example: <strong>Instagram</strong>.</p><h3>🗓️ Week 1 — Foundations</h3><ul><li><strong>What is System Design?</strong>: The process of building software systems like Instagram that can <strong>scale</strong> (handle millions of users), maintain <strong>reliability</strong> (stay available without crashing), and ensure <strong>maintainability</strong> (are easy to update and extend).</li><li><strong>4 Components:</strong> Client, Server, Database, APIs.</li><li><em>Client</em>: Your Instagram app,</li><li><em>Server</em>: Processes requests like “upload photo”</li><li><em>API: </em>Fetches user feed,</li><li><em>Database</em>: Stores user data, posts, reels etc.</li></ul><h4><strong>Videos to Watch</strong>:</h4><ul><li><a href="https://www.youtube.com/watch?v=xpDnVSmNFX0">System Design Primer — How to start? (Gaurav Sen)</a></li><li><a href="https://www.youtube.com/watch?v=UzLMhqg3_Wc">Intro to System Design (Tech Dummies Narendra L)</a></li></ul><h3>🗓️ Week 2 — Scaling Basics &amp; Load Balancing</h3><ul><li><strong>Vertical Scaling:</strong> Add more CPU/RAM to Instagram’s SINGLE web server. Works early on but becomes costly.</li><li><strong>Horizontal Scaling:</strong> Add many web servers to handle traffic. Instagram uses hundreds of servers to handle profile views and post uploads simultaneously.</li><li><strong>Load Balancer:</strong> Routes user requests evenly across servers. If 3 instagram servers exist, 1st handles uploads, 2nd handles profile fetches, and the 3rd handles likes.</li><li><strong>Stateless Service:</strong> Each request from your Instagram app( fetching your feed, liking a post, or adding a comment) is independent and can be handled by any server. Hence, the servers don’t store session information or “remember” users between requests. Instead, your app holds a token or cookie that is sent with every request, allowing the server to identify and authenticate the user each time.</li></ul><h4><strong>Videos to Watch</strong>:</h4><ul><li><a href="https://www.youtube.com/watch?v=K0Ta65OqQkY">Load Balancing Explained — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=-W9F__D3oY4">Scalability &amp; Availability — Tech Dummies</a></li></ul><h3>🗓️ Week 3 — Databases &amp; Data Partitioning</h3><ul><li><strong>SQL vs NoSQL:</strong></li><li>SQL (Postgres/MySQL) → stores structured data (users, posts).</li><li>NoSQL (Cassandra): stores large, flexible data (likes, comments feed).</li><li><strong>Replication:</strong> Instagram’s user data is copied across multiple databases around the world. If one database (say, in India) fails, another replica (like one in Singapore or Europe) instantly takes over, ensuring the app stays available without interruption.</li><li><strong>Sharding:</strong> Data between these databases is split by user ID or region.<br>Users A–M → DB1; N–Z → DB2 → prevents one database from overloading.</li><li><strong>CAP Theorem:</strong> Instagram chooses <em>Availability + Partition Tolerance</em> over strict consistency: a like might appear a few seconds late but app stays up.</li></ul><h4><strong>Videos to Watch:</strong></h4><ul><li><a href="https://www.youtube.com/watch?v=-t1_ffaFXao">Database Sharding — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=E6G2Z4lprNY">Replication &amp; Consistency — Tech Dummies</a></li></ul><h3>🗓️ Week 4 — Caching &amp; CDN</h3><ul><li><strong>Caching:</strong> Cache store frequently accessed data in memory (Redis, Memcached). When you open Instagram, your feed loads from cache instead of querying the DB each time.</li><li><strong>CDN:</strong> Stores static content (images, stories, profile pics) on servers worldwide for faster load. Your photo stored on a CDN node in India is served instantly to nearby users.</li><li><strong>Latency vs Throughput:</strong> Caching/CDN reduce latency (delay) and increase throughput (requests/sec).</li></ul><h4><strong>Videos to watch</strong>:</h4><ul><li><a href="https://www.youtube.com/watch?v=RMr2VgSxs9M">Caching in Distributed Systems — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=R7Sxn6XskxE">CDN Explained — Tech Dummies</a></li></ul><h3>🗓️ Week 5 — Event-Driven Architecture</h3><ul><li><strong>Message Queues:</strong> Decouple services to handle background jobs.<br>When you upload a photo, Instagram immediately shows it in your feed, but <em>processing</em> (resizing, compressing, tagging) happens asynchronously via message queues e.g: Kafka.</li><li><strong>Publish–Subscribe:</strong> When one event occurs, it is notified to all services that are subscribed. Eg: When a “Photo Uploaded” on Instagram, updates are sent to the Feed Service, Notification Service, and Search Index.</li><li><strong>Eventual Consistency :</strong> Every activity eg: “like”, “comment”, “upload” is stored as an event for auditing. These events are synced between the caches, and hence sometimes your likes or comments might appear 1–2 seconds late.</li></ul><h4><strong>Videos to Watch</strong>:</h4><ul><li><a href="https://www.youtube.com/watch?v=FveRAy_9BZk">Message Queues — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=_Rnb3RziK3Y">Event-Driven Architecture — Tech Dummies</a></li></ul><h3>🗓️ Week 6 — Networking &amp; Microservices</h3><ul><li><strong>Protocols (HTTP/TCP/WebSockets): </strong>Different protocols are used for client-server communications. Eg: HTTP for browsing profiles, WebSockets for real-time likes/comments.</li></ul><h4><strong>Microservices vs Monolith:</strong></h4><ul><li><strong>Monolith</strong>: Initially Instagram was a monolith, i.e. all its logic (login, posting, comments, notifications) lived in one single big codebase and ran as one giant program. This led to slower deployments, and if one part crashed everything crashed.</li><li><strong>Microservice Archtecture</strong>: Now Instagram breaks the big app into many smaller, independent services(User, Post, Feed, Search, Notification), each having its independent deployment, code base and database.</li><li><strong>API Gateway</strong>: It is a traffic controller for all the requests that come into Instagram’s backend. Instead of users talking directly to dozens of microservices (User, Feed, Notification, etc.), they talk to one single endpoint — the API Gateway, which then navigates them to these microservices.</li></ul><h4><strong>Videos to Watch:</strong></h4><ul><li><a href="https://www.youtube.com/watch?v=5yTazHkDR4o">Microservices Architecture — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=d4Y2DkKbx74">API Gateway &amp; Discovery — Tech Dummies</a></li></ul><h3>🗓️ Week 7 — High-Level Design Case Studies</h3><ul><li><strong>Most Frequently asked Case Studies</strong>: Design → Instagram / Twitter feed, WhatsApp / Messenger, YouTube / Netflix streaming, URL Shortener (Bitly), Uber / Ola, Amazon / Flipkart, Notification service, Real-time chatbot</li></ul><h4><strong>Videos to Practice HLD</strong>:</h4><ul><li><a href="https://www.youtube.com/watch?v=VJpfO6KdyWE">Design Instagram — Gaurav Sen</a></li><li><a href="https://www.youtube.com/watch?v=E5RjzSK0fvY">Design Instagram Feed — Tech Dummies</a></li></ul><h3>🗓️ Week 8 — Low-Level Design (LLD)</h3><ul><li><strong>Understand Design Patterns</strong></li><li><strong>Creational:</strong> Singleton, Factory, Builder, Prototype</li><li><strong>Structural:</strong> Adapter, Decorator, Proxy, Composite, Facade</li><li><strong>Behavioral:</strong> Observer, Strategy, Command, State, Chain of Responsibility</li><li><strong>Practice UML: </strong>Class, Sequence and Component diagrams</li><li><strong>Most Frequently asked Case Studies: </strong>Parking Lot system, Elevator system, Movie Booking system, Splitwise, Logger system, Rate limiter, Cache design (LRU, LFU)</li></ul><h4><strong>Videos to Practice LLD</strong>:</h4><ul><li><a href="https://www.youtube.com/playlist?list=PL564gOx0bCLpXQp4-WwY5N24PuL0EBe4n">LLD Examples — CodeKarle Playlist</a></li><li><a href="https://www.youtube.com/watch?v=3baWzvEDfgU&amp;utm_source=chatgpt.com">System Design Interview Prep — Striver × Gaurav Sen</a></li></ul><h3>🗓️ Week 9 — Interview Prep</h3><p>This is THE BEST Video for Interview Prep:</p><p><a href="https://youtu.be/3baWzvEDfgU?si=CgebX5SSzbSme6hC">System Design Interview Prep — Striver × Gaurav Sen</a></p><h3>📌 Final Thoughts</h3><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1010a7cbd476" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[System Design was HARD until I Learned these 30 Concepts]]></title>
            <link>https://medium.com/@praveenmudalgeri05/system-design-was-hard-until-i-learned-these-30-concepts-2928dd7f7799?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/2928dd7f7799</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Sat, 20 Dec 2025 17:59:03 GMT</pubDate>
            <atom:updated>2025-12-20T17:59:03.442Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>System Design</strong> can feel overwhelming especially when you’re just starting out and don’t know where to begin.</p><p>But once you understand the <strong>core concepts</strong> and <strong>building blocks</strong>, it becomes much less intimidating — whether you’re preparing for <strong>interviews</strong> or <strong>designing scalable systems</strong> at work.</p><p>In this article, I’ll walk you through the <strong>30 most important System Design concepts</strong> every developer should know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rTezOxMTzUX921ox.png" /></figure><h3>1. Client-Server Architecture</h3><p>Almost every web application that you use is built on this simple yet powerful concept called <strong>client-server architecture</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GPa7kk3JQrk9ssZC.png" /></figure><p>On one side, you have a <strong>client</strong> — this could be a web browser, a mobile app, or any other frontend application.</p><p>and on the other side, you have a <strong>server</strong> — a machine that runs continuously, waiting to handle incoming requests.</p><p>The client sends a request to <strong>store, retrieve, or modify data</strong>.</p><p>The server receives the request, processes it, performs the necessary operations, and sends back a response.</p><p>This sounds simple, but there’s a big question: <strong><em>How does the client even know where to find the server?</em></strong></p><h3>2. IP Address</h3><p>A client doesn’t magically know where a server is, it needs an <strong>address</strong> to locate and communicate with it.</p><p>On the internet, computers identify each other using <strong>IP addresses</strong>, which work like phone numbers for servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ox0uJdEcXOJd3U8spbj9fw.png" /></figure><p>Every publicly deployed server has a <strong>unique IP address</strong>. When a client wants to interact with a service, it must send requests to the correct IP address.</p><p>But there’s a problem:</p><ul><li>When we visit a website, we don’t type its IP address — we just enter the website name.</li><li>We can’t expect users (or even systems) to memorize a string of random numbers for every service they connect to.</li><li>And if we migrate our service to another server, its IP address may change — breaking all direct connections.</li></ul><h3>3. DNS</h3><p>Instead of relying on hard-to-remember IP addresses, we use something much more human-friendly: <strong>domain names</strong>.</p><p>But, we need a way to map a domain name to it’s corresponding IP address.</p><p>This is where <strong>DNS (or Domain Name System)</strong> comes in. It maps easy to remember domain names (like algomaster.io) to their corresponding IP addresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8rJMJWyQ929Fo0A-h7uyRA.png" /></figure><p>Here’s what happens behind the scenes:</p><ol><li>When you type algomaster.io into your browser, your computer asks a DNS server for the corresponding IP address.</li><li>Once the DNS server responds with the IP, your browser uses it to establish a connection with the server and make a request.</li></ol><p>You can find the IP address of any domain using the <strong>ping</strong> command. Just open your terminal and type ping followed by the domain name. And it’ll return the IP address currently assigned to that domain.</p><h3>4. Proxy / Reverse Proxy</h3><p>When you visit a website, your request doesn’t always go directly to the server — sometimes, it passes through a <strong>proxy</strong> or <strong>reverse proxy</strong> first.</p><p>A <strong>proxy server</strong> acts as a <strong>middleman</strong> between your device and the internet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*SEu_hPk86dWmRgiiZQmLoA.png" /></figure><p>When you request a webpage, the proxy forwards your request to the target server, retrieves the response, and sends it back to you.</p><p>Proxy hides your IP address, keeping your location and identity private.</p><p>A <strong>reverse proxy</strong> works the other way around. It intercepts client requests and forwards them to backend servers based on predefined rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*H0QuLAZdxcA4ID4COSXv9Q.png" /></figure><p>Allowing direct access to servers can pose <strong>security risks</strong>, exposing them to threats like <strong>hackers</strong> and <strong>DDoS attacks</strong>.</p><p>A reverse proxy mitigates these risks by acting as a controlled entry point that regulates incoming traffic and hides server IPs.</p><p>It can also act as a load balancer, distributing traffic across multiple servers.</p><h3>5. Latency</h3><p>Whenever a client communicates with a server, there’s always some delay. One of the biggest causes of this delay is <strong>physical distance</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oECfpbpm_K8-01fJZ1Z6Yw.png" /></figure><p>For example, if our server is in <strong>New York</strong>, but a user in <strong>India</strong> sends a request, the data has to travel halfway across the world — and then the response has to make the same long trip back.</p><p>This round-trip delay is called <strong>latency</strong> — the total time it takes for data to travel between the client and the server. High latency can make applications feel slow and unresponsive.</p><p>One way to <strong>reduce latency</strong> is by deploying our service across <strong>multiple data centers worldwide</strong>.</p><p>This way, users can connect to the <strong>nearest</strong> server instead of waiting for data to travel across the globe.</p><p><strong><em>Once a connection is made, how do clients and servers actually communicate?</em></strong></p><h3>6. HTTP/HTTPS</h3><p>Every time you visit a website, your browser and the server communicate using a set of rules called <strong>HTTP (Hypertext Transfer Protocol)</strong>.</p><p>That’s why most URLs start with http:// or its secure version, <a href="https://./">https://.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*52OuEe2LLp-Tt1b_aLIBSQ.png" /></figure><p>Here’s how it works:</p><ul><li>The <strong>client</strong> sends a request to the server. This request includes a <strong>header</strong> (containing details like the request type, browser type, and cookies) and sometimes a <strong>request body</strong> (which carries additional data, like form inputs).</li><li>The <strong>server</strong> processes the request and responds with an <strong>HTTP response</strong> — either returning the requested data or an error message if something goes wrong.</li></ul><p>HTTP has a major security flaw, it <strong>sends data in plain text</strong>. This is a serious problem, especially for sensitive information like passwords, credit card details, and personal data.</p><p>That’s why modern websites use <strong>HTTPS (Hypertext Transfer Protocol Secure)</strong> instead. HTTPS encrypts all data using <strong>SSL/TLS,</strong> ensuring that even if someone intercepts the request, they can’t read or alter it.</p><p>But clients and servers don’t directly exchange raw HTTP requests and response.</p><p>HTTP is just a <strong>protocol</strong> for transferring data but it doesn’t define:</p><ul><li>How requests should be structured</li><li>What format responses should be in</li><li>or how different clients should interact with the server.</li></ul><p>This is where <strong>APIs (or Application Programming Interfaces)</strong> come in.</p><h3>7. APIs</h3><p>Think of an API as a <strong>middleman</strong> that allows clients (like web and mobile apps) to communicate with servers without worrying about low-level details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1T7qygSuYWXNjT-HdLItA.png" /></figure><p>Almost every digital service you use — social media, e-commerce, online banking, ride-hailing apps — is built on APIs working together behind the scenes.</p><p>Here’s how it typically works:</p><ol><li>A <strong>client</strong> sends a request to an API.</li><li>The <strong>API</strong>, hosted on a server, processes the request, interacts with databases or other services, and prepares a response.</li><li>The <strong>API sends back the response</strong> in a structured format, usually <strong>JSON</strong> or <strong>XML</strong>, which the client understands and can display.</li></ol><p>APIs provide a <strong>layer of abstraction</strong> — the client doesn’t need to know <strong>how</strong> the server processes the request, only that it <strong>returns the expected data</strong>.</p><p>But, not all APIs are built the same. Different API styles exist to serve different needs. Two of the most popular ones are <strong>REST</strong> and <strong>GraphQL</strong>.</p><h3>8. Rest API</h3><p>Among the different API styles, <strong>REST (Representational State Transfer) </strong>is the most widely used.</p><p>A <strong>REST API</strong> follows a set of rules that define how clients and servers communicate over HTTP in a structured way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NfxPDbHqHDyfosvdV5ggjQ.png" /></figure><p>Rest is:</p><ul><li><strong>Stateless:</strong> Every request is independent; the server doesn’t store client state.</li><li><strong>Resource-Based:</strong> Everything is treated as a resource (e.g., /users, /orders, /products).</li><li><strong>Uses Standard HTTP Methods:</strong> Clients interact with resources using <strong>HTTP methods</strong> like:</li><li><strong>GET</strong> → Retrieves data (e.g., fetching a user profile).</li><li><strong>POST</strong> → Creates new data (e.g., adding a new user).</li><li><strong>PUT/PATCH</strong> → Updates existing data (e.g., changing user settings).</li><li><strong>DELETE</strong> → Removes data (e.g., deleting an account).</li></ul><p>REST APIs are great because they’re <strong>simple, scalable, and easy to cache</strong>, but they have <strong>limitations</strong>, especially when dealing with complex data retrieval.</p><p>REST endpoints often return <strong>more data than needed</strong>, leading to inefficient network usage. If an API doesn’t return related data, the client may need to <strong>make multiple requests</strong> to retrieve all required information.</p><p>To address these challenges, GraphQL was introduced in 2015 by Facebook.</p><h3>9. GraphQL</h3><p>Unlike REST, which forces clients to retrieve <strong>fixed sets of data</strong>, <strong>GraphQL</strong> lets clients <strong>ask for exactly what they need — nothing more, nothing less</strong>.</p><p>With a REST API, if you need a user details, user profile details along with their recent posts, you might have to <strong>make multiple requests</strong> to different endpoints:</p><ol><li>GET /api/users/123 → fetch user details</li><li>GET /api/users/123/profile → fetch user profile</li><li>GET /api/users/123/posts → fetch user’s posts</li></ol><p>With GraphQL, you can <strong>combine those requests into one</strong> and fetch exactly the data you need in a single query:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unIqUjpkus7M51bJjZpx7A.png" /></figure><p>The <strong>server responds with only the requested fields</strong>, reducing unnecessary data transfer and improving efficiency.</p><p>However, GraphQL also comes with trade-offs — it <strong>requires more processing on the server side</strong> and isn’t as easy to cache as REST.</p><p>When a client makes a request, they usually want to <strong>store or retrieve data</strong>.</p><p>But this brings up another question — <strong><em>where is the actual data stored?</em></strong></p><h3>10. Databases</h3><p>If our application deals with <strong>small amounts of data</strong>, we could store it <strong>in memory</strong>.</p><p>But modern applications handle <strong>massive volumes of data</strong> — far more than what memory can efficiently handle.</p><p>That’s why we need a <strong>dedicated server for storing and managing data</strong> — a <strong>database</strong>.</p><p>A database is the backbone of any application.<strong> </strong>It ensures that data is stored, retrieved, and managed efficiently while keeping it secure, consistent, and durable.</p><p>When a client requests to <strong>store</strong> or <strong>retrieve</strong> data, the <strong>server communicates with the database</strong>, fetches the required information, and returns it to the client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5D0K9wBYmudouuhDfTgUQ.png" /></figure><p>But not all databases are the same. Different applications have different <strong>scalability, performance, and consistency</strong> requirements, which is choosing the right type of database is important.</p><p>In system design, we typically choose between <strong>SQL and NoSQL databases</strong>.</p><h3>11. SQL vs NoSQL</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUJ7J04jfZuM7MdW-19rUg.png" /></figure><p>SQL databases store data in tables with a <strong>strict predefined schema</strong> and follow the <strong>ACID properties</strong>.</p><ul><li><strong>Atomicity</strong> — A transaction is <strong>all-or-nothing</strong> (it either completes fully or not at all).</li><li><strong>Consistency</strong> — Data always remains <strong>valid</strong> and follows defined rules.</li><li><strong>Isolation</strong> — Transactions <strong>don’t interfere</strong> with each other.</li><li><strong>Durability</strong> — Once data is saved, it <strong>won’t be lost</strong>, even if the system crashes.</li></ul><p>Because of these guarantees, SQL databases are ideal for applications that require <strong>strong consistency and structured relationships</strong>, such as <strong>banking systems</strong>.</p><blockquote><em>Examples of popular SQL databases include: MySQL and PostgreSQL</em></blockquote><p>NoSQL databases on the other hand are designed for <strong>high scalability and performance</strong>.</p><p>They <strong>don’t require a fixed schema</strong> and use different data models, including:</p><ul><li><strong>Key-Value Stores</strong> — Fast lookups for simple key-value pairs (e.g., Redis).</li><li><strong>Document Stores</strong> — Store flexible, JSON-like documents (e.g., MongoDB).</li><li><strong>Graph Databases</strong> — Best for highly connected data (e.g., Neo4j).</li><li><strong>Wide-Column Stores</strong> — Optimized for large-scale, distributed data (e.g., Cassandra).</li></ul><p>So, <strong>which one should you use?</strong> It depends on the system requirements.</p><ul><li>If you need <strong>structured, relational data with strong consistency</strong> → <strong>SQL is a better choice.</strong></li><li>If you need <strong>high scalability, flexible schemas, or fast reads/writes at scale</strong> → <strong>NoSQL is a better choice.</strong></li></ul><p>Many modern applications <strong>use both SQL and NoSQL together.</strong></p><p>For example, an e-commerce platform might:</p><ul><li>Store customer orders in SQL (because they require strict consistency).</li><li>and store Product recommendations in NoSQL (because they need flexible and fast lookups).</li></ul><h3>12. Vertical Scaling</h3><p>As our <strong>user base grows</strong>, so does the number of <strong>requests hitting our application servers</strong>.</p><p>Initially, a <strong>single server</strong> might be enough to handle the load. But, as traffic increases, that single server can become a bottleneck, slowing everything down.</p><p>One of the quickest solutions is to <strong>upgrade the existing server</strong> by adding more CPU, RAM or storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKj07pIaYmjUm2FjZJkopw.png" /></figure><p>This approach is called <strong>Vertical Scaling (Scaling Up)</strong> — making a single machine more powerful.</p><p>But there are some major limitations with this approach:</p><ol><li><strong>Hardware limits →</strong> You can’t keep upgrading a server forever. Every machine has a maximum capacity.</li><li><strong>Cost →</strong> More powerful servers become exponentially more expensive.</li><li><strong>Single Point of Failure (SPOF)</strong> <strong>→</strong> if this one server crashes, the entire system <strong>goes down</strong>.</li></ol><p>So, while vertical scaling is a quick fix, it’s not a long-term solution for handling high traffic and ensuring system reliability.</p><p>Lets look at a better approach — one that makes our system more scalable and fault tolerant.</p><h3>13. Horizontal Scaling</h3><p>Instead of upgrading a single server, what if we <strong>add more servers</strong> to share the load?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-aJduMdKVvClL14of1DPg.png" /></figure><p>This approach is called <strong>Horizontal Scaling (Scaling Out)</strong> — where we <strong>distribute the workload across multiple machines</strong>.</p><p>This approach is better because:</p><ul><li><strong>More servers = More capacity →</strong> The system can handle increasing traffic more effectively.</li><li><strong>No Single Point of Failure →</strong> If one server goes down, others can <strong>take over</strong>, improving reliability.</li><li><strong>Cost-effective → </strong>Instead of investing in a single, super-expensive machine, we can use multiple affordable ones.</li></ul><p>But horizontal scaling introduces a new challenge: <strong><em>how do clients know which server to connect to?</em></strong></p><p>This is where a <strong>Load Balancer</strong> comes in.</p><h3>14. Load Balancers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KWTXhHkfy84s3HPPGj1XsQ.png" /></figure><p>A <strong>Load Balancer</strong> sits between <strong>clients</strong> and <strong>backend servers</strong>, acting as a <strong>traffic manager</strong> that distributes requests across multiple servers.</p><p>If one server crashes, the Load Balancer automatically redirects traffic to another healthy server.</p><p><strong><em>But how does a Load Balancer decide which server should handle the next request?</em></strong></p><p>It uses <strong>Load Balancing algorithms</strong>, such as:</p><ol><li><strong>Round Robin</strong> <strong>→</strong> Requests are sent to servers sequentially, one after another in a loop.</li><li><strong>Least Connections</strong> <strong>→</strong> Requests are sent to the server with the fewest active connections.</li><li><strong>and IP Hashing</strong> <strong>→</strong> Requests from the same IP address always go to the <strong>same server</strong>, which helps with <strong>session consistency.</strong></li></ol><p>So far, we’ve talked about scaling our application servers, but as traffic grows, the volume of data also increases.</p><p>At first, we can scale a database <strong>vertically</strong> by adding more CPU, RAM, and storage, but there’s a limit to how much a single machine can handle.</p><p>So, let’s explore other <strong>database scaling techniques</strong> that help manage large volumes of data efficiently.</p><h3>15. Database Indexing</h3><p>One of the quickest and most effective ways to speed up database <strong>read queries</strong> is <strong>indexing</strong>.</p><p>Think of it like the index page at the back of a book — instead of flipping through every page, you jump directly to the relevant section.</p><p>A <strong>database index</strong> works the same way. It’s is a super-efficient lookup table that helps the database quickly locate the required data without scanning the entire table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GT4KHdwXj5BONeAYn_XLwQ.png" /></figure><p>An index stores column values along with pointers to the actual data rows in the table.</p><p>Indexes are typically created on <strong>columns that are frequently queried</strong>, such as:</p><ul><li>Primary keys</li><li>Foreign keys</li><li>Columns used in WHERE conditions</li></ul><p>But be careful — while indexes <strong>speed up reads, they slow down writes</strong> (INSERT, UPDATE, DELETE) since the index needs to be updated whenever data changes.</p><p>That’s why we should <strong>only index the most frequently accessed columns</strong>.</p><p>Indexing significantly improves <strong>read performance</strong>, but <strong><em>what if even indexing isn’t enough, and our database can’t handle the growing number of read requests?</em></strong></p><p>That’s where our next database scaling technique <strong>Replication</strong> comes in.</p><h3>16. Replication</h3><p>Just like we added more application servers to handle traffic, we can scale our database by creating <strong>copies </strong>of it across multiple servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*jOevbx5ioYZsHlaO3DqJ0Q.png" /></figure><p>Here’s how it works:</p><ul><li>We have <strong>one primary database</strong> (also called the <strong>Primary Replica</strong>) that handles all <strong>write operations</strong> (INSERT, UPDATE, DELETE).</li><li>We have <strong>multiple read replicas</strong> that handle <strong>read queries (SELECT)</strong>.</li><li>Whenever data is written to the primary database, it gets copied to the read replicas so that they stay in sync.</li></ul><p>Replication improves the read performance since read requests are spread across multiple replicas, reducing the load on each one.</p><p>This also improves availability since if the primary replica fails, a read replica can take over as the new primary.</p><p>Replication is great for scaling read heavy applications, <strong><em>but what if we need to scale writes or store huge amounts of data?</em></strong></p><h3>17. Sharding</h3><p>Let’s say our service now has <strong>millions of users, </strong>and our database has grown to<strong> terabytes of data</strong>.</p><p>A <strong>single database server</strong> will eventually struggle to handle all this data efficiently.</p><p>Instead of keeping everything in one place, we <strong>split the database into smaller, more manageable pieces</strong> and distribute them across <strong>multiple servers</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kIlkVlvJyPtfl-AU8-Ftgg.png" /></figure><p>This technique is called <strong>Sharding</strong>.</p><ul><li>We divide the database into smaller parts called <strong>shards</strong>.</li><li>Each <strong>shard contains a subset</strong> of the total data.</li><li>Data is distributed based on a <strong>sharding key</strong> (e.g., <strong>user ID</strong>).</li></ul><p>By distributing data this way, we:</p><ul><li><strong>Reduce database load</strong> → Each shard handles <strong>only a portion</strong> of queries.</li><li><strong>Speed up read and write performance</strong> → Queries are distributed across multiple shards instead of hitting a single database.</li></ul><p>Sharding is also referred to as horizontal partitioning since it splits data by rows.</p><p><strong><em>But what if the issue isn’t the number of rows, but rather the number of columns?</em></strong></p><p>In such cases, we use <strong>Vertical Partitioning</strong>, where we <strong>split the database by columns</strong>. Let’s explore that next.</p><h3>18. Vertical Partitioning</h3><p>Imagine we have a <strong>User table</strong> that stores:</p><ul><li>profile details (name, email, profile picture)</li><li>login history (last_login, IP addresses)</li><li>and billing information (billing address, payment details)</li></ul><p>As this table <strong>grows</strong>, queries become <strong>slower</strong> because the database must scan <strong>many columns</strong> even when a request only needs a <strong>few specific fields.</strong></p><p>To optimize this, we use <strong>Vertical Partitioning where we split user table into smaller, more focused tables</strong> based on usage patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BISlpMgn71gSAgMQ-yf9mg.png" /></figure><ul><li><strong>User_Profile</strong> → Stores name, email, profile picture.</li><li><strong>User_Login</strong> → Stores login timestamps.</li><li><strong>User_Billing</strong> → Stores billing address, payment details.</li></ul><p>This improves <strong>query performance</strong> since each request only scans <strong>relevant columns</strong> instead of the entire <strong>table</strong>.</p><p>It reduces <strong>unnecessary disk I/O</strong>, making data retrieval quicker.</p><p>However, no matter how much we optimize the database, <strong>retrieving data from disk is always slower than retrieving from memory</strong>.</p><p><strong><em>What if we could store frequently accessed data in memory for lightning-fast access?</em></strong></p><p>This is called <strong>caching</strong>.</p><h3>19. Caching</h3><p><strong>Caching</strong> is used to optimize the performance of a system by <strong>storing frequently accessed data in memory</strong> instead of repeatedly fetching it from the database.</p><p>One of the most common caching strategies is the <strong>Cache Aside Pattern.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iEsmiK2Q7osHgE8ae8V_KQ.png" /></figure><p>Here’s how it works:</p><ol><li>When a user <strong>requests a data</strong>, the application first check the <strong>cache</strong>.</li><li>If the data is <strong>in the cache</strong>, it’s returned <strong>instantly, avoiding a database call</strong>.</li><li>If the data <strong>is not in the cache</strong>, the application <strong>retrieves it from the database</strong>, stores it in the cache for future requests, and returns it to the user.</li><li>Next time, the same data is requested, it’s served <strong>directly from cache</strong>, making the request much faster.</li></ol><p>To prevent outdated data from being served, we use <strong>Time-to-Live (TTL)</strong> — an expiration time set on cached data so it gets <strong>automatically refreshed</strong> after a certain period.</p><blockquote><em>Popular caching tools include </em><strong><em>Redis and Memcached.</em></strong></blockquote><p>Lets look at the next database scaling technique.</p><h3>20. Denormalization</h3><p>Most relational databases use <strong>Normalization</strong> to store data efficiently by breaking it into separate tables.</p><p>For example, in an e-commerce system:</p><ul><li>The <strong>Users table</strong> stores user details.</li><li>The <strong>Orders table</strong> stores their orders.</li><li>The <strong>Products table</strong> stores product details.</li></ul><p>While this <strong>reduces redundancy</strong>, it also <strong>introduces joins. </strong>When retrieving data from multiple tables, the database must <strong>combine them using JOIN operations</strong>, which can slow down queries as the dataset grows.</p><pre>SELECT o.order_id, u.name, u.email, o.product, o.amount<br>FROM orders o<br>JOIN users u ON o.user_id = u.user_id;</pre><p><strong>Denormalization</strong> reduces the number of joins by combining related data into a single table, even if it means some data gets duplicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*34aJ3UmcXsGaXfwhYvFObw.png" /></figure><p><strong>Example:</strong> Instead of keeping Users<strong> </strong>and<strong> </strong>Orders in separate tables, we create UserOrders table that stores user details along with their latest orders.</p><p>Now, when retrieving a user’s order history, we don’t need a <strong>JOIN operation</strong> — the data is already stored together leading to faster queries and better read performance.</p><pre>SELECT order_id, user_name AS name, user_email AS email, product, amount<br>FROM orders;</pre><p>Denormalization is often used in read-heavy applications where speed is more critical but the downside is it leads to <strong>increases storage usage</strong> and more <strong>complex update requests</strong>.</p><h3>21. CAP Theorem</h3><p>As we scale our system across multiple <strong>servers, databases, and data centers</strong>, we enter the world of <strong>distributed systems</strong>.</p><p>One of the fundamental principles of distributed systems is the <strong>CAP Theorem</strong>, which states that: No distributed system can achieve all three of the following at the same time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dxoITBXD_EJlX3yE.png" /></figure><ul><li><strong>Consistency ©</strong> <strong>→</strong> Every node always returns the <strong>most recent data</strong>.</li><li><strong>Availability (A)</strong> <strong>→</strong> The system <strong>always responds</strong> to requests, even if some nodes are down (but the data may not be the latest).</li><li><strong>Partition Tolerance (P)</strong> <strong>→</strong> The system continues operating even if there’s a <strong>network failure</strong> between nodes.</li></ul><p>Since <strong>network failures (P) are inevitable</strong>, we must choose between:</p><ul><li><strong>Consistency + Partition Tolerance (CP)</strong> → Ensures every request gets the latest data but may reject requests during failures. Example: <strong>SQL databases like MySQL</strong>.</li><li><strong>Availability + Partition Tolerance (AP)</strong> → Ensures the system always responds, even if some data is stale. Example: <strong>NoSQL databases like Cassandra and DynamoDB</strong>.</li></ul><p>In distributed NoSQL databases, achieving <strong>instant consistency</strong> across all servers <strong>is too slow</strong>.</p><p>Instead, we use <strong>Eventual Consistency</strong> — which means:</p><ul><li><strong>Not all nodes are updated instantly</strong>, but given enough time, they <strong>eventually</strong> sync and return the same data.</li><li>This allows the system to remain <strong>highly available and fast</strong>, even under extreme loads.</li></ul><p><strong>How Eventual Consistency Works:</strong></p><ol><li>A user updates data in <strong>one replica</strong> of the database.</li><li>The system <strong>immediately</strong> acknowledges the update, ensuring <strong>high availability</strong>.</li><li>The update is then <strong>propagated asynchronously to other replicas</strong>.</li><li>After a short delay, all replicas have the latest data, ensuring <strong>consistency over time</strong>.</li></ol><h3>22. Blob Storage</h3><p>Most modern applications don’t just store text records, they also need to handle <strong>images</strong>, <strong>videos</strong>, <strong>pdfs</strong> and other <strong>large files</strong>.</p><p>But here’s the problem: <strong>Traditional databases are not designed to store large, unstructured files efficiently</strong>.</p><p><strong><em>So, what’s the solution?</em></strong></p><p>We use <strong>Blob Storage like Amazon S3</strong> — a <strong>highly scalable and cost-effective</strong> way to store <strong>large, unstructured files</strong> in the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/0*KkEXy4QxAkUG7n13.png" /></figure><p>Blobs are the individual files like images, videos or documents.</p><p>These blobs are stored inside <strong>logical containers or buckets</strong> in the cloud.</p><p>Each file gets a <strong>unique URL</strong>, making it easy to retrieve and serve over the web.</p><blockquote><strong><em>Example:</em></strong><em> </em><a href="https://my-bucket-name.s3.amazonaws.com/videos/tutorial.mp4"><em>https://my-bucket-name.s3.amazonaws.com/videos/tutorial.mp4</em></a></blockquote><p>There are several advantages with using blob storage like:</p><ul><li><strong>Scalability</strong> → It can store petabytes of data effortlessly.</li><li><strong>Pay-as-you-go pricing</strong> → You only pay for storage and retrieval that you actually use.</li><li><strong>Automatic replication</strong> → Data is copied across multiple data centers and availability zones for durability.</li><li><strong>Easy access</strong> → Files can be retrieved using REST APIs or direct URLs.</li></ul><p>A common use case is to stream audio or video files to user application in real-time.</p><p>But streaming directly from blob-storage can be slow, especially if the data is stored in a <strong>distant location</strong>.</p><h3>23. CDN</h3><p>For example, imagine you’re in <strong>India</strong> trying to watch a YouTube video that’s hosted on a server in <strong>California</strong>.</p><p>Since the video data has to <strong>travel across the world</strong>, this could lead to <strong>buffering and slow load times</strong>.</p><p>A <strong>Content Delivery Network (or CDN)</strong> solves this problem by <strong>delivering content faster</strong> to users based on their location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZgkadrIoiNtaKMzd.png" /></figure><p>source: <a href="https://www.cloudflare.com/learning/cdn/what-is-a-cdn/">https://www.cloudflare.com/learning/cdn/what-is-a-cdn/</a></p><p>A CDN is a global network of distributed servers that work together to deliver <strong>web content</strong> (like HTML pages, JavaScript files, stylesheets, images, and videos) to users based on their <strong>geographic location</strong>.</p><p>Instead of serving content from a <strong>single data center</strong>, a CDN <strong>caches static contents</strong> on multiple <strong>edge servers located worldwide</strong>.</p><p>When a user requests content, the <strong>nearest CDN server</strong> delivers it instead of reaching all the way to the <strong>origin server</strong>.</p><p>Since content is served from the <strong>closest CDN node</strong>, users experience <strong>faster load times</strong> with minimal buffering.</p><h3>24. WebSockets</h3><p>Most web applications use <strong>HTTP</strong>, which follows a <strong>request-response</strong> model.</p><ol><li>The client sends a request.</li><li>The <strong>server</strong> processes the request and sends a response.</li><li>If the client needs new data, it must <strong>send another request</strong>.</li></ol><p>This works fine for <strong>static web pages</strong> but it’s <strong>too slow and inefficient for real-time applications</strong> like: live chat apps, stock market dashboards and online multiplayer games.</p><p>With HTTP, the only way to get real-time updates is through <strong>polling</strong> — sending repeated requests every few seconds.</p><p>But polling is inefficient because it increases server load and wastes bandwidth, as <strong>most responses are empty</strong> (when there’s no new data).</p><p>WebSockets solve this problem by allowing <strong>continuous, two-way communication</strong> between the <strong>client and server</strong> over a <strong>single persistent connection</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vl5LHv7vSmitfjK3kfd_Ng.png" /></figure><p>Here is how WebSockets work:</p><ol><li>The <strong>client initiates a WebSocket connection</strong> with the server.</li><li>Once established, the connection <strong>remains open</strong>.</li><li><strong>The server can push updates</strong> to the client <strong>at any time</strong>, without waiting for a request.</li><li>The client can also send messages <strong>instantly to the server.</strong></li></ol><p>This enables real-time interactions and eliminates the need for polling.</p><p>WebSockets enable real-time communication between a client and a server, <strong><em>but what if a server needs to notify another server when an event occurs?</em></strong></p><p>Example:</p><ul><li>When a user makes a payment, Stripe needs to notify your application <strong>instantly</strong>.</li><li>If someone pushes code to GitHub, a CI/CD system (e.g., Jenkins) should be triggered automatically.</li></ul><p>Enter Webhooks.</p><h3>25. Webhooks</h3><p>Instead of constantly <strong>polling an API</strong> to check if an event has occured, <strong>Webhooks</strong> allow a server to <strong>send an HTTP request</strong> to another server as soon as the event occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBIUfSyHfK99P6tkcg3wWw.png" /></figure><p>Here’s how it works:</p><ul><li>The <strong>receiver (your app)</strong> registers a webhook URL with the <strong>provider (e.g., Stripe, GitHub, Twilio)</strong>.</li><li>When an event occurs (e.g., user makes a payment), the <strong>provider sends an HTTP POST request to the webhook URL</strong> with event details.</li><li>Your app <strong>processes the incoming request</strong> and updates data accordingly.</li></ul><p>This saves server resources and reduces unnecessary API calls.</p><h3>26. Microservices</h3><p>Traditionally, applications were built using a <strong>monolithic architecture</strong>, where:</p><ul><li>All features (e.g., authentication, payments, orders, shipping) are inside <strong>one large codebase</strong>.</li><li>If one part of the system <strong>fails or needs scaling</strong>, the <strong>entire system</strong> is affected.</li><li><strong>Deployment is risky</strong> — one bad update can take down the entire app.</li></ul><p><strong>Example:</strong> Imagine an e-commerce app where the <strong>order</strong>, <strong>payment, inventory</strong>, and <strong>shipping </strong>modules are all tightly connected in a <strong>single codebase</strong>.</p><p>If the inventory system crashes, the entire app could go down.</p><p>Monoliths work <strong>fine for small applications</strong>, but for <strong>large-scale systems</strong>, they become <strong>hard to manage, scale, and deploy</strong>.</p><p>The solution is to break down your application into smaller, independent services called <strong>micro-services</strong> that work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WXBqwhaOM20oW93K.png" /></figure><p>Each <strong>microservice</strong>:</p><ol><li>Handles <strong>a single responsibility</strong></li><li>Has its <strong>own database and logic</strong>, so it can scale <strong>independently.</strong></li><li>Communicates with other microservices using <strong>APIs or message queues</strong>.</li></ol><p>This way services can be scaled and deployed individually without affecting the entire system.</p><p>However, when multiple microservices need to communicate, direct <strong>API calls</strong> aren’t always efficient — this is where <strong>Message Queues</strong> come in.</p><h3>27. Message Queues</h3><p>In a <strong>monolithic system</strong>, functions call each other <strong>directly</strong> and wait for a response.</p><p>But in a <strong>microservices-based system</strong>, this approach is inefficient because:</p><ul><li>If one service is <strong>slow</strong> or <strong>down</strong>, everything waits.</li><li>High traffic can <strong>overload</strong> a single service.</li><li><strong>Synchronous communication</strong> (waiting for immediate responses) doesn’t scale well.</li></ul><p>A <strong>Message Queue</strong> enables services to <strong>communicate asynchronously</strong>, allowing requests to be processed <strong>without blocking</strong> other operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uA9qeeaIgKRLN94bq3cKGQ.png" /></figure><p><strong>Here’s How It Works:</strong></p><ol><li>A <strong>producer</strong> (e.g., checkout service) <strong>places a message in the queue</strong> (e.g., “Process Payment”).</li><li>The <strong>queue temporarily holds the message</strong> until a <strong>consumer</strong> (e.g., payment service) is ready to process it.</li><li>The <strong>consumer retrieves the message</strong> and processes it.</li></ol><p>Using message queues, we can decouple services and improve the scalability and fault tolerance.</p><blockquote><em>Common message queue systems include: Apache Kafka, Amazon SQS and RabbitMQ.</em></blockquote><p>Using message queues, we can prevent overload on internal services within our system.</p><p><strong><em>But, how do we prevent overload for the public APIs and services we deploy.</em></strong></p><p>We use <strong>rate limiting</strong>.</p><h3>28. Rate Limiting</h3><p>Imagine a <strong>bot starts making thousands of requests per second</strong> to your website.</p><p>Without restrictions, this could:</p><ul><li><strong>Crash your servers</strong> by consuming all available resources.</li><li><strong>Increase cloud costs</strong> due to excessive API usage.</li><li>and <strong>degrade performance</strong> for legitimate users.</li></ul><p>Rate Limiting <strong>restricts the number of requests</strong> a client can send within a specific time frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79a798g6MhHy5YdMtWutjw.png" /></figure><p><strong>Here’s How It Works:</strong></p><ol><li>Every user or IP address is assigned a <strong>request quota</strong> (e.g., <strong>100 requests per minute</strong>).</li><li>If they <strong>exceed this limit</strong>, the server <strong>blocks additional requests temporarily</strong> and <strong>returns an error (HTTP 429 — Too Many Requests)</strong>.</li></ol><p>There are various rate limiting algorithms. Some of the popular ones are:</p><ul><li><strong>Fixed Window</strong> → Limits requests based on a fixed time window (e.g., 100 requests per minute).</li><li><strong>Sliding Window</strong> → More flexible version that dynamically adjusts limits to smooth out request bursts.</li><li><strong>Token Bucket</strong> → Users get <strong>tokens</strong> for requests, which <strong>replenish over time</strong> at a fixed rate.</li></ul><p>We don’t need to implement our own rate limiting system — this can be handled by something called an <strong>API gateway</strong>.</p><h3>29. API Gateways</h3><p>An API Gateway is a <strong>centralized service</strong> that handles <strong>authentication, rate limiting, logging and monitoring, and request routing</strong>.</p><p>Imagine a <strong>microservices-based application</strong> with multiple services.</p><p>Instead of exposing each service <strong>directly</strong>, an <strong>API Gateway</strong> acts as a <strong>single entry point</strong> for all client requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2E8_SuXNUReGWKgkM-ccA.png" /></figure><p><strong>How API Gateways Work:</strong></p><ol><li>The <strong>client sends a request</strong> to the API Gateway.</li><li>The Gateway <strong>validates the request</strong> (e.g., authentication, rate limits).</li><li>It <strong>routes the request</strong> to the appropriate micro-service.</li><li>The response is sent back <strong>through the Gateway</strong> to the client.</li></ol><p>API gateway simplifies API management and improves scalability and security.</p><blockquote><em>Popular API Gateway solutions include </em><strong><em>NGINX, Kong and AWS API Gateway.</em></strong></blockquote><h3>30. Idempotency</h3><p>In distributed systems, <strong>network failures</strong> and <strong>service retries</strong> are common. If a user <strong>accidentally refreshes a payment page</strong>, the system might receive <strong>two payment requests</strong> instead of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fhP9oihG63dY5x2VfP1djw.png" /></figure><p>Idempotency ensures that <strong>repeated requests</strong> produce the <strong>same result</strong> as if the request was made <strong>only once</strong>.</p><p><strong>Here’s how it works:</strong></p><ol><li>Each request is assigned a <strong>unique ID</strong> (e.g., request_1234).</li><li>Before processing, the system checks if the request <strong>has already been handled</strong>.</li><li>If yes → It <strong>ignores the duplicate request</strong>.</li><li>If no → It <strong>processes the request normally</strong>.</li></ol><p>Idempotency prevents duplicate transactions and ensures data consistency in distributed systems.</p><p>Thank you so much for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2928dd7f7799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Systems — A Deep Dive]]></title>
            <link>https://medium.com/@praveenmudalgeri05/distributed-systems-a-deep-dive-3a9e04ab391c?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a9e04ab391c</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Wed, 17 Dec 2025 03:20:37 GMT</pubDate>
            <atom:updated>2025-12-17T03:20:37.349Z</atom:updated>
            <content:encoded><![CDATA[<h3>Distributed Systems — A Deep Dive</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*n4ICw4gM9SIEMpDPQ20r4Q.png" /></figure><p>Distributed systems aren’t just a concept; they’re the invisible machinery holding our digital world together.</p><p>Every time you send a message, book a taxi, or stream a song, dozens of nodes across the world work together to make it look simple. But simplicity on the surface hides chaos underneath.</p><p>We’ll see how these systems serve millions globally while maintaining control over the chaos.</p><p>But first, let’s understand why we need distributed systems…</p><h3>Why Distributed Systems?</h3><p>Traditionally, software systems ran on a single machine.</p><p>But as the internet became popular, we realized a single machine might be insufficient at scale. That’s how the idea of vertical and horizontal scaling came in…</p><h3>1. Vertical Scaling</h3><p>A simple way to scale is by adding:</p><ul><li>more CPU,</li><li>more memory,</li><li>faster processors.</li></ul><p>However, it’s impossible to scale vertically forever because of hardware limitations &amp; cost effectiveness.</p><h3>2. Horizontal Scaling</h3><p>Instead of relying on one powerful server,</p><blockquote>“What if many small servers shared the work?”</blockquote><p>And those servers coordinate to act as one unit. That’s the key idea behind a DISTRIBUTED SYSTEM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aHcKAIDVDjxfQnv8.png" /></figure><h3>What are Distributed Systems?</h3><p>When you hear “distributed systems”, you might think of buzzwords like clusters, microservices, or Kubernetes…</p><p>But the core idea is actually simple:</p><blockquote><em>A distributed system is a group of independent servers that work together and appear to the user as one system.</em></blockquote><p>Each server has its own CPU, memory, and processes.</p><p>They communicate over a network to achieve one shared goal. There’s no shared RAM, no shared global clock, and no single machine that “knows everything.”</p><p>These features define distributed systems:</p><p><strong>No Shared Memory</strong></p><ul><li>Each node works independently and can’t directly read another machine’s variables.</li><li>All communication happens through messages sent over the network.</li><li>We need idempotency<a href="https://substack.com/home/post/p-178294033#footnote-1-178294033">1</a>, retry logic, and consensus algorithms<a href="https://substack.com/home/post/p-178294033#footnote-2-178294033">2</a> to maintain system correctness even when messages are lost or fail.</li></ul><p><strong>No Global Clock</strong></p><ul><li>There is no universal time across machines.</li><li>Clocks drift, and network delays make timing unpredictable.</li><li>This makes it difficult to determine the exact order in which events occurred.</li></ul><p>To solve this, we use <a href="https://aeron.io/docs/distributed-systems-basics/logical-clocks/">logical time</a>:</p><p><em>Lamport Clocks</em></p><blockquote><em>If timestamp </em>a<em> is less than </em>b<em>, then event </em>a<em> happened before event </em>b<em> in logical order.</em></blockquote><p><em>Vector Clocks</em></p><blockquote><em>Extend Lamport clocks by tracking event counts from every node, allowing systems to detect concurrency and understand ordering across many processes.</em></blockquote><p><strong>Message Communication</strong></p><ul><li>Nodes communicate using protocols such as <a href="https://grpc.io/">gRPC</a><a href="https://substack.com/home/post/p-178294033#footnote-3-178294033">3</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Session">HTTP</a><a href="https://substack.com/home/post/p-178294033#footnote-4-178294033">4</a>, <a href="https://kafka.apache.org/">Kafka events</a><a href="https://substack.com/home/post/p-178294033#footnote-5-178294033">5</a>, and so on.</li><li>Messages may not arrive, may arrive late, may be duplicated, or may even arrive out of order.</li><li>A distributed system stays correct not because communication is perfect, but because it handles failures gracefully.</li></ul><p>Let’s keep going…</p><h3>Distributed Systems in Real Life</h3><p>Distributed systems aren’t just theory — they’re everywhere, and in everything that we use:</p><ul><li><strong>Google Search</strong>: a massive network of crawlers, indexers, and ranking nodes running across many data centers.</li><li><strong>Netflix</strong>: region-wide services for streaming, recommendations, authentication, video transcoding, and content delivery.</li><li><strong>DynamoDB &amp; Cassandra</strong>: distributed key–value stores that replicate data across nodes for availability and scale.</li><li><strong>Kafka</strong>: distributed, fault-tolerant event log for asynchronous communication.</li><li><strong>Stripe</strong>: event-driven architectures built on reliable messaging and idempotent operations.</li></ul><h3>Distributed vs Decentralized vs Parallel Systems</h3><p>These three terms sound similar, but they describe different designs…</p><h3>1. Distributed Systems</h3><p>A distributed system is a group of independent nodes that work together to present themselves as a single, logical system.</p><p>Examples:</p><ul><li><a href="https://cloud.google.com/spanner">Google Spanner</a> (globally distributed database)</li><li><a href="https://kafka.apache.org/">Apache Kafka</a> (distributed log system)</li></ul><p>There is often a “leader or a control plane” responsible for managing consistency, replication, and communication between nodes.</p><h3>2. Decentralized Systems</h3><p>A decentralized system removes or minimizes the need for a single point of control.</p><p>Examples:</p><ul><li>Bitcoin and Ethereum (blockchain networks)</li><li>BitTorrent (peer-to-peer file sharing)</li></ul><p>Each node operates more autonomously, and coordination occurs through peer-to-peer consensus, not central orchestration.</p><h3>3. Parallel Systems</h3><p>They run multiple computations simultaneously, but on a single machine or a tightly coupled cluster with shared memory.</p><p>They typically have:</p><ul><li>One global clock,</li><li>A shared memory space.</li></ul><p><strong>Examples:</strong></p><ul><li>GPU workloads</li><li>Multithreaded programs</li><li>High-performance computing clusters</li></ul><h3>TL;DR</h3><ul><li>Distributed systems: loosely coupled, message-passing, reliability-oriented</li><li>Decentralized systems: trustless, peer-based, no central authority</li><li>Parallel systems: tightly coupled, shared memory, performance-oriented</li></ul><p>If many machines need to work together, they must communicate and share state efficiently.</p><p>The next question then is:</p><blockquote>“How do machines communicate effectively at scale?”</blockquote><h3>Communication in Distributed Systems</h3><p>When machines communicate in a distributed system, all interactions occur over a network. And networks come with their own set of messy realities.</p><p>Onward.</p><h3>1. Network Realities</h3><p>By default, the network is unreliable… Messages can:</p><ul><li>Be lost: never reach the destination</li><li>Be duplicated: delivered more than once</li><li>Be corrupted: bits flipped during transit</li><li>Arrive out of order: later messages come before earlier ones</li><li>Experience latency: unpredictable delays</li></ul><p>Designing a distributed system is really about expecting the WORST and still making the system work.</p><p>As Murphy’s law states:</p><blockquote>“If something can go wrong, it will go wrong.”</blockquote><h3>2. TCP: Reliability Layer</h3><p>Distributed systems rely heavily on TCP<a href="https://substack.com/home/post/p-178294033#footnote-6-178294033">6</a> because it provides:</p><ul><li>Reliable delivery: data gets delivered reliably, or the sender is notified of failure</li><li>Ordering: packets arrive in the order they were sent</li><li>Error checking: corrupted packets are detected and retransmitted</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*btxEyV4rRgXKAjH4.png" /></figure><p>TCP establishes this connection via a 3-way handshake:</p><ol><li>SYN: Client says, <em>“I want to connect.”</em></li><li>SYN-ACK: Server replies, <em>“Got it, ready to proceed.”</em></li><li>ACK: Client confirms, <em>“Great, let’s begin.”</em></li></ol><h3>3. TLS: Security Layer</h3><p>You can consider TCP as a “mailman” and TLS<a href="https://substack.com/home/post/p-178294033#footnote-7-178294033">7</a> as the envelope with a lock and signature. TLS ensures:</p><ul><li>Encryption<a href="https://substack.com/home/post/p-178294033#footnote-8-178294033">8</a>: nobody can read the data</li><li>Authentication: you know who you’re talking to</li><li>Integrity: messages can’t be altered without detection</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Zr0fqwo0urWgqgC.png" /></figure><p>During the TLS handshake, both sides:</p><ul><li>Choose cipher suites</li><li>Agree on the TLS version</li><li>Verify the server’s TLS certificate</li><li>Generate session keys for encrypted communication</li></ul><p>Reliability loses its meaning if someone can intercept or tamper with the data. In modern systems, every RPC and network call should use TLS.</p><blockquote><em>If servers spread worldwide, how do they find each other?</em></blockquote><h3>4. DNS: Finding the Right Node</h3><p>That’s where DNS (Domain Name System) comes in:</p><ul><li>It resolves human-readable names to IP addresses.</li><li>And acts as a basic service discovery<a href="https://substack.com/home/post/p-178294033#footnote-9-178294033">9</a> mechanism.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LKeRgSehjNu-rysZ.png" /></figure><p>In large distributed systems, DNS is usually layered with service registries or load balancers. But at the core, you always need a way to translate a name into a network endpoint.</p><p>Communication alone is not enough… For a distributed system to function as a single, coherent system, the nodes must also coordinate their actions.</p><p>And that brings us to the next major challenge: coordination.</p><h3>Coordination Challenges in Distributed Systems</h3><p>Once machines start working together, they need to stay in sync. Yet coordinating many nodes creates new problems.</p><p>Let’s look at the key challenges:</p><h3>1. Failure Detection</h3><p>Determining whether a machine has failed may seem simple… but it’s not.</p><blockquote><em>“If a node doesn’t respond, is it dead, or just slow?”</em></blockquote><p>Distributed systems usually detect failures using these two ways:</p><ul><li>Heartbeat mechanism</li><li>Heartbeat messages are small signals that nodes send to each other to show they are still alive.</li><li>These messages serve as health checks, enabling each node to determine whether its peers are functioning properly or have failed.</li><li>Gossip protocol</li><li>Each node regularly sends a small “I am alive” message to others.</li><li>System considers the node dead if these messages stop for too long.</li><li>Like real gossip, nodes share what they know with others.</li></ul><p>These techniques help to detect failures in a noisy, unreliable network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*efWhZtZzqV4YK8Uz.gif" /></figure><p>But you’ve got to find a balance between speed and accuracy:</p><ul><li>Check often → false positives (you think a node failed when it hasn’t)</li><li>Check slowly → system reacts late to real failures</li></ul><blockquote><em>If machines are spread around the world, it becomes hard to know which event happened first between two different servers.</em></blockquote><p>So let’s read the solution to this problem:</p><h3>2. Event Ordering and Timing</h3><p>If we use time.Now() to decide “which event happened when,” we run into issues:</p><p>Each machine has its own local clock… and these clocks can drift<a href="https://substack.com/home/post/p-178294033#footnote-10-178294033">10</a>.</p><p>Because of this, it is almost impossible to keep all clocks perfectly in sync across many machines.</p><p>In distributed systems, we rarely care about the exact timestamp. What we really need to know is which event happened first.</p><p>To solve this, we use algorithms called “logical clocks”.</p><h4>Lamport Clocks</h4><p>Lamport clocks use a simple counter on each machine:</p><ul><li>Each process starts with a counter at 0</li><li>For every local event, increase the counter: LC = LC + 1</li><li>When sending a message, attach the current counter value</li><li>When receiving a message, update the counter to:<br>LC = max(local LC, received LC) + 1</li></ul><p>Example:</p><p>Consider two machines — M1 and M2.</p><ul><li>M1 sends a message with LC = 5</li><li>M2 receives it when its own clock is LC = 3</li><li>So M2 updates its clock to max(3, 5) + 1 = 6</li></ul><p>From the above interactions:</p><ul><li>Send event has LC = 5</li><li>Receive event has LC = 6</li></ul><p>This keeps the correct order<a href="https://substack.com/home/post/p-178294033#footnote-11-178294033">11</a>: send &lt; receive.</p><p>It tells you what happened before what… but not if two events were independent… so it cannot detect concurrency.</p><p>Let’s understand why:</p><ul><li>M1 has LC = 5 and does event A → LC(A) = 6</li><li>M2 has LC = 2 and does event B → LC(B) = 3</li></ul><p>These two events are independent; there is no communication between the machines.</p><p>But Lamport clocks say 3 &lt; 6, so it looks like B happened before A, even though they were concurrent. So Lamport clocks only show ordering, not concurrency.</p><p>Vector clocks fix this problem!</p><h4>Vector Clocks</h4><p>They extend the idea on which Lamport’s clocks are based:</p><p>“What if every machine knows every other machine’s order?”</p><p>If there are N machines, the vector keeps N entries, and each entry tracks that machine’s event count, M[i].</p><p>This means:</p><ul><li>Every event carries a snapshot of what that machine knows about others’ state.</li><li>When a machine receives a message, it merges the knowledge using element-wise maximum.</li></ul><p>This gives each machine a partial history of the entire system.</p><p>With vector clocks, a system can tell:</p><ul><li>Which event happened first,</li><li>Whether two events happened at the same time (concurrently).</li></ul><p>In a distributed system, we need a single source of truth for all operations… so it helps to have one machine act as the leader.</p><p>Let’s dive in!</p><h3>3. Leader Election</h3><p>The leader becomes the source of truth, and the other machines (followers) replicate its state.</p><p>But if all servers share the same design, how can we determine which server qualifies to become the “leader”?</p><p>Algorithms like Raft<a href="https://substack.com/home/post/p-178294033#footnote-12-178294033">12</a> solve this in a clean and predictable way:</p><ul><li>All nodes start as followers.</li><li>If no leader is found, the followers enter a candidate state to become the leader.</li><li>Voting occurs, and the node that gets the most votes becomes the leader.</li></ul><p>Once elected, the leader coordinates updates, and followers replicate its log.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_fNUHBwYdixgSl5m.png" /></figure><p>Raft is popular because it’s simple, has well-defined states (follower, candidate, leader), and provides strong safety guarantees. It’s used in many distributed databases and consensus systems.</p><h3>4. Data Replication and Consistency</h3><p>Maintaining data consistency across many machines is one of the most challenging problems in distributed systems.</p><p>Because data is stored on different nodes, every write must be replicated to other nodes to ensure consistency.</p><p>But not all systems need the same level of strictness… different consistency models exist depending on what the application needs:</p><p><strong>Linearizability</strong></p><ul><li>Every operation looks instant and globally ordered.</li><li>Readers always see the latest write.</li><li>This is the strictest model and is expensive at scale because every operation requires coordination across machines.</li><li>Plus, it often adds latency.</li></ul><p><strong>Sequential Consistency</strong></p><ul><li>Each node’s operations appear in order, but different nodes may see different global orders.</li><li>It’s easier to achieve than linearizability.</li><li>Writes can be done locally first and then replicated asynchronously to others.</li></ul><p><strong>Eventual Consistency</strong></p><ul><li>If no new writes occur, all replicas will eventually converge on the same value.</li><li>Used in high-availability systems like DynamoDB and Cassandra.</li><li>You trade immediate correctness for higher availability and better partition tolerance.</li></ul><h4>CAP Theorem</h4><p>These consistency choices connect directly to the CAP theorem, which says a distributed system can only provide two out of three at the same time:</p><ul><li>Consistency ©: Every read returns the latest write (or an error).</li><li>Availability (A): Every request gets a response (even if it’s outdated).</li><li>Partition Tolerance (P): The system keeps working even if the network breaks into parts.</li></ul><p>So why not all three?</p><p>Network partitions (P) are common in distributed systems. When they occur, the system must choose between:</p><ul><li>Consistency ©: Stop serving outdated data until the partition heals.</li><li>Availability (A): Serve whatever data is available, even if outdated.</li></ul><p>Thus, a system can be CP, AP, or CA, but never all three at the same time. (There is always a tradeoff.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a9e04ab391c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[50 System Design Concepts for Beginners in 90 Minutes [2026 Edition]]]></title>
            <link>https://medium.com/@praveenmudalgeri05/50-system-design-concepts-for-beginners-in-90-minutes-2026-edition-84f98c225fd0?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/84f98c225fd0</guid>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 15:04:57 GMT</pubDate>
            <atom:updated>2025-12-16T15:04:57.277Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3OOpH0ferYSNGXAt8KXcqg.png" /></figure><p>When you start learning system design, the hardest part is not the concepts themselves.</p><p>It is figuring out where to find clear explanations in one place.</p><p>That is why having a single guide that covers all the essentials is such a game-changer.</p><p>Therefore, we have designed this guide to cover 50 of the most important system design concepts.</p><p>Think of it as your one-stop reference for understanding how real systems scale, stay reliable, communicate, and handle data.</p><p>My goal is to walk you through fifty important ideas using short explanations and simple examples so everything clicks quickly.</p><h3>I. Core Architecture Principles</h3><h3>Vertical vs Horizontal Scaling</h3><ul><li><strong>Vertical scaling</strong> means upgrading a single machine, like adding more CPU, RAM, or faster storage.</li><li><strong>Horizontal scaling</strong> means adding more machines and spreading work across them.</li></ul><p>Vertical is easier but hits hardware limits and becomes expensive.</p><p>Horizontal is harder because you need load balancing, stateless services, and shared storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MZzIhlt7J4PEjOpK.jpeg" /></figure><p>Think of it this way: vertical is one superhero getting stronger, horizontal is building a team.</p><h3>CAP Theorem</h3><p><strong>CAP Theorem</strong> says that in the presence of a network partition, a distributed system must choose between <strong>Consistency</strong> and <strong>Availability</strong>. Consistency means every user sees the same data at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l0jOC4MtwBxyZmya.jpeg" /></figure><p>Availability means the system always responds, even if the data might be slightly stale.</p><p>You cannot have perfect consistency and perfect availability when your network is broken, so you decide which one to sacrifice for your use case.</p><h3>PACELC Theorem</h3><p><strong>PACELC</strong> extends CAP and says: if there is a Partition, choose Availability or Consistency; Else choose Latency or Consistency.</p><p>Even when the network is fine, you still trade off slow but consistent reads vs fast but eventually consistent reads. Systems that sync across regions often pay in latency to keep strong consistency.</p><p>It explains why some databases are fast but slightly stale, while others are slower but always accurate.</p><h3>ACID vs BASE</h3><p><strong>ACID</strong> is about strict, reliable transactions: Atomicity, Consistency, Isolation, Durability. It suits financial systems, inventory, and anything where mistakes are very costly.</p><p><strong>BASE</strong> stands for Basically Available, Soft state, Eventual consistency and is used in large distributed systems that need to stay up and respond quickly.</p><p>BASE systems might show temporary inconsistencies but fix themselves over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h-TxtZEV36wrllBX.png" /></figure><p>In practice, many architectures combine both, using ACID for core money flows and BASE for things like feeds and analytics.</p><h3>Throughput vs Latency</h3><ul><li><strong>Throughput</strong> is how many requests your system can handle per second.</li><li><strong>Latency</strong> is how long a single request takes from start to finish.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T68u-tPrifyrbxto.jpeg" /></figure><p>You can often increase throughput by doing more work in parallel, but that may increase latency if queues build up.</p><p>Think of a restaurant that takes many orders at once but makes customers wait longer. Good system design tries to balance both: enough throughput for peak load but low latency for a smooth user experience.</p><h3>Amdahl’s Law</h3><p><strong>Amdahl’s Law</strong> says that the speedup from parallelization is limited by the part that cannot be parallelized.</p><p>If 20 percent of your system is always sequential, no amount of extra machines will fix that bottleneck.</p><p>Let me break it down.</p><p>If your request always has to hit a single master database, that master will cap your performance. This law reminds you to hunt for bottlenecks instead of just adding more servers.</p><h3>Strong vs Eventual Consistency</h3><ul><li><strong>Strong consistency</strong> means all users see the same data immediately after a write.</li><li><strong>Eventual consistency</strong> means updates spread over time and nodes may briefly disagree.</li></ul><p>Strong consistency is easier to reason about but usually slower and less available under failures.</p><p>Eventual consistency is great for large-scale systems like timelines or counters where perfect freshness is not critical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pvDRZq-bBK9_22Nf.png" /></figure><p>The key is to choose the model that matches the user experience you need.</p><h3>Stateful vs Stateless Architecture</h3><ul><li>A <strong>stateful</strong> service remembers user context between requests, often storing session data locally.</li><li>A <strong>stateless</strong> service treats every request as new, relying on external stores like caches or databases for any state.</li></ul><p>Stateless services are easier to scale horizontally because any instance can handle any request.</p><p>Stateful systems can be simpler to code but harder to load balance and fail over.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CRLYIeJQkDFkOgSg.png" /></figure><p>In modern cloud systems, we try to push state into databases and keep services as stateless as possible.</p><h3>Microservices vs Monoliths</h3><p>A <strong>monolith</strong> is a single application that contains many features in one deployable unit.</p><p><strong>Microservices</strong> split features into separate services that communicate over the network.</p><p>Microservices help teams work independently and scale different parts separately, but introduce complexity around communication, debugging, and data consistency.</p><p>Monoliths are simpler to start with and often fine up to a certain scale. Here is the tricky part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bswgyjLsCQ3tVDIa.png" /></figure><p>Many great systems start as monoliths and gradually evolve into microservices when the pain is real.</p><h3>Serverless Architecture</h3><p><strong>Serverless</strong> lets you run small functions in the cloud without managing servers directly. You pay only when your code runs, and the platform handles scaling and infrastructure for you.</p><p>It is ideal for event-driven workloads such as webhooks, background jobs, or light APIs with spiky traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3xU_zumV-1GLvE1_.png" /></figure><p>The tradeoff is less control over long-running tasks, cold starts, and sometimes a higher cost at very high volumes.</p><p>Think of serverless as “functions as a service,” perfect for glue code and lightweight services.</p><h3>II. Networking and Communication</h3><h3>Load Balancing</h3><p><strong>Load balancing</strong> spreads incoming traffic across multiple servers so no single server gets overloaded. It improves both reliability and performance, since a single server’s failure does not bring down the entire system.</p><p>Load balancers can be hardware devices or software services. They often support health checks so they stop sending traffic to unhealthy instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*93AL2gWiF8CyP1ps.png" /></figure><p>From an interview point of view, they are your first building block when scaling horizontally.</p><h3>Load Balancing Algorithms</h3><p>Common <strong>load balancing algorithms</strong> include Round Robin, Least Connections, and IP Hash.</p><ul><li><strong>Round Robin</strong> cycles through servers in order and is simple to implement.</li><li><strong>Least Connections</strong> sends traffic to the server with the fewest active connections, which helps when requests vary in length.</li><li><strong>IP Hash</strong> uses a hash of the client IP so the same user usually goes to the same server, which helps with simple session stickiness.</li></ul><p>Picking the right algorithm affects fairness, resource usage, and user experience.</p><h3>Reverse Proxy vs Forward Proxy</h3><ul><li>A <strong>reverse proxy</strong> sits in front of servers and represents them to clients. It hides the inner topology, can do TLS termination, caching, compression, and routing.</li><li>A <strong>forward proxy</strong> sits in front of clients and represents them to the outside world, often for security, caching, or content filtering.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*tz_1CbPKxVXnnysV.png" /></figure><p>Think of a reverse proxy as the reception desk of a company that hides all the internal rooms, and a forward proxy as a gateway your laptop must pass through to reach the internet.</p><p>Knowing the difference helps when you talk about API gateways and corporate proxies.</p><h3>API Gateway</h3><p>An <strong>API gateway</strong> is a special reverse proxy that acts as a single entry point for all API calls in a microservices system. It handles routing to the right service, rate limiting, authentication, logging, and sometimes response shaping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rw5TBYy9AFfPwtLN.png" /></figure><p>This reduces complexity on the client side since clients only talk to one endpoint.</p><p>If you put too much logic in the gateway, it can become a bottleneck or a mini monolith of its own. Good designs keep it focused and thin.</p><h3>CDN (Content Delivery Network)</h3><p>A <strong>CDN</strong> is a network of geographically distributed servers that cache static content like images, videos, and scripts closer to users.</p><p>When a user requests content, they are routed to the nearest CDN node, which greatly reduces latency. This also offloads traffic from your origin servers, improving scalability and resilience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LF1zaHFzBxOLWW4M.png" /></figure><p>CDNs are essential for global applications and front-end performance.</p><p>Think of them as “local copies” of your website’s heavy files sprinkled around the world.</p><h3>DNS (Domain Name System)</h3><p><strong>DNS</strong> maps human readable domain names to IP addresses.</p><p>When you type a website name, your device queries DNS to find the numeric address of the server.</p><p>has multiple layers of caching, so responses are fast after the first lookup. It can also be used to perform simple load balancing by returning different IPs for the same name.</p><p>Understanding DNS helps you reason about why name changes take time to propagate and why some outages are caused by misconfigured DNS.</p><h3>TCP vs UDP</h3><ul><li><strong>TCP</strong> is a reliable, connection-oriented protocol. It guarantees ordered, error checked delivery by using acknowledgments and retries.</li><li><strong>UDP</strong> is connectionless and does not guarantee delivery or order, which makes it much faster and lighter.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9ttMA6UMd2mrZThq.png" /></figure><p>TCP suits APIs, web pages, and file transfers where accuracy matters.</p><p>UDP works well for real time applications like video calls or games where occasional packet loss is acceptable.</p><p>Think of TCP as registered mail and UDP as quick postcards.</p><h3>HTTP/2 and HTTP/3 (QUIC)</h3><ul><li><strong>HTTP/2</strong> introduced multiplexing, which lets multiple requests share a single TCP connection, reducing overhead. It also brought features like header compression and server push.</li><li><strong>HTTP/3</strong> runs over QUIC, which is built on UDP and improves connection setup time and performance on unreliable networks. These versions mainly aim to reduce latency and better use modern network conditions.</li></ul><p>For you as an engineer, the key idea is: fewer connection setups and better use of a single connection.</p><h3>gRPC vs REST</h3><ul><li><strong>REST</strong> typically uses HTTP with JSON and focuses on resources like /users or /orders. It is simple, human-readable, and widely used for public APIs.</li><li><strong>gRPC</strong> uses HTTP/2 and binary encoded messages (protobuf), which are smaller and faster over the wire. It also supports bidirectional streaming and strong typing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n66cE1UFpqlmXZVF.png" /></figure><p>In microservices, gRPC is often preferred for service-to-service calls, while REST is common for external clients.</p><p>Use REST when readability and compatibility matter, gRPC when performance and contracts matter.</p><h3>WebSocket and Server-Sent Events (SSE)</h3><p><strong>WebSockets</strong> create a full-duplex connection where client and server can send messages to each other at any time.</p><p><strong>SSE</strong> allows the server to push events to the client over a one way channel using HTTP.</p><p>WebSockets are great for chats, multiplayer games, and live collaboration.</p><p>SSE is simpler and fits cases like live score updates or notifications where only the server needs to push updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8OcvWFmKbtF8dkRn.png" /></figure><p>Both solve real-time communication problems that plain HTTP cannot handle well.</p><h3>Long Polling</h3><p><strong>Long polling</strong> is a technique where the client sends a request and the server holds it open until there is new data or a timeout.</p><p>When the response comes back, the client immediately opens another request. This simulates real time updates over plain HTTP without special protocols.</p><p>It is less efficient than WebSockets but easier to implement and works through most proxies and firewalls.</p><p>Think of it as asking “anything new?” and waiting quietly until there is an answer.</p><h3>Gossip Protocol</h3><p>A <strong>gossip protocol</strong> lets nodes in a distributed system share information by periodically talking to random peers.</p><p>Over time, information spreads like gossip in a social group until everyone has roughly the same view. It is used to share membership, health status, or configuration in a fault tolerant way.</p><p>The protocol is eventually consistent and does not rely on a central authority. This makes it ideal for large clusters where nodes frequently join and leave.</p><h3>III. Database and Storage Internals</h3><h3>Sharding (Data Partitioning)</h3><p><strong>Sharding</strong> splits data across multiple machines, each holding a subset of the data. Common strategies include range-based sharding, hash-based sharding, and directory-based sharding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4e_oW_yrdj4kjQVp.png" /></figure><p>The main goal is to scale storage and throughput by avoiding a single giant database node.</p><p>The tricky part is choosing a shard key that avoids hot spots where one shard gets most of the traffic. Once you shard, moving data between shards (resharding) becomes an important operational challenge.</p><h3>Replication Patterns (Master Slave, Master Master)</h3><p><strong>Replication</strong> means keeping multiple copies of data on different nodes.</p><ul><li>In <strong>master slave</strong> (or primary replica), one node handles writes and replicates changes to others that serve reads.</li><li>In <strong>master master</strong> (multi-primary), multiple nodes accept writes and reconcile conflicts.</li></ul><p>Replication improves read performance and availability, but makes consistency harder, especially when writes go to multiple nodes.</p><p>In interviews, expect to talk about how replication lag affects reads and how failover works when a master dies.</p><h3>Consistent Hashing</h3><p><strong>Consistent hashing</strong> is a technique to distribute keys across nodes in a way that minimizes data movement when nodes are added or removed.</p><p>Keys and nodes are placed on a logical ring, and each key belongs to the next node on the ring.</p><p>When a node joins or leaves, only a small portion of keys need to move. This property is very helpful in distributed caches and databases.</p><p>Think of it as a smooth mapping that does not get scrambled when the cluster size changes.</p><h3>Database Indexing (B Trees, LSM Trees)</h3><p><strong>Indexes</strong> speed up queries by organizing data in a way that allows fast lookup.</p><p>B Trees are balanced trees that keep data sorted and let you find ranges efficiently, common in relational databases.</p><p>LSM Trees batch writes in memory and periodically flush them to disk, which makes writes very fast but reads more complex.</p><p>The tradeoff is write heavy vs read heavy workloads.</p><p>The key idea is that indexes are a separate structure that must be updated on every write, which is why too many indexes hurt insert performance.</p><h3>Write Ahead Logging (WAL)</h3><p><strong>Write Ahead Logging</strong> records changes to a log before applying them to the main database.</p><p>If a crash happens in the middle of a transaction, the system can replay the log to restore a consistent state. WAL ensures durability and atomicity of transactions. It also allows techniques like replication from the log stream. Let me tell you why it is important.</p><p>Without WAL, a crash could leave your data in a half updated, corrupt state.</p><h3>Normalization vs Denormalization</h3><ul><li><strong>Normalization</strong> organizes data into tables that reduce redundancy and dependencies, following rules like first normal form, second normal form, and so on. This avoids anomalies on updates and inserts.</li><li><strong>Denormalization</strong> intentionally duplicates data to speed up reads and reduce joins. In high scale systems, denormalization is common for read heavy paths, such as storing user names along with posts instead of joining every time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WxrNFqDXwd3dDem4.png" /></figure><p>The real skill is knowing where you can safely denormalize without breaking consistency.</p><h3>Polyglot Persistence</h3><p><strong>Polyglot persistence</strong> means using multiple types of databases within the same system, each chosen for what it does best. You might use a relational database for transactions, a document store for logs, a key value store for caching, and a graph database for relationships.</p><p>Instead of forcing everything into one database, you pick the right tool for each job.</p><p>The tradeoff is more operational complexity and more knowledge required from the team.</p><h3>Bloom Filters</h3><p>A <strong>Bloom filter</strong> is a space efficient data structure that quickly answers “might this item be in the set?” with possible false positives but no false negatives. It uses multiple hash functions to set bits in a bit array when items are inserted.</p><p>To check membership, you test the same bits; if any bit is zero, the item is definitely not present.</p><p>Databases and caches use Bloom filters to avoid unnecessary disk lookups or cache misses.</p><p>Think of them as fast gatekeepers that say “definitely not” or “maybe.”</p><h3>Vector Databases</h3><p><strong>Vector databases</strong> store and query vectors, which are numeric representations of data such as text, images, or audio. These vectors come from models like embeddings and allow similarity search, such as “find documents most similar to this one.”</p><p>Instead of exact equality comparisons, they use distance metrics like cosine similarity or Euclidean distance. This is essential for modern search, recommendation, and AI assistant systems.</p><p>In interviews, it is enough to know that vector databases support nearest neighbor search over high-dimensional data.</p><h3>IV. Reliability and Fault Tolerance</h3><h3>Rate Limiting</h3><p><strong>Rate limiting</strong> controls how many requests a user, IP, or API key can make in a given time window. It protects your system from abuse, accidental traffic spikes, and runaway loops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g9kd_xzjuI3kRvTC.png" /></figure><p>Common strategies include fixed window, sliding window, and token bucket.</p><p>Rate limits are often enforced at the API gateway or load balancer.</p><p>Think of them as safety brakes that keep shared resources from being overwhelmed.</p><h3>Circuit Breaker Pattern</h3><p>A <strong>circuit breaker</strong> monitors calls to a remote service and “opens” if there are too many failures.</p><p>When open, it immediately fails new requests instead of trying the broken service again.</p><p>After a cooldown period, it allows a few trial calls to see if the service has recovered and closes if they succeed. This pattern prevents cascading failures where one slow service drags down the entire system.</p><p>Here is the tricky part. Circuit breakers must be tuned carefully so they do not open too aggressively or too late.</p><h3>Bulkhead Pattern</h3><p>The <strong>bulkhead pattern</strong> isolates parts of a system so a failure in one area does not sink everything. This can mean separate connection pools, thread pools, or even entire service clusters for different features.</p><p>If one bulkhead is flooded with traffic, others keep working.</p><p>The name comes from ship bulkheads that contain flooding in one compartment.</p><p>In design discussions, using bulkheads shows you are thinking about fault isolation and blast radius.</p><h3>Retry Patterns and Exponential Backoff</h3><p><strong>Retries</strong> help recover from transient errors like network timeouts or temporary overload.</p><p><strong>Exponential backoff</strong> means each retry waits longer than the previous one, such as 1 second, 2 seconds, 4 seconds, and so on. This prevents your client from hammering a service that is already struggling.</p><p>Good retry policies also use jitter (small randomness) to avoid thundering herds.</p><p>Let me break it down.</p><p>Retries without backoff can make outages worse instead of helping.</p><h3>Idempotency</h3><p>An operation is <strong>idempotent</strong> if performing it multiple times has the same effect as performing it once.</p><p>For example, “set user status to active” is idempotent, while “increment account balance by 10” is not.</p><p>Idempotency is critical when systems use retries, because the same request may be sent more than once.</p><p>APIs often require idempotency keys on operations like payments to avoid double charging.</p><p>In interviews, always mention idempotency when you talk about at least once delivery or retries.</p><h3>Heartbeat</h3><p>A <strong>heartbeat</strong> is a periodic signal sent by a service or node to indicate that it is alive and healthy.</p><p>Monitoring systems or coordinators listen for heartbeats.</p><p>If they stop receiving them, they mark the node as down and trigger failover or scaling actions.</p><p>Heartbeats are simple but powerful tools for liveness detection. Think of them as the system’s “pulse checks.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RxHRHqsWp0jj5rGW.jpeg" /></figure><h3>Leader Election (Paxos, Raft)</h3><p><strong>Leader election</strong> is the process of choosing a single node to act as a coordinator among many.</p><p>Algorithms like Paxos and Raft ensure that only one leader is chosen and that all nodes eventually agree on who that leader is.</p><p>The leader handles tasks like assigning work, managing metadata, or ordering writes. If the leader fails, a new one is elected automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-jXXx5nBXBLrh-T8.png" /></figure><p>You do not need to memorize the math for interviews, but you should know that consensus algorithms power many critical systems like metadata stores and distributed logs.</p><h3>Distributed Transactions (SAGA Pattern)</h3><p>A <strong>distributed transaction</strong> spans multiple services or databases.</p><p>The <strong>SAGA pattern</strong> models such a transaction as a sequence of local steps with compensating actions for rollbacks.</p><p>Instead of locking everything like a single ACID transaction, each service performs its part and publishes an event. If something fails, compensating steps attempt to undo previous changes. This fits naturally with microservices and eventual consistency.</p><p>The tradeoff is more complex logic and the possibility of partial failures that must be handled gracefully.</p><h3>Two Phase Commit (2PC)</h3><p><strong>Two Phase Commit</strong> is a protocol that tries to provide atomic transactions across multiple nodes.</p><ul><li>In the first phase, the coordinator asks all participants if they can commit.</li><li>In the second phase, if everyone agrees, it tells them to commit; otherwise, it tells them to roll back.</li></ul><p>2PC provides strong guarantees but can block if the coordinator fails, and it is expensive at scale due to locking.</p><p>In modern cloud systems, 2PC is often avoided for high throughput paths and replaced by patterns like SAGA.</p><h3>V. Caching and Messaging</h3><h3>Caching</h3><p><strong>Caching</strong> stores frequently accessed data in a fast storage layer, usually memory, to reduce latency and backend load.</p><p>Common cache layers include in process caches, external key value stores, and CDNs. Caching is especially effective for read heavy workloads and expensive computations.</p><p>Here is the tricky part. Stale data and invalidation make caching harder than it first appears.</p><p>As the saying goes, cache invalidation is one of the hard problems in computer science.</p><h3>Caching Strategies (Cache Aside, Write Through, etc.)</h3><ul><li><strong>Cache aside</strong> means the application reads from the cache, and on a miss, loads from the database and writes to the cache.</li><li><strong>Write through</strong> writes to the cache and database at the same time, ensuring cache and source are always in sync.</li><li><strong>Write back</strong> writes to the cache first and flushes to the database later, which is fast but risky if the cache fails.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FMdrF-Vg5mP2wC-p.jpeg" /></figure><p>Each strategy balances freshness, complexity, and performance differently.</p><p>Interviewers love when you mention which strategy you would pick for a given scenario.</p><h3>Cache Eviction Policies (LRU, LFU)</h3><p><strong>Cache eviction policies</strong> decide which items to remove when the cache is full.</p><ul><li><strong>LRU</strong> (Least Recently Used) evicts items that have not been accessed recently, assuming recent items are more likely to be used again.</li><li><strong>LFU</strong> (Least Frequently Used) evicts items that are rarely accessed, focusing on long term popularity.</li></ul><p>Some systems use random, FIFO, or advanced algorithms.</p><p>The key idea is that cache space is limited, so you want to keep the most valuable items in memory.</p><h3>Message Queues (Point to Point)</h3><p>A <strong>message queue</strong> allows one component to send messages to another without needing both to be online at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ah_Xztt4p2R-h6S5.jpeg" /></figure><p>In a point to point model, messages in a queue are consumed by one receiver and then removed. This decouples sender and receiver so they can scale and fail independently.</p><p>Queues are great for background jobs, email sending, and processing heavy tasks asynchronously.</p><p>Think of them as a todo list shared between services.</p><h3>Pub Sub (Publish Subscribe)</h3><p>In <strong>pub sub</strong>, publishers send messages to topics, not directly to consumers.</p><p>Subscribers listen to topics they care about and receive copies of relevant messages. This enables broadcast style communication and loose coupling between producers and consumers.</p><p>Multiple services can react to the same event in different ways, such as logging, analytics, and notifications.</p><p>In interviews, pub sub often appears in event driven designs like activity feeds or event sourcing.</p><h3>Dead Letter Queues</h3><p>A <strong>dead letter queue</strong> stores messages that could not be processed successfully after several attempts.</p><p>Instead of retrying forever and blocking the main queue, these messages are moved aside.</p><p>Engineers can inspect the dead letter queue to debug issues, fix data, or replay messages later. This pattern improves resiliency and keeps your system from getting stuck on “poison messages.”</p><p>Think of it as a holding area for problematic jobs.</p><h3>VI. Observability and Security</h3><h3>Distributed Tracing</h3><p><strong>Distributed tracing</strong> tracks a single request as it flows through multiple services. Each service adds a trace ID and span information so you can reconstruct the full path of a request. This is extremely helpful when debugging slow responses or failures in microservice architectures.</p><p>Without tracing, you just see errors in isolation. With it, you see the whole story across services, queues, and databases.</p><h3>SLA vs SLO vs SLI</h3><p>An <strong>SLA</strong> (Service Level Agreement) is an external promise to customers, such as “99.9 percent uptime per month.”</p><p>An <strong>SLO</strong> (Service Level Objective) is an internal target that engineers aim to meet, usually stricter than the SLA. An <strong>SLI</strong> (Service Level Indicator) is the actual measured metric, like real uptimes or request success rates.</p><p>Think of SLA as the contract, SLO as the goal, and SLI as the scoreboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uOoPq_hc1h2imcet.png" /></figure><p>In interviews, using these terms correctly shows maturity in thinking about reliability.</p><h3>OAuth 2.0 and OIDC</h3><p><strong>OAuth 2.0</strong> is a framework for delegated authorization. It lets users grant an application limited access to their resources without sharing passwords.</p><p><strong>OIDC</strong> (OpenID Connect) builds on OAuth 2.0 to add authentication, letting clients verify who the user is and get user identity information. This is the basis of many “Login with X” flows.</p><p>The key idea is that an authorization server issues tokens that clients and APIs can trust.</p><h3>TLS/SSL Handshake</h3><p><strong>TLS/SSL</strong> secures communication between client and server by encrypting data in transit.</p><p>During the <strong>handshake</strong>, the client and server agree on encryption algorithms, exchange keys securely, and verify certificates.</p><p>Once the handshake completes, all subsequent data is encrypted and safe from eavesdropping. This is what puts the little lock icon in your browser.</p><p>Without TLS, anyone on the network could read or modify sensitive traffic.</p><h3>Zero Trust Security</h3><p><strong>Zero Trust</strong> is a security model that says: “Never trust, always verify.” It assumes that threats can exist both outside and inside the network.</p><p>Every request must be authenticated, authorized, and encrypted, even if it comes from within your data center or VPC. Access is granted based on identity, device posture, and context, not just on being “inside the firewall.”</p><p>In modern architectures, Zero Trust is becoming the default approach to secure system design.</p><h3>Reference</h3><ul><li><a href="https://bit.ly/3KFuqrC"><strong>Grokking the System Design Interview</strong></a></li></ul><h3>Key Takeaways</h3><ul><li>System design is mostly about understanding trade-offs: consistency vs. availability, latency vs. throughput, simplicity vs. flexibility.</li><li>Scaling is not just “add more servers.” You must think about load balancing, sharding, replication, and bottlenecks.</li><li>Reliability patterns like rate limiting, circuit breakers, retries, and bulkheads exist because failures are normal in distributed systems.</li><li>Caching, queues, and pub-sub are your best friends for performance and decoupling, but they introduce new challenges around consistency and ordering.</li><li>Observability and security concepts such as tracing, SLIs, OAuth, TLS, and Zero Trust are essential for systems that are not just fast but also safe and debuggable.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84f98c225fd0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Merge Two Sorted Lists]]></title>
            <link>https://medium.com/@praveenmudalgeri05/merge-two-sorted-lists-c91533650af3?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c91533650af3</guid>
            <category><![CDATA[leetcode]]></category>
            <category><![CDATA[leetcode-solution]]></category>
            <category><![CDATA[leetcode-easy]]></category>
            <category><![CDATA[merge-two-sorted-lists]]></category>
            <category><![CDATA[merging-two-sorted-lists]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Mon, 17 Nov 2025 16:08:41 GMT</pubDate>
            <atom:updated>2025-11-17T16:08:41.620Z</atom:updated>
            <content:encoded><![CDATA[<p><em>(Amazon Interview Question)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*vH117jAwOJbUS0-MG5Yhyg.jpeg" /><figcaption>Fig: Merge Two Sorted Lists</figcaption></figure><h4>Problem Overview</h4><blockquote>Difficulty: Easy</blockquote><blockquote>Leetcode Pattern: Linked Lists + Two Pointers</blockquote><p>You are given the heads of two sorted linked lists <strong>list1</strong> and <strong>list2</strong>.</p><p>Merge the two lists into one sorted list and return the head of the merged linked list.</p><pre>Input:<br>- list1 = [1, 2, 4]<br>- list2 = [1, 3, 4]<br>Output:<br>- [1, 1, 2, 3, 4, 4]</pre><pre>Input:<br>- list1 = []<br>- list2 = [4]<br>Output:<br>- [4]</pre><h4><strong>Step 1: Clarify Requirements</strong></h4><ul><li><strong>Can one or both the lists be empty?</strong></li></ul><p>→ Yes</p><ul><li><strong>Will both the lists be sorted?</strong></li></ul><p>→ Yes</p><ul><li><strong>Can the lists be of different sizes?</strong></li></ul><p>→ Yes</p><h4><strong>Step 2: Discuss Approaches</strong></h4><ol><li><strong>Combine and Sort (<em>Suboptimal</em>)</strong></li></ol><ul><li><strong>Logic:</strong></li></ul><p>→ Convert both the lists to arrays</p><p>→ Combine the two arrays</p><p>→ Sort the two arrays</p><p>→ Use the sorted array to build a new linked list</p><p>→ Return this list at the end</p><ul><li><strong>Big O:</strong></li></ul><p>→ Time: O((m + n) log(m + n))</p><p>→ Space: O(m + n)</p><p>2. <strong>Merge with Two Pointers (<em>Optimal</em>)</strong></p><ul><li><strong>Logic:</strong></li></ul><p>→ Use two pointers (one on each list)</p><p>→ Compare current nodes</p><p>→ Attach the smaller one to result</p><p>→ Move that pointer forward</p><p>→ Continue until both lists are done</p><ul><li><strong>Big O:</strong></li></ul><p>→ Time: O(m + n)</p><p>→ Space: O(1)</p><h4><strong>Step 3: Write Code (Optimal)</strong></h4><p><strong>Python</strong></p><pre>def mergeTwoLists(l1, l2):<br>  # create dummy head to start merged list<br>  dummy = ListNode()<br>  cur = dummy<br><br>  # loop while both lists have nodes<br>  while l1 and l2:<br>    # pick smaller value node<br>    if l1.val &lt; l2.val:<br>      cur.next = l1<br>      l1 = l1.next<br>    else:<br>      cur.next = l2<br>      l2 = l2.next # move forward<br><br>  # attach remaining nodes from either list<br>  cur.next = l1 or l2<br>  return dummy.next # return merged list head</pre><p><strong>Java</strong></p><pre>public ListNode mergeTwoLists(ListNode l1, ListNode l2) {<br>  // create dummy head to simplify merging<br>  ListNode dummy = new ListNode(0);<br>  ListNode cur = dummy;<br><br>  // merge untill one list ends<br>  while (l1 != null &amp;&amp; l2 != null) {<br>    // choose smaller node<br>    if (l1.val &lt; l2.val) {<br>      cur.next = l1;<br>      l1 = l1.next;<br>    } else {<br>      cur.next = l2;<br>      l2 = l2.next;<br>    }<br>    curr = cur.next; // move pointer<br>  }<br><br>  //connect leftover part<br>  curr.next = (l1 != null) ? l1 : l2;<br>  return dummy.next; // head of the merged list<br>}</pre><p><strong>C++</strong></p><pre>ListNode* mergeTwoLists(ListNode* l1, ListNode* l2) {<br>  // dummy node to build result easily<br>  ListNode dummy(0);<br>  ListNode* cur = &amp;dummy;<br><br>  // iterate while both lists have nodes<br>  while (l1 &amp;&amp; l2) {<br>    // attach smaller node first<br>    if(l1-&gt;val &lt; l2-&gt;val) {<br>      cur-&gt;next = l1;<br>      l1 = l1-&gt;next;<br>    } else {<br>      cur-&gt;next = l2;<br>      l2 = l2-&gt;next;<br>    }<br>    cur = cur-&gt;next; // advance pointer<br>  }<br>  <br>  // attach remaining nodes if any<br>  cur-&gt;next = l1 ? l1 : l2;<br>  return dummy.next; // merged list h<br>}</pre><h4><strong>Step 4: Answer Follow-Ups</strong></h4><ul><li><strong>What if both lists are empty?</strong></li></ul><p>→ Return None. Nothing to merge.</p><ul><li><strong>Can we merge without a dummy node?</strong></li></ul><p>→ Yes, but logic gets messy.</p><ul><li><strong>Can we do it recursively?</strong></li></ul><p>→ Yes, but not if lists are huge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c91533650af3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prompt Engineering: Template Library]]></title>
            <link>https://medium.com/@praveenmudalgeri05/prompt-engineering-template-library-bafb0b9261bb?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/bafb0b9261bb</guid>
            <category><![CDATA[writing-prompts]]></category>
            <category><![CDATA[automate]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[students]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Sat, 09 Aug 2025 07:35:59 GMT</pubDate>
            <atom:updated>2025-08-09T07:35:59.545Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EpiKIYNFWd7wLH36" /><figcaption>Photo by <a href="https://unsplash.com/@berctk?utm_source=medium&amp;utm_medium=referral">Berke Citak</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Tired of starting from scratch every time you need a good prompt?<br> Here’s a collection of simple, ready-to-use templates for work, personal life, and creative projects.<br> Just copy, fill in the blanks, and get results faster.</p><p>Here are battle-tested templates you can use immediately:</p><ol><li>Problem–Solution Template</li></ol><pre>Problem: [SPECIFIC_ISSUE]<br>Constraints: [LIMITATIONS]<br>Goal: [DESIRED_OUTCOME]<br><br>Provide:<br>Root cause analysis (2–3 key factors)<br>Solution options (rank by feasibility)<br>Implementation roadmap<br>Success metrics<br><br>Format as executive summary with bullet points.</pre><p>2. Comparison Analysis Template</p><pre>Act as an objective analyst comparing [ITEMS_TO_COMPARE].<br>Evaluation criteria:<br>[CRITERION_1]<br>[CRITERION_2]<br>[CRITERION_3]<br><br>For each item, provide:<br>Strengths (top 3)<br>Weaknesses (top 3)<br>Best use case<br>Score out of 10<br><br>Present as comparison table with final recommendations.</pre><p>3. Step-by-Step Guide Template</p><pre>You are an experienced [DOMAIN] instructor known for clear, actionable tutorials.<br><br>Create a beginner-friendly guide for: [TOPIC]<br>Structure:<br>Prerequisites (what they need first)<br>Materials/tools required<br>Steps (numbered, with explanations)<br>Common mistakes to avoid<br>Next steps for continued learning<br><br>Target audience: [DESCRIBE_AUDIENCE]<br> Tone: Encouraging but practical<br> Length: [WORD_COUNT]</pre><p>4.Brainstorming / Idea Generation Template</p><pre>You are a creative strategist tasked with generating innovative ideas for [GOAL_OR_PROJECT].  <br><br>Context: [BRIEF_DESCRIPTION]  <br>Constraints: [LIST_LIMITATIONS]  <br>Target Audience: [AUDIENCE_DESCRIPTION]  <br><br>Provide:  <br>- 5 unique ideas (brief explanation each)  <br>- Why each idea might work  <br>- Potential risks or challenges  <br>- Quick win vs long-term potential classification  <br><br>Output as a bullet list with concise notes.</pre><p>5. Content Outline Template</p><pre>You are a content strategist creating a structured outline for [CONTENT_TYPE] about [TOPIC].  <br><br>Target Audience: [AUDIENCE_DESCRIPTION]  <br>Tone: [FORMAL/INFORMAL/ENGAGING]  <br><br>Structure:  <br>1. Introduction (hook + purpose)  <br>2. Main sections (3–5 key topics)  <br>3. Supporting examples or data  <br>4. Call to action or conclusion  <br><br>For each section, provide:  <br>- Working title  <br>- 2–3 bullet points of content ideas  <br>- Suggested media (images, charts, quotes)</pre><p>6. Decision-Making Matrix Template</p><pre>You are a decision consultant helping choose between [NUMBER] options for [SITUATION].  <br><br>Criteria:  <br>- [CRITERION_1]  <br>- [CRITERION_2]  <br>- [CRITERION_3]  <br><br>For each option, provide:  <br>- Score for each criterion (1–10)  <br>- Short justification for score  <br>- Overall weighted score  <br><br>Present as a table followed by a one-paragraph recommendation.  </pre><p>7. Email Drafting Assistant</p><pre>You are a professional copywriter helping draft a clear, concise, and polite email for [PURPOSE].  <br><br>Context: [BRIEF_BACKGROUND]  <br>Recipient: [RECIPIENT_DETAILS]  <br>Tone: [FORMAL/FRIENDLY/NEUTRAL]  <br><br>Structure:  <br>- Subject line (short &amp; relevant)  <br>- Greeting  <br>- Body (main points in logical order)  <br>- Call to action or next steps  <br>- Closing &amp; signature  <br><br>Ensure grammar is correct and the message is easy to understand.</pre><p>8. Daily Task Prioritization Template</p><pre>You are a productivity coach helping plan and prioritize today’s tasks.  <br><br>List of tasks: [TASK_1, TASK_2, TASK_3...]  <br>Available time: [HOURS]  <br>Energy level: [HIGH/MEDIUM/LOW]  <br><br>Provide:  <br>- Priority ranking (High/Medium/Low)  <br>- Time estimate for each task  <br>- Suggested order of execution  <br>- Quick wins to complete first  <br>- Optional tasks to postpone or delegate  <br><br>Format as a numbered priority list with time blocks.  </pre><p>9. Meal Planning &amp; Grocery List Template</p><pre>You are a nutrition planner creating a balanced weekly meal plan.  <br><br>Diet type: [VEGETARIAN/NON-VEGETARIAN/VEGAN]  <br>Calories per day: [CALORIES]  <br>Restrictions: [ALLERGIES/INGREDIENTS_TO_AVOID]  <br>Budget: [BUDGET_RANGE]  <br><br>Provide:  <br>- Breakfast, lunch, dinner for 7 days  <br>- 1–2 snack options per day  <br>- Nutritional balance (protein, carbs, fats)  <br>- Consolidated grocery shopping list  <br><br>Format meal plan as a table and grocery list as bullet points.</pre><p>10. Conflict Resolution &amp; Response Template</p><pre>You are a communication coach helping draft a calm, constructive, and solution-focused response to a disagreement or complaint.  <br><br>Context: [BRIEF_DESCRIPTION_OF_SITUATION]  <br>Parties involved: [NAMES/ROLES]  <br>Desired outcome: [RESOLUTION_GOAL]  <br>Tone: [PROFESSIONAL/CALM/UNDERSTANDING]  <br><br>Provide:  <br>- Acknowledgment of the other person’s perspective  <br>- Brief explanation of your viewpoint  <br>- Areas of agreement (if any)  <br>- Proposed solution or next steps  <br>- Invitation for further discussion  <br><br>Ensure the language avoids blame, stays respectful, and focuses on collaboration.  </pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bafb0b9261bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unique RAG Project Ideas for Engineering Students]]></title>
            <link>https://medium.com/@praveenmudalgeri05/unique-rag-project-ideas-for-engineering-students-23916f6b1700?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/23916f6b1700</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[students]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[projects]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Mon, 04 Aug 2025 10:28:41 GMT</pubDate>
            <atom:updated>2025-08-04T10:28:41.788Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RqmlILNBki9ibRw7kXJ__A.png" /><figcaption>Unique RAG Project Ideas by Complexity and Domain — A comprehensive visualization of 18 innovative RAG project concepts organized by implementation difficulty and application area</figcaption></figure><h3>Unique Project Categories by Complexity</h3><h3>Beginner-Level Projects (2–4 weeks)</h3><p>1. Personal Knowledge Assistant with Smart Chunking</p><ul><li>Extend your basic concept with semantic chunking rather than fixed-size chunks</li><li>Implement hierarchical summarization for long documents</li><li>Add temporal awareness for document versioning and updates</li><li>Innovation: Context-aware chunk boundaries using sentence transformers</li></ul><p>2. Domain-Specific RAG for Cricket Analytics</p><ul><li>Perfect for your cricket interest! Create a RAG system for cricket statistics, match analysis, and player comparisons</li><li>Integrate live match data with historical statistics</li><li>Generate match predictions and player performance insights</li><li>Innovation: Sports-specific entity recognition and statistical reasoning</li></ul><h3>Intermediate-Level Projects (6–10 weeks)</h3><p>3. Multimodal RAG with Vision Integration</p><ul><li>Combine your document processing with image understanding using models like CLIP or ColPali</li><li>Process documents containing charts, diagrams, and images</li><li>Generate descriptions for visual content and integrate with text retrieval</li><li>Innovation: Cross-modal retrieval where text queries can find relevant images and vice versa</li></ul><p>4. Real-Time Streaming RAG System</p><ul><li>Implement continuous data ingestion using Apache Kafka or similar streaming platforms</li><li>Handle real-time updates to the knowledge base without system downtime</li><li>Add temporal reasoning to understand when information becomes outdated</li><li>Innovation: Event-driven architecture for dynamic knowledge base updates</li></ul><p>5. Code-Aware RAG for Software Development</p><ul><li>Build a system that understands code syntax, documentation, and comments</li><li>Integrate with GitHub repositories for live code analysis</li><li>Generate contextual code suggestions and documentation</li><li>Innovation: AST-based code understanding combined with natural language processing</li></ul><h3>Advanced-Level Projects (10–16 weeks)</h3><p>6. Graph-Enhanced RAG (GraphRAG)</p><ul><li>Implement knowledge graph integration using Neo4j or similar graph databases</li><li>Create entity relationship mapping from unstructured text</li><li>Enable multi-hop reasoning across connected entities</li><li>Innovation: Hybrid vector-graph retrieval for complex relational queries</li></ul><p>7. Multi-Agent Collaborative RAG</p><ul><li>Design specialized agents for different data sources (databases, documents, web)</li><li>Implement agent coordination and task decomposition</li><li>Add self-reflection and error correction mechanisms</li><li>Innovation: Agents that can learn from each other and improve retrieval strategies</li></ul><p>8. Edge-Optimized RAG for Resource-Constrained Environments</p><ul><li>Implement model quantization and embedding pruning for mobile deployment</li><li>Design adaptive caching strategies for frequently accessed content</li><li>Create progressive loading of knowledge bases</li><li>Innovation: On-device RAG with intelligent resource management</li></ul><h3>Expert-Level Projects (16–24 weeks)</h3><p>9. Modular RAG Architecture (Your Proposed System Enhanced)<br>Building on your excellent foundation, consider these advanced enhancements:</p><ul><li>Plugin Ecosystem: Create a marketplace-like system where different retrieval strategies, embedding models, and generators can be dynamically loaded</li><li>Auto-Optimization: Implement reinforcement learning to automatically tune hyperparameters based on query performance</li><li>Federated Learning: Enable distributed training across multiple knowledge bases while preserving privacy</li><li>Innovation: Self-configuring RAG system that adapts to new domains automatically</li></ul><p>10. Production-Ready RAG with Advanced Observability</p><ul><li>Implement comprehensive monitoring and alerting systems</li><li>Add A/B testing framework for comparing different RAG configurations</li><li>Create detailed performance analytics and cost optimization features</li><li>Innovation: ML-driven system optimization with predictive scaling</li></ul><h3>Technical Implementation Strategies</h3><h3>Core Technologies to Consider</h3><p>Based on current industry trends, here are the most impactful technology choices:</p><p>Embedding Models: Move beyond basic embeddings to domain-specific models like E5-large-v2, BGE-base-en-v1.5, or Alibaba-NLP/gte-Qwen2–7B-instruct</p><p>Vector Databases: Consider Qdrant for high performance, Weaviate for multimodal support, or Pinecone for managed solutions</p><p>Orchestration: Implement using LangChain or LlamaIndex for rapid development, but consider custom frameworks for unique requirements</p><h3>Evaluation Framework</h3><p>Implement robust evaluation using modern frameworks:</p><ul><li>RAGChecker for fine-grained analysis</li><li>RAGAS for comprehensive pipeline evaluation</li><li>Custom metrics specific to your use case</li></ul><h3>Recommended Project Selection</h3><p>For Maximum Impact: Choose the Multimodal RAG with Vision Integration or Graph-Enhanced RAG. These represent cutting-edge developments that are highly relevant to current industry needs.</p><p>For Technical Challenge: The Multi-Agent Collaborative RAG offers the most learning opportunities and demonstrates advanced AI system design principles.</p><p>For Practical Application: Your enhanced Modular RAG Architecture with the suggested improvements would create a genuinely useful tool for the broader developer community.</p><p>For Personal Interest: The Cricket Analytics RAG would be unique, personally engaging, and demonstrate domain expertise.</p><h3>Key Success Factors</h3><ol><li>Focus on Innovation: Don’t just implement existing solutions — add novel features or improvements</li><li>Demonstrate Understanding: Show deep comprehension of RAG principles, not just implementation</li><li>Practical Value: Ensure your system solves real problems and could be used by others</li><li>Proper Evaluation: Implement comprehensive testing and metrics</li><li>Documentation: Create excellent documentation and presentation materials</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23916f6b1700" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My First Open Source Contribution]]></title>
            <link>https://medium.com/@praveenmudalgeri05/my-first-open-source-contribution-9b316b86d1a5?source=rss-8fe1c26d01d7------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b316b86d1a5</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[contribution]]></category>
            <category><![CDATA[open-source-contribution]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Praveen Mudalgeri]]></dc:creator>
            <pubDate>Mon, 28 Jul 2025 06:25:20 GMT</pubDate>
            <atom:updated>2025-07-28T06:25:20.300Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQRgAA_08wTpfDlU-2MB5g.png" /></figure><p><em>Ever since I started learning Python, I’d wanted to be part of the open source community. This year, that dream finally became a reality — I made my very first contribution to an established open source project on GitHub, and it got merged! In this blog, I’d like to share the experience, what I learned, and why you should consider contributing too.</em></p><h3>How It Started</h3><p>Like many newcomers, I was a bit intimidated at first. Would my code be good enough? Would I be able to figure out their project structure?</p><p>One day, while exploring repositories that welcome contributions, I stumbled upon <a href="https://github.com/geekcomputers/Python">geekcomputers/Python</a> — a large, well-known repository filled with scripts and automation bits.</p><p>Browsing through the issues and PRs, I found a maintainability task: updating the ccxt dependency and fixing Python warnings in the codebase. The pull request I worked on aimed to:</p><ul><li>Update the ccxt library version in requirements.txt from 4.4.95 to 4.4.96</li><li>Resolve various Python syntax warnings and invalid escape sequence issues throughout the repository</li><li>Ensure the codebase compiles and runs cleanly, free of unnecessary warnings</li></ul><h3>What I Actually Did</h3><p>Here’s a breakdown of my contributions:</p><ul><li><strong>Bumped ccxt dependency</strong> in requirements.txt to keep the library up to date. Dependency updates are often overlooked but are crucial for security and compatibility.</li><li><strong>Fixed invalid escape sequences</strong> across the codebase. These are sneaky warnings that clutter outputs and can potentially cause bugs on different systems. I replaced problematic paths and strings with raw format or proper escape characters.</li><li><strong>Ran the full test suite and syntax checks</strong> to verify that my changes didn’t break anything. No new features were added, but the project became cleaner and easier to maintain.</li></ul><h3>The Pull Request Journey</h3><p>Once I finished the changes, I submitted a pull request describing exactly what was done. I referenced related issues so maintainers could see the context, documented all modifications, and ensured the changes were easy for reviewers to follow.</p><p>🔗 <a href="https://github.com/geekcomputers/Python/pull/2802">Here’s the pull request</a></p><p>After some review and friendly feedback, my pull request was merged! It felt amazing to see my work included in a project used by so many people.</p><h3>What I Learned</h3><ul><li><strong>Open source isn’t just about big features.</strong> Maintenance, documentation, and minor fixes are just as valuable.</li><li><strong>Clear communication matters.</strong> A well-written pull request description speeds up the review process and helps maintainers understand your intent.</li><li><strong>Testing is critical.</strong> Always check your changes locally before pushing.</li><li><strong>The community is welcoming.</strong> Most projects are eager for help, especially on issues marked <em>“good first issue”</em> or <em>“help wanted”</em>.</li></ul><h3>Why You Should Contribute</h3><p>If you’re reading this and feel hesitant, I encourage you to find a project, read their guidelines, and take the plunge. Your first contribution doesn’t need to be monumental. Even fixing a typo can be a great way to start.</p><p><em>Here’s to many more contributions ahead!</em></p><p>If you have questions about getting started with open source, feel free to reach out in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b316b86d1a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>