<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by CodeKarle on Medium]]></title>
        <description><![CDATA[Stories by CodeKarle on Medium]]></description>
        <link>https://medium.com/@codekarle?source=rss-45907e1366ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by CodeKarle on Medium</title>
            <link>https://medium.com/@codekarle?source=rss-45907e1366ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:50:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@codekarle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DNS Lookup]]></title>
            <link>https://medium.com/@codekarle/dns-lookup-8157c5d6fb7a?source=rss-45907e1366ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/8157c5d6fb7a</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[CodeKarle]]></dc:creator>
            <pubDate>Fri, 14 Feb 2025 05:22:09 GMT</pubDate>
            <atom:updated>2025-02-14T05:22:09.064Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>What happens when you type google.com in your browser and hit enter?</strong></h4><p>How do we go from typing google.com to a page with the colourful Google logo and a search bar with the ability to answer any and all of our questions? How does your browser even know which page to show? And why do some pages take longer to load whereas others appear almost instantaneously?</p><p>Well, all of this happens through a process called <strong>DNS resolution</strong>. But before we get into that, let us understand what DNS really is.</p><p><strong>So, what is DNS?</strong></p><p>DNS, <strong>Domain Name System</strong>, is basically the internet’s Yellow Pages. It allows you to reach Facebook’s home page without remembering its IP address, just like Yellow Pages helps you call Anand Sweets<a href="#_msocom_1">[1]</a> <a href="#_msocom_2">[2]</a> without having to remember their contact number. We as users access web pages through their domain names like <a href="http://www.codekarle.com/">www.codekarle.com</a>, but web browsers interact with servers through IP addresses like 192.168.1.1, to fetch the web page. DNS translates the domain names to IP addresses so that browsers can understand the request and fetch the relevant web page.</p><p><strong>How does DNS work?</strong></p><p>The process of translation of domain names to IP addresses is known as <strong>DNS resolution</strong>. Once the browser makes the request to the DNS server, the query passes through a bunch of other servers to find the domain name — IP address mapping. Let us look at this whole process in a little more detail.</p><p>The DNS resolution pipeline has four levels of servers that work together to find the correct IP address.</p><p>DNS recursor -&gt; Root Nameserver -&gt; TLD Nameserver -&gt; Authoritative Nameserver</p><p><strong>DNS Recursor </strong>is the server responsible for receiving requests from web browsers and makes further requests, if needed, to find the correct IP address. It does this by making multiple requests till it finds the authoritative nameserver that has the mapping for requested domain name. If no record is found it returns a timeout error. To optimise the lookup at this level itself, our recursive resolver maintains a cache where it stores the recently looked up records. This is why you’ll notice, some pages that are opened more frequently take less time to load.</p><p><strong>Root Nameserver</strong>, our next stop, is actually the first step in the process of DNS resolution. If the DNS recursor is unable to find the mapping in its cache, it reaches out to the root nameserver. Root nameserver has information about the root zone i.e. <strong>Top Level Domains </strong>like .com, .org, .net, .in, .au, etc. For each TLD the root server returns the IP address for its TLD’s content.</p><p>There are 13 root servers which are reachable with 13 IP addresses. For better reach, collectively 1300 instances of root servers run across the globe, but at any time from any given location, only 13 root servers are accessible.</p><p><strong>TLD Nameserver </strong>is our next stop. This server contains information of all domain names ending in a common extension. For example if we search <a href="http://www.codekarle.com/">www.codekarle.com</a>, the recursive resolver at this point will reach out to the .com TLD nameserver which will respond with the authoritative nameserver for that codekarle.com.</p><p><strong>Authoritative Nameserver</strong> is the last step in the resolution process. It contains information specific to the domain name and responds with the IP address of the server found.</p><p>Now this seems like a fairly long process. Isn’t there a way to optimize this? Of course there is and the most obvious thing that comes to mind is <strong>caching</strong>. Usually there is some level of caching implemented at every level to avoid further look ups and save some time. But even before the process begins, we have a local copy of recent DNS lookups on our machine itself..</p><p><strong>DNS lookup and caching</strong></p><p><strong>Browser cache </strong>— Most browsers maintain a local DNS cache where they store the mappings for more frequently accessed domain names. If you think about it, it kind of makes sense, since closer the cache, faster the rendering and better the experience. So browser level DNS Cache is the first place we check.</p><p><strong>OS cache</strong> — If the mapping is not found in browser DNS cache, OS level DNS resolver is the second place we check. The process responsible for this check is called the <em>stub resolver.</em> If the record is found by the stub resolver, IP address is returned to the browser which fetch the web page and renders it. If not, the request now goes out of our machine. Some routers also maintain a DNS cache, and this is the next place we check. Again, if the mapping is found, IP address is sent back to the browser, otherwise the request is sent to the ISP’s DNS recursor. ISP’s recursor checks in its own DNS cache, if the record is found, it is returned to the browser. Otherwise, the DNS lookup is initiated.</p><p><strong>So, to summarise…</strong></p><p>-&gt; When the user types, let’s say, codekarle.com in the browser and hits enter, the browser first checks its DNS cache. If the IP address is found, it fetches the webpage and renders it.</p><p>-&gt; If not, OS level DNS cache is the next stop.</p><p>-&gt; If the IP address is not found here as well, the router level DNS cache, if it is maintained, is the next stop.</p><p>-&gt; Same drill, if the IP address is found it is returned to the browser, otherwise the request is sent to the ISP’s recursive resolver..</p><p>-&gt; ISP checks its own cache, if the IP address is found it is returned to the browser, otherwise the recursive resolver reaches out the the Root Nameserver.</p><p>-&gt; Root Nameserver returns the address of the relevant TLD nameserver, in our case it will be .com nameserver to the resolver.</p><p>-&gt; Resolver now reaches out to the TLD server which responds with the authoritative nameserver for codekarle.com.</p><p>-&gt; Resolver now reaches out to the codekarle.com authoritative nameserver to fetch the IP address and returns it to the web browser.</p><p>-&gt; Now the browser makes an HTTP request to this IP address which returns the webpage to the browser to render.</p><p>That’s all about DNS resolution from our side. If there is something we missed, let us know in the comments. Do check out our website and YouTube channel for more such content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8157c5d6fb7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is CAP Theorem?]]></title>
            <link>https://medium.com/@codekarle/what-is-cap-theorem-def637917f03?source=rss-45907e1366ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/def637917f03</guid>
            <category><![CDATA[software-design]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[system-architecture]]></category>
            <dc:creator><![CDATA[CodeKarle]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 11:10:55 GMT</pubDate>
            <atom:updated>2025-02-04T11:10:55.599Z</atom:updated>
            <content:encoded><![CDATA[<p>Has it ever happened to you that your friend posted something really cool on Facebook but you can only see it in your feed after a while or get notified after a while? And how many times have we seen a “Sorry! We couldn’t find the page!” on Amazon? Why does this happen? And no, it is not always because of a bad network or heavy traffic!</p><p>To answer this we need to understand the CAP Theorem and the various trade-offs in distributed systems, but before that let us understand more about distributed systems.</p><p><strong>What are Distributed Systems?</strong></p><p>Traditionally, softwares used to be monolithic in nature. Everything was in one place, one database with all the data, one massive repo with all the code! There were very few inter-system interactions over the network, so lesser security threats, no need to worry about consistency since there is just one database, use a relational DB and problem solved! Partition tolerance was probably not even a concept! But these systems were very complex, difficult to maintain, costly to scale and just could not handle load efficiently. If a part of the system breaks… the system breaks! Yeah, doesn’t sound very promising!</p><p>This is when the concept of distributed systems came in. The huge monolithic system was broken down into microservices, each with their own independent data-store. The systems were more fault tolerant since now there was no single point of failure. We can have load balancers, so huge traffic? Not a problem anymore! Where vertical scaling was previously the only approach, horizontal scaling, which is much more cost efficient, now became a thing. Since we were running multiple instances of our server anyways, we might as well distribute them geographically, working towards lower latency. The huge repo was broken down into multiple systems so the code was more easily manageable, maintainable, understandable, and a lot less complex! This is why distributed systems gained a lot of popularity in the last few decades. But distributed systems came with their own set of complications. Remember the trade-offs we mentioned? Let us have a deeper look at them.</p><p><strong>CAP Theorem</strong></p><p>CAP Theorem, proposed by Eric Brewer, states that “Of the three properties of shared-data systems — Consistency, Availability and tolerance to Network Partitioning — only two can be achieved at any given moment in time.” Meaning there are times we might need to prioritise two out of the three properties as it is not possible to guarantee all three. This is not to say that an available system can never be consistent or the other way round, but there will come a time where we will have to decide between availability and consistency.</p><p>Based on this theory, distributed systems are classified into three types — CP (Consistent and Partition Tolerant), AP (Available and Partition Tolerant) and CA (Consistent and Available). Let us try to understand these properties and classifications with some examples.</p><p><strong>Consistency</strong> is when all the nodes in the system see the same data at all times. If multiple users are trying to access the database, they should all see the same data. If the data is distributed across multiple partitions or databases each write is followed by several other updates to the system so that every instance of the data is updated and only then will our system allow further reads, even at the cost of availability. This type of system is a <strong>CP</strong> system. Let us look at an example. Say you are trying to transfer some money from bank account A to bank account B. If $100 is deducted from account A, then $100 should be credited to account B. That means, there will be an update to both the accounts across all the instances and partitions of the database that store information about balance in accounts A and B. If any of the updates fails and the data is left inconsistent, the transaction will not be completed and $100 will be returned to account A, with a notification sent to the account holder saying their transaction failed. Only if all the updates are successfully done, money will actually be transferred and further reads will show the updated balance.</p><p>Let’s say this payment was a part of Amazon’s checkout flow. When you buy the product, order, inventory and payment systems will be updated. If the payment fails because the system could not be consistent, the complete transaction will need to be rolled back, including any updates done to the inventory and order systems, to ensure the entire system remains consistent.</p><p><strong>Availability </strong>is when every request sent to the system generates a valid response. This is usually achieved by replicating the data across multiple nodes. Even if these nodes haven’t had the time to sync up yet to maintain consistency, they will still provide the users a valid response. Let us consider our Amazon example again. So our payment failed due to some inconsistency and order could not be placed. Here consistency was important. But we can’t say the same about cart. If a user has added a product to their cart and it is reflected on the node they are accessing, it is alright if other nodes take some time getting the same information. Just because that data cannot be communicated to other nodes at this point in time, not allowing the users to add products to their cart seems like a bad user experience. This is where consistency can take a back seat and we will prioritise availability. Such systems are called <strong>AP</strong> systems.</p><p><strong>Partition Tolerance </strong>is the ability of a system to function even if two or more of the nodes are unable to communicate with each other. This usually happens when the nodes are distributed across the network. Distributed systems, by nature, are a group of servers distributed across various geographical locations to provide the best possible user experience with lowest possible latency and downtime. And since network connectivity is something that sits outside our system and cannot be guaranteed, we need to ensure our systems can still function in case of a network partition.This is why, while designing a distributed system, the decision comes down to concurrency vs availability. This is also why the third category <strong>CA</strong> cannot exist in a distributed system.</p><p>Again, this doesn’t mean that our entire system has to be CP or CA. Remember how we used Amazon checkout flow as an example for both CP and AP systems? That is because it is not practical to compromise availability and user experience just for the sake of consistency and it is also not practical to make the entire system inconsistent just to provide a good user experience. We can easily make parts of our system CA or CP as per requirement by just breaking it down to multiple microservices that can function independently.</p><p><strong>How do you use CAP theorem in a system design interview?</strong></p><p>So the first time CAP Theorem comes in, is when you are calling out the non-functional requirements right in the beginning of your discussion. You need to clarify what parts of your system need to be available at all times and what parts of the system need to be consistent. This way you can get a clear picture of how the system will be broken down so you can make the trade-off easily. Think back to our Amazon example.</p><p>Now when you are discussing individual microservices and the data stores they are going to use, again, CAP Theorem comes in when you are choosing the database. Let’s say, you understand the structure of your data (or lack of it), scale at which you will be working and what kind of query pattern you will work with and decide you need to use a Columnar DB but are confused between Cassandra and HBase. This is when you need to think if your service needs to be CP or AP. If the trade-off is not needed, Cassandra is much easier to set up, so that would be an easier choice. But Cassandra is by default an AP database and if consistency is of utmost importance, it makes much more sense to use HBase since it is a CP store.</p><p>That is all about CAP Theorem from our end. If you feel there is something we missed, let us know in the comments and we will try to cover it in the next few articles.</p><p>Do have a look at our website <a href="https://www.codekarle.com/">CodeKarle </a>and our <a href="https://www.youtube.com/@codeKarle/videos">YouTube</a> channel for more such content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=def637917f03" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My System Design Interview Checklist]]></title>
            <link>https://medium.com/@codekarle/my-system-design-interview-checklist-6d75cf9ef0ce?source=rss-45907e1366ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d75cf9ef0ce</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[system-design-interview]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[CodeKarle]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 04:25:46 GMT</pubDate>
            <atom:updated>2025-01-26T04:25:46.845Z</atom:updated>
            <content:encoded><![CDATA[<h3>Must-knows for a system design interview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zOtxkCeum5P4ioYS" /><figcaption>System Design Interview Checklist</figcaption></figure><p><em>That dreaded system design interview. I remember the first system design question I was asked. “Design WhatsApp”, he said. I didn’t know where to start! I was a recent graduate. Data structures and Algorithms were the only things I knew. I am sure you can guess how that interview went. Then after enough research I made myself a checklist of components, of sorts, to navigate me through my next system design interviews and boy does it work!</em></p><p>So I’ll stop with the chit-chat and get right into it!</p><h3>User Interface</h3><p>First things first, how are the users interacting with your system? Is it via a mobile app or on a laptop or a smart TV? This gives us an indication of what kinds of limitations and hidden requirements we are dealing with. For example, in the case of a mobile user, we might have to handle fluctuating networks, for a smart TV and laptop user kind of scenario we might have to support different file formats and different resolutions and aspect ratios.</p><h3>Load Balancer</h3><p>You cannot design a distributed system without a load balancer. Load balancing means distributing the requests between various nodes of a distributed system, so as to decrease response time and improve resource utilization. By distributing the load between various nodes, we rule out the chance of having a single point of failure and also ensure that a single node is not being overloaded while another server might be sitting idle.</p><h3>Web Sockets</h3><p>Or to be more specific <em>web socket handler</em> and <em>web socket manager</em>. I know, these are not standard terms. But it is something I came across recently on a youtube video and it makes total sense. A <strong>Web Socket Handler</strong> is basically a server that keeps an open web socket connection with all active user devices. Any request from the user will enter the system via a web socket handler and any notifications or responses to the user will be sent via the web socket handler as well. Now if we are working on a large-scale application, with users distributed across various locations, we are going to need web socket handlers distributed across various geographical locations. Think <em>low latency</em>. Now users could also switch between various web socket handlers due to let’s say network fluctuations, or simply because they are now physically closer to another web socket handler. In this case, we need a central repository to keep track of which user is connected to which web socket handler. This is done by a <strong>Web Socket Manager</strong>.</p><h3>Database</h3><p>Well, not just database, storage solutions in general. If we need to save images or files, we will need blob storage like Amazon S3, if we are storing payment records, we need it to follow ACID properties and need a relational database in such a case. If we are storing product-related information on a platform like Amazon we will probably need a document DB like Mongo DB. If we were building Twitter or Facebook, we cannot design our system without a Caching solution like Redis. There are so many things we need to consider while deciding which storage solution to use! Here is one of the best articles I came across during my prep -&gt; <a href="https://www.codekarle.com/system-design/Database-system-design.html">Choosing the best Database Solution in a System Design Interview</a> by CodeKarle.</p><h3>CDN</h3><p>Again, if our users are distributed geographically, it makes sense to maintain copies of frequently accessed data in a data center closer to the users’ location. For example, a lot of traffic for “Scared Games” on Netflix probably comes from India. So it makes sense to keep the data in a data center close to India to reduce the latency and improve the user experience. Here is a short video by Akamai, explaining briefly exactly what a CDN does -&gt; <a href="https://youtu.be/l6X_IxyGHHU">What is a CDN</a></p><h3>Analytics Components</h3><p>Remember, no matter what kind of a system you are creating, there is always scope for analytics. It is one of those things that are never called out during requirement gathering but are always required. This is when we use a stream processing software like Kafka. Whatever goes on in the system, a user logs in or logs out, gets dropped off from a call, cancels an order, cancels a payment, anything! It is all useful information for us! Whenever an event occurs, it should be written to Kafka. On Kafka, we could have some simple spark streaming jobs running for real-time processing. Or we could simply dump the data in a Hadoop cluster and later run some fancy machine learning algorithms for detailed analytics.</p><h3>Pluggable components</h3><p>If we were designing Twitter, post length is one of the limitations we need to handle. But what if a user wants to share a link? That would take up half the post! Which is why we need a <em>URL shortening service</em>. No matter what system you build, there will always be something or the other that needs to be notified to the user. This is where you might need a <em>notification service</em>. Notification services are actually a great way to understand the requirement for pluggability. What if as per our requirement, we need to send in-app notifications if a friend likes a profile picture on Facebook. So we build that into our system. Now a new requirement comes in to send out an email notification to users who haven’t accessed the app in more than 3 days. We can’t add another service to handle email notifications. This is why our notification service needs to be pluggable, so we can add handlers as we go and support new notification methods easily.</p><p>Another example of an independent service being used as a separate component would be an <em>asset delivery service</em>. If you were building something like Netflix, we need a system to load video content to CDNs depending on the incoming traffic, manage what resolutions and aspect ratio data needs to be sent to which user, takes care of transcoding the main file to all other supported file formats, etc. I will attach a link to one of the videos I referred to, to understand how an asset delivery system might work.</p><h3>Monitoring and Alerting</h3><p>Are you reaching milestones? Meeting deadlines? Is your system working as expected? Be it any application, no matter how well it is planned or designed, there is always a chance for something to go wrong. A third-party dependency could crash, a server could go down, maybe a server’s CPU utilization is very high while another is underutilized, maybe the system could not scale enough during the Black Friday Sale! Wouldn’t it be great to know these things beforehand?</p><p>But how? By monitoring your system of course! And alerting. Anything that is behaving unexpectedly or is reaching its limit, must be called out BEFORE it gets out of hand. This is another one of those things that are not called out initially but are very important while building a system. <a href="https://youtu.be/YyOXt2MEkv4?t=2056">Here</a> is an example of how you can do that.</p><h3>Brownie Points</h3><p>Once you have come up with a solution that meets all the requirements called out in the requirement gathering stage, it is your time to actually shine and stand out from other applicants. This is when you consider <strong>edge cases</strong> that people usually don’t consider in interviews. Like how does Google handle disputed areas like India-Pakistan-China borders in Google Maps? Or you could demonstrate how you can <strong>extend your solution</strong> to support a much larger scale than intended. Like can your video conferencing system be extended to live broadcast an event?</p><h3>Resources you might want to look at</h3><p><em>What is a load balancer? -&gt; </em><a href="https://medium.com/@itIsMadhavan/what-is-load-balancer-and-how-it-works-f7796a230034"><em>https://medium.com/@itIsMadhavan/what-is-load-balancer-and-how-it-works-f7796a230034</em></a></p><p><em>How to choose the best storage solution in your system design interview? -&gt; </em><a href="https://www.codekarle.com/system-design/Database-system-design.html"><em>https://www.codekarle.com/system-design/Database-system-design.html</em></a></p><p><em>What exactly is a CDN? -&gt; </em><a href="https://youtu.be/l6X_IxyGHHU"><em>https://youtu.be/l6X_IxyGHHU</em></a></p><p><em>How to scale your system with Redis -&gt; </em><a href="https://youtu.be/CtV-QymAeIc"><em>https://youtu.be/CtV-QymAeIc</em></a></p><p><em>How an asset delivery system works (with Netflix example) -&gt; </em><a href="https://youtu.be/lYoSd2WCJTo"><em>https://youtu.be/lYoSd2WCJTo</em></a></p><p><em>Here’s another medium article I wrote about choosing the best DB for a system design interview -&gt; </em><a href="https://towardsdatascience.com/choosing-the-right-database-in-a-system-design-interview-b8af9c6dc525?source=friends_link&amp;sk=d78ef7e0b0f4071636c57681b9fce1c8"><em>https://towardsdatascience.com/choosing-the-right-database-in-a-system-design-interview-b8af9c6dc525?source=friends_link&amp;sk=d78ef7e0b0f4071636c57681b9fce1c8</em></a></p><p><em>A comprehensive guide for cracking the next system design interview by CodeKarle -&gt; </em><a href="https://www.codekarle.com/"><em>https://www.codekarle.com/</em></a></p><p><em>Commonly asked system design interview questions -&gt; </em><a href="https://www.youtube.com/watch?v=EpASu_1dUdE&amp;list=PLhgw50vUymyckXl3D1IlXoVl94wknJfUC"><em>https://www.youtube.com/watch?v=EpASu_1dUdE&amp;list=PLhgw50vUymyckXl3D1IlXoVl94wknJfUC</em></a></p><p>I hope this can help you come up with a checklist of your own to refer to during your next system design interview. Think I missed an important step? Let me know in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d75cf9ef0ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the right database]]></title>
            <link>https://medium.com/@codekarle/choosing-the-right-database-cbfa5934a01f?source=rss-45907e1366ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbfa5934a01f</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[CodeKarle]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 18:03:43 GMT</pubDate>
            <atom:updated>2025-01-16T18:03:43.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to choose the right storage solution for your requirements</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*AbxBEDbju4Rl2WC_W3eeAg.png" /><figcaption>Storage Solutions and Databases — How to choose the best one? — Photo by Jan Antonin Kolar on Unsplash</figcaption></figure><p><em>Imagine — You’re in a system design interview and need to pick a database to store, let’s say, order-related data in an e-commerce system. Your data is structured and needs to be consistent, but your query pattern doesn’t match with a standard relational DB’s. You need your transactions to be isolated, and atomic and all things ACID… But OMG it needs to scale infinitely like Cassandra!! So how would you decide what storage solution to choose? Well, let’s see!</em></p><p>First of all, what kind of data are we working with? Is it records or file systems or audio/video content? And what kind of processing do we intend to do on that data? Do we need to search for something or run sophisticated analytics algorithms?</p><h4>What are the different types of storage solutions out there?</h4><p>Based on our requirements and how we want to use or access our data, we could be looking for the following storage solutions:</p><p><strong>Caching Solutions </strong>— If we are designing a read-heavy system like Twitter or Facebook, we might end up catching a lot of data, even complete timelines, to meet the low latency requirement. Some options here would be <em>Redis</em> or <em>Memcached</em>.</p><p><strong>File system storage </strong>— If we are designing some sort of asset delivery service, where we might need to store images or audio/video files, we might need to use something called blob storage. An extremely popular example is <em>Amazon S3.</em></p><p><strong>Text search engine</strong> — What if we are designing a system like Amazon and need to implement a search feature. The thing about search features is that we need to consider typos as well. Suppose a user wants to search for “<em>shirt</em>” but types “<em>shrt</em>” instead. Now if we don’t show any results, it would be very poor user experience. Our system needs to be smart enough to show results for “<em>shirt</em>” or “<em>shorts</em>”. This is known as fuzzy search and this is where we use text search engines like <em>Elasticsearch</em>.</p><p><strong>Data Warehouses </strong>— I know! We have been discussing data and storage all this time, so how can we not consider Big Data! Sometimes we just need to dump all the data in a single store where we can later perform all sorts of analytics. These systems are used more for offline reporting that usually transactions. This is where we end up using data warehouse solutions like <em>Hadoop</em>.</p><p>Now, you might have noticed that we have been talking about “storage solutions” not “databases”. So let’s have a look at <strong>Databases </strong>now!</p><h4>SQL? NoSQL? What is going on here?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*TROQQ-6mszRqt2GN1fJiqQ.png" /><figcaption>Image by Author</figcaption></figure><p>Well, there are a few factors basis which we can decide what kind of database to go with, these factors being — the structure of the <strong>data</strong>, <strong>query pattern</strong>, and <strong>scale</strong>.</p><p>A little confusing I know! This is why I will attach the video which explains this in a lot more detail, but we are also going to go over it in the coming sections, so read on.</p><p><a href="https://youtu.be/cODCpXtPHbQ?si=EErMsGP3s3oTCHD7">Choosing the best Database in a System Design Interview — CodeKarle</a></p><p>Now, <strong>scale</strong>,<strong> structure</strong>, and<strong> query pattern</strong>. Right. If the information is structured and can be represented as a table, and if we need our transactions to be atomic, consistent, isolated, and durable (ACID), we go with a <strong>relational database</strong>. The most commonly used one would be <em>MySQL</em>.</p><p>Now if the ACID properties are not required, well you could still use a relational database, or you could go with a NoSQL alternative. But if your data lacks a structure, it cannot be represented as a table and now we <em>need</em> to use a NoSQL DB like <em>MongoDB, Cassandra, HBase, Couchbase,</em> etc. And this is where the query pattern becomes a deciding factor.</p><blockquote>BTW, Elasticsearch is kind of a special case of document DB.</blockquote><p>If we have a vast variety of attributes in our data and a vast variety of queries, we use a <strong>Document DB</strong> like <em>MongoDB</em> or <em>Couchbase</em>. But if we have to work on an extremely large scale but have few types of queries we need to run, then we go for a <strong>Columnar DB</strong> like <em>Cassandra</em> or <em>HBase</em>. And even between columnar DBs, as you might know, HBase is built on top of Hadoop. So while setting up HBase we first need to set up Hadoop and the related components, and then set up HBase on top of it. This adds a level of complexity while setting up the system, so I would personally go for Cassandra if only for the sake of simplicity. Performance-wise both give similar results.</p><p>Now, the thing with Columnar databases like Cassandra is that they work majorly by partitioning and duplicating the data. So if you can choose the partition key such that all of your queries use the common partition key in the where clause, Cassandra is the way to go.</p><p>There is a more detailed article about <a href="https://www.codekarle.com/system-design/Database-system-design.html">how to choose the best storage solution</a>, which explains when it might make sense to use a columnar DB vs a document DB with an example of how Uber interacts with their driver and rider sides of the system. Let me try to explain this using the same scenario.</p><p>Let’s say Uber has saved the ride related information in a Cassandra with driver id as the partition key. Now when we run a query to fetch all data for a particular driver on a day, it fetches it based on the partition key driver id. This was the partition side of Cassandra’s solution. Now, what if we try to query a customer’s rides on a particular day by customer id. Now the query will be sent out to all the partitions and there goes the efficiency! This is where the replication side of the solution comes in. We can simply replicate the whole data and now use customer id as the partition key. Now when a query comes in based on customer id, it will be directed to the instance using customer id as the partition key. And this is why Cassandra can scale infinitely. Remember the query pattern for Cassandra? We mentioned it is only useful if there is a limited variety of queries. That is because we can only replicate the data so many times.</p><h3>Let’s get real though, is one database enough?</h3><p>Now, we have seen various storage solutions and also how to choose between various DBs based on our requirements and the type of information we need to store. But is that enough? Referring to another example from the above-mentioned article, there are some cases where a single DB is not enough for our requirement. For example, in the case of Amazon, orders data needs to follow ACID properties, but it needs to be infinitely scalable like a Columnar DB. In such a case we will use a combination of databases like MySQL + Cassandra. Now, all the information about ongoing orders, which need to follow ACID properties, will be stored in a MySQL DB and once they are completed, we can move them to Cassandra, which can be used as a permanent store. So as long as we require the ACID properties, the data remains in a relational DB and then is moved to a columnar DB which can scale according to the size of data. Problem solved!</p><p>I hope this can help you come up with a checklist of sorts to refer to during your next system design interview. Think I missed an important database/storage solution? Let me know in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbfa5934a01f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>