<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ram Karnani on Medium]]></title>
        <description><![CDATA[Stories by Ram Karnani on Medium]]></description>
        <link>https://medium.com/@ramkarnani24?source=rss-a73b78eb1f0a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*oBxg-CSUiyKJXJfp.</url>
            <title>Stories by Ram Karnani on Medium</title>
            <link>https://medium.com/@ramkarnani24?source=rss-a73b78eb1f0a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:58:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ramkarnani24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Reading yaml configurations in Python]]></title>
            <link>https://medium.com/@ramkarnani24/reading-yaml-configurations-in-python-40f54ce69655?source=rss-a73b78eb1f0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/40f54ce69655</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[yaml]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[configuration-file]]></category>
            <category><![CDATA[configuration]]></category>
            <dc:creator><![CDATA[Ram Karnani]]></dc:creator>
            <pubDate>Thu, 27 Feb 2020 12:38:24 GMT</pubDate>
            <atom:updated>2020-02-27T12:38:24.975Z</atom:updated>
            <content:encoded><![CDATA[<p>I prefer to read configurations in python using <strong>yaml</strong> file when I have to specify multiple interrelated configurations in one file.</p><p>So how to go around it: <em>all you need to do is</em> <strong><em>bucket these configurations under different headers</em></strong><em>.</em></p><p>Consider the following <strong>request_conf.yaml</strong> configuration file:</p><pre>---<br>- server:<br>    username: &#39;ram&#39;<br>    url: &#39;http://localhost:3000/&#39;<br>- execution:<br>    timeout: 60 #seconds</pre><p>And here is how to read these configurations in a <strong>python</strong> file:</p><pre>import yaml</pre><pre>def <strong>load_conf_file</strong>(config_file):<br>   with <strong>open</strong>(config_file, &quot;r&quot;) as f:<br>       <strong>config</strong> = yaml.safe_load(f)<br>       <strong>server_conf</strong> = config[0][&quot;server&quot;]<br>       <strong>exec_conf</strong> = config[1][&quot;execution&quot;]<br>   return <strong>server_conf</strong>, <strong>exec_conf</strong></pre><pre><strong>server_conf,exec_conf = load_conf_file(&quot;&lt;path to config file&gt;&quot;)</strong></pre><pre><strong>username</strong>=server_conf[&quot;username&quot;]<br><strong>url</strong>=server_conf[&quot;url&quot;]</pre><pre><strong>timeout</strong>=exec_conf[&quot;timeout&quot;]</pre><p>That’s it!</p><p>Happy reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40f54ce69655" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reading configurations in Scala]]></title>
            <link>https://medium.com/@ramkarnani24/reading-configurations-in-scala-f987f839f54d?source=rss-a73b78eb1f0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f987f839f54d</guid>
            <category><![CDATA[configuration]]></category>
            <category><![CDATA[configuration-file]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[config]]></category>
            <dc:creator><![CDATA[Ram Karnani]]></dc:creator>
            <pubDate>Mon, 29 Apr 2019 05:14:42 GMT</pubDate>
            <atom:updated>2019-04-29T05:14:42.822Z</atom:updated>
            <content:encoded><![CDATA[<p>There are multiple ways to read the configuration files in Scala but here are two of my most preferred approaches depending on the structure of the configurations:</p><h3>Reading configurations from <strong>JSON-type .conf</strong> files</h3><p>Use this approach <strong>when you have to specify multiple interrelated configurations (wherein some of them might be related to each other). </strong>All you need to do is- <em>bucket these configurations under different headers.</em></p><p>Consider the following illustration:-</p><p><strong>application.conf</strong></p><pre>com.ram.batch {<br>  spark {<br>    app-name = <strong>&quot;my-app&quot;<br>    </strong>master = <strong>&quot;local&quot;<br>    </strong>log-level = <strong>&quot;INFO&quot;<br>  </strong>}<br>  mysql {<br>    url = <strong>&quot;jdbc:mysql://localhost/ram&quot; </strong><em>// Ignore PORT if its 3306<br>    </em>username = <strong>&quot;root&quot;<br>    </strong>password = <strong>&quot;mysql&quot;<br>  </strong>}<br>}</pre><p>In the above file, you bucket the configurations related to <strong>spark/mysql</strong> under the respective headers to improve the readability. You can also have <strong>nested structures with any depth</strong> using approach.</p><p>So, lets see how to read these configurations:</p><ol><li>Add the following dependency to your <strong>pom.xml</strong></li></ol><pre>&lt;<strong>dependency</strong>&gt;<br>    &lt;<strong>groupId</strong>&gt;com.typesafe&lt;/<strong>groupId</strong>&gt;<br>    &lt;<strong>artifactId</strong>&gt;config&lt;/<strong>artifactId</strong>&gt;<br>    &lt;<strong>version</strong>&gt;1.2.0&lt;/<strong>version</strong>&gt;  // You can change this version<br>&lt;/<strong>dependency</strong>&gt;</pre><p>2. Use the following lines of code to read the config parameters:</p><pre><strong>// </strong>Assuming that <strong>application.conf</strong> is in the root folder of your application</pre><pre>// ConfigFactory looks for the name <strong>application.conf by default</strong> so you need not give this name</pre><pre><strong>val config</strong> = ConfigFactory.<em>load</em>(&quot;application.conf&quot;).getConfig(&quot;com.ram.batch&quot;) </pre><pre><strong>val sparkConfig</strong> = config.getConfig(&quot;spark&quot;)</pre><pre><strong>val mysqlConfig</strong> = config.getConfig(&quot;mysql&quot;)</pre><pre><strong>val appName</strong> = sparkConfig.getString(BatchConstants.<em>GET_APP_NAME</em>)</pre><pre>println(appName) // prints <strong>my-app</strong></pre><h3>Reading configurations from .properties file</h3><p>Use this approach <strong>when you have a set of unrelated configurations and you need to bundle them in a single file</strong>(this file may be environment specific i.e. stage/dev/prod)</p><p>Consider the following illustration:-</p><p><strong>application.properties</strong></p><pre><strong>hbase_table_name</strong>=mytable<br><strong>hbase_txn_family</strong>=txn<br><strong>hbase.zookeeper.quorum</strong>=localhost<br><strong>zookeeper.znode.parent</strong>=/hbase-secure<br><strong>hbase.zookeeper.property.clientPort</strong>=2181</pre><p>In the above file, you specify the configurations simply as a <strong>key-value</strong> map i.e. as a set of <strong>properties</strong>.</p><p>Use the following snippet to read this type of properties:</p><pre><strong>// </strong>Assuming that <strong>application.properties</strong> is in the root folder of your application</pre><pre><strong>val url</strong> = getClass.getResource(&quot;application.properties&quot;)<br><strong>val properties: Properties</strong> = new Properties()<br><br>if (url != null) {<br>  val source = Source.fromURL(url)<br>  properties.load(source.bufferedReader())<br>}<br>else {<br> logger.error(&quot;properties file cannot be loaded at path &quot; +path)<br> throw new FileNotFoundException(&quot;Properties file cannot be loaded);<br>}<br>   <br><strong>val table</strong> = properties.getProperty(&quot;<strong>hbase_table_name</strong>&quot;)<br><strong>val zquorum</strong> = properties.getProperty(&quot;<strong>localhost</strong>&quot;)<br><strong>val port</strong> = properties.getProperty(&quot;<strong>2181</strong>&quot;)</pre><p>That’s it!</p><p>Hope it helps. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f987f839f54d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing Scalable Systems: Part 2]]></title>
            <link>https://medium.com/@ramkarnani24/designing-scalable-systems-part-2-45c103ed112e?source=rss-a73b78eb1f0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/45c103ed112e</guid>
            <category><![CDATA[cap-theorem]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Ram Karnani]]></dc:creator>
            <pubDate>Thu, 10 Jan 2019 16:05:24 GMT</pubDate>
            <atom:updated>2019-01-10T17:01:51.614Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KlNIQet26hCixdUccmeBGA.jpeg" /></figure><p>The <a href="https://medium.com/@ramkarnani24/designing-scalable-systems-part-1-60279fc312f1?fbclid=IwAR2J-aoCy0qPC04mYQAuAZH-T4kxKNoQcqo5-OgAfQV-29FM35gPdByhvXc">part 1 of this blog</a> talked about CAP Theorem and the associated trade-off between availability and consistency.</p><p>Also, it ended with a question on how to achieve “Availability”... Did you get the chance to think about that?</p><h3>On a high-level, you can achieve availability using either: <strong>Fail-over Mechanism </strong>or<strong> Replication.</strong></h3><h3><strong>Understanding Fail-over Mechanism</strong></h3><p>Let’s say you get an opportunity to travel abroad for work, what would be the first thing that you would look out for?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*W4j4ub8u96bXm1oTNz69Zw.jpeg" /></figure><p>Forex card is like a debit card but you can load multiple currencies in it and when you apply for a Forex card with your bank, you generally get 2 cards(at least my bank gave me 2 💳).</p><p>Let’s say you have arrived in the foreign land and immediately go on a shopping spree(I wouldn’t recommend doing it 👀).</p><p>A question pops up in your mind - “Which of the two cards do I use?” My advice: Go ahead and use either one.</p><p>You keep on using this card, keep buying YSL, Gucci, Prada. But one fine day, you lose the card. Now, what do you do?</p><p>Now your second card would come to rescue as if it was on a <strong>standby </strong>mode and was just waiting for you to lose the first card (another scenario could be that you swiped your first card so much that it’s magnetic strip wore off).</p><p>But are you worried about your first card? Don’t be..!!</p><p>As soon as you use your second card, your first card would be <strong>automatically blocked</strong> and you don’t need to call your bank to get it blocked. Also, all the purchases you made using either cards will be written <strong>against the same bank account</strong>.</p><p>Now you have to admit that you didn’t have to stop your shopping spree due to card loss/damage. It was indeed a <strong>HIGHLY AVAILABLE SERVICE</strong>.</p><p><strong>Fail-over mechanism works just like that:</strong></p><p>There is one <strong>active</strong> node to serve the clients and the other nodes operate in the <strong>standby</strong> mode to achieve High Availability.</p><p>They are managed by a <strong>distributed</strong> <strong>coordination service </strong>such as <strong>Zookeeper</strong> (Just like the bank in the above example) to ensure that only one node is active at a time while the rest are on standby.</p><p>Analogous to the case where all your purchases went against a single bank account, all the updates to the active node are also written to a <strong>highly available storage shared</strong> between all the nodes. In the event of a failure, any standby node will resume work from where the active node failed and re-run the logs from the shared storage to achieve the same state as the active node.</p><p>Analogous to the case where your first card got blocked after you started using the second card, a mechanism formally known as <strong>Fencing </strong>is used to ensure that client updates don’t go to the previously active node.</p><p><strong>[Optional Read: For those of you who are familiar with HDFS]: </strong>The HDFS Namenode (centerpiece of the HDFS file system) makes use of this mechanism to achieve High Availability. Each namenode runs a lightweight failover controller process to monitor its heartbeat. In the event of a failure, failover mechanism is triggered and namenode is transitioned to the standby node which re-runs the edit logs from the shared storage such as Quorum Journal Manager(QJM) or NFS Filer and presume its role.</p><h3><strong>What is Replication?</strong></h3><p>Coming to replication, it is very important to replicate data across all nodes to ensure that the system is highly available to serve various requests.</p><p>There are essentially three classical approaches to synchronize the replicated data. What is common in all of these approaches? <strong>A client update arrives at some node, it is then processed right there and then passed as a downstream update to the rest of the nodes.</strong></p><p>To see how these 3 approaches differ from each other, imagine a scenario where your team decides to launch a feedback campaign to gather customer opinions.</p><p>One solution is to appoint an individual from your team to collect all the feedback and then pass it on to the team members. This is precisely our first approach namely <strong>Primary Copy,</strong> all the client updates are processed by one primary copy node and then propagated to the other nodes in the cluster, a system commonly known as <strong>Master-Slave system.</strong></p><p>Second solution is, all of you going out to take the feedback and then passing it on to each other. The second approach — <strong>Multi-Master</strong> does exactly this i.e. allows all the nodes to process the client updates and then propagate these updates to the other nodes, a system commonly known as <strong>Master-Master system.</strong></p><p>Third solution is to go out in groups for getting the feedback, to ensure that you don’t lose any of the feedbacks and jot down the same feedback after discussion and arriving at a consensus. This is the third approach of <strong>Quorum Consensus</strong>, which requires the update to be completed at a quorum of nodes and then acknowledged to the client, i.e, the distributed transaction/update request has to obtain minimum number of votes in order to be allowed to perform an operation. This quorum can be achieved either using Primary Copy or Multi-Master.</p><blockquote><strong>In a nutshell</strong>: High Availability can be achieved using any of the above stated approaches but they all have one issue in common: <strong>Latency would be present in the system.</strong></blockquote><h3>Can CAP theorem help circumvent this issue? Oops… it overlooked it altogether!</h3><p>As per the CAP theorem, there 3 types of systems possible i.e. <strong>Consistent and Available(CA), Available and Partition Tolerant(AP), Consistent and Partition Tolerant(CP)</strong>. But if we think critically, there is an <strong>asymmetry</strong> in the role of <strong>C </strong>and <strong>A. </strong>In real-life scenarios, partitioning almost always exists — so CA system is platonic and you have a choice between CP and AP only. To understand it clearly, let’s take the following scenarios:</p><ol><li>When the system is <strong>consistent and partition tolerant: </strong>In the presence of network partitioning, we sacrifice availability for consistency. But what would happen if the network partitioning is not present? The system would always be available.</li><li>When the system is <strong>available and partition tolerant: </strong>In the presence of network partitioning, we sacrifice consistency for availability. But what would happen if the network partitioning is not present? The system would still operate with the same loosened consistency model.</li></ol><p>So, while designing your system, if you choose to sacrifice <strong>Availability, </strong>you actually need to make this choice only in the case of network partitioning, but if you choose to sacrifice <strong>Consistency</strong>, you would have to do it all the time — whether or not network partitioning exists.</p><h3>Because of the asymmetry between the roles of C,A and P and no considerations for the Latency bit, Daniel Abadi proposed <a href="http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html"><strong>PACELC</strong></a>(pronounced: pass-elk).</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*jGK-zHisTfh82WiQYUBrPw.jpeg" /></figure><blockquote>Stated in words:<strong> </strong>If <strong>network partitioning</strong> is present, you would have to make a tradeoff between <strong>availability</strong> and <strong>consistency</strong> <strong>otherwise</strong> in the absence of network partitioning, system would <strong>always</strong> be <strong>available</strong> so you would need to make a tradeoff between <strong>latency</strong> and <strong>consistency</strong>.</blockquote><p>Before going into the details of this tradeoff, lets talk about one <strong>more important concept </strong>which affects the state of a distributed system — <strong>Distributed Caching</strong>.</p><p>Leading technology companies require hundreds of GBs of cache to serve the customers faster. But since this much<strong> In-Memory cache</strong> can’t be made available on a single system, we need to use <strong>Distributed caches </strong>like Memcache, EHCache, JBoss Cache etc.</p><p>Depending on your system size, either of in-memory or distributed cache can be used, so lets talk about the different caching modes:</p><ol><li><strong>Write-through</strong>: <em>Write the data to cache and disk memory</em><strong> </strong>in an atomic manner.</li><li><strong>Write-behind</strong>: Just <em>write the data to cache</em> and it would <em>later be moved to disk memory</em> in an <em>asynchronous </em>manner by some background thread.</li><li><strong>Write-back</strong>: Don’t write anything to the cache, simply write the data <em>directly to disk memory</em><strong>.</strong></li></ol><p>It was just an overview of different caching modes but hey…. did you notice something? Are you able to relate this with <strong>PACELC</strong>?</p><p>Lets take a critical look on the ‘<strong>LC</strong>’ part of PACELC and try to correlate it with the caching modes explained above:</p><p><strong>Write-through </strong>cache is writing the data in an <strong>atomic manner</strong> to both cache and disk memory implying<strong> strong consistency </strong>is present but at the cost of what?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*nsSOcXK3Tyh5mBIhiLG29A.jpeg" /><figcaption>Write-through cache</figcaption></figure><p>The atomic write operation would be <strong>slow</strong> in comparison to a single cache write and hence you are essentially <strong>trading-off higher</strong> <strong>write latency for strong consistency</strong>.</p><p><strong>Write-behind </strong>cache is only writing to cache, not to memory and therefore it would be very <strong>fast</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*us3syCsM-1T58yi0Ck_ALA.jpeg" /><figcaption>Write-behind cache</figcaption></figure><p>All the write requests would be queued in <strong>write-behind queue</strong> and would be written to the disk memory by some background process after a configurable delay.</p><p><strong>But what will happen in case of a machine failure?</strong></p><p>All the cache data present in this machine would be<strong> lost! </strong>But here’s a catch -you can use any of the <strong>replication mechanisms</strong> discussed above to take a <strong>backup of the write-behind queue</strong> and this data can be eventually written to the disk memory.</p><p>So the trade-off in Write Through Cache<strong> </strong>is <strong>Eventual consistency for Lower write latency.</strong></p><p><strong>Write-back</strong> is directly writing to disk memory. This will ensure that the data has <strong>strong</strong> <strong>consistency. </strong>But what would happen in case of <strong>read operation</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*HObJU1kzvHEeWd46XYSpyg.jpeg" /><figcaption>Write-back cache</figcaption></figure><p>Well, you would have to go to the memory, get the data in cache and then serve it to the client. So here you are making a tradeoff between <strong>read latency and strong consistency.</strong></p><blockquote><strong>Summing up:</strong> We talked about <strong>multiple approaches to availability</strong>, introduced the<strong> missing piece of latency</strong> in our CAP to reach <strong>PACELC</strong> and understood PACELC with the help of different caching modes.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45c103ed112e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing Scalable Systems: Part 1]]></title>
            <link>https://medium.com/@ramkarnani24/designing-scalable-systems-part-1-60279fc312f1?source=rss-a73b78eb1f0a------2</link>
            <guid isPermaLink="false">https://medium.com/p/60279fc312f1</guid>
            <category><![CDATA[scalable-systems]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cap-theorem]]></category>
            <category><![CDATA[consistency]]></category>
            <dc:creator><![CDATA[Ram Karnani]]></dc:creator>
            <pubDate>Tue, 01 Jan 2019 16:40:18 GMT</pubDate>
            <atom:updated>2019-01-10T18:56:05.943Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*-1p6MLl3MT4XGnyd80KSuw.jpeg" /></figure><blockquote><strong>How do you design a system?</strong></blockquote><p>Lets look at a simplified view of a payments company where the following requests could come:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*DuhsSP-ZUnRGoFNaCmfEuA.jpeg" /><figcaption>System 1</figcaption></figure><ol><li><strong>Money transfer requests</strong> come in and load balancer distributes these requests to different machines. Each machine executes the write transaction and commits it to our SQL db.</li><li>User <strong>requests</strong> transaction reports, these requests come to the load balancer and then to machines and these machine execute read transaction on our SQL db to generate the transaction reports and serve them to the user.</li></ol><p>This system should work without any glitches but<strong> is it scalable?</strong></p><p>In a situation where the number of users and transactions keep on increasing, wouldn’t our SQL db become a bottleneck? How? Because there is an upper limit to how many concurrent read/write you can perform.</p><p>This simple yet powerful use case helps us to correct our initial question-</p><blockquote><strong>How would you design a scalable system?</strong></blockquote><ol><li>To conquer this problem of scalability and handle the <strong>ever increasing load</strong>, one of the solution could be <strong>Vertical Scaling</strong> i.e. using powerful computers with lots of RAM, CPUs etc. to handle the user load and work on the data. But again, there would be an upper limit to the performance you achieve using the hardware modifications.</li><li>Another solution is to use the commodity hardware and perform <strong>Horizontal scaling </strong>i.e. work with Distributed Systems.</li></ol><blockquote><strong>Restating</strong>: <strong>Our goal is to strive for scalability which can be made possible using Horizontal scaling i.e. using Distributed systems instead of a Centralized system.</strong></blockquote><p>A simple solution for our above stated problem would be to use the following architecture:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*2RkzmCvJycXBuFTMyam_dQ.jpeg" /></figure><ol><li>Write operation happens on SQL db and is propagated through Kafka to HDFS Ingest pipeline.</li><li>HDFS will either be serving the requests directly(in case of streaming data access pattern) or will be first moving the data to a fast serving NoSQL like HBase(Reporting DB) and then serving the requests.</li></ol><p>Now, as you can see, there can be multiple approaches to solve a problem but all of these approaches require an understanding of how distributed systems work.</p><p>Simply stated, <strong>Distributed systems </strong>refer to different computers connected via network and talking to each other. But in the presence of such an environment, how would you answer the following questions:</p><p><strong>— How to maintain data state?<br> — How to perform the computations on data?</strong></p><blockquote><strong>You can think of it as similar to OOP concept of class: <br>Field are similar to maintaining the state and methods are equivalent to computations on data.</strong></blockquote><ol><li>The question of maintaining the data state needs an understanding of topics like CAP theorem, NoSQL/RDBMS, Distributed caching, HTTP caching etc.</li><li>And the question of Computations on data requires the understanding of:<br> — Network patterns<br> — Load balancers<br> — Parallel computing<br> — Event-driven architecture</li></ol><p>I will be talking about <strong>CAP theorem</strong> in this blog and will cover rest of the topics in my subsequent blogs.</p><blockquote>CAP theorem states that in case of network partitioning, <strong>you can either achieve consistency or availability </strong>i.e. only 2 types of system are possible: <strong>CP or AP</strong>.</blockquote><p>Hey, but what is that CP and AP? Here’s what each letter stands for:-</p><p><strong>C = Consistency </strong>= data should be consistent on read/write i.e. everyone should have single view of data.<br><strong>A = Availability</strong> = data should be highly available i.e. there should not be any downtime associated with data read/write.<br><strong>P = Partition Tolerance </strong>= refers to network partitions</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*U2ypGsrhj9wAe9YdtZAPxw.jpeg" /><figcaption><strong>Lets dig deeper into CAP</strong></figcaption></figure><p><strong>Network Partitioning:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*fW0XnzASEeRgtpzlEPZKNQ.jpeg" /></figure><p>Imagine this simple scenario where nodes are connected to each other using some network pattern and lets say some network issue arises between two subsets of nodes. As shown below, now we have two different systems where a read/write request can reach i.e. Network-1 and Network-2.</p><p>With respect to above image, lets consider both the systems.</p><ol><li><strong>CP: Consistency with Network Partitioning:</strong> Imagine a scenario where a write request comes in — it could either go to Network-1 and Network-2. <em>But since the network issue is present, you can’t achieve consistency between both the networks</em>. Hence to achieve consistency, you would have to <em>make one of the network unavailable</em> and then you would be able to make your system <em>highly consistent</em> since all the requests whether read or write would be served by Network-1.</li><li><strong>AP: Availability with Network Partitioning:</strong> Lets consider a scenario where both the networks are available i.e can serve read/write request. <em>But because of network partitioning, you can’t achieve consistency between both the systems. </em>So although the systems are available at all times, they will achieve eventual consistency, when the network issue is resolved.</li></ol><p>These constrains are present because network partitioning can happen in the world of distributed systems.</p><p>But what about the centralized systems?</p><p>In an <strong>RDBMS</strong> like system, we have a centralized database and hence network partitioning won’t be present. That is the reason they provide the <strong>ACID</strong> properties and thereby support consistency and availability whereas NoSQL stores are an example of distributed database system i.e. data is partitioned across multiple machines and hence they could either run in CP mode or AP mode.</p><p>Here is a Venn diagram to illustrate the CAP theorem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*WPnv_6sG9k4oG3S1A09MDA.jpeg" /><figcaption>CAP Theorem</figcaption></figure><p>So, this was a high-level overview of CAP terms and I hope you are left with questions like-</p><ol><li><strong>How can I achieve availability?</strong></li><li><strong>What could be different levels of consistency?</strong></li></ol><p>These are huge topics in themselves and require a lot of discussion. So until I come up with the answers in my next blog, here are some pointers for these questions:</p><p>1. <strong>Availability</strong> can either be achieved using <strong>fail-over mechanisms</strong> or using <strong>replication</strong> where replication mode could be either: Active(Push) or Passive(Pull).</p><p>2. There are different types of <strong>consistency</strong> though the prominent ones are <strong>Strict consistency, Sequential consistency, Linearizable consistency </strong>and<strong> Eventual consistency</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60279fc312f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>