<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alok Kumar Singh on Medium]]></title>
        <description><![CDATA[Stories by Alok Kumar Singh on Medium]]></description>
        <link>https://medium.com/@mail-alok87?source=rss-f4ee7d11538e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qITmZ6KSzyhoEGhte6AtUQ.jpeg</url>
            <title>Stories by Alok Kumar Singh on Medium</title>
            <link>https://medium.com/@mail-alok87?source=rss-f4ee7d11538e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:41:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mail-alok87/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Posting from a new location]]></title>
            <link>https://mail-alok87.medium.com/posting-from-a-new-location-2b31dc150dd3?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b31dc150dd3</guid>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Sat, 26 Nov 2022 10:12:01 GMT</pubDate>
            <atom:updated>2022-11-26T10:12:01.187Z</atom:updated>
            <content:encoded><![CDATA[<p>The new sweet spot for my blog posts <a href="https://alok87.in./">https://alok87.in</a></p><p>I share the blogs via LinkedIn <a href="https://www.linkedin.com/in/alok87/recent-activity/shares/">https://www.linkedin.com/in/alok87/recent-activity/shares/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b31dc150dd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Continuing on Weak Isolation Levels…]]></title>
            <link>https://mail-alok87.medium.com/continuing-on-weak-isolation-levels-8dde49a39205?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dde49a39205</guid>
            <category><![CDATA[isolation]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[race-condition]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Sat, 26 Feb 2022 22:49:55 GMT</pubDate>
            <atom:updated>2022-02-26T23:17:28.726Z</atom:updated>
            <content:encoded><![CDATA[<h4>Databases by default do not prevent us from all concurrency issues. Let’s understand more issues and their solutions.</h4><p>In the last <a href="https://mail-alok87.medium.com/weak-isolation-levels-c183a8c01760">blog</a>, we looked at DirtyRead, DirtyWrite and Read Skew concurrency problems and learnt how Read Committed and the Snapshot Isolation levels solve them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Nta81AY2FOkcTR2QTWD-7g.png" /><figcaption>But there are more problems!</figcaption></figure><p>Let’s learn about other concurrency problems that can still occur:</p><h3><strong>Lost Updates</strong></h3><p>If two transactions perform “Read, Modify and Write” concurrently, the change made by one of the transaction could get lost if one makes a change on the old value and not the latest value of the data object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*iSFekQm7J_l5_90ZWeYYbg.png" /><figcaption>Both tx1 and tx2 get the value of likes as 199, and tx1 incremented and updated it and then tx2 incremented it. But tx2 incremented 199 and not 200, so the update by tx1 got lost!</figcaption></figure><p>The read that resulted in the lost update is known as a Fuzzy Read. The lost update is a very common problem. Few scenarios of it occurring:</p><ul><li>Incrementing counter concurrently like above likes example. It can happen in a program as well. <a href="https://go.dev/play/p/JtCPUMPpWZG">Run this</a> program multiple times, you would see different result !</li><li>Updating JSON documents.</li><li>Updating Lists.</li></ul><p>Note: lost update can happen in Read Committed and Snapshot Isolation setting, as it depends on the application if it “read-modified and wrote”?</p><p>Let’s look at ways to prevent it:</p><h4><strong>Prevent Lost Update using Atomic Operations</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*UFtbR5-o_NBJihp3.png" /><figcaption>No lost update using database atomic operations! Mic drop the read-modify-write cycle !</figcaption></figure><p>Avoid <em>“Read-Modify-Write”</em> cycles and use atomic update operations provided by the database. Most of the database provides concurrency safe update operations using exclusive locks on the database object.</p><ul><li>MySQL updates: <em>update article set </em><strong><em>likes=likes+1</em></strong><em> where id=5;</em></li><li><a href="https://dev.mysql.com/blog-archive/partial-update-of-json-values/">Partially JSON value update</a>s on MySQL.</li><li><a href="https://redis.io/commands/INCR"><em>INCR likes</em></a><em> </em>in Redis.</li></ul><h4>Prevent Lost Update using Explicit Locks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*4LaEG_00LJ3jD8-4" /><figcaption>application explicitly locks the data it wants to read-modify and write!</figcaption></figure><p>The application can prevent lost updates by explicitly locking the database objects that need to be updated.</p><pre>BEGIN;</pre><pre>// lock all the rows returned by the query and read it<br>SELECT * FROM notification where `date` &gt;= &#39;2011-05-03&#39; FOR UPDATE;</pre><pre>// modify the previous read and write<br>UPDATE...  </pre><pre>COMMIT;</pre><p>Databases can detect Lost updates in concurrently executing read-modify-write cycles. It aborts the transaction for applications to retry. Snapshot Isolation mode in most databases can detect Lost updates, but it is very important to know that some databases <strong>like MYSQL/InnoDb’s Repeatable-read(snapshot isolation) do not detect lost updates and in such databases concurrently executing read-modify-write cycles are not safe.</strong></p><h4>Prevent Lost Update using Compare and Set</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*Q_gCIICSyVN9yFNUBgkLfg.png" /></figure><p>Databases which do not provide transactions provide “Compare and Set” operation to prevent lost updates by allowing updates to only happen if the value has not changed since you last read it. Details on whether the compare and set is a thread safe operation would depend on the database implementation and isolation setting. Please look at it before using it.</p><p>Let’s move on to more problems that can still occur…</p><h3>Write Skew</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aoG2GNmbHYlS4-t3eTjGYQ.png" /><figcaption>one patient got assigned to 2 doctors due to write skew</figcaption></figure><p>Example speaks louder than words: Say, we have an online doctor consultation application. Patient come online and start looking for a doctor. We broadcast patients to live doctors. Doctor, on accepting the request to consult, starts a transaction, checks if the patient is free to consult. If yes, then doctor <strong>self assigns</strong> the patient for consultation. In the above example, both the Dr Prashant and Dr Jignesh self assigned to consult Shubham. Really hope the Shubham gets better, now two excellent doctors are treating him :D. This is the write-skew concurrency problem.</p><p>There was no dirty write: Dr Prashant updated his data with Shubham, and Dr Jignesh also did the same, which is valid in Snapshot isolation.</p><p>Both Prashant and Jignesh read the same object<strong> but updated different objects </strong>resulting in write skew. <strong>Also, write skew is very less obvious, hard to detect, databases cannot auto detect it. </strong>As it is said, if a problem can happen will happen. The application developer really needs to be careful of this race condition.</p><h4>Preventing Write Skew</h4><p>Databases do not have auto detection of write skew. Being careful and aware of these issues is the first step. Also, it can be prevented by:</p><ul><li>Explicitly locking the rows the transaction depends on. or…</li><li>Multi object constraints using triggers or materialized views. or…</li><li>Using serialization isolation level, if that is an option.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*dRcqrqoNETErg_LVe_vUHA.png" /></figure><p>I would like to write on Phantoms, 2 Phase and 3 Phase Locking in other blog(s).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dde49a39205" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Weak Isolation Levels]]></title>
            <link>https://medium.com/practo-engineering/weak-isolation-levels-c183a8c01760?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c183a8c01760</guid>
            <category><![CDATA[isolation]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Thu, 24 Feb 2022 22:08:21 GMT</pubDate>
            <atom:updated>2022-02-26T22:54:33.386Z</atom:updated>
            <content:encoded><![CDATA[<h4>Database Isolation levels can get confusing and hard to understand. But understanding it is a must to build reliable systems since databases by default provide weaker isolations which do not prevent all concurrency issues.</h4><p><em>I wrote this after a joyful read of Martin Kleppmann’s Design Data-Intensive Application. In this write up, I would work on covering the isolation levels and the concurrency problem they solve. For those interested in the detailed read, I refer you to the </em><a href="https://www.goodreads.com/book/show/23463279-designing-data-intensive-applications"><em>book</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*0iQ5fL1Us06g9DfB" /></figure><p>Building a fault-tolerant system is hard. Many things can go wrong. Transactions in databases have helped in building reliable fault-tolerant systems for decades. Transaction simplifies application development and almost all relational databases support transactions.</p><p>Transaction <strong>groups several read and write into one logical unit </strong>which is executed as one operation. But why do we need to do that? Why cannot we run single operations? We can, but handling all the failure scenarios while preventing the concurrency issues and keeping the group of operations reliable is hard. Transactions in the database make it easy for the application developers to access a database without worrying about error scenarios and concurrency issues.</p><p>The well-known acronym ACID helps in understanding the safety guarantees transaction provides. Every database implementation of ACID is slightly different, as <strong>ACID is an acronym</strong> coined by Joe Hellerstein <strong>and it is not an open standard that every database followed. </strong>Let’s briefly understand ACID from the perspective of the problems they solve for the application developer.</p><h4><strong>Atomicity</strong></h4><p>As said earlier, a transaction is a group of operations, a logical unit that executes together. This logical unit can either succeed or fail when executed. It can never be half complete. That’s atomicity. As coined by Martin in his book, it makes more sense to call it <strong>A for Abortability </strong>from what safety guarantee it brings. Because of this property, application developers can safely abort transactions on errors and have all changes from that transaction discarded safely. On abort, the application developers can safely retry transactions and there is no worry of duplicates as the guarantee ensures no changes happened in the earlier transaction. This makes things easy for the application.</p><h4>Consistency</h4><p>Joe Hellerstein coined ACID, and has remarked that C in ACID was “<em>tossed to make the acronym work</em>”. Consistency means certain invariants about your data should always be true and writes should preserve the invariants. This is true for some invariants like foreign key constraints, unique constraints, <strong>but in general, the application is responsible for defining its transaction correctly so that the invariants are obeyed and consistency is preserved. </strong>So it is right to say that consistency is an application responsibility and is not a safety guarantee provided by the database. <strong>C for nothing? </strong>People who do not remember what was C in ACID, do not worry it was nothing!</p><h4>Durability</h4><p><strong>D for Durability</strong> guarantee means data written in successful transactions will never get lost. That’s the guarantee. But let’s consume this information with more context:</p><ul><li>In a single node database, durability means the data has been written to nonvolatile storage as a Hard disk or SSD. Write-Ahead Log in databases allows data recovery in case of disk corruption.</li><li>In a replicated database, durability means the data has been copied to some number of nodes and the guarantee here is provided by waiting for replication to other nodes complete. Write is successful if the data has been successfully replicated to N nodes.</li><li>No Durability can be guaranteed if all the hard disks and backups are destroyed or if the <a href="https://www.jpl.nasa.gov/asteroid-watch/next-five-approaches">Meteor hits</a> the Earth! There is no <strong>absolute</strong> guarantee.</li></ul><h4>Isolation</h4><p>There is no problem if multiple running transactions are accessing different data in the database. But if they access the same data in the database, then concurrency problems, i.e. race conditions, may arise. <strong>I for Isolation</strong> is the guarantee that transactions are completely isolated from each other when executing concurrently. For a transaction, it is like it is the only executing transaction in the database. The guarantee provided by the database here is the result after multiple concurrently ran transactions finish is the same if the transactions were executed serially one by one (Strongest isolation level).</p><p>In real life, executing transactions serially is a tremendous performance hit, hence most of the database gives weaker isolation levels as defaults. <strong>The weaker isolation levels protect us from some concurrency issues, but not all.</strong> So, an application developer needs to understand them. Let’s understand the weaker isolation levels and the concurrency issues they solve.</p><p>Level 0 to above are in order of weaker to stronger isolation order.</p><h3>Level 0: Read Committed</h3><p>Most basic isolation level and it solves the Dirty Read and Dirty Write phenomena:</p><p><strong>Dirty Read:</strong> When a transaction reads data that has not been committed.</p><p><strong>Dirty Write:</strong> When a transaction overwrites the value written by an ongoing transaction.</p><p><strong>Problems with Dirty Read: </strong>Listing down a few unintended outcomes that can occur.</p><ol><li>If a transaction updates many objects in a database and other transactions saw only some changes, then the other transaction can take incorrect decisions and the database will look inconsistent to it. An excellent example to explain is: Say, Alice has two bank accounts in a Bank with 500Rs in each and she requested the bank to transfer 100Rs from one account to another.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBzdehcDl397WsNy_XnoOA.png" /><figcaption>The bank transfer transaction was half completed and alice tried to check her balance and got confused as she expected the total balance to never drop. Had she checked after the tx committed by the bank, she would have seen the expected total balance.</figcaption></figure><p>2. If a transaction reads un-committed data and later that data was rolled back it can become really confusing for the application and cause bugs that are hard to trace and fix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*ndc_7kjqd95Yple9SfbEPA.png" /><figcaption>tx1 change was rolled back but tx2 used the uncommitted amount and would have taken some decisions.</figcaption></figure><p><strong>Problems with Dirty Write</strong></p><p>Last write wins causing intermingled transactions making a mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqeQccdtV6rLy_nMghc_rQ.png" /><figcaption>the room got booked for charlie and the guest was bob who was expected to go to alice’s meeting room.</figcaption></figure><p>Read-Commited isolation level prevents the above problems. You get to read only the committed data from the database and you can only overwrite the last committed data.</p><p><strong>How does the database implement Read Committed?</strong></p><ul><li><strong>Row Level Write Locks</strong> are used to lock the data in a transaction. Other transactions have to wait for the lock to release before they can change the same row. This is how dirty writes are prevented!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*anZhUPZRviXx578FZP-tFw.png" /><figcaption>the second tx on right hangs waiting for the tx on left to finish(abort/commit)</figcaption></figure><ul><li><strong>No read-locks, databases can remember old values:</strong> To prevent dirty read databases, do not use read locks because a single long-running transaction can block many read-only transactions to wait until the long-running transaction finishes. So, if a transaction holds the write lock on data, the database remembers both the old and new values of that data. It gives back the old value to all the transactions that want to read that data. This Is how dirty reads are prevented!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IMLUlM3NU0WjIgnMM3ZFhg.png" /><figcaption>the tx on left sees the value it changed, and the tx on right sees the last committed value</figcaption></figure><h3>Level 1: Repeatable Read / Snapshot Isolation</h3><p>This is the default isolation level for many relational databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*fanAXVKTdWGLtM9b6qn-Sw.png" /></figure><p>Read Committed solved a few of the concurrency problems/phenomena. But there are more concurrency issues that can occur. Let’s look at them:</p><p><strong>Read Skew</strong></p><p>The transaction can read different objects at different times, causing a read skew. This is clearly understood with an example: say Alice has two accounts A and B in a Bank and it is a joint account with Bob. Bob made a transfer from account A to B without informing Alice and Alice&#39;s balance may look inconsistent as she did not expect the total balance to drop which has not dropped but it looked to Alice as it did(she saw 900 in total). This anomaly is known as non-repeatable Read or Read-Skew.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lswK-QWyKUJBHs4TiyGBcg.png" /><figcaption>though bob&#39;s transaction is totally valid, since alice saw different accounts balance at a different time, it resulted in a read skew!</figcaption></figure><p>Wait! How is it different from Dirty Read? It is not a dirty read as both the reads of Alice are from committed data. <strong>It was just the timing of the read causing the problem.</strong> This <strong>timing anomaly </strong>is expected in a Read Committed Isolation level.</p><p><strong>Solution</strong>: Repeatable Read or Snapshot Isolation level solves this by giving each transaction <strong>a point in time consistent snapshot of the database. </strong>The point is time is the start of the transaction. The transaction during the execution cannot see any other data or changes that were committed after it started. Almost all relational databases provide this isolation level.</p><p><strong>How does the database implement Snapshot Isolation?</strong></p><ul><li><strong>Row Level Write Locks: </strong>This is exactly how Read Committed Isolation level does it. A transaction acquires the lock and then updates the data object, blocking and making other transactions wait to write the same data object.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MfGTTieZKcjxTh7HmqVatw.png" /><figcaption>the right tx hangs waiting for the first one to finish</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1MrlyjABbzTQiz4isma3BA.png" /><figcaption>also the tx2(right) does not get impacted by the lock tx1(left) acquired, <strong>reads are not impacted by write!</strong></figcaption></figure><ul><li><strong>No read-locks, databases can remember old values: </strong>Yes this is the same as Read-Committed Isolation’s implementation, the only thing is database here needs to remember not just 2 values but many versions of the data object. Since each transaction will see its own version for the read. This technique is known as Multi-Version Concurrency Control (MVCC).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l0I75BX3FNfTAf9BjxgPGQ.png" /><figcaption>tx1(left) makes 2 updates, tx2(middle) started after first update, tx3(right) started after second update. Both tx2 and tx3 see data at the point when they started. Repeatable read!</figcaption></figure><p><strong>MVCC Implementation</strong></p><ol><li>Unique ever-increasing transaction ID for each transaction. tx1, tx2, tx3,…</li></ol><p>2. Table Row gets the following extra metadata fields:</p><ul><li>created_by: contains the transaction id which caused the change.</li><li>deleted_by: contains transaction id that requested deletion. deletion happens async by the garbage collection process when it is sure no tx can access the deleted data.</li></ul><p>3. The update is translated into a delete and create. So in case of updates, 2 rows are created: one with deleted_by updated with tx id and the other row with created_by updated with tx id.</p><p>4. Visibility rules: Reads use the transaction IDs to determine which database object it can see and what is hidden.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMC1K6cgqukMcR3BcHKRDA.png" /><figcaption>MVCC implementation</figcaption></figure><p>There are many other types of concurrency problems that can still occur, and there is a variety of solutions made to solve them.</p><p>Click <a href="https://mail-alok87.medium.com/continuing-on-weak-isolation-levels-8dde49a39205"><em>here</em></a> to go to the<a href="https://mail-alok87.medium.com/continuing-on-weak-isolation-levels-8dde49a39205"> 2nd part of this blog</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*HqKHjY8GnEqRNPoz.gif" /><figcaption>what isolation level is this?</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c183a8c01760" width="1" height="1" alt=""><hr><p><a href="https://medium.com/practo-engineering/weak-isolation-levels-c183a8c01760">Weak Isolation Levels</a> was originally published in <a href="https://medium.com/practo-engineering">Practo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-sourcing Tipoca Stream]]></title>
            <link>https://medium.com/data-science/open-sourcing-tipoca-stream-f261cdcc3a13?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f261cdcc3a13</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[realtime]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Fri, 18 Jun 2021 20:05:31 GMT</pubDate>
            <atom:updated>2021-06-21T12:35:16.898Z</atom:updated>
            <content:encoded><![CDATA[<h4>Take the data from Kafka to Redshift in near-realtime</h4><p>The story of making a cloud-native near real-time data pipeline using Kafka, KafkaConnect, and RedshiftSink in AWS. CDC + Sink. RedshiftSink is the new open source project from Practo. It comes with a high-performance low overhead data loader for loading data in Redshift with rich <a href="https://github.com/practo/tipoca-stream/blob/master/MASKING.md">masking support </a>to ensure you can create universal data access in your organization in a privacy-preserving way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*omYOg7ceJzciV6nbaZZ_5A.png" /><figcaption>Tipoca Stream 2021: Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/arch.png">Source</a></figcaption></figure><p>Dive right into our <a href="https://github.com/practo/tipoca-stream">repo on GitHub</a> to get started. 🏊</p><h3>Backstory</h3><p>In 2015, we wrote<a href="https://starwars.fandom.com/wiki/Tipoca_City"> Tipoca</a>, an internal transactional data copier, to bootstrap Practo’s warehouse. It used to copy data from our transactional databases to Redshift with PII masking support. It refreshed data every night, doing a full recreate applying the latest mask configurations. For the first time, everyone at Practo got access to all data and the ability to join data across various systems while preserving our customers’ privacy.</p><p>Back then, building distributed systems required a lot of work. We built a system with different processes, running across multiple workers, and managing their coordination with retries. It has become much easier to build distributed systems over the years with the help of open-source technologies and tools. This is how Tipoca looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGCGZdhUHULn1h88FuQ-mQ.png" /><figcaption>Tipoca in 2015: Image by author</figcaption></figure><h3>Motivation to go beyond Tipoca</h3><ol><li>Real-time data analysis.</li><li>Faster Redshift queries. Heavy batch load occupies Redshift resources. The need to sync the data diff.</li><li>Data-driven products.</li><li>Recreating data every day is not scalable and can be expensive.</li><li>Remove analytics dependencies on RDS read replicas.</li></ol><p>This led to the birth of Tipoca Stream, bringing to an end Tipoca’s 5-year run.</p><h3>Introducing Tipoca Stream</h3><p>In July 2020, we began discussing plans to move to a<a href="https://en.wikipedia.org/wiki/Change_data_capture"> CDC</a> platform. The idea was to stream data from transactional databases running on RDS to Redshift in near real-time. We opted for Kafka primarily for two reasons — first, Kafka is open; second, it has a good data ecosystem around it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omYOg7ceJzciV6nbaZZ_5A.png" /><figcaption>Tipoca Stream 2021: Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/arch.png">Source</a></figcaption></figure><p>The first part (CDC) was available as open-source. We used<a href="http://debezium.io/"> Debezium</a> to write the RDS streams to Kafka.<a href="https://strimzi.io/"> Strimzi</a> had open-source <strong>KafkaConnect</strong> and <strong>Kafka</strong> as <strong>Kubernetes CRDs.</strong></p><pre>kubectl get kafka<br>kubectl get kafkaconnect<br>kubectl get kafkaconnector</pre><p>We used AVRO to store data in compact binary format in Kafka. AVRO is fast and with AVRO, you can define and store the data schema in Schema Registry.<a href="https://github.com/practo/tipoca-stream/blob/22ba8370550891e8b64c667785136e51b603fb38/redshiftsink/pkg/schemaregistry/schemaregistry.go#L12-L19"> This is the interface</a> for the schema registry we use.</p><p>RedshiftSink solves the second part. It sinks the data from Kafka topics to Redshift. It has a rich <a href="https://github.com/practo/tipoca-stream/blob/master/MASKING.md">masking</a> support and supports all schema migrations.</p><p>More eyes make better software! we are excited to open-source RedshiftSink. It is available as a Kubernetes RedshiftSink operator. 🎉</p><pre>kubectl get redshiftsink</pre><p>You <strong>can just jump into the</strong> <a href="https://github.com/practo/tipoca-stream">repo on GitHub</a> to try out this for yourself. Read on to understand the core concepts of the implementation below.</p><h3>Core Components</h3><h3><strong>RedshiftBatcher</strong></h3><p>RedshiftBatcher reads the data from Kafka, masks them and uploads them as micro batches to S3. It then signals the RedshiftLoader for loading these batches to Redshift.</p><p><strong>Why batch on a streaming platform?</strong></p><p>Redshift is good at ingesting millions of records quickly using<a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html"> COPY</a> command. Each file copy has an overhead, hence there is a balance required between the number of files and the number of records in each file. That is why we copy in micro batches. You can control these using maxSizePerBatch and maxWaitSeconds.</p><pre>spec:<br>  kafkaTopicRegexes: &quot;db.inventory.*&quot;<br>  batcher:<br>    sinkGroup:<br>      reload:<br>        maxReloadingUnits: 2<br>        maxSizePerBatch: 10Mi<br>        maxWaitSeconds: 30</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*G_QTi1vfaiaMMXOM" /><figcaption>Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/arch-batcher.png">Source</a></figcaption></figure><p>Every topic is one table and has its own Kafka Consumer Group. Following happens for every topic in the batcher:</p><ul><li>The Manager routine creates and gracefully shuts down consumer groups.</li><li>Sarama consumer groups watch over the Kafka topics and populate the read channel with new messages.</li><li>Reader writes the messages received to the read buffer.</li><li>Reader flushes the batch to the process channel when the batch size gets hit or when the time elapses. Buffered channel is used to separate out the receiving and processing of messages.</li><li>Orchestrator routine makes the processing of multiple batches happen concurrently. It signals the load of these batches to Kafka when processing completes.</li><li>Commit to Kafka marking the offsets as processed.</li></ul><p><strong>Responsibilities of the batcher:</strong></p><ul><li><strong>Transform:</strong> Transform every Debezium message in the batch to Redshift loadable format. We forked and changed the open-source<a href="https://github.com/practo/tipoca-stream/tree/master/redshiftsink/pkg/redshift"> Redshift</a> Library to keep all communication to Redshift as part of this library. We annotate every message with two additional fields: operation and kafka-offset. RedshiftLoader uses these fields to perform merge operations before loading to Redshift.</li><li><strong>Mask:</strong> Masking has always been the most important requirement for us. The data that goes from RDS to Redshift is completely masked by default, unless the developer has specified not to mask it. Unmasking configurations are kept in GitHub as files. Developers and analysts specify what not to mask, as GitHub Pull Requests. Once the security engineer reviews and approves the change, RedshiftSink Operator takes care of applying these mask changes to Redshift. RedshiftBatcher CPU requirement in the first time sink is also high because of the mask processing it has to do. There is room for <a href="https://github.com/practo/tipoca-stream/issues/173">improvement</a> here. Following are the masking features, for details on them please check<a href="https://github.com/practo/tipoca-stream/blob/master/MASKING.md"> this</a>.</li><li><strong>Upload the batch to S3</strong>: Once the transformation is complete, we upload the batch in S3. Batch write is an idempotent operation and so reprocessing of a batch is not a problem. But we do graceful shutdowns and the chance of reprocessing is very rare. Rare since the commits to Kafka are asynchronous to keep the batcher fast.</li><li><strong>Signal:</strong> For the topic db.inventory.customers, the batcher reads the messages from this topic, processes the data and uploads to S3. Then it adds a job in the topic loader-db.inventory.customers which is read by the RedshiftLoader to load the batch in Redshift. Below is the job schema:</li></ul><pre>var JobAvroSchema string = `{<br>    &quot;type&quot;: &quot;record&quot;,<br>    &quot;name&quot;: &quot;redshiftloader&quot;,<br>    &quot;fields&quot;: [<br>        {&quot;name&quot;: &quot;upstreamTopic&quot;, &quot;type&quot;: &quot;string&quot;},<br>        {&quot;name&quot;: &quot;startOffset&quot;, &quot;type&quot;: &quot;long&quot;},<br>        {&quot;name&quot;: &quot;endOffset&quot;, &quot;type&quot;: &quot;long&quot;},<br>        {&quot;name&quot;: &quot;csvDialect&quot;, &quot;type&quot;: &quot;string&quot;},<br>        {&quot;name&quot;: &quot;s3Path&quot;, &quot;type&quot;: &quot;string&quot;},<br>        {&quot;name&quot;: &quot;schemaId&quot;, &quot;type&quot;: &quot;int&quot;},<br>        {&quot;name&quot;: &quot;maskSchema&quot;, &quot;type&quot;: &quot;string&quot;},<br>        {&quot;name&quot;: &quot;skipMerge&quot;, &quot;type&quot;: &quot;string&quot;, &quot;default&quot;: &quot;&quot;}<br>    ]<br>}`</pre><h3>RedshiftLoader</h3><p>RedshiftLoader loads the processed batches from S3 to Redshift.</p><p>The RedshiftLoader watches over the topics written by the batcher. It is very important to perform the load using as few loaders as possible because of the limited Redshift connections. In a single loader pod, we share Redshift connections across all loader routines.</p><pre>spec:<br>  loader:<br>    redshiftGroup: non_pii<br>    redshiftSchema: inventory<br>    sinkGroup:<br>      all:<br>        maxSizePerBatch: 1Gi</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*yoZQWfTLQdeJ3QeG" /><figcaption>Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/arch-loader.png">Source</a></figcaption></figure><p>You specify how much to load by maxSizePerBatch !</p><p>Loader expects the same schema for every message in the batch. It is the responsibility of the batcher to flush the existing batch if it finds a new schema for the message.</p><p>Both Debezium and the RedshiftBatcher store the schema in the Schema Registry. The batcher and loader both speak to the schema registry when required to get the schema related information.</p><p><strong>Responsibilities of the loader</strong></p><ul><li><strong>Schema Migration:</strong> The development team keeps changing the RDS schema with their code releases. We need to keep the redshift schema in sync with the changes in schema happening in RDS. So, we wrote a schema migrator. By reading the first message in the batch, the loader figures out if it needs to perform the schema migration. It does that by extracting the schema-id from the job payload. Then it gets the schema info from the schema registry and constructs an input Redshift table using that. Then it compares the input Redshift table with the actual table in Redshift and decides on the schema migration strategy. Redshift does not support all the SQL migration commands, for example, they support very few ALTER commands. The first strategy is to attempt an in-place migration, and if the first strategy is not possible, we recreate the complete table and replace it. The second strategy can be really time taking for big-tables. We have a plan to take the big schema migration out of the RedshiftLoader and let the operator perform the big schema operations.</li><li><strong>Merge Load: </strong>As stated earlier, COPY is the most efficient way to add data to Redshift. But COPY alone can only do Inserts. Therefore, we need to use a merge technique using a stage table to perform the updates in Redshift. YELP Redshift connector<a href="https://engineeringblog.yelp.com/2016/10/redshift-connector.html"> article</a> and the<a href="https://docs.aws.amazon.com/redshift/latest/dg/merge-examples.html"> AWS documentation</a> has a good explanation of using this technique. Though the YELP article was very helpful.</li></ul><h3><strong>RedshiftSink Operator</strong></h3><p>Kubernetes operators helped in keeping the orchestration and the operation out into the operator. Thank you Kubernetes!</p><pre>kubectl get deploy redshiftsink-operator</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*GGmP-AibXglaIEVS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*vVecgq4TftDapHiE" /><figcaption>Image <a href="https://twitter.com/kelseyhightower/status/1298343780307755008?s=20">Source</a></figcaption></figure><p>We solved the following using Kubernetes<a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/resource-management.md#the-kubernetes-resource-model-krm"> declarative APIs</a>:</p><ol><li>Live Mask Reloading.</li><li>Users get a simple clean API.</li></ol><p>The operator has helped us manage the first time sink of tables and also helps us in reloading tables in Redshift when mask configurations are updated.</p><p>Operator helps us solve the use cases described above but is not mandatory. You can install a batcher and loader and run them in any way possible. Kubernetes is not mandatory!</p><h3>Live Mask Reloading</h3><p>Mask changes require recreating the table with new mask configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*ZWmMrBkazPndlCXd" /><figcaption>Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/arch-operator.png">Source</a></figcaption></figure><p>Let’s understand live mask reloading with an example. In the example, we are sinking the topics that match the regex: db.inventory.*</p><pre>apiVersion: tipoca.k8s.practo.dev/v1<br>kind: RedshiftSink<br>metadata:<br>  name: inventory<br>spec:<br>  kafkaTopicRegexes: &quot;^db.inventory*&quot;<br>  batcher:<br>    mask: true<br>    maskFile: <a href="https://github.com/practo/tipoca-stream/blob/master/redshiftsink/pkg/transformer/masker/database.yaml">&quot;</a><a href="https://github.com/practo/tipoca-stream/blob/master/pkg/transformer/masker/database.yaml">database.yaml</a>&quot;<br>  loader:<br>    redshiftSchema: &quot;datawarehouse&quot;<br>    redshiftGroup:  &quot;nonpii&quot;</pre><p>Batcher masks all the data it reads from Kafka. The user specifies what not to mask (non-PII) in GitHub, in this case<a href="https://github.com/practo/tipoca-stream/blob/master/redshiftsink/pkg/transformer/masker/database.yaml"> db.yaml</a>. For simplicity, let’s assume the mask versions of db.yaml increments by v1, v2, v3,… on every change.</p><p><strong>What’s a SinkGroup?</strong></p><p>The operator performs the sink to Redshift by creating a SinkGroup. SinkGroup is a group of RedshiftBatcher and RedshiftLoader pods.</p><p>Three parameters that make a sink group are the 1) topics being handled by the group, 2) the mask version being used, and 3) the target redshift table. For example, customers is the target table for the topic db.inventory.customers and customers_ts_reload is the temporary table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*jpUDmPO455e0GXtV" /><figcaption>Image by author, <a href="https://github.com/practo/tipoca-stream/blob/master/build/sinkgroup.png">Source</a></figcaption></figure><pre>I0218 10:24:21.226806 status.go:229] rsk/inventory allTopics:   126<br>I0218 10:24:21.226827 status.go:230] rsk/inventory diffTopics:  6<br>I0218 10:24:21.226832 status.go:231] rsk/inventory released:    120<br>I0218 10:24:21.226837 status.go:232] rsk/inventory reloading:   6<br>I0218 10:24:21.226838 status.go:233] rsk/inventory reload-dupe: 6<br>I0218 10:24:21.226848 status.go:234] rsk/inventory realtime:    0</pre><p>On creating the RedshiftSink resource for the first time, the operator assigns all the topics to the Reload SinkGroup, which starts loading data with mask version <strong>v1</strong> to the Redshift temporary tables.</p><p>Meanwhile, the operator keeps monitoring the consumer group lag for the topics being reloaded. It marks the reloading topics as realtime when the lag satisfies the release condition as specified in the spec.</p><p>When the operator finds the topics have reached realtime, then it reconfigures all the SinkGroups to stop sinking the real-time topics and attempts to release them.</p><p><strong>What’s a release?</strong></p><p>Release is the set of operations performed by the operator in Redshift when it makes the realtime tables live.</p><ol><li>Replace the current live tables with the realtime temporary tables.</li><li>Run schema grants to the specified Redshift group.</li><li>Move the released topics to Main SinkGroup.</li></ol><p><strong>What happens when a new column needs to be unmasked in a table?</strong></p><p>This requires the table to be recreated with the new mask configuration.</p><p>Everything happens as the first time sink as described above, the only difference being: we provide the updates to the current live table with old mask configuration until the complete reload of the table with new mask configuration is complete. It’s done using ReloadDupeSink Group. We do this because reloading a big table takes time and during that time we still want the currently live tables to get real-time updates. Once the table gets released, the operator notifies the users of the release in Slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AgJkjNSued4YM42nB4VDNw.png" /><figcaption>slack bot notifying the users of the release: Image by author</figcaption></figure><p><strong>What’s a Deployment Unit?</strong></p><pre>batcher:<br>  sinkGroup:<br>    reload:<br>      maxReloadingUnit: 20<br>      deploymentUnit:<br>        podTemplate:<br>          resources:<br>            requests:<br>              cpu: 3<br>              memory: 3Gi<br>            limits:<br>              cpu: 3<br>              memory: 3Gi</pre><p>Batcher pods require more CPU to process big tables fast. To process messages at 80k messages per second it requires close to 3 cores of CPU. After the first sink gets finished and the table has reached realtime, the resources required are very less, for us it is 50m. This is the reason we have sinkGroup specification. Using this you can specify different pod resources for all the three sinkGroups.</p><pre>maxReloadingUnit 20</pre><p>The reload sinkGroup runs a unit (pod) for every topic. But the main and reloadDupe sinkGroups runs all topics together as one unit (pod). Say there are 1000 topics for a database, we do not reload all the topics together but we sink at max maxReloadingUnit topics with the shortest lag first.</p><h3>Challenges</h3><p>Redshift is an analytic powerhouse but to use it well a lot of things needs to go right to achieve peak performance.</p><ul><li>Setting the right SORT and DIST Keys.</li><li>VACUUMING the tables to keep them sorted and reclaiming unused disk blocks.</li><li>Optimal utilisation of node slices for efficient copy. Deciding the size of the files and the number of files for an efficient copy.</li><li>Minimising the s3 look ups on the files during COPY.</li><li>Using the right compression.</li></ul><p>After we moved ~50% of our database to realtime, table lag started spilling over the threshold.</p><p>🤖: You have a Pagerduty alert ! 📟 🔥</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*-Mi8dhQSPahLnkGU" /><figcaption>Prometheus alerts for tipoca-stream: Image by author</figcaption></figure><p>So we started measuring. We added metrics for the load duration, frequency of load (timeline), load speed, etc. <a href="https://github.com/practo/tipoca-stream/blob/master/REDSHIFTSINK.md#metrics-1">These</a> are available as Prometheus metrics.</p><p>Working on the compression, node slice utilisation, keeping s3 lookups low, we have solved many of our load bottlenecks. We now<strong> load at &gt; 2 Mb/second</strong> a jump from ~100 kb/second and have set the right defaults for everyone to benefit from this.</p><p>AWS continues to make Redshift better with every release. Releases like AUTO SORT keys, AUTO DIST keys, AUTO VACUUM are great! But we expect more :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nlYfTIKM52oAvsv0ENAFoA.png" /><figcaption>Load Timeline: Image by author</figcaption></figure><p>The<strong> </strong><a href="https://stackoverflow.com/questions/67278497/stream-many-tables-in-realtime-to-redshift-bottleneck-parallel-concurrent-load"><strong>greatest challenge is keeping</strong></a><strong> </strong>the lag of 1000s of tables in Redshift &lt; 10mins with many concurrent loads happening.<strong> </strong>We have 1000+ tables in Redshift but we use not over 400 in a day. This is the reason we now throttle loads for the unused table when needed.<strong> </strong>This feature makes sure the <strong>tables which are in use are always near realtime</strong> and keep the Redshift less burdened. This was very useful.</p><p>Looking forward to <a href="https://www.youtube.com/watch?v=EOwg8NwZKkI">Redshift AQUA </a>release in Mumbai.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Zl8I3eEIKx24ucc88pULQ.png" /><figcaption>count(redshift_scan_query_total) &gt; 0, gives total no of tables in use!: Image by author</figcaption></figure><p><strong>How does the throttling work?</strong></p><p>Follow <a href="https://github.com/practo/tipoca-stream/blob/master/REDSHIFTSINK.md#enable-redshiftsink-exporter-optional-recommended">this</a> to enable throttling. It works like this:</p><ul><li><strong>First, determine the table is in use: </strong>We export the Redshift STL_SCAN as a Prometheus metric redhshift_scan_query_total<strong>. </strong>We consider a table is in use if the scan query total is greater than zero. Scan query table queries on a view of the last X days to determine if the table is in use.</li><li><strong>Then, throttle based on table’s usage: </strong>Next, we allow only 10 concurrent loads in Redshift. If the table is in use, we never throttle and always load it. Otherwise, we throttle the table load to a maximum 1 hour if need be. If the table gets accessed during this 1 hour throttling window, we just make it real time straightaway.</li></ul><p>We have successfully moved most of our databases to real time now, and things have been great!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*XNdKY4-lcZLem8qcPilMaQ.png" /><figcaption>from our warehouse lag dashboard: Image by author</figcaption></figure><p>By open-sourcing the project, we hope to give back to the community. Looking forward to learning from and collaborating with everyone to continue making it better.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f261cdcc3a13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/open-sourcing-tipoca-stream-f261cdcc3a13">Open-sourcing Tipoca Stream</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playing with Prometheus…]]></title>
            <link>https://mail-alok87.medium.com/playing-with-prometheus-c939fe6d1df7?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c939fe6d1df7</guid>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Fri, 14 May 2021 19:41:57 GMT</pubDate>
            <atom:updated>2021-05-14T19:55:38.511Z</atom:updated>
            <content:encoded><![CDATA[<h4>Short story of playing with Prometheus metrics and functions to measure, observe, then optimize.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cwWKI6H0KRUnKzA6gvrPLw.png" /><figcaption>All loads running</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*An9EMMOFuF4sNhh46gzByA.png" /><figcaption>Load of the product being optimized</figcaption></figure><h3><strong>Timeline Metric</strong></h3><p>We wanted to visualize and optimize the number of load tasks running to Redshift. Redshift UI gives a similar view for the query runtime.</p><p>Prometheus <a href="https://prometheus.io/docs/concepts/metric_types/#gauge">Gauge</a> metric helped us make this possible. Sweet and simple!</p><pre>rsk_loader_running{product=&quot;communicator&quot;, topic=&quot;sms&quot;} 1<br>rsk_loader_running{product=&quot;book&quot;, topic=&quot;appointment&quot;} 0</pre><p>Whenever a task run, we set the Gauge to 1, and we set it back to 0 when task finishes. Our scrap interval of this metric is always greater than the time to complete one task. So not a problem.</p><p>We can easily visualize timeline in Grafana using the below query:</p><pre>sum by (rsk) (rsk_loader_running &gt; 0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*VXDmadzBpxu342eqOUoADQ.png" /><figcaption>sum(rsk_loader_running &gt; 0)</figcaption></figure><h3>Duration Metrics</h3><p><strong>Load Speed (bytes/second)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C9kpDsH4JUAbnQdGqIUuwQ.png" /><figcaption>Notice the vertical lines, they are the time of load; the graph is just connected!</figcaption></figure><p>Next we wanted to measure the speed. Bytes loaded per second to Redshift.</p><p>For this, we used <a href="https://prometheus.io/docs/practices/histograms/">Histogram</a> metric.</p><pre>rate(rsk_loader_bytes_loaded_sum[5m])/<br>rate(rsk_loader_bytes_loaded_count[5m])</pre><pre>/  </pre><pre>rate(rsk_loader_seconds_sum[5m]) / rate(rsk_loader_seconds_count[5m])</pre><p>Similarly, we measure the no of messages, data loaded per load, duration per load, messages processed per load, etc..</p><h3>Counting the no of loads that happened in last 15minutes?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*bY9ecbnXdFyqDww2ro0jcg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FK8s68U9u3GbyGOIW5_8-g.png" /><figcaption>This shows the load happening for a task in last 30 minutes</figcaption></figure><p>When load happens, the metric becomes one, and when it finishes, it becomes zero again. How do you measure how many loads happened in last 15 minutes using Prometheus query then ?</p><p>We can use Prometheus <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#changes">changes</a> to find the count here!</p><pre>changes(rsk_loader_running{task=”task1&quot;}[15m])</pre><p>This will return the no of changes happened for this metric. It would be 6 in this case. 6 times the values have changed from 0 to 1 or 1 to 0.</p><p>Now but only 3 loads ran right! Yes!</p><p>So, we divide it by 2 and take the <a href="https://prometheus.io/docs/prometheus/latest/querying/functions/#ceil">ceil</a>, that’s it, we get our total runs.</p><pre>ceil(changes(rsk_loader_running{task=”task1&quot;}[15m])/2)</pre><p>Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c939fe6d1df7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launching Worker Pod Autoscaler]]></title>
            <link>https://medium.com/practo-engineering/launching-worker-pod-autoscaler-3f6079728e8b?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f6079728e8b</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[queue]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[hpa]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Thu, 12 Dec 2019 16:47:48 GMT</pubDate>
            <atom:updated>2019-12-14T06:42:14.033Z</atom:updated>
            <content:encoded><![CDATA[<h3>Launching Worker Pod Autoscaler — Solving specific problems with worker scaling in Kubernetes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*NJoDmZ_kT9VIIHI5ORE3Qw.png" /></figure><p>When you are finding a doctor, booking an appointment or consulting a doctor online on Practo, a lot of work is done asynchronously using Job queue systems (a.k.a <strong>workers</strong>). This is a critical part of our architecture and it plays an important role in making our systems scalable and reliable.</p><p>We use <a href="https://aws.amazon.com/sqs/">AWS SQS</a> as the message queue and we process on a scale of about 800 million jobs per month. These workers are run as a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Kubernetes Deployment</a> with every worker running as a separate pod. This helps to deploy, scale, monitor and troubleshoot each worker process independently.</p><p>This article focuses only on the<strong> “scaling part”</strong> of these worker deployments.</p><h3>Implementation 1 — Using Custom Metrics and HPA</h3><p>When we started with Kubernetes we used <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">HPA</a> with its <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-custom-metrics">Custom Metric Support</a> and <a href="https://prometheus.io/">Prometheus</a> to scale the workers. This is also the default recommended solution from the community for scaling worker pods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*uZ8c6hbMjVBAIA5KzhLXxQ.png" /></figure><p>Horizontal Pod Autoscaler(HPA) is a controller which scales Kubernetes Deployment typically based on CPU metric and supports scaling Kubernetes Deployment based on any custom metric. <strong>The only requirement is the user needs to make sure the custom metric is available for consumption by the Kubernetes process</strong>.</p><p>We exported the SQS Cloudwatch metrics to Prometheus by making the <a href="https://github.com/prometheus/cloudwatch_exporter">Prometheus Cloudwatch Exporter</a> compatible for this use case. Here is the code for it — <a href="https://github.com/practo/cloudwatch_exporter">https://github.com/practo/cloudwatch_exporter</a></p><p>This was our first implementation of scaling our background workloads. We ran this for about a year or so. Based on the feedback from the various service teams and our experience, the <strong>following were the problems with this implementation:</strong></p><ol><li>Slow scale-ups: A lot of services required scaling to be near real-time. If there is a surge in the messages in the queue then the workers should scale instantly. But this was not happening because the Cloudwatch metrics are <strong>only available at 5-minute intervals for SQS</strong>. Solving for this required a combination of long-polling, <a href="https://docs.aws.amazon.com/sdk-for-go/api/service/sqs/#SQS.GetQueueAttributes">SQS metrics API</a> and Cloudwatch metrics.</li><li>Scaling down was causing issues: <strong>HPA does not allow to scale up and down based on different metrics</strong>. But it can be different based on the use case. For example, we were scaling up on SQS metric — <strong>ApproximateNumberOfMessagesVisible </strong>but we could not scale down based on the same metric because for high throughput workers when they were consuming the queue very fast this metric was always zero.</li><li>Non-idle workers were scaling down: Since it was a target-based scaling when the Queue length metric dropped it started to scale down the workers which were not idle.</li><li>Costs were not optimized: Since metrics were polled from Cloudwatch at fixed intervals and not intelligently as required, this incurred a significant AWS cost towards Cloudwatch. This can be optimized by fetching only the relevant metrics only when we require them.</li><li>Using custom metrics was not as straightforward as running a few kubectl commands and required a good effort from the user to make the whole setup work. Also, <strong>HPA was requiring us to decouple the metrics from the scaler which means we cannot derive the scaling metrics using the currently running pods already known to the scaler</strong>.</li></ol><p>So to solve the above problems, we wrote <a href="https://github.com/practo/k8s-worker-pod-autoscaler">Worker Pod Autoscaler.</a></p><h3>Implementation 2 — Worker Pod Autoscaler</h3><p><a href="https://github.com/practo/k8s-worker-pod-autoscaler">practo/k8s-worker-pod-autoscaler</a></p><p>Scaling workloads based on queue length is a fairly common use case. Therefore we decided to build a generic, performant and very easy to use open-source autoscaler for workers using <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Kubernetes CRDs</a>. Following were the primary goals of this project:</p><ol><li>Superfast scale-ups: Near real-time scaling of workers. As soon as a job comes in the queue the containers should scale if needed.</li><li>Solve the scale down problems for the workers: The solution should not scale down workers that are processing a job. Scale down metric should not be restricted to the same metric as scale-up, it should be based on the queue provider.</li><li>Built-in Queue Exporters: Kubernetes HPA with custom metrics requires org effort in exporting custom metric and storing them to be made accessible to the Kubernetes API. This makes sense for Kubernetes to support several use cases. But the current HPA implementation is restrictive to use for queue-based scaling. Also, <strong>built-in</strong> help in focussing the user worry about the application and frees the user from maintaining metric exporters. The project should build generic metric exporters (pollers) with every queue provider integration.</li><li>Support for scaling up from and down to zero pods to optimize costs.</li><li>Ease of Use:<strong> The whole setup of worker autoscaling should start working by running a few kubectl commands.</strong></li><li>Generic Platform Independent Autoscaler: The autoscaler should extend to support any message queuing service and should work in any cloud or on-premise Kubernetes environment.</li><li>Open Source: The solution should be completely open-sourced for a bigger community to use, contribute and make it better.</li></ol><p>Now, let’s dig into how Worker Pod Autoscaler aka WPA works!</p><h3>Worker Pod Autoscaler</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*GYE9qyNEmbz_Zw0xltY-Yw.png" /></figure><p>When the WPA controller pod starts it install the WPA custom resource in Kubernetes if it is not present and it starts the following go-routines:</p><ul><li>WPA Goroutine: WPA controller go-routine is the control loop which scales the deployment to the required number of replicas based on the queue specified for the deployment.</li><li>Poller Goroutine(s): There is a <a href="https://github.com/practo/k8s-worker-pod-autoscaler/blob/6451f18135b9489948b275b038cfe0ae60ead01a/pkg/queue/poller.go#L81-L109">poller manager</a> who starts and stops go-routines (poller) for each WPA resource. Each poller polls the queue provider metrics for the exact metric keeping the API calls to minimum. The sync intervals here can be configured based on use cases. The concurrency here is one of the reasons for near real-time scaling.</li></ul><p>The algorithm for deciding the desired replicas can be read h<a href="https://github.com/practo/k8s-worker-pod-autoscaler/blob/bf3ef92082ae7032ce32f771f7b292fb30c21c43/pkg/controller/controller.go#L355-L397">ere.</a></p><p>Explaining WPA working with a simple example.</p><h4>Install WPA</h4><p>Worker pod autoscaler can be easily installed using <em>hack/install.sh</em>. Please <a href="https://github.com/practo/k8s-worker-pod-autoscaler#install-the-workerpodautoscaler">follow this</a> for detailed information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7TJ2wa85rdPvslMPKqzYBg.png" /></figure><h3>Example</h3><p>Create a deployment and a WPA resource to manage the scaling of it.</p><pre>kubectl create -f artifacts/example-deployment.yaml<br>kubectl create -f artifacts/example-wpa.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8DkK8XwHlEZr3tej-AUHw.png" /></figure><p><strong>WPA will start scaling example-deployment from 0 to 30 if the messages in the queue are greater than 150. </strong>The workers will scale down to 0 when there are no jobs in the queue.</p><p><strong>That’s it! Two kubectl commands and your worker deployment starts scaling!</strong></p><p>We have been running WPA in production from <a href="https://github.com/practo/k8s-worker-pod-autoscaler/releases/tag/v0.1.0">the last 4 months</a>. We have got great feedback from our developers in Practo. It is now in <a href="https://github.com/practo/k8s-worker-pod-autoscaler/releases/tag/v0.2.2">v0.2.2 release</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-53gTvemBnUw0WMu_TZuYQ.png" /><figcaption><strong>Unused workers were stopped reducing memory usage from 2 GB to 100 MB for one of the services.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XSNQzSmt9O0pvVWfTt3Zmw.png" /><figcaption><strong>Cloudwatch costs came down by 50% because only relevant metrics are fetched more intelligently.</strong></figcaption></figure><h4>Future Work</h4><p>SQS is the only queue provider supported at present. But it can easily be extended to any other queue provider. <a href="https://github.com/practo/k8s-worker-pod-autoscaler/blob/9e7cd690da1c8c0f6aa3490853f7e805d629b4bd/pkg/queue/queueing_service.go#L5-L8">This interface</a> needs to be implemented and a <a href="https://github.com/practo/k8s-worker-pod-autoscaler/blob/9e7cd690da1c8c0f6aa3490853f7e805d629b4bd/pkg/queue/sqs.go">simple poller like this</a> needs to be written down to bring in a new queue provider support! Pull requests are welcome :)</p><p>Thanks to the Kubernetes team for making <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">CRDs</a> and <a href="https://github.com/kubernetes/sample-controller">sample controller</a>. Please feel free to open issues in <a href="https://github.com/practo/k8s-worker-pod-autoscaler/issues">Github</a> if you have any questions or concerns. We hope WPA is of use to you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f6079728e8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/practo-engineering/launching-worker-pod-autoscaler-3f6079728e8b">Launching Worker Pod Autoscaler</a> was originally published in <a href="https://medium.com/practo-engineering">Practo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Networking with Kubernetes]]></title>
            <link>https://medium.com/practo-engineering/networking-with-kubernetes-1-3db116ad3c98?source=rss-f4ee7d11538e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3db116ad3c98</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tcp-ip]]></category>
            <dc:creator><![CDATA[Alok Kumar Singh]]></dc:creator>
            <pubDate>Mon, 23 Oct 2017 08:11:46 GMT</pubDate>
            <atom:updated>2022-02-27T14:30:43.853Z</atom:updated>
            <content:encoded><![CDATA[<p>Containers are changing the landscape of how the applications are developed and run. With containers coming into picture, the application design has become more distributed making the network a very critical component. Container networking is somewhat complicated and not that easily understood. The types and options to network containers are many and each should be understood properly to make a right network choice to build a secure, robust and adaptable production system.</p><p>The container networking across the industry were implemented using the core Linux network technologies like IPRouting, IPTables for fire-walling, port-forwarding and load-balancing, IPAM for IP allocations, DNS, IP-Masquerading, network address translators, network interfaces, L<a href="https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/">inux namespaces</a>, bridges etc. I intend to speak about these technologies in a series of blogs, some of them might be part of the other blog posts in the series. (Note: Please read it in desktop, to the see the well formatted tables)</p><h3>Contents</h3><ul><li>Networking modes</li><li>Pod to pod communication within the same node</li><li>Pod to pod communication across nodes</li><li>Simple networking implementations</li></ul><h3>Networking Modes</h3><p>As I said there are various options to network containers. A lot of standardisation efforts are happening on configuring the network interfaces in the container community. But before getting into standardisation in the industry and understanding kubernetes networking in particular, let us first understand different networking modes and their story a little bit. It is useful to understand this as it forms the fundamental of how any container orchestration system works.</p><p><strong>Link mode</strong></p><p>It all started with docker introducing the Docker link feature to allow containers to discover each other and communicate within the same host. The main focus of this approach was just to make the containers communicate and to make them reachable from outside. When containers were linked, information of the source container was made present in the recipient container by setting environment variables and updating /etc/hosts of the recipient containers for the source container IP addresses. This was not a scalable approach as the IP of the source container can change and then it was impossible to keep updating the environment variables and /etc/hosts across all the dependant containers. This mode is deprecated now.</p><p><strong>Host network mode</strong></p><p>Host network mode is also called native networking and was one of the early simple networking modes. In this mode, when a container starts, it uses the host <a href="https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/">network namespace</a> and host IP address. This basically means the containers can see all the network interfaces of the host. And to expose a container outside, users need to choose a port from the host port space and configure the IPTables for the appropriate firewall rules to expose the container service.</p><p>Few advantages of this approach are 1) Performance is better as there are no NAT translations 2) Easier to troubleshoot 3) Simple design.</p><p>Mesos uses host network mode. Even <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf">Borg</a>, the predecessor to Kubernetes used the IP address of the host and thus shares the host-port space and thus operated in host mode. The major drawback with this approach is that there is a need to manage the ports that the containers are attached to. This might lead to port conflicts and applications are required to pre-declare the ports they use.</p><p><strong>Bridged network mode</strong></p><p>Bridge mode offers an improvement over the host mode as there is no need to maintain ports. It is the most common type of container network. In this, a virtual bridge container0 is created in the host. Each container which gets created gets its own <a href="https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/">network namespace</a> and gets an IP address from the bridge subnet. Since every container has its own namespace, they have their own port space and hence there are no port conflicts. But to communicate outside, these containers still uses hostIP and hence the network address translation is required to translate containerIp:Port to hostIp:Port. These NAT translations are done in Linux using IPTables.</p><p>Good thing about this approach is it isolates the container and also give them their own port space which prevents port conflicts but the NAT translation brings in the performance overhead.</p><p>Bridge method could be extended to multiple hosts but it requires maintaining ports or dynamically allocating ports. A simple design that supported multi host networking without the infrastructure complexity of managing the ports and without having to do any NAT translations was required.</p><p>Here comes the sun…</p><p><strong>IP per </strong><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#what-is-a-pod"><strong>pod</strong></a><strong> mode</strong></p><p>Kubernetes took a completely different approach to network containers. We could call it an “Internet like” approach to networking. The fundamental change was that every pod will have their own IP address and they could be accessible from the other pods regardless of which host the pod is on and there won’t be any NAT translations in pod to pod or node to pod or pod to node communications.</p><p>Let us now dig into IP-per-pod mode and understand the various communications that happen in the kubernetes cluster.</p><p>Pods and hosts send packets to each other to communicate. Packets are the most basic unit of network transmission.(Actually the layer of communication determines if it is a segment or a packet or a frame getting transmitted, but let’s just call it a packet for the discussion) Basic contents of packets are -</p><ul><li>Sender and Receiver IP Address</li><li>Payload: Actual Data being transmitted i.e. content of the packet</li><li>MAC address is added to the packet when it is sent as frames in Layer4</li><li>Sender and destination port informations</li></ul><h3>Pod to pod communication in the same node:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*lAfpMbHRf266utcd4xmLjQ.gif" /></figure><p><strong>Note</strong>: The red dot in all the diagrams is the packet getting transmitted.</p><p><strong>Network namespaces</strong></p><p>The above diagram tells us how the packet flows for a pod to pod communication in a single node(pod1 to pod2 communication). It is a bridge network within the same node. Let us understand it in detail. We have a machine, it is called a node in kubernetes. It has an IP 172.31.102.105 belonging to a subnet having CIDR 172.31.102.0/24. The node has an network interface eth0 attached. It belongs to root <a href="https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/">network namespace</a> of the node. For pods to be isolated, they were created in their own <a href="https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/">network namespaces</a> — these are pod1 n/w ns and pod2 n/w ns . The pods are assigned IP addresses 100.96.243.7 and 100.96.243.8 from the CIDR range 100.96.0.0/11. We will discuss on how the IP addresses are assigned to the pods later in network implementation section as it is implementation specific.</p><p><strong>Network interfaces and veth pairs</strong></p><p>Now, we need to connect the root and pod network namespaces to talk to each other. This is done in Linux by using veth pipe pairs (a cable with two ends, whatever data that comes in one will come out of other and vice versa). So lets call them eth0 &lt;--&gt; vethpod1 and eth0 &lt;--&gt; vethpod2</p><p>To view the above namespaces in Linux we need to create a symlink for the created pod namespace as the ip netns utility looks for namespaces in a particular directory /var/run/netns. Let us call the pod container names as pod1 and pod2</p><pre>// view namespaces<br>admin@ip-172-31-102-105:~$ pid=&quot;$(sudo docker inspect -f &#39;{{.State.Pid}}&#39; &quot;pod1&quot;)&quot;</pre><pre>admin@ip-172-31-102-105:~ $<strong> </strong>sudo ln -sf /proc/$pid/ns/net &quot;/var/run/netns/pod1&quot;</pre><pre>admin@ip-172-31-102-105:~ $<strong> </strong>ip netns list<br><strong>pod1</strong></pre><p>Here, we list down the interfaces attached in the namespace of pod1 that is eth0@if41. This is one end of veth1 pair.</p><pre>// one end, eth0 of the pair eth0 &lt;--&gt; vethpod1<br>admin@ip-172-31-102-105:~ $ <strong>sudo ip netns exec pod1 ip a</strong><br><strong>3: eth0@if18:</strong> &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 8973 qdisc noqueue state UP group default<br>    link/ether <strong>0a:58:64:60:1d:0e</strong> brd ff:ff:ff:ff:ff:ff<br>    inet <strong>100.96.243.7/24</strong> scope global eth0<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::f8c9:64ff:fe4e:aac9/64 scope link tentative dadfailed<br>       valid_lft forever preferred_lft forever</pre><p>Let us find the other end of theveth pair. This pair is present in the root namespace, so we can use the ip a command to find it. It can be found by grepping the docker container name, pod1. The name is veth&lt;podname&gt;</p><pre>// other end, vethpod1 of the pair eth0 &lt;--&gt; vethpod1<br>admin@ip-172-31-102-105:~ $ <strong>ip a | grep -A3 &quot;vethpod1&quot;</strong><br><strong>8: vethpod1: </strong>&lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue master docker0 state UP group default<br>    link/ether ee:70:85:2a:3e:6c brd ff:ff:ff:ff:ff:ff<br>    inet6 fe80::ec70:85ff:fe2a:3e6c/64 scope link<br>       valid_lft forever preferred_lft forever</pre><p><strong>Routing and bridges</strong></p><p>As soon as the pod1 emits a packet, it travels via eth0&lt;-&gt;vethpod1 pair to reach the root network namespace.</p><p>The interface to send the packet is determined by checking the route table. Our packet has the destination ip of pod2e i.e. 100.96.243.8. Route table tells to forward any traffic coming for the range 100.96.243.0/24 to cbr0. cbr0 is a bridge. Let us look at the route table in the node</p><pre>// route table of the node<br>admin@ip-172–31–102–105:~$ <strong>ip route</strong><br>default via 172.31.103.1 dev eth0<br>100.64.0.0/10 dev flannel0 proto kernel scope link src 100.96.243.0<br><strong>100.96.243.0/24 dev cbr0 proto kernel scope link src 100.96.243.1</strong><br>172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1<br>172.31.102.0/24 dev eth0 proto kernel scope link src 172.31.102.105</pre><p>vethpod1 then sends the packet to the bridge cbr0</p><p>We can find the interfaces attached to the bridge using the below command which shows up our pod veth interfaces. Cool !</p><pre>// interfaces attached to cbr0 in node<br>admin@ip-172-31-102-105:~ $ <strong>sudo brctl show</strong><br>bridge name bridge id         STP enabled interfaces<br><strong>cbr0</strong>        8000.0a5864601d01 no          <strong>vethpod1</strong><br>                                          <strong>vethpod2</strong>     <br>docker0     8000.024202a3b77c no</pre><p>Next, the bridge finds the destination mac address for pod2 in the packet to be 0a:28:64:60:2d:0e</p><p>We can confirm the mac address by doing below —</p><pre>// destination mac address, mac address of pod2<br>admin@ip-172-31-102-105:~ $ sudo ip netns exec <strong>pod2</strong> ip a<br><strong>6: eth0@if21:</strong> &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 8973 qdisc noqueue state UP group default<br>    link/ether <strong>0a:28:64:60:2d:0e</strong> brd ff:ff:ff:ff:ff:ff<br>    inet <strong>100.96.243.8/24</strong> scope global eth0<br>       valid_lft forever preferred_lft forever</pre><p>Bridge maintains a forwarding table where previously learnt* mapping for MAC address and ports are kept and does layer2 transmissions using mac addresses.</p><pre>// forwarding table, FDB<br>admin@ip-172-31-102-105:~$ sudo brctl showmacs cbr0<br>port no mac addr            is local?   ageing timer<br><strong>48</strong>      <strong>0a:28:64:60:2d:0e</strong>   yes         0.00<br>96      0a:58:64:60:1d:0e   yes         0.00</pre><blockquote>*Mac learning: In our case, the bridge receives a packet from MAC address of pod1 <em>0a:58:64:60:1d:0e</em>. If it does not finds this address in the forwarding table, it adds an entry for the source MAC address and port. Next time if it has to send packets to this MAC address it looks up this table and finds the port to forward the packet to. This is known as<em> MAC learning</em>.</blockquote><p>We can clearly see that our destination mac address in the packet is 0a:28:64:60:2d:0e. Bridge will do a forward table lookup to find the port of this address and send the packet to that port i.e. <strong>48</strong>. We can find out which interface this 48 belongs to by doing below —</p><pre>// port - interface mapping in the bridge<br>admin@ip-172-31-102-105:~$ sudo brctl showstp cbr0 | grep -A2 vethpod2</pre><pre><strong>vethpod2</strong> (<strong>48</strong>)<br> port id  804e   state       <strong>forwarding</strong></pre><p>Packet then reaches vethpod2 interface. From vethpod2, the packet gets piped to the eth0 of the pod2 namespace. Congratulations! our packet reached the destination!</p><p>We now know how the packet moves and reaches our other pod in the same node. Also, the linux utilities used here can help in debugging routing problems in the node.</p><p>It gets more interesting when the communication happens across the nodes.</p><h3>Pod to pod communications across nodes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*UzzwpHoB-4Giy7YLvcZuKg.gif" /></figure><p>Pod to pod communication within the same node was simple and involved basic Linux networking concepts. According to the kubernetes networking requirement, the pod IP should be accessible across nodes. We will get to how the pod IP is made accessible across nodes, when we discuss the specific network implementations. For now, let us just go ahead with the packet flow from a pod to a another pod in a different node.</p><p><strong>Routing</strong></p><p>Now again, the pod1 emits a packet, but for pod4 this time. It travels via eth0-vethpod1 pair to reach the root network namespace and then to the bridge cbr0 in node1</p><pre>// route table node1<br>admin@ip-172–31–102–105:~$ <strong>ip route</strong><br><strong>default via 172.31.103.1 dev eth0</strong><br>100.64.0.0/10 dev flannel0 proto kernel scope link src 100.96.243.0<br><strong>100.96.243.0/24 dev cbr0 proto kernel scope link src 100.96.243.1</strong><br>172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1<br>172.31.102.0/24 dev eth0 proto kernel scope link src 172.31.102.105</pre><p>Our packet this time has the destination ip 100.96.33.8 and say, it has the destination<em> </em>MAC address<em> </em>ab:e8:64:62:20:0e. In this case, our bridge cbr0 won’t find this MAC address in the forwarding table. So, it will perform an ARP lookup to find the MAC address from an IP address which will fail again. We can run the below command to look at the node ARP table.</p><pre>// arp table<br>admin@ip-172-31-102-105:~$ sudo arp<br>Address           HWtype HWaddress           Iface<br>ip-172-31-103-43  ether  12:b8:2c:97:c4:98   eth0<br>100.96.243.8      ether  0a:28:64:60:2d:0e   cbr0 <br>100.96.243.7      ether  0a:58:64:60:1d:0e   cbr0</pre><p>As the IP100.96.33.8belongs to a different node and this node has never been communicated with before, ARP fails(you can see the arp table above has nothing for this IP). Then the route tables are looked up again and on finding nothing specific, the packet is forwarded to the default interface eth0 of the node1.</p><p>Now the time has come for the packet to leave the node1 and go outside into the wire into the cloud. How the packet gets from the node1 to the node2 is very network implementation specific and will be discussed in the section of specific network implementations.</p><p>Our packet reaches node2 at its eth0. It is then IP-forwarded to cbr0 seeing the route table of the node2. Check the table below.</p><pre>// route table node2<br>admin@ip-172–31–103–67:~$ <strong>ip route</strong><br>default via 172.31.103.1 dev eth0<br>100.64.0.0/10 dev flannel0 proto kernel scope link src 100.96.33.0<br><strong>100.96.33.0/24 dev cbr0 proto kernel scope link src 100.96.33.1</strong><br>172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1<br>172.31.103.0/24 dev eth0 proto kernel scope link src 172.31.103.67</pre><p>cbr0 does the arp/fdb lookups and send the packet to the vethpod4 pipe and then our packets finally reaches the pod4 network namespace’s eth0. Awesome, our packet reached the other node’s pod !</p><h3>Simple networking implementations</h3><p>Kubernetes does not really care on how the network is implemented until the Kubernetes’ basic network requirements of ip-per-pod is implemented by the network provider. We will start with the simplest implementations first.</p><p>We discussed above how the packet gets routed when it does not find the destination ip within the node. How does it get routed to reach the other node?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*IUSm6gzIYajxiwsovfErUA.gif" /><figcaption>Node to node communication</figcaption></figure><p><strong><em>a) Networking using cloud routing tables: </em></strong>In this implementation every node is allocated a POD CIDR range and every node has access to change the cloud routing table. Whenever a node comes up, the cloud routing table is updated for it. It is basically telling the cloud that any packet that comes for pod range 102.96.33.0/24, route it to 172.31.103.67.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*OhWBNLRg7xE1LMEEV6fYsA.jpeg" /><figcaption>Node communication using cloud routing tables</figcaption></figure><p>Couple of drawbacks with this approach</p><ul><li>Multiple route tables needs to be maintained in case of multi AZ cluster which has single AZ NATs. Most clusters are provisioned by <a href="http://github.com/kubernetes/kops">kops</a> and it only supports kubenet networking for using the cloud routing tables and allows only one route table. Hence you can not use kops for private networking with cloud routing tables. For details, look at this <a href="https://github.com/kubernetes/kops/issues/1335">issue</a>.</li><li>Some cloud providers limit the route table entries, 50 in case of AWS. It means that the cluster can have only 50 nodes. This is a bottleneck.</li></ul><p><strong><em>b) Networking using node route tables</em></strong>: Another approach to allow nodes to communicate could be to write a program that configures all the nodes internal route tables for every node &amp; POD-CIDR pair that comes up. We can make node1 and node2 communicate to each other by adding the below route entries for the respective nodes. Difficult to maintain this setup though, and no one really does this.</p><pre>admin@172.31.102.105$ <strong>ip route add 100.96.33.0/24 via 172.31.103.67<br></strong>admin@172.31.103.67$ <strong>ip route add 100.96.243.0/24 via 172.31.102.105</strong></pre><p>In the<strong> upcoming blog(s) </strong>in the series, we will dig into other networking implementations like cloud native networking, SDNs, CNIs, overlay networks, etc. and discuss pod to service communications, internal and external communications in the kubernetes cluster<strong>.</strong></p><p>Update: future blogs not done as there are many such articles already there.</p><blockquote>I wrote this blog series as a means to understand the networking in kubernetes. And it was made after reading many articles and by understanding concepts in the internet. But I would like to particularly thank <a href="https://twitter.com/thockin?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor">Tim Hockin</a> and Michael Rubin for making such a wonderful <a href="https://kubernetes.io/">system</a> and explaining it in style in this <a href="https://www.youtube.com/watch?v=y2bhV81MfKQ">video</a>. Also, want to thank <a href="https://twitter.com/bboreham">Bryan Boreham</a> from the kubernetes and weave community for helping me understand some of the stuffs. And to <a href="https://twitter.com/krhapr">Hari</a> who suggested to use the blogging approach to learn things.</blockquote><blockquote>Follow us on <a href="https://twitter.com/practodev">twitter</a> for regular updates. If you liked this article, please hit the Hand icon to recommend it. This will help other Medium users find it.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3db116ad3c98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/practo-engineering/networking-with-kubernetes-1-3db116ad3c98">Networking with Kubernetes</a> was originally published in <a href="https://medium.com/practo-engineering">Practo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>