<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chintan Betrabet on Medium]]></title>
        <description><![CDATA[Stories by Chintan Betrabet on Medium]]></description>
        <link>https://medium.com/@chintanbetrabet?source=rss-fd8e48f1139a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Efo_v5QylmQtK5aN.jpg</url>
            <title>Stories by Chintan Betrabet on Medium</title>
            <link>https://medium.com/@chintanbetrabet?source=rss-fd8e48f1139a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:54:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chintanbetrabet/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[India vs Australia, a poetic story of justice]]></title>
            <link>https://chintanbetrabet.medium.com/india-vs-australia-a-poetic-story-of-justice-7731ea476449?source=rss-fd8e48f1139a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7731ea476449</guid>
            <category><![CDATA[sports]]></category>
            <category><![CDATA[indian-cricket-team]]></category>
            <category><![CDATA[virat-kohli]]></category>
            <category><![CDATA[australian-cricket]]></category>
            <category><![CDATA[cricket]]></category>
            <dc:creator><![CDATA[Chintan Betrabet]]></dc:creator>
            <pubDate>Sun, 07 Feb 2021 17:38:12 GMT</pubDate>
            <atom:updated>2021-02-15T06:14:27.484Z</atom:updated>
            <content:encoded><![CDATA[<p>A poetic recollection of how India and Australia’s riveting rivalry has unfolded since Ricky Ponting and co last conquered the Indian Subcontinent</p><p>In many ways, Cricket has been a funny game</p><p>Playing home and away, teams are never the same</p><p>When India plays Australia, neither is a great host</p><p>When they visit the other, the record is nothing to boast</p><p>For the Aussies, 2004 was a landmark year</p><p>Finally, they conquered India, their Final Frontier</p><p>It was the end of so many years of hurt and pain</p><p>But the wait would be long before they would win again</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UmkzK2CSfvyZm_HzqKmfsQ.jpeg" /><figcaption>Australia winning the Border Gavaskar trophy with a 2–1 in 2004 (<a href="https://www.cricket.com.au/news/australia-india-2004-winners-wish-steve-smith-well-fourth-test-dharamsala-katich-gilchrist/2017-03-25">source</a>)</figcaption></figure><p>The Indians fought hard in the 2007–08 tour Down Under</p><p>Denied a deserved win by multiple umpiring blunders</p><p>Taking it in their stride the Men in Blue roared back</p><p>Scripting a famous win on Perth’s pacy track</p><figure><img alt="The infamous Sydney test of 2008 which included the “Monkeygate” scandal as well as a number of umpiring errors" src="https://cdn-images-1.medium.com/max/760/1*BQl7RYXBY-MlkQLO-7TqDA.jpeg" /><figcaption>The infamous Sydney test of 2008 included the “Monkeygate” scandal as well as a number of umpiring errors (<a href="https://www.theweek.in/news/sports/2020/07/19/two-mistakes-bucknor-relieves-infamous-india-australia-2008-sydney-test.html">source</a>)</figcaption></figure><p>Contests like these soon became a thing of the past</p><p>In away conditions, batting lineups barely did last</p><p>India struggled with pace as the Aussies did against spin</p><p>It would be another 10 years for both without an away test win</p><p>It was in Pune, 2017 that Australia pulled off a shock</p><p>Steve Smith powered them home with a captains knock</p><p>The joy was short-lived and normalcy was soon restored</p><p>A dominant performance and the Indians won the series once more</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*0Nofa2a17FI5BfdCWX1eHg.jpeg" /><figcaption>Captain Steve Smith led Australia to a 333 run win in the opening game, a first test match win on Indian shores since 2004 but lost the series 2–1 eventually. (<a href="https://indianexpress.com/article/sports/cricket/india-vs-australia-2017-india-set-to-chase-441-after-steve-smith-ton-takes-australia-to-285-4542847/">source</a>)</figcaption></figure><p>People often push their own limits to win at any cost</p><p>But by bending the rules, more than the game can be lost</p><p>By tampering with the ball, Smith and Warner didn’t play fair</p><p>It did not go unnoticed, leading to a year-long ban for the pair</p><p>Without their stalwarts, the Australians had lost their charm</p><p>Tumbling defeats to Pakistan and South Africa was certainly an alarm</p><p>Visiting Australia in 2018, the Indians would have licked their lips</p><p>Against the depleted Aussies, this could be the best of their trips</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*owUMI9zbLbEIJAnzQONCrg.jpeg" /><figcaption>Steve Smith and David Warner were banned for a year for their involvement in ball-tampering, greatly affecting the performance of the Australian team</figcaption></figure><p>In their own kingdom though, they are difficult to dethrone</p><p>Even without their leader, the Aussies held their own</p><p>They lost the first game in Adelaide but it was extremely tight</p><p>It showed that the weakened Kangaroos were still up for a fight</p><p>The Indians were now flying high having won the first bout</p><p>Winless in Australia since 2007, it was the end of a drought</p><p>If they held on to this lead, it would be a story to narrate</p><p>A chance to at last set a 100-year record straight</p><p>Their hopes though soon came crashing to the ground</p><p>In the next game at Perth, the home team turned it around</p><p>A pacer’s paradise soon became spinner Lyon’s lair</p><p>The teams walked away with series all square</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_qxE6sZpqDAVcV_EJf37Q.jpeg" /><figcaption>Lyon took a record seventh 5 wicket haul against India to set up Australia’s win and level the series (<a href="https://timesofindia.indiatimes.com/sports/cricket/india-in-australia/india-vs-australia-2nd-test-special-nathan-lyon-earns-praise-from-sachin-tendulkar/articleshow/67117215.cms">source</a>)</figcaption></figure><p>Form may be temporary, what’s permanent is class</p><p>Whatever happened at Perth, the Indians let pass</p><p>Learning from their mistakes, they took the loss in their stride</p><p>Heading to Melbourne to once again turn the tide</p><p>A dominant batting performance batted the hosts out of the game</p><p>For all the fight in Perth, this time the Aussies turned out tame</p><p>A chance to follow on, the Indians choose to forego</p><p>But they did just enough to land the knockout blow</p><p>After the latest setback though the Aussies looked down and out</p><p>A monstrous total by India almost put the series beyond doubt</p><p>It rained and rained till a draw ensured a series win at last</p><p>Kohli’s men achieved what no Indian team had done in the past</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*6IOVNkzaSlf7QD1ffqT2Xg.jpeg" /><figcaption>This was India’s first-ever series win on Australian soil (<a href="https://timesofindia.indiatimes.com/sports/cricket/india-in-australia/india-vs-australia-india-make-history-win-first-ever-cricket-test-series-in-australia/articleshow/67414248.cms">source</a>)</figcaption></figure><p>To say the Indians avenged the loss of 2004, many experts would disagree</p><p>India was at their peak back then, the current Aussies missed Smith’s pedigree</p><p>History was made but not against their strongest eleven</p><p>But who could stop the Indians who were in the seventh heaven?</p><p>A lot would happen around the world before their next exchange</p><p>Nobody anticipated how much a single year could change</p><p>From the Covid-19 pandemic, the world was beginning to rise</p><p>An Indian tour to Australia in 2020, was a sight for sore eyes</p><p>With the return of Warner and Smith to the team, it would be a stern test</p><p>Between the leaders of the World Test Championship, a battle of the best</p><p>For the Indians, the IPL and quarantine meant both tired body and mind</p><p>Many questioned whether they would survive the rigorous grind</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*kXAgEFqI6mk5djgKm9UsTA.png" /><figcaption>A long tour of Australia awaited India amidst strict quarantine regulations (<a href="https://timesofindia.indiatimes.com/sports/cricket/news/india-vs-australia-soft-quarantine-over-team-india-checks-into-new-hotel/articleshow/79425991.cms">source</a>)</figcaption></figure><p>With a first-innings lead at Adelaide for India, many plaudits were earned</p><p>But in an eventful year, to expect the unexpected we had learned</p><p>With a fiery attack from Cummins, the Australians used their pace</p><p>The stunned Indian batting lineup soon vanished without a trace</p><p>Breaking records in their last visit, the Indians were at it again</p><p>Bowled out for their lowest score, they looked like lambs in a lion’s den</p><p>The demons of this defeat would long continue to spook</p><p>An uphill task lay ahead to prove their previous win was no fluke</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*f6ir9mV3555GjoL5ildtpQ.jpeg" /><figcaption>India was bowled out their lowest score ever in Test cricket (<a href="https://www.espncricinfo.com/story/aus-vs-ind-1st-test-adelaide-stats-india-hit-record-low-with-36-all-out-1244674">source</a>)</figcaption></figure><p>With pacer Shami injured and Kohli returning home on a personal errand</p><p>For many a realistic possibility of a 4–0 whitewash certainly beckoned</p><p>It was up to Rahane and his youngsters to bring India back from the brink</p><p>In the armor of this ferocious beast, they had to find a chink</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fsLONGnDbiDaGCEgMsyAng.jpeg" /><figcaption>Former cricketers began to call the series a potential 4–0 whitewash after the Adelaide debacle (<a href="https://www.insidesport.co/ind-vs-aus-test-series-india-will-lose-4-0-predicts-4-big-test-cricketers-can-india-fight-back/">source</a>)</figcaption></figure><p>On Boxing Day in Melbourne, the Australians batted first</p><p>A stunning performance from India and their fortunes were reversed</p><p>Inspired by their captain’s century, India had the Aussies on the mat</p><p>The skipper a man of few words, did all the talking with his bat</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hd0J61S5UNN1kcJGTK9NHg.jpeg" /><figcaption>A captain’s century led India to a victory in the second test and revived their hope in the series (<a href="https://www.news18.com/cricketnext/news/india-vs-australia-captains-ton-from-ajinkya-rahane-gives-india-invaluable-first-innings-lead-3218303.html">source</a>)</figcaption></figure><p>Things were about to get ugly as the rivals made each other bleed</p><p>In a tense encounter, neither team would ever concede</p><p>Heading to Sydney, both of them readied their guns</p><p>For all the cricket lovers, this battle promised to be fun</p><p>At the SCG, Steve Smith shone brightly on his home soil</p><p>Scoring a stylish century, he made the visiting bowlers toil</p><p>Behind in the first innings, India had their backs to the wall</p><p>Between them and safety stood a target so massively tall</p><p>The equation was 309 runs to score off the final day</p><p>Starting with 8 wickets in hand, the game could go either way</p><p>The toil in the field had taken its toll, injuries leaving their batting thin</p><p>It was a tough dilemma between holding on or going for a win</p><p>To knockout the remaining batsmen, Australia assembled their crew</p><p>Using all their firepower they battered the Indians black and blue</p><p>Every bouncer hitting the batsman made a whole nation flinch</p><p>But the brave batsman held on for a draw making the journey inch by inch</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGz_IN5Dw9yEHK2qUeMbDw.jpeg" /><figcaption>Ashwin and Vihari battled injuries to hold on Australia to a draw at Sydney (<a href="https://www.espncricinfo.com/series/india-in-australia-2020-21-1223867/australia-vs-india-3rd-test-1223871/match-report">source</a>)</figcaption></figure><p>To call it a great escape wouldn’t be an overstatement for sure</p><p>It was a battle of minds which was fought with great composure</p><p>It wasn’t all rosy in the Indian camp as the draw came at a price</p><p>The loss of Ashwin and Bumrah meant India now needed the luck of the dice</p><p>Against all the odds India reached Australia’s Gabba fortress</p><p>Never had the home team lost here in 32 years no less</p><p>With an inexperienced bowling line up, few gave India a chance</p><p>Yet for an assault on Australia’s bastion, they readied their combat stance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsA1GDBRZH5pJn27CptgHg.jpeg" /><figcaption>Australia had an amazing record at the Gabba, having not lost in 32 years. (<a href="https://www.news18.com/cricketnext/news/india-vs-australia-can-india-breach-fortress-gabba-where-australia-have-remained-undefeated-for-over-32-years-3280214.html">source</a>)</figcaption></figure><p>To defend their kingdom, Australia put 326 on the board</p><p>The Indians were soon 5 down with less than half of that scored</p><p>A lower-order collapse was imminent, another dog fight on the cards</p><p>What followed though was certainly a story for the bards</p><p>Compared to their rivals, India’s lower order always lagged</p><p>But on this monumental occasion, the tail certainly wagged</p><p>Making the beleaguered bowlers suffer in the beating sun</p><p>Sundar and Thakur bridged the colossal deficit run by run</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TrocLWjJHXxuwnWSDQDKFQ.jpeg" /><figcaption>Thakur and Sundar fought hard, setting up India’s third-highest 7th wicket partnership in Australia (<a href="https://timesofindia.indiatimes.com/sports/cricket/india-in-australia/india-vs-australia-shardul-was-eager-to-get-to-his-fifty-i-knew-a-six-was-coming-says-sundar/articleshow/80315470.cms">source</a>)</figcaption></figure><p>With the ball once again the bowlers showed a lot of heart</p><p>Putting on a great show to topple Australia’s apple cart</p><p>Their efforts meant a victory for India was still within a sniff</p><p>But climbing mount 328 at the Gabba would a slippery cliff</p><p>Thrown into the deep waters, Gill thrived like a fish</p><p>Setting up a thrilling chase just as his captain would wish</p><p>Ruling out defeat was the specialty of Dravid, India’s wall</p><p>In his absence was his student Pujara standing tall</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*4omgdSYklpkYXKk1MH-lvw.jpeg" /><figcaption>Pujara held the fort against the Australian assault at the Gabba (<a href="https://www.news.com.au/sport/cricket/josh-hazlewood-accused-of-classless-sledge-after-hitting-indian-star-in-the-head/news-story/6668a019f0ab24c5bf5087c8a3c23a47">source</a>)</figcaption></figure><p>As the day passed on Australia’s hopes began to decline</p><p>Sloppy fielding and missed chances were not a good sign</p><p>A series that they were expected to win even before the first toss</p><p>Now looked like a genuine possibility of a humbling loss</p><p>As the target grew closer and overs began to dwindle</p><p>Silent prayers grew across India and hopes began to kindle</p><p>To salvage a draw, Australia bowled down the leg side</p><p>Exploiting the rule book as in Test matches, it is not called a wide</p><p>With T20 star Rishabh Pant still present on the pitch</p><p>The batting stance and shots slowly began to switch</p><p>Playing a finisher’s knock he consistently found the boundary</p><p>Taking the team to victory but narrowly missing out on a century</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nas6hgSPJ1Y5E1Tbd2NZZQ.jpeg" /><figcaption>Pant played a heroic counter-attacking knock to lead India to a memorable win (<a href="https://www.dnaindia.com/cricket/report-michael-vaughan-admits-egg-on-his-face-as-he-gets-india-vs-australia-4-0-prediction-horribly-wrong-as-ajinkya-rahane-india-national-cricket-team-register-historic-win-2869256">source</a>)</figcaption></figure><p>Turning doubters into believers, the young team silenced all critics</p><p>A series win after the first test debacle defied all arithmetic</p><p>They say pride often comes before a great fall</p><p>Australia’s defeat at the Gaba is a lesson for us all</p><p>Finally, the ghosts of the tours of 2004 and 2008 have been put to rest</p><p>This thrilling series win certainly ranks among India’s greatest</p><p>A fighting spirit in hostile conditions shows the team is in safe hands</p><p>Hope to see more such performances when they play in foreign lands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z02sKK121WTpgYRUUy8V7w.jpeg" /><figcaption>A win for India, a proud moment in the nation’s history, and hopefully a promise for the future (<a href="https://www.thestatesman.com/sports/india-conquer-gabba-beating-australia-fourth-test-win-four-match-series-2-1-1502947383.html">source</a>)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7731ea476449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating to Mutual TLS: What Istio does not tell you]]></title>
            <link>https://chintanbetrabet.medium.com/migrating-to-mutual-tls-what-istio-does-not-tell-you-7749e2de6133?source=rss-fd8e48f1139a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7749e2de6133</guid>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[secutity]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[mutual-tls]]></category>
            <dc:creator><![CDATA[Chintan Betrabet]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 08:29:23 GMT</pubDate>
            <atom:updated>2020-11-18T08:38:46.694Z</atom:updated>
            <content:encoded><![CDATA[<p>Istio is a powerful tool to enforce secure communication but the path from development to production is not as simple as it sounds.</p><p>The article has been co-authored by <a href="https://medium.com/u/b312f21c7733">Saurabh Shekhar</a></p><h3><strong>Enabling mTLS with Istio</strong></h3><p>There are multiple resources on the subject, you may explore the internet for the same or find a relevant article of mine <a href="https://chintanbetrabet.medium.com/securing-microservices-with-istio-79e3cf3a1ed2">here</a>.</p><p>The summary is to use a combination of PeerAuthentication and Destination rules to configure and enforce mTLS on both the sender side as well as receiver side However, it may not always be as simple as that.</p><h3>What to do when this does not work?</h3><p>Most of the references and examples in the Istio documentation are largely toy programs intended to illustrate how mutual TLS (<strong>mTLS</strong>) works. However, there are many cases where this won&#39;t work out of the box for you. Once you have applied the <strong>STRICT</strong> policy for PeerAuthentication and <strong>ISTIO_MUTUAL </strong>traffic policy Destination Rule, these are multiple issues that could surface in a pre-existing setup.</p><p>Some common problems are documented <a href="https://istio.io/latest/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule">here</a> but there are many more which I have encountered while migrating to Istio. The most common is a 503 error returned from the envoy proxy when sending a request between two pods. The best way to identify the issue is to check the envoy proxy logs using:</p><pre>kubectl logs PODNAME -c istio-proxy -n NAMESPACE</pre><p>However, it&#39;s best to take note of the following known causes before you proceed further into the envoy logs.<br>1. <strong>Missing sidecars</strong></p><p>This is one of the most common causes observed. When you try to communicate from a pod that does not have a sidecar to a pod having a sidecar with a strict authentication policy, all requests will be rejected as a mutual TLS handshake is not established in the absence of envoy sidecars.</p><p>You can use <a href="https://kiali.io/">Kiali</a> for visibility on your workloads. In this case, it will show a warning similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*t7SdsmLpK-gfdPwOBL1VGw.png" /></figure><p>This can be fixed by enabling automatic sidecar injection through labels/annotations at the namespace or deployment level.</p><pre>kind: Namespace<br>apiVersion: v1<br>metadata: <br>  name: foo<br>  labels:<br>    sidecar.istio.io/inject: “true”</pre><p>This will try to inject sidecars into all pods created in the foo namespace.</p><p><strong>Note: </strong>To reflect changes, a redeploy is needed, existing pods are <strong>not</strong> affected by this change.</p><p>If you want to enable for only certain pods/deployments, you can modify the label at a deployment level:</p><pre>apiVersion: apps/v1beta1<br>kind: Deployment<br>metadata:<br>  name: test-deploy<br>  namespace: foo<br>  labels:<br>    name: test-deploy<br>spec:<br>  replicas: 1<br>  strategy:<br>      type: RollingUpdate<br>      rollingUpdate:<br>        maxSurge: 80%<br>  template:<br>    metadata:<br>      annotations:<br>        sidecar.istio.io/inject: &quot;true&quot;<br>      labels:<br>        name: test-deploy<br>    spec:<br>      containers:<br>      - name: test<br>        image: test:latest</pre><p>The section below is used to denote that sidecar should be injected in all the pods of this deployment, irrespective of whether it is dictated at the namespace level</p><pre>template:<br>    metadata:<br>      annotations:<br>        sidecar.istio.io/inject: &quot;true&quot;</pre><p>Using these policies, you should be able to eradicate surprises introduced by new services that do not have sidecars injected.</p><p>2. <strong>Naming convention of service ports</strong></p><p>Istio is extremely reliant on port naming to establish communication. Check the examples given below:</p><pre>kind: Service<br>apiVersion: v1<br>metadata:<br>  name: working-http-svc<br>  namespace: foo<br>  labels:<br>    app: working-http-svc<br>spec:<br>  selector:<br>    name: working-http-svc<br>  type: ClusterIP<br>  ports:<br>  - name: http-svc-port<br>    protocol: TCP<br>    port: 80<br>    targetPort: 3000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  labels:<br>    app: not-working-http-svc<br>    name: not-working-http-svc<br>  name: not-working-http-svc<br>  namespace: foo<br>    <br>spec:<br>  ports:<br>  - name: not-working-http-svc-port<br>    port: 8125<br>    protocol: TCP<br>    targetPort: 28125<br>  selector:<br>    app: not-working-http-svc<br>  type: ClusterIP</pre><p>The correct naming convention is <strong>$protocol-$service</strong> as illustrated in <strong>working-http-svc </strong>where the port name is of the form: <strong>http-svc-port </strong>where we have HTTP as a prefix to denote the protocol.</p><p>The <strong>not-working-http-svc </strong>has a port name called <strong>not-working-http-svc-port </strong>which is missing the protocol prefix<strong>. </strong>The envoy on the client-side is unable to recognize “<strong>not” </strong>(prefix before the ‘<strong>-</strong>’) as a valid protocol for communication and hence aborts establishing a mutual TLS connection. This will cause all requests sent to this service to receive 503 errors.</p><p>This issue is thrown as a warning on Kiali as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*1Kgjs9ggjNjioeMdTmz-4w.png" /></figure><p>So the fix for this is simple. Rename the port in the configuration to <strong>tcp-app-port </strong>to resolve this issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*HeBsOAyfS4uvvFSNdh6g6Q.png" /></figure><p>3. <strong>Upstream naming and headers in Nginx</strong></p><p>Nginx is a popular reverse proxy and is a common choice to behave as the ingress service for an application. In Kubernetes, it can be used to route requests to independent upstream microservices based on the path prefix. An upstream can be defined as follows:</p><pre>upstream payments-backend {<br>  server payments.foo.svc.cluster.local;<br>}</pre><p>This can be referred to from a location block as:</p><pre>location  /pay {<br>      proxy_pass payments-backend;<br>  }</pre><p>Without an envoy proxy, this will work seamlessly. However, there are a few hiccups once the envoy proxy is introduced. While internally, Nginx is able to handle the routing from the name: <strong>payments-backend </strong>to the service: <strong>payments, </strong>Envoy is unable to identify this as a request to be routed within the Kubernetes cluster and routes the request via the <strong>Passthrough Cluster. </strong>Hence, it does not attempt to initiate mTLS. This will cause requests to be dropped by the receiving <strong>payments </strong>service as it has a strict authentication policy. To fix this, we need the sidecar to understand that the request is internal to the cluster and needs mTLS to be established. There are two ways to achieve this:</p><p><strong>a) Host headers</strong></p><p>In the above case, by default, envoy will consider the <strong>Host </strong>header as <strong>payments-backend </strong>and since it does not have a lookup entry corresponding to this name internal to the cluster, it treats it as an external request. We can explicitly set the host header as follows:</p><pre>location  /pay {<br>    proxy_set_header Host payments.foo.svc.cluster.local;<br>    proxy_pass payments-backend;<br>}</pre><p>This informs the proxy that the request is going to <strong>payments.foo.svc.cluster.local </strong>and since it deciphers that this is a valid service within the cluster, it honors the destination rule and initiates mTLS.</p><p><strong>b) Renaming the upstream</strong></p><p>Alternatively, you can rename the upstream to:</p><pre>upstream payments {<br>  server payments.foo.svc.cluster.local;<br>}</pre><p>and the location to:</p><pre>location  /pay {<br>    proxy_pass payments;<br>}</pre><p>This is cleaner than modifying headers and is more intuitive since the upstream name and the server for the upstream have the same name.</p><p><strong>Proxy HTTP version directive</strong></p><p>This however is not the end of the problem. Proxy pass needs an explicit directive to use HTTP 1.1 as envoy does not work with HTTP 1.0 connections. The directive is as follows:</p><pre>location  /pay {<br>    proxy_http_version 1.1;<br>    proxy_pass payments-backend;<br>}</pre><p>These suggested changes resolved the issues we faced while implementing mTLS with Nginx as a reverse proxy.</p><p><strong>4</strong>. <strong>Graceful termination of Istio containers</strong></p><p>As described in the architecture diagrams above, all requests routed to a pod are intercepted by the envoy sidecar and then proxied to the container running the HTTP service. In most cases, these services are stateless and its pods can be terminated instantly without any loss of data.</p><p>However, we may have use cases where a pod needs to gracefully handle termination such as:</p><p>a) Sending notifications to listeners or reporting cause of termination</p><p>b) Releasing locks on shared resources in a distributed system</p><p>c) Draining existing requests and sending callbacks on completion</p><p>All of these require communication with external services and will be routed via the envoy sidecar proxy.</p><p>These operations can be handled gracefully within the application using:</p><ol><li>Trapping Sigterm signals within the container</li><li>Overriding <strong>terminationGracePeriodSeconds </strong>for a pod</li></ol><p>However, when we introduce an additional hop through the envoy proxy, these external calls will fail as the envoy container will be terminated once Kubernetes sends a <strong>Sigterm</strong> signal to the pod. Envoy gracefully handles all the existing requests and allows them to drain but any new request, either ingress or egress will fail with <strong>503 </strong>as envoy has been terminated and is unable to handle new requests.</p><p><strong>Solution</strong></p><p>The desired solution is to ensure that the proxy sidecar is not terminated before the main container running the service is terminated. The problem is tricky since sidecar injection is performed by Istio and we do not own the lifecycle of the istio-proxy container. There is however a way to tackle the problem.</p><p><strong>Step 1:</strong> Disable the automatic injection of sidecars</p><p>This is essential as manual injection of sidecars allows us to control the final YAML which will be applied. To perform this, disable automatic sidecar injection at the namespace level:</p><pre>kind: Namespace<br>apiVersion: v1<br>metadata: <br>  name: foo<br>  labels:<br>    sidecar.istio.io/inject: “false”</pre><p>You will accordingly need to add the annotations at a deployment level as:</p><pre>template:<br>    metadata:<br>      annotations:<br>        sidecar.istio.io/inject: &quot;true&quot;</pre><p><strong>Step 2: </strong>Inject the sidecars manually, along with correct Pre stop lifecycle hooks</p><p>First, generate the required YAML with istio proxy using:</p><pre>istioctl kube-inject -f $your_yaml &gt; $destination_yaml</pre><p>After that, modify the lifecycle hook for the istio-proxy container as follows:</p><pre>conatainers:<br>- name: istio-proxy<br>  lifecycle:<br>    preStop:<br>      exec:<br>        command:<br>          - &#39;/bin/sh&#39;<br>          - &#39;-c&#39;<br>          - &#39;while [ 1==1 ]; do curl localhost:3000/health; if [ $? -gt 0 ]; then  break; fi; sleep 1; done&#39;</pre><p>The above pre stop hook will wait as long as the process on port 3000 (main container process) return 200 OK for /health API. This ensures that while the main service is up and running, istio-proxy will not be terminated. It is assumed that the graceful termination behavior was already working for the main process and istio-proxy will honor the same.</p><p>Once done, you can use <strong>kubectl </strong>to apply this modified YAML configuration.</p><p>You may refer to an effort made to simplify the process <a href="https://github.com/chintanbetrabet/istio-graceful-termination">here</a>. The aim is to build a config driven tool which will generate the appropriate YAML files with pre stop hooks, both with and without istio-proxy injection.</p><p>5. <strong>Confirmation of encryption of traffic</strong></p><p>The migration path from non-Istio to Istio with mTLS has the following phases:</p><p>i) Introduce sidecars and destination rules</p><p>ii) Enforce the Permissive mode of peer authentication to provide backward compatibility</p><p>iii) Verify that traffic encrypted using mTLS</p><p>iv) Enforce the Strict mode of peer authentication so that any new communication will not be allowed unless it is following mTLS</p><p>Step (iii) is the most critical since we need to ensure traffic is encrypted with mTLS before we enforce strict mode as without this, requests will fail and this is not acceptable in production environments.</p><p>Apart from manually verifying the N X (N-1) modes of combinations of service to service communication between N services, you can use <strong>Prometheus</strong></p><p>Istio exposes a Prometheus dashboard which can be used to monitor all Istio requests. It can be accessed using:</p><pre><strong>istioctl dashboard prometheus</strong></pre><p>You can also use the Grafana dashboard with:</p><pre><strong>istioctl dashboard grafana</strong></pre><p>One key metric is:</p><pre><strong>istio_requests_total{connection_security_policy!=”mutual_tls”}</strong></pre><p>The metric should <strong>not increase</strong> with traffic once your services are expected to be moved to mTLS communication. If this is increasing, you should dig deeper and identify which of your communications are not secured. You can use the following tags:</p><p>a) <strong>destination_workload</strong></p><p>b) <strong>source_workload</strong></p><p>c) <strong>request_protocol</strong></p><p>You can check the steps mentioned earlier in the article to ascertain the cause of the communication not being secured.</p><h3><strong>Conclusion</strong></h3><p>The path to productionizing mTLS with zero downtime is a tricky path. However, if you follow the steps mentioned here, the journey will certainly be smoother. Hope the experiences documented here can be used both by engineers presently working on migrating to istio as well as those aiming to do so in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7749e2de6133" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing microservices with Istio]]></title>
            <link>https://chintanbetrabet.medium.com/securing-microservices-with-istio-79e3cf3a1ed2?source=rss-fd8e48f1139a------2</link>
            <guid isPermaLink="false">https://medium.com/p/79e3cf3a1ed2</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[tls]]></category>
            <dc:creator><![CDATA[Chintan Betrabet]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 08:26:48 GMT</pubDate>
            <atom:updated>2020-11-18T08:40:48.442Z</atom:updated>
            <content:encoded><![CDATA[<p>Using the power of mTLS to secure services with minimal application intervention</p><h3>What is Istio?</h3><p><a href="https://istio.io/">Istio</a> is a service mesh used to manage microservices on Kubernetes. It is used to connect, control, secure, and observe microservices at scale. It consists of two major components:</p><ol><li>Istio control plane</li><li>Envoy proxy</li></ol><p>You can find details on how to set up Istio <a href="https://istio.io/latest/docs/setup/getting-started/">here</a>. It is suggested that the reader also checks the compatibility matrix to ensure your Istio and Kubernetes (and correspondingly the istioctl and kubectl) versions are compatible.</p><p>Before diving in further, let us have a quick look at the architecture of Istio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*greTwQ5VRN4yfEEGts0g2A.png" /></figure><p>You can also refer to the <a href="https://istio.io/latest/docs/ops/deployment/architecture/">source</a> from the official documentation</p><p>The control plane runs in the <strong>istio-system</strong> namespace and consists of:<br>1. Pilot: responsible for the lifecycle envoy sidecars</p><p>2. Citadel: handles all certificates used for communication and their rotation and propagation</p><p>3. Galley: manages the configuration for Istio</p><p>The main takeaway from this diagram though is that all service communication, both ingress, and egress (between pods) occur via the sidecar which acts as a proxy. This uses E<a href="https://www.envoyproxy.io/">nvoy</a> for intercepting the traffic and provide all the monitoring, tracking, and security functionalities promised by Istio.</p><h3>Why Istio?</h3><p>For internal communication, most servers tend to use HTTP for communication as it does not have the overheads of certificate management. While this is acceptable in certain scenarios, compliance requirements like the Health Insurance Portability and Accountability Act (<a href="https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act"><strong>HIPAA</strong></a><strong>)</strong> will be violated if any channel transmitting Patient Health Information (<strong>PHI</strong>) is unencrypted.</p><p>One to achieve this is to migrate services to HTTPS however, there are certain hurdles:</p><ol><li><strong>Application-level changes</strong></li></ol><p>All applications and microservice owners will need to change the servers from running on HTTP protocols to HTTPS protocols. Further, all client-side calls also need to be migrated to use HTTPS instead of HTTP.</p><p>2. <strong>Trust stores for certificates</strong></p><p>Running HTTPS servers needs:</p><p>a) A public key and a private key for SSL encryption as well as a CA certificate authority to sign them</p><p>b) All clients to trust the server’s certificate to establish a handshake</p><p>3. <strong>Rotation of certificates</strong></p><p>All SSL certificates have a limited time before expiration after which connections will no longer be considered trusted. The upgrade to new certificates can be a tedious process for the reasons mentioned above and adds a burden to the service owners.</p><p>Istio takes care of all of the above problems in a very concise and application-agnostic manner.</p><h3>Istio and Mutual TLS</h3><p><strong>Certificate management</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Juw7UG6KjWslLvebwI_0-A.png" /></figure><p>This diagram from the <a href="https://istio.io/latest/docs/concepts/security/#pki">source</a> shows how certificate management is done efficiently by Istio to ensure communication between envoy sidecars can be encrypted using self-signed certificates. It also handles the rotation of the certificates so there is no operational overhead for the same.</p><p>These certificates can be used along with Istio policies to enforce <strong>mutual TLS </strong>to secure communication between workloads.</p><p><strong>Configuring Mutual TLS</strong></p><p>For the internal mechanism of mutual TLS, <a href="https://istio.io/latest/docs/concepts/security/#mutual-tls-authentication">this</a> is a good reference. Let us focus on how to achieve this for our applications.</p><p>The Envoy sidecars take care of establishing mutual TLS connections based on the following configurations:<br>1. <strong>Destination Rules</strong> (client-side)</p><p>These rules are configured on the client-side to ensure that the client initiates a mutual TLS connection in case the destination matches one of the rules specified. An example is given below:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: DestinationRule<br>metadata:<br>  name:  default-istio-mtls<br>  namespace:  test-ns<br>spec:<br>  host: &#39;*.test-ns.svc.cluster.local&#39;<br>  trafficPolicy:<br>    tls:<br>      mode: ISTIO_MUTUAL</pre><p>Breaking it down, we are declaring: a destination rule, with the name <strong>default-istio-mtls </strong>in the <strong>test-ns </strong>namespace which will enforce a traffic policy with TLS mode as <strong>ISTIO_MUTUAL </strong>(mutual TLS) for any host which matches <strong>*.test-ns.svc.cluster.local </strong>or any communication to a service endpoint within the same namespace.</p><p>This can be overwritten at a service level for instance:</p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: DestinationRule<br>metadata:<br>  name:  my-svc-custom-rule<br>  namespace:  test-ns<br>spec:<br>  host: &#39;my-svc.test-ns.svc.cluster.local&#39;<br>  trafficPolicy:<br>    tls:<br>      mode: SIMPLE</pre><p>So for my-svc service, a mutual TLS handshake will not be initiated. This is <strong>not recommended</strong> in production for compliance reasons but it may be useful when debugging issues while migrating to mutual TLS.</p><p>2. <strong>PeerAuthentication</strong> (server-side)</p><p>On the server-side, we use these policies to govern which requests will be accepted by the receiver’s envoy sidecar. This can be of two types:</p><p>a) Permissive (Allow plain text as well as mutual TLS encrypted requests)</p><p>b) Strict (Allow only mutual TLS encrypted requests)</p><p>These can be defined in a hierarchical manner with precedence take in the order of port &gt; service &gt; namespace &gt; mesh policy.</p><p>Examples:</p><pre>apiVersion: security.istio.io/v1beta1<br>kind: PeerAuthentication<br>metadata:<br>  name:  istio-strict<br>  namespace:  foo<br>spec:<br>  mtls:<br>    mode: STRICT<br>---<br>apiVersion: security.istio.io/v1beta1<br>kind: PeerAuthentication<br>metadata:<br>  name: istio-permissive-service<br>  namespace: foo<br>spec:<br>  selector:<br>    matchLabels:<br>      name: permissive-service<br>  mtls:<br>    mode: PERMISSIVE<br>---<br>apiVersion: security.istio.io/v1beta1<br>kind: PeerAuthentication<br>metadata:<br>  name: port-level-permissive<br>  namespace: foo<br>spec:<br>  selector:<br>    matchLabels:<br>      app: port-level-permissive<br>  mtls:<br>    mode: STRICT<br>  portLevelMtls:<br>    3000:<br>      mode: DISABLE</pre><p>This policy enforces strict TLS at the namespace level but permissive mode for <strong>permissive-service. </strong>Further for <strong>port-level-permissive </strong>service, only communication on port 3000 will be permissive and all other ports will be strict.</p><p>As illustrated, a combination of Destination rules and PeerAuthentication can be used to enforce mutual TLS and secure your communication.</p><h3><strong>Conclusion</strong></h3><p>Istio makes life much easier when it comes to securing microservices. The information mentioned here though is largely introductory. For more details on debugging issues when migrating to Istio and mTLS, you can check out my article <a href="https://chintanbetrabet.medium.com/migrating-to-mutual-tls-what-istio-does-not-tell-you-7749e2de6133">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79e3cf3a1ed2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KISS: Kubernetes Is Simple Silly]]></title>
            <link>https://chintanbetrabet.medium.com/kiss-kubernetes-is-simple-silly-a9e8822f3af3?source=rss-fd8e48f1139a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9e8822f3af3</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[pods]]></category>
            <dc:creator><![CDATA[Chintan Betrabet]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 09:53:06 GMT</pubDate>
            <atom:updated>2020-11-13T09:53:06.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*54Fc0LttiyCkVZrrGOSYGw.png" /></figure><h4>Glossary of commonly used terms in Kubernetes</h4><p>For a new learner, Kubernetes (commonly referred to as K8s) can be a steep learning curve. This article covers a brief conceptual overview to kickstart your journey</p><h3>Kubernetes Cluster</h3><p>A Kubernetes cluster has two components:</p><ol><li>master nodes which run the Kubernetes related daemons (Kube API, Kube proxy, Kube DNS, Kube dashboard,)</li><li>Cluster nodes belonging to one or more node pools which serve as the underlying physical resources for all the containers</li></ol><h3>Node pools</h3><p>A homogenous set of physical resources that provide the underlying resources for the cluster. A cluster can have one or more instance groups with labels to provide information to the Kubernetes scheduler on what can be run on the provided hardware.</p><h3>Containers</h3><p>A container is a resource running a single image (generally a docker image) containing all executable packages, runtime, operating system, system libraries. This is analogous to docker containers used by other orchestration tools.</p><h3>Pod</h3><p>Pods are the smallest deployable units of computing that can be created and managed in Kubernetes. A pod is a set of one or more containers deployed and run together, sharing the same local network space. The recommendation is to run one container on a pod so that you can atomically control a process at a Kubernetes level but multi-container pods are not uncommon.</p><h3>ReplicaSet</h3><p>A ReplicaSet ensures that a specified number of pod replicas are running at any time. In other words, a ReplicaSet makes sure that a pod or a homogeneous set of pods is always up and available.</p><h3>Deployment</h3><p>A Deployment controller provides a declarative wrapper on top of replica sets defining the templates used to build the homogenous pods and the replication number. Even if you need to just run one container instance of a particular service, it is recommended to create a deployment for it with a configuration of 1 replica.</p><h3>Horizontal Pod Scaler</h3><p>With Horizontal Pod Autoscaling, Kubernetes automatically scales the number of pods in a replication controller, deployment, or replica set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).</p><h3>Daemonset</h3><p>A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.</p><p>Some typical uses of a DaemonSet are:</p><ul><li>running a cluster storage daemon, such as glusterd, ceph, on each node.</li><li>running a logs collection daemon on every node, such as fluentd or logstash.</li></ul><h3>StatefulSets</h3><ul><li>Used to manage stateful applications.</li><li><em>Provides guarantees about the ordering and uniqueness</em> of these Pods</li><li>StatefulSet maintains a sticky identity for each of their Pods</li><li>Used typically for cases where predictability of deletion and creation order is important</li></ul><h3>Service</h3><p>A Service is an abstraction that defines a logical set of Pods and a policy by which to access them — sometimes called a micro-service. So if a deployment from the previous step exposed a web service through a port, then a service is how you would expose the deployment.</p><p>A service can be exposed outside using a load balancer or a cluster IP that is reachable from within the K8s cluster. The load balancer is a cloud-native LB like ELB in AWS. The cluster IP is a virtually routable IP.</p><p>Additionally, Kubernetes also gives each service a routable A record of the form $service-name.$namespace.svc.cluster.local</p><h3>Configmap</h3><p>Every K8s cluster can have one or more config maps. A configmap is a key-value store that is that can be made available to each container as part of the environment variables.</p><h3>Network Topology</h3><p>Every Kubernetes cluster satisfies the following requirements</p><ul><li>all pods can communicate with all other pods without NAT</li><li>all nodes can communicate with all pods (and vice-versa) without NAT</li><li>the IP that a pod sees itself as is the same IP that others see it as</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9e8822f3af3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>