<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by adil on Medium]]></title>
        <description><![CDATA[Stories by adil on Medium]]></description>
        <link>https://medium.com/@adil?source=rss-fdac53119065------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by adil on Medium</title>
            <link>https://medium.com/@adil?source=rss-fdac53119065------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:50:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Magic Behind Docker-in-Docker (DinD)]]></title>
            <link>https://adil.medium.com/the-magic-behind-docker-in-docker-dind-3cdb97dacc55?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cdb97dacc55</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[containerization]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 20:55:24 GMT</pubDate>
            <atom:updated>2026-01-04T15:26:55.491Z</atom:updated>
            <content:encoded><![CDATA[<p>Docker-in-Docker (DinD) is a popular technique in CI/CD pipelines to enable containerized builds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WaoaxBSPXtuivgK-" /><figcaption>Photo by <a href="https://unsplash.com/@hudsoncrafted?utm_source=medium&amp;utm_medium=referral">Debby Hudson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>But why is DinD necessary?<br>Does it actually run a Docker container inside another Docker container?</p><p>Let’s say you use Gitlab CI (or Jenkins, Circle CI etc.). In your .gitlab-ci.yml file, you define a task like this:</p><pre>image: ubuntu:25.10<br><br>services:<br>  - docker:dind<br><br>variables:<br>  DOCKER_HOST: tcp://docker:2375/<br>  DOCKER_TLS_CERTDIR: &quot;&quot;<br><br>stages:<br>  - test123<br><br>run-tests123:<br>  stage: test123<br>  before_script:<br>    - apt-get update &amp;&amp; apt-get install -y docker.io<br>  script:<br>    - docker pull ailhan/web-debug:latest<br>    - docker run ailhan/web-debug:latest ./runtests.sh</pre><h3>The problem definition</h3><p>You want to download the <a href="https://hub.docker.com/r/ailhan/web-debug">ailhan/web-debug</a> image, run it as a container, and perform some tests.</p><p>Forget about the above-mentioned example yaml file.</p><p>You run CI jobs in a Docker container or a Kubernetes pod.</p><p>You want to get the latest container image from artifactory and run some tests.</p><p>When your CI job runs inside a container or Kubernetes pod, it cannot directly launch sibling Docker containers because Docker requires access to the host’s Docker daemon, which is not accessible inside the container by default.</p><p>There should be a mechanism that allows you to launch another Docker container on the host machine.</p><h3>The solution</h3><p>This is where DinD (Docker-in-Docker) comes into play.</p><p>It sounds like we will run a Docker container within a Docker container.</p><p>Actually, it’s not exactly running a container fully nested inside another container.</p><p>We will run the CI job (run-tests123) inside a special container (docker:dind), which allows you to run another Docker container on the host machine.</p><p>DinD doesn’t allow you to run a Docker container fully nested inside another container. Instead, it runs a Docker daemon inside a special container (docker:dind) with elevated privileges.</p><p>Your CI job connects to the Docker daemon running inside the DinD container, which acts like a Docker host and lets you launch sibling containers.</p><p>DinD allows you to create another container (ubuntu:25.10) on the host machine.</p><p>We install the Docker CLI inside the Ubuntu container to communicate with the Docker daemon running inside the DinD container.</p><p>Thanks to the environment variable DOCKER_HOST, it will connect to the Docker daemon on the host machine.</p><p>After that, we will be able to run <a href="https://hub.docker.com/r/ailhan/web-debug">ailhan/web-debug</a> container and perform the tests.</p><h3>What’s the magic behind the Docker-in-Docker container?</h3><p>It’s actually a simple container. It simply installs Docker and runs the container with the --privileged flag. The --privileged flag grants the container extended privileges, allowing it to interact closely with the host system and its resources (for more details, see cgroups).</p><p>This allows the CI Job to run another container on the host machine.</p><p>Here’s the docker-in-docker’s Dockerfile:<br><a href="https://github.com/jpetazzo/dind/blob/master/fedora/Dockerfile">https://github.com/jpetazzo/dind/blob/master/fedora/Dockerfile</a></p><p>In 2013, Docker implemented the --privileged flag that makes DinD possible:</p><p><a href="https://github.com/moby/moby/commit/280901e5fbd0c2dabd14d7a9b69a073f6e8f87e4">https://github.com/moby/moby/commit/280901e5fbd0c2dabd14d7a9b69a073f6e8f87e4</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cdb97dacc55" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Setup Transit Gateway + Site to Site VPN and NAT Gateway?]]></title>
            <link>https://adil.medium.com/how-to-setup-transit-gateway-site-to-site-vpn-and-nat-gateway-c26d6663873e?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/c26d6663873e</guid>
            <category><![CDATA[site-to-site-vpn]]></category>
            <category><![CDATA[transit-gateway]]></category>
            <category><![CDATA[nat-gateway]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Wed, 11 Dec 2024 23:06:17 GMT</pubDate>
            <atom:updated>2024-12-12T08:15:14.811Z</atom:updated>
            <content:encoded><![CDATA[<p>I have previously published two articles explaining the separate installation of Transit Gateway and IPSec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xRGGAPLLBfKoPgqA" /><figcaption>Photo by <a href="https://unsplash.com/@thomasjsn?utm_source=medium&amp;utm_medium=referral">Thomas Jensen</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>They are here:</p><p><a href="https://adil.medium.com/how-to-setup-ipsec-between-3-linux-servers-5596798eac11">How to Setup IPSec Between 3 Linux Servers?</a><br><a href="https://adil.medium.com/how-to-enable-peering-between-multiple-inter-region-vpcs-via-transit-gateway-36fa47d2d53a">How to Enable Peering Between Multiple (Inter-Region) VPCs via Transit Gateway?</a></p><p>In this article, I’ll walk through establishing the transit gateway and site-to-site VPN together.</p><p>Using <a href="https://adil.medium.com/how-to-enable-peering-between-multiple-inter-region-vpcs-via-transit-gateway-36fa47d2d53a">the tutorial</a> and a little tweak to the configuration, I set up a Linux virtual machine on Digitalocean and installed Strongswan.</p><p>The start parameter will be set to <strong>add</strong>.</p><p>In IPsec, one side of the tunnel should initiate the connection. In this tutorial, the AWS side will establish the connection.</p><p><a href="https://serverfault.com/questions/556885/using-strongswan-whats-the-difference-between-auto-add-and-auto-start">what’s the difference between auto=add, and auto=start?</a></p><p>Network configurations for instance in Digitalocean are as follows:</p><blockquote>Public IP Address: 164.92.144.165<br>Private IP Address: 10.110.0.2<br>Subnet: 10.110.0.0/20</blockquote><h3><strong>Create Customer Gateway and Transit Gateway</strong></h3><p>I’ll create a customer gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*syLsPnWpU04Zk_i-7VNMdA.png" /></figure><p>Enter the public IP address of your pair and the name of the customer gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*Kph7jHl65E1iBy7ImrIPpQ.png" /></figure><p>The customer gateway will be available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lr97erS3bj6lC0G5YV2dsg.png" /></figure><p>I’ll create a transit gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0sGv5SJM3oMo46njlU8hLQ.png" /></figure><p>Name your transit gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*cPlXxZwR5bznVltj0RPP3Q.png" /></figure><p>The transit gateway will be available after a time;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMzsrOlNX7kBkMTigvGHvw.png" /></figure><h3><strong>Initiate the VPN Connection</strong></h3><p>We will be able to start the VPN connection using both the customer gateway and the transit gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q2m6vJ6Gx8g4MoRdeJuRMQ.png" /></figure><p>Fill in the VPN details:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*yUPLXwHzgB487YofSlQf5w.png" /></figure><p>Scroll down on the same page to add a random pre-shared key to the tunnels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*vEV_SaffG784Bnvmg4yr8A.png" /></figure><p>Click the Create VPN connection button.</p><p>The VPN connection will be created, and AWS will generate two tunnels with two distinct Public IP addresses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KvEP5ersKUCbUPI__U17DQ.png" /></figure><p>Let’s examine the startup action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eg5yaZy6PMPBjctz2wGqOw.png" /></figure><p>The tunnel’s startup action is Default, which indicates <strong>add</strong>. As stated at the start of the article, the AWS side will initiate the VPN connection. Therefore, the startup action need to be set to <strong>start.</strong></p><p>(<a href="https://serverfault.com/questions/556885/using-strongswan-whats-the-difference-between-auto-add-and-auto-start">what’s the difference between auto=add, and auto=start?</a>)</p><p>Let’s change the tunnel’s configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dLlf92pPgfib9Fn2lQhLxA.png" /></figure><p>Select <strong>start</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*UD_8GbzgI3Flf2OL3PF0vQ.png" /></figure><p>The startup action will be changed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VLPkOmLDFtJFUKxol0WJUQ.png" /></figure><h3><strong>Redirect the traffic from the VPC to the VPN</strong></h3><p>I have an EC2 instance operating on a private subnet.</p><blockquote>Public IP Address: N/A<br>Private IP Address: 172.16.8.157<br>Private Subnet: 172.16.8.0/24<br>Network: 172.16.0.0/20</blockquote><p>I am going to create another subnet to be utilized for the NAT Gateway: 192.0.11.0/20</p><h3><strong>What? Your network is 172.16.0.0/20, but will you create a subnet with 192.0.x.x?</strong></h3><p>Yes, I will add another CIDR to my VPC:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5rRZnbr6kvwcyMB_yb5jg.png" /></figure><p>Add a new IPv4 CIDR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7PZKieUl1RgbzYrnjcJcRg.png" /></figure><p>Add 192.0.0.0/20:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXYfVexUfow7ZKQWKOFI-Q.png" /></figure><p>Added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKjFA9kOItMK1DIDghyRmQ.png" /></figure><p><strong>Why not 192.168.x.x? Because it is restricted by AWS:</strong></p><p><a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html">https://docs.aws.amazon.com/vpc/latest/userguide/vpc-cidr-blocks.html</a></p><h3><strong>Create a VPN subnet</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3kp68misB4oH0U48IExk-g.png" /></figure><p>Fill out the subnet form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*6IpKBCm4BYnhEUZ0Q91m5g.png" /></figure><p>Click Create subnet.</p><p>The subnet gets created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qogeRTxctyhLX9G59mEWVg.png" /></figure><p>Let’s set up a NAT gateway utilizing this subnet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*B9dcQWLi8-nmh9p5Tztueg.png" /></figure><p>It will become available after a while:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_AtQhMYIZKLO02rHEB_gwQ.png" /></figure><p>I will update the route table of the subnet of my EC2 instance;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qhixiYijMCLsRT8fpwm3Vw.png" /></figure><p>Route table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-tA3a5WfWWkVGgHKHJycwg.png" /></figure><p>I will add the Digitalocean subnet (10.110.0.0/20) to the route table and select the NAT Gateway I created for the VPN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJ0-VLOImX5dFA8kaVWXMA.png" /></figure><p>Added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdlJBOq9dm5guryLZcJJWQ.png" /></figure><h3><strong>Attach VPC To the Transit Gateway</strong></h3><p>We have not added the VPC to the transit gateway. However, when we choose Transit Gateway during VPN Creation, VPN is automatically added to the transit gateway.</p><p>Go to the Transit Gateway Attachments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*-UApl365rM1Ug9hmRsTh8w.png" /></figure><p>Add the VPC to the Transit Gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*OG9e5JwpGrK4wQRvzdOFfw.png" /></figure><p>Added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7QwaNakWGWsOOMJNMFZJg.png" /></figure><h3><strong>Update Transit Gateway Route Table</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C5agI4_SPs1YmUXY-Wj8sA.png" /></figure><p>Redirect all traffic that arrives at the transit gateway to the VPN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*DrS2lB7yfCg5cxbH0Hglow.png" /></figure><p><strong>Any traffic? I assumed just <em>10.110.0.0/20</em> would be forwarded to the VPN.</strong></p><p>Correct. Please keep in mind that we routed the traffic (10.110.0.0/20) from our private subnet to Transit Gateway. So, any traffic arriving at TGW will be forwarded to the VPN.</p><p>So, in the future, if you want to access another subnet over VPN, you must change the route table in VPC. You do not need to modify TGW.</p><p><strong>The route table was updated:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgB0FOhQWIaqY2xhJ15P4g.png" /></figure><p>The route state seems to be a black hole as the VPN connection has not been established.</p><p>Return to the VPN subnet page and redirect all VPN subnet traffic to TGW.</p><p>Go to the VPN subnet’s route table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCeLr6woF0lg7R44TZwI0g.png" /></figure><p>Edit routes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1pGlFuHZgK2IjYaxpqB9Q.png" /></figure><p>Redirect all traffic arriving on the VPN subnet to Transit Gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z1761FTyheWrznP1w5hWzg.png" /></figure><p>The route table was updated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4z4NcoaJFHPHe5rZ6ZXWg.png" /></figure><p>I will update the IPsec configuration in Digitalocean:</p><pre>root@digitalocean-vpn:~# cat /etc/ipsec.conf<br>config setup<br>      uniqueids = yes<br>conn aws-digitalocean-adil<br>      type=tunnel<br>      auto=add<br>      authby=secret<br>      leftsubnet=10.110.0.0/20<br>      left=164.92.144.165<br>      right=34.246.150.189<br>      rightsubnet=192.0.11.0/24</pre><p>And key;</p><pre>root@digitalocean-vpn:~# cat /etc/ipsec.secrets<br>164.92.144.165 : PSK &quot;add_some_long_string_here&quot;</pre><p>The VPN connection is UP:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*et4gtpNaqq9D1y8nX0fQgg.png" /></figure><p>Route state is no longer a blackhole:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Cui-rbdqbyIlvaTSnghfQ.png" /></figure><p>I setup a Nginx server on the Digitalocean VM. I’ll attempt to connect it from EC2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*Vc8DrffrD8pt5zl5R_1miw.png" /></figure><p>IP configuration and traceroute output from EC2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQbAP7bqwTt80I2l_j0Uyg.png" /></figure><p>Remember that the IP address 192.0.11.111 is the IP address of NAT Gateway:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-DK5iHJ1iK-X16dwyVG9og.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c26d6663873e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why use DNS ALIAS record instead of CNAME in the Cloud?]]></title>
            <link>https://adil.medium.com/why-use-dns-alias-record-instead-of-cname-in-the-cloud-ca995b7a364d?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca995b7a364d</guid>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[route-53]]></category>
            <category><![CDATA[load-balancing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sat, 04 May 2024 15:40:51 GMT</pubDate>
            <atom:updated>2024-05-13T13:33:10.712Z</atom:updated>
            <content:encoded><![CDATA[<p>Having a custom domain on top of a cloud resource is essential.</p><p><strong>***** </strong><a href="https://www.linkedin.com/posts/adililhan_cloud-aws-cloudsecurity-activity-7193134936143396865-qdco/"><strong>You can follow me on LinkedIn</strong></a><strong> *****</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jFESz-2oT_BmdxD7" /><figcaption>Photo by <a href="https://unsplash.com/@joannakosinska?utm_source=medium&amp;utm_medium=referral">Joanna Kosinska</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Cloud providers create a resource (such as a load balancer), and provide you with a domain, such as: <br>some-production-loadbalancer-123456.eu-west-1.elb.amazonaws.com</p><p>You add this domain as a CNAME record to app.yourcompany.com</p><h3><strong>What happens if you delete the resource on the Cloud?</strong></h3><p>If you remove the resource but leave the CNAME record on the custom domain (app.yourcompany.com), you will encounter the Dangling CNAME problem.</p><h3><strong>What is the Dangling CNAME issue?</strong></h3><p>Let’s have a look at an actual example;</p><p>I created a domain: blog.example.adililhan.com <br>I created a load balancer on AWS: <br>adil-blog-test-1393565408.eu-west-1.elb.amazonaws.com</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bAf6XT-Qt0F6cofClx0g1w.png" /></figure><p>I added the load balancer’s domain name as a CNAME to my domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjqMdxQVmZQaDT4iK7xR-g.png" /></figure><p>Just for testing, I will send a request:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*UVPvzeYjjOqtxtnvVjMztw.png" /></figure><p>I received a response from the AWS resource (an EC2 instance).</p><p>I will delete the load balancer and see what happens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Q8gHgp9riB_hpuUVNhtyw.png" /></figure><p>DNS Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cGysWsSYEeoemq5lBHGGTA.png" /></figure><p>AWS has no longer returned anything for the load balancer domain since it was deleted.</p><p>I will create a new load balancer with the same name: adil-blog-test</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHCbbdaid-IptCU8YQ9FOA.png" /></figure><p>This is the new load balancer’s domain name:</p><p>adil-blog-test-1109044799.eu-west-1.elb.amazonaws.com</p><p>It is quite similar to the previous one. Only the numbers are different.</p><h3><strong>This is the problem: Random numbers</strong></h3><p>AWS uses the following naming convention to establish domains for its load balancers: $YourLoadBalancerName-$RandomNumber.$RegionName.elb.amazonaws.com</p><p>Attackers may identify 3 variables in your DNS output:</p><pre>$YourLoadBalancerName -&gt; adil-blog-test<br>$RandomNumber -&gt; 1109044799<br>$RegionNumber -&gt; eu-west-1</pre><p>So, the attacker cannot control just one variable: <strong>$RandomNumber</strong></p><p>I removed the load balancer, but maintained its domain name in my custom domain ( blog.example.adililhan.com).</p><p>So, if the attacker deletes/creates a sufficient number of load balancers, the attacker will eventually get the same number of removed one.</p><h3><strong>Solution: Use DNS Alias Records</strong></h3><p>DNS has a lot of distinct record types: A, AAAA, CNAME, MX, etc.</p><p>The ALIAS record type is not one of the standard record types.</p><p>Only a few service providers, such as <strong>AWS</strong>, <strong>CloudFlare</strong>, and <strong>Azure</strong>, support the <strong>DNS ALIAS</strong> record type.</p><blockquote>The ALIAS record type is comparable to CNAME. You provide the load balancer’s domain name in the service provider configuration.</blockquote><blockquote>The service provider will check the returning IP addresses from the load balancer’s domain address and update your domain’s A records accordingly.</blockquote><blockquote>If the load balancer’s IP address changes, your custom domain’s A records are immediately updated.</blockquote><p>Let’s add the load balancer’s domain name as an ALIAS record.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJzVt6fKiG_B9BjOycyWBA.png" /></figure><p>Check the DNS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1wzjN10FLEjD_1j-BNemQ.png" /></figure><p>As a result, the attacker cannot determine the load balancer&#39;s domain name.</p><p>For testing purposes, I will send a request:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnYQ7e-Q3kbe6w1BxvWlgA.png" /></figure><p>I deleted the load balancer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5h3jWTxjtV0dS-OtTLFMbg.png" /></figure><p>Upon detecting the deletion of the load balancer, AWS proceeded to remove the IP address associated with the load balancer from my custom domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wPVvnfgHyPUQg9csA2UTg.png" /></figure><p>The ALIAS record eliminates the need to monitor CNAME entries on your domains.</p><p>This technique is also known as DNS Flattening.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca995b7a364d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Two Network Interfaces on the Same Subnet: ARP Flux Issue]]></title>
            <link>https://adil.medium.com/two-network-interfaces-on-the-same-subnet-arp-flux-issue-2a6c96523799?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a6c96523799</guid>
            <category><![CDATA[cloud-networking]]></category>
            <category><![CDATA[tcp-ip]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[network-engineering]]></category>
            <category><![CDATA[arp-protocol]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sun, 28 Apr 2024 12:17:24 GMT</pubDate>
            <atom:updated>2024-04-30T11:54:40.501Z</atom:updated>
            <content:encoded><![CDATA[<p>It might be necessary to utilize several network interfaces within a single subnet. Separating application traffic from replication traffic on the services (like Kafka, Elasticsearch, etc.) is a common use case.</p><p><strong>Linux uses the weak host model; hence, having multiple network interfaces might lead to unexpected network issues.</strong></p><h3><strong>Table of Contents</strong></h3><p>· <a href="#ce8f">What is the host model?</a><br>· <a href="#c3db">What is the ARP Flux problem?</a><br>· <a href="#60a8">Why is having two MAC addresses a problem?</a><br>· <a href="#e01c">Reproduce the ARP Flux Problem</a><br>· <a href="#5b3d">Let’s fix the multiple MAC addresses issue:</a><br>· <a href="#89ad">Routing Problem</a><br>· <a href="#7b04">Fix the routing issue on the multiple network interfaces</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1yc1SwX4ipfbgs3F" /><figcaption>Photo by <a href="https://unsplash.com/@zacong?utm_source=medium&amp;utm_medium=referral">Zac Ong</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What is the host model?</h3><p>Either the <strong>weak host model</strong> or the <strong>strong host model</strong> is used by modern operating systems.</p><p><strong>Weak host model (</strong><em>a.k.a. Weak End System Model or Weak ES Model</em><strong>)</strong>: If the network packet&#39;s destination IP address matches one of the IP addresses on its interfaces, the operating system will accept the packet.</p><p><strong><em>TL; DR:</em></strong> The operating system acts like a router for its network interfaces.</p><p>Linux uses weak host model.</p><p><strong>Strong host model</strong> <strong>(</strong><em>a.k.a. Strong End System Model or Strong ES Model</em><strong>)</strong>: If the IP address does not match the IP address on the interface that the traffic was routed through, the operating system does not accept the packet.</p><p><strong><em>TL; DR</em></strong>: The operating system does not act like a router for its network interfaces.</p><p>Newer Windows systems use strong host model.</p><h3><strong>What is the ARP Flux problem?</strong></h3><p>If the operating system gets an ARP request and there are multiple network interfaces on the same subnet, the MAC addresses of both will be returned due to the <strong>weak host model</strong>.</p><h3><strong>Why is having two MAC addresses a problem?</strong></h3><p>Having two MAC addresses isn’t an issue. The problem is that the OS returns multiple MAC addresses. The network routers could drop your packets. In particular, if the cloud provider’s source/destination check option is set, the packet will be dropped since the returned MAC address does not match the MAC address for the IP address.</p><h3><strong>Reproduce the ARP Flux Problem</strong></h3><p>The setup is as follows:</p><p>A virtual machine will act as a client.<br>Another virtual machine will act as a server.</p><p>The server has two interfaces:</p><pre>eth0: 10.211.55.13<br>eth1: 10.211.55.14<br>gateway: 10.211.55.1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dm4mrcqw3G8qvlqV47sR3A.png" /></figure><p>The client has one interface:</p><pre>eth0: 10.211.55.7<br>gateway: 10.211.55.1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1MlqtIzCcaK3VpfTcmJNyw.png" /></figure><p>I can ping both IP addresses from the client:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*dNn67toGGBZiuHpaUaXw5Q.png" /></figure><p>Let’s look at the ARP packets sent by the server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*n0gT_FxlMCWwaPG6yCcchg.png" /></figure><p>The server returns two MAC addresses.</p><h3><strong>Let’s fix the multiple MAC addresses issue:</strong></h3><p>I’ll activate arp_ignore on the server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ro6akZQy9cm-RU2M9m-1Aw.png" /><figcaption>Source: <a href="https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt">https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt</a></figcaption></figure><pre>echo 2 &gt; /proc/sys/net/ipv4/conf/all/arp_ignore</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ad5CeaR7XXrPBOG52NmmHQ.png" /></figure><p>The server returns one MAC address for each IP address.</p><h3><strong>Routing Problem</strong></h3><p>Let’s examine the Tcpdump output on the server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qUgMnaMw58lgknTuUPvR4A.png" /></figure><p>Notice the difference;</p><p>✅ ✅ When I sent a ping request to <strong>10.211.55.13</strong>, the request reached the server via its interface <strong>eth0</strong>, and the response was sent from the server’s <strong>eth0</strong> interface.</p><p>❗️❗️When I sent a ping request to <strong>10.211.55.14</strong>, the request reached the server via its interface <strong>eth1</strong>, and the response was sent from the server’s <strong>eth0</strong> interface</p><h3><strong>Fix the routing issue on the multiple network interfaces</strong></h3><p>Create a route table for each interface on the server:</p><pre>echo &quot;100 eth0rt&quot; &gt;&gt; /etc/iproute2/rt_tables<br>echo &quot;101 eth1rt&quot; &gt;&gt; /etc/iproute2/rt_tables</pre><p>Add necessary rules to the tables on the server:</p><pre>ip route add 10.211.55.0/24 dev eth0 src 10.211.55.13 table eth0rt<br>ip route add default via 10.211.55.1 dev eth0 src 10.211.55.13 table eth0rt<br>ip rule add from 10.211.55.13 table eth0rt<br>ip rule add to 10.211.55.13 table eth0rt<br><br>ip route add 10.211.55.0/24 dev eth1 src 10.211.55.14 table eth1rt<br>ip route add default via 10.211.55.1 dev eth1 src 10.211.55.14 table eth1rt<br>ip rule add from 10.211.55.14 table eth1rt<br>ip rule add to 10.211.55.14 table eth1rt</pre><p>Screenshot from the server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zwCdBfXgDgra_4-xTUDBg.png" /></figure><p>Let’s test the ping:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q94F9jXVTUUNxxIajQbeWg.png" /></figure><p>It works</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6c96523799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Rename enX0 to eth0 on Ubuntu/Debian?]]></title>
            <link>https://adil.medium.com/how-to-rename-enx0-to-eth0-on-ubuntu-debian-deb14f07d665?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/deb14f07d665</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[network-engineering]]></category>
            <category><![CDATA[network-interface-card]]></category>
            <category><![CDATA[aws-networking]]></category>
            <category><![CDATA[linux-networking]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Fri, 26 Apr 2024 21:09:53 GMT</pubDate>
            <atom:updated>2024-04-26T21:09:53.219Z</atom:updated>
            <content:encoded><![CDATA[<p>The interface name in the latest Ubuntu/Debian versions is determined by the hypervisor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*d-zVw2Up_BqJQdo0" /><figcaption>Photo by <a href="https://unsplash.com/@lukash?utm_source=medium&amp;utm_medium=referral">Lukas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For example, an Ubuntu instance running on Xen will have an interface similar to enX0. The interface name on other hypervisors may be something like ens5.</p><p>You can rename the interface name like this:</p><pre>sed -i &#39;s/enX0/eth0/g&#39; /etc/netplan/50-cloud-init.yaml</pre><p>After that, run this command:</p><pre>netplan apply</pre><p>or reboot the server.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=deb14f07d665" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Mirror Traffic to Another Pod in Kubernetes?]]></title>
            <link>https://adil.medium.com/how-to-mirror-traffic-to-another-pod-in-kubernetes-5edb2b7a5bba?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/5edb2b7a5bba</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mirroring]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes-networking]]></category>
            <category><![CDATA[cloud-network]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sat, 13 Apr 2024 09:50:36 GMT</pubDate>
            <atom:updated>2024-05-13T13:32:21.073Z</atom:updated>
            <content:encoded><![CDATA[<p>Mirroring network traffic in production has a lot of benefits, such as debugging, troubleshooting, testing, monitoring, etc.</p><p>***** <a href="https://www.linkedin.com/in/adililhan/"><strong>You can follow me on LinkedIn</strong></a><strong> *****</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qxk8iDjOYsOW8AUA" /><figcaption>Photo by <a href="https://unsplash.com/@jovisjoseph?utm_source=medium&amp;utm_medium=referral">Jovis Aloor</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Some Kubernetes plugins allow you to <strong>mirror traffic</strong> from <strong>one pod</strong> to <strong>another</strong>. In this tutorial, we will be using <strong>Istio</strong>, a well-known Kubernetes traffic management component.</p><p><strong>Istio</strong> provides a clear and concise <a href="https://istio.io/latest/docs/setup/install/istioctl/">installation document</a>. After installing <strong>istioctl</strong> on your computer,</p><p>Install <strong>istio</strong> with its CNI component:</p><pre>cat &lt;&lt;EOF &gt; istio-cni.yaml<br>apiVersion: install.istio.io/v1alpha1<br>kind: IstioOperator<br>spec:<br>  components:<br>    cni:<br>      enabled: true<br>EOF<br>istioctl install -f istio-cni.yaml -y</pre><p><strong>*** Note: Istio CNI is required to mirror the network ***</strong></p><p>Enable <a href="https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/">istio Sidecar</a> in your namespace (I’ll use the default namespace):</p><pre>kubectl label namespace default istio-injection=enabled</pre><p>Check if Istio is operating correctly:</p><pre>istioctl analyze</pre><h3><strong>Traffic Mirroring</strong></h3><p>Deploy two pods.</p><p>One will function like a production app, while the other will behave like a production app with debug mode turned on.</p><p><em>00-pod.yaml</em></p><pre>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    app: payment<br>    type: production<br>  name: payment-app<br>spec:<br>  containers:<br>    - image: ailhan/web-debug<br>      name: payment-app<br>      imagePullPolicy: Always<br>      ports:<br>      - containerPort: 80<br>      env:<br>      - name: TEXT<br>        value: &quot;Hello from the Payment App&quot;<br>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    app: payment<br>    type: debug<br>  name: payment-app-debug<br>spec:<br>  containers:<br>    - image: ailhan/web-debug<br>      name: payment-app-debug<br>      imagePullPolicy: Always<br>      ports:<br>      - containerPort: 80<br>      env:<br>      - name: TEXT<br>        value: &quot;Payment App - DEBUG Mode On&quot;</pre><p>Apply:</p><pre>➜  ~ kubectl apply -f 00-pod.yaml<br>pod/payment-app created<br>pod/payment-app-debug created</pre><p><em>01-service.yaml</em></p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: payment-app-svc<br>spec:<br>  ports:<br>  - port: 80<br>    targetPort: 80<br>    name: http-payment-app<br>  selector:<br>    app: payment</pre><p><strong>*** Note: A name must be added to the port The naming convention is </strong><strong>&lt;protocol&gt;[-&lt;suffix&gt;]***</strong></p><p>Apply:</p><pre>➜  ~ kubectl apply -f 01-service.yaml<br>service/payment-app-svc created</pre><p>Let’s test it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*5ilKxbWeul0dy3RHNkR7hg.png" /></figure><p>The requests are randomly distributed between two pods. It works.</p><p>For mirroring, I’ll use the payment-app-debug pod.</p><p><em>02-destination-rule.yaml</em></p><p>The destination rules define the rules that virtual services will follow.</p><pre>---<br>apiVersion: networking.istio.io/v1alpha3<br>kind: DestinationRule<br>metadata:<br>  name: payment-app-destination-rule<br>spec:<br>  host: payment-app-svc<br>  subsets:<br>  - name: main<br>    labels:<br>      type: production<br>  - name: mirror<br>    labels:<br>      type: debug</pre><p>Apply:</p><pre>➜  ~ kubectl apply -f 02-destination-rule.yaml<br>destinationrule.networking.istio.io/payment-app-destination-rule created</pre><p><em>03-virtual-service.yaml</em></p><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: VirtualService<br>metadata:<br>  name: payment-app-svc-virtual-service<br>spec:<br>  hosts:<br>  - payment-app-svc<br>  http:<br>  - route:<br>    - destination:<br>        host: payment-app-svc<br>        subset: main<br>      weight: 100<br>    mirror:<br>      host: payment-app-svc<br>      subset: mirror<br>    mirrorPercentage:<br>      value: 100</pre><p>Apply:</p><pre>➜  ~ kubectl apply -f 03-virtual-service.yaml<br>virtualservice.networking.istio.io/payment-app-svc-virtual-service created</pre><p>Let’s test it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*ETJ0BL68zNCqWlibyIwa9Q.png" /></figure><p>Istio’s virtual service intercepts the actual Service component. All requests are sent to the type: production app. Let’s examine the logs!:</p><p>Logs for the Production app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*s7N90gmIiFgMO86v92fy_A.png" /></figure><p>Logs for the Debug app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*rZyQ49JYZQVi28MuoABgpw.png" /></figure><p>It works. I got a response from the production app. However, the debug app got the same requests as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5edb2b7a5bba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How can an EC2 instance call itself via Internal NLB?]]></title>
            <link>https://adil.medium.com/how-can-an-ec2-instance-call-itself-via-internal-nlb-04201825534d?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/04201825534d</guid>
            <category><![CDATA[docker-networking]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[network-engineering]]></category>
            <category><![CDATA[nat]]></category>
            <category><![CDATA[aws-load-balancer]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 13:59:14 GMT</pubDate>
            <atom:updated>2024-04-07T18:56:25.711Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine that you have an application running on multiple EC2 instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A25FMwcsER7EAyPl" /><figcaption>Photo by <a href="https://unsplash.com/@jwimmerli?utm_source=medium&amp;utm_medium=referral">jean wimmerlin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The app has an internal network load balancer in front of it so that other lambda functions or apps in a different VPC can access it (via VPC peering).</p><p>Assume for the moment that you wish to call the application itself using the internal load balancer.</p><p><strong>It will fail. Why?</strong></p><p>Due to the internal load balancer&#39;s default behavior of keeping the source IP address.</p><p><strong>How so?</strong></p><p>Assume the following configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSFCtcsIJrdBMybJ7cXHIw.png" /></figure><p>If <strong>Instance 2</strong> receives a request sent from <strong>Instance 1</strong> to <strong>the load balancer</strong>, the request will be <strong>successful</strong>, and the network connections will appear as follows:</p><p><strong>Tcpdump on Instance 1</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3dI70IXLoHjIFu-k8kQ5A.png" /></figure><p><strong>Tcpdump on Instance 2</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABQ0v46mUycWymGI-yMBHA.png" /></figure><p><strong>Instance 1</strong> sent a request to <strong>the load balancer</strong>’s IP address and received a response back from the load balancer.</p><p><strong>Instance 2</strong> got a request from <strong>Instance 1</strong>’s IP address and responded to that address.</p><p><strong>Why does Instance 1 appear to have received its response from the load balancer when Instance 2 has directly returned the response to Instance 1?t</strong></p><h3><strong>Magic!</strong></h3><p>No, not really. This is what is called <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation">Client IP preservation</a>. NLB can <strong>preserve clients&#39; source IP</strong> addresses. The <strong>target server</strong> (Instance 2) can see the client’s (Instance 1) actual IP address rather than the load balancer’s IP address.<br>Since the target server is not aware of the load balancer’s IP address, the target server will try to send the request to <strong>the client directly</strong>.</p><p>However, <strong>AWS</strong> is aware of this fundamental issue. In their <strong>NAT Table</strong>, they modify the source IP address for Instance 1. In this way, Instance 1 thinks the answer is from the load balancer.</p><p><strong>What happens if Instance 1&#39;s request goes to Instance 1 rather than Instance 2?</strong></p><p><strong>Tcpdump</strong> on Instance 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9jxUJsdBrZT7SCAhWrPLvQ.png" /></figure><p>The request is sent to <strong>the load balancer</strong>. Because the NLB <strong>preserves</strong> the <strong>source IP and port</strong>, Instance 1 recognizes it as an incoming request.</p><p>The OS says, “<em>I was expecting a response from </em><strong><em>10.0.0.139:80</em></strong><em>, but instead, I got one from </em><strong><em>10.0.0.140:39648</em></strong><em>. I didn’t expect this to happen.”</em> In other words, the connection is never established and eventually times out.</p><h3><strong>How to fix this problem?</strong></h3><p><strong>1) Disable Client IP preservation</strong></p><p>The easiest way to fix it is to turn off client IP preservation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gks2jUNUq8LRuWgEzDgMbA.png" /></figure><p>Disable it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lWFCP_7Pdy-oVI6qchvekA.png" /></figure><p><strong>2) Keep Client IP preservation on and enable NAT loopback/hairpinning</strong></p><p>Sometimes, the client’s IP is an absolute need, and you want to keep the client’s IP preserved.</p><p>If the source and destination IP addresses are the same or belong to the same subnet, NAT loopback redirects the traffic before it reaches the load balancer:</p><pre>iptables -t nat -A OUTPUT -p tcp --source 10.0.0.140 --dport 80 -d 10.0.0.139 -j DNAT --to-destination 10.0.0.140:80</pre><p><strong>tcpdump on Instance 1</strong> (when Instance 1 attempts to send a request to the load balancer):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TGKuU9cAELRfsUnPK5Z5Jw.png" /></figure><p>The request was looped back thanks to the NAT rule.</p><p><strong>tcpdump on instance 1</strong> (when the lambda function sends a request to the load balancer):</p><p>The source IP address of the lambda function can be observed in Instance 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYfC9IruEe-Ne1cntLE8jQ.png" /></figure><h3><strong>But I use Docker!</strong></h3><p>Even though Docker does not encourage modifying iptables rules, you may want to use this rule to fix the issue (use at your own risk):</p><pre>sysctl -w net.ipv4.ip_forward=1<br>iptables -I PREROUTING -t nat -p tcp --dport 80 -d 10.0.0.139 -j DNAT --to-destination 10.0.0.140:80</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=04201825534d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to fix conntrack: table full, dropping packets in Kubernetes?]]></title>
            <link>https://adil.medium.com/how-to-fix-conntrack-table-full-dropping-packets-in-kubernetes-07f561a432aa?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/07f561a432aa</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[conntrack]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sat, 30 Mar 2024 23:07:37 GMT</pubDate>
            <atom:updated>2024-11-03T14:31:14.873Z</atom:updated>
            <content:encoded><![CDATA[<p>The Kubernetes network stack has many levels.</p><p><a href="https://www.linkedin.com/posts/adililhan_kubernetes-cloud-linux-activity-7180806256612245504-maEE"><strong>You can follow me on LinkedIn</strong></a></p><p>One of the most important layers is <strong>conntrack</strong>.</p><h3>Table of contents:</h3><p>· <a href="#2f9a">What is conntrack?</a><br>· <a href="#daa4">What is the conntrack limit?</a><br>· <a href="#58da">Why does conntrack get full?</a><br>· <a href="#4484">In Kubernetes, DNS queries are not so simple</a><br>· <a href="#273c">How to fix this issue?</a><br>· <a href="#e234">Solution 1: Add Ram</a><br>· <a href="#5773">Solution 2: Put a dot</a><br>· <a href="#6252">Solution 3: Modify the nf_conntrack_max value via DaemonSet</a><br>· <a href="#c947">Solution 4: Reduce the number of DNS connections</a><br>· <a href="#b7a1">Solution 5: Send sequential DNS requests</a><br>· <a href="#4d43">Solution 6: UDP is fragile; consider TCP</a><br>· <a href="#ecbe">Solution 7: Use NodeLocal DNS Cache</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s_XlYrkRpQ_lcPU0" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>You may have seen this error log on your system:</p><pre>nf_conntrack: table full, dropping packets</pre><h3><strong>What is conntrack?</strong></h3><p><strong>conntrack</strong> stands for “<strong>Connection Tracking</strong>”. conntrack is a component of the Netfilter framework in Linux Kernel. conntrack functions as a database for network connections that travel through a Linux server. It tracks the state of each network connection (TCP, UDP, ICMP, etc).</p><p>The state of a network connection is critical for packet filtering and NAT rules in Kubernetes.</p><h3><strong>What is the conntrack limit?</strong></h3><p>Conntrack has a maximum limit to maintain active network traffic. The Linux Kernel automatically sets the maximum value depending on the Linux Server’s RAM (<a href="https://github.com/torvalds/linux/blob/v6.8/net/netfilter/nf_conntrack_core.c#L2673-L2694">source code</a>).</p><p>However, each Linux distribution or cloud provider may assign a different value.</p><p>Enter your Kubernetes node and get your k8s node’s limit:</p><pre>cat /proc/sys/net/netfilter/nf_conntrack_max</pre><p>Get the current number of entries in conntrack:</p><pre>cat /proc/sys/net/netfilter/nf_conntrack_count</pre><p>In your monitoring system, you may create an alarm like this:</p><pre>(&lt;nf_conntrack_count&gt;/&lt;nf_conntrack_max&gt;) * 100 &gt;= 95</pre><p><em>Alert: 95% of the maximum conntrack limit is reached.</em></p><h3><strong>Why does </strong>conntrack<strong> get full?</strong></h3><p>By default, <strong>conntrack</strong> retains an entry for a <strong>TCP</strong> connection for <strong>120</strong> seconds and an entry for a UDP connection for <strong>30</strong> seconds. If no action occurs in the flow, the entry is immediately removed from <strong>conntrack</strong>.</p><p>As a result, short-lived connections are bad for <strong>conntrack</strong>. As an example, a simple DNS request will be retained in conntrack’s memory for a duration of <strong>30</strong> seconds.</p><h3>In Kubernetes, DNS queries are not so simple</h3><p>Prior to reviewing DNS requests in Kubernetes, it is good to analyze the /etc/resolv.conffile located within a Kubernetes pod (<em>My test pod runs on EKS</em>).</p><pre>root@web-debug:~# cat /etc/resolv.conf<br>search default.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal<br>nameserver 10.100.0.10<br>options ndots:5</pre><p>If the domain name has less than <strong>5 dots</strong>, it will be passed through all domain names in the search field (<em>default.svc.cluster.local</em>, <em>svc.cluster.local, </em>etc).</p><p>If the Kubernetes DNS server cannot find any DNS records, the DNS request will be directed to an external DNS server.</p><p>Does this seem complicated? I’ll enable CoreDNS <a href="https://coredns.io/plugins/log/">logs</a> to check what’s going on.</p><p>I’ll execute a simple curl command:</p><pre>root@web-debug:~# curl google.com<br>&lt;HTML&gt;&lt;HEAD&gt;&lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html;charset=utf-8&quot;&gt;<br>...</pre><p>CoreDNS logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFbrM7i7AkYRucUWoyDmgQ.png" /></figure><p>What on earth is going on in there!?</p><p>Google.com searched over many domain zones (<em>default.svc.cluster.local</em>, <em>svc.cluster.local</em>, etc.).</p><p>CoreDNS couldn’t identify any DNS entries in those zones, thus it returned NXDOMAIN (non-existent domain). Finally, the DNS request was routed to an external DNS server, which responded successfully (NOERROR).</p><p><a href="https://github.com/bminor/glibc/blob/glibc-2.39/nss/getaddrinfo.c#L643">By default, glibc’s getaddrinfo function sends out concurrent A and AAAA requests.</a></p><p>By default, CoreDNS returns 30 seconds for TTL. So DNS queries will be cached in the client for 30 seconds.</p><p>Assume you are operating a service that repeatedly calls internal/external services/domains. <strong>Boom</strong>! <strong>Conntrack</strong>’s capacity will run out.</p><p>DNS connections stored in the conntrack memory on the Kubernetes node:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaUClC4KeX8wak8NleHWiA.png" /></figure><h3><strong>How to fix this issue?</strong></h3><p>Either you increase the conntrack limit or you make fewer DNS queries.</p><h3><strong>Solution 1: Add Ram</strong></h3><p><strong>Quick fix: add extra RAM to the Kubernetes node or raise the instance type if you are in the cloud.</strong></p><p>My t2.medium (4 GiB Memory) Kubernetes node has the nf_conntrack_max value of 131072.</p><p>My t3.xlarge (16 GiB Memory) Kubernetes node has the nf_conntrack_max value of 262144</p><h3><strong>Solution 2: Put a dot</strong></h3><p>If the domain ends with a dot, DNS clients will not check for it in the search field.</p><pre>curl google.com.</pre><p>CoreDNS logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkvlRrv3TiNPXZsU8ZySow.png" /></figure><h3><strong>Solution 3: Modify the nf_conntrack_max value via DaemonSet</strong></h3><p>It is possible to increase the Kubernetes node’s <strong>nf_conntrack_max</strong> value via DaemonSet:</p><pre>---<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: nf-conntrack-fix<br>spec:<br>  selector:<br>    matchLabels:<br>      app: nf-conntrack-fix<br>  template:<br>    metadata:<br>      labels:<br>        app: nf-conntrack-fix<br>    spec:<br>      hostNetwork: true<br>      hostPID: true<br>      initContainers:<br>      - name: dependency-install<br>        command: [&quot;/bin/sh&quot;]<br>        args: [&quot;-c&quot;, &quot;nsenter --mount=/proc/1/ns/mnt -- sh -c &#39;sysctl -w net.nf_conntrack_max=140000&#39;&quot;]<br>        image: alpine:latest<br>        securityContext:<br>          privileged: true<br>      containers:<br>      - name: pause<br>        image: public.ecr.aws/eks-distro/kubernetes/pause:3.8</pre><h3><strong>Solution 4: Reduce the number of DNS connections</strong></h3><p>Let’s assume you call an external API with this domain: api.site.com</p><p>If you lower the number of dots to two, DNS requests should not hit the search field.</p><p>Example yaml:</p><pre>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    app: web-debug<br>  name: web-debug<br>spec:<br>  containers:<br>    - image: ailhan/web-debug<br>      name: web-debug<br>      imagePullPolicy: Always<br>  dnsConfig:<br>    options:<br>      - name: ndots<br>        value: &quot;2&quot;</pre><p>CoreDNS logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z4pmb3YACZ0xXGKvLuHpdw.png" /></figure><h3><strong>Solution 5: </strong>Send sequential DNS requests</h3><p>By default, DNS clients do queries for <strong>A</strong> and <strong>AAAA</strong> records using the same socket. (Please note the IP column in the CoreDNS logs). Some clients, servers, or hardware are unable to support concurrent requests. This is why DNS clients continue to query the same domain.</p><p>It is good to send DNS queries through different source ports:</p><pre>spec.template.spec:<br>  dnsConfig:<br>        options:<br>        - name: single-request-reopen</pre><p>Yaml:</p><pre>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    app: web-debug<br>  name: web-debug<br>spec:<br>  containers:<br>    - image: ailhan/web-debug<br>      name: web-debug<br>      imagePullPolicy: Always<br>  dnsConfig:<br>    options:<br>      - name: single-request-reopen</pre><p>/etc/resolv.conf should look as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ERIzif6DGeNT8HAFwoFLLA.png" /></figure><p>CoreDNS Logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u5b73UHDEhxJhvw3ybi8oQ.png" /></figure><h3>Solution 6: UDP is fragile; consider TCP</h3><p>UDP connections are likely to be lost. Thereby, DNS clients continue to query the domain.</p><p>YAML</p><pre>---<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  labels:<br>    app: web-debug<br>  name: web-debug<br>spec:<br>  containers:<br>    - image: ailhan/web-debug<br>      name: web-debug<br>      imagePullPolicy: Always<br>  dnsConfig:<br>    options:<br>      - name: use-vc</pre><p>CoreDNS logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77NsgwtU27UGm8MjFwwlTA.png" /></figure><h3>Solution 7: Use NodeLocal DNS Cache</h3><p><a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/">Quote from Kubernetes.io:</a></p><pre>NodeLocal DNSCache improves Cluster DNS performance by running a DNS caching agent on cluster nodes as a DaemonSet.</pre><p><a href="https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/#configuration">The DNS caching agent is very simple to install.</a></p><p><a href="https://github.com/aws/containers-roadmap/issues/303#issuecomment-512229440">If you are using EKS, you can get variable values from here.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=07f561a432aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Develop a Firewall in C as a Linux Kernel Module?]]></title>
            <link>https://adil.medium.com/how-to-develop-a-firewall-in-c-as-a-linux-kernel-module-46899fa7d1e6?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/46899fa7d1e6</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[netfilter]]></category>
            <category><![CDATA[linux-kernel-module]]></category>
            <category><![CDATA[c-programming]]></category>
            <category><![CDATA[linux-kernel]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 22:08:48 GMT</pubDate>
            <atom:updated>2024-03-17T22:08:48.756Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://adil.medium.com/understanding-network-packets-ip-header-udp-header-a672398553d0"><strong>Part 1: Understanding Network Packets: IP Header, UDP Header</strong></a><strong><br></strong><a href="https://adil.medium.com/how-to-use-iptables-to-block-a-specific-dns-request-2dfb5ca7340b"><strong>Part 2: How To Use IPTables to Block a Specific DNS Request?</strong></a></p><p>iptables, nftables, and ipvs, which are firewalls/load balancers on Linux, are being built on top of Netfilter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nUVmCSMdcm41w8Oj" /><figcaption>Photo by <a href="https://unsplash.com/@feliixlam?utm_source=medium&amp;utm_medium=referral">Félix Lam</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Can we develop our own firewall?</strong></h3><p>It is not that easy, but yes.</p><p>I have <strong>seven</strong> distinct modules created. The modules will assist you in creating a basic Linux firewall.</p><p>The Git repository will teach you how to use the <strong>Netfilter</strong> framework to filter network packages, extract valuable information, and allow or block them.</p><p>I added the kernel modules to my GitHub account: <a href="https://github.com/adililhan/Firewall-as-a-Linux-Kernel-Module"><strong>Firewall as a Linux-Kernel Module</strong></a></p><p>The modules&#39; complexity is progressively raised. It is highly recommended that you proceed through the modules in order of number.</p><p><a href="https://asciinema.org/a/647411">This is the video where you can learn how to build and deploy a module.</a></p><h3><strong>Part 1 &amp; Part 2</strong></h3><p>It is strongly advised that you read the <a href="https://adil.medium.com/understanding-network-packets-ip-header-udp-header-a672398553d0">first</a> and <a href="https://adil.medium.com/how-to-use-iptables-to-block-a-specific-dns-request-2dfb5ca7340b">second</a> articles before proceeding to the modules.</p><p><a href="https://github.com/adililhan/Firewall-as-a-Linux-Kernel-Module">Firewall-as-a-Linux-Kernel-Module</a></p><p>Enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46899fa7d1e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding SPF and CNAME Records for Emails]]></title>
            <link>https://adil.medium.com/understanding-spf-and-cname-records-for-emails-93eeadf8c1da?source=rss-fdac53119065------2</link>
            <guid isPermaLink="false">https://medium.com/p/93eeadf8c1da</guid>
            <category><![CDATA[gmail]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[outlook]]></category>
            <category><![CDATA[dns-servers]]></category>
            <dc:creator><![CDATA[adil]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 23:24:45 GMT</pubDate>
            <atom:updated>2024-03-13T08:31:21.206Z</atom:updated>
            <content:encoded><![CDATA[<p>Sending email from a subdomain can be difficult. This can be more difficult if there is a CNAME on the subdomain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zp3-2UmFIOB6242p" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I created a domain: blog.adililhan.com</p><p>I will send an email to my Gmail and Outlook accounts from <strong><em>info@blog.adililhan.com</em></strong></p><h3><strong>Scenario 1: Primary Domain has SPF. No SPF on Subdomain</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*oUNTohz7US6eP2HUBA4rgg.png" /></figure><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2Mavu3syRR5JTEQ3eHFag.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*NKBWI9K0_oB6htdP4GgAEw.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*uRx8l0-cZ0qLSlKjEPfr8Q.png" /></figure><h3>Scenario 2: No SPF in the primary domain. Subdomain Has SPF</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*3FvsvwboM4oHIaWyTOoFwA.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*s7G3eH88I9iXhldF6dZSdg.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*YlwA3Yctv6aXkqFhKxPNJg.png" /></figure><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6bnjPxF4uguNdAokqs6sQ.png" /></figure><h3><strong>Scenario 3: No SPF in the primary domain. No SPF in the subdomain</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*7PJi8zNzOJ4aGkzad_wx7g.png" /></figure><p>SPF Tool</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ayaWPZNKdrxRAV_74Oneyg.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*-cSmzPze26pxggbBK8J6Ow.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*3K1zPkh5M0Bi0r280yO9yQ.png" /></figure><h3>Scenario 4: Block all domains in the primary domain. The subdomain has the correct SPF Record</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*9l_KNKXOruqab2UITLjUlw.png" /></figure><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*dyV60tektkDAoj4EJOBENw.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*KgrAH9dRpsmQgXKSG3Uy2g.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*dowLX1_Mc-nyP3B50LQMng.png" /></figure><h3>Scenario 5: Block all domains in the primary domain. No SPF in the subdomain</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*zTBXmpglJmeJNH6TnX9y8Q.png" /></figure><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eKzdIlGQ58u9x5F99hUi1g.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*91PatTDRyDHlWlFe6Mxv0w.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*rTHUTPAH7SUyE6_0pUN5dg.png" /></figure><p><strong>We’ve made sure that you need an SPF record on the subdomain so that you can send emails from it.</strong></p><h3>CNAME and SPF Record</h3><p>A <strong>CNAME</strong> record <strong>can’t coexist</strong> with any other DNS type.</p><p>I will attempt to create a CNAME record for <em>blog.adililhan.com:</em></p><pre>blog.adililhan.com -&gt; CNAME -&gt; anotherblog.adililhan.com</pre><p>The DNS editor I use doesn’t allow me to add the CNAME record:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*Nu3cEq77x2b5ZmWWMGCyng.png" /></figure><p>Because I already have a DNS record for blog.adililhan.com :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*pxRmvPsO3v8ZHkgE3l1e5Q.png" /></figure><p>I will remove those DNS records and try to add a CNAME record.</p><p>I managed to add the CNAME record:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*TY5FBqWFyjGqVULLdnpeWA.png" /></figure><p>I purposely created a CNAME chain:</p><pre>blog.adililhan.com-&gt; CNAME anotherblog.adililhan.com-&gt; CNAME blog123.adililhan.com</pre><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLbpsG7WrGu7Cv5ZS39xjA.png" /></figure><p>If your domain has a CNAME, you need to add your SPF records to the last domain in the chain .The last domain is blog123.adililhan.com</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*kQJI4gndkDKHC-LFL1FVRA.png" /></figure><p>SPF Tool:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAbD2XZlnS7tZxYYsVbNcw.png" /></figure><p>Outlook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*pjkcKzeWPaBvpwWeHgmNrQ.png" /></figure><p>Gmail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*jBkyl7ubc2J8TO6CjNHtnA.png" /></figure><h3><strong>Return-Path and SPF</strong></h3><p>I sent emails using Mailgun. I cannot modify the <strong>Return-Path</strong> in Mailgun. <strong>It is important to remember that the domain in the <em>Return-Path</em> field—rather than the domain in the <em>From</em> field—will be used for the SPF check.</strong></p><pre>From: info@blog.adililhan.com<br>Return-Path: admin@blog.xyz.com</pre><p>blog.xyz.com must have the correct SPF records.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93eeadf8c1da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>