<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dinesh Kumar Ramasamy on Medium]]></title>
        <description><![CDATA[Stories by Dinesh Kumar Ramasamy on Medium]]></description>
        <link>https://medium.com/@dramasamy?source=rss-79ada7cd5ae5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-zzpv0tSisJTN9KR3CWQnQ.jpeg</url>
            <title>Stories by Dinesh Kumar Ramasamy on Medium</title>
            <link>https://medium.com/@dramasamy?source=rss-79ada7cd5ae5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 04:06:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dramasamy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Life of a Packet in ISTIO — Control Plane— Part 3]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-istio-control-plane-part-3-899e565da4d2?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/899e565da4d2</guid>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <category><![CDATA[microservicesarchitecture]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[service-mesh]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Mon, 25 Mar 2024 01:12:03 GMT</pubDate>
            <atom:updated>2024-03-25T01:16:45.875Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in ISTIO — Control Plane— Part 3</h3><p><a href="https://dramasamy.medium.com/life-of-a-packet-in-istio-envoy-proxy-part-2-27950e820b04?source=friends_link&amp;sk=fdeab38bcabd0a7e5594cf14a8471bab">In Part 2 </a>of the “Life of a Packet in ISTIO” series, we explored the Envoy proxy configuration and its significance in managing inbound and outbound traffic within the Istio service mesh. Envoy, a CNCF project developed by Lyft, operates as a sidecar proxy alongside each service instance, intercepting and directing traffic between services. We discussed Envoy’s essential features, such as load balancing, traffic routing, and security, which are crucial for Istio’s functionality.</p><p>The article provided a detailed overview of the processing steps that a packet undergoes when it reaches an Envoy proxy, including ingress, network filtering, HTTP stream processing, load balancing, and egress. A hands-on demo was presented to demonstrate how to configure Envoy as a front proxy using YAML configuration, enabling round-robin load balancing and securing communication with TLS authentication.</p><p>Furthermore, we explored tracing in Envoy, which helps developers understand communication between services in a distributed application. A hands-on demo illustrated how to enable Zipkin tracing in Envoy and visualize traces using Jaeger. The article also introduced Envoy XDS, a core component of Istio, which dynamically configures Envoy sidecars with service mesh configuration. A hands-on demo showcased how to use Envoy XDS by starting the proxy container and upstream HTTP echo servers.</p><p>Overall, Part 2 provided a comprehensive understanding of Envoy’s role in Istio’s traffic management, tracing capabilities, and dynamic configuration updates. In Part 3, we will delve further into other critical aspects of Istio’s architecture.</p><p>In this article, we will delve deeper into another critical component of Istio’s architecture: Pilot. We will explore Pilot’s role in Istio’s control plane, including service discovery, traffic management, and configuration distribution. We will also discuss how Pilot generates a detailed service topology map and ensures that the Istio mesh can handle service-to-service communication efficiently, reliably, and responsively.</p><blockquote>As you read this article, you will come across the term “<strong>Pilot</strong>” frequently when discussing service discovery, traffic management, and configuration distribution within the Istio service mesh. It’s worth noting that if you’re familiar with Istio’s architecture in versions prior to 1.5, you might know about Pilot, Galley, Citadel, and Mixer as separate control plane components, each handling different aspects of the service mesh. However, starting with Istio 1.5, these components have been combined into a single binary and deployment called <em>istiod</em>.</blockquote><h3>Pilot’s Role in ISTIO’s Control Plane</h3><p>Pilot is a crucial component of the Istio service mesh. It is responsible for converting high-level configuration and service discovery information into specific configuration objects that Envoy can understand, such as listeners, clusters, and endpoints. Whenever a new service is deployed or updated within the Kubernetes cluster, Pilot automatically updates the Envoy configuration to reflect these changes. This involves creating or modifying listeners to intercept traffic, updating clusters to define how traffic is managed, and maintaining an up-to-date list of healthy endpoints for traffic routing. This dynamic configuration ensures that the Istio mesh can handle service-to-service communication efficiently, reliably, and responsively.</p><p><strong>Service Registration</strong></p><ul><li>Pilot stays up-to-date with the Kubernetes API server by subscribing to updates. Whenever there are changes to services or endpoints (such as when a new service is added or an existing service is modified), Pilot is notified immediately.</li><li>These notifications are received through the Kubernetes event stream, which Pilot listens to as part of its regular operation. This guarantees that Pilot always has the most recent information about the status of services within the Kubernetes cluster.</li></ul><p><strong>Discovery</strong></p><p>With the information received from the Kubernetes API server, Pilot performs several critical functions:</p><ul><li>When Pilot receives updates from the Kubernetes API server, it creates an internal service registry that contains information about all services, endpoints, and their health statuses.</li><li>Pilot obtains this data from Kubernetes Custom Resource Definitions (CRDs), such as ServiceEntries and Endpoints.</li><li>Additionally, Pilot collects metadata and health check information for each service instance, which is also stored in the registry.</li></ul><p><strong>Topology Generation</strong></p><p>Using the data gathered from Kubernetes CRDs, Pilot creates a detailed service topology map. This map is essential for several reasons.</p><ul><li><strong>Dynamic Representation: </strong>The topology map dynamically illustrates how services are related and interact with each other within the mesh. This includes their endpoints, dependencies, and the policies that affect their communication.</li><li><strong>Routing and Policies: </strong>The topology map includes routing rules, load balancing policies, and failover strategies that govern how traffic flows through the mesh and how services should interact with one another.</li></ul><p><strong>Endpoint Selection</strong></p><ul><li>When a request arrives at a sidecar proxy, the proxy uses pre-configured rules and policies to determine where to forward the request. This decision is based on factors such as routing rules (e.g., URI paths, headers), load balancing policies (e.g., round-robin, least connections), and the health status of potential destination endpoints.</li><li>The proxy selects a healthy endpoint from its local cache of the service registry, which is regularly synchronized with Pilot. The selection criteria include the load balancing policy and the health checks of the endpoints.</li><li>Once the destination endpoint is chosen, the sidecar proxy sends the request directly to that endpoint, without routing the request back through Pilot or any other part of Istio’s control plane.</li></ul><p><strong>Health Monitoring</strong></p><ul><li>Istio’s Pilot (or Istiod in newer versions, which combines Pilot with other Istio control plane components) doesn’t perform health checks directly. Instead, it relies on the health status information managed by Kubernetes. Pilot receives periodic updates from Kubernetes about the status of service instances (such as new instances becoming ready or existing instances becoming unavailable).</li><li>For Kubernetes-managed services, Pilot’s service registry is updated based on Kubernetes endpoints, which reflect the results of Kubernetes health checks.</li><li>For Envoy’s active health checks, the health checks are performed between the Envoy proxies themselves, and unhealthy endpoints can be dynamically excluded from load balancing without directly updating Pilot’s service registry. This is more about real-time traffic management than service registry management.</li></ul><p><strong>Dynamic Updates</strong></p><ul><li>As services and endpoints are added, modified, or removed in Kubernetes, Pilot updates the service registry and the service topology map dynamically. This ensures that if a service instance is considered unhealthy and is either removed from Kubernetes endpoints or marked as unhealthy by Envoy health checks, the routing configurations distributed by Pilot will reflect these changes, preventing traffic from being directed to unhealthy instances.</li><li>Pilot distributes these updates to all sidecar proxies within the service mesh, ensuring that they have the most up-to-date information about service endpoints and configurations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J1LvGeNC2-opfOsuNIOuIA.png" /><figcaption>Dynamic Service Discovery and Configuration in Istio</figcaption></figure><h3>Envoy as SideCar</h3><p>Envoy Proxy, a critical component of the Istio service mesh, uses several abstractions to manage traffic: listeners, clusters, and endpoints. These elements play essential roles in how Envoy processes incoming and outgoing requests within the mesh, ensuring efficient and flexible traffic management. Here’s how they relate to the Istio mesh:</p><ul><li><strong>Listeners: </strong>Listeners are configuration objects that tell Envoy where to listen for incoming connections and enable sidecar proxies to intercept and manage traffic entering and leaving the service pods.</li><li><strong>Clusters:</strong> A cluster is a collection of endpoints (service instances) to which Envoy can route traffic, representing services within the mesh. Istio configures Envoy with the necessary clusters based on the services defined in the mesh, along with policies for load balancing, retries, and other traffic management features.</li><li><strong>Endpoints: </strong>Endpoints are the individual instances of a service (e.g., pods in Kubernetes) to which traffic can be routed. They are the actual destinations for network connections managed by Envoy. In an Istio mesh, endpoints correspond to the pods running the components of the microservices. Pilot dynamically updates the Envoy proxies with the latest endpoint information, ensuring traffic is always routed to the appropriate service instance.</li></ul><h3>Step-by-Step Guide to Istio Service Discovery</h3><p>Let’s use a simple example to understand how Istio manages service discovery and configuration distribution. We will use a <em>productpage </em>application and two versions of a <em>reviews </em>application. After deploying these services and exposing them via Kubernetes services, we will use Istio to manage traffic between them. This setup will demonstrate how Istio communicates service discovery information to Envoy sidecars and dynamically updates the configuration based on changes within the cluster.</p><h4>Prerequisites</h4><ul><li>Istio Installed: Ensure Istio is installed in your Kubernetes cluster. If not, follow Istio’s installation instructions. <br>- <a href="https://kind.sigs.k8s.io/docs/user/quick-start/">kind — Quick Start (k8s.io)</a><br>- <a href="https://istio.io/latest/docs/setup/install/istioctl/">Istio / Install with Istioctl</a></li><li>Namespace Prepared: Confirm automatic sidecar injection is enabled in your working namespace (e.g., default). If necessary, enable it by running:</li></ul><pre>kubectl label namespace default istio-injection=enabled</pre><h4>Step 1: Deploy the Services</h4><p>Create the following deployments and services in Kubernetes. We are using simplified versions of the BookInfo application’s productpage and reviews services for clarity.</p><h4>ProductPage Deployment and Service</h4><ul><li>Save the following YAML as productpage.yaml and apply it with kubectl apply -f productpage.yaml:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96d58b738235a30adb4ea24f8d7a10a6/href">https://medium.com/media/96d58b738235a30adb4ea24f8d7a10a6/href</a></iframe><h4>Reviews v1 and v2 Deployments and Service</h4><ul><li>Save the following YAML as reviews.yaml and apply it with kubectl apply -f reviews.yaml</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff8691721d596f2195ffb891861a190a/href">https://medium.com/media/ff8691721d596f2195ffb891861a190a/href</a></iframe><p>Once the services are deployed, and sidecars are injected, let’s explore the initial Envoy configuration.</p><p><strong>Check Clusters:</strong></p><blockquote>A cluster is a group of logically similar upstream hosts that Envoy proxy connects to.</blockquote><pre>$ POD_NAME=$(kubectl get pod -l app=productpage -n default -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br>$ istioctl proxy-config clusters $POD_NAME --namespace default<br><br>SERVICE FQDN                                            PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE<br>                                                        9080      -          inbound       ORIGINAL_DST     <br>BlackHoleCluster                                        -         -          -             STATIC           <br>InboundPassthroughClusterIpv4                           -         -          -             ORIGINAL_DST     <br>PassthroughCluster                                      -         -          -             ORIGINAL_DST     <br>agent                                                   -         -          -             STATIC           <br>istio-egressgateway.istio-system.svc.cluster.local      80        -          outbound      EDS              <br>istio-egressgateway.istio-system.svc.cluster.local      443       -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     80        -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     443       -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     15021     -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     15443     -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     31400     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   443       -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15010     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15012     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15014     -          outbound      EDS              <br>kube-dns.kube-system.svc.cluster.local                  53        -          outbound      EDS              <br>kube-dns.kube-system.svc.cluster.local                  9153      -          outbound      EDS              <br>kubernetes.default.svc.cluster.local                    443       -          outbound      EDS              <br>productpage.default.svc.cluster.local                   9080      -          outbound      EDS              <br>prometheus_stats                                        -         -          -             STATIC           <br>reviews.default.svc.cluster.local                       9080      -          outbound      EDS              <br>sds-grpc                                                -         -          -             STATIC           <br>xds-grpc                                                -         -          -             STATIC           <br>zipkin                                                  -         -          -             STRICT_DNS  </pre><p><strong>Check Listeners:</strong></p><blockquote>A listener is a named network location (e.g., port, unix domain socket, etc.) that can be connected to by downstream clients. Envoy exposes one or more listeners that downstream hosts connect to.</blockquote><pre>$ istioctl proxy-config listeners $POD_NAME --namespace default<br><br>ADDRESSES     PORT  MATCH                                                                                           DESTINATION<br>10.96.0.10    53    ALL                                                                                             Cluster: outbound|53||kube-dns.kube-system.svc.cluster.local<br>0.0.0.0       80    Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 80<br>0.0.0.0       80    ALL                                                                                             PassthroughCluster<br>10.96.0.1     443   ALL                                                                                             Cluster: outbound|443||kubernetes.default.svc.cluster.local<br>10.96.160.151 443   ALL                                                                                             Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local<br>10.96.63.230  443   ALL                                                                                             Cluster: outbound|443||istio-egressgateway.istio-system.svc.cluster.local<br>10.96.85.208  443   ALL                                                                                             Cluster: outbound|443||istiod.istio-system.svc.cluster.local<br>0.0.0.0       9080  Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 9080<br>0.0.0.0       9080  ALL                                                                                             PassthroughCluster<br>10.96.0.10    9153  Trans: raw_buffer; App: http/1.1,h2c                                                            Route: kube-dns.kube-system.svc.cluster.local:9153<br>10.96.0.10    9153  ALL                                                                                             Cluster: outbound|9153||kube-dns.kube-system.svc.cluster.local<br>0.0.0.0       15001 ALL                                                                                             PassthroughCluster<br>0.0.0.0       15001 Addr: *:15001                                                                                   Non-HTTP/Non-TCP<br>0.0.0.0       15006 Addr: *:15006                                                                                   Non-HTTP/Non-TCP<br>0.0.0.0       15006 Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0                        InboundPassthroughClusterIpv4<br>0.0.0.0       15006 Trans: raw_buffer; App: http/1.1,h2c; Addr: 0.0.0.0/0                                           InboundPassthroughClusterIpv4<br>0.0.0.0       15006 Trans: tls; App: TCP TLS; Addr: 0.0.0.0/0                                                       InboundPassthroughClusterIpv4<br>0.0.0.0       15006 Trans: raw_buffer; Addr: 0.0.0.0/0                                                              InboundPassthroughClusterIpv4<br>0.0.0.0       15006 Trans: tls; Addr: 0.0.0.0/0                                                                     InboundPassthroughClusterIpv4<br>0.0.0.0       15006 Trans: tls; App: istio,istio-peer-exchange,istio-http/1.0,istio-http/1.1,istio-h2; Addr: *:9080 Cluster: inbound|9080||<br>0.0.0.0       15006 Trans: raw_buffer; Addr: *:9080                                                                 Cluster: inbound|9080||<br>0.0.0.0       15010 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 15010<br>0.0.0.0       15010 ALL                                                                                             PassthroughCluster<br>10.96.85.208  15012 ALL                                                                                             Cluster: outbound|15012||istiod.istio-system.svc.cluster.local<br>0.0.0.0       15014 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: 15014<br>0.0.0.0       15014 ALL                                                                                             PassthroughCluster<br>0.0.0.0       15021 ALL                                                                                             Inline Route: /healthz/ready*<br>10.96.160.151 15021 Trans: raw_buffer; App: http/1.1,h2c                                                            Route: istio-ingressgateway.istio-system.svc.cluster.local:15021<br>10.96.160.151 15021 ALL                                                                                             Cluster: outbound|15021||istio-ingressgateway.istio-system.svc.cluster.local<br>0.0.0.0       15090 ALL                                                                                             Inline Route: /stats/prometheus*<br>10.96.160.151 15443 ALL                                                                                             Cluster: outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local<br>10.96.160.151 31400 ALL                                                                                             Cluster: outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local</pre><p><strong>Check Routes:</strong></p><pre>$ istioctl proxy-config routes $POD_NAME --namespace default<br><br>NAME                                                          VHOST NAME                                                    DOMAINS                                              MATCH                  VIRTUAL SERVICE<br>15010                                                         istiod.istio-system.svc.cluster.local:15010                   istiod.istio-system, 10.96.85.208                    /*                     <br>15014                                                         istiod.istio-system.svc.cluster.local:15014                   istiod.istio-system, 10.96.85.208                    /*                     <br>istio-ingressgateway.istio-system.svc.cluster.local:15021     istio-ingressgateway.istio-system.svc.cluster.local:15021     *                                                    /*                     <br>80                                                            istio-egressgateway.istio-system.svc.cluster.local:80         istio-egressgateway.istio-system, 10.96.63.230       /*                     <br>80                                                            istio-ingressgateway.istio-system.svc.cluster.local:80        istio-ingressgateway.istio-system, 10.96.160.151     /*                     <br>kube-dns.kube-system.svc.cluster.local:9153                   kube-dns.kube-system.svc.cluster.local:9153                   *                                                    /*                     <br>9080                                                          productpage.default.svc.cluster.local:9080                    productpage, productpage.default + 1 more...         /*                     <br>9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     <br>InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     <br>inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     <br>inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     <br>                                                              backend                                                       *                                                    /healthz/ready*        <br>InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     <br>                                                              backend                                                       *                                                    /stats/prometheus*   </pre><p>Initially, you might not see custom routes until we define VirtualServices. These commands give you a snapshot of how traffic is configured to flow between the productpage and the reviews services, as seen by the productpage&#39;s sidecar proxy.</p><p>To understand the flow of egress traffic from the productpage service to the reviews service within an Istio service mesh, it&#39;s crucial to grasp how Istio&#39;s control plane (specifically istiod in versions 1.5 and later, which subsumes the functionality previously attributed to Pilot) uses routes and endpoints to direct traffic. This process is central to enabling the dynamic, resilient, and secure service-to-service communication that Istio is known for. Here&#39;s an end-to-end explanation, simplifying the concepts of routes and endpoints as per your request:</p><h3>Overview of the Flow</h3><p>When a request is made from the productpage service to the reviews service within an Istio-enabled Kubernetes cluster, several steps are involved in routing the request correctly and ensuring that it reaches a healthy instance of the reviews service. This process leverages the configurations defined in VirtualServices, DestinationRules, and the dynamic service discovery provided by istiod.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*HTStO76UNJwYVr9PY-s6dQ.png" /></figure><h3>Routes</h3><p>Routes in Istio are defined through VirtualServices. They tell the Envoy sidecar proxies how to route traffic to different services based on rules like URI paths, HTTP headers, or even weights for A/B testing or canary rollouts. In the istioctl proxy-config routes output:</p><ul><li>You see entries like 9080 for both productpage.default.svc.cluster.local and reviews.default.svc.cluster.local. These entries represent routing rules that apply to traffic directed to these services on port 9080.</li><li>The “VHOST NAME” and “DOMAINS” columns indicate the virtual host within the proxy that handles this traffic, essentially grouping routing rules.</li><li>The “MATCH” and “VIRTUAL SERVICE” columns would show specific conditions and the associated VirtualService resource that defines the routing behavior (if custom routing rules were applied).</li></ul><h3>Endpoints</h3><p>Endpoints are specific instances (pods) of a service that are healthy and available to handle requests. istiod communicates with Envoy sidecars to provide them with up-to-date lists of endpoints for each service, using the Endpoint Discovery Service (EDS). In the istioctl proxy-config endpoints output:</p><ul><li>The “ENDPOINT” column shows the IP addresses and ports of service instances that Envoy can route traffic to. For instance, entries under outbound|9080||reviews.default.svc.cluster.local represent the IPs and ports of the reviews service pods.</li><li>The “STATUS” indicates the health status of each endpoint, ensuring that traffic is only routed to healthy instances.</li><li>The “CLUSTER” refers to the logical grouping of these endpoints as defined by Istio configurations (e.g., reviews.default.svc.cluster.local).</li></ul><h3>The Egress Traffic Flow from productpage to reviews</h3><ol><li>Request Initiation: A user or an application makes a request to the productpage service, which in turn needs to retrieve data from the reviews service.</li><li>Envoy Sidecar Interception: The request from productpage is intercepted by its Envoy sidecar proxy, based on the predefined routing rules (as seen in the routes output).</li><li>Route Evaluation: The Envoy proxy evaluates the request against the routing rules defined in the associated VirtualService (e.g., headers, paths) and selects the appropriate route for the reviews service.</li><li>Endpoint Selection: Based on the selected route and the load balancing policies defined in DestinationRule, the Envoy proxy consults the list of healthy endpoints for the reviews service (as updated by istiod).</li><li>Forwarding: The Envoy proxy forwards the request to one of the healthy reviews service instances, based on the load balancing policy and the health statuses of the endpoints.</li><li>Response Path: The response from the reviews service travels back through its Envoy sidecar, to the productpage&#39;s Envoy sidecar, and finally to the original requester.</li></ol><p>This flow demonstrates Istio’s capability to dynamically manage service-to-service communication within the mesh, leveraging real-time updates on service topology and health statuses to ensure efficient and reliable request routing and load balancing.</p><h4>Step 3: Define VirtualService for Reviews</h4><p>Create a VirtualService to control traffic flow to different versions of the reviews service based on HTTP headers.</p><p>This configuration tells Istio to check the incoming request for a header named end-user. If the header is present and its value is exactly tester, the request is routed to reviews-v2. All other traffic is routed to reviews-v1. This allows you to selectively route traffic based on client identity or any other criteria you choose to implement, making it highly useful for testing new features or releases with a subset of your users before rolling them out more broadly.</p><ul><li><strong>Hosts</strong>: Specifies the hostnames that this rule applies to. Matching the reviews service means it affects traffic intended for the reviews service.</li><li><strong>HTTP Route</strong>: Defines routing rules:<br>– The first rule routes traffic to the v2 subset when the end-user header exactly matches tester. This targets traffic to pods labeled with version: v2, as defined in the DestinationRule.<br>– The second route rule, which doesn’t specify a match condition, <strong>acts as the default path</strong>, routing all other traffic to the v1 subset.</li></ul><p>Save the following as reviews-virtualservice.yaml and apply it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dd9f40e4667555ceddfc72a37cdb401/href">https://medium.com/media/5dd9f40e4667555ceddfc72a37cdb401/href</a></iframe><p><strong>Create DestinationRule for Reviews:</strong></p><p>Define a DestinationRule to specify <strong>subsets </strong>corresponding to the two versions of the reviews service. This will allow us to route traffic selectively to each version.</p><ul><li><strong>Host</strong>: Specifies the service this rule applies to. In this case, it’s reviews, which matches the Kubernetes service name.</li><li><strong>Subsets</strong>: Define logical divisions of the reviews service based on labels. Here, two subsets are defined:<br>–v1: Represents pods of the reviews service with the label version: v1.<br>–v2: Represents pods of the reviews service with the label version: v2.</li></ul><p>The purpose of these subsets is to provide identifiable groups of service instances that can be referenced in routing rules.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60fb1e2d47a8fb51865dcaa3cf8143e0/href">https://medium.com/media/60fb1e2d47a8fb51865dcaa3cf8143e0/href</a></iframe><h4>How VS and DS works together?</h4><ol><li><strong>Traffic Routing:</strong> When a request is made to the reviews service, Istio evaluates the <em>VirtualService </em>rules to determine how the traffic should be routed.<br>– If the request includes an end-user header with the value tester, the request is routed to instances of the reviews service labeled with version: v2 because of the match condition in the <em>VirtualService </em>that references the v2 subset defined in the DestinationRule.<br>– All other requests, which don’t meet the specified match condition, fall through to the default route, which directs traffic to instances labeled with version: v1.</li><li><strong>Dynamic Configuration:</strong> This setup allows for dynamic configuration of traffic routing based on request attributes (like headers) without needing to update the application or the service’s deployment configuration. Changes can be made by adjusting the VirtualService or DestinationRule resources.</li><li><strong>Use Cases:</strong> This model is particularly useful for canary rollouts, A/B testing, or any scenario where you want to control traffic distribution to different versions of a service based on specific criteria.</li></ol><p>In summary, <em>DestinationRules </em>define policies and subsets for traffic targets within the mesh, while <em>VirtualServices </em>specify how traffic is routed to these targets. Working together, they enable fine-grained control over traffic routing, allowing for sophisticated traffic management strategies.</p><h4>Demonstrating Dynamic Configuration with Pilot</h4><p>Now that we’ve applied a more complex routing rule with our VirtualService, Pilot will distribute this configuration to the Envoy sidecars of the pods that match the hosts field, in this case, the reviews service.</p><p>You can verify the updated Envoy configurations to reflect this new routing logic by repeating the inspection commands:</p><ol><li>Review Cluster Configuration: Check the clusters again to ensure they include both subsets (v1 and v2).</li></ol><pre>$ POD_NAME=$(kubectl get pod -l app=productpage -n default -o jsonpath=&#39;{.items[0].metadata.name}&#39;)<br>$ istioctl proxy-config clusters $POD_NAME --namespace default<br><br>SERVICE FQDN                                            PORT      SUBSET     DIRECTION     TYPE             DESTINATION RULE<br>                                                        9080      -          inbound       ORIGINAL_DST     <br>BlackHoleCluster                                        -         -          -             STATIC           <br>InboundPassthroughClusterIpv4                           -         -          -             ORIGINAL_DST     <br>PassthroughCluster                                      -         -          -             ORIGINAL_DST     <br>agent                                                   -         -          -             STATIC           <br>istio-egressgateway.istio-system.svc.cluster.local      80        -          outbound      EDS              <br>istio-egressgateway.istio-system.svc.cluster.local      443       -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     80        -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     443       -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     15021     -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     15443     -          outbound      EDS              <br>istio-ingressgateway.istio-system.svc.cluster.local     31400     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   443       -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15010     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15012     -          outbound      EDS              <br>istiod.istio-system.svc.cluster.local                   15014     -          outbound      EDS              <br>kube-dns.kube-system.svc.cluster.local                  53        -          outbound      EDS              <br>kube-dns.kube-system.svc.cluster.local                  9153      -          outbound      EDS              <br>kubernetes.default.svc.cluster.local                    443       -          outbound      EDS              <br>productpage.default.svc.cluster.local                   9080      -          outbound      EDS              <br>prometheus_stats                                        -         -          -             STATIC     <br>reviews.default.svc.cluster.local                       9080      -          outbound      EDS              reviews-destination.default<br>reviews.default.svc.cluster.local                       9080      v1         outbound      EDS              reviews-destination.default<br>reviews.default.svc.cluster.local                       9080      v2         outbound      EDS              reviews-destination.default <br>sds-grpc                                                -         -          -             STATIC           <br>xds-grpc                                                -         -          -             STATIC           <br>zipkin                                                  -         -          -             STRICT_DNS </pre><p><strong>Difference:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H9R7WHNn3OZkfol7PEQ61g.png" /><figcaption>Before and After VirtualService and DestinationRules</figcaption></figure><p>2. Review Route Configuration: This will show the detailed routing rules based on the end-user header.</p><pre>$ istioctl proxy-config routes $POD_NAME --namespace default   <br>NAME                                                          VHOST NAME                                                    DOMAINS                                              MATCH                  VIRTUAL SERVICE<br>15010                                                         istiod.istio-system.svc.cluster.local:15010                   istiod.istio-system, 10.96.85.208                    /*                     <br>15014                                                         istiod.istio-system.svc.cluster.local:15014                   istiod.istio-system, 10.96.85.208                    /*                     <br>istio-ingressgateway.istio-system.svc.cluster.local:15021     istio-ingressgateway.istio-system.svc.cluster.local:15021     *                                                    /*                     <br>80                                                            istio-egressgateway.istio-system.svc.cluster.local:80         istio-egressgateway.istio-system, 10.96.63.230       /*                     <br>80                                                            istio-ingressgateway.istio-system.svc.cluster.local:80        istio-ingressgateway.istio-system, 10.96.160.151     /*                     <br>80                                                            reviews.default.svc.cluster.local:80                          reviews.default.svc.cluster.local                    /*                     reviews.default<br>80                                                            reviews.default.svc.cluster.local:80                          reviews.default.svc.cluster.local                    /*                     reviews.default<br>kube-dns.kube-system.svc.cluster.local:9153                   kube-dns.kube-system.svc.cluster.local:9153                   *                                                    /*                     <br>9080                                                          productpage.default.svc.cluster.local:9080                    productpage, productpage.default + 1 more...         /*                     <br>9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     reviews.default<br>9080                                                          reviews.default.svc.cluster.local:9080                        reviews, reviews.default + 1 more...                 /*                     reviews.default<br>InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     <br>inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     <br>inbound|9080||                                                inbound|http|9080                                             *                                                    /*                     <br>                                                              backend                                                       *                                                    /healthz/ready*        <br>InboundPassthroughClusterIpv4                                 inbound|http|0                                                *                                                    /*                     <br>                                                              backend                                                       *                                                    /stats/prometheus*</pre><p>After adding VirtualServices and DestinationRules in Istio, the configuration and behavior of the Envoy proxies within the service mesh are significantly altered to accommodate the defined routing policies and service subsets. Let’s break down the impact of these changes and understand the end-to-end flow when egress traffic goes from the productpage service to the reviews service.</p><p><strong>Difference:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7zam5ZxZiG6IPS_S713YGg.png" /><figcaption>Before and After VirtualService and DestinationRules</figcaption></figure><h3>Before VirtualServices and DestinationRules</h3><p>Initially, without specific VirtualServices and DestinationRules, traffic routing within the Istio mesh is based on standard Kubernetes service discovery and load balancing. Envoy proxies would route requests to the reviews service without any fine-grained control over which version of the service (e.g., v1 or v2) the traffic should be directed to.</p><h3>After Adding VirtualServices and DestinationRules</h3><ol><li><strong>DestinationRule Impact: </strong>By introducing a DestinationRule for the reviews service, we define subsets based on service versions (v1 and v2). This allows for more granular control over traffic routing to different versions of the reviews service.<br> — <strong>In the </strong><strong>clusters output, you now see clusters for </strong><strong>reviews differentiated by subsets (</strong><strong>v1 and </strong><strong>v2). This indicates that Envoy is aware of these subsets and can route traffic accordingly.</strong></li><li><strong>VirtualService Impact:</strong> A VirtualService specifies how traffic to a service (or from one service to another) is routed. By applying a VirtualService to the reviews service, we can create rules that determine how traffic is directed between the subsets defined in the DestinationRule.<br> — <strong>In the </strong><strong>routes output, you now see routes for the </strong><strong>reviews service indicating how traffic is matched (e.g., based on headers or URIs) and which subset it is routed to. The existence of these routes is directly tied to the rules defined in the </strong><strong>VirtualService.</strong></li></ol><p>If you’re seeing port 80 (<strong><em>If you noticed this, well done!</em></strong>) in the istioctl proxy-config routes output for the reviews service despite not specifying it in your configuration, and after applying a VirtualService that does not mention port 80, it’s to Istio&#39;s handling of unspecified ports in the VirtualService routing definitions. Here&#39;s a deeper look into why this might happen and how Istio processes such configurations:</p><h4>Default Port Handling in Istio</h4><p>When a VirtualService does not explicitly specify a port in the destination of a routing rule, Istio might use default port assignments based on the protocol or service configuration. HTTP services, for instance, commonly use port 80 as a default when no other port is specified. This behavior helps ensure that the traffic routing rules can still apply even if specific port details are omitted, but it can lead to confusion if the actual service uses a different port.</p><h4>How Istio Interprets the Configuration</h4><p>Your VirtualService configuration for the reviews service routes traffic based on the end-user header but does not specify ports in the routing destinations. Given this:</p><ul><li><strong>Implicit Default Port: </strong>In the absence of a specific port in the VirtualService&#39;s route.destination field, Istio might implicitly use a default port (commonly 80 for HTTP) for routing decisions.</li><li><strong>Protocol Inference:</strong> Istio infers the protocol (HTTP in this case) from the VirtualService definition. If the service definition in Kubernetes for reviews lacks a specified port or uses a name implying HTTP (e.g., http), Istio could default to using port 80 for HTTP traffic routing.</li></ul><h4>Addressing the Issue</h4><ol><li><strong>Specify Ports Explicitly: </strong>To ensure that traffic is routed to the correct port, you can explicitly specify the port in your VirtualService routing rules. For instance, if the reviews service is intended to be accessed on port 9080, modify the VirtualService to include the port number:</li><li><strong>Review Service Definitions:</strong> Ensure the Kubernetes Service definition for reviews specifies the correct target port (9080), and it&#39;s consistently used across your configurations.</li><li><strong>Protocol Naming Conventions:</strong> Verify the port naming conventions in your Kubernetes Service definition align with the expected protocol, which can influence Istio&#39;s default behavior.</li><li><strong>Check for Other Influencing Resources:</strong> Look for other Istio resources, such as Gateways or DestinationRules, that might influence traffic routing and port selection, ensuring they are consistent with your intended configuration.</li></ol><p><a href="https://gist.github.com/dramasamy/c1c97276f1c74b94e77c7136529b05ba">vs-with-port.yaml (github.com)</a></p><p>By explicitly defining all aspects of the traffic routing, including ports, in your VirtualService and ensuring consistency across your service mesh configuration, you can control and predict Istio&#39;s traffic routing behavior more accurately, avoiding unexpected defaults and ensuring traffic flows as intended.</p><h3>End-to-End Flow of Egress Traffic from productpage to reviews</h3><ol><li><strong>Initiation:</strong> A request is made to the productpage service, which then needs to fetch data from the reviews service.</li><li><strong>Interception by Envoy: </strong>The request from productpage to reviews is intercepted by the Envoy sidecar proxy in the productpage pod.</li><li><strong>Route Matching: </strong>The Envoy proxy uses the VirtualService configuration to match the request based on specified criteria (e.g., URI, headers). The VirtualService dictates whether the request should go to reviews-v1 or reviews-v2 based on these rules.</li><li><strong>Subset Selection:</strong> Once the route is determined, the Envoy proxy consults the DestinationRule to understand the subsets (versions) available for the reviews service. The DestinationRule helps Envoy make a decision on which subset (e.g., v1 or v2) the request should be routed to.</li><li><strong>Endpoint Discovery: </strong>The Envoy proxy then looks up the healthy endpoints within the chosen subset by querying the service discovery information provided by istiod. This is where the EDS (Endpoint Discovery Service) comes into play, dynamically providing the list of healthy endpoints for reviews-v1 or reviews-v2.</li><li><strong>Request Forwarding: </strong>The Envoy proxy forwards the request to one of the healthy endpoints in the selected subset. This decision also considers load balancing policies defined in the DestinationRule.</li><li><strong>Response Routing:</strong> The response from the reviews service is routed back through the Envoy proxies to the client that initiated the request to the productpage.</li></ol><p>This demonstrates the dynamic nature of traffic routing within an Istio service mesh, enabled by the configuration of <em>VirtualServices </em>and <em>DestinationRules</em>. These Istio resources empower operators to define sophisticated traffic routing and load balancing policies that react to real-time changes in the service landscape.</p><p>This detailed setup, incorporating a <em>DestinationRule </em>for defining subsets and a <em>VirtualService </em>for header-based routing, illustrates how Istio enables sophisticated traffic management within the service mesh. By leveraging these features, you can perform canary releases, A/B testing, and ensure that new features are gradually and safely exposed to your users. Additionally, it demonstrates how Istio’s Pilot component dynamically manages and distributes service discovery and configuration information to Envoy proxies, ensuring the mesh adapts in real-time to configured policies.</p><h3>In the Next Part</h3><p>In the next article of our Istio deep dive series, we will explore Istio’s extensive traffic management capabilities. Building upon our foundational understanding of how Istio’s Pilot component orchestrates service discovery and configuration distribution, we’ll take a closer look at how to harness Istio for advanced traffic routing, resilience patterns like circuit breaking, fault injection, traffic mirroring, and rate limiting.</p><p>Each of these features enables us to create more robust, reliable, and observable microservice architectures. We’ll go through practical examples to see these capabilities in action, helping us to fine-tune our mesh’s behavior to meet the needs of complex, real-world applications. Stay tuned for an in-depth exploration of how to leverage Istio to its full potential for managing traffic within your service mesh.</p><h3>References</h3><p>To deepen your understanding and further explore the concepts discussed in this series, consider the following resources:</p><ol><li><a href="https://istio.io/latest/"><strong>Istio Official Documentation</strong></a><strong>:</strong> The primary source of comprehensive documentation covering all aspects of Istio, from getting started to advanced configuration.</li><li><a href="https://www.envoyproxy.io/"><strong>Envoy Proxy Documentation:</strong></a> Since Istio uses Envoy as its sidecar proxy, understanding Envoy can be very beneficial.</li><li><a href="https://kubernetes.io/docs/home/"><strong>Kubernetes Documentation:</strong></a><strong> </strong>A deep understanding of Kubernetes is essential when working with Istio, as it operates within the Kubernetes ecosystem.</li></ol><p>These resources provide a solid foundation for both beginners and experienced users looking to leverage Istio within their service mesh for efficient service-to-service communication, traffic management, security, and observability.</p><h3>Disclaimer</h3><p>This article is for informational purposes only and does not provide any technical advice or recommendations. The views expressed in this article are solely those of the author and do not represent the views of any company or organization. Any action you take based on the information in this article is at your own risk.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=899e565da4d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in ISTIO — Envoy Proxy — Part 2]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-istio-envoy-proxy-part-2-27950e820b04?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/27950e820b04</guid>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[istio-service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[envoy-proxy]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 01:01:31 GMT</pubDate>
            <atom:updated>2023-03-13T01:12:48.554Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in ISTIO — Envoy Proxy — Part 2</h3><p><strong>Recap</strong>: In <a href="https://dramasamy.medium.com/life-of-a-packet-in-istio-part-1-8221971d77de?source=friends_link&amp;sk=325b2b612c0d7db562696553e0d2275e">Part 1 of the “<em>Life of a Packet in ISTIO</em>”</a> series, we explored the concept of a service mesh and Istio, an open-source service mesh that provides features like traffic management, observability, policy enforcement, and service identity and security. We then delved into the traffic interception process in Istio’s data plane, which is composed of a collection of proxy services represented as sidecar containers in each Kubernetes pod using an extended Envoy proxy. We discussed sidecar injection and the roles of the istio-proxy and istio-init containers. We then covered traffic interception for inbound and outbound traffic, and analyzed the NAT table rules that handle traffic routing.</p><p>In this next part (2) of the “<em>Life of a Packet in ISTIO</em>” series, We will introduce Envoy proxy configuration, and discuss how it works for inbound and outbound traffic. By the end of this part, you will have a clear understanding of how Istio handles traffic between microservices in the service mesh.</p><p>In the world of the service mesh, Envoy has emerged as one of the most popular proxy servers used to handle the traffic between services. Developed by Lyft and now a graduated project of the Cloud Native Computing Foundation (CNCF), Envoy provides a scalable, high-performance, and extensible solution for handling network traffic between services.</p><p>Envoy is designed to work as a sidecar proxy, running alongside each service instance in a service mesh. It intercepts all traffic between services, providing features such as load balancing, traffic routing, traffic management, and security. Envoy is a powerful tool that can be configured in many different ways to meet the specific needs of your service mesh.</p><p>In this post, we will take a deep dive into the configuration of Envoy, exploring the different components that make up an Envoy configuration file, and how they work together to create a robust and flexible service mesh. We will also provide examples of how to configure Envoy for different use cases, such as service discovery, routing, and traffic shaping.</p><p>Whether you’re just getting started with Envoy or are a seasoned user looking to deepen your understanding of its configuration options, this blog post will provide the knowledge you need to take your service mesh to the next level.</p><blockquote>If you’re interested in learning more about Kubernetes, check out my ‘Life of a Packet in Kubernetes’ blog for a deep dive into the Kubernetes network model and how packets flow in the cluster. You can find the blog post here: <a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051?source=friends_link&amp;sk=9525ced03e4683afca531c21914625eb">https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051?source=friends_link&amp;sk=9525ced03e4683afca531c21914625eb</a></blockquote><p>Here are some of the specific ways that Envoy features help in Istio:</p><ol><li><strong>Traffic management:</strong> Envoy’s dynamic routing capabilities allow Istio to manage traffic at the service mesh level. This enables Istio to perform traffic routing, load balancing, and service discovery without requiring changes to application code. Istio uses Envoy’s advanced routing features, including path-based routing, header-based routing, and fault injection, to implement advanced traffic management policies.</li><li><strong>Security: </strong>Envoy provides Istio with strong security features, including encryption and authentication. Istio uses Envoy’s TLS and mTLS support to secure communication between services in the service mesh. Envoy’s advanced access control features, including rate limiting and RBAC, allow Istio to enforce fine-grained security policies across the service mesh.</li><li><strong>Observability</strong>: Envoy’s observability features enable Istio to monitor and troubleshoot the service mesh. Envoy’s rich telemetry data, including detailed metrics and logs, can be used to diagnose issues and optimize performance. Istio uses Envoy’s distributed tracing capabilities to provide end-to-end tracing across the service mesh, allowing users to understand how requests flow through the mesh and identify performance bottlenecks.</li><li><strong>Extensibility:</strong> Envoy’s modular architecture allows Istio to easily add new functionality to the service mesh. Istio can add new Envoy filters or configure existing filters to implement custom logic, enabling users to extend Istio’s capabilities as needed.</li></ol><p>Overall, Envoy’s rich feature set makes it an ideal choice for powering Istio’s service mesh platform. Envoy’s advanced traffic management, security, observability, and extensibility features enable Istio to provide a powerful and flexible platform for managing microservices at scale.</p><h3>Envoy Proxy Processing Steps: How a Packet is Handled</h3><p>When a packet arrives at an Envoy proxy, it goes through a series of steps before it reaches its final destination. These steps include the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s3MxPjCOF7EM25Fc6ccB_Q.png" /></figure><p>At the ingress phase, the Envoy listener component accepts incoming connections from the downstream client on specific ports and protocols such as TCP, HTTP, or HTTPS. The listener filter chain provides SNI and other pre-TLS details and matches a network filter chain based on destination IP CIDR range, SNI, ALPN, source ports, etc. A transport socket, such as the TLS transport socket, is associated with the filter chain for secure communication.</p><p>The network filter chain decrypts the data from the TCP connection using the TLS transport socket on network reads, and the HTTP connection manager filters are the last in the chain. The HTTP/2 codec in the HTTP connection manager deframes and demultiplexes the decrypted data stream from the TLS connection into independent streams, handling each request and response.</p><p>For each HTTP stream, a corresponding HTTP filter chain is created and run. The request passes through custom filters that can read and modify the request. The router filter is the most important HTTP filter, sitting at the end of the chain, and selects a route and a cluster based on the headers of the incoming request. The headers are then forwarded to an upstream endpoint in that cluster. The router filter obtains an HTTP connection pool from the cluster manager for the matched cluster to handle this process.</p><p>Cluster-specific load balancing is performed to find an endpoint, and circuit breakers are checked to determine if a new stream is allowed. A new connection to the endpoint is created if the endpoint’s connection pool is empty or lacks capacity. The upstream endpoint connection’s HTTP/2 codec multiplexes and frames the request’s stream with any other streams going to that upstream over a single TCP connection. The upstream endpoint connection’s TLS transport socket encrypts these bytes and writes them to a TCP socket for the upstream connection.</p><p>The request, consisting of headers, and optional body and trailers, is proxied upstream, and the response is proxied downstream. The response passes through the HTTP filters in the opposite order from the request, starting at the router filter and passing through custom filters before being sent downstream. When the response is complete, the stream is destroyed. Post-request processing is done, including updating stats, writing to the access log, and finalizing trace spans.</p><p>Finally, the egress configuration specifies the endpoint information for the downstream client, including transport protocol, encryption, and other settings. Envoy can be configured to forward the response to the downstream client, using the appropriate protocol and transport mechanism, allowing you to configure filters to process the response from the upstream service, add response headers, set cookies, perform caching, and more, and also to configure the access log filters to log the response.</p><p>Overall, the Envoy proxy configuration is highly customizable and can be tailored to meet the specific needs of your application architecture. By carefully configuring each phase of the packet life cycle, you can achieve advanced traffic management, observability, and security features for your modern applications.</p><h3>Hands-On Demo: Envoy front proxy</h3><p>Let’s see a practical example of how to configure an Envoy proxy using the provided YAML configuration. In this example, we are configuring a listener to accept incoming traffic on port 8000 and forward it to a cluster named “service” using round-robin load balancing. The cluster is defined as a strict DNS type and has three endpoints, each with an address of service_blue, service_green, and service_red, all listening on port 3000.</p><p>The connection manager is configured to generate request IDs, use automatic codec type detection, and prefix statistics with “ingress_http.” It also specifies a virtual host named “service” that accepts traffic for any domain and a route matching any requests with a prefix of “/service.” The route is then forwarded to the “service” cluster.</p><p>To secure the communication between Envoy and its clients, we are using a self-signed certificate pair for Transport Layer Security (TLS) authentication. The certificate pair is fed to Envoy as an inline string, but it can also be provided as a file or fetched remotely via Secret Discovery Service (SDS) in a dynamic configuration scenario.</p><p>Overall, this configuration demonstrates how Envoy can be used as a powerful and flexible proxy to handle incoming traffic and distribute it to a cluster of endpoints with load balancing and security features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ShoNXirF6uSXB5vDIIsnw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e7bf071900166a56aee73c8e68cb7ea/href">https://medium.com/media/9e7bf071900166a56aee73c8e68cb7ea/href</a></iframe><blockquote>To try out the example with the given configuration, please clone the GitHub repository at <strong>https://github.com/dramasamy/training.git </strong>and navigate to the envoy directory. From there, you can run docker-compose up to start the Envoy proxy container and test the configuration.</blockquote><p>Here are the steps to execute docker-compose and browse the page to see the Envoy load balancing:</p><p>Step 1: Clone the repository by running the following command in your terminal:</p><pre>git clone https://github.com/dramasamy/training.git</pre><p>Step 2: Navigate to the envoy directory by running the following command:</p><pre>cd training/envoy/front-proxy</pre><p>Step 3: Execute docker-compose to start the containers by running the following command:</p><pre>docker compose up -d</pre><p>Step 4: Once the containers are up and running, open your browser and navigate to https://localhost:8000/service. This will show you the response from one of the three services, with load balancing being performed by Envoy.</p><p>Step 5: Refresh the page a few times and observe that the response changes between the three services (blue, green, and red), demonstrating the round-robin load balancing performed by Envoy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ydN3_7Dgwp9TAE_s4nBl3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCuKXiTqAhbFk87Htu9gKg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UpX0yvXZwqAJJLpMWrz9Bg.png" /></figure><h3>Tracing in Envoy</h3><p>Tracing is an essential feature that helps developers in understanding how different services are communicating with each other. When applications are distributed, the request can span across multiple services, making it challenging to understand what’s happening across the network. Envoy provides an easy way to capture and visualize traces through its Zipkin tracing feature.</p><h4>Hands-On Demo: Jaeger Tracing</h4><p>The provided configuration file demonstrates how to enable the Zipkin tracer in Envoy. In this configuration, Envoy acts as a front proxy and listens on port 8000. The traffic_direction field indicates that it is an outbound listener. The tracing field is enabled, which enables the Zipkin tracer with the collector&#39;s configuration information. The collector cluster is specified as &quot;jaeger,&quot; and the collector endpoint is specified as &quot;/api/v2/spans.&quot; The shared_span_context field is set to false, indicating that the span context is not propagated between services.</p><p>The route configuration is specified under virtual_hosts. The decorator field is used to set the operation name in the span. In this example, the decorator is set to &quot;checkAvailability.&quot; The use_remote_address field is set to true, indicating that Envoy should use the client&#39;s IP address to determine the source address of the request.</p><p>The clusters section defines the different backend services and their associated clusters. In this example, there is only one backend service called service1. The load_assignment field defines how the endpoints should be assigned to the service. In this case, the endpoint is defined as service1:8000.</p><p>Overall, this configuration file enables Envoy to capture trace information using the Zipkin tracer and propagate span context across services. Developers can use this information to gain insight into how different services are communicating and to debug issues in their distributed applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NlTfyn7ASW0vC12pnCZ3BA.png" /></figure><p>Step 1: Navigate to the ‘training/envoy/jaeger’ directory:</p><pre>cd training/envoy/jaeger/</pre><p>Step 2: Start the docker containers by running:</p><pre>docker-compose up -d</pre><p>Step 3: Open two browser tabs and navigate to <a href="http://localhost:8000/trace/1">http://localhost:8000/trace/1</a> and <a href="http://localhost:8000/trace/2">http://localhost:8000/trace/2</a></p><p>Step 4: Refresh the pages a few times to generate some traffic.</p><p>Step 5: Open another browser tab and navigate to <a href="http://localhost:10000/search">http://localhost:10000/search</a> to access Jaeger’s search page.</p><p>Step 6: In the Jaeger search page, select ‘front-proxy’ as the service from the dropdown menu and click the ‘Find Traces’ button.</p><p>Step 7: You should now see a list of traces generated by Envoy, including the traces generated by the traffic to <a href="http://localhost:8000/trace/1">http://localhost:8000/trace/1</a> and <a href="http://localhost:8000/trace/2">http://localhost:8000/trace/2</a>. Click on a trace to view its details and explore the distributed tracing functionality of Envoy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-68SIhHczLv8YBkzb_eGdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMYRHm3MTSa8G66jX36K5g.png" /></figure><p>One of the most important benefits of tracing from Envoy is that it will take care of propagating the traces to the Jaeger service cluster. However, in order to fully take advantage of tracing, the application has to propagate trace headers that Envoy generates, while making calls to other services. In the flask app (see trace function in envoy/shared/flask/tracing/service.py) acting as service1 propagates the trace headers while making an outbound call to service2.</p><h3>Envoy UDP Proxy</h3><p>The User Datagram Protocol (UDP) example in Envoy provides a simple demonstration of how Envoy can be used to proxy UDP traffic. The example includes an upstream server that listens for UDP traffic on port 5005 and an Envoy proxy that listens for UDP traffic on port 10000 and proxies it to the upstream server. Additionally, Envoy provides an admin endpoint on port 10001 that provides statistics on UDP traffic. This example demonstrates how Envoy can be used to handle not only HTTP but also UDP traffic.</p><h4>Hands-On Demo: UDP proxy</h4><p>Step 1: Change the directory to <em>envoy/udp</em> in the Envoy repository.</p><pre>cd training/envoy/udp/</pre><p>Step 2: Start the docker containers by running:</p><pre>docker-compose up -d</pre><p>This starts Envoy and an upstream server on port 5005.</p><p>Step 2: Send some UDP messages Send packets to the upstream server, proxied by Envoy using netcat. Run the following commands:</p><pre>echo -n HELO | nc -4u -w1 127.0.0.1 10000<br>echo -n OLEH | nc -4u -w1 127.0.0.1 10000</pre><p>Step 3: Check the logs of the upstream UDP listener server To view the packets that were sent, check the logs of the upstream server by running the following command:</p><pre>lab1 $ docker compose logs service-udp<br>service-udp_1  | Listening on UDP port 5005<br>service-udp_1  | HELO<br>service-udp_1  | OLEH</pre><p>Step 4: View the Envoy admin UDP stats View the stats provided by the Envoy admin endpoint by running the following command:</p><pre>lab1 $ curl -s http://127.0.0.1:10001/stats | grep udp | grep -v &quot;\: 0&quot;<br>cluster.service_udp.default.total_match_count: 15<br>cluster.service_udp.max_host_weight: 1<br>cluster.service_udp.membership_change: 1<br>cluster.service_udp.membership_healthy: 1<br>cluster.service_udp.membership_total: 1<br>cluster.service_udp.udp.sess_tx_datagrams: 2<br>cluster.service_udp.update_attempt: 15<br>cluster.service_udp.update_no_rebuild: 14<br>cluster.service_udp.update_success: 15<br>cluster.service_udp.upstream_cx_tx_bytes_total: 8<br>udp.service.downstream_sess_rx_bytes: 8<br>udp.service.downstream_sess_rx_datagrams: 2<br>udp.service.downstream_sess_total: 2<br>udp.service.idle_timeout: 2<br>cluster.service_udp.upstream_cx_connect_ms: No recorded values<br>cluster.service_udp.upstream_cx_length_ms: No recorded values</pre><p>This will display the non-zero stats, including upstream connection data and downstream session data.</p><h3>Dynamic XDS Configuration — Updating Envoy’s Configuration on the Fly</h3><p>Envoy XDS is a core component of Istio, which is a service mesh that provides features such as traffic management, security, and observability to microservices-based applications.</p><p>Istio uses the Envoy proxy as a sidecar container alongside each service instance to handle incoming and outgoing traffic for that instance. Envoy XDS is used by Istio to dynamically configure the Envoy sidecars with the service mesh configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a-tErRePcGywR7liNxr_Qw.png" /></figure><p>Here is a high-level overview of how Envoy XDS works.</p><ol><li>The configuration server serves as a centralized repository for all the configuration information that the Envoy proxy needs to route traffic in the service mesh.</li><li>The configuration server provides dynamic configuration updates to the Envoy proxy via the XDS (Discovery Service) protocol. The XDS protocol uses gRPC to enable communication between the control plane and the data plane.</li><li>When a new service is added to the service mesh or when an existing service is updated, the configuration server sends a new configuration update to the Envoy proxy.</li><li>The Envoy proxy applies the new configuration update and begins routing traffic according to the new rules.</li><li>The configuration server can also provide health check information to Envoy, which enables Envoy to route traffic only to healthy instances of a service.</li><li>In this way, the control plane and the data plane work together to provide dynamic service discovery and traffic routing in the service mesh.</li></ol><h4>Hands-On Demo: Using Envoy XDS</h4><p>Step 1: Change the directory to <em>envoy/dynamic-config-cp</em> in the Envoy repository.</p><pre>cd training/envoy/dynamic-config-cp/</pre><p>Step 2: Start the proxy container along with two upstream HTTP echo servers, service1 and service2.</p><pre>docker-compose up -d proxy</pre><p>Step 3: As the control plane has not yet been started, nothing should be responding on port 10000. To check this, dump the proxy’s static_clusters configuration by running the command curl -s http://localhost:19000/config_dump | jq &#39;.configs[1].static_clusters&#39;. The output should show the cluster named xds_cluster configured for the control plane.</p><pre>lab1 $ curl -s http://localhost:19000/config_dump | jq &#39;.configs[1].static_clusters&#39;<br>[<br>  {<br>    &quot;cluster&quot;: {<br>      &quot;@type&quot;: &quot;type.googleapis.com/envoy.config.cluster.v3.Cluster&quot;,<br>      &quot;name&quot;: &quot;xds_cluster&quot;,<br>      &quot;type&quot;: &quot;STRICT_DNS&quot;,<br>      &quot;load_assignment&quot;: {<br>        &quot;cluster_name&quot;: &quot;xds_cluster&quot;,<br>        &quot;endpoints&quot;: [<br>          {<br>            &quot;lb_endpoints&quot;: [<br>              {<br>                &quot;endpoint&quot;: {<br>                  &quot;address&quot;: {<br>                    &quot;socket_address&quot;: {<br>                      &quot;address&quot;: &quot;go-control-plane&quot;,<br>                      &quot;port_value&quot;: 18000<br>                    }<br>                  }<br>                }<br>              }<br>            ]<br>          }<br>        ]<br>      },<br>      &quot;typed_extension_protocol_options&quot;: {<br>        &quot;envoy.extensions.upstreams.http.v3.HttpProtocolOptions&quot;: {<br>          &quot;@type&quot;: &quot;type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions&quot;,<br>          &quot;explicit_http_config&quot;: {<br>            &quot;http2_protocol_options&quot;: {}<br>          }<br>        }<br>      }<br>    },<br>    &quot;last_updated&quot;: &quot;2023-03-12T23:16:57.273Z&quot;<br>  }<br>]</pre><p>Step 4: No dynamic_active_clusters have been configured yet:</p><pre>lab1 $ curl -s http://localhost:19000/config_dump  | jq &#39;.configs[1].dynamic_active_clusters&#39;<br>null</pre><p>Step 5: Start up the go-control-plane service by running the command docker compose up --build -d go-control-plane. You may need to wait a moment or two for it to become healthy.</p><p>Step 6: Once the control plane has started and is healthy, you should be able to make a request to port 10000, which will be served by service1.</p><pre>lab1 $ curl http://localhost:10000<br>Request served by service1<br><br>GET / HTTP/1.1<br><br>Host: localhost:10000<br>Accept: */*<br>User-Agent: curl/7.85.0<br>X-Envoy-Expected-Rq-Timeout-Ms: 15000<br>X-Forwarded-Proto: http<br>X-Request-Id: 8df92d24-6656-44d2-90d9-53f00d3af091</pre><p>Step 7: To verify that the dynamic configuration is working, dump the proxy’s dynamic_active_clusters configuration by running the command curl -s http://localhost:19000/config_dump | jq &#39;.configs[1].dynamic_active_clusters&#39;. The output should show that the example_proxy_cluster is pointing to service1 and has a version of 1.</p><pre>lab1 $ curl -s http://localhost:19000/config_dump | jq &#39;.configs[1].dynamic_active_clusters&#39;<br>[<br>  {<br>    &quot;version_info&quot;: &quot;1&quot;,<br>    &quot;cluster&quot;: {<br>      &quot;@type&quot;: &quot;type.googleapis.com/envoy.config.cluster.v3.Cluster&quot;,<br>      &quot;name&quot;: &quot;example_proxy_cluster&quot;,<br>      &quot;type&quot;: &quot;LOGICAL_DNS&quot;,<br>      &quot;connect_timeout&quot;: &quot;5s&quot;,<br>      &quot;dns_lookup_family&quot;: &quot;V4_ONLY&quot;,<br>      &quot;load_assignment&quot;: {<br>        &quot;cluster_name&quot;: &quot;example_proxy_cluster&quot;,<br>        &quot;endpoints&quot;: [<br>          {<br>            &quot;lb_endpoints&quot;: [<br>              {<br>                &quot;endpoint&quot;: {<br>                  &quot;address&quot;: {<br>                    &quot;socket_address&quot;: {<br>                      &quot;address&quot;: &quot;service1&quot;,<br>                      &quot;port_value&quot;: 8080<br>                    }<br>                  }<br>                }<br>              }<br>            ]<br>          }<br>        ]<br>      }<br>    },<br>    &quot;last_updated&quot;: &quot;2023-03-12T23:26:24.879Z&quot;<br>  }<br>]</pre><h3>Summary</h3><p>In part two, we discussed load balancing and tracing in Envoy. We started by explaining the basic front proxy example and then added load balancing to it using the round-robin algorithm. Next, we explored how tracing works in Envoy and provided an example configuration for it. We also asked the user to execute Docker Compose and browse the page to see the load balancing and tracing in action.</p><p>Additionally, we introduced the concept of Envoy XDS and explained a sample configuration for it, including CDS and LDS. Finally, we briefly discussed how Envoy XDS works in Istio.</p><p>Stay tuned for part three, where we will dive into Istio’s control plane components.</p><h3>References:</h3><ul><li>Envoy documentation: <a href="https://www.envoyproxy.io/docs/envoy/latest/">https://www.envoyproxy.io/docs/envoy/latest/</a></li><li>Envoy GitHub repository: <a href="https://github.com/envoyproxy/envoy">https://github.com/envoyproxy/envoy</a></li><li>Istio documentation: <a href="https://istio.io/latest/docs/">https://istio.io/latest/docs/</a></li><li>Jaeger documentation: <a href="https://www.jaegertracing.io/docs/">https://www.jaegertracing.io/docs/</a></li><li>Zipkin documentation: <a href="https://zipkin.io/">https://zipkin.io/</a></li></ul><h3>Disclaimer</h3><p>This article is for informational purposes only and does not provide any technical advice or recommendations. The views expressed in this article are solely those of the author and do not represent the views of any company or organization. Any action you take based on the information in this article is at your own risk.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27950e820b04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in ISTIO — Part 1]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-istio-part-1-8221971d77de?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/8221971d77de</guid>
            <category><![CDATA[traffic-management]]></category>
            <category><![CDATA[service-mesh]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[istio]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Wed, 09 Mar 2022 05:55:17 GMT</pubDate>
            <atom:updated>2024-03-25T01:17:42.215Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in ISTIO — Part 1</h3><p>The term “service mesh” has probably insinuated your consciousness sometimes if you are working with backend systems. Istio has been criticized as notoriously complex, but today’s Istio is much more straightforward. In this series, I will try to follow an HTTP request in the ISTIO service mesh to deep dive to find how it handles the inbound and outbound traffic by intercepting the traffic.</p><h4>Topics — Part 1 (Traffic Interception)</h4><ol><li>Sidecar injection</li><li>Init-container iptable configuration</li><li>Traffic Interception in SideCar Proxy</li></ol><h4>Topics — Part 2 (SideCar proxy config)</h4><ol><li>Envoy proxy configuration introduction</li></ol><p><strong>Link:</strong> <a href="https://dramasamy.medium.com/life-of-a-packet-in-istio-envoy-proxy-part-2-27950e820b04?source=friends_link&amp;sk=fdeab38bcabd0a7e5594cf14a8471bab">Envoy Proxy</a></p><h4>Topics — Part 3 (Architecture)</h4><ol><li>Service Discovery and Configuration</li></ol><h4>Topics — Part 4 (Traffic Management)</h4><ol><li>Ingress Gateway</li><li>Egress Gateway</li><li>Gateway, VirtualService, and Destination rules</li></ol><h4>Topics — Part 5 (Observability)</h4><ol><li>Metrics</li><li>Tracing</li></ol><h4>Topics — Part 6 (Security)</h4><ol><li>TLS</li></ol><blockquote>Life of a Packet in Kubernetes</blockquote><p><a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051">Life of a Packet in Kubernetes — Part 1</a></p><h3>What is a Service Mesh?</h3><blockquote>Modern applications are typically architected as distributed collections of microservices, with each collection of microservices performing some discrete business function. A service mesh is a dedicated infrastructure layer that you can add to your applications. It allows you to transparently add capabilities like observability, traffic management, and security, without adding them to your own code. The term “service mesh” describes both the type of software you use to implement this pattern and the security or network domain that is created when you use that software.</blockquote><blockquote>As the deployment of distributed services, such as in a Kubernetes-based system, grows in size and complexity, it can become harder to understand and manage. Its requirements can include <strong>discovery</strong>, <strong>load balancing</strong>, <strong>failure recovery</strong>, <strong>metrics</strong>, and <strong>monitoring</strong>. A service mesh also often addresses more complex operational requirements, like <strong>A/B testing</strong>, <strong>canary </strong>deployments, <strong>rate limiting</strong>, <strong>access control</strong>, <strong>encryption</strong>, and <strong>end-to-end authentication</strong>.</blockquote><h3>What is Istio?</h3><blockquote>Istio is an open-source service mesh that layers transparently onto existing distributed applications. Istio’s powerful features provide a uniform and more efficient way to secure, connect, and monitor services. Istio is the path to load balancing, service-to-service authentication, and monitoring — with few or no service code changes. Istio provides a complete solution to satisfy the diverse requirements of microservice applications by providing behavioral insights and operational control over the service mesh as a whole. It provides a number of key capabilities uniformly across a network of services:</blockquote><blockquote><strong>Traffic Management:</strong> Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions.</blockquote><blockquote><strong>Observability:</strong> Gain an understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues.</blockquote><blockquote><strong>Policy Enforcement:</strong> Apply the organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing the application code.</blockquote><blockquote><strong>Service Identity and Security:</strong> Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustability.</blockquote><h3>How it Works</h3><blockquote>Istio has two main components: the <strong>data plane</strong> and the <strong>control plane.</strong></blockquote><blockquote><strong>Data plane</strong>: The data plane, or data layer, is composed of a collection of proxy services represented as sidecar containers in each Kubernetes pod, using an extended <a href="https://www.envoyproxy.io/">Envoy</a> proxy. Those sidecars mediate and control all network communication between the microservices while also collecting and reporting useful telemetry data.</blockquote><blockquote><strong>Control plane</strong>: The control plane, or control layer, consists of a single binary called <em>istiod</em> that is responsible for converting high-level routing rules and traffic control behavior into Envoy-specific configurations, then propagating them to sidecars at runtime. Additionally, the control plane provides security measures enabling strong service-to-service and end-user authentication with built-in identity and credential management while enforcing security policies based on service identity.</blockquote><blockquote>The control plane takes your desired configuration, and its view of the services, and dynamically programs the proxy servers, updating them as the rules or the environment changes.</blockquote><h4>I’ll be using the book info application from the ISTIO documentation as an example to explore and explain ISTIO.</h4><p><a href="https://istio.io/latest/docs/examples/bookinfo/">Bookinfo Application</a></p><h3><strong>Traffic routing <em>without</em> ISTIO</strong></h3><p>A Kubernetes service can be used to easily expose an application deployed on a set of pods using a single endpoint. Services keep track of the changes in IP addresses and DNS names of the pods and expose them to the end-user as a single IP or DNS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*7b9XuN7mfSawDYZGrzp4Vg.png" /></figure><blockquote>Here is the link to the ‘<strong><em>Life of a Packet in Kubernetes — Part 3</em></strong>’ to know more about Kubernetes services.</blockquote><p><a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-3-dd881476da0f">Life of a Packet in Kubernetes — Part 3</a></p><h3><strong>Traffic routing <em>with</em> ISTIO</strong></h3><p>Envoy Proxy is deployed as a <strong>sidecar</strong> to the relevant service in the same Kubernetes pod. A sidecar is just a container that runs on the same Pod as the application container, because it shares the same volume and network as the main container, it can “help” or enhance how the application operates.</p><figure><img alt="Book info app with ISTIO" src="https://cdn-images-1.medium.com/max/591/1*a6mQ5pbjXLZ6Ii8ta87y2Q.png" /></figure><h3>SideCar Injection</h3><p>In simple terms, sidecar injection is adding the configuration of additional containers to the pod template. The added containers needed for the Istio service mesh are,</p><h4>istio-proxy (sideCar)</h4><p>istio-proxy This is the actual sidecar proxy (based on Envoy).</p><p>This allows Istio to extract a wealth of signals about traffic behavior as <a href="https://istio-releases.github.io/docs/concepts/policy-and-control/attributes.html">attributes</a>, which in turn it can use in <a href="https://istio-releases.github.io/docs/concepts/policy-and-control/mixer.html">Mixer</a> to enforce policy decisions and be sent to monitoring systems to provide information about the behavior of the entire mesh. The sidecar proxy model also allows you to add Istio capabilities to an existing deployment with no need to rearchitect or rewrite code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f6069ca22fcc58fbb772cdf822d5d511/href">https://medium.com/media/f6069ca22fcc58fbb772cdf822d5d511/href</a></iframe><p>istio-proxy container runs with restricted privileges as a user 1337. As this is reserved, the UID (User ID) for an application workload must be different and must not conflict with 1337. The 1337 UID has been chosen arbitrarily by the Istio team to bypass traffic redirection to istio-proxy container. You can also see 1337 being used as an argument to istio-iptables when initializing iptables. As this container is actively running along with the application workload, Istio also ensures that if it’s compromised, it only has read-only access to the root filesystem.</p><h4>istio-init</h4><p>istio-init This <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">init container</a> is used to set up the iptables rules so that inbound/outbound traffic will go through the sidecar proxy. An init container is different than an app container in the following ways:</p><ul><li>It runs before an app container is started and it always runs to <strong>completion</strong>.</li><li>If there are many init containers, each should be complete with success before the next container is started.</li></ul><p>So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a5140518d1b39aaee017a2df74ee9a5/href">https://medium.com/media/9a5140518d1b39aaee017a2df74ee9a5/href</a></iframe><p>To minimize the attack surface, securityContext stanza in the istio-init container signifies that the container runs with root privileges (runAsUser: 0), however all Linux capabilities are dropped with the exception of the NET_ADMIN and NET_RAW capabilities. These capabilities provide the istio-init init container with runtime privileges to rewrite the application pod’s iptables.</p><h4>istio-init iptable rules</h4><p>How does the sidecar proxy grab the inbound and outbound traffic to and from the container? It is done by the iptable rules created by theistio-init container command.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb284f63f41e4c584f8f9c94b6b7e8ea/href">https://medium.com/media/eb284f63f41e4c584f8f9c94b6b7e8ea/href</a></iframe><h4>SideCar injection methods</h4><p>In the <strong>manual injection method,</strong> you can use <a href="https://istio.io/latest/docs/reference/commands/istioctl">istioctl</a> to modify the pod template and add the configuration of the two containers previously mentioned. For both manual as well as automatic injection, Istio takes the configuration from the istio-sidecar-injector configuration map (configmap) and the mesh’s istio configmap.</p><blockquote>istioctl kube-inject -f application.yaml | kubectl apply -f -</blockquote><p>Most of the time, you don’t want to manually inject a sidecar every time you deploy an application, using the <a href="https://istio.io/latest/docs/reference/commands/istioctl">istioctl</a> command, but would prefer that <strong>Istio automatically inject the sidecar</strong> to your pod. This is the recommended approach and for it to work, all you need to do is to label the namespace where you are deploying the app with istio-injection=enabled. Istio relies on Mutating Admission Webhook for injecting the sidecar.</p><blockquote>kubectl label namespace &lt;namespaceName&gt; istio-injection=enabled</blockquote><p>Here’s the process that Kubernetes mutating admission controller handles in the sidecar injection:</p><ol><li>First, the istio-sidecar-injector mutating configuration injected during the Istio installation process (shown below) sends a webhook request with all pod information to the istiod controller.</li><li>Next, the controller modifies the pod specification in runtime introducing an init and sidecar container agents to the actual pod specification.</li><li>Then, the controller returns the modified object back to the admission webhook for object validation.</li><li>Finally after validation, the modified pod specification is deployed with all the sidecar containers.</li></ol><p>For the full configuration, take a look at kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml.</p><h3>Traffic Interception (Inbound traffic)</h3><p>TCP requests that are sent or received from the Pod will be hijacked by iptables. After the inbound traffic is hijacked, it is processed by the Inbound Handler and then forwarded to the application container for processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*84uJcFmc5cw89A_vyD4C0w.png" /></figure><p>In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.</p><p>This has two important side effects that cause the behavior to differ from standard Kubernetes:</p><ul><li>Applications binding only to lo will receive traffic from other pods, when otherwise this is not allowed.</li><li>Applications binding only to eth0 will not receive traffic.</li></ul><p>Applications that bind to both interfaces (which is typical) will not be impacted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*pNbHTESzu9o8_8lwu3ghSw.png" /></figure><p>Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.</p><p>Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with <a href="https://istio.io/latest/blog/2021/zero-config-istio/">zero configuration</a>. Additionally, it avoids unintended exposure of applications binding only to lo.</p><h3>Traffic Interception (Outbound traffic)</h3><p>The outbound traffic is hijacked by iptables and then forwarded to the Outbound Handler for processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*5Zhpxk90ArZHDgP_HyxLjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*4p4Ya0mMHBzgXG0PRRPhxQ.png" /></figure><h3>iptable analysis</h3><p>Following are the NAT table rules that handle inbound and outbound traffic routing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0752c821c39e106f0ad3b9a4cd80ee3/href">https://medium.com/media/a0752c821c39e106f0ad3b9a4cd80ee3/href</a></iframe><h4>Outbound Traffic Routing</h4><ol><li>Product Page service sends requests to the reviews service which is intercepted by netfilter again and forwarded to the export traffic OUTPUT chain</li><li>OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain</li><li>Traffic that sends non-localhost requests and is istio proxy user space is forwarded to ISTIO_REDIRECT chain</li><li>ISTIO_REDIRECT chain is directly redirected to the 15001 outlet traffic port monitored by envoy</li><li>After a series of export traffic governance actions, envoy continues to send response data, which will be intercepted by netfilter and forwarded to the export traffic OUTPUT chain</li><li>OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain</li><li>The traffic is directly returned to the POSTROUTING and then reaches the Reviews service.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*FcrI_Hwey764CXnb8Qeh7g.png" /></figure><p><strong>Inbound Traffic Routing</strong></p><ol><li>The product page service sends a TCP connection request to the reviews service</li><li>The request enters the Pod kernel space where the reviews service is located, is intercepted by netfilter, passes through the PREROUTING chain, and then forwarded to the ISTIO_INBOUND chain</li><li>ISTIO_INBOUND chain is defined by this rule — A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT intercepts and forwards to ISTIO_IN_REDIRECT</li><li>ISTIO_IN_REDIRECT chain is directly redirected to the 15006 inlet traffic port monitored by the envoy.</li><li>After a series of inbound traffic governance actions, the envoy sends a TCP connection request to review the service. This step belongs to outbound traffic for the envoy and will be intercepted by netfilter and forwarded to the outbound traffic OUTPUT chain</li><li>OUTPUT chain forwarding traffic to ISTIO_OUTPUT chain</li><li>The destination is localhost, which can’t be matched to the forwarding rule chain. RETURN directly to the next chain.</li><li>The request from sidecar arrives at the review service 9080 port</li></ol><h3>References:</h3><pre><a href="https://istio.io/">https://istio.io/</a><br><a href="https://www.solo.io/blog/">https://www.solo.io/blog/</a><br><a href="https://istio.io/latest/blog/">https://istio.io/latest/blog/</a><br><a href="https://github.com/istio">https://github.com/istio</a><br><a href="https://github.com/istio/istio/wiki/Design-Doc-Links">https://github.com/istio/istio/wiki/Design-Doc-Links</a><br><a href="https://github.com/istio/old_pilot_repo/blob/master/doc/design.md">https://github.com/istio/old_pilot_repo/blob/master/doc/design.md</a></pre><h3>Disclaimer</h3><p>This article does not provide any technical advice or recommendation; if you feel so, it is my personal view, not the company I work for.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8221971d77de" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Pod Deployment — Behind the scenes]]></title>
            <link>https://dramasamy.medium.com/kubernetes-pod-deployment-behind-the-scenes-5cc59982cec7?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cc59982cec7</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Sat, 26 Feb 2022 20:37:33 GMT</pubDate>
            <atom:updated>2022-08-13T23:19:34.973Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Pod Deployment — Behind the scenes</h3><p>Picture link:</p><p><a href="https://raw.githubusercontent.com/dramasamy/training/master/kubernetes/kubernetes_deployment_flow.png">https://raw.githubusercontent.com/dramasamy/training/master/kubernetes/kubernetes_deployment_flow.png</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-RoDt9W0R2Qi6oVrxNA0_w.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cc59982cec7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in Kubernetes — Part 4]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-4-4dbc5256050a?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dbc5256050a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[load-balancer]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Mon, 10 May 2021 02:42:47 GMT</pubDate>
            <atom:updated>2021-05-10T02:42:47.386Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in Kubernetes — Part 4</h3><p>This is part 4 of the series on <em>Life of a Packet in Kubernetes</em> we’ll be tackling Kubernetes’s Ingress resource and Ingress controller. An Ingress Controller is a <a href="https://kubernetes.io/docs/concepts/architecture/controller/">controller</a> that watches the Kubernetes API server for updates to the <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress resource</a> and reconfigures the Ingress load balancer accordingly.</p><h4>Nginx Controller and LoadBalancer/Proxy</h4><p>An ingress controller is usually an application that runs as a pod in a Kubernetes cluster and configures a load balancer according to Ingress Resources. The load balancer can be a software load balancer running in the cluster or hardware or cloud load balancer running externally. Different load balancers require different ingress controllers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBThYKgU3n2HWeTWg_AEJw.png" /></figure><p>The basic idea behind the Ingress is to provide a way of describing higher-level traffic management constraints, specifically for HTTP(S). With Ingress we can define rules for routing traffic without creating a bunch of Load Balancers or exposing each service on the node. It can be configured to give services externally-reachable URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting and content-based routing.</p><h3>Configuration Options</h3><p>The Kubernetes Ingress Controller uses ingress classes to filter Kubernetes Ingress objects and other resources before converting them into configuration. This allows it to coexist with other ingress controllers and/or other deployments of the Kubernetes Ingress Controller in the same cluster: a Kubernetes Ingress Controller will only process configuration marked for its use.</p><h4>Prefix Based</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/654cf655f3c808ed4c27596ade43a30e/href">https://medium.com/media/654cf655f3c808ed4c27596ade43a30e/href</a></iframe><h4>Host-Based</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2893fea54a7037eb9074ca98e884fdc9/href">https://medium.com/media/2893fea54a7037eb9074ca98e884fdc9/href</a></iframe><h4>Host + Prefix</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9001b4c772ab7b14c331db589055d33c/href">https://medium.com/media/9001b4c772ab7b14c331db589055d33c/href</a></iframe><p>Ingress is one of the built-in APIs which doesn’t have a built-in controller, and an ingress controller is needed to actually implement the Ingress API. There are many ingress controllers out there, will look at <strong>Nginx</strong> and <strong>Contour</strong>.</p><p>Ingress is made up of an Ingress API object and an Ingress controller. As mentioned earlier, Kubernetes Ingress is an API object that describes the desired state for exposing services deployed to a Kubernetes cluster. So, to make it work as an Ingress controller you will require the actual implementation of the Ingress API to read and process the Ingress resource’s information.</p><p>Since the Ingress API is actually <em>just</em> metadata, the Ingress controller does the heavy lifting. Various ingress controllers <a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/#additional-controllers">are available</a> and it is important to choose the right one carefully for each use case.</p><p>It’s also possible to have multiple ingress controllers in the same cluster and to set the desired ingress controller for each Ingres. Usually, we end up using a combination of these controllers for different scenarios in the same cluster. For example, we may have one for handling the external traffic coming into the cluster which includes bindings to SSL certificates, and has another internal one with no SSL binding that handles in-cluster traffic.</p><h3>Deployment options</h3><h4>Contour + Envoy</h4><p>The Contour Ingress controller is a collaboration between:</p><ul><li>Envoy, which provides the high-performance reverse proxy.</li><li>Contour, which acts as a management server for Envoy and provides it with configuration</li></ul><p>These containers are deployed separately, Contour as a Deployment and Envoy as a Daemonset, although other configurations are possible. Contour is a client of the Kubernetes API. Contour watches Ingress, HTTPProxy, Secret, Service, and Endpoint objects, and acts as the management server for its Envoy sibling by translating its cache of objects into the relevant JSON stanzas: Service objects for CDS, Ingress for RDS, Endpoint objects for EDS, and so on).</p><p>The following example shows EnvoyProxy with <strong>host network</strong>-enabled (0.0.0.0:80)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77Qj-DGe4IM42h_0NPcaCA.png" /></figure><h4>Nginx</h4><p>The goal of the Nginx Ingress controller is the assembly of a configuration file (nginx.conf). The main implication of this requirement is the need to reload NGINX after any change in the configuration file. <em>Though it is important to note that we don’t reload Nginx on changes that impact only an </em><em>upstream configuration (i.e Endpoints change when you deploy your app)</em>. We use <a href="https://github.com/openresty/lua-nginx-module">lua-nginx-module</a> to achieve this.</p><p>On every endpoint change, the controller fetches endpoints from all the services it sees and generates corresponding Backend objects. It then sends these objects to a Lua handler running inside Nginx. The Lua code in turn stores those backends in a shared memory zone. Then for every request Lua code running in <a href="https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md">balancer_by_lua</a> context detects what endpoints it should choose upstream peer from and applies the configured load balancing algorithm to choose the peer. Then Nginx takes care of the rest. This way we avoid reloading Nginx on endpoint changes. In a relatively big cluster with frequently deploying apps, this feature saves a significant number of Nginx reloads which can otherwise affect response latency, load balancing quality (after every reload Nginx resets the state of load balancing) and so on.</p><h4>Nginx + Keepalived — High Availablity Deployment</h4><p>The <a href="https://keepalived.readthedocs.io/en/latest/">keepalived</a> daemon can be used to monitor services or systems and to automatically failover to standby if problems occur. We configure a <a href="https://www.digitalocean.com/community/tutorials/how-to-use-floating-ips-on-digitalocean">floating IP address</a> that can be moved between worker nodes. If a worker goes down, the floating IP will be moved to another worker automatically, allowing nginx to bind to a new IP address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mC5rQ-OAC2GVUUdOwSIM4A.png" /></figure><h4>MetalLB —Nginx with LoadBalancer Service (For the private clusters with a small Public IP address pool)</h4><p>MetalLB hooks into your Kubernetes cluster, and provides a network load-balancer implementation. In short, it allows you to create Kubernetes services of type “LoadBalancer” in clusters that don’t run on a cloud provider. In a cloud-enabled Kubernetes cluster, you request a load-balancer, and your cloud platform assigns an IP address to you. In a bare-metal cluster, MetalLB is responsible for that allocation. Once MetalLB has assigned an external IP address to a service, it needs to make the network beyond the cluster-aware that the IP “lives” in the cluster. MetalLB uses standard routing protocols to achieve this: ARP, NDP, or BGP.</p><p>In layer 2 mode, one machine in the cluster takes ownership of the service and uses standard address discovery protocols (<a href="https://en.wikipedia.org/wiki/Address_Resolution_Protocol">ARP</a> for IPv4, <a href="https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol">NDP</a> for IPv6) to make those IPs reachable on the local network. From the LAN’s point of view, the announcing machine simply has multiple IP addresses.</p><p>In BGP mode, all machines in the cluster establish <a href="https://en.wikipedia.org/wiki/Border_Gateway_Protocol">BGP</a> peering sessions with nearby routers that you control and tell those routers how to forward traffic to the service IPs. Using BGP allows for true load balancing across multiple nodes, and fine-grained traffic control thanks to BGP’s policy mechanisms.</p><p>MetalLB Pods,</p><ul><li>controller is the cluster-wide MetalLB controller, in charge of IP assignment (<strong>deployment</strong>)</li><li>speaker is the per-node daemon that advertises services with assigned IPs using various advertising strategies (<strong>daemonset</strong>)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pn3RNZ_3c1yfF4k9UHjzfg.png" /></figure><p>Note: Metal LB can be used by any services in the cluster by defining the service type as ‘LoadBalancer’ but it is not practical to get a big public IP address pool to use with MetalLB</p><h3>References</h3><ol><li><a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">https://kubernetes.io/docs/concepts/services-networking/ingress/</a></li><li><a href="https://www.nginx.com/products/nginx-ingress-controller/">https://www.nginx.com/products/nginx-ingress-controller/</a></li><li><a href="https://www.keepalived.org/">https://www.keepalived.org/</a></li><li><a href="https://www.envoyproxy.io/">https://www.envoyproxy.io/</a></li><li><a href="https://projectcontour.io/">https://projectcontour.io/</a></li><li><a href="https://metallb.universe.tf/">https://metallb.universe.tf/</a></li></ol><h3>Disclaimer</h3><p>This article does not provide any technical advice or recommendation; if you feel so, it is my personal view, not the company I work for.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dbc5256050a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Path — AirShip]]></title>
            <link>https://dramasamy.medium.com/learning-path-airship-cb41c367b191?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb41c367b191</guid>
            <category><![CDATA[open-dev]]></category>
            <category><![CDATA[open-infrastructure]]></category>
            <category><![CDATA[airship]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[telco]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Thu, 12 Nov 2020 15:11:54 GMT</pubDate>
            <atom:updated>2023-03-14T16:52:26.240Z</atom:updated>
            <content:encoded><![CDATA[<h3>Learning Path — AirShip</h3><p>As infrastructure becomes more integrated into services for companies large and small, it also becomes more complex. This is particularly true for telcos such as AT&amp;T, which is why the company founded the Airship project to help cleanly manage infrastructure. Airship is a collection of loosely coupled but interoperable open source tools that declaratively automate cloud provisioning.</p><p>This series will explore how this brand-new open source project works to streamline the deployment of tools such as OpenStack and Kubernetes. Airship 1.0 is available to try; however, I have not figured out how to install the AirSkiff (Lightweight Airship for Dev) on my CentOS server as the installation scripts are for Ubuntu (Yet to try the nested virtualization).</p><p>As mentioned earlier, Airship is a collection of loosely coupled but interoperable open-source tools; hence it is crucial to understand how those tools work. Let’s start with the theory to understand all Airship components and how it works.</p><p>We will also explore what is new in Airship 2.0 and the open-source tools part of <a href="https://www.airshipit.org/collateral/Airship_2.0_White_Paper.pdf">2.0</a> (Not yet available)</p><h3>Prerequisites</h3><ol><li>OpenStack</li><li>Kubernetes</li><li>Helm</li></ol><p>There are many other prerequisites, but the above ones are mandatory.</p><h3>What is Airship?</h3><p><a href="https://www.airshipit.org/">Airship</a> allows cloud operators to manage the entire lifecycle of sites, including their creation, minor updates, configuration changes, and major uplifts such as OpenStack upgrades. Airship accomplishes this using a unified, declarative, fully containerized, cloud-native platform.</p><h3>Why Airship?</h3><ol><li><strong>Declarative</strong>: Sites are declared using YAML. This includes both hard assets such as network configuration and bare-metal hosts as well as soft assets like helm charts, their overrides, and container images. Operators manage the documents and Airship does the implementation.</li><li><strong>One Workflow for Lifecycle Management</strong>: We need a system that is predictable with lifecycle management at its core. This means one workflow handling both initial deployments and site updates. In another word, new deployment and an update to an existing site should be virtually identical.</li><li><strong>Containers Are the New and Only Unit of Software Delivery</strong>: Containers are the unit of software delivery for Airship. Everything that can be a container is a container. This allows us to progress environments from development to testing and finally production with confidence that the same software is being used.</li><li><strong>Flexible for Different Architectures and Software</strong>: Airship is delivering environments both very small and large with a wide range of configurations. AT&amp;T uses Airship to manage its entire cloud platform, not just OpenStack.</li></ol><h3>AirShip 1.0 Elements</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yPAA6IJm2c-WmtA1yvW5Sw.png" /></figure><ol><li>Create override yaml to customize the default configuration</li><li>Invoke the shipyard command to initiate the site deployment</li></ol><pre>shipyard create action deploy_site</pre><p>3. Use <em>update_site </em>to make changes in the site (Day 2)</p><p>It might be hard to follow now, but we will deepdive in the upcoming articles.</p><h3><strong>Pegleg</strong></h3><p>Pegleg is a document aggregator that provides early linting and validations via <a href="https://airship-deckhand.readthedocs.io/">Deckhand</a>, a document management micro-service within <a href="https://www.airshipit.org/">Airship</a>.</p><p>Pegleg supports local and remote <a href="https://git-scm.com/">Git</a> repositories. Remote repositories can be cloned using a variety of protocols — HTTP(S) or SSH. Afterward, specific revisions within those repositories can be checked out, their documents aggregated, linted, and passed to the rest of <a href="https://www.airshipit.org/">Airship</a> for orchestration, allowing document authors to manage their site definitions using version control.</p><h3>Shipyard</h3><p><a href="https://opendev.org/airship/shipyard">Shipyard</a> is the directed acyclic graph controller for Kubernetes and OpenStack control plane life cycle management.</p><h4><strong>Designs and Secrets</strong></h4><p>Site designs, including the configuration of bare metal host nodes, network design, operating systems, Kubernetes nodes, Armada manifests, Helm charts, and any other descriptors that define the build out of a group of servers enter the Airship via Shipyard. Secrets, such as passwords and certificates, use the same mechanism. The designs and secrets are stored in Airship’s Deckhand, providing for version history and secure storage among other document-based conveniences.</p><h4>Actions</h4><p>Interaction with the site’s control plane is done via invocation of actions in Shipyard. Each action is backed by a workflow implemented as a directed acyclic graph (DAG) that runs using Apache Airflow. Shipyard provides a mechanism to monitor and control the execution of the workflow.</p><p>Shipyard adopts the <strong>Falcon</strong> web framework and uses Apache <strong>Airflow</strong> as the backend engine to programmatically author, schedule and monitor workflows.</p><p>The current workflow is as follows,</p><ol><li>Initial region/site data will be passed to Shipyard from either a human operator or Jenkins</li><li>The data (in YAML format) will be sent to <a href="https://github.com/openstack/airship-deckhand">Deckhand</a> for validation and storage</li><li>Shipyard will make use of the post-processed data from DeckHand to interact with <a href="https://github.com/openstack/airship-drydock">Drydock</a>.</li><li>Drydock will interact with <a href="https://github.com/openstack/airship-promenade">Promenade</a> to provision and deploy bare metal nodes using Ubuntu MAAS and a resilient Kubernetes cluster will be created at the end of the process</li><li>Once the Kubernetes clusters are up and validated to be working properly, Shipyard will interact with <a href="https://github.com/openstack/airship-armada">Armada</a> to deploy OpenStack using <a href="https://github.com/openstack/openstack-helm">OpenStack Helm</a></li><li>Once the OpenStack cluster is deployed, Shipyard will trigger a workflow to perform basic sanity health checks on the cluster</li></ol><h3>Drydock</h3><p>Drydock is a provisioning orchestrator for baremetal servers that translates a YAML-based declaritive site topology into a physical undercloud that can be used for building out a enterprise Kubernetes cluster. It uses plugins to leverage existing provisioning systems to build the servers allowing integration with the provisioning system that best fits the goals and environment of a site.</p><h4>Capabilities</h4><ul><li>Initial IPMI configuration for PXE booting new servers.</li><li>Support for Canonical MAAS provisioning.</li><li>Configuration of complex network topologies including bonding, tagged VLANs and static routes</li><li>Support for running behind a corporate proxy</li><li>Extensible boot action system for placing files and SystemD units on nodes for post-deployment execution</li><li>Supports Keystone-based authentication and authorization</li></ul><h3>Deckhand</h3><p>Deckhand provides document revision management, storage and mutation functionality upon which the rest of the <a href="https://www.airshipit.org/">Airship</a> components rely for orchestration of infrastructure provisioning. Deckhand understands declarative YAML documents that define, end-to-end, the configuration of sites: from the hardware — encompassing network topology and hardware and host profile information — up to the software level that comprises the overcloud.</p><h4>Core Responsibilities</h4><ul><li>layering — helps reduce duplication in configuration by applying the notion of inheritance to documents</li><li>substitution — provides separation between secret data and other configuration data for security purposes and reduces data duplication by allowing common data to be defined once and substituted elsewhere dynamically</li><li>revision history — maintains well-defined collections of documents within immutable revisions that are meant to operate together, while providing the ability to rollback to previous revisions</li><li>validation — allows services to implement and register different kinds of validations and report errors</li><li>secret management — leverages existing OpenStack APIs — namely <a href="https://docs.openstack.org/barbican/latest/api/">Barbican</a> — to reliably and securely store sensitive data</li></ul><h3>Armada</h3><p>Armada is a tool for managing multiple Helm charts with dependencies by centralizing all configurations in a single Armada YAML and providing life-cycle hooks for all Helm releases.</p><p>The Armada Python library and command line tool provide a way to synchronize a Helm (Tiller) target with an operator’s intended state, consisting of several charts, dependencies, and overrides using a single file or directory with a collection of files. This allows operators to define many charts, potentially with different namespaces for those releases, and their overrides in a central place. With a single command, deploy and/or upgrade them where applicable.</p><p>Armada also supports fetching Helm chart source and then building charts from source from various local and remote locations, such as Git endpoints, tarballs or local directories.</p><p>It will also give the operator some indication of what is about to change by assisting with diffs for both values, values overrides, and actual template changes.</p><p>Its functionality extends beyond Helm, assisting in interacting with Kubernetes directly to perform basic pre- and post-steps, such as removing completed or failed jobs, running backup jobs, blocking on chart readiness, or deleting resources that do not support upgrades. However, primarily, it is an interface to support orchestrating Helm.</p><h4>Core Responsibilities</h4><ul><li>Multiple Chart Deployments and Upgrades driven by Armada Manifests</li><li>Manage multiple chart dependencies using Chart Groups</li><li>Enhancing base Helm functionality</li><li>Supports Keystone-based authentication and authorization</li></ul><h3>Promenade</h3><p>Promenade is a tool for bootstrapping a resilient, self-hosted Kubernetes cluster and managing its life-cycle.</p><p>Bootstrapping begins by provisioning a single-node cluster with a complete, configurable Airship infrastructure. After hosts are added to the cluster, the original bootstrapping node can be re-provisioned to avoid subtle differences that could result in future issues.</p><p>Promenade provides cluster resiliency against both node failures and full cluster restarts. It does so by leveraging <a href="https://github.com/kubernetes/helm">Helm</a> charts to manage core Kubernetes assets directly on each host, to ensure their availability.</p><h3>Divingbell</h3><p>Divingbell is a lightweight solution for:</p><ol><li>Bare metal configuration management for a few very targeted use cases</li><li>Bare metal package manager orchestration</li></ol><h4>What problems does it solve?</h4><ol><li>To plug gaps in day 1 tools (e.g., Drydock) for node configuration</li><li>To provide a day 2 solution for managing these configurations going forward</li><li>[Future] To provide a day 2 solution for system level host patchingBerth</li></ol><h3>Open-Source Elements</h3><ol><li>OpenStack</li><li>Docker</li><li>Kubernetes</li><li>Helm</li><li>OpenStack-Helm</li><li>Argo (AS2.0)</li><li>Kustomize (AS2.0)</li><li>Cluster API (AS2.0)</li><li>Metal Kube (AS2.0)</li><li>Ironic (AS 2.0)</li><li>HAProxy</li><li>PostgreSQL</li><li>MariaDB</li><li>Apache AirFlow</li><li><a href="https://maas.io/">MaaS</a> (Metal-As-A-Service)</li><li>Python 3.5+</li></ol><p><em>AS2.0 — AirShip 2.0</em></p><h3>Content Courtesy</h3><p><a href="https://www.airshipit.org/">airshipit.org</a></p><p><a href="https://airshipit.readthedocs.io/">https://airshipit.readthedocs.io/</a></p><p><a href="https://opendev.org/">https://opendev.org/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb41c367b191" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in Kubernetes — Part 3]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-3-dd881476da0f?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd881476da0f</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Thu, 05 Nov 2020 15:07:55 GMT</pubDate>
            <atom:updated>2020-11-05T15:30:13.482Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in Kubernetes — Part 3</h3><p>This is part 3 of the series on <em>Life of a Packet in Kubernetes</em>. We’ll be tackling how Kubernetes’s kube-proxy component uses iptables to control the traffic. It’s important to know the role of kube-proxy in Kubernetes environment and how it uses iptables to control the traffic.</p><p>Note: There are many other plugins/tools to control the traffic flow, but in this article will look at the kube-proxy + iptables combo.</p><p>We’ll start with various communication models provided by Kubernetes and their implementation. If you are already aware of the magic words ‘<strong><em>Service</em></strong><em>, </em><strong><em>ClusterIP</em></strong><em> and </em><strong><em>NodePort</em></strong>’ concept, please jump to the kube-proxy/iptables section.</p><h4><strong>Part 1</strong> —<a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051?source=friends_link&amp;sk=9525ced03e4683afca531c21914625eb"> Basic container networking</a></h4><h4>Part 2 —<a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-2-a07f5bf0ff14?source=friends_link&amp;sk=2e9f25db9d4f5d9ca85ecd3a08db1c07"> Calico CNI</a></h4><h4>Part 3:</h4><ol><li>Pod-to-Pod</li><li>Pod-to-External</li><li>Pod-to-Service</li><li>External-to-Pod</li><li>External Traffic Policy</li><li>Kube-Proxy</li><li>iptable rules processing flow</li><li>Network Policy basics</li></ol><h3>Pod-to-Pod</h3><p>kube-proxy is not involved in Pod-to-Pod communication as the CNI would configure the Node and Pod with required routes. All the containers can communicate with all other containers without NAT; all nodes can communicate with all containers (and vice-versa) without NAT.</p><p><strong>Note</strong>: Pod’s IP address is not static (There are ways to get the static IP, but the default configuration doesn’t guarantee a static IP address). CNI will allocate a new IP address upon Pod restart as CNI never maintains IP address and Pod mapping. FYI, Pod name itself not static in case of <strong><em>Deployment</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiwHL77EjJa_f2j_9gRakg.png" /></figure><p>Practically, the Pods in a Deployment should use a Load-Balancer type of entity to expose the application as the app is stateless, and there will be more than one Pod hosting the application. Load-Balancer type of entity is called ‘<strong>Service</strong>’ in Kubernetes.</p><h3>Pod-to-external</h3><p>For the traffic that goes from pod to external addresses, Kubernetes uses <a href="https://en.wikipedia.org/wiki/Network_address_translation">SNAT</a>. What it does is replace the pod’s internal source IP:port with the host’s IP:port. When the return packet comes back to the host, it rewrites the pod’s IP:port as the destination and sends it back to the original pod. The whole process is transparent to the original pod, who doesn’t know the address translation.</p><h3>Pod-to-Service</h3><h4>ClusterIP</h4><p>Kubernetes has a concept called “service,” which is simply an L4 load balancer in front of pods. There are several different types of services. The most basic type is called ClusterIP. This type of service has a unique VIP address that is only routable inside the cluster.</p><p>It would not be easy to send traffic to a particular application using just pod IPs. The dynamic nature of a Kubernetes cluster means pods can be moved, restarted, upgraded, or scaled in and out of existence. Additionally, some services will have many replicas, so we need some way to load balance between them.</p><p>Kubernetes solves this problem with <em>Services</em>. A Service is an API object that maps a single virtual IP (VIP) to a set of pod IPs. Additionally, Kubernetes provides a DNS entry for each service’s name and virtual IP so that services can be easily addressed by name.</p><p>The mapping of virtual IPs to pod IPs within the cluster is coordinated by the kube-proxy process on each node. This process sets up either iptables or IPVS to automatically translate VIPs into pod IPs before sending the packet out to the cluster network. Individual connections are tracked, so packets can be properly de-translated when they return. IPVS and iptables can load balancing of a single service virtual IP into multiple pod IPs, though IPVS has much more flexibility in the load balancing algorithms it can use. Virtual IP doesn’t actually exist in the system interface; it lives in iptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*eTLb5fFc9TMK_Z0aLB328w.png" /></figure><blockquote>‘Service’ definition from the Kubernetes document — An abstract way to expose an application running on a set of Pods as a network service. With Kubernetes you don’t need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.</blockquote><p><strong>FrontEnd Deployment:</strong></p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: webapp<br>  labels:<br>    app: webapp<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: webapp<br>  template:<br>    metadata:<br>      labels:<br>        app: webapp<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: nginx:1.14.2<br>        imagePullPolicy: IfNotPresent<br>        ports:<br>        - containerPort: 80</pre><p><strong>Backend Deployment:</strong></p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: auth<br>  labels:<br>    app: auth<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: auth<br>  template:<br>    metadata:<br>      labels:<br>        app: auth<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: nginx:1.14.2<br>        imagePullPolicy: IfNotPresent<br>        ports:<br>        - containerPort: 80</pre><p><strong>Service:</strong></p><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: <strong>frontend</strong><br>  labels:<br>    app: frontend<br>spec:<br>  ports:<br>  - port: 80<br>    protocol: TCP<br>  type: ClusterIP<br>  selector:<br>    <strong>app: webapp</strong><br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: <strong>backend</strong><br>  labels:<br>    app: backend<br>spec:<br>  ports:<br>  - port: 80<br>    protocol: TCP<br>  type: ClusterIP<br>  selector:<br>    <strong>app: auth</strong><br>...</pre><p>Now the FrontEnd Pod can connect to the backend via the ClusterIP or the <strong>DNS</strong> entry added by the Kubernetes. A cluster-aware DNS server, such as CoreDNS, watches the Kubernetes API for new Services and creates a set of DNS records for each one. If DNS has been enabled throughout your cluster, all Pods should automatically resolve Services by their DNS name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBeXhZD_qjFR-K_CR9_bgg.png" /></figure><h4>NodePort (External-to-Pod)</h4><p>Now we have the DNS that can be used to communicate between the services in the cluster. However, the external requests can’t reach the service that lives inside the cluster as the IP address are virtual and Private.</p><p>Let’s try to reach the <strong>frontEnd Pod IP address from the external server. </strong>(Note: At this point, no service has been created for the FrontEnd service)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*h56rjw17zc-MA8xPNPHz1w.png" /><figcaption>NodePort 1.1</figcaption></figure><p>Can’t reach the Pod IP as it is a private IP address that can’t be routable.</p><p>Let’s create a NodePort service to expose the FrontEnd service to the external world. If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767). Each node proxies that port (the same port number on every Node) into your Service. Your Service reports the allocated port in its .spec.ports[*].nodePort field.</p><pre><strong>---<br>apiVersion</strong>: v1<br><strong>kind</strong>: Service<br><strong>metadata</strong>:<br>  <strong>name</strong>: frontend<br><strong>spec</strong>:<br>  <strong>type</strong>: NodePort<br>  <strong>selector</strong>:<br>    <strong>app</strong>: <strong>webapp</strong><br>  <strong>ports</strong>:<br>      <em># By default and for convenience, the `targetPort` is set to the same value as the `port` field.</em><br>    - <strong>port</strong>: 80<br>      <strong>targetPort</strong>: 80<br>      <em># Optional field</em><br>      <em># By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)</em><br>      <strong>nodePort</strong>: <strong>31380</strong></pre><pre>...</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*P31o5gFyWLtePg5grMYUAA.png" /><figcaption>NodePort 1.2</figcaption></figure><p>Now we can access the frontend service via &lt;anyClusterNode&gt;:&lt;nodePort&gt;. If you want a specific port number, you can specify a value in the nodePort field. The control plane will either allocate you that port or report that the API transaction failed. This means that you need to take care of possible port collisions yourself. You also have to use a valid port number, one that&#39;s inside the range configured for NodePort use.</p><h3>External Traffic Policy</h3><blockquote>ExternalTrafficPolicy denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. “Local” preserves the client source IP and avoids a second hop for NodePort type services, but risks potentially imbalanced traffic spreading. “Cluster” obscures the client source IP and may cause a second hop to another node, but should have good overall load-balancing</blockquote><h4>Cluster Traffic Policy</h4><p>This is the <strong>default</strong> external traffic policy for Kubernetes Services. The assumption here is that you always want to route traffic to all pods (across all the nodes) running a service with equal distribution.</p><p>One of the caveats of using this policy is that you may see unnecessary network hops between nodes as you ingress external traffic. For example, if you receive external traffic via a NodePort, the NodePort SVC may (randomly) route traffic to a pod on another host when it could have routed traffic to a pod on the same host, avoiding that extra hop out to the network.</p><p>Packet flow in <strong>Cluster</strong> traffic policy is as follows,</p><ul><li>client sends the packet to node2:31380</li><li>node2 replaces the source IP address (SNAT) in the packet with its own IP address</li><li>node2 replaces the destination IP on the packet with the pod IP</li><li>packet is routed to node 1 or 3, and then to the endpoint</li><li>the pod’s reply is routed back to node2</li><li>the pod’s reply is sent back to the client</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*ANQuIVp-HNnEBpoROrZ5Pw.png" /><figcaption>NodePort 1.3</figcaption></figure><h4>Local Traffic Policy</h4><p>With this external traffic policy, kube-proxy will add proxy rules on a specific NodePort (30000–32767) only for pods that exist on the same node (local) instead of every pod for a service regardless of where it was placed.</p><p>You’ll notice that if you try to set externalTrafficPolicy: Local on your Service, the Kubernetes API will require you are using the LoadBalancer or NodePort type. This is because the “Local” external traffic policy is only relevant for external traffic, which only applies to those two types.</p><p>If you set service.spec.externalTrafficPolicy to the value Local, kube-proxy only proxies proxy requests to local endpoints and does not forward traffic to other nodes. This approach preserves the original source IP address. If there are no local endpoints, packets sent to the node are dropped, so you can rely on the correct source-ip in any packet processing rules you might apply a packet that makes it through to the endpoint.</p><pre><strong>---<br>apiVersion</strong>: v1<br><strong>kind</strong>: Service<br><strong>metadata</strong>:<br>  <strong>name</strong>: frontend<br><strong>spec</strong>:<br>  <strong>type</strong>: NodePort<br>  <strong>externalTrafficPolicy</strong>: Local<br>  <strong>selector</strong>:<br>    <strong>app</strong>: <strong>webapp</strong><br>  <strong>ports</strong>:<br>      <em># By default and for convenience, the `targetPort` is set to the same value as the `port` field.</em><br>    - <strong>port</strong>: 80<br>      <strong>targetPort</strong>: 80<br>      <em># Optional field</em><br>      <em># By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)</em><br>      <strong>nodePort</strong>: 31380</pre><pre>...</pre><p>Packet flow in <strong>Local</strong> traffic policy as follows,</p><ul><li>client sends the packet to node1:31380, which <em>does</em> have endpoints</li><li>node1 routes packet to the endpoint with the correct source IP</li><li>node1 won’t route the packet to node3 as the policy is Local</li><li>the client sends a packet to node2:31380, which doesn&#39;t have any endpoints</li><li>packet is <strong>dropped</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-IS_Hzwwhwk6GpfiPjfKw.png" /><figcaption>NodePort 1.4</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*Q6JTBRBHzo-fl7EDxLvf6Q.png" /><figcaption>NodePort 1.5</figcaption></figure><h4>Local traffic policy in LoadBalancer Service type</h4><p>If you’re running on Google Kubernetes Engine/GCE, setting the same service.spec.externalTrafficPolicy field to Local forces nodes <em>without</em> Service endpoints to remove themselves from the list of nodes eligible for load-balanced traffic by deliberately failing health checks. So there won’t be any traffic drops. This model is great for applications that ingress a lot of external traffic and avoid unnecessary hops on the network to reduce latency. We can also preserve true client IPs since we no longer need SNAT traffic from a proxying node! However, the biggest downsides to using the “Local” external traffic policy, as mentioned in the Kubernetes docs, is that traffic to your application may be imbalanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qk4hRbd0E1b-JxRR0_bFQ.png" /></figure><h3>Kube-Proxy (iptable mode)</h3><p>The component in Kubernetes that implements ‘Service’ is called <strong>kube-proxy.</strong> It sits on every node and programs complicated iptables rules to do all kinds of filtering and NAT between pods and services. If you go to a Kubernetes node and type iptables-save, you’ll see the rules inserted by Kubernetes or other programs. The most important chains are KUBE-SERVICES, KUBE-SVC-* and KUBE-SEP-*.</p><ul><li>KUBE-SERVICES is the entry point for service packets. What it does is that match the destination IP:port and dispatch the packet to the corresponding KUBE-SVC-* chain.</li><li>KUBE-SVC-* chain acts as a load balancer and distributes the packet to KUBE-SEP-*chain equally. Each KUBE-SVC-* has the same number of KUBE-SEP-* chains as the number of endpoints behind it.</li><li>KUBE-SEP-* chain represents a <strong>S</strong>ervice <strong>E</strong>nd<strong>P</strong>oint. It simply does DNAT, replacing service IP:port with pod&#39;s endpoint IP:Port.</li></ul><p>For DNAT, conntrack kicks in and tracks the connection state using a state machine. The state is needed because it needs to remember the destination address it changed to, and changed it back when the returning packet came back. Iptables could also rely on the conntrack state (ctstate) to decide the destiny of a packet. Those 4 conntrack states are especially important:</p><ul><li><em>NEW</em>: conntrack knows nothing about this packet, which happens when the SYN packet is received.</li><li><em>ESTABLISHED</em>: conntrack knows the packet belongs to an established connection, which happens after the handshake is complete.</li><li><em>RELATED</em>: The packet doesn’t belong to any connection, but it is affiliated to another connection, which is especially useful for protocols like FTP.</li><li><em>INVALID</em>: Something is wrong with the packet, and conntrack doesn’t know how to deal with it. This state plays a centric role in this Kubernetes issue.</li></ul><p>This is how the TCP connection works between pod and service; The sequence of events is:</p><ul><li>Client pod from the left-hand side sends a packet to a service: 2.2.2.10:80</li><li>The packet is going through iptables rules in the client node, and the destination is changed to pod IP, 1.1.1.20:80</li><li>Server pod handles the packet and sends back a packet with destination 1.1.1.10</li><li>The packet is going back to the client node, conntrack recognizes the packet and rewrites the source address back to 2.2.2.10:80</li><li>Client pod receives the response packet</li></ul><p><strong>GIF visualization:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1u1d1WU1SqTiiJFyHO-X_A.gif" /></figure><h3>iptables</h3><p>In the Linux operating system, the firewalling is taken care of using netfilter. Which is a kernel module that decides what packets are allowed to come in or to go outside.iptables are just the interface to netfilter. The two might often be thought of as the same thing. A better perspective would be to think of it as a backend (netfilter) and a frontend (iptables).</p><h4>chains</h4><p>Each chain is responsible for a specific task,</p><ul><li><strong>PREROUTING</strong>: This chain decides what happens to a packet as soon as it arrives at the network interface. We have different options, such as altering the packet (for NAT probably), dropping a packet, or doing nothing at all and letting it slip and be handled elsewhere along the way.</li><li><strong>INPUT</strong>: This is one of the popular chains as it almost always contains strict rules to avoid some evildoers on the internet harming our computer. If you want to open/block a port, this is where you’d do it.</li><li><strong>FORWARD</strong>: This chain is responsible for packet forwarding. Which is what the name suggests. We may want to treat a computer as a router, and this is where some rules might apply to do the job.</li><li><strong>OUTPUT</strong>: This chain is the one responsible for all your web browsing among many others. You can’t send a single packet without this chain allowing it. You have a lot of options, whether you want to allow a port to communicate or not. It’s the best place to limit your outbound traffic if you’re not sure what port each application is communicating through.</li><li><strong>POSTROUTING</strong>: This chain is where packets leave their trace last, before leaving our computer. This is used for routing among many other tasks just to make sure the packets are treated the way we want them to.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*DceFvQRD6B42oGSIMezHYg.png" /></figure><p><strong>FORWARD</strong> chain only works if the <em>ip_forward</em> enabled in the Linux server, that’s the reason the following command is important while setting up and debugging the Kubernetes cluster.</p><pre>node-1# sysctl -w net.ipv4.ip_forward=1<br>net.ipv4.ip_forward = 1</pre><pre>node-1# cat /proc/sys/net/ipv4/ip_forward<br>1</pre><p>The above change is not persistent. To permanently enable the IP forwarding on your Linux system, edit /etc/sysctl.conf and add the following line:</p><pre>net.ipv4.ip_forward = 1</pre><h3>tables</h3><p>We are going to focus on the NAT table, but the following are the available tables.</p><ul><li><strong>Filter</strong>: This is the default table. In this table, you would decide whether a packet is allowed in/out of your computer. If you want to block a port to stop receiving anything, this is your stop.</li><li><strong>Nat</strong>: This table is the second most popular table and is responsible for creating a new connection. Which is shorthand for Network Address Translation. And if you’re not familiar with the term, don’t worry. I’ll give you an example below.</li><li><strong>Mangle</strong>: For specialized packets only. This table is for changing something inside the packet either before coming in or leaving out.</li><li><strong>Raw</strong>: This table is dealing with the raw packet, as the name suggests. Mainly this is for tracking the connection state. We’ll see examples of this below when we want to allow success packets from SSH connection.</li><li><strong>Security</strong>: It is responsible for securing your computer after the filter table. Which consists of <a href="https://www.redhat.com/en/topics/linux/what-is-selinux">SELinux</a>. If you’re not familiar with the term, it’s a powerful security tool on modern Linux distributions.</li></ul><blockquote>Please read <a href="https://www.digitalocean.com/community/tutorials/a-deep-dive-into-iptables-and-netfilter-architecture">THIS</a> article for more detailed info on iptables.</blockquote><h3>iptable configuration in Kubernetes</h3><p>Let’s deploy an <strong>Nginx</strong> application with replica count two in minikube and dump the iptable rules.</p><p>ServiceType: <strong>NodePort</strong></p><pre>master # kubectl get svc webapp<br>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>webapp NodePort <strong>10.103.46.104</strong> &lt;none&gt; 80:31380/TCP 3d13h<br>master # kubectl get ep webapp <br>NAME ENDPOINTS AGE<br>webapp <strong>10.244.120.102</strong>:80,<strong>10.244.120.103</strong>:80 3d13h<br>master # </pre><p>The ClusterIP doesn’t exist anywhere, its a virtual IP exists in iptable Kubernetes adds a DNS entry in CoreDNS.</p><pre>master # kubectl exec -i -t dnsutils -- nslookup webapp.default</pre><pre>Server:  10.96.0.10</pre><pre>Address: 10.96.0.10#53</pre><pre>Name: webapp.default.svc.cluster.local</pre><pre>Address: <strong>10.103.46.104</strong></pre><p>To hook into packet filtering and NAT, Kubernetes will create a custom chain KUBE-SERVICES from iptables; it will redirect all PREROUTING AND OUTPUT traffic to custom chain KUBE-SERVICES, refer to below,</p><pre>$ sudo iptables -t nat -L PREROUTING | column -t<br>Chain            PREROUTING  (policy  ACCEPT)                                                                    <br>target           prot        opt      source    destination                                                      <br>cali-PREROUTING  all         --       anywhere  anywhere     /*        cali:6gwbT8clXdHdC1b1  */                 <br><strong>KUBE-SERVICES</strong>    all         --       anywhere  anywhere     /*        kubernetes             service   portals  */<br>DOCKER           all         --       anywhere  anywhere     ADDRTYPE  match                  dst-type  LOCAL</pre><p>After using KUBE-SERVICES chain hook into packet filtering and NAT, Kubernetes can inspect traffics to its services and apply SNAT/DNAT accordingly. At the end of the KUBE-SERVICES chain, it will install another custom chain KUBE-NODEPORTS to handle traffics for a specific service type NodePort.</p><p>If the traffic is for ClusterIP, the <strong>KUBE-SVC-2IRACUALRELARSND </strong>chain will process the traffic; else, the next chain will process the traffic, that is <strong>KUBE-NODEPORTS.</strong></p><pre>$ sudo iptables -t nat -L KUBE-SERVICES | column -t<br>Chain                      KUBE-SERVICES  (2   references)                                                                                                                                                                             <br>target                     prot           opt  source          destination                                                                                                                                                             <br><strong>KUBE-MARK-MASQ</strong>             tcp            --   !10.244.0.0/16  10.103.46.104   /*  default/webapp                   cluster  IP          */     tcp   dpt:www                                                                          <br><strong>KUBE-SVC-2IRACUALRELARSND</strong>  tcp            --   anywhere        10.103.46.104   /*  default/webapp                   cluster  IP          */     tcp   dpt:www                                                                                                                                             <br><strong>KUBE-NODEPORTS</strong>             all            --   anywhere        anywhere        /*  kubernetes                       service  nodeports;  NOTE:  this  must        be  the  last  rule  in  this  chain  */  ADDRTYPE  match  dst-type  LOCAL</pre><p>Let’s check what the chains are part of <strong>KUBE-NODEPORTS,</strong></p><pre>$ sudo iptables -t nat -L <strong>KUBE-NODEPORTS</strong> | column -t<br>Chain                      KUBE-NODEPORTS  (1   references)                                            <br>target                     prot            opt  source       destination                               <br>KUBE-MARK-MASQ             tcp             --   anywhere     anywhere     /*  default/webapp  */  tcp  dpt:31380<br><strong>KUBE-SVC-2IRACUALRELARSND</strong>  tcp             --   anywhere     anywhere     /*  default/<strong>webapp</strong>  */  tcp  dpt:31380</pre><p>From this point, the processing is the same for ClusterIP and NodePort. Please take a look at the iptable flow diagram as follows.</p><pre><strong># statistic  mode  random -&gt; Random load-balancing between endpoints.</strong><br>$ sudo iptables -t nat -L <strong>KUBE-SVC-2IRACUALRELARSND</strong> | column -t<br>Chain                      KUBE-SVC-2IRACUALRELARSND  (2   references)                                                                             <br>target                     prot                       opt  source       destination                                                                <br><strong>KUBE-SEP-AO6KYGU752IZFEZ4</strong>  all                        --   anywhere     anywhere     /*  default/webapp  */  statistic  mode  random  probability  0.50000000000<br><strong>KUBE-SEP-PJFBSHHDX4VZAOXM</strong>  all                        --   anywhere     anywhere     /*  default/webapp  */</pre><pre><br>$ sudo iptables -t nat -L KUBE-SEP-AO6KYGU752IZFEZ4 | column -t<br>Chain           KUBE-SEP-AO6KYGU752IZFEZ4  (1   references)                                               <br>target          prot                       opt  source          destination                               <br>KUBE-MARK-MASQ  all                        --   10.244.120.102<strong> </strong> anywhere     /*  default/webapp  */       <br><strong>DNAT</strong>            tcp                        --   anywhere        anywhere     /*  default/webapp  */  tcp  <strong>to:10.244.120.102:80</strong></pre><pre><br>$ sudo iptables -t nat -L KUBE-SEP-PJFBSHHDX4VZAOXM | column -t<br>Chain           KUBE-SEP-PJFBSHHDX4VZAOXM  (1   references)                                               <br>target          prot                       opt  source          destination                               <br>KUBE-MARK-MASQ  all                        --   10.244.120.103  anywhere     /*  default/webapp  */       <br><strong>DNAT</strong>            tcp                        --   anywhere        anywhere     /*  default/webapp  */  tcp  <strong>to:10.244.120.103:80</strong></pre><pre><br>$ sudo iptables -t nat -L KUBE-MARK-MASQ | column -t<br>Chain   KUBE-MARK-MASQ  (24  references)                         <br>target  prot            opt  source       destination            <br>MARK    all             --   anywhere     anywhere     MARK  or  0x4000</pre><p><em>Note: Trimmed the output to show only the required rules for readability.</em></p><p><strong>ClusterIP</strong>:</p><blockquote>KUBE-SERVICES → KUBE-SVC-XXX → KUBE-SEP-XXX</blockquote><p><strong>NodePort</strong>:</p><blockquote>KUBE-SERVICES → KUBE-NODEPORTS → KUBE-SVC-XXX → KUBE-SEP-XXX</blockquote><p>Note: The NodePort service will have a ClusterIP assigned to handle internal and external traffic.</p><p>Visual representation of above iptable rules,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*-uNjKPq2E9wJnGid22fguA.png" /></figure><h4>ExtrenalTrafficPolicy: Local</h4><p>As discussed before, using “externalTrafficPolicy: Local” will preserve source IP and drop packets from the agent node has no local endpoint. Let’s take a look at the iptable rules in the node with no local endpoint.</p><pre>master # kubectl get nodes<br>NAME           STATUS   ROLES    AGE    VERSION<br>minikube       Ready    master   6d1h   v1.19.2<br>minikube-m02   Ready    &lt;none&gt;   85m    v1.19.2</pre><p>Deploy Nginx with externalTrafficPolicy Local.</p><pre>master # kubectl get pods nginx-deployment-7759cc5c66-p45tz -o wide<br>NAME                                READY   STATUS    RESTARTS   AGE   IP               NODE       NOMINATED NODE   READINESS GATES<br>nginx-deployment-7759cc5c66-p45tz   1/1     Running   0          29m   10.244.120.111   <strong>minikube</strong>   &lt;none&gt;           &lt;none&gt;</pre><p>Check externalTrafficPolicy,</p><pre>master # kubectl get svc webapp -o wide -o jsonpath={.spec.externalTrafficPolicy}<br><strong>Local</strong></pre><p>Get the service,</p><pre>master # kubectl get svc webapp -o wide<br>NAME     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR<br>webapp   NodePort   10.111.243.62   &lt;none&gt;        80:30080/TCP   29m   app=webserver</pre><p>Let’s check the iptable rules in node <em>minikube-m02; there</em> should be a DROP rule to drop the packets as there is no local endpoint.</p><pre>$ sudo iptables -t nat -L KUBE-NODEPORTS</pre><pre>Chain KUBE-NODEPORTS (1 references)</pre><pre>target prot opt source destination</pre><pre>KUBE-MARK-MASQ tcp — 127.0.0.0/8 anywhere /* default/webapp */ tcp dpt:30080</pre><pre><strong>KUBE-XLB-2IRACUALRELARSND</strong> tcp — anywhere anywhere /* default/webapp */ tcp dpt:30080</pre><p>Check <strong>KUBE-XLB-2IRACUALRELARSND </strong>chain<strong>,</strong></p><pre>$ sudo iptables -t nat -L KUBE-XLB-2IRACUALRELARSND</pre><pre>Chain KUBE-XLB-2IRACUALRELARSND (1 references)</pre><pre>target prot opt source destination</pre><pre>KUBE-SVC-2IRACUALRELARSND all — 10.244.0.0/16 anywhere /* Redirect pods trying to reach external loadbalancer VIP to clusterIP */</pre><pre>KUBE-MARK-MASQ all — anywhere anywhere /* masquerade LOCAL traffic for default/webapp LB IP */ ADDRTYPE match src-type LOCAL</pre><pre>KUBE-SVC-2IRACUALRELARSND all — anywhere anywhere /* route LOCAL traffic for default/webapp LB IP to service chain */ ADDRTYPE match src-type LOCAL</pre><pre><strong>KUBE-MARK-DROP </strong>all — anywhere anywhere /* default/webapp has no local endpoints */</pre><p>If you take a closer look, there is no issue with the Cluster level traffic; only the nodePort traffic will be dropped on this node.</p><p>‘<strong>minikube</strong>’ node iptable rules,</p><pre>$ sudo iptables -t nat -L KUBE-NODEPORTS</pre><pre>Chain KUBE-NODEPORTS (1 references)</pre><pre>target prot opt source destination</pre><pre>KUBE-MARK-MASQ tcp — 127.0.0.0/8 anywhere /* default/webapp */ tcp dpt:30080</pre><pre>KUBE-XLB-2IRACUALRELARSND tcp — anywhere anywhere /* default/webapp */ tcp dpt:30080</pre><pre>$ sudo iptables -t nat -L KUBE-XLB-2IRACUALRELARSND</pre><pre>Chain KUBE-XLB-2IRACUALRELARSND (1 references)</pre><pre>target prot opt source destination</pre><pre>KUBE-SVC-2IRACUALRELARSND all — 10.244.0.0/16 anywhere /* Redirect pods trying to reach external loadbalancer VIP to clusterIP */</pre><pre>KUBE-MARK-MASQ all — anywhere anywhere /* masquerade LOCAL traffic for default/webapp LB IP */ ADDRTYPE match src-type LOCAL</pre><pre>KUBE-SVC-2IRACUALRELARSND all — anywhere anywhere /* route LOCAL traffic for default/webapp LB IP to service chain */ ADDRTYPE match src-type LOCAL</pre><pre>KUBE-SEP-5T4S2ILYSXWY3R2J all — anywhere anywhere /* Balancing rule 0 for default/webapp */</pre><pre>$ sudo iptables -t nat -L KUBE-SVC-2IRACUALRELARSND</pre><pre>Chain KUBE-SVC-2IRACUALRELARSND (3 references)</pre><pre>target prot opt source destination</pre><pre>KUBE-SEP-5T4S2ILYSXWY3R2J all — anywhere anywhere /* default/webapp */</pre><h3>Headless Services</h3><p><em>-Copied from Kubernetes documentation-</em></p><p>Sometimes you don’t need load-balancing and a single service IP. In this case, you can create what is termed “headless” Services by explicitly specifying &quot;None&quot; for the cluster IP (.spec.clusterIP).</p><p>You can use a headless Service to interface with other service discovery mechanisms without being tied to Kubernetes’ implementation.</p><p>For headless Services, a cluster IP is not allocated, kube-proxy does not handle these Services, and there is no load balancing or proxying done by the platform. How DNS is automatically configured depends on whether the Service has selectors defined:</p><h4>With selectors</h4><p>For headless services that define selectors, the endpoints controller creates Endpoints records in the API, and modifies the DNS configuration to return records (addresses) that point directly to the Pods backing the Service.</p><pre>master # kubectl get svc webapp-hs</pre><pre>NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE</pre><pre>webapp-hs   ClusterIP   <strong>None</strong>         &lt;none&gt;        80/TCP    24s</pre><pre>master # kubectl get ep webapp-hs</pre><pre>NAME        ENDPOINTS                             AGE</pre><pre>webapp-hs   <strong>10.244.120.109:80,10.244.120.110:80</strong>   31s</pre><h4>Without selectors</h4><p>For headless services that do not define selectors, the endpoints controller does not create Endpoints records. However, the DNS system looks for and configures either:</p><ul><li>CNAME records for <a href="https://kubernetes.io/docs/concepts/services-networking/service/#externalname">ExternalName</a>-type Services.</li><li>A records for any Endpoints that share a name with the Service for all other types.</li></ul><p>If there are external IPs that route to one or more cluster nodes, Kubernetes Services can be exposed on those externalIPs. Traffic that ingresses into the cluster with the external IP (as the destination IP) on the Service port will be routed to one of the Service endpoints. externalIPsare not managed by Kubernetes and are the responsibility of the cluster administrator.</p><h3>Network Policy</h3><p>By now, you might have got an idea of how the network policy is implemented in Kubernetes. Yes, the <strong>iptables</strong> again; this time, the CNI takes care of implementing the network policy, not the <strong>kube-proxy. </strong>This section should have been added to the Calico (Part 2); however, I feel this is the right place to have the network policy details.</p><p>Let’s create three services — frontend, backend, and db.</p><p>By default, pods are non-isolated; they accept traffic from any source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*ChWgesRJcLqOQY3xnE0EPA.png" /></figure><p>However, there should be a traffic policy to isolate the DB pods from the FrontEnd pods to avoid any traffic flow between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*tqFTrpWy1zrT6crG7oQNlQ.png" /></figure><p>I would suggest you read <a href="https://www.stackrox.com/post/2019/04/setting-up-kubernetes-network-policies-a-detailed-guide/">THIS</a> article to understand the Network Policy configuration. This section will focus on how the network policy is implemented in Kubernetes instead of configuration deep dive.</p><p>I have applied a network policy to isolate <em>db</em> from the <em>frontend</em> pods; this results in no connection between the <em>frontend</em> and <em>db</em> pods.</p><p>Note: Above picture shows the ‘service’ symbol instead of the ‘pod’ symbol to make life easier as there can be many pods in a given service. But, the actual rules are applied per <strong>Pod</strong>.</p><pre>master # kubectl exec -it <strong>frontend</strong>-8b474f47-zdqdv -- /bin/sh</pre><pre># curl <strong>backend</strong></pre><pre>backend-867fd6dff-mjf92</pre><pre># curl <strong>db</strong></pre><pre>curl: (7) Failed to connect to db port 80: Connection timed out</pre><p>However, the <em>backend</em> can reach the <em>db</em> service without any issue.</p><pre>master # kubectl exec -it <strong>backend</strong>-867fd6dff-mjf92 -- /bin/sh</pre><pre># curl <strong>db</strong></pre><pre>db-8d66ff5f7-bp6kf</pre><p>Let’s take a look at the NetworkPolicy — Allow ingress from the service if it has a label ‘<strong>allow-db-access</strong>’ set to ‘<strong>true.</strong>’</p><pre>---<br>apiVersion: networking.k8s.io/v1<br>kind: NetworkPolicy<br>metadata:<br>  name: allow-db-access<br>spec:<br>  podSelector:<br>    matchLabels:<br>      <strong>app: &quot;db&quot;</strong><br>  policyTypes:<br>  - Ingress<br>  <strong>ingress</strong>:<br>  - from:<br>    - podSelector:<br>        matchLabels:<br>          <strong>networking/allow-db-access: &quot;true&quot;</strong><br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: backend<br>  labels:<br>    app: backend<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: backend<br>  template:<br>    metadata:<br>      labels:<br>        app: backend<br>        <strong>networking/allow-db-access: &quot;true&quot;</strong><br>    spec:<br>      volumes:<br>      - name: workdir<br>        emptyDir: {}<br>      containers:<br>      - name: nginx<br>        image: nginx:1.14.2<br>        imagePullPolicy: IfNotPresent<br>        ports:<br>        - containerPort: 80<br>        volumeMounts:<br>        - name: workdir<br>          mountPath: /usr/share/nginx/html<br>      initContainers:<br>      - name: install<br>        image: busybox<br>        imagePullPolicy: IfNotPresent<br>        command: [&#39;sh&#39;, &#39;-c&#39;, &quot;echo $HOSTNAME &gt; /work-dir/index.html&quot;]<br>        volumeMounts:<br>        - name: workdir<br>          mountPath: &quot;/work-dir&quot;<br>...</pre><p>Calico converts the Kubernetes network policy into Calico’s native format,</p><pre>master # calicoctl get networkPolicy --output yaml<br>apiVersion: projectcalico.org/v3<br>items:<br>- apiVersion: projectcalico.org/v3<br>  kind: NetworkPolicy<br>  metadata:<br>    creationTimestamp: &quot;2020-11-05T05:26:27Z&quot;<br>    name: knp.default.allow-db-access<br>    namespace: default<br>    resourceVersion: /53872<br>    uid: 1b3eb093-b1a8-4429-a77d-a9a054a6ae90<br>  spec:<br>    ingress:<br>    - action: Allow<br>      destination: {}<br>      source:<br>        selector: projectcalico.org/orchestrator == &#39;k8s&#39; &amp;&amp; networking/allow-db-access<br>          == &#39;true&#39;<br>    order: 1000<br>    selector: projectcalico.org/orchestrator == &#39;k8s&#39; &amp;&amp; app == &#39;db&#39;<br>    types:<br>    - Ingress<br>kind: NetworkPolicyList<br>metadata:<br>  resourceVersion: 56821/56821</pre><p>The iptables rule plays an important role in enforcing the policy by using the ‘<em>filter</em>’ table. It’s hard to do reverse engineering as the Calico uses advanced concepts like <em>ipset</em>. From the iptables rules, I see that the packets are allowed to <em>db</em> pod only if the packets are from the backend, and that’s exactly our network policy is.</p><p>Get the workload endpoint details from the calicoctl.</p><pre>master # calicoctl get workloadEndpoint<br>WORKLOAD                         NODE       NETWORKS        INTERFACE         <br>backend-867fd6dff-mjf92          minikube   10.88.0.27/32   cali2b1490aa46a   <br>db-8d66ff5f7-bp6kf               minikube   10.88.0.26/32   <strong>cali95aa86cbb2a </strong>  <br>frontend-8b474f47-zdqdv          minikube   10.88.0.24/32   cali505cfbeac50</pre><p><strong><em>cali95aa86cbb2a</em></strong> — Host side end of veth pair that is in use by <em>db</em> pod.</p><p>Let’s get the iptables rules related to this interface.</p><pre>$ sudo iptables-save | grep cali95aa86cbb2a<br>:cali-fw-cali95aa86cbb2a - [0:0]<br>:cali-tw-cali95aa86cbb2a - [0:0]<br>-A cali-from-wl-dispatch -i cali95aa86cbb2a -m comment --comment &quot;cali:R489GtivXlno-SCP&quot; -g cali-fw-cali95aa86cbb2a<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:3XN24uu3MS3PMvfM&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:xyfc0rlfldUi6JAS&quot; -m conntrack --ctstate INVALID -j DROP<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:wG4_76ot8e_QgXek&quot; -j MARK --set-xmark 0x0/0x10000<br>-A cali-fw-cali95aa86cbb2a -p udp -m comment --comment &quot;cali:Ze6pH1ZM5N1pe76G&quot; -m comment --comment &quot;Drop VXLAN encapped packets originating in pods&quot; -m multiport --dports 4789 -j DROP<br>-A cali-fw-cali95aa86cbb2a -p ipencap -m comment --comment &quot;cali:3bjax7tRUEJ2Uzew&quot; -m comment --comment &quot;Drop IPinIP encapped packets originating in pods&quot; -j DROP<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:0pCFB_VsKq1qUOGl&quot; -j cali-pro-kns.default<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:mbgUOxlInVlwb2Ie&quot; -m comment --comment &quot;Return if profile accepted&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:I7GVOQegh6Wd9EMv&quot; -j cali-pro-ksa.default.default<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:g5ViWVLiyVrKX91C&quot; -m comment --comment &quot;Return if profile accepted&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br>-A cali-fw-cali95aa86cbb2a -m comment --comment &quot;cali:RBmQDo38EoPmxJ0I&quot; -m comment --comment &quot;Drop if no profiles matched&quot; -j DROP<br>-A cali-to-wl-dispatch -o cali95aa86cbb2a -m comment --comment &quot;cali:v3sEoNToLYUOg7M6&quot; -g cali-tw-cali95aa86cbb2a<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:eCrqwxNk3cKw9Eq6&quot; -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:_krp5nzavhAu5avJ&quot; -m conntrack --ctstate INVALID -j DROP<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:Cu-tVtfKKu413YTT&quot; -j MARK --set-xmark 0x0/0x10000<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:leBL64hpAXM9y4nk&quot; -m comment --comment &quot;Start of policies&quot; -j MARK --set-xmark 0x0/0x20000<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:pm-LK-c1ra31tRwz&quot; -m mark --mark 0x0/0x20000 -j <strong>cali-pi-_tTE-E7yY40ogArNVgKt</strong><br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:q_zG8dAujKUIBe0Q&quot; -m comment --comment &quot;Return if policy accepted&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:FUDVBYh1Yr6tVRgq&quot; -m comment --comment &quot;Drop if no policies passed packet&quot; -m mark --mark 0x0/0x20000 -j DROP<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:X19Z-Pa0qidaNsMH&quot; -j cali-pri-kns.default<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:Ljj0xNidsduxDGUb&quot; -m comment --comment &quot;Return if profile accepted&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:0z9RRvvZI9Gud0Wv&quot; -j cali-pri-ksa.default.default<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:pNCpK-SOYelSULC1&quot; -m comment --comment &quot;Return if profile accepted&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br>-A cali-tw-cali95aa86cbb2a -m comment --comment &quot;cali:sMkvrxvxj13WlTMK&quot; -m comment --comment &quot;Drop if no profiles matched&quot; -j DROP</pre><pre>$ sudo iptables-save -t filter | grep cali-pi-_tTE-E7yY40ogArNVgKt<br>:cali-pi-_tTE-E7yY40ogArNVgKt - [0:0]<br>-A cali-pi-_tTE-E7yY40ogArNVgKt -m comment --comment &quot;cali:M4Und37HGrw6jUk8&quot; -m set --match-set <strong>cali40s:LrVD8vMIGQDyv8Y7sPFB1Ge</strong> src -j MARK --set-xmark 0x10000/0x10000<br>-A cali-pi-_tTE-E7yY40ogArNVgKt -m comment --comment &quot;cali:sEnlfZagUFRSPRoe&quot; -m mark --mark 0x10000/0x10000 -j RETURN<br></pre><p>By checking the <em>ipset, </em>it is clear that the ingress to db pod allowed only from the backend pod IP <strong>10.88.0.27</strong></p><pre>[root@minikube /]# ipset list<br><strong>Name: cali40s:LrVD8vMIGQDyv8Y7sPFB1Ge</strong><br>Type: hash:net<br>Revision: 6<br>Header: family inet hashsize 1024 maxelem 1048576<br>Size in memory: 408<br>References: 3<br>Number of entries: 1<br>Members:<br><strong>10.88.0.27</strong></pre><p>I’ll update <a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-2-a07f5bf0ff14">Part 2</a> of this series with more detailed steps to decode the calico iptables rules.</p><h3>References:</h3><pre><a href="https://kubernetes.io/">https://kubernetes.io</a><br><a href="https://www.projectcalico.org/">https://www.projectcalico.org/</a><br><a href="https://rancher.com/">https://rancher.com/</a> <br><a href="http://www.netfilter.org/">http://www.netfilter.org/</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd881476da0f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in Kubernetes — Part 2]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-2-a07f5bf0ff14?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a07f5bf0ff14</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[calico]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Mon, 19 Oct 2020 03:10:11 GMT</pubDate>
            <atom:updated>2020-10-29T05:44:06.510Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in Kubernetes — Part 2</h3><p>As we discussed in <a href="https://medium.com/@dramasamy/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051">Part 1</a>, CNI plugins play an essential role in Kubernetes networking. There are many third-party CNI plugins available today; Calico is one of them. Many engineers prefer Calico; one of the main reasons is its ease of use and how it shapes the network fabric.</p><p>Calico supports a broad range of platforms, including Kubernetes, OpenShift, Docker EE, OpenStack, and bare metal services. The Calico node runs in a Docker container on the Kubernetes master node and on each Kubernetes worker node in the cluster. The calico-cni plugin integrates directly with the Kubernetes kubelet process on each node to discover which Pods are created and add them to Calico networking.</p><p>We will talk about installation, Calico modules (Felix, BIRD, and Confd), and routing modes.</p><p><strong>What is not covered? </strong>Network policy — It needs a separate article, therefore skipping that for now.</p><h3>Topics — Part 2</h3><ol><li>Requirements</li><li>Modules and its functions</li><li>Routing modes</li><li>Installation (calico and calicoctl)</li></ol><h3>CNI Requirements</h3><ol><li>Create veth-pair and move the same inside container</li><li>Identify the right POD CIDR</li><li>Create a CNI configuration file</li><li>Assign and manage IP address</li><li>Add default routes inside the container</li><li>Advertise the routes to all the peer nodes (Not applicable for VxLan)</li><li>Add routes in the HOST server</li><li>Enforce Network Policy</li></ol><p>There are many other requirements too, but the above ones are the basic. Let’s take a look at the routing table in the Master and Worker node. Each node has a container with an IP address and default container route.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QDQXN3-9b3aRWGkh4Kyx7g.png" /><figcaption>Basic Kubernetes network requirement</figcaption></figure><p>By seeing the routing table, it is evident that the Pods can talk to each other via the L3 network as the routes are perfect. What module is responsible for adding this route, and how it gets to know the remote routes? Also, why there is a default route with gateway 169.254.1.1? We will talk about that in a moment.</p><p>The core components of Calico are Bird, Felix, ConfD, Etcd, and Kubernetes API Server. The data-store is used to store the config information(ip-pools, endpoints info, network policies, etc.). In our example, we will use Kubernetes as a Calico data store.</p><h3>BIRD (BGP)</h3><p>The bird is a per-node BGP daemon that exchanges route information with BGP daemons running on other nodes. The common topology could be node-to-node mesh, where each BGP peers with every other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eO0PdPOZpMGNzrZFSvcbcg.png" /></figure><p>For large scale deployments, this can get messy. There are Route Reflectors for completing the route propagation (Certain BGP nodes can be configured as Route Reflectors) to reduce the number of BGP-BGP connections. Rather than each BGP system having to peer with every other BGP system with the AS, each BGP speaker instead peers with a router reflector. Routing advertisements sent to the route reflector are then reflected out to all of the other BGP speakers. For more information, please refer to the <a href="https://tools.ietf.org/html/rfc4456">RFC4456</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MO7b2yM6KE1OS5N5xApCiA.png" /></figure><p>The BIRD instance is responsible for propagating the routes to other BIRD instances. The default configuration is ‘BGP Mesh,’ and this can be used for small deployments. In large-scale deployments, it is recommended to use a Route reflector to avoid issues. There can be more than one RR to have high availability. Also, external rack RRs can be used instead of BIRD.</p><h3>ConfD</h3><p>ConfD is a simple configuration management tool that runs in the Calico node container. It reads values (BIRD configuration for Calico) from etcd, and writes them to disk files. It loops through pools (networks and subnetworks) to apply configuration data (CIDR keys), and assembles them in a way that BIRD can use. So whenever there is a change in the network, BIRD can detect and propagate routes to other nodes.</p><h3>Felix</h3><p>The Calico Felix daemon runs in the Calico node container and brings the solution together by taking several actions:</p><ul><li>Reads information from the Kubernetes etcd</li><li>Builds the routing table</li><li>Configures the IPTables (kube-proxy mode IPTables)</li><li>Configures IPVS (kube-proxy mode IPVS)</li></ul><p>Let’s look at the cluster with all Calico modules,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8VvXjPh7FsWZY7-gVpZ0Q.png" /><figcaption>Deployment with ‘<strong>NoSchedule</strong>’ Toleration</figcaption></figure><p><strong>Something looks different?</strong> Yes, the one end of the <strong><em>veth </em></strong>is dangling, not connected anywhere; It is in kernel space.</p><p><strong>How the packet gets routed to the peer node?</strong></p><ol><li>Pod in master tries to ping the IP address 10.0.2.11</li><li>Pod sends an ARP request to the gateway.</li><li>Get’s the ARP response with the MAC address.</li><li><strong>Wait</strong>, who sent the ARP response?</li></ol><p>What’s going on? How can a container route at an IP that doesn&#39;t exist? Let’s walk through what’s happening. Some of you reading this might have noticed that 169.254.1.1 is an IPv4 link-local address. The container has a default route pointing at a link-local address. The container expects this IP address to be reachable on its directly connected interface, in this case, the containers eth0 address. The container will attempt to ARP for that IP address when it wants to route out through the default route.</p><p>If we capture the ARP response, it will show the MAC address of the other end of the <strong><em>veth </em></strong>(cali123). So you might be wondering how on earth the host is replying to an ARP request for which it doesn’t have an IP interface. The answer is proxy-arp. If we check the host side VETH interface, we’ll see that proxy-arp is enabled.</p><pre>master $ cat /proc/sys/net/ipv4/conf/cali123/proxy_arp<br>1</pre><blockquote><strong>“Proxy ARP</strong> is a technique by which a proxy device on a given network answers the <a href="https://en.wikipedia.org/wiki/Address_Resolution_Protocol">ARP</a> queries for an <a href="https://en.wikipedia.org/wiki/IP_address">IP address</a> that is not on that network. The proxy is aware of the location of the traffic’s destination, and offers its own <a href="https://en.wikipedia.org/wiki/MAC_address">MAC address</a> as the (ostensibly final) destination.<a href="https://en.wikipedia.org/wiki/Proxy_ARP#cite_note-1">[1]</a> The traffic directed to the proxy address is then typically routed by the proxy to the intended destination via another interface or via a <a href="https://en.wikipedia.org/wiki/Tunneling_protocol">tunnel</a>. The process, which results in the node responding with its own MAC address to an ARP request for a different IP address for proxying purposes, is sometimes referred to as <em>publishing”</em></blockquote><p>Let’s take a closer look at the worker node,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6eEzMjdJIwOpGosHs8h8Q.png" /></figure><p>Once the packet reaches the kernel, it routes the packet based on routing table entries.</p><p><strong>Incoming traffic</strong></p><ol><li>The packet reaches the worker node kernel.</li><li>Kernel puts the packet into the cali123.</li></ol><h3>Routing Modes</h3><p>Calico supports 3 routing modes; in this section, we will see the pros and cons of each method and where we can use them.</p><ul><li><strong>IP-in-IP</strong>: default; encapsulated</li><li><strong>Direct/NoEncapMode</strong>: unencapsulated (Preferred)</li><li><strong>VXLAN</strong>: encapsulated (No BGP)</li></ul><h4>IP-in-IP (Default)</h4><p>IP-in-IP is a simple form of encapsulation achieved by putting an IP packet inside another. A transmitted packet contains an outer header with <strong>host</strong> source and destination IPs and an inner header with <strong>pod</strong> source and destination IPs.</p><p>Azure doesn’t support IP-IP (As far I know); therefore, we can’t use IP-IP in that environment. It’s better to disable IP-IP to get better performance.</p><h4>NoEncapMode</h4><p>In this mode, send packets as if they came directly from the pod. Since there is no encapsulation and de-capsulation overhead, direct is highly performant.</p><p>Source IP check must be disabled in AWS to use this mode.</p><h4>VXLAN</h4><p>VXLAN routing is supported in Calico 3.7+.</p><blockquote><strong>VXLAN stands for Virtual Extensible LAN</strong>. VXLAN is an encapsulation technique in which layer 2 ethernet frames are encapsulated in UDP packets. VXLAN is a network virtualization technology. When devices communicate within a software-defined Datacenter, a VXLAN tunnel is set up between those devices. Those tunnels can be set up on both physical and virtual switches. The switch ports are known as <strong>VXLAN Tunnel Endpoints (VTEPs)</strong> and are responsible for the <strong>encapsulation and de-encapsulation of VXLAN</strong> packets. Devices without VXLAN support are connected to a switch with VTEP functionality. The switch will provide the conversion from and to VXLAN.</blockquote><p>VXLAN is great for networks that do not support IP-in-IP, such as Azure or any other DC that doesn’t support BGP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlZBdZeOf-gs5vU2Mm0xZA.png" /></figure><h3>Demo — IPIP and UnEncapMode</h3><p>Check the cluster state before the Calico installation.</p><pre>master $ kubectl get nodes<br>NAME           STATUS     ROLES    AGE   VERSION<br>controlplane   <strong>NotReady   </strong>master   40s   v1.18.0<br>node01         <strong>NotReady   </strong>&lt;none&gt;   9s    v1.18.0<br><br>master $ kubectl get pods --all-namespaces<br>NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE<br>kube-system   coredns-66bff467f8-52tkd               0/1     <strong>Pending   </strong>0          32s<br>kube-system   coredns-66bff467f8-g5gjb               0/1     <strong>Pending   </strong>0          32s<br>kube-system   etcd-controlplane                      1/1     Running   0          34s<br>kube-system   kube-apiserver-controlplane            1/1     Running   0          34s<br>kube-system   kube-controller-manager-controlplane   1/1     Running   0          34s<br>kube-system   kube-proxy-b2j4x                       1/1     Running   0          13s<br>kube-system   kube-proxy-s46lv                       1/1     Running   0          32s<br>kube-system   kube-scheduler-controlplane            1/1     Running   0          33s</pre><p>Check the CNI bin and conf directory. There won’t be any configuration file or the calico binary as the calico installation would populate these via volume mount.</p><pre>master $ cd /etc/cni<br>-bash: cd: /etc/cni: No such file or directory</pre><pre>master $ cd /opt/cni/bin<br>master $ ls<br>bridge  dhcp  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sample  tuning  vlan</pre><p>Check the IP routes in the master/worker node.</p><pre>master $ ip route<br>default via 172.17.0.1 dev ens3<br>172.17.0.0/16 dev ens3 proto kernel scope link src 172.17.0.32<br>172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown<br>curl https://docs.projectcalico.org/manifests/calico.yaml -O</pre><p>Download and apply the calico.yaml based on your environment.</p><pre>curl https://docs.projectcalico.org/manifests/calico.yaml -O<br>kubectl apply -f calico.yaml</pre><p>Let’s take a look at some useful configuration parameters,</p><pre>cni_network_config: |-<br>    {<br>      &quot;name&quot;: &quot;k8s-pod-network&quot;,<br>      &quot;cniVersion&quot;: &quot;0.3.1&quot;,<br>      &quot;plugins&quot;: [<br>        {<br>          &quot;type&quot;: &quot;<strong>calico</strong>&quot;,<strong> &gt;&gt;&gt; Calico&#39;s CNI plugin</strong><br>          &quot;log_level&quot;: &quot;info&quot;,<br>          &quot;log_file_path&quot;: &quot;/var/log/calico/cni/cni.log&quot;,<br>          &quot;datastore_type&quot;: &quot;kubernetes&quot;,<br>          &quot;nodename&quot;: &quot;__KUBERNETES_NODE_NAME__&quot;,<br>          &quot;mtu&quot;: __CNI_MTU__,<br>          &quot;ipam&quot;: {<br>              &quot;type&quot;: &quot;<strong>calico-ipam</strong>&quot; <strong>&gt;&gt;&gt; Calico&#39;s IPAM instaed of default IPAM</strong><br>          },<br>          &quot;policy&quot;: {<br>              &quot;type&quot;: &quot;k8s&quot;<br>          },<br>          &quot;kubernetes&quot;: {<br>              &quot;kubeconfig&quot;: &quot;__KUBECONFIG_FILEPATH__&quot;<br>          }<br>        },<br>        {<br>          &quot;type&quot;: &quot;portmap&quot;,<br>          &quot;snat&quot;: true,<br>          &quot;capabilities&quot;: {&quot;portMappings&quot;: true}<br>        },<br>        {<br>          &quot;type&quot;: &quot;bandwidth&quot;,<br>          &quot;capabilities&quot;: {&quot;bandwidth&quot;: true}<br>        }<br>      ]<br>    }</pre><pre># Enable IPIP<br>- name: CALICO_IPV4POOL_IPIP<br>    value: &quot;Always&quot; <strong>&gt;&gt; Set this to &#39;Never&#39; to disable IP-IP</strong><br># Enable or Disable VXLAN on the default IP pool.<br>- name: CALICO_IPV4POOL_VXLAN<br>    value: &quot;Never&quot;</pre><p>Check POD and Node status after the calico installation.</p><pre>master $ kubectl get pods --all-namespaces<br>NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE<br>kube-system   calico-kube-controllers-799fb94867-6qj77   0/1     ContainerCreating   0          21s<br>kube-system   calico-node-bzttq                          0/1     PodInitializing     0          21s<br>kube-system   calico-node-r6bwj                          0/1     PodInitializing     0          21s<br>kube-system   coredns-66bff467f8-52tkd                   0/1     Pending             0          7m5s<br>kube-system   coredns-66bff467f8-g5gjb                   0/1     ContainerCreating   0          7m5s<br>kube-system   etcd-controlplane                          1/1     Running             0          7m7s<br>kube-system   kube-apiserver-controlplane                1/1     Running             0          7m7s<br>kube-system   kube-controller-manager-controlplane       1/1     Running             0          7m7s<br>kube-system   kube-proxy-b2j4x                           1/1     Running             0          6m46s<br>kube-system   kube-proxy-s46lv                           1/1     Running             0          7m5s<br>kube-system   kube-scheduler-controlplane                1/1     Running             0          7m6s</pre><pre>master $ kubectl get nodes<br>NAME           STATUS   ROLES    AGE     VERSION<br>controlplane   Ready    master   7m30s   v1.18.0<br>node01         Ready    &lt;none&gt;   6m59s   v1.18.0</pre><p>Explore the CNI configuration as that’s what Kubelet needs to set up the network.</p><pre>master $ cd /etc/cni/net.d/<br>master $ ls<br>10-calico.conflist  calico-kubeconfig<br>master $<br>master $<br>master $ cat 10-calico.conflist<br>{<br>  &quot;name&quot;: &quot;k8s-pod-network&quot;,<br>  &quot;cniVersion&quot;: &quot;0.3.1&quot;,<br>  &quot;plugins&quot;: [<br>    {<br>      &quot;type&quot;: &quot;calico&quot;,<br>      &quot;log_level&quot;: &quot;info&quot;,<br>      &quot;log_file_path&quot;: &quot;/var/log/calico/cni/cni.log&quot;,<br>      &quot;datastore_type&quot;: &quot;kubernetes&quot;,<br>      &quot;nodename&quot;: &quot;controlplane&quot;,<br>      &quot;mtu&quot;: 1440,<br>      &quot;ipam&quot;: {<br>          &quot;type&quot;: &quot;calico-ipam&quot;<br>      },<br>      &quot;policy&quot;: {<br>          &quot;type&quot;: &quot;k8s&quot;<br>      },<br>      &quot;kubernetes&quot;: {<br>          &quot;kubeconfig&quot;: &quot;/etc/cni/net.d/calico-kubeconfig&quot;<br>      }<br>    },<br>    {<br>      &quot;type&quot;: &quot;portmap&quot;,<br>      &quot;snat&quot;: true,<br>      &quot;capabilities&quot;: {&quot;portMappings&quot;: true}<br>    },<br>    {<br>      &quot;type&quot;: &quot;bandwidth&quot;,<br>      &quot;capabilities&quot;: {&quot;bandwidth&quot;: true}<br>    }<br>  ]<br>}</pre><p>Check the CNI binary files,</p><pre>master $ ls<br>bandwidth  bridge  <strong>calico  calico-ipam</strong> dhcp  flannel  host-device  host-local  install  ipvlan  loopback  macvlan  portmap  ptp  sample  tuning  vlan<br>master $</pre><p>Let’s install the calicoctl to give good information about the calico and let us modify the Calico configuration.</p><pre>master $ cd /usr/local/bin/<br>master $ curl -O -L  <a href="https://github.com/projectcalico/calicoctl/releases/download/v3.16.3/calicoctl">https://github.com/projectcalico/calicoctl/releases/download/v3.16.3/calicoctl</a><br>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current<br>                                 Dload  Upload   Total   Spent    Left  Speed<br>100   633  100   633    0     0   3087      0 --:--:-- --:--:-- --:--:--  3087<br>100 38.4M  100 38.4M    0     0  5072k      0  0:00:07  0:00:07 --:--:-- 4325k<br>master $ chmod +x calicoctl<br>master $ export DATASTORE_TYPE=kubernetes<br>master $ export KUBECONFIG=~/.kube/config</pre><pre># Check endpoints - it will be empty as we have&#39;t deployed any POD<br>master $ calicoctl get workloadendpoints<br>WORKLOAD   NODE   NETWORKS   INTERFACE</pre><pre>master $</pre><p>Check BGP peer status. This will show the ‘worker’ node as a peer.</p><pre>master $ calicoctl node status<br>Calico process is running.</pre><pre>IPv4 BGP status<br>+--------------+-------------------+-------+----------+-------------+<br>| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |<br>+--------------+-------------------+-------+----------+-------------+<br>| 172.17.0.40  | node-to-node mesh | up    | 00:24:04 | Established |<br>+--------------+-------------------+-------+----------+-------------+</pre><p>Create a busybox POD with two replicas and master node toleration.</p><pre>cat &gt; busybox.yaml &lt;&lt;&quot;EOF&quot;<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: busybox-deployment<br>spec:<br>  selector:<br>    matchLabels:<br>      app: busybox<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        app: busybox<br>    spec:<br>      tolerations:<br>      - key: &quot;node-role.kubernetes.io/master&quot;<br>        operator: &quot;Exists&quot;<br>        effect: &quot;NoSchedule&quot;<br>      containers:<br>      - name: busybox<br>        image: busybox<br>        command: [&quot;sleep&quot;]<br>        args: [&quot;10000&quot;]<br>EOF</pre><pre>master $ kubectl apply -f busybox.yaml<br>deployment.apps/busybox-deployment created</pre><p>Get Pod and endpoint status,</p><pre>master $ kubectl get pods -o wide<br>NAME                                 READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES<br>busybox-deployment-8c7dc8548-btnkv   1/1     Running   0          6s    192.168.196.131   node01         &lt;none&gt;           &lt;none&gt;<br>busybox-deployment-8c7dc8548-x6ljh   1/1     Running   0          6s    192.168.49.66     controlplane   &lt;none&gt;           &lt;none&gt;</pre><pre>master $ calicoctl get workloadendpoints<br>WORKLOAD                             NODE           NETWORKS             INTERFACE<br>busybox-deployment-8c7dc8548-btnkv   node01         192.168.196.131/32   calib673e730d42<br>busybox-deployment-8c7dc8548-x6ljh   controlplane   192.168.49.66/32     cali9861acf9f07</pre><p>Get the details of the host side veth peer of master node busybox POD.</p><pre>master $ ifconfig <strong>cali9861acf9f07</strong><br>cali9861acf9f07: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1440<br>        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20&lt;link&gt;<br>        ether <strong>ee:ee:ee:ee:ee:ee</strong>  txqueuelen 0  (Ethernet)<br>        RX packets 0  bytes 0 (0.0 B)<br>        RX errors 0  dropped 0  overruns 0  frame 0<br>        TX packets 5  bytes 446 (446.0 B)<br>        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</pre><p>Get the details of the master Pod’s interface,</p><pre>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- ifconfig<br>eth0      Link encap:Ethernet  HWaddr 92:7E:C4:15:B9:82<br>          inet addr:192.168.49.66  Bcast:192.168.49.66  Mask:255.255.255.255<br>          UP BROADCAST RUNNING MULTICAST  MTU:1440  Metric:1<br>          RX packets:5 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:0<br>          RX bytes:446 (446.0 B)  TX bytes:0 (0.0 B)</pre><pre>lo        Link encap:Local Loopback<br>          inet addr:127.0.0.1  Mask:255.0.0.0<br>          UP LOOPBACK RUNNING  MTU:65536  Metric:1<br>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:1000<br>          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)</pre><pre>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- ip route<br><strong>default via 169.254.1.1 dev eth0<br>169.254.1.1 dev eth0 scope link</strong><br>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- <strong>arp</strong><br>master $</pre><p>Get the master node routes,</p><pre>master $ ip route<br>default via 172.17.0.1 dev ens3<br>172.17.0.0/16 dev ens3 proto kernel scope link src 172.17.0.32<br>172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown<br>blackhole 192.168.49.64/26 proto bird<br>192.168.49.65 dev calic22dbe57533 scope link<br>192.168.49.66 dev cali9861acf9f07 scope link<br><strong>192.168.196.128/26 via 172.17.0.40 dev tunl0 proto bird onlink</strong></pre><p>Let’s try to ping the worker node Pod to trigger ARP.</p><pre>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- ping 192.168.196.131 -c 1<br>PING 192.168.196.131 (192.168.196.131): 56 data bytes<br>64 bytes from 192.168.196.131: seq=0 ttl=62 time=0.823 ms</pre><pre>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- arp<br>? (169.254.1.1) at <strong>ee:ee:ee:ee:ee:ee</strong> [ether]  on eth0</pre><p>The MAC address of the gateway is nothing but the <strong><em>cali9861acf9f07</em></strong>. From now, whenever the traffic goes out, it will directly hit the kernel; And, the kernel knows that it has to write the packet into the <strong><em>tunl0</em></strong> based on the IP route.</p><p>Proxy ARP configuration,</p><pre>master $ cat /proc/sys/net/ipv4/conf/cali9861acf9f07/proxy_arp<br>1</pre><p><strong>How the destination node handles the packet?</strong></p><pre>node01 $ ip route<br>default via 172.17.0.1 dev ens3<br>172.17.0.0/16 dev ens3 proto kernel scope link src 172.17.0.40<br>172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown<br>192.168.49.64/26 via 172.17.0.32 dev <strong>tunl0 </strong>proto bird onlink<br>blackhole 192.168.196.128/26 proto bird<br>192.168.196.129 dev calid4f00d97cb5 scope link<br>192.168.196.130 dev cali257578b48b6 scope link<br><strong>192.168.196.131 </strong>dev <strong>calib673e730d42 </strong>scope link</pre><p>Upon receiving the packet, the kernel sends the right <strong><em>veth </em></strong>based on the routing table.</p><p>We can see the IP-IP protocol on the wire if we capture the packets. Azure doesn’t support IP-IP (As far I know); therefore, we can’t use IP-IP in that environment. It’s better to disable IP-IP to get better performance. Let’s try to disable and see what’s the effect.</p><p><strong>Disable IP-IP</strong></p><p>Update the ipPool configuration to disable IPIP.</p><pre>master $ calicoctl get ippool default-ipv4-ippool -o yaml &gt; ippool.yaml<br>master $ vi ippool.yaml</pre><p>Open the ippool.yaml and set the IPIP to ‘<strong>Never</strong>,’ and apply the yaml via calicoctl.</p><pre>master $ calicoctl apply -f ippool.yaml<br>Successfully applied 1 &#39;IPPool&#39; resource(s)</pre><p>Recheck the IP route,</p><pre>master $ ip route<br>default via 172.17.0.1 dev ens3<br>172.17.0.0/16 dev ens3 proto kernel scope link src 172.17.0.32<br>172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown<br>blackhole 192.168.49.64/26 proto bird<br>192.168.49.65 dev calic22dbe57533 scope link<br>192.168.49.66 dev cali9861acf9f07 scope link<br>192.168.196.128/26 via 172.17.0.40 dev <strong>ens3</strong> proto bird</pre><p>The device is no more <strong><em>tunl0; it</em></strong> is set to the management interface of the master node.</p><p>Let’s ping the worker node POD and make sure all works fine. From now, there won’t be any IPIP protocol involved.</p><pre>master $ kubectl exec busybox-deployment-8c7dc8548-x6ljh -- ping 192.168.196.131 -c 1<br>PING 192.168.196.131 (192.168.196.131): 56 data bytes<br>64 bytes from 192.168.196.131: seq=0 ttl=62 time=0.653 ms</pre><pre>--- 192.168.196.131 ping statistics ---<br>1 packets transmitted, 1 packets received, 0% packet loss<br>round-trip min/avg/max = 0.653/0.653/0.653 ms</pre><blockquote>Note: Source IP check should be disabled in AWS environment to use this mode.</blockquote><h3>Demo — VXLAN</h3><p>Re-initiate the cluster and download the <strong>calico.yaml </strong>file to apply the following changes,</p><ol><li>Remove <strong>bird</strong> from <strong><em>livenessProbe </em></strong>and <strong><em>readinessProbe</em></strong></li></ol><pre>livenessProbe:<br>            exec:<br>              command:<br>              - /bin/calico-node<br>              - -felix-live<br>              <strong>- -bird-live &gt;&gt; Remove this</strong><br>            periodSeconds: 10<br>            initialDelaySeconds: 10<br>            failureThreshold: 6<br>          readinessProbe:<br>            exec:<br>              command:<br>              - /bin/calico-node<br>              - -felix-ready<br>             <strong> - -bird-ready &gt;&gt; Remove this</strong></pre><p>2. Change the calico_backend to ‘vxlan’ as we don’t need BGP anymore.</p><pre>kind: ConfigMap<br>apiVersion: v1<br>metadata:<br>  name: calico-config<br>  namespace: kube-system<br>data:<br>  # Typha is disabled.<br>  typha_service_name: &quot;none&quot;<br>  # Configure the backend to use.<br>  calico_backend: &quot;<strong>vxlan</strong>&quot;</pre><p>3. Disable IPIP</p><pre># Enable IPIP<br>- name: CALICO_IPV4POOL_IPIP<br>    value: &quot;<strong>Never</strong>&quot; <strong>&gt;&gt; Set this to &#39;Never&#39; to disable IP-IP</strong><br># Enable or Disable VXLAN on the default IP pool.<br>- name: CALICO_IPV4POOL_VXLAN<br>    value: &quot;Never&quot;</pre><p>Let’s apply this new yaml.</p><pre>master $ ip route<br>default via 172.17.0.1 dev ens3<br>172.17.0.0/16 dev ens3 proto kernel scope link src 172.17.0.15<br>172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown<br>192.168.49.65 dev calif5cc38277c7 scope link<br>192.168.49.66 dev cali840c047460a scope link<br>192.168.196.128/26 via 192.168.196.128 dev <strong>vxlan.calico</strong> onlink</pre><pre><strong>vxlan.calico</strong>: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1440<br>        inet 192.168.196.128  netmask 255.255.255.255  broadcast 192.168.196.128<br>        inet6 fe80::64aa:99ff:fe2f:dc24  prefixlen 64  scopeid 0x20&lt;link&gt;<br>        ether <strong>66:aa:99:2f:dc:24</strong>  txqueuelen 0  (Ethernet)<br>        RX packets 0  bytes 0 (0.0 B)<br>        RX errors 0  dropped 0  overruns 0  frame 0<br>        TX packets 0  bytes 0 (0.0 B)<br>        TX errors 0  dropped 11 overruns 0  carrier 0  collisions 0</pre><p>Get the POD status,</p><pre>master $ kubectl get pods -o wide<br>NAME                                 READY   STATUS    RESTARTS   AGE   IP                NODE           NOMINATED NODE   READINESS GATES<br>busybox-deployment-8c7dc8548-8bxnw   1/1     Running   0          11s   192.168.49.67     controlplane   &lt;none&gt;           &lt;none&gt;<br>busybox-deployment-8c7dc8548-kmxst   1/1     Running   0          11s   192.168.196.130   node01         &lt;none&gt;           &lt;none&gt;</pre><p>Ping the worker node POD from</p><pre>master $ kubectl exec busybox-deployment-8c7dc8548-8bxnw -- ip route<br>default via 169.254.1.1 dev eth0<br>169.254.1.1 dev eth0 scope link</pre><p>Trigger the ARP request,</p><pre>master $ kubectl exec busybox-deployment-8c7dc8548-8bxnw -- arp<br>master $ kubectl exec busybox-deployment-8c7dc8548-8bxnw -- ping 8.8.8.8<br>PING 8.8.8.8 (8.8.8.8): 56 data bytes<br>64 bytes from 8.8.8.8: seq=0 ttl=116 time=3.786 ms<br>^C<br>master $ kubectl exec busybox-deployment-8c7dc8548-8bxnw -- arp<br>? (169.254.1.1) at <strong>ee:ee:ee:ee:ee:ee</strong> [ether]  on eth0<br>master $</pre><p>The concept is as the previous modes, but the only difference is that the packet reaches the vxland, and it encapsulates the packet with node IP and its MAC in the inner header and sends it. Also, the UDP port of the vxlan proto will be 4789. The etcd helps here to get the details of available nodes and their supported IP range so that the vxlan-calico can build the packet.</p><p>Note: VxLAN mode needs more processing power than the previous modes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlZBdZeOf-gs5vU2Mm0xZA.png" /></figure><h3>Disclaimer</h3><p>This article does not provide any technical advice or recommendation; if you feel so, it is my personal view, not the company I work for.</p><h3><strong>References</strong></h3><pre><a href="https://docs.projectcalico.org/">https://docs.projectcalico.org/</a><br><a href="https://www.openstack.org/videos/summits/vancouver-2018/kubernetes-networking-with-calico-deep-dive">https://www.openstack.org/videos/summits/vancouver-2018/kubernetes-networking-with-calico-deep-dive</a><br><a href="https://kubernetes.io/">https://kubernetes.io/</a><br><a href="https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_network/calico.html">https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_network/calico.html</a><br><a href="https://github.com/coreos/flannel">https://github.com/coreos/flannel</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a07f5bf0ff14" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life of a Packet in Kubernetes — Part 1]]></title>
            <link>https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-1-f9bc0909e051?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9bc0909e051</guid>
            <category><![CDATA[cni]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[namespaces]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Sat, 17 Oct 2020 19:06:18 GMT</pubDate>
            <atom:updated>2022-03-09T06:08:31.049Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life of a Packet in Kubernetes — Part 1</h3><p>Kubernetes cluster networking can be more than a bit confusing, even for engineers with hands-on experience working with virtual networks and request routing. This article will help to understand the fundamental Kubernetes networking. The initial plan was to deep dive into Kubernetes networking’s complexities by following the journey of an HTTP request to a service running on a Kubernetes cluster. However, the life of a packet will not be complete without namespaces, CNI, and calico. We will start with Linux networking and cover other topics later.</p><p>This article is already too <strong>long, </strong>therefore divided the topics into several parts to ensure we give each topic time.</p><h3>Topics — Part 1</h3><ol><li>Linux namespaces</li><li>Container networking</li><li>What is CNI?</li><li>Pod network namespace</li></ol><p><strong>Link</strong>: This article.</p><h3>Topics — Part 2</h3><ol><li>Calico CNI</li></ol><p><strong>Link</strong>: <a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-2-a07f5bf0ff14">Life of a Packet in Kubernetes — Part 2</a></p><h3>Topics — Part 3</h3><ol><li>Pod-to-Pod</li><li>Pod-to-External</li><li>Pod-to-Service</li><li>External-to-Pod</li><li>External Traffic Policy</li><li>Kube-Proxy</li><li>iptable rules processing flow</li><li>Network Policy basics</li></ol><p><strong>Link</strong>: <a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-3-dd881476da0f">Life of a Packet in Kubernetes — Part 3</a></p><h3>Topics — Part 4</h3><ol><li>Ingress Controller</li><li>Ingress Resources Example</li><li>Nginx</li><li>Envoy+Contour</li><li>Ingress with MetalLB</li></ol><p>Link: <a href="https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-4-4dbc5256050a?source=friends_link&amp;sk=b2011d9850110c86e2fcd1d5eb3360d6">Life of a Packet in Kubernetes — Part 4</a></p><h3>Topics — Part 5</h3><ol><li>ISTIO service mesh</li></ol><p>Link: <a href="https://dramasamy.medium.com/life-of-a-packet-in-istio-part-1-8221971d77de?source=friends_link&amp;sk=325b2b612c0d7db562696553e0d2275e">Life of a Packet in ISTIO — Part 1</a></p><h3>Linux Namespaces</h3><p>Linux namespaces comprise some of the fundamental technologies behind most modern-day container implementations. At a high level, they allow for the isolation of global system resources between independent processes. For example, the PID namespace isolates the process ID number space. This means that two processes running on the same host can have the same PID!</p><p>This level of isolation is clearly useful in the world of containers. Without namespaces, a process running in container A <em>could</em>, for example, umount an important filesystem in container B, or change the hostname of container C, or remove a network interface from container D. By namespacing these resources, the process in container A isn’t even aware that the processes in containers B, C, and D exist.</p><ol><li>Mount — isolate filesystem mount points</li><li>UTS — isolate hostname and domain name</li><li>IPC — isolate interprocess communication (IPC) resources</li><li>PID — isolate the PID number space</li><li>Network — isolate network interfaces</li><li>User — isolate UID/GID number spaces</li><li>Cgroup — isolate cgroup root directory</li></ol><p>Most container implementations make use of the above namespaces in order to provide the highest level of isolation between separate container processes. Although note that the cgroup namespace is slightly more recent than the others and isn’t as widely used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*HE24C_5PRUFO2GHlQiN9aQ.png" /><figcaption>PC: Google</figcaption></figure><h3>Container Networking (Network Namespace)</h3><p>Before we jump into understanding the various options provided by CNI and Docker, let’s explore the core technology that powers container networking. The Linux kernel has various features that have been developed to provide multi-tenancy on hosts. Namespaces provide functionality that offers different kinds of isolation, with network namespace being the one that provides network isolation.</p><p>It’s very easy to create network namespaces using the <strong>ip</strong> command in any Linux operating system. Let’s create two different network namespaces and name them as client and server.</p><pre>master# ip netns add client<br>master# ip netns add server<br>master# ip netns list<br>server<br>client</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*PIR6n4AfNncg-oqMk1REIw.png" /></figure><p>Create a <strong>veth</strong> pair to connect these network namespaces. Think of a <strong>veth</strong> pair as a network cable with connectors at both ends.</p><pre>master# ip link add veth-client type veth peer name veth-server<br>master# ip link list | grep veth<br>4: veth-server@veth-client: &lt;BROADCAST,MULTICAST,M-DOWN&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000<br>5: veth-client@veth-server: &lt;BROADCAST,MULTICAST,M-DOWN&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*KsL6knbiIBgmCGMHTDrXvg.png" /></figure><p>The <strong>veth</strong> pair (cable) exists on the host network namespace; Now let’s move the two ends of the <strong>veth</strong> pair to their respective namespaces that we created earlier.</p><pre>master# ip link set veth-client netns client<br>master# ip link set veth-server netns server<br>master# ip link list | grep veth # doesn’t exist on the host network namespace now</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*vnyFh0GOhqzR44fAFJoGSg.png" /></figure><p>Let’s verify the <strong>veth</strong> ends actually exist in the namespaces. We’ll start with the <strong>client</strong> namespace</p><pre>master# ip netns exec client ip link<br>1: lo: &lt;LOOPBACK&gt; mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>5: veth-client@if4: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000<br>    link/ether ca:e8:30:2e:f9:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1</pre><p>Now let’s check the <strong>server</strong> namespace</p><pre>master# ip netns exec server ip link<br>1: lo: &lt;LOOPBACK&gt; mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>4: veth-server@if5: &lt;BROADCAST,MULTICAST&gt; mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000<br>    link/ether 42:96:f0:ae:f0:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0</pre><p>Now let’s assign IP addresses to these interfaces and bring them up</p><pre>master# ip netns exec client ip address add 10.0.0.11/24 dev veth-client<br>master# ip netns exec client ip link set veth-client up<br>master# ip netns exec server ip address add 10.0.0.12/24 dev veth-server<br>master# ip netns exec server ip link set veth-server up<br>master#<br>master# ip netns exec client ip addr<br>1: lo: &lt;LOOPBACK&gt; mtu 65536 qdisc noop state DOWN group default qlen 1<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>5: veth-client@if4: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000<br>    link/ether ca:e8:30:2e:f9:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1<br>    inet 10.0.0.11/24 scope global veth-client<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::c8e8:30ff:fe2e:f9d2/64 scope link<br>       valid_lft forever preferred_lft forever<br>master#<br>master# ip netns exec server ip addr<br>1: lo: &lt;LOOPBACK&gt; mtu 65536 qdisc noop state DOWN group default qlen 1<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>4: veth-server@if5: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000<br>    link/ether 42:96:f0:ae:f0:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>    inet 10.0.0.12/24 scope global veth-server<br>       valid_lft forever preferred_lft forever<br>    inet6 fe80::4096:f0ff:feae:f0c5/64 scope link<br>       valid_lft forever preferred_lft forever<br>master#</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*5iksV7EzJEAJG--6d9K_Tg.png" /></figure><p>Using the ping command, we can verify the two network namespaces have been connected and are reachable,</p><pre>master# ip netns exec client ping 10.0.0.12<br>PING 10.0.0.12 (10.0.0.12) 56(84) bytes of data.<br>64 bytes from 10.0.0.12: icmp_seq=1 ttl=64 time=0.101 ms<br>64 bytes from 10.0.0.12: icmp_seq=2 ttl=64 time=0.072 ms<br>64 bytes from 10.0.0.12: icmp_seq=3 ttl=64 time=0.084 ms<br>64 bytes from 10.0.0.12: icmp_seq=4 ttl=64 time=0.077 ms<br>64 bytes from 10.0.0.12: icmp_seq=5 ttl=64 time=0.079 ms</pre><p>If we would like to create more network namespaces and connect them together, it might not be a scalable solution to create a <strong>veth</strong> pair for every combination of namespaces. Instead, one can create a Linux bridge and hook up these network namespaces to the bridge to get connectivity. And that’s exactly how Docker sets up networking between containers running on the same host!</p><p>Let’s create namespaces and attach the same to the bridge.</p><pre># All in one<br>BR=bridge1<br>HOST_IP=172.17.0.33</pre><pre>ip link add client1-veth type veth peer name client1-veth-br<br>ip link add server1-veth type veth peer name server1-veth-br<br>ip link add $BR type bridge<br>ip netns add client1<br>ip netns add server1<br>ip link set client1-veth netns client1<br>ip link set server1-veth netns server1<br>ip link set client1-veth-br master $BR<br>ip link set server1-veth-br master $BR<br>ip link set $BR up<br>ip link set client1-veth-br up<br>ip link set server1-veth-br up<br>ip netns exec client1 ip link set client1-veth up<br>ip netns exec server1 ip link set server1-veth up<br>ip netns exec client1 ip addr add 172.30.0.11/24 dev client1-veth<br>ip netns exec server1 ip addr add 172.30.0.12/24 dev server1-veth<br>ip netns exec client1 ping 172.30.0.12 -c 5<br>ip addr add 172.30.0.1/24 dev $BR<br>ip netns exec client1 ping 172.30.0.12 -c 5<br>ip netns exec client1 ping 172.30.0.1 -c 5</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*uN5lIUU8wDO8EUL8gnQwNA.png" /></figure><p>Using the ping command, we can verify the two network namespaces have been connected and are reachable,</p><pre>controlplane $ ip netns exec client1 ping 172.30.0.12 -c 5<br>PING 172.30.0.12 (172.30.0.12) 56(84) bytes of data.<br>64 bytes from 172.30.0.12: icmp_seq=1 ttl=64 time=0.138 ms<br>64 bytes from 172.30.0.12: icmp_seq=2 ttl=64 time=0.091 ms<br>64 bytes from 172.30.0.12: icmp_seq=3 ttl=64 time=0.073 ms<br>64 bytes from 172.30.0.12: icmp_seq=4 ttl=64 time=0.070 ms<br>64 bytes from 172.30.0.12: icmp_seq=5 ttl=64 time=0.107 ms</pre><p>Let’s ping the HOST_IP from the namespace,</p><pre>controlplane $ ip netns exec client1 ping $HOST_IP -c 2<br><strong>connect: Network is unreachable</strong></pre><p>Yes, ‘Network is unreachable’ because there is no route configured in the newly created namespaces. Let’s add the default route,</p><pre>controlplane $ ip netns exec client1 ip route add default via 172.30.0.1<br>controlplane $ ip netns exec server1 ip route add default via 172.30.0.1<br>controlplane $ ip netns exec client1 ping $HOST_IP -c 5<br>PING 172.17.0.23 (172.17.0.23) 56(84) bytes of data.<br>64 bytes from 172.17.0.23: icmp_seq=1 ttl=64 time=0.053 ms<br>64 bytes from 172.17.0.23: icmp_seq=2 ttl=64 time=0.121 ms<br>64 bytes from 172.17.0.23: icmp_seq=3 ttl=64 time=0.078 ms<br>64 bytes from 172.17.0.23: icmp_seq=4 ttl=64 time=0.129 ms<br>64 bytes from 172.17.0.23: icmp_seq=5 ttl=64 time=0.119 ms</pre><pre>--- 172.17.0.23 ping statistics ---<br>5 packets transmitted, 5 received, 0% packet loss, time 3999ms<br>rtt min/avg/max/mdev = 0.053/0.100/0.129/0.029 ms</pre><p>Now the ‘<strong>default</strong>’ route to reach the external network is the bridge, hence the namespaces can consume any external network services,</p><pre>controlplane $ ping 8.8.8.8 -c 2<br>PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.<br>64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=3.40 ms<br>64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=3.81 ms</pre><pre>--- 8.8.8.8 ping statistics ---<br>2 packets transmitted, 2 received, 0% packet loss, time 1001ms<br>rtt min/avg/max/mdev = 3.403/3.610/3.817/0.207 ms</pre><h4>How to reach the private network from external servers?</h4><p>As you can see, the machine we’re demoing on already have Docker installed, which has led to the creation of the <strong>docker0</strong> bridge. It’s not easy to run a webserver from a network namespace context as all the Linux namespaces should interwork to simulate that scenario. Let’s use the Docker to simulate the scenario.</p><pre>docker0   Link encap:Ethernet  HWaddr 02:42:e2:44:07:39<br>          inet addr:172.18.0.1  Bcast:172.18.0.255  Mask:255.255.255.0<br>          UP BROADCAST MULTICAST  MTU:1500  Metric:1<br>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:0<br>          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)</pre><p>Let’s spin up an Nginx container now and inspect the same.</p><pre>controlplane $ docker run -d --name web --rm nginx<br>efff2d2c98f94671f69cddc5cc88bb7a0a5a2ea15dc3c98d911e39bf2764a556<br>controlplane $ WEB_IP=`docker inspect -f &quot;{{ .NetworkSettings.IPAddress }}&quot; web`<br>controlplane $ docker inspect web --format &#39;{{ .NetworkSettings.SandboxKey }}&#39;<br>/var/run/docker/netns/c009f2a4be71</pre><p>Since Docker doesn’t create the <strong>netns</strong> in the default location, <strong>ip netns list</strong> doesn’t show this network namespace. We can create a symlink to the expected location to overcome that limitation.</p><pre>controlplane $ container_id=web<br>controlplane $ container_netns=$(docker inspect ${container_id} --format &#39;{{ .NetworkSettings.SandboxKey }}&#39;)<br>controlplane $ mkdir -p /var/run/netns<br>controlplane $ rm -f /var/run/netns/${container_id}<br>controlplane $ ln -sv ${container_netns} /var/run/netns/${container_id}<br>&#39;/var/run/netns/web&#39; -&gt; &#39;/var/run/docker/netns/c009f2a4be71&#39;<br>controlplane $ ip netns list<br><strong>web </strong>(id: 3)<br>server1 (id: 1)<br>client1 (id: 0)</pre><p>Let’s check the IP address in the <strong>web </strong>namespace,</p><pre>controlplane $ ip netns exec web ip addr<br>1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1<br>    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>    inet 127.0.0.1/8 scope host lo<br>       valid_lft forever preferred_lft forever<br>11: eth0@if12: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default<br>    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0<br>    inet <strong>172.18.0.3</strong>/24 brd 172.18.0.255 scope global eth0<br>       valid_lft forever preferred_lft forever</pre><p>Let’s check the IP address in the docker container,</p><pre>controlplane $ WEB_IP=`docker inspect -f &quot;{{ .NetworkSettings.IPAddress }}&quot; web`<br>controlplane $ echo $WEB_IP<br><strong>172.18.0.3</strong></pre><p>It is very clear that the docker uses all the Linux namespaces and isolates things from the host. Let’s try to reach the WebApp that runs in the <strong>web </strong>network namespace from the HOST server.</p><pre>controlplane $ curl $WEB_IP<br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br>&lt;title&gt;Welcome to nginx!&lt;/title&gt;<br>&lt;style&gt;<br>    body {<br>        width: 35em;<br>        margin: 0 auto;<br>        font-family: Tahoma, Verdana, Arial, sans-serif;<br>    }<br>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;<br>&lt;p&gt;If you see this page, the nginx web server is successfully installed and<br>working. Further configuration is required.&lt;/p&gt;</pre><pre>&lt;p&gt;For online documentation and support please refer to<br>&lt;a href=&quot;<a href="http://nginx.org/">http://nginx.org/</a>&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;<br>Commercial support is available at<br>&lt;a href=&quot;<a href="http://nginx.com/">http://nginx.com/</a>&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;</pre><pre>&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>Is it possible to reach this webserver from an external network? Yes, by adding port forwarding.</p><pre>controlplane $ iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination $WEB_IP:80<br>controlplane $ echo $HOST_IP<br>172.17.0.23</pre><p>Let’s try to reach the webserver using the HOST IP address.</p><pre>node01 $ curl 172.17.0.23<br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br>&lt;title&gt;Welcome to nginx!&lt;/title&gt;<br>&lt;style&gt;<br>    body {<br>        width: 35em;<br>        margin: 0 auto;<br>        font-family: Tahoma, Verdana, Arial, sans-serif;<br>    }<br>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;<br>&lt;p&gt;If you see this page, the nginx web server is successfully installed and<br>working. Further configuration is required.&lt;/p&gt;</pre><pre>&lt;p&gt;For online documentation and support please refer to<br>&lt;a href=&quot;<a href="http://nginx.org/">http://nginx.org/</a>&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;<br>Commercial support is available at<br>&lt;a href=&quot;<a href="http://nginx.com/">http://nginx.com/</a>&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;</pre><pre>&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;<br>node01 $</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*okCmYUfC2slSB9E7MhJNXQ.png" /></figure><p>The CNI plugin executes the above commands (Not exactly, but similar) to set up the loopback interface, eth0, and assign the IP address to the container. The Container run time (i.e Kubernetes, PodMan, etc) makes use of CNIs to setup the POD network. Let’s discuss the CNI in the next section.</p><h3>What is CNI?</h3><p>“<em>A CNI plugin is responsible for inserting a network interface into the container network namespace (e.g. one end of a veth pair) and making any necessary changes on the host (e.g. attaching the other end of the veth into a bridge). It should then assign the IP to the interface and setup the routes consistent with the IP Address Management section by invoking the appropriate IPAM plugin.</em>”</p><p><strong>Wait, Looks familiar?</strong> Yes, we have seen this in the “<strong><em>Container Networking</em></strong>” section.</p><p>CNI (<em>Container Network Interface</em>), a <a href="https://cncf.io/">Cloud Native Computing Foundation</a> project, consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers, along with a number of supported plugins. CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted. Because of this focus, CNI has a wide range of support and the specification is simple to implement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*YrxFRE-3cmaiwR1HS5wviA.png" /></figure><p>Note: The runtime can be anything — i.e Kubernetes, PodMan, cloud foundry, etc</p><p><strong>Detailed CNI Specification</strong></p><p><a href="https://github.com/containernetworking/cni/blob/master/SPEC.md">https://github.com/containernetworking/cni/blob/master/SPEC.md</a></p><p>Here are a couple of points I found interesting during my first read-through…</p><ul><li>The spec defines a container as being a Linux network namespace. We should be comfortable with that definition as container runtimes like Docker create a new network namespace for each container.</li><li>Network definitions for CNI are stored as JSON files.</li><li>The network definitions are streamed to the plugin through STDIN; i.e there are no configuration files sitting on the host for the network configuration.</li><li>Other arguments are passed to the plugin via environmental variables</li><li>A CNI plugin is implemented as an executable.</li><li>The CNI plugin is responsible for wiring up the container. i.e it needs to do all the work to get the container on the network. In Docker, this would include connecting the container network namespace back to the host somehow.</li><li>The CNI plugin is responsible for IPAM which includes IP address assignment and installing any required routes.</li></ul><blockquote>Let’s try to <strong>simulate the Pod creation</strong> manually without Kubernetes and assign IP via CNI plugin instead of using too many CLI commands. At end of this demo, you will understand what is a Pod in Kubernetes.</blockquote><p>Step1: Download the CNI plugins.</p><pre>controlplane $ mkdir cni<br>controlplane $ cd cni<br>controlplane $ curl -O -L <a href="https://github.com/containernetworking/cni/releases/download/v0.4.0/cni-amd64-v0.4.0.tgz">https://github.com/containernetworking/cni/releases/download/v0.4.0/cni-amd64-v0.4.0.tgz</a><br>  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current<br>                                 Dload  Upload   Total   Spent    Left  Speed<br>100   644  100   644    0     0   1934      0 --:--:-- --:--:-- --:--:--  1933<br>100 15.3M  100 15.3M    0     0   233k      0  0:01:07  0:01:07 --:--:--  104k<br>controlplane $ tar -xvf cni-amd64-v0.4.0.tgz<br>./<br>./macvlan<br>./dhcp<br>./loopback<br>./ptp<br>./ipvlan<br>./bridge<br>./tuning<br>./noop<br>./host-local<br>./cnitool<br>./flannel<br>controlplane $</pre><p>Step 2: Create a CNI configuration in a JSON format.</p><pre>cat &gt; /tmp/00-demo.conf &lt;&lt;&quot;EOF&quot;<br>{<br>    &quot;cniVersion&quot;: &quot;0.2.0&quot;,<br>    &quot;name&quot;: &quot;demo_br&quot;,<br>    &quot;type&quot;: &quot;bridge&quot;,<br>    &quot;bridge&quot;: &quot;cni_net0&quot;,<br>    &quot;isGateway&quot;: true,<br>    &quot;ipMasq&quot;: true,<br>    &quot;ipam&quot;: {<br>        &quot;type&quot;: &quot;host-local&quot;,<br>        &quot;subnet&quot;: &quot;10.0.10.0/24&quot;,<br>        &quot;routes&quot;: [<br>            { &quot;dst&quot;: &quot;0.0.0.0/0&quot; },<br>            { &quot;dst&quot;: &quot;1.1.1.1/32&quot;, &quot;gw&quot;:&quot;10.0.10.1&quot;}<br>        ]<br>    }<br>}<br>EOF</pre><p>CNI Configuration parameters,</p><pre><strong>-:CNI generic parameters:-</strong></pre><pre><strong>cniVersion</strong>: The version of the CNI spec in which the definition works with<br><strong>name</strong>: The network name<br><strong>type</strong>: The name of the plugin you wish to use.  In this case, the actual name of the plugin executable<br><strong>args</strong>: Optional additional parameters<br><strong>ipMasq</strong>: Configure outbound masquerade (source NAT) for this network<br><strong>ipam</strong>:<br>    <strong>type</strong>: The name of the IPAM plugin executable<br>    <strong>subnet</strong>: The subnet to allocate out of (this is actually part of the IPAM plugin)<br>    <strong>routes</strong>:<br>        <strong>dst</strong>: The subnet you wish to reach<br>        <strong>gw</strong>: The IP address of the next hop to reach the dst.  If not specified the default gateway for the subnet is assumed<br><strong>dns</strong>:<br>    <strong>nameservers</strong>: A list of nameservers you wish to use with this network<br>    <strong>domain</strong>: The search domain to use for DNS requests<br>    <strong>search</strong>: A list of search domains<br>    <strong>options</strong>: A list of options to be passed to the receiver</pre><p>Step 3: Create a container with a ‘<strong>none</strong>’ network so that the container won’t have any IP address to use. You can use any image for this container, however, I’m using the ‘<strong>pause</strong>’ container to mimic Kubernetes.</p><pre>controlplane $ docker run --name pause_demo -d --rm --network none kubernetes/pause<br>Unable to find image &#39;kubernetes/pause:latest&#39; locally<br>latest: Pulling from kubernetes/pause<br>4f4fb700ef54: Pull complete<br>b9c8ec465f6b: Pull complete<br>Digest: sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105<br>Status: Downloaded newer image for kubernetes/pause:latest<br>763d3ef7d3e943907a1f01f01e13c7cb6c389b1a16857141e7eac0ac10a6fe82<br>controlplane $ container_id=pause_demo<br>controlplane $ container_netns=$(docker inspect ${container_id} --format &#39;{{ .NetworkSettings.SandboxKey }}&#39;)<br>controlplane $ mkdir -p /var/run/netns<br>controlplane $ rm -f /var/run/netns/${container_id}<br>controlplane $ ln -sv ${container_netns} /var/run/netns/${container_id}<br>&#39;/var/run/netns/pause_demo&#39; -&gt; &#39;/var/run/docker/netns/0297681f79b5&#39;<br>controlplane $ ip netns list<br>pause_demo<br>controlplane $ ip netns exec $container_id ifconfig<br>lo        Link encap:Local Loopback<br>          inet addr:127.0.0.1  Mask:255.0.0.0<br>          UP LOOPBACK RUNNING  MTU:65536  Metric:1<br>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:1<br>          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)</pre><p>Step 4: Invoke the CNI plugin with the CNI configuration file.</p><pre>controlplane $ CNI_CONTAINERID=$container_id CNI_IFNAME=eth10 CNI_COMMAND=ADD CNI_NETNS=/var/run/netns/$container_id CNI_PATH=`pwd` ./bridge &lt;/tmp/00-demo.conf<br>2020/10/17 17:32:37 Error retriving last reserved ip: Failed to retrieve last reserved ip: open /var/lib/cni/networks/demo_br/last_reserved_ip: <strong>no such file or directory</strong><br>{<br>    &quot;ip4&quot;: {<br>        &quot;ip&quot;: &quot;10.0.10.2/24&quot;,<br>        &quot;gateway&quot;: &quot;10.0.10.1&quot;,<br>        &quot;routes&quot;: [<br>            {<br>                &quot;dst&quot;: &quot;0.0.0.0/0&quot;<br>            },<br>            {<br>                &quot;dst&quot;: &quot;1.1.1.1/32&quot;,<br>                &quot;gw&quot;: &quot;10.0.10.1&quot;<br>            }<br>        ]<br>    },<br>    &quot;dns&quot;: {}</pre><ul><li><strong>CNI_COMMAND=ADD</strong> — Action (Available values: ADD/DEL/CHECK)</li><li><strong>CNI_CONTAINER=pause_demo</strong> — We’re telling CNI that the network namespace we want to work is called ‘<strong>pause_demo</strong>’</li><li><strong>CNI_NETNS=/var/run/netns/pause_demo</strong>— The path to the namespace in question</li><li><strong>CNI_IFNAME=eth10</strong>— The name of the interface we wish to use on the container side of the connection</li><li><strong>CNI_PATH=`pwd`</strong> — We always need to tell CNI where the plugin executables live. In this case, since we’re already in the ‘cni’ directory we just have the variable reference pwd (present working directory). You need the ticks around the command pwd for it to evaluate correctly. The formatting here seems to be removing them but they are in the command above correctly</li></ul><blockquote>I strongly recommend you to read the <a href="https://github.com/containernetworking/cni/blob/master/SPEC.md"><strong>CNI specification</strong></a> to get more information about the plugin and their functions. You can use more than one plugin in the same JSON file to chain the actions; i.e add firewall rule etc.</blockquote><p>step 5: Running the above command returns a couple of things. First — it returns an error since the IPAM driver can’t find the file it uses to store IP information locally. If we run this again for a different namespace, we wouldn’t get this error since the file is created the first time we run the plugin. The second thing we get is a JSON return indicating the relevant IP configuration that was configured by the plugin. In this case, the bridge itself should have received the IP address of 10.0.10.1/24 and the namespace interface would have received 10.0.10.2/24. It also added the default route and the 1.1.1.1/32 route that we defined in the network configuration JSON. Let’s check what it did,</p><pre>controlplane $ ip netns exec pause_demo ifconfig<br><strong>eth10     </strong>Link encap:Ethernet  HWaddr 0a:58:0a:00:0a:02<br>          inet addr:<strong>10.0.10.2</strong>  Bcast:0.0.0.0  Mask:255.255.255.0<br>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1<br>          RX packets:18 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:0<br>          RX bytes:1476 (1.4 KB)  TX bytes:0 (0.0 B)</pre><pre>lo        Link encap:Local Loopback<br>          inet addr:127.0.0.1  Mask:255.0.0.0<br>          UP LOOPBACK RUNNING  MTU:65536  Metric:1<br>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:1<br>          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)</pre><pre>controlplane $ ip netns exec pause_demo ip route<br><strong>default </strong>via 10.0.10.1 dev eth10<br><strong>1.1.1.1 </strong>via 10.0.10.1 dev eth10<br>10.0.10.0/24 dev eth10  proto kernel  scope link  src 10.0.10.2<br>controlplane $</pre><p>CNI creates a bridge and configures it as per the configuration,</p><pre>controlplane $ ifconfig<br>cni_net0  Link encap:Ethernet  HWaddr 0a:58:0a:00:0a:01<br>          inet addr:<strong>10.0.10.1</strong>  Bcast:0.0.0.0  Mask:255.255.255.0<br>          inet6 addr: fe80::c4a4:2dff:fe4b:aa1b/64 Scope:Link<br>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1<br>          RX packets:7 errors:0 dropped:0 overruns:0 frame:0<br>          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0<br>          collisions:0 txqueuelen:1000<br>          RX bytes:1174 (1.1 KB)  TX bytes:1545 (1.5 KB)</pre><p>Step 6: Start a web server and share the ‘<strong>pause</strong>’ containers namespace.</p><pre>controlplane $ docker run --name web_demo -d --rm --network <strong>container:$container_id</strong> nginx<br>8fadcf2925b779de6781b4215534b32231685b8515f998b2a66a3c7e38333e30</pre><p>Step 7: Browse the web page using pause containers IP address</p><pre>controlplane $ curl `cat /var/lib/cni/networks/demo_br/<strong>last_reserved_ip</strong>`<br>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br>&lt;title&gt;Welcome to nginx!&lt;/title&gt;<br>&lt;style&gt;<br>    body {<br>        width: 35em;<br>        margin: 0 auto;<br>        font-family: Tahoma, Verdana, Arial, sans-serif;<br>    }<br>&lt;/style&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;<br>&lt;p&gt;If you see this page, the nginx web server is successfully installed and<br>working. Further configuration is required.&lt;/p&gt;</pre><pre>&lt;p&gt;For online documentation and support please refer to<br>&lt;a href=&quot;<a href="http://nginx.org/">http://nginx.org/</a>&quot;&gt;nginx.org&lt;/a&gt;.&lt;br/&gt;<br>Commercial support is available at<br>&lt;a href=&quot;<a href="http://nginx.com/">http://nginx.com/</a>&quot;&gt;nginx.com&lt;/a&gt;.&lt;/p&gt;</pre><pre>&lt;p&gt;&lt;em&gt;Thank you for using nginx.&lt;/em&gt;&lt;/p&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>Now let’s see the definition of a Pod in the following section.</p><h4>Pod Network Namespace</h4><p>The first thing to understand in Kubernetes is that a <strong>POD </strong>is not actually the equivalent to a container, but is a collection of containers. And all these containers of the same collection share a network stack. Kubernetes manages that by setting up the network itself on the <strong>pause </strong>container, which you will find for every pod you create. All other containers attach to the network of the pause container which itself does nothing but provide the network. Therefore, it is also possible for one container to talk to a service in a different container, which is in the same definition of the same pod, via the localhost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*DkhwDvRK9L5cKecEAyQ-dg.png" /></figure><h3>References:</h3><pre><a href="https://man7.org/linux/man-pages/man7/namespaces.7.html">https://man7.org/linux/man-pages/man7/namespaces.7.html</a><br><a href="https://github.com/containernetworking/cni/blob/master/SPEC.md">https://github.com/containernetworking/cni/blob/master/SPEC.md</a><br><a href="https://github.com/containernetworking/cni/tree/master/cnitool">https://github.com/containernetworking/cni/tree/master/cnitool</a><br><a href="https://github.com/containernetworking/cni">https://github.com/containernetworking/cni</a><br><a href="https://tldp.org/HOWTO/BRIDGE-STP-HOWTO/set-up-the-bridge.html">https://tldp.org/HOWTO/BRIDGE-STP-HOWTO/set-up-the-bridge.html</a> <br><a href="https://kubernetes.io/">https://kubernetes.io/</a><br><a href="https://www.dasblinkenlichten.com/">https://www.dasblinkenlichten.com/</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9bc0909e051" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High-Performance Containerized Applications in Kubernetes]]></title>
            <link>https://dramasamy.medium.com/high-performance-containerized-applications-in-kubernetes-f494cef3f8e8?source=rss-79ada7cd5ae5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f494cef3f8e8</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Dinesh Kumar Ramasamy]]></dc:creator>
            <pubDate>Mon, 12 Oct 2020 21:22:48 GMT</pubDate>
            <atom:updated>2020-10-12T23:03:58.043Z</atom:updated>
            <content:encoded><![CDATA[<p>The Single Root I/O Virtualization (SR-IOV) specification is a standard for a type of PCI device assignment that can share a single device with multiple pods. SR-IOV enables you to segment a compliant network device, recognized on the host node as a physical function (PF), into multiple virtual functions (VFs), and make them available for direct IO to the POD.</p><p>You can use SR-IOV network devices with additional networks on your Kubernetes cluster for applications that require high bandwidth or low latency.</p><p>In this article, we will talk about how to set up the POD to use SR-IOV VF by using the device plugin and Multus. The assumption here is that the worker nodes have enough VFs assigned; and you are familiar with the SR-IOV concept. <a href="https://docs.openstack.org/mitaka/networking-guide/config-sriov.html">This </a>OpenStack documentation will help to set up SR-IOV VFs.</p><blockquote><strong><em>A disadvantage of SR-IOV is that it has specific vendor hardware dependency. When used for CNFs, it is difficult to assign a network function that is fully portable from node to node unless the nodes have identical NICs.</em></strong></blockquote><p>Kubernetes currently supports the discovery of CPU and Memory primarily to a minimal extent. Very few devices are handled natively by Kubelet. It is not a sustainable solution to expect every hardware vendor to add their vendor-specific code inside Kubernetes to make their devices usable.</p><h3><strong>Requirements</strong></h3><ul><li>I want to use a particular device type in my pod.</li><li>I should be able to use that device without writing custom Kubernetes code.</li><li>I want a consistent and portable solution to consume hardware devices across k8s clusters.</li></ul><p>This can be achieved by using the “Device Plugin”. Instead of customizing the code for Kubernetes itself, vendors can implement a device plugin that you deploy either manually or as a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset">DaemonSet</a>. The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters, and other similar computing resources that may require vendor-specific initialization and setup</p><p>The <em>SRIOV network device plugin</em> is a Kubernetes device plugin for discovering and advertising SRIOV network virtual functions (VFs) in a Kubernetes host. To deploy workloads with SRIOV VF this plugin needs to work together with the CNI plugins.</p><h3><strong>How does it work?</strong></h3><p>Device plugins are simple gRPC servers that may run in a container deployed through the pod mechanism or in bare metal mode.</p><p>The device plugin is structured in 3 parts:</p><ol><li><strong>Registration</strong>: The device plugin advertises its presence to Kubelet</li><li><strong>ListAndWatch</strong>: The device plugin advertises a list of Devices to Kubelet and sends it again if the state of a Device changes</li><li><strong>Allocate</strong>: When creating containers, Kubelet calls the device plugin’s Allocate function so that it can run device-specific instructions (GPU cleanup, QRNG initialization, ...) and instruct Kubelet how to make the device available in the container.</li></ol><p>These servers implement the gRPC interface and once the device plugin makes itself known to kubelet, kubelet will interact with the device through two simple functions:</p><ol><li>A ListAndWatch function for the kubelet to discover the devices and their properties as well as notify of any status change (device became unhealthy).</li><li>An Allocate function which is called before creating a user container consuming any exported devices</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sz6qfbPa7AxNE2q5fIS_EA.jpeg" /></figure><p><em>PC: Kubernetes Device Manager Proposal</em></p><h3>Installation and Configuration</h3><p>The end result will be similar to the following picture except for the SRIOV-CNI and the DPDK userspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*BxPlAq_P6Ik-KbR2Ny5XPQ.jpeg" /><figcaption>PC: redhat.com</figcaption></figure><p>When setting up the cluster the admin knows what kind of devices are present on the different machines and therefore can select what devices to enable. The cluster-admin knows his cluster has Intel NICs therefore he deploys the device plugin that supports Intel installed NIC through: kubectl create -f deployments/k8s-v1.16/sriovdp-daemonset.yaml</p><p>Note: Make sure that the <em>host-device</em> plugin is installed which is a MUST to move an already-existing device into a container.</p><p><a href="https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin">k8snetworkplumbingwg/sriov-network-device-plugin</a></p><h4>Supported SR-IOV NICs</h4><p>The following NICs were tested with this implementation. However, other SR-IOV capable NICs should work as well.</p><ul><li>Intel® Ethernet Controller X710 Series 4x10G — PF driver: v2.4.6 — VF driver: v3.5.6</li></ul><p>please refer to the Intel download center for installing the latest <a href="https://downloadcenter.intel.com/product/82947/Intel-Ethernet-Controller-X710-Series">Intel Ethernet Controller-X710-Series</a> drivers</p><ul><li>Intel® 82599ES 10 Gigabit Ethernet Controller</li><li>PF driver: v4.4.0-k</li><li>VF driver: v3.2.2-k</li></ul><p>please refer to the Intel download center for installing latest <a href="https://ark.intel.com/products/41282/Intel-82599ES-10-Gigabit-Ethernet-Controller">Intel-® 82599ES 10 Gigabit Ethernet</a> drivers</p><ul><li>Mellanox ConnectX®-4 Lx EN Adapter</li><li>Mellanox ConnectX®-5 Adapter</li></ul><p>Network card drivers are available as a part of the various Linux distributions and upstream. To download the latest Mellanox NIC drivers, click <a href="http://www.mellanox.com/page/software_overview_eth">here</a>.</p><p><strong>Device Driver Information:</strong></p><p><a href="https://www.kernel.org/doc/Documentation/networking/device_drivers/">https://www.kernel.org/doc/Documentation/networking/device_drivers/</a></p><p>The <em>SRIOV network device plugin</em> needs the configuration to discover and advertise SRIOV network virtual functions (VFs) in a Kubernetes host. Please find the configmap as follows,</p><p><strong>sriovdp-config.yaml (AWS)</strong></p><pre># Vendor:Devices<br># 1d0f:0ec2 - ENA PF<br># 1d0f:1ec2 - ENA PF with LLQ support<br># 1d0f:ec20 - ENA VF<br># 1d0f:ec21 - ENA VF with LLQ support</pre><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: sriovdp-config<br>  namespace: kube-system<br>data:<br>  config.json: |<br>    {<br>        &quot;resourceList&quot;: [{<br>                &quot;resourceName&quot;: &quot;amazon_ena&quot;,<br>                &quot;selectors&quot;: {<br>                    &quot;vendors&quot;: [&quot;1d0f&quot;],<br>                    &quot;devices&quot;: [&quot;ec20&quot;],<br>                    &quot;drivers&quot;: [&quot;ena&quot;]<br>                }<br>            }<br>        ]<br>    }</pre><p><strong>sriovdp-config.yaml (Intel)</strong></p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: sriovdp-config<br>  namespace: kube-system<br>data:<br>  config.json: |<br>    {<br>        &quot;resourceList&quot;: [{<br>                &quot;resourceName&quot;: &quot;intel_sriov_netdevice&quot;,<br>                &quot;selectors&quot;: {<br>                    &quot;vendors&quot;: [&quot;8086&quot;],<br>                    &quot;devices&quot;: [&quot;154c&quot;, &quot;10ed&quot;],<br>                    &quot;drivers&quot;: [&quot;i40evf&quot;, &quot;ixgbevf&quot;]<br>                }<br>            }<br>        ]<br>    }</pre><p><strong>sriovdp-daemonset.yaml:</strong></p><pre>---<br>apiVersion: v1<br>kind: ServiceAccount<br>metadata:<br>  name: sriov-device-plugin<br>  namespace: kube-system<br><br>---<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: kube-sriov-device-plugin-amd64<br>  namespace: kube-system<br>  labels:<br>    tier: node<br>    app: sriovdp<br>spec:<br>  selector:<br>    matchLabels:<br>      name: sriov-device-plugin<br>  template:<br>    metadata:<br>      labels:<br>        name: sriov-device-plugin<br>        tier: node<br>        app: sriovdp<br>    spec:<br>      hostNetwork: true<br>      hostPID: true<br>      nodeSelector:<br>        beta.kubernetes.io/arch: amd64<br>      tolerations:<br>      - key: node-role.kubernetes.io/master<br>        operator: Exists<br>        effect: NoSchedule<br>      serviceAccountName: sriov-device-plugin<br>      containers:<br>      - name: kube-sriovdp<br>        image: nfvpe/sriov-device-plugin:v3.2<br>        imagePullPolicy: IfNotPresent<br>        args:<br>        - --log-dir=sriovdp<br>        - --log-level=10<br>        securityContext:<br>          privileged: true<br>        volumeMounts:<br>        - name: devicesock<br>          mountPath: /var/lib/kubelet/<br>          readOnly: false<br>        - name: log<br>          mountPath: /var/log<br>        - name: config-volume<br>          mountPath: /etc/pcidp<br>      volumes:<br>        - name: devicesock<br>          hostPath:<br>            path: /var/lib/kubelet/<br>        - name: log<br>          hostPath:<br>            path: /var/log<br>        - name: config-volume<br>          configMap:<br>            name: sriovdp-config<br>            items:<br>            - key: config.json<br>              path: config.json<br><br>---<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: kube-sriov-device-plugin-ppc64le<br>  namespace: kube-system<br>  labels:<br>    tier: node<br>    app: sriovdp<br>spec:<br>  selector:<br>    matchLabels:<br>      name: sriov-device-plugin<br>  template:<br>    metadata:<br>      labels:<br>        name: sriov-device-plugin<br>        tier: node<br>        app: sriovdp<br>    spec:<br>      hostNetwork: true<br>      hostPID: true<br>      nodeSelector:<br>        beta.kubernetes.io/arch: ppc64le<br>      tolerations:<br>      - key: node-role.kubernetes.io/master<br>        operator: Exists<br>        effect: NoSchedule<br>      serviceAccountName: sriov-device-plugin<br>      containers:<br>      - name: kube-sriovdp<br>        image: nfvpe/sriov-device-plugin:ppc64le<br>        imagePullPolicy: IfNotPresent<br>        args:<br>        - --log-dir=sriovdp<br>        - --log-level=10<br>        securityContext:<br>          privileged: true<br>        volumeMounts:<br>        - name: devicesock<br>          mountPath: /var/lib/kubelet/<br>          readOnly: false<br>        - name: log<br>          mountPath: /var/log<br>        - name: config-volume<br>          mountPath: /etc/pcidp<br>      volumes:<br>        - name: devicesock<br>          hostPath:<br>            path: /var/lib/kubelet/<br>        - name: log<br>          hostPath:<br>            path: /var/log<br>        - name: config-volume<br>          configMap:<br>            name: sriovdp-config<br>            items:<br>            - key: config.json<br>              path: config.json<br>---<br>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: kube-sriov-device-plugin-arm64<br>  namespace: kube-system<br>  labels:<br>    tier: node<br>    app: sriovdp<br>spec:<br>  selector:<br>    matchLabels:<br>      name: sriov-device-plugin<br>  template:<br>    metadata:<br>      labels:<br>        name: sriov-device-plugin<br>        tier: node<br>        app: sriovdp<br>    spec:<br>      hostNetwork: true<br>      hostPID: true<br>      nodeSelector:<br>        beta.kubernetes.io/arch: arm64<br>      tolerations:<br>      - key: node-role.kubernetes.io/master<br>        operator: Exists<br>        effect: NoSchedule<br>      serviceAccountName: sriov-device-plugin<br>      containers:<br>      - name: kube-sriovdp<br>        image: alexeyperevalov/arm64-sriov-device-plugin<br>        imagePullPolicy: IfNotPresent<br>        args:<br>        - --log-dir=sriovdp<br>        - --log-level=10<br>        securityContext:<br>          privileged: true<br>        volumeMounts:<br>        - name: devicesock<br>          mountPath: /var/lib/kubelet/<br>          readOnly: false<br>        - name: log<br>          mountPath: /var/log<br>        - name: config-volume<br>          mountPath: /etc/pcidp<br>      volumes:<br>        - name: devicesock<br>          hostPath:<br>            path: /var/lib/kubelet/<br>        - name: log<br>          hostPath:<br>            path: /var/log<br>        - name: config-volume<br>          configMap:<br>            name: sriovdp-config<br>            items:<br>            - key: config.json<br>              path: config.json</pre><p>The device plugin lands on all the nodes of the cluster and if it detects that there are no VFs it terminates (assuming restart: OnFailure). However, when there are VFs it reports them to Kubelet and starts its gRPC server to monitor devices and hook into the container creation process.</p><p>Devices reported by Device Plugins are advertised as Extended resources of the shape vendor-domain/vendor-device.</p><h3><strong>Install Multus CNI</strong></h3><p>Multus CNI is a container network interface (CNI) plugin for Kubernetes that enables attaching multiple network interfaces to pods. Typically, in Kubernetes, each pod only has one network interface (apart from a loopback) — with Multus you can create a multi-homed pod that has multiple interfaces. This is accomplished by Multus acting as a “meta-plugin”, a CNI plugin that can call multiple other CNI plugins.</p><p><a href="https://github.com/intel/multus-cni#quickstart-installation-guide">intel/multus-cni</a></p><h3><strong>Kubernetes Deployment</strong></h3><p>Deploy the application with annotations so that the discovered VF will be attached to the POD.</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: hpsample</pre><pre>---<br>apiVersion: &quot;k8s.cni.cncf.io/v1&quot;<br>kind: NetworkAttachmentDefinition<br>metadata:<br>  name: sriov-dev1<br>  namespace: hpsample<br>  annotations:<br>    k8s.v1.cni.cncf.io/resourceName: intel.com/amazon_ena<br>spec:<br>  config: |<br>    {<br>        &quot;type&quot;: &quot;host-device&quot;,<br>        &quot;cniVersion&quot;: &quot;0.3.0&quot;,<br>        &quot;name&quot;: &quot;sriov-network&quot;,<br>        &quot;ipam&quot;: {}<br>    }</pre><pre>---<br>apiVersion: extensions/v1beta1<br>kind: Deployment<br>metadata:<br>  name: hp-deployment-v1<br>  namespace: hpsample<br>spec:<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        name: hp-deployment<br>        version: v1<br>      annotations:<br>        k8s.v1.cni.cncf.io/networks: hpsample/sriov-dev1<br>    spec:<br>      containers:<br>      - name: hpcontainer<br>        image: busybox:1.28<br>        imagePullPolicy: IfNotPresent<br>        command: [&#39;sh&#39;, &#39;-c&#39;, &#39;echo The app is running! &amp;&amp; sleep 3600&#39;]<br>        resources:<br>          requests:<br>            intel.com/amazon_ena: &#39;1&#39;<br>          limits:<br>            intel.com/amazon_ena: &#39;1&#39;<br>        securityContext:<br>          privileged: true</pre><h3>References</h3><p><a href="https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/">https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/</a></p><p><a href="https://github.com/containernetworking/plugins">https://github.com/containernetworking/plugins</a></p><p><a href="https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin">https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin</a></p><p><a href="https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/master/docs/vf-setup.md">https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/master/docs/vf-setup.md</a></p><p><a href="https://github.com/kubernetes/community/tree/master/contributors/design-proposals">https://github.com/kubernetes/community/tree/master/contributors/design-proposals</a></p><p><a href="https://docs.openstack.org/mitaka/networking-guide/config-sriov.html">https://docs.openstack.org/mitaka/networking-guide/config-sriov.html</a></p><p><a href="https://docs.openshift.com/container-platform">https://docs.openshift.com/container-platform</a></p><p><a href="https://github.com/Mellanox/docker-sriov-plugin">https://github.com/Mellanox/docker-sriov-plugin</a></p><p><a href="https://www.openness.org/docs/doc/enhanced-platform-awareness/openness-sriov-multiple-interfaces">https://www.openness.org/docs/doc/enhanced-platform-awareness/openness-sriov-multiple-interfaces</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f494cef3f8e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>