<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Allan John on Medium]]></title>
        <description><![CDATA[Stories by Allan John on Medium]]></description>
        <link>https://medium.com/@allanjohn909?source=rss-79dade5b4723------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tmqVFJk3s9RDg804ilrt7A.png</url>
            <title>Stories by Allan John on Medium</title>
            <link>https://medium.com/@allanjohn909?source=rss-79dade5b4723------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:25:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@allanjohn909/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Integrating Cilium with Gateway API, IPv6, and BGP for Advanced Networking Solutions]]></title>
            <link>https://allanjohn909.medium.com/integrating-cilium-with-gateway-api-ipv6-and-bgp-for-advanced-networking-solutions-5b41b0ca0090?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b41b0ca0090</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ipv6]]></category>
            <category><![CDATA[kubernetes-networking]]></category>
            <category><![CDATA[cilium]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 23:10:31 GMT</pubDate>
            <atom:updated>2025-09-01T17:50:42.291Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2luavVVsbmGenG7aEdUng.jpeg" /></figure><h3>Introduction</h3><p>This document is similar to <a href="https://allanjohn909.medium.com/harnessing-the-power-of-cilium-a-guide-to-bgp-integration-with-gateway-api-on-ipv4-7b0d058a1c0d">a previous document</a>, I have already published with the sane configuration but with ipv4. So if you already read the article, then you will notice a lot of text is being copied from there ;)</p><p>The setup architecture is similar to the previous document, a k3d cluster with 2 agents.</p><h3>Setup</h3><p>The setup is going to be</p><ul><li>a K3d cluster without CNI and kube-proxy disabled.</li><li>Cilium is installed as CNI which also takes care of the routing which was handled by kube-proxy. BGP control plane is enabled, and a set of IP’s are configured for Loadbalancers and advertised with BGP.</li><li>Deploy <a href="https://cilium.io/use-cases/gateway-api/">Gateway API</a> instead of Ingress controller and test a single Gateway.</li><li><a href="https://frrouting.org/">FRR</a> is deployed on the host machine to connect to these Loadbalancer IPs. Our setup looks like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*TQCCO5aUHmG9u7XFIdWmJQ.png" /></figure><p>Like the setup in the previous document, we follow the same setup. We configure Cilium to act as a BGP peer and then advertise any new Loadbalancer IPs to the docker bridge IP(172.50.0.1/32) and FRR will be configured to listen to the K3D node IPs with the docker bridge IP(172.50.0.1/32) as the router. So when the connection is established between FRR and cilium BGP, FRR will publish these new routes to the system route table, and the LoadBalancer IPs will be reachable via host system. Then we use HAProxy to make the Loadbalancer IP reachable from internet via the public IP of the server, because the IPs are not publicly routable</p><p>Due to limitations on pure IPv6 setup, on cilium, or because of the underlying host configuration(I tested with IPv6), Its not possible to setup IPv6 routing between docker and FRR. This is why we setup BGP communication to be over IPv4.</p><p>There was a lot of research in reddit forums and other tech forums, documentations and many trial and error configuration to end up with this configuration that works. You can use this configuration as a baseline and work on different configurations to make it work for use case.</p><h3>Prerequistes</h3><p><strong>Enable IPv6 on Docker</strong></p><p>Modify the /etc/docker/daemon.json file and add the following information:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0032dd0795f5a26da1dca518fb96e0dd/href">https://medium.com/media/0032dd0795f5a26da1dca518fb96e0dd/href</a></iframe><p>Since IPv6 IPs are supposed to be internet routable, there are some limitations with IPv6 as private IPs and docker also is not configured to work properly with IPv6 right out of the box. So for testing purposes, I have configured a random range. If you are testing, you can can use the same range or different range of your choice</p><p><strong>Enable IPv6 kernel modules</strong></p><p>Since cilium uses IPtables to write routes to kernel, we need to enable additional kernel modules</p><pre>sudo modprobe -v ip6table_filter<br>sudo modprobe -v ip6_tables<br>sudo modprobe -v ip6table_mangle<br>sudo modprobe -v ip6table_raw<br>sudo modprobe -v iptable_nat<br>sudo modprobe -v ip6table_nat<br>sudo modprobe -v iptable_filter<br>sudo modprobe -v xt_socket</pre><p>Add this configuration in kernel modules configuration file to persist the modules after restart</p><pre>echo &quot;ip6table_filter<br>ip6_tables<br>ip6table_mangle<br>ip6table_raw<br>iptable_nat<br>ip6table_nat<br>iptable_filter<br>xt_socket&quot; | sudo tee /etc/modules-load.d/modules.conf</pre><p>Check if the modules are loaded:</p><pre>lsmod | grep xt_socket<br><br>xt_socket              16384  0<br>nf_socket_ipv4         16384  1 xt_socket<br>nf_socket_ipv6         20480  1 xt_socket<br>nf_defrag_ipv6         24576  2 nf_conntrack,xt_socket<br>nf_defrag_ipv4         16384  2 nf_conntrack,xt_socket<br>x_tables               57344  9 ip6table_filter,ip6table_raw,iptable_filter,ip6table_nat,xt_socket,ip6_tables,ip_tables,iptable_nat,ip6table_mangle</pre><p><strong>Ensure sysctl parameters are enabled</strong></p><pre>echo &quot;net.core.devconf_inherit_init_net=1<br>net.netfilter.nf_conntrack_max=196608<br>net.ipv4.conf.all.forwarding = 1<br>net.ipv6.conf.all.forwarding = 1&quot; | sudo tee /etc/sysctl.d/01-sysctl.conf &gt; /dev/null<br><br>sudo sysctl -p</pre><p>Tools required before starting</p><ul><li>helm</li><li>k3d</li><li>docker</li><li>cilium-cli (version &gt;1.14.1)</li><li>frr (Can be used with Kube router or bird. Haven’t tried them yet. so..)</li><li>haproxy</li></ul><h3>Lets deploy:</h3><blockquote>All configs and scripts can be found in this <a href="https://github.com/lovelinuxalot/local-k8s-tests/tree/main/k3d-cilium-gateway-ipv6">github repo</a></blockquote><p>Create a docker network to start with. Make sure you are enabling IPv6. We are also using IPv4 addresses for the host machines</p><pre>docker network create \<br>      --driver bridge \<br>      --subnet &quot;172.50.0.0/16&quot; \<br>      --gateway &quot;172.50.0.1&quot; \<br>      --ip-range &quot;172.50.0.0/16&quot; \<br>      --ipv6 \<br>      --subnet &quot;2001:3200:3200::/64&quot; \<br>      &quot;cilium&quot;</pre><p>Since K3D runs with docker, and docker has issues with IPv6, in our setup, we will run docker with IPv4 Gateway, which means the docker containers will have both IPv4 and IPv6 stack, however we will use the IPv6 the maximum and only use IPv4, when there is a limitation.</p><p>To create a k3d cluster with cilium, we first need to run some commands to mount bpf anf cgroups. Its better if we run the commands by an k3d-entrypoint-cilium.sh for k3d, rather running the commands after the container for k3d is up. So we create a script and mount it as entrypoint.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e78ba7e31f5604c717edb4e4af58440/href">https://medium.com/media/3e78ba7e31f5604c717edb4e4af58440/href</a></iframe><p>Create a k3d config file</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0b2760e3dfd216615340b43df63384b/href">https://medium.com/media/c0b2760e3dfd216615340b43df63384b/href</a></iframe><blockquote>Note: K3D config does not support relative paths. Fix will be to replace the volume mount path to an absolute path.</blockquote><p>Deploy k3d cluster</p><pre>k3d cluster create -c k3d-ipv6-config.yaml</pre><p>When the cluster is deployed, there will be only couple of pods in pending state. This is normal as we need to deploy a CNI.</p><pre>docker ps<br>CONTAINER ID   IMAGE                      COMMAND                  CREATED         STATUS         PORTS                     NAMES<br>cb7da708c3bb   rancher/k3s:v1.30.1-k3s1   &quot;/bin/k3d-entrypoint…&quot;   5 minutes ago   Up 4 minutes                             k3d-cilium-cluster-agent-1<br>86699ebe1fbb   rancher/k3s:v1.30.1-k3s1   &quot;/bin/k3d-entrypoint…&quot;   5 minutes ago   Up 4 minutes                             k3d-cilium-cluster-agent-0<br>b5856ce06ef3   rancher/k3s:v1.30.1-k3s1   &quot;/bin/k3d-entrypoint…&quot;   5 minutes ago   Up 5 minutes   0.0.0.0:42663-&gt;6443/tcp   k3d-cilium-cluster-server-0<br><br>kubectl get pods -A<br>NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE<br>kube-system   coredns-576bfc4dc7-j6v46                  0/1     Pending   0          5m17s<br>kube-system   local-path-provisioner-75bb9ff978-xqcdj   0/1     Pending   0          5m17s<br>kube-system   metrics-server-557ff575fb-lllqt           0/1     Pending   0          5m17s</pre><p>And for Cilium, we need a couple of CRDs to be installed first.</p><pre>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_gatewayclasses.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_gateways.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_httproutes.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.1.0/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml</pre><p>Cilium will be installed as a Helm chart. The values for cilium:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/010c7230d1c4ae3ae7e479af84103412/href">https://medium.com/media/010c7230d1c4ae3ae7e479af84103412/href</a></iframe><p>Notable values:</p><p>tunnel: disabled : Tunneling does not work properly with IPv6, so its best to disable it for now, until cilium issues a proper fix</p><p>bgpControlPlane.enabled: true : We are deploying latest rc version, to test the BGPClusterConfig which replaces BGPPeerConfig</p><p>enableIPv6BIGTCP: true : This is something I want to test the performance. This is something to improve the performance of number of transactions per second. This would be better over JumboFrames, where all network devices in the network should support JumboFrames.</p><p>In the latest version, Envoy is deployed along with the cilium pods. Since the scope of the story is to deploy a pure IPv6 cluster, I am not exploring if i want to use it or get rid off it.</p><p>Installing helm chart now:</p><pre>helm upgrade --install cilium cilium/cilium --version 1.16.0-rc.0 \<br>       --namespace=kube-system -f cilium-values.yaml</pre><p>Wait for a while, so that the cilium operator will get installed, and then cilium pods will be deployed to each node and then all pods should be in running state.</p><pre>kubectl get pods -A<br>NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE<br>kube-system   local-path-provisioner-75bb9ff978-xqcdj   1/1     Running   0          35m<br>kube-system   coredns-576bfc4dc7-j6v46                  1/1     Running   0          35m<br>kube-system   metrics-server-557ff575fb-lllqt           1/1     Running   0          35m<br>kube-system   cilium-operator-5bdff49868-nhl2m          1/1     Running   0          8m39s<br>kube-system   cilium-bfjp2                              1/1     Running   0          8m39s<br>kube-system   cilium-frv2r                              1/1     Running   0          8m39s<br>kube-system   cilium-znx8l                              1/1     Running   0          8m39s<br>kube-system   cilium-envoy-wdmgs                        1/1     Running   0          8m39s<br>kube-system   cilium-envoy-jk5tq                        1/1     Running   0          8m39s<br>kube-system   cilium-envoy-xtqlt                        1/1     Running   0          8m39s<br>kube-system   hubble-ui-59bb4cb67b-bz72q                2/2     Running   0          8m39s<br>kube-system   hubble-relay-75fb6597d7-7gzln             1/1     Running   0          8m39s</pre><p>Cilium status can also be checked with the cilium-cli tool</p><pre>cilium status<br>    /¯¯\<br> /¯¯\__/¯¯\    Cilium:             OK<br> \__/¯¯\__/    Operator:           OK<br> /¯¯\__/¯¯\    Envoy DaemonSet:    OK<br> \__/¯¯\__/    Hubble Relay:       OK<br>    \__/       ClusterMesh:        disabled<br><br>DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3<br>Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1<br>DaemonSet              cilium-envoy       Desired: 3, Ready: 3/3, Available: 3/3<br>Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1<br>Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1<br>Containers:            cilium-envoy       Running: 3<br>                       hubble-relay       Running: 1<br>                       cilium-operator    Running: 1<br>                       cilium             Running: 3<br>                       hubble-ui          Running: 1<br>Cluster Pods:          5/5 managed by Cilium<br>Helm chart version:<br>Image versions         cilium             quay.io/cilium/cilium:v1.16.0-rc.0@sha256:bc88ac635a871293d5d2837196e53adba1ea55f79cd3f5cba802dd488312fd2a: 3<br>                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.13.1@sha256:0e0eed917653441fded4e7cdb096b7be6a3bddded5a2dd10812a27b1fc6ed95b: 1<br>                       hubble-ui          quay.io/cilium/hubble-ui:v0.13.1@sha256:e2e9313eb7caf64b0061d9da0efbdad59c6c461f6ca1752768942bfeda0796c6: 1<br>                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.29.5-8fccf45a8ab9da13824e0f14122d5db35673f3bb@sha256:f2c0b275aebe14c7369c8396c4461c787b12b823aba0c613ebbed7a3f92f288e: 3<br>                       hubble-relay       quay.io/cilium/hubble-relay:v1.16.0-rc.0@sha256:22b7f87db6a7a00d10e4ad8c316324368693b0e7f158055b7f81f39fb27928e2: 1<br>                       cilium-operator    quay.io/cilium/operator-generic:v1.16.0-rc.0@sha256:78b9951cd6d92e7c954b9d7d2791cf52c83895441147deec3906c03363fd1169: 1</pre><p>Now we need to define a set of IPs for Loadbalancer ranges, when services are created as type Loadbalancer or for Gateway API</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/142a43d4ce88419ea0f9cd9ce357b42b/href">https://medium.com/media/142a43d4ce88419ea0f9cd9ce357b42b/href</a></iframe><pre>kubectl apply -f ippool.yaml</pre><p>With IPv4, cilium automatically figures out the IP of the node to set as Router-id for bgp configuration. With IPv6, its not really automated, so we need to annotate nodes with setting the IPs manually.</p><pre>kubectl annotate node k3d-cilium-cluster-agent-1 --overwrite cilium.io/bgp-virtual-router.64512=&quot;router-id=172.50.0.1&quot;<br>kubectl annotate node k3d-cilium-cluster-agent-0 --overwrite cilium.io/bgp-virtual-router.64512=&quot;router-id=172.50.0.1&quot;<br>kubectl annotate node k3d-cilium-cluster-server-0 --overwrite cilium.io/bgp-virtual-router.64512=&quot;router-id=172.50.0.1&quot;</pre><p>Now to use the BGP Control Plane v2, Instead of a single BGP Peering Policy, we need to define a couple of Kubernetes Objects. You can find more information <a href="https://docs.cilium.io/en/latest/network/bgp-control-plane/bgp-control-plane-v2/">here</a> on how it works</p><p>So we need a couple of configurations</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2ad402aa06f26680b222eaabde8baae/href">https://medium.com/media/c2ad402aa06f26680b222eaabde8baae/href</a></iframe><p>This is the BGP cluster config, here we define the label for nodes to apply the rule. and bgpInstances. The peerAddress is the IPv6 gateway of the docker network and this clusterConfig refers to a peerConfig, which is described below.</p><p>Now with v2 configuration we can apply separate rules or different configuration for BGP within a single cluster which was not possible with v1 configuration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4864c98e6252df51fc2dd133a35daa6/href">https://medium.com/media/d4864c98e6252df51fc2dd133a35daa6/href</a></iframe><p>The Peer config references BGP Advertisement config which is selected with a labelSelector</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1153541b5ee216ae6911fb78a8a2fdc/href">https://medium.com/media/d1153541b5ee216ae6911fb78a8a2fdc/href</a></iframe><p>This config will advertise routes for all services that have Loadbalancer type. I have disabled the selector, because I want to deploy a Gateway API and Ineed the routes be advertised. I could also add a label to the Gateway service, but I am playing around :D</p><p>Lets deploy the config</p><pre>kubectl apply -f bgp-config.yaml</pre><p>To test, we will create a new service pointing to the hubble service and then create a Gateway and add an HTTP route to it</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45c45a17c408940dabf8784d58abfb82/href">https://medium.com/media/45c45a17c408940dabf8784d58abfb82/href</a></iframe><p>Note that I have added a selector for Gateway to be applied to namespaces with shared-gw: true label. So now we need to annotate it and apply the config</p><pre>kubectl label namespace kube-system shared-gw=true<br>kubectl apply -f hubble.yaml</pre><p>lets now check Gateway, HTTPRoute and services</p><pre>kubectl -n kube-system get gateway,svc,httproute<br>NAME                                               CLASS    ADDRESS   PROGRAMMED   AGE<br>gateway.gateway.networking.k8s.io/shared-gw-ipv6   cilium   2004::1   True         13m<br><br>NAME                                    TYPE           CLUSTER-IP           EXTERNAL-IP   PORT(S)                  AGE<br>service/kube-dns                        ClusterIP      1001:cafe:43:9::a    &lt;none&gt;        53/UDP,53/TCP,9153/TCP   96m<br>service/metrics-server                  ClusterIP      1001:cafe:43:9::be   &lt;none&gt;        443/TCP                  96m<br>service/cilium-envoy                    ClusterIP      None                 &lt;none&gt;        9964/TCP                 69m<br>service/hubble-peer                     ClusterIP      1001:cafe:43:9::87   &lt;none&gt;        443/TCP                  69m<br>service/hubble-relay                    ClusterIP      1001:cafe:43:9::cf   &lt;none&gt;        80/TCP                   69m<br>service/hubble-ui                       ClusterIP      1001:cafe:43:9::80   &lt;none&gt;        80/TCP                   69m<br>service/hubble-ipv6                     ClusterIP      1001:cafe:43:9::a1   &lt;none&gt;        80/TCP                   13m<br>service/cilium-gateway-shared-gw-ipv6   LoadBalancer   1001:cafe:43:9::cc   2004::1       80:30604/TCP             13m<br><br>NAME                                                 HOSTNAMES                AGE<br>httproute.gateway.networking.k8s.io/hubble-ipv6-gw   [&quot;hubble.example.com&quot;]   13m</pre><p>As you can see, the gateway has one Loadbalancer IP which was from the assigned Loadbalancer IPPools which we added earlier and the same IP can be seen coming from the service. Also the httproute has got a DNS name assigned to it</p><p>Lets install FRR for advertising and receiving routes.</p><pre>sudo apt install frr -y</pre><p>Enable BGP for frr, by editing /etc/frr/daemons and change the value of bgpd from no to yes</p><p>Edit the file /etc/frr/frr.conf and add the following config</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68057cb89a7563288984e0e93855cbb9/href">https://medium.com/media/68057cb89a7563288984e0e93855cbb9/href</a></iframe><ul><li>We are setting the localASN number as 64513, because we used this number as the peerASN in cilium</li><li>We set the peer router-id as the gateway IP of the docker network created for the cluster</li><li>We define each node in the cluster as neighbour in the configuration to receive announcements</li></ul><p>Now start frr</p><pre>sudo systemctl start frr</pre><p>After a couple of seconds you can check the status in both cilium and frr for peering status</p><pre>cilium bgp peers<br>Node                          Local AS   Peer AS   Peer Address        Session State   Uptime   Family         Received   Advertised<br>k3d-cilium-cluster-agent-0    64512      64513     2001:3200:3200::1   established     15m38s   ipv6/unicast   1          2<br>k3d-cilium-cluster-agent-1    64512      64513     2001:3200:3200::1   established     15m28s   ipv6/unicast   1          2<br>k3d-cilium-cluster-server-0   64512      64513     2001:3200:3200::1   established     15m38s   ipv6/unicast   1          2</pre><pre>sudo vtysh -c &quot;show bgp summary&quot;<br><br>IPv4 Unicast Summary:<br>BGP router identifier 172.50.0.1, local AS number 64513 VRF default vrf-id 0<br>BGP table version 0<br>RIB entries 0, using 0 bytes of memory<br>Peers 3, using 60 KiB of memory<br>Peer groups 1, using 64 bytes of memory<br><br>Neighbor          V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc<br>2001:3200:3200::2 4      64512        35        35        0    0    0 00:15:56        NoNeg    NoNeg N/A<br>2001:3200:3200::3 4      64512        35        35        0    0    0 00:15:56        NoNeg    NoNeg N/A<br>2001:3200:3200::4 4      64512        35        35        0    0    0 00:15:46        NoNeg    NoNeg N/A<br><br>Total number of neighbors 3<br><br>IPv6 Unicast Summary:<br>BGP router identifier 172.50.0.1, local AS number 64513 VRF default vrf-id 0<br>BGP table version 1<br>RIB entries 1, using 96 bytes of memory<br>Peers 3, using 60 KiB of memory<br>Peer groups 1, using 64 bytes of memory<br><br>Neighbor          V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc<br>2001:3200:3200::2 4      64512        35        35        1    0    0 00:15:56            1        1 N/A<br>2001:3200:3200::3 4      64512        35        35        1    0    0 00:15:56            1        1 N/A<br>2001:3200:3200::4 4      64512        35        35        1    0    0 00:15:46            1        1 N/A<br><br>Total number of neighbors 3</pre><p>We can see that the routes are being advertised and received. If there is something wrong with the connectivity, we can see the advertised and recevied routes to be 0 from cilium-cli and Up/Down status will show a “Never” state.</p><p>Lets check the routes, to see if FRR is creating proper routes:</p><pre>ip -6 r s<br>::1 dev lo proto kernel metric 256 pref medium<br>2001:db8::/64 dev docker0 metric 1024 linkdown pref medium<br>2001:3200::/64 dev docker0 proto kernel metric 256 linkdown pref medium<br>2001:3200::/64 dev docker0 metric 1024 linkdown pref medium<br>2001:3200:3200::/64 dev br-455d8ca1d613 proto kernel metric 256 pref medium<br>2001:3200:3200::/64 dev docker0 metric 1024 linkdown pref medium<br>2001:3200:3200::/56 dev docker0 metric 1024 linkdown pref medium<br>2004::1 nhid 32 proto bgp metric 20 pref medium<br> nexthop via 2001:3200:3200::4 dev br-455d8ca1d613 weight 1<br> nexthop via 2001:3200:3200::2 dev br-455d8ca1d613 weight 1<br> nexthop via 2001:3200:3200::3 dev br-455d8ca1d613 weight 1</pre><p>Now lets do a curl to see if we can view the page</p><pre>curl --connect-to &#39;hubble.example.com:80:[2004::1]:80&#39; http://hubble.example.com<br>&lt;!doctype html&gt;&lt;html&gt;&lt;head&gt;&lt;meta charset=&quot;utf-8&quot;/&gt;&lt;title&gt;Hubble UI&lt;/title&gt;&lt;base href=&quot;/&quot;/&gt;&lt;meta name=&quot;color-scheme&quot; content=&quot;only light&quot;/&gt;&lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=edge&quot;/&gt;&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,user-scalable=0,initial-scale=1,minimum-scale=1,maximum-scale=1&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;32x32&quot; href=&quot;favicon-32x32.png&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;16x16&quot; href=&quot;favicon-16x16.png&quot;/&gt;&lt;link rel=&quot;shortcut icon&quot; href=&quot;favicon.ico&quot;/&gt;&lt;script defer=&quot;defer&quot; src=&quot;bundle.main.eae50800ddcd18c25e9e.js&quot;&gt;&lt;/script&gt;&lt;link href=&quot;bundle.main.1d051ccbd0f5cd57832e.css&quot; rel=&quot;stylesheet&quot;&gt;&lt;/head&gt;&lt;body&gt;&lt;div id=&quot;app&quot; class=&quot;test&quot;&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt;</pre><h3>HAProxy Setup</h3><p>Lets try to make it DNS resolvable. One way to do it is to update the domain AAAA record with the IP address of the server and use HAProxy to route the request to backend Loadbalancer IP that is advertised. If done properly, with support from ISP, then we can actually use publicly routable IP address range and use it, and run something else, other than docker, like testing on a proper cluster.</p><p>When running on cloud, we can make use of external DNS to update the records. In local setups, HAProxy is king :D</p><p>So lets test with HAProxy</p><pre>sudo apt install haproxy -y</pre><p>Now replace /etc/haproxy/haproxy.cfg file with the following config</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/56a62741fa63d5d4470cf38026e28107/href">https://medium.com/media/56a62741fa63d5d4470cf38026e28107/href</a></iframe><p>Here we will then choose to update the DNS to point to the public IPv6 address of the server or machine that we are running. However, to make it more simple, I will add to my hosts file, so I can check on my browser to confirm it works.</p><p>Now start HAProxy</p><pre>sudo systemctl start haproxy</pre><p>On my local machine, I will add to my hosts file</p><pre>echo &quot;xxxx:xxxx:xxxx:xxxx:0000:0000:0000:0001 hubble.example.com&quot; | sudo tee -a /etc/hosts&gt;/dev/null</pre><p>Go to browser and go to the domain “hubble.example.com”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frDNLUbJRfEiDiLKyamvyg.png" /></figure><blockquote><em>Please note that the domain name I used is a test one, please replace it with your own domain name. The domain name from registrar and on the http-route should match</em></blockquote><p>I also tested with and without BigTCP enabled and the performance was comparatively better on random test. I cannot test it properly unless I have a proper setup. To learn more about BigTCP, you can check it <a href="https://isovalent.com/blog/post/big-tcp-on-cilium/">here</a>. Its an amazing article to read about it. For a concise information, BigTCP enables CNI to perform faster by sending data with bigger sizes than default 64kb without any change in network devices, where JumboFrames, also another way for faster performance requires physical network devices that supports JumboFrames</p><h3>Conclusion</h3><p>Cilium as an excellent CNI tool is still proving to be well efficient and with amazing performance and features. Its worth to explore working with pure IPv6 stack and have fun in learning about the limitations that it brings and how to overcome it.</p><p>I hope you enjoyed this blog,</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b41b0ca0090" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harnessing the Power of Cilium: A Guide to BGP Integration with Gateway API on IPv4]]></title>
            <link>https://allanjohn909.medium.com/harnessing-the-power-of-cilium-a-guide-to-bgp-integration-with-gateway-api-on-ipv4-7b0d058a1c0d?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b0d058a1c0d</guid>
            <category><![CDATA[cilium]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[bgp]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 15:10:44 GMT</pubDate>
            <atom:updated>2024-06-27T23:12:43.498Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ych6z4ANpmtSaKWt" /><figcaption>Photo by <a href="https://unsplash.com/@sammoghadamkhamseh?utm_source=medium&amp;utm_medium=referral">Sam Moghadam Khamseh</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>This document describes setting up a simple K3d cluster with Cilium as CNI and Cilium Gateway API instead of Ingress Controller. It also describes using Cilium BGP Control Plane feature to announce Loadbalancer IPs. Before diving in, a small introduction on BGP</p><h3>Border Gateway Protocol</h3><p>Border Gateway Protocol (BGP) is the primary protocol used to exchange routing information between different autonomous systems on the internet. An autonomous system (AS) is a collection of IP networks and routers under the control of a single organisation that presents a common routing policy to the internet. Each router or peer in the network announces the route information, which the peers connected on the other end receives and pass the information on.</p><p>Before thinking BGP is cool, there are slight drawbacks. BGP trusts network operators to use it securely and send the right data. However, it lacks built-in security, so if mistakes or attacks happen, there can be problems.</p><p>Enough with the theory, lets get into the cool stuff</p><h3>Setup</h3><p>The setup is going to be</p><ul><li>a K3d cluster without CNI and kube-proxy disabled.</li><li>Cilium is installed as CNI which also takes care of the routing which was handled by kube-proxy. BGP control plane is enabled, and a set of IP’s are configured for Loadbalancers and advertised with BGP.</li><li>Deploy <a href="https://cilium.io/use-cases/gateway-api/">Gateway API</a> instead of Ingress controller and test a single Gateway.</li><li><a href="https://frrouting.org/">FRR</a> is deployed on the host machine to connect to these Loadbalancer IPs. Our setup looks like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*xTK4FnsRj8t_SWBBPdw2OQ.png" /><figcaption>Architecture for K3D with Cilium</figcaption></figure><p>So in here, we configure Cilium to act as a BGP peer and then advertise any new Loadbalancer IPs to the docker bridge IP(172.50.0.1/32) and FRR will also be configured to listen to the K3D node IPs with the docker bridge IP(172.50.0.1/32) as the router. So when the connection is established between FRR and cilium BGP, FRR will publish these new routes to the system route table, and the LoadBalancer IPs will be reachable via host system. Then we use HAProxy to make the Loadbalancer IP reachable from internet via the public IP of the server, because the IPs are not publicly routable</p><p>Pardon me for this setup, this is just for local setup. When running on cloud, the IPs will be from a public network, which can be reachable from internet, so when the IPs are advertised, they will be reachable from the internet and External DNS can be used to sync the hostnames with DNS Zones.</p><h3>Prerequisites</h3><p>Tools required before starting</p><ul><li>helm</li><li>k3d</li><li>docker</li><li>cilium-cli (version &gt;-1.14.1</li><li>frr</li><li>haproxy</li></ul><p>Some things to check or enable (which i couldnt find in docs, but from github issues)</p><h4>Enable Ipv4 Forwarding:</h4><pre>echo &quot;net.ipv4.ip_forward=1&quot; | sudo tee /etc/sysctl.d/01-sysctl.conf &gt; /dev/null<br>sudo sysctl -p</pre><h4>Enable Socket match kernel module</h4><p>The module is part of netfilter kernel module which matches ipv4 or ipv6 packets to associated sockets.</p><pre>lsmod | grep xt_socket</pre><p>If the output is empty, then enable it</p><pre>sudo modprobe xt_socket -v<br>insmod /lib/modules/6.1.46-0-lts/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko.gz<br>insmod /lib/modules/6.1.46-0-lts/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko.gz<br>insmod /lib/modules/6.1.46-0-lts/kernel/net/netfilter/xt_socket.ko.gz</pre><p>check again if the module is loaded</p><pre>lsmod | grep xt_socket<br>xt_socket 16384 0<br>nf_socket_ipv4 16384 1 xt_socket<br>nf_socket_ipv6 20480 1 xt_socket<br>nf_defrag_ipv6 24576 3 nf_conntrack,xt_socket,xt_TPROXY<br>nf_defrag_ipv4 16384 3 nf_conntrack,xt_socket,xt_TPROXY<br>x_tables 61440 15 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,xt_socket,xt_comment,ip6_tables,xt_TPROXY,xt_CT,iptable_raw,ip_tables,iptable_nat,ip6table_mangle,iptable_mangle,xt_mark</pre><h3>Lets start…</h3><blockquote>All configs and scripts can be found in this <a href="https://github.com/lovelinuxalot/local-k8s-tests/tree/main/k3d-cilium-gateway-ipv4">github repo</a></blockquote><p>First create a separate docker network and call it cilium .</p><pre>docker network create \<br>      --driver bridge \<br>      --subnet &quot;172.50.0.0/16&quot; \<br>      --gateway &quot;172.50.0.1&quot; \<br>      --ip-range &quot;172.50.0.0/16&quot; \<br>      &quot;cilium&quot;</pre><p>To create a k3d cluster with cilium, we first need to run some commands to mount bpf anf cgroups. Its better if we run the commands by an k3d-entrypoint-cilium.sh for k3d, rather running the commands after the container for k3d is up. So we create a script and mount it as entrypoint.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cc83c0b75eeded6ba22b8166364e3c1/href">https://medium.com/media/3cc83c0b75eeded6ba22b8166364e3c1/href</a></iframe><p>Create a k3d config file</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6de5340960f48ff6f01d93df09b2dd62/href">https://medium.com/media/6de5340960f48ff6f01d93df09b2dd62/href</a></iframe><p>Notable configurations:</p><ul><li>kubeAPI.hostIP: 127.172.50.1 This is set like this so that we can access the k3d cluster via the docker interface</li><li>nodeLabels This added so that later on, we can add the BGP peering policy based on the nodes. So that the policy can be applied to certain nodes only.</li><li>Most of the other configs are normal, like disabling default loadbalancer, kube-proxy, flannel-cni, adding a cluster and service cidr</li></ul><p>Deploy k3d cluster</p><pre> k3d cluster create -c k3d-config.yaml</pre><p>When the cluster is deployed, there will be only couple of pods in pending state. This is normal as we need to deploy a CNI.</p><pre>docker ps<br>CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS          PORTS                         NAMES<br>5a17e139be15   rancher/k3s:v1.25.7-k3s1   &quot;/bin/k3d-entrypoint…&quot;   17 seconds ago   Up 9 seconds                                  k3d-cilium-cluster-agent-1<br>073d0f5d9b76   rancher/k3s:v1.25.7-k3s1   &quot;/bin/k3d-entrypoint…&quot;   17 seconds ago   Up 9 seconds                                  k3d-cilium-cluster-agent-0<br>3740871a7ecb   rancher/k3s:v1.25.7-k3s1   &quot;/bin/k3d-entrypoint…&quot;   17 seconds ago   Up 16 seconds   127.172.50.1:6443-&gt;6443/tcp   k3d-cilium-cluster-server-0<br><br>kubectl get pods -A<br>NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE<br>kube-system   coredns-597584b69b-7l77z                  0/1     Pending   0          1s<br>kube-system   metrics-server-5f9f776df5-sftwl           0/1     Pending   0          1s<br>kube-system   local-path-provisioner-79f67d76f8-bg9b8   0/1     Pending   0          1s</pre><p>And for Cilium, we need a couple of CRDs to be installed first.</p><pre>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_gatewayclasses.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_gateways.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_httproutes.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v0.7.0/config/crd/experimental/gateway.networking.k8s.io_referencegrants.yaml</pre><p>Cilium will be installed as a Helm chart. When installing Cilium as CNI as a replacement for kube-proxy, we need to add the k8sServiceHost and k8sServicePort values and set kubeProxyReplacement: strict . Because we need cilium to be the one to do all the network routing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6030cf502f4f7085a6ad2894274c277/href">https://medium.com/media/c6030cf502f4f7085a6ad2894274c277/href</a></iframe><ul><li>ipam.mode: kubernetes : To pool IPs from Kubernetes, other option would be cluster-pool , which will get IPs from a configured CIDR which is useful for multi-cluster configuration. For single cluster, kubernetes will be fine.</li><li>k8s-require-ipv4-pod-cidr : This configuration will make cilium to use the configured cluster-cidr from the cluster. Otherwise, cilium will pick a range of 10.0.0.0 and start using those IPs. For some reason, the cilium IPs are not being announced via BGP if the pod CIDRs are exported (will come to it later)</li><li>tunnel:vxlan This is the default mode, if tunnel is used. This is not necessary for single cluster, but will be good for a multi-cluster setup. Tunnel will also create additional encapsulation, which means its good for small setup, for larger clusters with many applications, then we need to adjust the MTU of the interfaces to use jumbo-frames or value of 9000. (If I am successful with changing this one, I will update or add a new page)</li><li>I have disabled the ingress controllers so that I can specifically install Gateway API</li></ul><p>Install Cilium Helm chart</p><pre>helm upgrade --install cilium cilium/cilium --version 1.14.1 \<br>       --namespace=kube-system -f cilium-values.yaml</pre><p>Wait for a while, so that the cilium operator will get installed, and then cilium pods will be deployed to each node and then all pods should be in running state.</p><pre>kubectl get pods -A<br>NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE<br>kube-system   cilium-operator-d5dd47988-58gn7           1/1     Running   0          59s<br>kube-system   cilium-7bm9g                              1/1     Running   0          59s<br>kube-system   cilium-lm5zc                              1/1     Running   0          59s<br>kube-system   cilium-wh5pf                              1/1     Running   0          59s<br>kube-system   local-path-provisioner-79f67d76f8-bg9b8   1/1     Running   0          2m34s<br>kube-system   coredns-597584b69b-7l77z                  1/1     Running   0          2m34s<br>kube-system   hubble-relay-6bb96bd796-d596q             1/1     Running   0          59s<br>kube-system   metrics-server-5f9f776df5-sftwl           1/1     Running   0          2m34s<br>kube-system   hubble-ui-869b75b895-s2m8f                2/2     Running   0          59s</pre><p>Cilium status can also be checked with the cilium-cli tool</p><pre>cilium status<br>    /¯¯\<br> /¯¯\__/¯¯\    Cilium:             OK<br> \__/¯¯\__/    Operator:           OK<br> /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)<br> \__/¯¯\__/    Hubble Relay:       OK<br>    \__/       ClusterMesh:        disabled<br><br>Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1<br>Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1<br>Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1<br>DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3<br>Containers:            cilium             Running: 3<br>                       hubble-relay       Running: 1<br>                       hubble-ui          Running: 1<br>                       cilium-operator    Running: 1<br>Cluster Pods:          5/5 managed by Cilium<br>Helm chart version:    1.14.1<br>Image versions         hubble-ui          quay.io/cilium/hubble-ui:v0.12.0@sha256:1c876cfa1d5e35bc91e1025c9314f922041592a88b03313c22c1f97a5d2ba88f: 1<br>                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.12.0@sha256:8a79a1aad4fc9c2aa2b3e4379af0af872a89fcec9d99e117188190671c66fc2e: 1<br>                       cilium-operator    quay.io/cilium/operator-generic:v1.14.1@sha256:e061de0a930534c7e3f8feda8330976367971238ccafff42659f104effd4b5f7: 1<br>                       cilium             quay.io/cilium/cilium:v1.14.1@sha256:edc1d05ea1365c4a8f6ac6982247d5c145181704894bb698619c3827b6963a72: 3<br>                       hubble-relay       quay.io/cilium/hubble-relay:v1.14.1@sha256:db30e85a7abc10589ce2a97d61ee18696a03dc5ea04d44b4d836d88bd75b59d8: 1</pre><p>Now we need to define a set of IPs for Loadbalancer ranges, when services are created as type Loadbalancer or for Gateway API</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28e2befedc87336cbe14d04d1299ea58/href">https://medium.com/media/28e2befedc87336cbe14d04d1299ea58/href</a></iframe><p>and install it:</p><pre>kubectl apply -f ippool.yaml</pre><p>Previously, IPPool management and assigning of IPs to Loadbalancer services was handled by MetalLB. Since BGP control plane feature is within cilium, we might not need it. In previous versions of cilium, it would be best to use with MetalLB</p><p>Now we need to define a BGP Peering policy for cilium</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4d49a56553a0ce123e508bda444fc60/href">https://medium.com/media/a4d49a56553a0ce123e508bda444fc60/href</a></iframe><ul><li>nodeSelector is added to apply the peering policy to the nodes that has the label. I have added this label to all nodes for now</li><li>localASN : this is the localASN number for the cilium BGP router</li><li>exportPodCIDR : This switch is to whether advertise the pod IPs or not. Currently we dont need it. Its cool to have it enabled and connect to the pod directly, but this is not really secure IMHO. So keeping it false</li><li>neighbors: This is where we configure the neighbours for the BGP peering. We are setting it to the docker network gateway, as this is the entrypoint of communication between the docker network and host machine.</li><li>peerASN: The ASN number of the peer, it can be the same as the localASN, but for testing and to see if the connection is established, we will use a different private ASN</li><li>serviceSelector: This configuration is to set which service IP’s should be advertised. TO advertise all , the operator should be of the value NotIn</li></ul><p>Apply the Peering policy</p><pre>kubectl apply -f bgpp.yaml</pre><p>Check the status of the bgp with cilium-cli</p><pre>cilium bgp peers<br>Node                          Local AS   Peer AS   Peer Address   Session State   Uptime   Family         Received   Advertised<br>k3d-cilium-cluster-agent-0    64512      64513     172.50.0.1     active          0s       ipv4/unicast   0          0<br>                                                                                           ipv6/unicast   0          0<br>k3d-cilium-cluster-agent-1    64512      64513     172.50.0.1     active          0s       ipv4/unicast   0          0<br>                                                                                           ipv6/unicast   0          0<br>k3d-cilium-cluster-server-0   64512      64513     172.50.0.1     active          0s       ipv4/unicast   0          0<br>                                                                                           ipv6/unicast   0          0</pre><p>As you can see the Session State is active, which means the BGP router is running in cilium. Now to configure on the host machine as the next router, we will install and configure frr</p><pre>sudo apt install frr -y</pre><p>Enable BGP for frr, by editing /etc/frr/daemons and change the value of bgpd from no to yes</p><p>Edit the file /etc/frr/frr.conf and add the following config</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdb0f1ceb10fea5a846725a39fa90ad3/href">https://medium.com/media/cdb0f1ceb10fea5a846725a39fa90ad3/href</a></iframe><ul><li>We are setting the localASN number as 64513, because we used this number as the peerASN in cilium</li><li>We set the peer router-id as the gateway IP of the docker network created for the cluster</li><li>We define each node in the cluster as neighbour in the configuration to receive announcements and set the remote-asn</li></ul><p>Now start frr</p><pre>sudo systemctl start frr</pre><p>After a couple of seconds you can check the status in either cilium or frr for peering status</p><pre>cilium bgp peers<br>Node                          Local AS   Peer AS   Peer Address   Session State   Uptime   Family         Received   Advertised<br>k3d-cilium-cluster-agent-0    64512      64513     172.50.0.1     established     10m49s   ipv4/unicast   3          1<br>                                                                                           ipv6/unicast   0          0<br>k3d-cilium-cluster-agent-1    64512      64513     172.50.0.1     established     10m47s   ipv4/unicast   3          1<br>                                                                                           ipv6/unicast   0          0<br>k3d-cilium-cluster-server-0   64512      64513     172.50.0.1     established     10m51s   ipv4/unicast   3          1<br>                                                                                           ipv6/unicast   0          0</pre><pre>sudo vtysh -c &#39;show bgp summary&#39;<br><br>IPv4 Unicast Summary (VRF default):<br>BGP router identifier 172.50.0.1, local AS number 64513 vrf-id 0<br>BGP table version 18<br>RIB entries 0, using 920 bytes of memory<br>Peers 3, using 4338 KiB of memory<br><br>Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc<br>172.50.0.2      4      64512      1619      1624        0    0    0 00:10:54            1        3 N/A<br>172.50.0.3      4      64512      1619      1620        0    0    0 00:10:52            1        3 N/A<br>172.50.0.4      4      64512      1619      1620        0    0    0 00:10:50            1        3 N/A<br><br><br>Total number of neighbors 3</pre><p>The State/PfxRcd status with frr shows the connection is established with connection time established in Up/Down field. On cilium-cli, the Session State is now changed to established</p><p>If you check the route table, you will see the default routes there, nothing interesting there.</p><pre>route -n<br>Kernel IP routing table<br>Destination     Gateway         Genmask         Flags Metric Ref    Use Iface<br>0.0.0.0         xxx.xxx.xxx.1    0.0.0.0         UG    0      0        0 eth0<br>10.21.0.0       172.50.0.2      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>10.21.1.0       172.50.0.3      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>10.21.2.0       172.50.0.4      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>xxx.xxx.xxx.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0<br>172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0<br>172.50.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-7821810f4fc8</pre><p>Lets create a Gateway and deploy an HTTP-route</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f1b91e5ab5e36a4a22ff7e0eb4cb632/href">https://medium.com/media/6f1b91e5ab5e36a4a22ff7e0eb4cb632/href</a></iframe><ul><li>gatewayClassName: cilium : we will be using the default gateway controller created by the helm chart</li><li>listeners : creating a listener configuration</li><li>allowedRoutes: adding access control, for now I am going to leave it open, we can use Selector and add a label and namespaces with this label will be able to use this gateway for adding HTTPRoutes</li><li>parentRefs: Mapping HTTPRoute to the proper Gateway</li><li>rules: Adding rules for the HTTPRoute configuration</li><li>backendRefs: Connecting the HTTPRoute to the Hubble UI service</li></ul><p>Now deploy the config</p><pre>kubectl apply -f hubble.yaml</pre><p>lets now check Gateway, HTTPRoute and services</p><pre>kubectl get httproute -n kube-system<br>NAME     HOSTNAMES   AGE<br>hubble               1m<br><br>kubectl get gateway -n kube-system<br>NAME        CLASS    ADDRESS         PROGRAMMED   AGE<br>shared-gw   cilium   172.50.201.37   True         1m<br><br>kubectl get services -n kube-system<br>NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                  AGE<br>kube-dns                   ClusterIP      10.201.0.10      &lt;none&gt;          53/UDP,53/TCP,9153/TCP   29m<br>metrics-server             ClusterIP      10.201.130.169   &lt;none&gt;          443/TCP                  29m<br>hubble-peer                ClusterIP      10.201.167.135   &lt;none&gt;          443/TCP                  28m<br>hubble-relay               ClusterIP      10.201.219.116   &lt;none&gt;          80/TCP                   28m<br>hubble-ui                  ClusterIP      10.201.196.229   &lt;none&gt;          80/TCP                   28m<br>cilium-gateway-shared-gw   LoadBalancer   10.201.222.86    172.50.201.37   80:31031/TCP             1m20s</pre><p>Nice, we have a Service Loadbalancer for our Gateway object from cilium</p><p>Check the host route table again</p><pre>route -n<br>Kernel IP routing table<br>Destination     Gateway         Genmask         Flags Metric Ref    Use Iface<br>0.0.0.0         xxx.xxx.xxx.1    0.0.0.0         UG    0      0        0 eth0<br>10.21.0.0       172.50.0.2      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>10.21.1.0       172.50.0.3      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>10.21.2.0       172.50.0.4      255.255.255.0   UG    20     0        0 br-7821810f4fc8<br>xxx.xxx.xxx.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0<br>172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0<br>172.50.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-7821810f4fc8<br>172.50.201.37   172.50.0.2      255.255.255.255 UGH   20     0        0 br-7821810f4fc8</pre><p>Now the advertised route via BGP is updated in the host machine route table</p><p>If you check with FRR, to see the advertised routes by taking a neighbor</p><pre>sudo vtysh -c &#39;show ip bgp neighbors 172.50.0.3 advertised-routes&#39;<br>BGP table version is 19, local router ID is 172.50.0.1, vrf id 0<br>Default local pref 100, local AS 64513<br>Status codes:  s suppressed, d damped, h history, * valid, &gt; best, = multipath,<br>               i internal, r RIB-failure, S Stale, R Removed<br>Nexthop codes: @NNN nexthop&#39;s vrf id, &lt; announce-nh-self<br>Origin codes:  i - IGP, e - EGP, ? - incomplete<br>RPKI validation codes: V valid, I invalid, N Not found<br><br>   Network          Next Hop            Metric LocPrf Weight Path<br>*&gt; 10.21.0.0/24     0.0.0.0                                0 64512 i<br>*&gt; 10.21.1.0/24     0.0.0.0                                0 64512 i<br>*&gt; 10.21.2.0/24     0.0.0.0                                0 64512 i<br>*&gt; 172.50.201.37/32 0.0.0.0                                0 64512 i<br><br>Total number of prefixes 4</pre><p>Now lets do a curl to see if we can view the page</p><pre>curl http://172.50.201.37/<br>&lt;!doctype html&gt;&lt;html&gt;&lt;head&gt;&lt;meta charset=&quot;utf-8&quot;/&gt;&lt;title&gt;Hubble UI&lt;/title&gt;&lt;base href=&quot;/&quot;/&gt;&lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=edge&quot;/&gt;&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,user-scalable=0,initial-scale=1,minimum-scale=1,maximum-scale=1&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;32x32&quot; href=&quot;favicon-32x32.png&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;16x16&quot; href=&quot;favicon-16x16.png&quot;/&gt;&lt;link rel=&quot;shortcut icon&quot; href=&quot;favicon.ico&quot;/&gt;&lt;script defer=&quot;defer&quot; src=&quot;bundle.main.104f057a7d45238d9d45.js&quot;&gt;&lt;/script&gt;&lt;link href=&quot;bundle.main.3818224e482785607640.css&quot; rel=&quot;stylesheet&quot;&gt;&lt;/head&gt;&lt;body&gt;&lt;div id=&quot;app&quot;&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt;</pre><p>And there it is !!!</p><h3>Local DNS testing of route</h3><blockquote>Please note that the domain name I used is a test one, please replace it with your own domain name. The domain name from registrar and on the http-route should match</blockquote><p>Since this is a local setup, we cannot add hostnames to the HTTPRoute and expect that DNS will work. We can make it work by modifying the HTTPRoute to add hostname and point the hostname in /etc/hosts of the host machine. Lets try that now</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a5f7c548ef0f085067ed32590b40714/href">https://medium.com/media/2a5f7c548ef0f085067ed32590b40714/href</a></iframe><p>As you can see, its the same HTTPRoute, but with a slight difference. We add hostnames there. Lets apply and check the status of HTTPRoute</p><pre>kubectl apply -f hubble-with-hostnames.yaml</pre><pre>kubectl get httproute -n kube-system<br>NAME     HOSTNAMES                AGE<br>hubble   [&quot;hubble.example.com&quot;]   22m</pre><p>As you can see the hostname is added. But this will not work now, because the domain name is not resolvable. Lets try to add it in hosts file and check</p><pre>echo &quot;172.50.201.37 hubble.example.com&quot; | sudo tee -a /etc/hosts&gt;/dev/null</pre><p>and now with curl</p><pre>curl hubble.example.com<br>&lt;!doctype html&gt;&lt;html&gt;&lt;head&gt;&lt;meta charset=&quot;utf-8&quot;/&gt;&lt;title&gt;Hubble UI&lt;/title&gt;&lt;base href=&quot;/&quot;/&gt;&lt;meta http-equiv=&quot;X-UA-Compatible&quot; content=&quot;IE=edge&quot;/&gt;&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width,user-scalable=0,initial-scale=1,minimum-scale=1,maximum-scale=1&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;32x32&quot; href=&quot;favicon-32x32.png&quot;/&gt;&lt;link rel=&quot;icon&quot; type=&quot;image/png&quot; sizes=&quot;16x16&quot; href=&quot;favicon-16x16.png&quot;/&gt;&lt;link rel=&quot;shortcut icon&quot; href=&quot;favicon.ico&quot;/&gt;&lt;script defer=&quot;defer&quot; src=&quot;bundle.main.104f057a7d45238d9d45.js&quot;&gt;&lt;/script&gt;&lt;link href=&quot;bundle.main.3818224e482785607640.css&quot; rel=&quot;stylesheet&quot;&gt;&lt;/head&gt;&lt;body&gt;&lt;div id=&quot;app&quot;&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt;</pre><p>Nice!!!</p><p>Before continuing, don’t forget to remove the entries from /etc/hosts</p><h3>HAProxy Setup</h3><p>Lets try to make it DNS resolvable. One way to do it is to update the domain A record with the IP address of the server and use HAProxy to route the request to backend Loadbalancer IP that is advertised</p><p>Like explained before, when running on cloud, we can make use of external DNS to update the records. In local setups, HAProxy is king :D</p><p>So lets test with HAProxy</p><pre>sudo apt install haproxy -y</pre><p>Now replace /etc/haproxy/haproxy.cfg file with the following config</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ef57da3f4e03909e85d574580ca7587/href">https://medium.com/media/0ef57da3f4e03909e85d574580ca7587/href</a></iframe><p>The important part is the last section, where the frontend and backend is defined, the other config is there by default.</p><p>We are checking the domain name and forwarding it to the proper backend configured with LoadBalancer IP</p><p>So now is the time to configure DNS with the public IP of the server and point it to the backend. I replaced my domain name with the example.com domain.</p><p>Now start HAProxy</p><pre>sudo systemctl start haproxy</pre><p>Go to browser and go to the domain “hubble.example.com”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EaOPEEgOYr9QowmqwHnTxA.png" /></figure><blockquote>Please note that the domain name I used is a test one, please replace it with your own domain name. The domain name from registrar and on the http-route should match</blockquote><h3>Conclusion</h3><p>Cilium is an excellent CNI tool and has started to bring Gateway API into existence which has higher functions compared to Ingress controller. The capabilities and features cannot be compared. Here I presented to you how cilium can be utilized in a simple small cluster setup. This can also be used in large scale, but the configuration and the tools will differ.</p><p>I hope you enjoyed this blog</p><blockquote>Update: A pure-IPv6 stack based kubernetes deployment with k3d <a href="https://allanjohn909.medium.com/integrating-cilium-with-gateway-api-ipv6-and-bgp-for-advanced-networking-solutions-5b41b0ca0090">blog</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b0d058a1c0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[eBPF Essentials: A Glimpse Into Kernel Magic]]></title>
            <link>https://allanjohn909.medium.com/ebpf-essentials-a-glimpse-into-kernel-magic-6e90dd6c814f?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e90dd6c814f</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ebpf]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[system-administration]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Thu, 16 Mar 2023 15:00:01 GMT</pubDate>
            <atom:updated>2023-03-17T09:16:58.829Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nk8uqPrz8HbCnmVtj_vAg.jpeg" /><figcaption>Courtesy to unsplash.com</figcaption></figure><h3><strong>Introduction</strong></h3><p>In this blog, I’ll be introducing eBPF, a versatile and efficient technology that has made significant strides within the Linux kernel ecosystem. As we discuss eBPF’s core concepts and a bit of its internals, please note that we’ll touch upon some Linux kernel specifics. My aim is to provide a learning experience that enables you to grasp eBPF’s fundamentals and appreciate its powerful capabilities. Let’s get started!</p><blockquote>This will be a bit long to read as the content was not so much big for 2 blogs. I hope its worth it!</blockquote><h3>What is BPF?</h3><p>Berkeley Packet Filter (BPF) is a technology that originated in 1992, designed to efficiently filter and capture packets within UNIX-based operating systems. It functions as a virtual machine, complete with a distinct instruction set, storage objects, and supportive helper functions. The Linux kernel features a BPF runtime responsible for executing these instructions, using both an interpreter and a Just-In-Time (JIT) compiler that transforms BPF instructions into native executable code. To ensure kernel integrity and prevent crashes, a dedicated verifier rigorously checks BPF instructions for safety before they are executed, regardless of whether they are interpreted or JIT compiled.</p><p>BPF offers an efficient and secure method for interacting with the Linux Kernel, enabling system customization and control through programmable kernel enhancements.</p><p>In layman’s terms, BPF is much like a mini-computer nestled within the larger computer system, seamlessly executing small programs without impacting the main computer’s operations. This concept can be compared to JavaScript, which runs miniature programs on a webpage, such as tracking mouse clicks or scrolling positions. Despite running in the background, JavaScript ensures that the webpage and its users remain unaffected.</p><h3>eBPF: Addressing BPF’s Limitations</h3><p>While BPF proved to be efficient and useful for packet filtering and capturing, it had certain limitations that restricted its scope and adaptability in modern systems:</p><ol><li>Limited instruction set: BPF’s original instruction set was relatively simple and tailored for packet filtering, making it insufficient for more complex tasks and use cases.</li><li>Lack of extensibility: The original BPF framework wasn’t designed for easy extensibility, making it challenging to add new features or adapt it for different applications beyond packet filtering.</li><li>Inflexible data structures: BPF’s initial data structures were quite rigid, limiting the flexibility needed to handle complex data manipulation and analysis in modern systems.</li></ol><p>These limitations prompted the development of eBPF (Extended Berkeley Packet Filter), which addresses these issues and greatly expands BPF’s capabilities:</p><ol><li>Expanded instruction set: eBPF features an enhanced instruction set that allows for more complex operations and supports a broader range of use cases.</li><li>Improved extensibility: The eBPF framework is designed to be more easily extended, <strong>enabling the addition of new features and applications beyond packet filtering, such as performance monitoring, security, and observability.</strong></li><li>Flexible data structures: eBPF introduces new, v<strong>ersatile data structures like maps and arrays</strong>, providing the flexibility needed for sophisticated data manipulation and analysis.</li></ol><p>In short, eBPF was developed to overcome the limitations of the original BPF, offering a more powerful, flexible, and extensible solution that caters to the demands of modern systems.</p><h4>Ok, Why not Kernel Modules?</h4><p>Kernel modules have traditionally been used to extend the functionality of the Linux kernel and its a valid question why don’t we use them instead of eBPF?</p><p>Well, thats where eBPF stands out.</p><ul><li>Kernel modules require reload of the kernel when adding or removing modules. An eBPF program can be added, updated, modified or removed without reloading kernel.</li><li>eBPF programs are checked by a verifier before loading into kernel. Kernel modules on the other hand do not have this feature, which can cause kernel panic errors due to bugs or worse can introduce security vulnerabilities.</li><li>eBPF provides rich and advanced data structures which is better than the Kernel module data structures.</li><li>eBPF programming is easier than kernel engineering because of safety, flexibility, learning curve, tooling and libraries</li></ul><h4>Short history</h4><p>eBPF was conceived by Alexei Starovoitov, who was exploring innovative approaches to develop software-defined networking (SDN) solutions. At its proposal stage, Daniel Borkmann, a kernel engineer at Red Hat, collaborated with Alexei to refine eBPF for integration into the kernel as a versatile virtual machine, ultimately replacing the existing BPF. Since its successful inclusion, eBPF has garnered contributions from numerous developers, further expanding its capabilities and applications.</p><h3>eBPF Applications in real-world</h3><p>eBPF has found numerous real-world applications, enabling developers and operators to gain deeper insights into system performance, improve security, and customize system behavior. Some of these applications include:</p><ol><li><strong>Networking</strong>: eBPF is widely used for implementing software-defined networking (SDN) solutions, load balancing, and network function virtualization (NFV). Companies like Cilium leverage eBPF to create high-performance networking and security solutions for containerized environments, such as Kubernetes.</li><li><strong>Performance monitoring</strong>: eBPF enables the development of advanced performance monitoring and observability tools like BCC (BPF Compiler Collection) and BPFtrace. These tools provide fine-grained visibility into system and application behavior, helping developers identify bottlenecks, optimize performance, and troubleshoot issues.</li><li><strong>Security</strong>: eBPF can be employed for real-time security monitoring and enforcement. It allows for the implementation of custom security policies, intrusion detection systems, and sandboxing solutions that enhance system protection without compromising performance.</li><li><strong>Tracing and profiling</strong>: eBPF can be used to create powerful tracing and profiling tools for various layers of the system, from the kernel to user-space applications. This enables developers to understand how their code interacts with the kernel, identify performance issues, and optimize resource usage.</li></ol><p>These real-world scenarios demonstrate the versatility and potential of eBPF in improving various aspects of modern computing systems, from networking and performance to security and observability.</p><h3>BPF Runtime</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*UkkSHmvmV2xILtcg7-XfMA.png" /><figcaption>BPF Runtime Internals</figcaption></figure><p>This is an illustrations on how eBPF programs are executed. This is an architecture of BPF Runtime.</p><ol><li><strong>BPF custom program</strong>: The custom code written by developers using eBPF’s instruction set to perform specific tasks within the kernel.</li><li><strong>BPF Verifier</strong>: A kernel component that checks the eBPF program for safety and correctness before allowing it to execute.</li><li><strong>BPF JIT compiler</strong>: A Just-In-Time compiler that translates the eBPF bytecode into native machine code for improved performance, if <em>bpf_jit_enabled</em> is true.</li><li><strong>Interpreter</strong>: An alternative execution mode for eBPF programs, which interprets the bytecode directly without JIT compilation. This is when <em>bpf_jit_enabled</em> is false.</li><li><strong>BPF Context</strong>: A data structure passed to an eBPF program when it is executed. The context provides the eBPF program with the necessary information about the event or data it is processing. The contents of the context depend on the type of eBPF program being executed and the specific use case it is designed for. For example, if an eBPF program is attached to a network socket for packet processing, the context would contain information about the packet, such as the source and destination addresses, the protocol used, and the payload.</li><li><strong>Maps</strong>: Key-value data structures that allow eBPF programs to store and share data between the kernel and user space.</li><li><strong>Helper functions</strong>: Kernel-provided functions that eBPF programs can call to perform specific operations, such as accessing network packets or interacting with system resources.</li></ol><h3>How eBPF is efficient?</h3><p>Lets take an example of how an observability tool is used to generate a cli histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*dDrYP19DTt-pK6Fe4JGdfQ.png" /><figcaption>Generate histogram before and using BPF</figcaption></figure><p>Before BPF, to produce a histogram, the steps would be:</p><ol><li>In the kernel: <br>- Enable instrumentation for CPU events.<br>- For each event, write a record to the perf buffer. If tracepoints are used (preferred), the record includes various metadata fields about the CPU.</li><li>In user-space: <br>- Copy the event buffer from the kernel to user space periodically.<br>- Iterate over each event, extracting the ‘bytes’ field from the metadata. Other fields are ignored.<br>- Generate a histogram summary based on the ‘bytes’ field.</li></ol><p>Writing events to per buffer and copying it to user space introduce high-performance overhead for high performance systems, such as transferring ten thousand CPU trace records to user space for parsing and summarization every second.</p><p>With BPF, the steps for the byte size program are simplified:</p><ol><li>In the kernel: <br>- Enable instrumentation for CPU events and attach a custom BPF program, defined by byte size</li><li>In the kernel: <br>- For each event, execute the BPF program. It retrieves only the ‘bytes’ field and stores it in a custom BPF map histogram.<br>- Read the BPF map histogram once and display the output.</li></ol><p>This avoids the cost of copying events to user space and reprocessing them. The only data copied to user space is shown in the below output. The “count” column, which is a sum of number.</p><p>This approach eliminates the need to copy events to user space and reprocess them, as well as the transfer of unused metadata fields. The only data transferred to user space is the “count” column, which is the sum of number.</p><p>This is a sample output of one of the eBPF tool,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e324a4342cb90ba8015115f692408c44/href">https://medium.com/media/e324a4342cb90ba8015115f692408c44/href</a></iframe><h3>Traditional tools vs eBPF tools.</h3><p>Traditional tools and eBPF tools both serve to monitor, troubleshoot, and analyze system performance. However, there are significant differences between the two approaches:</p><h4>1. Scope of information:</h4><ul><li>Traditional tools often rely on pre-existing kernel counters, logs, and interfaces (such as procfs, sysfs, or perf) to gather information. They may provide limited visibility into the system’s internals and may not be customizable to suit specific needs.</li><li>eBPF tools allow for more fine-grained and customizable data collection, enabling developers to create custom programs that access kernel data structures, monitor system events, and even modify kernel behavior.</li></ul><h4>2. Performance impact:</h4><ul><li>Traditional tools can sometimes impose a significant performance overhead when collecting data, particularly if they involve frequent context switches between kernel and user space or require substantial data processing.</li><li>eBPF tools generally have lower performance overhead, as they run directly in the kernel and can process data more efficiently. eBPF programs are also verified for safety and optimized by the JIT compiler for better performance.</li></ul><h4>3. Flexibility and extensibility:</h4><ul><li>Traditional tools are often limited by the capabilities provided by the kernel interfaces they rely on. Customizing or extending these tools can be challenging and may require kernel modifications.</li><li>eBPF tools are highly flexible and extensible, allowing developers to create custom eBPF programs tailored to their specific requirements. The eBPF ecosystem also offers various libraries and frameworks that simplify the development of new eBPF tools.</li></ul><h4>4. Applicability:</h4><ul><li>Traditional tools typically focus on specific tasks or areas of the system (e.g., networking, storage, or CPU performance) and may not be suitable for cross-domain analysis or correlation.</li><li>eBPF tools can be applied to a wide range of use cases, including networking, security, observability, and performance monitoring, making it easier to perform cross-domain analysis and gain deeper insights into system behavior.</li></ul><p>While traditional tools can still be useful for specific tasks or in certain situations, eBPF tools offer greater visibility, flexibility, and performance, making them a preferred choice for many modern monitoring and analysis needs.</p><h3>Real life use-case: Tracing</h3><p>As system admins, we all know what an amazing tool is <strong>strace. </strong>Its a very good tool to check all system calls being called. I personally have used this tool a lot, even on production servers. One thing I came to know, after learning about BPF, was that this command creates a lot of overhead to generate the output and running this tool on a production server can be catastrophic! It might be okay, if there is redundancy.</p><p>Lets take a look at <strong>strace</strong> and <strong>trace</strong>, where trace is an eBPF tool.</p><p>This is an output of <strong>strace</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f528f0af83a55f3e922a5146d5eb2424/href">https://medium.com/media/f528f0af83a55f3e922a5146d5eb2424/href</a></iframe><p>And this is an output from <strong>trace</strong>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c8efba371f8d58de59cc501d08d678f/href">https://medium.com/media/0c8efba371f8d58de59cc501d08d678f/href</a></iframe><p>As you can see, how the data is shown is much more better to view than strace, although strace is a good tool.</p><h3>eBPF in Observability:</h3><p>Some of the common tools used in observability:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d50fd4045f193dc8a301456a523720e1/href">https://medium.com/media/d50fd4045f193dc8a301456a523720e1/href</a></iframe><p>Additional eBPF tools that can be used to drill down analysis:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26331f76206de9f69f78585f4cf177f3/href">https://medium.com/media/26331f76206de9f69f78585f4cf177f3/href</a></iframe><p>These tools can be installed for Linux OS with kernel version above 4.7 either by cloning the git-repo or by installing from package-managers. For more information on the tools, check these pages for bcc tools and bpftrace</p><p>Some of the popular tools thats worth mentioning:</p><ul><li>Cilium for networking</li><li>BCC for tracing</li><li>XDP for high-performance packet processing</li></ul><h3>Building your own eBPF tool</h3><p>You can build your own eBPF tools if you want. Currently you can leverage <strong>bcc</strong>(Berkley Compiler Collection) or <strong>bpftrace</strong>. These are the two popular frontend tools used to create custom tools. You can refer to these links for more tools and understanding them to make more complex or sophisticated tools out of it.</p><ul><li><a href="https://github.com/iovisor/bpftrace">GitHub - bpftrace/bpftrace: High-level tracing language for Linux</a></li><li><a href="https://github.com/iovisor/bcc#tools">GitHub - iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more</a></li></ul><h3>Limitations and Challenges</h3><p>As with every new technology thats in the market, there are some limitations with eBPF. Some of them are:</p><ul><li>Use of eBPF was started from Linux Kernel version 4.1 to 4.7. So to have the least amount of eBPF tools available should be 4.7. Although 4.7 would have only limited number of tools, the latest version will have and support a lot of tools that is available</li><li>Learning curve can be a bit difficult, as developing a new tool will require some knowledge about the frontend tools like bcc, bpftrace. Compared to Kernel engineering the learnign curve is small.</li></ul><h3>Future of eBPF</h3><p>The future of eBPF is promising, as its adoption is expected to grow across industries and platforms, including cloud providers and container orchestration systems like Kubernetes. The ongoing development of the eBPF ecosystem, with improved tooling, libraries, and enhanced capabilities, will simplify eBPF programming and enable more sophisticated applications. Additionally, cross-platform support, such as Microsoft’s eBPF for Windows project, will broaden eBPF’s applicability. As eBPF gains momentum, we can anticipate deeper integration with existing monitoring, security, and management tools, allowing users to benefit from eBPF without abandoning or relearning their current toolsets. Overall, eBPF’s future is poised for growth, solidifying its position as a powerful and versatile technology in the Linux kernel and beyond.</p><h3>Conclusion</h3><p>In summary, eBPF is a powerful tool that has made a big difference in how we work with the Linux kernel. It has improved upon traditional BPF and opened up new possibilities in areas like networking, monitoring, security, and more. As eBPF continues to grow and more people start using it, its impact will only become greater. eBPF is an important technology to learn about because it helps us create better and more efficient solutions for Linux systems.</p><p>Hope you enjoyed reading. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e90dd6c814f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CICD for Terraform with GitHub Actions]]></title>
            <link>https://allanjohn909.medium.com/cicd-for-terraform-with-github-actions-3062e58711b2?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/3062e58711b2</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Mon, 24 Oct 2022 11:16:56 GMT</pubDate>
            <atom:updated>2022-10-24T11:16:56.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwAzdScwyFi0RCm5J1vcew.jpeg" /></figure><h3>Introduction</h3><p>This post is about using Terraform in CICD pipeline with GitHub actions.</p><p>Most of the time we make use of popular CICD tools like Jenkins for Infra deployment. Because its nice as we can have stages for deployment, add custom variables or jobs and we can also add prompts for user inputs to continue, which is really important. With GitHub actions, its fully automated and it can be risky sometimes, because with automated, there is a pretty good chance, that a mistake in the config and wipe off the full infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*VcSkp79QNrmbl7W61C4dGg.jpeg" /><figcaption>something like this ;)</figcaption></figure><p>What if we can use GitHub Actions in a safe way for CICD for Terraform? Lets get to it</p><h3>Configuration</h3><h4>Terraform Plan</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64e894b7353cc47ddd52e446183c7e90/href">https://medium.com/media/64e894b7353cc47ddd52e446183c7e90/href</a></iframe><p>If you are familiar with GitHub Actions workflw, then this config looks really simple:</p><ul><li>name section is the name of the workflow</li><li>on defines when the workflow should be triggered. Here when there is a Pull Request created to the main branch and when there is changes for files inside infrastructure</li><li>env defines the environment variables required for this particular workflow or for the jobs that is going to run. The variables are related to terraform config that is run. All those secrets are stored in GitHub Secrets.</li><li>jobs section contains all the jobs that needs to run for this workflow. There is:<br>- Install terraform<br>- Check terraform formatting is proper<br>- Initialize terraform<br>- Validate terraform<br>- Run terraform Plan<br>- Run commands to update the PR if its passed or failed</li><li>Terraform is installed all the time, because I am using a public ubuntu image and I am not sure if proper version of terraform is installed or not. This additional help is better than building my own custom image and using it.</li><li>The posting info job uses actions/github-script@v6 . You will need to give more attention on the format on printing information, otherwise it might look a bit ugly :P</li></ul><h4>Terraform Apply</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/753ef1456485ec68a72554dd7a3b150d/href">https://medium.com/media/753ef1456485ec68a72554dd7a3b150d/href</a></iframe><p>So the jobs are similar to the ones in the terraform plan workflow. Except in here we are running plan and apply as well together. I can actually merge both these workflows into one and then check if its PR or not to run the apply part.</p><p>At this point of time when I am writing, there was an issue with the actions/github-script@v6 action. When the Pull Request is merged, then for some reason, the GITHUB_TOKEN cannot write on the closed PR discussion. I tried to use the same action to use with the PR state in closed. Still it didn’t work. So I had to use a different action, which someone created due to the same issue.</p><p>So now create a new branch from main and add some TF code and create a a PR against the main branch. This is how my plan looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*tQszbA7EmyvlHioD6LDVnQ.png" /><figcaption>Terraform plan</figcaption></figure><p>And when I merge the PR, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*y7hKfmSFT2Dt0b5R1P1kGw.png" /><figcaption>terraform apply</figcaption></figure><h3>Next Steps</h3><p>This is for setting up simple infrastructure. In this infra, the tfvars file is inside the infrastructure folder in the repo. For more complex environment like multiple environments, then it would be better to use <strong>terraform workspaces</strong>, provide separate tfvars files and change the directory structure itself and based on branch, deploy to different environments.</p><p>For example:</p><ul><li>PR and merge from feature to development branch, goes to DEV</li><li>PR and merge from development to main branch, goes to TEST/UAT</li><li>And creation of tag and deploying it will go to PROD.</li></ul><p>This is what I would see as a proper infra deployment. With this base workflows, you can also easily integrate tests with <strong>localstack </strong>or <strong>mock </strong>environments</p><h3>Conclusion</h3><p>CICD for infrastructure deployment is getting more and more as a requirement, because Infrastructure as a Code is moving as a defacto standard for provisioning Infra on any provider whether its cloud or on-premise. It would be nice to have CICD for infra with unit and integration testing. I hope this post helped you to understand how to build a CICD pipeline. With this info as a base, you can actually build complex pipelines.</p><p>Hope you enjoyed and have a nice day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3062e58711b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SSO Authentication for Applications in Kubernetes]]></title>
            <link>https://blog.devgenius.io/sso-authentication-for-applications-in-kubernetes-aedc3c189d89?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/aedc3c189d89</guid>
            <category><![CDATA[dex]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops-practice]]></category>
            <category><![CDATA[oidc]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Wed, 19 Oct 2022 11:19:16 GMT</pubDate>
            <atom:updated>2022-10-27T17:27:16.275Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gt1BQvURX6Sx5-lhO7kUFA.jpeg" /><figcaption>courtesy to unsplash.com</figcaption></figure><h3>Introduction</h3><p>This post is about securing applications and ingresses with SSO authentication and authorization. Ingresses can be protected with username and passwords. If many teams are working on a cluster with many applications, it would be better to use an SSO authentication mechanism with RBAC(Role bases access control). This makes life easier as the groups that are allowed / denied access can be configured easily.</p><h3>Current setup</h3><p>I have a Traefik ingress controller and there are lot of ingresses. And I want to use GitHub as an OIDC provider.</p><h3>Prerequisite</h3><p>To understand and follow this documentation, I assume the reader has a bit of knowledge on deploying apps with helm and when making changes to the values.yaml file for an app, it is assumed that the chart release is upgraded with new values.</p><h3>DEX as OIDC Provider</h3><p>There are many OIDC providers that are available, but I chose to go with Dex, because</p><ul><li>Its not tied up with Kubernetes, I can use Dex for protecting my applications or all ingresses.</li><li>Its feature rich</li><li>It has many plugins which can integrate with many OIDC providers and can act like a single hub for authentication and authorization in the cluster.</li></ul><p>So instead of using GitHub directly for OIDC config, I will use Dex as the OIDC provider inside the cluster and configure GitHub plugin on Dex for OIDC.</p><p>For applications that can communicate directly with an OIDC provider, can be protected by adding the OIDC config in the application configuration itself. This is cool. But what is uncool, is about ingresses or applications that do not support OIDC configuration. So lets see how we can do this.</p><h3>Creating a GitHub OAuth Application</h3><p>First things first</p><p>Go to github.com &gt; Your Profile &gt; Settings &gt; Developer Settings &gt; OAuth Apps &gt; New OAuth App</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*kpmL8zwUoMilKhKBb85hmg.png" /></figure><p>The Application Name , Homepage URL, Application Description can be anything. The important is Authorization callback URL . This should be the url to Dex that will be deployed.</p><p>When the App is registered, a ClientID and a ClientSecret is generated. Copy this in a safe place.</p><p>Now lets create a secret in Kubernetes to store this credentials that will be used by Dex</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/421a8505826244eaba8df8c436997002/href">https://medium.com/media/421a8505826244eaba8df8c436997002/href</a></iframe><p>and apply it in cluster</p><pre>kubectl apply -f dex-github-secret.yaml</pre><h3>Dex as OIDC provider for Applications that CAN talk OIDC</h3><p>I have ArgoCD for GitOps in my cluster. ArgoCD can talk to OIDC providers directly. So I will configure ArgoCD to use Dex as OIDC. The auth workflow goes like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7oztBfRa4YRoacVvfFNHew.jpeg" /><figcaption>DEX workflow for app with OIDC support</figcaption></figure><ol><li>User requests the application ArgoCD</li><li>ArgoCD forwards auth request to the OIDC provider configured, which is Dex.</li><li>Dex will check for tokens already exists or not. Since there is not one, so it will send the request to the OIDC connector configured.</li><li>The GitHub OAuth app authenticates the user in the request and returns the feedback to the configured Authorization Callback URL in the OAuth app, which is the Dex url</li><li>Dex verifies the response and creates signed ID tokens to the client that requested via the RedirectURI configured in staticClients.</li><li>ArgoCD will now forward the requested resource as its authenticated with a valid token</li></ol><p>This is the workflow explained in simple terms. In reality there is lot of stuff happening, like authentication and creation of signed ID tokens or JWT tokens and so on. For that I would advise to check more into how OIDC authentication works. Or wait for sometime, when I write one about it ;)</p><p>Now lets deploy Dex. Since I am using Kubernetes, I will prefer to deploy via helm chart. So my values.yaml will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d0803e1e2ef0f2714e3764bfca7ec90/href">https://medium.com/media/4d0803e1e2ef0f2714e3764bfca7ec90/href</a></iframe><p>The storage part mentions where to store the tokens, I use kubernetes cluster for storage. And use Github connector.</p><p>If loadAllGroups is true then anyone logged in GitHub will be able to pass the authorization. Here I am telling Dex to authorize users from only myorg and who belongs to the teams myteam1 and myteam2. If I dont provide the teams config, then anyone part of the org myteam will be able to be authorized.</p><blockquote>One important thing to note is that, if this Oauth App is a personal user created, then this app should be authorized by the organization. When using for the first time, Github will ask you to send request to the organization to approve the request.</blockquote><p>In the staticClients section, I have added one for ArgoCD. The id and secret will be used as clientID and clientSecret in the oidc.config respectively</p><p>For more configuration on Github connector, please check <a href="https://dexidp.io/docs/connectors/github/">here</a></p><p>Lets deploy Dex</p><pre>kubectl apply -f dex-github-secret.yaml</pre><pre>helm repo add dex <a href="https://charts.dexidp.io/">https://charts.dexidp.io</a><br>helm install --generate-name --wait dex/dex -n dex -f values.yaml </pre><p>When Dex is ready, configure ArgoCD config to add OIDC provider. I am not explaining it here. A TL;DR; would be to edit the config map argocd-cm and add the OIDC config and try to login. If properly configured, then the OIDC config should work.</p><h3>Dex as OIDC provider for Applications that CANNOT talk OIDC</h3><p>Now lets look at using Dex for applications that do not have OIDC support. We have two options.</p><h4>Option 1: Traefik Forward Auth</h4><p>Traefik Forward Auth is a simple SSO/Oauth authentication tool for Traefik as ingress controller.</p><p>The request workflow will be like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*kgHnBOb7itenlIxHMRdtRg.jpeg" /></figure><ol><li>User requests for App to Traefik Ingress controller</li><li>Traefik redirects request to Traefik Forward Auth, through a middleware that is configured on Traefik</li><li>Traefik Forward Auth forwards the request to Dex</li><li>Dex forwards redirects request to GitHub</li><li>GitHub accepts the login and sends the auth request to Dex</li><li>Dex redirects the request to Traefik Forward Auth</li><li>Traefik Forward Auth checks the response from Dex and sends 200/403 based on the response</li><li>Traefik then sends the user request to the requested resource or app.</li></ol><h4>Configure Traefik Forward Auth</h4><p>Before creating Traefik Forward Auth, we need to add a static client section in Dex. We already did the same for ArgoCD. So append the following to staticClients section of the values.yaml file for Dex:</p><pre>- id: traefik-fwd-auth<br>  name: Traefik Forward Auth OIDC Dex App<br>  redirectURIs:<br>  - <a href="https://auth.verdictcyber.com/_oauth">https://auth.mydomain.com/_oauth</a><br>  secret: random-secret</pre><p>The redirectURIs should be the ingress url for Traefik Forward Auth which will configure soon.</p><p>The values for the Traefik Forward Auth helm chart is like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d95fc157dc9bc9bc780a701172bb0880/href">https://medium.com/media/d95fc157dc9bc9bc780a701172bb0880/href</a></iframe><p>So here I am setting</p><ul><li>default oidc provider</li><li>oidc section contains OIDC information to pass through. The issuerUrl is the configured Dex application URL. The clientID and clientSecret are added here and this should be exactly the same one as you provide in the staticClients section</li><li>cookie.insecure to true , so that the cookie is allowed from non https domains because I have ssl termination on traefik. A random secret can be provided here. This will be stored as a secret in the namespace where Traefik Forward Auth is installed. The domain should be the value of the domain</li><li>authHost is important, as this is the hostname that Traefik Forward Auth configures for authentication</li><li>middleware.enabled set to true will create a middleware automatically. This middleware contains information on the auth url and service to point to.</li><li>An ingress for Traefik Forward Auth</li></ul><blockquote>The original helm chart may have issues when installing, since, I had some of them. So it would be better to clone this chart and modify the chart as your requirement. For me the issue was with the ingress config, because I was using the latest kubernetes version and the version was outdated.</blockquote><h4>Deploy Traefik Forward Auth</h4><pre>helm repo add itscontained <a href="https://charts.itscontained.io/">https://charts.itscontained.io</a><br>helm install itscontained/traefik-forward-auth -n traefik -f values.yaml</pre><p>After installing,modify an unauthenticated ingress by adding the annotation:</p><pre>traefik.ingress.kubernetes.io/router.middlewares: traefik-traefik-forward-auth@kubernetescrd</pre><p>The value of annotation will be in the format &lt;namespace&gt;-traefik-forward-auth@kubernetescrd . If namespace or kubernetescrd is omitted, then this auth will not work.</p><p>Try to login and if everything is good, then you should be redirected to GitHub via Dex and if authentication works, then you will be authorized to access the Ingress</p><h4>Option 2: Oauth2-Proxy</h4><p>Oauth2-Proxy is a reverse proxy that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group.</p><p>The authentication workflow for Oauth2-Proxy looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*ydrDoyfwHjVN8UbwF_BNMw.jpeg" /></figure><ol><li>User requests the App to Traefik</li><li>Traefik forwards the request to requested ingress, which is the Oauth2-Proxy</li><li>Oauth2-Proxy redirects request to Dex, its OIDC provider</li><li>Dex redirects request to GitHub, its OIDC provider</li><li>GitHub accepts login and forwards request to Dex via callback URL</li><li>Dex forwards request to Oauth2-Proxy via Redirect URL</li><li>Oauth2-Proxy authorizes request and sends the request to the Upstream URL configured</li></ol><p>The difference between Traefik Forward Auth and Oauth2-Proxy is that, The ingresses are separate for Traefik Forward Auth and the application ingress. In Oauth2-Proxy, the app ingress is pointing to Oauth2-Proxy as it is acting like a proxy for the application</p><h4>Configure Oauth2-Proxy</h4><p>First a static client on DEX. So append the DEX values.yaml with the following in staticClients section</p><pre>- id: oauth2-proxy-oidc<br>  name: Oauth2-Proxy OIDC Dex App<br>  redirectURIs:<br>  - https://myapp.mydomain.com/oauth2/callback<br>  secret: random-secret</pre><p>The values for Oauth2-Proxy for a sample app is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45e83f0d1c2dfe4da7190603959f4963/href">https://medium.com/media/45e83f0d1c2dfe4da7190603959f4963/href</a></iframe><p>so there is:</p><ul><li>config section for the OIDC config, provided in Dex staticClients section</li><li>ingress with tls config. Note that, I am using a wildcard dns certificate, stored as a secret in all namespaces.</li><li>extraArgs, where more config is added. Like the oidc-issuer-url, redirect-url, allowed-group, upstream. In upstream is where we configure where to send request if auth is successful. Here it is pointing to a kubernetes service named myapp running in default namespace at port 8080</li></ul><h4>Deploy Oauth2-Proxy</h4><pre>helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests<br>helm install my-release oauth2-proxy/oauth2-proxy -n default -f values.yaml</pre><p>If everything is working good, then the url <a href="https://myapp.mydomain.com/">https://myapp.mydomain.com</a> will show a screen for Oauth2-Proxy and then redirects to Dex, who will redirect to GitHub and then to the app.</p><h3>The important question. Traefik Forward Auth or Oauth2-Proxy?</h3><p>Lets take a look why Oauth2-Proxy is not ideal in this scenario. When there is multiple ingresses that needs to be protected, then you will need same number of Oauth2-Proxy helm charts, which is not a good idea because then it will be Oauth2-Proxies all over the place.</p><p>Although Oauth2-Proxy can be used as a central authentication system for all ingresses, which means one instance can be used for all auth. This will work perfectly fine, when you have <strong>NGINX </strong>as your ingress controller. Then you deploy the Oauth2-Proxy normally with setting reverse-proxy to false and then remove upstream parameter, and set the hostname something like oauth.mydomain.comfrom the above config. Also deploy it in a namespace like security</p><p>Then on the application ingress add the following annotation</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: myapp<br>  annotations:<br>    kubernetes.io/ingress.class: nginx<br>    nginx.ingress.kubernetes.io/auth-signin: <a href="https://oauth.example.com/oauth2/start">https://oauth.mydomain.com/oauth2/start</a><br>    nginx.ingress.kubernetes.io/auth-url: <a href="http://oauth2-proxy.security.svc.cluster.local/oauth2/auth">http://oauth2-proxy.security.svc.cluster.local/oauth2/auth</a><br>...</pre><ul><li>The auth-siginredirects any needed login to the Oauth2-Proxy Ingress.</li><li>The auth-url annotation can access the Oauth2-Proxy internally via its service to verify a submitted token</li></ul><p>Sadly this annotations will not work with Traefik Ingress Controller. Which is why Traefik Forward Auth is best in this case.</p><h3>Can we use both?</h3><p>Yes, its fine to have both as there will be use-cases for that. For example, you are part of a Platform Engineering team and you created Traefik Forward Auth so that customers use this middleware to protect their ingresses and access is allowed only for the customer provided groups. Now in this cluster, if you as part of a team need to deploy something, like Prometheus for example, and expose the ingress and protect it with your own groups(not customer groups), then an instance of Oauth2-Proxy will do the trick here without much complications</p><p>Also its unwise to have more than, lets say 3 or 5 Oauth2-Proxy instances to be running in the cluster, unless its fine for you. For me, that will look like an ugly solution</p><h3>Conclusion</h3><p>Dex as an OIDC provider, is a really cool tool. One can integrate all OIDC providers into one hub inside the cluster and use each provider specific config for authentication. And using Traefik Forward Auth or Oauth2-Proxy adds more security for ingresses or applications that do not support in-built OIDC config. The perfect choice here is Traefik Forward Auth, because in my setup I have Traefik as the ingress controller. It should also be noted that, if I use NGINX ingress controller, I will not be able to use Traefik Forward Auth at all, as it requires Traefik because of the Middleware configuration.</p><p>I hope you have liked this post and understood a bit of the concept of using this in your setup</p><p>Let me know your comments, if you have some other options available, I will be happy to try and document it as well!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aedc3c189d89" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/sso-authentication-for-applications-in-kubernetes-aedc3c189d89">SSO Authentication for Applications in Kubernetes</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS API Gateway + Lambda with Serverless Framework]]></title>
            <link>https://awstip.com/aws-api-gateway-lambda-with-serverless-framework-226eeceeea91?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/226eeceeea91</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[serverless-apps]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[api-gateway]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Thu, 06 Oct 2022 20:41:12 GMT</pubDate>
            <atom:updated>2022-11-08T16:48:18.449Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qza_LCcauNGS7-mLEnEDsQ.png" /><figcaption>AWS API Gateway + AWS Lambda with Serverless Framework</figcaption></figure><h3>Introduction</h3><p>This post is about deploying an application with AWS Lambda with frontend with API gateway using Serverless Framework. For those who does not know, in simple terms, Serverless Framework is a tool which will read a yaml config file and create serverless resources in any cloud. For AWS, Serverless framework will generate a Cloudformation template and deploy the resources by Cloudformation.</p><h3>Why Serverless Framework? Why not Terraform?</h3><p>Well, this would be an interesting debate, but IMO, I would prefer to use Serverless as this will build everything from a single config file, whereas with Terraform, resources configuration needs to be specified with the dependencies. In short, with Serverless framework, the deployment is very easy. I can provide with you an example, and try to do it with Terraform ;)</p><p>Lets get to it then!</p><h3>Application description</h3><p>The objective is that, when an API is called, a namespace if it does not exist, should be created along with a deployment, service and ingress in a kubernetes cluster</p><p>For this approach, I will use API Gateway and Lambda for setting up the API. I was first skeptic to use KubeConfig in Lambda, but I thought GitOps was better and secure than passing KubeConfig, especially if MFA is enabled when accessing cluster. Lambda function will update a Git repo. <a href="https://argo-cd.readthedocs.io/">ArgoCD</a> on the kubernetes cluster sync the changes and apply the manifests in the repo into the cluster.</p><h3>Code Prep</h3><p>If serverless framework is not installed, install by running the command</p><pre>npm install -g serverless</pre><p>Configure serverless to use AWS with credentials:</p><pre>sls config credentials --provider aws --key &lt;AWS_ACCESS_KEY_ID&gt; --secret &lt;AWS_SECRET_ACCESS_KEY&gt;</pre><p>To start a project, run the following command</p><pre>serverless create --template aws-python3 --name deploy-app --path deploy-app</pre><p>Since I am using python for coding, I used the template aws-python3 . You can use your preferred language and choose the template of your choice. The directory structure looks like this:</p><pre>.<br>└── deploy-app<br>    ├── handler.py<br>    └── serverless.yml</pre><p>Lets modify the handler.py to add the code. My code looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a56656e384b2f5f6c93f6368c944b463/href">https://medium.com/media/a56656e384b2f5f6c93f6368c944b463/href</a></iframe><p>If you don’t understand python very much, the code will:</p><ul><li>Get some inputs from body of the API call</li><li>Create a folder with value of app_group in the repo</li><li>Add a namespace.yaml file if it does not exist</li><li>Create a folder with the name of the app and add a app.yaml file</li><li>These *.yaml files are actually jinja templates which we will create in a sec</li><li>Commit and Push</li><li>Return status code and message</li></ul><p>So create a folder templates in the same folder and lets add the template files</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c86a222b8c883b4e3f9b4a083e7a31c/href">https://medium.com/media/3c86a222b8c883b4e3f9b4a083e7a31c/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac8029beeca53d8458ea5774669c16ea/href">https://medium.com/media/ac8029beeca53d8458ea5774669c16ea/href</a></iframe><p>So these are simple templates which will get to be as yaml files and updates the Git repo.</p><p>Note that, I am creating Ingress in this application and I already have ArgoCD for GitOps and Traefik as Ingress Controller inside the cluster.</p><p>The requirements.txt is required so that the lambda layer should be created for the code. the contents look like this:</p><pre>gitdb==4.0.9<br>GitPython==3.1.27<br>Jinja2==3.1.2<br>MarkupSafe==2.1.1<br>smmap==5.0.0</pre><p>Now the code part is over, lets start with serverless.yaml</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5e59518ca16d87219875151a4dc20b0/href">https://medium.com/media/c5e59518ca16d87219875151a4dc20b0/href</a></iframe><p>Don’t worry, I will explain in detail what this means, I don’t know honestly how to add only piece of codes from gists. Lets take a look</p><pre>service: deploy-app</pre><pre>frameworkVersion: &#39;3&#39;<br>provider:<br>  name: aws<br>  stage: prod<br>  region: eu-central-1<br>  runtime: python3.8<br>  versionFunctions: false</pre><p>This one is fairly easy, its the service name, then the provider config with region, name of the provider, runtime for lambda function. The stage is where the function should be deployed under the API gateway stage. You can start with dev while testing and then change it to prod when the code is ready</p><pre>plugins:<br>  - serverless-python-requirements<br>  - serverless-ssm-fetch</pre><p>This section describes what plugins are required for serverless to deploy the app. The serverless-python-requirements plugin is to create the python lambda layer and the serverless-ssm-fetch plugin is for serverless to fetch values from the <strong>SSM Parameter store. </strong>This is purely optional. you can substitute the value directly instead of using SSM. Since I have these values in a central location, I prefer to use SSM Parameter store</p><p>Lets examine the function bit by bit:</p><pre>functions:<br>  deploy-app:<br>    name: ${sls:stage}-deploy-app<br>    handler: handler.main<br>    layers:<br>      - Ref: PythonRequirementsLambdaLayer<br>      - arn:aws:lambda:eu-west-1:553035198032:layer:git-lambda2:8<br>    memorySize: 256<br>    timeout: 30<br>    architecture: x86_64</pre><p>So we have the</p><ul><li>function name</li><li>handler which is the &lt;python-filename&gt;.&lt;main-function-name&gt;</li><li>memorySize for the function to run</li><li>timeout for the function.</li><li>architecture of the function.</li><li>layers. This is an interesting thing here. So we are adding a Cloudformation template like Ref which is defined in the bottom in custom section. This is because Cloudformation first needs to create the layer and the output value of it will be passed into this key. The second layer is where we are using a layer that is already created by someone. In the code, we need git binary in the code. So we are adding this layer in lambda</li></ul><p>Continuing function explaining…</p><pre>function:<br>  deploy-app:<br>  ...<br>    events:<br>      - http:<br>          path: deploy<br>          method: post<br>          request:<br>            parameters:<br>              headers:<br>                X-header-from-deploy-app: true<br>           schemas:<br>             application/json:<br>                schema: ${file(request.json)}<br>                name: PostDeployApp<br>                description: &quot;Post request that deploys the app in the cluster&quot;</pre><p>So events for API gateway with http request. you can use rest as well.</p><ul><li>path for the API gateway URL</li><li>method for the API</li></ul><p>I am calling this API from within a portal which already checks the user is authenticated or not. So I don’t need authentication here, so for extra security, I am adding a header in the API call. and the header should be X-header-from-deploy-app . The value can be anything, just need a header with this key.</p><p>For additional security and not the Lambda function getting executed unnecessarily, I am adding a check for validating the content in body of the API. If the body is validated, then the request should pass through and invoke the Lambda function.</p><p>The body validation content should look like as its defined in request.json file. The content looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e583cd98dd5ebabff62575369e3c20b/href">https://medium.com/media/9e583cd98dd5ebabff62575369e3c20b/href</a></iframe><p>For authentication, you can also use AWS Cognito. For a simple app, double authentication check is overkill, so I skipped it, but did not forgot about putting a bit of security, just in case.</p><pre>function:<br>  deploy-app:<br>    ...<br>    package:<br>      patterns:<br>        ...</pre><p>This is to tell serverless when packaging, what to avoid and remove instead of packaging from the folder where we are working</p><pre>custom:<br>  pythonRequirements:<br>    dockerizePip: true<br>    layer:<br>      name: deploy-app-layer<br>      description: &quot;Layer which contains pip packages&quot;<br>      compatibleRuntimes:<br>        - python3.8<br>      compatibleArchitectures:<br>        - x86_64</pre><p>This is the section where we are saying how serverless should package the function.</p><ul><li>dockerizePip will require docker on local machine, so the image is built locally</li><li>then there is layer config for building the layer</li></ul><p>To add SSM parameter vars as env vars, we need to get the serverless plugin for serverless installed, then we need to add the code for it in 2 places.</p><p>In custom section:</p><pre>custom:<br>  ...<br>  serverlessSsmFetch:<br>    GIT_USER: /lambda/git_user<br>    GIT_PASS: /lambda/git_pass~true<br>    GIT_REPO: /lambda/git_repo</pre><p>Here we are telling serverless where to look for the variables, which path to look for in SSM parameter store. Notice that the parameter GIT_PASS has ~true in the end. This value is a SecureString in SSM Parameter store. If this is not added to the value, then the value that is stored in the environment variable will be encypted. Adding this string allows serverless to store the value as decrypted string in the env vars section of the Lambda function.</p><p>To add these variables as environment variables in the function , we add it in the function section</p><pre>function:<br>  deploy-app:<br>    ssmToEnvironment:<br>      - GIT_USER<br>      - GIT_PASS<br>      - GIT_REPO</pre><p>And thus we are ready with the code-prep part.</p><h3>Deployment</h3><p>To deploy, first we need to install the plugins, which can be installed by running the command:</p><pre>sls plugin install --name serverless-python-requirements<br>sls plugin install --name serverless-ssm-fetch</pre><p>After installing, now our directory tree structure should look similar to this:</p><pre>.<br>└── deploy-app<br>    ├── handler.py<br>    ├── package.json<br>    ├── package-lock.json<br>    ├── README.md<br>    ├── request.json<br>    ├── requirements.txt<br>    ├── serverless.yml<br>    └── templates<br>        ├── app.yaml.j2<br>        └── namespace.yaml.j2</pre><p>Now lets deploy by running</p><pre>sls deploy</pre><p>After waiting for sometime and seeing some nice logs from Cloudformation, serverless will deploy the application.</p><h3>Testing,</h3><p>Using postman, add the URL from the API Gateway portal something like: <a href="https://randomstring/">https://randomstring</a>jumbled/deploy . Set method to POST, add a header with key X-header-from-deploy-app and set value to true, in the body section pass:</p><pre>{<br>  &quot;app_group&quot;: &quot;testgroup&quot;,<br>  &quot;app_name&quot;: &quot;webserver&quot;,<br>  &quot;namespace&quot;: &quot;webservice&quot;,<br>  &quot;app_image&quot;: &quot;nginx&quot;,<br>  &quot;app_port&quot;: &quot;80&quot;,<br>  &quot;app_url&quot;: &quot;webserver.mytestwebsite.com&quot;<br>}</pre><p>And the output I get will be a status code of 200 and output of:</p><pre>{<br>  message: &quot;App deployed&quot;,<br>  app_url: &quot;webserver.mytestwebsite.com&quot;<br>}</pre><h3>Should I use Terraform or Serverless framework?</h3><p>Serverless framework makes it easy to deploy serverless app from one place with all necessary moving parts required for the application. Once you get a hang of it, then its really nice to use it. This can also be combined with Terraform where, all dependent values or variables required from Terraform can be pushed to SSM parameter store. This is one way to make Terraform and Serverless framework independent. Otherwise, it would be like to call Serverless framework from within Terraform, where Terraform will prepare the config file.</p><p>Its upto the requirement and architecting the whole application or project. Sometimes Terraform fits better. In this case, I wanted to use Serverless framework and I think it was easy for me, than deploying it with Terraform</p><h3>Conclusion</h3><p>For me, I like Serverless framework to deploy serverless applications. For beginners to cloud or serverless, Serverless framework can be easier than learning Terraform. However those who know Terraform well, will stick to it and its also a good choice where we can have a single tool for deployment of all infrastructure resources. However, we can also use a mix of both Terraform and Serverless framework, where Terraform deploys the base infra like network, IAM , security group etc and use Serverless framework for deploying serverless applications</p><p>But in the end, its a user who needs to decide what to do. But its nice to test and play around and explore new technologies in the market.</p><p>Hope you enjoyed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=226eeceeea91" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-api-gateway-lambda-with-serverless-framework-226eeceeea91">AWS API Gateway + Lambda with Serverless Framework</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is EKS actually better or worse?]]></title>
            <link>https://allanjohn909.medium.com/is-eks-actually-better-or-worse-9a251406b801?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a251406b801</guid>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[aws-eks]]></category>
            <category><![CDATA[experience-learning]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Wed, 05 Oct 2022 20:31:23 GMT</pubDate>
            <atom:updated>2022-10-24T07:54:25.453Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b1aHDN40wxdI_1Hs5ewj3Q.jpeg" /><figcaption>Amazon EKS</figcaption></figure><h3>Introduction</h3><p>This post is about my experience with Amazon EKS for deploying a kubernetes cluster as a noob on EKS.</p><h3>Current situation</h3><p>I have experience working on local kubernetes cluster like k3d, minikube and cloud provided clusters like AKS. Since the concepts are similar to local, I thought this might be similar and easy task to deploy EKS.</p><h3>Trying to understand EKS</h3><p>First I started to deploy EKS with Terraform. So I created the usual stuff to start with:</p><ul><li>A VPC</li><li>2 x public subnets</li><li>2 x private subnets</li><li>1 x NAT gateway</li><li>1 x Internet gateway</li><li>Security groups</li><li>Route tables with Routes</li></ul><p>Then started the EKS deployment. At first I was confused, because in AKS, there is a default nodepool deployed by default. On researching I found there is 3 types of nodegroups</p><ol><li><strong>EKS Managed Nodegroups</strong>: These are similar to what Azure provides. AWS provides us with the instances, and they take care of the OS patches on the nodes. This nodes have bootstrap script provided by AWS which is easy to find and join the cluster automatically. This configuration is preferred most of the time. Like deploying a test cluster.</li><li><strong>Self managed Nodegroups</strong>: This is used when we need full control over the instances. Like custom bootstrap script, instance role etc. This means we need to fully take care of the instance, OS upgrades and patches. This is really required when working in an organization with certain standards or tools that needs to be installed on the nodes and so on.</li><li><strong>Fargate Profile</strong>: This is fully managed EKS with no control over the compute resources. In here everything is managed by AWS. Users can deploy manifests in this cluster. I would not choose this anyway, I prefer some hands on stuff and not some black magic that does everything ;)</li></ol><p>Configured and deployed a t2.medium Managed Nodegroup. This worked but the nodes were not able to connect to the cluster. On debugging I found out the security group permissions was not proper. So I opened eveything for the time being, because the nodegroups were deployed in private subnets and EKS on public and private subnets</p><p>Finally got the cluster up, and the only thing I can see was coredns, aws-node and kube-proxy pods. Looks like they are the ones provided by AWS by default.</p><p>Tried to install an ingress controller, and the loadbalancer cannot connect to the instances. Looks like a security group issue and improper tags</p><p>After much frustration and spending like over 4 days on a simple EKS deployment, I got a suggestion from a friend than <a href="https://eksctl.io/"><strong>eksctl</strong></a><strong> </strong>was the tool to go with when playing around. And use terraform for production grade setup. So I started to look into it.</p><p>To be honest, eksctl looked much simple and easy to deploy. So basically it deploys EKS cluster from cli arguments or from a yaml config file. And I had a running cluster with all config I wanted within an hour! And there was no problem with network issues, cluster connection from nodes, ingress controller deployment, cluster autoscaler or anything at all that I got with deployment with terraform!!!</p><p>In a nutshell, eksctl is a tool which will prepare a Cloudformation template and run Cloudformation to deploy an EKS cluster based on the configuration that is provided. So now my idea changed like Terraform generates the config file for EKS and eksctl deploys the cluster with the configuration file.</p><h3>Verdict?</h3><p>For me personally, it was hard to understand first the concepts, but after wrapping my head around it, I can say EKS is a pretty good one when compared to other cloud provided clusters.</p><p>It allows to build the EKS cluster in any way the user requires. The cluster is totally empty with default configuration. This means the user can bring his own custom tools.</p><p>For example: AKS provides default cluster autoscaler. EKS does not provide, so we can choose to deploy any autoscaler tools like cluster-autoscaler or Karpenter in any namespace of your choice, Same goes for CSI drivers.</p><p>A disadvantage would be that when in need of a test cluster immediately, one can like AKS better or keep a configuration nearby at all times to deploy EKS with default config. Another one is about the kubernetes version. AKS is far ahead when compared with EKS in terms of kubernetes releases.</p><h3>Conclusion</h3><p>EKS is a very good option with kind of feature that allows one to configure the cluster in their own way, which I would say other Cloud providers are not providing it as such. Its like a managed service which can be customized to ones needs. And when getting to learn more on how to handle it, EKS is very impressive and really good with options like spot instances for nodegroups. In the end I am a pretty big fan . Let me know your thoughts if this is the same for you or if there is something you would like to add.</p><p>Hope you enjoyed, have a nice day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a251406b801" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crossplane — Explained]]></title>
            <link>https://allanjohn909.medium.com/crossplane-explained-cbf785cece29?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/cbf785cece29</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[crossplane]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Fri, 24 Jun 2022 13:00:22 GMT</pubDate>
            <atom:updated>2023-02-22T15:13:22.099Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crossplane — Explained</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*22uYItEPqkkPR1NBjcNIbw.jpeg" /><figcaption>No idea what to put here :)</figcaption></figure><p>I was looking to understand what Crossplane in simple terms and everywhere, literally everywhere, the explanation for Crossplane is exactly copied information from the main Crossplane Documentation. I wanted to try from my side to explain what I understood in simple terms. If there are any mistakes, please let me know so I can correct the doc</p><h3>Introduction</h3><p>Since the start of Kubernetes Cluster, many things started to make its way into the IT landscape with Kubernetes as the heart of it. And out of the many, one is Infrastructure deployment.</p><p>In the past, we would need to rely on Cloud agnostic deployment tools like Terraform, Pulumi, etc. And this required to have knowledge on the tool and proficient when building large-scale infrastruture.</p><p>Someone might had an idea to build a tool, which will also deploy infra with simple way and it doesn’t matter which cloud is it. And that’s how, I believe, Crossplane was started.</p><p>We can think, why its so special and what’s the difference between other tools right now in the market? One answer, Immediate Reconciliation. So as you know any application running in a Kubernetes cluster is controlled by a Controller manager which takes care of constant reconciliation of resource created to match the state that is stored in the ETCD state. So for infrastructure deployment, the change is reflected immediately. No wasting time in deployment. This is the real bonus. Because if someone manage to change in Cloud through Portal or Console, Crossplane reverts it back in matter of seconds. Perfect for Gitops!!</p><h3>So what’s Crossplane?</h3><p>Crossplane, in simple terms, can be called an infrastructure deployment tool, which will deploy resources in any Cloud using just a yaml file. A simple yaml file.</p><p>Sounds cool? Then lets take a look at how does it look under the hood.</p><h3>Inner Workings</h3><p>So Crossplane is deployed as a Helm Chart or a kubernetes manifest file. This is basically an application, which when installed, will install all the CRDs or Custom Resource Definitions to make the Crossplane run. List of CRDS that are installed can be found <a href="https://doc.crds.dev/github.com/crossplane/crossplane">here</a>.</p><p>Then the next step will be to install a Provider. The Provider is kind of a plugin which when installed, will install Cloud Provider specific CRDs for the resources to be created. List of Providers can be found <a href="https://crossplane.io/docs/v1.8/concepts/providers.html#providers">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*bazL5aecfmi6LS1ZroHKkA.jpeg" /></figure><p>For example, provider-aws is actually a mix of package and CRDs. Package contains the functions to create the resource and CRDS contain information on what are the parameters or input that is required for the resource to be created. A sample Provider yaml config will look like this:</p><pre>apiVersion: pkg.crossplane.io/v1<br>kind: Provider<br>metadata:<br>  name: provider-aws<br>spec:<br>  package: &quot;crossplane/provider-aws:master&quot;</pre><p>And ProviderConfig is to provide the credentials or config for the Provider to run. In this case, it will be the AWS credentials and configuration. A sample ProviderConfig looks like this:</p><pre>apiVersion: aws.crossplane.io/v1beta1<br>kind: ProviderConfig<br>metadata:<br>  name: aws-provider<br>spec:<br>  credentials:<br>    source: Secret<br>    secretRef:<br>      namespace: crossplane-system<br>      name: aws-creds<br>      key: key</pre><p>ControllerConfig is a configuration which can be used to override the default values provided when deploying Provider. The changes that I think is important is package version for the provider or resources, nodeSelector, etc. There are many other CRDs that are related to Crossplane. I am just touching only the important stuff.</p><p>For more information on what CRDs are installed as part of provider-aws, check <a href="https://doc.crds.dev/github.com/crossplane/provider-aws">here</a></p><h3>Crossplane Resources.</h3><p>Now that the Crossplane is setup and connected, Its time to deploy resources. In Terraform, we have the possibility to deploy a resource or a couple of resources bundled into a module . Just like that we can create a ManagedResource which is exactly the 1-to-1 resource on the cloud. For example RDSInstance. Then there is CompositeResource which is a combination of couple of ManagedResource combined into one. For example, myDBInstance which can be a mix of RDSInstance, NetworkRules, DB etc.</p><p>With ManagedResource, Its fairly simple deployment, because there is already the CRDs installed. A simple example:</p><pre>apiVersion: database.aws.crossplane.io/v1beta1<br>kind: RDSInstance<br>metadata:<br>  name: rdspostgresql<br>spec:<br>  forProvider:<br>    region: us-east-1<br>    dbInstanceClass: db.t2.small<br>    masterUsername: masteruser<br>    allocatedStorage: 20<br>    engine: postgres<br>    engineVersion: &quot;12&quot;<br>    skipFinalSnapshotBeforeDeletion: true<br>  writeConnectionSecretToRef:<br>    namespace: crossplane-system<br>    name: aws-rdspostgresql-conn</pre><p>So this RDSInstance kind is created by the database.aws.crossplane.io/v1beta1 API.</p><p>With CompositeResource, its a different story. This resource requires a CRD called CompositeResourceDefinition or XRD . This contains what are the input parameters that is required for the resource. Then there is a Composition which is the main resource that is being created. The configuration here will take the input and can be managed to create a CompositeResource which is a combination of multiple ManagedResource . Infrastructure Teams, prefer Compositeresource over ManagedResource , because this gives them leverage to put static values for some parameters or lock some input parameter, so that the developers who creates the resource cannot override those parameter.</p><p>A Composite Resource is always a Cluster object. To create a Composite resource will deploy a Cluster Object and then a Claim is created out of it, which is what the endusers see in their namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wln2ygOTt0OzYw_tncBjEg.jpeg" /><figcaption>Composite Resource Scope</figcaption></figure><p>As you see in the picture, The end user will be creating a Claim, which is a request sent to Crossplane, who will read the Composition and XRD for the Composite resource requested, then make the API query to the Cloud via Provider and create the resource on the Cloud and then sends the confirmation info to the Claim, which will make the resource available in the namespace.</p><p>An example of a Composite Resource for Database looks like this:</p><pre>apiVersion: database.example.org/v1alpha1<br>kind: XPostgreSQLInstance<br>metadata:<br>  name: my-db<br>spec:<br>  parameters:<br>    storageGB: 20<br>  compositionRef:<br>    name: production<br>  writeConnectionSecretToRef:<br>    namespace: crossplane-system<br>    name: my-db-connection-details</pre><p>So all the other parameters can be defaulted in Composition or cannot be overridden. Composition and XRD configuration for a Composite resource can be found <a href="https://crossplane.io/docs/v1.8/concepts/composition.html#defining-composite-resources">here</a></p><h3>So is Crossplane Good?</h3><p>So far, I have been working on AWS and Azure. It looks like Provider for AWS is really collaborative and engaging and the development is very fast, so for AWS, Crossplane will be a good choice for infra deployments. But of course, a K8s cluster needs to be in place.</p><p>For Azure and GCP, the development is slow, so the resources that can be deployed are really low. Alternative providers can also be used like provider-terraform or provider-terrajet, which basically uses Terraform to deploy resources under Crossplane. This is not an ideal solution IMHO, but if someone wants it so badly, then they can use it.</p><h3>Conclusion</h3><p>Crossplane is a really interesting tool and has the potential to be a very good one for infrastructure deployment. It helps to do a lot of automated infrastructure creation/deletion of resources for testing. On the other hand, its still in Beta phase, and gaining a large infrastructure crowd. If you think this is “beta” and not so good, remember, Terraform was once in beta stage, and we are still running huge, large-scale deployments. Choice is always yours!</p><p>Hope you enjoyed!</p><h3>New Edit:</h3><p>There are now stable providers available for crossplane, provided by Upbound.io. These are very good and stable from my tests, so I believe now its time to play around more without much issues and have a crossplane deployed infra.!!!</p><p>More info on providers can be found <a href="https://marketplace.upbound.io/providers">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbf785cece29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Terraform Provider-Part-IV — Import and Build]]></title>
            <link>https://allanjohn909.medium.com/building-a-terraform-provider-part-iv-import-and-build-173b771b9eca?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/173b771b9eca</guid>
            <category><![CDATA[terraform-provider]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Mon, 06 Jun 2022 09:07:53 GMT</pubDate>
            <atom:updated>2022-06-06T09:07:53.027Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Terraform Provider-Part-IV — Import and Build</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Q-aCzVSIFmffyBpFJQhuQ.jpeg" /><figcaption>Image from Unsplash</figcaption></figure><h3>Introduction</h3><p>This is a continuation of <a href="https://allanjohn909.medium.com/building-a-terraform-provider-part-iii-resource-creation-db5b6eeb559e">Part-IV</a>. We have all the stuff for doing CRUD operations. But what about an already existing resource? It would be nice to have the import function working as well for a Terraform provider to be deemed complete. So lets get on it.</p><h3>Import</h3><p>First we need to add the Import function. The purpose is to</p><ul><li>Get the ID of the resource provided in the terraform import command</li><li>Read the Record with the ID</li><li>Set the fqdn, ipaddress, and ID of the resource to be imported and pass it to Import Context</li></ul><p>We have a slight issue with Import. Because when we create a resource using the Custom RestAPI, we are passing Owners list, to be made as owners when the record is being created. So this will be stored in the state. But when we are importing, Azure API does not return any info on Owners. So when we import, and try to run a terraform plan, there will be changes on updating the Owners list. So when importing the resource, we always need to run a plan and apply to add the Owners to the state.</p><p>So lets get on with the function to Import the state and then add the context. Modify the file records/resource_dnsARecord.go, and add the function:</p><pre>//resourceRestAPIImport - Function to import a resource<br>func resourceRestAPIImport(d *schema.ResourceData, m interface{}) (imported []*schema.ResourceData, err error) {<br>	//Calling the client<br>	c := m.(*client.Client)<br><br>	// Get the resource ID<br>	recordID := d.Id()<br><br>	// Read the Record information<br>	res, err := c.GetRecord(recordID)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Setting the values from the response into a way terraform understands<br>	d.Set(&quot;fqdn&quot;, res.Properties.Fqdn)<br>	for _, ip := range res.Properties.ARecords {<br>		d.Set(&quot;ipaddress&quot;, ip.Ipv4Address)<br>	}<br><br>	// Set the ID with the resource ID<br>	d.SetId(res.ID)<br><br>	// Appending data to the imported data list<br>	imported = append(imported, d)<br><br>	// Return imported data<br>	return imported, nil<br>}</pre><p>Now modify the context and add the following code in the context section:</p><pre>...<br>// resourceARecord - Function that implements all reequests for the records<br>func resourceDnsARecord() *schema.Resource {<br>	return &amp;schema.Resource{<br>            ...<br>            DeleteContext: resourceARecordDelete,<br>            Importer: &amp;schema.ResourceImporter{<br>	        State: resourceRestAPIImport,<br>        },</pre><p>And now we have an import section!</p><h3>Build</h3><p>Now we can build our provider for testing. I have a nice Makefile here which makes life easier</p><pre>TEST?=$$(go list -buildvcs=false ./... | grep -v &#39;vendor&#39;)<br>HOSTNAME=terraform.test.com<br>NAMESPACE=alpha<br>NAME=rest<br>BINARY=terraform-provider-${NAME}<br>VERSION=0.0.2<br>OS_ARCH=linux_amd64</pre><pre>default: install</pre><pre>build:<br> go build -buildvcs=false -o ${BINARY}</pre><pre>release:<br> goreleaser release --rm-dist --snapshot --skip-publish  --skip-sign</pre><pre>install: build<br> mkdir -p ~/.terraform.d/plugins/${HOSTNAME}/${NAMESPACE}/${NAME}/${VERSION}/${OS_ARCH}<br> mv ${BINARY} ~/.terraform.d/plugins/${HOSTNAME}/${NAMESPACE}/${NAME}/${VERSION}/${OS_ARCH}</pre><pre>test: <br> go test -i $(TEST) || exit 1                                                   <br> echo $(TEST) | xargs -t -n4 go test $(TESTARGS) -timeout=30s -parallel=4</pre><pre>testacc: <br> TF_ACC=1 go test $(TEST) -v $(TESTARGS) -timeout 120m</pre><p>All of this is kind of self-explanatory. Something to note is:</p><ul><li>Added parameter -buildvcs=false for the build process. So when you are building multiple packages in Go, you have the possibility to add the pacakges to be used as a git repo, or in the same repo, like I did. So when we build like that, if we do not enable this option, then Go will cry a lot and will not build the binary</li><li>Created HOSTNAME and NAMESPACE for Terraform build. This is something to understand, because Terraform downloads plugins from internet when terraform init is run. So if the binary built is supposed to be non-public, then the only possible solutions are to setup a private terraform registry, or copy the binaries manually to proper location where Terraform is running. If the HOSTNAME is a domain name, then Terraform always downloads, but if there is a subdomain, Terraform first checks locally and if the plugin is there, it will not download.</li></ul><p>So to build the package, simply run</p><pre>make install</pre><p>This will build and place the provider in proper directory</p><h3>Test</h3><p>Time to test the package. In examples, modify the main.tf and the code:</p><pre>terraform {<br>  required_providers {<br>    pe = {<br>      version = &quot;~&gt; 0.0.2&quot;<br>      source  = &quot;terraform.test.com/alpha/records&quot;<br>    }<br>  }<br>}</pre><pre>provider &quot;records&quot; {<br>  username = &quot;tester&quot;<br>  password = &quot;tester@123&quot;<br>  custom_client_url = &quot;<a href="https://vpce-0b5742412431d07a3-uf1mv5sk.execute-api.eu-central-1.vpce.amazonaws.com/">h</a>ttps://localhost:23543&quot;<br>}</pre><pre>resource &quot;dns_a_record&quot; &quot;test_record&quot; {<br>  ipaddress = &quot;192.168.8.9&quot;<br>  fqdn = &quot;tester.private.hosted.zone.net&quot;</pre><pre>  owners {<br>    owner_type = &quot;User&quot;<br>    resource_owner = &quot;<a href="/service/mailto:allan.pandaplackal@allianz.de">allanj</a>ohn909@gmail.com&quot;<br>  }<br>}</pre><p>As you can see, I am adding the new built provider in required_providers section and adding the provider{} with the credentials and inputs. I add this variables just for testing, I usually keep the provider empty and pass all the variables through environment variables.</p><p>And then I am creating a resource, as configured in the Provider with the values.</p><p>If all looks good, the provider can create, update, delete and import resource now!.</p><h3>Deployment</h3><p>The best option for deploying the Terraform provider is using the Hashicorp Terraform Registry. But if the provider is something to be non-public, then the options are:</p><ul><li>Create a Private Registry with the instance running Terraform has access to the registry and push the provider there</li><li>Copy the binary manually to the instance where Terraform is running. The plugin can be stored in a different location, but then it should be installed using the <a href="https://www.terraform.io/cli/config/config-file#provider-installation">provider installation</a> method provided by Terraform</li></ul><h3>Conclusion</h3><p>So now we have a simple Terraform Provider which is build, and I hope you understood. I wrote this doc, because it was really hard for me to look a lot of places to build one, and everywhere there was a lack of documentation. So I thought the next guy who wants to build can make use of this doc. Hope you enjoyed !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=173b771b9eca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Terraform Provider-Part-III— Resource Setup]]></title>
            <link>https://blog.devops.dev/building-a-terraform-provider-part-iii-resource-creation-db5b6eeb559e?source=rss-79dade5b4723------2</link>
            <guid isPermaLink="false">https://medium.com/p/db5b6eeb559e</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform-provider]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Allan John]]></dc:creator>
            <pubDate>Sun, 05 Jun 2022 15:53:04 GMT</pubDate>
            <atom:updated>2022-06-07T07:33:04.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjSGgtPTGY7rmQFjIWQTXg.jpeg" /><figcaption>Images from Unsplash</figcaption></figure><h3>Introduction</h3><p>This is a continuation of <a href="https://allanjohn909.medium.com/building-a-terraform-provider-part-ii-auth-and-configure-provider-129570a851f">Part-II</a>. We have prepared a client and configured a Provider. Its time to do the fun stuff, Resources. There is Resources and Datasources provided by Terraform. For now, we are only looking only at Resources and how to create a resource.</p><h3>Resources</h3><p>For this, lets create a file in records/resource_dnsARecord.go. Its a good idea to have all files which contain resources code should be prefixed with resource_ and Data sources prefixed with datasource_. Lets start with simple and work our way in.</p><pre>package records<br> <br>import (<br>	&quot;context&quot;<br>	fmt &quot;fmt&quot;<br>	&quot;strings&quot;<br>	&quot;terraform-provider-rest/client&quot;<br>	&quot;time&quot;<br><br>	&quot;github.com/hashicorp/terraform-plugin-sdk/v2/diag&quot;<br>	&quot;github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema&quot;<br>)</pre><pre>func resourceDnsARecord() *schema.Resource {<br>   return &amp;schema.Resource{<br>     CreateContext: resourceDnsARecordCreate,<br>     ReadContext:   resourceDnsARecordRead,<br>     UpdateContext: resourceDnsARecordUpdate,<br>     DeleteContext: resourceDnsARecordDelete,<br>     Schema: map[string]*schema.Schema{},<br>   }<br> }</pre><pre>func resourceDnsARecordCreate(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>   // Warning or errors can be collected in a slice type<br>   var diags diag.Diagnostics<br>    return diags <br>}</pre><pre>func resourceDnsARecordRead(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>   // Warning or errors can be collected in a slice type<br>   var diags diag.Diagnostics<br>    return diags<br> }</pre><pre>func resourceDnsARecordUpdate(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>   return resourceOrderRead(ctx, d, m)<br> }</pre><pre>func resourceDnsARecordDelete(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>   // Warning or errors can be collected in a slice type<br>   var diags diag.Diagnostics<br>    return diags<br> }</pre><p>We have a boilerplate for a Terraform resource. This resource is returned with a Context, based on what the context is. And each context has its own functions. You will notice that we have functions for Create, Read, Update and Delete. This is basically a terraform resource complete functions, but one is missing, its the Import. To add Import, we need to first fix the Read function. We will come back to it later.</p><p>We now need to populate a schema for the Resource. Just like we did for the Provider in <a href="https://allanjohn909.medium.com/building-a-terraform-provider-part-i-initial-setup-22e20bd19fd0">Part-I</a>, we will replace the Schema: map[string]*schema.Schema{},. So now the resource function will look like this:</p><pre>// resourceDnsARecord - Function that implements all requests for the records<br>func resourceDnsARecord() *schema.Resource {<br>	return &amp;schema.Resource{<br>		CreateContext: resourceDnsARecordCreate,<br>		ReadContext:   resourceDnsARecordRead,<br>		UpdateContext: resourceDnsARecordUpdate,<br>		DeleteContext: resourceDnsARecordDelete,<br>		Schema: map[string]*schema.Schema{<br>		    &quot;last_updated&quot;: &amp;schema.Schema{<br>		        Type:     schema.TypeString,<br>			Optional: true,<br>			Computed: true,<br>		    },<br>		    &quot;ipaddress&quot;: &amp;schema.Schema{<br>		        Type:     schema.TypeString,<br>			Required: true,<br>		    },<br>		    &quot;fqdn&quot;: &amp;schema.Schema{<br>			Type:     schema.TypeString,<br>			Required: true,<br>		    },<br>		    &quot;owners&quot;: &amp;schema.Schema{<br>			Type:     schema.TypeSet,<br>			Required: true,<br>			MinItems: 1,<br>			Elem: &amp;schema.Resource{<br>			    Schema: map[string]*schema.Schema{<br>			        &quot;type&quot;: &amp;schema.Schema{<br>				    Type:     schema.TypeString,<br>				    Required: true,<br>				},<br>				&quot;name&quot;: &amp;schema.Schema{<br>				    Type:     schema.TypeString,<br>				    Required: true,<br>				},<br>			    },<br>			},<br>		    },<br>		},<br>	}<br>}</pre><p>Here the schema is added for the resource, so we have:</p><ul><li>ipaddress : For IPaddress of the Record to create</li><li>fqdn: The FQDN or the full address of the Record to be created. Note that the domain name should be part of a private DNS Zone.</li><li>owners : To add owners for the record created. This is a block with 2 items, The type of the Owner which can be a User, Group, or Service Principal. The name will be a name of the owner type provided</li></ul><p>These parameters are required by the custom RestAPI which creates the DNS record in the Private Hosted Zone. My plan was to create a list of maps. Unfortunately, Terraform do not support that. Instead, it provides either a Set or List type. I went with Set because, the order in the blocks does not change the resource, whereas, List will change the resource.</p><p>Now that we have a resource, we can add this resource to the ResourcesMap in Providers. Modify the records/providers.go and replace :</p><pre>ResourcesMap: map[string]*schema.Resource{},</pre><p>with</p><pre>ResourcesMap: map[string]*schema.Resource{<br>    &quot;dns_a_record&quot;: resourceDnsARecord(),<br>},</pre><h3>Models</h3><p>Now we have the Terraform schema, time to create the models. Modify the client/models.go file and add the following:</p><pre>package client</pre><pre>// Records Struct for Azure Get Request response<br>type Records struct {<br> Etag       string     `json:&quot;etag,omitempty&quot;`<br> ID         string     `json:&quot;ID,omitempty&quot;`<br> Name       string     `json:&quot;name,omitempty&quot;`<br> Properties Properties `json:&quot;properties,omitempty&quot;`<br> Type       string     `json:&quot;type,omitempty&quot;`<br>}</pre><pre>// ARecord A Record structure required for CustomClient to Create Record<br>type ARecord struct {<br> ID        string  `json:&quot;id,omitempty&quot;`<br> Ipaddress string  `json:&quot;ipaddress&quot;`<br> Fqdn      string  `json:&quot;fqdn&quot;`<br> Owners    []Owner `json:&quot;owners&quot;`<br>}</pre><pre>// Owner Struct, part of ARecord Struct<br>type Owner struct {<br> Type string `json:&quot;type&quot;`<br> Name string `json:&quot;name&quot;`<br>}</pre><pre>// Properties Struct, part of Records struct<br>type Properties struct {<br> ARecords         []Ipaddress `json:&quot;aRecords&quot;`<br> Fqdn             string      `json:&quot;fqdn,omitempty&quot;`<br> IsAutoRegistered bool        `json:&quot;isAutoRegistered,omitempty&quot;`<br> TTL              float64     `json:&quot;ttl,omitempty&quot;`<br> Metadata         any         `json:&quot;metadata,omitempty&quot;`<br>}</pre><pre>// Ipaddress struct, part of Properties struct<br>type Ipaddress struct {<br> Ipv4Address string `json:&quot;ipv4Address&quot;`<br>}</pre><h3>Implementation</h3><h4>Create</h4><p>To create the resource function, we need to first define what should happen when Terraform wants to create the resource. So in our case we just need to create an A Record in Azure Private DNS Hosted Zone, using the custom client. So first we need to modify the client/dns_records.go file. This file contains all CRUD functions related to the records. Lets add the function to create function</p><pre>package client<br><br>import (<br>	&quot;encoding/json&quot;<br>	&quot;fmt&quot;<br>	&quot;io/ioutil&quot;<br>	&quot;net/http&quot;<br>	&quot;strings&quot;<br>)<br><br>// CreateRecord - Create new A Record with CloudTribe client<br>func (c *Client) CreateRecord(aRecord ARecord) (interface{}, error) {<br><br>	// Convert Request Data/Body to JSON<br>	rb, err := json.Marshal(aRecord)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Prepare the URL, Method and Payload fo the Client<br>	url := fmt.Sprintf(&quot;%s/dns/a-record&quot;, c.CustomHostURL)<br>	method := &quot;POST&quot;<br>	payload := strings.NewReader(string(rb))<br><br>	// Create a new Request<br>	req, err := http.NewRequest(method, url, payload)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Add Proper Headers<br>	req.Header.Add(&quot;Content-Type&quot;, &quot;application/json&quot;)<br>	req.Header.Add(&quot;Authorization&quot;, fmt.Sprintf(&quot;Bearer %s&quot;, c.CustomToken))<br>	req.Host = &quot;custom-api-restapi.testing.com&quot;<br><br>	// Make the Request to create record<br>	res, err := c.CustomHTTPClient.Do(req)<br>	if err != nil {<br>		return nil, err<br>	}<br>	defer res.Body.Close()<br><br>	// Read the ResponseBody content<br>	body, err := ioutil.ReadAll(res.Body)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Check Status Code<br>	if res.StatusCode != http.StatusOK {<br>		return nil, fmt.Errorf(&quot;status: %d, body: %s&quot;, res.StatusCode, body)<br>	}<br><br>	// Preparing Response Body to return and convert it to Golang Map Object<br>	var responseBody map[string]interface{}<br>	err = json.Unmarshal([]byte(body), &amp;responseBody)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Get the ID of the resource from the Response Body<br>	recordID := responseBody[&quot;private-dns-record-id&quot;]<br><br>	// Return ID of the record created<br>	return recordID, nil<br><br>}</pre><p>This function is similar to getting the Bearer token from the custom RestAPI. So we are preparing the URL, method and Payload for creating where the payload is coming from Terraform config and then make an HTTP request to the custom API. You will notice that the response of the API is parsed as a map, rather than struct. The reason is that I dont need any other information that is received except for the ID of the A record created. And the function is returning this ID, which can be used as the ID of the resource created with Terraform.</p><p>Now for the Terraform part, modify the file records/resource_dnsARecord.go and add the function for Creating resource.</p><pre>// resourceDnsARecordCreate - Create the record<br>func resourceDnsARecordCreate(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br><br>	// Calling the client<br>	c := m.(*client.Client)<br><br>	// Warning or errors can be collected in a slice type<br>	var diags diag.Diagnostics<br><br>	// Get the list of Owners<br>	owners := d.Get(&quot;owners&quot;).(*schema.Set).List()<br><br>	// Create an empty list of Owners<br>	var ownersMap []client.Owner<br><br>	// Loop over the owners list and append them to the empty list to get list of owners map<br>	for _, owner := range owners {<br>		i := owner.(map[string]interface{})<br><br>		ownersMod := client.Owner{<br>			Type: i[&quot;type&quot;].(string),<br>			Name: i[&quot;name&quot;].(string),<br>		}<br><br>		ownersMap = append(ownersMap, ownersMod)<br>	}<br><br>	// Prepare the JSON Data for API Payload<br>	aRecord := client.ARecord{<br>		Ipaddress: d.Get(&quot;ipaddress&quot;).(string),<br>		Fqdn:      d.Get(&quot;fqdn&quot;).(string),<br>		Owners:    ownersMap,<br>	}<br><br>	// Call the function to to create Record with Payload<br>	record, err := c.CreateRecord(aRecord)<br><br>	// If error, print it out<br>	if err != nil {<br>		diags = append(diags, diag.Diagnostic{<br>			Severity: diag.Error,<br>			Summary:  fmt.Sprintf(&quot;Something wrong with Provider to Delete record: %s&quot;, err),<br>			Detail:   fmt.Sprintf(&quot;Something wrong with Provider to create record: %s&quot;, err),<br>		})<br>		return diags<br>	}<br><br>	// Set the ID of the record with the resource ID returned from the CreateRecord Function<br>	d.SetId(record.(string))<br><br>	// Call the resourceARecordRead function<br>	resourceARecordRead(ctx, d, m)<br><br>	return diags<br>}</pre><p>So the function:</p><ul><li>Gets the client from Provider,</li><li>Gets the owners blocks and then create a temporary owners list and add each owner block as map element inside the list</li><li>Create an aRecord object and add the values to it</li><li>Call the CreateRecord method to create the resource by the custom client</li><li>Get the ID of the resource created and set it as the ID of the resource that the provider creates.</li><li>Runs the Read context function. This is to ensure that once the resource is created, and if its read by terraform the state matches.</li></ul><h4>Read</h4><p>Lets proceed with the implementation of Read as this is an important function required by other Terraform context functions as well as import.</p><p>So lets first create the function to do the API calls to Azure. Modify the file client/dns_records.go and add the following</p><pre>//GetRecord - Get the A record of the resource using Azure Client with ID of the resource<br>func (c Client) GetRecord(recordID string) (*Records, error) {<br><br>	// Prepare config for API request<br>	url := fmt.Sprintf(&quot;https://management.azure.com%s?api-version=2018-09-01&quot;, recordID)<br>	method := &quot;GET&quot;<br>	payload := strings.NewReader(``)<br><br>	// Prepare the request<br>	req, err := http.NewRequest(method, url, payload)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Add the proper Headers<br>	req.Header.Add(&quot;Authorization&quot;, fmt.Sprintf(&quot;Bearer %s&quot;, c.AzToken))<br>	req.Header.Add(&quot;Content-Type&quot;, &quot;application/json&quot;)<br><br>	// Make the Request<br>	res, err := c.AzHTTPClient.Do(req)<br>	if err != nil {<br>		return nil, err<br>	}<br>	defer res.Body.Close()<br><br>	// Read the response body<br>	body, err := ioutil.ReadAll(res.Body)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Check Status code<br>	if res.StatusCode != http.StatusOK {<br>		return nil, fmt.Errorf(&quot;status: %d, body: %s&quot;, res.StatusCode, body)<br>	}<br><br>	// Preparing Response Body to return and convert it Record Struct<br>	responseBody := Records{}<br>	err = json.Unmarshal(body, &amp;responseBody)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	return &amp;responseBody, err<br>}</pre><p>This is also similar to making a request to Azure. Since we need couple of items from the response like Properties and ID we need a struct which is defined as Records . So in here, we are preparing the request and calling Azure API, return the response</p><p>As the function is ready, we can go forward with modifying the Read Context in Terraform. So modifying records/resource_dnsARecord.go and add the following for Read</p><pre>// resourceDnsARecordRead - Read the Record information. Required to check if the record exits/ to update / to import<br>func resourceDnsARecordRead(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>	// Calling the client<br>	c := m.(*client.Client)<br><br>	// Warning or errors can be collected in a slice type<br>	var diags diag.Diagnostics<br><br>	// Get the ID of the resource<br>	recordID := d.Id()<br><br>	// Call the function to Get the Record information<br>	res, err := c.GetRecord(recordID)<br>	if err != nil {<br>		return diag.FromErr(err)<br>	}<br><br>	// If a Record Exists, then prepare the Record Data in a way that terraform understands.<br>	// Owners is omitted, because Owners is not returned by the Azure API call to GET the resource<br>	if res != nil {<br>		// Set the value of FQDN<br>		d.Set(&quot;fqdn&quot;, strings.TrimSuffix(res.Properties.Fqdn, &quot;.&quot;))<br><br>		// Set the value of IP address<br>		for _, ip := range res.Properties.ARecords {<br>			if err := d.Set(&quot;ipaddress&quot;, ip.Ipv4Address); err != nil {<br>				diags = append(diags, diag.Diagnostic{<br>					Severity: diag.Error,<br>					Summary:  &quot;Unable to read data&quot;,<br>					Detail:   fmt.Sprintf(&quot;Error: %s and output is :%s&quot;, err, res.Properties.ARecords),<br>				})<br>				return diags<br>			}<br>		}<br><br>	}<br><br>	// Only use it for debugging manually<br>	//diags = append(diags, diag.Diagnostic{<br>	//	Severity: diag.Warning,<br>	//	Summary:  &quot;Output&quot;,<br>	//	Detail:   fmt.Sprintf(&quot;Output is :%s&quot;, res),<br>	//})<br>	//if err := d.Set(&quot;name&quot;, record.name); err != nil {<br>	//	return diag.FromErr(err)<br>	//}<br><br>	return diags<br>}</pre><p>Here, we are calling the GetRecord function to get the records and properly set the data in a way Terraform schema is written. So we have the fqdn and ipaddress properly added.</p><h4>Update</h4><p>Lets add the function for modifying the A record. Note that only the ipaddresscan be changed. Because changing fqdn will result in recreation of the resource.</p><p>Modifying client/dns_records.go to add update function</p><pre>// UpdateRecord - Update a record using Az Client<br>func (c *Client) UpdateRecord(recordID string, updateRecord Records) (interface{}, error) {<br><br>	// Convert data into JSON<br>	rb, err := json.Marshal(updateRecord)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Prepare config for API request<br>	url := fmt.Sprintf(&quot;https://management.azure.com%s?api-version=2018-09-01&quot;, recordID)<br>	method := &quot;PATCH&quot;<br>	payload := strings.NewReader(string(rb))<br><br>	// Create a new Request<br>	req, err := http.NewRequest(method, url, payload)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Add Proper Headers<br>	req.Header.Add(&quot;Authorization&quot;, fmt.Sprintf(&quot;Bearer %s&quot;, c.AzToken))<br>	req.Header.Add(&quot;Content-Type&quot;, &quot;application/json&quot;)<br><br>	// Make the request<br>	res, err := c.AzHTTPClient.Do(req)<br>	if err != nil {<br>		return nil, err<br>	}<br>	defer res.Body.Close()<br><br>	// Read the ResponseBody<br>	body, err := ioutil.ReadAll(res.Body)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Check Status Code<br>	if res.StatusCode != http.StatusOK {<br>		return nil, fmt.Errorf(&quot;status: %d, body: %s&quot;, res.StatusCode, body)<br>	}<br><br>	// Return Body and error<br>	return body, err<br>}</pre><p>So basically this function updates the resource using the PATCH method. And the data is provided by the Update context. Modifying records/resource_dnsARecord.go for update:</p><pre>// resourceDNSARecordUpdate - Update a Resource. Only IP address<br>func resourceDNSARecordUpdate(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>	// Calling the client<br>	c := m.(*client.Client)<br><br>	var diags diag.Diagnostics<br><br>	// Get the ID of the resource<br>	recordID := d.Id()<br><br>	// Checking if there is change for the attributes. However only `ipaddress` is supported<br>	if d.HasChange(&quot;ipaddress&quot;) {<br><br>		// Preparing the Payload, therefore, creating the struct<br>		var updateRecord client.Records<br>		var updateProperties client.Properties<br>		var aRecord []client.Ipaddress<br>		var ip client.Ipaddress<br><br>		// Setting  the IP address<br>		ip.Ipv4Address = d.Get(&quot;ipaddress&quot;).(string)<br>		aRecord = append(aRecord, ip)<br>		updateProperties.ARecords = aRecord<br>		updateRecord.Properties = updateProperties<br><br>		// Calling UpdateRecord function to update Record with provided payload<br>		_, err := c.UpdateRecord(recordID, updateRecord)<br><br>		if err != nil {<br>			diags = append(diags, diag.Diagnostic{<br>				Severity: diag.Error,<br>				Summary:  fmt.Sprintf(&quot;Something wrong with Provider to Delete record: %s&quot;, err),<br>				Detail:   fmt.Sprintf(&quot;Something wrong with Provider to create record: %s&quot;, err),<br>			})<br>			return diags<br>		}<br><br>		// Set `last_updated` to know when the record was last updated<br>		d.Set(&quot;last_updated&quot;, time.Now().Format(time.RFC850))<br>	}<br><br>	// Call resourceARecordRead function<br>	return resourceARecordRead(ctx, d, m)<br>}</pre><p>So basically, we are checking if ipaddress has changed and if there is a change then prepare the payload and call the updateRecord function. In addition to that a new field will also be updated called last_updated . This can be omitted if not required, because this is just to add this info in the state file that when it was updated.</p><h4>Delete</h4><p>Adding function for delete in client/dns_records.go</p><pre>func (c *Client) DeleteRecord(recordID string) (interface{}, error) {<br><br>	// Prepare config for API Request<br>	url := fmt.Sprintf(&quot;https://management.azure.com%s?api-version=2018-09-01&quot;, recordID)<br>	method := &quot;DELETE&quot;<br>	payload := strings.NewReader(``)<br><br>	// Initialize the Request<br>	req, err := http.NewRequest(method, url, payload)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Add Proper Headers<br>	req.Header.Add(&quot;Authorization&quot;, fmt.Sprintf(&quot;Bearer %s&quot;, c.AzToken))<br>	req.Header.Add(&quot;Content-Type&quot;, &quot;application/json&quot;)<br><br>	// Make the Request<br>	res, err := c.AzHTTPClient.Do(req)<br>	if err != nil {<br>		return nil, err<br>	}<br>	defer res.Body.Close()<br><br>	// Read the ResponseBody<br>	body, err := ioutil.ReadAll(res.Body)<br>	if err != nil {<br>		return nil, err<br>	}<br><br>	// Check StatusCode<br>	if res.StatusCode != http.StatusOK {<br>		return nil, fmt.Errorf(&quot;status: %d, body: %s&quot;, res.StatusCode, body)<br>	}<br><br>	// Return ResponseBody and error<br>	return body, err<br>}</pre><p>So we run the API call to Azure using DELETE method and return the response. Modifying the Terraform context for Delete in records/resource_dnsARecord.go</p><pre>// resourceDnsARecordDelete - Delete the Record<br>func resourceDnsARecordDelete(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {<br>	// Calling the client<br>	c := m.(*client.Client)<br><br>	var diags diag.Diagnostics<br><br>	// Get the resource ID<br>	recordID := d.Id()<br><br>	// Calling the DeleteRecord function to delete the Record<br>	_, err := c.DeleteRecord(recordID)<br><br>	if err != nil {<br>		diags = append(diags, diag.Diagnostic{<br>			Severity: diag.Error,<br>			Summary:  fmt.Sprintf(&quot;Something wrong with Provider to Delete record: %s&quot;, err),<br>			Detail:   fmt.Sprintf(&quot;Something wrong with Provider to Delete record: %s&quot;, err),<br>		})<br>		return diags<br>	}<br><br>	// Setting ID to empty, otherwise terraform thinks the resource still exists<br>	d.SetId(&quot;&quot;)<br><br>	return diags<br>}</pre><p>So we are getting the ID of the resource, and calling the DeleteRecord function to delete the record and once the response is good and without error, we are setting empty value for the ID. Otherwise Terraform state will still keep the resource if the ID is not empty and since this ID was set by us in the resourceDnsARecordCreate function, we need to unset it.</p><h3>Conclusion</h3><p>Now our Terraform provider is getting more functions and its starting to look like a provider. In the next <a href="https://allanjohn909.medium.com/building-a-terraform-provider-part-iv-import-and-build-173b771b9eca">Part</a>, we will take a look at how to add import function</p><p>Hope you enjoyed!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db5b6eeb559e" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/building-a-terraform-provider-part-iii-resource-creation-db5b6eeb559e">Building a Terraform Provider-Part-III— Resource Setup</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>