<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhanesh Malviya on Medium]]></title>
        <description><![CDATA[Stories by Dhanesh Malviya on Medium]]></description>
        <link>https://medium.com/@DhaneshMalviya?source=rss-6a25c22fd178------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*a9sR4dTo1p4kXtyE.jpg</url>
            <title>Stories by Dhanesh Malviya on Medium</title>
            <link>https://medium.com/@DhaneshMalviya?source=rss-6a25c22fd178------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:34:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@DhaneshMalviya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Install RabbitMQ on 3 node on-premise Kubernetes Cluster]]></title>
            <link>https://blog.devops.dev/install-rabbitmq-on-3-node-on-premise-kubernetes-cluster-245ed7988a59?source=rss-6a25c22fd178------2</link>
            <guid isPermaLink="false">https://medium.com/p/245ed7988a59</guid>
            <category><![CDATA[rabbitmq-cluster]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <dc:creator><![CDATA[Dhanesh Malviya]]></dc:creator>
            <pubDate>Fri, 25 Nov 2022 13:36:58 GMT</pubDate>
            <atom:updated>2022-12-05T19:37:15.837Z</atom:updated>
            <content:encoded><![CDATA[<p>Having <a href="https://medium.com/@DhaneshMalviya/ingress-with-metallb-loadbalancer-on-local-4-node-kubernetes-cluster-a0445357048">installed ingress</a> on our local Kubernetes cluster, I needed RabbitMQ cluster for one of our applications. Before Kubernetes, I had installed a 3 Node RabbitMQ cluster on Ubuntu which worked perfectly. I wanted to utilize the Kubernetes Cluster and see how RabbitMQ fits in.</p><p>Let’s dive in with a quick introduction to RabbitMQ.</p><p>RabbitMQ is a message-queuing software (also known as a message broker) that facilitates message exchange between distributed applications, allowing them to communicate efficiently and reliably.</p><p>We will use Kubernetes autoscaling and container deployment capabilities to deploy RabbitMQ instances into our 3 Node cluster. This will ease the hassles of deploying on separate VMs where RabbitMQ instances can go unbounded regarding server resource consumption. Deployment within the Kubernetes cluster provides us with more fine-grained control over the resources and abstracts out the deployment.</p><h4>Install the RabbitMQ Cluster Operator</h4><p>Execute below command to install the RabbitMQ cluster operator that makes the entire process of deploying RabbitMQ resources easy. Basically, it creates a separate namespace with name “rabbitmq-system” and installs all the resources ( Service Accounts, Resource Definitions, RabbitMQ Cluster, Roles, Bindings, Deployments etc..) required to create and manage a RabbitMQ cluster.</p><pre>kubectl apply -f &quot;https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml&quot;</pre><p>The output of the above command will be as below</p><pre>ccloud@k8scontrol:~$ ccloud@k8scontrol:~$ kubectl apply -f &quot;https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml&quot;<br>namespace/rabbitmq-system unchanged<br>customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com unchanged<br>serviceaccount/rabbitmq-cluster-operator unchanged<br>role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role unchanged<br>clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role unchanged<br>rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding unchanged<br>clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding unchanged<br>deployment.apps/rabbitmq-cluster-operator unchanged<br>ccloud@k8scontrol:~$</pre><h4>Installing Local Path Provisioner</h4><p>We need a storage area where RabbitMQ can persist the messages and logs. RabbitMQ requires the presence of Volumes that can be accessed to store the messages and logs. Since RabbitMQ instances run within a container in the Pod, the lifetime of the storage is limited to the lifetime of the Pod or container. The on-disk files within the container are ephemeral and can be lost when containers crash (RabbitMQ instances). Kubernetes “<a href="https://kubernetes.io/docs/concepts/storage/volumes/">Volume</a>” abstraction solves this problem. At its core, a volume is a directory, possibly with some data in it, which is accessible to the containers in a pod. Using Persistent Volumes in Kubernetes will allow us to store the data beyond the lifetime of a Pod and container crashes.</p><p><a href="https://github.com/rancher/local-path-provisioner">Local Path Provisioner</a> provides a way for the Kubernetes users to utilize the local storage in each node. Based on the user configuration, the Local Path Provisioner will create either hostPath or local based persistent volume on the node automatically. It utilizes the features introduced by Kubernetes <a href="https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/">Local Persistent Volume feature</a>, but makes it a simpler solution than the built-in <a href="https://kubernetes.io/docs/concepts/storage/volumes/#local">local </a>volume feature in Kubernetes.</p><p>Execute the below command to install Local Path Provisioner.</p><pre>kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.23/deploy/local-path-storage.yaml<br><br>kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class=true</pre><p>In this setup, the directory /opt/local-path-provisioner will be used across all the nodes as the path for provisioning (a.k.a, store the persistent volume data). The provisioner will be installed in local-path-storage namespace by default.</p><p>The mentioned directory is created on our worker node and as you can see below one of the pods is using this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*eVJdJyvAUuCtI_IVTEHE9w.png" /><figcaption>Persistent volume used by Pod</figcaption></figure><h4>Deploy a RabbitMQ Cluster using the Operator</h4><p>Now that we have Operator and Persistent Volumes in place, let’s go ahead and create a simple RabbitMQ cluster. ( I was constrained by resources on 3rd worker node, resulting 3rd RabbitMQ instance failing).</p><p>Store the below content in “rabbitmq.yaml” file on the control plane server.</p><pre>apiVersion: rabbitmq.com/v1beta1<br>kind: RabbitmqCluster<br>metadata:<br>  name: rabbitmq-server<br>spec:<br>  replicas: 3</pre><p>Then run the below command.</p><pre>kubectl apply -f rabbitmq-cluster.yaml<br><br>#in case you want to edit the file after applying you can use below command<br>kubectl edit rabbitmqcluster definition</pre><p>You should see all the pods in running state</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*Dvby_Q-odZDTOb4NdDSoHg.png" /><figcaption>RabbitMQ pods in running state</figcaption></figure><p>This will create a RabbitMQ cluster called rabbitmq-server in the default namespace. You can see the RabbitMQ Cluster as it is being created:</p><pre>#execute below command<br><br>watch kubectl get all<br><br>-------------- Output ----------<br><br>Every 2.0s: kubectl get all                                                                                                                                                    k8scontrol: Fri Nov 25 11:19:57 2022<br>NAME                            READY   STATUS    RESTARTS   AGE<br>pod/rabbitmq-server-server-0    1/1     Running   0          3m20s<br>pod/rabbitmq-server-server-1    1/1     Running   0          3m20s<br>pod/rabbitmq-server-server-2    1/1     Running   0          3m20s<br><br>NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                        AGE<br>service/kubernetes              ClusterIP      10.96.0.1        &lt;none&gt;            443/TCP                        17d<br>service/rabbitmq-server         ClusterIP      10.110.124.255   &lt;none&gt;            5672/TCP,15672/TCP,15692/TCP   3m20s<br>service/rabbitmq-server-nodes   ClusterIP      None             &lt;none&gt;            4369/TCP,25672/TCP             3m20s<br>service/webapp-service          NodePort       10.111.192.164   &lt;none&gt;            80:32613/TCP,81:31138/TCP      2d1h<br><br>NAME                       READY   UP-TO-DATE   AVAILABLE   AGE<br>deployment.apps/webapp     3/3     3            3           2d1h<br><br>NAME                                  DESIRED   CURRENT   READY   AGE<br>replicaset.apps/webapp-fd89cbcfb      3         3         3       2d1h<br><br>NAME                                      READY   AGE<br>statefulset.apps/rabbitmq-server-server   3/3     3m20s<br><br>NAME                                         REFERENCE           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE<br>horizontalpodautoscaler.autoscaling/webapp   Deployment/webapp   &lt;unknown&gt;/60%   3         20        3          17d<br><br>NAME                                           ALLREPLICASREADY   RECONCILESUCCESS   AGE<br>rabbitmqcluster.rabbitmq.com/rabbitmq-server   True               True               3m20s</pre><p>Incase any of your pods are in “pending” state, either the Persistent Volume is not created or the Pod was schedule on a resource constrained server (CPU / RAM). You may need to adjust these limits.</p><h4>View RabbitMQ Logs</h4><p>To see if RabbitMQ has started successfully, let’s inspect the logs from one of the Pods. Execute below command</p><pre>kubectl logs rabbitmq-server-server-0<br><br>------- Output ----------<br><br>2022-11-25 11:16:52.636142+00:00 [notice] &lt;0.263.0&gt; WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables<br>2022-11-25 11:16:52.644842+00:00 [info] &lt;0.221.0&gt; ra: starting system coordination<br>2022-11-25 11:16:52.644947+00:00 [info] &lt;0.221.0&gt; starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-server-server-0.rabbitmq-server-nodes.default/coordination/rabbit@rabbitmq-server-server-0.rabbitmq-server-nodes.default<br>2022-11-25 11:16:52.647162+00:00 [info] &lt;0.270.0&gt; ra system &#39;coordination&#39; running pre init for 0 registered servers<br>2022-11-25 11:16:52.648457+00:00 [info] &lt;0.271.0&gt; ra: meta data store initialised for system coordination. 0 record(s) recovered<br>2022-11-25 11:16:52.648938+00:00 [notice] &lt;0.276.0&gt; WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables<br>2022-11-25 11:16:52.652390+00:00 [info] &lt;0.221.0&gt;<br>2022-11-25 11:16:52.652390+00:00 [info] &lt;0.221.0&gt;  Starting RabbitMQ 3.10.2 on Erlang 24.3.4 [jit]<br>2022-11-25 11:16:52.652390+00:00 [info] &lt;0.221.0&gt;  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.<br>2022-11-25 11:16:52.652390+00:00 [info] &lt;0.221.0&gt;  Licensed under the MPL 2.0. Website: https://rabbitmq.com<br><br>  ##  ##      RabbitMQ 3.10.2<br>  ##  ##<br>  ##########  Copyright (c) 2007-2022 VMware, Inc. or its affiliates.<br>  ######  ##<br>  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com<br><br>  Erlang:      24.3.4 [jit]<br>  TLS Library: OpenSSL - OpenSSL 1.1.1o  3 May 2022<br><br>  Doc guides:  https://rabbitmq.com/documentation.html<br>  Support:     https://rabbitmq.com/contact.html<br>  Tutorials:   https://rabbitmq.com/getstarted.html<br>  Monitoring:  https://rabbitmq.com/monitoring.html<br><br>  Logs: /var/log/rabbitmq/rabbit@rabbitmq-server-server-0.rabbitmq-server-nodes.default_upgrade.log<br>        &lt;stdout&gt;<br><br>  Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf<br>                  /etc/rabbitmq/conf.d/10-operatorDefaults.conf</pre><h4>Access the RabbitMQ Management UI</h4><p>Let’s access the management UI and check our cluster status</p><p>When we created the cluster a Secret was automatically created in the default namespace which holds the credentials to access RabbitMQ management UI. Let’s retrieve those and connect</p><pre>username=&quot;$(kubectl get secret rabbitmq-server-default-user -o jsonpath=&#39;{.data.username}&#39; | base64 --decode)&quot;</pre><pre>echo &quot;username: $username&quot;</pre><pre>password=&quot;$(kubectl get secret rabbitmq-server-default-user -o jsonpath=&#39;{.data.password}&#39; | base64 --decode)&quot;</pre><pre>echo &quot;password: $password&quot;</pre><p>Note down the username and password</p><p>to access the cluster we will port-forward to a localport on the system and access the management UI</p><pre>kubectl port-forward &quot;service/rabbitmq-server&quot; 15672 --address=0.0.0.0</pre><p>Open the browser and access the UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qWJavJ61GHlHKzdXQ_LEKw.png" /></figure><p>use the credentials noted down earlier to login. Here’s our cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oDfYfrQiX_0iF_rUelNhSA.png" /></figure><h4>Attach a Workload to the Cluster</h4><p>We will use the perf-test utility to test connection and perform load testing.</p><pre>username=&quot;$(kubectl get secret rabbitmq-server-default-user -o jsonpath=&#39;{.data.username}&#39; | base64 --decode)&quot;<br>password=&quot;$(kubectl get secret rabbitmq-server-default-user -o jsonpath=&#39;{.data.password}&#39; | base64 --decode)&quot;<br>service=&quot;$(kubectl get service rabbitmq-server -o jsonpath=&#39;{.spec.clusterIP}&#39;)&quot;<br>kubectl run perf-test --image=pivotalrabbitmq/perf-test -- --uri amqp://$username:$password@$service</pre><p>Let’s check the perf-test logs now. Execute below command.</p><pre>kubectl logs --follow perf-test</pre><p>You should see something similar to below logs</p><pre>id: test-132856-195, time: 203.659s, sent: 6998 msg/s, received: 2765 msg/s, min/median/75th/95th/99th consumer latency: 140940841/141225997/141311699/141378448/141608292 µs<br>id: test-132856-195, time: 204.659s, sent: 6363 msg/s, received: 2937 msg/s, min/median/75th/95th/99th consumer latency: 141391490/141824843/142206514/142350685/142358044 µs<br>id: test-132856-195, time: 205.677s, sent: 7500 msg/s, received: 2205 msg/s, min/median/75th/95th/99th consumer latency: 141928565/142429067/142703367/142813462/142815306 µs<br>id: test-132856-195, time: 206.681s, sent: 7603 msg/s, received: 3340 msg/s, min/median/75th/95th/99th consumer latency: 142457071/142855447/143160536/143327541/143381349 µs<br>id: test-132856-195, time: 207.773s, sent: 12818 msg/s, received: 2606 msg/s, min/median/75th/95th/99th consumer latency: 143391977/143654479/143875454/144111913/144127466 µs<br>id: test-132856-195, time: 208.775s, sent: 10158 msg/s, received: 3405 msg/s, min/median/75th/95th/99th consumer latency: 143749389/144167811/144484820/144663698/144715927 µs<br>id: test-132856-195, time: 209.908s, sent: 6177 msg/s, received: 2857 msg/s, min/median/75th/95th/99th consumer latency: 144729605/145117042/145362793/145543171/145596323 µs<br>id: test-132856-195, time: 210.908s, sent: 7635 msg/s, received: 2951 msg/s, min/median/75th/95th/99th consumer latency: 145158203/145583761/145896949/146066036/146083453 µs<br>id: test-132856-195, time: 211.944s, sent: 14124 msg/s, received: 2750 msg/s, min/median/75th/95th/99th consumer latency: 146127201/146464727/146679496/146880352/146894715 µs<br>id: test-132856-195, time: 212.957s, sent: 8793 msg/s, received: 3115 msg/s, min/median/75th/95th/99th consumer latency: 146495276/146900070/147261323/147447530/147448351 µs<br>id: test-132856-195, time: 213.961s, sent: 8238 msg/s, received: 2792 msg/s, min/median/75th/95th/99th consumer latency: 147305145/147699882/147886826/148180906/148185149 µs<br>id: test-132856-195, time: 214.961s, sent: 2681 msg/s, received: 2961 msg/s, min/median/75th/95th/99th consumer latency: 147549546/148034349/148369180/148457036/148501473 µs<br>id: test-132856-195, time: 215.961s, sent: 11952 msg/s, received: 2986 msg/s, min/median/75th/95th/99th consumer latency: 148512910/149123704/149283632/149433514/149445530 µs<br>id: test-132856-195, time: 216.980s, sent: 6868 msg/s, received: 2348 msg/s, min/median/75th/95th/99th consumer latency: 149493971/149828828/149988517/150165894/150172098 µs<br>id: test-132856-195, time: 217.987s, sent: 6317 msg/s, received: 3148 msg/s, min/median/75th/95th/99th consumer latency: 149988651/150446924/150714149/150907679/150920234 µs<br>id: test-132856-195, time: 219.064s, sent: 7089 msg/s, received: 2301 msg/s, min/median/75th/95th/99th consumer latency: 150954163/151513356/151697627/151838895/151848992 µs<br>id: test-132856-195, time: 220.096s, sent: 7398 msg/s, received: 2769 msg/s, min/median/75th/95th/99th consumer latency: 151154335/151675064/151867981/152030411/152084837 µs<br>id: test-132856-195, time: 221.124s, sent: 14234 msg/s, received: 2912 msg/s, min/median/75th/95th/99th consumer latency: 152071242/152513197/152832349/153001740/153014179 µs<br>id: test-132856-195, time: 222.124s, sent: 6999 msg/s, received: 2730 msg/s, min/median/75th/95th/99th consumer latency: 153064992/153390997/153491626/153613371/153633886 µs<br>id: test-132856-195, time: 223.125s, sent: 6992 msg/s, received: 2651 msg/s, min/median/75th/95th/99th consumer latency: 153642946/154137159/154458128/154587772/154616244 µs<br>id: test-132856-195, time: 224.131s, sent: 8853 msg/s, received: 3063 msg/s, min/median/75th/95th/99th consumer latency: 154538580/154951535/155225034/155370960/155423380 µs<br>id: test-132856-195, time: 225.131s, sent: 8907 msg/s, received: 2505 msg/s, min/median/75th/95th/99th consumer latency: 154718639/155157104/155567078/155637840/155650745 µs<br>id: test-132856-195, time: 226.145s, sent: 7529 msg/s, received: 2689 msg/s, min/median/75th/95th/99th consumer latency: 155685529/156108240/156350836/156649477/156654597 µs<br>id: test-132856-195, time: 227.353s, sent: 5793 msg/s, received: 2649 msg/s, min/median/75th/95th/99th consumer latency: 156330774/156738534/156833521/157057873/157067654 µs<br>id: test-132856-195, time: 228.355s, sent: 7619 msg/s, received: 2495 msg/s, min/median/75th/95th/99th consumer latency: 157302753/157724764/157922206/158012960/158021937 µs<br>id: test-132856-195, time: 229.355s, sent: 14139 msg/s, received: 3149 msg/s, min/median/75th/95th/99th consumer latency: 157687667/158147941/158447354/158621179/158642450 µs<br>id: test-132856-195, time: 230.365s, sent: 1118 msg/s, received: 2629 msg/s, min/median/75th/95th/99th consumer latency: 158943146/159228555/159450361/159607999/159616742 µs<br>id: test-132856-195, time: 231.365s, sent: 7740 msg/s, received: 3105 msg/s, min/median/75th/95th/99th consumer latency: 158773897/159320809/159720324/159969453/160011655 µs<br>id: test-132856-195, time: 232.380s, sent: 14314 msg/s, received: 2945 msg/s, min/median/75th/95th/99th consumer latency: 159408590/159864280/160081125/160226061/160237369 µs<br>id: test-132856-195, time: 233.380s, sent: 8569 msg/s, received: 3049 msg/s, min/median/75th/95th/99th consumer latency: 160370140/160740511/161159766/161318471/161335359 µs<br>id: test-132856-195, time: 234.427s, sent: 6399 msg/s, received: 2723 msg/s, min/median/75th/95th/99th consumer latency: 160657847/161096471/161397229/161573925/161620728 µs<br>id: test-132856-195, time: 235.476s, sent: 7278 msg/s, received: 3336 msg/s, min/median/75th/95th/99th consumer latency: 161372436/161900537/162050448/162265722/162300765 µs<br>id: test-132856-195, time: 236.476s, sent: 8271 msg/s, received: 2619 msg/s, min/median/75th/95th/99th consumer latency: 162398960/162616294/163061408/163211983/163267262 µs<br>id: test-132856-195, time: 237.476s, sent: 6363 msg/s, received: 2925 msg/s, min/median/75th/95th/99th consumer latency: 162512246/163012055/163208898/163453416/163464489 µs<br>id: test-132856-195, time: 238.476s, sent: 8907 msg/s, received: 3158 msg/s, min/median/75th/95th/99th consumer latency: 163468942/164000564/164182027/164348512/164440080 µs</pre><p>Checking the admin UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NGRnff5vabvBP_gxezkDyg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*so7P0YgxyULUKw1yR8gUeA.png" /><figcaption>Messages sent to the queue by perf-test</figcaption></figure><p>Thats all about setting up the RabbitMQ Cluster on Local Kubernetes Cluster. I had many failed attempts too here while trying to setup. But finally I got this up and running. Thanks for being with me till now.</p><p>I hope you find this article useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=245ed7988a59" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/install-rabbitmq-on-3-node-on-premise-kubernetes-cluster-245ed7988a59">Install RabbitMQ on 3 node on-premise Kubernetes Cluster</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ingress with MetalLB Loadbalancer on local 4-Node Kubernetes Cluster]]></title>
            <link>https://medium.com/@DhaneshMalviya/ingress-with-metallb-loadbalancer-on-local-4-node-kubernetes-cluster-a0445357048?source=rss-6a25c22fd178------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0445357048</guid>
            <category><![CDATA[net-core-6]]></category>
            <category><![CDATA[metallb]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[Dhanesh Malviya]]></dc:creator>
            <pubDate>Wed, 23 Nov 2022 11:00:09 GMT</pubDate>
            <atom:updated>2022-11-25T13:38:08.160Z</atom:updated>
            <content:encoded><![CDATA[<h3>Ingress with MetalLB Loadbalancer on local 3-Node Kubernetes Cluster</h3><p>In the <a href="https://medium.com/@DhaneshMalviya/deploy-applications-on-the-on-premise-kubernetes-cluster-from-private-registry-9b8d1608642f">previous article</a>, we had set up a 3-node Kubernetes Cluster and deployed a .Net Core C# micro-front end and backend OpenAPI service into a single Pod on Kubernetes Cluster. We then exposed the App as a service with NodePort Type, accessible on individual nodes on the dynamic ports. Now accessing the service on individual nodes would not be useful to us as we have multiple replicas of our app running on different nodes. We need to load balance it and utilize Kubernetes power to auto-scale without any manual intervention. To achieve the load balancing, we will be using Kubernetes Ingress Object.</p><p>Let’s start with what an Ingress is. Ingress in Kubernetes allows us to access the services within the cluster externally over HTTP &amp; HTTPS, with features like Load Balancing and SSL termination. The routing to resources (services) within the cluster is controlled by rules defined on <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> resources. Ingress can be considered a kind of reverse proxy that exposes the upstream app servers to the outside world.</p><p><a href="https://kubernetes.io/docs/concepts/services-networking/service/">Services</a> in Kubernetes can sometimes act as internal or external load balancers. The definition of services is stored in the Kubernetes API server. Kube-Proxy is one such component that implements services on nodes using IP tables as rules. On the other hand, there are Ingress Controller implementations that use Service concepts from API to find endpoints and direct traffic to them, effectively skipping iptables implementation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*fOld1_QAz8P001vee6VMxA.jpeg" /><figcaption>Ingress</figcaption></figure><h4>Ingress in Cloud Environment</h4><p>Cloud environments provide network load balancers on demand which can be utilized by Kubernetes to direct traffic to the ingress controller to access the services and apps inside the cluster. On the other hand, we’ve been working in the current article series on the Local Kubernetes Cluster hosted within a private cloud environment or on-premise. Our on-premise setup lacks the cloud feature for provisioning load balancers to connect to our ingress.</p><h4>Ingress in an on-premise setup</h4><p>In the current setup that I am working on the network topology is as below. Note that due to constraints in the network load balancers and new public IP / Domain, I reused existing infrastructure by just forwarding all the traffic to the ingress from one of our public-facing servers using IIS rewrite rules and ARR server farms. A new server farm was created with the Ingress IP Pools. (I’ll cover what these IP pools are in the next section below)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*ZSvbpdZrYJlqMeULkyQZ_w.jpeg" /><figcaption>Deployment of Kubernetes in a Private Environment</figcaption></figure><p>We will be using <a href="https://metallb.universe.tf/">MetalLB </a>to provide us with a Load Balancer feature and NGINX Ingress controller for our Local Kubernetes setup</p><p><a href="https://metallb.universe.tf/">MetalLB</a> provides a network load-balancer implementation for Kubernetes clusters that do not run on a supported cloud provider, effectively allowing the usage of LoadBalancer Services within any cluster.</p><p>Let’s install the NGINX Ingress controller on our cluster using the below command. The below command will install all the Objects required to create Ingress and manage the cluster in the “ingress-Nginx” namespace.</p><pre># Run this on control plane<br>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml</pre><p>A few pods should start post-executing the above command</p><pre>kubectl get pods --namespace=ingress-nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*8LEKIYHGgVHPd80_FpTcoQ.png" /></figure><p>The following command will wait for the ingress controller pod to be up, running, and ready:</p><pre>kubectl wait --namespace ingress-nginx \<br>  --for=condition=ready pod \<br>  --selector=app.kubernetes.io/component=controller \<br>  --timeout=120s</pre><p>Let’s see the services created by NGINX ingress. Execute the below command. Note the EXTERNAL-IP in the output. This indicates that our applications are not accessible externally from the cluster. In case we need to access we need to provide an External Load Balancer for which Ingress acts as an entry point for our applications over HTTP. Further in this post below, when we install MetalLB, the EXTERNAL-IP will be assigned with an IP from the Pool provided to MetalLB.</p><pre>kubectl -n ingress-nginx get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jaLD_PuAMhglmCmDY-erKw.png" /></figure><h4>Creating the Ingress</h4><p>In the <a href="https://medium.com/@DhaneshMalviya/deploy-applications-on-the-on-premise-kubernetes-cluster-from-private-registry-9b8d1608642f">previous article</a>, we create a .net core app (reactui &amp; webapi), bundled it in the same pod, and deployed it on our Kubernetes cluster. We will utilize the same setup to access the webapi from the Ingress in the next section. To quickly test the ingress, we will deploy httpd server and expose it locally. The ingress will distribute the requests to httpd server.</p><h4>Testing Ingress with HTTPD WebServer</h4><p>Let’s create a simple web server and its associated service.</p><pre>#Run below commands on control plane<br><br>kubectl create deployment demo --image=httpd --port=80 --replicas=3<br>kubectl expose deployment demo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*lxysS1fGBK_3UZZI1LIOWA.png" /></figure><p>Then create an ingress resource. The following example uses a host that maps to localhost:</p><pre>kubectl create ingress demo-localhost --class=nginx \<br>  --rule=&quot;demo.localdev.me/*=demo:80&quot;</pre><blockquote>An interesting thing to note when you finish executing the above command is that the hostname “demo.localdev.me” can be ping’d from individual worker nodes. I had not made any entry into the hosts file. Kubernetes Networking (Calico), I suppose comes into picture .</blockquote><blockquote>**** I am further investigating how this happens. Ther are few things in Kubernetes which feels like magic.. another one is the MetalLB picking up IP from the Pool and acting as a load balancer).</blockquote><p>Now, forward a local port to the ingress controller:</p><pre>kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80</pre><p>At this point, if you access <a href="http://demo.localdev.me:8080/,">http://demo.localdev.me:8080/,</a> you should see an HTML page telling you “It works!”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oF6Ojxl_fOaNT_0MSww_aA.png" /><figcaption>HTTP GET request on Control Plane to access the services via Ingress</figcaption></figure><h4>Testing Ingress with .net app deployments</h4><p>We will be using the deployment we created in the previous article. The deployment comprises a React UI app &amp; .Net Core WebAPI. These two applications run in the same Pod and exposes port 80 &amp; 81 respectively on the containers.</p><p>In case you have created a “demo” deployment from the previous section, let&#39;s delete those and set up a fresh deployment to avoid confusion.</p><pre>ccloud@k8scontrol:~$ kubectl delete service demo<br>service &quot;demo&quot; deleted<br>ccloud@k8scontrol:~$ kubectl get pods<br>No resources found in default namespace.<br>ccloud@k8scontrol:~$ kubectl get deployments<br>No resources found in default namespace.<br>ccloud@k8scontrol:~$ kubectl delete ingress demo-localhost<br>ingress.networking.k8s.io &quot;demo-localhost&quot; deleted</pre><p>Create a new file deployment. yaml with the following content.</p><pre>---<br><br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: webapp<br>  labels:<br>    app: web<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: web<br>  template:<br>    metadata:<br>      labels:<br>        app: web<br>    spec:<br>      containers:<br>        - name: reactui<br>          image: csik8sregistry.com/reactui<br>          ports:<br>            - containerPort: 80<br>        - name: webapi<br>          image: csik8sregistry.com/webapi<br>          ports:<br>            - containerPort: 81<br>      imagePullSecrets:<br>        - name: csiregcred</pre><p>execute below command</p><pre> kubectl apply -f deployment.yaml</pre><p>Let’s create a service for our deployment to make it accessible to ingress.</p><pre>kubectl expose deployment webapp --type=NodePort --name=webapp-service</pre><p>The above command creates a service. In case we want to access the applications they are available over the NodePort on ports (32613 &amp; 31138). You can access the apps on all the nodes as we had 3 replicas specified in the “deployment.yaml” file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*KTqyPyRtyGeX6kEByCDnZg.png" /></figure><p>We will now install MetalLB on our cluster.</p><h4>Installing MetalLB</h4><p>The instructions for MetalLB are taken from this <a href="https://metallb.universe.tf/installation/">article</a>.</p><p>Check if the “strictARP” mode is set to true in kube-proxy config map.</p><pre>kubectl edit configmap -n kube-system kube-proxy</pre><p>and then set.</p><pre>apiVersion: kubeproxy.config.k8s.io/v1alpha1<br>kind: KubeProxyConfiguration<br>mode: &quot;ipvs&quot;<br>ipvs:<br>  strictARP: true</pre><p>To install MetalLB apply the following command.</p><pre>kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml</pre><p>output of above command will be displayed as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4nMyblmgp1FjrI9AZjgHGw.png" /></figure><p>This will deploy MetalLB to your cluster, under the metallb-system namespace. The components in the manifest are:</p><ul><li>The metallb-system/controller deployment. This is the cluster-wide controller that handles IP address assignments.</li><li>The metallb-system/speaker daemonset. This is the component that speaks the protocol(s) of your choice to make the services reachable.</li><li>Service accounts for the controller and speaker, along with the RBAC permissions that the components need to function.</li></ul><p>Now we are all set to create Ingress and set MetalLB as the LoadBalancer to access our apps externally.</p><blockquote>MetalLB requires a pool of IP addresses in order to be able to take ownership of the ingress-nginx Service. This pool can be defined through IPAddressPool objects in the same namespace as the MetalLB controller. This pool of IPs <strong>must</strong> be dedicated to MetalLB&#39;s use, you can&#39;t reuse the Kubernetes node IPs or IPs handed out by a DHCP server.</blockquote><p>Let’s provide MetalLB with the Pool of IP addresses that it can own.</p><p>Create a new file “metallb-config.yaml” with following content. Note that IP range is provided which are free for MetalLB to consume. These IP addresses are not used by any server on the network.</p><pre>---<br>apiVersion: metallb.io/v1beta1<br>kind: IPAddressPool<br>metadata:<br>  name: default<br>  namespace: metallb-system<br>spec:<br>  addresses:<br>  - 192.168.100.244-192.168.100.246<br>  autoAssign: true<br>---<br>apiVersion: metallb.io/v1beta1<br>kind: L2Advertisement<br>metadata:<br>  name: default<br>  namespace: metallb-system<br>spec:<br>  ipAddressPools:<br>  - default</pre><p>execute the following command</p><pre>kubectl apply -f metallb-config.yaml</pre><p>Output of the above command is as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*Tt9KdB5LlHnF3oPubBkCSw.png" /></figure><p>Let’s check the nginx-ingress controller service again. as you can see, the EXTERNAL-IP is now populated with an IP provided from the Pool. This IP can now be used to access our applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OoYRD_IY25LsFXQgqH_G2g.png" /></figure><p>Our Kubernetes Cluster now supports a Load Balancer Service which can be checked with below command (Its same as the above). You can check with following command</p><pre>kubectl get service ingress-nginx-controller --namespace=ingress-nginx</pre><p>Once you have the external IP address (or FQDN), set up a DNS record pointing to it. Then you can create an ingress resource. The following example assumes that you have set up a DNS record for <a href="http://www.demo.io/">www.demo.io:</a></p><p>Here we are setting up a rule to access our “webapi” application which is accessible on port 81 in our deployment.</p><pre>kubectl create ingress ingress-webapp --class=nginx   --rule=&quot;demo.localdev.me/*=webapp-service:81&quot;</pre><p>You can access the application from browser. I have made host entry for demo.localdev.me pointing to 192.168.100.244.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mrPKKlPpfgatpP4n3UtvQ.png" /></figure><p>As I wanted to access the application from over the internet rather than internally and had more than one IP (MetalLB Pool), I decided to add these IP’s to the server farm within IIS ARR module. This gave me the flexibility to direct the external traffic to whichever IP is chosen. In Production, we would have a FQDN mapped to public IP which will rarely change or that could be another use case that I am not aware of . But this architecture gave me reasonable insights into how the apps deployed on internal Kubernetes Setup can be exposed over the internet.</p><p>This was an amazing learning for me. I hope you find this documentation useful to setup your Kubernetes Cluster on-premises.</p><p>Below is the snapshot of few errors that I encountered while experimenting with ingresses. I had mistakenly input the incorrect “service name” while creating the ingress using this command <strong>“kubectl create ingress ingress-webapp — class=nginx — rule=”demo.localdev.me/*=webapp-service:81&quot;“</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*-nbYPucCb0BeNN6D9GnOrQ.png" /></figure><pre>ccloud@k8scontrol:~/kubernetes-deployment$ kubectl describe ingress ingress-webapp<br>Name:             ingress-webapp<br>Labels:           &lt;none&gt;<br>Namespace:        default<br>Address:          192.168.100.244<br>Ingress Class:    nginx<br>Default backend:  &lt;default&gt;<br>Rules:<br>  Host                 Path  Backends<br>  ----                 ----  --------<br>  cvcp.csinfocomm.com<br>                       /   webapp:80 (&lt;error: endpoints &quot;webapp&quot; not found&gt;)<br>Annotations:           &lt;none&gt;<br>Events:<br>  Type    Reason  Age                    From                      Message<br>  ----    ------  ----                   ----                      -------<br>  Normal  Sync    2m14s (x2 over 2m39s)  nginx-ingress-controller  Scheduled for sync<br>ccloud@k8scontrol:~/kubernetes-deployment$</pre><p>Here are some good resources that help me setup MetalLB on the on-premise Kubernetes Cluster — <a href="https://kubernetes.github.io/ingress-nginx/deploy/">Link 1</a>, <a href="https://kubernetes.github.io/ingress-nginx/deploy/baremetal/">Link 2</a>, <a href="https://metallb.universe.tf/installation/">MetalLB installation</a>.</p><p><a href="https://metallb.universe.tf/concepts/layer2/">Under the hood</a> how MetalLB takes control of the IP address from the provided IP Address Pool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0445357048" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy .Net 6 & React App on local Kubernetes cluster from Private Registry]]></title>
            <link>https://medium.com/@DhaneshMalviya/deploy-applications-on-the-on-premise-kubernetes-cluster-from-private-registry-9b8d1608642f?source=rss-6a25c22fd178------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b8d1608642f</guid>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[net-core-6]]></category>
            <category><![CDATA[containerization]]></category>
            <dc:creator><![CDATA[Dhanesh Malviya]]></dc:creator>
            <pubDate>Mon, 21 Nov 2022 14:07:17 GMT</pubDate>
            <atom:updated>2022-11-25T14:09:40.462Z</atom:updated>
            <content:encoded><![CDATA[<p>In continuation of the <a href="https://medium.com/@DhaneshMalviya/setup-3-node-kubernetes-cluster-on-ubuntu-22-04-d4652ac17207">first part</a> of the series, I needed to pull app images from a private repository. This private repo could be anywhere on the cloud or an on-premise server.</p><p>This article describes the deployment process of applications from a private Kubernetes Registry setup in-house. For setting up a Private repository on the cloud or on-premise Servers follow this article.</p><p>We’ll begin with creating a Pod file first, deploy that to our servers, and then create a Kubernetes Deployment file.</p><p>we will be using YAML (human readable text-based format) for instructing Kubernetes to deploy our application. Our application will utilize the NodePort type (the hosted app will be accessible on worker nodes via worker-node-host: port combination). We will then go ahead and extend this application via services &amp; ingress in Kubernetes to provide us with an API Gateway kind of feature. We will then have a single entry point to access our services rather than the worker NodePort.</p><p>We will use the .Net Core 6 Web API app and React UI app, containerize them and deploy it to our local Kubernetes Cluster. Note that we will be doing this from a private registry setup in-house. You can also use Docker Hub for your private or public repositories. This is trivial when using Visual Studio 2022. You can also prepare your own containers for projects in other languages. However, to keep this article simple, I’ll use the .Net Core 6 App with swagger &amp; a front-end project in ReactJS.</p><p>Let’s begin.</p><h4>Tools &amp; Software:</h4><ul><li>Visual Studio 2022 Community Edition</li><li>.Net Core 6</li><li>Docker Desktop</li><li>Kubernetes Cluster (<a href="https://medium.com/@DhaneshMalviya/setup-3-node-kubernetes-cluster-on-ubuntu-22-04-d4652ac17207">see this article</a> to set up your own cluster)</li></ul><h4>Step 1: Prepare .Net Core 6 Web API &amp; ReactJS project</h4><p>Start with creating a new .Net core 6 Web API project in Visual Studio 2022</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIG-msHYJGXa0y6PRd3Kgw.png" /></figure><p>Keep the “Enable Docker” option unchecked, we will add support later</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdFZn73HnFyIS2GvCOg8_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vHbJvG6mreDYn_If_W982w.png" /></figure><p>Now create a new UI project with ReactJS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*Jh1z-xkzBkec4FVLDOh5FA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rwi-bsbAt-Cijn_QusEC4A.png" /></figure><p>Keep the option “Add Integration for Empty ASP.NET Web API project” checked. This will make it easier for our ReactJS app to call the Web API project. This is basically required so the same port that ReactJS is running o n can be utilized for proxying the requests to the WebAPI project. The configuration for this can be found in <strong>setupProxy.js</strong> file in React project. More information on this can be found <a href="https://learn.microsoft.com/en-us/visualstudio/javascript/tutorial-asp-net-core-with-react?view=vs-2022">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOclwTGejWTEOyYKZu-4Jg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E-ZNlQxA5HU9x66HjRwG1Q.png" /><figcaption>setupProy.js forwarding requests to WebAPI</figcaption></figure><p>This will take few minutes depending on your internet speed as the npm modules are being installed.</p><p>Once the installation is complete, run the project to see it working.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P0zEBpT3lZtyO3a67LV4Jg.png" /><figcaption>Swagger WebAPI project</figcaption></figure><p>When you run the project, only the web-api may start. To startup both the Web API and ReactJS project tothere, change the below configuration by right-click on the solution and selecting “Set Startup Projects..”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grjqQNTgFnrt7Ace0ByIjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYl4BKd80B1rK5Acuy5TEw.png" /></figure><p>Save and re-run the projects. This will also start a command prompt to serve the ReactJS UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*msERiI2srteKw_ZHZU6feg.png" /></figure><p>Navigate to the mentioned URL : https:/localhost:3000</p><p>You may need to change the port of WebAPI in setupProxy.js in case port other than 5001 is assigned to WebAPI project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FARxexXjhsKOsjay36xzYw.png" /></figure><p>Here’s the working setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OlMItWo5i5oI1o-6KBrQ8w.png" /></figure><h4>Step 2: Add support for Docker to our WebAPI project.</h4><p>Let’s display the swagger UI always for our testing purposes since we will run our application in container. Ideally the swagger UI is visible only in development environment. To enable this make changes to program.cs as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dRQqmyDEd-sFT_DGfC-GEQ.png" /></figure><p>We could have created a project with docker support in the beginning, however, to demonstrate that any existing project can also be containerized, we are performing this step here.</p><p>Let’s first add docker support to the web API project. Right click -&gt; Add -&gt; Docker support as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlSiutO-kxnabfpSzNosmw.png" /></figure><p>Depending on the type of container select the option. I am deploying to Linux containers.</p><p>This will create a new Dockerfile which will have below content.</p><pre>#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.</pre><pre>FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base</pre><pre>WORKDIR /app</pre><pre>EXPOSE 80</pre><pre>EXPOSE 443</pre><pre>FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build</pre><pre>WORKDIR /src</pre><pre>COPY [&quot;KubernetesSample/WebAPI.csproj&quot;, &quot;KubernetesSample/&quot;]</pre><pre>RUN dotnet restore &quot;KubernetesSample/WebAPI.csproj&quot;</pre><pre>COPY . .</pre><pre>WORKDIR &quot;/src/KubernetesSample&quot;</pre><pre>RUN dotnet build &quot;WebAPI.csproj&quot; -c Release -o /app/build</pre><pre>FROM build AS publish</pre><pre>RUN dotnet publish &quot;WebAPI.csproj&quot; -c Release -o /app/publish /p:UseAppHost=false</pre><pre>FROM base AS final</pre><pre>WORKDIR /app</pre><pre>COPY --from=publish /app/publish .</pre><pre>ENTRYPOINT [&quot;dotnet&quot;, &quot;WebAPI.dll&quot;]</pre><p>To build the docker image for deployment, right click on Dockerfile -&gt; Build Docker Image as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spaOQ_zfPP427Ih_sSxYLA.png" /></figure><p>Once the docker image is built successfully you can see the image in Docker Desktop as shown in image below or execute the below command.</p><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6RxkhBA5FZgo_58_WG3M3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*84pBr0dtBa7dQqOuGmvznw.png" /><figcaption>Output of command “docker images”</figcaption></figure><h4>Step 3: Test our WebAPI Docker Container</h4><p>It’s time to test the docker container on our desktop so we can be sure that it will run on the Kubernetes Cluster as expected.</p><p>On Docker Desktop, choose the webapi container image and click “run” as shown below. To access our API we need to map the container port : 80 to another port on our host machine (8080), so that the application running within a docker container will be exposed on our development machine on port 8080. The application can be accessed on <a href="http://localhost:8080/">http://localhost:8080</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YdeNGOUy_6VxgknQVA_PsQ.png" /></figure><p>Successful run of the container is displayed below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aPaGbQAE3FrfZCwM7uvfSg.png" /></figure><p>Successful run of our web API within container should display the swagger API page as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uCN_Qe-PnDTKQ9iMeDDkbg.png" /></figure><h4>Step 5: Containerize ReactJS App</h4><p>Create a Dockerfile in the ReactJS project directory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0vUs6F8Ven3t6stXBGGyQ.png" /></figure><p>Configure the .dockerignorefile . Copy and paste the below snippet.</p><pre>**/node_modules</pre><p>Create a new folder named nginx. Create a new nginx.conf file in this folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LZ7ccnVUCxsb0TRsL0Up8w.png" /></figure><p>Build the docker image for ReactJS app using the below command. The -t parameter tags the docker image for reference.</p><pre>docker build -t reactui .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6BwcISVTzI_h1KerGRy9NA.png" /></figure><p>A successful build of the docker image should show the image in Docker Desktop UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7WL_1pMIB0EKZYA9HPW9bA.png" /></figure><p>Or you can check the image using below command</p><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*z-LJTMKhrNBJ1dtJoztYKw.png" /></figure><p>Lets verify whether our ReactJS app is running. Hit the run button in the “images” section and configure port 8081 for UI to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RURLzjfpfWbys7INZ6FnlQ.png" /></figure><p>The image below shows the React App running and loaded from the Docker Container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRRPJbit50Y1Qt3Z6uRRRg.png" /></figure><p>There’s one problem with the ReactJS app container, in that, it does not point to the WebAPI project. Now this problem can be solved by providing externalized configurations to the container when the container starts. We will orchestrate this when we deploy it to our Kubernetes Cluster. For now, we are testing deployment on Kubernetes Cluster from private repositories. So let’s stick to that.</p><h4>Step 3 : Push the generated images to a private repository</h4><p>We will push the Docker Image generated in previous step to our private repository for deployment on Kubernetes Cluster.</p><p>The images list is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fTU0PsKQDCMuf3qkEqhRZQ.png" /></figure><p>The private docker registry is configured and hosted on one of the internal servers (VM’s). The setup process for the private registry can be found <a href="https://medium.com/@DhaneshMalviya/setup-private-kubernetes-repository-4228a16f1aff">here</a>.</p><p>Run below command to transfer the image to private registry. The private registry is located at csik8sregistry.com (which is a local server).</p><pre># First we need to tag the image to specify the repository to push to</pre><pre>$ docker image tag webapi csik8sregistry.com/webapi<br>$ docker image tag reactui csik8sregistry.com/reactui</pre><p>The above commands will result into two additional docker images of our apps based on the previous images. The list of images can also be found from cmd prompt using “docker images” comand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGFsxb98akOO9C6F_8mj7Q.png" /><figcaption>New tagged images</figcaption></figure><p>Now lets push the images to the private registry using below commands.</p><pre>$ docker image push csik8sregistry.com/webapi<br>$ docker image push csik8sregistry.com/reactui</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lqFWALYupqnjWs8F-R4dvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*mdpmvgKw26oxYM_DV8BLsQ.png" /></figure><p>That’s all, we have now successfully pushed the images from .net projects to our private docker registry. It’s now time to deploy it on our Kubernetes Nodes. The next step is the preparatory phase for the deployment and covers the process of creating deployment files.</p><h4>Step 3: Create Kubernetes Pod Deployment file</h4><p>We will deploy both the React UI and Web API applications in the same Kubernetes Pod. The other approach could be to deploy each application in their own Pods. The former approach is useful when you have dependent applications and you need to ensure that both run in the same environment. So, running in the same Pod will ensure that both apps will be on the same Node and share the resources.</p><p>Note : For running our applications in container and deployment on Kubernetes environment within a single Pod, I am configuring the reactui app to run on port 80 within the container. similarly, webapi will run on port 81 within the container.</p><ul><li>The mapping of ports is done in the nginx.conf file for reactui project and Dockerfile for webapi project.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzF0Y8RhSRiucII7aYmmeA.png" /><figcaption>Mapping of port for WebAPI</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOwlFYukaaji1Rr2wsuShg.png" /><figcaption>Mapping of Port for ReactJS project served by nginx on port 80</figcaption></figure><p>Create a new folder in home directory on the Kubernetes Master (Control Plane) Node.</p><pre>$ mkdir kubernetes-deployment<br>$ cd kubernetes-deployment/</pre><pre>$ nano WebAppPod.yaml</pre><p>Paste the following in WebAppPod.yaml</p><pre>---</pre><pre>apiVersion: v1</pre><pre>kind: Pod</pre><pre>metadata:</pre><pre>---</pre><pre>apiVersion: v1</pre><pre>kind: Pod</pre><pre>metadata:<br>  name: kubernetes-app<br>  labels:<br>    app: webapp<br>spec:<br>  containers:<br>    - name: front-end<br>      image: csik8sregistry.com/reactui<br>      ports:<br>        - containerPort: 80<br>    - name: web-api<br>      image: csik8sregistry.com/webapi<br>      ports:<br>        - containerPort: 81<br><strong>  imagePullSecrets:<br>    - name: csiregcred</strong></pre><p>Let&#39;s go through the above file one by one. Both applications will be deployed as a single unit on Nodes in the Kubernetes Cluster.</p><ol><li>We are creating a Kubernetes Object of kind Pod. We might instead specify a service, Deployment, Job and so on..</li><li>Next, we specify the Pod Name and assign labels to it for querying using Kubectl Commands</li><li>The spec section contains the containers that we want Kubernetes to pull from the private registry and deploy on the cluster. Note that the containers are the ones we created earlier and are present in the private registry. You can specify any other images here from any other registry.</li><li>We have tagged both the images and assigned ports to it. These ports are where the application will be listening within the container. To access it over the network we will have to map the port to another Port on the Worker Node with NodePort Type. In case of cluster deployment, Kubernetes will dynamically assign the port to our containers on the Worker Node.</li><li>You can also specify more complex properties in the YAML file, such as commands to execute when the container starts, the working directory, data directory, and pull a new image every time the container is instantiated. More properties can be found in the <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/">Kubernetes Object Reference</a></li><li>Lastly, we have specified the “secret” Kubernetes Object for specifying the credentials of the Private Registry to pull the image</li></ol><p>Let’s create a new Pod</p><p>Run the below command to instruct Kubernetes to create a new Pod on one of the Nodes.</p><pre>kubectl create -f WebAppPod.yaml</pre><p>Keep a note that in case the Pods failed to create as can be seen in the image below, you can execute “kubectl describe pod kubernetes-app” command to obtain more information for the failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*te_qeFfczYoP_njSQwkBDg.png" /></figure><p>In my case I had forgotten to provide the secret that is to be used to pull the image from Private Registry.</p><p>You can use this command (kubectl describe pod kubernetes-app) to investigate further. <a href="https://stackoverflow.com/questions/58713651/2-containers-using-the-same-port-in-kubernetes-pod#:~:text=There%20should%20be%20a%20good,%3A%20sidecar%2C%20ambassador%20and%20adapter.">Here’s</a> a good stack overflow issue describing the same.</p><pre># <strong> This is the sample output of describe pod command. Reason of failure was that both the apps (in different containers) were configured to run on the same port within the Pod</strong><br>ccloud@k8scontrol:~/kubernetes-deployment$ kubectl describe pod kubernetes-app<br>Name:             kubernetes-app<br>Namespace:        default<br>Priority:         0<br>Service Account:  default<br>Node:             k8snode2/192.168.100.242<br>Start Time:       Mon, 07 Nov 2022 10:52:38 +0000<br>Labels:           app=webapp<br>Annotations:      cni.projectcalico.org/containerID: 36f1f03be1fc121d19820c57262e01a3050662962a44e47cd642ebd6cfec6bfd<br>                  cni.projectcalico.org/podIP: 10.244.185.212/32<br>                  cni.projectcalico.org/podIPs: 10.244.185.212/32<br>Status:           Running<br>IP:               10.244.185.212<br>IPs:<br>  IP:  10.244.185.212<br>Containers:<br>  front-end:<br>    Container ID:   containerd://582c3fcf4fbb400d07501d646d713c2c1a58161a7eef22da3d302a4243c027aa<br>    Image:          csik8sregistry.com/reactui<br>    Image ID:       csik8sregistry.com/reactui@sha256:88c71acfbc937ec8a1f4a3d1c0a0910c7ecfd723f33b6bcaa1f73315df1286f3<br>    Port:           <strong>80/TCP</strong><br>    Host Port:      0/TCP<br>    State:          Running<br>      Started:      Mon, 07 Nov 2022 10:52:39 +0000<br>    Ready:          True<br>    Restart Count:  0<br>    Environment:    &lt;none&gt;<br>    Mounts:<br>      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r6wqk (ro)<br>  web-api:<br>    Container ID:   containerd://0886f2226cd24768232d404ccad4694f0b121c23a3a2078dc1efed804f83fc61<br>    Image:          csik8sregistry.com/webapi<br>    Image ID:       csik8sregistry.com/webapi@sha256:40c58a82670f752ec51afab3d86a15f3d289b7814330833b4e7ff3ac29fa96ee<br>    Port:           <strong>80/TCP</strong><br>    Host Port:      0/TCP<br>    State:          Waiting<br>      Reason:       CrashLoopBackOff<br>    Last State:     Terminated<br>      Reason:       Error<br>      Exit Code:    139<br>      Started:      Mon, 07 Nov 2022 10:52:41 +0000<br>      Finished:     Mon, 07 Nov 2022 10:52:42 +0000<br>    Ready:          False<br>    Restart Count:  1<br>    Environment:    &lt;none&gt;<br>    Mounts:<br>      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r6wqk (ro)<br>Conditions:<br>  Type              Status<br>  Initialized       True<br>  Ready             False<br>  ContainersReady   False<br>  PodScheduled      True<br>Volumes:<br>  kube-api-access-r6wqk:<br>    Type:                    Projected (a volume that contains injected data from multiple sources)<br>    TokenExpirationSeconds:  3607<br>    ConfigMapName:           kube-root-ca.crt<br>    ConfigMapOptional:       &lt;nil&gt;<br>    DownwardAPI:             true<br>QoS Class:                   BestEffort<br>Node-Selectors:              &lt;none&gt;<br>Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s<br>                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s<br>Events:<br>  Type     Reason     Age                From               Message<br>  ----     ------     ----               ----               -------<br>  Normal   Scheduled  21s                default-scheduler  Successfully assigned default/kubernetes-app to k8snode2<br>  Normal   Pulling    21s                kubelet            Pulling image &quot;csik8sregistry.com/reactui&quot;<br>  Normal   Pulled     20s                kubelet            Successfully pulled image &quot;csik8sregistry.com/reactui&quot; in 116.765944ms<br>  Normal   Created    20s                kubelet            Created container front-end<br>  Normal   Started    20s                kubelet            Started container front-end<br>  Normal   Pulled     20s                kubelet            Successfully pulled image &quot;csik8sregistry.com/webapi&quot; in 117.376736ms<br>  Normal   Started    18s (x2 over 20s)  kubelet            Started container web-api<br>  Normal   Pulled     18s                kubelet            Successfully pulled image &quot;csik8sregistry.com/webapi&quot; in 116.473791ms<br>  Warning  BackOff    15s (x2 over 16s)  kubelet            Back-off restarting failed container<br>  Normal   Pulling    0s (x3 over 20s)   kubelet            Pulling image &quot;csik8sregistry.com/webapi&quot;<br>  Normal   Created    0s (x3 over 20s)   kubelet            Created container web-api<br>  Normal   Pulled     0s                 kubelet            Successfully pulled image &quot;csik8sregistry.com/webapi&quot; in 114.472715ms</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9KIj_8b9a-DvIcEHxkWnQ.png" /></figure><p>Let’s delete and re-create the pod</p><pre>#<strong>Use this command only if you need to delte the Pod. Just for reference.</strong><br>kubectl delete pod kubernetes-app</pre><pre># <strong>After applying the secret in Pod file, I was able to create containers running successfully.</strong></pre><p>Now we have Pod successfully running, let’s go ahead and do the same thing for Deployment. Basically, we will instruct Kubernetes to manage a set of replicas of the Pod that we created above. This way Kubernetes will ensure that a certain number of our application instances are always running.</p><h4>Kubernetes Deployment Use Cases</h4><p>It’s important to understand why do we need Kubernetes Deployment in the first place.</p><ul><li><strong>Availability of workload </strong>— Deployments will ensure that the specified number of copies will be constantly running and available on the cluster. In case any Node dies, Kubernetes will automatically start the Pod on another Node in the Cluster.</li><li><strong>Scaling </strong>— Kubernetes makes it easy to manage the number of replicas that should always be running. In fact, it can also be configured to autoscale based on the number of requests to service.</li><li><strong>Multi-version deployments</strong> — Kubernetes makes it easy to manage multiple versions of the application deployments within the same cluster. Pause, edit or Rollback the deployments provide more control over the process of deployment.</li><li><strong>Exposing service over the network </strong>— Ideally, when extending our services over public internet we usually get a reverse proxy into our deployment architecture which offers a feature to load balance and connect the upstream servers. Kubernetes offers such a kind feature via Services with a single command that can provide you with similar features.</li></ul><h4>Preparing the Deployment Spec file</h4><p>We will prepare a deployment spec file where we specify what goes into the Pod and how many replicas we want to be always running. I have a 3 worker node setup, so I am setting out 3 replicas to be always running. Setting other properties like selectors, provide us ability to further define the Pods that are affected by this Deployment, the number of seconds to wait before declaring the Pod “ready”. Let’s go ahead and create a Deployment File that specifies 3 replicas of the Pod at any point in time.</p><p>Create a new <strong>deployment.yaml</strong> file with the following content</p><pre>---</pre><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: webapp<br>  labels:<br>    app: <strong>web</strong><br>spec:<br>  <strong>replicas: 3</strong><br>  selector:<br>    matchLabels:<br>      app: <strong>web</strong><br>  template:<br>    metadata:<br>      labels:<br>        app: <strong>web</strong><br>    spec:<br>      containers:<br>        - name: reactui<br>          image: csik8sregistry.com/reactui<br>          ports:<br>            - containerPort: 80<br>        - name: webapi<br>          image: csik8sregistry.com/webapi<br>          ports:<br>            - containerPort: 81<br><strong>      imagePullSecrets:<br>        - name: csiregcred</strong></pre><p>Here we have specified two containers (reactui and webapi) that will be part of this deployment and labelled as “web”. Remember to provide Secret that will be used to pull images from the private registry. The selector matches these labels and is useful when we need to change any properties like scale up / down the deployment. Let’s roll out the deployment. Execute below command</p><pre># <strong>This command will create a new deployment with 3 Pods running 1 instance on every Node in the cluster. There are 3 Nodes in my cluster setup</strong>.<br>$ kubectl create -f deployment.yaml</pre><p>successful deployment should look like below. Execute below command</p><pre>$ kubectl get deployment</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*FNUo0pvNpnT7UkEgwgS2yw.png" /></figure><p>To get information about the replicas, execute below commands</p><pre>kubectl get replicasets<br>kubectl describe replicasets</pre><h4>Updating a Deployment</h4><p>Once deployment is created, we can update it in multiple ways. One of them is to edit the deployment file and specify the changes there, and run the below command(note “apply” instead of “create” is used)</p><pre>$ kubectl apply -f deployment.yaml</pre><p>Second option is to edit the specific object using below command. It will open up a editor where you can modify the settings. Once you save, the settings will be applied to the running deployment. Let’s say you changed the number of replicas, then, Kubernetes will apply those settings to the running deployment and ensure the specified number of replicas running.</p><pre>$ kubectl edit deployment.v1.apps/webapp</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*ITReAPfFTaiA3yrDUYpjJQ.png" /><figcaption>On saving the edited deployment</figcaption></figure><h4>Other ways of scaling</h4><p>You can execute the below command instead of modifying any of the files.</p><pre>kubectl scale deployment.v1.apps/webapp --replicas=2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*G_i6IoyH-kG34GWtFyEi-g.png" /><figcaption>Output of above command</figcaption></figure><p>Let’s autoscale the deployment and instruct Kubernetes to ensure that our pods are never using more than 60% of their available CPU capacity, minimum and maximum number of pods that could be running at any point in time.</p><pre>kubectl autoscale deployment.v1.apps/webapp --min=3 --max=20 --cpu-percent=60</pre><h4>Create a Service for our Deployment from command line</h4><p>It’s time to see our apps in action and see if the pages from both the apps load correctly.</p><p>The application container that we deployed in the Pod are only accessible within the Cluster. We need to access our application outside the cluster. Kubernetes “Service” object provides access into the cluster along with loadbalacing feature to the nodes where are apps are deployed.</p><p>Create a service object to expose our deployment</p><pre># <strong>This command makes our apps accessible over network on the ports specified during deployment</strong></pre><pre>$ kubectl expose deployment webapp --type=NodePort --name=webapp-service</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKcLmpBTd-pwPd-jLaqdQQ.png" /><figcaption>Exposing Kubernetes Deployment</figcaption></figure><p>The applications should now be accessible on each of the worker nodes on a specific port that the service has exposed. To check the ports execute below command</p><pre># <strong>As can be seen in image below, our applications are exposed over ports 30390 (reactui) and 31649 (webapi)</strong></pre><pre>$ kubectl get service webapp-service</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*byUNW6nJerTc74NP63pUWg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4ZBXsPyL7BKpel-XwV1uQ.png" /><figcaption>ReactUI App on Worker Node 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-4_Ej_sVnlGjz49Cshyrg.png" /><figcaption>WebAPI App on Worker Node 1</figcaption></figure><p>Similiar, to above images, you will find the applications exposed on Worker Node 2, Worker Node 3. In case you have specified more replicas than your Worker Nodes, Kubernetes will create more pods and assign them different ports on the same worker node. This means, there may be more than 1 Pod on the same Worker Node.</p><p>To list out all the Pods of our webapp deployment, we can execute below command with the selector specified in the deployment.yaml</p><pre>$ kubectl get pods --selector=&quot;app=web&quot; --output=wide</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGtl8-u833ilb1177QHEoA.png" /><figcaption>Web App comprising of 2 containers running in Pods on 3 Worker Nodes</figcaption></figure><p>That’s it for Kubernetes Deployment. Hope this entire article was exciting and fulfilled your search for Kubernetes Deployment from a private registry.</p><p>In the Next article, I will be covering exposing these apps over ingress using MetalLB load balancer as we are deploying on a Local Kubernetes Cluster.</p><h4>Useful Links and Resources</h4><p>This <a href="https://www.mirantis.com/blog/introduction-to-yaml-creating-a-kubernetes-deployment/">site </a>gives a good understanding of YAML markup and the process to deploy apps to Kubernetes.</p><p>This <a href="https://collabnix.com/creating-your-first-react-app-using-docker/">blog </a>provides good information on containerizing your ReactJS app.</p><p>Know more about <a href="https://kubernetes.io/docs/tasks/access-application-cluster/service-access-application-cluster/">Kubernetes Services</a> to access the app on worker Nodes and <a href="https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/">Cluster Wide</a> over Load Balancer.</p><p>Good <a href="https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0">blog</a> for understanding the difference between NodePort, LoadBalancer, and Ingress.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b8d1608642f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup 3 Node Kubernetes Cluster on Ubuntu 22.04]]></title>
            <link>https://medium.com/@DhaneshMalviya/setup-3-node-kubernetes-cluster-on-ubuntu-22-04-d4652ac17207?source=rss-6a25c22fd178------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4652ac17207</guid>
            <category><![CDATA[ubuntu-22-04]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <dc:creator><![CDATA[Dhanesh Malviya]]></dc:creator>
            <pubDate>Thu, 03 Nov 2022 14:33:08 GMT</pubDate>
            <atom:updated>2022-11-03T14:33:08.480Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently started using Kubernetes and its a fantastic tool for deploying microservices. There are many providers offering managed Kubernetes Cluster. But the way forward to learning is by deploying all the moving parts on your own to gain a better understanding. I searched through various websites to setup, however some lacked clear instructions and some were for older version of Kubernetes or Ubuntu. This is my attempt to document the procedure for an on-premise installation of Kubernetes. Let’s dive in.</p><p>Kubernetes is an opensource container orchestration tool that lets you deploy containerized applications. With Kubernetes, you can achieve automated deployments, scaling and management. You can learn more about the Kubernetes components and Architecture <a href="https://blog.knoldus.com/introduction-to-google-kubernetes/">here</a></p><p>Kubernetes cluster consists of one or more master nodes that manage one or more worker nodes. The applications are deployed on worker nodes within pods( Pods can contain one ore more containers which allow to manage multiple containers as a single entity and share the Pod’s resources)</p><p>In this guide, we will be setting up 1 master node along with 3 worker nodes. The physical resources on each node can be different with a minimum of the following for successful setup.</p><ul><li>Minimal install Ubuntu 22.04</li><li>Minimum 2GB RAM or more</li><li>Minimum 2 CPU cores / or 2 vCPU</li><li>20 GB free disk space on /var or more</li><li>Sudo user with admin rights</li><li>Internet connectivity on each node</li></ul><p>Here’s how our setup will look like</p><p>Master Node : 192.168.100.240 (k8scontrol)<br>Worker Node 1 : 192.168.100.241 (k8snode1)<br>Worker Node 2 : 192.168.100.242 (k8snode2)<br>Worker Node 3 : 192.168.100.243 (k8snode3)</p><p>Below image shows the setup and Kubernetes components involved. More details of the Kubernetes components is provided on <a href="https://kubernetes.io/docs/concepts/overview/components/">Kubernetes website</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*gKhVgXmw65UqwYTBIe7HDw.jpeg" /><figcaption>Kubernetes Components &amp; Setup Architecture</figcaption></figure><h4>Step 1 (Optional) : setting up hostname entries</h4><p>Login to master node and execute the following commands. These commands will setup the hostname for easy identification instead of using IP’s directly. This is optional.</p><pre>$ sudo hostnamectl set-hostname &quot;k8scontrol&quot;<br>$ exec bash</pre><p>Execute on Worker Node 1</p><pre>$ sudo hostnamectl set-hostname &quot;k8snode1&quot;<br>$ exec bash</pre><p>Execute on Worker Node 2</p><pre>$ sudo hostnamectl set-hostname &quot;k8snode2&quot;<br>$ exec bash</pre><p>Execute on Worker Node 3</p><pre>$ sudo hostnamectl set-hostname &quot;k8snode3&quot;<br>$ exec bash</pre><p>Add following entries to /etc/hosts on each node</p><pre>192.168.100.240 k8scontrol<br>192.168.100.241 k8snode1<br>192.168.100.242 k8snode2<br>192.168.100.243 k8snode3</pre><h4>Step 2 : Disabling Swap Memory &amp; add Kernel Settings</h4><p>Disabling swap ensures all the Kubernetes pods are tied up to RAM, do not utilize swap memory and provide guaranteed QoS (server resources) to the pods. <a href="https://medium.com/tailwinds-navigator/kubernetes-tip-why-disable-swap-on-linux-3505f0250263">Here’s</a> a good article that helped me on this topic.</p><p>Note we are using “sed” (stream editor) command to find and remove swap entries in /etc/fstab file</p><pre>#permanently disable the swap memory on master node<br>ccloud@k8scontrol:~$ sudo swapoff -a<br>ccloud@k8scontrol:~$ sudo rm /swap.img<br>ccloud@k8scontrol:~$ sudo sed &quot;-i.bak&quot; &#39;/swap.img/d&#39; /etc/fstab</pre><pre>#permanently disable the swap memory on all worker node 1<br>ccloud@k8snode1:~$ sudo swapoff -a<br>ccloud@k8snode1:~$ sudo rm /swap.img<br>ccloud@k8snode1:~$ sudo sed &quot;-i.bak&quot; &#39;/swap.img/d&#39; /etc/fstab</pre><pre>#permanently disable the swap memory on all worker node 2<br>ccloud@k8snode2:~$ sudo swapoff -a<br>ccloud@k8snode2:~$ sudo rm /swap.img<br>ccloud@k8snode2:~$ sudo sed &quot;-i.bak&quot; &#39;/swap.img/d&#39; /etc/fstab</pre><pre>#permanently disable the swap memory on all worker node 3<br>ccloud@k8snode3:~$ sudo swapoff -a<br>ccloud@k8snode3:~$ sudo rm /swap.img<br>ccloud@k8snode3:~$ sudo sed &quot;-i.bak&quot; &#39;/swap.img/d&#39; /etc/fstab</pre><p>Load the following kernel modules on the master and worker nodes</p><pre># <strong>Run below commands on all nodes to see the bridged traffic</strong><br>$ sudo tee /etc/modules-load.d/containerd.conf &lt;&lt;EOF<br>overlay<br>br_netfilter<br>EOF</pre><pre># enables/loads the overlay module in kernel</pre><pre>$ sudo modprobe overlay</pre><pre>#br_netfilter - This module is required to enable transparent masquerading and to facilitate Virtual Extensible LAN (VxLAN) traffic for communication between Kubernetes pods across the cluster</pre><pre>$ sudo modprobe br_netfilter</pre><p>Set the following Kernel parameters for Kubernetes, using <a href="https://www.linuxtechi.com/tee-command-examples-in-linux/">tee command</a></p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo tee /etc/sysctl.d/kubernetes.conf &lt;&lt;EOF<br>net.bridge.bridge-nf-call-ip6tables = 1<br>net.bridge.bridge-nf-call-iptables = 1<br>net.ipv4.ip_forward = 1<br>EOF</pre><p>reload the above changes (reads from all system directories and from the one mentioned above ../etc/sysctl.d/kubernetes.conf)</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo sysctl --system</pre><h4>Step 3 : Installing docker / containerd runtime</h4><p>For our setup we are using containerd runtime to run containers. Kubernetes supports container runtimes such as <a href="https://containerd.io/docs/">containerd</a>, <a href="https://cri-o.io/#what-is-cri-o">CRI-O</a>, and any other implementation of the <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md">Kubernetes CRI (Container Runtime Interface)</a>.</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo apt-get update<br>$ sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates</pre><p>Enable Docker Repository</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg</pre><pre>$ sudo add-apt-repository &quot;deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable&quot;</pre><p>Run the following commands to install containerd</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo apt update<br>$ sudo apt install -y containerd.io</pre><p>Configure containerd so that it starts using systemd as cgroup.</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ containerd config default | sudo tee /etc/containerd/config.toml &gt;/dev/null 2&gt;&amp;1<br>$ sudo sed -i &#39;s/SystemdCgroup \= false/SystemdCgroup \= true/g&#39; /etc/containerd/config.toml</pre><p>Restart and enable containerd service</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo systemctl restart containerd<br>$ sudo systemctl enable containerd</pre><h4>Step 4 : Add apt repository for Kubernetes</h4><p>Execute below commands to add apt repositories for Kubernetes</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -<br>$ sudo apt-add-repository &quot;deb http://apt.kubernetes.io/ kubernetes-xenial main&quot;</pre><h4>Step 5 : Install Kubernetes Components (kubeadm, kubectl and kubelet)</h4><p>kubeadm : tool to setup cluster without much complex configuration and does the heavy lifting in terms of setting up the cluster components. (for development purposes, setting up a single node cluster can be done using minikube)</p><pre># <strong>Run below commands on all nodes</strong></pre><pre>$ sudo apt update<br>$ sudo apt install -y kubelet kubeadm kubectl<br>$ sudo apt-mark hold kubelet kubeadm kubectl</pre><p>Also make sure the below ports are accessible and open on the master and worker nodes. If you are setting up in cloud make sure these ports are open accessible from every node in firewall configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*G3gI-T7D_4EbcGDnwetrdw.png" /><figcaption>Ports for Kubernetes components</figcaption></figure><h4>Step 6 : Initialize Kubernetes Cluster using kubeadm</h4><p>Thanks for hanging out so far. We are now ready to initialize the Kubernetes cluster. Run the following command on master node only. Below command will initialize the cluster on master node and assign IP’s to all the pods with the specified CIDR format.</p><pre># <strong>Run this command only on master node</strong></pre><pre>$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=k8scontrol</pre><p>Once the setup completes, you will get below output. The output consists of commands that you need to run on the master node and worker nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5MH-Ho2QTsbZKWT7VLiDA.png" /><figcaption>Output of kubeadm init command</figcaption></figure><p>The above output shows that we kubeadm init was successful. The output consists of commands that need to be run in order to communicate with the cluster. The output also shows command that needs to be run on worker node to make it a part of the cluster.</p><p>so let’s go ahead and run the commands to interact with the cluster.</p><pre># <strong>Run this command only on master node</strong></pre><pre>$ mkdir -p $HOME/.kube<br>$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config<br>$ sudo chown $(id -u):$(id -g) $HOME/.kube/config</pre><p>To view our cluster status and node status, run the following command</p><pre>$ kubectl cluster-info<br>$ kubectl get nodes</pre><p>The output of the commands above will be as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*H6_v9Of_SN_0j6qljWod6w.png" /><figcaption>kubectl cluster-info command</figcaption></figure><h4>Step 7: Add worker nodes to cluster</h4><p>Now lets join the worker nodes to the master using the command that was in the output earlier. just copy paste on the worker nodes.</p><pre># <strong>Run this command only on worker nodes</strong></pre><pre>$ sudo kubeadm join k8scontrol:6443 --token n26y5i.thy19qec7vgo2ml7 \<br>        --discovery-token-ca-cert-hash sha256:7410e9117dc43ce8e194558f0753f907690058d61872f7ff9a9f09457fa30ccd</pre><p>below is the output of the worker node 1. You will get similar outputs from other worker nodes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W2SNTKfhyS1NFcaKR3youw.png" /><figcaption>Output from worker node 1</figcaption></figure><p>having formed the cluster and added worker nodes to the master, we can now check whether the nodes are active or not. Run the below command to check the status of the nodes.</p><pre>$ kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*b1bzkoIkmbY0IAxblZFfVg.png" /><figcaption>kubectl get nodes output</figcaption></figure><p>as can be seen in the output above, the nodes are in “NotReady” . we will activate these nodes by installing CNI (Container Network Interface) or network add-on plugins like Calico, Flannel, Weave-net. Calico worked better for my setup. I somehow had problem with Flannel.</p><h4>Step 8: Install Calico Pod Network Add-On</h4><p>Run following curl and kubectl command to install Calico network plugin from the master node.</p><pre>$ curl https://projectcalico.docs.tigera.io/manifests/calico.yaml -O<br>$ kubectl apply -f calico.yaml</pre><p>The output of the commands above will look like below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eM3ncFCC2HsPtSAzr4e7dQ.png" /><figcaption>Kubectl Calico output</figcaption></figure><p>the above commands will create pods in kube-system namespace. you can read more about namespaces <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">here</a>. Basically namespaces allow you to group resources together and isolate within a single cluster. These are good fits for multiple teams or projects and require isolation.</p><p>verify the status of the pods in the kube-system namespace</p><pre>$ kubectl get pods -n kube-system</pre><p>the output of above command will look like below. Ensure that all the pods are in the running state. There may be time where the pods may be crashing with “CrashloopBackoff” as the status. There could be multiple reasons for which you can use “kubectl describe pod podname” command to get more insights into the failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*Bn2E_FyQGUBZqexEngfKnQ.png" /><figcaption>Pod statuses</figcaption></figure><p>Lets check our nodes status. All the nodes should be active now and in ready state. run the below command</p><pre>$ kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*0KiihCJqtwHSQXYHlEJVPg.png" /><figcaption>All nodes in running state</figcaption></figure><p>That’s perfect. All nodes have been successfully added to the cluster, functional and are in ready state.</p><h4>Step 9 : Testing Kubernetes Installation</h4><p>Its time to put the setup to test. For this, we will pull an nginx web server image and deploy it on our cluster. Post deployment we should be able to access this server directly on each of the nodes port. Another way to access the server would be to create a “service” and “ingress” that will allow us to access the web server via a single entry point through ingress.</p><p>For now, lets just use node port approach to test our setup.</p><p>Execute the following command on master node to create a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Kubernetes Deployment</a> for nginx. This will create pods with nginx containers on the nodes and ensure these are always running. The number of instances can be controlled with the “replicas” parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*VUClfTQE3K8BNFr4ySIPIg.png" /></figure><p>Lets view our deployment details using the command. As seen in the output, by default only one replica is create which means that kubernetes controller will ensure that one instance of nginx will be always running on any of the worker nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*Y3SyGDLOUlprVoKJ1xfvDg.png" /></figure><p>Lets see the number of pods created for our deployment. Run below command</p><pre>#Run this command on master node</pre><pre>$ kubectl get pods</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*sTHmv0y1YPDUROQRer5a3Q.png" /></figure><p>Lets scale up our instances to 3, since we have a 3 node cluster. Run below command on master node</p><pre>$ kubectl scale --replicas=3 deployment/nginx</pre><p>Here’s the result of our scaling</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*K4mb334ANeRljhq0KRCZ5w.png" /></figure><p>To make the nginx service accessible over network, we will create a <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Kubernetes Service</a> which will expose the port on pod that nginx is listening on to a port on the worker node.</p><p>Run below command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*i0pYoSIx0YIqEENFoogZGA.png" /></figure><p>This being a NodePort type, Kubernetes assigns the service a port in the range of 32000 or greater</p><pre>ccloud@k8scontrol:~$ <strong>kubectl get svc</strong><br>NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE<br>kubernetes   ClusterIP   10.96.0.1        &lt;none&gt;        443/TCP        21h<br>nginx        NodePort    10.110.141.179   &lt;none&gt;        80:<strong>32314</strong>/TCP   108s</pre><p>Take a note of the port above (32314) and point your browser to the worker nodes <a href="http://192.168.100.241:32314/">http://192.168.100.241:32314</a> . This will display the default nginx page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1TKYFfXA8Mn6zqYjoYGZg.png" /><figcaption>Worker node 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmEJNYfFnaufcdQu4SR8aw.png" /><figcaption>Worker node 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjl-xf_aZ-radp5I2SH93w.png" /><figcaption>Worker node 3</figcaption></figure><p>That’s a long way we have come up. Thanks for staying up here. Our Kubernetes on-premise setup is successful!</p><h4>Reference Materials</h4><p>You can refer these articles which were helpful, <a href="https://www.linuxtechi.com/setup-docker-private-registry-on-ubuntu/">Link1</a>, <a href="https://hakanmazi123.medium.com/how-to-install-kubernetes-cluster-with-docker-on-ubuntu-20-04-49d4ec102dcf">Link 2</a>, <a href="https://www.cloudsigma.com/how-to-install-and-use-kubernetes-on-ubuntu-20-04/">Link 3</a></p><p>I was setting up the cluster on-premise. I also required a private docker registry to pull inhouse created application containerized images. I’ll post those updates in another article and provide the link here.</p><h4>Useful commands</h4><p>I did a lot of goof ups while installing Kubernetes referring various articles. Some of them worked only upto certain extent with pods, nodes failing frequently. I had re-installed Ubuntu a couple of times just to start with a clean slate. However, sometime it was enough to just remove the Kubernetes packages. Here are few commands for reference.</p><pre># Reset the kubeadm cluster setup<br>sudo kubeadm reset</pre><pre># Remove all Kubernetes packages</pre><pre>sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*   <br>sudo apt-get autoremove  <br>sudo rm -rf ~/.kube<br>sudo rm -rf /etc/cni /etc/kubernetes /var/lib/dockershim /var/lib/etcd /var/lib/kubelet /var/run/kubernetes ~/.kube/*</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4652ac17207" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>