<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohan P Edala on Medium]]></title>
        <description><![CDATA[Stories by Mohan P Edala on Medium]]></description>
        <link>https://medium.com/@mohanpedala?source=rss-3f1161b894cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*nZJaTIzKrlr_t7tuyC08MQ.jpeg</url>
            <title>Stories by Mohan P Edala on Medium</title>
            <link>https://medium.com/@mohanpedala?source=rss-3f1161b894cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:50:28 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohanpedala/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Enable Workload Identity on GKE cluster]]></title>
            <link>https://medium.com/@mohanpedala/enable-workload-identity-on-gke-cluster-9ff139baa2ea?source=rss-3f1161b894cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ff139baa2ea</guid>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Mohan P Edala]]></dc:creator>
            <pubDate>Tue, 26 May 2020 00:53:52 GMT</pubDate>
            <atom:updated>2020-06-06T22:38:26.182Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Scenario</strong>: A microservice in GKE cluster should communicate with a pubsub topic using kubernetes service account, GCP IAM, workload identity and RoleBindings</blockquote><p><strong><em>What is workload identity</em></strong>: It is the recommended way to access Google Cloud services from within <strong>GKE</strong> due to its improved security properties and manageability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*6nS2WJDTSSK6nwHTbTTeeQ.png" /></figure><blockquote><strong>Enable workload-identity</strong></blockquote><p><strong>gcloud reference:</strong></p><ul><li>Enable workload identity on existing gke cluster</li></ul><pre>gcloud container clusters update CLUSTER_NAME --zone us-west1-a --workload-pool=PROJECT_ID.svc.id.goog</pre><blockquote>Node pool modification: Modify an existing node pool to enable GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool.</blockquote><ul><li>list node pools to identify the node-pool name in the cluster</li></ul><pre>gcloud container node-pools list --cluster=CLUSTER_NAME --zone us-west1-a --project &lt;PROJECT_ID&gt;</pre><ul><li>update node pools with workload-metadata</li></ul><pre>gcloud container node-pools update CLUSTER_NAME_NODE_POOL_NAME --cluster=CLUSTER_NAME --zone us-west1-a --workload-metadata-from-node=GKE_METADATA</pre><ul><li>describe an existing node pool to verify the `workloadMetadataConfig`</li></ul><pre>gcloud container node-pools describe CLUSTER_NAME_NODE_POOL_NAME --cluster=CLUSTER_NAME --zone us-west1-a</pre><p><strong>Terraform refernce:</strong></p><ul><li>Enable “workload identity” in `google_container_cluster` resource</li></ul><pre>workload_identity_config {<br>    identity_namespace = &quot;${var.google_project}.svc.id.goog&quot;<br>}</pre><ul><li>Now enable workload identity in node pools `google_container_node_pool`.</li></ul><pre>workload_metadata_config {<br>    node_metadata = &quot;GKE_METADATA_SERVER&quot;<br>}</pre><ul><li>Binding google service account and K8s service account</li></ul><pre>resource &quot;google_service_account_iam_binding&quot; &quot;gsa_ksa_binding&quot; {<br>    service_account_id = <br>projects/${var.project}/serviceAccounts/${var.gke_sa_email}&quot;<br>    role = &quot;roles/iam.workloadIdentityUser&quot;<br>    members = [<br>        &quot;serviceAccount:${var.project}.svc.id.goog[flux/flux]&quot;,<br>        &quot;serviceAccount:${var.project}.svc.id.goog[flux/default]&quot;<br>    ]<br>}</pre><ul><li>annotating k8s service account</li></ul><pre>resource “null_resource” “annotate_ksa” {<br>triggers = {<br>    cluster_ep = “${google_container_cluster.gke_cluster.endpoint}” #kubernetes cluster endpoint<br>}<br>    provisioner “local-exec” {<br>        command = &lt;&lt;EOT<br>            gcloud container clusters get-credentials $${K8S_CLUSTER} — zone $${K8S_ZONE} — project $${K8S_PROJECT}<br>            kubectl annotate serviceaccount — namespace flux flux iam.gke.io/gcp-service-account=$${GCP_SA_EMAIL}<br>            kubectl annotate serviceaccount — namespace flux default iam.gke.io/gcp-service-account=$${GCP_SA_EMAIL}<br>        EOT<br>        environment = {<br>            GCP_SA_EMAIL = “${var.gke_sa_email}”<br>            K8S_CLUSTER = <br>${google_container_cluster.gke_cluster.name}”<br>            K8S_ZONE = “${var.location}”<br>            K8S_PROJECT = “${var.project}”<br>        }<br>    }<br>    depends_on = <br>“google_service_account_iam_binding.gsa_ksa_binding”]<br>}</pre><blockquote><strong>Create Kubernetes service account (KSA), Google service account (GSA), annotate KSA and rolebinding KSA and GSA</strong></blockquote><ul><li>Create KSA</li></ul><pre>kubectl create serviceaccount --namespace K8S_NAMESPACE KSA_NAME</pre><ul><li>GSA creation</li></ul><pre>gcloud iam service-accounts create GSA_NAME</pre><ul><li>Rolebinding between GSA and KSA</li></ul><pre>gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member &quot;serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAME_SPACE/KSA]&quot; GSA@PROJECT_ID.iam.gserviceaccount.com</pre><ul><li>Annotate KSA</li></ul><pre>kubectl annotate serviceaccount --namespace NAME_SPACE KSA iam.gke.io/gcp-service-account=GSA@PROJECT_ID.iam.gserviceaccount.com</pre><blockquote><strong>pubsub topic and rolebinding with GSA</strong></blockquote><ul><li>Create pubsub topic</li></ul><pre>gcloud pubsub topics create TOPIC_NAME</pre><ul><li>Create a pubsub iam rolebinding</li></ul><pre>gcloud pubsub topics add-iam-policy-binding TOPIC_NAME --member=GSA --role = &quot;roles/pubsub.publisher&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ff139baa2ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cert-Manager using venafi “ClusterIssuer”]]></title>
            <link>https://medium.com/@mohanpedala/cert-manager-using-venafi-clusterissue-4a88624d2376?source=rss-3f1161b894cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a88624d2376</guid>
            <category><![CDATA[cert-manager]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Mohan P Edala]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 02:30:18 GMT</pubDate>
            <atom:updated>2020-04-28T15:23:05.659Z</atom:updated>
            <content:encoded><![CDATA[<p>Cert-Manager is a Kubernetes add-on to automate the management and issuance of TLS certificates from various issuing sources. It will ensure certificates are valid and up to date periodically, and attempt to renew certificates at an appropriate time before expiry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*En4nRMqrfS4ZL_2PmrzvWw.png" /></figure><ul><li>helm3</li><li>ingress-nginx</li><li>create an account in venafi-cloud (if you use venafi-cloud instead of venafi-tpp)</li><li>cert-manager helm chart (minimum cert-manager v0.13.1)</li><li>Creating a Venafi Issuer resource</li></ul><blockquote><strong>Install Helm3</strong></blockquote><ul><li>macOS installation using Homebrew</li></ul><pre>brew install helm</pre><ul><li>Windows installation using chocolatey</li></ul><pre>choco install kubernetes-helm</pre><ul><li>Linux installation</li></ul><pre>sudo snap install helm --classic</pre><blockquote><strong>Deploy nginx</strong></blockquote><pre>helm install nginx stable/nginx-ingress --namespace nginx --set rbac.create=true --set controller.publishService.enabled=true</pre><blockquote><strong>Install cert-manager</strong></blockquote><ul><li>Add jetstack helm repo</li></ul><pre>helm repo add jetstack https://charts.jetstack.io</pre><ul><li>Update helm repo</li></ul><pre>helm repo update</pre><ul><li>Create a cert-manager namespace</li></ul><pre>kubectl create ns cert-manager</pre><ul><li>Install helm repo</li></ul><pre>helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.13.1</pre><blockquote><strong>Configure cert-manger</strong></blockquote><p>Choose one of the venafi cert issuer from below</p><ul><li>venafi-tpp server</li><li>venafi cloud</li></ul><p><strong>venafi-tpp server</strong></p><ul><li>Create kubernetes secret to authenticate with your Venafi TPP server</li></ul><pre>apiVersion: v1<br>data:<br>  password: WU9VUl9UUFBfUEFTU1dPUkRfSEVSRQ==<br>  username: WU9VUl9UUFBfVVNFUk5BTUVfSEVSRQ==<br>kind: Secret<br>metadata:<br>  name: venafi-tpp-secret<br>  namespace: cert-manager</pre><ul><li>Create a Venafi ClusterIssuer resource, which represents a certificate authority within Kubernetes.</li></ul><pre>apiVersion: cert-manager.io/v1alpha2<br>kind: ClusterIssuer<br>metadata:<br>  name: venafi-cluster-issuer<br>spec:<br>venafi:<br>  zone: &quot;Default&quot; # Set this to the Venafi policy zone you want to use<br>  tpp:<br>    url: https://venafi-tpp.example.com/vedsdk # Change this to the URL of your TPP instance<br>    caBundle: &lt;base64 encoded string of caBundle PEM file, or empty to use system root CAs&gt;<br>    credentialsRef:<br>name: venafi-tpp-secret # this secret is created above</pre><ul><li>Check the clusterissuer event for Verified issuer with Venafi server</li></ul><pre>kubectl describe clusterissuer venafi-cluster-issuer</pre><pre>Status:<br>Conditions:<br>Last Transition Time:  2019-07-17T15:46:00Z<br>Message:               Venafi issuer started<br>Reason:                Venafi issuer started<br>Status:                True<br>Type:                  Ready<br>Events:<br>Type    Reason  Age   From          Message<br>----    ------  ----  ----          -------<br>Normal  Ready   14s   cert-manager  Verified issuer with Venafi server</pre><p><strong>venafi cloud</strong></p><ul><li>Fetch your API key by clicking the profile name or service account in the top right of the control panel interface.</li><li>Create a kubernetes secret for apikey</li></ul><pre>apiVersion: v1<br>data:<br>  apikey: &lt;API_KEY&gt;<br>kind: Secret<br>metadata:<br>  name: venafi-cloud-secret<br>  namespace: cert-manager</pre><ul><li>Create a Venafi ClusterIssuer resource, which represents a certificate authority within Kubernetes.</li></ul><pre>apiVersion: cert-manager.io/v1alpha2<br>kind: ClusterIssuer<br>metadata:<br>  name: venafi-cluster-issuer<br>spec:<br>venafi:<br>    zone: &quot;Default&quot; # Set this to the Venafi policy zone you want to use<br>    cloud:<br>    apiTokenSecretRef:<br>      name: venafi-cloud-secret<br>      key: apikey</pre><ul><li>Check the clusterissuer event for Verified issuer with Venafi server</li></ul><pre>kubectl describe clusterissuer venafi-cluster-issuer</pre><pre>Status:<br>Conditions:<br>Last Transition Time:  2019-07-17T15:46:00Z<br>Message:               Venafi issuer started<br>Reason:                Venafi issuer started<br>Status:                True<br>Type:                  Ready<br>Events:<br>Type    Reason  Age   From          Message<br>----    ------  ----  ----          -------<br>Normal  Ready   14s   cert-manager  Verified issuer with Venafi server</pre><blockquote><strong>Request a Certificate</strong></blockquote><ul><li>Create certificate-request.yaml this request should be created per namespace and app</li></ul><pre>apiVersion: cert-manager.io/v1alpha2<br>kind: Certificate<br>metadata:<br>  name: &lt;service_name&gt;-tls<br>  namespace: &lt;NAMESPACE_NAME&gt;<br>spec:<br>  secretName: &lt;service_name&gt;-tls<br>  commonName: &#39;domain.com&#39;<br>  dnsNames:<br>  - &#39;domain.com&#39;<br>  issuerRef:<br>    kind: ClusterIssuer<br>    name: venafi-cluster-issuer</pre><ul><li>Check the certificate status to verify the status is true</li></ul><pre>kubectl get certificate &lt;service_name&gt;-tls -n &lt;NAMESPACE_NAME&gt;</pre><pre>NAME                 READY   SECRET               AGE<br>&lt;service_name&gt;-tls   True    &lt;service_name&gt;-tls   96m</pre><ul><li>To verify the certificate content in the Secret resource use the below command</li></ul><pre>kubectl get secret -n &lt;NAMESPACE_NAME&gt; &lt;service_name&gt;-tls -o yaml | grep tls.crt | awk &#39;{print $2}&#39; | base64 -d | openssl x509 -noout -text</pre><blockquote><strong>Test run deploy sample app</strong></blockquote><ul><li>Deploy helloworld app</li></ul><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: hello-app<br>  namespace: &lt;NAMESPACE_NAME&gt;<br>spec:<br>  type: ClusterIP<br>  ports:<br>  - port: 8080<br>    protocol: TCP<br>    targetPort: 8080<br>  selector:<br>    app: hello-app</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: hello-app<br>  namespace: &lt;NAMESPACE_NAME&gt;<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: hello-app<br>  template:<br>    metadata:<br>      labels:<br>        app: hello-app<br>    spec:<br>      containers:<br>      - name: hello-app<br>        image: gcr.io/google-samples/hello-app:1.0<br>        resources:<br>          requests:<br>            cpu: 500m<br>            memory: 500Mi</pre><ul><li>Create Ingress resource</li></ul><pre>apiVersion: networking.k8s.io/v1beta1<br>kind: Ingress<br>metadata:<br>  name: helloworld<br>  namespace: &lt;NAMESPACE_NAME&gt;<br>  annotations:<br>    kubernetes.io/ingress.class: nginx<br>spec:<br>  tls:<br>  - hosts:<br>    - hello.domain.com<br>    secretName: &lt;service_name&gt;-tls<br>  rules:<br>  - host: hello.domain.com<br>    http:<br>      paths:<br>      - path: /helloworld<br>        backend:<br>          serviceName: hello-app<br>          servicePort: 8080</pre><ul><li>open the browser and see if you can access <a href="https://hello.domain.com/">https://hello.domain.com</a> with a venafi issued cert.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a88624d2376" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CoreDNS pods DNS resolution issue]]></title>
            <link>https://medium.com/@mohanpedala/coredns-pods-dns-resolution-issue-2282110a0b37?source=rss-3f1161b894cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2282110a0b37</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[coredns]]></category>
            <category><![CDATA[resolutions]]></category>
            <category><![CDATA[issues]]></category>
            <dc:creator><![CDATA[Mohan P Edala]]></dc:creator>
            <pubDate>Mon, 21 Oct 2019 22:48:50 GMT</pubDate>
            <atom:updated>2024-06-16T22:02:31.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*Tsj9abmWZcuyBaD9qCn0Hw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*XNaTGMtYn0kVkwhwNbG0Kw.png" /></figure><p>Hey All I wanted to share my story about the recent dns issue which we faced.</p><p><strong>Root cause</strong>: There was a minor network glitch which caused interruption between the cluster and DNS server which led to dns resolution timeout</p><p><strong>Error</strong>:</p><pre>[ERROR] plugin/errors: 2 svc-example.example-prod.svc.cluster.local.exmaple.com. A: unreachable backend: read udp : i/o timeout<br>[ERROR] plugin/errors: 2 master01.example.com. A: unreachable backend: read udp x.x.x.x:36063-&gt;x.x.x.x:53: i/o timeout<br>[ERROR] plugin/errors: 2 master02.example.com. A: unreachable backend: read udp x.x.x.x:36063-&gt;x.x.x.x:53: i/o timeout<br>[ERROR] plugin/errors: 2 master03.example.com. A: unreachable backend: read udp x.x.x.x:36063-&gt;x.x.x.x:53: i/o timeout<br>[ERROR] plugin/errors: 2 node1.example.com. A: unreachable backend: read udp x.x.x.x:36063-&gt;x.x.x.x:53: i/o timeout<br>[ERROR] plugin/errors: 2 node2.example.com. A: unreachable backend: read udp x.x.x.x:36063-&gt;x.x.x.x:53: i/o timeout</pre><p>CoreDNS fail to resolve internal and external names(external: redis, splunk, etc) (internal: masters, nodes, svc, pods, etc)</p><p>As part of troubleshooting we started using busyboxto see if the dns resolution is happening as expected or not.</p><pre>kubectl run -it --rm --restart=Never --image=busybox busybox</pre><p>Run the below nslookup command</p><pre>/ # nslookup kubernetes.default.svc.cluster.local</pre><pre>Output:</pre><pre>Server:         10.96.0.10<br>Address:        10.96.0.10#53</pre><pre>*** Can&#39;t find kubernetes.default.svc.cluster.local: No answer</pre><p>But the output seems to be showing that the resolution is not happening as expected. we suspected that something went bad with CoreDNS pods.</p><p>Then we came to know that some times busybox cannot do the proper dns resolution as the dns tools</p><p>so we spun a new pod which has the dns tools</p><pre>kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools</pre><p>Run the below nslookup command.</p><pre>dnstools# nslookup kubernetes.default.svc.cluster.local</pre><pre>Output:</pre><pre>Server:  10.96.0.10<br>Address: 10.96.0.10#53</pre><pre>Name: kubernetes.default.svc.cluster.local<br>Address: 10.96.0.1</pre><p>The above output shows that dns resolution is happening.</p><p>But even after the network issue was resolved the issue in CoreDNS pods still exists.</p><h4>Identifying the Issue</h4><p>Our investigation into the DNS resolution issue unfolded as follows:</p><ul><li><strong>Monitoring with Datadog</strong>: Upon receiving the Alerts , we focused on CPU and memory utilization across Kubernetes nodes.</li><li><strong>Detecting Abnormal Compute Usage</strong>: Datadog revealed an anomaly in CPU consumption on one of the nodes hosting CoreDNS pods. This unusual spike indicated that CoreDNS was monopolizing CPU resources, potentially starving other critical processes.</li><li><strong>Verification on the VM</strong>: To delve deeper, we accessed the affected VM directly. Here, we observed that the number of CoreDNS child processes had reached its maximum limit, indicating a resource exhaustion issue.</li></ul><h4><strong>Mitigation Steps</strong>:</h4><ul><li><strong>Restarting the Pod</strong>: As an immediate measure, we restarted the CoreDNS pod. Scheduler scheduled the pod in a new node, thereby resuming the dns calls in the cluster.</li><li><strong>Removing the Starving Node</strong>: Subsequently, we decommissioned the node experiencing resource starvation from the Kubernetes cluster. This step prevented further disruptions and ensured other pods continued to operate smoothly.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2282110a0b37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Certificate rotation — Manual Process]]></title>
            <link>https://medium.com/@mohanpedala/kubernetes-certificate-rotation-manual-process-c0d0433b4eed?source=rss-3f1161b894cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0d0433b4eed</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[certificate]]></category>
            <category><![CDATA[rotation]]></category>
            <dc:creator><![CDATA[Mohan P Edala]]></dc:creator>
            <pubDate>Wed, 18 Sep 2019 22:24:33 GMT</pubDate>
            <atom:updated>2025-02-13T16:34:39.050Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Certificate rotation — Manual Process</h3><blockquote>This rotation process is done on kubernetes v 1.9 and 1.10</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*rhzQ9EH-aeaq36huuYgZjQ.png" /></figure><p><strong>Download cfssl</strong></p><pre>mkdir ~/bin</pre><pre>curl -s -L -o ~/bin/cfssl <a href="https://pkg.cfssl.org/R1.2/cfssl_linux-amd64">https://pkg.cfssl.org/R1.2/cfssl_linux-amd64</a></pre><pre>curl -s -L -o ~/bin/cfssljson <a href="https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64">https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64</a></pre><pre>chmod +x ~/bin/{cfssl,cfssljson}</pre><pre>export PATH=$PATH:~/bin</pre><p><strong>Create workspace in local machine</strong></p><pre>mkdir -p ~/desktop/certs/{admin,controller_manager,master1,mastern,node1,noden,ca,api,scheduler,front-proxy}</pre><p><strong>Copy ca cert and ca key from k8s cluster to the local workspace</strong></p><pre>scp ~/desktop/certs user@instancename:/etc/kubernetes/ssl/ca.*</pre><p><strong>Create Certificate Authority</strong> <strong>config file</strong></p><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/ca/ca-config.json<br>{<br>    &quot;signing&quot;: {<br>        &quot;default&quot;: {<br>            &quot;expiry&quot;: &quot;87600h&quot;<br>        },<br>        &quot;profiles&quot;: {<br>            &quot;server&quot;: {<br>                &quot;expiry&quot;: &quot;8760h&quot;,<br>                &quot;usages&quot;: [<br>                    &quot;signing&quot;,<br>                    &quot;key encipherment&quot;,<br>                    &quot;server auth&quot;<br>                ]<br>            },<br>            &quot;client&quot;: {<br>                &quot;expiry&quot;: &quot;8760h&quot;,<br>                &quot;usages&quot;: [<br>                    &quot;signing&quot;,<br>                    &quot;key encipherment&quot;,<br>                    &quot;client auth&quot;<br>                ]<br>            },<br>            &quot;peer&quot;: {<br>                &quot;expiry&quot;: &quot;8760h&quot;,<br>                &quot;usages&quot;: [<br>                    &quot;signing&quot;,<br>                    &quot;key encipherment&quot;,<br>                    &quot;server auth&quot;,<br>                    &quot;client auth&quot;<br>                ]<br>            }<br>        }<br>    }<br>}<br>EOF</pre><h3><strong>Generate Node Kubelet Server and Client Certs</strong></h3><p><strong>Kubelet Server Cert creation</strong>:</p><ul><li>Get the Output of the below command from the Node. Make a note of the host name and IP. This output is generated from the old cert.</li></ul><pre>openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep &quot;Subject Alternative Name&quot; -A 1</pre><ul><li>Create server.json to create kubelet server certificates for the nodes</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/node1/server.json<br>{<br>    &quot;CN&quot;: &quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>    &quot;hosts&quot;: [<br>        &quot;&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>        &quot;&lt;HOST_IPADDRESS&gt;&quot;,<br>        &quot;&lt;DOCKER/CONTAINERD_NETWORK_IP&gt;&quot;, ## example:172.17.0.1<br>        &quot;&lt;WEAVE_IP&gt;&quot;<br>    ],<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    },<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:nodes&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run the below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=server ~/desktop/certs/node1/server.json | cfssljson -bare ~/desktop/certs/node1/kubelet-server</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep &quot;Subject Alternative Name&quot; -A 1 &amp;&amp; openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep &quot;Not After&quot;</pre><p><strong>Kubelet Client Cert creation:</strong></p><ul><li>Get the Output of the below command from the Node. Make a note of the host name and IP. This output is generated from the old cert.</li></ul><pre>openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -noout -text | grep -e &quot;Subject:&quot;</pre><ul><li>Create client.json to create kubelet client certificates for the nodes</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/node1/server.json<br>{<br>    &quot;CN&quot;: &quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    },<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:nodes&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run the below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=server ~/desktop/certs/node1/client.json | cfssljson -bare ~/desktop/certs/node1/kubelet-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/node1/kubelet-server.pem -noout -text | grep &quot;Not After&quot;</pre><h3>Generate Master Kubelet Server and Client Certs</h3><p><strong>Kubelet Server Cert creation</strong>:</p><ul><li>Get the Output of the below command from the Master. Make a note of the host name and IP. This output is generated from the old cert.</li></ul><pre>openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-server.crt -noout -text | grep &quot;Subject Alternative Name&quot; -A 1</pre><ul><li>Create server.json to create kubelet server certificates.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/master1/server.json<br>{<br>    &quot;CN&quot;: &quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>    &quot;hosts&quot;: [<br>        &quot;&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>        &quot;&lt;HOST_IPADDRESS&gt;&quot;,<br>        &quot;&lt;DOCKER/CONTAINERD_NETWORK_IP&gt;&quot;, ## example:172.17.0.1<br>        &quot;&lt;WEAVE_IP&gt;&quot;<br>    ],<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    },<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:masters&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run the below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=server ~/desktop/certs/master1/server.json | cfssljson -bare ~/desktop/certs/master1/kubelet-server</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep &quot;Subject Alternative Name&quot; -A 1 &amp;&amp; openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep &quot;Not After&quot;</pre><p><strong>Kubelet Client Cert creation:</strong></p><ul><li>Get the Output of the below command from the Master. Make a note of the host name. This output is generated from the old cert.</li></ul><pre>openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in /var/lib/kubelet/pki/kubelet-client.crt -noout -text | grep -e &quot;Subject:&quot;</pre><ul><li>Create client.json to create kubelet client certificates.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/master1/server.json<br>{<br>    &quot;CN&quot;: &quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    },<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:masters&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run the below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=server ~/desktop/certs/node1/client.json | cfssljson -bare ~/desktop/certs/node1/kubelet-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/master1/kubelet-server.pem -noout -text | grep &quot;Not After&quot;</pre><h3><strong>Kube Controller Manager Client Cert Creation:</strong></h3><ul><li>Create the below controller-manager cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/controller_manager/client.json<br>{<br>    &quot;CN&quot;: &quot;system:kube-controller-manager&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>ccfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/controller_manager/client.json | cfssljson -bare ~/desktop/certs/controller_manager/cm-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/controller_manager/cm-client.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/controller_manager/cm-client.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/controller_manager/cm-client.pem -noout -text | grep grep &quot;Not After&quot;</pre><h3>Kube Scheduler Client Cert Creation:</h3><ul><li>Create the below controller-manager cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/scheduler/client.json<br>{<br>    &quot;CN&quot;: &quot;system:kube-scheduler&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/scheduler/client.json | cfssljson -bare ~/desktop/certs/scheduler/cm-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/scheduler/cm-client.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/scheduler/cm-client.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/scheduler/cm-client.pem -noout -text | grep grep &quot;Not After&quot;</pre><h3>API Server Client Cert Creation:</h3><ul><li>Create the below apiserver client cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/api/client.json<br>{<br>    &quot;CN&quot;: &quot;system:kube-apiserver-client&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:masters&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/api/client.json | cfssljson -bare ~/desktop/certs/api/apiserver-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/api/apiserver-client.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/api/apiserver-client.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/api/apiserver-client.pem -noout -text | grep grep &quot;Not After&quot;</pre><h3>API Server Server Cert Creation:</h3><ul><li>Create the below apiserver server cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/api/server.json<br>{<br>    &quot;CN&quot;: &quot;kube-apiserver&quot;,<br>    &quot;hosts&quot;: [<br>        &quot;&lt;ENTER MASTER-1 HOST NAME fqdn&gt;&quot;,<br>        &quot;&lt;MASTER-1 HOST_IPADDRESS&gt;&quot;,<br>        &quot;&lt;ENTER MASTER-N HOST NAME fqdn&gt;&quot;,<br>        &quot;&lt;MASTER-N HOST_IPADDRESS&gt;&quot;,<br>        &quot;&lt;KUBE_NETWORK_IP&gt;&quot;, ## example: 10.233.0.1<br>        &quot;kubernetes&quot;,<br>        &quot;kubernetes.default&quot;,<br>        &quot;kubernetes.default.svc&quot;,<br>        &quot;kubernetes.default.svc.cluster.local&quot;,<br>        &quot;localhost&quot;,<br>        &quot;127.0.0.1&quot;<br>    ],<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:masters&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/api/server.json | cfssljson -bare ~/desktop/certs/api/apiserver</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/api/apiserver.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/api/apiserver.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/api/apiserver.pem -noout -text | grep &quot;Subject Alternative Name&quot; -A 1 &amp;&amp; openssl x509 -in ~/desktop/certs/api/apiserver.pem -noout -text | grep grep &quot;Not After&quot;</pre><h3>Front Proxy Client Cert Creation:</h3><ul><li>Create the below front proxy client cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/front-proxy/client.json<br>{<br>    &quot;CN&quot;: front-proxy-client,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/front-proxy/client.json | cfssljson -bare ~/desktop/certs/front-proxy/front-proxy-client</pre><ul><li>Verification</li></ul><pre>openssl x509 -in ~/desktop/certs/front-proxy/front-proxy-client.pem -noout -text | grep -e &quot;Issuer&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/front-proxy/front-proxy-client.pem -noout -text | grep -e &quot;Subject:&quot; &amp;&amp; openssl x509 -in ~/desktop/certs/front-proxy/front-proxy-client.pem -noout -text | grep grep &quot;Not After&quot;</pre><h3>Administrator Cert Creation:</h3><ul><li>Create the below admin cert config.</li></ul><pre>cat &lt;&lt; EOF &gt; ~/desktop/certs/admin/client.json<br>{<br>    &quot;CN&quot;: &quot;kubernetes-admin&quot;,<br>    &quot;key&quot;: {<br>        &quot;algo&quot;: &quot;rsa&quot;,<br>        &quot;size&quot;: 2048<br>    }<br>    &quot;names&quot;: [<br>        {<br>            &quot;O&quot;: &quot;system:masters&quot;<br>        }<br>    ]<br>}<br>EOF</pre><ul><li>Run below command to generate server certs.</li></ul><pre>cfssl gencert -ca=~/desktop/certs/ca/ca.crt -ca-key=~/desktop/certs/ca/ca.key -config=~/desktop/certs/ca/ca-config.json -profile=client ~/desktop/certs/admin/client.json | cfssljson -bare ~/desktop/certs/admin/admin</pre><blockquote>Once the certificates are created move the certificates to the specific masters and nodes.</blockquote><h3><strong>Update Node Kubelet Certs:</strong></h3><ul><li>Copy the certs(kubelet-client.pem, kubelet-client-key.pem, kubelet-server.pem, kubelet-server-key.pem) in to Node 1</li></ul><pre>scp ~/desktop/certs/node1/*.pem root@node1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes and take a backup of the existing certs.</li></ul><pre>systemctl stop kubelet &amp;&amp; cd /var/lib/kubelet/pki &amp;&amp; mkdir -p /tmp/wrk-cfg &amp;&amp; mv * /tmp/wrk-cfg</pre><ul><li>Copy new certs in to /var/lib/kubelet/pki</li></ul><pre>cp *.pem /var/lib/kubelet/pki/</pre><ul><li>Merge kubelet-server.pem and kubelet-server-key.pem in to kubelet-server-&lt;date&gt;.pem</li></ul><pre>cat kubelet-server.pem &gt; kubelet-server-2019-09-16.pem &amp;&amp; cat kubelet-server-key.pem &gt;&gt; kubelet-server-2019-09-16.pem</pre><ul><li>link kublet-server-&lt;date&gt;.pem with kubelet-server-current.pem</li></ul><pre>ln -s kubelet-server-2019-09-16.pem kubelet-server-current.pem</pre><ul><li>Verify if kubelet.conf is pointed to the new client certificates. Verify the client-certificate and client-key</li></ul><pre>cat /etc/kubernetes/kubelet.conf</pre><blockquote><em>How to embed the client certificate in kubelet.conf or How to point the kubelet.conf to the client certificate.</em></blockquote><p>Embed certificates:</p><pre>export KUBE_APISERVER=https://&lt;IP_ADDRESS_API_SERVER&gt;:6443</pre><pre>cd /etc/kubernetes &amp;&amp; kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.crt --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kubelet.conf &amp;&amp; kubectl config set-credentials --client-certificate=/var/lib/kubelet/pki/kubelet-client.pem --client-key=/var/lib/kubelet/pki/kubelet-client-key.pem --embed-certs=true --kubeconfig=kubelet.conf &amp;&amp; kubectl config set-context kubernetes --cluster=kubernetes --user=&quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot; --kubeconfig=kubelet.conf &amp;&amp; kubectl config use-context kubernetes --kubeconfig=kubelet.conf &amp;&amp; cd ~</pre><p>Point kubelet.conf to the new certificates: Edit the kubelet.conf client-certificate and client-key point them to the right kublet-client certificates.</p><ul><li>Append tls-cert and tls-keyto KUBELET_ARGS in kubelet.env as shown below</li></ul><pre>vi /etc/kubernetes/kubelet.env</pre><pre>--tls-cert-file=/var/lib/kubelet/pki/kubelet-server.pem --tls-private-key-file=/var/lib/kubelet/pki/kubelet-server-key.pem</pre><ul><li>Now Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><blockquote>Repeat the same procedure for the other nodes.</blockquote><h3><strong>Update Master Components Certificates:</strong></h3><p><strong>Master kubelet certs.</strong></p><ul><li>Copy the certs(kubelet-client.pem, kubelet-client-key.pem, kubelet-server.pem, kubelet-server-key.pem) in to Node 1</li></ul><pre>scp ~/desktop/certs/m1/*.pem root@node1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes and take a backup of the existing certs.</li></ul><pre>systemctl stop kubelet &amp;&amp; cd /var/lib/kubelet/pki &amp;&amp; mkdir -p /tmp/wrk-cfg &amp;&amp; mv * /tmp/wrk-cfg</pre><ul><li>Copy new certs in to /var/lib/kubelet/pki</li></ul><pre>cp *.pem /var/lib/kubelet/pki/</pre><ul><li>Merge kubelet-server.pem and kubelet-server-key.pem in to kubelet-server-&lt;date&gt;.pem</li></ul><pre>cat kubelet-server.pem &gt; kubelet-server-2019-09-16.pem &amp;&amp; cat kubelet-server-key.pem &gt;&gt; kubelet-server-2019-09-16.pem</pre><ul><li>link kublet-server-&lt;date&gt;.pem with kubelet-server-current.pem</li></ul><pre>ln -s kubelet-server-2019-09-16.pem kubelet-server-current.pem</pre><ul><li>Verify if kubelet.conf is pointed to the new client certificates. Verify the client-certificate and client-key</li></ul><pre>cat /etc/kubernetes/kubelet.conf</pre><blockquote><em>How to embed the client certificate in kubelet.conf or How to point the kubelet.conf to the client certificate.</em></blockquote><p>Embed certificates:</p><pre>export KUBE_APISERVER=https://&lt;IP_ADDRESS_API_SERVER&gt;:6443</pre><pre>cd /etc/kubernetes &amp;&amp; kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.crt --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kubelet.conf &amp;&amp; kubectl config set-credentials --client-certificate=/var/lib/kubelet/pki/kubelet-client.pem --client-key=/var/lib/kubelet/pki/kubelet-client-key.pem --embed-certs=true --kubeconfig=kubelet.conf &amp;&amp; kubectl config set-context kubernetes --cluster=kubernetes --user=&quot;system:node:&lt;ENTER HOST NAME fqdn&gt;&quot; --kubeconfig=kubelet.conf &amp;&amp; kubectl config use-context kubernetes --kubeconfig=kubelet.conf &amp;&amp; cd ~</pre><p>Point kubelet.conf to the new certificates: Edit the kubelet.conf client-certificate and client-key point them to the right kublet-client certificates.</p><ul><li>Append tls-cert and tls-keyto KUBELET_ARGS in kubelet.env as shown below</li></ul><pre>vi /etc/kubernetes/kubelet.env</pre><pre>--tls-cert-file=/var/lib/kubelet/pki/kubelet-server.pem --tls-private-key-file=/var/lib/kubelet/pki/kubelet-server-key.pem</pre><ul><li>Now Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><p><strong>Controller-Manager</strong></p><ul><li>Copy the certs(cm-client.pem, cm-client-key.pem) in to Master 1</li></ul><pre>scp ~/desktop/certs/controller_manager/*.pem root@master1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes and take a backup of the entire /etc/kubernetes directory.</li></ul><pre>systemctl stop kubelet &amp;&amp; cp -R /etc/kubernetes/* /tmp/wrk-cfg</pre><ul><li>Remove controller_manager old certs in /etc/kubernetes/ssl if present.</li><li>Copy new certs in to /etc/kubernetes/ssl</li></ul><pre>cp cm-client* /etc/kubernetes/ssl</pre><ul><li>Export apiserver ip you can find it in the /etc/kubernetes/controller-manager.conf</li></ul><pre>export KUBE_APISERVER=https://&lt;API_SERVER_IP_ADDRESS&gt;:6443</pre><ul><li>Run the below command.</li></ul><pre>cd /etc/kubernetes &amp;&amp; kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.crt --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=controller-manager.conf &amp;&amp; kubectl config set-credentials system:kube-controller-manager --client-certificate=/etc/kubernetes/ssl/cm-client.pem --client-key=/etc/kubernetes/ssl/cm-client-key.pem --embed-certs=true --kubeconfig=controller-manager.conf &amp;&amp; kubectl config set-context kubernetes --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=controller-manager.conf &amp;&amp; kubectl config use-context kubernetes --kubeconfig=controller-manager.conf &amp;&amp; cd ~</pre><ul><li>Restart Controller Manager pods</li></ul><pre>kubectl delete po &lt;controller_manager_pod_name&gt; -n kube-system</pre><ul><li>Now Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><p><strong>Scheduler</strong></p><ul><li>Copy the certs(scheduler.pem, scheduler-client-key.pem) in to Master 1</li></ul><pre>scp ~/desktop/certs/scheduler/*.pem root@master1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes.</li></ul><pre>systemctl stop kubelet</pre><ul><li>Remove schedulerold certs in /etc/kubernetes/ssl if present.</li><li>Copy new certs in to /etc/kubernetes/ssl</li></ul><pre>cp scheduler* /etc/kubernetes/ssl</pre><ul><li>Export apiserver ip you can find it in the /etc/kubernetes/scheduler.conf</li></ul><pre>export KUBE_APISERVER=https://&lt;API_SERVER_IP_ADDRESS&gt;:6443</pre><ul><li>Run the below command.</li></ul><pre>cd /etc/kubernetes &amp;&amp; kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.crt --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=scheduler.conf &amp;&amp; kubectl config set-credentials system:kube-scheduler --client-certificate=/etc/kubernetes/ssl/scheduler-client.pem --client-key=/etc/kubernetes/ssl/scheduler-client-key.pem --embed-certs=true --kubeconfig=scheduler.conf &amp;&amp; kubectl config set-context kubernetes --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=scheduler.conf &amp;&amp; kubectl config use-context kubernetes --kubeconfig=scheduler.conf &amp;&amp; cd ~</pre><ul><li>Restart Controller Manager pods</li></ul><pre>kubectl delete po &lt;scheduler_pod_name&gt; -n kube-system</pre><ul><li>Now Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><p><strong>APISERVER</strong> <strong>and Front-Proxy</strong></p><ul><li>Copy the certs (apiserver-key.pem, apiserver-client-key.pem, apiserver-client.pem, apiserver.pem, front-proxy-client-key.pem, front-proxy-client.pem) in to Master 1</li></ul><pre>scp ~/desktop/certs/apiserver/*.pem root@master1.example.com:~/<br>scp ~/desktop/certs/front-proxy/*.pem root@master1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes.</li></ul><pre>systemctl stop kubelet</pre><ul><li>Move apiservermanifest.</li></ul><pre>mv /etc/kubernetes/manifests/kube-apiserver.yaml ~/</pre><ul><li>Remove apiserver and front-proxy-clientold client and server certs in /etc/kubernetes/ssl if present.</li></ul><p><strong><em>NOTE</em></strong><em>: Donot delete front-proxy.crt</em></p><ul><li>Copy new certs in to /etc/kubernetes/ssl</li></ul><pre>cp apiserver* /etc/kubernetes/ssl &amp;&amp; cp front-proxy-client-* /etc/kubernetes/ssl</pre><ul><li>Verify if kube-apiserver.yaml is pointed to the new certs api and front-proxy certs in the directory/etc/kubernetes/ssl</li><li>Move apiservermanifest back to /etc/kubernetes/manifest directory.</li></ul><pre>mv ~/kube-apiserver.yaml /etc/kubernetes/manifests/</pre><ul><li>Now Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><blockquote>Check if all the pods are running as expected by using the below command.</blockquote><pre>kubectl get po -n kube-system -w</pre><blockquote>Check the cluster and the nodes status</blockquote><pre>kubectl get cs<br>NAME                 STATUS    MESSAGE              ERROR<br>scheduler            Healthy   ok<br>controller-manager   Healthy   ok<br>etcd-0               Healthy   {&quot;health&quot;: &quot;true&quot;}</pre><pre>kubectl get nodes</pre><pre>NAME                  STATUS   ROLES    AGE   VERSION<br>master1.example.com   Ready    master   1y   v1.10.11<br>master2.example.com   Ready    master   1y   v1.10.11<br>masterN.example.com   Ready    master   1y   v1.10.11<br>node1.example.com     Ready    node     1y   v1.10.11<br>node2.example.com     Ready    node     1y   v1.10.11<br>nodeN.example.com     Ready    node     1y   v1.10.11</pre><p><strong>Administrator</strong></p><ul><li>Copy the certs(admin.pem, admin-key.pem) in to Master 1</li></ul><pre>scp ~/desktop/certs/admin/*.pem root@master1.example.com:~/</pre><ul><li>Stop the kubelet service before performing any changes.</li></ul><pre>systemctl stop kubelet</pre><ul><li>Export apiserver ip you can find it in the /etc/kubernetes/scheduler.conf</li></ul><pre>export KUBE_APISERVER=https://&lt;API_SERVER_IP_ADDRESS&gt;:6443</pre><ul><li>Run the below command.</li></ul><pre>cd /etc/kubernetes &amp;&amp; kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.crt --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=admin.conf &amp;&amp; kubectl config set-credentials kubernetes-admin --client-certificate=~/admin.pem --client-key=~/admin-key.pem --embed-certs=true --kubeconfig=admin.conf &amp;&amp; kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=admin.conf &amp;&amp; kubectl config use-context kubernetes --kubeconfig=admin.conf &amp;&amp; cd ~</pre><ul><li>Backup and Update the .kube/config file.</li></ul><pre>mv ~/.kube/config /tmp/wrk-cfg/ &amp;&amp; cp admin.conf ~/.kube/config &amp;&amp; chmod 700 ~/.kube/config</pre><ul><li>Start the kubelet and check the status if it is up and running.</li></ul><pre>systemctl start kubelet &amp;&amp; systemctl status kubelet</pre><blockquote>Repeat the same procedure for other master servers</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0d0433b4eed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>