<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Terpase Theophilus Gagher on Medium]]></title>
        <description><![CDATA[Stories by Terpase Theophilus Gagher on Medium]]></description>
        <link>https://medium.com/@phil16terpasetheo?source=rss-d069ca822be1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*uR84DG469ow0RgfK</url>
            <title>Stories by Terpase Theophilus Gagher on Medium</title>
            <link>https://medium.com/@phil16terpasetheo?source=rss-d069ca822be1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:05:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@phil16terpasetheo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Install Velero On EKS Cluster and Backup Cluster to S3]]></title>
            <link>https://medium.com/@phil16terpasetheo/install-velero-on-eks-cluster-and-backup-cluster-to-s3-6fdb069cda5f?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fdb069cda5f</guid>
            <category><![CDATA[velero]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 21:34:06 GMT</pubDate>
            <atom:updated>2025-01-07T12:00:15.727Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*-agm3uFiMdfm2Db8.png" /></figure><h4>Introduction</h4><p>Project Repo: <a href="https://github.com/philcz16/velero-demo">https://github.com/philcz16/velero-demo</a></p><p>Let’s look at a scenario where a firm has its Kubernetes environment fully set up let’s say this firm, follows best practices including GitOps, and loves to use cloud-native technologies so in this case, they have all application stack, monitoring, and observability stack, service meshes, CICD, even down to databases running on Kubernetes.</p><p>The firm’s cluster and applications are all healthy and nothing breaks until something happens where the entire cloud region goes down and they experience data loss. Another occurrence could occur when the cluster administrator decides to upgrade the cluster to a newer version and the process isn’t followed correctly which could cause data loss.</p><h4>Velero</h4><p>Velero is an open-source tool used to backup, restore, perform disaster recovery, and migrate Kubernetes clusters, resources, and persistent volumes.</p><p>Velero solves the problem of having to backup and restore the K8s cluster, so if you experience data loss on your Kubernetes cluster, you can easily restore all your resources. In the scenario described above, Velero helps to safely upgrade a cluster because all you’ll need to do is back up the existing cluster, create a new cluster with the latest version, and restore all your backed-up data into that cluster.</p><p>In this article, we will create a Kubernetes cluster using AWS EKS and set up Velero on the cluster. We will also proceed to configure Velero to store cluster backups into Amazon S3 and demonstrate how to back up and restore Kubernetes cluster objects.</p><h4>Prerequisites</h4><p>The article sets up the EKS cluster with VPC networking resources and uses Terraform helm provider to install Velero. We will also create an IAM user that gives permission to write to S3 and pass the access keys into the helm chart as Kubernetes secrets all through Terraform.</p><p>To follow along, you’ll need the following installed:</p><ul><li>AWS CLI</li><li>Terraform</li><li>Ensure that the AWS user you’re using has access to create other users</li><li>Velero CLI</li><li>Kubectl installed</li></ul><h4>Create VPC and Subnets</h4><p>We will begin by creating the VPC and subnets needed to provision the EKS cluster. We will be using the VPC module provided by AWS. Begin by creating a main.tf file</p><pre>provider &quot;aws&quot; {<br>  region = var.region<br>}<br></pre><pre>provider &quot;helm&quot; {<br>  kubernetes {<br>    host                   = module.eks.cluster_endpoint<br>    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)</pre><pre>    exec {<br>      api_version = &quot;client.authentication.k8s.io/v1beta1&quot;<br>      command     = &quot;aws&quot;<br>      # This requires the awscli to be installed locally where Terraform is executed<br>      args = [&quot;eks&quot;, &quot;get-token&quot;, &quot;--cluster-name&quot;, module.eks.cluster_name]<br>    }<br>  }<br>}</pre><pre>data &quot;aws_availability_zones&quot; &quot;available&quot; {<br>}</pre><pre># create VPC</pre><pre>module &quot;vpc&quot; {<br>  source = &quot;terraform-aws-modules/vpc/aws&quot;</pre><pre>  name = &quot;${var.project_name}-vpc&quot;<br>  cidr = &quot;10.0.0.0/16&quot;</pre><pre>  azs             = slice(data.aws_availability_zones.available.names, 0, 3)<br>  private_subnets = [&quot;10.0.1.0/24&quot;, &quot;10.0.2.0/24&quot;, &quot;10.0.3.0/24&quot;]<br>  public_subnets  = [&quot;10.0.101.0/24&quot;, &quot;10.0.102.0/24&quot;, &quot;10.0.103.0/24&quot;]</pre><pre>  enable_nat_gateway = true<br>  enable_vpn_gateway = false<br>  single_nat_gateway = true</pre><pre>  tags = {<br>    Project = &quot;${var.project_name}&quot;<br>  }<br>}</pre><p>We begin by setting up the AWS and helm provider used to install the helm chart you’ll notice that we use two variables for region and project_name. Proceed to create a var.tf file to define our variables.</p><pre>variable &quot;project_name&quot; {<br>  type = string<br>}</pre><pre>variable &quot;region&quot; {<br>  type = string<br>}</pre><p>You can assign values to these variables we declared. To do this, create a terraform.tfvars file and enter the content below</p><pre>project_name = &quot;velero&quot;</pre><pre>region = &quot;us-east-1&quot;</pre><h4>Create the EKS Cluster</h4><p>Create the EKS cluster using the AWS EKS module provided by Terraform. To do this, create a file named eks.tf and then paste in the content below</p><pre>################################################################################<br># EKS Module<br>################################################################################</pre><pre>data &quot;aws_caller_identity&quot; &quot;current&quot; {}</pre><pre>module &quot;eks&quot; {<br>  source = &quot;terraform-aws-modules/eks/aws&quot;</pre><pre>  cluster_name    = var.project_name<br>  cluster_version = &quot;1.30&quot;</pre><pre>  # Gives Terraform identity admin access to cluster which will<br>  # allow deploying resources (Karpenter) into the cluster<br>  enable_cluster_creator_admin_permissions = true<br>  cluster_endpoint_public_access           = true</pre><pre>  cluster_addons = {<br>    coredns                = {}<br>    eks-pod-identity-agent = {}<br>    kube-proxy             = {}<br>    vpc-cni                = {}<br>  }</pre><pre>  vpc_id                   = module.vpc.vpc_id<br>  subnet_ids               = module.vpc.private_subnets<br>  control_plane_subnet_ids = module.vpc.intra_subnets</pre><pre>  eks_managed_node_groups = {<br>    istio-ng = {<br>      ami_type       = &quot;AL2023_x86_64_STANDARD&quot;<br>      instance_types = [&quot;t3.medium&quot;]</pre><pre>      min_size     = 2<br>      max_size     = 5<br>      desired_size = 2<br></pre><pre>    }<br>  }</pre><pre>  access_entries = {<br>    # One access entry with a policy associated<br>    user-access = {<br>      kubernetes_groups = []<br>      principal_arn     = &quot;arn:aws:iam::${data.aws_caller_identity.current.account_id}:root&quot;</pre><pre>      policy_associations = {<br>        admin_user = {<br>          policy_arn = &quot;arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy&quot;<br>          access_scope = {<br>            type = &quot;cluster&quot;<br>          }<br>        }<br>      }    }<br>  }</pre><pre>}</pre><p>With this, we have successfully set up the EKS cluster we can then create the resources needed to get Velero to work on AWS EKS.</p><h4>Create Velero Credentials</h4><p>For Velero to have access to our AWS account to store backups inside an s3 bucket, we need to create an IAM user for Velero and grant the user the least privileges needed. These privileges are strictly only access to EC2 snapshots and S3 bucket objects.</p><p>Create a file, name it velero.tf paste the content below while we take a look at what it creates</p><pre>resource &quot;random_string&quot; &quot;random&quot; {<br>  length           = 10<br>  special          = false<br>  upper            = false <br>}</pre><pre>module &quot;s3_bucket&quot; {<br>  source = &quot;terraform-aws-modules/s3-bucket/aws&quot;</pre><pre>  bucket = &quot;valero-bucket-${random_string.random.result}&quot;<br>  acl    = &quot;private&quot;</pre><pre>  control_object_ownership = true<br>  object_ownership         = &quot;ObjectWriter&quot;</pre><pre>  versioning = {<br>    enabled = true<br>  }<br>}</pre><pre>resource &quot;aws_iam_user&quot; &quot;valero&quot; {<br>  name = &quot;valero-user&quot;</pre><pre>  tags = {<br>    Project = &quot;Valero-project&quot;<br>  }<br>}</pre><pre>resource &quot;aws_iam_user_policy&quot; &quot;valero&quot; {<br>  name   = &quot;valero-policy&quot;<br>  user   = aws_iam_user.valero.name<br>  policy = &lt;&lt;POLICY<br>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:DescribeVolumes&quot;,<br>                &quot;ec2:DescribeSnapshots&quot;,<br>                &quot;ec2:CreateTags&quot;,<br>                &quot;ec2:CreateVolume&quot;,<br>                &quot;ec2:CreateSnapshot&quot;,<br>                &quot;ec2:DeleteSnapshot&quot;<br>            ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:PutObject&quot;,<br>                &quot;s3:AbortMultipartUpload&quot;,<br>                &quot;s3:ListMultipartUploadParts&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::${module.s3_bucket.s3_bucket_id}/*&quot;<br>            ]<br>        },<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;s3:ListBucket&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::${module.s3_bucket.s3_bucket_id}&quot;<br>            ]<br>        }<br>    ]<br>}<br>POLICY <br>}</pre><pre>resource &quot;aws_iam_access_key&quot; &quot;valero&quot; {<br>  user    = aws_iam_user.valero.name<br>}</pre><p>It begins by creating a random string, uses the random string as the s3 bucket name, creates the IAM user for Velero, adds the IAM policy we talked about, and finally, generates access keys for the IAM user.</p><h4>Install Velero</h4><p>Finally, create a helm release using Terraform that will be installed on the Kubernetes cluster. This release will install Velero on the cluster, including a values file that contains our S3 bucket name, and pass in the access keys credentials into the values file for the Velero helm chart to create it as a secret.</p><p>Create a file named velero-helm.tf and paste the content below</p><pre>resource &quot;helm_release&quot; &quot;velero&quot; {<br>  name       = &quot;velero&quot;<br>  repository = &quot;https://vmware-tanzu.github.io/helm-charts/&quot;<br>  chart      = &quot;velero&quot;<br>  version    = &quot;7.1.5&quot;<br>  namespace  = &quot;velero&quot;<br>  create_namespace =  true<br>  depends_on = [ module.eks ]</pre><pre>  values = [<br>    &quot;${templatefile(&quot;values.yaml&quot;, { access_key = &quot;${aws_iam_access_key.valero.id}&quot;, secret_access_key = &quot;${aws_iam_access_key.valero.secret}&quot;, bucket_name = &quot;${module.s3_bucket.s3_bucket_id}&quot; })}&quot;<br>  ]</pre><pre>}</pre><p>This creates a helm release and then points in a values file. The values file is referenced as a template file which allows us to put dynamic values into the file before we pass it into the helm chart.</p><p>So we will need to create the values file. In your project root directory, create a values.yaml file and paste the content below:</p><pre>initContainers:<br>  - name: velero-plugin-for-aws<br>    image: velero/velero-plugin-for-aws:v1.4.1<br>    imagePullPolicy: IfNotPresent<br>    volumeMounts:<br>      - mountPath: /target<br>        name: plugins<br>credentials:<br>  useSecret: true<br>  secretContents:<br>    cloud: |<br>      [default]<br>      aws_access_key_id= ${access_key}<br>      aws_secret_access_key= ${secret_access_key}</pre><pre>configuration:<br>  backupStorageLocation:<br>  - bucket: ${bucket_name}<br>    provider: aws<br>  volumeSnapshotLocation:<br>  - config:<br>      region: us-east-1<br>    provider: aws</pre><p>The values file specifies an init container for the AWS environment. It creates access key secret values and configures the bucket name.</p><p>If you followed along, you should have a project repo similar to the screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*UGosdZYQzDWiAlbNRPqsIA.png" /></figure><h4>Set Up Infrastructure</h4><p>To provision the infrastructure on AWS, proceed to execute the commands below in your project directory terminal</p><pre>terraform init<br>terraform validate<br>terraform apply --auto-approve</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*XyB8QY-u-AZo_o6qj3YDHw.png" /></figure><p>Once terraform completes, connect to the cluster using the command below:</p><pre>aws eks update-kubeconfig --region us-east-1 --name velero</pre><p>Confirm that Velero is running on the cluster using the command below</p><pre>kubectl get pods -n velero</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*LKwMK13tadQlD-LdymGv8g.png" /></figure><p>From here, check the logs to confirm that it is running without errors.</p><h4>Demonstrate Velero BackUps</h4><p>To demonstrate Velero backups, we will create a simple nginx deployment in the default namespace. We will backup the default namespace to an S3 bucket and restore the backup.</p><p>Let’s create a sample nginx deployment. You can kubectl apply the manifest below:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx-deployment<br>  labels:<br>    app: nginx<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: nginx:latest<br>        ports:<br>        - containerPort: 80</pre><p>This will create 3 nginx pods</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*9RLog8twVHc0BoNZTpbZdA.png" /></figure><blockquote>NB Backups with velero can be automated using cron syntax that can be added in velero helm chart values file.</blockquote><p>Now let’s proceed to perform a manual backup using Velero CLI.</p><pre>velero backup create demo</pre><p>Describe the backup</p><pre>velero backup describe demo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*nE76dMF8cOZ9zPA0BHpDqw.png" /></figure><p>you can also list backups that were created using the command below:</p><pre>velero get backups</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*Vj-wcTW5Gygo8oczzmhhIg.png" /></figure><p>Let’s proceed to S3 to confirm that the backup is created successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95eGkuqk-tXSR1EQWqYYXA.png" /></figure><p>To demonstrate the restore process, we will delete the Nginx app currently running and then recreate the deployment back from the backup that was created.</p><pre>kubectl delete -f .\manifest.yaml</pre><p>Use the command below to restore the previous backup that we created</p><pre>velero restore create --from-backup demo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*kxCl_rdSLc8fN_ZuEb3Irg.png" /></figure><p>We can then check if the deployment is restored from the backup created. You can see that the deployment has been restored from the backup we created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*H4f3TLkMnytYTqjJRPiMGA.png" /></figure><h4>Clean Ups</h4><p>To delete all AWS resources created, navigate to your project directory and execute the command below</p><pre>terraform destroy --auto-approve</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fdb069cda5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure SSO authentication to ArgoCD and Harbor with Keycloak on EKS Cluster]]></title>
            <link>https://blog.devops.dev/configure-sso-authentication-to-argocd-and-harbor-with-keycloak-on-eks-cluster-1791b1aa27b0?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1791b1aa27b0</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[sso-single-sign-on-portal]]></category>
            <category><![CDATA[harbor]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Sun, 04 Aug 2024 12:11:46 GMT</pubDate>
            <atom:updated>2025-01-07T12:03:54.518Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*MsQ2T2yOh2C3eN-LnnlYiw.png" /></figure><p>Project repo: <a href="https://github.com/philcz16/keycloak">https://github.com/philcz16/keycloak</a></p><h4>Introduction</h4><p>Single Sign On (SSO) authentication is a more secure and easy method for users to sign into different applications. This method eliminates the need for users to keep passwords and also have to create passwords each time they want to sign into an application. A perfect example of this is when you try to sign into a website and you see an option like “Sign in with Google”. This method will authenticate the user into the application without revealing the user’s credentials to the authentication provider.</p><p>This means that when you sign in to an application using SSO, let’s say you used to sign in with Google, the process happens in a way that your Google password isn’t revealed in the authentication process.</p><p>In this article, we will set up our Authentication provider on a Kubernetes cluster using a great open-source tool called Keycloak. And we will implement SSO login to ArgoCD and Harbor registry using Keycloak.</p><h3>Keycloak</h3><p>What exactly is keycloak? Keycloak, is an opensource identity and access management tool. This tool, allows you to manage authentication to applications with minimum effort. So you won’t need to write code to your backend to implement SSO logins or you wont have any need to write your own identity management tool. All you need to do is to integrate the application with Keycloak and Keycloak does all the hard work.</p><p>It supports many features such as Single Sign On (SSO) which is what we will implement, Identity and Social Login which is login with Google, Facebook, or GitHub. It also provides many features which will be out of scope of this article if we are to talk about it. Check out their page: <a href="https://www.keycloak.org/">https://www.keycloak.org/</a></p><h4>Prerequisites</h4><p>To follow along with the article, you’ll need to have the requirements listed below:</p><ul><li>Kubernetes Cluster (The article uses EKS)</li><li>Helm Installed</li><li>If you don’t have a cluster, you’ll need Terraform installed</li><li>A domain in a domain provider (This article uses cPanel)</li><li>AWS CLI Installed</li></ul><h3>Provisioning the Cluster</h3><p>To begin with, we will first provision a cluster on AWS using their managed service for Kubernetes EKS. The project’s repository provides has a folder named /kubernetes-cluster . The folder contains terraform configuration files to provision a cluster and Networking on AWS. To spin up a the cluster on AWS, navigate to the repo folder in a terminal and execute the commands below:</p><pre>cd kubernetes-cluster<br>terraform init --upgrade<br>terraform apply --auto-approve</pre><p>It should take around 15–25mins to provision the cluster completely and the nodes.</p><p>After the terraform configuration applies successfully, execute the command below to get kubectl to connect to the cluster. Don’t ask why the cluster name is realworld… Just a random name.</p><pre>aws eks update-kubeconfig --region us-east-1 --name realworld-cluster</pre><p>Test the connection if you can reach the cluster.</p><pre>kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*rgYNxBWmYxzoFD98NU4zDA.png" /></figure><h3>Install Nginx Ingress</h3><p>In this article, we will deploy Keycloak SSO provider, ArgoCD, and Harbor. So it will be wise to use an ingress to access each service. Using host based routing.</p><p>I have a sub domain of demo.waaron.org so to implement host based routing for all three services, we will use the hosts:</p><p>argocd.demo.waaron.org — → ArgoCD Service</p><p>keycloak.demo.waaron.org — -&gt; Keycloak Service</p><p>harbor.demo.waaron.org — -&gt; Harbor Service.</p><p>To install nginx ingress, first add the helm repository using the command below. The command installs ingress nginx using the manifest file in the repo folder.</p><pre>kubectl apply -f ingress-nginx.yml</pre><p>Once the helm chart is installed, let’s proceed to confirm if all kubernetes objects created by the ingress nginx are running without errors:</p><pre>kubectl get all -n ingress-nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*agbWmi7Q1BEoqLf8AhNLGQ.png" /></figure><h4>Map an A or CNAME Record</h4><p>Once Nginx controller is fully set up and you have an external IP provided to you, head over to your domain provider and map and “A” record pointing to that external IP. In my case I have a loadbalancer dns name, I’ll use a CNAME record for all three endpoints. Notice that i used the same load balancer endpoint for all domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*e5DUhDL79wjmJBMRG4byEA.png" /></figure><h3>Install Certificate Manager</h3><p>Since we will be creating public domains, it will be wise to create generate TLS certificates inside the kubernetes cluster with a certificate provider. This will allow us to accept HTTPS connections through all endpoints.</p><p>To install cert manager, first add the helm repository using the command below:</p><pre>helm repo add jetstack https://charts.jetstack.io --force-update</pre><p>Install the helm chart using the command below:</p><pre>helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.15.2 --set crds.enabled=true</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*lwHcUpAeYlRMslmfdn4xlw.png" /></figure><p>Confirm that cert manager resources are running</p><pre>kubectl get all -n cert-manager</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*s8jhFRyyEBmo7-Ivw3GCZA.png" /></figure><h4>Provision a Cluster Issuer</h4><p>To set up certificates we will need an issuer which will sign the certificate. This issuer is typically a provider like letsencrypt that will approve that the domain is valid.</p><p>In the repo folder, navigate to /cert-manager and apply the manifest issuer.yml</p><pre>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: letsencrypt-issuer<br>spec:<br>  acme:<br>    server: https://acme-v02.api.letsencrypt.org/directory<br>    email:  phil16terpasetheo@gmail.com<br>    privateKeySecretRef:<br>      name: letsencrypt-issuer-key<br>    solvers:<br>    - http01:<br>        ingress:<br>          class: nginx</pre><p>It creates a cluster wide issuer from lets encrypt. Ensure you replace the email with your personal email.</p><pre>kubectl apply -f issuer.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*NftZy9QTl8yg_p9F0SpKWg.png" /></figure><p>Now we need to provision certificates for argocd, harbor, and keycloak. And each certificate needs to be in it’s own namespace. So we will begin by creating namespaces for all three services.</p><p>Still on the /cert-manager directory, there’s a file called namespace.yml the content of the file is below:</p><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: argocd</pre><pre>---</pre><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: harbor</pre><pre>---</pre><pre>apiVersion: v1<br>kind: Namespace<br>metadata:<br>  name: keycloak</pre><p>Create all namespaces using the command below:</p><pre>kubectl apply -f namespace.yml</pre><h4>Create TLS Certificates</h4><p>Next, we will generate TLS certificates for ArgoCD, Harbor and Keycloak. Each of these certificates will be signed by the Issuer we created earlier and will allow us to access each service using HTTPS connections.</p><p>Still on /cert-manager directory, there’s a file called certificate.yml let’s look at the content of this file</p><pre>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: harbor-cert<br>  namespace: harbor<br>spec:<br>  dnsNames:<br>    - harbor.demo.waaron.org<br>  secretName: harbor-tls<br>  issuerRef:<br>    name: letsencrypt-issuer<br>    kind: ClusterIssuer</pre><pre>---</pre><pre>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: argocd-cert<br>  namespace: argocd<br>spec:<br>  dnsNames:<br>    - argocd.demo.waaron.org<br>  secretName: argocd-tls<br>  issuerRef:<br>    name: letsencrypt-issuer<br>    kind: ClusterIssuer</pre><pre>---</pre><pre>apiVersion: cert-manager.io/v1<br>kind: Certificate<br>metadata:<br>  name: keycloak-cert<br>  namespace: keycloak<br>spec:<br>  dnsNames:<br>    - keycloak.demo.waaron.org<br>  secretName: keycloak-tls<br>  issuerRef:<br>    name: letsencrypt-issuer<br>    kind: ClusterIssuer</pre><p>The file content simply generates certificates in each service namespace , take note of the DNS sub domains I used to match with what i created in the CNAME record i created earlier.</p><p>Execute the command below to apply the certificates:</p><pre>kubectl apply -f certificate.yml</pre><p>Wait for a while for it to enter a ready state and Confirm that all certificates are created and issued.</p><pre>kubectl get certificates -n keycloak<br>kubectl get certificates -n argocd<br>kubectl get certificates -n harbor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*PuVkb-iYxD_pd3SECNRzOg.png" /></figure><h3>Install ArgoCD</h3><p>Let’s proceed to install ArgoCD and configure it to use the domain and TLS certificate we created.</p><pre>helm repo add argo https://argoproj.github.io/argo-helm</pre><p>In the repository folder, there’s a file called argocd-values.yml lets look at the content of the file:</p><pre>global:<br>  domain: argocd.demo.waaron.org</pre><pre>server:<br>  ingress:<br>    enabled: true<br>    ingressClassName: &quot;nginx&quot;<br>    hostname: &quot;argocd.demo.waaron.org&quot;<br>    tls: false<br>    extraTls:<br>    - hosts:<br>      - argocd.demo.waaron.org<br>      secretName: argocd-tls<br>    annotations:<br>      nginx.ingress.kubernetes.io/backend-protocol: HTTPS</pre><p>What the file does is it simply sets the ingress class name as nginx and the host name as the one we have in the CNAME record. Take note of the secret name we set. It should match the secret name created in the certificate.</p><p>Install the helm chart using the command below:</p><pre>helm upgrade --install argo-cd argo/argo-cd --version 7.3.10 --reuse-values -f .\argocd-values.yml --namespace argocd --create-namespace</pre><p>Once the chart is successfully applied, confirm that argocd resources are created successfully using the command below:</p><pre>kubectl get pods -n argocd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*e19f9mgytvdXV64fOBvhNg.png" /></figure><p>You can also check the ingress created by the argocd helm chart</p><pre>kubectl get ingress -n argocd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Yx36CNRHmQnBiPwq9UmQA.png" /></figure><p>You can see the host mapped correctly and the address pointing to the load balancer’s DNS name. On your browser, open the host for argocd and you should get a secured connection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZYb49VGAchXsGXP9Hms5uQ.png" /></figure><p>Now we have ArgoCD installed, let’s proceed to install Keycloak.</p><h3>Install Keycloak</h3><p>We will still use helm to install Keycloak. First add the helm repo using the command below. We will be using Keycloack from bitnami helm chart:</p><pre>helm repo add bitnami https://charts.bitnami.com/bitnami</pre><p>Let’s also set up a values file to get keycloak to point to the SSL certificate that we have inside the kubernetes cluster.</p><p>In the repo folder, there’s a file named keycloak-values.yml let’s take a look at the file</p><pre>production: true<br>auth:<br>  createAdminUser: true<br>  adminUser: admin<br>  adminPassword: admin12345<br>tls:<br>  enabled: true<br>  autoGenerated: true<br>ingress:<br>  enabled: true<br>  hostname: keycloak.demo.waaron.org<br>  ingressClassName: nginx<br>  annotations:<br>    nginx.ingress.kubernetes.io/backend-protocol: HTTPS<br>  servicePort: https<br>  tls: true<br>  extraTls:<br>  - hosts:<br>    - keycloak.demo.waaron.org<br>    secretName: keycloak-tls<br>postgresql:<br>  enabled: true<br>  postgresqlPassword: psqldbpass12345</pre><p>The values file begins by setting up the default admin password and user, then proceed to set up the domain with the TLS secret we created in the certificates step. Finally, it provisions postgresql instance inside the cluster with a password.</p><p>Install the chart using the command below:</p><pre>helm upgrade --install keycloak bitnami/keycloak  --reuse-values -f .\keycloak-values.yml --namespace keycloak --create-namespace</pre><p>Confirm that all keycloak resources are running.</p><pre>kubectl get pods -n keycloak</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*qLIGK06LE6g38VvBJ6xWvQ.png" /></figure><p>Access keycloak UI using the domain you set</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q0pZg7WuluxqSz6sd-Wmqw.png" /></figure><h3>Install Harbor</h3><p>Let’s proceed to finally install harbor using helm. First add harbor helm chart using the command below</p><pre>helm repo add harbor https://helm.goharbor.io</pre><p>In the repository root folder, there’s a values file for harbor called harbor-values.yml the content of the file is given below:</p><pre>expose:<br>  type: ingress<br>  tls:<br>    enabled: true<br>    certSource: secret<br>    secret:<br>      secretName: harbor-tls<br>  ingress:<br>    hosts:<br>      core: harbor.demo.waaron.org<br>    kubeVersionOverride: &quot;&quot;<br>    className: nginx<br>    annotations:<br>      nginx.ingress.kubernetes.io/backend-protocol: HTTP<br>      # note different ingress controllers may require a different ssl-redirect annotation<br>      # for Envoy, use ingress.kubernetes.io/force-ssl-redirect: &quot;true&quot; and remove the nginx lines below<br>      ingress.kubernetes.io/ssl-redirect: &quot;true&quot;<br>      ingress.kubernetes.io/proxy-body-size: &quot;0&quot;<br>      nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;<br>      nginx.ingress.kubernetes.io/proxy-body-size: &quot;0&quot;</pre><p>Install the helm chart using the command below:</p><pre>helm upgrade --install harbor harbor/harbor --namespace harbor --create-namespace --reuse-values -f ./harbor-values.yml</pre><p>Confirm that harbor resources are provisioned successfully</p><pre>kubectl get pods -n harbor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*95-cGpchMvhIA6_vGhaHAg.png" /></figure><p>Access harbor UI using the domain you set for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QF9kGZQ9aG6AV8rTlDR5A.png" /></figure><h3>Configure SSO login with Keyclock on ArgoCD</h3><p>Congrats on reaching this far. Now we will set up the SSO login to Argocd using keycloak. Login to the Keycloak UI from the domain host you set earlier mine is keycloak.demo.waaron.org</p><p>Sign in using the admin as username and admin12345 as password. You should have a UI like the screen shot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RomsBO39zuKXMOMelmqDug.png" /></figure><p>Click on Clients and create a new client. Note that we are using the defaul “master” realm. To create a client, click on Clients &gt; Create Client.</p><p>Name enter “argocd” as client ID and “ArgoCD Client” as client name click next. Ensure that you selected OpenID connect in the client type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*931o47d8bFIefmV8XvyNQg.png" /></figure><p>Turn on client authentication and ensure you check mark Standard flow and Direct Access grants. Click next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8n40ulYiGS7N5k1uy_JPPQ.png" /></figure><p>Enter the domain URL you set for ArgoCD and the home URL to be /applications. Enter your argocd domain and append /auth/callback behind it for rediect URIs.</p><p>Also enter + for valid post logout URIs. And your argocd domain as web origin. Click on Save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*91aJ2dyLASGf1O0f7ejyxw.png" /></figure><p>After clicking save, there should be a credentials tab on that page copy the client secret and save somewhere. We will use it in argocd.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHsoXkzZ443wjyc0oCRpmg.png" /></figure><p>Assuming the user is authenticated to argocd, we will need to specify a groups claim for that user to assume. To do this, we will need to create a client scope.</p><p>On the left tab click on Client Scopes and click on Create scope. Name the scope groups. Also ensure that you enable “Include in token scope” and click on Save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fi5twZABVbfJgV3-S0g0zg.png" /></figure><p>Now we need to add a mapper that will point the groups claim to the token when the user requests the group scope. Click on the tab mappers and click on “Configure a new mapper” select <strong>Group Membership. </strong>Set the name and the token claim name to <strong>groups. </strong>Also disable full group path. Click Save</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MaT3Njq8BF0BYiYXctAHag.png" /></figure><p>Now head back to the Client we created earlier in the clients tab with name “argocd”, click on client scopes click on “add a client scope”. Select the scope we created called groups. Add it as defualt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*tQwrLhBgUC7PVfj0bdBiKg.png" /></figure><p>Finally, on the left tab, click on groups and <strong>ArgoCDAdmins.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*8a_AQt0yuT7bPMPKLY8wHw.png" /></figure><p>Click on the members tab and add a member. Add the admin user as a member of the group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*XtwO15H4cQmTNumuBjmRSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*iaAeOjmzSQlpyfiNjQMFIQ.png" /></figure><h4>Enable SSO on ArgoCD</h4><p>Now we can head over to argocd running our kubernetes cluster and enable SSO login with keycloak. Before you proceed, encrypt the client secret you copied as base64 a good tool to use can be: <a href="https://www.base64encode.org/">https://www.base64encode.org/</a></p><p>Edit argocd secret and add the keycloak secret using the keyname <em>oidc.keycloak.clientSecret</em></p><pre>kubectl edit secret argocd-secret -n argocd</pre><p>The secret should look like this: “Ensure you use the base64 value”</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: argocd-secret<br>data:<br>  ...<br>  oidc.keycloak.clientSecret: ODMwODM5NTgtOGVjNi00N2IwLWE0MTEtYThjNTUzODFmYmQy   <br>  ...</pre><p>Edit argocd configmap using the command below:</p><pre>kubectl edit configmap argocd-cm -n argocd</pre><p>Edit the config map and add the oidc config data key. Also confirm that the url and issuer are the domain hosts you’ve set in previous steps.</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: argocd-cm<br>data:<br>  url: https://argocd.demo.waaron.org<br>  oidc.config: |<br>    name: Keycloak<br>    issuer: https://keycloak.demo.waaron.com/realms/master<br>    clientID: argocd<br>    clientSecret: $oidc.keycloak.clientSecret<br>    requestedScopes: [&quot;openid&quot;, &quot;profile&quot;, &quot;email&quot;, &quot;groups&quot;]</pre><p>Finally, let’s attach the groups e created earlier in keycloak. Edit the config map using the command below:</p><pre>kubectl edit configmap argocd-rbac-cm -n argocd</pre><p>Add the data given below with key policy.csv</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: argocd-rbac-cm<br>data:<br>  policy.csv: |<br>    g, ArgoCDAdmins, role:admin</pre><p>Now check the ArgoCD UI, refresh and confirm if the SSO login button is up. You can see login via Keycloak.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*geX8Qjnd1V5knUzN3OF1kQ.png" /></figure><p>If you click on it, it will redirect you to keycloak to login then back to the applications page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0MTO0eG-pU0AHl6eGq93A.png" /></figure><p>If Argocd refuses to authenticate you, in a scenario where it displays the applications page then forcefully take you back to login again, simply refresh argocd pods using the command below</p><pre>kubectl rollout restart deployment argo-cd-argocd-server -n argocd</pre><h3>Configure SSO login with Keyclock on Harbor</h3><p>OIDC setup on Herbor is more easier. All we have to do is set up the client on keycloak and enter it’s details and secret using the Harbor admin dashboard.</p><p>Navigate back to you keycloak dashboard on the domain and create a new client. Set the client ID as harbor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xE5wDPHLjZGVwXRJ4dJmKQ.png" /></figure><p>Enable client authentication</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pujjMBOfSvsN_5yvRPUjAQ.png" /></figure><p>Set root URL and home URL as the host name you created for harbor. In my case harbor.demo.waaron.org. Also set redirect URI to <a href="https://harbor.demo.waaron.org/c/oidc/callback">https://harbor.demo.waaron.org/c/oidc/callback</a> Click on save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F1rNnlYfVUFfCyPAMJWbJg.png" /></figure><p>Navigate to the secret tab and copy the secret value Somewhere.</p><p>Now head over to your harbor host name on your browser in my case, harbor.demo.waaron.org and login as admin. username: admin password: Harbor12345</p><p>On Harbor UI, on the left pane click on Administration &gt; Configuration &gt; Authentication Select OIDC</p><p>For name enter Keycloak. Also Enter <a href="https://keycloak.demo.waaron.org/realms/master">https://keycloak.demo.waaron.org/realms/master</a> as OIDC endpoint (Ensure you edit the host to yours). Also fill in the client id as harbor and secret what you copied earlier. For client scope, enter openid,profile,rolesand click save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4bfJb-yR_iJdH7RNjFkCcw.png" /></figure><p>Test the connection and save.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JrH92Xmp7CSUd5HtwV8ltw.png" /></figure><p>On Harbors login page, you should see login with Keycloak</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_EwNFTu696YCRM4FIzsZg.png" /></figure><p>Congrats! you’ve successfully configured OIDC authentication on Harbor and ArgoCD using keycloak! Let’s proceed to clean up.</p><h3>Clean Ups</h3><p>Let’s proceed to remove Harbor, ArgoCD, Keycloak and Ingress Nginx.</p><h4>Uninstall Harbor</h4><p>To uninstall harbor, execute the command below:</p><pre>helm uninstall harbor -n harbor</pre><h4>Uninstall ArgoCD</h4><p>Uninstall argocd using the command below</p><pre>helm uninstall argo-cd --namespace argocd</pre><h4>Uninstall Keycloak</h4><p>To clean up keycloak uninstall the helm chart with the command below</p><pre>helm uninstall keycloak --namespace keycloak</pre><h4>Uninstall Nginx Ingress</h4><p>To remove nginx ingress, ensure you’re in the project repo folder and execute the command below:</p><pre>kubectl delete -f .\ingress-nginx.yaml</pre><p>You can also proceed to delete all certificates also.</p><p>Finally delete the cluster:</p><pre>cd kubernetes-cluster<br>terraform destroy --auto-approve</pre><h3>Conclusion</h3><p>In this long tutorial, we went through the setting up Keycloak, Harbor, and ArgoCD on an EKS cluster and configuring SSO on ArgoCD and Harbor uaing Keycloak. Thank you for following along!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1791b1aa27b0" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/configure-sso-authentication-to-argocd-and-harbor-with-keycloak-on-eks-cluster-1791b1aa27b0">Configure SSO authentication to ArgoCD and Harbor with Keycloak on EKS Cluster</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install Kyverno and Harbor on Kubernetes and Enforce only Images from Harbor using Policies]]></title>
            <link>https://medium.com/@phil16terpasetheo/install-kyverno-and-harbor-on-kubernetes-and-enforce-only-images-from-harbor-using-policies-8c45545d1532?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c45545d1532</guid>
            <category><![CDATA[policy-as-code]]></category>
            <category><![CDATA[harbor]]></category>
            <category><![CDATA[kyverno]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Sat, 20 Jul 2024 09:05:31 GMT</pubDate>
            <atom:updated>2025-01-07T12:04:14.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*_54js0YZ3SwADyDbV5veqg.png" /></figure><h3>Introduction</h3><p>Article Repo Link: https://github.com/philcz16/kyverno</p><p>Security in Kubernetes clusters and ensuring best security practices in k8s is now something that isn’t only talked about on Kubernetes podcasts or events but something that is now in practice and is here to stay.</p><p>With the new tech concept of Infrastructure as Code and Everything as Code, we are now introduced with new concept of Policies as code in Kubernetes. What is policy as code?</p><h4>Policy As Code</h4><p>This practice is the use of code to define and manage rules and conditions. In this concept, policies are written out in a programming language such as Python or YAML. The type of language used depends on the policy-as-code management or enforcing tool.</p><p>In this article, we will install another kubernetes tool called Harbour that allows us to manage a local image registry in Kubernetes. Then, we will demonstrate the policy as code strategy to enforce pods on the Kubernetes cluster to only allow images from the local registry.</p><h3>Hands On</h3><p>The hands on is in three simple steps, listed below:</p><ul><li>Install Harbor on Kubernetes Cluster</li><li>Write out Policy as code to enforce only images from our local Harbor Registry.</li><li>Test the policy.</li></ul><p>To follow along, you’ll need the following:</p><ul><li>Helm Installed</li><li>Kubernetes cluster (you can use a local cluster like Minikube)</li><li>Docker installed</li></ul><h4>Spin Up a Cluster and Install Nginx Ingress</h4><p>In this article, we make use of minikube to spin up a local cluster on a windows machine. You can use your linux/Mac to spin up a local cluster. You can also use a cluster on the cloud like EKS or AKS.</p><p>Spin up a cluster on your local machine using the command below:</p><pre>minikube start</pre><p>To access Harbor’s web interface, we will need an ingress controller. Using minikube, you can add nginx ingress using the command below. This will install nginx ingress and configure it on minikube’s cluster.</p><pre>minikube addons enable ingress</pre><p>Confirm that the nginx controller is running.</p><pre>kubectl get pods -n ingress-nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*vfTC9K5lQdBMNaFNQ52Czw.png" /></figure><h4>Install Harbor Registry</h4><p>Next, we’ll use Helm to install Harbor and edit some values in the values file to configure the web interface for the local image registry.</p><p>Add the Habor’s helm repo using the command below:</p><pre>helm repo add harbor https://helm.goharbor.io</pre><p>We need to edit some values in the helm chart’s values file. To do this, we will download the helm chart to a local folder, edit some values, nd finally install the chart using the local folder’s version we have.</p><p>Use the command below to download the helm chart. This will create a folder named <em>harbor</em> with all the configuration files for the helm chart.</p><pre>helm fetch harbor/harbor --untar</pre><p>If you open the workdir on vscode like me, you should have a folder there named Harbor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*LZByadJwDq1CVGqiUzBviQ.png" /></figure><p>In the values.yml file, edit the host name for harbor registry. Under {{ <em>expose.ingress.hosts.core }} </em>For this article, I used the hostname registry.local.harbor. You can leave as default but if you change it, ensure that you set {{ <em>externalURL }} </em>to the same value as your host value.</p><p>Finally, edit the ingress class name under {{ expose.ingress.className }} in values.yaml file to your ingress class name. If you’re using minikube and the nginx controller addon, enter “nginx” as the the className. The full file of my values file is in the repo provided in the article introduction.</p><p>Finally, install harbor using the directory folder name. The command is given below:</p><pre>helm install harbor ./harbor -n harbor --create-namespace</pre><p>Confirm that harbor’s resources are all running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*KZgrtQCji0IcjYKEPm5img.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*Xfxq0aPGWsLB_Pvv9qigeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*ImhPdfsjYnAJguMe5_1yew.png" /></figure><p>If all harbor’s resources are running and created successfully, we can then proceed to access the web interface for the registry and also try to push an image to the registry running in Kubernetes.</p><p>Harbor creates it’s own ingress using the ingress class we provided in the values file. you can use the command below to confirm</p><pre>kubectl get ingress -n harbor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*tCML6xQAM2xVOvrJn6hOBg.png" /></figure><p>So we can access the Harbor registry using the endpoint <a href="https://registry.local.harbor/">https://registry.local.harbor</a> but before we do that, we need to modify the host machine’s hostname to be able to resolve the domain name on our local computer.</p><p>If you’re using a windows machine, navigate to the directory C:\Windows\System32\drivers\etc and edit the hosts file in Administrator mode. Add the line below in the file.</p><pre>127.0.0.1  registry.local.harbor</pre><p>If you’re using a UNIX/MAC machine, open the hosts file in the directory /etc/hosts and add the line below</p><pre>127.0.0.1  registry.local.harbor</pre><p>Finally, open a seperate terminal from the one you’ve been using and execute the command below:</p><pre>minukube tunnel</pre><p>This will open a session for us to access all services and ingress defined in the minikube cluster. You can then access the registry Interface in your browser usign the URL <a href="https://registry.local.harbor/">https://registry.local.harbor</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DorB9cnUHDtbMg6KC28cHw.png" /></figure><p>Login using the default credentials below. Note that you can modify this in the values file.</p><pre>Username: admin<br>Password: Harbor12345</pre><p>Don’t hesitate to create a new project using the UI. Create a project and name it k8s. We’re going to push an nginx image to this project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*EoQCLQeY_SiEQhecmFb_lQ.png" /></figure><h4>Push an image to Local Harbor Registry</h4><p>On your local machine, execute the command to login to the registry use the same credentials you used to login to the web interface to login to the registry.</p><pre>docker login https://registry.local.harbor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*rn0cdzeSjiwwJBiJTezO-A.png" /></figure><p>If you get any certificate error when trying to login, if you’re on UNIX system edit /etc/docker/daemon.json add the line below</p><pre>{<br>  &quot;insecure-registries&quot; : [&quot;registry.local.harbor&quot;]<br>}</pre><p>If you’re on a Windows or Mac, got to Settings &gt; Docker Engine and add the line. Save and restart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*P_tyCClNTCV6qdUsyFCXSw.png" /></figure><p>Next, we will pull an nginx image and tag it with our harbor image url and then push the image to harbor local registry running in kubernetes.</p><pre>docker pull nginx:alpine</pre><p>Tag the image with our harbor registry URL</p><pre>docker tag nginx:alpine registry.local.harbor/k8s/nginx:alpine</pre><p>Finally, push the image to the harbor registry.</p><pre>docker push  registry.local.harbor/k8s/nginx:alpine</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UL0_qu31bYyUZI0uAR8p-g.png" /></figure><h3>Kyverno — Policy as Code</h3><p>In this section, we will allow only images from our registry in Harbor on a specific namespace using kyverno. So we will write a policy that will only allow images from ONLY the harbor registry.</p><h4>Install Kyverno</h4><p>We will use helm to install Kyverno on the Minikube Kubernetes cluster. Add kyverno helm repo using the command below:</p><pre>helm repo add kyverno-repo https://kyverno.github.io/kyverno/</pre><p>Install the helm chart using the command below</p><pre>helm install kyverno kyverno-repo/kyverno -n kyverno-ns --create-namespace</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*OOyNt0YPvr6ssCcmulK3QA.png" /></figure><p>Also confirm that all of kyverno’s resources are created successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*76ZyG46b0ShiX57rVPFCDQ.png" /></figure><h4>Writing Kyverno Policy</h4><p>In this step, we will write out the policy. The policy written below only allows images from the harbor registry we configured above on a specific namespace in our case, app-namespace. create a file disallow_any_repo.yaml</p><pre>apiVersion : kyverno.io/v1<br>kind: ClusterPolicy<br>metadata:<br>  name: check-images<br>spec:<br>  validationFailureAction: Enforce<br>  background: false<br>  rules:<br>  - name: check-registry<br>    match:<br>      any:<br>      - resources:<br>          kinds:<br>          - Pod<br>          namespaces:<br>          - app-namespace<br>      <br>    preconditions:<br>      any:<br>      - key: &quot;{{request.operation}}&quot;<br>        operator: NotEquals<br>        value: DELETE<br>    validate:<br>      message: &quot;unknown registry&quot;<br>      foreach:<br>      - list: &quot;request.object.spec.initContainers&quot;<br>        pattern:<br>          image: &quot;registry.local.harbor/*&quot;<br>      - list: &quot;request.object.spec.containers&quot;<br>        pattern:<br>          image: &quot;registry.local.harbor/*&quot;</pre><p>Let’s have a quick look at the policy. The policy is of kind ClusterPolicy, which means it is going to be deployed cluster wide, it’s name is check-images.</p><p>The fun part happens in the spec where we specify Enforce, which means that it will block any request that isn’t allowed by the rule and will throw an error. The second option other than Enfore is Audit. Audit will not block the request, rather it will send audit messages to the audit management tool about the request and it’s’ state.</p><p>In the snippet below, you can see that it checks for pods that are deployed in the namespace of app-namespace. So any pod outside the app namespace can use any registry image but pods in the app-namespace can only use the registry from harbor.</p><pre>- name: check-registry<br>    match:<br>      any:<br>      - resources:<br>          kinds:<br>          - Pod<br>          namespaces:<br>          - app-namespace</pre><p>Finally, we check if the request is not a delete operation and then add the logic to validate the image. If you look closely, you can see that the manifest uses our image url from harbor “registry.local.harbor” . Let’s test this rule.</p><p>Apply the policy.</p><pre>kubectl apply -f .\disallow_any_repo.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*TTt57ObUINslkwTww1o3NQ.png" /></figure><p>To test the policy, let’s first of all create the app-namespace</p><pre>kubectl create ns app-namespace</pre><p>Then try to apply an nginx configuration that uses the public dockerhub registry:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx-deployment<br>spec:<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: nginx:alpine<br>        ports:<br>        - containerPort: 80</pre><pre>kubectl apply -f nginx.yaml -n app-namespace</pre><p>You’ll see that request will be rejected with an error message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*D6Guf7Lpu6jg0PW1wv7_gQ.png" /></figure><p>But when we edit the image URLto be a URL from our Harbor registry, The resource is created successfully.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx-deployment<br>spec:<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  replicas: 1<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: registry.local.harbor/k8s/nginx:alpine<br>        ports:<br>        - containerPort: 80</pre><pre>kubectl apply -f nginx.yaml -n app-namespace</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*8OawyGmPTqEVV6ruy6oXdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*yAo-J9hsWCWhguYix3MBoQ.png" /></figure><h3>Conclusion</h3><p>Writing policies to ensure that compliance and best practices are followed in your kubernetes environment is key to advanced security. This can aid your cluster security. Kyverno is a great tool and can be used to write out more complex policies to meet your cluster security needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c45545d1532" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure Alert Manager with Slack Notifications on Kubernetes Cluster — Helm Kube-Prometheus Stack]]></title>
            <link>https://medium.com/@phil16terpasetheo/configure-alert-manager-with-slack-notifications-on-kubernetes-cluster-helm-kube-prometheus-stack-112878c35f26?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/112878c35f26</guid>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Wed, 19 Jun 2024 13:28:32 GMT</pubDate>
            <atom:updated>2024-06-19T13:52:47.550Z</atom:updated>
            <content:encoded><![CDATA[<h3>Configure Alert Manager with Slack Notifications on Kubernetes Cluster — Helm Kube-Prometheus Stack</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*LUPyVYFeEi4XZJT5L770KA.png" /></figure><h3>Intro</h3><p>Being aware of your application’s health is a game changing innovation in infrastructure configuration and application deployments. Especially in Kubernetes environments where there are multiple micro-service applications and might be hard to point out which micro-service is experiencing downtimes due to the amount of pods, deployments and other components that make up apps on Kubernetes.</p><h4>Alert Manager</h4><p>Alert manager is a tool in Kubernetes integrated together with Prometheus to monitor application health using metrics. Alert manager specifically sends notification messages or alerts to your favorite alerting clients such as Emails, Amazon SNS and Slack too.</p><p>Alert manager is a powerful tool that sends in notification such as slack messages when any of your application being monitored with Prometheus is reported down or experiencing high traffic based on the rules you specify.</p><h3>Demo</h3><p>In this article, we will learn how to configure and deploy alert manger on a Kubernetes cluster, deploy a demo Golang application that scrapes metrics and finally, run a demo of the alerting rules by receiving message a slack message from alert manager when we manually take down the running Golang app.</p><h4>Prerequisites</h4><p>To follow along with the demo, you will need the following prerequisites:</p><ul><li>A running Kubernetes Cluster (If you don’t have one, a terraform script is provided to provision one for you on AWS EKS)</li><li>If you’ll use the terraform script, you’ll need an AWS account configured (Access Keys)</li><li>AWS CLI configured</li><li>Helm Installed</li><li>Git installed</li></ul><h4>Provision Cluster</h4><p>To provision a cluster on AWS EKS, the provided repository for this article contains Terraform script to provision networking resources and an EKS cluster. The script also updates your local machine’s Kube context to the EKS cluster context.</p><p>Clone the git repository using the command below:</p><pre>git clone https://github.com/philcz16/alertmanager-demo</pre><h4>Hands On</h4><p>First set up the Kubernetes cluster with terraform if you don’t have one. To spin up a cluster, on AWS EKS, clone the repository below:</p><pre>https://github.com/philcz16/alertmanager-demo-prom-helm</pre><p>Your project folder should have the structure like the screenshot below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*cDmKzBotx-0mkBn82kU5Bw.png" /><figcaption>folder structure</figcaption></figure><p>/application contains the golang application code that is deployed, /terraform contains the terraform script that provisions the EKS cluster. The YAML files in the root folder are manifests that we will use to set up the kube-prometheus monitoring stack.</p><blockquote>App.yml contains deployment file for the golang application deployed</blockquote><h4>Provision Cluster</h4><p>Navigate to /terrform and apply the terraform configuration provided. The configuration creates VPC, EKS cluster and other supported addons to run Prometheus on the cluster. It will also add the EKS cluster to your local machine Kube-Context.</p><p>To apply the terraform config, execute the command below:</p><pre>terraform init<br>terraform apply --auto-approve</pre><p>Once terraform successfully applies and all resources are created, you should be able to execute kubectl commands to connect directly to the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*HaJJii0yhPu29oJuClkVQQ.png" /></figure><blockquote>Note that all nodes created by the cluster are SPOT instances hence, cheaper</blockquote><h4>Install Kube-Prometheus Stack</h4><p>To install the Kube-Prometheus stack that installs Alertmanger, Prometheus and Node exporter, first add the repository:</p><pre>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update</pre><p>Create a namespace named monitoring where we will provision all resources related to Prometheus and alert manager.</p><pre>kubectl create namespace monitoring</pre><p>Install Kube-Prometheus using the command below. Note that it takes approximately 10mins to complete.</p><pre>helm install kube-prometheus prometheus-community/kube-prometheus-stack -n monitoring</pre><p>Verify the installation on the cluster:</p><pre>kubectl get pods -n monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*424SmO8MwM-6me6-cqG7hw.png" /></figure><p>To access prometheus UI, for this demo, we’ll be using port forwarding in kubernetes. Execute the command below:</p><pre>kubectl port-forward svc/kube-prometheus-kube-prome-prometheus -n monitoring 9090</pre><p>This will open a session in your terminal (don’t close the terminal to avoid session termination). You can then access prometheus UI on your browser via <strong>localhost:9090. </strong>Navigate to the <strong>alerts</strong> tab to see default alert rules created by kube-prometheus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bgyd2dLfsIG9uVnu1hzbag.png" /></figure><p>Notice that most of the Alert rules are in <strong>Inactive</strong> state. Alert manager has three states Inactive, Pending and Firing</p><p><strong>Inactive State </strong>this state is when the rule does not meet the requirement that has been set for it and the alerting conditions are not met, so Prometheus considers the application running functional.</p><p><strong>Pending State </strong>this state is when the alerting rules is met but still in the duration set before evaluation.</p><p><strong>Firing State </strong>this state is when the set duration for the condition set is exceeded, for example a downtime for 1min. Prometheus will then send alerts to alert manager which will then be forwarded to either slack or mail.</p><h4>Set Up Golang Application</h4><p>In this section, we will create a simple kubernetes deployment and service to expose the golang application. The golang application is built to also collect metrics in /metrics path of the API.</p><p>In the project repository, in /application folder, the Golang app code is there with a dockerfile. You can build and push your version of the application to a docker registry, or you can use my dockerhub image in the manifest below.</p><p>The content of the manifest file in a.yml is below:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: app-1<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: app-1<br>  template:<br>    metadata:<br>      labels:<br>        app: app-1<br>        monitoring: enabled # this is needed!!<br>    spec:<br>      containers:<br>        - name: app-1<br>          image: Philcz16/goapp:latest<br>          ports:<br>            - containerPort: 8080<br>              name: http-metrics<br>          resources:<br>            requests:<br>              memory: 128Mi<br>              cpu: 50m<br>            limits:<br>              memory: 128Mi<br>              cpu: 50m<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: app-1-service<br>  labels:<br>    monitoring: enabled # this is needed!!<br>spec:<br>  type: LoadBalancer<br>  ports:<br>    - port: 80<br>      targetPort: http-metrics<br>      name: http-metrics<br>  selector:<br>    app: app-1</pre><p>Keep note of the label <strong>monitoring: enabled </strong>this label is going to be used for Prometheus to identify the pod and service and collect monitoring metrics.</p><p>To apply the manifest, execute the command below</p><pre>kubectl apply -f app.yml</pre><p>Confirm the deployment and service:</p><pre>kubectl get pods<br>kubectl get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*HUuKu0IAe0oKzFqhjfc0sQ.png" /></figure><p>If you’re using the EKS cluster provided, you can access the app through the load balancer link/endpoint provided in the svc. Notice in the screenshot below, the app fetched metrics and return them on /metrics route.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8RZJV0wXczQfFks3BZGkA.png" /></figure><h4>Pod Monitor and Service Monitor</h4><p>The kube prometheus stack comes with custom CRDs. These custom CRDs specifically PodMonitor and ServiceMonitor, inform the Prometheus operator of which pods to scrape metrics from. The pods re targeted using labels which is the reason why we specified a custom label in our golang application deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/0*gJDPNI-gal_5EtQ5.png" /></figure><h4>Pod Monitor</h4><p>Let’s configure a pod monitor using the provided CRD created by the kube-prometheus stack.</p><p>Create file podmonitor.yml</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: PodMonitor<br>metadata:<br>  name: my-podmonitor<br>  namespace: default<br>spec:<br>  podMetricsEndpoints:<br>    - path: metrics<br>      port: http-metrics<br>  namespaceSelector:<br>    any: true<br>  selector:<br>    matchLabels:<br>      monitoring: enabled</pre><p>Notice that <strong>selector.matchLabels </strong>specifies the pod label that should be monitored which is what we specified in the deployment manifest.</p><p>apply the pod monitor manifest:</p><pre>kubectl apply -f podmonitor.yml</pre><h4>Update Helm</h4><p>Initially, we installed kube prometheus without a values file. To ensure that kube-prometheus picks up the new pod monitor we just created, we need to over write some default values.</p><p>Create a kube-prometheus.yaml add the content below:</p><pre>prometheus:<br>  prometheusSpec:<br>    podMonitorSelectorNilUsesHelmValues: false<br>    serviceMonitorSelectorNilUsesHelmValues: false</pre><p>Save the file and upgrade the helm chart installed using the command below</p><pre>helm upgrade -f kube-prometheus.yaml kube-prometheus prometheus-community/kube-prometheus-stack -n monitoring</pre><p>After applying the new change, wait some minutes for Prometheus to pick up the new pod monitor change. You can then query some of the application metrics using the query app_total_metrics</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogcbOpfsgr-FPDf0Ll57CQ.png" /></figure><p>You can see that Prometheus can query metrics coming from the Golang application we deployed.</p><h4>Send Alerts to Slack with Alert Manager</h4><p>Kube-prometheus comes with a custom CRD that you can use to create custom manifests to create alerting rules.</p><p>Create a prometheus-rule.yml file and paste the content below. The rule sends alerts when the pod number for app-1 is less than 2. It also forwards a message in the annotations.</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: PrometheusRule<br>metadata:<br>  labels:<br>    release: kube-prometheus<br>  name: app-1-rule<br>  namespace: monitoring<br>spec:<br>  groups:<br>    - name: &quot;app-1.rules&quot;<br>      rules:<br>        - alert: PodDown<br>          for: 1m<br>          expr: sum(up{container=&quot;app-1&quot;}) &lt; 2<br>          labels:<br>            severity: critical<br>            app: app-1<br>          annotations:<br>            # these annotations are going to be used in the alert that will be send to the slack/email/etc...<br>            message: The deployment has less than 1 pod running.<br>            summary: the current deployment &quot;app-1&quot; need to have at least 2 replicas running<br>            description: there seems to be a mismatch in the number of pod replicas between your alert configuration and the pod replicas</pre><p>Here’s a quick break down:</p><ul><li>Metadata: Custom metadata for the rule to be created.</li><li>Groups: Organize rules into groups.</li><li>For: Duration the alert stays in the pending state before entering firing state.</li><li>Expr: PromQL expression to evaluate the alert condition.</li><li>Labels: are used mainly for filtering on alerting so we can use those labels to know to which destination we need to direct this message, an example would be:<br>- if the alert contains a label with severity=critical direct it to the DevOps team lead<br>- if the alert contains a label with severity=warning direct it to the Senior DevOps member</li><li>Annotations: are metadata that we pass to the alert, which can easily be messages and text to include in the alert.</li></ul><p>Apply the manifest:</p><pre>kubectl apply -f prometheus-rule.yml</pre><p>Wait a few minutes and then check the Prometheus UI dashboard in the alerting tab. You should find a new alert PodDown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-GeQikXmLjita6osf27BQ.png" /></figure><p>Let’s attempt to scale down the golang app deployment and check if the rule will be tagged Firing</p><pre>kubectl scale deployments.apps app-1 --replicas=1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLDdQW6dZ8oDdyprCfqo_Q.png" /></figure><p>With the alerting rules working, let’s proceed to scale it back up and then finally configure the slack integration with the alert manager.</p><pre>kubectl scale deployments.apps app-1 --replicas=2</pre><h4>Slack Integration</h4><p>Create a slack channel and name it #alerts click on the channel name and select integrations tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*c8npLfxEaB1-c--3YqCstw.png" /></figure><p>Click on Add an App. Search for <strong>Incoming WebHooks.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*YFc-Bd5A3ijxKzSFJhYM0A.jpeg" /></figure><p>Click on Add to Slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9GcTT1bj4hZMHAnDiuQ_A.jpeg" /></figure><p>Copy the Webhook URL, keep the URL safe. It should be treated as secret.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Zd6UAPGACnlci-1wJT0y3w.jpeg" /></figure><p>Update the kube-prometheus.yml file for kube-prometheus helm chart to include the slack integration config. update the file with your slack API URL you copied earlier.</p><pre>prometheus:<br>  prometheusSpec:<br>    podMonitorSelectorNilUsesHelmValues: false<br>    serviceMonitorSelectorNilUsesHelmValues: false<br><br>alertmanager:<br>  config:<br>    global:<br>      slack_api_url: &lt;API URL here&gt;<br>  alertmanagerSpec:<br>    alertmanagerConfigNamespaceSelector:<br>    alertmanagerConfigSelector:<br>    alertmanagerConfigMatcherStrategy:<br>      type: None</pre><p>Upgrade the helm chart to include the alert manager configuration</p><pre>helm upgrade -f kube-prometheus.yaml kube-prometheus prometheus-community/kube-prometheus-stack -n monitoring</pre><p>Finally, let’s configure the alerts manager configuration to forward the notification from prometheus to our slack channel</p><p>Create a file alert-config.yml and paste the content below:</p><pre>apiVersion: monitoring.coreos.com/v1alpha1<br>kind: AlertmanagerConfig<br>metadata:<br>  name: global-alert-manager-configuration<br>  namespace: monitoring<br>  labels:<br>    release: kube-prometheus<br>spec:<br>  receivers:<br>    - name: slack-notifications<br>      slackConfigs:<br>        - channel: &quot;#alerts&quot; # your channel name that you have created<br>          sendResolved: true<br>          iconEmoji: &quot;:bell:&quot;<br>          text: &quot;&lt;!channel&gt; \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}\nmessage: {{ .CommonAnnotations.message }}&quot;<br>  route:<br>    matchers:<br>      # name of the label to match<br>      - name: app<br>        value: app-1<br>        matchType: =<br>    groupBy: [&quot;job&quot;, &quot;severity&quot;]<br>    groupWait: 30s<br>    receiver: slack-notifications<br>    groupInterval: 1m<br>    repeatInterval: 1m</pre><p>If you slack channel has a different name, you can update it in the manifest. The manifest is explained below:</p><p><strong>Receivers </strong>they specify if it’s an Email or Slack notification</p><p><strong>Matchers </strong>specify the labels defined in the Prometheus rules. In this demo or label is app and the values is app-1.</p><p><strong>Route </strong>specifies conditions or intervals when sending the notifications.</p><p>Apply the manifest using the command below:</p><pre>kubectl apply -f alert-config.yml</pre><h4>Test Alerts Manager</h4><p>Finally, let’s test the alerting set up by scaling the Golang application deployment replicas down to 1</p><pre>kubectl scale deployments.apps app-1 --replicas=1</pre><p>Wait for a few minutes for Prometheus to pick up the the downtime</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWFwDW9xO9ShlBY8bdwJsw.png" /></figure><p>Once it’s in the Firing state, it should trigger and send a slack notification to your channel that a down time occurred.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFWCTxt0cYqRId408THgmQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*XFEQI3x_JuHlboCxaOAjFw.png" /></figure><p>You can see that it has triggered and sent a message that the app replicas are below 2.</p><p>Scale the app back to 2 replicas</p><pre>kubectl scale deployments.apps app-1 --replicas=1</pre><p>It should also trigger a slack message notifying that the application is back up and healthy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*hFrOokbD5MXJjytlOanVxA.png" /></figure><h4>Clean Ups</h4><p>First remove all manifests and kube-prometheus CRD resources applied using the command below</p><pre>kubectl delete -f .\alert-config.yml<br>kubectl delete -f .\podmonitor.yml<br>kubectl delete -f .\prometheus-rule.yml</pre><p>Remove the kube prometheus stack using helm</p><pre>helm uninstall kube-prometheus -n monitoring   </pre><p>Remove the Golang application deployment manifest</p><pre>kubectl delete -f .\app.yml</pre><p>Finally, remove the EKS cluster using terraform in /terraform folder</p><pre>terraform destroy --auto-approve</pre><h4>Conclusion</h4><p>In this article, we demonstrated how alert manager can be used to send push alerts to your slack channel when your Kubernetes deployment is down. Using the Kube-prometheus helm chart and custom CRDs that come with it can be a game changer to deploying and configuring Prometheus alerts easily.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=112878c35f26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure Alert Manager with Slack Notifications on Kubernetes Cluster]]></title>
            <link>https://medium.com/@phil16terpasetheo/configure-alert-manager-with-slack-notifications-on-kubernetes-cluster-711387dbcaad?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/711387dbcaad</guid>
            <category><![CDATA[alertmanager]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Fri, 14 Jun 2024 23:23:46 GMT</pubDate>
            <atom:updated>2024-06-14T23:23:46.481Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*LUPyVYFeEi4XZJT5L770KA.png" /></figure><h3>Intro</h3><p>Being aware of your application’s health is a game changing innovation in infrastructure configuration and application deployments. Especially in Kubernetes environments where there are multiple micro-service applications and might be hard to point out which micro-service is experiencing downtimes due to the amount of pods, deployments and other components that make up apps on Kubernetes.</p><h4>Alert Manager</h4><p>Alert manager is a tool in Kubernetes integrated together with Prometheus to monitor application health using metrics. Alert manager specifically sends notification messages or alerts to your favorite alerting clients such as Emails, Amazon SNS and Slack too.</p><p>Alert manager is a powerful tool that sends in notification such as slack messages when any of your application being monitored with Prometheus is reported down or experiencing high traffic based on the rules you specify.</p><h4>Demo</h4><p>In this article, we will learn how to configure and deploy alert manger on a Kubernetes cluster, deploy a demo Golang application that scrapes metrics and finally, run a demo of the alerting rules by receiving message a slack message from alert manager when we manually take down the running Golang app.</p><h4>Prerequisites</h4><p>To follow along with the demo, you will need the following prerequisites:</p><ul><li>A running Kubernetes Cluster (If you don’t have one, a terraform script is provided to provision one for you on AWS EKS)</li><li>If you’ll use the terraform script, you’ll need an AWS account configured (Access Keys)</li><li>AWS CLI configured</li></ul><h4>Hands On</h4><p>First set up the Kubernetes cluster with terraform if you don’t have one. To spin up a cluster, on AWS EKS, clone the repository below:</p><pre>https://github.com/philcz16/alertmanager-demo</pre><p>Once the files have been successfully applied, and the cluster is up and running, update your local machine’s kubeconfig using the command below:</p><pre>aws eks --region us-east-1 update-kubeconfig --name my-cluster</pre><p>Create a namespace where we will deploy all Prometheus related resources:</p><pre>kubectl create namespace prometheus</pre><p>Still on the repository folder, apply the cluster role bindings and service accounts that gives Prometheus required access to the cluster.</p><pre>kubectl apply -f authorization.yml</pre><p>Next, deploy the sample Golang application that deploys a web app and an api route <strong>/metrics</strong>. This API route will scrape metrics and return it upon get request to the API route.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: appdeployment<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>        - name: my-app-container<br>          image: Philcz16/goapp<br>          imagePullPolicy: Always<br>          ports:<br>          - containerPort: 8080<br>         <br>---</pre><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: appservice<br>spec:<br>  selector:<br>    app: my-app<br>  ports:<br>    - protocol: TCP<br>      port: 80<br>      targetPort: 8080<br>  type: NodePort</pre><h4>Apply Configurations</h4><p>Prometheus and alert manager uses configuration files. These files are where you specify where Prometheus should scrape metrics from (In this case our app), for alert manager rules, and finally the endpoint to slack channel.</p><h4>Alert Manager Config</h4><p>Configure alert manger. In the provided repository, in <strong>configmaps.yml </strong>, replace your slack API URL in the manifest file. You can easily set up a slack URL <a href="https://amanranjanverma.medium.com/how-to-create-a-webhook-url-for-a-slack-channel-b5b58b1d208a"><em>here</em></a><em>.</em></p><pre>#configmaps.yml</pre><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: alert-rules<br>  namespace: monitoring<br>data:<br>  alert.rules: |<br>    groups:<br>    - name: example<br>      rules:<br>      - alert: PodDown<br>        expr: kube_pod_container_status_ready{namespace=&quot;prometheus&quot;, pod=&quot;appdeployment-5544868c9c-5wpl4&quot;} == 0<br>        for: 1m<br>        labels:<br>          severity: critical<br>        annotations:<br>          summary: &quot;Pod {{ $labels.pod }} is down&quot;<br>          description: &quot;The pod {{ $labels.pod }} in namespace {{ $labels.namespace }} has been down for more than 1 minute.&quot;n job has been down for more than 1 minute&quot;<br></pre><pre>---</pre><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: prometheus-server-conf<br>  namespace: prometheus<br>data:<br>  prometheus.yml: |<br>    global:<br>      scrape_interval: 15s<br>      evaluation_interval: 15s<br>    scrape_configs:<br>      - job_name: &#39;prometheus&#39;<br>        static_configs:<br>          - targets: [&#39;localhost:9090&#39;]<br>      - job_name: &#39;my-app&#39;<br>        static_configs:<br>          - targets: [&#39;my-app:8080&#39;]<br></pre><pre>---</pre><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: alertmanager-config<br>  namespace: monitoring<br>data:<br>  alertmanager.yml: |<br>    global:<br>      resolve_timeout: 5m<br>    route:<br>      group_by: [&#39;alertname&#39;]<br>      group_wait: 30s<br>      group_interval: 5m<br>      repeat_interval: 1h<br>      receiver: &#39;slack-notifications&#39;<br>    receivers:<br>    - name: &#39;slack-notifications&#39;<br>      slack_configs:<br>      - api_url: &#39;&lt;webhook-url&gt;&#39;<br>        channel: &#39;#alerts&#39;</pre><p>Next, apply the config maps manifest files.</p><pre>kubectl apply -n prometheus -f configmaps.yml</pre><p>Finally, on configuring prometheus, create the deployments and service for prometheus and alert manager</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: prometheus<br>  namespace: prom<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: prometheus<br>  template:<br>    metadata:<br>      labels:<br>        app: prometheus<br>    spec:<br>      containers:<br>      - name: prometheus<br>        image: prom/prometheus:latest<br>        volumeMounts:<br>        - name: config-volume<br>          mountPath: /etc/prometheus<br>        ports:<br>        - containerPort: 9090<br>      volumes:<br>      - name: config-volume<br>        configMap:<br>          name: prometheus-server-conf</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: prometheus<br>  namespace: prom<br>spec:<br>  selector:<br>    app: prometheus<br>  ports:<br>    - protocol: TCP<br>      port: 9090<br>      targetPort: 9090</pre><pre>---</pre><pre>#alertmanager<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: alertmanager-config<br>  namespace: prom<br>data:<br>  alertmanager.yml: |</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: alertmanager<br>  namespace: prom<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: alertmanager<br>  template:<br>    metadata:<br>      labels:<br>        app: alertmanager<br>    spec:<br>      containers:<br>      - name: alertmanager<br>        image: prom/alertmanager:latest<br>        volumeMounts:<br>        - name: config-volume<br>          mountPath: /etc/alertmanager<br>        ports:<br>        - containerPort: 9093<br>      volumes:<br>      - name: config-volume<br>        configMap:<br>          name: alertmanager-config</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: alertmanager<br>  namespace: prom<br>spec:<br>  selector:<br>    app: alertmanager<br>  ports:<br>    - protocol: TCP<br>      port: 9093<br>      targetPort: 9093</pre><p>Confirm you have your app’s pods running</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*4j65K6vgyXRDu1a6.png" /></figure><h4>Testing Alert Manager</h4><p>To demonstrate alert manager’s power, let’s scale down the Golang application deployment pods to zero.</p><pre>kubectl scale deployment appdeployment --replicas=0 -n prometheus</pre><p>Wait for at least 2mins and you should get a slack message in your channel notifying that the app is down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*7en4MRqgeqKVWcBT.png" /></figure><h4>Conclusion</h4><p>In this article, we demonstrated how alert manager can be used to send push alerts to your slack channel when your Kubernetes deployment is down.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=711387dbcaad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bootstrap Kubernetes Cluster with ArgoCD]]></title>
            <link>https://awstip.com/bootstrap-kubernetes-cluster-with-argocd-6ef0125626ef?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ef0125626ef</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 18:29:46 GMT</pubDate>
            <atom:updated>2025-01-07T12:04:33.298Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bootstrap EKS Cluster with ArgoCD using Terraform, Helm</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/0*KiQoRDhhqdE4mzv5.png" /></figure><p>Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Bootstrapping a Kubernetes (K8s) cluster with ArgoCD involves setting up ArgoCD to manage the applications running in your Kubernetes cluster from the very beginning, typically during the initial cluster setup stage.</p><p>In this article, we will learn how to bootstrap our Kubernetes cluster with ArgoCD. To achieve this, we will use terraform to provision a kubernetes cluster on AWS (EKS) and also use terraform to install ArgoCD helm chart to the cluster.</p><h4>Prerequisites</h4><p>To follow along, you’ll need the following installed/configured on the environment or machine you’ll work on.</p><ul><li>Terraform Installed</li><li>AWS CLI installed and configured</li><li>AWS Account and Access Keys Configured</li></ul><h3>Running Your Cluster</h3><p>For the sake of this project, we will clone my <a href="https://github.com/philcz16/manifests">repository</a> which contains Terraform files to configure EKS on AWS(other cloud providers can be used as well) and manifest files which will run on our cluster. Follow these steps:</p><ol><li>Clone the repository containing Terraform files.</li></ol><pre>https://github.com/philcz16/manifests<br>cd terraform</pre><p>Initialize, plan, and apply the Terraform files to provision your cluster on AWS:</p><pre>terraform init<br>terraform plan<br>terraform apply</pre><p>The terraform configuration provisions the following listed below:</p><ul><li>EKS cluster on AWS</li><li>Node and Node group using SPOT instances</li><li>Update Kube config on the machine you’re running terraform from.</li><li>Install ArgoCD helm chart on the provisioned cluster</li></ul><p>After updating your cluster to receive manifests configuration, apply the files in the manifest folder with the command below. The command applies a manifest of a golang application.</p><pre>kubectl apply -f app.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Mbsn8Qgc9GQWFO1V.png" /></figure><h3>Bootstrapping the Manifest with ArgoCD</h3><p>Once your manifest files have been applied, the next step is to bootstrap them to ArgoCD to enhance continuous delivery. Follow these steps:</p><p>Check that argocd is installed and the pods are running.</p><pre>kubectl get pods -n argocd</pre><p>when the pods are ready, map your port to access ArgoCD in the browser</p><pre>kubectl port-forward svc/argocd-server -n argocd 8080:443</pre><p>Upon access, you will be required to log in with a username and a password. The username is admin, and the password can be retrieved using the following command:</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot; | base64 -d</pre><p>When the password has been retrieved, install <a href="https://argo-cd.readthedocs.io/en/stable/cli_installation/">ArgoCD CLI</a> and login to ArgoCD with argocd login localhost:8080</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*-_Qe72L6Dd9Zkhz4.png" /></figure><h3>Adding Your Repository and Cluster to ArgoCD</h3><p>Once logged in successfully, connect the GitHub repo that contains the manifest with the following command:</p><pre>argocd repo add https://github.com/username/repourl --username &lt;your-github-username&gt; --password &lt;your-personal-access-token&gt;</pre><p>You will see the output below:</p><p><em>Repository ‘https://github.com/philcz16/manifests’ added</em></p><p><em>Note: To get your GitHub password, use your GitHub token, which can be generated in developer’s settings.</em></p><p>Once your repo has been connected successfully, add your cluster to the ArgoCD server using the following command:</p><pre>kubectl config get-contexts<br>argocd cluster add &lt;context-name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*12BRVcKpijneeXTu.png" /></figure><h3>Creating and Syncing Your Application</h3><p>Once the cluster has been added successfully, proceed to create your app and configure your ArgoCD using:</p><pre>argocd app create appname \<br>   --repo https://github.com/username/repourl \<br>   --path manifests/ \<br>   --dest-server https://kubernetes.default.svc \<br>   --dest-namespace argocd</pre><p>If the app was successfully created, you should get a message like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*W62FlJd4__zEC1ek_d2lqg.png" /></figure><p>Finally, sync your app using the following command:</p><pre>argocd app sync newapp</pre><p>If the sync is successful, you will get a response like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*rAE7idOFHWF7DL3v.png" /></figure><p>When you see this response, you can go on to Login into your ArgoCD via browser to check it out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*OzoubXCOQKRMWjH_.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ef0125626ef" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/bootstrap-kubernetes-cluster-with-argocd-6ef0125626ef">Bootstrap Kubernetes Cluster with ArgoCD</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install Kube-Prometheus-Stack, Loki, and Vault on Kubernetes using App of Apps Pattern and Helm…]]></title>
            <link>https://blog.devops.dev/install-kube-prometheus-stack-loki-and-vault-on-kubernetes-using-app-of-apps-pattern-and-helm-959d02f21959?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/959d02f21959</guid>
            <category><![CDATA[loki]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[argo-cd]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Mon, 03 Jun 2024 18:11:53 GMT</pubDate>
            <atom:updated>2025-01-07T12:04:51.115Z</atom:updated>
            <content:encoded><![CDATA[<h3>Install Kube-Prometheus-Stack, Loki, and Vault on Kubernetes using App of Apps Pattern and Helm Umbrella Charts with ArgoCD</h3><p>Using GitOps pattern in Kubernetes deployments is now a game changer in modern Infrastructure and complex application deployments on Kubernetes. Using tools like argocd to keep track of application in a git repository to keep the application in sync with the manifests stored in the repository enhances DevOps practices and Infrastructure as Code Practices.</p><p>In my previous article, I explained what argocd is and how to bootsrap a cluster using argo cd. In this article, we will see the limitations of deploying helm charts using argocd and how to go about it.</p><p>Also, to demonstrate this strategy, we will install Prometheus, Grafana, Loki and Hashicorp’s Vault using Argocd app of apps strategy.</p><h3>ArgoCD Recap</h3><p>What does this term mean in ArgoCD terms? Before we head there, let’s fist give a brief intro to what ArgoCD is. ArgoCD is a tool that ensures Continuos Delivery and Deployments to a kubernetes cluster. It is installed in the cluster as a kubernetes controller to keep track of kubernetes resources and compare the state of those resources with predefined written manifests. These manifests are the single source of truth for the application state and ArgoCD ensures that the application is in sync with what is defined in the Git SCM and the actual state in the cluster.</p><p>A typical example is when you store your kubernetes deployment file in git and you connect it with an argo cd app, ArgoCD ensures that the deployment is in sync in the cluster and what you predefined in your github repo. So even if you delete that deployment in the cluster, ArgoCD will create it back and ensure it is in sync with the predefined manifest in git.</p><h3>App of Apps</h3><p>In our previous article, we created a sample ArgoCD app using the ArgoCD CLI tool. You can also create apps using the console UI but think of a scenario where you have multiple applications to create? You’ll have to keep creating the apps using the UI or keep executing CLI commands to create these argoCD apps.</p><p>Another blocker is that creating these apps using the UI or CLI doesn’t follow GitOps practices. We need way to create our ArgoCD apps in a Git repository and also deploy kubernetes resources too from a Git repo.</p><p>That’s where app of apps comes in. It’s a strategy introduced by the Argocd team that allows you to create a single app either using the UI or CLI but the only job of that app is to pick up other ArgoCD apps manifests in a Github repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*-AAcVpjwLsDdpgzx.png" /><figcaption>App of Apps Strategy</figcaption></figure><p>This strategy ensures that all your argocd app configuration is stored in git and is not directly managed outside git. It ensures GitOps practices are followed and all application configurations are managed fully in git.</p><p>So what’s with the Umbrella chart strategy?</p><h3>Drawbacks of Deploying Helm Charts with ArgoCD</h3><p>Deploying Self managed Helm charts with ArgoCD is a game changer but there’s a draw back. ArgoCD does not allow you to pass in values file for helm charts that you don not manage yourself for example, let’s say you want to install a bitnami helm chart and you want to pass your own values file, ArgoCD app configuration doesn’t directly support that.</p><p>To pass helm values, you have to put it directly inside the argocd app configuration, which is not what we want. What if the values are so many, how do you manage indentation? What if you need to create multiple variations of the same chart with different values files?</p><h3>Solution: Using Umbrella Helm Charts</h3><p>To solve this problem, of passing value files to argocd, we make use of a strategy in Helm charts deployments called Umbrella charts. This strategy allows us to create a single helm chart and then install other charts from repositories by adding them as dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/0*cw8a5JE81ALy4w1X.jpg" /></figure><p>Adding these managed external charts as dependences to a primary chart allows us to seamlessly attach values files and then we can create argocd apps to point to those values files under the umbrella chart this will ensure resilience when managing these apps and you can seamlessly add your own custom values files to the argocd app manifest.</p><p>Let’s see this concept in action by deploying a demo. To follow along, ensure that you have a working kubernetes cluster. If you don’t, there’s a project repo provided that you can clone to get terraform files to spin up a cluster on AWS EKS for you.</p><h3>Prerequisites</h3><p>To follow along with this demo, you’ll need the following</p><ol><li>Kubernetes Cluster (Terraform is provided if you don’t have one)</li><li>Kubectl Installed</li></ol><p>If you don’t have a running cluster, you’ll need the prerequisites below:</p><ol><li>Terraform installed</li><li>AWS Credentials</li></ol><h3>Provision the Kubernetes Cluster (Optional)</h3><p>In this step, you provision a Kubernetes cluster using terraform on AWS EKS. The terraform configuration is in the terraform folder of the repo provided in this article in the introduction section.</p><p>Navigate to the directory and execute the commands:</p><pre>terraform init<br>terraform apply --auto-approve</pre><p>The terraform configuration creates a VPC, Subnets, EKS Cluster, Node Group, 3 Nodes (SPOT)</p><p>It takes a while to complete, around 20mins</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVpzx3bsWrTBEQwg2WjmZQ.png" /><figcaption>terraform apply</figcaption></figure><h3>Install ArgoCD</h3><p>Once the cluster is provisioned connect to the cluster using the command below</p><pre>aws eks update-kubeconfig --region us-east-1 --name realworld-cluster</pre><p>Once the context is updated, Install ArgoCD with kubectl using the command below:</p><pre>kubectl create namespace argocd</pre><pre>kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*kTdJAxGmt8xGgS7jCqcBww.png" /></figure><p>It will create a bunch of resources. Confirm if it got created successfully using the command below</p><pre>kubectl get pods -n argocd</pre><p>Ensure all pods are running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*mEnxGukFT6sZTiM_DJmj-Q.png" /></figure><p>For this demo, we will access the ArgoCD UI dashboard by using a port forward from the kubernetes Service to our local machines. This is because load balancers are expensive.</p><h3>Accessing ArgoCD UI Dashboard</h3><p>Use the command below to access argoCD UI on your local browser. Ensure that you keep the terminal running and open another terminal to execute other commands.</p><pre>kubectl port-forward svc/argocd-server -n argocd 8080:443</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*YWghEC1SuzojVPg1YDed-A.png" /></figure><p>With this, you can access argocd on your local browser on localhost:8080. If you get an error saying the SSL certificate is unsafe, select advanced and proceed to the argoCD site.</p><p>To get ArgoCD default password, execure the command below in another terminal</p><pre>kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=&quot;{.data.password}&quot;</pre><p>It prints the password as base64</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*UCVwojnVL0IiNJAT_ph6TA.png" /></figure><p>You can decode the password from base64 from the online base64 tool <a href="https://www.base64decode.org/">https://www.base64decode.org/</a></p><p>Use the decoded password to login to argocd using admin as the username.</p><h3>Create Umbrella Chart</h3><p>In this section, we will create the umbrella chart that has it’s child charts under. In the Directory you’re working on, initialize a helm chart using the command:</p><pre>helm create umbrella-chart</pre><p>This will initialize a sample nginx helm chart this chart is what we will modify and add other dependencies. Feel free to delete the template folder and the values.yaml file.</p><p>Edit the Chart.yml file and paste in the content below</p><pre>apiVersion: v2<br>name: umbrella-chart<br>description: A Helm chart for Kubernetes<br></pre><pre>type: application</pre><pre>version: 0.1.0</pre><pre>appVersion: &quot;1.16.0&quot;</pre><pre>dependencies:</pre><pre>- name: loki-stack<br>  version: 2.10.2<br>  repository: <a href="https://grafana.github.io/helm-charts">https://grafana.github.io/helm-charts</a></pre><pre>- name: vault<br>  version: 0.28.0<br>  repository: <a href="https://helm.releases.hashicorp.com/">https://helm.releases.hashicorp.com</a></pre><p>In the dependencies block, you can see that we add two external charts that we do not manage the loki-stack and hashicorp’s vault. The loki-stack is a grafana managed chart that installs grafana, loki, promtail and Prometheus including some alerting managers.</p><p>So here’s the fun part, the values file In the same umbrella helm chart folder, create a values file for loki and vault.</p><p>Paste the content below for loki.yaml</p><pre>test_pod:<br>  enabled: true<br>  image: bats/bats:1.8.2<br>  pullPolicy: IfNotPresent</pre><pre>loki:<br>  enabled: true<br>  isDefault: true<br></pre><pre>promtail:<br>  enabled: true<br>  config:<br>    logLevel: info<br>    serverPort: 3101<br>    clients:<br>      - url: <a href="http://{{/">http://{{</a> .Release.Name }}:3100/loki/api/v1/push</pre><pre>fluent-bit:<br>  enabled: true</pre><pre>grafana:<br>  enabled: true<br>  sidecar:<br>    datasources:<br>      label: &quot;&quot;<br>      labelValue: &quot;&quot;<br>      enabled: true<br>      maxLines: 1000<br>  image:<br>    tag: 10.3.3</pre><pre>prometheus:<br>  enabled: true<br>  isDefault: true</pre><pre>filebeat:<br>  enabled: true<br>  filebeatConfig:<br>    filebeat.yml: |<br>      # logging.level: debug<br>      filebeat.inputs:<br>      - type: container<br>        paths:<br>          - /var/log/containers/*.log<br>        processors:<br>        - add_kubernetes_metadata:<br>            host: ${NODE_NAME}<br>            matchers:<br>            - logs_path:<br>                logs_path: &quot;/var/log/containers/&quot;<br>      output.logstash:<br>        hosts: [&quot;logstash-loki:5044&quot;]</pre><pre>logstash:<br>  enabled: false<br>  image: grafana/logstash-output-loki<br>  imageTag: 1.0.1<br>  filters:<br>    main: |-<br>      filter {<br>        if [kubernetes] {<br>          mutate {<br>            add_field =&gt; {<br>              &quot;container_name&quot; =&gt; &quot;%{[kubernetes][container][name]}&quot;<br>              &quot;namespace&quot; =&gt; &quot;%{[kubernetes][namespace]}&quot;<br>              &quot;pod&quot; =&gt; &quot;%{[kubernetes][pod][name]}&quot;<br>            }<br>            replace =&gt; { &quot;host&quot; =&gt; &quot;%{[kubernetes][node][name]}&quot;}<br>          }<br>        }<br>        mutate {<br>          remove_field =&gt; [&quot;tags&quot;]<br>        }<br>      }<br>  outputs:<br>    main: |-<br>      output {<br>        loki {<br>          url =&gt; &quot;http://loki:3100/loki/api/v1/push&quot;<br>          #username =&gt; &quot;test&quot;<br>          #password =&gt; &quot;test&quot;<br>        }<br>        # stdout { codec =&gt; rubydebug }<br>      }</pre><pre># proxy is currently only used by loki test pod<br># Note: If http_proxy/https_proxy are set, then no_proxy should include the<br># loki service name, so that tests are able to communicate with the loki<br># service.<br>proxy:<br>  http_proxy: &quot;&quot;<br>  https_proxy: &quot;&quot;<br>  no_proxy: &quot;&quot;</pre><p>Enter the values content for vault.yaml file</p><pre>server:<br>  affinity: &quot;&quot;<br>  ha:<br>    enabled: true<br>    raft: <br>      enabled: true</pre><p>Your folder structure should look match the screenshot below assuming you deleted the template folder and the values folder of the default chart configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*ChZ33fIx5UJjjvAh0yGT4A.png" /></figure><p>With this configuration, we have successfully created a helm umbrella chart with two dependencies.</p><h3>Create ArgoCD Apps</h3><p>In this section, we will create the ArgoCD applications one for the Loki stack and another for Hashicorp vault. Create a folder and name it /apps . In the apps folder, create a file loki.yml file and paste the content below:</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: loki<br>  # annotations:<br>  #   argocd.argoproj.io/sync-options: ServerSideApply=true<br>spec:<br>  destination:<br>    name: &#39;&#39;<br>    namespace: monitoring<br>    server: &#39;https://kubernetes.default.svc&#39;<br>  # syncPolicy:<br>  #   syncOptions:<br>  #   - Replace=true<br>  source:<br>    path: umbrella-chart<br>    repoURL: &#39;https://github.com/philcz16/app-of-apps&#39;<br>    targetRevision: HEAD<br>    helm:<br>      valuesFiles:<br>        - loki.yaml<br>  sources: []<br>  project: default</pre><p>You can see that in the argocd manifest, we point at the github repository in the umbrella-chart folder. We then target the loki.yml values files we created earlier. You can see that using this method, we can easily specify values files for applications we don’t specifically manage in an ArgoCD app.</p><p>Let’s do the same for HashiCorp vault chart. Create a file in the same /apps folder and name it vault.yml</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: vault<br>  # annotations:<br>  #   argocd.argoproj.io/sync-options: ServerSideApply=true<br>spec:<br>  destination:<br>    name: &#39;&#39;<br>    namespace: default<br>    server: &#39;https://kubernetes.default.svc&#39;<br>  # syncPolicy:<br>  #   syncOptions:<br>  #   - Replace=true<br>  source:<br>    path: umbrella-chart<br>    repoURL: &#39;https://github.com/philcz16/app-of-apps&#39;<br>    targetRevision: HEAD<br>    helm:<br>      valuesFiles:<br>        - vault.yaml<br>  sources: []<br>  project: default</pre><p>Finally, let’s create the ArgoCD root app that will create the other apps this concept is the app of apps strategy. Head over to the ArgoCD UI (localhost:8080 if you are still using the port forward) create a new app, and select Edit as YAML. Paste the Yaml content below</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: app-of-apps<br>  namespace: argocd<br><br>spec:<br>  destination:<br>    namespace: argocd<br>    server: https://kubernetes.default.svc<br>  project: default<br>  source:<br>    path: apps<br>    repoURL: https://github.com/philcz16/app-of-apps<br>    targetRevision: HEAD</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fvxu90xLlc9KApchyeW-7w.png" /></figure><p>Click Save and click create. For this demo, the sync policy is set to manual, so we will sync the application using the ArgoCD UI. You can see that ArgoCD has picked up the two apps we created in git that will install the helm charts we specified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTPqBL1eBAJhQRTsbHNvtw.png" /></figure><p>Before we Sync the loki stack and the vault app, create the monitoring namespace that the loki stack will create resources into.</p><pre>kubectl create ns monitoring</pre><p>Click on the loki application and select click on Sync. Ensure that you check Server-side Apply. Click on Synchronize. This will create all the monitoring resources in the kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*_Lz94F1IP9r2gDjskwc8mQ.png" /></figure><p>Once the sync is completed you should get a response of Sync OK</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*caV9AK6ICH23ZKQSqDbYTw.png" /></figure><p>We can confirm the monitoring resources were created successfully using the command below:</p><pre>kubectl get all -n monitoring</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*cscn4BBofR6FdHanUrmPfA.png" /></figure><p>You can see that all loki resources have been created successfully. Next, let’s proceed to create Vault resources. Select the Vault app and click on sync. Also ensure you check Server side apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*2Rg-cJPWRRWffqM-2GA77w.png" /></figure><p>To confirm that vault resources are created successfully, check resources created in the default namespace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*kQNAXFlG9OAl9gOYQhiRmg.png" /></figure><p>Finally, to demonstrate the GitOps workflow, let’s add one more app to the umbrella helm chart that will deploy Grafana and Prometheus. Add the code block below to the Chart.yml file under dependencies to deploy Grafana and Prometheus.</p><pre>- name: kube-prometheus-stack<br>  version: 59.1.0<br>  repository: https://prometheus-community.github.io/helm-charts</pre><p>Also add a values file named kube-prom.yml you can use <a href="https://gitlab.tikalk.dev/matan.amiel/kube-prometheus-stack/-/blob/main/values.yaml">this</a> values file content or the one in the article repository.</p><p>Create an argocd app in the /apps folder by creating a file kube-prom.yml and paste the content below.</p><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: kube-prometheus<br>  # annotations:<br>  #   argocd.argoproj.io/sync-options: ServerSideApply=true<br>spec:<br>  destination:<br>    name: &#39;&#39;<br>    namespace: monitoring<br>    server: &#39;https://kubernetes.default.svc&#39;<br>  # syncPolicy:<br>  #   syncOptions:<br>  #   - Replace=true<br>  source:<br>    path: umbrella-chart<br>    repoURL: &#39;https://github.com/philcz16/app-of-apps&#39;<br>    targetRevision: HEAD<br>    helm:<br>      valuesFiles:<br>        - kube-prom.yaml<br>  sources: []<br>  project: default</pre><p>You can see that the app points to the kube-prom.yaml values file in the umbrella-chart folder. Save the changes and push to the git repository.</p><p>Upon pushing new changes, sync the app of apps you have in ArgoCD notice that argocd has picked up the new kube-prometheus app we added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*uG3Y43zzoCHKEL_n_5XYIw.png" /></figure><p>Sync all applications individually from ArgoCD UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*wXE5IqbM56GquSyPk4Wj8A.png" /></figure><h3>Clean Ups</h3><p>If you followed along using the terraform script provided in the repo codebase, navigate to the /terraform folder and execute the command below:</p><pre>terraform destroy --auto-approve</pre><p>The command removes all resources created using terraform.</p><h3>Conclusion</h3><p>In this article, we saw the draw backs of specifying custom values files into helm charts we do not manage when deploying to a cluster with ArgoCD. Using the umbrella chart method ensures full control and flexibility of installing public helm charts into a kubernetes cluster with argoCD.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=959d02f21959" width="1" height="1" alt=""><hr><p><a href="https://blog.devops.dev/install-kube-prometheus-stack-loki-and-vault-on-kubernetes-using-app-of-apps-pattern-and-helm-959d02f21959">Install Kube-Prometheus-Stack, Loki, and Vault on Kubernetes using App of Apps Pattern and Helm…</a> was originally published in <a href="https://blog.devops.dev/">DevOps.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy Helm chart to EKS Cluster using CICD pipeline with GitHub Actions]]></title>
            <link>https://medium.com/@phil16terpasetheo/deploy-helm-chart-to-eks-cluster-using-cicd-pipeline-with-github-actions-4ed098da8dab?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ed098da8dab</guid>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Wed, 15 May 2024 15:50:06 GMT</pubDate>
            <atom:updated>2025-01-07T12:05:15.656Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In the previous article, we built a CICD pipeline that builds a docker image and pushes it to DockerHub. In this article, we will create a helm chart and deploy our docker image to it. We will also extend the pipeline to automate the deployment of the helm chart and update the image to the latest version.</p><p>Repo link: <a href="https://github.com/philcz16/github_actions_eks_helm">https://github.com/philcz16/github_actions_eks_helm</a></p><p><a href="https://medium.com/@phil16terpasetheo/build-a-ci-cd-pipeline-using-github-actions-on-an-aws-ec2-self-hosted-runners-1abf30078e71">Build a CI/CD Pipeline using Github Actions on an AWS EC2 self hosted runners.</a></p><h4>What is Helm?</h4><p>Helm is a tool that helps you to create, package, and manage Kubernetes applications. It combines all the configuration files needed to deploy your app into a single reusable package.</p><p>You can see it as the package manager of Kubernetes. For example chocolatey for Windows, homebrew for Mac, apt for Linux and Helm for Kubernetes.</p><h4>EKS?</h4><p>Yeah! EKS (Elastic Kubernetes Service). This is AWS’s managed Kubernetes service, it’s so cool that you don’t have to manage the control plane yourself but rather have AWS manage it for you with High Availability.</p><p>So all we have to do in EKS is specify the number of nodes (EC2 Instances), and the instance type. It gets more complex based on requirements, features, and other factors such as application workload type, scaling, load balancing etc.</p><h3>Prerequisites</h3><p>To follow along with the article, you will need the following:</p><ul><li>AWS Account</li><li>Helm Installed</li><li>EKS Cluster Running</li></ul><blockquote>Note: If you don’t have a cluster running, a terraform script is provided to provision the cluster for you.</blockquote><ul><li>GitHub Account</li><li>Terraform Installed</li></ul><h3>CICD Configuration</h3><p>You can clone the repo provided and follow along with the demo, or you can delete the <em>/helm-chart </em>and<em> /.github, </em>folders. This section is broken down into three sections — Helm Chart, Terraform Infrastructure and the CICD pipeline.</p><h3>Helm Chart</h3><p>Helm as a package manager provides us with template we can use, then edit it to fit our application. To create the helm chart for golang app, create a folder in the code base directory and name the folder <em>helm-chart . </em>The folder will be where we will create the helm chart.</p><p>Switch to /helmchart folder in your codebase workdir and execute the command:</p><pre>helm create docker-cicd</pre><p>In the command above will create a helm chart named docker-cicd. The chart will be a folder containing some demo template files, below is a screenshot of my file tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*BgyW6kSiHQQo0_ETnmE8yA.png" /><figcaption>Full workdir</figcaption></figure><p>After creating helm repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*A4CmsEbC8L6AUAI7yS0B-A.png" /></figure><p>By default, helm will initialise a chart that deploys nginx. In our case, we need to deploy our own image not nginx. To do this, edit the <em>values.yaml</em> file in line 7, to 11, edit to the code snippet below:</p><pre>image:<br>  repository: &quot;username/imagename&quot;<br>  pullPolicy: IfNotPresent<br>  # Overrides the image tag whose default is the chart appVersion.<br>  tag: &quot;latest&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*plGmAbvPEkROwI1Jycahow.png" /><figcaption>Screenshot of my codebase</figcaption></figure><p>The code block, removes nginx image and replaces it with <em>username/imagename</em>. Don’t worry about it being invalid the plan is to get the pipeline to automatically fill in the username, image name, and tag name so just let the <em>username</em> and <em>imagename</em> be like that, don’t edit it to your own.</p><p>Next, let’s configure the terraform infrastructure.</p><h4>Terraform Infrastructure (Optional)</h4><p>If you already have an EKS cluster running in AWS you can use, you can skip this part and jump straight to the GitHub actions section. Else, navigate to <em>/terraform</em> folder in the codebase directory and execute the command:</p><pre>terraform init</pre><blockquote>Note: Ensure you have set up your AWS CLI and access keys on your local machine before executing the terraform commands.</blockquote><p>Next, apply the terraform script:</p><pre>terraform apply --auto-approve</pre><p>The terraform script will create the following:</p><ul><li>VPC</li><li>Private &amp; Public Subnets</li><li>EKS Cluster</li><li>1 t2.medium node</li></ul><blockquote>Note: The terraform script will take some time to provision approximately ~10–15min</blockquote><h4>CICD Pipeline</h4><p>In the previous article we explained what CICD is and what github actions is. In this article, we dive straight to hands on how to use github actions.</p><p>In the project work directory, create a folder <em>/.github/workflows</em> don’t forget to add the dot before the github. In the workflows folder, create a yml file. In this article, we named it <em>deploy.yml </em>you can name it anything doesn’t matter.</p><p>In the yml file, paste the file below inside.</p><pre>name: Deploy to EKS</pre><pre>on:<br> <br>  push:<br>    branches: [ master ]<br>  workflow_dispatch:</pre><pre>env:<br>  EKS_CLUSTER_NAME: cicd-cluster<br>  AWS_REGION: us-east-1<br>  DOCKER_REPOSITORY: goapp</pre><pre>jobs:<br>  <br>  build:<br>    <br>    name: Deployment<br>    runs-on: ubuntu-latest</pre><pre>    steps:</pre><pre>    - name: Set short git commit SHA<br>      id: commit<br>      uses: prompt/actions-commit-hash@v2</pre><pre>    - name: Check out code<br>      uses: actions/checkout@v2<br>    <br>    - name: Configure AWS credentials<br>      uses: aws-actions/configure-aws-credentials@v1<br>      with:<br>        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}<br>        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}<br>        aws-region: ${{env.AWS_REGION}}</pre><pre>    - name: Login to Docker Hub<br>      uses: docker/login-action@v3<br>      with:<br>        username: ${{ secrets.DOCKERHUB_USERNAME }}<br>        password: ${{ secrets.DOCKERHUB_TOKEN }}</pre><pre>    - name: Build, tag, and push image to Dockerhub<br>      env:<br>        DOCKER_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}        <br>        IMAGE_TAG: ${{ steps.commit.outputs.short }}<br>      run: |<br>        docker build -t $DOCKER_USERNAME/$DOCKER_REPOSITORY:$IMAGE_TAG .<br>        docker push $DOCKER_USERNAME/$DOCKER_REPOSITORY:$IMAGE_TAG</pre><pre>    - name: Update kube config<br>      run: aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_REGION</pre><pre>    - name: Update Image tag<br>      env:       <br>        IMAGE_TAG: ${{ steps.commit.outputs.short }}<br>      run: |<br>        sed -i.bak &quot;s|latest|$IMAGE_TAG|g&quot; helm-chart/docker-cicd/values.yaml<br>    <br>    - name: update docker Username and Image name<br>      env: <br>        USER_NAME: ${{ secrets.DOCKERHUB_USERNAME }}<br>        IMAGE_NAME: ${{ env.DOCKER_REPOSITORY }}<br>      run: |<br>        sed -i.bak &quot;s|username|$USER_NAME|g&quot; helm-chart/docker-cicd/values.yaml &amp;&amp; \<br>        sed -i.bak &quot;s|imagename|$IMAGE_NAME|g&quot; helm-chart/docker-cicd/values.yaml</pre><pre>    - name: Deploy helm chart<br>      uses: WyriHaximus/github-action-helm3@v3<br>      with:<br>        exec: helm upgrade --install docker-cicd helm-chart/docker-cicd/</pre><p>Let’s explain each step in the pipeline.</p><ul><li>The first step get the short SHA of the github commit</li><li>The second step, clones the codebase into the pipeline runner. In this pipeline, we make use of github already provided runner</li><li>Third step configures AWS Access keys to authenticate to AWS EKS</li><li>Fourth step login to DockerHub account for authentication to push the image</li><li>Fifth step builds new versions of the golang application, tags it and push it into the DockerHub account authenticated in previous step.</li><li>Sixth step updates the kube config of the kubernetes cluster. The kube config provides the certificates and endpoint of the cluster running in EKS</li><li>Seventh Step uses a special linux tool called sed. The sed tool is used to replace certain texts in files in linux. In this step, we make use of the sed command to remove the latest tag in the helm chart and replace it with the github short SHA.</li><li>Eight step uses same sed command to dynamically replace your dockerhub username and image name. Remember we didn’t add our correct docker username and image name in the helm chart. This step will replace what’s there with the correct image name and username.</li><li>Finally, we deploy a new version of the helm chart using the helm upgrade command. We also include — install to install the chart if it is not existing in the kubbernetes cluster.</li></ul><h4>Secrets</h4><p>Don’t push your code yet, else it will trigger the pipeline and it will fail. Before we trigger the pipeline, we need to set up some github secrets head over you your github repo in Settings &gt; Secrets and Variables &gt; Actions</p><p>Click on New Repository Secret. Your secret name should match the below</p><p>AWS_ACCESS_KEY_ID — Your aws access key</p><p>AWS_SECRET_ACCESS_KEY — Your aws secret access key</p><p>DOCKERHUB_TOKEN — Add your dockerhub password/ token</p><p>DOCKERHUB_USERNAME — Add your dockerhub username</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*il_nOokVEcHAzwdcjAe67g.png" /></figure><p>After adding the repository secret, push the changes you have locally and check the actions tab for the pipeline trigger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgUMa1uwqAaD11SY8hC2gA.png" /><figcaption>Workflow pipeline</figcaption></figure><p>If successful, you should have a new image tag in docker hub and the app running in kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*osPKHvVufNoMIx-btDYDVw.png" /><figcaption>Pushed images with tags</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*g4XcOJF-fYsfDgLOeB9DVA.png" /><figcaption>pods deployed by helm chart</figcaption></figure><h4>Accessing the app</h4><p>The helm chart uses a loadbalancer service type in line 42 of the values.yaml file. So we can access the app using the load balancer endpoint provided by Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pqr8PVJqnnkDGqPM5nJCwA.png" /><figcaption>Load balancer endpoint</figcaption></figure><p>Head over to your bowser, paste the link and add the port 8080 behind the endpoint to access the application</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyd2IF4emNRm02MRHrR45A.png" /><figcaption>Access the App</figcaption></figure><h3>Clean Ups</h3><p>In this section, we will clean up the whole set up we created. To do this, we will first delete the helm chart which will remove everything about the app in the k8s cluster.</p><p>To delete the helm chart, on your local computer that is connected to the k8s cluster execute the command:</p><pre>helm uninstall docker-cicd</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*K3iLQHwf-lTS5vTkvrNeYA.png" /><figcaption>Uninstall helm chart</figcaption></figure><blockquote>If you don’t have the kubernetes cluster connected on local you can connect to it using command below</blockquote><pre>aws eks update-kubeconfig --name cicd-cluster --region us-east-1</pre><p>Finally, remove the EKS cluster created using terraform if you followed along with the terraform section:</p><p>In the /terraform folder of the codebase execute the command:</p><pre>terraform destroy --auto-approve</pre><p>The command will take a while to finish executing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*WjZvKuvv1DpJ3Cl8k2_7YQ.png" /><figcaption>terraform destroy</figcaption></figure><h3>Conclusion</h3><p>In this article, we learned how to configure a CICD pipeline that will be automatically be triggered when there’s a change in a GitHub repo. The pipeline builds new image versions and deploys a new version of a helm chart to Kubernetes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ed098da8dab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a CI/CD Pipeline using Gitlab CICD on a self-managed runner.]]></title>
            <link>https://medium.com/@phil16terpasetheo/build-a-ci-cd-pipeline-using-gitlab-cicd-on-a-self-managed-runner-8880856aa519?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/8880856aa519</guid>
            <category><![CDATA[gitlab-cicd]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Fri, 10 May 2024 19:10:18 GMT</pubDate>
            <atom:updated>2024-05-10T19:10:18.837Z</atom:updated>
            <content:encoded><![CDATA[<p>Following our previous article on how to build a <a href="https://medium.com/@phil16terpasetheo/build-a-ci-cd-pipeline-using-github-actions-on-an-aws-ec2-self-hosted-runners-1abf30078e71">CI/CD pipeline using Github Actions on a self-hosted runner</a>, we will explore doing the same thing but this time, we will be using Gitlab CICD</p><p>CI/CD is a continuous software development method involving ongoing building, testing, deploying, and monitoring of iterative code changes. Learning GitLab CI/CD is invaluable for catching bugs early in development and ensuring adherence to established code standards in production deployments.</p><p>In this article, we are going to setup a GitLab CI/CD to build and push a golang note taking application to a container registry. We are going to do this with the following steps.</p><ol><li>Setup Self-managed GitLab Runner</li><li>Create a .gitlab-ci.yml file and define pipeline jobs</li><li>View the status of our pipeline and jobs</li></ol><h3>Setup Self-managed Gitlab Runner</h3><p>A <a href="https://docs.gitlab.com/runner/"><strong>GitLab Runner</strong></a> is an application that works with GitLab CI/CD to run jobs in a pipeline. When you push code changes, GitLab uses the Runner to automatically build, test, and deploy your project according to the instructions you’ve defined in your CI/CD configuration.<br>To setup a self-managed runner follow the steps below:</p><figure><img alt="Gitlab runner screen" src="https://cdn-images-1.medium.com/max/1024/1*mcfmO0lOhFrJWjE9dVr5IQ.png" /></figure><ol><li>On the left sidebar, select <strong>Search or go to</strong> and find your project.</li><li>Select <strong>Settings &gt; CI/CD</strong>.</li><li>Expand the <strong>Runners</strong> section.</li><li>Select <strong>New project runner</strong>.</li><li>Select the operating system where GitLab Runner is install</li><li>In the <strong>Tags</strong> section, in the <strong>Tags</strong> field, enter the job tags to specify jobs the runner can run. If there are no job tags for this runner, select <strong>Run untagged</strong>.</li><li>Optional. In the <strong>Runner description</strong> field, to add a runner description that displays in GitLab, enter a runner description.</li><li>Optional. In the <strong>Configuration</strong> section, add additional configurations.</li><li>Select <strong>Create runner.</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*34hXXcdR73XDyUjtsuMAWw.png" /></figure><p>This should create a runner successfully, keep the authentication token show on the next screen in a safe place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CvrmU9OA4MSy9NAXVGJdcA.png" /></figure><p>Use the script below to install the gitlab runner package on the instance you want to use as a runner. In our case we will be using an AWS EC2 instance.</p><pre># Download the binary for your system<br>sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64<br><br># Give it permission to execute<br>sudo chmod +x /usr/local/bin/gitlab-runner<br><br># Create a GitLab Runner user<br>sudo useradd --comment &#39;GitLab Runner&#39; --create-home gitlab-runner --shell /bin/bash<br><br># Install and run as a service<br>sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner<br>sudo gitlab-runner start</pre><p>We can proceed to register our runner with the command. Replace the token with yours generated in the windwo above.</p><pre>gitlab-runner register  --url https://gitlab.com  --token glrt-2_xxxxxxxxxxxx</pre><p>We can see now, when you visit the runners page, you’ll see that the runner has been registered, indicated by the green circle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZAos89DnQtvPGWNnXXZFg.png" /></figure><h3>Create a .gitlab-ci.yml file and define pipeline jobs</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f05acd598a09d9b2e8b511ac33348a3/href">https://medium.com/media/5f05acd598a09d9b2e8b511ac33348a3/href</a></iframe><p>From the pipeline defined, our runner will pick up all jobs with tags <strong><em>self-hosted</em></strong> since this was the tag we defined for our runner.</p><p>Our pipeline has only one stage: <strong>build - </strong>in this stage the pipeline is defined to build a docker image and push that image to an image registry.</p><h3>View the status of our pipeline and jobs</h3><p>We can commit our changes and push the latest changes to the remote branch to trigger the start of the pipeline.</p><p>Select<strong> Build</strong> and then <strong>pipelines </strong>to see all the jobs for that repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DBAo8QNmwttljDXTz4cgPw.png" /></figure><p>I hope you followed all the instructions and everything works, cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8880856aa519" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a CI/CD Pipeline using Github Actions on an AWS EC2 self hosted runners.]]></title>
            <link>https://medium.com/@phil16terpasetheo/build-a-ci-cd-pipeline-using-github-actions-on-an-aws-ec2-self-hosted-runners-1abf30078e71?source=rss-d069ca822be1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1abf30078e71</guid>
            <category><![CDATA[cicd]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <dc:creator><![CDATA[Terpase Theophilus Gagher]]></dc:creator>
            <pubDate>Wed, 08 May 2024 22:35:20 GMT</pubDate>
            <atom:updated>2024-05-08T22:35:20.590Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview of CI/CD</h3><p>CI/CD which stands for continuous integration and continuous delivery/deployment automates all the processes involved in getting new code from a commit into production, the aim of CI/CD is to streamline and accelerate the software development lifecycle (SDLC)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*mplx5L5FM0zFK2-V" /></figure><p>In this article, we will learn how to implement a CI/CD pipeline using github actions on a self-hosted runner. We will automate the process of building a docker image, tagging the image and pushing to a container registry like <a href="https://hub.docker.com/">dockerhub</a>.</p><p>GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production.</p><p>GitHub Actions goes beyond just DevOps and lets you run workflows when other events happen in your repository. For example, you can run a workflow to automatically add the appropriate labels whenever someone creates a new issue in your repository.</p><p>GitHub provides Linux, Windows, and macOS virtual machines to run your workflows, or you can host your own self-hosted runners in your own data center or cloud infrastructure.</p><h3>Setup a self-hosted runner in the repo</h3><p>For the hosted-runner I will be making use of an AWS EC2 instance, you can read <a href="https://docs.aws.amazon.com/efs/latest/ug/gs-step-one-create-ec2-resources.html">AWS documentation</a> on how to provision an EC2 instance.</p><p>Let’s get to business. The first step is to go to your repository’s settings . Select runners under Actions and click on New self-hosted runner</p><figure><img alt="Create and add a new self-hosted runner" src="https://cdn-images-1.medium.com/max/1024/1*WSDPAR1Jkmod-WjA6VshZA.png" /></figure><p>As you can see, we have already connected our instance as a runner</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsRFwowc1mn319imD63m4g.png" /><figcaption>our EC2 instance has been connected as a github actions self-runner</figcaption></figure><h3>Create workflow and configure events to trigger</h3><p>First, create a yaml file stored in .github/workflows folder in the root directory of your repository. This example workflow will be trigger on push and pull requests on branch “main”</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58c5b58bbdf12ee96cb307da63e9f73f/href">https://medium.com/media/58c5b58bbdf12ee96cb307da63e9f73f/href</a></iframe><p>Next we will add some secrets contained in our workflow like DOCKER_USERNAME and DOCKER_KEY which are the username and password needed to login to dockerhub to push our image after building.</p><p>Go to your repository’s settings . Select secrets and variablesunder Actions and click on New repository secret to add the key and value.</p><h3>Conclusion</h3><p>In this article we learnt how to build a github actions CI/CD workflow running on a self-hosted runner that builds a docker image, tags the image and pushes to dockerhub. In coming articles we will be extending the pipeline to create a helm chart and deploy to an EKS cluster provisioned with terraform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1abf30078e71" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>