Following are the topics which are covered in this blog

• Manual Scheduling
• Taints and Toleration
• Node Selector
• Node Affinity

Manual Scheduling: in manual scheduling we schedule pod manually on node, to understand this concept we have to understand how kube-scheduler work, whenever we create a pod through definition file scheduler looks for the pod which have no “nodeName” property attached to them and ignored the pod which have on it, “nodeName” property is assigned under pod.spec section when we create our pod , if there is no nodeName specified then scheduler pick the pod and placed it on a best available node available at that time in k8-cluster. What if we specified nodeName while creating our pod the pod is ignored by the scheduler and it is directly placed on that node which is specified on it, by specifying nodeName we directly scheduled our pod on node of our choice. We can use manual scheduling when there is no scheduler available on k8-cluster in this situation pod will goes into pending state until and unless we specify the nodeName in pod definition file while we create it. Below is the example of definition file where we select node2 to put the pod on

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 8080
  nodeName: node2

What if we want to schedule the pod on specific node and we forget to mention that nodeName while creation and it’s already created, kubernetes wont allow us to edit the pod file and add nodeName property to it but we can achieve this by creating a binding object and then sending object data by a POST request to binding api through this way it mimics what actual scheduler do. Below is the example of binding object and the api request

apiVersion: v1
kind: Binding
metadata:
  name: nginx
target:
  apiVersion: v1
  kind: Node
  name: node2

##binding api requestrequest
$ curl --header "Content-Type:application/json" --request POST \
  --data '{"apiVersion":"v1", "kind":"Binding" ...}' \
  http://$SERVER/api/v1/namespace/default/pods/$PODNAME/binding

Taint and Toleration: is one of the main concept through which we can understand and learn about how nodes are restricted to deploy POD’s on it. To understand Taint and Toleration we draw an analogy by which it become easy to grasp the concept

Analogy: Suppose we went for camping and there are lot of bugs and mosquitos the first thing we will do we apply bug/mosquito repellent on ourself now all the species of bug and mosquito will not come on our body and bite us but some specie of bug and mosquito have toleration against that repellent it means that species of bug/mosquito can come on our body and bite us. As you can see i use the word a bug/mosquito who are tolerated to that repellent can come because there is a possibility that those species of bug and mosquito not come on our body and goes to some other person.

In Kubernetes we will replace ourself as a node, mosquito/bug as a pod, repellent as a Taint. If there is a taint apply on the node only the pod which have Toleration against that Taint can be deployed on that node. Having a Toleration against the Taint does not guarantee that scheduler will put the tolerated pod on that specific node which have Taint, but there is a possibility that it can be deployed tolerated pod on node which have same taint as the pod have toleration for.

Taint: Taint can only be applied on node, by default master-node in k8-cluster is tainted, this is the reason scheduler does not put any pod on it.

Taint Effect: a taint effect defines the action of scheduler in the case if the pod does not tolerate the node taint. There are 3 effects or ways of Pod scheduling which are applied with Taint on nodes. Following are the effects

• NoSchedule: In NoSchedule un-tolerated pods which cant tolerate the Taint which are applied on node are not scheduled by the scheduler. Existing pod will continue to run on the node regardless of the toleration exist on the pod or not
• PreferNoSchedule: The scheduler will avoid this node until and unless there is no node left to shceduled pod on. Existing pods which are running on that pod are un-effected
• NoExecute: In NoExecute not only the future pods effected but also the current pod will be evicted if they dont have the Toleration for that node, if it have a toleration it will continue to run. We can specify the eviction time on pod name by tolerationSeconds it will continue to run on node until the seconds not finished

Below is the command to Taint a node

##kubectl taint node node-name key=value:taint-effect
$ kubectl taint node k8-worker1 black-taint=black:NoSchedule

Toleration: Toleration can only be applied on pod, putting Toleration for any Tainted node does not guarantee that the pod will put on that node, it is only guaranteed if the pod is part of the daemonset. To create a toleration on pod we specify the Tolerations under pod.spec section in definition file. Below is the example of pod definition file which have toleration against the node2 on which we created Taint above

apiVersion: v1
kind: Pod
metadata:
  name: tolerated-nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 8080
  tolerations:
  - key: black-taint
    operator: Equal
    value: black
    effect: NoSchedule

Node Selector: Node Selector are used to select node for pod scheduling based on certain criteria which are defined inside pod definition file under spec section with the property name nodeSelector. We can select node based on the labels of node, then the scheduler search for these labels of node and match with those mentioned under nodeSelector if any node with same label found the scheduler deploy the pod on that specific node if not then pod is deployed on any node. If multiple node found with same label then scheduler can choose the best one based on it’s algorithm.

Prior to define the nodeSelector in pod definition file we must ensure that the node with same label node exist in our cluster, to label the node we can use the below commands. We can define multiple labels on single node

## choose one of your node and labeled it
$ kubectl label nodes k8-worker1 disktype=ssd

##overwrite the label of same node with different value
$ kubectl label --overwrite nodes k8-worker1 disktype=hdd

##comfirm if the label is applied on your node, getting all node in default namespace with lables
$ kubectl get nodes k8-worker1 --show-labels

##to check if the node with the label exist in our cluster
$ kubectl get nodes --show-lables | grep -i hdd

##To remove the label from a node, provide the key without any value
$ kubectl label --overwrite nodes k8-worker1 disktype-

Below is pod definition file with nodeSelector, we have label k8-worker2 node as disktype=hhd

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 8080
  nodeSelector:
    disktype: hhd

PS: Node selector have some limitation like we cant instruct scheduler to place pod on one of the node based on multiple selector, like selecting node which have cpu=8 or cpu=10

Node Affinity: Node Affinity is same as the node selector but with overcoming the limitation which are in the node selector, nodeAffinity provides advanced capabilities to select node based on there labels and can defined multiple labels, scheduler search for the node labels and place pod on the node which have matched labels. It is defined in the pod definition under spec.affinity there are multiple keys defined under it, to understand we will see the the structure of file then dig into each key

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:alpine
    ports:
    - containerPort: 8080
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - hhd
            

requiredDuringSchedulingIgnoredDuringExecution: This is the type of nodeAffinity, it is not so difficult to understand this key, this key instruct the scheduler that matched node label is required when pod is scheduling and if after matched node label is removed after the pod is scheduled then ignore it and don't evict the pod it will continue to run on that node

preferredDuringSchedulingIgnoredDuringExecution: This is the second type of nodeAffinity, as the key is self explanatory it instruct the scheduler that matched node label are preferred not mandatory during scheduling of pod and once it scheduled on pod later if the node label is removed it will ignore it and countinue to run pod on that same node. We use this type where the pod placement is more important then the node which is it being placed on

matchExpression: it is used to instruct scheduler to check the node labels against the match expression which are combination of key, operator and values. The matchExpression can be one or multiple

key: key of the node selector

operator: operator is the criteria of matching nodes label with defined key value, following are the operators(In, NotIn, Exists, DoesNotExist)

values: values of the node selector, it can be one or multiple

Conclusion: In this article we have built a good understanding of pod scheduling and different technique to scheduled pod in Kubernetes.

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi

Introduction : In modern tech infrastructure provisioning and managing it have become easy this can be done through writing some instruction in a specific format and then executing those configuration files, there are multiple tools which can be used for provisioning infrastructure but so far Terraform by Hashicorp is one of the best tool for IAC(Infrastructure As Code). In this article we will understand the core concepts of terraform from basics then we advanced our concepts in 2 separate article in which we will discuss more advance topics

Following are the topics which will be covered in this article

• What is IAC and it’s types
• What is Terraform and it’s 3 phases
• Mutable vs Immutable Infrastructure
• Installing Terraform on your local machine
• HashiCorp Configuration Language (HCL) and Configuration Directory
• Terraform Providers
• Write → Initialize → Plan → Apply → Update → Destroy Infrastructure
• Terraform State and it’s purpose
• Terraform State Commands

What is IAC:

Background: Before cloud computing traditional infrastructure is provisioned by setting up physical hardware resources in their on premises data center, the main drawback in managing physical on premises data center is that if we have to provisioned some computational or hardware resources it will take a good amount of time which became a bottleneck for the organization who need to scale up and down rapidly.

Then comes the cloud provider’s organization which need to scale fast shift from on premises data center to cloud providers like(AWS, GCP, AZURE etc..) cloud providers give flexibility to orchestrate resources whenever they want these resources will be provisioned in few minutes with some click on their dashboard which saves lot of time and managing cost.

Till now seems like cloud providers solves our problems related to saving time and managing cost but let give it a second thought what if these organization have large infrastructure spans around multiple cloud provider including on premises resources and they have to update or orchestrate their infrastructure often according to their needs, by going on to the dashboard of each cloud provider and changing resources will become cumbersome and time consuming too.

IAC: What if all the resources are provisioned and scaled by executing a file which contain all the instruction for creating and updating infrastructure in it and these instruction write in specific format is called Infrastructure As Code (IAC). Using IAC we can manage any infrastructure components as code such as (Database, Computation, Storage, Network e.tc) and also the application configuration.

There are multiple tools to address IAC which can be broadly divided into 3 main type based on their functionality

Configuration Management: The purpose of configuration management tool is to configure application and services on existing infrastructure and also to monitor resources on these, it can be run on multiple remote resources at once. These tools are idempotent means once anything is configure on a resource it will not configure once it run again. Tools which are fall into this category are (Ansible, Chef, Puppet, Saltstack).

Server Templating: The purpose of sever templating tool to spinup servers which are identically configured to a defined standard, it is used to create custom images of a machine which contains pre installed application and services in it. Infrastructure which is orchestrated using server templating are immutable unlike configuration management tool. Tools which fall into this category are (Docker, Vagrant and Packer)

Provisioning : These tool are used for provisioning infrastructure and it component on cloud such as (Database, Computation, Storage, Network e.tc) multiple resources can be provisioned in one go using these tools. Tools which fall into this category are (Terraform, CloudFormation), in this article we will discuss one of the provisioning tool Terraform in detail.

What is Terraform

With terraform we can write our infrastructure using dedicated configuration language called HCL and then saved it in a .ts file once we execute these file our infrastructure is provisioned exactly as defined in configuration file. Terraform allows us to build, manage and destroy infrastructure in matter of minutes, these infrastructure component can be on private-public cloud or on prem, it can also manage low level component like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.

Every object which terraform can manages is called resource and terraform manages it resource using providers through it’s API, providers helps terraform to work with any platform or services using it’s API, we have discuss providers in detail in different section below.

3 Phases: The core terraform workflow consist of 3 phases Write, Plan and Apply. First we write our infrastructure in Hashicorp Configuration Language then we plan our infrastructure in which terraform creates an execution plan of how resources are created or updated based on the configuration file and finally we apply our plan in this stage terraform execute the plan which is created earlier by performing each operation in correct order.

Mutable Infrastructure:

In mutable infrastructure the component or the services running on it can be me modified without recreating the resource again. For instance if we have to update nginx version on 4 of our server we can do it by only updating the nginx version without recreating server with newer nginx version. By this approach a configuration drift can happen if one of the servers fail to update the nginx software then it leave infrastructure in unstable state

Immutable Infrastructure:

In immutable infrastructure service or software can’t be updated or changed in any form until and unless it’s underline component is also recreated with the updated changes. For instance if we have to update new version of nginx on to our 4 servers with immutable infrastructure the server will be recreated with new version of nginx, by this approach it minimizes the configuration drift and become easy to manage underline infrastructure resources.

Installing Terraform on your local machine:

Installing terraform on your machine is an straight process where we download the official binaries from terraform and then install it using package manager of our system, below is the command and result snippet of installing terraform in our Ubuntu machine
You can also refer to official guide to install terraform on your machine
https://developer.hashicorp.com/terraform/install

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

HashiCorp Configuration Language (HCL):

HCL is an language format of Terraform in which we write instructions for creating, updating and removing resources of our infrastructure, file in which these instruction are written is called configuration file and have a .ts extension. We can name these file with any valid string but the default configuration file is called main.ts

HCL file mainly contain resource block these blocks defined what attributes our resource will have and these attributes can be vary between different resources provided by the providers(plugins) but syntax of resource block are mainly the same, resource block is used to create and update different resources on our infrastructure these resource can be from creating or updating EC2 instance on amazon to creating or updating local_file on the server instance. Below is the syntax of a resource block

Configuration Directory

The directory where configuration file are located is called terraform configuration directory, all the commands should be run within the configuration directory. Configuration directory can contain multiple configuration, variable files but it contain only single terraform.tfstate and plan file. Configuration Directory is the main directory where all the configuration file repleted to our architecture is located.

Terraform Providers :

In order to manages resources terraform uses plugins called provider which interact with the terraform API in order to create, update and delete resources and services on it. A resources manages by terraform can be on cloud, on prem or locally in order to manage these resources terraform uses different providers which are responsible for interacting with specific resource or service like aws plugin interact with resources on AWS cloud whereas local plugin allows us to interact with local resources like files on local system.

There are 3 tiers of providers, below are the details

Official Provider: These are the providers which are officially maintain by hashicorp itself

Partner: These are the partner of hashicorp and participates in Hashicorp Technology Program, plugins are maintained, managed, validated and publish by partner themself

Community: Providers which are maintain and publish by individuals maintainers or by the group of maintainers into terraform registry

To check all the providers and their documentation go to official website of terraform registry [https://registry.terraform.io/]

Write → Initialize → Plan → Apply → Update → Destroy Infrastructure

Write: We start by writing infrastructure in a defined hcl format in a .ts extension file, hcl file mainly contains the resource blocks which instruct how to create or update resources in our infrastructure other block may include data source block or provisioner block in our hcl file. Below is the example of hcl file name main.tf we have used local_file provider

resource "local_file" "rand_new_pet"{
  filename = "/root/my-files/my-pet.txt",
  content= "My new pet name is max"
}

Initialize: In order to orchestrate or manage any resource in our infrastructure we first required the respective providers into our configuration directory. When we run the initialize command terraform creates a new secret directory name .terraform in this secret directory all our required plugins are placed, below are the command and snippet

## initializing terraform config directory by downloading required provider
$ terraform init

Plan: After initializing the configuration directory we creates plan by running the blow command, when the command is run terraform creates an execution plan of how to execute instruction which are specified in configuration file(hcl), below are the command and snippets

## creating plan for creating resources defined in our config file, 
## the -out options is used to store the execution plan in a file, its optional
$ terraform plan -out local-infra-plan

Apply: Once we plan our infrastructure now we can apply changes in real world by executing our plan with apply command, through this command our changes will be reflected in matters of minutes, below are the command and snippets

## applying plan which we just saved in a file, 
## if we have saved plan we can pass it if not then apply can be executed without it
$ terraform apply local-infra-plan

PS: Every resources created by terraform have a unique id which helps terraform to manage them, we can see those id when resource is created or we can check it later in terraform.tfstate file

Update: If in future we need to update any resource in our infrastructure this can be easily done by following the core work flow of terraform again which is changing the configuration file then plan it and finally apply it (Change → Plan → Apply), below are the snippets

Whenever new changes are being done the resource is recreated because of terraform immutable infrastructure, we can verify it by seeing the difference in resource id which terraform assign for every new resources

Destroy: If we want to destroy our whole infrastructure define in configuration file located at configuration directory we can do so by using one simple command destroy, keep in mind this command will remove all our infrastructure resources in one go so it’s better to create destroy plan first and then execute this plan, below are the command and snippets

## we can destroy our infrastructure by using simple destroy command, 
## but it's better to create a destruction plan save it in a file and then apply that plan
$ terraform plan -destroy -out local-infra-plan 

$ terraform apply local-infra-plan

Terraform State and it’s purpose

Terraform maintain a local state of infrastructure which hold all the details of every resources which are in real world, the state is created when we run apply command first time and stored it in a file name “terraform.tfstate”. The state files is in json format which maps every detail of real world resources in it, this state file is consider as a single source of truth, once terraform.tfstate file created by default it refreshes every time whenever plan or apply command run and matches the changes in configuration file with the current refresh state this procedure helps terraform to identifies the changes which are being done on our infrastructure and once new configuration apply then terraform changes it state file with the updated changes.

A state file is considered as a blue print of our real world infrastructure, it eliminates the configuration drift between the real world resources and configuration file if any drift exist terraform state helps to create an execution plan based on the difference which eliminates the configuration drift. State file not only maps the resources itself but also maps the metadate of it like resource dependencies, we will learn more about resource dependency in chapter2

We can use terraform state for increasing the performance as well like whenever we apply new resources or create plan terraform refresh the state by reconciling it local state from the real world infrastructure if our infrastructure is large having ton of resources then it can take lot of time to get the updated state from our infrastructure instead reconciling every time from real world infrastructure we can use terraform.tfstate file for reconciliation. By using — refresh false option with every commands, we can skip refresh and local state is used for reconciliation

PS: Every user who are working on same infrastructure should have latest terraform.tfstate file on their machines, we should never share these file through version control system such as git or bitbucket instead it should be posted on remote data storage like s3 or terraform cloud because state file is sensitive in nature and holds secrets of our infrastructure

Terraform State Commands:

Terraform state file is in json format and should not be edited manually instead terraform provides sets of command to interact or edit with it’s state file. Command which is used is terraform state state followed by sub command which are described below with result snippet

## it will only print the resource address in current state 
$ terraform state list

## it will show the detail of specific resource by providing resource id
$ terraform state show local_file.rand_new_pet

## moving item within same state file or moving resources from one state file to another state file
## one usecase of this command is to renaming the resource, below is the commnad. 
## After state file change we have to reanme resource name manually in config file too
$ terraform state mv local_file.rand_new_pet local_file.rand_old_pet

## it used to download and display the current remote state of our infrastrucuture
$ terraform state pull

## it is used to delete item from state, this command is used if we dont want to mannage the resource through terraform
## the resource itself will not be removed by this command
$ terraform state rm local_file.rand_new_pet

Conclusion: In this article we have discuss core concepts of terraform, so far we understand how terraform works and what is the workflow of terraform by implementing basic infrastructure on local machine in par2 and advance topic we will create infrastructure on cloud and discuss more advance topics until the stay tuned…

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi

Introduction This project is an actual depiction of setting up the production pipeline and architecture by following the best DevSecOps practices and tools, we have used Kubernetes to deploy our application, Ansible to configure our k8-cluster and configure many other tools. In this project we have cover a wide array of topics and concept which are essentials for setting up and configuring tools for production. This project perfectly covers all the aspect of modern DevSecOps practices.

Prerequisites: There is no compulsory prerequisites concept in this project, but a basic understanding of CI/CD pipeline including Kubernetes and Ansible is compulsory. We will use AWS for provisioning our servers but you can use any cloud provider. Dockerhub and Gmail account is required

Following are the topics which are covered in this article
1. Setting up EC2 instance
2. Installing and Configuring Jenkins on Jenkins/Ansible server
3. Installing Docker
4. Installing Sonarqube
5. Installing Trivy
6. Installing and configuring Ansible Control node
7. Configuring Kubernetes Cluster through Ansible
8. Setting up monitoring-server using Prometheus and Grafana
9. Configuring node exporter on jenkins server
10. Configuring node exporter as a daemonset on k8-cluster
11. Setting up CI/CD Pipeline

Lets Dive into Building

First we need to orchestrate our ec2 instances, we will spin up 4 instance with following configurations

• Jenkins Server/Ansible Control Node(t2.large, 35Gb): On this server we will install and configure multiple tools like jenkins, ansible, sonarqube, docker, kubectl and trivy
• Monitoring Server(t2.medium, 15Gb): This server is solely responsible for monitoring, we will install and configure Prometheus, Grafana and node exporter
• Kubernetes Master Server(t2.mdeium, 8Gb): On this server we will configure k8 master node through ansible playbook, it will act as an ansible managed node and the k8-master-controlplane
• Kubernetes Worker Server(t2.medium, 8Gb): This server will act as an k8 worker and the managed ansible node, we will install and configure k8 worker node through ansible playbook

Before we start orchestrating the instances on AWS we will select the nearest region from the “region selector” located at top right corner, In my case “Canada Central” is the nearest one, select according to yours

Create Security Group: Next we will create security group for our server, to control inbound and outbound traffic. All 4 instance will share same security group all inbound TCP traffic is allowed and the SSH access will only be allowed to our home ip address

Go to search bar and search EC2 and then from ec2 dashboard select Security Group

Next add basic details like name and description of security group “plg1-sg”

After creating security group edit inbound rules, allow all inbound TCP traffic and allow SSH to only your ip

Setting up EC2 instance: Now we will start creating our ec2 instance, i have show steps to spin up 1 instance “jenkins-server”, all other 3 ec2 instance can be orchestrated following exactly same steps with their own instance name, type and storage

From the ec2 Dashboard select “Launch Instance”

Then add the instance detail like instance name “jenkins-server”, os “ubuntu 22.04”, instance type “t2.large”

Then from key pair section, we will create a new key pair and name it “plg1-ec2” then select the key format i have selected .ppk key format you can select either .pem or .ppk. Now we will create the key and save it on our local machine, it will be showed once so keep it safely. This key is used for connecting to our server, i will use this same key for all our instances

Now from the network settings section, “Select existing security group” and from drop down select the one which we created earlier named “plg1-sg”

Finally edit storage for our jenkins server we need atleast 35Gb storage and the click on launch instance

Now spin-up 3 more ec2 instances named “monitoring-server”, “k8-master” and “k8-worker” . At final all the 4 server will look like below

After orchestrating all 4 servers, we have to configured our server and install all the required tools and packages on each of them. First we will start from jenkins server

Login into jenkins server using the .ppk key, from “putty” software on your local machine, open it specify public ip in our case it is 3.96.135.216 then from left pane select Connection->SSH->Auth->Credentials and specify private key which we generated earlier and then select open button

Installing and Configuring Jenkins: On jenkins server we will start by changing hostname and then installing jenkins on it, write following command on the shell as root user

##changing hostname
hostnamectl hostname jenkins-server

##Installing JDK
sudo apt update -y
sudo apt install openjdk-11-jre -y
##check JDK version
java --version

##Installing Jenkins
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y

After installing, we can see jenkins service is running

Now we have to configure jenkins admin user and plugins on it, in order to do so we need to “Unlock Jenkins”, to unlock it we need to get the “initialAdminPassword” it’s located at /var/lib/jenkins/secrets/initialAdminPassword to get the pswd type below command

Now open the jenkins UI on web-browser by typing jenkins url which will be your server ip followed by the port 8080, in my case it is “3.96.135.216:8080” and then pass the initial-admin-pswd

Now Click on Install Suggested Plugin

After suggested plugin installed, create your first admin account and click Save and Continue

At last add your jenkins url and click Save and Finish

Installing Docker on jenkins-server, write below command on shell

##Installing Docker
sudo apt update
sudo apt install docker.io -y

##Adding jenkins and ubuntu user into docker group
sudo usermod -aG docker jenkins
sudo usermod -aG docker ubuntu

##Restarting docker service and changing docker.sock file privilege
sudo systemctl restart docker
sudo chmod 777 /var/run/docker.sock

Installing Sonarqube we will run sonarqube as an docker container on jenkins-server, as we already installed and configure docker on our jenkins server now we can write below command to spin-up our sonarqube docker container

##-d is for running container in detached mode and -p is for defining port
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

##checking docker runing
docker ps

Now we will open the sonarqube ui on web-browser by putting <jenkins-server-public-ip>:9000, then login into default account. Below are the default credentials
username: admin
pswd: admin

On next screen reset the pswd and click update

After resetting the paswd we can see our sonarqube dashboard

Installing Trivy on jenkins-server, we will use it to scan the vulnerabilities in our docker image. Below are the commands

##installing pre-requisite packages
sudo apt-get install wget apt-transport-https gnupg lsb-release

##getting aquasecurity, adding key and listing the package in source list
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list

##installling trivy
sudo apt-get update
sudo apt-get install trivy

Installing and configuring Ansible Control node: we will use jenkins server as an controlled ansible node which is used to managed and installing services on k8-master and k8-worker node

To install ansible on control-node(jenkins-server) we only required 2 packages python and ansible-core, ansible is not required to installed on managed node just python is required on all node in order to work with ansible, so we just have to make sure python is installed on k8-master and k8-worker node

##install python package
sudo apt install python3.6

##install ansible-core
sudo apt install ansible-core

##check ansible version
ansible --version

##creating ansible user, with the shell access
sudo i
useradd -m -s /bin/bash ansible
passwd ansible

Ansible connect to other nodes with the help of ssh, to do so we have to make sure we are able to ssh on our both k8-master and k8-worker node from jenkins server. We will create a private, public key pair in our jenkins server(ansible user) at location ~ dir and then put the public key inside both managed server at location /home/$USER/.ssh/authorized_keys

First create pub, private key pair at ansible user home dir by following command
Note: while generating key don’t enter any passphrase, let everything be default

## switch to ansible user
$ sudo su ansible
$ cd
$ mkdir .ssh
$ cd .ssh

## generating key with the name aws_ec2 at location ~/.ssh
$ ssh-keygen -t rsa -b 4096 ~/.ssh/aws_ec2
$ ls .ssh

Now change the private key file permission to read and write and give this access to ansible user only, below is the command

##changing and viewing file permission
chmod 600 .ssh/aws_ec2
ls -l .ssh

We will update ansible.cfg file to include the default private key path, by this we don't have to specify the path in-order to ssh into managed nodes, below are the command

##adding privates ssh key path to ansible cfg file
echo private_key_file=/home/ansible/.ssh/aws_ec2 > /etc/ansible/ansible.cfg

##viewing edited file
cat /etc/ansible/ansible.cfg

Now we copy our public key and place it into both k8-master and k8-worker server at location /home/k8/.ssh/authorized_keys
Login into k8-master server, first change the hostname to “k8-master” and create a “k8” user, below are the commands

##changing hostname
hostnamectl hostname k8-master

##verifying hostname change
hostnamectl 

##creating k8 user, with the shell access
sudo -i
useradd -m -s /bin/bash k8
passwd k8

##adding k8 user to sudo group
sudo usermod -aG sudo k8
sudo su k8

Now we will make .ssh dir at k8 user and paste the public key inside authorized_keys file which we have copied from the jenkins-server , after that we will exit from the server. Below are the command

##creating .ssh if dir not exist
mkdir -p .ssh

##creating authorized_keys file with the public key in it
echo <public-key> >> ~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys

##logging out from the k8-master server 
exit
exit

We will repeat the step for copying public key to k8-worker server. First we will login into the k8-worker server change the hostname to “k8-worker1” and then create “k8" user and at final paste the public key at k8 user ~/.ssh/authorized_keys and then exit from the server

##changing hostname
hostnamectl hostname k8-worker1

##verifying hostname change
hostnamectl

##creating k8 user, with the shell access
sudo -i
useradd -m -s /bin/bash k8
passwd k8

##adding k8 user to sudo group
sudo usermod -aG sudo k8
sudo su k8

##creating .ssh if dir not exist
mkdir -p .ssh

##creating authorized_keys file with the public key in it
echo <public-key> >> ~/.ssh/authorized_keys
cat ~/.ssh/authorized_keys

##logging out from the k8-master server 
exit
exit

Now after placing public key at both the k8-master and k8-worker server we will test if our ansible control node(jenkins-server) able to connect to both the server

Login in into jenkins-server then switch to ansible user. First we have to create a dir name “ansible_k8_mannaged_nodes” in that dir we will create our ansible file structure, below is the command

sudo su ansible

## making our ansible dir ansible_k8_mannaged_nodes and changing working dir
mkdir ansible_k8_mannaged_nodes
cd ansible_k8_mannaged_nodes

In “ansible_k8_mannaged_nodes” dir we will create “inventory” file our managed node host information will be placed in it, below is the command

sudo vi inventory
###########file content below this line#################
[k8-master-node]
k8-master

[k8-worker-node]
k8-worker1
###########file content above this line#################

After creating inventory file we will create “host_var” dir and inside the created dir we create host-variable files, the host variable file name will be the same as the host name we specified in our inventory file in our case it will be “k8-master.yaml” and “k8-worker1.yaml”. Inside both the variable files we will specify the all the variables which related to that host including the connection method and credentials, below is the command

mkdir host_vars
cd host_vars

sudo vi k8-master.yaml
###########file content below this line#################
ansible_connection: ssh
ansible_host: k8@35.183.181.173
###########file content above this line#################

sudo vi k8-worker1.yaml
###########file content below this line#################
ansible_connection: ssh
ansible_host: k8@3.98.97.29
###########file content above this line#################

##get to home dir
cd

Now it’s time to test our connection to both the managed nodes, to test it we will use ansible ping command, below is the command

cd ansible_k8_mannaged_nodes

##ping k8-master node
ansible k8-master -m ping -i inventory 

##ping k8-worker node
ansible k8-worker1 -m ping -i inventory

Configuring Kubernetes Cluster through Ansible: Now we will start configuring Kubernetes cluster through ansible, in this step we will create roles, ansible role help us to automatically loads the artifacts of our respective host.
First create a dir name “roles” inside “ansible_k8_mannaged_nodes” dir , below is the command and the snippet of file structure so far

mkdir roles

Now we have to create our roles according to the tasks which have to be performed on our nodes, we have separated our whole task into 2 main roles. The first role task named “cluster-setup” will setup all the configuration and packages required both on k8-master and k8-worker node. The second role task name “master-setup”, these are the task which are only run on the k8-master-node after the first batch of task performed(cluster-setup)

Creating directory

cd && cd ansible_k8_mannaged_nodes/roles
mkdir -p cluster-setup master-setup
mkdir cluster-setup/tasks && mkdir master-setup/tasks
tocuh cluster-setup/tasks/main.yaml && tocuh cluster-setup/tasks/main.yaml

The file structure for “role” will look like below snippet

Now we have to edit the file content of our role “cluster-setup”, below is the content of the file “ansible_k8_mannaged_nodes/roles/cluster-setup/tasks/main.yaml”

    - name: "updating package initially"
      apt:
        update_cache: yes
      become: true

    - name: "swap-off"
      shell: 
        cmd: swapoff -a; sed -i '/swap/d' /etc/fstab
      become: true

    - name: "install pre-requisite module"
      apt: name={{ item }} state=present
      with_items: "{{ pre_dependencies }}"
      become: true

    - name: "Getting apt-key for k8"
      shell:
        cmd: curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
      become: true

    - name: "adding k8 packege in apt sorce list"
      shell: 
        cmd: echo deb https://apt.kubernetes.io/ kubernetes-xenial main | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
      become: true
        
    - name: "updating package after adding k8 in sorce list"
      apt:
        update_cache: yes
      become: true

    - name: "install module"
      apt: name={{ item }} state=present
      with_items: "{{ dependencies }}"
      become: true

    - name: "start docker"
      shell:
        cmd: sudo systemctl start docker
      become: true

    - name: "enable docker"
      shell:
        cmd: sudo systemctl enable docker
      become: true
    
    - name: "adding current user to docker group"
      shell:
        cmd: sudo usermod -aG docker $USER
      become: true

    - name: "update the iptables"
      shell:
        cmd: |
          cat << EOF | sudo tee /etc/sysctl.d/k8s.conf
          net.bridge.bridge-nf-call-ip6tables = 1
          net.bridge.bridge-nf-call-iptables = 1
          EOF
      become: true

    - name: "applying systemctl param"
      shell:
        cmd: sudo sysctl --system
      become: true

    - name: "changing docker c-groupc config file"
      shell:
        cmd: echo '{"exec-opts":["native.cgroupdriver=systemd"]}' | sudo tee /etc/docker/daemon.json
      become: true

    - name: "reloading systemd"
      shell:
        cmd: sudo systemctl daemon-reload
      become: true

    - name: "restart docker"
      shell:
        cmd: sudo systemctl restart docker
      become: true

Now we have to edit the master-setup.yaml, below is the content of the file “ansible_k8_mannaged_nodes/roles/master-setup/tasks/main.yaml”

    - name: "pulling packages from kubeadm"
      shell:
        cmd: sudo kubeadm config images pull
      become: true

    - name: "restart kubelet"
      shell:
        cmd: sudo systemctl restart kubelet
      become: true

    - name: "assigning private ip"
      shell:
        cmd: sudo kubeadm init >> k8-conection-string
      become: true

    - name: "mkdir .kube at $HOME"
      shell:
        cmd: mkdir -p $HOME/.kube
      become: true

    - name: "copying k8/admin.conf to /home/$USER/.kube/config"
      shell:
        cmd: sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
      become: true

    - name: "changing user and group ownership recursively of /home/$USER/.kube/config"
      shell:
        cmd: sudo chown $(id -u):$(id -g) $HOME/.kube/config
      become: true

    - name: "activating calico pod networking"
      shell:
        cmd: kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml
      become: true

After adding the roles task of both cluster-setup and master-setup, now we will add our variable for both the host, these variables are required in the role task file

Edit the “ansible_k8_mannaged_nodes/host_vars/k8-master.yaml” file with below host variables for k8-master

pre_dependencies:
  - apt-transport-https
  - ca-certificates
  - curl
  - gpg
dependencies:
  - kubectl 
  - kubeadm 
  - kubelet 
  - kubernetes-cni 
  - docker.io

Edit the “ansible_k8_mannaged_nodes/host_vars/k8-worker1.yaml” file with below host variables for k8-worker1 host

pre_dependencies:
  - apt-transport-https
  - ca-certificates
  - curl
  - gpg
dependencies:
  - kubectl 
  - kubeadm 
  - kubelet 
  - kubernetes-cni 
  - docker.io

After editing both the files our final file content will look like below snippet . As from the below snippet we can see our variables for both the host are almost same

Now we are almost ready to spin-up our k8-cluster through ansible control node, but to run these tasks on our role we need a playbook which can play our role tasks

Create a playbook named “k8-playbook.yaml”, below is the file content, as you can notice i have written “any_errors_fatal: true” for both our plays this is because if on any step error occur we have to exit from the ansible immediately without going on to the next task as most of the steps are critical

-
 name: Deploying k8-cluster
 hosts: k8-master-node k8-worker-node
 any_errors_fatal: true
 roles:
  - cluster-setup


- 
  name: Setting up k8-master
  hosts: k8-master
  any_errors_fatal: true
  roles:
    - master-setup

Now we are ready to run our playbook we will use below command, make sure you are on “ansible” user and in the “ansible_k8_mannaged_nodes” directory

ansible-playbook -i inventory k8-playbook.yaml 

If everything goes well every task will be executed smoothly and K8-cluster will be setup, this comes in hand when in real enterprise environment we have 100 of worker node, we can’t manually go into each worker-node and setup k8-cluster. Below are the result snippet

Now we have to connect our worker-nodes to our k8-cluster, in order to do so we have to run the join command on every worker node on cluster through playbook, we already save the join command which we get from the otuput of the command kubeadm init and its located in the file “k8-conection-string”
Login into the k8-master(k8-user) server to get the join command, and copy it and then exit from k8-master server

Now logged back into our jenkins-server(ansible-user) which is our ansible control node and create a ansible play to join the worker node to our clusters, below are the commands

cd ansible_k8_mannaged_nodes
vi k8-joining-node-playbook.yaml

Below is the content of the file “k8-joining-node-playbook.yaml”

-
  name: Joining Worker Nodes
  hosts: k8-worker-node
  tasks:
    - name: kubeadm join
      shell:
        cmd: kubeadm join 172.31.1.118:6443 --token mwrjxo.3se77rnljzb8lggh --discovery-token-ca-cert-hash swqha256:76546e1bec5c5dd52ec205904930822ec8eeeb1abcc40a925fd4d95c5a6f800c
      become: true

After editing the “k8-joining-node-playbook.yaml” file, we will now run the playbook by below command

ansible-playbook -i inventory k8-joining-node-playbook.yaml

By joining worker nodes now we can check our nodes, by logging into the worker node and running below command

kubectl get nodes -o wide

Finally we have completely setup our Kubernetes cluster and its fully functional, now we will exit from k8-master server

Setting up monitoring-server, in monitoring server we will first create the Prometheus user, below are the commands

##creating prometheus user with no shell and home
sudo useradd --system --no-create-home --shell /bin/false prometheus

##to verify if the user created or not
cat /etc/passwd | grep prometheus

In this step we will download the prometheus file and then untar the file below is the command

##dowloading prometheus 
wget https://github.com/prometheus/prometheus/releases/download/v2.49.0-rc.1/prometheus-2.49.0-rc.1.linux-amd64.tar.gz

##untaring the prometheus file
tar -xvf prometheus-2.49.0-rc.1.linux-amd64.tar.gz

After un-taring we will we move files from “prometheus-2.49.0-rc.1.linux-amd64”, below are the command

##changing dir
cd prometheus-2.49.0-rc.1.linux-amd64

##moving files
sudo mv prometheus promtool /usr/local/bin/
sudo mkdir -p /data /etc/prometheus
sudo mv consoles console_libraries/ prometheus.yml /etc/prometheus/

##changing permision to prometheus user
sudo chown -R prometheus:prometheus /etc/prometheus/ /data/

##verifying the permision of file if changed or not
stat /etc/prometheus
stat /data

checking prometheus version

prometheus --version

Now we will create the systemd file of prometheus in order for it to run as an service, edit the “/etc/systemd/system/prometheus.service” file and update it with the below content

sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus \
 - config.file=/etc/prometheus/prometheus.yml \
 - storage.tsdb.path=/data \
 - web.console.templates=/etc/prometheus/consoles \
 - web.console.libraries=/etc/prometheus/console_libraries \
 - web.listen-address=0.0.0.0:9090 \
 - web.enable-lifecycle
[Install]
WantedBy=multi-user.target

After editing the file we will enable it and start the prometheus service, below is the command

sudo systemctl enable prometheus.service
sudo systemctl start prometheus.service
systemctl status prometheus.service

we will check the default port of prometheus by below command, and then open it in web browser by using following url <monitoring-server-public-ip>:<prometheus-port>, in our case it is 35.183.234.130:9090

sudo lsof -i -P -n | grep LISTEN

Once prometheus is up and running now we want node_exporter to export machine resources data like(CPU,RAM etc), then tese data will be showed on Grafana dashboard used as a “Data Source” for Grafana

First we will create node_exporter user then downloading node_exporter package and extract it and at last moving the node_exporter dir to /usr/local/bin/, below is the command

##creating node_exporter user with no shell and home
sudo useradd --system --no-create-home --shell /bin/false node_exporter

##to verify if the user created or not
cat /etc/passwd | grep node_exporter

##downloading the node exporter package
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz

##untar the node_exporter package and moving the node_exporter
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo mv node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

check node_exporter version, below is the command

node_exporter --version

Now we will create the systemd file of node_exporter in order for it to run as an service, edit the “/etc/systemd/system/node_exporter.service” file and update it with the below content

sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
 - collector.logind
[Install]
WantedBy=multi-user.target

After editing the file we will enable it and start the node_exporter service, below is the command

sudo systemctl enable node_exporter
sudo systemctl enable node_exporter
systemctl status node_exporter.service

we will now add the node exporter to prometheus target file “/etc/prometheus/prometheus.yml”, inorder to get the metrics send by the node_exporter

sudo vim /etc/prometheus/prometheus.yml

edit the file with the below content

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

now our file will look like this in the below snippet

promtool is used to validate our changes in our promethus configuration, we use it to validate our changes which just made, below is the command

promtool check config /etc/prometheus/prometheus.yml

once the changes is confirmed and validated, now we can push our changes to prometheus server by using curl post command to our server, below is the command

curl -X POST http://localhost:9090/-/reload

The changes can be verified by going on to /target path on our prometheus server, in our case the url is “http://35.183.234.130:9090/targets”

Now we will install Grafana tool in our monitoring server in order to visualize our metrics data, coming from our prometheus target paths, below are the commands

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg - dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update

now install grafana

sudo apt-get install grafana

after installing grafana we will enable the grafana service, below are the commands

sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server.service
sudo systemctl status grafana-server.service

in order to login into grafana on web we required the port, to get port we will use below commands

sudo lsof -i -P -n | grep LISTEN

copy your grafana url <monitoring-server-public-ip>:<grafana-port> on web browser, in our case it will be “35.183.234.130:3000”. At first time we have to reset the pswd, the default username and pswd is “admin”

After login into grafana, we first add a data source by clicking on “Add your first data source” then select prometheus then giving it a name and the prometheus url which is in our case is the mointoring-server-public-ip:9090 “http://35.183.234.130:9090”

Now click “Save & test”

Now we will configure the dashboard for our data source, going to the dashboard section and importing a dashboard by its dashboard id and click “Load”

Below is the dashboard snippet

Configuring node exporter on jenkins server, in order to monitor the jenkins-server we need an agent which export the server resource data like node_exporter and then add it to prometheus target then finally create an dashboard on grafana

First installing prometheus plugin on jenkins, below is the snippet

Now we will edit “/etc/prometheus/prometheus.yml” to add in our jenkins-server target, below is the commad to edit the content

sudo vim /etc/prometheus/prometheus.yml

- job_name: "jenkins"
    static_configs:
      - targets: ["<jenkins-server-public-ip>:8080"]

now testing the config file which we have just edited through promtool, below is the command

promtool check config /etc/prometheus/prometheus.yml

if its validated then we will push our changes to prometheus server, below is the command

curl -X POST http://localhost:9090/-/reload

We now check if our jenkins metrics appears on our prometheus /target or not, to do so we will open our prometheus url with /target path in it. In our case it is “http://35.183.234.130:9090/target”

In this step we will create the dashboard on grafana for our jenkins metrics, to do so we will first import jenkins dashboard clicking on
New->Import and then providing the jenkins dashboard id and Load it

After that we will select the default prometheus from drop-down menu and click import it

Below is the jenkins-server grafana dashbaord snippet

Configuring node exporter as a daemonset on k8-cluster: Till now we have configured our grafana dashboard for the 2 servers jenkins-server and the monitoring-server itself, now we will configure grafana dashboard for our k8-worker and k8-master server. On both the server we will take leverage of kubernetes and deploying a daemonset of node_exporter service and not installing node_exporter directly on the server

First login into k8-mater server and create a namespace named “monitoring”, below is the command

kubectl create ns monitoring
kubectl get ns 

In this step we create daemonset, we will create it by creating definition file. For creating definition file we create a pod definition file using imperative command as it mostly match to it and then edit this file accordingly

##creating definition folder, all the definition file will be placed here
mkdir k8-definitions

##generating a definition and then saving it
kubectl run node-exporter --image=prom/node-exporter \
  --port=9100 \
  -n monitoring \
  --dry-run=client -o yaml > k8-definitions/node-exporter-ds.yaml
  
##now we will edit the file to make it a daemonset as our requirement
vi k8-definitions/node-exporter-ds.yaml

##after editing file create daemonset
kubectl apply -f k8-definitions/node-exporter-ds.yaml

Below is the final content of daemonset definition which will goes into the “k8-definitions/node-exporter-ds.yaml” file.

kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: node-exporter
    spec:
      tolerations: 
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - args:
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        - --collector.netclass.ignored-devices=^(veth.*)$
        name: node-exporter
        image: prom/node-exporter
        ports:
          - containerPort: 9100
            protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/sys
          mountPropagation: HostToContainer
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      volumes:
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root

NOTE: If we see our “node-exporter-ds.yaml” definition file i have added a tolerations in it of type “NoSchedule” with the key “node-role.kubernetes.io/control-plane”. This is because by default master node is tainted, the purpose of taint is to make sure no other pod is deployed on node except only the one’s which have toleration for that specific taint. We can find the taint on node by checking the config, below is the command and snippet

##it will check Taints key if exist in node config
kubectl describe node k8-master | grep Taint

From the below snippet we can see our daemonset pod is deployed on both the k8-node “k8-master” and “k8-worker1”

After creating the daemonset now we will create the service of node-exporter through definition file named “node-exporter-sv.yaml” for our daemonsets, below is the command

kubectl create service nodeport node-exporter --tcp=9100 \
  -n monitoring \
  --dry-run=client -o yaml > k8-definitions/node-exporter-sv.yaml

##editing the file
vi k8-definitions/node-exporter-sv.yaml

##after editing file create service
kubectl apply -f k8-definitions/node-exporter-sv.yaml

##listing services in monitoring namespace
kubectl get svc -n monitoring

Below is the final content of service definition which will goes into the “k8-definitions/node-exporter-sv.yaml” file

kind: Service
apiVersion: v1
metadata:
  name: node-exporter
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9100'
spec:
  selector:
      app.kubernetes.io/name: node-exporter
  type: NodePort
  ports:
  - name: node-exporter
    protocol: TCP
    port: 9100
    targetPort: 9100

As we can see from the above snippet, k8 have assigned us random valid nodeport for our service, this is the port “32025” which we will use with our public ip for the node_exporter url, in our case it is “35.183.181.173:32025” and “3.98.97.29:32025”, below are the snippets

Exit from the k8-worker host machine and then logged back in to the “monitoing-server”

Now we will add these node_exporter url for the metrics at prometheus target file “/etc/prometheus/prometheus.yml”, below is the content

sudo vim /etc/prometheus/prometheus.yml

  - job_name: "k8-master"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["35.183.181.173:32025"]

  - job_name: "k8-worker1"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["3.98.97.29:32025"]

After making change in file we will validate the changes and push the changes to prometheus server if its validated

##validating the changes
promtool check config /etc/prometheus/prometheus.yml

##pushing the changes to prometheus server
curl -X POST http://localhost:9090/-/reload

Now we can see our node_exporter metrics url at prometheus /target path

Now we can check the metrics of k8 hardware resources on grafana, go the the grafana dashboard and select Node Exporter, we can see the the kuberentes metrics there

Setting up CI/CD Pipeline: in last we have to configure our build pipeline on jenkins, we have to install multiple plugins and configure them. we will start with the below plugins Download following plugins Eclipse Temurin installer , SonarQube Scanner, NodeJS

Login into your jenkins account, in our case the jenkins url is “3.96.135.216:8080”, click “Manage Jenkins” from left pane then click “Plugins” then click on “Available Plugins” from left pane, type the listed plugins then select all the listed plugins and then click on Install

Once Installed now we have to configure NodeJS and JDK tools, Goto Dashbaord->Manage Jenkins->Tools
Then go-to Jdk section and add following details below in snippet

Go to NodeJs section and add below details, i’ve selected “16.0.0” version because my “resume-app” is not comaptible with latest version

Now we will setup sonarqube which we install earlier, to include this in our pipeline, open sonarqube url <jenkins-server-public-ip>:9000 in our case it is “3.96.135.216:9000”

First we have to get the user authentication token for this we have to go to Administration->Security->Users then click on the small list icon under token column

Now we will provide the token name as in our case it is “plg1” and then click on generate, we will copy the generated token and keep it somewhere

We will add this token to Jenkins Credentials, go to Dashbaord->Manage Jenkins->Credentials->System->Global credentials, then click on “Add Credentials” and paste the token we just created, the id i have give is “sonarqube-token”, below are the details in snippet

Now we will add sonarqube to jenkins system, go to Manage Jenkins->System then on “Sonarqube servers” section we will add the name and sonarqube server url, below is the snippet

Then we will add sonarqube-scanner into tools, go to
Manage Jenkins->Tools and scroll to “SonarQube Scanner” section add it with below details in snippet

Now we have to create a webhook for jenkins on sonarqube, go to sonarqube url click on Administration->Configuration and then select Webhooks. Click on create and create the webhook with below details in snippet

Below is the created webhook for jenkins on sonarqube

After creating webhook, now we will create sonarqube project Manually

Below are the details in snippet

Click on setup, then on next screen provide existing token then continue

On 2nd step select other as your build step and Linux as OS

Now we will add a post-build alert in our pipeline, we will send email in as an post-build alert with the information about the build. For this we need an sender email address we are using gmail for this

First add an Email extension plugin

Then generate an app password from the gmail, we will use this app password in our credentials, put App name “Jenkins” Click on Create

Below is the snippet of generated App password

Now we will add this “app-password” in our jenkins credentials, go to Manage Jenkins->Credentials->System->Global credentials then click on “Add Credentials”

Below are the email credentials detail in snippet

Once added it will look like below

Once the credentilas setup for email, we will move to the “SMTP server” in our Jenkins System settings, go to Manage Jenkins->System and scroll down to “E-mail Notification” and the below details shown in snippet. Provide SMTP server-name, server-port and email credentials

We can test if email is sending or not by clicking on test check-box

We are almost ready to make our initial build with sonarqube and postbuild alert but we need a code repo(an node.js application). For the demo purpose i have get the react-resume-app from the github and named it “plg1-resume” you can use this repo or any node-js appliaction
Note: If you get the dependencies issue you can change the node.js version from Dashboard->Manage Jenkins->Tools Node.js section, as i am using version “16.0.0"

This repo is private, in order to pull from this repo we need credentials. We will setup ssh access to our github account from our jenkins server, first we will create private-public key put the public-key on github and then we will provide this private key in jenkins-credentials

Login into jenkins server switch user to jenkins then generate ssh key , below is the code

su jenkins

##generating key leave everything as default
ssh-keygen

##copy public key "id_rsa.pub"
cat .ssh/id_rsa.pub

Copy the public key and paste it into “SSH and GPG keys” section under profile settings, in our case we named it “plg1-jenkins-server”, below is the snippet

Now verify if the ssh connection is established to our github from jenkins-server, below is the command

ssh -T git@github.com

Now copy the private ssh key(“id_rsa”) which we have just created located at jenkin user in the jenkins-server and then create a jenkins credentials with it. Go to Manage Jenkins->Credentials->System->Global Credentials, add new credentials with the details below in the snippet, use your git email and private key which you just copied

We are done with setting up git credential, now we can easily pull the code from our private repo using the “git ssh url”

Now we will setup the initial pipeline, go to the Jenkins dashboard select “New item” from left pane select the “Pipeline” project and named it “Resume” and click “OK”

In next screen scroll-down to the Pipeline section and add script, below is the initial pipeline stage script

pipeline{
    agent any
    tools{
        jdk 'jdk'
        nodejs 'nodejs'
    }
    environment {
        SCANNER_HOME=tool 'sonarqube-server'
    }
    stages {
        stage('Workspace Cleaning'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'master', credentialsId: 'git_ssh_key', url: 'git@github.com:syedtahamashhadi/plg1-resume.git'
            }
        }
        stage("Sonarqube Analysis"){
            steps{
                 withSonarQubeEnv('sonarqube-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Resume \
                    -Dsonar.projectKey=Resume \
                    '''
                }
            }
        }
        stage("Quality Gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }  
    }
    post {
     always {
        emailext attachLog: true,
            subject: "'${currentBuild.result}'",
            body: "Project: ${env.JOB_NAME}<br/>" +
                "Build Number: ${env.BUILD_NUMBER}<br/>" +
                "URL: ${env.BUILD_URL}<br/>",
            to: 'syedtaha2016@gmail.com',
            attachmentsPattern: 'trivyfs.txt,trivyimage.txt'
        }
    }
}

Now click on Apply and then Save, after saving the script we will create the build by clicking on “Build Now” on left pane, once build is sucessfully completed we can check our sonarqube code analysis on our sonarqube server under project section

Now we will setup OWASP build step, first we will install required plugins
“OWASP Dependency-Check Plugin” an then add it in
Tools->”Dependency-Check installations”

Below is the details to put in Dependency-Check

Now we will add the build step for OWASP in our “Resume” pipeline, below is the build step for it

stage('OWASP DP SCAN') {
  steps {
      dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'OWASP-DP-CHECK'
      dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
  }
}
stage('TRIVY FS SCAN') {
    steps {
        sh "trivy fs . > trivyfs.txt"
    }
}

Click on save and make build of it, once build is sucessfull we see the dependency check report on the left pane, as shown in below snippet

Now we have to setup docker build step, in order to do so we need docker-hub account then we push the docker image to our private docker repo, create one at https://hub.docker.com/. Once we have our docker hub account we need to pass our docker hub credentials in our jenkins credentials management

First add required docker plugins into jenkins, below are the required plugin
Docker, Docker Commons, Docker Pipeline, Docker API, docker-build-step

Adding docker credentials, as shown in below snippet

After adding credentials we can add the build step in it, below is the build step

 stage('Docker Build and Push to Docker Hub') {
    steps{
        script{
            withDockerRegistry(credentialsId: 'docker_hub_cred', toolName: 'docker'){
                sh "docker build -t resume ."
                sh "docker tag resume spawn328/resume:latest"
                sh "docker push spawn328/resume:latest"
            }
        }
    }
}
stage('TRIVY'){
    steps{
        sh 'trivy image spawn328/resume:latest > trivyimage.txt'
    }
}

In-order to make a docker image we required Dockerfile of our code, this file must be placed at root dir of our project folder. Below is the Dockerfile for our react-resume app. If your application is different it can differ slightly

Now we will create the build, from the below snippet we can see its successful

Once build is successful our docker image is pushed to our docker hub repo

Now we have to configure and add last step in our build process which is to deploy the docker image as pod on to our Kubernetes cluster. To do so we first need to install all required plugin into Jenkins. Below are the plugins

Kubernetes, Kubernetes Credentials, Kubernetes Client API, Kubernetes CLI, Kubernetes Credential Provider

Now in-order to deploy our image on K8 cluster, we need jenkins able to access k8-cluster , we can do so by getting the .kube/config details and adding it to jenkins credentials as a secret file

Login into k8-master server et the config details from .kube/config located at home dir of k8 user, below is the code

su k8

##copy the output and save it in a file name "k8-config.txt" at local machine
cat .kube/config

Now add this file as a credentials on jenkins, below is the snippet

Now we are ready to include our last build step which is to deploy application on kubernetes cluster, below is the build step
Note: deployment.yaml and service.yaml file is included into our code repo, also i will mention both the definition file here too, place both the file inside k8 dir located at root of the project folder

deolyment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: resume-fe
  name: resume-fe
  namespace: webapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: resume-fe
  template:
    metadata:
      labels:
        app: resume-fe
    spec:
      containers:
      - image: spawn328/resume:latest
        name: resume
        ports:
        - containerPort: 80

service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: resume-fe
  name: resume-fe
  namespace: webapp
spec:
  ports:
  - name: 80-80
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 31000
  selector:
    app: resume-fe
  type: NodePort

Note: To save time and resources, I have not implemented a full Ingress controller for our Kubernetes services.

Build step for deploying application on kubernetes

stage('Deploy to Kubernetes'){
  steps{
      script{
          dir('k8') {
              withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: 'webapp', restrictKubeConfigAccess: false, serverUrl: '') {
                      sh 'kubectl apply -f deployment.yaml'
                      sh 'kubectl apply -f service.yaml'
                      sh 'kubectl get svc -n webapp'
                      sh 'kubectl get all -n webapp'
              }   
          }
      }
  }
}

Our final build script will look like below

pipeline{
    agent any
    tools{
        jdk 'jdk'
        nodejs 'nodejs'
    }
    environment {
        SCANNER_HOME=tool 'sonarqube-server'
    }
    stages {
        stage('Workspace Cleaning'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'master', credentialsId: 'git_ssh_key', url: 'git@github.com:syedtahamashhadi/plg1-resume.git'
            }
        }
        stage("Sonarqube Analysis"){
            steps{
                withSonarQubeEnv('sonarqube-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Resume \
                    -Dsonar.projectKey=Resume \
                    '''
                }
            }
        }
        stage("Quality Gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonarqube-token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('OWASP DP SCAN') {
            steps {
                dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'OWASP-DP-CHECK'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
        stage('Docker Build and Push to Docker Hub') {
            steps{
                script{
                    withDockerRegistry(credentialsId: 'docker_hub_cred', toolName: 'docker'){
                        sh "docker build -t resume ."
                        sh "docker tag resume spawn328/resume:latest"
                        sh "docker push spawn328/resume:latest"
                    }
                }
            }
        }
        stage('TRIVY'){
            steps{
                sh 'trivy image spawn328/resume:latest > trivyimage.txt'
            }
        }
        stage('Deploy to Kubernetes'){
            steps{
                script{
                    dir('k8') {
                        withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: 'webapp', restrictKubeConfigAccess: false, serverUrl: '') {
                                sh 'kubectl apply -f deployment.yaml'
                                sh 'kubectl apply -f service.yaml'
                                sh 'kubectl get svc'
                                sh 'kubectl get all'
                        }   
                    }
                }
            }
        }
    }
    post {
     always {
        emailext attachLog: true,
            subject: "'${currentBuild.result}'",
            body: "Project: ${env.JOB_NAME}<br/>" +
                "Build Number: ${env.BUILD_NUMBER}<br/>" +
                "URL: ${env.BUILD_URL}<br/>",
            to: 'syedtaha2016@gmail.com',
            attachmentsPattern: 'trivyfs.txt,trivyimage.txt'
        }
    }
}

Note: In our Jenkins pipeline script at “Deploy to Kubernetes stage” once our deployment is done we dont need to do deploy our application again and again for same application intead set it with the new image by using following command “kubectl set image deployment/resume-fe resume=spawn328/resume:latest”

Now hit the Build Now button, once build is successful to view the application we use public ip of any k8 worker node and use the nodeport, in our case it is “3.98.97.29:31000”. below is the deployed react-resume app snippet

Conclusion:
If you come this far and set things on your end too it means you have covered lot of topics and tools, by completing this project and understanding each step it will make you confident enough to use these tools in your actual environment. Having said that the technology space changes too fast new things come and go but the topics which are covered in this project will be here to stay for a long time.

See you in the next topic, until then keep hustling and keep learning!

Introduction: To understand how process work and how we can diagnose and manage it is an big asset for any linux user. In this blog we will deep dive into understanding how process work and if some issue arise with the processes then how we can diagnose it and then based on our diagnosis and understanding we can take best actions to resolve it

Following are the topics which we will cover in this blog

• Process
• Foreground and Background Process
• Process States
• Process niceness
• Controlling Process
• Daemons
• Process Signals

Process: A process is sometime mistakenly taken as an program where it is not true the actual definition can be said as “process is the running instance of any program in our machine which have unique id named PID assigned by the kernel”. Linux creates process for every program that executes or run for instance if we type ls on command line it creates a short live process for the program ls, some process are longer like process of a node js application or some process closed when system goes off like daemon processes.

Whenever system boots up the first process which runs is init process and it gets the PID 1 every other process are derived from this first process and init become parent or grandparent of every other process, init is a daemon process which runs until system shutdown. We can use the below command to see all the processes of current user.

## displaying all process of current user in user readable format
## the ax combination is for listing all process and the u option is for human readable format
$ ps -aux | grep

## we can customize the ouptput by passing -o option
$ ps -aux 

## to check process runing by specific user, if we want output in user oriented format pass u
$ ps -U osboxes

## if we want to check specific process, we can do it by passing the pid of it
$ ps u 1

## if we know the process name we can use below command, a option is for listing command
$ pgrep -a containerd

To continuously monitor the logs we use htop command it is the newer version of top command, htop continuously shows the process sorted by highest cpu usage it gives detailed information of each process along with the usage of overall resources. Below is the output snippet

At any given time a single program can have multiple instance of its process running and a program can have multiple threads running simultaneously (multithreading) each of it considered as a separate process of a running program, every process has a pid(Process ID), ppid(Parent Process ID) and pgid(Group Process ID), the max number of pid is mention in file /proc/sys/kernel/pid_max we can edit this number and if the pid reach to it max number then the process restart again at PID 300

Processes are run in isolation it only runs in the user space, other user can not have control or any type of access on process running on other user space this enhance the security of the system. Process cant access hardware directly in order to access the hardware it indirectly access hardware through system calls, hardware can only be accessed by kernel directly

Forgeround and Backgorund Process: A process can run in background like a daemon process or in foreground like an htop command on shell, a background process will not interrupt user or holds shell whereas foreground process holds the shell but we can put foreground process into background and free up the shell for further operations, the background process runs at lower priority which allow user to interact with other tasks smoothly

Adding ampersand (&) after a command will run the command in the background, like putting & after htop command

By pressing CTRL+Z suspend foreground processes and CTRL+C terminates a foreground process, for running process in background type bg and for running it in foreground type fg

jobs command is used to list all the background and stopped process of current user

Process States: A process can be in different state in its lifetime, we have discuss all the process state below

• Running : A process is in running state when it’s consuming resources like cpu or memory, a process can be in queue before going into running state a queued process goes into running state when the priority elevated it
• Waiting: A process is in waiting state when its wait for some response from task or required I/O operation to be complete in order to run again
• Stopped : A stopped process is the process which goes into the temporary stop state instructed by the user or from other process, in this state process is in hold and can be resumed again
• Zombie: A process which is completely terminated and releases all it resources except it’s entry in process table is called Zombie process

Process niceness: process niceness refer to how nice process is to other processes we can measure the niceness from (-20 lowest) to (+19 highest), the niceness value set the priority of process execution (lowest the value highest the priority) or (highest the value lowest the priority). The default value of niceness of any process is inherited from the process which rans them. We can change these value of any process but it can only be raised by normal user and only root user can lower the niceness value, every process have the niceness value which effect the execution of process and it can be controlled by nice and renice commands difference between both of them is nice allow user to set the value during execution of process and renice allow user to change it later, below are the examples.

## to se the process of current user with nice value we use l option, nice value is under NI cloumn
##we can include ax option to list all process
$ ps -l

## to set the nice value at time of the launch, nice command will append the value
$ nice -n 4 bash

## to check which process is parent to other processes in a tree format, f option is for tree format output
$ ps -fax

## using renice to change the nice value by specfying the PID
$ renice 7 8617

Controlling Process: It is better to limit resources of each user process consumption, this is the responsibility of system admin to put a restriction and limitation for every user resource consumption by there processes, an admin can put restriction so the server does not get exhausted or can expand it limitation so task can easily be performed, this can be done through ulimit command it is an built-in command which is used for displaying and limiting resource used by processes for each user.

Through ulimit we can restrict resource like memory, cpu time or max number of process on the system, ulimit can only limit user process for current session for permanently setting values we have to edit /etc/security/limits.conf file. Limit is classified into two types

• An “hard limit” can’t be override by regular user for instance if hard limit says a user cant create more then 12 process then user can not run more then 12 process, its the max value of resources used by a user.
• An “soft limit” is the initial limit of a user to create and after if needed it can be manually increase by user but they cant go above the hard limit. “-” a dash identifies both the hard and soft limit, which mean if a user have 16 process limit of dash “-” then they can use all the 16 process anytime they want

## viewing detail ulimit report
$ ulimit -a 

##viewing detailed soft limit
$ ulimit -Sa

##viewing detailed hard limit
$ ulimit -Ha

##setting max file size a user can make
$ ulimit -f 10

## limit number of open file
$ ulimit -n 1200

## hard limit number of open file
$ ulimit -H -n 1200

## soft limit number of open file
$ ulimit -H -n 1000

Daemons: daemon process is an background process which runs continuously in background until the system shutdown, daemon process usually starts with the system bootup and unlike other process it does not respond to signals from the keyboard. Daemon process handle all request in background and ensure everything runs smoothly, many process in linux system are daemon process and they usually ends with “d” like httpd, crond, lpd and vsftpd. Daemon process provides better security and control compared to other processes as it only operate when needed and does not required any controlling terminal or any I/O device.

We can list daemons process by following the command instructed in this stackexchange question
https://unix.stackexchange.com/questions/175380/how-to-list-all-running-daemons

Process Signals: process signals is a kind of interruption for the process and once signal received by process it will act accordingly, these are kind of asynchronous event for process which tell process to do specific action synchronously. Process signal can be triggered by other process or by user, every process signal starts with SIG. SIGSTOP and SIGKILL are the only 2 signal which cant be ignored else for other signal it depends upon on how application is programmed to act on these signal or not. We can only send signal to process which are owned by current user

## to list all signals, l option is for listing
$ kill -l

## we can use signal with thier whole name like SIGTERM followed by PID
## or we can use with thier signal name like TERM followed by PID
## or we can use the signal id only like SIGTERM have an ID 15 followed by PID
## terminating sshd process
$ kill -SIGTERM 7941

## same as the above command, if no signal defined it sends SIGTERM signal by default
$ kill 7941

## killing process by its name, if we only know the name of the process we use pkil command
$ pkill -TERM sshd

PS: Both SIGTERM and SIGKILL do same job but the main difference is SIGKILL kills the process in abrupt fashion where SIGTERM tries to kill the process in a polite fashion and it can be handled by the program unlike SIGKILL

Conclusion: In this article we have learned about processes in linux system and how we can list or manage these process by using commands, we can take advantage of these command in diagnosing the issue related to process execution or failure.

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi

Replicasets:

The literal meaning of replication is to create an exact copy of same thing and these same copies together can be refer as an replicasets. In kubernetes context creating multiple replicas of same exact pods are called the replicasets, but why we create it and what purpose it serves, to understand this we have to understand following scenarios

High Availability: Suppose our application is running as a single pod and for any reason our pod crashed or our node failed, in this situation our user faces the downtime to counter this situation we need high availability which can be guaranteed by replica set it ensures that specified number of pod replicas are running all the time across our k8 cluster

Load Balancing: Load to our application can be increased anytime and if there is only one node there is high chances that it can be crashed, replicaset helps in this situation by distributing loads evenly on our pods. These pods can be on same server or distributed on different server node, by this way load is distributed to different server machines. Replicasets can spans around on different worker node or on single worker node and it ensures the balance of traffic load to each pod.

PS: Replicaset is early known as the Replication Controller which is exactly the same thing but an older technology there is no major difference except that replica set required selector definition and Replication Controller does not, now we refer Replication Controller as Replicaset

Replicaset Definition File: To create any object/resource in kuberenetes we need a definition file for it, which describe how the object will look like in YAML format. In Replicaset definition file we use following main keys

- apiVersion: api version of the object which “apps/v1”
- kind: kind of the definition file which is “ReplicaSet”
- template: a pod definition under the spec section
- selector: we use selector to select the pods which is included in replicaset selected based on their name tag, however it is not the required field
- replicas: number of pods defined which includes in the replicaset

Below is the example of replicaset definition file named
“my-webapp-rs.yaml”

apiVersion: apps/v1 
kind: ReplicaSet
metadata:
  name: my-webapp-rs
  labels:
    app: my-webapp
    type: frontend
spec:
  template:
    metadata:
      name: my-webapp-pod
      labels: 
        app: my-webapp
        type: frontend
    spec:
      containers:
      - name: nginx-container
        image: nginx  
  replicas: 3
  selector:
    matchLabels:
      type: frontend

Creating Replica Set and Managing it: In this section we will see and understand some commands use to create and manage replicaset, with example on actual K8-cluster(1 worker and 1 controlplane node)

## creating rs by definition file
kubectl apply -f my-webapp-rs.yaml 

##geting rs in default namespace
kubectl get rs

## scaling replicas in rs
kubectl scale --replicas=4 rs/my-webapp-rs

## deleting replicaset
kubectl delete rs 

Note: replicaset assign a random hash id to its pods, the hash id is added after the rs name

DaemonSets:

Daemonsets are like replicasets as it helps us to deploy multiple instances of the pods span around the worker-nodes, but the main difference is that daemonsets guaranteed the deployment of single pod on every worker-node in kubernetes cluster and it ensures a pod is present every time on a each node. It runs one copy of pod on every worker node, if in future new node is added it automatically adds new pod on it or similarly if node is removed it automatically removes it.

Earlier daemonset uses the node-name approach to bypass scheduler and put pod one every node but from v1.12 kubernetes uses Node Affinity and default scheduler to schedule pods on each node. Below are some use-cases where we can take leverage of daemonsets

Monitoring and Logging Agent: Now in every production architecture we need ways to get the monitoring and logs data which can provide us the detail information about the resource utilization and logs of the application that are running, to do so we have to deploy some sort of monitoring and logging agent like(Prometheus and Grafana) on our instances(which are mostly server running in cloud or in premises). The best way is to deploy these agents as a daemonset on k8 cluster, by this way our logging and monitoring agent can be deployed on each worker node and if new node is added or removed theses agent can be added or removed automatically. Deploying these agents as an daemonset ensures we get monitoring and logging data seamlessly without any interruption.

Below are the 2 other use cases other then monitoring and logging agent where we can deploy pods as daemonsets

Kube-proxy: kube-proxy is one of the main component of k8-cluster which is required on every worker node, to make sure kube-proxy is present on every worker node we can deploy it as a daemonsets

Networking: Some networking solution required some agent to be deployed on every worker node, we can use daemonset to deployed it on every worker node

PS: Daemonsets ensures the deployment of pod on every single node in k8-cluster but not on the Tainted node like the master-node as it is tainted by default. To deploy pod on tainted nodes too we have to create Toleration for that node by specifying the taint name and type under toleration key in definition file

Daemonset Definition File: Creating ds definition file is almost same as the replicaset we have to specify type as Daemonset except we don't use replicas in it

Below is the example of daemonset definition file named “my-webapp-ds.yaml”, but first as we have only 2 nodes in our k8-cluster so in order to deploy pods on master node along with worker node we have to create Tolerations for the Taints which is on master-node

apiVersion: apps/v1 
kind: DaemonSet
metadata:
  name: my-monitoring-agent-ds
  labels:
    app: my-montioring-agent
    type: monitoring-agent
spec:
  selector:
    matchLabels:
      app: my-montioring-agent
      type: monitoring-agent
  template:
    metadata:
      name: my-montioring-agent-pod
      labels: 
        app: my-montioring-agent
        type: monitoring-agent
    spec:
      tolerations: 
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: nginx-container
        image: nginx

Creating Daemonsets and Managing it: In this section we will see and understand some commands use to create and manage daemonsets, with example on actual K8-cluster(1 worker and 1 controlplane node)

##creating daemonset
kubectl create -f my-webapp-ds.yaml

##getting ds
kubectl get ds

##deleting ds
kubectl delete ds

Deployments:

Deployment are used to deploy our containerized application on pods in actual development/production env and it caters all the underline process which are required in deploying the application and controls it behavior, it come’s after the replicaset in hierarchy of kubernetes object. It give the capabilities of deployment like rolling-updates, saclling-up, undo-changes, pausing pod instances and resuming it. If we want to deploy any application on kubernetes we use deployment due to its underline capacity to handle deployment of application on kubernetes cluster seamlessly

Deployment Definition File: Creating deployment definition file is almost same as the replicaset, we have to specify type as Deployment

Below is the example of deployment definition file named “my-webapp-dp.yaml”

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: my-webapp-dp
  labels:
    app: my-webapp
    type: frontend
spec:
  template:
    metadata:
      name: my-webapp-pod
      labels: 
        app: my-webapp
        type: frontend
    spec:
      containers:
      - name: nginx-container
        image: nginx  
  replicas: 2
  selector:
    matchLabels:
      type: frontend

Creating Deployment and Managing it: In this section we will see and understand some commands use to create and manage ds, with example on actual K8-cluster(1 worker and 1 control plane node)

##creating deployment by my-webapp-dp
kubectl apply -f my-webapp-dp.yaml

##setting new image to deployment
kubectl set image deployment/my-webapp-dp nginx-container=redis-alpine

##checking rollout status of current change on deployment
kubectl rollout status deployment/my-webapp-dp

##checking rollout history of mentioned deployment
kubectl rollout history deployment/my-webapp-dp

##rolling back to previous deployment state
kubectl rollout undo deployment/my-webapp-dp

##scale rs of deployment
kubectl scale deployment/my-webapp-dp --replicas=3

##pause rollouts for new changes
kubectl rollout pause deployment/nginx-deployment

##resume rollouts for new changes
kubectl rollout resume deployment/nginx-deployment

Conclusion: In this blog we have cover main topics of kubernetes which is related to pod orchestrations by using different methods and techniques, as in kuberentes it is important to understand how pods are scheduled using these main techniques most of the time we use one of the methods which we discuss in this blog to scheduled pods. Other then these method pod can also be scheduled as a single pod using definition file or scheduling it as a static pod by putting pod definition file in manifest folder.

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi/

Introduction: In linux system in order to automate some processes which usually need to perform repeatedly from command line, we bundle those set of command instruction in a file known as shell script and then these scripts is parsed by an interpreter like bash and executed on the fly is called bash shell scripting. We used bash for scripting because it is simple and powerful language which is widely used by all unix based operating systems and also an default shell of most distribution. In this blog we will only discuss the prerequisite commands and techniques which comes handy and need to be known before moving to the actual bash slanguage syntax which we will cover in Part-2.

First line of script (#!/bin/bash): whenever we start writing our script we have to specify a line which consist of “#!” also known as shebang followed by full path of the command interpreter which is “/bin/bash” with no space between them, we have to make sure it will always present in start of our bash script if it’s not there our script will not work and after first line we can start writing our script.

PS: As we are using bash script the above explanation is applicable for other command shells like csh or ksh

Return value: In linux every program or script returns a value when they exit, not only programs or script but also methods which are used in script generate these value after they complete there execution. These return value are stored in special variable “$” the value in it indicates if the program, script or method exit successfully or not which helps parent process to monitor the exit state. These variable “$” holds numeric value where 0 indicates successful exit other then 0 it mean error or other issue.

Below is the example of exit code of “rm” command if file remove successfully it return 0 if not then it returns 1.The return value of unsuccessful exit code can vary between different programs, similarly if we try to list directory which is not exist by “ls” command it will return 2. To check return value on command line we use “echo $?”

Script parameters: we can pass arguments to bash script from the command line while we run the script, these parameters(argument) can be any number or a string character in order to access these parameter value inside a script we use $ sign followed by a number each parameter got its variable name dynamically which is the numeric number from $1 to the $nth number, the variable is assigned according to their passing order where $0 variable holds the script name itself and $* holds all the parameter values in it.

In below example we have written a script name var_name.sh which prints the parameter that are passed from the command line

#!/bin/bash
echo script name : $0
echo first perameter : $1
echo second perameter : $2
echo third perameter : $2
echo All perameters : $*

Now we will run script but first make script executable with below commands

$ chmod +x ./var_name.sh
$ ./var_name.sh first second third

Comments: Like other scripting languages we can follow the same syntax as Python to comment a single line which is #, comments are ignored by bash interpreter and it moves to the next line. Comments are useful in defining what operation is being performed next, in large script where multiple things are done it become essential to add comments

#!/bin/bash
#echo This line will be ignored
echo This line will be executed

Output redirection: every commands create an output stream which is shown on the terminal, in order to redirect the output to specific location we use output redirection. Following are the 2 pattern of output redirection

first one is using >(greater then sign) after the command and then followed by the location of file where generated output need to be saved. The single greater then sign(>) will completely removes the content of the file and add the current output

second one is using >>(doble greater then sign) it will work as same as the above one but the difference is the (>>) sign will append the content of file and not removing old content of file. Below is the pattern and example snippet

$ date > /home/osboxes/current_date_1
$ date >> /home/osboxes/current_date_2

/dev/null and /dev/zero: consider these 2 as your rescuer for dumping all your unnecessary data both are special file tag as c type but both are slightly different from each other how they process data. /dev/null is also known as black-hole of linux galaxy anything goes in will disappear and /dev/zero act same where all write request to it are discarder but it generates infinite stream of zeros.

We can redirect logs and output generated by any command to these location, it’s come’s handy when there is some command in our script which generate output that is not useful for us then we can redirect them to /dev/null.

Below is the snippet where we have redirect the output of “find /etc” command to /dev/null, the only output which is showed is from the stderrr stream. If we want to ignore the stderr stream we can use this command
“find /etc >& /dev/null”

Logging error outputs: A output can be generated from two type of stream stdout and stderr, there can be scenarios where we want to log the error generated by any command for the purpose of investigating the issue this can be achieve by error redirection. Syntax of error redirection is same as normal redirection the only change is we add 2 before >>. Below is the example of code and snippet

# redirecting error to err_log file, it will only show stdout on screen
$ find /etc 2>> ./err_logfile

# it will show nothing on screen, redirect the stdout stream to /dev/null and stderr to ./error_log file
$ find /etc >> /dev/null 2>> ./err_log

Special characters: Following are some special characters which can be used in a bash script

Command substitution: command substitution is a way of substituting commands with other command which allow commands to create dynamically, it become so much useful when our command output depends upon the result of other command. The output of one command is an input for other command through command nesting commands

There are 2 ways to substitute commands both have different syntax but the result generated by both of them are same, the first syntax is substitute command written in $(your command) and the second syntax is a substitute code written inside `your command`.

For instance i have to change the user and group ownership of file to current login user but i don't know what my user id and group id is? instead of first checking my user id and group id I can directly substitute these commands with “chown” command in one line, below is the example code where i used both syntaxes and result snippet.

sudo chown $(id -u):`id -g` /home/osboxes/error_log

Generating random data: random characters and numbers are very useful in shell scripting especially when we want to assign some unique name to a file or to a directory with unique characters in their name or it can be useful in generating a file with random character for payload testing. There are few ways to generate random character or number, the popular one is to generate random number by using default RANDOM variable of shell other way is to use openssl to get the random characters, below are the example code and snippet

# getting random number each time
$ echo $RANDOM

# generating random character of 18 bytes in base64 format, we can also use different option with openssl rand command
$ openssl rand -base64 18

# generating random character of 128 bytes in base64 format and saving it in a file
$ openssl rand -base64 -out random-data 128

Temporary files and directories: we can create temporary files and directories which are only available until the end of user current session and these file/directories only accessible to user who created it, these temporary files/dir are located in “/tmp” directory. Temporary files and directories come handy when we need to create files and folder which only needed by current user for placing some temporary information or file that are only useful until script completely executes itself or until current login session

To create temp file or dir we use command name “mktemp” followed by the location of file which is “/tmp” and then finally random file/dir name, following is the syntax for naming random file/dir “temp.XXXX” where XXXX is the random char which will be automatically changed into character when file is created

For creating random directory the syntax is almost same as for creating random file the only difference is we have to provide “-d” option to “mktemp” command. Below is the example code of creating both random file & directory with result snippet attached

$ mktemp /tmp/temp.XXXX

$ mktemp -d /tmp/temp.XXXX

Debugging script: we can run script in debug mode by specifying -x while we run the script or can debug specific part of our script by combining it to +x with the set keyword. When we run script in debug mode it prints each line of command in script which it executes. Below is the example of both the ways and snippet

# this command will run scipt in debug mode
$ bash -x debug_script.sh

In below snippet we have set debug mode for the specific set of lines in out shell script file.

Conclusion: Till now we have cover all the essentials command and techniques which create advance possibilities of doing things particularly in bash shell, now in part2 we will cover all the syntax of bash shell scripting, until then keep practice and stay tuned.

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi/

Introduction: In kuberentes etcd-database holds the current state of our cluster, in order to be vigilant with our infrastructure on kubernetes we should always have a recent backup of etcd either on cloud or on different backup machine, by taking backup we make sure that if something went wrong with our kuberentes cluster we can restore it to previous working state through our recent backup of etcd database. In this project we will cover each step of how we can automate our process of taking backup of etcd on cloud(amazon s3) which include setting up an IAM user then configuring s3 bucket and aws-cli on control plane machine followed by writing a bash shell script for taking backup and restoring it and in the end we will use crontab to schedule our backup.

Following are the topics which will be covered in this project

• what is etcd
• etcdctl
• Configuring AWS IAM
• Configuring AWS S3
• Configuring AWS CLI
• etcd backup through bash shell script
• etcd restore through bash shell script
• scheduling backup (crontab)

what is etcd : etcd is an key value persistent storage in which we can store our data, in kuberentes etcd plays an vital role as it holds the current state of complete k8 cluster any changes made to the cluster are reflected into the etcd database no change is complete in k8 cluster until and unless it is recorded into the etcd, it is the component of control-plane node and kube-apiserver is responsible to communicate and update information into the etcd database directly.

It is important for us to make sure that we always have a backup of current working state of etcd database because any issue with our controlplane machine or any misconfiguration can effect our applications or services running on kuberentes cluster, for instance if we have to perform some maintenance task in our k8-cluster and unfortunately after the maintenance resources which are previously running fine are now missing or not performing as expected in this situation the backup of etcd database which we take before the maintenance or at any time comes to rescue as we can now restore our previous working state from the recent etcd backup.

etcdctl: etcdctl is a default command line client for etcd database through which we can interact with our etcd db, by using it we can perform multiple actions like taking snapshot of our etcd db then restoring from that snapshot or getting status of snapshot. In this project we will use etcdctl to interact with etcd database and we will use version 3 of it, in order to work with v3 we have to use ETCDCTL_API=3 in start of every etcdctl command or if you are an root user you can set the env variable by exporting it.

To install etcdctl on ububtu machine type below command

$ sudo apt update
$ sudo apt install etcd-client

AWS IAM: AWS IAM also known as (Amazon Web Service Identity Access Management), it is an popular aws service to manage workforce and workload across multiple teams and groups and the primary purpose of it to securely control access to our infrastructure and service which are on aws. On AWS IAM dashboard we as an admin create Roles which have certain parameter and specific access to our services on aws then these Roles are attached to specific User or Group, we can attached different policies to our user by creating new one or selecting from the aws pre configured policies

We will create IAM role for our user which have access for s3 service then we will create access key these access keys are then used to configure aws cli

Step1 : Login into aws-console account and then go to s3 dashboard to create user and leave everything as default

Step2: Then on next page attach policy name (AmazonS3FullAccess) to our user and leave everything as default

Step3: Then on next page create user, tags are optional

Step4: Click on User then Security Credentials Tab and then in access key section click on Create access key

Step5: On Access Key page select Application running outside AWS

Step6: On next page Add description and then click on Create access key

Step7: Securely copy Access Key and Secret Access Key it will be only shown once

AWS S3: We will take our etcd backup on cloud, aws have a popular service name s3 it is used to store large amount of data on cloud which act as an simple storage service, there are lot of other purpose of s3 but for our project we will only use it for storing our etcd backup.

Step1: Go to s3 dashboard and select Create bucket, on first page select your region and give it a unique global name

Step2: Scroll Down and select Enable radio button in Default encryption section and finally click on Create bucket

PS: While configuring s3 you can use Amazon S3 Lifecycle Rule, it helps to set expiry on your saved object and auto remove it which is effective in cleaning up outdated backups which saves our cost too

Configuring AWS CLI: We will interact with s3 through aws cli from our control plane machine, in order to interact with it we need to install aws-cli into our control-plane machine and then after we will configure access key and region for aws-cli

Step1: Install official package from aws through curl request, it will be downloaded as zip file we have to unzip it’s content. Below are the command and result snippet

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Step2: File will be extracted with the name aws, run the installer located in that dir

Step3: configure aws using aws configure command and then passing access key, access secret key and region. These configuration will be saved in home inside the .aws dir

Step4: Verifying the connection to aws services by listing content of s3 bucket

etcd backup through shell script: Now we will take backup of our etcd database from the control-plane where it is located, we can do so by typing each command 1 by 1 but in order to take backup at every day at specific time we have to automate this process for this purpose we will use the bash scripting and then add this script file as a cronjob into crontab

Step1: Change etcd-data directory from etcd object definition file located at /etc/kubernetes/manifest/etcd.yml, open this file with your favorite editor and change mounted data dir to “/opt/etcd-restore” under the key “spec.volumes[1].hostpath.path”

Step2: Create bash shell script file where we write our script to take backup and named it “backup-script.sh” and then make compressed archive file of our backup with tar and finally upload it to your s3 bucket. Below is the content of backup-script.sh, I have added comment on each step in order to understand what each step do

#!/bin/sh

# create /opt/etcd-backup dir, in this directory daily etcd-database backup will be stored
mkdir -p /opt/etcd-backup

# creating file name variable which have value of filename with current datetime
FILE_NAME=etcd-$(date +"%F%T").db

# creting etcd path variable where full path for current backup file is defined
ETCD_PATH=/opt/etcd-backup/$FILE_NAME

# taking etcd database snapshot with etcdctl and saving it into /opt/etcd-backup dir, we have to specify 4 keys in order to interact with etcd server
# --endpoints it is the url of etcd server
# --cacert it is the full path where ca.crt file located
# --cert it is the full path where server.crt file located
# --key it is the full path where server.key file located
ETCDCTL_API=3 etcdctl \
        --endpoints=https://127.0.0.1:2379 \
        --cacert=/etc/kubernetes/pki/etcd/ca.crt \
        --cert=/etc/kubernetes/pki/etcd/server.crt \
        --key=/etc/kubernetes/pki/etcd/server.key \
        snapshot save $ETCD_PATH

# we will delete backup file if it is older then 7 days from etcd-backup directory
# value under [] is an test statement followed by actual statement
# if /opt/etcd-backup dir exist then delete files from it which are 7 days older
[ -d /opt/etcd-backup ] && find /opt/etcd-backup -cmin +10080 -delete

# archiving and compressing using tar with .gz utility of etcd-backup folder,
# after compression etcd-backup.tar.gz file will be located at current dir where this script is located
tar czfp etcd-backup.tar.gz --absolute-names /opt/etcd-backup/

# finally uploading our tar compressed backup to s3 buket
# quiet option will run command in quiet mode
# storage-class option is for defining which storage class to used for storing object
aws s3 cp ./etcd-backup.tar.gz s3://taha-k8-etcd-backup --storage-class STANDARD

Step3: change permission of backup-script.sh file to 744 which mean only owner have full read, write, execute access where group and other user have only read access. Below is the command and result snippet

$ chmod 744 ./backup-script.sh

Step4: now we will run script using below command

$ ./backup-script.sh

Step5: after running backup-script successfully we will verify each step by looking for backup file at each location by below commands. For verification check time of each file

## verifying if compressed backup file uploaded to s3 bucket
$ aws s3 ls s3://taha-k8-etcd-backup

## verifying if new backup compressed file created
$ls -l etcd-backup.tar.gz

## verifying if new backup file created at /opt/etcd-backup folder
$ls /opt/etcd-backup

etcd restore through shell script: at some point due to some technical reason our kubernetes cluster resources are not present or not performing as expected and now we have to restore our last backup of etcd-database from cloud(s3-bucket), this task can be done manually by writing each command step by step but i have written bash shell script for restoring backup too, the script for restoring backup name as “restore-backup.sh” and it’s located at root home dir. Below are the steps to restore our backup and run our applications again on k8-cluster :-)

Step1: Create file name restore-backup.sh with your favorite editor and then open it to add step for restoration of etcd-database with the last backup file from cloud(s3-bucket), I have added comment on each step in order to understand what each step do

#!/bin/sh
# removing /opt/etcd-restore folder if it exist, 
# if we dont remove our data dir then etcd will throw err "Error: data-dir path exists" 
[ -d /opt/etcd-restore ] && rm -r /opt/etcd-restore

# creating aws-etcd-restore directory where we will put our backup file downloaded from s3
mkdir -p aws-etcd-restore

# syncing our local folder with s3 bucket, it only download content which are not in sync and vice versa
aws s3 sync s3://taha-k8-etcd-backup ./aws-etcd-restore

# extracting compressed tar file to our backup-dir whcih is /opt/etcd-backup
[ -f ./aws-etcd-restore/etcd-backup.tar.gz ] && tar -xf ./aws-etcd-restore/etcd-backup.tar.gz -C /

# picking the last updated file whcih is the most recent one and assign it to RECENT_BACKUP variable
RECENT_BACKUP=/opt/etcd-backup/$(ls -Art /opt/etcd-backup/ | tail -n 1)

# printing out the RECENT_BACKUP variable which holds full path to most recent etcd-backup file
echo $RECENT_BACKUP

# restoring most recent backup using etcdctl 
# by specifying --data-dir restoration will goes into the specified dir which is in our case is /opt/etcd-restore
ETCDCTL_API=3 etcdctl snapshot restore --data-dir /opt/etcd-restore $RECENT_BACKUP

Step2: change permission of backup-script.sh file to 744 which mean only owner have full read, write, execute access where group and other user have only read access. Below is the command and result snippet

$ chmod 744 ./restore-script.sh

Step3: now we will run script using below command

$ ./restore-script.sh

scheduling backup (crontab): we will use crontab to register our cronjob entry for running our backup script at every night 12 AM, use full path to your backup-script.sh. Following are the step to schedule our job with crontab

Step1: run “sudo crontab -e” without quotes command to open crontab file, it will then ask you to select editor. Use sudo for root user and and if you are normal user then no need to use sudo

Step2 : add cronjob entry in end of file, our will be
“0 0,12 * * * /root/backup-script.sh” without quotes, always use full path.

Step3: after adding cronjob verify our entry by using following command “sudo crontab -l” without quotes

Conclusion: In this project we have cover vast array of topic which directly or indirectly help you in task automation. Our primary purpose in this project is to automate etcd-dabase backup with uploading backups to cloud(s3-bucket), further more if you want to challenge yourself create backups of High Availability cluster which have multiple control plane node.

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi/

Introduction: A kubernetes cluster is consist of one or more control nodes and worker nodes, these control plane node have multiple component which works all together to make kubernetes cluster functional in proper way. In this blog we will understand each component in control-plane node and how these component works and communicate with each other. In k8-cluster master/controlplane node is like the brain of the cluster which is responsible for running all the components which are required for running our application, before doing hands-on with kubernetes like deploying application on the k8-cluster it is better to understand what are it’s components and how they function together.

Following are the topics which we discuss in this blog

• kube-apiserver
• kube-controller-manager
• node-controller
• replication controller
• etcd
• kube-scheduler

kube-apiserver: kube-api-server is central point of communication in Kubernetes it handles all internal and and external api-calls and then forward it to other components after authentication and validation of the request, it is an single point of contact to “etcd database” which directly interact with it. kube-apiserver provides sophisticated REST api’s through which we can interact with other components in the cluster, all our “kubectl” commands are passed through the kube-api-server after validation and authentication the request then passed to respective components like if we want to get pods in default ns (kubectl get pods -n default) this request will first go for authentication and validation of request and of the user once its done the request will finally pass to etcd database for retrieving the current pods state in default namespace. kube-api-server is used for updating and passing information within the cluster components in other words it is an center point of communication between all the components whenever any request is initiated or any information need to be passed within the cluster components.

kube-apiserver is installed as a binary on the system and run as a service when we setup our cluster through the hard-way. When we setup cluster through kubeadm it’s automatically run as a pod in kube-system namespace on the control-plane node(master-node).

kube-controller-manager: it is used to manages different controller within kuberenetes and it consist of multiple controller like(node-controller, replication controller, namespace controller) we will discuss the main one below. The primary purpose of kube-controller-manager is to verify the current state of cluster by communicating through kube-api-server if the current state of cluster does not matched it will instruct necessary controller to act accordingly in-order to match the state of the cluster.

kube-controller-manager can be downloaded from the official kubernetes website and run as a service or when we setup cluster through kubeadm it automatically run as a pod in kube-system namespace on the control-plane node(master-node). We can define different options to customize kube-controller-manager by editing kube-controller-manager.service file if it is installed as a service or we can edit the kube-controller-manager definition file in /etc/kubernetes/manifests folder if it’s running as a pod. We can enable and disable different controller or can edit the grace-period, monitoring-period and eviction period by editing service or definition file. We can also checks the option of running kube-controller-manager through process command

## getting kube-controller-manager pod, if it's running through kubeadm
kubectl get pod -n kube-system | grep kube-controller-manager

## for editing pod definition file of kube-controller-manager, if running through kubeadm
sudo vi /etc/kubernetes/manifest/kube-controller-manager.yml

##checking current option of kube-controller-manager
ps -aux | grep kube-controller-manager

node-controller: node controller is one of the main controller in kube-controller-manager it continuously monitors the status of the node in cluster through kube-api-server if any node not responding or not working correctly the node-controller will act accordingly by shifting the resources from unfunctional node to functional node. By default it checks status of node at every 5 sec if it doesn't get the response from the node for 40 sec it marks as Unreachable-Node/NotReady once node is marked unreachable the node-controller will give time of 5 min to node for coming back if the node doesn't respond within 5 min it removes the pod which are running on that node and assign them to different functional node

replication-controller: replication controller is responsible to monitor the replica-set, it make sure the desired number of pod in replica-set are always up , if any pod goes down replication-controller will create another pod to match the number of pod defined in replicaset definition file

etcd: if we define etcd in a nutshell we can say that it is a persistent storage where cluster information and current state of the kubernetes-cluster are stored in an key:value format and the only component which can directly interact with the etcd is an kube-apiserver any change in our k8-cluster are updated in etcd db through kube-apiserver. As kubernetes is designed in the way that all it’s component are decoupled but the only exception is with the etcd we cant remove it as if we do all the information of our cluster will be removed too, we have to make sure our etcd data is protected and should not be exploit or lost by any mean. Before we do any maintenance on k8-cluster we have to make sure that we have taken backup of our etcd db which hold the current state of our kubernetes cluster, this backup can be done through a default client name “etcdctl”

An etcd database can be installed through the package-manager and run as a service or if we setup it through kubeadm it will run as a pod in the kube-system. The default port of etcd is 2379 and the default client is etcdctl we can use it to store or retrieve data from our etcd db.

Note: In High-Availability Cluster there can be multiple master nodes and it will have corresponding etcd database, in this scenario we have to make sure that each etcd should know each other this can be done by editing the service or definition file on each master node and add url of each etcd db in the “- -initial-cluster” parameter like below
- -initial-cluster controller-0=https://{master-node1-ip}:2380,controller-1=https://{master-node2-ip}:2380,n..

PS: We will cover an important topic of how to take etcd backup and then restoring it in detail, in a different blog

kube-scheduler: kube-scheduler is used to schedule pods on the node in kubernetes cluster, the pods are not scheduled randomly in fact kubernetes have an dedicated algorithm which runs to check the best fit node available to deploy pod and once find it deploy the pod immediately on it or we can use our custom scheduler instead of default scheduler of kubernetes . The kube-scheduler algorithm checks different factor for deploying pod like what are the resources required for pod we can instruct kube-scheduler to select pod based on the parameter like min/max computation and memory resources required by a pod these resources parameter are defined in the pod definition file, once the requested resource node found then kube-scheduler will deploy or re-deploy pods. Pods can be in pending state until the requested parameter are met, several factors derive in the placement of pod on node along with the kube-scheduler algorithm like we can also used node binding or Taint and Toleration to deploy pods on the specific node of our choice.
We can download kube-scheduler binaries from the official kubernetes website and run as a service on machine or if the cluster is setup through kubeadm then scheduler will run as a pod on master node in kube-system namespace

Conclusion: So far we have cover all the component which are required in kubernetes cluster on controlplane/master node and also understand what purpose each component serve in kubernetes cluster. These components are setup differently in our cluster based on how we are setting things up like we can do it by hardway by installing each component and run them as a service on our machine or we can initiate our cluster through kubeadm where all our components are deploy as a pod on master node in the kube-system namespace and then join our worker node to master node through kubeadm join command, kubeadm is the easiest and fastest way to setting things up.
PS: In this blog we haven’t cover 2 component (kubelet and kube-proxy) as these components are not the main components of controlplane-node and it is common to worker node also, so we will discuss these component in detail in different blog

See you in the next topic, until then keep hustling and keep learning!

Follow me on
https://www.linkedin.com/in/taha-mashhadi/