(Introduction + Basics + Why it Matters)

1. What is the Shared Responsibility Model?

Let’s imagine you rent an apartment.

• The landlord makes sure the building is safe, the lifts are working, fire alarms are installed, and water and electricity lines are fine.
• But you, as the tenant, must lock your door, keep your valuables safe, and make sure you don’t leave the stove on.

That’s exactly how AWS works.

• AWS (the landlord) is responsible for securing the infrastructure — the physical servers, data centers, networking, and hardware.
• You (the customer) are responsible for what you do inside — like securing your data, managing access, and configuring services correctly.

This clear line of division is called the AWS Shared Responsibility Model.

👉 AWS secures “Security OF the Cloud”.
👉 Customers secure “Security IN the Cloud”.

2. Why Does It Matter?

A lot of companies think:
“Hey, we are on AWS, so everything is automatically secure!” ❌
That’s wrong.

Here’s a real-world example:

• In 2019, a major financial company had a data breach.
• The issue wasn’t with AWS servers. The issue was that the company misconfigured their S3 bucket (they left it open to the internet).
• Who was at fault? The customer. Because S3 is secure by default, but the customer didn’t configure access properly.

That’s why AWS created the Shared Responsibility Model — so that both AWS and customers know who is responsible for what.

3. High-Level Division

Here’s the most important line you should remember:

• AWS Responsibility → Security OF the Cloud
  (infrastructure, hardware, software that runs cloud services)
• Customer Responsibility → Security IN the Cloud
  (data, applications, identity, access control, configurations)

Think of it like an airport:

• The airport authority takes care of runways, buildings, and security systems.
• But you, the passenger, must carry valid ID, not bring dangerous items, and keep your luggage safe.

Both sides need to do their part.

4. Cloud Service Models and Responsibility

Responsibilities also change depending on the cloud service model you use:

1. IaaS (Infrastructure as a Service) → e.g., Amazon EC2

• AWS secures hardware, networking, and virtualization.
• You secure the OS, apps, and data.

2. PaaS (Platform as a Service) → e.g., AWS Elastic Beanstalk

• AWS manages infrastructure + runtime.
• You just secure your app and data.

3. SaaS (Software as a Service) → e.g., Amazon WorkMail

• AWS manages almost everything.
• You only manage your data and users.

AWS Responsibilities: Security “OF” the Cloud

1. Big Idea

AWS says:
👉 “We take care of everything that makes the cloud run. You only focus on what you put inside the cloud.”

So AWS is responsible for Security OF the Cloud.

This includes:

• Physical data centers
• Networking infrastructure
• Hardware & virtualization layer
• Global AWS services availability

Let’s break this down step by step.

2. Physical Security

Think about it: AWS has data centers all over the world. These are like super secure banks, but instead of gold, they store data.

AWS responsibilities here:

• Guards at the gate
• Biometric access (fingerprint, retina scan)
• 24/7 CCTV monitoring
• Fire suppression systems
• Climate control (servers don’t like heat 🔥)

Real-life example:
You never have to worry about someone breaking into the building where your EC2 instance runs. AWS already guards that.

3. Hardware & Infrastructure

AWS owns and manages:

• Servers
• Storage devices (disks)
• Networking equipment (switches, routers)

They make sure:

• Hardware failures are detected and replaced quickly.
• Data is replicated across availability zones (so if one disk dies, your data is still safe).

👉 Example: If an AWS hard disk fails in their data center, AWS replaces it. You don’t even know it happened.

4. Networking

AWS provides a global secure network.

They handle:

• Firewalls at the network level
• DDoS protection with AWS Shield
• Secure communication between regions & AZs
• Encryption in transit (TLS)

👉 Example: When you send data from one AWS region to another, you don’t need to worry about hackers sniffing packets in between. AWS already encrypts and secures the backbone network.

5. Virtualization Layer

Most AWS services run on virtual machines. AWS manages the hypervisor that divides physical servers into multiple VMs.

They make sure:

• One customer’s VM can’t “peek” into another customer’s VM.
• Isolation between accounts is always maintained.

👉 Example: If your EC2 runs on the same physical server as another company’s EC2, don’t worry. AWS guarantees they cannot see or touch your instance.

6. Managed Services

For managed services like S3, RDS, DynamoDB, AWS takes care of:

• Patching the underlying OS
• Ensuring uptime (99.99% SLA in many cases)
• Scaling infrastructure on demand

👉 Example: You don’t need to install antivirus or patch the OS inside Amazon S3. AWS already secures it.

7. Compliance & Certifications

AWS spends millions of dollars getting certifications like:

• ISO 27001
• SOC 1/2/3
• PCI DSS
• HIPAA

This means if your business needs to follow these rules, AWS already has a strong foundation.

👉 Example: If you’re building a healthcare app, AWS already meets HIPAA standards at the infrastructure level. You only need to ensure your app complies.

8. Availability & Redundancy

AWS ensures their cloud is always available:

• Multiple data centers in each region (called Availability Zones).
• Backup power, internet, cooling.
• Disaster recovery plans.

👉 Example: Even if one AZ in Mumbai goes down, AWS automatically shifts load to another AZ in the same region.

9. Summary of AWS Responsibilities

• Data center security
• Physical hardware
• Networking backbone
• Hypervisor/virtualization layer
• Managed service patching
• Global availability
• Compliance certifications

In short:
AWS provides the secure foundation. Customers build on top

1. Introduction

When you use apps like Swiggy, Zomato, Ola, Uber, or Paytm, have you ever thought about how these apps connect to their backend servers?

For example:

• You open Zomato → search for “Pizza”.
• The app shows you a list of restaurants.
• You place an order → payment happens → delivery tracking starts.

Behind the scenes, all this happens because of something called APIs.

What is an API?

API stands for Application Programming Interface.

In simple words:

• API is like a messenger between two systems.
• It takes your request, tells the server what you need, and brings back the response.

👉 Real-life example:
Think of a restaurant waiter.

• You (the customer) ask for food → that’s the request.
• The waiter takes your order to the kitchen (server).
• The kitchen prepares the food and gives it back to the waiter.
• The waiter serves it to you → that’s the response.

Here:

• You = Client (like your mobile app)
• Kitchen = Server (where the business logic is running)
• Waiter = API

This is exactly how apps talk to servers.

What is an API Gateway?

Now imagine a big hotel with 20+ different kitchens — Indian, Chinese, Italian, Dessert, Beverages.
If customers directly run to the kitchens, it will create chaos.

So the hotel keeps one reception desk (gateway).

• You place your order at reception.
• Reception decides which kitchen to send your order to.
• Then it brings back the food.

This reception desk is like an API Gateway.

👉 Definition:
API Gateway is a service that acts as a front door for your applications (mobile apps, web apps, etc.) to access data, business logic, or backend services.

It manages:

• Who can enter (security)
• How many people can enter at a time (rate limiting)
• Where to send requests (routing)
• And keeps logs of everything(cloudwatch)

Why AWS API Gateway?

There are many API Gateways (Kong, Apigee, etc.), but AWS API Gateway is fully managed and deeply integrated with AWS services like Lambda, DynamoDB, EC2, S3.

It is serverless, meaning:

• You don’t manage any servers
• You pay only when someone uses your API
• It automatically scales from 1 request per day to millions per second

That’s why it is very popular for startups, enterprises, and even personal projects.

2. Why do we need AWS API Gateway?

Let’s take a simple example.

Imagine you are building an Online Shopping App.

• Mobile app → sends requests like Get Product List, Add to Cart, Make Payment.
• Backend → multiple services: product service, order service, payment service.

Problem without API Gateway:

• Your mobile app has to directly talk to each backend service.
• You must manage authentication for each service separately.
• Each service might have a different endpoint.
• You will face issues in scaling, throttling, and monitoring.

This becomes very complex very fast.

Solution with API Gateway:

• Your mobile app talks only to one endpoint (API Gateway).
• API Gateway then routes the request to the correct backend service.
• It manages authentication, throttling, caching, and logging for you.

👉 Think of it like Swiggy delivery partner.

• Customer places an order on Swiggy app.Swiggy assigns a delivery person.
• That delivery person coordinates with restaurant, pickup, and drop.
• Customer only interacts with Swiggy → not directly with the restaurant.

This is exactly what API Gateway does.

Next part →Core Concepts of AWS API Gateway (Endpoints, Methods, Resources, Stages, Integrations, etc.) with real-life examples.

In modern software development, ensuring smooth deployments with minimal downtime is critical. Two popular advanced deployment strategies that achieve this are Canary Deployments and Blue-Green Deployments. These strategies help reduce risks, enable faster rollbacks, and improve user experience. Additionally, observability is crucial for monitoring these rollouts effectively.

What is a Canary Deployment?

A Canary Deployment is a strategy where a new version of an application is gradually released to a small subset of users before rolling it out to everyone. This approach allows teams to monitor performance, identify issues, and mitigate risks before full deployment.

How Canary Deployment Works:

1. Release to a Small Group: Deploy the new version to a small percentage of users.
2. Monitor Metrics: Track performance, errors, and user feedback.
3. Gradual Rollout: If everything looks good, slowly increase traffic to the new version.
4. Rollback if Needed: If issues arise, revert to the previous version immediately.

Advantages of Canary Deployment:

• Risk Mitigation: Issues affect only a small group of users.
• Faster Rollbacks: Quick recovery if something goes wrong.
• Performance Insights: Real-time feedback from actual users.

Tools for Canary Deployments:

• Kubernetes with Argo Rollouts or Flagger.
• Service meshes like Istio for traffic splitting.

What is a Blue-Green Deployment?

A Blue-Green Deployment involves running two identical environments: Blue (current version) and Green (new version). Traffic is switched from the blue environment to the green one instantly.

How Blue-Green Deployment Works:

1. Deploy Green: Deploy the new version (Green) alongside the current version (Blue).
2. Switch Traffic: Redirect user traffic from Blue to Green.
3. Monitor and Validate: Ensure the Green environment performs well.
4. Keep Blue for Rollback: Retain Blue for immediate fallback if issues arise.

Advantages of Blue-Green Deployment:

• Zero Downtime: Traffic switches instantly without downtime.
• Instant Rollback: Return to Blue immediately if needed.
• Seamless User Experience: Users experience no service interruptions.

Tools for Blue-Green Deployments:

• AWS Elastic Load Balancing (ELB)Kubernetes Services with Load Balancers
• CI/CD tools like GitLab CI/CD or Jenkins

Observability in Progressive Rollouts

Observability is essential for monitoring deployments and ensuring their success. It involves collecting and analyzing data from logs, metrics, and traces.

Key Components of Observability:

• Metrics: Measure system performance (e.g., CPU usage, error rates).
• Logs: Capture detailed records of events and errors.
• Traces: Track requests across services for better visibility.

Tools for Observability:

• Prometheus & Grafana: Collect and visualize metrics.
• Loki & Promtail: Centralize and analyze logs.
• Jaeger or OpenTelemetry: Monitor distributed traces.

Combining Strategies with Observability

In large-scale deployments, teams often combine Canary and Blue-Green strategies with strong observability to enhance reliability. For example:

• Start with a Canary Deployment: Release gradually and observe performance.
• Switch to Blue-Green for Full Rollout: Once confident, redirect all traffic.
• Use Observability Tools: Continuously monitor and quickly respond to issues.

Canary and Blue-Green deployments are powerful strategies for achieving safe, efficient, and zero-downtime deployments. Combining these approaches with observability ensures successful rollouts, quick issue detection, and fast recovery. Using tools like Kubernetes, Prometheus, and GitLab CI/CD, teams can achieve seamless and reliable software delivery.

Completion Steps :-

Phase 1 :Continuous Integration on Azure Devops Platform

Step 1 : Setup Azure Repos

Step 2 : Set Azure Container Registeries

Step 3: Create Azure Pipelines

Step 4: Setup agent pool

Step 5: Setup Pipelines for Voting and worker microservices

Phase 2 : Continuous Deployment using gitops

Step 1: Kubernetes Cluster Creation using Azure Kubernetes Service (AKS)

Step 2: Setup Azure CLI

Step 3: Install and configure Argo-CD

Step 4: Deploy Application on argocd

Phase 3: Automating the whole cicd part

Step 1: Creating a shell script

Step 2 : Added a new stage in pipeline

Step 1 : Setup Azure Repos

1. Go to your Azure portel and search for devops organization and create a new organization

2. click on Azure Repos and then click on Import option

3. Paste the github project repo link

https://github.com/Aakibgithuber/Azure-devops-CI-CD-GitOps.git

4. Provide the github project repo URL and then click on Import

5. It will fetch all the Files and branches from github

6. Now you have to set main branch as default branch

7. Select branches from left menu and then click on three dots of main branch where you find a option to set main branch as a default branch

8. We set main branch as default one beacause the latest code will always have to be in main branch

Step 2 : Set Azure Container Registeries

What is Azure Container Registeries → It is central repositories in azure for storing docker images just like dockerhub

1. Go to azure portel and search for container registeries

2. Create a new container registery

3. Keep the setting as it is mentioned in below image

4. Your central repository for docker images is ready to use

Step 3: Create Azure Pipelines

What is Azure Pipelines → It helps you automatically build, test, and deploy your code just like jenkins It’s like a robot that takes your code, checks if it works correctly, and sends it to where it needs to be (like a website or app store) without you doing it manually every time

1. Click on Pipelines Option present in the left menu

2. In our case we fetch the code from github and store it in Azure Repos so we have to select Azure repos from options given below

3. These are the pre templates of your pipeline code in our case we are using docker for building and pushing the image to ACR

4. Select the subscription I only have pay as you go type so i need to chose this

5. Select the container registry we made earliar in the blog

6. It will provide a pre template of pipeline code now we have to make some changes

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger: #This option will only trigger a pipeline when you only make any change in the result folder
 paths: 
  include:
  - result/*

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: 'e05f07b8-d307-47cb-87d1-c75b657fa09b'
  imageRepository: 'resultapp'
  containerRegistry: 'microcicd.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)/result/Dockerfile' 
  tag: '$(Build.BuildId)'

pool: 
 name: 'azureagent'

stages:
- stage: Build
  displayName: Build 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'build'
        Dockerfile: 'result/Dockerfile'
        tags: '$(tag)'

- stage: push
  displayName: push 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'push'
        tags: '$(tag)'

1. Trigger

trigger:
  paths:
    include:
      - result/*

• This tells Azure Pipelines to only trigger this pipeline when you make a change to files in the result folder.
  For example, if we update result/Dockerfile, the pipeline will run. If you change something outside this folder, it won't trigger.

2. resources

resources:
- repo: self

• This tells the pipeline to use the same repository (self) where this YAML file is stored.

3. variables

variables:
  dockerRegistryServiceConnection: 'e05f07b8-d307-47cb-87d1-c75b657fa09b'
  imageRepository: 'resultapp'
  containerRegistry: 'microcicd.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)/result/Dockerfile'
  tag: '$(Build.BuildId)'

• These are reusable values (variables) used later in the pipeline:

1. dockerRegistryServiceConnection: Refers to the Azure service connection that allows the pipeline to communicate with the Azure Container Registry (ACR). This is the unique ID of the service connection.
2. imageRepository: The name of the repository in the container registry where the image will be stored. Here, it’s called resultapp.
3. containerRegistry: The URL of our Azure Container Registry (microcicd.azurecr.io).
4. dockerfilePath: The path to the Dockerfile. $(Build.SourcesDirectory) is a predefined variable pointing to the root of the source code.
5. tag: A unique identifier for the Docker image, generated automatically for each pipeline run. $(Build.BuildId) is a unique number provided by Azure Pipelines for each build.

4. pool

pool:
  name: 'azureagent'

• Specifies the pool of agents (virtual machines) to run the pipeline. Here, we’re using the azureagent pool provided by Azure.

5. stages

This section defines the pipeline’s main stages. we have two stages: Build and Push.

This error occurs because the agent pool specified in your pipeline (azureagent) Does not exist there is no agent pool with the name azureagent in our Azure DevOps organization. so we need to setup agent pool for azureagent

Step 4: Setup agent pool

An agent pool in Azure DevOps is like a group of workers (computers or virtual machines) that perform tasks in our pipeline, such as building, testing, or deploying our code.

How It Works:

• When we run a pipeline, it looks for an available “worker” in the agent pool to carry out the tasks we have defined.
• Each worker is called an agent, and it runs one job at a time.

Why It’s Useful:

1. Resource Management: It allows we to manage and organizeyour agents into groups, so we can use them for specific pipelines or projects.

Types of Agent Pools:

1. Microsoft-Hosted Agent Pool:

• Provided by Azure DevOps.
• Includes pre-configured virtual machines (e.g. Windows, Linux, macOS).
• You don’t need to maintain or set up anything just use it.

2. Self-Hosted Agent Pool:

• Machines that you set up and maintain (e.g. your own servers or VMs).
• Useful if you need:
• More customization (e.g., special software or configurations).
• To avoid limitations in Microsoft-hosted agents.

In our Case we are going to use Self-Hosted Agent Pool

First we need to create Virtual Machine in which our agent is being hosted

1. Go to VM section and click on create

2. Now click on Agent pools option under project setting option

3. Click on Add pool

4. Select self hosted from the options and provide a name to your agent pool and right tick on grant access permission on all pipelines

5. You have provided with certain command to run on your VM that setup your agent

6. Go to your VM and copy the public IP

7. Take ssh of your vm and run the follwing commands

for ssh you need to run (please ensure that your pem file must be present in that particular location)

ssh -i <.pem_file> azureuser@<public_ip>

8. Now Run the folloiwng commands

wget <paste_the_copied_link>
apt update

9. Now list the items and run the following command to extract the file

tar zxvf vsts-agent-linux-x64-4.248.0.tar.gz

This command extracts the contents of the tarball into the current directory

10. After Extracting you have to run .sh file to setup agent

/.config.sh

11. you have to provide server URL which you can get from the documentation self-hosted linux agents on azure →documentation link

12. Copy the server url and paste it there(edit the organization part and provide your devops organization name)now it will asked you to provide the personal acess token(PAT) which you need to generate

14. On the agents tab click on setting > personal access tokens

15. Provide a name to your token select your organization and also provide full authorization acess to your tokenn

16. Copy Your token and paste it on terminal

17. Now enter your agent pool name that we created earliar

18. As you see your agent is offline to make it available for our use we need to install docker and provide neccessary permission to docker

sudo apt install docker.io
sudo usermod -aG docker azureuser 
sudo systemctl restart docker
./run.sh

19. Your agent is now online is ready to use

20. If you are Going to pipeline section you will see your pipeline will autmatically triggered by the agent

Now everything is setup you all need to make two more pipelines for voting and worker microservices

Step 5: Setup Pipelines for Voting and worker microservices

1. go to pipelines section select the code repo as azure repos , chose the pre-template as the docker and select your subscription

Pipeline code for Voting Service

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
 paths:
  include:
  - vote/*

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: 'e05f07b8-d307-47cb-87d1-c75b657fa09b'
  imageRepository: 'votetapp'
  containerRegistry: 'microcicd.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)/vote/Dockerfile'
  tag: '$(Build.BuildId)'

pool:
 name: 'azureagent'

stages:
- stage: Build
  displayName: Build 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'build'
        Dockerfile: 'vote/Dockerfile'
        tags: '$(tag)'

- stage: push
  displayName: push 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'push'
        tags: '$(tag)'

Pipeline code for Worker Service

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
 paths:
  include:
  - worker/*

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: 'e05f07b8-d307-47cb-87d1-c75b657fa09b'
  imageRepository: 'workerapp'
  containerRegistry: 'microcicd.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)/worker/Dockerfile'
  tag: '$(Build.BuildId)'

pool:
 name: 'azureagent'

stages:
- stage: Build
  displayName: Build 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'build'
        Dockerfile: 'worker/Dockerfile'
        tags: '$(tag)'

- stage: push
  displayName: push 
  jobs:
  - job: Build
    displayName: Build the image
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'push'
        tags: '$(tag)'

As of now you have successfully created 3 pipelines and all your pipelines successfully build and push docker images to Azure Container Registeries

Here we completed our Phase 1 that is CI part on Azure Devops Platform Let’s move to CD part

Phase 2 : Continuous Deployment

Step 1: Kubernetes Cluster Creation using Azure Kubernetes Service (AKS)

1. Go to Azure portal and search for kubernetes service
2. click on create Kubernetes cluster from the dropdown

3. Keep the Everything as it is shown in the image

4. Click on nodepool to update the no. of instances used in autoscaling

What is NodePool →A node pool in Azure Kubernetes Service (AKS) is simply a group of virtual machines (VMs) that act as the workers for your Kubernetes cluster. These VMs are where your applications run. Each node pool contains multiple VMs called nodes and each of these nodes has the tools Kubernetes needs to manage and run your applications

5. your Cluster is now Ready to orchestrate

Step 2: Setup Azure CLI

1. Go to your Terminaland Run the Following commands to install azure CLI

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az --version

2. Now for login run —

az login

3. It will Redirect you to web broswer

4. click on continue select your account and then go back to your terminal

5. You are now login to access your azure account via CLI

az account show

Step 3: Install and configure Argo-CD

1. On your termninal run the following command

az aks get-credentials --name AzureDevops --overwrite-existing --resource-group Demo-RG

This command fetches the access credentials for your Azure Kubernetes Service (AKS) cluster, so you can connect to it and manage it using kubectl

• --name AzureDevops: This specifies the name of your AKS cluster. In this case, the cluster is named AzureDevops.
• --resource-group Demo-RG: This tells Azure which resource group your cluster is in
• --overwrite-existing: If your current configuration already has credentials for a cluster with the same name, this option replaces them with the new credentials you’re fetching. This avoids confusion or conflicts if you were previously connected to another cluster

2. Installation of Kubectl

Run the following command to install kubectl on your VM

sudo apt update
sudo apt install curl -y
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

3. Create a namespace for argocd

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

4. Above command will downloads the install.yaml file from the Argo CD GitHub repository.

• The file contains a predefined set of instructions to install Argo CD into your Kubernetes cluster.
• Kubernetes will create all the necessary resources (like Deployments, Services, Pods, etc.) required to run Argo CD in the argocd namespace

Kubectl get pods -n argocd

5. To access the UI of Argocd we need credentials that is stored as secret

kubectl get secrets -n argocd

6. Open the yaml file for admin secret

kubectl edit secrets argocd-initial-admin-secret -n argocd

7. Copy the password

8. Initially it is base64 encoded and we need to decode the password

#run the following command to encode the secret
echo <encoded_password> |base64 --decode

Base64 encoding is a way to convert data (like text, images, or files) into a format that can be safely transmitted or stored as plain text

Why it’s called “Base64”: There are 64 characters in the Base64 encoding scheme (A-Z, a-z, 0–9, +, /). These are used to represent the data

If you want to encode the word “Cat”:

1. Convert the text into its binary format: Cat becomes 01000011 01100001 01110100.
2. Split it into chunks and encode each part using Base64 rules. The result: Q2F0.

So, the Base64 encoding of “Cat” is Q2F0

9. Copy the decoded password and paste it in notepad

kubectl get svc -n argocd

# This will display a list of all services in the argocd namespace

10. Checkout service for the argocd-server

11. As we see the argocd server expose as ClusterIP we need to change this to NodePort

If you want to open Argo CD’s UI in your web browser or interact with its API from outside the cluster, ClusterIP won’t work. Switching to NodePort exposes it so you can reach it via <Node IP>:<NodePort>

For example:

• ClusterIP: Accessible only inside Kubernetes (e.g., http://argocd-server:8080).
• NodePort: Accessible outside (e.g., http://<NodeIP>:32000).

Think of ClusterIP as a private door that can only be opened from inside your house (the cluster). Changing it to NodePort adds a public door that anyone outside your house (the cluster) can use, so you can easily access Argo CD from your browser or tools

11. Run the Following command to edit the argocd-server manifest

kubectl edit svc argocd-server -n argocd

12. Look for type and change It to NodePort

kubectl get svc -n argocd

13. Go to your azure portal type vmss

14. click on agent pool>networking

15. Create a New port rule and open the port no. 30645 that is our argocd port

16. Go back to your terminal

# this command will provide the externalip of node
kubectl get nodes -o wide

17. copy the external ip and go to your browser type http://<public_ip>:<port_no.>

18. Now go back to broswer and proceed to advanced

19. username == admin password is what you decoded earliar in the blog

Step 4: Deploy Application on argocd

1. Connect With Azure Repos

1. First you need to conned the Azure repo which contain the source code
2. click on setting>repositories

3. Change the connection method to HTTPS

4. For repository url go to azure repos and then click on clone

5. Paste in the URL section

6. As it is a private repo so we need to token that we had generated earliar for connection paste your token here

7. We need to change the Application refresh timout to 10 sec by default it is 3 min. So we need to edit the config file of argocd and add the section of application refresh timeout

kubectl edit configmap argocd-cm -n argocd

#Restart the argocd server
kubectl rollout restart deployment argocd-server -n argocd

2. Create a new Application

1. Click on New app section on argocd

2. Keep all the details as it shown in the below images

3. In the path section provide the manifest location. All the manifest is stored in k8s-specification folder

4. Your Application is now deployed on Azure Kubernetes Cluster via argo-cd

5. Go to your terminal check for the pods and services and open the no. in which your services is running

kubectl get pods
kubectl get svc

6. go to yourazure portal and search for vmss>nodepool and open the 31000,31001 port that is for vote app

7. collect the node external ip and again fo to your web broswer type http://<node_ip>:<port_no.>

kubectl get nodes -o wide

Here we done Continuous Deployment Part of our Application using GitOps Approach

Phase 3: Automating the whole cicd part

Step 1: Creating a shell script

1. Go to Azure repos click on three dots and make a new repository
2. folder name==scripts , file name ==updateK8sManifests.sh

3. Paste the Following code in the .sh file

#!/bin/bash

set -x

# Set the repository URL
REPO_URL="https://<ACCESS-TOKEN>@dev.azure.com/<AZURE-DEVOPS-ORG-NAME>/voting-app/_git/voting-app"

# Clone the git repository into the /tmp directory
git clone "$REPO_URL" /tmp/temp_repo

# Navigate into the cloned repository directory
cd /tmp/temp_repo

# Make changes to the Kubernetes manifest file(s)
# For example, let's say you want to change the image tag in a deployment.yaml file
sed -i "s|image:.*|image: <ACR-REGISTRY-NAME>/$2:$3|g" k8s-specifications/$1-deployment.yaml

# Add the modified files
git add .

# Commit the changes
git commit -m "Update Kubernetes manifest"

# Push the changes back to the repository
git push

# Cleanup: remove the temporary directory
rm -rf /tmp/temp_repo

Don’t forget to change the token name , ACR name and your organization name

What Does This Script Do?

1. Clones a Git repository from Azure DevOps into a temporary directory.
2. Modifies a Kubernetes deployment YAML file to update the image tag.
3. Commits and pushes the changes back to the repository.
4. Cleans up by deleting the temporary directory

Step 2 : Added a new stage in pipeline

1. Go to Voting pipeline and edit a new stage Update

What does this stage do → This stage will run the script that we created earliar

# $1=vote $2=repository_name=votetapp $3=latest tag of the image these arguments
# will used in the script we created or at the line given below

sed -i "s|image:.*|image: <ACR-REGISTRY-NAME>/$2:$3|g" k8s-specifications/$1-deployment.yaml

#THE above line will change the image tag to the lastest one

2. when the Update stage is completed the changes will reflect in the vote-deployment.yaml file

3. Let’s make a change in app.py that will trigger a new pipeline

4. Push Stage will push the image into ACR with the latest tag

5. Then update stage will run the script that update the image name in the deployment.yaml with the latest tag

6. Argocd will look for a change in k8s-specifications folder and found a new image as the tag was changed to latest one

7. As this change in the k8s-specifications folder will envoke the argocd to change application according to new image

8. But how argocd will fetch these changes from private container registeries for this we need to create secrets

9. For this to resolve we need to create secrets that allow argocd to pull images from private repos

10. Go to ACR > Access keys and click on generate and copy the username and password and go back to your terminal

11. Copy the below command and make the accordingly

service-principal-id == username of your ACR

Service-principal-password == Password you copied

kubectl create secret docker-registry <secret-name> \
  --namespace argocd \
  --docker-server=<your-acr-name>.azurecr.io \
  --docker-username=<your-acr-username> \
  --docker-password=<your-acr-password> \
  --docker-email=<your-email>

12. Go to your Vote-deployment.yaml and add the follwing line

imagePullSecrets:
- name: acr-secret

13. Now Again go to your vote/app.py to make changes that will triggerd the pipeline again

Go to your Web browser and and check for the change

Here is your updated application

From here we have completed our blog Thanks for reading

Follow me on linkedin

Follow me on github

In this blog, we will break down how Prowler works , why AWS security is important , and how you can use this tool to make sure your AWS environment is safe

Why Security in AWS is Essential

When you use AWS, both you and AWS share responsibility for keeping things secure. AWS is responsible for the security of the cloud , like managing data centers and hardware . But you are responsible for securing everything inside the cloud, like how you manage user access, set up servers , and protect data

If something is not set up properly in your AWS account like if sensitive data is not protected it could lead to a data theft or other security problems . Regularly checking your setup to make sure it follows best security practices helps prevent these issues. That is where Prowler can help you to enhance your AWS security

What is Prowler

Prowler is a free tool that helps you check your AWS account for security problems . It checks your AWS setup and finds any weak spots that could lead to attacks

Prowler looks at different parts of your AWS account , such as:

• Networking (VPC) : Are your network configurations secure, and are access points restricted?
• Virtual machines (EC2) : Are your servers secure?
• Security Groups : Are your firewall rules properly configured to limit access ?
• User permissions (IAM) : Do your users and roles have the right permissions and access ?
• Storage (S3) : Are your files secure, and who can access them?
• Logging (CloudTrail) : Is all activity in your AWS account being monitored
• Database (RDS) : Is your data encrypted and protected from unauthorized access ?
• Backups (EBS/Snapshot) : Are your backups secure and regularly tested for recovery ?
• Multi-Factor Authentication (MFA) : Is MFA enabled for all users to add an extra layer of security ?

It also tells you how to fix any issues if finds .

Key Features of Prowler

1. CIS Benchmark Checks

The CIS Benchmark is a set of rules that explains how to keep an AWS environment secure . Prowler checks if your AWS setup follows these rules , making sure there are no unsafe settings

2. Scans large no. of AWS Services

AWS has many services (like EC2, S3, and IAM), and each one needs to be configured securely. Prowler scans many of these services to find security issues, such as:

1. EC2 instances allowing anyone on the internet to connect.
2. S3 buckets that are publicly accessible when they should not be.
3. Not having encryption, which keeps sensitive data safe

3. Helps in Following the rules and regulations.

If your company needs to follow certain security rules (like GDPR or HIPAA), Prowler can help by checking if your AWS environment meets these requirements. For example , it will verify if your data is encrypted or if you are keeping track of all changes and activity in your account.

4. Customizable

Prowler comes with many built in checks , but you can also create your own checks if you have specific needs . This makes it flexible for companies with unique security requirements.

5. Detailed Reports

After Prowler runs a scan, it creates a report showing all the security issues it found. The report also suggests how to fix these problems. This helps both technical teams and management understand what needs to be done to improve security.

How to Install and Setup Prowler

The first step in using Prowler is to install it on your computer or system. Prowler is a command line tool , which means you will be using a terminal or command prompt to run it

1.Installation of Prowler :

1. Open a terminal on your computer. If you are using Windows, you can use Git Bash or PowerShell. On macOS or Linux, use the default terminal. In my case I am using ubuntu
2. Run the following command in your terminal to download Prowler:

git clone https://github.com/prowler-cloud/prowler

3. This command clone the github repository in your local machine

4. Run the follwing comands to install prowler

cd prowler
apt install python3-poetry
poetry shell
poetry install
python prowler.py -v

5. Now Prowler is installed and ready to run.

2. Configure Your AWS Credentials

To allow Prowler to scan your AWS environment, it needs permission to access your AWS account. This is done through AWS IAM credentials. Before running Prowler, make sure you have the AWS CLI is installed in your local machine. If you don’t have the aws cli installed then follow the below command to install the aws cli

1. Installing aws-cli

sudo su
apt update 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt-get install unzip -y
unzip awscliv2.zip
sudo ./aws/install
aws --version

2. Configure aws

1. Create an IAM user

click on iam

2. click on user →create user

3. Give a name to your user and tick on provide user access to management console and then click on I want an IAM user option

5. choose a password for your user →click next

6. Attach the policies directly to your iam user → click next

note →I will provide the administrator accesss for now but we careful while attaching the policies at your workapce

review and create user

7. click on create user

8. download your password file if it is autogenerated otherwise it is your’s choice

9. Now click on your Prowler IAM user →security credentials

10. scroll down to access keys and create an access keys

11.choose aws cli from the options listed

12. click next and download you csv file for username and password

13. go to your terminal and type →aws configure

14. Now it is ask to your access key and secret key for this open your csv file and paste the access and secret key and remain everything default

15. Now you are ready to configure aws from your terminal

16. Once your AWS credentials are properly set up, Prowler will be able to scan your AWS environment for security issues.

3. Run a Basic Security Scan

Now you have Prowler installed and your AWS credentials configured, you can run your first security scan. This basic scan checks your AWS environment against the CIS AWS Foundations Benchmark that is a set of security practices.

To see a prowler dashboard run the following command

prowler dashboard

To start the basic scan, type the following command into your terminal:

prowler aws #if you are working with aws

What this command does:

• It runs all the built-in checks for different AWS services like EC2 , S3 , IAM and more.
• Prowler will start listing the results of the scan directly in your terminal. It will show which checks passed and which ones failed .

For each failed check , Prowler will tell you what the issue is (for exmple “S3 bucket is publicly accessible”) and how to fix it ( Turn off public access).

To check the no. of services prowler support

prowler aws --list-services

4. Focus on Specific Services

Sometimes, you might only want to check a particular AWS service instead of running a full scan . For example → you might want to check the security of your S3 storage or your EC2 instances without scanning everything.

To do this, you can run more focused scans. For example, if you want to check just your S3 storage, use the following command:

./prowler -g s3

This command only checks S3 buckets and makes sure that they are configured securely . If you want to check EC2 instances you can use :

prowler aws --service s3 

This is helpful when you are trying to fix an issue with a particular service or when you want to regularly monitor specific areas of your setup.

5. Generating Detailed Reports

After Prowler finishes scanning your AWS environment , you might want to save the results in a report. Prowler allows you to export the results in different formats like HTML , CSV or JSON .

For example , if you want a clean , easy to read HTML report that you can share with your team , run this command :

prowler aws --output prowler_report --output-format html

This will run the scan and save the output in prowler_report.html in the current directory.

prowler aws — service s3 — output s3_report — output-format html

This command will check S3-related configurations and save the report as s3_report.html.

This command creates an HTML file with all the results of the scan, including which checks passed or failed and recommendations on how to fix issues. You can open this report in a web browser, making it easy to review and share.

Other formats, such as CSV for use in spreadsheets and JSON , are also available, depending on how you want to work with the data.

Best Practices for AWS Security Using Prowler

Now that you understand how to use Prowler, let’s explore some best practices to help you get the most out of it . These tips will help you use Prowler helps you maintain the security of your AWS environment continuously, rather than just checking it once.

1. Automate Regular Security Scans

Cloud security is an ongoing effort, not something you do just once Even if your AWS environment is secure today, changes happen new resources get added, configurations change, and new risks can appear . This is why it is important to regularly scan your AWS account.

You can set up Prowler to run these scans automatically on a regular basis. By doing this, you won’t need to remember to manually run checks every week or month. There are several ways to automate these scans :

• Use AWS Lambda: You can set up Lambda functions to automatically run Prowler on a schedule, like daily or weekly
• Add to CI/CD pipeline: If you are using a CI/CD pipeline, integrate Prowler as part of your deployment process. This way, Prowler can check your environment each time you deploy new updates or make changes to your AWS infrastructure.

By automating scans, you ensure that potential security risks are detected quickly before they become major issues.

2. Regularly Review IAM Permissions

One of the most important aspects of AWS security is managing who has access to your AWS account and what they are allowed to do. AWS Identity and Access Management (IAM) controls this, and it is essential to regularly check that permissions are set correctly .

Over the time many users may collect permissions that they don’t need anymore, or new users might be given too broad access. This can increase the risk of accidental or malicious actions.

Prowler helps you regularly review your IAM setup by doing checks for

• Users or roles have too many permissions .
• There are any unused accounts or roles that should be deleted.
• Multi factor authentication (MFA) is enabled to add an extra layer of security.

Make it a habit to audit IAM permissions and clean up unused or excessive permissions. Always apply the principle of least privilege, which means giving users only the access they need and nothing more.

3. Implement Encryption Across Services

Encryption ensures that your data is protected, even if someone manages to access it. AWS allows you to encrypt data both at rest (when it is stored, like in S3 or RDS databases) and in transit (when it is being moved, like over the internet).

Prowler checks if your data is encrypted correctly by doing checks

• Are your S3 buckets encrypted ?
• Are your RDS databases using encryption to protect stored data ?
• Is encryption being used for data in transit, like when it is sent from your website to a database ?

Make sure encryption is enforced for all sensitive data , and regularly use Prowler to confirm that these settings remain in place.

4. Strengthen Security Group Settings

AWS security groups act like a firewall for your EC2 instances. They control who can access your servers and what kind of traffic is allowed in or out. It is important to keep these settings tight to prevent unauthorized access.

One common mistake is leaving security groups too open. For example allowing SSH from anywhere on the internet is risky. Prowler can help you identify:

• Security groups that are too permissive, such as allowing anyone to connect to your servers.
• Unnecessary open ports that might be exposing your EC2 Instances

After running a Prowler scan, review any flagged security groups and restrict access to only what is necessary. This will help reduce the chances of an external attack.

To wrap up :

Keeping your AWS environment secure is really important today. Using tools like Prowler helps you check and improve your security regularly , so your data stays safe and meets necessary rules. Remember security is not just a one time job it is something you need to do all the time. By following good practices and checking your settings more often, you can protect your cloud systems from risks and keep your business running well. Stay alert and stay safe!

Project Overview

In this project we deploy 10 microservices in kubernetes cluster using jenkins pipeline….first we launch a Base server in which all the task is performed….then we take ssh of that EC2….next we install docker , jenkins , configure java ,eksctl , kubectl etc… moving forward to setup k8s cluster in aws that is EKS and atlast we run a multibranch pipeline that fetch code from github and deploy it on EKS cluster that is all about the project let’s begin

Completion steps →

Step 1 : Set a Base Instance

Step 2 : create an IAM user with specific permission

Step 3: Kubernetes

Step 4: Install Essential tools

Step 5: Setup jenkins

Step 6: create jenkins pipeline

Step 1 : Set a Base Instance

1. Go to aws console and launch a instance with given configuration

a. AMI = ubuntu 24

b. Type = t2.large

c. Storage = 30GB

d. ports to be opened = 22 , 80 ,443 ,8080

2. Take a ssh of your ec2 >>click on connect>>ssh client

3. copy the command and open your cmd and locate the download folder where your key pair (.pem file) is stored

Step 2 : create an IAM user with specific permission

1. go to aws console and search for IAM and then click on user>>create user

• AmazonEC2FullAccess
• AmazonEKSClusterPolicy
• AmazonEKSWorkerNodePolicy
• AWSCloudFormationFullAccess
• IAMFullAccess

2. create an Inline policy for eks

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "eks:*",
            "Resource": "*"
        }
    ]
}

3. After this create a access keys and secret keys to work with awscli in base ec2

4. Download your secret and acess keys that we need to use later in the project

Step 3: Kubernetes

1. go to your ec2 and run

aws configure

2. Now you are log into aws via cli with limited permission provided in earliar steps

3. Create an Eks cluster using the following command

eksctl create cluster --name=EKS-1 --region=ap-south-1 --zones=ap-south-1a,ap-south-1b --without-nodegroup

4. Associate OIDC Provider →An OIDC provider (OpenID Connect provider) is like an identity verification service for applications. It ensures that people or systems accessing your application or cloud resources are who they claim to be.

Imagine you want to enter a secure building. At the entrance:

1. You show your ID card to prove your identity.
2. The security guard checks your ID against a trusted database.

In this case:

• Your ID card is the identity token.
• The OIDC provider is the trusted system (the database) that confirms your ID is valid.

eksctl utils associate-iam-oidc-provider --region ap-south-1 --cluster EKS-1 --approve

5. Create Node Group:

eksctl create nodegroup --cluster=EKS-1 --region=ap-south-1 --name=node2 --node-type=t3.medium --nodes=3 --nodes-min=2 --nodes-max=4 --node-volume-size=20 --ssh-access --ssh-public-key=DevOps --managed --asg-access --external-dns-access --full-ecr-access --appmesh-access --alb-ingress-access

6. update kubeconfig →This command aws eks update-kubeconfig is used to configure your local Kubernetes kubectl command-line tool to interact with an Amazon EKS (Elastic Kubernetes Service) cluster.

aws eks update-kubeconfig --region ap-south-1 --name EKS-1

a. Create a namesapce:

A namespace in Kubernetes is like a folder for organizing resources. It helps you group and manage Kubernetes objects (like pods, services, or secrets) that belong to the same team, project, or environment.

kubectl create namespace webapps
kubectl get namespaces

b. Creating Service Account:

A Service Account in Kubernetes is like a special user for applications running in your cluster. It allows pods (your running applications) to interact with the Kubernetes API or other systems securely.

vim svc-acc.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: webapps

kubectl apply -f svc-acc.yaml

c. Create role :

It is used to define a Role or RoleBinding for an application in your cluster. It specifies what actions the application (or a service account used by the application) is allowed to perform on specific resources within a namespace.

• A Role in Kubernetes grants permissions within a single namespace.
• It specifies what actions (like get, list, create, delete) are allowed on certain resources (like Pods, Services, ConfigMaps).

vim app-role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: webapps
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

d. Bind the role to service account:

• A RoleBinding binds a Role to a user, group, or service account, allowing them to use the permissions defined in the Role.

vim role-bind.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rolebinding
  namespace: webapps 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role 
subjects:
- namespace: webapps 
  kind: ServiceAccount
  name: jenkins 

e. Create a token by utilizing a service account in the desired namespace

A secret token in Kubernetes is used to allow a pod or application to securely prove its identity and access resources it has permission for. It’s like a password or keycard for your application.

vim secret.yaml

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: mysecretname
  annotations:
    kubernetes.io/service-account.name: jenkins

kubectl apply -f secret.yaml -n webapps

Now run →

kubectl describe secret mysecretname -n webapps

copy the token and paste it to your notepad for future refrence

Step 4: Install Essential tools

back to EC2 and run the following commands first

sudo su
apt update

1. Now make a shell script file to install the docker , jenkins , awscli etc….

vim install.sh

#!/bin/bash
sudo apt update -y
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
sudo apt update -y
sudo apt install temurin-17-jdk -y
/usr/bin/java --version

#install jenkins
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
sudo systemctl start jenkins
sudo systemctl status jenkins

#install docker
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker ubuntu
sudo usermod -aG docker jenkins
newgrp docker
sudo chmod 777 /var/run/docker.sock
sudo systemctl restart jenkins


# Install AWS CLI 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt-get install unzip -y
unzip awscliv2.zip
sudo ./aws/install


# Install kubectl
sudo apt update
sudo apt install curl -y
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

# Install eksctl
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

2. Above script will install jenkins , docker ,kubectl ,eksctl etc….. tools

Step 5: Setup jenkins

1. Go to your browser and type>> public_ip:8080 (to access jenkins console)

2. for this go to your ec2 and connect it

3. run the below commands

sudo su
cat /var/lib/jenkins/secrets/initialAdminPassword

output is your password and paste it to your jenkins

4. Install the suggested plugins

5. Setup your jenkins user

Welcome to jenkins dashboard

a. Install Plugins listed below

1. docker (all plugins)
2. kubernetes (all plugins)
3. Multibranch scan webhook trigger

b. Setup docker credentials

1. go to your jenkins →manage jenkins →credentials →global →add credentials
2. provide your username and password of your dockerhub
3. id==docker

c. setup git credentials →click on me to see how to generate token in github

1. Now copy that token from github and
2. go to your jenkins →manage jenkins →credentials →global →add credentials
3. provide your username and password of your github
4. id==github

d. set k8s token

1. go to credentials section chose secret text in kind section and paste your token in secret id==k8-token and then click on create

Step 6: create jenkins pipeline

1. click on New item and chose multibranch pipeline

2. Now chose git from add source option and paste the project repo url there

3. Now we need to setup webhook in github to trigger the pipeline

4. go to your project repo in github and click on setting>>webhooks>>Add a webhook

5. go back to your jenkins pipeline dashboard

Here is website having 10 microservice running in

Usual Error in Jenkins Pipeline →make sure to change api server endpoint in jenkins file

Now this project is completed here !!!

follow me on linkedin

follow me on github

When you are devleoping a software you have different parts of the system databases, servers, API and they all need to talk to each other . That is where software architecture diagrams come in handy . These diagrams give you a visual breakdown of how everything fits together , making it easier to plan , build , and explain your software .

For any job , having the right tools makes all the difference . Here is a eight great tools that can help you create clear and useful software architecture diagrams

1. Cloudairy Cloudchart

Cloudairy Cloudchart is a cloud focused diagram tool designed to help you easily map out cloud infrastructures . Whether you are working with AWS , Azure and Google Cloud , it has everything you need to create clear and detailed architecture diagrams specific to cloud environments

Features of Cloudairy Cloudchart :

• Cloud Service Icons : Provides icons and shapes for cloud services like AWS , Azure , and Google Cloud and many more .
• Drag and Drop Editor : Quickly build diagrams by dragging shapes into place .
• Templates : 500+ cloud/DevOps/flowchart templates that help you start your diagram quickly
• Real Time Collaboration : Multiple users can work on the same diagram at once .
• Export Features : Easily export your diagrams in various formats, including animated SVG, GIF, PNG and more allowing you to share or embed them anywhere while keeping the quality intact.
• Easy Sharing : Share diagrams with others through links or integrate with other tools for easy access .

Why you will like it :

• You will love how Cloudairy Cloudchart makes it easy to show cloud systems clearly , with special icons for cloud services
• Drag and drop editor for quick creation of cloud system diagrams
• It is super easy to use , so you can build diagrams quickly without needing to learn anything complicated
• It is great for teamwork because everyone can work on the diagram together at the same time

If your software architecture is built on cloud infrastructure , Cloudairy Cloudchart is the perfect tool for creating precise , easy to read diagrams

2. Lucidchart

Lucidchart is one of the easiest tools to use when it comes to making diagrams . It is all online so you don’t need to install anything , and it works great for teams . You can drag and drop shapes , create flowcharts , and even work with your team in real time

Features of Lucidchart :

• Drag and Drop Editor : Easily add and arrange shapes and lines by dragging and dropping them
• Real Time Collaboration : Work on diagrams with your team at the same time and see changes as they happen
• Pre-Made Templates : Use ready made templates to save time and get started quickly
• Integration : Connects with other tools like Google Workspace and Slack for smoother workflows

Why you will like it :

• Super easy to use , even if you have never made a diagram before
• You and your team can edit diagrams together
• Has ready made templates , so you don’t have to start from scratch

3. Draw.io (Diagrams.net)

Draw.io is a free , no frills tool that is perfect for simple diagramming needs . It is web-based , so you don’t have to download anything . Even though it is simple , it gets the job done for a lot of developers .

Features of Draw.io :

• Free to Use : No cost to use the tool just go to the website and start creating
• Cloud Storage Integration : Save your diagrams directly to Google Drive , OneDrive or Dropbox
• Offline Mode : You can work on your diagrams without an internet connection if you download the app .
• Export Options : Save your diagrams in various formats like PNG , PDF , or SVG for easy sharing

Why you will like it:

• It is free for use
• Works with Google Drive , OneDrive , and other cloud services .
• No account needed just open the website and start creating .

4. Gliffy

Gliffy is another web based tool that is easy to pick up and use . Like Lucidchart , it is drag and drop , but it also integrates with project management tools like Jira and Confluence , which is handy for teams working in Agile environments

Features of Gliffy :

• User Friendly Interface : Simple drag and drop design makes it easy to create diagrams
• Variety of Templates : Use built-in templates for different types of diagrams like flowcharts and UML diagrams
• Jira and Confluence Integration : Works well with Jira and Confluence, useful for teams in Agile environments .
• Real Time Collaboration: Collaborate with others in real time to update and refine diagrams together .

Why you will like it:

• Simple, clean interface easy to use .
• Great for UML diagrams , flowcharts , and wireframes .
• Works well with Jira and Confluence for teams using Agile .

5. Excalidraw

Excalidraw is an online tool that provides a simple , sketchy style for creating diagrams and drawings .

Features of Excalidraw :

• Hand Drawn Style : Create diagrams that look like they are drawn by hand , giving a more casual and approachable feel
• Real Time Collaboration : Work on diagrams with others in real time and see changes instantly .
• Easy to Use Interface : Simple and intuitive , making it quick to pick up and start drawing .
• Export Options : Save your diagrams in formats like PNG or SVG for easy sharing and embedding

Why you will like it :

• You will enjoy creating diagrams with a hand drawn look , giving your designs a more personal and relaxed feel .
• It is really easy to use , so you can quickly start drawing without spending time learning complex tools
• You can collaborate with others in real time , which is perfect for team projects where everyone can work together at once

6. C4 Model

The C4 Model is not a tool but a method . It helps you create diagrams that show your system at different levels big picture views down to the code . You can use it with tools like Structurizr to make the diagrams more structured .

Features of C4 Model :

• Structured Approach : Helps you create diagrams at different levels of detail ( system , container , component , and code )
• Clarity : Provides a clear way to break down and understand complex systems .
• Consistent : Keeps your diagrams organized and consistent across different levels .
• Tool Integration : Use with tools like Structurizr to automate and manage your diagrams .

Why you will like it :

• Focuses on breaking down your system into simple , understandable parts
• Great for visualizing complex systems step by step
• Works with Structurizr to automate the diagram making process

7. PlantUML

PlantUML is a different kind of tool . Instead of dragging shapes around , you write code to describe your diagram , and PlantUML turns that into an image . It is great for developers who are more comfortable writing than designing .

Features of Plant UML :

• Text Based Diagramming : Write text code to describe your diagram , and PlantUML generates the visual for you
• Support for Many Diagram Types : Create UML diagrams , sequence diagrams , and more using code
• IDE Integration : Works with popular development environments like IntelliJ and Visual Studio Code
• Automated Updates : Easily update diagrams by changing the code , which is useful for keeping documentation in sync

Why you will like it :

• You can write simple text to generate complex diagrams
• Works with many IDEs like IntelliJ and Visual Studio Code
• Perfect for automating diagram creation from code

8. Miro

Miro is a versatile online whiteboard tool that is excellent for brainstorming and visualizing ideas . It is great for creating collaborative diagrams and allows for a more free form approach to diagramming.

Features of Miro :

• Interactive Whiteboard : Provides a flexible canvas where you can freely arrange shapes , text , and sticky notes
• Collaboration : Work with your team in real time , adding comments and feedback directly on the board .
• Templates : It offers a variety of templates for different types of diagrams
• Integration : Connects with tools like Slack , Google Drive and Microsoft Teams for easy sharing and collaboration

Why you will like it :

• Ideal for collaborative and idea mapping
• Flexible and allows you to create diagrams , flowcharts , and even mind maps
• Supports real time collaboration with your team

All of these tools have something different to offer , so it really comes down to what works best for you

Cloudairy Cloudchart is perfect for anyone dealing with Cloud architecture, flowchart, mindmap, devops and more , thanks to its handy templates and cloud specific icons

Lucidchart is great if you areworking with a team and need real time collaboration .

If you are after something free and flexible , Draw.io is a solid choice

For simplicity and easy integration with other tools , Gliffy is a good option.

Excalidraw gives you that sketchy , casual style , which can be easy to use .

If you prefer a structured way of breaking down complex systems , the C4 Model will be really helpful

Miro is awesome for working together , while PlantUML is perfect if you prefer creating diagrams with code

Whatever your need , there is a tool here to help you design your system in the way that suits you best

What are you waiting for just go and design your diagrams !!

Topics covered in this blog

1. Introduction about the project
2. Three-Tier Web Application Architecture
3. Deployment Flowchart
4. DevOps Pipeline to Deploy Applications in Development, QA, and Production

Phase 1 : Testing Locally

Step 1 : setup a Base EC2

Step 2 : IAM Role for EC2

Step 3 : Clone the github Repository

Step 4 : Setup Docker and docker compose

Step 5 : Deploy Locally using docker-compose

Phase 2 : Setup CI/CD Pipeline

Step 1→ Setup Sonarqube

Step 2 →Setup jenkins

Step 3 → Install Plugins

Step 4 → Setup Credentials

Step 5 → Setup Tools and configure global setting

Step 6 → Pipeline script

Phase 3 : Setup Multiple Pipeline for Quality Assuarance and for Production

• Pipeline for Development
• Pipeline for Quality assurance
• Pipeline for Production

1. Introduction

In today’s world, many websites and applications we use every day, like online stores, social media platforms, or banking apps, are built using something called a three-tier architecture. This design pattern helps developers organize their code and make sure the application works smoothly, securely, and efficiently.

The three tiers (or layers) in this architecture are:

1. Presentation Layer (Frontend) — This is what users see and interact with, like the buttons, images, and forms on a website.
2. Logic Layer (Backend) — This handles the “behind the scenes” work, like processing user data, running calculations, or managing tasks.
3. Data Layer (Database) — This stores and retrieves all the important information, like user profiles, product details, or transaction history.

Each of these layers works together to make sure the website or application runs well. The three-tier architecture is very popular because it keeps things organized, makes applications easier to update, and helps handle large numbers of users without slowing down.

In this blog, we’ll take a detailed look at how these layers work, how the application is deployed step-by-step, and how developers use tools like DevOps pipelines to automate the process of building and updating the application in different environments such as Development (Dev), Quality Assurance (QA), and Production (Prod).

we are going to deploy this three-tier application with docker compose

Docker Compose is a tool that makes it easier to manage multi-container Docker applications.

Think of Docker as a way to package your app into a “container,” which includes everything it needs to run, like code, libraries, and system tools. But many apps need more than one container to work. For example, a web app might need one container for the web server and another for the database.

Docker Compose lets you define and run these multiple containers with a simple text file called docker-compose.yml. In this file, you list all the containers your app needs and how they should interact. Then, with one command, Docker Compose will start everything up, making sure all the containers work together as intended.

In short, Docker Compose is like an organizer for your Docker containers, making it easy to manage and run complex apps with just a few commands

2. Three-Tier Web Application Architecture

Web applications today, whether it’s an online shopping platform or a social media site, are often built using a three-tier architecture. This design is popular because it splits the application into three distinct parts, making it easier to manage and maintain. Each part has a specific job to do, and they all work together to give users a smooth experience.

Let’s break down these three parts:

1) Presentation Layer (The Frontend)

Think of this as the face of your application. It’s what users see and interact with when they visit your website. This layer includes all the visual stuff — buttons, images, forms, menus, and text.

For example, if you’re on an shopping website, the homepage, the search bar, the product listings, and the “Add to Cart” button are all part of this presentation layer. When you click on a product to view more details, you’re interacting with this layer.

Technologies used Here :

• HTML (for the structure of web pages)
• CSS (to make things look nice and styled)
• JavaScript (to make the website interactive, like responding when you click buttons)

2) Logic Layer (The Backend)

This is where the magic happens. The logic layer handles all the heavy lifting. When you, the user, click a button or submit a form on the frontend, this middle layer processes the request, does the calculations, and makes decisions based on what you want to do.

Let’s say you’re shopping online and you add something to your cart. The backend checks whether the item is available, updates your cart, and calculates the total. It doesn’t just store or display the information — it works with it.

The logic layer is like the “brain” of the application. It processes what the user does on the frontend and interacts with the database to get or save data. Once it processes everything, it sends the final result back to the frontend for the user to see.

Technologies used here:

• Programming languages like Python, Java, Node.js, or Ruby.
• Frameworks such as Django (for Python) or Spring (for Java).
• APIs (Application Programming Interfaces), which let the frontend and backend talk to each other.

3) Data Layer (The Database)

The data layer is where all the important information is stored. This could be anything from user profiles, order histories, or product details. The database holds onto this data and sends it back to the logic layer whenever it’s needed.

For example, when you log in to your account on a website, the backend checks the data layer for your username and password. If it finds a match, it lets you log in. If not, it shows an error message.

The data layer stores, updates, and retrieves data as the backend requests it. When you submit a form or need to see certain information (like product availability), the backend checks with the database to get the info and then processes it.

Technologies used here :

• SQL databases like MySQL or PostgreSQL.
• NoSQL databases like MongoDB (which is great for unstructured data).
• Cloud databases like AWS RDS or Google Cloud SQL for managing large-scale applications.

How Do These Layers Interact With Each Other

Let’s walk through a real-world example. Imagine you’re on a website, looking to buy a new pair of shoes:

1. You search for “sneakers” in the search bar (this is happening on the frontend).
2. The search request is sent to the backend, where the logic layer processes it. The backend knows it needs to look in the database for sneakers.
3. The database (data layer) is queried, and it sends back a list of all the sneakers available in the store.
4. The backend receives this data, processes it, and sends the result back to the frontend, which displays all the sneakers on your screen.

Each layer has its own job, but they rely on each other to keep the application running smoothly. This separation of responsibilities makes the system more organized and scalable. If you need to change the design, you only modify the frontend. If you need to update how the system handles data, you focus on the backend or database.

If you ever wonder how websites like Amazon or Netflix handle millions of users at the same time, it’s thanks to architectures like this one!

3. Deployment Flowchart for a Three-Tier Web Application

1. Coding Part

• Frontend code : Developers design and build the part of the app you see and interact with.
• Backend code: Developers write the code that handles data processing and logic.

2. Code Management

• Written code is managed by source code management tools like git Think of it as a shared workspace where everyone’s changes are recorded.
• When after your code is ready you have to push it in a central repository like github

3. CI/CD Pipeine

• You can use any CI/CD tool like jenkins , circle ci etc. for me it is jenkins
• Jenkins is the CI/CD tool which automate the whole Software Development Lifecycle Process
• It Fetches code from github , tests it and deployed on docker container

4. Containerization

• The app is packed into containers (using Docker), making it easy to run consistently across different environments.

5. Set Up Different Environments

• Development (Dev): Where the initial testing happens.
• Quality Assurance (QA): Where more detailed testing and bug fixes are done.
• Production (Prod): The live version of the app where real users interact with it.

6. Deploy to Dev Environment

• The new code is first deployed to the Dev environment. Here, developers check if everything works as expected.

7. Testing in QA Environment

• Next, the code moves to QA. Testers perform more rigorous testing to ensure there are no bugs or performance issues.

8 . Get Approval for Production

• Once QA testing is complete and the code is stable, it’s reviewed for final approval before going live.

9. Deploy to Production

• The code is deployed to the Production environment. This is where users can see and use the updated app.

10. Repeat and Improve

• The process starts again with new code updates, improving the app based on user feedback and performance data.

4. DevOps Pipeline to Deploy Applications in Development, QA, and Production :

Phase 1 : Testing Locally

Step 1 : setup a Base EC2

Step 2 : IAM Role for EC2

Step 3 : Clone the github Repository

Step 4 : Setup Docker and docker compose

Step 5 : Deploy Locally using docker-compose

Step 1 → setup a Base EC2

1. go to aws console and launch a instance

2. Choose ubuntu from the list and create a new key pair →my key and open ports http and https

and then click on launch instance

3. click on EC2 →security →click on security groups→edit the inbound rules to open the following port no.

Step 2 : IAM Role for EC2

Why we need IAM role for EC2 → It is used by your ec2 instance to create EKS cluster and manage s3 bucket by applying this IAM role it gives the authenticity to your ec2 to do changes in aws account

1. creating IAM role

1. on the search bar type IAM

2. click on Roles on the left side

3. click on create role and choose EC2 from the dropdown

4. click on next

5. choose administrator access on permission sections

6. click on next and give a name to your role

7. click on create role option and your IAM role is created

2. Attach IAM role with your EC2

1. go to EC2 section
2. click on actions → security → modify IAM role option

3. choose the role from dropdown and click on update IAM role

Now go back to your EC2 and connect with your EC2 instance

Run the following command

sudo su
apt update -y

Step 3 →Clone the github Repository

On your EC2 Instance clone the github repo for the source code

git clone https://github.com/Aakibgithuber/wanderlust-3-tier-project.git
ls

Step 4 → Setup docker and docker compose

There is a file called script.sh run that file using below command to setup docker and docker compose

bash script.sh # it will install all the necessary things

Step 5 →Deploy with docker compose

1. you just need to run a single command to up all the containers the frontend , the backend , the database containers and connect all of them

docker-compose up -d

Copy public ip and paste to your web broswer

1. public_ip:5173

As you see featured section is not accessible to do that we have to run a following commands

docker ps #give containers name 

Copy mongo container name and run

docker exec -it <container_name> mongoimport --db wanderlust --collection posts --file ./data/sample_posts.json --jsonArray

In output it gives 10 documents imported successfully

Now again if you hit the url it will give the whole website

Note: If you getting error in loading backend and in featured post then you need to update the current ip address of your machine in /backend/ .env.sample and in /frontend/.env.sample files

Phase 2 : The Pipeline

Step 1→ Setup Sonarqube

Step 2 →Setup jenkins

Step 3 → Install Plugins

Step 4 → Setup Credentials

Step 5 → Setup Tools and configure global setting

Step 6 → Pipeline script

1.Sonarqube →

copy the public ip of your machine

1. go to your browser and type →<publicip>:9000

sonarqube window open

2. iniatially username and password is admin

3. update your password

4. welcome window of Sonarqube

2. Jenkins →

1. on browser type →<public_ip>:8080 for password

2. for this go to your ec2 and connect it

3. run the below commands

sudo su
cat /var/lib/jenkins/secrets/initialAdminPassword

output is your password and paste it to your jenkins

4. Install the suggested plugins

5. Setup your jenkins user

Welcome to jenkins dashboard

Step 3 → Install Plugins listed below

1. Eclipse Temurin Installer (Install without restart)
2. SonarQube Scanner (Install without restart)
3. docker compose build step
4. NodeJs Plugin (Install Without restart)
5. owasp →The OWASP Plugin in Jenkins is like a “security assistant” that helps you find and fix security issues in your software. It uses the knowledge and guidelines from the Open Web Application Security Project (OWASP) to scan your web applications and provide suggestions on how to make them more secure. It’s a tool to ensure that your web applications are protected against common security threats and vulnerabilities.
6. Prometheus metrics →to moniter jenkins on grafana dashboard
7. Download all the docker realated plugins

Step 4 →add credentials of Sonarqube and Docker

1st we genrate a token for sonarqube to use in jenkins credentials as secret text

a. setup sonarqube credentials

1. go to http://publicip:9000
2. now enter your username and password
3. click on security →users →token →generate token
4. token_name==jenkins

4. copy the token and go to your jenkins →manage jenkins →credentials →global →add credentials

5. select secret text from dropdown

6. secret text ==your token , id =jenkins →click on create

b. setup projects in sonarqube for jenkins

1. go to your sonarqube server
2. click on projects
3. in the name field type gpt
4. click on set up

click on setup

click on above option

click on generate

click on continue ….

Sonarqube project for jenkins is setup now

c. setup sonarqube

d. Setup docker credentials

1. go to your jenkins →manage jenkins →credentials →global →add credentials
2. provide your username and password of your dockerhub
3. id==docker

credentials for both are setup

Step 5 →Now we are going to setup tools for jenkins

go to manage jenkins → tools

a. add jdk

1. click on add jdk and select installer adoptium.net
2. choose jdk 17.0.8.1+1version and in name section enter jdk 17

b. add node js

1. click on add nodejs
2. enter node16 in name section
3. choose version nodejs 16.2.0

c. add docker →

1. click on add docker
2. name==docker
3. add installer ==download from docker.com

d. add sonarqube →

1. add sonar scanner
2. name ==sonar-scanner

e. add owasp dependency check →

Adding the Dependency-Check plugin in the “Tools” section of Jenkins allows you to perform automated security checks on the dependencies used by your application

1. add dependency check
2. name == DP-Check
3. from add installer select install from github.com

Configure global setting for sonarube and setup webhooks

1. go to manage jenkins →Configure global setting →add sonarqube servers
2. name ==sonar-server
3. server_url==http://public_ip:9000
4. server authentication token == jenkins →it is created in sonarqube security configurations

5. let’s run the Pipeline →

1. go to new item →select pipeline →in the name section type gpt-pipeline
2. scroll down to the pipeline script and copy paste the following code

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/Aakibgithuber/wanderlust-3-tier-project.git'
            }
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=docker-compose \
                    -Dsonar.projectKey=docker-compose '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
                }
            }
        }
        stage('OWASP FS SCAN') {
            steps {
                dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.json"
            }
        }
          // Docker Compose Build Stage with Timeout
        stage('Docker-compose Build') {
            steps {
                script {
                    timeout(time: 1, unit: 'MINUTES') { // Timeout set to 1 minute
                        sh 'docker-compose up -d'
                    }
                }
            }
        }

        stage('Docker-compose Push') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker', toolName: 'docker') {
                        // Tag and push backend and frontend images
                        sh "docker tag devpipeline-backend aakibkhan1212/devpipeline-backend:latest"
                        sh "docker tag devpipeline-frontend aakibkhan1212/devpipeline-frontend:latest"
                        
                        sh "docker push aakibkhan1212/devpipeline-backend:latest"
                        sh "docker push aakibkhan1212/devpipeline-frontend:latest"
                    }
                }
            }
        }
        stage("TRIVY"){
            steps{
                sh "trivy image aakibkhan1212/devpipeline-backend > trivy.json"
                sh "trivy image aakibkhan1212/devpipeline-frontend > trivy.json"
            }
        }
    }
}

your application is successfully deployed checkout the dockerhub and to acess your application go to your browser and type

“public_ip:5173”

Phase 3 : Setup Multiple Pipeline for Quality Assuarance and for Production

Why we need to Deploy it Multiple Times

We deploy the application in three different stages — Development, Quality Assurance (QA), and Production — to ensure the application works correctly at every step and to minimize the chances of errors in the final product. Here’s why each stage is important:

1. Development (Dev)

In the Development stage, the application is tested by developers. This is where new features are added, bugs are fixed, and code changes are made. It’s like a “testing ground” where developers can try out new things without worrying about breaking anything important.

• Purpose: To develop and test new features in a safe environment.
• Why it’s needed: It helps catch issues early before the code moves to the next stage.

2. Quality Assurance (QA)

In the QA stage, testers check if everything works as expected. They test the application in a controlled environment, trying to find bugs, issues, or performance problems. This stage simulates a real-world scenario but is still separate from the actual users.

• Purpose: To test the stability, performance, and functionality of the application.
• Why it’s needed: It helps ensure the application is ready for users by catching any issues that were missed in the development stage.

3. Production (Prod)

The Production stage is where the application goes live for actual users. This is the final environment where the real customers or users interact with the application. It’s crucial that only well-tested, stable code reaches this stage, so users don’t experience issues.

• Purpose: To serve the final, stable version of the application to users.
• Why it’s needed: This ensures that only high-quality, reliable code is deployed, providing a smooth experience for users.

Pipeline for Development

In the Development pipeline, the goal is to test new features or bug fixes in a safe environment where developers can make changes without impacting the live application. It’s all about fast feedback to help developers iterate quickly.

Steps:

• Checkout Code: Pull the latest code from the repository so developers have the most up-to-date version.
• Build Docker Images: Create Docker images for each service (frontend, backend, and database) to package the application in a standardized way.
• Run Unit Tests: These tests focus on checking small, individual parts of the application (like functions or modules) to make sure they work as expected.
• Push Docker Images (Optional): You can choose to push the images to a Docker registry, but it’s optional at this stage since the images are usually local.
• Deploy to Development: The application is deployed in the Development environment where developers can interact with it to see if their changes worked.
• Basic Integration Tests: Optionally, run tests to ensure different parts of the application (like frontend and backend) work together correctly.
• Trigger the QA Pipeline: Once the development work is complete and basic tests pass, the pipeline automatically triggers the QA process for further testing.

pipeline {
    agent any

    stages {
        stage('Checkout Code') {
            steps {
                git 'https://github.com/Aakibgithuber/docker-compose-project.git'
            }
        }

        stage('Build Docker Images') {
            steps {
                sh 'docker-compose build'
            }
        }

        stage('Run Unit Tests') {
            steps {
                sh 'docker-compose run backend pytest'
            }
        }

        stage('Deploy to Development') {
            steps {
                sh 'docker-compose up -d'
            }
        }
    }

    post {
        success {
            echo "Development deployment successful, triggering QA pipeline in 10 seconds..."
            sleep 10
            build job: 'QA_Pipeline'
        }
        failure {
            echo "Development deployment failed."
        }
    }
}

2. Pipeline for Quality Assurance

In the QA pipeline, the focus shifts from development to testing the entire application thoroughly. QA teams simulate real-world conditions to ensure the application works as expected for users. The QA environment is closer to what the production environment will be, so this is where more advanced tests are run.

Steps:

• Pull Docker Images: The pipeline pulls the Docker images that were built and tested in the Development pipeline, ensuring that the exact same version of the application is being tested in QA.
• Deploy to QA: The application is deployed in the QA environment so that testers can interact with it and run their tests.
• Run Functional Tests: These tests check that the application functions correctly from a user’s perspective. They make sure all features work as expected.
• Run End-to-End Tests: These tests simulate real user scenarios, verifying that all parts of the application (frontend, backend, and database) work seamlessly together.
• Run Performance Tests: Performance or load testing checks how well the application handles heavy traffic and stress. This ensures the application can scale and perform well under real-world conditions.
• Manual QA Approval: Once the QA team is satisfied with the tests, there’s a manual approval step to ensure human oversight before the application moves to production.
• Trigger the Production Pipeline: If all tests pass and QA approves, the pipeline automatically triggers the Production deployment process

pipeline {
    agent any

    stages {
        stage('Pull Docker Images') {
            steps {
                sh 'docker-compose pull'
            }
        }

        stage('Deploy to QA') {
            steps {
                sh 'docker-compose up -d'
            }
        }

        stage('Run Functional Tests') {
            steps {
                sh 'docker-compose run frontend npm run test'
            }
        }

        stage('Run End-to-End Tests') {
            steps {
                sh 'docker-compose run backend behave tests/'
            }
        }

        stage('Run Performance Tests') {
            steps {
                sh 'docker-compose run backend stress --cpu 8 --timeout 10'
            }
        }
    }

    post {
        success {
            echo "QA testing successful, triggering Production pipeline in 10 seconds..."
            sleep 10
            build job: 'Production_Pipeline'
        }
        failure {
            echo "QA testing failed."
        }
    }
}

3. Pipeline for Production

The Production pipeline is where the final version of the application is deployed for real users. This environment must be stable and reliable, so the pipeline focuses on smooth deployment and post-deployment checks to ensure everything runs as expected.

Steps:

• Pull Docker Images: The Production pipeline pulls the same Docker images that were built and tested in Development and QA, ensuring consistency.
• Approval Step: A manual approval step is typically included here to ensure that nothing is deployed to production without explicit authorization. This step adds an extra layer of protection.
• Deploy to Production: The final version of the application is deployed to the production environment, where real users will interact with it.
• Post-Deployment Smoke Tests: After the deployment, simple smoke tests are run to verify that the application is up and running correctly. This ensures that the deployment didn’t break anything critical.
• Monitoring & Alerts: In production, it’s important to have monitoring systems and alerts set up to keep track of the application’s performance and notify the team if anything goes wrong.
• Notify Team: Once the production deployment is successful, notifications (via Slack, email, etc.) are sent to the team, confirming that the application is live and stable

pipeline {
    agent any

    stages {
        stage('Pull Docker Images') {
            steps {
                sh 'docker-compose pull'
            }
        }

        stage('Manual Approval') {
            steps {
                input message: 'Approve deployment to production?'
            }
        }

        stage('Deploy to Production') {
            steps {
                sh 'docker-compose up -d'
            }
        }

        stage('Run Smoke Tests') {
            steps {
                sh 'curl http://your-production-url'
            }
        }

        stage('Post-Deployment Notifications') {
            steps {
                echo "Deployment to production was successful!"
                // Here you can add additional steps for notifications like Slack or email
            }
        }
    }

    post {
        success {
            echo "Production deployment successful."
        }
        failure {
            echo "Production deployment failed."
        }
    }
}

Here we completed Our blog if you like it do clap , follow and share the blog

Follow me on github

Follow me on Linkedin

The above animation Diagram is designed with Cloudairy Cloudchart that is powerful architectural design tool

It provides you the innovative features to enhance your design process such as

1. ✅𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐌𝐚𝐠𝐢𝐜: Bring your ideas to life with our AI-powered flowchart generator.
2. ✅𝐒𝐡𝐚𝐩𝐞 𝐘𝐨𝐮𝐫 𝐖𝐨𝐫𝐥𝐝: Choose from an extensive library of BPMN, dataflow, enterprise, flowchart, and geometric shapes.
3. ✅𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐞 𝐢𝐧 𝟑𝐃: Bring your cloud infrastructure to life with our 3D icons for AWS, Azure, GCP, and Kubernetes.
4. ✅𝐄𝐧𝐫𝐢𝐜𝐡 𝐘𝐨𝐮𝐫 𝐃𝐞𝐬𝐢𝐠𝐧𝐬: Search and import images, icons, and GIFs directly into your diagrams.
5. ✅200+Templates: Accelerate your design with 200+ pre-built cloud architecture templates.

Cloudairy Cloudchart link →https://cloudairy.com

Today we are going to deploy youtube clone . In this project we also integrate slack notification for jenkins . This is one of the resume ready project with each and every steps Do follow the project guys

Completion steps →

Step 1: Set a base EC2 on AWS

Step 2 : IAM Role for EC2

Step 3 : Configure EC2

Step 4 : Setup jenkins pipeline

Step 5 : ci-cd pipeline to build and push the image to docker hub

Step 6 : Integrate Slack notification for pipeline

Step 7 : Kuberntes Cluster creation using terraform

Step 8→ Creation of deployment for EKS

Step 9→ Destroy all the Insrastructure

Step 1 : Set a base EC2 on AWS

1. go to aws console and launch a instance

2. Choose ubuntu from the list and create a new key pair →my key and open ports http and https

and then click on launch instance

Step 2 : IAM Role for EC2

Why we need IAM role for EC2 → It is used by your ec2 instance to create EKS cluster and manage s3 bucket by applying this IAM role it gives the authenticity to your ec2 to do changes in aws account

1. creating IAM role

1. on the search bar type IAM

2. click on Roles on the left side

3. click on create role and choose EC2 from the dropdown

4. click on next

5. choose administrator access on permission sections

6. click on next and give a name to your role

7. click on create role option and your IAM role is created

2. Attach IAM role with your EC2

1. go to EC2 section
2. click on actions → security → modify IAM role option

3. choose the role from dropdown and click on update IAM role

why we need IAM Role →

Imagine you have a robot (EC2 instance) that does tasks for you in a big factory (AWS environment). Now, this robot needs to access different rooms (AWS services like S3, DynamoDB, etc.) to perform its tasks.

Here’s where IAM (Identity and Access Management) comes in:

1. Robot Needs a Key to Enter Rooms: The IAM Role is like giving your robot a special key. This key allows the robot to enter specific rooms (access certain AWS services). Without this key, the robot can’t get in.
2. Different Keys for Different Robots: Each robot (EC2 instance) can have its own key (IAM Role) with specific permissions. So, one robot may have a key to enter the storage room (access S3), while another robot has a key to enter the database room (access DynamoDB).
3. No Need for Hardcoding Passwords: Using IAM Roles means you don’t have to hardcode passwords (access credentials) into the robot. It’s like not writing down passwords on the robot itself. The robot just uses its key when needed.
4. Easily Change Permissions: If you want to change what a robot can do, you just change the permissions on its key (IAM Role). No need to reprogram the robot or give it a new password; just update the permissions on its key.
5. Secure and Controlled Access: IAM Roles help keep things secure. You can control exactly what each robot is allowed to do. This way, if one robot is compromised or needs a different role, you can easily adjust its permissions without affecting others.

Now go back to your EC2 and connect with your EC2 instance

Step 3 : Configure EC2

you have to install git , docker , jenkins , kubectl etc for docker container and kubernetes deployment

1. run the following commands

apt update 
vim run.sh

2. Enter the following commands to your .sh file

#!/bin/bash
sudo apt update -y
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
sudo apt update -y
sudo apt install temurin-17-jdk -y
/usr/bin/java --version

#install jenkins
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
sudo systemctl start jenkins
sudo systemctl status jenkins

#install docker
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker ubuntu
sudo usermod -aG docker jenkins
newgrp docker
sudo chmod 777 /var/run/docker.sock
sudo systemctl restart jenkins

# install trivy
sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy -y


# Install Terraform
sudo apt install wget -y
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

# Install kubectl
sudo apt update
sudo apt install curl -y
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

3. run the .sh file by command bash run.sh

Step 4 : Setup jenkins pipeline

From the above step everything is installed all we need to setup jenkins ci-cd pipeline

1. copy the public ip of your ec2 and paste it on browser type →<public_ip>:8080

2. for this go to your ec2 and connect it

3. run the below commands

sudo su
cat /var/lib/jenkins/secrets/initialAdminPassword

output is your password and paste it to your jenkins

4. Install the suggested plugins

5. Setup your jenkins user

Welcome to jenkins dashboard

Step 5 : ci-cd pipeline to build and push the image to docker hub

1. Install Plugins listed below

1. Terrform
2. slack notification
3. Global Slack Notifier
4. Eclipse Temurin Installer (Install without restart)
5. NodeJs Plugin (Install Without restart)
6. Parameterized Trigger (to trigger another pipeline if one completed)
7. Download all the docker realated plugins

Note → if jenkins is not running on the given website then go to your ec2 security groups and open the 8080 port no.

A. Setup docker credentials

1. go to your jenkins →manage jenkins →credentials →global →add credentials
2. provide your username and password of your dockerhub
3. id==docker

B. Setup tools for jenkins

go to manage jenkins → tools

a. add jdk

1. click on add jdk and select installer adoptium.net
2. choose jdk 17.0.8.1+1version and in name section enter jdk 17

b. add node js

1. click on add nodejs
2. enter node16 in name section
3. choose version nodejs 16.2.0

c. add docker →

1. click on add docker
2. name==docker
3. add installer ==download from docker.com

d. terraform

1. go to manage jenkins →tools →search for terraform
2. add terraform
3. provide the name in the name field and untick the install automatically option and give the path /usr/bin/
4. As terraform is installed in this section it takes it from there

5. click on apply and save

C. The Pipeline Script

1. go to new item →select pipeline →in the name section type Youtube Pipeline 1

2. scroll down to the pipeline script and copy paste the following code

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/Aakibgithuber/deployment-of-youtube.git'
            }
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage("Docker Build & Push"){
            steps{
                script{
                   withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){   
                       sh "docker build -t youtube-clone ."
                       sh "docker tag youtube-clone aakibkhan1212/youtube-clone:latest "
                       sh "docker push aakibkhan1212/youtube-clone:latest "
                       sh "docker "
                    }
                }
            }
        }
        stage('Deploy to container'){
            steps{
                sh 'docker run -d --name youtube-clone -p 3000:3000 aakibkhan1212/youtube-clone:latest'
            }
        }
    }
}

If you want to add owasp and sonarqube then you could make changes by following the previous project blog

your application is successfully deployed checkout the dockerhub and to acess your application go to your browser and type

http://your_public_ip:3000

Here is your application

Step 6 : Integrate Slack notification for pipeline

1. go to slack.com and make a account

2. choose any of the option

3. Create a Workspace

4. click on next

5. Create a new channel

6. Now you have to browse Slack app directory and search for jenkins

7. click on add to slack and choose a channel

8. It will generate token for the jenkins

9. Now you have to go to Jenkins →manage jenkins →global and scroll down to Slack Notifications

10. On credentials add the token in the secret text dropdown

11. Click on Test Connection and you will see Success blink on

12. Now again go to your youtube Pipeline and click on configure then add the following code to the pipeline script

pipeline {
    agent any
    tools {
        jdk 'jdk17'
        nodejs 'node16'
    }
    stages {
        stage('Clean Workspace') {
            steps {
                cleanWs()
            }
        }
        stage('Checkout from Git') {
            steps {
                git branch: 'main', url: 'https://github.com/Aakibgithuber/deployment-of-youtube.git'
            }
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('Docker Build & Push') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker', toolName: 'docker') {
                        sh "docker build -t youtube-clone ."
                        sh "docker tag youtube-clone aakibkhan1212/youtube-clone:latest"
                        sh "docker push aakibkhan1212/youtube-clone:latest"
                    }
                }
            }
        }
        stage('Deploy to Container') {
            steps {
                sh 'docker run -d --name youtube-clone -p 3000:3000 aakibkhan1212/youtube-clone:latest'
            }
        }
    }
    post {
        success {
            slackSend(channel: '#all-the-cloud-hub', color: 'good', message: "Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL}) was successful.")
        }
        failure {
            slackSend(channel: '#all-the-cloud-hub', color: 'danger', message: "Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL}) failed.")
        }
        always {
            echo 'Build finished, check Slack for notifications.'
        }
    }
}

13. Now again build the Pipeline and check for the slack notifications

14. It shows success on slack for the pipeline that we created on jenkins

Step 7 : Kuberntes Cluster creation using terraform

1. clone the github repo by →

a. mkdir super_mario

b. cd super_mario

c. git clone https://github.com/Aakibgithuber/deployment-of-youtube.git

d. cd deployment-of-youtube/

e. cd EKS-TF

f. edit the backend.tf file by → vim backend.tf

Note →make sure to provide your bucket and region name in this file otherwise it doesn’t work and IAM role is also associated with your ec2 which helps ec2 to use other services such S3 bucket

NOW RUN →

1. terraform init

When we run terraform init, it sets up your working area, downloads necessary plugins, and makes sure everything is in place so that you can start using Terraform to create, update, or manage your infrastructure. It’s like getting all the tools and materials ready before you start building something amazing with your computer.

2. terraform validate

Terraform validate reviews our code to catch any syntax errors or mistake and gives an output success if everything is no error in the file

3. terraform plan

Terraform plan is used to see what changes will be made to your infrastructure. By using this command we could review and confirm that everything looks good before giving the final approval to build or modify our application infrasructure It is like the blueprint of the construction project before actually creating or changing anything with Terraform

4. terraform apply

terraform apply --auto-approve

Running terraform apply --auto-approve is like telling computer, "Go ahead and build everything exactly as planned without asking me for confirmation each time." It's a way to automate the deployment of your infrastructure without needing our constant input. When we execute this command, Terraform reads our code, figures out what needs to be created or changed, and then goes ahead with the construction, skipping the usual step where it checks with you for approval.

It takes your 5 to 10 min for completion

5. Below command is used to update the configuration of EKS

aws eks update-kubeconfig --name EKS_CLOUD --region us-east-1

The command aws eks update-kubeconfig --name EKS_CLOUD --region us-east-1 is like telling our computer, "Hey, I'm using Amazon EKS (Elastic Kubernetes Service) in the 'us-east-1' region, and I want to connect to it. you could use your desired location

Step 8→ Creation of deployment for EKS

1. change the directory where deployment and service files are stored use the command → cd ..
2. create the deployment

kubectl apply -f deployment.yaml

deployment.yaml file is like a set of instructions that tells a computer system, "Hey, here's how you should run and manage a particular application " . It provides the necessary information for a computer system to deploy and manage a specific software application. It includes details like what the application is, how many copies of it should run, and other settings that help the system understand how to keep the application up and running smoothly.

3. Service file is included in the deployment file

service.yaml file is like a set of rules that helps computers find and talk to each other within a software application. It's like a directory that says, "Hey, this is how you can reach different parts of our application." It specifies how different parts of your application communicate and how other services or users can connect to them.

4. run → kubectl get all

copy the load balancer ingress and paste it on browser and your application is running there

don’t forget to destory everything that’s saves of aws bill and you aws account too

Load Balancer Ingress →

It is a mechanism that helps distribute incoming internet traffic among multiple servers or services, ensuring efficient and reliable delivery of requests.

It’s like having a receptionist at a busy office building entrance who guides visitors to different floors or departments, preventing overcrowding at any one location. In the digital world, a Load Balancer Ingress helps maintain a smooth user experience, improves application performance, and ensures that no single server becomes overwhelmed with too much traffic.

Step 9→ Destroy all the Insrastructure

1 . Below commands delete or destroy your cluster

cd EKS-TF
terraform destroy --auto-approve

after 3 -5 mins all things are destroyed

3. Now go to your EC2 and terminate your Instance

Here we completed another project don’t forget to create a good animation diagram for this project using cloudairy cloudchart

Cloudairy cloudchart website link →https://cloudairy.com

Over the years, I’ve seen a lot of different architecture diagrams. Some were very detailed, while others were more basic. However, many of them had a problem — they didn’t clearly show what was most important about the system.

Common Problems with Software Diagrams

When I first started working in software development, I was exposed to a wide range of architecture diagrams. Some were overly complex, while others were too simplistic. Many diagrams suffered from common problems that made them difficult to understand. For example:

1. Mixing Levels of Detail:One big issue is when a diagram tries to show too much at once. For example, a diagram might start by showing the overall system but then suddenly dive into the small details. This can be confusing because it mixes different levels of detail. It’s important to keep things consistent so the diagram is easy to follow.
2. Inconsistent Use of Colors and Sizes:Another problem is using too many colors or sizes to represent different things. While colors can help highlight important parts, they can also make the diagram harder to read if not used carefully. Similarly, using different sizes for boxes or elements to show importance can be misleading, especially as the system changes over time.
3. Overly Complex Diagrams:Some diagrams try to include every possible detail about the system, making them overwhelming to look at. When a diagram is too complex, it becomes difficult to extract the most important information. Instead of clarifying the architecture, it can actually make it harder to understand.

I’ve made these mistakes too. I’ve used various tools like Whiteboard, MS Paint, Photoshop, draw.io , canva to create diagrams. No matter which tool you use, it’s easy to get caught up in making things look perfect instead of focusing on what really matters — clearly communicating the system’s structure.

Finding a Better Way

So, after making these mistakes , I found a better way to create clear and useful software architecture diagrams. This better way is called the C4 model.

while using the C4 model approach you also need a right tool for your architecture diagrams one of the best tool for creating these diagrams is cloudairy cloudchart

Using Cloudairy Cloudchart for Diagrams

While the C4 model helps organize your diagrams, you also need the right tools to create and update them. One of the best tools for this is Cloudairy Cloudchart.

How to Use Cloudairy Cloudchart for C4 Diagrams

Here’s how you can use Cloudairy Cloudchart to create diagrams using the C4 model:

1. Start with a Template: Cloudairy Cloudchart offers templates for different types of diagrams. Choose a template that matches the type of diagram you want to create.

2. Add Elements: Use Cloudairy Cloudchart drag-and-drop tools to add the elements you need, like users, systems, and relationships for a context diagram, or containers for a container diagram.

3. Customize Your Diagram: You can customize your diagram’s colors, labels, and shapes to make it clear and visually appealing.

4. Collaborate in Real-Time: Invite your team to work on the diagram with you. You can leave comments, suggest changes, and make sure everyone is on the same page.

5. Share Your Diagram: Once your diagram is ready, you can easily share it by embedding it in documents, presentations, or project management tools.

Why Use Cloudairy Cloudchart ?

Cloudairy Cloudchart is an online tool that makes it easy to create software architecture diagrams. Some of the features of Cloudairy Cloudchart :

1. Easy to Use: It has a simple drag-and-drop interface, so you can create diagrams quickly without spending a lot of time learning how to use the tool.
2. Collaboration Features: It lets multiple people work on a diagram at the same time. This is great for teams that need to brainstorm, design, and update their architecture together, especially if team members are in different locations.
3. Templates and Shapes: It has lots of templates and shapes to help you create different types of diagrams, including those needed for the C4 model.
4. Access from Anywhere: Since it is cloud-based, you can access your diagrams from anywhere, on any device. This is especially useful for remote teams.
5. Version Control: It tracks changes to your diagrams, so you can see how they’ve evolved over time and go back to earlier versions if needed.
6. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐌𝐚𝐠𝐢𝐜: Bring your ideas to life with our AI-powered flowchart generator
7. 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐞 𝐢𝐧 𝟑𝐃: Bring your cloud infrastructure to life with our 3D icons for AWS, Azure, GCP, and Kubernetes
8. 𝐕𝐞𝐫𝐬𝐚𝐭𝐢𝐥𝐞 𝐒𝐭𝐢𝐜𝐤𝐲 𝐍𝐨𝐭𝐞𝐬: Brainstorm, plan, and collaborate effortlessly with various sticky notes
9. 𝐄𝐧𝐫𝐢𝐜𝐡 𝐘𝐨𝐮𝐫 𝐃𝐞𝐬𝐢𝐠𝐧𝐬: Search and import images, icons, and GIFs directly into your diagrams
10. 𝐂𝐮𝐬𝐭𝐨𝐦𝐢𝐳𝐞 𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠: Make Cloudchart truly yours with custom colours, icons, and backgrounds
11. 200+Templates: Accelerate your design with 200+ pre-built cloud architecture templates

What is the C4 Model?

The C4 model is a method for creating software diagrams that break down a system into different levels of detail. This helps people understand the system from different perspectives without getting overwhelmed.

Here’s a simple explanation of the C4 model:

1. Context Diagrams (Level 1): These diagrams show the system as a whole and how it interacts with users and other systems. This is the big picture, helping everyone, even non-technical people, understand the system’s main purpose. Example: Imagine you’re working on an online banking system. A context diagram would show the banking system itself, the customers who use it, and other systems it interacts with, such as an email service or a mainframe banking system. This gives everyone a clear understanding of the system’s overall structure and its external connections.
2. Container Diagrams (Level 2): These diagrams zoom in a bit and show the main parts of the system, like web applications, databases, or services. This level helps technical people see how the system is structured. Example: In the online banking system, a container diagram might show the web application that customers use, the database that stores account information, and a payment service that handles transactions. This helps the team see how the system is organized and how the different parts fit together
3. Component Diagrams (Level 3): These diagrams break down each part of the system into smaller pieces, showing how these pieces work together. This level is useful for developers who need to understand how different parts of the system are connected. Example: Within the web application container of the online banking system, a component diagram might show different modules such as login, account management, and transaction processing. This helps developers understand the internal structure of the web application and how the components interact.
4. Code Diagrams (Level 4): These are the most detailed diagrams, showing the actual code structure or design. These diagrams are for developers who need to see how the code is organized. Example: A code diagram might show the classes and methods used in the transaction processing module of the online banking system. This helps developers see how the code is organized and how it supports the overall architecture.

The C4 model uses simple elements like boxes and lines to represent different parts of the system. It encourages clear labeling and a simple layout, making the diagrams easy to understand and collaborate on.

The C4 approach

C4 supports the idea of evolutionary architecture — where the architecture can change and grow as needed. The C4 model allows you to start with a simple diagram and add more details as the system evolves. This makes it perfect for teams that work in an agile way, where things are always changing.

For example, imagine you’re working on an online banking system. A Level 1 context diagram would show the whole system, including how it connects with users and other systems like email or mainframe systems. This gives everyone a clear understanding of the system’s overall structure.

At Level 2, you would create a container diagram that shows the main parts of the system, like the web application, database, and payment service. This helps the team see how the system is put together.

At Level 3, you would create a component diagram that breaks down each main part into smaller pieces, such as the different modules of the web application. This helps developers understand how everything fits together.

Finally, at Level 4, you would create code diagrams that show the details of the code, such as classes or database tables. This level helps developers see how the code is organized and how it supports the system.

Using PlantUML for Diagrams

While the C4 model helps you create clear diagrams, updating them as the system changes can still take a lot of time. This is where PlantUML comes in handy.

PlantUML is a tool that lets you create diagrams using plain text. Instead of using a drawing program, you can write your diagrams as code. This has several advantages:

1. Version Control: Because PlantUML diagrams are just text, you can save them in version control systems like GIT. This makes it easy to track changes and collaborate with others.
2. Collaboration: PlantUML’s text format makes it easy to share and edit diagrams with your team. You can review and update diagrams directly in the code.
3. Automation: You can automate the process of creating and updating diagrams by integrating PlantUML with your development pipeline. For example, you could set up a Jenkins job to generate the latest diagrams every time the code changes.
4. Integration: PlantUML works well with popular coding tools like Visual Studio Code and IntelliJ IDEA, so you can create and update diagrams without leaving your coding environment.

Here’s an example of how you might use PlantUML to create a Level 1 System Context diagram for an online banking system:

@startuml
!include C4_Context.puml

LAYOUT_WITH_LEGEND()

title System Context diagram for E-commerce Platform

Person(customer, "Online Shopper", "A customer who buys products through the e-commerce platform.")
Person(admin, "System Administrator", "Manages the platform and ensures its smooth operation.")
System(ecommerce_system, "E-commerce Platform", "Allows customers to browse products, place orders, and make payments.")

System_Ext(payment_gateway, "Payment Gateway", "Processes online payments for orders.")
System_Ext(warehouse_system, "Warehouse Management System", "Manages inventory and shipping.")

Rel(customer, ecommerce_system, "Browses and purchases products")
Rel(admin, ecommerce_system, "Administers and monitors the platform")
Rel(ecommerce_system, payment_gateway, "Processes payments")
Rel(ecommerce_system, warehouse_system, "Sends order and shipping details")
@enduml

In this example, the diagram represents an E-commerce Platform and its interactions with an online shopper, a system administrator, a payment gateway, and a warehouse management system. It shows the relationships between the different entities involved in the system

Other Diagrams and Tools

PlantUML also lets you create other types of diagrams, like System Landscape Diagrams, Dynamic Diagrams, and Deployment Diagrams. These additional diagrams can help you understand more complex parts of the system.

There are also plugins for tools like Visual Studio Code and IntelliJ IDEA that let you create and view diagrams right in your coding environment. If you prefer, you can even set up a local PlantUML server using Docker:

docker run -d -p 8080:8080 plantuml/plantuml-server:tomcat

This gives you more control and ensures that your diagrams aren’t dependent on external services.

Challenges with PlantUML

While PlantUML is a great tool, it does have some challenges. For example, positioning elements like arrows and boxes can be tricky, especially in complex diagrams. Unlike graphical tools where you can drag and drop, with PlantUML, you have to adjust the layout using code, which can take time.

Another drawback is that PlantUML requires a server component to generate diagrams, which might not be as convenient as some other tools that work directly in the browser. However, the benefits of version control, collaboration, and automation often outweigh these challenges.

Software architecture diagrams are powerful tools for understanding and communicating how a system works. But to be effective, these diagrams need to be clear, consistent, and easy to update. The C4 model provides a simple yet powerful way to create diagrams that meet these goals.

By using tools like PlantUML, you can make it easier to create, update, and share your diagrams, ensuring they stay useful as your system evolves. With the right approach and tools, you can create diagrams that help your team understand the system better and work more effectively.

So, may your diagrams be clear, easy to understand, and always up-to-date. With the cloudairy cloudchart and PlantUML, you’ll have everything you need to create great software architecture diagrams.

cloudairy cloudchart sign up link → https://chart.cloudairy.com/signup