Introduction

In this project, we’ll build a CI/CD pipeline for a voting application deployed through microservices. Each microservice is written in a different language — Python, Node.js, and .NET — creating a polyglot environment. The application uses Redis for caching and Postgres as its primary database. Our goal is to streamline development and deployment workflows using Azure DevOps and Kubernetes.

Overview of Azure DevOps vs. Azure Portal

Azure DevOps is a comprehensive suite of tools that supports the entire software lifecycle, from code repositories and CI/CD pipelines to project tracking and testing.In this project, we’ll use Azure DevOps for managing our code, building CI/CD pipelines, and automating deployments.

Meanwhile, Azure Portal is the central management hub for Azure cloud resources, where we’ll manage services like Azure Kubernetes Service (AKS) and Azure Container Registry (ACR) used in our deployments.

Tools and Technologies Used

To achieve this, we’ll utilize several key tools and platforms:

• Azure DevOps Repos: Centralized source control for code.
• Azure DevOps Pipelines: Automates build, test, and deployment stages.
• SonarQube: Static code analysis to maintain quality and detect vulnerabilities.
• Trivy & OWASP Dependency-Check: Security scanners for container images and dependencies.
• Docker: Containerizes microservices for consistent deployment.
• Azure Container Registry (ACR): Stores Docker images for seamless Azure integration.
• Kubernetes: Orchestrates and scales microservices in a clustered environment.
• Argo CD: GitOps tool for managing and deploying Kubernetes resources.

By the end of this project, you’ll have a solid understanding of how to build a robust CI/CD pipeline for microservices, incorporating best practices for version control, testing, security, and deployment.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Setting Up Azure DevOps and Azure Portal Accounts

Before starting with the project setup, you’ll need accounts on both Azure DevOps and Azure Portal. Here’s how to get started on each platform:

Step 1: Create an Azure DevOps Account

• Go to Azure DevOps and click Sign in.
• If you have a Microsoft account, sign in.
• If not, click Create one! and follow the prompts.
• After signing in, you’ll be prompted to create an organization. Enter a name and select a region. Click Continue.
• Click New Project, enter a project name, select public or private, and click Create.

Step 2: Create an Azure Portal Account

• Go to Azure Portal and click Sign in.
• Use your existing Microsoft account or click Create one! to set up a new account.
• New users can activate a free trial that includes credits for various Azure services.
• Once signed in, explore the dashboard, where you’ll manage services like Azure Kubernetes Service (AKS) and Azure Container Registry (ACR) for this project.

With these accounts set up, you’ll have access to both Azure DevOps and Azure Portal, and you’ll be ready to proceed with the project configuration.

Importing the GitHub Repository into Azure Repos

Now that you have set up your Azure DevOps account, we can import our existing GitHub repository into Azure Repos. Follow these steps:

Step 1: Copy the GitHub Repository URL

• Navigate to my GitHub repository and find the source code here
• On the repository page, click on the green Code button.
• Copy the URL displayed (use HTTPS or SSH format as preferred)

Step 2: Import the Repository into Azure Repos

• Go back to your Azure DevOps project and click on Repos in the left sidebar.
• Click on the Import a Repository option.
• In the Clone URL field, paste the copied GitHub repository URL.
• Choose a name for your new Azure Repo and adjust any other settings if necessary.
• Click the Import button. Azure DevOps will start importing the repository, including its history and contents.
• Once the import is complete, you’ll see your repository listed under Repos in Azure DevOps.

Setting Up Azure Container Registry (ACR)

Now that you’ve imported our GitHub repository into Azure Repos, the next step is to create an Azure Container Registry (ACR) to store and manage your Docker images. Follow these steps:

Step 1: Access Azure Portal

Open your web browser and navigate to Azure Portal.
Enter your Microsoft account credentials to sign in.

Step 2: Create a New Container Registry

1. Navigate to Container Registries
In the Azure Portal, click on Create a resource in the top left corner. In the search bar, type Container Registry and select Container Registry from the dropdown.

2. Click on Create
Click the Create button to start the registry creation process.

3. Fill in the Required Information

• Subscription: Select your Azure subscription.
• Resource Group: Choose an existing resource group or create a new one.
• Registry Name: Enter a unique name for your registry (e.g., my-voting-app-acr).
• Location: Choose the Azure region where you want your registry to be hosted.
• SKU: Select the pricing tier (Basic, Standard, or Premium) based on your needs. The Basic tier is sufficient for small projects.

4. Configure Networking Options (Optional)
You can configure networking options, but for most cases, the default settings will suffice.
5. Review and Create
Click on the Review + create button. After validating the details, click Create to provision your ACR instance.

Step 3: Access Your New ACR

1. Go to Resource
After the deployment is complete, click on Go to resource to access your new Azure Container Registry.

2. Explore the ACR Interface
Here, you can manage your container images, configure access policies, and set up webhooks.

Note on Azure Kubernetes Service (AKS) Costs

To manage costs effectively, we’ll first set up the CI/CD pipeline before configuring the Azure Kubernetes Service (AKS). This approach allows us to validate our deployment process without incurring significant expenses. We will return to set up AKS later in the project.

Setting Up CI/CD Pipelines in Azure DevOps

Now that we have our repository and ACR set up, we can proceed to create CI/CD pipelines in Azure DevOps. This will automate the building, testing, and deployment of our voting application.

Step 1: Navigate to Pipelines

1. Access Azure DevOps
   Open your Azure DevOps project and select Pipelines from the left sidebar.
2. Create a New Pipeline
   Click on the New pipeline button to begin the setup.

Step 2: Select Your Source

1. Choose Your Repository

• Select Azure Repos Git as the source for your pipeline.
• Choose the repository you previously imported containing your voting application code.

Step 3: Configure the Pipeline

1. Select a Pipeline Template
   Choose Docker build and push as your starter template. This template is optimized for building Docker images and pushing them to a container registry.

2. Edit the Pipeline YAML
We will be editing the pipeline YAML to add more stages beyond just building and pushing the Docker images.

Each of our microservices will have its own pipeline, but the core steps will remain consistent.

Configuring Triggers

In Azure DevOps, triggers determine when a pipeline runs automatically based on changes to your code repository.

For our voting application, which consists of three microservices — vote, worker, and result — we will set up 3 different triggers for each microservice. For example, if you are setting up the pipeline for the result microservice, you will use the following trigger configuration in your YAML:

trigger:
 paths:
  include:
    - result/*

This configuration means that the pipeline will automatically run whenever there are changes made to any files in the result directory.

You will need to replace the default trigger with the relevant value for the microservice you are configuring: - result/*, — vote/* or — worker/*

Agent Pool Creation

To run CI/CD tasks, Azure DevOps needs an agent — a machine that performs the actual build and deployment processes. Users with an Azure DevOps subscription can use the default Azure Pipelines agent pool, so they can skip this step.

However, if you’re using the free version of Azure DevOps, you’ll need to set up and maintain your own agent by creating a VM and adding it to an agent pool in Azure DevOps.

Step 1: Create a VM for the Agent

1: Go to the Azure Portal: Sign in to the Azure Portal.

2. Create a Virtual Machine:

• In the Azure Portal, navigate to Virtual Machines and click on Create.
• Select the required options, such as the Region, Resource Group, VM Name, and Size. For DevOps purposes, a standard DS1_v2 size VM should be sufficient.
• Choose an operating system (typically Ubuntu or Windows Server, depending on your preference and needs).
• Configure Administrator Account settings for SSH or password-based access.

3. Configure Networking and Storage:

• Under Networking, ensure your VM has a public IP (or configure it for private access if you’re connecting through a VPN).
• You may keep the default settings for storage unless you have specific requirements.

4. Review and Create the VM:

• After configuring, review your settings and click Create to start the VM deployment.
• Once the VM is created, make a note of its IP address for later access.

Step 2: Generate a Personal Access Token (PAT) for Agent Authentication

To allow your agent to connect to Azure DevOps, you need a Personal Access Token (PAT).

1: In Azure DevOps, click on your profile icon and select Personal Access Tokens.

2. Click + New Token:

• Set a name (e.g., “Agent Token”).
• Choose Organization for the scope.
• Under Permissions, select Full Access (or at minimum Agent Pools and Pipelines).
• Set an expiration date.

3. Click Create, and copy the token. You’ll use it during the agent configuration.

Step 3: Add the VM Agent to Your Azure DevOps Project

1: Open Azure DevOps Project Settings: In Azure DevOps, go to Project Settings by selecting the gear icon in the bottom left.

2: Create an Agent Pool:

• Under Pipelines, select Agent Pools.
• Click on Add pool, provide a name (e.g., Custom Agent Pool), and click Create.

3: Download and Configure the Agent:

• In the Agent Pools section, select the new pool you created and click New agent.
• Choose the operating system of your VM and follow the instructions to download the agent.
• When prompted for authentication, paste your PAT from Step 2.
• After downloading, connect to your VM, install the agent, and configure it using the setup commands provided by Azure DevOps.

4: Run and Verify the Agent

• Once installed, start the agent on your VM.
• Back in Azure DevOps, you should see the agent listed as Online in the agent pool.

Now your agent is ready and can be referenced in your pipeline configuration for running CI/CD tasks!

Make sure you update your pipeline with the new created agentpool.

 # Agent VM image name
pool:
 name: 'azureagent'

Note: Always make sure the agent is running before executing any CI/CD pipelines in Azure DevOps. If the agent is stopped or offline, your pipelines won’t be able to run. If you set up the agent as a service, it should automatically restart with the VM. Otherwise, if you’re using ./run.sh, remember to keep the terminal session open, or restart it as needed.

Tools to Install on the Self-Hosted Agent VM

1. Docker: Required for containerizing applications.
2. Security and Quality Scanning Tools: Trivy and SonarQube Scanner: These are needed if you’re integrating security scans in our pipeline. Install them on the agent to allow direct scanning as part of CI/CD.

# Docker (if not already installed)
sudo apt-get update && sudo apt-get install -y docker.io
# Pull the SonarQube Docker image
docker pull sonarqube
# Run SonarQube in a Docker container
docker run -d --name sonarqube -p 9000:9000 sonarqube
# Download and install Trivy
wget https://github.com/aquasecurity/trivy/releases/latest/download/trivy_0.39.1_Linux-64bit.deb
sudo dpkg -i trivy_0.39.1_Linux-64bit.deb

Final Note

After installing these tools, you might also need to add the agent user to the docker group to allow Docker commands without sudo

sudo usermod -aG docker $(whoami)

Replace $(whoami) with the actual username of the agent on your VM. Remember to restart the agent or log out and log back in for this change to take effect.

Note: We’ll use Azure DevOps task templates for some of our pipeline steps. These templates provide a ready-to-use setup, saving time by filling in common fields and configurations automatically. Once the templates are in place, we’ll customize the specific settings.

Now let’s go back to our pipeline and configure the different stages.

Pipeline Variables Section

In this section, we’ll define the variables that our pipeline will use. These variables are essential for maintaining consistency and flexibility in the CI/CD process. Here’s an example of how you might configure your variables:

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '31a607b0-f8ba-466b-a965-c66a0b69044f'
  imageRepository: 'workerapp'
  containerRegistry: 'mariaazurecicd.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)/result/Dockerfile'
  tag: '$(Build.BuildId)'

Continuous integration (CI)

Code Quality Analysis & Dependency Scanning

We will be using SonarQube for our code analysis and OWASP Dependency-Check to scan for vulnerable dependencies.

SonarQube can be storage-intensive, especially with larger projects or multiple microservices. Here are key recommendations:

• Upgrade Storage on the Current Agent: If SonarQube runs on the same server as your Azure DevOps agent, consider adding extra storage to prevent memory issues.
• Set Up a Dedicated SonarQube Server: For improved performance and isolation, set up SonarQube on a separate VM with sufficient storage.

Access SonarQube UI

To access the SonarQube UI, ensure that port 9000 is open on your firewall. You can access the SonarQube interface by navigating to http://<your-server-ip>:9000 in your web browser after starting the SonarQube container. The default login credentials are:

• Username: admin
• Password: admin (you should change this after the first login).

Creating a Service Connection for SonarQube

1: Navigate to Project Settings: Go to your Azure DevOps project and click on the Project Settings gear icon at the bottom left.

2. Select Service connections: Under the Pipelines section, click on Service connections.

3. Add a New Service Connection:

• Click on New service connection.
• Select SonarQube from the list and click Next.

4. Enter Connection Details:

• Server URL: Enter the URL of your SonarQube server (e.g., http://<your-server-ip>:9000).
• Authentication: You can use either a token or username/password for authentication. If you opt for a token, you need to create one in SonarQube (under My Account > Security) and enter it here.

5. Name the Connection: Give your service connection a recognizable name (e.g., sonar-svc).

6. Verify and Save: Click Verify to check the connection, then click Save to finalize.

Using Azure DevOps Tasks:

1: Navigate to Pipeline Tasks:

• Open the pipeline editor in Azure DevOps.
• Click on the Tasks tab to add pre-configured tasks.

2. Adding SonarQube Tasks:

Prepare Analysis Configuration:

• In the search bar under Tasks, search for “SonarQube.”
• Select SonarQubePrepare (version 7 or latest).
• Set the SonarQube Service Connection to 'sonar-svc' (or the name you configured for your SonarQube server in Azure DevOps).
• Set Project Key and Project Name to match your application, e.g., voteapp.
• For Source Files, use the value . to specify the root directory

Run Code Analysis:

• Add another task, SonarQubeAnalyze.
• Specify the Java version if needed by selecting JAVA_HOME_17_X64 under the jdkversion field, particularly if your project includes Java components.

3. Add OWASP Dependency-Check Task:

• For the dependency check, there isn’t a built-in Azure DevOps task, so you’ll use a Script task.
• Search for and add a Script task, then enter the following script:

# Step to run Dependency-Check
    - script: |
        echo "Running Dependency-Check..."
        dependency-check.sh --project "voteapp" --scan . --outdir ./dependency-check-report --format ALL
      displayName: 'Run OWASP Dependency-Check'

Here is the full yaml pipeline for this stage remember to replace with your actual values.

stages:
  # Code Quality and Dependency Scanning Stage
  - stage: SonarAnalysis
    displayName: SonarAnalysis
    jobs:
      - job: SonarAnalysis
        displayName: SonarAnalysis
        steps:
          # Pull the latest code from the repository
          - checkout: self
            fetchDepth: 0  # Fetch full commit history for detailed analysis
            
          # Prepare SonarQube analysis
          - task: SonarQubePrepare@7
            inputs:
              SonarQube: 'sonar-svc'  # Replace 'sonar-svc' with your SonarQube service connection name
              scannerMode: 'cli'
              configMode: 'manual'
              cliProjectKey: 'voteapp'  # Unique key for the project in SonarQube
              cliProjectName: 'voteapp'
              cliSources: '.'  # Set the root directory for SonarQube to analyze
              extraProperties: 'sonar.java.binaries=.'  # Specify binaries location if needed for Java analysis

          # Run the SonarQube analysis
          - task: SonarQubeAnalyze@7
            inputs:
              jdkversion: 'JAVA_HOME_17_X64'  # Specify Java version if your project includes Java files

          # Run OWASP Dependency-Check for vulnerabilities
          - script: |
              echo "Running Dependency-Check..."
              dependency-check.sh --project "voteapp" --scan . --outdir ./dependency-check-report --format ALL
            displayName: 'Run OWASP Dependency-Check'

Building Docker Image

In this stage, we’ll build our Docker image using the Azure DevOps Docker task. (I’m assuming you are all familiar with using Azure DevOps tasks from our previous SonarQube configuration, so I’ll keep the remaining steps brief.)

The following YAML configuration outlines the build process:

# BUILD STAGE (Docker)
- stage: Build
  displayName: Build
  jobs:
  - job: Build
    displayName: Build
    steps:
    - task: Docker@2
      displayName: Build
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'build'
        Dockerfile: 'worker/Dockerfile'
        tags: '$(tag)'

Configuration Notes:

• containerRegistry: Ensure that this variable points to the service connection created for your Azure Container Registry.(Make sure you have configured this service connection).
• repository: This should match the name of the image repository where the Docker image will be stored.
• Dockerfile: Update the path to the Dockerfile if your Dockerfile is located in a different directory.
• tags: The tag will default to the build ID; however, you can customize it if needed.

Docker Image Scanning with Trivy

In this stage, we will scan the built Docker image for vulnerabilities using Trivy. The following YAML configuration outlines the scanning process:

# SCAN IMAGE STAGE (Trivy)
- stage: TrivyScan
  displayName: Trivy Scan
  jobs:
  - job: TrivyScan
    displayName: Trivy Image Scan
    steps:
    - script: |
        echo "Scanning Docker image with Trivy..."
        trivy image --exit-code 1 --severity HIGH,CRITICAL $(containerRegistry)/$(imageRepository):$(tag) || true
      displayName: 'Run Trivy Scan'
      env:
        TRIVY_SEVERITY: 'HIGH,CRITICAL'
        TRIVY_SKIP_UPDATE: 'true' # Skip the database update to speed up scans

Explanation:

• trivy image: This command scans the specified Docker image for vulnerabilities.
• — exit-code 1: The scan will return a non-zero exit code if vulnerabilities are found, allowing for further action in the pipeline.
• — severity HIGH,CRITICAL: This limits the scan to only report high and critical severity vulnerabilities, which helps focus on the most significant risks.
• TRIVY_SKIP_UPDATE: This environment variable is set to true to skip the database update, which speeds up the scanning process.

Docker Image Push

Using the Azure DevOps Docker task we will push the built Docker image to our container registry.

The following YAML configuration outlines the pushing process:

# PUSH STAGE (Docker)
- stage: Push
  displayName: Push
  jobs:
  - job: Push
    displayName: Push
    steps:
    - task: Docker@2
      displayName: Push
      inputs:
        containerRegistry: '$(dockerRegistryServiceConnection)'
        repository: '$(imageRepository)'
        command: 'push'
        tags: '$(tag)'

Explanation:

• Docker@2 task: This task is used to push the Docker image to the specified container registry.
• containerRegistry: This input specifies the service connection for the Azure Container Registry that was established during the pipeline setup.
• repository: This input defines the name of the image repository where the image will be pushed.
• command: The command is set to push, indicating that we want to push the image to the container registry.
• tags: This input defines the tag for the image being pushed, typically using the build ID or version number for identification.

Update Script (Kubernetes Manifest Files)

In this stage, we will update our existing Kubernetes manifests files and ensure that ArgoCD can detect and deploy the latest Docker image. The following YAML configuration outlines the update process:

# UPDATE SCRIPT STAGE
- stage: Update
  displayName: Update 
  jobs:
    - job: Update
      displayName: Update
      steps:
        - script: sudo apt-get update
          displayName: Update package list

        - script: sudo apt-get install -y dos2unix
          displayName: Install dos2unix

        - script: dos2unix scripts/updateK8sManifests.sh
          displayName: Convert script to Unix format

        - task: ShellScript@2
          displayName: Run Update Script
          inputs:
            scriptPath: 'scripts/updateK8sManifests.sh'
            args: 'vote $(imageRepository) $(tag)'

Explanation:

• Updating Package List: Ensures the system has the latest package information.
• Installing dos2unix: Converts Windows line endings to Unix format for script compatibility (For the Windows users).
• Converting Script to Unix Format: Applies dos2unix to updateK8sManifests.sh for correct line endings.
• Running the Update Script: Executes updateK8sManifests.sh with the specified microservice, repository, and tag, allowing ArgoCD to track changes and deploy the updated image.

Run the Pipeline

1: Trigger the Pipeline:

• Once the pipeline is configured and ready, navigate to Azure DevOps.
• Under your project, go to Pipelines and select the pipeline you want.
• Click on Validate and save > Run Pipeline to manually trigger it.

2. Monitor the Pipeline Execution:

• After triggering, monitor the pipeline’s progress in the Azure DevOps UI.
• You can view logs and check each stage to ensure everything is running smoothly.

3. Check for Success or Failure:

• The pipeline will provide a success or failure status after execution.
• If it fails, review the logs to identify and resolve any issues before re-running.

4. Confirm Deployment:

• Once the pipeline completes successfully, confirm the application deployment by checking ArgoCD or the relevant Kubernetes cluster to ensure the updated application is deployed. (Which we will do at the end of this project don’t worry about this part for now)

5. Repeat the steps for all the remaining microservices.

Continuous Deployment

AKS Cluster Set Up

Since our CI/CD pipeline will ultimately deploy to an Azure Kubernetes Service (AKS) cluster, the next step is to set up the AKS cluster.

Here, we’ll create the cluster through the Azure Portal UI, which simplifies the process and provides flexibility in configuring resources.

1: Log in to Azure Portal
Go to Azure Portal and log in to your account.

2. Navigate to AKS Service
In the search bar, type Azure Kubernetes Service and select it.

3. Create a New Cluster

• Click + Create and select Kubernetes cluster.
• Choose your Subscription and Resource Group ( you can use the resource grp we created at the beginning or create a new one if necessary).
• Enter a Cluster name and select your preferred Region.

4. Configure Node Pools

• Set the Node size and Node count according to your application’s needs. For example, you might start with 3 nodes of type Standard_DS2_v2.
• Choose options like auto-scaling if you expect variable loads.

5. Authentication

• Choose your authentication method (e.g., System-assigned managed identity for access control).
• Enable RBAC (Role-Based Access Control) if you want more granular permissions.

6. Networking and Integration Options

• Configure your Network settings based on your requirements (you can use default settings for simplicity or define specific VNet and Subnet).
• Decide on integration options like Container Insights for monitoring.

7. Review + Create

• Click on Review + create to review your configurations.
• Once confirmed, click Create to deploy the cluster.

8. Get Cluster Credentials
After deployment, retrieve the cluster credentials:

• Go to your AKS resource, and in the left menu, select Connect.
• Follow the instructions to set up kubectl access if needed. Azure will provide a command you can run locally if your machine is already configured with the Azure CLI and kubectl.

Argo CD Setup

ArgoCD is a declarative GitOps tool for Kubernetes that automates application deployments by syncing configurations from a Git repository to your cluster.

It continuously monitors for changes, ensuring the cluster state matches the desired state defined in Git, enabling automated, version-controlled, and reliable deployments.

Here’s how to set up ArgoCD in your Kubernetes environment:

Step 1: Install ArgoCD

a) Create a Namespace for ArgoCD

kubectl create namespace argocd

b) Apply the ArgoCD Manifests

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Step 2: Expose the ArgoCD UI

a) Change the Service Type to LoadBalancer

For Linux/MacOS:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

For Windows:

kubectl patch svc argocd-server -n argocd -p '{\"spec\": {\"type\": \"LoadBalancer\"}}'

b) Retrieve the Load Balancer IP

kubectl get svc argocd-server -n argocd

This command will provide you with the external IP address of the ArgoCD server.

Step 3: Access the ArgoCD UI

Open your web browser and navigate to the Load Balancer IP address obtained from the previous step. You can access the ArgoCD UI from there. Make sure you have opened the inbound port of the argocd server.

Step 4: Retrieve ArgoCD Admin Password

Get the Initial Admin Password

Retrieve the password using the following command

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d

This command will output the initial admin password required to log into the ArgoCD UI.

Add Update Script to Git Repository

To ensure that ArgoCD tracks updates for automatic deployment, add the updateK8sManifests.sh script to your Git repository.

This allows ArgoCD to continuously monitor the repository for changes and trigger deployments when necessary.

Steps:

1. Add the updateK8sManifests.sh script to your Git repository, ideally in a scripts folder or similar directory.
2. Ensure the script contains logic to update the Kubernetes manifests with the new Docker image information.
3. Commit and push the changes to the repository so that ArgoCD can detect them.

Note: ArgoCD will automatically sync and deploy the changes when it detects updates to the repository, following GitOps principles.

Connecting ArgoCD to Azure DevOps using Personal Access Token (PAT)

1: Generate a Personal Access Token (PAT) in Azure DevOps:

• Go to Azure DevOps > Your Profile > Personal Access Tokens.
• Click New Token, give it a name, select the required scopes (at least “Code” or “Read & Write” for accessing repositories), and click Create.
• Copy the generated token as you will need it to connect ArgoCD.

2. Add the Azure DevOps Repository in ArgoCD:

• Log into your ArgoCD UI.
• Go to Settings > Repositories and click + New Repository.
• Select Git as the repository type and enter the URL of your Azure DevOps repository (e.g., https://dev.azure.com/{organization}/{project}/_git/{repository}).
• Choose Username/Password as the authentication method.
• For the Username, you can use your Azure DevOps username (or devops).
• For the Password, paste the PAT you generated.
• Click Save to add the repository.

Adjusting the Timeout Reconciliation Time for ArgoCD (Optional)

In ArgoCD, the reconciliation process is responsible for ensuring that the state of your Kubernetes cluster matches the desired state defined in the Git repository.

The default reconciliation timeout in ArgoCD is 5 minutes (300 seconds).

If you want to adjust this to a shorter value (e.g., 10 seconds), you can update the timeout.reconciliation setting in the argocd-cm ConfigMap.

1. Access ArgoCD ConfigMap

ArgoCD stores configuration settings in a ConfigMap, which you need to modify.

Run the following command to open the ConfigMap in an editor.

kubectl edit configmap argocd-cm -n argocd

2. Modify the Reconciliation Timeout

In the opened ConfigMap, you’ll need to add or modify the timeout.reconciliation setting under the data section. Set the value to 10s for 10 seconds.

data:
  timeout.reconciliation: "10s"

3. Save the Changes

Once you’ve updated the value, save the changes and exit the editor.

4. Apply the Changes

The changes should take effect automatically. If needed, restart the ArgoCD server to apply the changes:

kubectl rollout restart deployment/argocd-server -n argocd

5. Verify the Changes

You can verify that the new reconciliation timeout is applied by checking the ArgoCD logs to see if the reconciliation interval reflects updated value.

kubectl logs deployment/argocd-server -n argocd

Deploy Your Application via ArgoCD

a) Login to the ArgoCD UI

• Use the default username is ‘admin’ and the retrieved admin password to log in to the ArgoCD UI.

b) Add Your Application

• In the ArgoCD UI, click on New App to create a new application.
• Provide the necessary details, including the Git repository URL where your Helm chart or Kubernetes manifests are stored, and configure the application settings.

c) Sync Your Application

• After adding the application, click Sync to deploy the application to your Kubernetes cluster.
• Monitor the deployment status and logs directly from the ArgoCD UI.

Troubleshooting ImagePullBackOff Error

Issue: The ImagePullBackOff error occurs because Kubernetes is unable to authenticate with the container registry when trying to pull the image.

Cause: Kubernetes cannot access the image repository because it does not have the necessary credentials to authenticate with the container registry.

Solution:

1: Verify Registry Credentials:

Ensure that your K8 cluster has the correct credentials to pull from the container registry.

This can be done by creating a Kubernetes secret for the registry credentials and linking it to your deployment.

For ACR, you can create a secret that stores the login credentials for ACR.

kubectl create secret docker-registry acr-secret \
  --docker-server=<your_acr_name>.azurecr.io \
  --docker-username=<your_service_principal_id> \
  --docker-password=<your_service_principal_password> \
  --docker-email=<your_email>

2. Update Deployment with ImagePullSecrets:

Once the secret is created, update your Kubernetes deployment YAML to reference this secret under imagePullSecrets.

Here’s an example of how to reference the secret in your Kubernetes deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: vote
  name: vote
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vote
  template:
    metadata:
      labels:
        app: vote
    spec:
      containers:
      - image: mariaazurecicd.azurecr.io/voteapp:57
        name: vote
        ports:
        - containerPort: 80
          name: vote
      imagePullSecrets:
       - name: acr-secret

3. Ensure the Service Principal Has Correct Permissions:

If you are using a service principal for authentication, make sure it has the correct permissions to pull images from your Azure Container Registry. The service principal needs the AcrPull role assigned.

To assign this role, run the following command:

az role assignment create --assignee <your_service_principal_id> --role AcrPull --scope /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.ContainerRegistry/registries/<acr_name>

4. Check for Role Propagation Delays:

After assigning the correct permissions, there might be a delay in propagating the permissions. You can verify the permissions with:

az role assignment list --assignee <your_service_principal_id>

If the permissions are not showing up or there is still an issue, it might be necessary to wait a few minutes and retry.

5. Verify Deployment:

Once the pipeline completes successfully, confirm the application deployment by checking ArgoCD to ensure the updated application is deployed.

Cleaning Up Resources

Here’s a brief list of resources to delete after completing your project.

1: Azure Kubernetes Service (AKS) Cluster

If the AKS cluster is no longer needed, delete it to stop any associated costs.

az aks delete --name <aks_cluster_name> --resource-group <resource_group> --yes --no-wait

2. Azure Container Registry (ACR)

If you used Azure Container Registry (ACR) for storing your container images, delete it manually:

az acr delete --name <acr_name> --resource-group <resource_group>

3. Persistent Volumes (PVs)

Any persistent volumes (e.g., Azure Disks or Azure Files) created for your AKS applications need to be manually deleted:

kubectl delete pv <pv_name>

4. Virtual Network (VNet) and Subnets

If you created a custom Virtual Network (VNet) for the AKS cluster, delete it manually:

az network vnet delete --name <vnet_name> --resource-group <resource_group>

5. Service Principal

If you created a service principal for the AKS cluster’s managed identity, it will not be automatically deleted. Remove it manually:

az ad sp delete --id <service_principal_id>

6. Kubernetes Resources (Optional)

Any Kubernetes resources (e.g., secrets, configmaps, deployments) you created in the AKS cluster will persist. You can delete them using kubectl:

kubectl delete all --all

Ensure that any network security groups (NSGs), public IP addresses, and other associated resources are also cleaned up if not needed.

Conclusion

In this project, we set up a comprehensive CI/CD pipeline with Azure DevOps and integrated it with Kubernetes using ArgoCD for GitOps.

We utilized SonarQube for code quality analysis, OWASP Dependency-Check for scanning dependencies for known vulnerabilities, docker for building and pushing images, and Trivy for scanning Docker images.

The pipeline handled the complete flow from building, scanning, to deploying Docker images in AKS, with updates managed by ArgoCD.

We also addressed troubleshooting common issues like ImagePullBackOff. Finally, we outlined the steps for cleaning up resources, including AKS, ACR, and custom VMs.

This workflow ensures automated deployment, monitoring, and security in a Kubernetes environment.

Acknowledgment

This project was significantly inspired by Abhishek Veeramalla, known for his excellent DevOps content. His content provided a solid foundation for my enhancement.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub , Medium and LinkTree

CI/CD Pipeline with Azure DevOps, ArgoCD, & Kubernetes was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

The MERN stack, comprising MongoDB for the database, Express.js as the backend framework, React as the frontend library, and Node.js to run the backend server, is a popular choice for building modern web applications.

However, setting up and managing the different components of a MERN application can sometimes be cumbersome, especially when transitioning between development and production environments.

Containerization simplifies this process by packaging each component of the application into lightweight, standalone containers. Docker, one of the most widely used containerization platforms, allows developers to encapsulate their applications along with all dependencies, ensuring consistency across various environments.

In this article, I’ll walk through how to containerize a MERN stack application using Docker and deploy it seamlessly with Docker Compose. By the end, you’ll have a fully containerized MERN application running smoothly with minimal configuration effort, ready for scaling and deployment in any environment.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Cloning the Application from GitHub

1: Install Git: Ensure that Git is installed on your local machine. If it’s not installed, download and install it from Git’s official website.

2: Open Terminal/Command Prompt: Open your terminal or command prompt where you will run Git commands.

3: Clone the Repository: Use the git clone command followed by the URL of the repository you want to clone. You can find the Source Code here.

You can get this URL from the GitHub repository page (usually found under the “Code” button).

git clone https://github.com/wangoimwangi/MERN-docker-compose.git

4: Navigate into the Cloned Directory: Once the cloning process is complete, navigate to the project folder.

cd your-repository-name

At this point, you should have a local copy of the MERN stack application, ready for containerization.

Creating the Dockerfiles

To containerize your MERN stack application, you’ll need to create two separate Dockerfiles: one for the frontend (React) and another for the backend (Node.js and Express). Each Dockerfile specifies how to build the respective Docker image for your application.

Frontend Dockerfile

Here’s the Dockerfile for the frontend:

# Use the official image as a parent image
# Description: Dockerfile for the client side of the MERN stack application
FROM node:18.9.1

# Set the working directory
WORKDIR /app

# Copy the package.json file from your host to your current location in the container
COPY package.json ./

# Install the dependencies defined in package.json
RUN npm install

# Inform Docker that the container is listening on port 5173 at runtime
EXPOSE 5173

# Copy the rest of your app's source code from your host to your image filesystem
COPY . .

# Run the specified command within the container to start the application
CMD ["npm", "run", "dev"]

Explanation:

1. Base Image: We start with the official Node.js image (node:18.9.1), which provides the necessary environment to run a Node.js application.
2. Working Directory: The WORKDIR /app command sets the working directory inside the container to /app, where all subsequent commands will be run.
3. Copying package.json: The COPY package.json ./ command copies the package.json file from your local machine into the container, allowing us to install the necessary dependencies.
4. Installing Dependencies: The RUN npm install command installs the application dependencies specified in package.json.
5. Port Exposure: The EXPOSE 5173 command informs Docker that the container will listen on port 5173 at runtime.
6. Copying Source Code: The COPY . . command copies the rest of the application source code into the container.
7. Starting the Application: Finally, the CMD ["npm", "run", "dev"] command runs the application in development mode when the container starts.

Backend Dockerfile

Here’s the Dockerfile for the backend:

FROM node:18.9.1

# Set the working directory
WORKDIR /app

# Copy the package.json file from your host to your current location in the container
COPY package.json ./

# Install the dependencies defined in package.json
RUN npm install

# Copy the rest of your app's source code from your host to your image filesystem
COPY . .

# Inform Docker that the container is listening on port 5050 at runtime
EXPOSE 5050

# Run the specified command within the container to start the application
CMD ["npm", "start"]

Explanation:

1. Base Image: Similar to the frontend, this Dockerfile also uses the official Node.js image.
2. Working Directory: The working directory is set to /app.
3. Copying package.json: The package.json file is copied into the container to install dependencies.
4. Installing Dependencies: Dependencies are installed using RUN npm install.
5. Copying Source Code: The COPY . . command brings the rest of the backend source code into the container.
6. Port Exposure: The EXPOSE 5050 command indicates that the backend service will run on port 5050.
7. Starting the Application: The CMD ["npm", "start"] command starts the backend server when the container is launched.

Database

In our MERN stack, we will be using MongoDB as the database to store and manage application data. For our Docker setup, we will use the official MongoDB image, which provides a lightweight and consistent environment for running MongoDB.

By creating these Dockerfiles, you’ve defined how to build the frontend and backend containers for your MERN stack application. Each Dockerfile encapsulates all necessary configurations, ensuring that your application can run consistently across different environments.

Setting Up Docker Compose

Docker Compose allows you to define and run multi-container Docker applications. In a MERN stack application, we typically have separate containers for the frontend, backend, and database (MongoDB). Setting up Docker Compose makes it easy to manage these containers as a single application.

Creating the docker compose File

1. In the root directory of your MERN stack application, create a new file named docker-compose.yml.
2. Open the docker-compose.yml file in your preferred code editor and add the following configuration:

services:
  backend:
    build: ./mern/backend
    ports:
      - "5050:5050" 
    networks:
      - mern_network
    environment:
      MONGO_URI: mongodb://mongo:27017/mydatabase  
    depends_on:
      - mongodb

  frontend:
    build: ./mern/frontend
    ports:
      - "5173:5173"  
    networks:
      - mern_network
    environment:
      REACT_APP_API_URL: http://backend:5050 

  mongodb:
    image: mongo:latest  
    ports:
      - "27017:27017"  
    networks:
      - mern_network
    volumes:
      - mongo-data:/data/db  

networks:
  mern_network:
    driver: bridge 

volumes:
  mongo-data:
    driver: local  # Persist MongoDB data locally

Configuration Breakdown:

Backend Service:

• Build Context: The backend is built from the Dockerfile located in the ./mern/backend directory.
• Ports: Exposes port 5050 on the host, allowing access to the backend API.
• Network: Connected to mern_network for inter-service communication.
• Environment Variables: MONGO_URI is set to connect to the MongoDB service using the hostname mongo and the specified database (mydatabase).
• Dependency: The backend service depends on the MongoDB service, ensuring it starts only after the database is up.

Frontend Service:

• Build Context: The frontend is built from the Dockerfile located in the ./mern/frontend directory.
• Ports: Exposes port 5173 for the frontend application.
• Network: Also connected to mern_network.
• Environment Variables: REACT_APP_API_URL is set to point to the backend service for API calls.

MongoDB Service:

• Image: Uses the latest official MongoDB image.
• Ports: Exposes port 27017 for database connections.
• Network: Connected to mern_network.
• Volumes: mongo-data is mounted to persist MongoDB data across container restarts.

Networks: A custom network named mern_network is created to allow seamless communication between the services.

Volumes: The mongo-data volume is defined to ensure that MongoDB data is stored persistently on the local system, safeguarding against data loss.

This configuration ensures that all components of the MERN stack are well-integrated and can communicate effectively, providing a solid foundation for development and deployment.

Running the Application

With your docker-compose.yml file set up, you can now build and run your MERN stack application using Docker Compose.

1. Open your terminal in the root directory of your project (where the docker-compose.yml file is located).
2. Run the following command to start the application:

docker-compose up --build

The --build flag ensures that Docker Compose builds the images as specified in your Dockerfiles.

This command will:

• Build the Docker images for the frontend and backend.
• Start the MongoDB service and initialize it with the specified volume for persistent data storage.
• Start the backend and frontend services, making them accessible on the configured ports.

3. Once the services are up and running, you can access your frontend at http://localhost:5173 and your backend at http://localhost:5050

4. When you’re finished testing or using the application, you can stop the running services by pressing CTRL + C in the terminal where Docker Compose is running.

To remove the containers and associated networks, run:

docker-compose down

Conclusion

By following these steps, you’ve successfully containerized a MERN stack application and deployed it using Docker Compose. This setup not only simplifies development but also ensures that your application is consistent across various environments. You can now focus on building features and scaling your application, confident in the robust infrastructure you’ve established.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Containerizing a MERN stack Application and Deploying using Docker Compose was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This project focuses on deploying a Go application using modern DevOps practices. The goal is to build and run the application locally, then containerize it and deploy it to a Kubernetes environment. Key components of this project include:

• Local Development: Building and running the Go application on a local machine.
• Dockerfile Creation: Creating a multistage Dockerfile for efficient containerization.
• Containerization: Using Docker to package the application into containers.
• AWS EKS Cluster: Setting up and configuring an Amazon EKS (Elastic Kubernetes Service) cluster.
• Kubernetes Manifests: Writing Kubernetes manifests for deployment and management.
• Ingress Controller: Installing and configuring an Ingress Controller for managing external access to the application.
• Local DNS Mapping: Configure Local DNS mapping to access the application via a custom domain.
• Helm: Using Helm to simplify and manage Kubernetes deployments.
• Continuous Integration: Implementing Continuous Integration (CI) with GitHub Actions for automated testing and building.
• Continuous Deployment: Using Argo CD for automated deployment of changes to the Kubernetes cluster, ensuring seamless updates and rollouts.

This project demonstrates a comprehensive approach to building, deploying, and managing containerized applications in a Kubernetes environment, leveraging CI/CD best practices for efficient and reliable software delivery.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Cloning the Application from GitHub

1: Install Git: Ensure that Git is installed on your local machine. If it’s not installed, download and install it from Git’s official website.

2: Open Terminal/Command Prompt: Open your terminal or command prompt where you will run Git commands.

3: Navigate to the Desired Directory: Change to the directory where you want to clone the repository. Use the cd command to navigate:

cd /path/to/your/directory

4: Clone the Repository: Use the git clone command followed by the URL of the repository you want to clone. You can find the Source Code here.

You can get this URL from the GitHub repository page (usually found under the “Code” button).

git clone https://github.com/wangoimwangi/go-web-app-devops.git

5: Navigate into the Cloned Directory: Change to the directory of the cloned repository:

cd your-repository

6: Verify the Clone: List the files in the directory to ensure that the repository has been cloned correctly:

ls

You now have the source code of the application on your local machine and can start working on it.

Build and Run the Go Application Locally

Before you containerize the Go application, it’s important to understand its functionality by building and running it locally.

This will help you verify that the application works as expected and give you a clear understanding of its behavior.

1: Ensure Go is Installed: Make sure that Go is installed on your local machine. If it’s not installed, download and install it from Go’s official website.

2:Navigate to the Application Directory: Open your terminal and change to the directory where your Go application’s source code is located. Use the cd command.

3: Build the Application: Run go build: This command compiles the Go source code into an executable binary. Execute the following command in the terminal:

go build -o main .

This will create an executable file named myapp (you can name it anything you like).

4: Run the Application: Run the compiled binary to start the application

go run main.go

5: Access the Application: Open your browser and navigate to http://localhost:8080/courses to interact with the application.

6: Observe Output: Check the application’s output and behavior to understand how it works.

Creating a Multistage Dockerfile

To containerize the Go application efficiently, we use a multistage Dockerfile. This approach helps in minimizing the final image size and ensuring a clean, production-ready environment.

Below is the Dockerfile used for this purpose:

# Containerize the go application that we have created
# This is the Dockerfile that we will use to build the image
# and run the container

# Start with a base image
FROM golang:1.21 as base

# Set the working directory inside the container
WORKDIR /app

# Copy the go.mod and go.sum files to the working directory
COPY go.mod ./

# Download all the dependencies
RUN go mod download

# Copy the source code to the working directory
COPY . .

# Build the application
RUN go build -o main .

#######################################################
# Reduce the image size using multi-stage builds
# We will use a distroless image to run the application
FROM gcr.io/distroless/base

# Copy the binary from the previous stage
COPY --from=base /app/main .

# Copy the static files from the previous stage
COPY --from=base /app/static ./static

# Expose the port on which the application will run
EXPOSE 8080

# Command to run the application
CMD ["./main"]

Explanation:

1. Base Image: The Dockerfile starts with the golang:1.21 image, which contains the Go runtime and tools necessary for building the application.
2. Setup Working Directory: Sets /app as the working directory inside the container.
3. Dependency Management: Copies go.mod and go.sum and runs go mod download to fetch dependencies.
4. Application Code: Copies the application source code into the container and builds the Go application.
5. Multistage Build: Switches to a distroless image to run the application, reducing the final image size by including only the runtime environment and the compiled binary.
6. Copy Binaries and Static Files: Copies the built binary and any static files from the previous stage.
7. Expose Port: Exposes port 8080 for accessing the application.
8. Run Command: Specifies the command to run the application when the container starts.

This Dockerfile ensures that the final image is lightweight and secure, focusing solely on running the application with minimal overhead.

Containerization

Containerization packages your application and its dependencies into a container, which can be consistently run on any container platform.

Here’s how to containerize your Go web application using Docker:

Prerequisites:

1: Install Docker: Download and install Docker from Docker’s official website if it’s not already installed.

2: Install Docker Desktop:

• Windows/Mac Users: Download and install Docker Desktop from Docker’s official website. Docker Desktop is necessary for managing Docker containers and provides a graphical user interface.
• Linux Users: Install Docker Engine by following the instructions on Docker’s official website. Docker Desktop is not available for Linux, but Docker Engine will allow you to run Docker containers.

3: Sign Up for a DockerHub Account: If you don’t have a DockerHub account, sign up at DockerHub to create one. DockerHub is a cloud-based registry service for Docker images and doesn’t require installation.

4: Ensure Docker Daemon is Running:

• Windows/Mac Users: Ensure Docker Desktop is running.
• Linux Users: Verify that the Docker service is running with:

sudo systemctl start docker

• You can check its status with:

sudo systemctl status docker

With these prerequisites in place, you’re ready to start containerizing your application.

Commands to Build, Run, and Push the Docker Container:

Build the Docker Image:

docker build -t <your-docker-username>/go-web-app .

Replace <your-docker-username> with your DockerHub username. This tags the image for uploading to DockerHub.

Log In to DockerHub:

Enter your DockerHub credentials to authenticate your Docker client.

docker login

Run the Docker Container:

This command maps port 8080 on your host to port 8080 in the container.

docker run -p 8080:8080 -d <your-docker-username>/go-web-app

Push the Docker Image to DockerHub:

This uploads the image to DockerHub, making it accessible from anywhere.

docker push <your-docker-username>/go-web-app

Verify the Image on DockerHub:

Log in to your DockerHub account at DockerHub.

Navigate to your repository and ensure the pushed image appears under the tags section. This confirms that your image was successfully uploaded.

Access the Application:

Open your browser and navigate to http://localhost:8080/courses to interact with the running application.

AWS EKS Cluster Setup

Setting up an AWS EKS (Elastic Kubernetes Service) cluster involves several key steps. Follow this guide to get your EKS cluster up and running.

Prerequisites

Before you begin, ensure you have the following tools installed and configured:

• kubectl: A command-line tool for working with Kubernetes clusters. For more information on installing or updating kubectl, see the official documentation.
• eksctl: A command-line tool for working with EKS clusters that automates many tasks. For more information on installing or updating eksctl, see the eksctl documentation.
• AWS CLI: A command-line tool for working with AWS services, including Amazon EKS. For more information on installing, updating, and uninstalling the AWS CLI, see the AWS CLI User Guide.

After installing the AWS CLI, configure it with:

aws configure

Verify all the tools are installed successfully by checking their versions.

Install and Configure EKS

Use eksctl to create the cluster:

eksctl create cluster --name demo-cluster --region us-east-1

Note: The EKS cluster creation process can take several minutes to complete. Please be patient during this step. Be aware that creating an EKS cluster will incur charges. If cost is a concern, consider deploying your application on Minikube as an alternative.

Update kubectl to use the new EKS cluster

aws eks --region us-east-1 update-kubeconfig --name demo-cluster

Verify via the AWS Console manager that the EKS Cluster is created.

Check the status of your nodes to ensure they are running:

kubectl get nodes

Set Up Networking and IAM Roles

• Configure Networking: Ensure your VPC, subnets, and security groups are properly configured to allow traffic to and from your EKS cluster.
• Set Up IAM Roles: Verify that IAM roles for EKS and worker nodes are correctly configured with the necessary permissions.

Kubernetes Manifest Creation

To deploy your containerized Go application to a Kubernetes cluster, you need to create several Kubernetes manifests.

These manifests define how your application is deployed, exposed, and accessed.

Create a folder calledK8s then inside the K8s folder, create another folder named manifests. Inside the manifests folder, create the following files: deployment.yaml, service.yaml, ingress.yaml

1: Create deployment.yaml: This manifest defines how your application is deployed, including the number of replicas and the Docker image to use.

Replace <docker-user-name> with your DockerHub username.

# This is a sample deployment manifest file for a simple web application.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-web-app
  template:
    metadata:
      labels:
        app: go-web-app
    spec:
      containers:
      - name: go-web-app
        image: <docker-user-name>/go-web-app
        ports:
        - containerPort: 8080

2. Create service.yaml: This manifest defines how your application is exposed within the Kubernetes cluster. It creates a Service that routes traffic to your application pods.

# Service for the application
apiVersion: v1
kind: Service
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: go-web-app
  type: ClusterIP

3: Create ingress.yaml: This manifest defines how your application is exposed to the outside world. It creates an Ingress resource that manages external access to the application.

Replace go-web-app.local with your actual domain or IP address if needed.

# Ingress resource for the application
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-web-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: go-web-app.local
    http:
      paths: 
      - path: /
        pathType: Prefix
        backend:
          service:
            name: go-web-app
            port:
              number: 80

4. Apply the Manifests: Use kubectl to apply these manifests to your EKS cluster:

kubectl apply -f K8s/manifests/deployment.yaml

kubectl apply -f K8s/manifests/service.yaml

kubectl apply -f K8s/manifests/ingress.yaml

5. Verify Deployment: Ensure your pods, services, and ingress are running correctly.

kubectl get pods

kubectl get services

kubectl get ingress

Ingress Controller Setup

An Ingress Controller manages and routes external traffic to services within your Kubernetes cluster. It listens for changes to Ingress Resources, which define how external HTTP and HTTPS requests are directed to your services.

The Ingress Controller configures the underlying load balancer or proxy (e.g., NGINX) based on the rules specified in the Ingress Resources, handling routing, SSL termination, and other traffic management tasks.

To manage external access to your services running on an AWS EKS cluster, you’ll need to set up an NGINX Ingress Controller.

1. Deploy NGINX Ingress Controller

Apply the NGINX Ingress Controller Manifest:

• Run the following kubectl command to deploy the NGINX Ingress Controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.1/deploy/static/provider/aws/deploy.yaml

• This manifest sets up the NGINX Ingress Controller, which will handle routing external requests to the appropriate services within your cluster.

2. Verify the Installation

Check the Status of the Ingress Controller Pods:

• Ensure that the NGINX Ingress Controller pods are running

kubectl get pods -n ingress-nginx

Check the Service:

• Verify that the NGINX Ingress Controller service is properly set up and accessible:

kubectl get services -n ingress-nginx

Verify the Load Balancer

• Check if the Load Balancer is created in the AWS Management Console.

Check the Ingress Controller Logs:

• To troubleshoot or verify that the controller is working correctly, you can check the logs of the Ingress Controller pods:

kubectl logs -n ingress-nginx <nginx-ingress-controller-pod-name>

By following these steps, you will have the NGINX Ingress Controller set up and ready to manage inbound traffic to your applications deployed on AWS EKS.

Setting Up Local DNS Mapping

To access your application using a custom domain like go-web-app.local, as specified in our ingress rules in ingress.yaml file you'll need to set up a local DNS mapping.

This will allow you to use the domain name specified in the Ingress Resource (go-web-app.local) instead of relying on the IP address provided by the Load Balancer. Here’s how to do it:

Steps to Set Up Local DNS Mapping

1: Obtain the Load Balancer IP Address

• Navigate to the EC2 Dashboard in the AWS Management Console.
• Select Load Balancers from the left-hand menu.
• Locate the Load Balancer associated with the NGINX Ingress Controller.
• Copy the DNS Name of the Load Balancer (e.g., abc123.elb.amazonaws.com).
• Use the DNS Name to find the Load Balancer’s public IP address or resolve the DNS name to an IP address using a command like:

nslookup <Load-Balancer-DNS-Name>

2: Edit the Hosts File

• For Windows:

a) Open Notepad or your preferred text editor as an administrator (right-click and select “Run as administrator”).

b) Open the file located at C:\Windows\System32\drivers\etc\hosts.

c) Add a new entry mapping the Load Balancer’s IP address to the domain name:

<Load-Balancer-IP> go-web-app.local

d) Save the changes to the hosts file.

• For macOS and Linux:

a) Open a terminal.

b) Use a text editor with sudo to open the file located at /etc/hosts

sudo nano /etc/hosts

c) Add a new entry mapping the Load Balancer’s IP address to the domain name:

<Load-Balancer-IP> go-web-app.local

d) Save and exit the editor (e.g., press Ctrl+X, then Y to confirm in nano).

3: Verify the Local DNS Mapping

• Open your web browser and navigate to http://go-web-app.local/courses
• Ensure that the application loads correctly and that you can access it via the specified domain.

By setting up this local DNS mapping, you ensure that requests to go-web-app.local are directed to your application's Load Balancer, matching the domain name specified in your Ingress Resource.

Helm Chart Configuration

Helm is a package manager for Kubernetes that helps manage Kubernetes applications using Helm charts.

Helm simplifies the deployment and management of applications on Kubernetes by packaging all necessary configurations into a single chart.

Helm Chart Structure

A Helm chart consists of several key components that define how your application is deployed and managed in Kubernetes:

a) Chart.yaml

This file contains metadata about the Helm chart, including its name, version, and description.

b) values.yaml

This is the default configuration file for the chart. It defines values that can be customized to modify the behavior of the chart without changing the underlying templates.

Users can override these values when deploying the chart.

c) templates/

This directory contains Kubernetes manifest templates that define the resources for your application, such as deployments, services, and ingresses.

Helm renders these templates into Kubernetes manifests using the values specified in values.yaml.

d) charts/

This directory can contain any sub-charts that your chart depends on. These are other Helm charts packaged as dependencies.

e) README.md

(Optional) This file provides documentation about the chart, including how to use and configure it.

Steps to Configure and Deploy Using Helm

a) Install Helm

• Follow the official Helm installation guide to install Helm on your local machine.

b) Create a Helm Chart

• Create a new Helm chart for your application:

helm create go-web-app-chart

This command creates a directory named go-web-app-chart with a default chart structure.

c) Customize the Helm Chart

• Navigate to the go-web-app-chart directory

cd go-web-app-chart

• Modify the templates/deployment.yaml, templates/service.yaml, and templates/ingress.yaml if necessary to suit your application’s specific needs. Use the manifest files we created before.

templates/deployment.yaml

# This is a sample deployment manifest file for a simple web application.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-web-app
  template:
    metadata:
      labels:
        app: go-web-app
    spec:
      containers:
      - name: go-web-app
        image: abhishekf5/go-web-app:{{ .Values.image.tag }}
        ports:
        - containerPort: 8080

templates/service.yaml

# Service for the application
apiVersion: v1
kind: Service
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: go-web-app
  type: ClusterIP

templates/ingress.yaml

# Ingress resource for the application
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-web-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: go-web-app.local
    http:
      paths: 
      - path: /
        pathType: Prefix
        backend:
          service:
            name: go-web-app
            port:
              number: 80

• Customize the default values.yaml file to match your application’s configuration:

# Default values for go-web-app-chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: abhishekf5/go-web-app
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "10016307834"

ingress:
  enabled: false
  className: ""
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific

d) Package and Deploy the Helm Chart

• Package the Helm chart:

helm package go-web-app-chart

• Deploy the Helm chart to your Kubernetes cluster:

helm install go-web-app-chart ./go-web-app-chart-<version>.tgz

Replace <version> with the actual version number of the chart, such as 1.0.0, for example, go-web-app-1.0.0.tgz.

e) Verify the Deployment

• Check the status of the deployed application:

kubectl get all -l app=go-web-app

Ensure that the application is running as expected and that the Ingress rules are applied correctly.

By following these steps, you’ll configure and deploy your application using Helm, allowing for simplified management and updates of your Kubernetes resources.

Continuous Integration with GitHub Actions

Continuous Integration (CI) is a key part of the DevOps pipeline that automates the process of integrating code changes into a shared repository frequently.

GitHub Actions allows you to create workflows that build, test, and deploy your application automatically.

Steps to Configure CI with GitHub Actions

a) Create a GitHub Actions Workflow File

• In your GitHub repository, navigate to the .github/workflows directory. If it doesn’t exist, create it.
• Create a new YAML file for your workflow, such as ci.yml.

b) Define the Workflow

• Open ci.yml and define the workflow using the following template:

# CICD using GitHub actions

name: CI/CD

# Exclude the workflow to run on changes to the helm chart
on:
  push:
    branches:
      - main
    paths-ignore:
      - 'helm/**'
      - 'k8s/**'
      - 'README.md'

jobs:

build:
    runs-on: ubuntu-latest

steps:
    - name: Checkout repository
      uses: actions/checkout@v4

- name: Set up Go 1.22
      uses: actions/setup-go@v2
      with:
        go-version: 1.22

- name: Build
      run: go build -o go-web-app

- name: Test
      run: go test ./...
  
  code-quality:
    runs-on: ubuntu-latest

steps:
    - name: Checkout repository
      uses: actions/checkout@v4

- name: Run golangci-lint
      uses: golangci/golangci-lint-action@v6
      with:
        version: v1.56.2
  
  push:
    runs-on: ubuntu-latest

needs: build

steps:
    - name: Checkout repository
      uses: actions/checkout@v4

- name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

- name: Login to DockerHub
      uses: docker/login-action@v3
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Build and Push action
      uses: docker/build-push-action@v6
      with:
        context: .
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.DOCKERHUB_USERNAME }}/go-web-app:${{github.run_id}}

update-newtag-in-helm-chart:
    runs-on: ubuntu-latest

needs: push

steps:
    - name: Checkout repository
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.TOKEN }}

- name: Update tag in Helm chart
      run: |
        sed -i 's/tag: .*/tag: "${{github.run_id}}"/' helm/go-web-app-chart/values.yaml

- name: Commit and push changes
      run: |
        git config --global user.email "<your-git-email-address>"
        git config --global user.name "<your-git-username>"
        git add helm/go-web-app-chart/values.yaml
        git commit -m "Update tag in Helm chart"
        git push

Replace values <your-git-email-address> and <your-git-username> with your actual values.

If you are not sure of the syntax of action to put on your workflow google search “githubactions market place”

Workflow Breakdown:

a) Trigger

• on: push: Runs the workflow on pushes to the main branch, excluding changes in specified paths like Helm charts and Kubernetes manifests.

b) Jobs

build:

• Checkout repository: Checks out the repository code.
• Set up Go: Configures the Go environment.
• Build: Compiles the Go application.
• Test: Runs tests for the application.

code-quality:

• Checkout repository: Checks out the repository code.
• Run golangci-lint: Executes golangci-lint to perform linting and check for code quality issues.

push:

• Checkout repository: Checks out the repository code.
• Set up Docker Buildx: Configures Docker Buildx for building multi-platform images.
• Login to DockerHub: Authenticates to Docker Hub using secrets.
• Build and Push Docker Image: Builds and pushes the Docker image to Docker Hub with a tag based on the GitHub run ID.

update-newtag-in-helm-chart:

• Checkout repository: Checks out the repository code.
• Update tag in Helm chart: Updates the image tag in the Helm chart values file.
• Commit and push changes: Commits and pushes the updated Helm chart to the repository.

c) Set Up DockerHub Credentials

• Log in to GitHub: Navigate to your repository.
• Go to Settings: Click on the “Settings” tab.
• Select Secrets and Variables: Choose “Secrets and Variables” and then “Actions”.
• Add New Repository Secret: Add the necessary secrets for DockerHub credentials:

DOCKERHUB_USERNAME: Your DockerHub username.

DOCKERHUB_TOKEN: A personal access token or password for DockerHub.

Ensure that your CI workflow (GitHub Actions YAML file) uses these secrets to authenticate with DockerHub.

d) Push the Workflow File

• Commit and push the ci.yml file to your GitHub repository

git add .github/workflows/ci.yml
git commit -m "Add GitHub Actions CI workflow"
git push

e) Monitor CI Runs

• Go to the Actions tab in your GitHub repository to view the progress and results of your CI runs.

By following this setup, you ensure that code changes trigger automated builds, tests, and deployments, integrating seamlessly into your development workflow.

Continuous Delivery with ArgoCD

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It enables you to manage and automate your application deployments directly from Git repositories.

Here’s how to set up ArgoCD in your Kubernetes environment:

Step 1: Install ArgoCD

a) Create a Namespace for ArgoCD

kubectl create namespace argocd

b) Apply the ArgoCD Manifests

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Step 2: Expose the ArgoCD UI

a) Change the Service Type to LoadBalancer

For Linux/MacOS:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

For Windows:

kubectl patch svc argocd-server -n argocd -p '{\"spec\": {\"type\": \"LoadBalancer\"}}'

b) Retrieve the Load Balancer IP

kubectl get svc argocd-server -n argocd

This command will provide you with the external IP address of the ArgoCD server.

Step 3: Access the ArgoCD UI

Open your web browser and navigate to the Load Balancer IP address obtained from the previous step. You can access the ArgoCD UI from there.

However, Instead of using the Load Balancer IP address directly, you can set up a local DNS entry or update your DNS settings to map a domain name (go-web-app.local) to the Load Balancer IP for the ArgoCD UI just like we did in the ingress section of this project.

Step 4: Retrieve ArgoCD Admin Password

Get the Initial Admin Password

Retrieve the password using the following command

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d

This command will output the initial admin password required to log into the ArgoCD UI.

Step 5: Deploy Your Application via ArgoCD

a) Login to the ArgoCD UI

• Use the default username is ‘admin’ and the retrieved admin password to log in to the ArgoCD UI at http://argocd.local

b) Add Your Application

• In the ArgoCD UI, click on New App to create a new application.
• Provide the necessary details, including the Git repository URL where your Helm chart or Kubernetes manifests are stored, and configure the application settings.

c) Sync Your Application

• After adding the application, click Sync to deploy the application to your Kubernetes cluster.
• Monitor the deployment status and logs directly from the ArgoCD UI.

This guide provides a comprehensive setup for integrating ArgoCD into your CI/CD pipeline, enabling automated deployments and efficient management of your applications.

Cleaning Up Resources

After completing your DevOps implementation, it is important to clean up all resources used in this project to avoid incurring any additional costs. Below are the steps to remove the resources you created during this project:

a) Delete Kubernetes Resources: To delete the Kubernetes resources (Deployment, Service, Ingress) that were created, use the following commands:

kubectl delete -f K8s/manifests/deployment.yaml
kubectl delete -f K8s/manifests/service.yaml
kubectl delete -f K8s/manifests/ingress.yaml

b) Remove Helm Releases: If you used Helm for deployment, you can delete the Helm release with:

helm uninstall go-web-app-chart

c) Delete AWS EKS Cluster: To delete the EKS cluster, use the eksctl command:

eksctl delete cluster --name demo-cluster --region us-east-1

Make sure to also manually verify and clean up any associated resources in the AWS Management Console if necessary.

d) Remove Docker Images from DockerHub: If you pushed Docker images to DockerHub, you can remove them using the following commands:

docker rmi <your-docker-username>/go-web-app

Log in to DockerHub and delete the repository or individual tags through the DockerHub web interface if required.

e) Clean Up Local DNS Mappings: Remove the local DNS mapping entry from your hosts file:

• For Windows: Edit C:\Windows\System32\drivers\etc\hosts and remove the line mapping the IP address to go-web-app.local.
• For macOS/Linux: Edit /etc/hosts and remove the corresponding line.

By following these steps, you will ensure that no unnecessary resources are left running, thus avoiding additional costs associated with cloud and container services.

Conclusion

This project successfully demonstrates the process of deploying a Go application using contemporary DevOps practices.

By building and running the application locally, creating a multistage Dockerfile for efficient containerization, and deploying the application to an Amazon EKS cluster, we’ve showcased a robust workflow for modern software delivery.

Acknowledgment

This project was significantly inspired by Abhishek Veeramalla, known for his excellent DevOps content.

His content provided a solid foundation for my enhancement. Check out his content here.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Comprehensive DevOps Implementation was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

What is Ansible?

Ansible is an open-source IT automation tool used for provisioning, configuration management, application deployment, and orchestration.

How Ansible Works

Ansible is agentless, meaning no software needs to be installed on managed nodes.

It connects via SSH (or other methods for non-SSH environments) and pushes small programs called Ansible modules to achieve the desired system state.

These modules are idempotent, ensuring changes are made only when necessary.

Project Overview

This Ansible Realtime Project aims to demonstrate the automation capabilities of Ansible in managing AWS EC2 instances.

This project involves creating EC2 instances with different Linux distributions, configuring password less SSH authentication, and automating instance shutdown operations.

Objectives

• Provisioning EC2 Instances:

Utilize Ansible loops to automate the creation of three EC2 instances — two running Ubuntu and one running Amazon Linux.

This step showcases how Ansible can efficiently manage multiple instances with different configurations.

• Passwordless Authentication Setup:

Implement passwordless SSH authentication between the Ansible control node and the newly created instances.

This step demonstrates the automation of security configurations, facilitating seamless management of the instances.

• Conditional Instance Management:

Automate the shutdown process for the Ubuntu instances using Ansible conditionals.

This task highlights the use of Ansible’s powerful conditional statements to perform operations based on specific criteria.

Prerequisites

1: Ansible Installation

Install Ansible on your control node. Use apt, yum, or pip.

For Linux/macOS:

# Debian/Ubuntu
sudo apt update
sudo apt install ansible

# RedHat/CentOS
sudo yum install ansible

# Using pip
pip install ansible

For Windows:

It’s recommended to use Windows Subsystem for Linux (WSL) for running Ansible.

wsl --install

First, install WSL and then follow the Linux/macOS instructions above within your WSL environment to install Ansible.

2: AWS Account & Credentials

Ensure you have an AWS account and configure AWS credentials using aws configure

Configure IAM user with permissions to manage EC2 instances.

• Log in to the AWS Management Console.
• Navigate to the IAM service.
• Click on “Users” in the sidebar and then “Add user.”
• Enter a user name (e.g., ansible-user).
• Select “Programmatic access” to enable API access.
• Click “Next: Permissions.”

Set Permissions:

1. Click on “Attach policies directly”
2. Search for “EC2FullAccess” and check the box next to it.
3. Click “Next: Tags” (optional) and then “Next: Review”
4. Click “Create user”

3: SSH Key Pair

Generate an SSH key pair for passwordless authentication

ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa

4: Install Required Packages

‘boto3’ : For AWS API interactions.

pip install boto3

AWS Collection: Install Ansible AWS collection.

ansible-galaxy collection install amazon.aws

5: Basic Ansible Knowledge

Understand basic concepts like playbooks and inventory files.

6: Network & Security Group

Ensure security groups allow SSH access and the network setup is correct.

By completing these steps, you’ll be ready to start the Ansible Realtime Project.

7: AWS EC2 AMI IDs

Obtain the AMI IDs for the Ubuntu and Amazon Linux distributions you plan to use. These IDs are required for launching EC2 instances.

• Go to the AWS Management Console and log in.
• Search for “EC2” and select it from the services list.
• Click on Launch Instance.
• In the Choose an Amazon Machine Image (AMI) section, enter “Ubuntu” or “Amazon Linux” in the search bar.
• Find the desired AMI and copy its AMI ID directly from this page. Do not launch the instance.
• Replace placeholder AMI IDs in your playbook with the copied AMI IDs.

Setting up Vault

Ansible Vault is a feature that allows you to keep sensitive data such as passwords or keys in encrypted files.

Here’s how to set up Vault for securely storing your AWS credentials

Create Vault Password:

Generate a strong password to use with Ansible Vault

openssl rand -base64 2048 > vault.pass

This command creates a file named vault.pass containing a 2048-bit random base64 string, which will serve as the password for encrypting and decrypting your Vault files.

Add AWS Credentials to Vault:

Use Ansible Vault to create an encrypted file for storing your AWS credentials.

ansible-vault create group_vars/all/pass.yml --vault-password-file vault.pass

This command opens a new file in your default text editor.

To enter your AWS credentials:

• Press i to enter insert mode.
• Enter your AWS credentials in the following format, replacing YOUR_ACCESS_KEY and YOUR_SECRET_KEY with your actual AWS access key and secret key.

ec2:
  access_key: YOUR_ACCESS_KEY
  secret_key: YOUR_SECRET_KEY

• After entering the credentials, press Esc to exit insert mode.
• Type :wq and press Enter to save and exit the editor.

Your AWS credentials are now stored securely in group_vars/all/pass.yml and can be accessed in your playbooks while remaining encrypted.

Usage Note:

Whenever you run an Ansible playbook that requires these credentials, which we will cover later on you will need to specify the Vault password file to decrypt them:

ansible-playbook YOUR_PLAYBOOK.yml --vault-password-file vault.pass

Replace YOUR_PLAYBOOK.yml with the actual name of your playbook file

In your Ansible playbook or role, you can reference these credentials.

By setting up Ansible Vault, you ensure that sensitive information is protected and can be securely accessed during your automation processes.

Provisioning EC2 Instances

The first task in this project is to provision three EC2 instances using Ansible.

Two instances will run Ubuntu, and one instance will run Amazon Linux.

This task demonstrates how Ansible can efficiently manage multiple instances with different configurations.

Concept of Idempotence

Idempotence is a key principle in Ansible, ensuring that running a task multiple times has the same effect as running it once.

In other words, idempotent tasks will only make changes if necessary, maintaining the desired state without performing redundant actions.

This guarantees that the system’s state remains consistent and predictable, regardless of how many times the playbook is executed.

Steps to Provision EC2 Instances:

1: Install Ansible AWS Collection:

Ensure you have the necessary Ansible collection for AWS installed.

ansible-galaxy collection install amazon.aws

2: Create Ansible Playbook:

Write a playbook to provision the EC2 instances (e.g., ec2_create.yaml).

Ensure that you update the AMI IDs according to your account.

---
- hosts: localhost
  connection: local

  tasks:
  - name: Create EC2 instances
    amazon.aws.ec2_instance:
      name: "{{ item.name }}"
      key_name: "server-Jenkins"
      instance_type: t2.micro
      security_group: default
      region: ap-southeast-2
      aws_access_key: "{{ec2.access_key}}"  # From vault as defined
      aws_secret_key: "{{ec2.secret_key}}"  # From vault as defined      
      network:
        assign_public_ip: true
      image_id: "{{ item.image }}"
      tags:
        environment: "{{ item.name }}"
    loop:
      - { image: "ami-09f5ddaab17f5ff43", name: "manage-node-1" } # Update AMI ID according 
      - { image: "ami-03f0544597f43a91d", name: "manage-node-2" } # to your account
      - { image: "ami-03f0544597f43a91d", name: "manage-node-3" }

Explanation of the Playbook
The provided playbook uses Ansible loops to provision three EC2 instances, avoiding task repetition and demonstrating efficient management of multiple instances with different configurations.

Here’s a breakdown of the playbook:

Hosts and Connection:

• hosts: localhost: The playbook runs on the local machine, which is the Ansible control node.
• connection: local: Specifies a local connection to execute tasks on the control node itself.

Tasks:

Create EC2 instances:

• amazon.aws.ec2_instance: An Ansible module for managing EC2 instances.
• name: "{{ item.name }}": Sets the name for each instance using the item.name variable.
• key_name: "server-Jenkins": Specifies the SSH key pair name for the instances. Replace this with our actual key pair name.
• instance_type: t2.micro: Defines the instance type.
• security_group: default: Assigns the default security group.
• region: ap-southeast-2: Specifies the AWS region.
• aws_access_key and aws_secret_key: Retrieves AWS credentials from the Ansible vault.
• network.assign_public_ip: true: Assigns a public IP to each instance.
• image_id: "{{ item.image }}": Sets the AMI ID for each instance using the item.image variable.
• tags.environment: "{{ item.name }}": Tags each instance with its name for easy identification.
• loop: Iterates over a list of dictionaries, each containing an image and name for the instances.

By using a loop, the playbook avoids repetitive tasks, making it more concise and easier to manage.

Each iteration creates an EC2 instance with the specified AMI and name.

3: Run the Playbook:

To execute the playbook and provision the EC2 instances, use the following command:

ansible-playbook ec2_create.yaml --vault-password-file vault.pass

Replace ec2_create.yaml with the actual name of your playbook file.

This task highlights Ansible’s capability to efficiently manage cloud infrastructure consistently and predictably, leveraging the concept of idempotence to ensure reliable provisioning of EC2 instances.

Go to your console management and ensure that all the 3 instances are created successfully

Passwordless Authentication Setup

The second task in the project involves setting up passwordless SSH authentication between the Ansible control node and the EC2 instances.

This ensures secure and seamless management of the instances without needing to manually enter passwords.

Methods Available:

When setting up passwordless authentication for your EC2 instances, you have two primary methods: using public key authentication or enabling password authentication.

Here’s a brief overview of each approach:

Method 1: Using Public Key Authentication

1. Generate SSH Key Pair:

Create a new SSH key pair on the Ansible control node. This key pair will be used to authenticate with the EC2 instances.

ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa

• -t rsa: Specifies the RSA algorithm for the key.
• -b 2048: Sets the key length to 2048 bits.
• -f ~/.ssh/id_rsa: Defines the file path to save the private key.

This command generates two files:

• ~/.ssh/id_rsa: The private key (keep this secure).
• ~/.ssh/id_rsa.pub: The public key (to be shared with EC2 instances).

2: Copy Public Key to EC2 Instances:

The public key needs to be copied to the ~/.ssh/authorized_keys file on each EC2 instance to allow passwordless SSH access.

Use ssh-copy-id to automate this process.

ssh-copy-id -f "-o IdentityFile <PATH TO PEM FILE>" ubuntu@<INSTANCE-PUBLIC-IP>

• -f: Forces copying of keys (overwrites existing keys).
• "-o IdentityFile ": Specifies the path to your private key file.
• ubuntu@: Username and IP address of the EC2 instance you are accessing.

Note: For instances created via automation tools like Ansible, public key authentication is typically handled by configuring the instance’s key pair during provisioning, so you generally do not need to manually use ssh-copy-id.

Method 2: Using Password Authentication

While less secure than public key authentication, enabling password authentication allows access using a password instead of a key.

This method is generally used when key-based access is not feasible.

1: Update SSH Configuration

Edit the SSH configuration file on your EC2 instance to allow password authentication.

sudo nano /etc/ssh/sshd_config.d/60-cloudimg-settings.conf

Change the PasswordAuthentication line to yes

PasswordAuthentication yes

2: Restart SSH Service

Apply the changes by restarting the SSH service.

sudo systemctl restart ssh

Summary

• Public Key Authentication: Preferred method for secure, passwordless access. Use ssh-copy-id for manual setups, or configure keys through automation tools like Ansible.
• Password Authentication: Enables access using a password, less secure but sometimes necessary. Update SSH configuration and restart the service to apply changes.

Conditional Instance Management

The goal of this task is to automate the shutdown of EC2 instances running Ubuntu using Ansible conditionals.

Ansible’s conditional statements allow you to execute tasks only if certain conditions are met.

You will use Ansible’s when condition to ensure that only instances with Ubuntu OS are shut down.

Concept:

Ansible’s when condition allows you to execute tasks based on the results of gathered facts or other conditions.

This ensures that the playbook performs actions only if certain criteria are met, which in this case means shutting down only Ubuntu instances.

Steps to Implement Conditional Instance Management:

1: Inventory File:

An inventory file is used by Ansible to specify the hosts that playbooks will manage.

Below is the inventory file (inventory.ini) for this project:

[all]
ec2_user@3.25.73.73
ubuntu@3.27.140.245
ubuntu@3.27.185.253

• Replace the example IP addresses with the actual public IP addresses of your EC2 instances.
• Save this configuration in a file named inventory.ini.

2: Define Conditions:

You need to set up conditions that determine when an instance should be managed (e.g., shut down).

These conditions might be based on tags, instance names, or other instance attributes.

3: Create an Ansible Playbook:

Write a playbook that includes tasks to manage the EC2 instances conditionally.

---
- hosts: all
  become: true

  tasks:
    - name: Shutdown ubuntu instances only
      ansible.builtin.command: /sbin/shutdown -t now
      when:
        ansible_facts['os_family'] == "Debian"

Explanation of the Playbook:

1: Hosts and Privilege Escalation:

• hosts: all: Specifies that the playbook will run on all hosts in the inventory.
• become: true: Runs tasks with elevated privileges (e.g., sudo) to allow the shutdown command to be executed.

2: Shutdown Ubuntu Instances Only:

• name: Describes the task, which is to shut down Ubuntu instances.
• ansible.builtin.command: Uses the command module to execute the shutdown command.
• /sbin/shutdown -h now: Issues the shutdown command to halt the system immediately. (-h is used for halt instead of -t, which is not a valid option for shutdown).
• when: The condition that controls when this task should run:
• ansible_facts[‘os_family’] == “Debian”: Ensures the task runs only if the operating system family of the instance is Debian.

Steps to Implement

1: Run the Playbook:

To execute the playbook and shut down Ubuntu instances, use the following command:

ansible-playbook ec2_shutdown.yaml

2: Verify Instance Shutdown:

Check your EC2 instances in the AWS Management Console to confirm that only Ubuntu instances have been shut down as specified.

Summary

By using Ansible’s when condition, you can efficiently manage EC2 instances based on their operating system.

This playbook demonstrates how to automate the shutdown of Ubuntu instances while ensuring other instances are not affected. This conditional approach provides flexibility and control in managing cloud infrastructure.

Conclusion

In this project, we used Ansible to:

1. Provision EC2 Instances: We automated the creation of EC2 instances with different configurations, using loops to manage multiple instances efficiently.
2. Set Up Passwordless Authentication: We configured secure, passwordless SSH access to EC2 instances, making management easier and more secure.
3. Manage Instances Conditionally: We automated the shutdown of Ubuntu instances using Ansible conditionals to perform actions based on specific criteria.

In summary, this project demonstrated how Ansible can simplify and automate cloud infrastructure tasks, ensuring efficient, reliable, and secure management of EC2 instances.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Ansible Automation for AWS EC2: Provisioning and Security was originally published in Towards AWS on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

As cloud-native applications become increasingly complex, managing infrastructure manually can be a daunting task.

Automation tools like Terraform offer a powerful solution, enabling us to define and provision our infrastructure as code.

In this project, I’ll walk you through the process of setting up an Amazon Elastic Kubernetes Service (EKS) cluster with autoscaling self-managed nodes and a custom Virtual Private Cloud (VPC) using Terraform.

We’ll cover:

1. Creating a VPC tailored for our EKS cluster.
2. Configuring security groups to control network traffic.
3. Setting up the EKS cluster for container orchestration.
4. Implementing autoscaling for worker nodes to handle varying workloads.

By the end of this guide, you’ll have a scalable, secure Kubernetes environment on AWS, ready to deploy your applications. Let’s dive in!

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Prerequisites

Before diving into setting up your EKS cluster with Terraform, ensure you have the following prerequisites in place:

1. AWS Account: An active AWS account is essential. If you don’t have one, you can sign up at aws.amazon.com.
2. AWS CLI: The AWS Command Line Interface (CLI) should be installed and configured on your local machine. You can follow the installation guide here. Ensure you’ve run aws configure to set up your credentials.
3. Terraform: Install Terraform on your local machine. The installation instructions can be found here.
4. kubectl: The Kubernetes command-line tool, kubectl, is required to interact with your EKS cluster. Install it following the instructions here.
5. Basic Knowledge of Kubernetes and Terraform: Familiarity with Kubernetes concepts and Terraform’s configuration syntax will help you follow along with the steps in this guide.

Once you have all these prerequisites set up, you’re ready to start creating your infrastructure.

Let’s move on to setting up the project structure.

Project Setup

Let’s set up the project structure and create the necessary Terraform files. Follow these steps to organize your project directory:

1: Create Project Directory: Start by creating a new directory for your Terraform project.

mkdir eks-terraform
cd eks-terraform

2: Create Terraform Files: Inside the eks-terraform directory, create the following files:

• eks-cluster.tf

Define the EKS cluster and node group resources.

• outputs.tf

Define the outputs to display important information after deployment.

• security-group.tf

Define the security groups for the EKS control plane and worker nodes.

• variables.tf

Define the variables used in your Terraform configuration.

• vpc.tf

Define the VPC and subnet resources.

• version.tf

This file specifies the required Terraform version and provider details.

• .ignore

Specify files and directories to ignore in version control.

Summary

Your project directory should now look like this:

eks-terraform/
├── eks-cluster.tf
├── outputs.tf
├── security-group.tf
├── variables.tf
├── vpc.tf
├── version.tf
└── .gitignore

EKS Cluster Creation

In this section, we’ll use Terraform to create the EKS cluster.

We will be leveraging a pre-built Terraform module from the Terraform Registry to simplify the process.

Terraform modules are self-contained packages of Terraform configuration files that are managed together.

They allow you to encapsulate and reuse resource configurations across different parts of your infrastructure.

Using modules has several advantages such as reusability, maintainability, consistency, and abstraction.

Inside the eks-cluster.tf file, paste this code:

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "20.8.4"
  cluster_name    = local.cluster_name
  cluster_version = var.kubernetes_version
  subnet_ids      = module.vpc.private_subnets

  enable_irsa = true

  tags = {
    cluster = "demo"
  }

  vpc_id = module.vpc.vpc_id

  eks_managed_node_group_defaults = {
    ami_type               = "AL2_x86_64"
    instance_types         = ["t3.medium"]
    vpc_security_group_ids = [aws_security_group.all_worker_mgmt.id]
  }

  eks_managed_node_groups = {
    node_group = {
      min_size     = 2
      max_size     = 6
      desired_size = 2
    }
  }
}

Explanation of the Code:

1: Module Source and Version:

source = "terraform-aws-modules/eks/aws"
version = "20.8.4"

This specifies the source of the module and the version to use.

The terraform-aws-modules/eks/aws module from the Terraform Registry simplifies the creation of EKS clusters.

Version 20.8.4 is specified to ensure compatibility and stability.

2: Cluster Name and Version:

cluster_name    = local.cluster_name
cluster_version = var.kubernetes_version

• local.cluster_name: This local variable holds the name of the EKS cluster.
• var.kubernetes_version: This variable specifies the Kubernetes version to be used in the EKS cluster.

3: Subnets and VPC ID:

subnet_ids = module.vpc.private_subnets
vpc_id = module.vpc.vpc_id

• module.vpc.private_subnets: This references the private subnets created in the VPC module.
• module.vpc.vpc_id: This references the VPC ID created in the VPC module.

4: IRSA (IAM Roles for Service Accounts):

enable_irsa = true

This enables IAM Roles for Service Accounts (IRSA), which allows for fine-grained access control on AWS resources for applications running on EKS.

5: Tags:

tags = {
  cluster = "demo"
}

• This adds tags to the EKS cluster for identification and management.

6: Managed Node Group Defaults:

eks_managed_node_group_defaults = {
  ami_type               = "AL2_x86_64"
  instance_types         = ["t3.medium"]
  vpc_security_group_ids = [aws_security_group.all_worker_mgmt.id]
}

This block sets default configurations for EKS managed node groups:

• ami_type: Specifies the Amazon Linux 2 AMI for the nodes.
• instance_types: Specifies the instance type for the nodes (e.g., t3.medium).
• vpc_security_group_ids: Associates the worker nodes with the specified security group.

7: Managed Node Groups:

eks_managed_node_groups = {
  node_group = {
    min_size     = 2
    max_size     = 6
    desired_size = 2
  }
}

This block defines the managed node group configuration:

• min_size: Minimum number of nodes in the node group.
• max_size: Maximum number of nodes in the node group.
• desired_size: Desired number of nodes in the node group.

VPC Creation

A Virtual Private Cloud (VPC) is a virtual network environment within a cloud provider like AWS that allows you to logically isolate and manage your resources securely.

It provides a customizable network configuration, including IP address ranges, subnets, route tables, and security settings, enabling you to create a virtual network infrastructure that closely resembles a traditional data center.

In this step, we create a VPC using Terraform. The VPC is essential for isolating and controlling network resources for your Amazon EKS cluster.

Here’s the provided code for vpc.tf, followed by a detailed explanation of each part:

provider "aws" {
  region = var.aws_region
}

data "aws_availability_zones" "available" {}

locals {
  cluster_name = "abhi-eks-${random_string.suffix.result}"
}

resource "random_string" "suffix" {
  length  = 8
  special = false
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.7.0"

  name                 = "abhi-eks-vpc"
  cidr                 = var.vpc_cidr
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets       = ["10.0.4.0/24", "10.0.5.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

Explanation of the code:

1: AWS Provider:

provider "aws" {
  region = var.aws_region
}

provider “aws”: Specifies that we are using the AWS provider and sets the region using a variable var.aws_region.

2: Data Source for Availability Zones:

data "aws_availability_zones" "available" {}

data “aws_availability_zones” “available”: Fetches the available AWS availability zones in the specified region.

3: Local Values

locals {
  cluster_name = "abhi-eks-${random_string.suffix.result}"
}

locals: Defines local values. Here, it creates a unique cluster_name by appending a random string suffix.

4: Random String Resource

resource "random_string" "suffix" {
  length  = 8
  special = false
}

resource “random_string” “suffix”: Generates a random string of 8 characters, excluding special characters, used to ensure unique resource names.

5: VPC Module

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.7.0"

• source: Specifies the source of the module terraform-aws-modules/vpc/aws, which is located in the Terraform Registry.
• version: Specifies the version of the module to use (5.7.0). This ensures consistency and compatibility with the module's features and functionality.

  name                 = "abhi-eks-vpc"
  cidr                 = var.vpc_cidr
  azs                  = data.aws_availability_zones.available.names

• name: Sets the name of the VPC to "abhi-eks-vpc".
• cidr: Specifies the CIDR block for the VPC, retrieved from the var.vpc_cidr variable. This defines the IP address range for the VPC.
• azs: Specifies the availability zones (AZs) where subnets will be created. This data is fetched using data.aws_availability_zones.available.names, which lists all available AZs in the specified region.

  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets       = ["10.0.4.0/24", "10.0.5.0/24"]

• private_subnets: Defines the CIDR blocks for private subnets within the VPC. Private subnets are typically used for resources that do not need direct access to the internet.
• public_subnets: Defines the CIDR blocks for public subnets within the VPC. Public subnets are used for resources that require internet access, such as load balancers or web servers.

  enable_nat_gateway   = true
  single_nat_gateway   = true

enable_nat_gateway : Enables a NAT gateway for outbound internet traffic from private subnets.

single_nat_gateway : Ensures that all private subnets share a single NAT gateway to minimize costs.

  enable_dns_hostnames = true
  enable_dns_support   = true

enable_dns_hostnames and enable_dns_support: Enable DNS hostnames and DNS resolution within the VPC, allowing instances in the VPC to resolve public DNS hostnames and receive DNS requests from the VPC itself.

6: Tags

tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

tags: Tags are metadata assigned to AWS resources for organization and identification purposes. Here, tags are applied to the VPC itself, indicating it belongs to the Kubernetes cluster named ${local.cluster_name}.

public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

public_subnet_tags and private_subnet_tags: Tags applied specifically to public and private subnets, respectively. These tags help in identifying subnet roles within the Kubernetes cluster environment.

Security Group Creation

Security groups are virtual firewalls that control inbound and outbound traffic for instances. They act as a fundamental component of network security, providing traffic control, stateful filtering, instance-level protection and flexibility & scalability.

• Ingress Rule: Allows inbound traffic from the specified CIDR blocks, which are private IP address ranges commonly used within networks.
• Egress Rule: Permits outbound traffic to any destination (0.0.0.0/0), allowing instances associated with the security group to communicate freely with the internet and other resources.

Here’s the Terraform code from security-groups.tf that creates an AWS security group and associated rules:

resource "aws_security_group" "all_worker_mgmt" {
  name_prefix = "all_worker_management"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group_rule" "all_worker_mgmt_ingress" {
  description       = "allow inbound traffic from eks"
  from_port         = 0
  protocol          = "-1"
  to_port           = 0
  security_group_id = aws_security_group.all_worker_mgmt.id
  type              = "ingress"
  cidr_blocks       = [
    "10.0.0.0/8",
    "172.16.0.0/12",
    "192.168.0.0/16",
  ]
}

resource "aws_security_group_rule" "all_worker_mgmt_egress" {
  description       = "allow outbound traffic to anywhere"
  from_port         = 0
  protocol          = "-1"
  security_group_id = aws_security_group.all_worker_mgmt.id
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

Explanation of the Code

1: aws_security_group Resource

aws_security_group.all_worker_mgmt: Defines an AWS security group named all_worker_management within the VPC specified by module.vpc.vpc_id. This security group will govern inbound and outbound traffic rules for associated instances.

2: aws_security_group_rule Resources

• aws_security_group_rule.all_worker_mgmt_ingress: Specifies an ingress (inbound) rule for the all_worker_mgmt security group. It allows traffic from specified CIDR blocks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) on any protocol (-1) and port (0).
• aws_security_group_rule.all_worker_mgmt_egress: Defines an egress (outbound) rule for the all_worker_mgmt security group. It permits all outbound traffic (0.0.0.0/0) on any protocol (-1) and port (0).

These security group rules ensure that instances within the associated VPC can communicate securely based on specified network traffic requirements.

Configure variables.tf

The variables.tf file defines the input variables that can be used to customize the Terraform configuration.

Here’s the provided code with a brief explanation of each variable:

variable "kubernetes_version" {
  default     = 1.27
  description = "Kubernetes version"
}

variable "vpc_cidr" {
  default     = "10.0.0.0/16"
  description = "Default CIDR range of the VPC"
}

variable "aws_region" {
  default     = "us-west-1"
  description = "AWS region"
}

Explanation of the Variables:

1:kubernetes_version

• Purpose: Specifies the version of Kubernetes to use for the EKS cluster.
• Default Value: 1.27
• Description: This variable allows you to specify which version of Kubernetes will be deployed in the EKS cluster. Keeping Kubernetes versions up-to-date ensures compatibility with the latest features and security patches.

2: vpc_cidr

• Purpose: Defines the CIDR (Classless Inter-Domain Routing) block for the VPC.
• Default Value: 10.0.0.0/16
• Description: This variable sets the IP address range for the VPC. The 10.0.0.0/16 range provides 65,536 IP addresses, which is typically sufficient for most use cases. This range can be adjusted based on your network requirements

3: aws_region

• Purpose: Specifies the AWS region where the infrastructure will be deployed.
• Default Value: us-west-1
• Description: This variable determines the AWS region for deploying your resources. Regions are geographic locations that allow you to place resources closer to your users or meet specific legal and regulatory requirements.

By defining these variables, you can easily modify your configuration without changing the actual Terraform code, promoting reusability and flexibility.

Configure outputs.tf

The outputs.tf file defines the outputs of your Terraform configuration, which are useful for obtaining information about the created resources. Here's the provided code with a brief explanation of each output:

output "cluster_id" {
  description = "EKS cluster ID."
  value       = module.eks.cluster_id
}

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane."
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.aws_region
}

output "oidc_provider_arn" {
  value = module.eks.oidc_provider_arn
}

output "zz_update_kubeconfig_command" {
  value = "aws eks update-kubeconfig --name " + module.eks.cluster_id
  value = format("%s %s %s %s", "aws eks update-kubeconfig --name", module.eks.cluster_id, "--region", var.aws_region)
}

Explanation of Outputs

1. cluster_id

• Description: The ID of the EKS cluster.
• Value: Retrieved from module.eks.cluster_id.
• Purpose: This output provides the unique identifier for the EKS cluster, which is essential for managing and referencing the cluster.

2:cluster_endpoint

• Description: The endpoint for the EKS control plane.
• Value: Retrieved from module.eks.cluster_endpoint.
• Purpose: This output provides the URL endpoint to interact with the EKS control plane. It’s necessary for configuring kubectl and other Kubernetes tools to communicate with the cluster.

3: cluster_security_group_id

• Description: The security group IDs attached to the cluster control plane.
• Value: Retrieved from module.eks.cluster_security_group_id.
• Purpose: This output provides the security group IDs, which are crucial for managing network access to the EKS control plane.

4: region

• Description: The AWS region where the infrastructure is deployed.
• Value: Retrieved from var.aws_region.
• Purpose: This output confirms the AWS region used for the deployment, which can be useful for cross-referencing and managing resources in a multi-region setup.

5: oidc_provider_arn

• Description: The ARN of the OIDC provider associated with the EKS cluster.
• Value: Retrieved from module.eks.oidc_provider_arn.
• Purpose: This output provides the Amazon Resource Name (ARN) for the OIDC provider, used for integrating IAM roles with Kubernetes service accounts for fine-grained access control.

6: zz_update_kubeconfig_command

• Description: A command to update the kubeconfig file with the EKS cluster configuration.
• Value: Constructs the command using the cluster ID and AWS region.
• Purpose: Although commented out, this output provides a ready-to-use command for updating the local kubeconfig file to interact with the EKS cluster. Uncomment and use it for convenience.

By defining these outputs, you make it easier to access and utilize key details of your infrastructure, facilitating management and integration with other tools and services.

Configure versions.tf

The versions.tf file sets the required versions for Terraform and the necessary providers.

This ensures compatibility and stability across your Terraform project. Here’s a breakdown of the provided code:

terraform {
  required_version = ">= 0.12"
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.1.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">=2.7.1"
    }
    aws = {
      source  = "hashicorp/aws"
      version = ">= 3.68.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.1.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1.0"
    }
    cloudinit = {
      source  = "hashicorp/cloudinit"
      version = "~> 2.2.0"
    }
  }
}

Explanation of the version.tf

1: required_version

• Purpose: Specify the minimum version of Terraform required to run the configuration.
• Value: >= 0.12 ensures compatibility with Terraform version 0.12 and newer. Terraform 0.12 introduced many new features and improvements, so this ensures those features are available.

2: required_providers

• Purpose: Defines the required providers and their versions for the configuration.

Providers:

random:

• Source: hashicorp/random
• Version: ~> 3.1.0 (compatible with 3.x versions, starting from 3.1.0)

kubernetes:

• Source: hashicorp/kubernetes
• Version: >= 2.7.1 (version 2.7.1 or newer)

aws:

• Source: hashicorp/aws
• Version: >= 3.68.0 (version 3.68.0 or newer)

local:

• Source: hashicorp/local
• Version: ~> 2.1.0 (compatible with 2.x versions, starting from 2.1.0)

null:

• Source: hashicorp/null
• Version: ~> 3.1.0 (compatible with 3.x versions, starting from 3.1.0)

cloudinit:

• Source: hashicorp/cloudinit
• Version: ~> 2.2.0 (compatible with 2.x versions, starting from 2.2.0)

Backend Configuration (Optional)

If you are using a remote backend (e.g., S3 for state files), you should configure it in a backend.tf file.

A remote backend in Terraform is a way to store the Terraform state file remotely, rather than locally on your machine.

This approach has several advantages, especially when working in a team or when managing large-scale infrastructure.

Benefits of Using a Remote Backend

State Management:

• Centralized State: Keeps the Terraform state file in a central location, enabling multiple team members to work on the same infrastructure without conflicts.
• State Locking: Prevents simultaneous updates to the state file, reducing the risk of corruption. Supports features like AWS S3 with DynamoDB for state locking.

Collaboration:

• Team Collaboration: Allows team members to share and manage infrastructure state efficiently, with everyone accessing the latest state.
• Versioning and History: Maintains a history of state changes, allowing rollbacks to previous states if needed.

Security:

• Secure Storage: Stores state files securely in solutions like AWS S3, with encryption and access controls.
• Backup: Provides reliable backup for your state files.

Scalability:

• Scalable Management: Makes it easier to manage larger and more complex infrastructures by ensuring state consistency and reliability.

Inside the backend.tf file paste this code:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "path/to/my/key"
    region         = "us-west-1"
    dynamodb_table = "my-lock-table"
    encrypt        = true
  }
}

• bucket: The name of the S3 bucket where the state file will be stored.
• key: The path within the bucket to the state file.
• region: The AWS region where the S3 bucket is located.
• dynamodb_table: The DynamoDB table used for state locking.
• encrypt: Ensures that the state file is encrypted in S3.

DynamoDB Table Configuration (Only if you have implemented Backend Configuration).

Amazon DynamoDB is a fully managed NoSQL database service provided by AWS, designed to deliver low-latency performance at any scale.

In addition to its typical use cases, a DynamoDB table is often utilized in AWS environments as part of Terraform’s remote backend configuration.

Specifically, it plays a critical role in state locking, ensuring that only one Terraform process can modify the state file at any given time.

This mechanism prevents concurrent modifications and maintains the integrity of infrastructure deployments managed by Terraform.

Paste this code inside the dynamodb.tf file:

resource "aws_dynamodb_table" "terraform_lock" {
  name         = "my-lock-table"  // Match the name used in backend.tf
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  tags = {
    Name = "my-lock-table"  // Tag the DynamoDB table for identification
  }
}

Explanation of the DynamoDB Table

• name = "my-lock-table": Specifies the name of the DynamoDB table as "my-lock-table", matching the name used in backend.tf for dynamodb_table.
• billing_mode = "PAY_PER_REQUEST": Sets the billing mode to "PAY_PER_REQUEST", which is suitable for on-demand usage.
• hash_key = "LockID": Defines "LockID" as the hash key attribute for the DynamoDB table.
• attribute {}: Defines the attribute "LockID" as a string ("S" type) to be used as the hash key.
• tags = { Name = "my-lock-table" }: Tags the DynamoDB table with a name for identification purposes, ensuring clarity in your AWS resources.

Default State Management in Terraform

When using Terraform, managing state files is crucial for tracking the state of your infrastructure resources and ensuring consistency across deployments. Terraform offers flexibility in how state files are managed:

Remote Backend and DynamoDB Configuration

If you configure a remote backend, such as AWS S3 with DynamoDB for state locking:

• State Locking: Ensures that only one Terraform process can modify the state file at a time, preventing concurrent modifications and maintaining infrastructure integrity.
• High Availability: Provides a centralized location for storing state files, accessible to team members for collaborative infrastructure management.

Default State Management

If you do not configure a remote backend and DynamoDB table:

• Local State Files: Terraform automatically manages state files locally in a file named terraform.tfstate.
• Single User Operations: Suitable for individual use or small teams where concurrent modifications are not a concern.
• Limitations: May lead to conflicts and state file corruption in collaborative environments or complex infrastructure setups.

Initialize and Validate Configuration

Before applying your Terraform configuration to provision or modify infrastructure, it’s essential to initialize and validate your Terraform configuration files.

This step ensures that your environment is properly set up and that the configuration is syntactically correct and ready for deployment.

1. Initialize Terraform

To initialize Terraform, you run the following command in your Terraform project directory:

terraform init

Explanation:

• Initialization: Downloads the necessary providers and modules specified in your configuration files (provider.tf, versions.tf), setting up your environment for Terraform operations.
• Backend Configuration: If using a remote backend (backend.tf), initializes connection settings like AWS S3 bucket and DynamoDB table for state storage and locking.

2. Validate Configuration

After initialization, validate your Terraform configuration files using:

terraform validate

Explanation:

• Validation: Checks the syntax and structure of your Terraform configuration files (*.tf files) to ensure they are correctly formatted and compliant with Terraform's expected syntax.
• Errors and Warnings: Identifies any errors or warnings in your configuration that could prevent successful deployment.

Finalize Configuration

After initializing and validating your Terraform configuration, the typical next steps involve planning, applying changes, and managing your infrastructure.

1: Plan Changes

terraform plan

• Purpose: Generates an execution plan showing what Terraform will do when you apply your configuration.
• Review: Review the plan to understand the actions Terraform will take, such as creating, modifying, or deleting resources.
• Validation: Ensures your configuration is correct and aligns with your intentions before making changes to your infrastructure.

2: Apply Changes

terraform apply

• Execution: Applies the changes defined in your configuration to your infrastructure.
• Interactive: Confirms changes before applying unless automated with -auto-approve.
• State Management: Updates the Terraform state file (terraform.tfstate or remote backend) to reflect the current state of deployed resources.

3: Review Outputs

terraform output

• Outputs: Displays any outputs defined in output.tf, such as resource IDs, endpoints, or other relevant information.
• Verification: Use outputs to verify successful deployment and gather necessary information for further operations or integrations.

4: Manage State

terraform state

• State Management: Provides commands to manage Terraform state, such as listing resources, moving states, or deleting resources from the state.
• Maintenance: Useful for troubleshooting, state file maintenance, or recovering from errors.

Managing Terraform Configurations with .gitignore

When using Terraform to manage infrastructure as code, it’s essential to properly manage your project’s version control.

One critical aspect of this is the .gitignore file, which helps ensure that sensitive or unnecessary files are not included in your Git repository.

Importance of .gitignore

• Sensitive Information: Excludes sensitive data such as credentials, API keys, and state files (terraform.tfstate or .terraform/) from being tracked by version control.
• Temporary Files: Ignores temporary files, logs, and artifacts generated during Terraform operations, reducing repository clutter.
• Enhanced Security: Protects sensitive information from being exposed inadvertently in version control history or during collaboration.

Paste this code inside the .gitignore file:

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

Explanation of .gitignore:

1: Local .terraform directories

**/.terraform/*

Excludes any .terraform directories and their contents from being committed to Git. These directories contain local Terraform plugins and modules.

2: .tfstate files

*.tfstate
*.tfstate.*

Ignores Terraform state files (*.tfstate) and any backup or temporary state files (*.tfstate.*) that may be generated during Terraform operations.

3: Crash log files

crash.log
crash.*.log

Excludes crash log files (crash.log and crash.*.log) from being tracked. These files are generated in case of Terraform command crashes or errors.

4: .tfvars files

*.tfvars
*.tfvars.json

Ignores Terraform variable files (*.tfvars and *.tfvars.json) which may contain sensitive data such as passwords, API keys, or other secrets.

5: Override files

override.tf
override.tf.json
*_override.tf
*_override.tf.json

Excludes override files (override.tf, override.tf.json, *_override.tf, *_override.tf.json) typically used to locally override Terraform configurations.

6: Include override files in version control

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

Provides a comment showing how to include specific override files (!example_override.tf) in version control if needed.

7: Include tfplan files

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

Comments suggesting to include tfplan files if you want to ignore the plan output generated by terraform plan -out=tfplan.

8: CLI configuration files

.terraformrc
terraform.rc

Ignores CLI configuration files (.terraformrc and terraform.rc) which may contain local or user-specific Terraform configurations.

Validate the resources creation via AWS Management Console (UI)

1. Login to AWS Console

• Open your web browser and navigate to the AWS Management Console.
• Sign in using your AWS account credentials.

2. Navigate to Each Resource Type

EC2 Instances:

• Services: Click on “Services” in the top-left corner.
• Compute: Select “EC2” under the Compute section.
• Instances: Verify that EC2 instances created by Terraform are listed under “Instances”.
• Details: Click on an instance to view its details, including instance type, status, security groups, and tags.

VPC (Virtual Private Cloud):

• Networking & Content Delivery: Click on “Services” and then “VPC”.
• Your VPCs: Ensure that the VPC created by Terraform is listed under “Your VPCs”.
• Subnets: Verify that the subnets (public and private) associated with your VPC are correctly configured.

Security Groups:

• Networking & Content Delivery: Click on “Services” and then “EC2”.
• Security Groups: Navigate to “Security Groups” under the “Network & Security” section.
• Verify: Check that the security groups created by Terraform are listed and configured with the expected inbound and outbound rules.

EKS Cluster:

• Compute: Select “Elastic Kubernetes Service” under Compute.
• Clusters: Ensure that the EKS cluster created by Terraform is listed under “Clusters”.
• Details: Click on the cluster name to review its details, including Kubernetes version, endpoint, node groups, and associated resources.

IAM Roles and Policies:

• Security, Identity, & Compliance: Click on “Services” and then “IAM”.
• Roles/Policies: Verify that IAM roles and policies created by Terraform are present under “Roles” and “Policies” respectively.
• Details: Click on a role or policy to inspect its permissions, trust relationships, and usage.

3. Review and Confirm Configuration

• Settings: For each resource type, review its configuration settings to ensure they match what you defined in your Terraform scripts (*.tf files).
• Tags: Check that tags applied to resources are correctly assigned and labeled.

4. Functional Testing (Optional)

• Interact: If applicable, interact with deployed resources to validate functionality, such as accessing EC2 instances via SSH, accessing Kubernetes cluster via kubectl, or testing network connectivity.

Clean Up Resources

1: Identify Resources

• List all resources provisioned by Terraform, including EC2 instances, VPCs, EKS clusters, IAM roles, and any other resources managed through your Terraform scripts.

2: Terminate EC2 Instances

• AWS Management Console: Navigate to “Services” > “EC2” > “Instances”.
• Select instances created by Terraform, verify their IDs, and terminate them.
• CLI: Use aws ec2 terminate-instances --instance-ids <instance-id> to terminate instances via the AWS CLI if preferred.

3: Delete EKS Cluster

• AWS Management Console: Navigate to “Services” > “Elastic Kubernetes Service”.
• Select the EKS cluster created by Terraform, and choose “Delete” to remove it.
• Ensure to delete associated resources such as node groups and any add-ons if applicable.

4: Remove VPC and Subnets

• AWS Management Console: Navigate to “Services” > “VPC”.
• Delete the VPC and associated subnets created by Terraform.
• Remove any remaining resources like route tables, internet gateways, or NAT gateways associated with the VPC.

5: Delete IAM Roles and Policies

• AWS Management Console: Navigate to “Services” > “IAM”.
• Remove IAM roles and policies created by Terraform that are no longer needed.
• Ensure to detach policies from roles before deletion if they are attached.

6: Clean Up Security Groups

• AWS Management Console: Navigate to “Services” > “EC2” > “Security Groups”.
• Delete any security groups created by Terraform that are no longer in use.

7: Verify Deletion

• Double-check the AWS Management Console to ensure all resources have been deleted successfully.
• Use the AWS CLI or SDKs to automate resource deletion if handling a large number of resources.

8: State File Management

• Remove local Terraform state files (terraform.tfstate and terraform.tfstate.backup) if you're not using a remote backend for state management.

9: Review Billing

• Monitor your AWS billing dashboard to ensure that resources have been successfully terminated and to verify that no unexpected charges occur.

Conclusion

This project showcased the effective use of Terraform for automating AWS infrastructure deployments.

By leveraging infrastructure as code principles, we achieved scalability, reliability, and cost efficiency.

Through automated provisioning of resources like EKS clusters, VPCs, and security groups, we ensured consistent deployments while maintaining robust security practices.

Moving forward, Terraform remains pivotal in enabling agile and efficient management of cloud infrastructure, supporting our goal of streamlined operations and optimized resource utilization.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Introduction

In today’s cloud-centric world, a secure and scalable network infrastructure is essential. Amazon Web Services (AWS) offers Virtual Private Cloud (VPC), enabling users to create isolated networks tailored to their needs.

This project will guide you through setting up a comprehensive VPC environment on AWS, covering key components such as subnets, gateways, load balancers, and security measures.

By the end of this tutorial, you’ll learn to build and manage a robust VPC architecture, ensuring your cloud resources are secure and highly available.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Project Overview

This project demonstrates how to create a Virtual Private Cloud (VPC) suitable for hosting servers in a production environment.

To enhance resilience, servers are deployed across two Availability Zones using an Auto Scaling Group and an Application Load Balancer.

For additional security measures, servers are placed in private subnets, while a public subnet allows inbound traffic through the Application Load Balancer.

Servers can access the internet via a NAT Gateway, which is deployed in both Availability Zones for improved resilience.

An Internet Gateway facilitates outbound Internet access for resources within the VPC.

A bastion host is used alongside Security Groups, EC2 instances, and Target Groups to securely manage access to resources within the VPC.

NOTE: Most of these step can also be automated using Terraform, but for the sake of simplicity, we will do it manually here.

Tools and Technologies Used

In this project, we leverage several key tools and technologies to build a robust Virtual Private Cloud (VPC) environment on AWS:

1: Amazon VPC (Virtual Private Cloud): Provides a logically isolated network to deploy AWS resources.

2: Availability Zones: Geographic regions with independent infrastructure to enhance fault tolerance and stability.

3: Subnets:

• Public Subnet: Allows resources to be accessible from the internet.
• Private Subnet: Provides enhanced security by isolating resources from direct internet access.

4: NAT Gateway: Enables instances in private subnets to access the internet while maintaining security.

5: Internet Gateway: Facilitates outbound internet access for resources within the VPC.

6: Application Load Balancer (ALB): Distributes incoming traffic across multiple targets such as EC2 instances, enhancing fault tolerance and scalability.

7: Security Groups: Acts as virtual firewalls to control inbound and outbound traffic at the instance level.

8: EC2 Instances: Virtual servers running in the AWS cloud that provide compute capacity.

9: Auto Scaling Group: Ensures the availability and scalability of EC2 instances by automatically adjusting capacity based on traffic demands.

10: Bastion Host: Acts as a secure gateway to access and manage instances in private subnets.

11: Target Groups: Directs Application Load Balancer traffic to registered targets, such as EC2 instances.

These tools and technologies collectively enable the creation of a resilient, scalable, and secure VPC environment suitable for hosting production applications on AWS.

Creating VPC

Creating a Virtual Private Cloud (VPC) is the first step in setting up a secure and isolated network environment within AWS.

A VPC allows you to launch AWS resources, such as EC2 instances, in a logically isolated virtual network that you define.

This section will walk through the process of creating a VPC, which will serve as the foundation for our entire infrastructure.

This VPC will include public and private subnets across multiple availability zones to ensure high availability and security for our applications.

Note: This step can also be automated using Terraform, but for the sake of simplicity, we will do it manually here.

1: Sign in to AWS Console:

• Go to AWS Management Console and sign in using your credentials.
• After signing in, you’ll land on the AWS Management Console dashboard.

2: Navigate to VPC Dashboard:

• Search for “VPC” in the search bar and select it from the results.

3: Start VPC Wizard:

• In the VPC dashboard, click on the “Create VPC” button to start the VPC creation process.

4: Choose a VPC Configuration:

• AWS provides several VPC configuration options. Let’s choose the “VPC and more” option.
• Add a name tag to identify your VPC.

5: Configure VPC Details:

• Enter the IPv4 CIDR block for your VPC. This defines the range of IP addresses that can be used within your VPC.
• Optionally, you can also specify an IPv6 CIDR block for your VPC.
• Leave other settings as default or configure as needed based on your requirements.

6: Configure Subnets:

• For Number of Availability Zones, choose 2, so that you can launch instances in multiple Availability Zones to improve resiliency.
• For the Number of public subnets, choose 2.
• For the Number of private subnets, choose 2.
• For NAT gateways, choose 1 per AZ to improve resiliency.
• For VPC endpoints, if your instances must access an S3 bucket, keep the S3 Gateway default. Otherwise, instances in your private subnet can’t access Amazon S3. There is no cost for this option, so you can keep the default if you might use an S3 bucket in the future. If you choose None, you can always add a gateway VPC endpoint later on.

• The wizard will automatically create route tables for your subnets.
• Ensure that routes are correctly configured, such as routing internet-bound traffic through an Internet Gateway for public subnets.
• For DNS options, clear Enable DNS hostnames.

7: Review and Create:

• Review your configurations details you’ve chosen for your VPC.
• Choose “Create VPC”.

• AWS will now create your VPC based on the configuration you’ve specified.
• This process usually completes within a few minutes please be patient.

8: Verification:

• After the VPC is created, verify its details in the VPC dashboard.
• Ensure all components (VPC, subnets, route tables) are correctly configured according to your requirements.

Creating Auto Scaling Group

Auto Scaling in AWS empowers applications to automatically adjust compute capacity based on demand, optimizing performance and costs.

This guide explores the straightforward steps to create an Auto Scaling group using the AWS Management Console.

By setting up Auto Scaling, you can ensure your infrastructure scales efficiently to meet fluctuating workload requirements, enhancing the resilience and cost-effectiveness of your AWS deployments.

1: Sign in to AWS Console:

• Go to AWS Management Console and sign in using your credentials.

2: Navigate to Auto Scaling:

• Search for “Auto Scaling” in the search bar and select it from the results.

3: Create an Auto Scaling Group:

• Click on the “Create Auto Scaling group” button to start creating a new Auto Scaling group.

4: Choose Launch Template or Launch Configuration:

• Click on “Create Launch Template” if you don’t have an existing one, or choose an existing Launch Template from the dropdown.

5: Create a Launch Template (if applicable):

If creating a new Launch Template, provide the following details:

• Template name: Enter a name and description for your Template.

• AMI (Amazon Machine Image): Select an AMI for your EC2 instances.

• Instance type: Choose the instance type for your EC2 instances (e.g., t2.micro).
• Key pair (if needed): Specify a key pair for SSH access to instances.

• Configure other instance details such as networking, as per your requirements.
• Ensure you select the previously created VPC, we will be using this throughout the project.

5: Configure Security Group

• Create a Security Group.

• Ensure that the security group allows SSH access and custom TCP port 8000.

• Configure other instance details such as storage as per your requirements.
• Click on “Create launch template” once all details are configured.

• Verify that the Launch template was created successfully.

6: Configure Auto Scaling Group:

After selecting or creating a Launch Template, configure the Auto Scaling group settings:

• Group name: Provide a name for your Auto Scaling group.

• Network: Choose the VPC and subnet where instances will be launched. Select the two private subnets and for the VPC select the one we created previously.

• Group size: Define the initial number of instances (2).
• Scaling Limits: Choose the minimum(1) and maximum(4) desired capacity.

7: Configure Scaling Policies (if applicable):

• Configure scaling policies (optional).
• Click “Next”.

• Add Notifications (optional).

8: Review and Create:

• Review all the configuration details you’ve chosen for your Auto Scaling group.
• Double-check the Launch Template, scaling policies, networking settings, and any other configurations.
• Click on “Create Auto Scaling group” to initiate the creation process.

9: Wait for Auto Scaling Group Creation:

• AWS will now create your Auto Scaling group based on the configuration you’ve specified.
• This process may take a few moments to complete please be patient.

10: Verification:

• After the Auto Scaling group is created, verify its details in the Auto Scaling dashboard.

• Go to the EC2 dashboard and ensure instances are launched according to scaling policies ( 2 instances in our case) and that they are integrated in two availability zones.

Creating Bastion Host

In AWS environments, maintaining secure access to private instances within a Virtual Private Cloud (VPC) is crucial for effective management and troubleshooting.

A bastion host, or jump server, serves as a secure gateway allowing authorized users to securely connect to instances in private subnets.

• Go to AWS Management Console and sign in using your credentials.
• In the AWS Management Console, search for “EC2” in the search bar.
• Click on “Launch Instance” to start the instance creation process.
• Select an appropriate AMI for your bastion host.
• Select an instance type that meets your performance and capacity requirements. A t2.micro instance type is often sufficient for a bastion host.
• Select an existing key pair or create a new key pair to securely access your instance.
• Download the key pair file (.pem) and keep it safe. You cannot download it again after it is created. It will be required to connect to your bastion host via SSH.
• Choose the VPC you created earlier for your project.
• Select a public subnet within your VPC to ensure the bastion host has a public IP address and internet access.
• Auto-assign Public IP: Enable this option to assign a public IP address to the bastion host.
• Configure storage settings as needed. The default settings are typically sufficient for a bastion host.
• Add tags to the instance for easy identification and organization. For example, you can add a tag with Key “Name” and Value “Bastion Host”.
• Create a new security group or use an existing one that allows inbound SSH (port 22) access from your IP address (or IP range) for secure SSH connections.
• Optionally, configure additional rules for any other necessary access (e.g., HTTPS for web-based management).
• Review all the instance details you’ve configured.
• Click on “Launch Instances” to start the bastion host instance.

Access the Bastion Host

• Once the instance state changes to “running”, note down the public IP address of the bastion host.
• Use SSH and the key pair file (.pem) to securely connect to the bastion host from your local machine or terminal:

ssh -i /path/to/your-key.pem ec2-user@bastion-host-public-ip

• Replace /path/to/your-key.pem with the path to your downloaded key pair file, and bastion-host-public-ip with the actual public IP address of your bastion host.
• Verify that you can successfully connect to the bastion host using SSH.

Alternatively for Windows users if you have MobaXterm you have use it to ssh into the bastion host like we have been doing with ec2 instances on our other project demos.

Securely Copying PEM File to Bastion Host

To securely access EC2 instances via SSH through a bastion host, you first need to transfer your SSH key (.pem file) to the bastion host.

This is particularly crucial when the EC2 instance resides in a private subnet without direct internet access.

In such cases, the bastion host serves as a secure gateway, allowing controlled SSH access from your local machine to the private instances within the AWS Virtual Private Cloud (VPC).

Below are the steps for securely copying your SSH key using the scp command.

• Ensure you have the .pem file (your private key) that you downloaded when you created the EC2 instance.
• Before transferring the file, make sure its permissions are set correctly to prevent unauthorized access:

chmod 400 /path/to/your-key.pem

• Open a terminal or command prompt on your local machine.
• Use the scp command to securely copy the .pem file to the bastion host.

scp -i /path/to/your-key.pem /path/to/your-key.pem ec2-user@bastion-host-public-ip:/home/ec2-user/

• Replace /path/to/your-key.pem with the actual path to your .pem file, and bastion-host-public-ip with the public IP address of your bastion host.
• Use /home/ubuntu/ instead of /home/ec2-user/ if you choose an Ubuntu AMI.

Example Sample:

scp -i ~/.ssh/my-key.pem ~/.ssh/my-key.pem ec2-user@123.456.789.0:/home/ec2-user/

• This command securely copies the .pem file to the home directory (/home/ec2-user/) of the ec2-user on the bastion host.
• If you’re prompted to enter a password, use the password associated with the SSH key pair for authentication.

• After the transfer completes, verify that the .pem file is in the correct directory on the bastion host:

ls /home/ec2-user/

• If you used an Ubuntu AMI, use:

ls /home/ubuntu/

SSH into EC2 Instance via Bastion Host

Once the .pem file is securely copied to the bastion host, you can proceed to SSH into your EC2 instance using the bastion host as a gateway:

1: SSH into Bastion Host:

We had done this in the above steps

ssh -i /path/to/your-key.pem ec2-user@bastion-host-public-ip

2: SSH into EC2 Instance from Bastion Host:

Once connected to the bastion host, SSH into the desired EC2 instance within your VPC using its private IP address and the .pem file:

ssh -i /home/ec2-user/your-key.pem ec2-user@private-ip-of-ec2-instance

Example for Ubuntu:

ssh -i /home/ubuntu/your-key.pem ubuntu@private-ip-of-ec2-instance

Creating a Simple HTML Application

After connecting to the EC2 instance via the Bastion host, if you don’t have an application to deploy, you can create a simple HTML application for demonstration purposes.

• On your EC2 instance, use vim to create and edit an HTML file:

vim index.html

• Press i to enter insert mode in vim.
• Paste the following HTML content:

<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<body>

<h1>My First AWS Project to demonstrate apps in private subnet</h1>

</body>
</html>

• Press Esc to exit insert mode.
• Type :wq and press Enter to save the file and exit vim.
• You can list the directory contents to confirm that index.html is created:

ls

Note: We intentionally deployed the application on only one instance to check if the load balancer will distribute 50% of the traffic to one instance, which will receive a response, and 50% to another instance, which will not receive a response.

This concept will be explained in the next part where we will create an Application Load Balancer.

Viewing Your HTML Application

To view your HTML application, you need to serve it through a web server. For simplicity, you can use Python’s built-in HTTP server.

1: Install Python (if not already installed):

sudo yum install python3 -y

For Ubuntu, use:

sudo apt-get install python3 -y

2: Start the HTTP Server:

python3 -m http.server 8000

Creating an Application Load Balancer

An Application Load Balancer (ALB) in AWS distributes incoming application traffic across multiple targets, such as EC2 instances, in different availability zones.

This ensures high availability and fault tolerance for your applications.

1: Sign in to AWS Console:

• Go to the AWS Management Console and sign in using your credentials.

2: Navigate to the EC2 Dashboard:

• In the AWS Management Console, search for “EC2” in the search bar.

3: Access Load Balancers:

• In the left-hand menu, under “Load Balancing,” click on “Load Balancers.”

4: Create Load Balancer:

• Click on the “Create Load Balancer” button.
• Choose “Application Load Balancer” and click on “Create.”

5: Configure Load Balancer Basics:

• Enter a name for your load balancer.
• Choose “internet-facing” to make the load balancer accessible over the internet.
• Choose “ipv4”.

6: Configure Network and Listeners:

• Select the VPC you created earlier.
• Select at least two availability zones and their corresponding public subnets.
• By default, a listener on port 80 (HTTP) is created. You can add more listeners if needed.

7: Configure Security Groups:

• Choose an existing security group or create a new one that allows inbound HTTP traffic (port 80).
• Ensure this security group also allows any other required inbound traffic.

8: Configure Routing:

• Create a new target group.
• Enter a name for your target group.
• Choose “Instance” to register EC2 instances.
• Select “HTTP” enter port 80
• Configure the health check settings (default settings are usually sufficient).

9: Register Targets:

• Select the EC2 instances you want to include in this target group.
• Click “Include as pending below”.

• Click “Create target group”.

10: Review and Create:

• Review all your settings for the Application Load Balancer.
• Click “Create” to launch the load balancer.

11: Verify Load Balancer:

• After the ALB is created, you will see it listed in the Load Balancers section.
• Note the DNS name of the ALB. This is the endpoint you will use to access your application.

Verification

1: Access the Load Balancer DNS:

• Open a web browser and navigate to the DNS name of the load balancer.

• You should see your application, verifying that the ALB is correctly distributing traffic.

2: Check Traffic Distribution:

• As noted earlier, only one instance has the application deployed.
• Access the ALB DNS multiple times to observe if 50% of the requests go to the instance with the application (receiving a response) and 50% go to the instance without the application (not receiving a response).

Clean up Resources

After completing your AWS VPC architecture project, it’s essential to clean up resources to prevent incurring unnecessary charges.

AWS bills for resources such as EC2 instances, NAT Gateways, and Load Balancers, even if they are not actively used.

Follow these steps to delete resources effectively:

1: Terminate EC2 Instances:

• Navigate to the EC2 Dashboard.
• Select the instances you created for the project.
• Click on “Instance State” and choose “Terminate.”
• Confirm the termination.

2: Delete Auto Scaling Groups:

• Go to the Auto Scaling Dashboard.
• Select the Auto Scaling group you created.
• Click on “Actions” and choose “Delete.”
• Confirm the deletion.

3: Remove Load Balancers:

• In the EC2 Dashboard, click on “Load Balancers.”
• Select the Application Load Balancer you created.
• Click on “Actions” and choose “Delete.”
• Confirm the deletion.

4: Delete NAT Gateways:

• Navigate to the VPC Dashboard.
• Click on “NAT Gateways.”
• Select the NAT Gateway you created.
• Click on “Actions” and choose “Delete NAT Gateway.”
• Confirm the deletion.
• Ensure to release the Elastic IP associated with the NAT Gateway to avoid charges.

5: Remove Security Groups:

• Go to the VPC Dashboard.
• Click on “Security Groups.”
• Select the security groups created for the project.
• Ensure no instances are using them, then delete the security groups.

6: Delete Subnets and VPC:

• In the VPC Dashboard, click on “Subnets.”
• Select each subnet created and delete them.
• After deleting all subnets, go back to the VPC Dashboard.
• Select the VPC you created and delete it.

7: Remove Internet Gateways:

• Navigate to the VPC Dashboard.
• Click on “Internet Gateways.”
• Detach the Internet Gateway from your VPC.
• Once detached, delete the Internet Gateway.

8: Release Elastic IP Addresses:

• Go to the EC2 Dashboard.
• Click on “Elastic IPs.”
• Select the Elastic IPs you allocated.
• Click on “Actions” and choose “Release Elastic IP Addresses.”

9: Delete Key Pairs:

• In the EC2 Dashboard, click on “Key Pairs.”
• Select the key pairs created for the project.
• Delete the key pairs.

Conclusion

In this project, we successfully created a robust Virtual Private Cloud (VPC) infrastructure suitable for a production environment.

By deploying resources across multiple availability zones, we ensured high availability and resiliency. Key components included public and private subnets, a bastion host for secure access, an auto scaling group for automated scaling of EC2 instances, and an Application Load Balancer to distribute incoming traffic efficiently.

This setup not only enhances security through the use of private subnets and security groups but also ensures that applications remain highly available and scalable.

This foundational architecture is critical for building resilient, secure, and scalable cloud applications.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Introduction

In this guide, we’ll dive into setting up continuous deployment (CD) using AWS CodeDeploy, a powerful managed service for automating application deployments.

Before implementing this project, it’s crucial to complete the continuous integration (CI) phase using AWS services like CodeCommit, CodeBuild, and CodePipeline.

I’ve already covered this essential step in a previous Medium article, which you can find here.

Let’s streamline our deployment process with AWS CodeDeploy and ensure a smooth transition from CI to CD.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Tools and Technologies Used

• AWS CodePipeline: Continuous integration and continuous delivery service that orchestrates your release process.
• AWS CodeDeploy: Managed deployment service that automates software deployments to a variety of computing services such as Amazon EC2 instances, AWS Lambda functions, or on-premises servers.
• Amazon EC2: Scalable virtual servers in the cloud used as deployment targets by AWS CodeDeploy.

These tools enable efficient deployment automation and scalability, ensuring reliable and consistent application releases without the need to manage underlying infrastructure.

Prerequisites

Detailed steps for setting up each prerequisite will be provided in the documentation.

1. AWS Account:

You need an AWS account to access and use AWS services, including AWS CodeDeploy.

2. GitHub Repository:

Have a GitHub repository where your application source code is stored. This repository will trigger automated deployments in AWS CodeDeploy upon code commits.

3. CodeDeploy IAM Role:

Create an IAM role with permissions for AWS CodeDeploy to access your deployment resources.

4. Deployment Target:

Prepare your deployment target, which could be:

• Amazon EC2 instances: Ensure your instances are configured with the necessary IAM roles and permissions. Steps for setting this up will be covered.
• AWS Fargate: Set up your Fargate profiles if deploying to serverless containers.

5. Deployment Package:

Prepare your application code packaged for deployment, typically stored in Amazon S3 or another accessible repository.

6. Deployment Configuration:

Define your deployment configuration in AWS CodeDeploy, including deployment groups and deployment settings.

Setting Up Git Repository

• Fork the GitHub Repository: Fork the GitHub repository I’ve prepared for this demo. You can find it here.

Create an EC2 Instance

Note that this step can also be automated using Terraform, but for the sake of simplicity, we will do it manually here.

Here is a detailed step-by-step guide to creating an EC2 instance:

1. Log into AWS Management Console:

• Navigate to the AWS Management Console.
• Sign in with your AWS credentials.

2. Open the EC2 Dashboard:

• In the AWS Management Console, search for “EC2” in the search bar.
• Select “EC2” from the list of services.

3. Launch Instance:

• Click the “Launch Instance” button on the EC2 Dashboard.

4. Add Tags:

• (Optional) Add tags to your instance for easier identification.

5. Choose an Amazon Machine Image (AMI):

• Select an appropriate AMI for your needs. For this example, you can choose “Ubuntu”.

6. Choose an Instance Type:

• Select an instance type. For most use cases, the “t2.micro” instance type is sufficient and is free tier eligible.

7. Select a Key Pair:

• Select an existing key pair or create a new key pair to securely access your instance.
• Download the key pair file (.pem) and keep it safe. You cannot download it again after it is created.

8. Configure Instance Details:

• Optionally, configure instance details such as network settings, subnets, IAM role, etc. You can leave these settings as default for now.

9. Configure Security Group:

• Create a new security group or select an existing one.
• Ensure that you have rules allowing SSH access (port 22) from your IP address.

10. Add Storage:

• The default storage settings should be sufficient.
• You can adjust the size if needed.

11. Review and Launch:

• Review your instance configuration.
• Click “Launch” to proceed.

Access Your Instance

Once the instance is launched, it will take a few minutes to initialize.

You can then connect to your instance using SSH with the downloaded .pem file.

We are using Mobaxterm to SSH to the EC2 Instance ( For Windows machines).

• Get MobaXterm and install it.
• Open MobaXterm from your desktop or the Start menu.
• Click the “Session” button, then select “SSH”.
• Enter the IP address or hostname of the server and specify username.
• Click on “Advanced SSH settings” tick the “Use private key” box and select the private key pair you used in your instance.
• Click “OK” to connect.

Installing AWS CodeDeploy Agent

1: Install the AWS CodeDeploy Agent:

The AWS CodeDeploy agent is required on your EC2 instance to facilitate communication with the AWS CodeDeploy service.

This agent manages deployments and ensures that your application updates are orchestrated correctly according to your deployment configurations.

For detailed installation instructions, refer to the AWS documentation here.

Ensure you select the documentation relevant to the operating system used on your EC2 instance.

2: Verify the Installation:

After installing the AWS CodeDeploy agent, verify its installation by running the following command on your EC2 instance:

sudo service codedeploy-agent status

Install Docker

Ensure Docker is installed to use Docker commands later in the project.

sudo apt install docker -y

Creating IAM Role and Granting Permissions

An IAM role in AWS defines the permissions and access controls for services like AWS CodeDeploy.

It enables secure management and deployment of applications by granting necessary permissions to EC2 instances, ensuring seamless interaction with AWS services.

1; Create an IAM Role:

• Navigate to the AWS Management Console and go to the IAM service.
• Click on “Roles” in the left-hand menu and then “Create role”.
• Choose the service that will use this role (EC2 in this case) and click “Next: Permissions”.

• In the “Filter policies” search box, type “AWSCodeDeployRole” and select the policy named AWSCodeDeployRole (for AWS managed policies).
• Alternatively, create a custom policy allowing the necessary permissions for AWS CodeDeploy. Click “Next”.

• Optionally, add tags to your role for better organization and click “Next: Review”.

• Provide a name and description for the role and click “Create role”.

2; Attach IAM Role to EC2 Instance:

• Once the role is created, navigate to the EC2 service in the AWS Management Console.
• Select your EC2 instance, click “Actions”, navigate to “Security”, and then “Modify IAM Role”.

• Choose the IAM role created in the previous step and click “Update IAM Role”.

Go to your terminal and restart your code deploy agent service.

sudo service codedeploy-agent restart

Then run the status again to verify it has started.

sudo service codedeploy-agent status

3: Grant Additional Permissions

• Navigate to the above created IAM Role.
• Click on the “Permissions” tab, then click “Add permissions” and then “Attach Policies”.

• In the search box, type “AmazonEC2FullAccess” and select the policy.
• Review the policy and click “Add permissions” to attach it to the role.
• Verify the role’s permissions include both AWSCodeDeployRole and AmazonEC2FullAccess.

Configure AWS Code Deploy Application

Configure AWS CodeDeploy to orchestrate and automate application deployments across your AWS infrastructure.

Follow these steps to set up deployment groups, define deployment settings, and integrate with other AWS services seamlessly.

• Go to the AWS Management Console and select the AWS CodeDeploy service.
• Click on “Create Application” button.
• Provide a name for your application.
• Select the compute platform where your application will be deployed (EC2/On-premises or AWS Lambda).
• Click on “Create application” to finalize and create the application.
• Once created, you will see the new application listed in your CodeDeploy dashboard.

Create Deployment Group in AWS CodeDeploy

Set up a deployment group in AWS CodeDeploy to define where and how your application will be deployed across your AWS infrastructure.

Follow these steps to configure deployment settings, choose deployment targets, and finalize the setup for seamless application deployments.

• Navigate to AWS CodeDeploy Console
• In the CodeDeploy dashboard, click on the name of the application you previously created.
• Click on the “Create deployment group” button.
• Provide a name for your deployment group.

• Configure deployment settings like service role.
• Choose the deployment type (In-place or Blue/green).

• Select the environment configuration (EC2/On-premises or AWS Lambda).
• Mention the key and value of this. (It will be matched with an Instance)

• You can configure advanced settings such as triggers, alarms, and notifications.
• Click on “Create deployment group” to finalize and create the deployment group.
• Once created, the new deployment group will be listed under your application in the CodeDeploy dashboard.

Create a Deployment in AWS CodeDeploy

• Navigate to AWS CodeDeploy Console
• In the CodeDeploy dashboard, click on the name of the application you previously created.
• Click on “Create deployment”.
• Select the deployment group you created earlier.

• Choose the source of your application code. Options include:

S3: Specify the S3 bucket name and object key.

GitHub: Provide the repository name and commit ID.

Bitbucket: Provide the repository name and commit ID.

Local file: Upload a local file as the revision.

• Optionally, provide a description for the deployment.

• Configure additional options (optional).
• Click on “Create deployment” to start the deployment process.

• Monitor the status of your deployment in the CodeDeploy dashboard to ensure it completes successfully.

Integrate AWS CodeDeploy with AWS CodePipeline

Integrating AWS CodeDeploy with AWS CodePipeline automates your deployment process, enabling continuous delivery (CD).

This integration ensures that every change in your source code repository triggers a new deployment, streamlining your workflow and improving deployment efficiency.

Here are the steps to set up this integration:

• Log in to the AWS Management Console.
• Go to the AWS CodePipeline service.
• Find and select the pipeline we set up in the previous project while working on the CI part of this project.
• Find the steps of creating the pipeline here.
• Click on “Edit” in the top-right corner of the pipeline details page.

• Click on “Add stage” at the appropriate point in your pipeline where you want the deployment to occur (typically after the build stage).
• Name the stage (e.g., “Deploy”) then click “Add Stage”.

• Provide a name for this action (e.g., “DeployAction”).
• Select “AWS CodeDeploy”.
• Select the region where your CodeDeploy application is located.

• Select the output artifact from the previous build stage.
• Choose the CodeDeploy application you created.
• Choose the CodeDeploy deployment group you created.
• Click on “Done”.
• Review the changes to ensure that the new deploy stage and action are correctly configured.
• Click on “Save pipeline changes”.

• Trigger the pipeline manually by clicking “Release change” or wait for an automated trigger from a new code commit.
• Monitor the progress of the pipeline in the CodePipeline console. Each stage (Source, Build, Deploy) will show its status.

Clean Up Resources

After completing the deployment, it’s important to clean up resources to avoid unnecessary charges. Ensure you:

1. Terminate EC2 Instances: Stop and terminate the EC2 instances used for deployment.
2. Delete the CodeDeploy Application: Remove the CodeDeploy application and deployment groups.
3. Remove IAM Roles and Policies: Delete any IAM roles and policies created specifically for this project.
4. Clean Up the CodePipeline: If no longer needed, delete the CodePipeline to prevent further triggers and resource usage.

By cleaning up, you maintain a tidy AWS environment and avoid incurring unexpected costs.

Conclusion

In this project, we extended our existing CI pipeline by integrating AWS CodeDeploy to automate the deployment process.

By following the outlined steps, we successfully set up a seamless CI/CD pipeline using AWS services.

This setup ensures efficient, automated deployments, enhancing our workflow and reducing manual intervention.

Feel free to refer to this guide for setting up similar projects, and explore the benefits of continuous delivery using AWS CodeDeploy.

For further information on the CI part of this project, you can refer here.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Introduction

In modern software development, Continuous Integration (CI) plays a crucial role in automating build and test processes, ensuring efficient code integration and quality assurance.

This project explores CI practices using AWS CodeBuild and AWS CodePipeline to automate the build, test, and validation of code changes.

This article focuses exclusively on the CI aspect. For Continuous Deployment (CD), please refer to the documentation here.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Tools and Technologies Used

This project utilizes a range of tools and technologies to streamline Continuous Integration (CI) with AWS CodeBuild and AWS CodePipeline:

• AWS CodeBuild: AWS’s fully managed build service for compiling source code, running tests, and producing software packages.
• AWS CodePipeline: A continuous integration and continuous delivery service that orchestrates workflows for building, testing, and deploying code changes.
• GitHub: A version control repository to store and manage source code, triggering automated builds in AWS CodeBuild upon code commits.
• Docker: A platform for developing, shipping, and running applications inside containers, providing consistency across development, testing, and deployment environments.
• Docker Hub: A cloud-based repository for Docker container images used in the application.
• AWS Systems Manager: Facilitates secure storage and management of Docker Hub credentials using Parameter Store, ensuring sensitive information is stored centrally and accessed securely.

These tools and technologies together automate build and test processes, ensure efficient integration of code changes, and enhance overall development productivity.

Prerequisites

Detailed steps for setting up each prerequisite will be provided in the documentation.

• AWS Account:

Sign up for an AWS account to access AWS CodeBuild, AWS CodePipeline, and other AWS services.

• GitHub Repository:

Have a GitHub repository where your application source code is stored. This repository will trigger automated builds in AWS CodeBuild upon code commits.

• AWS IAM Role:

Create an IAM role with necessary permissions to allow AWS CodePipeline and AWS CodeBuild to access AWS resources securely.

• AWS CLI:

Install and configure the AWS Command Line Interface (CLI) to interact with AWS services from your terminal.

• Docker Hub Account (Optional):

If using Docker containers, have a Docker Hub account to store and manage Docker images used in your application.

• Access to AWS Systems Manager (Optional):

Ensure access to AWS Systems Manager Parameter Store if storing sensitive credentials like Docker Hub credentials securely.

Setting Up Git Repository

• Fork the GitHub Repository: Fork the GitHub repository I’ve prepared for this demo. You can find it here.

Create IAM Role and Add Permissions

• Navigate to the AWS IAM Console.
• In the left-hand menu, click on “Roles” under “Access management”.
• Click on the “Create role” button.
• Select the service that will use this role, choose “AWS service” .

• Select “CodeBuild” for the service.

• Search for and select the following policies to attach to the role: AmazonEC2FullAccess and AmazonSSMFullAccess.

• Provide a name and description for the role, then click “Create role”.

• Optionally, add tags to your IAM role for better organization and management.
• Review everything then click “Create role”.

Configure AWS Systems Manager Parameter Store

To securely manage and access Docker credentials such as username, password, and Docker registry URL within AWS environments, the AWS Systems Manager Parameter Store provides a robust solution.

By storing these sensitive values as SecureString parameters, AWS ensures encryption using AWS Key Management Service (KMS), maintaining the confidentiality and integrity of your credentials.

This setup is crucial for integrating Docker-based workflows with AWS services like AWS CodeBuild and AWS CodePipeline, enabling seamless and secure CI/CD pipelines.

• Navigate to the AWS Systems Manager Console.
• Click on “Parameter Store” in the left-hand menu
• Click on the “Create parameter” button.
• Enter parameter details: Enter a name for your parameter, such as /docker/username, /docker/password, and /docker/registry-url.
• Optionally, provide a description to document the purpose of the parameter.

• Choose “SecureString” as the parameter type. This encrypts the parameter value with an AWS Key Management Service (KMS) key, ensuring secure storage.
• Optionally, you can choose a specific AWS KMS key to encrypt the parameter value. AWS manages the default KMS key if none is specified.
• Create the value
• Click on “Create parameter” to save.

The Docker credentials are securely stored in the AWS Systems Manager Parameter Store.

AWS CodeBuild

• Navigate to the AWS CodeBuild Console.
• Click on “Create build project” button.
• Enter a name for your CodeBuild project.

• Choose the source provider where your build will fetch the code. This could be AWS CodeCommit, GitHub, Bitbucket, or Amazon S3.
• Specify the branch, tag, or revision ID for the source code version.

• Define the build environment with details such as operating system, runtime version, and Docker configuration if applicable.

• Choose an “existing service role.”
• From the dropdown menu, select the IAM role you created in the previous steps. This role should have the necessary permissions, such as SSM Full Access, to access the Docker credentials stored in AWS Systems Manager and any other required resources.
• Ensure you enable the privileged box.

• Choose between using a buildspec file stored in your source code repository or providing a buildspec directly in the console.

Buildspec file

• Copy and paste this buildspec file.

version: 0.2

env:
  parameter-store:
    DOCKER_REGISTRY_USERNAME: /myapp/docker-credentials/username
    DOCKER_REGISTRY_PASSWORD: /myapp/docker-credentials/password
    DOCKER_REGISTRY_URL: /myapp/docker-registry/url
phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo "Installing dependencies..."
      - pip install -r day-14/simple-python-app/requirements.txt
  build:
    commands:
      - echo "Running tests..."
      - cd day-14/simple-python-app/
      - echo "Building Docker image..."
      - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin "$DOCKER_REGISTRY_URL"
      - docker build -t "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest" .
      - docker push "$DOCKER_REGISTRY_URL/$DOCKER_REGISTRY_USERNAME/simple-python-flask-app:latest"
  post_build:
    commands:
      - echo "Build completed successfully!"
artifacts:
  files:
    - '**/*'
  base-directory: ../simple-python-app

• Configure where build logs should be stored for monitoring and troubleshooting.
• Specify the build artifacts to be generated and where they should be stored (e.g., S3 bucket).
• Review the configuration details to ensure accuracy.
• Click on “Create build project” to create the CodeBuild project.

• The CodeBuild project is now created and ready to execute builds based on the specified configuration.

• Click “Start Build”.
• Use “Build Details” to see the status of the build.

By following these steps, you can create an AWS CodeBuild project that integrates with your source code repository and is configured to build Docker images or execute other build tasks as per your project requirements.

Verify the Image Was Pushed to Docker Hub

• Log in to your Docker Hub account.
• Navigate to the repository where you expected the image to be pushed.
• Verify that the image is present with the correct tag and was pushed recently.

By following these steps, you can confirm that your Docker image was successfully pushed to Docker Hub, ensuring the build process was completed correctly.

AWS Code Pipeline

In this step, we will set up an AWS CodePipeline to automate the continuous integration (CI) process.

CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

By integrating with your source repository and CodeBuild project, CodePipeline will automatically detect changes in your code and trigger the build process.

For this article, we’ll focus on the CI part, ensuring that every code change is automatically built and tested.

The deployment (CD) part is covered in another documentation, which you can find here.

• Open the AWS Management Console.
• Navigate to the AWS CodePipeline service.
• Click on “Create pipeline” to start the pipeline creation wizard.
• Enter a name for your pipeline.

• Select “New service role” to let AWS create a role for the pipeline or “Existing service role” if you have a preconfigured IAM role.

• Select the source provider for your code repository (e.g., AWS CodeCommit, GitHub, Bitbucket).
• Enter the name of your repository.
• Specify the branch to use (e.g., main or master).

• Choose AWS CodeBuild.
• Select the region where your CodeBuild project is located.
• Select the CodeBuild project you created earlier.
• Add any necessary environment variables required for the build process.

• Skip code deploy, since this guide focuses on CI, you might not need a deploy stage here.
• Review all the stages and configurations.
• Click on “Create pipeline” to finalize and create your CodePipeline.
• Trigger the pipeline manually by clicking “Release change” or wait for an automated trigger from a new code commit.
• Monitor the progress of the pipeline in the CodePipeline console. Each stage (Source, Build) will show its status.

Clean Up Resources

After completing the deployment, it’s important to clean up resources to avoid unnecessary charges. Ensure you:

1: Delete the CodePipeline: Select the pipeline you created and delete it.

2: Delete the CodeBuild Project: Select the build used in the pipeline and delete it.

3: Remove Secrets from AWS Systems Manager: Select and delete the parameters used to store Docker credentials.

4: Clean Up IAM Roles: Find the roles created for this project (e.g., CodePipeline and CodeBuild roles). Detach any policies attached to these roles and then delete the roles.

Verify Deletion:

Double-check that all resources have been deleted from the AWS Management Console to ensure there are no lingering resources that could incur charges.

Conclusion

In this project, we successfully set up a Continuous Integration (CI) pipeline using AWS CodePipeline and AWS CodeBuild, complemented by AWS Systems Manager for securely managing Docker credentials.

This setup ensures that every code change is automatically built and tested, streamlining our development workflow and maintaining code quality.

By leveraging these AWS services, we’ve created a scalable and efficient CI pipeline that supports rapid development and deployment.

For further information on the CD part of this project, you can refer here.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Introduction

In today’s fast-paced development environment, deploying applications in a scalable, reliable, and efficient manner is crucial.

This guide will walk you through deploying a containerized application using Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) and AWS Fargate.

By the end, you’ll understand how to leverage these tools to streamline your deployment process for various architectures, including 2-tier (web server and database) and 3-tier (web server, application server, and database) applications.

Feel free to connect with me on LinkedIn to discuss this post, or ask any questions.

Project Overview

In this project, we aim to deploy an application on AWS, utilizing EKS for orchestrating the Kubernetes control plane and Fargate to manage serverless compute resources for the worker nodes.

This setup ensures our application benefits from Kubernetes’ orchestration capabilities and Fargate’s scalability and simplicity.

Additionally, we will configure Ingress to manage external access to the services within our cluster, providing load balancing and SSL termination.

Tools and Technologies Used

We will be utilizing a variety of technologies and tools in this guide, including:

• Amazon EKS: Managed Kubernetes service simplifying control plane and node operations.
• AWS Fargate: Serverless compute engine for containers, eliminating the need for infrastructure management.
• Kubernetes: An open-source system for automating deployment, scaling, and management of containerized applications.
• Docker: A Platform for developing, shipping, and running applications inside containers.
• kubectl: Command-line tool for interacting with Kubernetes clusters.
• eksctl: Command-line tool for creating and managing EKS clusters.
• AWS CLI: Tool to manage AWS services from the command line.
• Ingress: Kubernetes resource managing external access to services, providing load balancing and SSL termination.

Prerequisites

Before we dive into the deployment process, ensure you have the following:

1. AWS Account: Sign up for an AWS account to create and manage resources like EKS clusters and Fargate profiles.
2. AWS CLI: Install and configure the AWS Command Line Interface (CLI) to manage AWS services. Follow the instructions here.
3. kubectl: Install the kubectl command-line tool for interacting with Kubernetes clusters. Follow the instructions here.
4. eksctl: Install eksctl, a command-line utility for creating and managing EKS clusters. Follow the instructions here.

Ensure all prerequisites are met before proceeding with the deployment steps.

Setting Up the EKS Cluster and Worker Nodes

Create an EKS Cluster:

• The first step is to create an EKS cluster. You can do this using the eksctl command-line tool, which simplifies cluster creation.
• Specify configuration options such as region, version, and node instance type.
• Run the command to create the cluster.

eksctl create cluster --name <cluster-name> --region <region-name> --fargate

Replace <cluster-name> with your desired cluster name and <region-name> with the region for deployment.

Note: The EKS cluster creation process can take several minutes to complete. Please be patient during this step.

Configure Fargate Profiles:

• Define Fargate profiles to specify which Kubernetes namespaces should use Fargate for running pods.
• Create Fargate profiles for the necessary namespaces using eksctl or the AWS Management Console.

eksctl create fargateprofile --cluster <cluster-name> --region <region-name> --name <fargate-profile-name> --namespace <namespace-name>

Ensure that you replace <cluster-name>, <region-name>, <fargate-profile-name>, and <namespace-name> with the appropriate values for your setup.

Note: With EKS, you have the flexibility to choose between EC2 instances, which are self-managed, and Fargate, which is managed by AWS, for your worker nodes.

Verify Setup:

• Access the AWS Management Console and navigate to the “Amazon EKS” service.
• Choose the EKS cluster you recently created and review its details, including its name, status, and configuration.
• In the selected cluster, find the “Fargate Profiles” tab on the cluster details page and confirm the successful creation of Fargate profiles.

Update kubeconfig

After creating the EKS cluster, you need to update your kubeconfig file to interact with the cluster using kubectl. Follow these steps:

• Use the AWS CLI to update the kubeconfig file with your new cluster’s details:

aws eks update-kubeconfig --name <cluster-name> --region <region-name>

Replace <cluster-name> with the name of your EKS cluster and <region-name> with the AWS region where your cluster is deployed.

• Confirm that the kubeconfig file has been updated correctly by running the following command:

kubectl config get-contexts

Configure IAM OIDC Provider

To enable IAM roles for service accounts and allow your EKS cluster to access AWS services securely, you need to configure an IAM OpenID Connect (OIDC) provider. Follow these steps:

1. Create the OIDC Identity Provider:

• Use the eksctl command to associate the IAM OIDC provider with your EKS cluster.

eksctl utils associate-iam-oidc-provider --cluster <cluster-name> --region <region-name> --approve

Replace <cluster-name> with the name of your EKS cluster and <region-name> with the AWS region where your cluster is deployed.

2. Verify the OIDC Provider Configuration:

• Go to the AWS Management Console.
• Navigate to the “IAM” service.
• In the left-hand menu, select “Identity providers.”
• You should see an entry for the OIDC provider associated with your EKS cluster.

Setup AWS Load Balancer (ALB) Add-On

1. Download IAM Policy:

Download the IAM policy for the AWS Load Balancer Controller:

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

2. Create IAM Policy:

Create an IAM policy using the downloaded JSON file:

aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

3. Create IAM Role:

Create an IAM role and attach the IAM policy to the service account in your EKS cluster:

eksctl create iamserviceaccount \
  --cluster=<your-cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<your-aws-account-id>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

Replace <your-cluster-name> with the name of your EKS cluster and <your-aws-account-id> with your AWS account ID.

Note: Configuring the IAM OIDC provider and associating IAM roles with service accounts enhances the security and flexibility of your EKS cluster by allowing fine-grained access control to AWS resources.

Deploy ALB Controller

a. Add Helm Repo: Add the EKS Helm repository.

helm repo add eks https://aws.github.io/eks-charts

b. Update the Repo: Update the Helm repository.

helm repo update eks

c. Install the ALB Controller: Install the AWS Load Balancer Controller using Helm:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \            
  -n kube-system \
  --set clusterName=<your-cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=<region> \
  --set vpcId=<your-vpc-id>

Replace <your-cluster-name>, <region>, and <your-vpc-id> with your specific values.

d. Verify Deployment: Ensure that the AWS Load Balancer Controller deployment is running.

kubectl get deployment -n kube-system aws-load-balancer-controller

Deploying the Application

To deploy your application on AWS EKS, you will need to create and apply the necessary Kubernetes resource files: deployment.yaml, service.yaml, and ingress.yaml.

Below are the steps to apply each file individually.

1. Deployment

What is a Deployment?

A Deployment in Kubernetes is used to manage a set of identical pods, ensuring the desired number of replicas are running at any given time.

It allows for easy updates, rollbacks, and scaling of applications.

First, create the namespace and deployment for your application.

deployment.yaml:

---
apiVersion: v1
kind: Namespace
metadata:
  name: game-2048
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: game-2048
  name: deployment-2048
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app-2048
  replicas: 5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app-2048
    spec:
      containers:
      - image: public.ecr.aws/l6m2t8p7/docker-2048:latest
        imagePullPolicy: Always
        name: app-2048
        ports:
        - containerPort: 80

Apply the deployment:

kubectl apply -f deployment.yaml

2. Service

What is a Service?

A Service in Kubernetes is an abstraction that defines a logical set of pods and a policy by which to access them.

Services enable network access to a set of pods, typically through a single IP address and a set of ports.

Next, create a service to expose your application.

service.yaml:

---
apiVersion: v1
kind: Service
metadata:
  namespace: game-2048
  name: service-2048
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  type: NodePort
  selector:
    app.kubernetes.io/name: app-2048

Apply the service:

kubectl apply -f service.yaml

3. Ingress

What is an Ingress?

An Ingress in Kubernetes manages external access to the services in a cluster, typically HTTP and HTTPS.

It provides load balancing, SSL termination, and name-based virtual hosting.

Finally, create an ingress resource to manage external access to your application.

ingress.yaml:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: game-2048
  name: ingress-2048
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: service-2048
              port:
                number: 80

Apply the ingress:

kubectl apply -f ingress.yaml

Verify the Deployment

After applying all the configurations, verify that your resources are running correctly:

Check the deployment:

kubectl get deployments -n game-2048

Check the service:

kubectl get services -n game-2048

Check the ingress:

kubectl get ingress -n game-2048

Ensure that the status of the resources indicates successful deployment and that the ingress has an address assigned by the ALB.

Accessing the Application via Ingress

Once the Ingress resource is set up, you can access your application through the ALB (Application Load Balancer) created by the Ingress controller.

Follow these steps to access your application:

1. Find the ALB DNS Name:

• The ALB DNS name is provided by the Ingress controller and can be found by describing the Ingress resource.

kubectl get ingress -n game-2048

This output as seen previously will give you an address.

The ADDRESS column contains the ALB DNS name.

2. Access the Application:

• Open your web browser and navigate to the ALB DNS name. You should see your application running.
• If your application was correctly deployed and the Ingress rules are properly configured, the application’s home page should be displayed.

3. Troubleshooting:

• If you cannot access the application, ensure that the Ingress controller is correctly installed and the ALB is in an Active state.
• Verify the Ingress resource configuration to ensure the paths and backend services are correctly specified.
• Check the status of the Ingress resource for any errors or warnings:

kubectl describe ingress ingress-2048 -n game-2048

This command provides detailed information about the Ingress resource, including any events or error messages.

Summary

By following these steps, you should be able to access your deployed application via the Ingress resource.

The Ingress controller uses the ALB to route external traffic to your Kubernetes services, providing a scalable and reliable access point for your application.

Deleting the EKS Cluster and Associated Resources

To avoid incurring unnecessary charges, it’s important to delete your EKS cluster and associated resources when they are no longer needed.

Follow these steps to clean up your environment:

1. Delete the EKS Cluster using eksctl:

• The easiest way to delete the EKS cluster is by using the eksctl command-line tool.
• This command will delete the cluster and all associated resources, such as worker nodes and networking components.

eksctl delete cluster --name <cluster-name> --region <region-name>

2. Verify Cluster Deletion:

• After running the delete command, verify that the cluster has been deleted by checking the status in the AWS Management Console or by using the AWS CLI:

aws eks describe-cluster --name <cluster-name> --region <region-name>

• If the cluster has been successfully deleted, this command will return a “ResourceNotFoundException”.

3. Delete Associated Resources Manually:

Although eksctl typically cleans up all resources, you should manually verify that there are no lingering resources that could incur charges.

Check and delete the following resources if necessary:

• Load Balancers: In the EC2 console, check for any load balancers created by the Ingress controller.
• EBS Volumes: In the EC2 console, check for any Elastic Block Store (EBS) volumes that were attached to your nodes.
• Elastic IPs: In the EC2 console, check for any Elastic IP addresses.
• Security Groups: In the VPC console, check for any security groups associated with your cluster.

4. Delete IAM Roles and Policies:

• If you created any IAM roles or policies for your cluster (e.g., for the AWS Load Balancer Controller), delete them to avoid unnecessary charges:

aws iam delete-role --role-name <role-name>
aws iam delete-policy --policy-arn arn:aws:iam::<your-aws-account-id>:policy/<policy-name>

Replace <role-name> with the name of the IAM role and <policy-name> with the name of the IAM policy.

By following these steps, you ensure that all components associated with your EKS cluster are deleted, preventing any additional charges from accruing.

Conclusion

Deploying an application on AWS EKS can be straightforward with the right tools and configuration.

By following these steps, you can leverage the power of Kubernetes while benefiting from AWS’s managed services.

If you have any questions or run into issues, feel free to leave a comment below!

Happy deploying!

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Serverless Application Deployment with AWS EKS and Fargate was originally published in DevOps.dev on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Managing cloud costs effectively is essential for any organization using AWS (Amazon Web Services). One way to optimize costs is by identifying and removing unused resources.

In this blog post, I’ll show you how to save on storage costs by detecting and deleting stale EBS (Elastic Block Store) snapshots.

EBS snapshots are backups of your EBS volumes, but they can become orphaned when instances are terminated or volumes are deleted. These unused snapshots take up space and incur unnecessary costs.

We’ll create a Lambda function that automatically finds and deletes these stale snapshots, helping you manage your AWS expenses more efficiently.

Whether you’re new to AWS or an experienced user, this guide will help you automate cost-saving measures in your cloud environment.

Prerequisites

Before you start with the project, ensure you have the following prerequisites in place:

• AWS Account: With permissions to manage Lambda, EBS snapshots, and EC2 instances.
• AWS CLI: Installed and configured on your machine.
• Basic IAM Understanding: To create roles and custom policies.
• Basic AWS Lambda Knowledge: To create and deploy the function.
• Python Knowledge: Since the Lambda function will be in Python.
• Boto3 Library: Familiarity, as it will be used for AWS service interactions.

Ensure these prerequisites are met to follow along effectively.

Feel free to connect with me on LinkedIn to discuss this post or ask any questions.

Create an EC2 Instance

This instance will serve as a resource to interact with during the setup of our Lambda function.

Note that this step can also be automated using Terraform, but for the sake of simplicity, we will do it manually here.

Here is a detailed step-by-step guide to creating an EC2 instance:

1. Log into AWS Management Console:

• Navigate to the AWS Management Console.
• Sign in with your AWS credentials.

2. Open the EC2 Dashboard:

• In the AWS Management Console, search for “EC2” in the search bar.
• Select “EC2” from the list of services.

3. Launch Instance:

• Click the “Launch Instance” button on the EC2 Dashboard.

4. Add Tags:

• (Optional) Add tags to your instance for easier identification.

5. Choose an Amazon Machine Image (AMI):

• Select an appropriate AMI for your needs. For this example, you can choose “Ubuntu”.

6. Choose an Instance Type:

• Select an instance type. For most use cases, the “t2.micro” instance type is sufficient and is free tier eligible.

7. Select a Key Pair:

• Select an existing key pair or create a new key pair to securely access your instance.
• Download the key pair file (.pem) and keep it safe. You cannot download it again after it is created.

8. Configure Instance Details:

• Optionally, configure instance details such as network settings, subnets, IAM role, etc. You can leave these settings as default for now.

9. Configure Security Group:

• Create a new security group or select an existing one.
• Ensure that you have rules allowing SSH access (port 22) from your IP address.

10. Add Storage:

• The default storage settings should be sufficient.
• You can adjust the size if needed.

11. Review and Launch:

• Review your instance configuration.
• Click “Launch” to proceed.

Verify the Volume

• Navigate to the created EC2 Instance under the “storage” section.
• This will open the details page for the volume, where you can see information like size, state, and type.
• Notice that this volume was automatically created during the instance setup process and is used as the root volume for your EC2 instance.

Creating Snapshots of the Volume

Currently we don’t have any snapshots available for our instance.

To confirm this navigate to the EC2 Dashboard by searching for “EC2” in the services menu.

Find the “Snapshots” section in the left-hand menu and click on it.

• To create a snapshot of the instance click on the “Create Snapshot” button.

• Choose the EBS volume for which you want to create a snapshot from the dropdown menu.
• Optionally, add a description for the snapshot to provide more context.

• Double-check the details you’ve entered to ensure accuracy.
• Once you’re satisfied, click on the “Create Snapshot” button to initiate the snapshot creation process.

AWS Lambda

Creating an AWS Lambda function allows you to execute code in response to events without provisioning or managing servers.

In our project, we aim to automate the identification and deletion of stale EBS snapshots using an AWS Lambda function.

Here’s how you can create this Lambda function using the AWS Management Console:

1. Log into AWS Management Console:

• Navigate to the AWS Management Console.
• Sign in with your AWS credentials.

2. Open the Lambda Dashboard:

• In the AWS Management Console, search for “Lambda” in the services.
• Select “Lambda” from the list of services.

3. Click on “Create function”:

• On the Lambda Dashboard, click on the orange “Create function” button.

4. Choose Authoring Method:

• Select “Author from scratch” to create a new Lambda function from scratch.

5. Configure Basic Information:

• Provide a meaningful name for your Lambda function.
• Select the runtime environment. Since we’re using Python, choose “Python 3.12”.

• Click on “Create function” to proceed.

Edit Configuration

• Navigate to the “configuration tab” of the created Lambda function and click “Edit”.

• Look for the section where you can adjust the function timeout.
• By default, the timeout is set to “3 seconds”. Change this to “10 seconds”.
• This value represents the maximum amount of time (in seconds) that Lambda will allow the function to run before terminating it.
• It is good practice to keep the execution time as small as possible since AWS will charge you using this as one of the parameters.
• Click “Save” to apply the changes.

IAM Role

In our project, the Lambda function is central to optimizing AWS costs by identifying and deleting stale EBS snapshots.

To accomplish this, it requires specific permissions. These include the ability to describe and delete snapshots, as well as to describe volumes and instances.

Roles are used to delegate access to AWS resources securely, without the need to share long-term credentials like access keys.

To achieve this follow these step by step guide below:

• In the Lambda function details page, click on the “Configuration” tab.
• Scroll down to the “Permissions” section to expand it.
• Click on the execution role link to open the IAM role configuration in a new tab.

• In the new tab that opens, you’ll be directed to the IAM Console with the details of the IAM role associated with your Lambda function.
• Scroll down to the “Permissions” section of the IAM role details page.
• Click on the “Add inline policy” button to create a new inline policy.

• Choose services as EC2 filter permissions.
• Search “Snapshot” and add these options “DescribeSnapshots”, “Delete Snapshots”.
• Also add these permissions as well “Describe Volume”, and “Describe Instances”.

• Under the Resources section select “All”.
• Click the “Next” button.

• Give the name of the policy then click the “Create Policy” button.

• Make sure that the created policy is attached to the existing role.

Writing the Lambda Function

Our Lambda function, powered by Boto3, automates the identification and deletion of stale EBS snapshots. Key features include:

1. Snapshot Retrieval: Fetching owned EBS snapshots and active EC2 instances.
2. Stale Snapshot Detection: Identifying unattached snapshots and checking volume-attachment status.
3. Exception Handling: Ensuring robustness with error management.
4. Cost Optimization: Efficiently managing resources to minimize storage costs.

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Get all EBS snapshots
    response = ec2.describe_snapshots(OwnerIds=['self'])

    # Get all active EC2 instance IDs
    instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    active_instance_ids = set()

    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
    for snapshot in response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        if not volume_id:
            # Delete the snapshot if it's not attached to any volume
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
        else:
            # Check if the volume still exists
            try:
                volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                if not volume_response['Volumes'][0]['Attachments']:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
            except ec2.exceptions.ClientError as e:
                if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                    # The volume associated with the snapshot is not found (it might have been deleted)
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

This script is pivotal in our AWS cost optimization strategy, showcasing the effectiveness of serverless computing for streamlining operations.

Testing the Lambda Function

To simulate a real-world scenario, delete the existing EC2 instance.

When an EC2 instance is deleted, AWS automatically removes the attached EBS volume as shown below.

However, any EBS snapshots associated with that volume remain in storage, even though they are no longer needed.

These snapshots, termed ‘stale,’ incur additional storage costs without serving any purpose.

Therefore, it’s crucial to regularly identify and remove such stale snapshots to optimize AWS storage costs effectively.

Once the Instance is deleted we can observe whether our Lambda function successfully identifies and removes any associated snapshots.

Follow the below steps:

• Navigate to the created EC2 instance and terminate it.
• Navigate to the created Lambda Function.
• Under the “Code” section in “lambda_function” paste the above Python code for your Lambda function.
• Ensure that your code includes necessary imports (e.g., import boto3) and the lambda_handler function.

• Save your Lambda Function to test it click on the “Test” Button.
• After running the test, review the output and logs to ensure that your function executed as expected.
• Debug any errors or issues that may arise during testing.

• Once your function passes testing, you can deploy it by clicking on the “Deploy” button.

The Lambda function automatically finds and deletes these stale snapshots, helping you manage your AWS expenses more efficiently.

• After deployment, monitor the execution of your Lambda function in the AWS Lambda console.
• Use CloudWatch Logs to view detailed logs and metrics for your function.

Conclusion

In this project, we’ve implemented a solution to automate the identification and deletion of stale EBS snapshots, leveraging AWS Lambda and Boto3.

By optimizing storage usage, we’ve reduced costs and improved resource efficiency.

This project demonstrates the effectiveness of automation in driving cost optimization within AWS, setting the stage for continued success in cloud management endeavors.

Connect with me

For more insights and future projects, feel free to connect with me on LinkedIn , GitHub and Medium.

Python’s Gurus🚀

Thank you for being a part of the Python’s Gurus community!

Before you go:

• Be sure to clap x50 time and follow the writer ️👏️️
• Follow us: Newsletter
• Do you aspire to become a Guru too? Submit your best article or draft to reach our audience.

Automated Cost Optimization for AWS EBS Snapshot Using Lambda was originally published in Python’s Gurus on Medium, where people are continuing the conversation by highlighting and responding to this story.