Introduction

This project uses Terraform and Ansible roles to deploy and configure a MYSQL server, then tests connections to the server using a custom Java application. The database server can only be accessed within the VPC where the web and database servers are deployed. However, the web server has to be publicly accessible on port 8080 to access the web application.

Checklist

Ansible server playbook

• Create a Terraform configuration file that meets the project requirements and includes an inventory template file that dynamically generates an Ansible inventory file.
• Configure an Ansible server to deploy database (MySQL) and web (Java) applications.
• Install required packages on the Ansible server
• Copy necessary playbook files to the Ansible server
• Copy application files to the Ansible server
• Copy MySQL role variable files to the Ansible server
• Copy the private key to access the web and database servers

Database server Playbook

• Use the database playbook to set custom variables for the MySQL role

Web server playbook

• Install necessary packages
• Copy application files to the server
• Copy inventory file
• Build the application
• Set the environment variable needed to connect to the database server and execute the application.

Terraform configuration for the database server

The Terraform configuration provisions 3 servers. It includes an Ansible server, a database server, and a web server. The web server can be accessed externally to access the web application, but the database server cannot. It is only accessible from the web server.

To do this, create two separate subnets: a public subnet and a private subnet.

resource "aws_subnet" "myapp-subnet-1" {
  vpc_id = aws_vpc.myapp-vpc.id
  cidr_block = var.subnet_cidr_block
  availability_zone = var.avail_zone
    tags = {
    Name: "${var.env_prefix}-subnet-1"
  }
}


resource "aws_subnet" "myapp-subnet-2" { # private subnet for database server
  vpc_id = aws_vpc.myapp-vpc.id
  cidr_block = var.subnet2_cidr_block
  availability_zone = var.avail_zone
    tags = {
    Name: "${var.env_prefix}-subnet-private"
  }
}

Create an Elastic IP for the NAT gateway.

resource "aws_eip" "nat_eip" { # creates elastic ip for nat gateway to access internet
    domain = "vpc"
}

Create an internet gateway, and associate the Elastic IP and public subnet with it.

 resource "aws_nat_gateway" "nat_gateway" { # creates a nat gateway and allocates the elastic ip to it
    allocation_id =  aws_eip.nat_eip.id
    subnet_id = aws_subnet.myapp-subnet-1.id

}

Create a private route table and add a route that includes the CIDR block and NAT gateway route.

resource "aws_route_table" "private-rtb" { # creates a route table for the private subnet and associates with the VPC
  vpc_id = aws_vpc.myapp-vpc.id

  route {
      cidr_block = "0.0.0.0/0"
      nat_gateway_id = aws_nat_gateway.nat_gateway.id
  }

Create a route table association resource to associate the private route table with the private subnet.

resource "aws_route_table_association" "private-rtb-association" {
    subnet_id = aws_subnet.myapp-subnet-2.id
    route_table_id = aws_route_table.private-rtb.id
}

Create a private security group that only accepts ingress connections from within the VPC

resource "aws_security_group" "private-sg" {
  vpc_id = aws_vpc.myapp-vpc.id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "TCP"
    cidr_blocks = [var.vpc_cidr_block]
  }

  ingress {
    from_port = 3306
    to_port = 3306
    protocol = "TCP"
    cidr_blocks = [var.vpc_cidr_block]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    prefix_list_ids = []
  }

  tags = {
    Name: "${var.env_prefix}-private-sg"
  }
}

In the "aws_instance" resource for the database server, include the private security group and subnet.

resource "aws_instance" "database-server" {
  ami = data.aws_ami.latest-amazon-linux-image.id
  instance_type = var.instance_type

  subnet_id = aws_subnet.myapp-subnet-2.id
  vpc_security_group_ids = [aws_security_group.private-sg.id]
  availability_zone = var.avail_zone

  key_name = aws_key_pair.ssh-key.key_name

  tags = {
    Name: "${var.env_prefix}-database-server"

  }
}

Terraform template files

Since the IP addresses of the provisioned servers cannot be known in advance, the Ansible hosts cannot be hardcoded. Dynamic inventory files can be created using a Terraform template file. After Terraform completes creating the infrastructure, it uses the template file to create and populate the inventory file.

A "local file" resource reads the template file and creates the inventory.ini file. It uses the content parameter to load the template file, and uses the filename parameter to specify the path where the inventory file will be created. The "local_file" resource also includes variables assigned the server's IP address resource attribute. The template uses these variables to populate the inventory file.

"local_file" resource

resource "local_file" "ansible_inventory" {
  filename = "../ansible/ansible-config/inventory.ini"

  content = templatefile("inventory.ini.tftpl", {

    ip_one = aws_instance.ansible-server.public_ip
    ip_two = aws_instance.database-server.private_ip
    ip_three = aws_instance.web-server.public_ip
    ip_three_private = aws_instance.web-server.private_ip


  }
  
  )


}

Inventory file template

[ansible-server]
server1 ansible_host=${ip_one} ansible_user=ec2-user

[database-server]
server2 ansible_host=${ip_two} ansible_user=ec2-user ansible_python_interpreter=/usr/bin/python3.9

[web-server]
server3 ansible_host=${ip_three_private} ansible_user=ec2-user


[web-server:vars]
db_server=${ip_two}


[database-server:vars]
web_server=${ip_three_private}

Example inventory file

[ansible-server]
server1 ansible_host=13.221.43.107 ansible_user=ec2-user

[database-server]
server2 ansible_host=10.0.20.230 ansible_user=ec2-user ansible_python_interpreter=/usr/bin/python3.9

[web-server]
server3 ansible_host=10.0.10.105 ansible_user=ec2-user


[web-server:vars]
db_server=10.0.20.230


[database-server:vars]
web_server=10.0.10.105

Setting up the Ansible server

Install required packages.

- name: Install required packages
  become: yes
  hosts: ansible-server
  tasks:
    - name: Update packages
      dnf:
        update_cache: yes

    - name: Install ansible
      dnf:
        name: ansible
        state: present


    - name: Install geerlingguy.mysql role
      community.general.ansible_galaxy_install:
        type: role
        name: geerlingguy.mysql
        dest: /home/ec2-user/.ansible/roles

This play installs and updates the cache with dnf using the "dnf" module. The next task installs Ansible on the server, and the last task installs geerlingguy's MySQL role to configure MySQL on the database server. The project's GitHub repository is available here: https://github.com/geerlingguy/ansible-role-mysql.

Copy playbooks to the Ansible server.

- name: Copy database config files to Ansible server
  hosts: ansible-server
  vars_files:
    - project-vars
  tasks:
    - name: Copy database playbook files
      copy:
        src: "{{ src_path_database }}"
        dest: /home/ec2-user/
    - name: Copy inventory file
      copy:
        src: "{{ src_path_inventory }}"
        dest: /home/ec2-user/database-config



- name: Copy web config files to Ansible server
  hosts: ansible-server
  vars_files:
    - project-vars
  tasks:
    - name: Copy web playbook files
      copy:
        src: "{{ src_path_web }}"
        dest: /home/ec2-user/
    - name: Copy inventory file
      copy:
        src: "{{ src_path_inventory }}"
        dest: /home/ec2-user/web-config

These plays use the "copy" module to copy the playbooks and inventory files to the Ansible server. The inventory files are dynamically generated from the Terraform template file and contain IP addresses for the web server and database servers.

Copy application files to the Ansible server.

- name: Copy java app to Ansible server
  hosts: ansible-server
  vars_files:
    - project-vars
  tasks:
    - name: Copy java files
      copy:
        src: "{{ src_path_java_app }}"
        dest: /home/ec2-user/

This play also uses the "copy" module to copy the application files to the Ansible server.

Copy modified Geerlingguy's MySQL vars file to the Ansible server.r

- name: Copy RedHat vars file to MySQL roles folder
  hosts: ansible-server
  become: yes
  vars_files:
    - project-vars
  tasks:
    - name: Copy vars file to ansible server
      copy: 
        src: "{{ src_path_redhat2023 }}"
        dest: /home/ec2-user/.ansible/roles/geerlingguy.mysql/vars/

Loading the Ansible role fails because the code block below searches for the OS-specific variable "{{ ansible_facts.os_family }}-{{ ansible_facts.distribution_major_version }}.yml" in the vars folder. The image name specified in the "terraform.tfvars" file uses Amazon 2023. The OS family does display "RedHat", but the major version is "2023", so the full name is "RedHat-2023". This file is not available in the project's "vars" section.

To work around this, create a "RedHat-2023.yml" file and copy it to the vars section in the user's home folder at "/home/ec2-user/.ansible/roles/geerlingguy.mysql/vars/". This allows the tasks section to find "RedHat-2023.yml" and execute code from that configuration file instead.

Original RedHat-10.yml file

---
__mysql_daemon: mariadb
__mysql_packages:
  - mariadb
  - mariadb-server
  - mariadb-connector-c
  - python3-PyMySQL
  - perl-DBD-MySQL
__mysql_slow_query_log_file: /var/log/mysql-slow.log
__mysql_log_error: /var/log/mariadb/mariadb.log
__mysql_syslog_tag: mariadb
__mysql_pid_file: /var/run/mariadb/mariadb.pid
__mysql_config_file: /etc/my.cnf
__mysql_config_include_dir: /etc/my.cnf.d
__mysql_socket: /var/lib/mysql/mysql.sock
# The entries controlled by this value should not be used with MariaDB >= 10.2.2
# See https://github.com/frappe/bench/issues/681#issuecomment-398984706
__mysql_supports_innodb_large_prefix: false

RedHat-2023.yml (modified to work on Amazon Red Hat Linux)

- mariadb123-client-utils.noarch
  - mariadb123-server-utils.noarch
  - mariadb-connector-c

The RedHat-2023.yml file was updated to include packages specific to Amazon Linux, like mariadb123-client-utils.noarch, mariadb123-server-utils.noarch, and mariadb-connector-c

Copy the private key for the database and web server to the Ansible server

- name: Copy private key for database and web server to Ansible server
  hosts: ansible-server
  vars_files:
    - project-vars
  tasks:
  - name: Copy private key
    copy:
      src: "{{ src_path_private_key }}"
      dest: /home/ec2-user/.ssh/
      mode: '0600'

The private key was copied to the ec2-user home directory and set to mode 600, so Amazon accepts it as a valid key file for accessing the database server and web server.

Configure Database server

Install packages required to run MySQL.

- name: Install MYSQL packages
  hosts: database-server
  become: yes
  tasks:
    - name: Install pip
      dnf:
        name: pip
        state: present

    - name: Install PyMySQL
      pip:
        name: PyMySQL

This play installs pip, and then uses the "pip" module to install PyMySQL, an important component for running MySQL on the server.

Configure MySQL server

- name: Configure MSQL Database server
  hosts: database-server
  become: yes
  vars_files:
    - project-vars

    # default root user does intial setup and creates sql_users

  vars:

    mysql_port: "3306"
    mysql_bind_address: "0.0.0.0"

    mysql_databases:
      - name: example_db

    mysql_users:
      - name: appuser
        password: "{{ sql_password }}"
        host: "{{ web_server }}"     #"%" from web_server vars in inventory.ini
        priv: "example_db.*:ALL"


  roles:
    - geerlingguy.mysql

The default values for running the MySQL server can be found here-https://github.com/geerlingguy/ansible-role-mysql/blob/master/defaults/main.yml, but the values can be customized or overridden for specific purposes. That is what the play above does. It sets the mysql_port and the bind address. It also provides the database to create on initialization. By default, the root user creates the SQL server, creates the user defined in "mysql_users" using the password in "project-vars", and uses the database variable “web_server” defined in the inventory.ini file.

Configure web server

Install required packages

- name: Install required packages
  become: yes
  hosts: web-server
  tasks:
    - name: Update packages
      dnf:
        update_cache: yes

    - name: Install java
      dnf:
        name: java-17-amazon-corretto-devel
        state: present

- name: Install Gradle
  hosts: web-server
  tasks:
    - name: Install SDK
      shell: 'curl -s "https://get.sdkman.io" | bash'

    - name: Install gradle
      shell: |
        source /home/ec2-user/.sdkman/bin/sdkman-init.sh
        sdk install gradle  

This play uses the "dnf" module to update the cache, install the JDK, SDK, and Gradle.

Copy the application folder and inventory file to the web server

- name: Copy Java files to web-server
  hosts: web-server
  vars_files:
    - project-vars
  tasks:
    - name: Copy java files to web-server
      copy:
        src: /home/ec2-user/java-app
        dest: /home/ec2-user/




- name: Copy inventory file # to get ip of database server
  hosts: web-server
  tasks:
    - name: Copy .ini file
      copy:
        src: /home/ec2-user/web-config/inventory.ini
        dest: /home/ec2-user/java-app/

These plays copy the application files and the inventory file to the web server; the inventory file variables are used to access the database server.

Build Java file

- name: Build the java file
  hosts: web-server
  tasks:

    - name: change directory and run gradle build
      command: "gradle build -p /home/ec2-user/java-app/"
      register: build_output

    - name: Display build build output
      ansible.builtin.debug:
        var: build_output.stdout_lines

Set environment variables and run the JAR file.

- name: Set env variables for MYSQL connection
  hosts: web-server
  vars_files:
    - project-vars

  environment:
    DB_SERVER: "{{ db_server }}"
    DB_NAME: example_db
    DB_USER: appuser
    DB_PWD: "{{ sql_password }}"
  
  tasks:

    - name: Execute java file
      shell: "java -jar /home/ec2-user/java-app/build/libs/build-tools-exercises-1.0-SNAPSHOT.jar"

The environment block sets DB_SERVER, DB_PWD, DB_USER, and DB_NAME. DB_NAME and DB_USER were already created on the SQL server by deploying the database playbook using the Ansible MySQL role. DB_SERVER comes from inventory.ini, and DB_PWD comes from project-vars.

Introduction

This project uses Terraform to provision an EC2 instance and Ansible to set up a Nexus server by installing required packages, automating setup via the Nexus REST APIs, and deploying a custom Java application to the Maven snapshot repository.

Checklist

• Use Terraform to provision an EC2 instance on AWS
• Install tools required to build the application
• Move the Java application to the Nexus server and build it
• Download and unpack the Nexus installer
• Create a Nexus user and group, and make the Nexus user the owner of both Nexus folders.
• Start Nexus and verify it runs
• Get the Nexus server's external IP
• Change the initial admin password
• Accept Nexus EULA
• Create Nexus role
• Create Nexus user
• Push JAR file to the repo

Provision an EC2 instance with Terraform

provider "aws" {
  region = "us-east-1"
}

variable vpc_cidr_block {}
variable subnet_cidr_block {}
variable avail_zone {}
variable env_prefix {}
variable my_ip {}
variable instance_type {}
variable public_key_location {}
variable image_name {}
variable ssh_key_private {}

resource "aws_vpc" "myapp-vpc" {
  cidr_block = var.vpc_cidr_block
  enable_dns_hostnames = true
  tags = {
    Name: "${var.env_prefix}-vpc"
  }
}

resource "aws_subnet" "myapp-subnet-1" {
  vpc_id = aws_vpc.myapp-vpc.id
  cidr_block = var.subnet_cidr_block
  availability_zone = var.avail_zone
    tags = {
    Name: "${var.env_prefix}-subnet-1"
  }
}

resource "aws_internet_gateway" "myapp-igw" {
  vpc_id = aws_vpc.myapp-vpc.id
  tags = {
    Name: "${var.env_prefix}-igw"
  }
}

resource "aws_default_route_table" "main-rtb" {
  default_route_table_id = aws_vpc.myapp-vpc.default_route_table_id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.myapp-igw.id
  }
  tags = {
    Name: "${var.env_prefix}-main-rtb"
  }
}

resource "aws_default_security_group" "default-sg" {
  vpc_id = aws_vpc.myapp-vpc.id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "TCP"
    cidr_blocks = [var.my_ip]
  }

  ingress {
    from_port = 8080
    to_port = 8080
    protocol = "TCP"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 8081
    to_port = 8081
    protocol = "TCP"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    prefix_list_ids = []
  }

  tags = {
    Name: "${var.env_prefix}-default-sg"
  }
}

data "aws_ami" "latest-amazon-linux-image" {
  most_recent = true
  owners = ["amazon"]
  filter {
    name = "name" 
    values = [var.image_name]
  }
  filter {
    name = "virtualization-type"
    values = ["hvm"]
  }
}

output "aws-ami_id" {
  value = data.aws_ami.latest-amazon-linux-image.id
}

output "ec2-public_ip_1" {
  value = aws_instance.myapp-server-one.public_ip
}


resource "aws_key_pair" "ssh-key" {
  key_name = "server-key"
  public_key = file(var.public_key_location)
}

resource "aws_instance" "myapp-server-one" {
  ami = data.aws_ami.latest-amazon-linux-image.id
  instance_type = var.instance_type

  subnet_id = aws_subnet.myapp-subnet-1.id
  vpc_security_group_ids = [aws_default_security_group.default-sg.id]
  availability_zone = var.avail_zone

  associate_public_ip_address = true
  key_name = aws_key_pair.ssh-key.key_name

  tags = {
    Name: "${var.env_prefix}-server-1"
  }

  

}

resource "null_resource" "configure_server" {

    triggers = {
        trigger = aws_instance.myapp-server-one.public_ip
    }

    provisioner "local-exec" {
        working_dir = "../ansible"
        command = "ansible-playbook --inventory ${aws_instance.myapp-server-one.public_ip}, --private-key ${var.ssh_key_private} --user ec2-user ../ansible/deploy_nexus.yaml"
    }



}

The Terraform configuration creates an Amazon Linux image and uses a null resource with a "local-exec" provisioner to run the "ansible-playbook" command after the instance is created. An ingress rule for port 8081 is added to the security group because the Nexus web server is accessible on port 8081 by default. The inventory parameter in "ansible-playbook" is set to the instance's public IP, and the null_resource is set to trigger when the public IP changes.

Install tools required to build the application.

- name: Install Java and Net-tools
  hosts: all
  become: yes
  tasks:
    - name: Update dnf repo
      dnf:
        update_cache: true
    - name: Install Java
      dnf:
        name: java-17-amazon-corretto-devel.x86_64
        state: present
    - name: Install Net-tools
      dnf:
        name: net-tools.x86_64
        state: present


- name: Install Gradle
  hosts: all
  tasks:
    - name: Install SDK
      shell: 'curl -s "https://get.sdkman.io" | bash'

    - name: Install gradle
      shell: |
        source /home/ec2-user/.sdkman/bin/sdkman-init.sh
        sdk install gradle  

Gradle is required to build the application and run Sonatype Nexus. A JDK (Java Development Kit) must be installed over a JRE, as the JDK compiler is needed to build the Java application with Gradle.

The first task from the first play uses the ansible dnf module to update the cache and install the Java 17 JDK. Net-tools is also installed to make the "netstat" command available, so network connections can be checked to determine what port Nexus is running on.

The second play uses SDKMAN to install Gradle on Amazon Linux, a method recommended in the Gradle user manual. SDKMAN is installed using curl, and the second task loads and initializes the SDKMAN environment into the current shell session so the sdk commands are available.

Move the Java application to the Nexus server and build it

- name: Copy Java files to web-server
  hosts: all
  tasks:
    - name: Copy java files to web-server
      copy:
        src: ../java-app # without trailing /, it creates the folder and copies contents into it. with / it copies contents without creating the folder
        dest: /home/ec2-user/


- name: Build the java file
  hosts: all
  tasks:

    - name: Run gradle build
      command: "gradle build -p /home/ec2-user/java-app/"
      register: build_output

    - name: Display build build output
      ansible.builtin.debug:
        var: build_output.stdout_lines

The first play uses the ansible copy module to transfer files from one location to another. In this case, it transfers the application folder from the Ansible server to the Nexus server at "/home/ec2-user".

The second play uses the Ansible command module to build the application, then uses the debug module to output the build result.

Download and unpack the Nexus installer.

- name: Download and unpack Nexus installer
  hosts: all
  become: yes
  tasks:

    - name: Get nexus folder stats
      stat:
        path: /opt/nexus
      register: stat_result

    - name: Download Nexus
      get_url:
        url: https://download.sonatype.com/nexus/3/latest-linux-x86_64.tar.gz
        dest: /opt/
      register: download_result

    - name: Display download destination
      ansible.builtin.debug:
        var: download_result.dest

    - name: Untar downloaded file
      unarchive:
        src: "{{ download_result.dest }}"
        dest: /opt/
        remote_src: yes
      when: not stat_result.stat.exists

    - name: Find nexus folder
      find:
        paths: /opt/
        pattern: "nexus-*"
        file_type: directory
      register: find_result


    - name: Rename nexus folder
      shell: mv {{ find_result.files[0].path }} /opt/nexus
      when: not stat_result.stat.exists

The Ansible stat module is used to get details about a file or folder. Nexus will be downloaded in the "/opt" directory, so details of that path need to be collected and registered to create conditionals, so rerunning the playbook doesn't overwrite the downloaded or unarchived folder, which may have unintended consequences and negatively affect the Nexus installation and configuration. Downloading files into the "/opt" directory requires root privileges, so "become:yes" switches from the regular "ec2-user" to root.

The second task uses the Ansible "get_url" module to download a file, using "dest" as a parameter for the URL, into a destination folder.

The fourth task untars the downloaded tar file using the Ansible "unarchive" module. It uses the "src" and "dest" parameters, and the "remote_src" parameter specifies that the source and destination are on the same remote server, instead of searching for the file on the local filesystem. The results from the "get_url" module were stored in a variable, "download_result". An attribute from that variable is then assigned to "src". Setting this value dynamically is more reliable than hardcoding the path because it ensures that if the path changes, the untar command can use the new file path. A conditional at the end of the tasks prevents the task from running if the Nexus path already exists.

When the Nexus tar file is downloaded, it is named like "nexus-xxxx" where xxxx is the version number. Since the latest version is being downloaded, the exact number is likely to change over time. The fifth task uses the Ansible "find" module to find the Nexus folder name, registers it into a variable, and uses the Ansible "shell" module to rename it to "/opt/nexus" so subsequent operations use the same path to refer to the same Nexus folder.

Create a Nexus user and group, and make the Nexus user the owner of both Nexus folders.

- name: Make nexus user owner of nexus folders
  hosts: all
  become: yes
  tasks:

    - name: Create nexus group
      group:
        name: nexus
        state: present

    - name: Create nexus user
      user:
        name: nexus
        group: nexus

    - name: Make nexus user owner of nexus folder
      file:
        path: /opt/nexus
        state: directory
        owner: nexus
        group: nexus
        recurse: yes

    - name: Make nexus user owner of sonatype-work folder
      file:
        path: /opt/sonatype-work
        state: directory
        owner: nexus
        group: nexus
        recurse: yes

The first task uses the Ansible "group" module to create a new group on the server, using the "name" and "state" parameters. Since root permissions are needed to create a group and a user, "become: yes" is added at the play level.

The second task uses the Ansible "user" module to create a new user "nexus" and assigns the user to the nexus group.

The third and fourth tasks use the Ansible "file" module to set the newly created nexus user as owner of the "nexus" and "sonatype-work" folders. "recurse: yes" applies this to the parent directory and all subdirectories and files.

Start Nexus and verify it runs.

- name: Start nexus with nexus user
  hosts: all
  become: yes
  become_user: nexus
  tasks:

    - name: Set run_as_user nexus
      blockinfile:
        path: /opt/nexus/bin/nexus.rc
        create: yes
        block: |
          run_as_user="nexus"

    - name: Start nexus
      command: /opt/nexus/bin/nexus start



- name: Verify nexus is running
  hosts: all
  tasks:


    - name: Check nexus process
      shell: ps aux | grep nexus
      register: nexus_process_status
    - debug: msg={{ nexus_process_status.stdout_lines }}
    
    - name: Wait 30 seconds
      pause:
        seconds: 30

    - name: Check with netstat
      command: netstat -plnt
      register: nexus_network_status
    - debug: msg={{ nexus_network_status.stdout_lines }}

For security reasons, the Nexus configuration file must be modified so the Nexus process runs as the nexus user, and not the root user. This can be done with the Ansible "blockinfile" module. In newer versions of Nexus, the "nexus.rc" file is no longer available in "/opt/nexus/bin" and must be created. This is done with the "create: yes" parameter. The "block parameter" specifies what values are written into the file. "run_as_user=nexus" is used to make sure the Nexus service runs under the "nexus" user.

The next task uses the command module to start Nexus

The shell and command modules run the "ps" and "netstat" commands to verify that the Nexus service started successfully. The results of both commands are printed to the screen.

Get Nexus server external IP

- name: Get external IP
  hosts: all
  tasks:

    - name: Get external public IP
      ansible.builtin.uri:
        url: https://api.ipify.org
        return_content: true
      register: public_ip
    - debug: msg={{ public_ip.content }}

    - name: Save nexus URL as a fact
      ansible.builtin.set_fact:
        nexus_url: "http://{{ public_ip.content | trim }}:8081"
    - debug: msg={{ nexus_url }}

Since the public IP will most likely change after running Terraform apply, the instance IP must be retrieved dynamically to allow subsequent Nexus configurations. This can be done using the ipify api. This works by using the Ansible "uri" module to access the "ipify" api, return its contents, and register it to a variable. The second task then sets the "public_ip" variable as an Ansible fact so it can be used in later plays.

Change initial admin password.

The initial Nexus admin password must be changed before proceeding with further configuration. Since the whole process will be automated, the REST API will be used to configure Nexus. The API reference link can be found here — https://help.sonatype.com/en/api-reference.html.

- name: Change initial admin password
  hosts: all
  vars_files:
    - project-vars
  tasks:

    - name: Retrieve initial admin password
      command: cat /opt/sonatype-work/nexus3/admin.password
      register: initial_password

    - name: Change admin password
      ansible.builtin.uri:
        url: "{{nexus_url}}/service/rest/v1/security/users/admin/change-password"
        method: PUT
        url_username: admin
        url_password: "{{initial_password.stdout}}"
        force_basic_auth: true
        headers:
          Content-Type: text/plain
        body: "{{ nexus_adminpass }}"
        status_code: 204
      no_log: false

The first task retrieves the admin password from "/opt/sonatype-work/nexus3/admin.password" and registers it to the "initial_password" variable.

The next task uses the "uri" module to access the "change-password" API, enters the initial password, and uses the request body to set a new password.

Accept Nexus EULA

- name: Accept Nexus EULA
  hosts: all
  vars_files:
    - project-vars
  tasks:

    - name: Retrieve Disclaimer
      ansible.builtin.uri:
        url: "{{ nexus_url }}/service/rest/v1/system/eula"
        method: GET
        url_username: admin
        url_password: "{{ nexus_adminpass }}"
        force_basic_auth: true
        return_content: true
        status_code: 200
      register: nexus_eula

    - name: Accept EULA
      ansible.builtin.uri:
        url: "{{ nexus_url }}/service/rest/v1/system/eula"
        method: POST
        url_username: admin
        url_password: "{{ nexus_adminpass }}"
        force_basic_auth: true
        body_format: json
        body:
          disclaimer: "{{ nexus_eula.json.disclaimer }}"
          accepted: true
        status_code: [200 ,204]

Before applications can be pushed to the repository, the EULA needs to be accepted by the admin user. If this isn't done, repo pushes will fail with very little detail provided in the output, so accepting the EULA is important. This can also be done with the REST API. The first task retrieves the EULA from the endpoint "/service/rest/v1/system/eula", returns the content, and registers it as a variable.

The next task then retrieves the variable, filters the disclaimer from the variable, and sends it back using the REST API with the "accepted" parameter set to "true". Getting the EULA uses a GET request, and accepting it uses a POST request.

Create Nexus role

- name: Create nexus role
  hosts: all
  vars_files:
    - project-vars
  tasks:

    - name:
      ansible.builtin.uri:
        url: "{{nexus_url}}/service/rest/v1/security/roles"
        method: POST
        url_username: admin
        url_password: "{{nexus_adminpass}}"
        force_basic_auth: true
        headers:
          Content-Type: application/json
        body_format: json
        body:
          id: "maven-snapshots-admin-role"
          name: "Maven Snapshots Admin Role"
          description: "Full access to Maven Snapshots"
          privileges:
            -  "nx-repository-view-maven2-maven-snapshots-*"
          roles: []
        status_code: 200
      no_log: false

This play uses the "/service/rest/v1/security/roles" API to create a role and assign a privilege to it. Roles must be created, but privileges are built into Nexus and must be assigned to the created roles. Since a Java snapshot will be pushed to the repo, the created role uses the "nx-repository-view-maven2-maven-snapshots-*" which assigns all repo view maven snapshot privileges to the user.

Create Nexus user

- name: Create new nexus user
  hosts: all
  vars_files:
    - project-vars
  tasks:

    - name:
      ansible.builtin.uri:
        url: "{{nexus_url}}/service/rest/v1/security/users"
        method: POST
        url_username: admin
        url_password: "{{nexus_adminpass}}"
        force_basic_auth: true
        headers:
          Content-Type: application/json
        body_format: json
        body:
          userId: "nexususer"
          firstName: "nexus"
          lastName: "user"
          emailAddress: "nexuser@nexus.user"
          password: "{{nexus_userpass}}"
          status: "active"
          roles:
            - "maven-snapshots-admin-role"
        status_code: 200
      no_log: false

This play uses the "/service/rest/v1/security/users" API to create a new user. The request body includes "userId", "firstName", "lastName", "emailAddress", "password", "status", and "roles".

Push JAR file to the repo.

- name: Push JAR file to repo
  hosts: all
  vars_files:
    - project-vars
  tasks:

# rest API does not support uploads to snapshot repos
# /<groupId>/<artifactId>/<version>/<artifactId>-<version>.<extension>
# body_format: json helps with the request size
# EULA must be accepted before pushing to repo

    - name: Push JAR file 
      ansible.builtin.uri:
        url: "{{nexus_url}}/repository/maven-snapshots/com/example/{{ artifact_name }}/{{ artifact_version }}/{{ artifact_name }}-{{ artifact_version }}.jar"
        method: PUT
        url_username: nexususer
        url_password: "{{nexus_userpass}}"
        force_basic_auth: true
        src: "{{ jar_path }}"
        body_format: json
        remote_src: true
        status_code: 201
      no_log: false

This play uploads the JAR file to the Nexus repository. The REST API does not support pushing to snapshot repos, so a direct request has to be made to the URL using the format "/repository/maven-snapshots/com/example/{{ artifact_name }}/{{ artifact_version }}/{{ artifact_name }}-{{ artifact_version }}.jar", using the PUT method. The "src" parameter contains the path to the JAR file, and "body_format: json" works for pushing large files.

Introduction

This project uses Terraform to provision an EKS cluster, uses Ansible to configure the cluster environment, deploys MYSQL using Helm, and deploys a test Java application that connects to the SQL server.

Checklist

• Use Terraform to provision an EKS cluster
• Use Ansible and Helm to provision and deploy MySQL to the cluster
• Use Ansible to deploy and connect the Java application to the MYSQL server

Deploy to EKS cluster with Terraform

vpc.tf — Networking configuration

provider "aws" {
    region = "us-east-1"
}

variable vpc_cidr_block {}
variable private_subnet_cidr_blocks {}
variable public_subnet_cidr_blocks {}
data "aws_availability_zones" "azs" {}

module "myapp-vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "6.7.2"

  name = "myapp-vpc"
  cidr = var.vpc_cidr_block
  private_subnets = var.private_subnet_cidr_blocks
  public_subnets = var.public_subnet_cidr_blocks
  azs = data.aws_availability_zones.azs.names

  enable_nat_gateway = true
  single_nat_gateway = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/myapp-eks-cluster" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/myapp-eks-cluster" = "shared"
    "kubernetes.io/role/elb" = 1


  }

  private_subnet_tags = {
    "kubernetes.io/cluster/myapp-eks-cluster" = "shared"
    "kubernetes.io/role/internal-elb" = 1

}
}

The networking configuration creates a vpc, and within the VPC specifies a cidr block, private and public subnets..

eks.tf — Cluster configuration

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.25.0"

   addons = {
    coredns                = {}
    eks-pod-identity-agent = {
      before_compute = true
    }
    kube-proxy             = {}
    vpc-cni                = {
      before_compute = true
    }
    aws-ebs-csi-driver = {}
  }
  
  name = "myapp-eks-cluster"
  kubernetes_version = "1.33"

  subnet_ids = module.myapp-vpc.private_subnets // outputs
  vpc_id = module.myapp-vpc.vpc_id # outputs

  endpoint_public_access = true
  enable_cluster_creator_admin_permissions = true


  eks_managed_node_groups = {
        dev = {  # name of the group
        ami_type       = "AL2023_x86_64_STANDARD"
        instance_types = ["t2.small"]

        min_size     = 1
        max_size     = 3
        desired_size = 3
        }
    }  

  tags = {
    environment = "development"
    applicaton = "myapp"
  }
}


module "aws_ebs_csi_pod_identity" {
source = "terraform-aws-modules/eks-pod-identity/aws"

name = "aws-ebs-csi"

attach_aws_ebs_csi_policy = true

associations = {
  this = {
    cluster_name = module.eks.cluster_name
    namespace = "kube-system"
    service_account = "ebs-csi-controller-sa"

  }

}

}


resource "null_resource" "configure_cluster" {

    triggers = {
        trigger_one = module.eks.cluster_name

    }

    provisioner "local-exec" {
        working_dir = "../ansible"
        command = "ansible-playbook ../ansible/deploy-kubernetes.yaml"
    }


}

In the eks.tf configuration file, the EKS cluster is assigned to a private subnet, endpoint public access is enabled, and a managed node group is created with a desired size of 3 nodes. A pod identity association is needed to allow the EBS CSI driver to create resources in AWS. It assigns EBS permissions via the “aws_ebs_csi” policy to the Kubernetes service account, which is created when the EBS CSI driver is installed in the cluster.

At the end of the configuration, a null resource is used along with a “local-exec” provisioner to run the Ansible playbook. The provisioner is set to trigger whenever the cluster name changes. Therefore, the provisioner will be triggered whenever the cluster is created with Terraform.

Using Ansible and Helm to deploy MYSQL to the cluster.

Update kubeconfig

- name: Update Kubeconfig
  hosts: localhost
  gather_facts: false
  vars_files:
    - project-vars
  tasks:

    - name: update kubeconfig
      ansible.builtin.shell: >
        aws eks update-kubeconfig
        --region us-east-1
        --name myapp-eks-cluster
        --kubeconfig "{{ kubeconfig_path }}"

This play uses the Ansible “shell” module to update kubeconfig once the cluster has been created, so subsequent deployments are made into the newly created cluster.

Install necessary packages

- name: Install Helm
  hosts: localhost
  tasks:
    
    - name: install Helm
      community.general.snap:
        name: helm
        state: present
        classic: true

The Ansible community snap module is used to install the Helm package using the snap package manager, so the Helm command is available to add repositories and install releases.

Deploy prerequisite manifest files to the cluster.

- name: Add configmap, secret, and storageclass to cluster
  hosts: localhost
  tasks:

    - name: Add configmap
      kubernetes.core.k8s:
        state: present
        src: ../kubernetes/configmap.yaml

    - name: Add mysql secret
      kubernetes.core.k8s:
        state: present
        src: ../kubernetes/secret.yaml

    - name: Add storageclass
      kubernetes.core.k8s:
        state: present
        src: ../kubernetes/gp3-storageclass.yaml

    - name: Create kubernetes Docker secret
      kubernetes.core.k8s:
        state: present
        src: ../kubernetes/secret-docker.yaml

The ConfigMap, Secret, StorageClass, and Docker secret resources need to be deployed so the correct configuration is available in the cluster before other resources like the MySQL server and Java application are deployed. The Kubernetes k8s module is used to deploy manifest files using the “state” and “src” parameters.

Helm tasks — Add MySQL repo, Nginx repo, and install MySQL release and ingress controller.

- name: Add Bitnami MySQL repo, Nginx Repo, install MySQL release and Nginx ingress controller
  hosts: localhost
  tasks:

    - name: Add Bitnami repo
      kubernetes.core.helm_repository:
        name: bitnami
        repo_url: https://charts.bitnami.com/bitnami

    - name: Add Nginx repo
      kubernetes.core.helm_repository:
        name: ingress-nginx
        repo_url: https://kubernetes.github.io/ingress-nginx

    - name: Install mysql release
      kubernetes.core.helm:
        name: mysql-release
        chart_ref: bitnami/mysql
        release_namespace: default
        values_files:
          - ../kubernetes/mysql-values.yaml

    - name: Install Nginx controller
      kubernetes.core.helm:
        name: ingress-nginx-controller
        chart_ref: ingress-nginx/ingress-nginx
        release_namespace: default

The Ansible “helm_repository” module is used to add the Bitnami MySQL repo and Nginx repo to the cluster, using the “name” and “repo_url” parameters. The Helm module is used to install the release using the name, chart reference, namespace, and a values file if needed.

Build the app, log in, and push to a private repository.

- name: Build the app and image
  hosts: localhost
  tasks: 

    - name: Build application
      command: "gradle build -p ../java-app"

 

- name: Login, build and push image to repo
  hosts: localhost
  vars_files:
    - project-vars
  tasks:

    - name: Authenticate to Docker
      community.docker.docker_login:
        registry_url: https://index.docker.io/v1/
        username: "{{ docker_user }}"
        password: "{{ docker_PAT }}"


    - name: Build an image and push it to a private repo
      community.docker.docker_image:
        build:
          path: ../java-app
        name: michael101/java-maven-app
        tag: kubernetes-app
        push: true
        source: build

The Ansible built-in command module is used to execute “gradle build” on the app. A JDK must first be installed, and for Amazon Linux, the SDK must be installed to install Gradle. Once the application is built, the community.docker docker_login module is used to log in to the private Docker repository, using the “registry_url”, “username”, and “password” parameters. The docker_image module is then used to build and push the image, using the “build,” “name”, “tag”, “push”, and “source” parameters.

Deploy Java app

- name: Deploy java app
  hosts: localhost
  tasks:

    - name: Deploy java app
      kubernetes.core.k8s:
        state: present
        src: ../kubernetes/java-deployment.yaml

The Java app manifest file is deployed into the cluster. The manifest file is configured with the required environment variables to connect to the SQL server obtained from the ConfigMap, and secrets that were deployed earlier.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-app
  namespace: default
  labels:
    app: custom-app
spec: 
  replicas: 2
  selector:
    matchLabels:
      app: custom-app
  template:
    metadata:
      labels:
        app: custom-app
    spec:
      imagePullSecrets:
      - name: docker-secret
      containers:
      - name: kubernetes-app
        image: michael101/java-maven-app:kubernetes-app
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
          - name: DB_USER
            valueFrom:
              configMapKeyRef:
                name: customapp-configmap
                key: DB_USER
          - name: DB_PWD
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: mysql-password
          - name: DB_NAME
            valueFrom:
              configMapKeyRef: 
                name: customapp-configmap
                key: DB_NAME
          - name: DB_SERVER
            valueFrom:
              configMapKeyRef:
                name: customapp-configmap
                key: DB_SERVER

---

apiVersion: v1
kind: Service
metadata:
  name: customapp-service
  namespace: default
spec:
  selector:
    app: custom-app
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

Deploy ingress controller

- name: Get external IP for load balancer and apply to ingress
  hosts: localhost
  tasks:

    - name: Get hostname for load balancer
      kubernetes.core.k8s_info:
        api_version: v1
        kind: Service
        namespace: default
        name: ingress-nginx-controller-controller 
      register: loadbalancer_hostname

    - name: Set ELB hostname
      ansible.builtin.set_fact:
        elb_hostname: "{{ loadbalancer_hostname['resources'][0]['status']['loadBalancer']['ingress'][0]['hostname'] }}"

    
    - name: Deploy ingress
      kubernetes.core.k8s:
        state: present
        definition: "{{ lookup('template', '../kubernetes/ingress-values.yaml') | from_yaml}}"

The kubernetes k8s_info module is used to get more information about a specified Kubernetes resource, using parameters like the “api_version”, “kind”, “namespace”, and “name”. In the above example, the module is used to gather information about the installed nginx controller and is registered to the “loadbalancer_hostname” variable. The hostname is then retrieved from that variable and set as an Ansible fact, so the information is available to other tasks in the play. In the “deploy ingress” task, the definition is used to dynamically set the load balancer hostname as a value in the ingress manifest file.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: customapp-ingress
    namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: "{{ elb_hostname }}"
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: customapp-service
              port:
                number: 8080

This project aims to create an EC2 instance, install and configure Docker, deploy an Nginx container, and set up a scheduled task to monitor application health.

GitHub Link — https://github.com/Michael2921/Automate-EC2-Docker-Boto3/

Requirements

• EC2 instance must be accessible externally. It must have a public IP.
• EC2 instance should allow SSH access for Docker installation and configuration
• EC2 instance should have a port open to access Nginx externally
• Import the key pair into AWS, and add it at instance creation
• Retrieve the instance ID and Public IP that will be used in later SSH connections.
• Install Docker
• Deploy and run Nginx
• Check the status of the Nginx container
• Monitor the application health using a scheduled task

Boto3 documentation for EC2 client — https://docs.aws.amazon.com/boto3/latest/reference/services/ec2.html

The Boto3 EC2 client has many methods (approximately 800). The methods that will be used for this project include:

run_instances() — creates the EC2 instance

describe_instances()-provides more details on instances

describe_instance_status()- provides details specifically on the status of instances.

import_key_pair()-imports a key pair from the local machine to AWS

authorize_security_group_ingress() — creates inbound security rules in the security group

Access EC2 instance externally.

The EC2 instance must be accessible externally, as there will be an external SSH connection to set up Docker and Nginx, and to access the web server later.

• The EC2 instance must be assigned a public IP address. This is already set by default on the default VPC.
• The VPC must have an internet gateway attached, and the subnet the EC2 instance runs in must have a route to the internet gateway (making it public). This allows traffic from the instance to the internet.

This is set up by default in the default VPC. This means an instance created in the default VPC gets a public IP address and can reach the internet via an internet gateway attached to the VPC.

• The security group associated with the VPC must allow inbound connections to ports that will be used (in this case, ports 22 and 8080)

This can be done using the “authorize_security_group_ingress()” method

create_security_group = ec2.authorize_security_group_ingress(
    GroupId='sg-0066ecb57ae1d22a3',
    IpPermissions=[
        {
            'IpProtocol': 'tcp',
            'FromPort': 22,
            'ToPort': 22,
            'IpRanges': [
                {
                    'Description': 'Security group created with Boto for inbound SSH access',
                    'CidrIp': '12.34.56.78/32'

                }
            ]
        },

{
            'IpProtocol': 'tcp',
            'FromPort': 8080,
            'ToPort': 8080,
            'IpRanges': [
                {
                    'Description': 'Security group created with Boto for inbound HTTP access',
                    'CidrIp': '12.34.56.78/32'

                }
            ]
        }
    ]
)

The method requires the security group ID, and the inbound rules are defined as dictionaries in the “IpPermissions” list. Running this adds the rules to the security group.

Import Key Pair.

A key pair needs to be loaded into the instance at the time of creation. For this to happen, the public key needs to be imported into the Key Pair section in AWS using “import_key_pair()”

ec2 = boto3.client("ec2")


file_location = 'C:\\Users\\user\\.ssh\\id_rsa.pub'

with open(file_location, 'r') as file:
    public_key_contents = file.read()

import_public_key = ec2.import_key_pair(
    KeyName='PublicKeyforEC2',
    PublicKeyMaterial=public_key_contents
)

print("Key pair is created")

An EC2 object is created. The file location on the local machine is specified with the “file_location” variable. “import_public_key()” reads the public key as a series of bytes. But instead of copying and pasting the public key, the public key contents can be read with file.read() and assigned to a variable. That variable is then assigned to “PublicKeyMaterial”. When executed, the public key pair is imported into AWS.

When creating the EC2 instance, the KeyName is specified as a parameter to add the public key.

created_instances = ec2.run_instances(
        ImageId = "ami-0d27e0fb3bac4d724",
        MinCount = 1,
        MaxCount = 1,
        KeyName='PublicKeyforEC2',
        InstanceType='t3.small'
)

Retrieve the Instance ID and Public IP address.

The Instance ID and Public IP address will be needed to perform further operations later. The instance ID will be used to retrieve the public IP, and the public IP will be needed for SSH access to install and configure Docker.

In Boto3, requests to the client have to be made using the syntax specified in the documentation. The response syntax is also specified in the documentation. The values in the request and response syntax usually have dictionaries in lists in dictionaries, and so on. So, learning how to access elements from lists or dictionaries is required.

Example of a request syntax in Boto3 “revoke_security_group_ingress()”

response = client.revoke_security_group_ingress(
    CidrIp='string',
    FromPort=123,
    GroupId='string',
    GroupName='string',
    IpPermissions=[
        {
            'IpProtocol': 'string',
            'FromPort': 123,
            'ToPort': 123,
            'UserIdGroupPairs': [
                {
                    'Description': 'string',
                    'UserId': 'string',
                    'GroupName': 'string',
                    'GroupId': 'string',
                    'VpcId': 'string',
                    'VpcPeeringConnectionId': 'string',
                    'PeeringStatus': 'string'
                },
            ],
            'IpRanges': [
                {
                    'Description': 'string',
                    'CidrIp': 'string'
                },
            ],
            'Ipv6Ranges': [
                {
                    'Description': 'string',
                    'CidrIpv6': 'string'
                },
            ],
            'PrefixListIds': [
                {
                    'Description': 'string',
                    'PrefixListId': 'string'
                },
            ]
        },
    ],
    IpProtocol='string',
    SourceSecurityGroupName='string',
    SourceSecurityGroupOwnerId='string',
    ToPort=123,
    SecurityGroupRuleIds=[
        'string',
    ],
    DryRun=True|False
)

Example of a response syntax in Boto3 “revoke_security_group_ingress()”

{
    'Return': True|False,
    'UnknownIpPermissions': [
        {
            'IpProtocol': 'string',
            'FromPort': 123,
            'ToPort': 123,
            'UserIdGroupPairs': [
                {
                    'Description': 'string',
                    'UserId': 'string',
                    'GroupName': 'string',
                    'GroupId': 'string',
                    'VpcId': 'string',
                    'VpcPeeringConnectionId': 'string',
                    'PeeringStatus': 'string'
                },
            ],
            'IpRanges': [
                {
                    'Description': 'string',
                    'CidrIp': 'string'
                },
            ],
            'Ipv6Ranges': [
                {
                    'Description': 'string',
                    'CidrIpv6': 'string'
                },
            ],
            'PrefixListIds': [
                {
                    'Description': 'string',
                    'PrefixListId': 'string'
                },
            ]
        },
    ],
    'RevokedSecurityGroupRules': [
        {
            'SecurityGroupRuleId': 'string',
            'GroupId': 'string',
            'IsEgress': True|False,
            'IpProtocol': 'string',
            'FromPort': 123,
            'ToPort': 123,
            'CidrIpv4': 'string',
            'CidrIpv6': 'string',
            'PrefixListId': 'string',
            'ReferencedGroupId': 'string',
            'Description': 'string'
        },
    ]
}

The Instance ID can be accessed using the “created_instances” variable from earlier

created_instances = ec2.run_instances(
        ImageId = "ami-0d27e0fb3bac4d724",
        MinCount = 1,
        MaxCount = 1,
        KeyName='PublicKeyforEC2',
        InstanceType='t3.small'
)

The variable is used to further filter down lists and dictionaries until arriving at the instance ID.

instance_id = created_instances['Instances'][0]['InstanceId']

The public IP can be accessed by running “describe_instances()”, while specifying the instance_id. The result of that is then filtered by the “Reservations”, “Instances”, and “PublicIpAddress” keys.

describe_instances_by_id = ec2.describe_instances(InstanceIds=[instance_id])
instance = describe_instances_by_id['Reservations'][0]['Instances'][0]
public_ipv4 = instance.get('PublicIpAddress')

Key pair is created
Security group is created
Instance is created, waiting for public ip
Instance ID is i-0270feb0d51f82b74
Public IP address is 44.195.42.215

Install Docker on the EC2 instance.

Now that inbound connections to the instance are confirmed and the necessary rules are created in the security group, Docker can be installed via SSH using the Paramiko Python library.

def ssh_access():

    ssh = paramiko.SSHClient()

    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    ssh.connect(public_ipv4, username='ec2-user', key_filename='C:\\Users\\user\\.ssh\\id_rsa')

    return ssh



def install_docker():

    ssh = ssh_access()

    print("Connecting to the server")

    print("Executing docker install commands...")

    commands_install_docker = [
        'sudo dnf install -y docker',
        'sudo systemctl start docker',
        'sudo usermod -aG docker ec2-user', # docker group is automatically created on install
        # docker group permissions will not be active in the existing SSH session. Close session and start a new one.



    ]

    for cmd in commands_install_docker:
        stdin, stdout, stderr = ssh.exec_command(cmd)
        stdout.channel.recv_exit_status()

        print(f"Command: {cmd}")
        print(stdout.read().decode())
        print(stderr.read().decode())

    ssh.close()

The function “ssh_access()” is created to abstract the code used to connect to the SSH client, add the host key, and connect to the EC2 instance using the private key.

The “install_docker()” method connects to the EC2 instance via SSH, runs the Docker installation commands, adds the default user to the group, and closes the SSH connection. The for loop executes each command, waits for it to finish, prints the command and its stdout or stderr, then moves to the next command. Once the user is added to the Docker group, the changes don’t take effect in the existing SSH session. The session must be closed and restarted before “ec2-user” can run Docker commands.

hecking status of instance...
Instance is running
Connecting to the server
Executing docker install commands...
Command: sudo dnf install -y docker
Waiting for process with pid 2000 to finish.
Amazon Linux 2023 Kernel Livepatch repository   435 kB/s |  89 kB     00:00    
Last metadata expiration check: 0:00:01 ago on Mon Oct 12 23:33:55 2026.
Dependencies resolved.
================================================================================
 Package                  Arch     Version                  Repository     Size
================================================================================
Installing:
 docker                   x86_64   25.0.16-1.amzn2023.0.4   amazonlinux    46 M
Installing dependencies:
 container-selinux        noarch   4:2.245.0-1.amzn2023     amazonlinux    58 k
 containerd               x86_64   2.2.7-1.amzn2023.0.1     amazonlinux    25 M
 iptables-libs            x86_64   1.8.8-3.amzn2023.0.2     amazonlinux   401 k
 iptables-nft             x86_64   1.8.8-3.amzn2023.0.2     amazonlinux   183 k
 libcgroup                x86_64   3.0-1.amzn2023.0.1       amazonlinux    75 k
 libnetfilter_conntrack   x86_64   1.0.8-2.amzn2023.0.2     amazonlinux    58 k
 libnfnetlink             x86_64   1.0.1-19.amzn2023.0.2    amazonlinux    30 k
 libnftnl                 x86_64   1.2.2-2.amzn2023.0.2     amazonlinux    84 k
 pigz                     x86_64   2.5-1.amzn2023.0.4       amazonlinux    83 k
 runc                     x86_64   1.3.6-1.amzn2023.0.1     amazonlinux   4.0 M

Deploy Nginx container.

The “install_and_run_nginx()” function initiates a connection to the EC2 instance via SSH, then pulls and runs an Nginx container in the background.

def install_and_run_nginx():

    ssh = ssh_access()

    print("Connecting to the server")

    print("Executing install and run nginx commands...")

    commands_install_nginx = [
        'docker run -p 8080:80 -d nginx:1.30.5',
        'docker ps'
    ]

    for cmd in commands_install_nginx:
        stdin, stdout, stderr = ssh.exec_command(cmd)
        stdout.channel.recv_exit_status()

        print(f"Command: {cmd}")
        print(stdout.read().decode())
        print(stderr.read().decode())

    ssh.close()

Connecting to the server
Executing install and run nginx commands...
Command: docker run -p 8080:80 -d nginx:1.30.5
0ca11cb46c56d38c2e461ec1ed32fe79f89d9411287c32e5d8a1c182761cdead

Unable to find image 'nginx:1.30.5' locally
1.30.5: Pulling from library/nginx
ecc510c1e359: Pulling fs layer
f4616bb1be1c: Pulling fs layer
f1a6a629b824: Pulling fs layer
3d774efa34e3: Pulling fs layer
cb44c3fe542b: Pulling fs layer
121907e00c1c: Pulling fs layer
220a6ac36368: Pulling fs layer
3d774efa34e3: Waiting
cb44c3fe542b: Waiting
121907e00c1c: Waiting
220a6ac36368: Waiting
f1a6a629b824: Verifying Checksum
f1a6a629b824: Download complete
3d774efa34e3: Verifying Checksum
3d774efa34e3: Download complete
ecc510c1e359: Verifying Checksum
ecc510c1e359: Download complete
f4616bb1be1c: Verifying Checksum
f4616bb1be1c: Download complete
cb44c3fe542b: Verifying Checksum
cb44c3fe542b: Download complete
220a6ac36368: Verifying Checksum
220a6ac36368: Download complete
121907e00c1c: Verifying Checksum
121907e00c1c: Download complete
ecc510c1e359: Pull complete
f4616bb1be1c: Pull complete
f1a6a629b824: Pull complete
3d774efa34e3: Pull complete
cb44c3fe542b: Pull complete
121907e00c1c: Pull complete
220a6ac36368: Pull complete
Digest: sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
Status: Downloaded newer image for nginx:1.30.5

Command: docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS                  PORTS                                   NAMES
0ca11cb46c56   nginx:1.30.5   "/docker-entrypoint.…"   1 second ago   Up Less than a second   0.0.0.0:8080->80/tcp, :::8080->80/tcp   jovial_maxwell

Connecting to the server
Executing install and run nginx commands...
Command: docker run -p 8080:80 -d nginx:1.30.5
0ca11cb46c56d38c2e461ec1ed32fe79f89d9411287c32e5d8a1c182761cdead

Unable to find image 'nginx:1.30.5' locally
1.30.5: Pulling from library/nginx
ecc510c1e359: Pulling fs layer
f4616bb1be1c: Pulling fs layer
f1a6a629b824: Pulling fs layer
3d774efa34e3: Pulling fs layer
cb44c3fe542b: Pulling fs layer
121907e00c1c: Pulling fs layer
220a6ac36368: Pulling fs layer
3d774efa34e3: Waiting
cb44c3fe542b: Waiting
121907e00c1c: Waiting
220a6ac36368: Waiting
f1a6a629b824: Verifying Checksum
f1a6a629b824: Download complete
3d774efa34e3: Verifying Checksum
3d774efa34e3: Download complete
ecc510c1e359: Verifying Checksum
ecc510c1e359: Download complete
f4616bb1be1c: Verifying Checksum
f4616bb1be1c: Download complete
cb44c3fe542b: Verifying Checksum
cb44c3fe542b: Download complete
220a6ac36368: Verifying Checksum
220a6ac36368: Download complete
121907e00c1c: Verifying Checksum
121907e00c1c: Download complete
ecc510c1e359: Pull complete
f4616bb1be1c: Pull complete
f1a6a629b824: Pull complete
3d774efa34e3: Pull complete
cb44c3fe542b: Pull complete
121907e00c1c: Pull complete
220a6ac36368: Pull complete
Digest: sha256:9bf97bd7714f5e24c1ccd545ecb9eb5435cb6d109c97cebb15e7e455e0239edb
Status: Downloaded newer image for nginx:1.30.5

Command: docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS                  PORTS                                   NAMES
0ca11cb46c56   nginx:1.30.5   "/docker-entrypoint.…"   1 second ago   Up Less than a second   0.0.0.0:8080->80/tcp, :::8080->80/tcp   jovial_maxwell

Status checks.

The EC2 instance must be running before these commands run; otherwise, they won’t work. The “check_instance_status()” function checks the status of the created instance. Then the while loop runs the “check_instance_function” and only executes the “install_docker” and “install_nginx” functions when the ec2 instance status == “running”

def check_instance_status():

    ec2_status = ec2.describe_instance_status(InstanceIds=instance_id)
    status = ec2_status['InstanceStatuses'][0]['InstanceState']['Name']
    return status


while True:
    print("Checking status of instance...")
    if check_instance_status() == "running":
        print("Instance is running")
        install_docker()
        install_and_run_nginx()
        break
    time.sleep(5)

Monitoring the application as a scheduled task.

A scheduled task checks the app's health every 5 seconds (response code == 200). A counter is declared. If the application is unhealthy (response code != 200) during a check, the counter increments by 1. If the counter reaches 5, the application is considered unhealthy, and the Nginx container is restarted.

def restart_container():

    ssh = ssh_access()

    stdin, stdout, stderr = ssh.exec_command("docker ps -q | head -n 1") # dynamically get the container ID
    container_id = stdout.read().decode().strip()

    stdin, stdout, stderr = ssh.exec_command(f"docker restart {container_id}") # starts container with dynamically retrieved container ID
    print(stdout.read().decode())
    print(stderr.read().decode())

    ssh.close()


check_counter = 0

# every 5 seconds, it checks for the status, for a total of 5 times (25 seconds) if 5 checks fail, it restarts the application.


def monitor_application():

    global check_counter

    response = requests.get(f'http://{public_ipv4}:8080')

    print("Monitoring application health...")
    if response.status_code != 200:
        print("Site is not reachable")
        check_counter += 1
        print(f"failed counter is {check_counter}")
        if check_counter == 5:
            print(f"failed counter is {check_counter}, restarting the container")
            restart_container()
            check_counter = 0


schedule.every(5).seconds.do(monitor_application)


while True:
    schedule.run_pending()

This project uses Terraform modules to provision an EKS cluster with a managed node group and Fargate profile. It deploys MYSQL, phpMyAdmin, and a custom Java application using Terraform and Jenkins in a CI/CD pipeline. It also creates a remote backend to store local Terraform state files.

GitHub Link — https://github.com/Michael2921/Terraform-EKS/tree/master

Create a Terraform folder in the project with the required files. Starting files include:

• vpc.tf — for the networking configuration
• providers.tf- includes provider information
• eks-cluster.tf — for the cluster configuration

Terraform modules

Using Terraform to deploy the VPC using the VPC Module

Terraform modules are reusable collections of Terraform resources and, like libraries or packages in programming languages, abstract complexity and extend Terraform provider features.

name, cidr, private_subnets, and public_subnets, enable_nat_gateway, and others are inputs in the Terraform EKS module. The enable_nat_gateway option is set to true to create a NAT gateway, so private subnets can reach the internet while the subnets themselves can't be reached from the internet.

The tags indicate that the private and public subnet resources can be shared among other Kubernetes clusters/resources and enable the public and private subnets to be used as internal and external load balancers.

Terraform Registry

The terraform. tfvars file assigns values to the variables defined in vpc.tf. It assigns values to the vpc_cidr_block, private_subnet_cidr_blocks, and public_subnet_cidr_blocks variables.

Using Terraform to deploy EKS using the EKS Module

The Terraform EKS module can be found here

Terraform Registry

Configuration:

The module installs needed add-ons like CoreDNS, the EKS Pod Identity agent, kube-proxy, and VPC CNI. “before_compute” means the add-ons are created before the compute resources are provisioned.

Endpoint public access is set to true so the cluster can be accessed locally (i.e with kubectl). "enable_cluster_creator_admin_permissions" sets the cluster creator as an admin.

The "eks_managed_node_groups" input sets the node group name, ami_type, instance_type, and node sizes.

Building the application, Docker image, and pushing to repo

EBS CSI Driver and MYSQL storage

To provision storage for the MySQL pods in AWS, the EBS CSI driver (Elastic Block Store Container Storage Interface) is required. This driver handles EBS storage tasks on AWS. To provision storage for MySQL on EKS, install the EBS CSI driver as an add-on and assign the appropriate policy.

Add the add-on to the Terraform configuration.

addons = {
    coredns = {}
    eks-pod-identity-agent = {
        before_compute = true
    }
    kube-proxy = {}
    vpc-cni = {
        before_compute = true
    }
    aws-ebs-csi-driver = {}

  }

Add the required policy to the node group.

iam_role_additional_policies = {
    AmazonEBSCSIDriverPolicy = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

Deploying MYSQL with Terraform

The Kubernetes and Helm providers will deploy MySQL with Terraform.

Two data sources are defined: "aws_eks_cluster" and "aws_eks_cluster_auth". The former retrieves the cluster endpoint, certificate authority, cluster name, and ARN. The other retrieves the auth token. The Kubernetes and Helm providers use these values to perform their functions.

Create a Kubernetes storage class for the MySQL pods to store their data; you can do this with Terraform using the "kubernetes_storage_class_v1" resource from the Kubernetes provider.

The "helm_release" resource from the Helm provider provides the information needed for Helm deployments.

Use "existingsecret" instead of hardcoding credentials in the Helm SQL values file. Create the root password, user password, and replication password in Jenkins.

And use it in the Jenkinsfile. "TF_VAR_ is used to define variables in Terraform"

Add AWS credentials as environment variables in the Jenkinsfile so Terraform can connect to AWS.

Add AWS credentials to Jenkins.

And reference them in the Jenkinsfile.

Terraform

Terraform init

Terraform plan

Use "terraform plan" to see a plan of what resources will be created/modified/destroyed.

The storage class will be created.

The VPC and all associated resources

Subnets

Add-ons

Nodegroup

Fargate profile

Deploying MYSQL pods with Terraform

Issues after running Terraform apply.

There were two issues according to the error output

• User cannot create storage classes
• Timeout while waiting for the EBS-CSI add-on to become active

Issues creating storage class

Running

kubectl auth can-i create storageclass

Outputs

So the user should be able to create storage classes.

This appeared to be due to a dependency issue and was resolved by adding a dependency in the storage class resource in mysql.tf to ensure the EKS cluster is created before the storage class is provisioned.

depends_on = [module.eks]

resource "kubernetes_storage_class_v1" "gp3" {
    depends_on = [module.eks] 
    metadata {
        name = "gp3"
    }

    storage_provisioner = "ebs.csi.aws.com"
    volume_binding_mode = "WaitForFirstConsumer"
    allow_volume_expansion = true
    reclaim_policy = "Delete"

    parameters = {
        type = "gp3"
        encrypted = "true"
        fsType = "ext4"
    }
}

After adding this dependency, "terraform apply" runs without issues.

EBS-CSI add-on unhealthy

Use

kubectl get pods -n kube-system 

To get a list of add-on pods

The ebs-csi-controllers have failed with a "CrashLoopBackOff" status

Running

kubectl logs ebs-csi-controller-799df95cb7-5pvbb -n kube-system

Gives output

This suggests there are no permissions, or permissions are insufficient, to start the EBS CSI add-on.

Terraform creates the aws-ebs-csi-driver add-on but doesn't automatically create the IAM role and pod identity association. Those can be created with the module "aws_ebs_csi_pod_identity"

  module "aws_ebs_csi_pod_identity" {
    source = "terraform-aws-modules/eks-pod-identity/aws"

    name = "aws-ebs-csi"

    attach_aws_ebs_csi_policy = true

    associations = {
      this = {
        cluster_name = module.eks.cluster_name
        namespace = "kube-system"
        service_account = "ebs-csi-controller-sa"
      }
    }
  }

After adding this module to the EKS cluster Terraform file and recreating the cluster, the EBS CSI drivers appear to run without issues.

Creating Kubernetes secrets with the Terraform Kubernetes Provider

Instead of creating the MySQL credentials in the Jenkinsfile with

kubectl create secret generic mysql-creds - from-literal=mysql-root-password=[root_password] - from-literal=mysql-password=[mysql_password] - from-literal=mysql-replication-password=[mysql_replication_password]

The credentials can be created during infrastructure provisioning with the Terraform Kubernetes provider

resource "kubernetes_secret_v1" "mysql_creds" {
    depends_on = [module.eks]
    metadata {
        name = "mysql-creds"
    }

    data = {
        mysql-root-password = var.mysql_root_password
        mysql-password = var.mysql_user_password
        mysql-replication-password = var.mysql_replication_password
    }

    type = "Opaque"
}

Volume permissions in the mysql helm_release resource give the MySQL pod the necessary read/write permissions when writing to the MySQL directory.

Volume permissions require the os-shell init container, which must be included in the mysql values file.

resource "helm_release" "mysql" {
    name = "mysql-release"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "mysql"
    version = "14.0.3" 
    timeout = 120

    // add values file here

    values = ["${file("helmvalues-mysql.yaml")}"]

    set = [{
        name = "volumePermissions.enabled"
        value = true
    }]

    depends_on = [
        kubernetes_storage_class_v1.gp3,
        kubernetes_secret_v1.mysql_creds
    ]


}

volumePermissions:
  enabled: true
  image:
    registry: docker.io
    repository: bitnamilegacy/os-shell
    tag: 12-debian-12-r50

Helm can install the mysql-release, but the pods fail

helm ls

kubectl get pod

kubectl describe pod mysql-release-primary-0

kubectl get pvc

The secondary MySQL pod has a PVC assigned and bound, but the primary pod does not.

kubectl describe pvc data-mysql-release-primary-0

The primary PVC does not have a storage class set.

Existing MySQL Helm custom values file

There is no storage class assigned to the primary

Making the necessary changes

After deleting the existing PVCs, uninstalling the release, and running "terraform apply" again, the issue is resolved.

Deploying PhpMyAdmin with the Helm provider in Terraform

Since the mysql.tf file already has the Helm provider and data source "data "aws_eks_cluster_auth" is declared, they don't have to be declared again in the phpmyadmin.tf file. The Terraform config file for phpmyadmin then looks like this. The “depends_on” block makes sure the storage class and mysql_creds secret are provisioned before phpmyadmin. The storage class is needed to deploy MySQL successfully, and the secret is needed for phpMyAdmin to connect to MySQL.

resource "helm_release" "phpmyadmin" {
    name = "phpmyadmin-release"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "phpmyadmin"
    version = "20.0.0" 
    timeout = 120


    values = ["${file("helmvalues-phpmyadmin.yaml")}"]

    depends_on = [
        kubernetes_storage_class_v1.gp3,
        kubernetes_secret_v1.mysql_creds
    ]


}

And the values file

db:
  host: "mysql-release-primary"
  port: 3306


service:
  type: ClusterIP
  ports:
    http: 80


global:
  security:
    allowInsecureImages: True

image: 
  registry: docker.io
  tag: latest
  repository: bitnamilegacy/phpmyadmin

When "terraform apply" runs from Jenkins, it creates the phpMyAdmin pod and service.

phpMyAdmin is running with a ClusterIP service, so it cannot be accessed externally. Port-forward the service to access it locally.

kubectl port-forward svc/phpmyadmin-release 8080:80

Log in with the "existing creds" secret type created with the resource "kubernetes_secret_v1" in mysql.tf and provided by Jenkins.

resource "kubernetes_secret_v1" "mysql_creds" {
    depends_on = [module.eks]
    metadata {
        name = "mysql-creds"
    }

    data = {
        mysql-root-password = var.mysql_root_password
        mysql-password = var.mysql_user_password
        mysql-replication-password = var.mysql_replication_password
    }

    type = "Opaque"
}

Deploying the Java app to a different namespace using Fargate

Pods will deploy to Fargate only if namespace=fargate, according to the eks-cluster.tf Terraform code

  fargate_profiles = {
    company-fargate = {
        selectors = [
            {
            namespace = "fargate"
        }
        ]


    }

  }

Create a new namespace resource with Terraform in the eks-cluster.tf file.

  resource "kubernetes_namespace" "fargate_namespace" {
    metadata {
      name = "fargate"
    }
  }

Edit the custom app deployment file to add the Fargate namespace so it deploys to Fargate.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: company-app
  namespace: fargate
  labels:
    app: company-app

Create a Docker secret in the companyapp.tf file so that Kubernetes can pull from the private Docker repository.

resource "kubernetes_secret_v1" "docker-creds" {
    depends_on = [module.eks]
    metadata {
        name = "docker-creds"
        namespace = "kubernetes_namespace.fargate_namespace.metadata[0].name"
    }

    type = "kubernetes.io/dockerconfigjson"

    data = {
        ".dockerconfigjson" = jsonencode({
            auths = {
                "https://index.docker.io/v1/" = {
                    username= var.docker_username
                    password = var.docker_PAT
                    auth = base64encode("${var.docker_username}:${var.docker_PAT}")
                }
            }
        })
    }


    
}

Create Docker credentials in Jenkins and add them to the Jenkinsfile.

Add the Fargate namespace, the secrets to pull from the repo, and the image name.

The "mysql-creds" secret also needs to be created in the Fargate namespace so the Fargate pods can use the MySQL credentials to connect to the database. "for_each" and "namespace" in the metadata block create the credentials in different namespaces.

resource "kubernetes_secret_v1" "mysql_creds" {
    depends_on = [module.eks, kubernetes_namespace_v1.fargate_namespace]
    for_each =  toset(["default", "fargate"])
    metadata {
        name = "mysql-creds"
        namespace = each.value
    }

    data = {
        mysql-root-password = var.mysql_root_password
        mysql-password = var.mysql_user_password
        mysql-replication-password = var.mysql_replication_password
    }

    type = "Opaque"
}

The Java application will use the ConfigMap to connect to the MYSQL Server in a different "default" namespace. The namespace must be specified for the PMA_HOST and DB_SERVER values. "mysql-release-primary" is the name of the service, "default" is the name of the namespace, and "svc.cluster.local" is appended at the end.

apiVersion: v1
kind: ConfigMap
metadata:
  name: companyapp-configmap
  namespace: fargate
data:
  PMA_HOST: mysql-release-primary.default.svc.cluster.local
  PMA_PORT: "3306"
  DB_USER: myuser
  DB_NAME: mydb
  DB_SERVER: mysql-release-primary.default.svc.cluster.local

Issues connecting Fargate Java pod to MySQL

After running "terraform apply" with the new configurations, the Fargate pods failed to connect to the MYSQL pods in the default namespace and ultimately failed.

Further troubleshooting revealed the MySQL security group (or worker node security group) did not allow connections to 3306 from the Fargate security group.

aws ec2 describe-security-groups --group-ids sg-03b6aed8c740303dd sg-0f663b0ae5b61b91b

Resolved this by adding a rule to allow connections to 3306 from the Fargate security group

aws ec2 authorize-security-group-ingress --group-id sg-03b6aed8c740303dd --protocol tcp --port 3306 --source-group sg-0f663b0ae5b61b91b

Fargate pods can now connect to MySQL pods using their IP address.

DNS issue

The Fargate pods can connect to the MYSQL pods with their IP address, but fail with DNS. This is important because the ConfigMap DB_SERVER references the MYSQL server in the default namespace with its hostname (mysql-release-primary.default.svc.cluster.local).

This issue is resolved by adding ingress rules to the worker node security group that allow UDP and TCP traffic on port 53 from the Fargate security group.

aws ec2 authorize-security-group-ingress --group-id sg-03b6aed8c740303dd --protocol udp --port 53 --source-group sg-0f663b0ae5b61b91b

aws ec2 authorize-security-group-ingress --group-id sg-03b6aed8c740303dd --protocol tcp --port 53 --source-group sg-0f663b0ae5b61b91b

Add these changes to the EKS Terraform configuration.

  resource "aws_vpc_security_group_ingress_rule" "fargate_to_mysql" { # allows fargate pods to access MySQL
  security_group_id            = module.eks.node_security_group_id # node security group
  referenced_security_group_id = module.eks.cluster_primary_security_group_id # cluster security group

  ip_protocol = "tcp"
  from_port   = 3306
  to_port     = 3306

   
}


resource "aws_vpc_security_group_ingress_rule" "fargate_to_node_dns_tcp" { #  allows fargate pods to access DNS over TCP
  security_group_id            = module.eks.node_security_group_id
  referenced_security_group_id = module.eks.cluster_primary_security_group_id

  ip_protocol = "tcp"
  from_port   = 53
  to_port     = 53

 
}

resource "aws_vpc_security_group_ingress_rule" "fargate_to_node_dns_udp" { #  allows fargate pods to access DNS over UDP
  security_group_id            = module.eks.node_security_group_id
  referenced_security_group_id = module.eks.cluster_primary_security_group_id

  ip_protocol = "udp"
  from_port   = 53
  to_port     = 53

  
}

After running "terraform apply" to apply the changes, the values inserted into the database from the Java app can be seen through phpMyAdmin

Splitting Terraform config files into different directories and creating a remote state.

I experienced many dependency issues while building this project. Terraform does not always know the order of creation (i.e., which resources should be created first), which can lead to a build failure. A way to resolve this is to create separate folders for the Terraform files that create the infrastructure, the files that provision the EKS cluster, and the pods that run in it. Then create a remote state in S3, and pass the data outputs from the infrastructure.tfstate to create the kubernetes resources.

Create the remote state.

Create an S3 bucket and reference it in the folder containing the infrastructure Terraform files.

terraform {
  required_version = ">= 0.12"
  backend "s3" {
    bucket = "terraform-eks20"
    key = "terraform/infrastructure.tfstate"
    region = "us-east-1"
  }
}

Project structure

Infrastructure folder

This contains all the Terraform code for provisioning the networking and EKS infrastructure.

vpc.tf

This file declares the remote state created in the S3 backend, and the "terraform.tf" state generated locally in the folder is sent to the remote backend.

“outputs.tf” creates outputs for values used when Kubernetes resources are created in the cluster.

output "cluster_name" {
    value = module.eks.cluster_name
}

output "cluster_endpoint" {
    value = module.eks.cluster_endpoint
}

output "cluster_certificate_authority_data" {
    value = module.eks.cluster_certificate_authority_data
    sensitive = true
}

eks-cluster.tf contains all the Terraform code for provisioning the EKS cluster.

Here is what the Jenkinsfile stage looks like

 stage('Provision Infrastructure (Networking + EKS)') { 

        environment {

            AWS_ACCESS_KEY_ID = credentials('jenkins_aws_access_key_id')
            AWS_SECRET_ACCESS_KEY = credentials ('jenkins_aws_secret_access_key')

        }
            steps {

                dir('terraform/infrastructure') {
               
                    echo "Provisioning Infrastructie"
                    sh 'terraform init'
                 //   sh 'terraform plan'
                    sh 'terraform apply --auto-approve'

                

                }
                
            }
        }

Kubernetes Folder

Contains the Terraform code for creating necessary Kubernetes resources that later pods will rely on. i,e storage class, secrets, namespaces

Remote-state.tf

Contains a call to the remote backend that stores the Terraform state from the infrastructure folder. Outputs in this state are used to create the Kubernetes resources with information from the already created infrastructure.

data "terraform_remote_state" "infrastructure" {
    backend = "s3"
    config = {
        bucket = "terraform-eks20"
        key = "terraform/infrastructure.tfstate"
        region = "us-east-1"
    }

}

Provider.tf

This contains the providers and data sources used to create resources in Kubernetes.

provider "aws" {
    region = "us-east-1"
}

provider "kubernetes" {
    host = data.terraform_remote_state.infrastructure.outputs.cluster_endpoint
    token = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.terraform_remote_state.infrastructure.outputs.cluster_certificate_authority_data)

}

data "aws_eks_cluster_auth" "cluster" {
    name = data.terraform_remote_state.infrastructure.outputs.cluster_name
}


}

The providers use the outputs from outputs.tf in the infrastructure folder to retrieve the cluster_endpoint and certification information. The token isn't generated from the remote state; it must be retrieved dynamically from the "aws_eks_cluster_auth" data source.

Here is what the Jenkinsfile stage looks like

stage('Provision Infrastructure (Networking + EKS)') { 

        environment {

            AWS_ACCESS_KEY_ID = credentials('jenkins_aws_access_key_id')
            AWS_SECRET_ACCESS_KEY = credentials ('jenkins_aws_secret_access_key')

        }
            steps {

                dir('terraform/infrastructure') {
               
                    echo "Provisioning Infrastructie"
                    sh 'terraform init'
                 //   sh 'terraform plan'
                    sh 'terraform apply --auto-approve'

                

                }
                
            }
        }

It reads the infrastructure remote state and data source

And creates the resources

kubectl get secret

kubectl get storageclass

kubectl get ns

kubectl get secret -n fargate

Helm folder

Configure a remote state for “kubernetes.tfstate” and reference it in the Helm folder to create MYSQL and phpMyAdmin pods.

In the Kubernetes folder, create a remote state with the key "terraform/kubernetes.tfstate" so the state file is stored there; in the Helm folder, create a remote-state.tf file that references the remote state created in the Kubernetes folder.

Kubernetes/kubernetes.tf file

terraform {
  required_version = ">= 0.12"
  backend "s3" {
    bucket = "terraform-eks20"
    key = "terraform/kubernetes.tfstate"
    region = "us-east-1"
  }
}

Helm/remote-state.tf file

data "terraform_remote_state" "kubernetes" {
    backend = "s3"
    config = {
        bucket = "terraform-eks20"
        key = "terraform/kubernetes.tfstate"
        region = "us-east-1"
    }

}

References the state file created in the Kubernetes folder

Run "terraform init" in Jenkins. It will ask if you want to copy the existing state to the new S3 backend.

Copy the Kubernetes folder local state file into the new remote backend. The following command is used to do that:

terraform init -migrate-state -force-copy

• migrate-state tells Terraform to move the local Kubernetes state into the new S3 backend
• force-copy answers "yes" to the Terraform prompt

Use "terraform state list" to see what resources will be copied to the remote backend.

The S3 bucket in AWS now contains the infrastructure state and the Kubernetes state.

Create an outputs.tf file in the Kubernetes folder to output the resources needed to deploy the MySQL pods successfully, i.e., mysql_creds and the storage class.

output "gp3_storageclass" {
    value = kubernetes_storage_class_v1.gp3.metadata[0].name

}

output "mysql_secrets" {
    value = {
        for namespace, secret in kubernetes_secret_v1.mysql_creds :
        namespace => secret.metadata[0].name
    }

}

The Helm folder would then use the newly created output values to retrieve the storage class and mysql_creds values.

But there is already a custom values YAML file for MySQL and phpMyAdmin. All the needed values are defined there instead.

architecture: replication
auth:
  database: mydb
  username: myuser
  existingSecret: mysql-creds

primary:
  persistence:
    enabled: true
    storageClass: gp3
    accessModes: ["ReadWriteOnce"]
    size: 10Gi


secondary:
  replicaCount: 2
  persistence:
    accessModes: ["ReadWriteOnce"]
    enabled: true
    storageClass: gp3
    size: 10Gi


global:
  security:
    allowInsecureImages: True

image: 
  registry: docker.io
  tag: latest
  repository: bitnamilegacy/mysql

volumePermissions:
  enabled: true
  image:
    registry: docker.io
    repository: bitnamilegacy/os-shell
    tag: 12-debian-12-r50

resource "helm_release" "phpmyadmin" {
    name = "phpmyadmin-release"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "phpmyadmin"
    version = "20.0.0" 
    timeout = 120


    values = ["${file("helmvalues-phpmyadmin.yaml")}"]


}

So the mysql.tf file will look like this

resource "helm_release" "mysql" {
    name = "mysql-release"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "mysql"
    version = "14.0.3" 
    timeout = 120

    // add values file here

    values = ["${file("helmvalues-mysql.yaml")}"]

    set = [{
        name = "volumePermissions.enabled"
        value = true
    }]

    


}

and phpmyadmin.tf will look like this

resource "helm_release" "phpmyadmin" {
    name = "phpmyadmin-release"
    repository = "https://charts.bitnami.com/bitnami"
    chart = "phpmyadmin"
    version = "20.0.0" 
    timeout = 120


    values = ["${file("helmvalues-phpmyadmin.yaml")}"]


}

After running "terraform apply" and deploying the two releases

Run

helm ls

To view the releases

kubectl get pod

To view the pods

All pods appear to be running without issues

Test access to MYSQL through phpMyAdmin

phpMyAdmin can connect to MySQL and uses the configured credentials

kubectl get pvc

All MYSQL pods have PVCs bound to them with the correct storage class

To save the Helm Terraform state file to the S3 backend, include the code in one of the Terraform files in the Helm folder, and use a key that doesn’t conflict with other remote states.

terraform {
  required_version = ">= 0.12"
  backend "s3" {
    bucket = "terraform-eks20"
    key = "terraform/helm.tfstate"
    region = "us-east-1"
  }
}

After running "terraform init -migrate-state -force-copy", a new remote backend will be created in S3 with the Helm Terraform local state file.

Deploy custom app to Fargate in a different namespace

Project structure

The necessary providers should be added in provider.tf

provider "aws" {
    region = "us-east-1"
}

provider "kubernetes" {
    host = data.terraform_remote_state.infrastructure.outputs.cluster_endpoint
    token = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.terraform_remote_state.infrastructure.outputs.cluster_certificate_authority_data)

}

data "aws_eks_cluster_auth" "cluster" {
    name = data.terraform_remote_state.infrastructure.outputs.cluster_name
}

remote-state.tf file should import the necessary state file from the remote backend

data "terraform_remote_state" "infrastructure" {
    backend = "s3"
    config = {
        bucket = "terraform-eks20"
        key = "terraform/infrastructure.tfstate"
        region = "us-east-1"
    }

}

companyapp.tf creates the Docker creds secret, then deploys the configmap, pods, and service using their manifest files

variable "docker_username" {
  type = string
}

variable "docker_PAT" {
  type = string
}


resource "kubernetes_secret_v1" "docker-creds" {
    metadata {
        name = "docker-creds"
        namespace = "fargate"
    }

    type = "kubernetes.io/dockerconfigjson"

    data = {
        ".dockerconfigjson" = jsonencode({
            auths = {
                "https://index.docker.io/v1/" = {
                    username = var.docker_username
                    password = var.docker_PAT
                    auth = base64encode("${var.docker_username}:${var.docker_PAT}")
                }
            }
        })
    }


    
}


resource "kubernetes_manifest" "companyapp_configmap" {
  manifest = yamldecode(
    file("companyapp-configmap.yaml")
  )
}


resource "kubernetes_manifest" "companyapp_deployment" {
  depends_on = [kubernetes_manifest.companyapp_configmap]
  manifest = yamldecode(
    file("companyapp-deployment.yaml")
  )
}

resource "kubernetes_manifest" "companyapp_service" {
  depends_on = [kubernetes_manifest.companyapp_configmap]
  manifest = yamldecode(
    file("companyapp-service.yaml")
  )
}

Configmap manifest file

apiVersion: v1
kind: ConfigMap
metadata:
  name: companyapp-configmap
  namespace: fargate
data:
  PMA_HOST: mysql-release-primary.default.svc.cluster.local
  PMA_PORT: "3306"
  DB_USER: myuser
  DB_NAME: mydb
  DB_SERVER: mysql-release-primary.default.svc.cluster.local

Deployment manifest file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: company-app
  namespace: fargate
  labels:
    app: company-app
spec: 
  replicas: 2
  selector:
    matchLabels:
      app: company-app
  template:
    metadata:
      labels:
        app: company-app
    spec:
      imagePullSecrets:
      - name: docker-creds
      containers:
      - name: company-app
        image: michael101/java-maven-app:company-app-1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
          - name: DB_USER
            valueFrom:
              configMapKeyRef:
                name: companyapp-configmap
                key: DB_USER
          - name: DB_PWD
            valueFrom:
              secretKeyRef:
                name: mysql-creds
                key: mysql-password
          - name: DB_NAME
            valueFrom:
              configMapKeyRef: 
                name: companyapp-configmap
                key: DB_NAME
          - name: DB_SERVER
            valueFrom:
              configMapKeyRef:
                name: companyapp-configmap
                key: DB_SERVER 

Service manifest file

apiVersion: v1
kind: Service
metadata:
  name: companyapp-service
  namespace: fargate
spec:
  type: LoadBalancer
  selector:
    app: company-app
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

After applying the configuration with "terraform apply"

kubectl get configmap -n fargate

kubectl get svc -n fargate

kubectl get secret -n fargate

kubectl get pod -n fargate

kubectl get pods -A -o wide

phpMyAdmin shows the app inserting test data into the database, meaning the company application deployed in Fargate at a different namespace can communicate with the MYSQL pods in the default namespace

Create a new remote backend with the companyapp local state file

In the companyapp.tf file, add the remote backend configuration to the file

terraform {
  required_version = ">= 0.12"
  backend "s3" {
    bucket = "terraform-eks20"
    key = "terraform/companyapp.tfstate"
    region = "us-east-1"
  }
}

Then, in the Jenkinsfile, run

terraform init -migrate-state -force-copy

This project demonstrates how to deploy multiple applications (phpmyadmin, MySQL, Java app) using a continuous deployment pipeline with Jenkins. It also covers role creation, autoscaling, and Fargate for AWS-managed nodes.

Tools used

eksctl, Amazon EKS, Kubernetes, Amazon ECR, Kubernetes, Fargate, Docker, Jenkins, Git

Create an EKS cluster

eksctl create cluster --name company-cluster --region us-east-1 --nodegroup-name company-nodegroup --node-type t2.micro --nodes 2 --nodes-min 2 --nodes-max 2

eksctl creates 2 availability zones, us-east-1a and us-east-1f, and 4 subnets: 2 public subnets and 2 private subnets. Each availability zone is assigned a public and private subnet. It also creates add-ons like vpc-cni, kube-proxy, and coredns.

EKS cluster in the AWS Management Console

Here is alist of add-ons that have been installed. They can be viewed from the kube-system namespace.

The Compute tab shows the EC2 instances that have been created and the node group.

The Networking tab shows the created VPC and subnets

The Add-ons tab shows the list of add-ons that have been installed

IAM

eksctl creates certain roles when the create cluster command is executed.

• EKS cluster IAM roles
• EC2 node IAM roles
• EKS service-linked roles

EKS cluster IAM roles

EKS assumes this role to interact with AWS resources and perform actions on services defined in the policy (AmazonEKSClusterPolicy) above.

EC2 node IAM roles

EC2 worker nodes assume this role to pull containers out of ECR, assign private IP addresses, attach, create, and delete network interfaces, and describe instances, instance types, and route tables, and so on.

EKS serivce-linked roles

These service-linked roles are created by AWS, with various policies attached. For example, the AWSServiceRoleForAmazonEKS role is attached with a policy that gives it the permissions to create security groups, create network interface permissions, and modify network interface attributes.

Other service-linked roles created by AWSinclude AWSServiceRoleForAmazonEKSNodegroup (for the eks-nodegroup), and AWSServiceRoleForAutoScaling (for the autoscaling service)

Deployment

Three containers need to be deployed. Java app, phpMyAdmin, and 2 MYSQL replicas.

The deployment feature will be tested with phpMyAdmin, as it’s the easiest to deploy and track. This can be used to eliminate and resolve easy-to-fix issues (e.g., access issues), so the more difficult ones (like MySQL deployment) can be dealt with a little easier.

External access is tested with the Elastic load balancer to see if phpMyAdmin is deployed correctly, and can be accessed externally.

Using Jenkins, kubectl needs

• The kubeconfig file
• AWS secret ID and key

To connect to EKS

The Jenkins container usually will not have a file editor (like nano or vi/m), so a config file has to be created from the host and copied into the container.

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <certificate-data>
    server: <endpoint-url>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: /usr/local/bin/aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - <cluster-name>

The certificate data, cluster-name, and endpoint-url have to be filled out in the config file

The Jenkinsfile should already have the AWS secret ID and key set as environment variables.

The file can be copied from the host to the Jenkins container using:

docker cp config [container id or name]:/var/jenkins_home/.kube

Kubectl has to be installed on the Jenkins container

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl; chmod +x ./kubectl; mv ./kubectl /usr/local/bin/kubectl

aws-iam-authenticator also needs to be installed on the Jenkins container

curl -Lo aws-iam-authenticator https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.6.11/aws-iam-authenticator_0.6.11_linux_amd64
chmod +x ./aws-iam-authenticator
mv ./aws-iam-authenticator /usr/local/bin

Deploying phpmyadmin with Jenkins

As explained, a simple phpmyadmin deployment was created to confirm that pods can be deployed and accessed externally.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: phpmyadmin
  labels:
    app: phpmyadmin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: phpmyadmin
  template:
    metadata:
      labels:
        app: phpmyadmin
    spec:
      containers:
      - name: phpmyadmin
        image: phpmyadmin
        ports:
        - containerPort: 80
        env:
        - name: PMA_HOST
          valueFrom:
            configMapKeyRef:
              name: companyapp-configmap
              key: PMA_HOST
        - name: PMA_PORT
          valueFrom:
            configMapKeyRef:
              name: companyapp-configmap
              key: PMA_PORT

---

apiVersion: v1
kind: Service
metadata:
  name: phpmyadmin-service
spec:
  type: LoadBalancer
  selector:
    app: phpmyadmin
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 80

Deployed phpmyadmin pod and service

phpmyadmin is accessible externally

Deploy stage for phpmyadmin in the Jenkinsfile

stage('Deploy') {
            environment {
                AWS_ACCESS_KEY_ID = credentials('jenkins_aws_access_key_id')
                AWS_SECRET_ACCESS_KEY = credentials('jenkins_aws_secret_access_key')
            }
            steps {
                script {
                    echo 'Deploying image...'
                    sh 'envsubst < eks-deployment/phpmyadmin-deployment.yaml | kubectl apply -f -'
                }
             
            }
        }

Installing Helm in the Jenkins container

Enter the Jenkins container as root with the following command

docker exec -it -u 0 [container name or id] /bin/bash

Run the following commands to install Helm in the container

$ curl -LO https://git.io/get_helm.sh
$ chmod 700 get_helm.sh
$ ./get_helm.sh

MYSQL persistence and storage

Since MySQL will be deployed on AWS, Elastic Block Storage (EBS) will be used to provide persistence for the MySQL pods. The aws-ebs-csi driver needs to be installed as an add-on to use EBS.

eksctl create addon --cluster company-cluster --name aws-ebs-csi-driver

Verify it is installed with

eksctl get addon --cluster company-cluster
kubectl get csidriver
aws eks list-addons --cluster-name company-cluster

I had issues installing this add-on, because there wasn’t enough capacity on the nodes. This made sense, as I was using a t2.micro instance type, and already had a lot of add-ons.

aws eks describe-addon --cluster-name company-cluster --addon-name aws-ebs-csi-driver
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver -o wide

{
    "addon": {
        "addonName": "aws-ebs-csi-driver",
        "clusterName": "company-cluster",
        "status": "UPDATING",
        "addonVersion": "v1.65.0-eksbuild.1",
        "health": {
            "issues": [
                {
                    "code": "InsufficientNumberOfReplicas",
                    "message": "The add-on is unhealthy because all deployments have all pods unscheduled 0/3 nodes are available: 3 Too many pods. no new claims to deallocate, preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod."
                }
            ]
        },
        "addonArn": "arn:aws:eks:us-east-1:420392929933:addon/company-cluster/aws-ebs-csi-driver/56d02efe-8bb8-c62f-c11e-ff71fdec99b1",
        "createdAt": "2026-09-01T12:21:02.974000-05:00",
        "modifiedAt": "2026-09-01T14:01:18.794000-05:00",
        "tags": {},
        "namespaceConfig": {
            "namespace": "kube-system"
        }
    }
}

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  16m                  default-scheduler  0/4 nodes are available: 1 Insufficient memory, 4 Too many pods. no new claims to deallocate, preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.
  Warning  FailedScheduling  100s (x11 over 16m)  default-scheduler  0/4 nodes are available: 4 Too many pods. no new claims to deallocate, preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.

4 pods appear to be the maximum capacity for t2.micro

kubectl get nodes -o custom-columns=NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,PODS:.status.capacity.pods

NAME                             TYPE       PODS
ip-192-168-17-215.ec2.internal   t2.micro   4
ip-192-168-23-13.ec2.internal    t2.micro   4
ip-192-168-37-62.ec2.internal    t2.micro   4
ip-192-168-5-48.ec2.internal     t2.micro   4
ip-192-168-52-207.ec2.internal   t2.micro   4

Fix 1: Create a new node group with a bigger instance type

eksctl create nodegroup --cluster company-cluster --name bigger-nodegroup --node-type t3.medium --nodes 3 --nodes-min 3 --nodes-max 3

The t3.medium instance type has maximum capacity of 17 pods.

I had 2 major issues after the instance type was switched to t3 medium

• The ebs-csi driver did not have the correct permissions
• The values in the mysql values were not entered in the correct format.

Fix 2: Create a pod identity association to associate the ebs-csi service account with an IAM role that has the AmazonEBSCSIDriverPolicy

eksctl create podidentityassociation --cluster company-cluster --namespace kube-system --service-account-name ebs-csi-controller-sa --role-name AmazonEKS_EBS_CSI_DriverRole --permission-policy-arns arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

Create a gp3 StorageClass configuration and apply it

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"
  fsType: ext4
reclaimPolicy: Delete

Fix 3: Filling out the MYSQL values file correctly

Values were entered in the MYSQL values yaml file in a way that was inconsistent with the MYSQL Bitnami chart. I resolved this by checking the manifest file to determine the exact structure of how values needed to be entered in the file

helm get values mysql -n default -a

Working StorageClass configuration

Working values file for MYSQL values file

Connecting phpMyAdmin to MySQL

I created a ConfigMap for sharing environment variables that will be used by phpmyadmin and company-app

Added environment variables to phpMyAdmin using the ConfigMap

And made changes to the Jenkinsfile

stage('Deploy') {
            environment {
                AWS_ACCESS_KEY_ID = credentials('jenkins_aws_access_key_id')
                AWS_SECRET_ACCESS_KEY = credentials('jenkins_aws_secret_access_key')
            }
            steps {
                script {
                    echo 'Deploying the configmap'
                    sh 'envsubst < eks-deployment/companyapp-configmap.yaml | kubectl apply -f -'                    
                   // echo 'Deploying mysql...'
                   // sh 'helm repo add bitnami https://charts.bitnami.com/bitnami' // add repo first
                   // sh 'helm install mysql --values eks-deployment/helmvalues-mysql.yaml bitnami/mysql' // install mysql from added repo
                    echo 'Deploying phpmyadmin'
                    sh 'envsubst < eks-deployment/phpmyadmin-deployment.yaml | kubectl apply -f -'
                }
             
            }
        }

Commented out the already deployed MySQL pods and added new lines for deploying the ConfigMap and phpMyAdmin.

As can be seen above, the database and phpMyAdmin seem to be communicating with each other.

Building the App

Now that phpmyadmin is able to communicate with the MYSQL server, the app and docker image will be built sent to ECR and deployed to EKS, so it can talk to other containers in the cluster and perform its functions (creating a table and adding values on the MYSQL server)

The application is built with “gradle build”. For this to work, gradle and jdk must be setup in the Tools section in Jenkins

Gradle can be installed automatically and the name should be added to the tools section in Jenkins

 tools {
            gradle 'gradle-9.8.0'
            jdk 'jdk-21'
     }

JDK can be added manually, if it is already installed in the Jenkins container. Use “which java” to find the JDK path in the container, then add it to Jenkins.

Building the Image and pushing to ECR

Contents of Dockerfile

Build the Dockerfile using:

docker build -t [IMAGE_NAME].

Log in to AWS ECR using the AWS CLI

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 420392929933.dkr.ecr.us-east-1.amazonaws.com

Tag the image with the ECR repo name

docker tag company-app:1.0 420392929933.dkr.ecr.us-east-1.amazonaws.com/company-app:1.0

Push the tagged image to ECR

docker push 420392929933.dkr.ecr.us-east-1.amazonaws.com/company-app:1.0

The image is successfully pushed to ECR

The build section looks like this in the Jenkinsfile

stage('Build Image') {
            steps {
                script {
                echo "Building the docker image"
                withCredentials([usernamePassword(credentialsId: 'ecr-credentials', passwordVariable: 'PASS', usernameVariable: 'USER')]){
                   sh "docker build -t ${DOCKER_REPO_SERVER}:${IMAGE_NAME} ."
                   sh "echo $PASS | docker login -u $USER --password-stdin ${DOCKER_REPO_SERVER} "
                   sh "docker push ${DOCKER_REPO_SERVER}:${IMAGE_NAME} "
                }
                }
       
            }
        }

The Java application is deployed to Kubernetes from the Jenkinsfile

stage('Deploy') {
            environment {
                AWS_ACCESS_KEY_ID = credentials('jenkins_aws_access_key_id')
                AWS_SECRET_ACCESS_KEY = credentials('jenkins_aws_secret_access_key')
            }
            steps {
                script {
                    echo 'Deploying the configmap'
                    sh 'envsubst < eks-deployment/companyapp-configmap.yaml | kubectl apply -f -'                    
                   // echo 'Deploying mysql...'
                   // sh 'helm repo add bitnami https://charts.bitnami.com/bitnami' // add repo first
                   // sh 'helm install mysql --values eks-deployment/helmvalues-mysql.yaml bitnami/mysql' // install mysql from added repo
                    echo 'Deploying phpmyadmin'
                    sh 'envsubst < eks-deployment/phpmyadmin-deployment.yaml | kubectl apply -f -'
                    echo 'Deploying companyapp'
                    sh 'envsubst < eks-deployment/companyapp-deployment.yaml | kubectl apply -f -'
                }
             
            }
        }

The application created a table and inserted values into it.

Checking with “kubectl get pod”

All pods appear to be running without issues

Setting up Autoscaling

The autoscaling feature automatically scales the number of nodes up or down based on the resources available to run the pods created.

The autoscaler needs access to create and delete EC2 instances on AWS

Create a custom policy on IAM

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeLaunchTemplateVersions"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

The autoscaler can use the EC2 service to list resources and use EC2 Auto Scaling to list and write resources.

Set up a connection between AWS IAM and the cluster

Create a new identity provider using the OpenID Connect provider URL

Create a role from the provider and associate the custom policy to it.

Add the custom-created ClusterAutoScaler policy and create the role

Editing the cluster autoscaler config file

The cluster autoscaler components need the tags to detect and communicate with the autoscaling group in AWS.

The cluster autoscaler deployment can be found here:

https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

The cluster name is added to the “command” section, and the tags can also be seen here

Add the created role to the autoscaler service account

Add the safe-to-evict: “false” tag, so the Autoscaler doesn’t accidentally evict itself when scaling down nodes.

Change the container image version to the version in the cluster

And the tag corresponding to the version here:

Tags · kubernetes/autoscaler

Add the region

Apply the config file with

kubectl apply -f cluster-autoscaler.yaml

Check the status of the autoscaler

kubectl get deployment -n kube-system cluster-autoscaler

List all the pods deployed on kube-system

kubectl get pod -n kube-system

Check out the logs for the autoscaler using:

kubectl logs cluster-autoscaler-686d9698f-sc8g4 -n kube-system

The autoscaler isn’t scaling down any nodes because the minimum, desired, and max nodes are the same: 3

I will change the minimum nodes to 1, desired nodes to 2, and maximum to 3 to test the autoscaler feature.

Checking the logs again

• The autoscaler could not evict the first node, as it was running the autoscaler, and we set a flag earlier for the autoscaler not to delete the node it is running on
• It began a removal simulation for a different node and moved existing pods to a different node
• It then removed the node
• It removed another node after moving the pods

After running “kubectl get node”, there is one node left. 2 nodes have been removed.

Configuring Fargate

Create a Fargate role for the “EKS Fargate pod” service

The role requires the “AmazonEKSFargatePodExecutionRolePolicy”

Configuring Fargate profile with eksctl

Fargate manages pods on the EC2 instances that AWS creates and manages. The instances are not visible in the customer’s AWS account. This feature is best used for deploying stateless applications. Stateful applications are better deployed using a nodegroup.

Create a Fargate profile with eksctl

To create a Fargate profile, selectors must be chosen to distinguish which pods will be deployed with Fargate.

Examples of pod selectors include namespaces and labels.

In our application, the Java app will be deployed using Fargate. It will be deployed in the default namespace, but labels will be added to make them identifiable for Fargate deployment.

The eksctl command will look like below:

eksctl create fargateprofile --cluster company-cluster --name company-fargate --namespace default --labels profile=fargate

This command will also create the necessary roles and policies required for Fargate to function.

The Fargate profile is created with the appropriate selector

After running “kubectl get node”

And “kubectl get pod”

2 Fargate instances were provisioned, one for each pod.

This project deploys a custom Java application, phpMyAdmin, and MySQL containers to a Kubernetes cluster on Linode Kubernetes Engine (LKE). The application connects to the MySQL container using environment variables, creates a table, and adds test data to it.

Tools Used:

Kubernetes, Linode Kubernetes Engine, Helm, Helmfile, Helm chart, Docker, Amazon ECR

Things to do:

• Create a Kubernetes cluster on LKE and connect to it from the command line
• Understand the application logic
• Create secrets to pull the image from Amazon ECR, and for MYSQL (includes user password, root password, and replication password)
• Create a ConfigMap to store non-sensitive data so other applications can reuse it
• Create a custom values file for MYSQL, and deploy with Helm
• Create deployment and service manifest files for PHPMyAdmin, and include the necessary environment variables
• Create deployment and service manifest files for the Java app, include necessary environment variables and secrets to access the database.
• Use Helm to deploy an ingress controller.
• Use an ingress rule to set the hostname, path, and service that is exposed externally.
• Create a chart using Helm. Create template files for deployment, service, configmap, and ingress.
• Create a default values file
• Create a custom values file for the application
• Use Helm to deploy the application
• Create a Helmfile and use it to deploy the application
• Port-forward phpMyAdmin for internal access

Create a Kubernetes cluster on LKE.

Provisioning

Download the LKE config file.

Set the KUBECONFIG variable to the LKE config file to connect to the cluster.

2. Understand the application logic

Databaseconfig.java

package com.example;

import com.mysql.cj.jdbc.MysqlDataSource;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.sql.Connection;
import java.sql.SQLException;


@Configuration
public class DatabaseConfig {

    private String user = System.getenv("DB_USER");
    private String password = System.getenv("DB_PWD");
    private String serverName = System.getenv("DB_SERVER");
    private String dbName = System.getenv("DB_NAME");
    private MysqlDataSource datasource = new MysqlDataSource();

    public DatabaseConfig() {
        datasource.setPassword(password);
        datasource.setUser(user);
        datasource.setServerName(serverName);
        datasource.setDatabaseName(dbName);


    }

    @Bean
    public Connection getConnection() throws SQLException {
        return datasource.getConnection();
    }
}

Strings “user”, “password”, “serverName”, and “dbName” are initialized and assigned to environment variables that will have already been set on the system. So the container running the application has to have those environment variables set.

package com.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import javax.annotation.PostConstruct;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

@SpringBootApplication
public class Application {

    private final Connection dbConnection;

    public Application(Connection dbConnection) {
        this.dbConnection = dbConnection;
    }

    public static void main(String[] args)
    {
        SpringApplication.run(Application.class, args);
    }

    @PostConstruct
    public void init()
    {
        Logger log = LoggerFactory.getLogger(Application.class);
        log.info("Java app started");

        try {
            Statement stmt = dbConnection.createStatement();
            createTable(stmt);
            generateData(stmt);
            stmt.close();
        } catch (SQLException e) {
            e.printStackTrace();
        }

    }

    private void createTable(Statement stmt) throws SQLException {
        String sqlStatement = "CREATE TABLE IF NOT EXISTS team_members(" +
                "member_id INT AUTO_INCREMENT PRIMARY KEY,\n" +
                "member_name VARCHAR(255),\n" +
                "member_role VARCHAR(255),\n" +
                "member_projects VARCHAR(255)" +
                ")";
        stmt.executeUpdate(sqlStatement);
    }

    private void generateData(Statement stmt) throws SQLException {
        String sqlQuery = "SELECT member_name, member_role FROM team_members";
        ResultSet resultSet = stmt.executeQuery(sqlQuery);
        if (!resultSet.next()) {
            String sqlStatement = "INSERT INTO team_members(member_name, member_role)\n" +
                    "VALUES ('Sarah', 'Full stack developer'),\n" +
                    "('Bobby', 'React developer'),\n" +
                    "('Ari', 'Java developer'),\n" +
                    "('Andrea', 'DevOps engineer'),\n" +
                    "('Bruno', 'IT operations')";
            stmt.executeUpdate(sqlStatement);
        }
    }

    public String getStatus() {
        return "OK";
    }
}

The createTable method creates a table named “team_members”, and generateData inserts data into it.

“gradle test” runs tests

“gradle build” builds the jar file

The JAR file is built in the build/libs folder.

Build the image and push it to Amazon ECR.

• Run the aws ecr get-login command to log in to AWS from the terminal.

aws ecr get-login-password - region us-east-1 | docker login - username AWS - password-stdin [ACCOUNT_ID].dkr.ecr.us-east-1.amazonaws.com

• Build the Docker image from the Dockerfile

docker build -t custom-app:1.0 .

• Tag it with the AWS account ID, region, and repo name

docker tag custom-app:1.0 [ACCOUNT_ID].dkr.ecr.us-east-1.amazonaws.com/custom-app:1.0

• Push the tagged Docker image to Amazon ECR

docker push [ACCOUNT_ID].dkr.ecr.us-east-1.amazonaws.com/custom-app:1.0

3. Create secrets to pull the image from Amazon ECR and to access MYSQL

• aws ecr get-login-password — region us-east-1 | docker login — username AWS — password-stdin [ACCOUNT_ID].dkr.ecr.us-east-1.amazonaws.com [saves creds to .configjson file]
• kubectl create secret docker-registry ecr-secret — from-file=.dockerconfigjson=”%USERPROFILE%\.docker\config.json” [creates a secret of type docker-registry using the config.json file from the previous step]
• kubectl create secret generic mysql-creds — from-literal=mysql-root-password=rootpassword — from-literal=mysql-password=mypassword — from-literal=mysql-replication-password=replicationpassword [creates a MYSQL root, user, and replication passwords]

4. Create a ConfigMap to store non-sensitive data

apiVersion: v1
kind: ConfigMap
metadata:
  name: customapp-configmap
data:
  PMA_HOST: mysql-primary
  PMA_PORT: "3306"
  DB_USER: myuser
  DB_NAME: mydb
  DB_SERVER: mysql-primary

The ConfigMap includes non-sensitive data required for both PHPMYadmin (PMA_HOST, PMA_PORT), and the java app (DB_USER, DB_NAME and DB_SERVER)

5. Create a custom values file for MySQL and deploy with Helm

architecture: replication
replicaCount: 2
auth:
  database: mydb
  username: myuser
  existingSecret: mysql-creds
persistence:
  storageClass: "linode-block-storage"
global:
  security:
    allowInsecureImages: True
image: 
  registry: docker.io
  tag: latest
  repository: bitnamilegacy/mysql

The above file is a basic overwritten-values file using the Bitnami Helm chart. It creates two replicas using the replication architecture, specifies the database, username, and uses existingSecret for the password. For persistence, it uses “linode-block-storage”. This provisions volumes for each replica on LKE.

charts/bitnami/mysql at main · bitnami/charts

Install with

helm install mysql --values helm-mysql.yaml bitnami/mysql

List releases installed using Helm with “helm ls” or “helm list”

6. Create deployment and service manifest files for phpMyAdmin

apiVersion: apps/v1
kind: Deployment
metadata:
  name: phpmyadmin
  labels:
    app: phpmyadmin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: phpmyadmin
  template:
    metadata:
      labels:
        app: phpmyadmin
    spec:
      containers:
      - name: phpmyadmin
        image: phpmyadmin
        ports:
        - containerPort: 80
        env:
        - name: PMA_HOST
          valueFrom:
            configMapKeyRef:
              name: customapp-configmap
              key: PMA_HOST
        - name: PMA_PORT
          valueFrom:
            configMapKeyRef:
              name: customapp-configmap
              key: PMA_PORT

---

apiVersion: v1
kind: Service
metadata:
  name: phpmyadmin-service
spec:
  selector:
    app: phpmyadmin
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 80

The deployment file contains environment variables needed to connect to MySQL, such as PMA_HOST and PMA_PORT. Values are defined in the ConfigMap.

env:
        - name: PMA_HOST
          valueFrom:
            configMapKeyRef:
              name: customapp-configmap
              key: PMA_HOST
        - name: PMA_PORT
          valueFrom:
            configMapKeyRef:
              name: customapp-configmap
              key: PMA_PORT

Under the “env” field, there is a name “PMA_HOST”; the actual value is gotten from “key: PMA_HOST”, which, according to the ConfigMap above, is “mysql-primary”

7. Create deployment and service manifest files for the Java app

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-app
  labels:
    app: custom-app
spec: 
  replicas: 2
  selector:
    matchLabels:
      app: custom-app
  template:
    metadata:
      labels:
        app: custom-app
    spec:
      imagePullSecrets:
      - name: ecr-secret
      containers:
      - name: custom-app
        image: 420392929933.dkr.ecr.us-east-1.amazonaws.com/my-app:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
          - name: DB_USER
            valueFrom:
              configMapKeyRef:
                name: customapp-configmap
                key: DB_USER
          - name: DB_PWD
            valueFrom:
              secretKeyRef:
                name: mysql-creds
                key: mysql-password
          - name: DB_NAME
            valueFrom:
              configMapKeyRef: 
                name: customapp-configmap
                key: DB_NAME
          - name: DB_SERVER
            valueFrom:
              configMapKeyRef:
                name: customapp-configmap
                key: DB_SERVER

---

apiVersion: v1
kind: Service
metadata:
  name: customapp-service
spec:
  selector:
    app: custom-app
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

8. Use Helm to deploy an ingress controller

helm install nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress --set controller.reportIngressStatus.enable=true

Deploying an Nginx controller with “controller.reportIngressStatus.enable=True” populates the status of the Ingress controller with the IP address of the LKE load balancer (NodeBalancer)

Using “kubectl get pod”

Using “kubectl get svc”

9. Use an ingress rule to set the hostname, path, and service that is exposed externally

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: customapp-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: 172-238-162-7.ip.linodeusercontent.com
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: customapp-service
              port:
                number: 8080

10. Create a Helm chart and template files for deployment, service, configmap, and ingress

helm create customapp

Delete unneeded template files and create what is needed.

Deployment template file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.appName }}
  labels:
    app: {{ .Values.appName }}
spec:
  replicas: {{ .Values.appReplicas }}
  selector:
    matchLabels:
      app: {{ .Values.appName }}
  template:
    metadata:
      labels:
        app: {{ .Values.appName }}
    spec:
      imagePullSecrets:
      - name: {{ .Values.secretKey }}
      containers:
      - name: {{ .Values.appName }}
        image: "{{ .Values.appImage }}:{{ .Values.appVersion }}"
        imagePullPolicy: Always
        ports: 
        - containerPort: {{ .Values.containerPort }}
        env:
        {{- range .Values.containerEnvVars }}
        - name: {{ .name }}
          valueFrom:
            {{ .keyRef }}:
                name: {{ .valueFrom.name }}
                key: {{ .valueFrom.key }}
        {{- end }}        

In the env field, “{{- range .Values.containerEnvVars }}” allows you to specify any number of environment variables, and {{ .keyRef }} lets you specify what keys to use, whether secretKeyRef or configmapKeyRef

Service template file

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.serviceName }}
spec:
  type: {{ .Values.serviceType }}
  selector:
    app: {{ .Values.appName }}
  ports:
  - protocol: TCP
    port: {{ .Values.servicePort }}
    targetPort: {{ .Values.containerPort }}

ConfigMap template file

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Values.configmapName }}
data:
{{- range $key, $value := .Values.data }}
    {{ $key }}: {{ $value | quote }}
{{- end}}

In the ConfigMap template file, you can specify the names of the keys and values. For example, PMA_HOST=[MYSQL_HOST] or DB_USER=[USERNAME]

Ingress template file

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: {{ .Values.ingressName }}
spec:
  ingressClassName: {{ .Values.ingressclassName }}
  rules:
  - host: {{ .Values.hostName }}
    http:
      paths:
        - path: {{ .Values.pathName }}
          pathType: {{ .Values.pathType }}
          backend:
            service:
              name: {{ .Values.serviceName }}
              port:
                number: {{ .Values.servicePort }}

11. Create a default values file

appName: appname
appReplicas: 1
secretKey: ecr-secret
appImage: 12345678.dkr.ecr.12345.amazonaws.com/1234
appVersion: 0.0
containerPort: 8080
containerEnvVars:
  - name: VAR_1
    keyRef: configMapKeyRef
    valueFrom:
      name: configmap-name1
      key: configmap-key1
  - name: VAR_2
    keyRef: configMapKeyRef
    valueFrom:
      name: configmap-name2
      key: configmap-key2
  - name: VAR_3
    keyRef: configMapKeyRef
    valueFrom:
      name: configmap-name3
      key: configmap-key3
  - name: VAR_4
    keyRef: secretKeyRef
    valueFrom:
      name: secret-name1
      key: secret-key1


serviceName: servicename
servicePort: 8080
serviceType: ClusterIP


ingressName: ingressname
ingressclassName: ingressclassName
hostName: custom-app.com
pathName: /
pathType: Prefix


configmapName: configmapname
data:
  DB_USER: dbuser
  DB_NAME: dbname
  DB_SERVER: dbserver

The default values are used if a custom values file isn’t specified when running Helm install. You can also override specific values using the “—— set” keyword.

12. Create a custom values file to override the default values file

appName: custom-app
appReplicas: 2
secretKey: ecr-secret
appImage: 420392929933.dkr.ecr.us-east-1.amazonaws.com/my-app
appVersion: "1.0"
containerPort: 8080
containerEnvVars:
  - name: DB_USER
    keyRef: configMapKeyRef
    valueFrom:
      name: customapp-configmap
      key: DB_USER
  - name: DB_NAME
    keyRef: configMapKeyRef
    valueFrom:
      name: customapp-configmap
      key: DB_NAME
  - name: DB_SERVER
    keyRef: configMapKeyRef
    valueFrom:
      name: customapp-configmap
      key: DB_SERVER
  - name: DB_PWD
    keyRef: secretKeyRef
    valueFrom:
      name: mysql-creds
      key: mysql-password

serviceName: customapp-service
servicePort: 8080
serviceType: ClusterIP

ingressName: customapp-ingress
ingressclassName: nginx
hostName: 172-238-162-7.ip.linodeusercontent.com

configmapName: customapp-configmap
data:
  DB_USER: myuser
  DB_NAME: mydb
  DB_SERVER: mysql-primary

Validate the chart with “helm template”, “helm lint”, and “helm — dry-run”

Use “helm lint” to check if your Helm chart is valid and follows Helm’s expected stucture/conventions.

helm lint -f customapp-values.yaml customapp

In the above command, “customapp-values.yaml” is the name of the custom values file, and “customapp” is the name of the chart folder, containing the template files.

Use “helm template” to render your Helm template into Kubernetes YAML

helm template -f customapp-values.yaml customapp

Use “helm install -dry-run” to validate the install against the Kubernetes cluster, and to check for any problems involving the cluster before installing

helm install -f values\customapp-values.yaml customapp charts\customapp --dry-run=server --debug

13. Use Helm to deploy the chart

helm install -f values\customapp-values.yaml customapp-helm charts\customapp

In the above command, “customapp-values.yaml” is the name of the custom values file, “customapp-helm” is the release name, and “charts\customapp” is the folder containing the template files.

14. Create a Helmfile and use it to deploy the application

A Helmfile is used to manage multiple Helm releases simultaneously, rather than installing or uninstalling them with a bunch of commands. A release file for the Java app will look like the following:

releases:
  - name: customapp-helm
    chart: charts/customapp
    values:
       - values/customapp-values.yaml

And can be installed using:

helmfile sync

List releases installed with Helmfile with ”helmfile list”, and uninstall the releases with Helmfile using “helmfile destroy”

helmfile list

15. Port-forward phpMyAdmin for internal access

The phpMyAdmin service is of type ClusterIP. This means it is not accessible externally, and it shouldn’t be, since phpMyAdmin manages the MYSQL database. We can, however, port-forward the phpMyAdmin service to our local machine so that we can access it internally.

kubectl port-forward service/phpmyadmin-service 8081:8081

As shown in the picture above, the phpMyAdmin webpage is accessible on localhost, and the Java application created a table named “team_members”, added columns, and inserted values into it.

The application consists of three containers: MongoDB and MongoExpress images from Docker Hub, and a custom application image written in Python and built with a Dockerfile. All containers are deployed using Docker Compose. Jenkins pulls the source code from the GitHub repository, scans the code and images for security issues using SonarQube and Trivy, and pushes to the master branch when the security scans pass. The secure image is then pushed to a private Docker repository and deployed to an EC2 instance.

This project demonstrates the use of Git, Jenkins, Docker, SonarQube, Trivy, AWS EC2, and ways to address vulnerabilities in a DevSecOps pipeline.

Application Overview

The application uses Docker Compose to build a MongoDB image and a MongoExpress image to access the database. It also uses a custom Python application to create a database, create a collection, and insert values into the collection.

MongoDB Image

mongo - Official Image | Docker Hub

The image requires environmental variables specifying the admin username and password. Adding a volume is recommended so the data is saved if the container goes down.

  mongodb:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    volumes:
      - mongo_data:/data/db

MongoExpress Image

mongo-express - Official Image | Docker Hub

The image needs environment variables for the MongoExpress username and password, and a variable that specifies how to access the MongoDB database.

  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_BASICAUTH_ENABLED: true
      ME_CONFIG_BASICAUTH_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      ME_CONFIG_BASICAUTH_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
      ME_CONFIG_MONGODB_URL: mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@${MONGODB_HOST}:${MONGODB_PORT}/

Custom Python Application

The application connects to MongoDB using the “uri” variable, creates a database and a collection, and inserts values into the collection.

from pymongo import MongoClient
from dotenv import load_dotenv
import os

load_dotenv()

# Connecting to the MongoDB server

uri = (
    f"mongodb://{os.getenv('MONGODB_USER')}:"
    f"{os.getenv('MONGODB_PASS')}@{os.getenv('MONGODB_HOST')}:{os.getenv('MONGODB_PORT')}/"

)


client = MongoClient(uri)

# Create database
db = client["test_database"]

# Create a collection
collection = db["users"]

# Add values
user_data = {"name": "Default_User", "age": 25, "skills": ["Python", "MongoDB"]}

result = collection.insert_one(user_data)

Final Docker Compose file

services:

  custom-app:
    image: ${IMAGE_NAME}
    restart: always
    depends_on:
      - mongodb
    environment:
      MONGODB_USER: ${MONGO_INITDB_ROOT_USERNAME}
      MONGODB_PASS: ${MONGO_INITDB_ROOT_PASSWORD}
      MONGODB_HOST: ${MONGODB_HOST}
      MONGODB_PORT: ${MONGODB_PORT}


  mongodb:
    image: mongo
    restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    volumes:
      - mongo_data:/data/db


  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_BASICAUTH_ENABLED: true
      ME_CONFIG_BASICAUTH_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      ME_CONFIG_BASICAUTH_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
      ME_CONFIG_MONGODB_URL: mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@${MONGODB_HOST}:${MONGODB_PORT}/


volumes:
  mongo_data:

Git Operations

In the project directory, initialize a new Git repo, set the remote URL, create a new branch, stage the files, commit the changes locally, and push to the remote repository.

git init [initializes git repo in current directory]
git remote set-url origin https://github.com/Michael2921/Basic_DevSecOps.git [sets the remote url for the git repository]
git switch -c dev [creates a new branch and switches to it]
git add . [stages files in current directory]
git commit -m "initial commit" [commits files to local repo]
git push -u origin dev [pushes commited changes to local repo. needs to be logged in]

Setting up Jenkins

• Create a multibranch pipeline in Jenkins.

• Add credentials for GitHub and MongoDB.

• Add repository as a branch source in Jenkins.

Create a starter Jenkinsfile.

pipeline {

    agent any

    environment {
        MONGODB_HOST = "mongodb"
        MONGODB_PORT = "27017"


    }

    stages {

        stage('Building image from Dockerfile.') {

            steps {
                sh 'docker build -t basic_devsecops:v1 .'



            }
        }



        stage('Running docker compose.') {

           steps{

            script {
                withCredentials([
                    usernamePassword(
                    credentialsId: 'mongodb-creds',
                    usernameVariable: 'MONGO_INITDB_ROOT_USERNAME',
                    passwordVariable: 'MONGO_INITDB_ROOT_PASSWORD')
               ])
               {
                    sh 'docker compose down -v'
                    sh 'docker compose -f docker-compose.yaml up'

                }


                }


                }


                }
                }

        }

Install Trivy in the Jenkins container.

docker exec -ut -u 0 [CONTAINER_ID] /bin/bash # enters container as root

apt-get update && sudo apt-get install wget gnupg lsb-release -y

apt-get update && apt-get install -y wget # installs wget

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | tee /usr/share/keyrings/trivy.gpg > /dev/null

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | tee /etc/apt/sources.list.d/trivy.list

apt-get update

apt-get install trivy -y

Install SonarQube as a Docker container.

docker run -d --name sonarqube-jenkins -v sona_data:/opt/sonarqube/data -v sona_logs:/opt/sonarqube/logs -v sona_extensions:/opt/sonarqube/extensions -p 9000:9000 sonarqube:community

The default password is admin: admin. You will be prompted to change the password on initial login. According to the official image on DockerHub, it is advised to create volumes with the directories /opt/sonarqube/data for data files, /opt/sonarqube/logs for logs, and /opt/sonarqube/extensions for 3rd-party plugins.

Setting up Jenkins | SonarQube Server 10.8 | Sonar Documentation

The above link describes how to integrate SonarQube into Jenkins.

Install the SonarQube scanner plugin on Jenkins (Manage Jenkins > Manage Plugins)

Create a SonarQube token in SonarQube (My account > Security > Generate token)

Add the token as a secret text in Jenkins credentials (Manage Jenkins -> Credentials -> Add credentials -> Secret text)

Add the SonarQube server address and credentials to the system configuration in Jenkins (Manage Jenkins > Configure System > SonarQube Servers)

Install the SonarScanner instance (Manage Jenkins > Tools > Add SonarQube scanner)

Configure the SonarQube webhook for the quality gate and to notify Jenkins of the task status. The webhook path should be configured in SonarQube, and the path must be /sonarqube-webhook/ (Administration > Configuration > Webhooks)

SonarQube Quality Gates

A quality gate in SonarQube is a pass/fail checkpoint that evaluates code against a set of conditions. This feature is helpful to stop a build from completing if it fails security checks.

SonarQube conducts quality gate checks on the default security gate using several criteria, including “New code” and “Overall code.” New code refers to lines recently added, modified, or touched, while overall code represents all source code, both legacy and newly merged.

To test the SonarQube scan, I created a custom quality gate and set conditions to fail the build if more than zero issues are detected in Overall Code.

Default SonarQube quality gate

Created SonarQube quality gate

Fixing security issues to pass SonarQube quality gate

My build failed after assigning the project to my new quality gate.

Looking in SonarQube, the issue responsible for the failure can be found.

It says my Python custom image runs with “root” as the default user, which is true, according to the Dockerfile above.

FROM python:3.15.0b4-slim

WORKDIR /home/app

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY app.py .

CMD ["python", "app.py"]

The updated Dockerfile looks like this. A new user was created with useradd, and the ownership of files in /home/app was set to the new user.

FROM python:3.15.0b4-slim

WORKDIR /home/app

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY app.py .

RUN useradd --create-home appuser && chown -R appuser:appuser /home/app

USER appuser

CMD ["python", "app.py"]

To push modified changes to the remote repository, run:

git add . [stages files in the current directory]
git commit -m "commit message" [commits modified files locally]
git push [pushes files to remote repository]

After making the changes and running the build again, the SonarQube scan passes the quality gate

Looking in SonarQube, the quality gate passed with no issues.

Remediating Vulnerabilities Found With Trivy

In my Jenkins pipeline, the SonarQube scan runs before the Trivy scan. SonarQube scans perform static code analysis, while Trivy scans containers for vulnerabilities.

I resolved the SonarQube issue by creating and using a non-root user. For Trivy, the Jenkinsfile is configured to fail the build if any high or critical vulnerabilities are detected.

stage('Trivy scan') {

            steps {

               sh 'trivy image --severity HIGH,CRITICAL --exit-code 1 basic_devsecops:v1'


            }

        }

The initial Trivy scans show multiple critical and high vulnerabilities.

Two effective methods for addressing these vulnerabilities are hardened base images and multi-stage builds.

Hardened base images include only essential packages, are regularly updated with security patches, typically run as non-root by default, and are designed to minimize CVE exposure.

Multi-stage builds ensure that only runtime requirements are included in the final production image, thereby reducing the attack surface.

Because this is a Python application, secure Python base images were selected.

• Hardened Images catalog | Python | Docker Hub
• python Container Image Versions | Chainguard

The Python image from dhi.io was used to build and install dependencies, while the cgr.dev image served as the runtime environment. SonarQube flagged an issue because it assumes the default user is root, whereas cgr.dev images use a non-root user. To address this, I updated the SonarQube quality gate, increasing the failure threshold for Overall Code issues from 1 to 2.

Updated Dockerfile:

FROM dhi.io/python:3-dev AS builder

WORKDIR /home/app

COPY requirements.txt .

RUN pip install --target=/home/app/dependencies -r requirements.txt

FROM cgr.dev/chainguard/python:latest-dev

WORKDIR /home/app

COPY --from=builder /home/app/dependencies /home/app/dependencies

COPY app.py .

ENV PYTHONPATH=/home/app/dependencies

CMD ["app.py"]

After running Trivy with hardened base images and using a multibranch build, the Trivy scan results show 0 critical and high vulnerabilities.

Pushing Secure Code to the Master Branch

When the code successfully passes SonarQube and Trivy scans, it needs to be pushed to the master branch so it can be deployed. Here is the stage doing that.

stage('Push secure code to master branch'){
        when {branch 'dev'} //master doesn't push to itself avoiding a build loop. work isn't done from master. only pushed from dev when security scans pass

            steps {
                   script {
                        sh 'echo "Pushing code to master branch"'
                        sh 'git config user.email "jenkins@example.com"'
                        sh 'git config user.name "jenkins"'
                        sh 'git status'
                        sh 'git branch'
                        sh 'git remote set-url origin https://${GITHUB_TOKEN}@github.com/Michael2921/Basic_DevSecOps.git'
                        sh 'git add .'
                        sh 'git commit -m "pushing secure code to master branch"'
                        sh 'git push -f origin HEAD:master'


                   }

            }

}

In the code block above, ‘git remote set-url origin’ requires a GitHub login. Since this is done non-interactively, a GitHub PAT (personal access token) needs to be generated in GitHub and added as a credential in Jenkins. “GITHUB_TOKEN” was already imported as an environment variable earlier.

enivronment {
GITHUB_TOKEN = credentials('github-token')
}

Generate a Personal Access Token in GitHub

• Go to profile in the top right corner ( Settings > Credentials > Personal Access Tokens (classic) and generate a token)
• Select the appropriate scope, and generate the token.

• Add Personal Access Token to Jenkins

Deploy Image to Private Docker Repository.

Before deploying the image to the AWS instance, it is advisable to push it to a repository. I selected a private DockerHub repository to reflect a typical production environment. The following stage outlines this process.

        stage('Docker login and push to repo') {
        when {branch 'master'}

            steps {
                script {
                    echo "Logging in to docker"
                    withCredentials([usernamePassword(credentialsId: 'docker-hub-repo', usernameVariable: 'USER', passwordVariable: 'PASS')]) {
                        sh 'echo $PASS | docker login -u $USER --password-stdin'
                        sh 'echo "Pushing secure image to repo"'
                        sh "docker push ${IMAGE_NAME}"

                  }

                }

            }

        }

Using withCredentials, I was able to log in with the credentials I created in Jenkins for Docker. The image has to be named in a specific way, using the account name + repo name; this naming method implicitly specifies where the image is pushed when “docker push” is executed. IMAGE_NAME was already set as an environment variable earlier.

 IMAGE_NAME = "michael101/basic-devsecops:v1"

Deploy image to EC2

In this stage, I will do a few things:

• Transfer necessary application files to the EC2 instance.
• Configure the required environment variables on the EC2 instance.
• Provide credentials for SSH access to the EC2 instance and for pulling the Docker image from the private repository.

Here is the Jenkinsfile:

stage('Deploying image to EC2'){
        when {branch 'master'} //deploys from master

            steps {
                script {
                    withCredentials([
                        usernamePassword(
                    credentialsId: 'mongodb-creds',
                    usernameVariable: 'MONGO_INITDB_ROOT_USERNAME',
                    passwordVariable: 'MONGO_INITDB_ROOT_PASSWORD'),
                        usernamePassword(
                        credentialsId: 'docker-hub-repo',
                        usernameVariable: 'USER',
                        passwordVariable: 'PASS'),

                        string(credentialsId: 'ec2-instance', variable: 'EC2_INSTANCE')

                    ]) {

                    def shellCmd = "bash ./commands.sh ${IMAGE_NAME}"
                    def ec2Instance = "ec2-user@${EC2_INSTANCE}"

                    sshagent(['basic-devsecops-ssh']) { //variables are set in the ssh command and a shell script containing commands to login to docker repo and deploy is executed
                        sh "scp -o StrictHostKeyChecking=no commands.sh docker-compose.yaml ${ec2Instance}:/home/ec2-user"
                        sh "ssh -o StrictHostKeyChecking=no ${ec2Instance} MONGO_INITDB_ROOT_USERNAME='${MONGO_INITDB_ROOT_USERNAME}' MONGO_INITDB_ROOT_PASSWORD='${MONGO_INITDB_ROOT_PASSWORD}' \
                        MONGODB_HOST='${MONGODB_HOST}' MONGODB_PORT='${MONGODB_PORT}' USER=${USER} PASS=${PASS} \
                        ${shellCmd}"

                    }

                    }
                }

            }
        }

The withCredentials block defines two credentials: one for DockerHub and another for SSH access to the EC2 instance.

Two variables are defined in this stage. One sets the username and IP address, while the other lists the commands to execute after establishing an SSH connection to the server.

def ec2Instance = "ec2-user@[ec2-Instance]"
def ShellCmd = "bash ./commands.sh"

Since the SSH session is non-interactive and quits after a single SSH command, I included the environment variables required to run the Docker Compose file and ran the commands.sh file (a shell script containing a list of commands) in the SSH session.

Here are the contents of commands.sh

#!/usr/bin/env bash

export IMAGE_NAME=$1
echo $PASS | docker login -u $USER --password-stdin
docker-compose down -v
docker-compose -f docker-compose.yaml up -d
echo "success"

The commands.sh file

• Assigns the first parameter while executing the command as the image name
• Runs the “docker login” command
• Runs the docker compose down command, while removing the volumes
• Runs the docker compose command to pull and run the images.
• Outputs a “success” message

Implement a CI/CD pipeline using GitHub Webhooks.

In Jenkins: Go to multibranch pipeline > Scan Repository Triggers.

In GitHub: Go to project > settings and create the payload URL

CI/CD overall logic

Dev branch

• Application is built from the dev branch
• SonarQube scans code from the dev branch
• If successful, an image is built from the scanned code
• The built image is then scanned with Trivy
• If the Trivy scan is successful, secure code is pushed to master.

Master branch

• The application is not built directly from the master branch. Only code that has passed both SonarQube and Trivy scans is merged into master.
• After secure code is pushed to master, the secure image is pushed to a private Docker repository.
• Then the secure image is pulled from the private Docker repository and deployed to an EC2 instance.