<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by suman15 on Medium]]></title>
        <description><![CDATA[Stories by suman15 on Medium]]></description>
        <link>https://medium.com/@suman15?source=rss-299760da75d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lflvlpGVgbRj6KreFtLX4w.jpeg</url>
            <title>Stories by suman15 on Medium</title>
            <link>https://medium.com/@suman15?source=rss-299760da75d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 07:56:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suman15/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building an End-to-End DevOps CI/CD Pipeline for an Amazon Prime Video Clone Using AWS, Terraform…]]></title>
            <link>https://aws.plainenglish.io/building-an-end-to-end-devops-ci-cd-pipeline-for-an-amazon-prime-video-clone-using-aws-terraform-ae0f2ab152f9?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae0f2ab152f9</guid>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 12:11:35 GMT</pubDate>
            <atom:updated>2026-11-11T20:31:35.652Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building an End-to-End DevOps CI/CD Pipeline for an Amazon Prime Video Clone Using AWS, Terraform, Jenkins, Docker, Kubernetes, Argo CD, Prometheus and Grafana</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*DoE5dsplKMkQTxAedWum9g.gif" /></figure><p><a href="https://github.com/suman526/amazon-prime-devops-platform/tree/main">Github</a> <a href="https://lnkd.in/p/dfBTaW2g">Linkedin</a></p><p>In this project, I built an end-to-end DevOps workflow to deploy an Amazon Prime Video clone application using multiple DevOps tools and AWS services.</p><p>The objective is to automate the complete process, starting from infrastructure provisioning and source-code management to continuous integration, security scanning, container image creation, deployment on Kubernetes, continuous delivery through Argo CD, and application monitoring using Prometheus and Grafana.</p><p>I also worked through several real-world troubleshooting scenarios during implementation, including Terraform configuration errors, AWS resource provisioning problems, Jenkins configuration issues, Kubernetes and Argo CD deployment challenges, and infrastructure cleanup.</p><p>This blog documents the project architecture, implementation workflow, tools used, technical decisions, troubleshooting experiences, and final outcome.</p><h3>Project</h3><p>The project involved deploying an Amazon Prime Video clone application through an automated DevOps pipeline. Instead of manually creating infrastructure and deploying the application, I used Terraform to provision AWS resources and Jenkins to automate the build and integration process. The application was packaged into a Docker image and pushed to Amazon Elastic Container Registry (ECR). The image was then deployed to an Amazon Elastic Kubernetes Service (EKS) cluster. Argo CD was used for continuous deployment, while Prometheus and Grafana were integrated for monitoring.</p><h3>Project objectives</h3><p>What are we going to build?</p><p>The project is divided into the following major activities:</p><p>1. Create an AWS account and configure AWS CLI.<br>2. Create an IAM user and generate access keys.<br>3. Create an EC2 key pair.<br>4. Clone the DevOps project repository from GitHub.<br>5. Create a Jenkins EC2 server using Terraform.<br>6. Install the required DevOps tools on the Jenkins server.<br>7. Configure Jenkins plugins, credentials and tools.<br>8. Create a Jenkins build pipeline.<br>9. Integrate GitHub, SonarQube, NPM and Aqua Trivy.<br>10. Build a Docker image and push it to Amazon ECR.<br>11. Clean up Docker images from the Jenkins server.<br>12. Create an Amazon EKS cluster using Terraform.<br>13. Create a separate Jenkins deployment pipeline.<br>14. Install Prometheus, Grafana and Argo CD using the deployment workflow.<br>15. Create an Argo CD application using the GitHub repository.<br>16. Access the deployed Amazon Prime clone application.<br>17. Configure the monitoring dashboard in Grafana.<br>18. Perform the cleanup pipeline.<br>19. Destroy the remaining AWS infrastructure using Terraform.</p><p>The main advantage of this approach is that the application build and deployment responsibilities are separated.</p><p>Jenkins builds and stores the image. Argo CD manages the Kubernetes application deployment based on the Git repository configuration.</p><h3>Install AWS CLI</h3><p>AWS CLI allows us to interact with AWS services directly from the terminal.</p><p>It is useful for authentication, checking resources, configuring EKS access, and executing AWS commands.</p><p>After installation, I checked the version:</p><pre>aws --version</pre><h3>Configure AWS CLI</h3><h4>Create an IAM user</h4><p>I opened the AWS Management Console and navigated to:</p><pre>IAM → Users → Create user</pre><p>I created a dedicated IAM user for the project.</p><p>In this workflow I used AdministratorAccess to simplify the lab setup.</p><p>However, this permission level is broad and should not be copied into production environments. A production implementation should follow the principle of least privilege and use narrowly scoped permissions.</p><h4>Generate access keys</h4><p>After creating the user, I navigated to:</p><pre>IAM → Users → Select User → Security credentials</pre><p>I generated an access key for CLI access.</p><p>The access key and secret key were then configured locally using:</p><p>These credentials were also required later for Jenkins to interact with AWS services in the demonstrated workflow.</p><p>To connect my local machine with AWS, I configured AWS CLI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*rMc6PZU81mmc4kDaxhTZhA.png" /></figure><p><strong>Security note:</strong> Access keys and secret keys should never be committed to GitHub or exposed in screenshots. In a production environment, prefer IAM roles and temporary credentials where possible.</p><h3>Install Visual Studio Code</h3><p>I used Visual Studio Code to manage the project files and Terraform configurations</p><h3>Create an EC2 key pair</h3><p>The Jenkins EC2 instance required SSH access.</p><p>I navigated to:</p><pre>EC2 → Network &amp; Security → Key Pairs → Create key pair</pre><p>I created the key pair with the name:</p><pre>key</pre><p>The private key was downloaded locally in PEM format.</p><p>The key pair name must match the name referenced in the Terraform configuration.</p><p>For example:</p><pre>key_name = &quot;key&quot;</pre><p>The private key should be stored securely and must never be uploaded to a public GitHub repository.</p><h3>Clone the Project Repository</h3><p>The project source code and configuration files were maintained in GitHub.</p><p>I used Git to clone the repository to my local machine.</p><p>The general command was:</p><pre>git clone &lt;GITHUB_REPOSITORY_URL&gt;</pre><p>After cloning the repository, I moved into the project directory:</p><pre>cd &lt;PROJECT_DIRECTORY&gt;</pre><p>I then opened the project in Visual Studio Code:</p><pre>code .</pre><h3>Project structure</h3><p>The repository contained different directories for infrastructure, application packaging, Kubernetes manifests, and pipeline scripts.</p><p>The logical structure is:</p><pre>DevOpsProject/<br>│<br>├── terraform_code/<br>│   ├── ec2_code/<br>│   │   └── Terraform configuration files<br>│   │<br>│   └── eks_code/<br>│       └── Terraform configuration files<br>│<br>├── Dockerfile<br>│<br>├── kubernetes/<br>│   ├── deployment.yaml<br>│   └── service.yaml<br>│<br>├── pipeline_script/<br>│   ├── build-pipeline<br>│   ├── deployment-pipeline<br>│   └── cleanup-pipeline<br>│<br>└── README.md</pre><p>The exact directory and file names should be updated to match the final version of my repository.</p><h3>Provisioning the Jenkins Server Using Terraform</h3><p>The first major infrastructure task was creating the Jenkins server.</p><p>Instead of launching an EC2 instance manually, I used Terraform to automate infrastructure provisioning.</p><p>Terraform is an Infrastructure as Code tool that allows infrastructure to be defined, planned, created, updated, and destroyed through configuration files.</p><h3>Why Terraform?</h3><p>Terraform was selected because:</p><p>Infrastructure can be defined as code, Resource creation can be automated, Configurations can be maintained in version control, Infrastructure can be recreated consistently, Changes can be reviewed through execution plans, Resources can be destroyed through Terraform.</p><h3>Terraform configuration</h3><p>EC2 Terraform directory.</p><p><a href="https://github.com/suman526/amazon-prime-devops-platform/blob/main/terraform_code/ec2_server/main.tf">amazon-prime-devops-platform/terraform_code/ec2_server/main.tf at main · suman526/amazon-prime-devops-platform</a></p><p>Copy the code from the above link, here I have explained each and every step in detailed manner.</p><h3>Initialize Terraform</h3><p>I opened PowerShell in the Terraform directory and initialized the working directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*tVPvz2GXD2Q-387_smODBw.png" /></figure><p>This downloaded the required provider plugins and initialized the Terraform working directory.</p><h3>Validate Terraform configuration</h3><p>Next, I checked whether the Terraform configuration was syntactically valid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*j_pMs-A_Gd74wIvh_F4dOQ.png" /></figure><h3>Review the execution plan</h3><p>Before creating infrastructure, I reviewed the execution plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONzS_eT9Lnd8NYcq9-fbcQ.png" /></figure><p>This displayed the resources Terraform intended to create or modify.</p><h3>Provision the infrastructure</h3><p>After reviewing the plan, I applied the configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*ctBwP7zAUUl6y0u6qvBlYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rRQcPZlzKnI_pOmpiiuBbg.png" /></figure><p>Terraform created the EC2 instance and the associated infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hua5SAW5xGrTi0GBRgr0ng.png" /><figcaption>jenkins server launched</figcaption></figure><p>The instance initialization process installed the required packages and services for the project.</p><p>The setup included components such as:</p><p>Java, Jenkins, Docker, AWS CLI, kubectl, Helm, Node.js-related dependencies, Aqua Trivy, SonarQube running in a container</p><p>The actual package versions and initialization commands should be taken from the final Terraform files in the repository.</p><h3>Access Jenkins</h3><p>Once the EC2 instance was running, I accessed Jenkins using the public IP address or URL generated by the <strong>infrastructure setup</strong>.</p><p>After logging in, I proceeded with the Jenkins configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7sWmD8bzvVzdwE3BRby9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JK_jgAmxCZUetfas7_C-tQ.png" /></figure><h3>Setting Up SonarQube</h3><p>SonarQube was integrated into the project to analyze source-code quality.</p><p>It helps identify issues such as:</p><ul><li>Bugs, Code smells, Code duplication, Maintainability issues, Security-related findings</li></ul><p>For this project, SonarQube was deployed as a Docker container instead of using a manual host installation.</p><h3>Why use Docker for SonarQube?</h3><p>Running SonarQube inside a container made the setup more manageable. The container could be started and managed independently of the Jenkins installation. The actual Docker image and configuration should be taken from the final setup used in the project.</p><h4>Access SonarQube</h4><p>After starting the SonarQube container, I accessed its web interface using the configured URL and port.</p><p>The default login credentials were used for the initial setup and should be changed after the first login.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XBnBO0elIUyeoVwyuvrRxw.png" /></figure><h3>Generate a SonarQube token</h3><p>To integrate SonarQube with Jenkins, I generated an authentication token.</p><p>The navigation path is:</p><pre>Administration / My Account → Security → Generate Token</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDZqe0wDe5V0qz9p8w0MPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*91v-G3mrue6muJPpDBD-NA.png" /></figure><p>I generated a token and stored it securely. The token is required later when configuring Jenkins credentials.</p><h4>Configure SonarQube webhook</h4><p>The SonarQube webhook was configured to communicate analysis results to Jenkins. The webhook configuration was available under:</p><pre>Administration → Configuration → Webhooks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YuZTGWkJF4cIPtpiKPmpKw.png" /></figure><p>I created a webhook pointing to the Jenkins endpoint. The endpoint format used in this setup was:</p><pre>http://&lt;JENKINS_URL&gt;/sonarqube-webhook/</pre><p>The trailing path is important for the Jenkins SonarQube integration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QM5ISiLB-rj8u83Bm1dXYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-NlBHZEfyDveSbe6hTw0w.png" /></figure><h3>Jenkins Configuration</h3><p>After installing Jenkins and setting up SonarQube, the next task was configuring Jenkins for the CI/CD workflow. The configuration included credentials, plugins, global tools, and the SonarQube server integration.</p><h4>Configure Jenkins credentials</h4><p>Jenkins needs credentials to communicate securely with external services.</p><p>I navigated to:</p><pre>Manage Jenkins → Credentials → System → Global credentials</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdr49zC3EdcpYyFHbjfQjQ.png" /></figure><p>The configuration included:</p><p>Credential Purpose SonarQube token → Connect Jenkins with SonarQube, AWS access key → Authenticate with AWS in the demonstrated pipeline, AWS secret key → Authenticate with AWS in the demonstrated pipeline</p><p>I used the Secret text credential type for the token and the relevant secret values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xyddzo_e6fa7XnuMDM7AQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PUeg34cFjw0gT7fxEEwm1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NW2YVYyngIv3Io0UcYYKEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UsPaTjpX8MDhyi7H3uw2sg.png" /></figure><h3>Install Jenkins plugins</h3><p>I navigated to:</p><pre>Manage Jenkins → Plugins → Available plugins</pre><p>The plugins used or required by the project workflow included:</p><p>1. <strong>SonarQube Scanner</strong>: Used to integrate Jenkins with SonarQube to check code quality and perform quality gate checks.</p><p>2. <strong>NodeJS</strong>: Manages and allows the use of different versions of Node.js within Jenkins.</p><p>3. <strong>Pipeline: Stage View</strong>: Provides a visual interface to monitor the stages of the Jenkins pipeline.</p><p>4. <strong>Docker</strong>: Allows Jenkins to manage Docker containers directly.</p><p>5. <strong>Docker Commons</strong>: Facilitates the sharing of common Docker configurations across Jenkins jobs.</p><p>6. <strong>Docker Pipeline</strong>: Supports building and running Docker containers within Jenkins pipelines.</p><p>7. <strong>Docker API</strong>: Provides an API to interact with Docker within Jenkins.</p><p>8. <strong>docker-build-step</strong>: Adds Docker-related build steps, like building images or running containers in freestyle jobs.</p><p>9. <strong>SSH Agent</strong>: Manages SSH credentials for remote server access within Jenkins pipelines.</p><p>10. <strong>Eclipse Temurin Installer</strong>: Used to install OpenJDK within Jenkins.</p><p><strong>11. Prometheus Metrics</strong></p><p>These plugins enabled Jenkins to integrate with the external tools and display pipeline execution information. After installing the plugins, Jenkins was restarted where required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5II5w2QPYlM1TJDrOWPJdw.png" /></figure><h3>Configure SonarQube server in Jenkins</h3><p>I navigated to:</p><pre>Manage Jenkins → System</pre><p>Under the SonarQube servers section, I added the SonarQube server configuration.</p><p>The configuration included:</p><ul><li>Server name</li><li>SonarQube URL</li><li>Authentication token</li></ul><p>The token generated earlier was selected from Jenkins credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lspGGpq25i5VvrIkGc-5SQ.png" /></figure><h4>Configure Global Tool Configuration</h4><p>Jenkins allows tools to be configured centrally.</p><p>I navigated to:</p><pre>Manage Jenkins → Tools</pre><p>The project required the following tool configurations:</p><p>ToolConfiguration → JDK: Java 17, SonarQube Scanner: Configured scanner installation, NodeJS: Node.js 16.20, DockerConfigured: Docker installation</p><p>The Node.js version was selected to match the application requirements used in the project. Central tool configuration makes it easier to reference tools from Jenkins pipelines without repeatedly defining their installation details.</p><p>Jenkins JDK configuration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaVBkEtbGRccPRiApe0Lww.png" /></figure><p>Jenkins SonarQube Scanner configuration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X41dY_Qmn3NaLQmaDTjfOA.png" /></figure><p>Jenkins NodeJS configuration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6lI21YsddmEzJoUnhVqiUA.png" /></figure><p>Jenkins Docker configuration</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3dukTTI7XT3gXxUwWuFgQ.png" /></figure><h3>Creating the Jenkins CI Pipeline</h3><p>The next major task was creating the continuous integration pipeline.</p><p>The purpose of this pipeline was to automate source-code checkout, code analysis, dependency installation, security scanning, Docker image creation, and pushing the image to Amazon ECR.</p><p>Open Jenkins Dashboard → Click New Item → Enter the name: build-pipeline → Select Pipeline and click OK.</p><p>Paste the deployment pipeline script from the original project repository or README file. Save the pipeline. Run it once so Jenkins can load the parameters. Then refresh the page.</p><p>The Build with Parameters option should be available. Now build with it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pcsBSQKGcOLXSD3iFdTE0Q.png" /></figure><h3>CI pipeline stages</h3><p><a href="https://github.com/suman526/amazon-prime-devops-platform/blob/main/pipeline_script/build_pipeline">amazon-prime-devops-platform/pipeline_script/build_pipeline at main · suman526/amazon-prime-devops-platform</a></p><p>You can view the complete build pipeline code and step-by-step explanation in my GitHub repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hyk1DaIdtIjRl67HTpkIqw.png" /></figure><p>The build pipeline is organized into these stages:</p><p>The exact stage names and commands should match the final Jenkinsfile in my GitHub repository.</p><h4>1. Git checkout</h4><p>The first stage checked out the application source code from GitHub. The repository URL and branch were configured in the pipeline. The project used the main branch in the demonstrated workflow. For a public repository, checkout can be performed without private repository credentials. For a private repository, a GitHub credential must be configured in Jenkins.</p><h4>2. SonarQube analysis</h4><p>The next stage executed the SonarQube scanner. The scanner analyzed the application source code and published the analysis results to the SonarQube server. The pipeline used the configured SonarQube server and scanner installation.</p><p>The logical command format was:</p><pre>sonar-scanner \<br>  -Dsonar.projectName=&lt;PROJECT_NAME&gt; \<br>  -Dsonar.projectKey=&lt;PROJECT_KEY&gt;</pre><p>The actual project key, scanner path, and any additional analysis parameters should be copied from the final pipeline script.</p><h4>3. SonarQube quality gate</h4><p>The pipeline waits for the SonarQube Quality Gate result. If it fails, the pipeline stops, preventing code that doesn’t meet the required quality standards from proceeding to the next stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMfPFgVqMxipkrsI5zM9Bw.png" /><figcaption>sonarqube portal</figcaption></figure><h4>4. NPM dependency installation</h4><p>The application was a Node.js project. Therefore, NPM was used to install the required application dependencies.</p><p>The pipeline executed:</p><pre>npm install</pre><p>This installed the dependencies defined by the project’s package configuration.</p><h4>5. Aqua Trivy security scanning</h4><p>Aqua Trivy was used to scan the project for known vulnerabilities.</p><p>In the workspace section you can check the trivy scan result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQZ7_cOmwgXSJe8vtvGOKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jMbMpTZD2CfCPMYdp8FWyA.png" /></figure><h4>6. Build Docker image</h4><p>Once the previous stages completed, Jenkins built the Docker image. Docker was used to package the application and its required runtime environment into a portable container image.</p><p>The general command format is:</p><pre>docker build -t &lt;IMAGE_NAME&gt;:&lt;TAG&gt; .</pre><p>The Dockerfile defined how the image was built. The actual Dockerfile used in the project should be included in the GitHub repository.</p><h4>7. Authenticate with Amazon ECR</h4><p>After building the image, Jenkins authenticated with Amazon ECR. ECR is AWS’s managed container image registry. It provides a private registry for storing and managing Docker images. The pipeline used AWS credentials and the configured AWS region to authenticate with the registry.</p><p>The authentication command follows this general pattern:</p><pre>aws ecr get-login-password --region &lt;AWS_REGION&gt; |<br>docker login \<br>  --username AWS \<br>  --password-stdin &lt;AWS_ACCOUNT_ID&gt;.dkr.ecr.&lt;AWS_REGION&gt;.amazonaws.com</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6namSUQr4IuFnhkTo-y3vw.png" /></figure><h4>8. Push Docker image to ECR</h4><p>After authentication, the image was tagged with the ECR repository address and pushed to the registry.</p><p>General command format:</p><pre>docker tag &lt;LOCAL_IMAGE&gt;:&lt;TAG&gt; &lt;ECR_URI&gt;:&lt;TAG&gt;</pre><pre>docker push &lt;ECR_URI&gt;:&lt;TAG&gt;</pre><p>The image is then available in the ECR repository for deployment to Kubernetes. Open the repository to verify that the image tags were pushed.</p><p>For example, if build number 8 was completed, the repository should contain the corresponding image version and the latest tag.</p><p>When another build runs, the next build number is added and the latest tag is updated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qp8D-PBevjpEhnVZGmrhA.png" /></figure><h4>9. Clean local Docker images</h4><p>After pushing the image to ECR, the local Jenkins server did not need to retain every image created during the build. The pipeline included image cleanup to reduce unnecessary disk usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R7DQKQYFCl6x-emoT40eEg.png" /></figure><p><strong>At this point, the CI/build pipeline is complete.</strong></p><h3>Provisioning Amazon EKS Using Terraform</h3><p>After completing the CI pipeline and pushing the Docker image to ECR, the next task was preparing the Kubernetes environment.</p><p>I used Amazon Elastic Kubernetes Service (EKS) to run the application. Instead of manually creating the cluster, I used Terraform.</p><h4>Why Amazon EKS?</h4><p>EKS is a managed Kubernetes service provided by AWS. It allows applications to run on Kubernetes without manually managing the entire control plane.</p><p>It provides:</p><ul><li>Managed Kubernetes control plane, Worker node integration, Kubernetes workload orchestration, AWS networking integration, Integration with other AWS services</li></ul><h3>Terraform EKS configuration</h3><p>Navigated to the EKS Terraform directory.</p><p>The configuration included the resources and modules required to create:</p><ul><li>VPC networking</li><li>Subnets</li><li>EKS cluster</li><li>Managed node groups</li><li>Required IAM roles and policies</li></ul><p>The project used a Terraform EKS module.</p><p>The configuration was responsible for creating the cluster and worker nodes.</p><p><a href="https://github.com/suman526/amazon-prime-devops-platform/tree/main/terraform_code/eks_code">amazon-prime-devops-platform/terraform_code/eks_code at main · suman526/amazon-prime-devops-platform</a></p><h4>Initialize and validate the EKS configuration</h4><p>I initialized the Terraform working directory</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HedQoMbYrOHUQnEo7RDkZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rhDARqxKv4xmzOF89ZRYVw.png" /></figure><p>I also reviewed the planned changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iEy1GKw3nznvMJ815PiCsw.png" /><figcaption>Terraform provisioned the EKS infrastructure</figcaption></figure><p>Cluster creation can take several minutes because AWS must create the control plane and associated resources.</p><p>The managed node group also requires time to become available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjxvoNAokAbO42YihA_vqQ.png" /><figcaption>EKS cluster in AWS console</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ujszo6PvYMGcBfP_-IeMHw.png" /><figcaption>EKS managed node group</figcaption></figure><p>At this stage, the EKS cluster is ready to host the application.</p><h3>Create the Jenkins Deployment Pipeline</h3><p>The build pipeline has already created the Docker image and pushed it to ECR. Now we need to deploy that image to the EKS cluster.</p><p>This pipeline performs the following activities:</p><p>Accepts the EKS cluster name as a parameter.<br>Authenticates with AWS.<br>Connects to the EKS cluster.<br>Uses Helm to install Prometheus and Grafana.<br>Applies the Argo CD manifests.<br>Changes the Argo CD service type to LoadBalancer.<br>Creates the required deployment environment.</p><p><a href="https://github.com/suman526/amazon-prime-devops-platform/blob/main/pipeline_script/deployment_pipeline">amazon-prime-devops-platform/pipeline_script/deployment_pipeline at main · suman526/amazon-prime-devops-platform</a></p><p>The deployment pipeline accepts the cluster name as an input.</p><p>This makes the pipeline reusable because the cluster name can be changed without modifying the pipeline script. It also uses the AWS credentials stored in Jenkins. After authentication, it configures access to the EKS cluster.</p><p><strong>Install Prometheus and Grafana through Helm</strong></p><p>The deployment pipeline uses Helm charts to install Prometheus and Grafana. Helm is a Kubernetes package manager. It allows Kubernetes applications to be installed using charts rather than manually applying every resource.</p><p><strong>Apply the Argo CD manifest</strong></p><p>The deployment pipeline applies the Argo CD Kubernetes manifest. The Argo CD service is changed from NodePort to LoadBalancer. This allows AWS to provision a load balancer and provide an external endpoint for the Argo CD interface.</p><p>Prometheus and Grafana are also exposed through their configured LoadBalancer services.</p><h3>Create the deployment pipeline in Jenkins</h3><p>Open Jenkins Dashboard → Click New Item → Enter the name: deployment-pipeline → Select Pipeline and click OK.</p><p>Paste the deployment pipeline script from the original project repository or README file.</p><p>Save the pipeline → Run it once so Jenkins can load the parameters → Then refresh the page → The Build with Parameters option should be available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gERXTJBaymdvDJeU1QsyA.png" /></figure><p>Provide the EKS cluster name → Open Build with Parameters.</p><p>Enter the EKS cluster name created by Terraform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*raV1nKqeeunzOfhKwGsmtw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VuNn-QleYTE1YNhpYXQWRQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2N89pYdoa8GVrkfm5kyfEA.png" /></figure><h4>Verify deployment pipeline execution</h4><p>The pipeline connects to EKS and configures the monitoring and Argo CD components.</p><p>Connect to the Jenkins server using SSH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*9xtTO-usEN6O9FUnrEav9g.png" /></figure><p>Here we will create a shell script named: access.sh</p><p>This script is available in the GitHub repository :</p><p><a href="https://github.com/suman526/amazon-prime-devops-platform/blob/main/access.sh">amazon-prime-devops-platform/access.sh at main · suman526/amazon-prime-devops-platform</a></p><p>The script performs the AWS configuration and EKS access steps needed to retrieve the service URLs and credentials.</p><p>Create a file on the Jenkins server and Make the script executable and then Execute the script</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*xe_Di9R7LL1Lsf_NlcRkPw.png" /></figure><p>It retrieves the following information:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ondmZ6tac2DoPx8J-jwS-Q.png" /></figure><h4>Now we will configure Argo CD to deploy the Amazon Prime application.</h4><p>Argo CD is a GitOps-based continuous deployment tool for Kubernetes. It allows us to manage Kubernetes application deployments using a Git repository as the source of the desired state.</p><h4>Access the Argo CD URL</h4><p>Copy the Argo CD URL generated by the access script. Open it in a browser.</p><h4>Log in to Argo CD</h4><p>Use the username and password retrieved through the access script. After logging in, change the initial password. Navigate to the user information or password settings. Set a new password and sign in again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgmZHMco1dNi-gAor5Dybg.png" /></figure><p>Create a new Argo CD application</p><p>Navigate to Applications → Click New App → Provide an application name</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tnf7Z7HLeNsuiAXxk1U2lw.png" /></figure><p>Configure synchronization policy → Select the Automatic sync policy.</p><p>This allows Argo CD to synchronize the desired configuration from Git.</p><p>Configure the GitHub repository → Copy the GitHub repository URL.</p><p>The repository contains the Kubernetes deployment and service manifests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BaZrU9CaHXIL3UHamMt6cw.png" /></figure><p>Configure the manifest path → The source path must point to that directory.</p><p><strong>Use the exact directory name from the repository.</strong></p><h4>Configure the destination</h4><p>Select the Kubernetes cluster destination. → I have used the default one</p><p>If a different namespace is required, configure it consistently in the manifests and application settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sSC2QCbjyjHx6PQuhBAzVQ.png" /></figure><h4><strong>Configure the application image</strong></h4><p>The deployment manifest contains the Docker image URI from Amazon ECR → The image URI can be copied from the ECR repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jeblSBbzRxXEWF3N__jYtw.png" /></figure><p>After verifying the repository, path, destination and manifests, click Create.</p><p>Argo CD will begin synchronizing the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-6-6WnBcU3xVXs9qD686CQ.png" /></figure><h4>Verify the Kubernetes Deployment</h4><p>While Argo CD is synchronizing the application, we can verify the Kubernetes resources through kubectl.</p><p>Check the services → This displays the Kubernetes services across namespaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nf7_2oNIx5uMiMNw7GazAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35X3Bh83dqAS9WJey7M4yQ.png" /></figure><h3>Verify through Argo CD</h3><p>Argo CD also provides a graphical view of the Kubernetes resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*29ilusuyTFCAebbJ8JoH2A.png" /></figure><p>We can see:</p><p>- Application status.<br>- Deployment.<br>- Service.<br>- Pods.<br>- Resource synchronization.<br>- Application health.</p><p>This is one of the major advantages of using Argo CD.</p><p>Instead of checking every resource only through kubectl commands, we can also view the application and its resources through the Argo CD interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bH6EsK2efeIx2i5izaewxw.png" /></figure><h4>Access the Amazon Prime Clone Application</h4><p>After the application is synchronized, Kubernetes provisions the LoadBalancer service. Locate the service corresponding to the Amazon Prime application.</p><p>Therefore, access the application using:</p><pre>http://&lt;LOAD_BALANCER_ENDPOINT&gt;:3000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ooots7qMFkTR8NeBMGIzOQ.png" /></figure><p>Copy the external address → application listens on port 3000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4PC2i5TAwQZkJgKUKqwMw.png" /></figure><p>The Amazon Prime clone application should now be accessible in the browser.</p><h3>Access Prometheus and Grafana</h3><p>After the deployment pipeline has installed the monitoring tools, we can access Prometheus and Grafana through their LoadBalancer endpoints.</p><p>The access script provides the URLs.</p><h4>Access Prometheus</h4><p>Copy the Prometheus URL generated by the script → Open it in the browser.</p><p>Prometheus provides access to the metrics and query interface.</p><p>The monitoring setup is used to collect and query the available metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lb4jGRwHnNCGTLxZYmld-A.png" /></figure><h4>Access Grafana</h4><p>Copy the Grafana URL generated by the script → Grafana may take approximately 5–10 minutes to become accessible after the LoadBalancer service is created.</p><p>If the endpoint is not immediately accessible:</p><p>1. Verify that the Grafana service has an external endpoint.<br>2. Check whether the Grafana pods are running.<br>3. Allow time for the load balancer to become available.<br>4. Check the protocol configured for the endpoint.</p><p>The tutorial demonstrates accessing the endpoint using HTTP where the service is configured for HTTP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8144-x7ebyeItlT6SQk9Q.png" /></figure><p>Log in to Grafana → Use the credentials retrieved through the access script.</p><p>Change default credentials where appropriate and do not expose them in public documentation.</p><h4>Import the Node Exporter Dashboard in Grafana</h4><p>Now that Grafana is accessible, we can create a monitoring dashboard.</p><p>We will now create Node Exporter dashboard with ID: 1860</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MfRi4MNlnVE_SsBw3LRtCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_8_p9w9tVVjjCrtmyZJAg.png" /></figure><p><strong>Grafana will load the Node Exporter dashboard.</strong></p><p>Configure the dashboard time range: Refresh interval: 5 seconds.</p><p>This allows the dashboard values to refresh periodically.</p><h4>Verify monitoring metrics</h4><p>The dashboard displays available metrics such as:</p><p>- CPU utilization.<br>- Memory utilization.<br>- System load.<br>- Uptime.<br>- Network-related metrics.<br>- Other Node Exporter metrics.</p><p>The metrics update according to the configured scrape interval and dashboard refresh interval.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Ok-E9FcLCJY-LV4rtBdCA.png" /></figure><p><strong>At this stage, the application deployment and monitoring workflow is complete.</strong></p><h3>Cleanup Pipeline: Remove Application and Monitoring Resources</h3><p>The cleanup pipeline removes project resources to avoid unnecessary AWS charges.</p><p>The cleanup pipeline handles:</p><ol><li>Application deployment and services.</li><li>Argo CD applications and manifests.</li><li>Prometheus and Grafana installations.</li><li>Kubernetes namespaces.</li><li>Helm repositories.</li><li>ECR repository.</li><li>Scheduling KMS key deletion.</li></ol><h3>Create the Cleanup Pipeline</h3><ul><li>Open the GitHub repository and copy the cleanup script.</li><li>Open Jenkins Dashboard and click New Item.</li><li>Enter cleanup-pipeline.</li><li>Select Pipeline and paste the cleanup script.</li><li>Save the configuration.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCxn3kMshfawIN__TcEzNw.png" /></figure><h3>Run and Verify Cleanup</h3><p>Execute the pipeline and check the Jenkins logs.</p><p>Verify that application, Argo CD, Prometheus, Grafana, Kubernetes namespaces, and ECR resources have been removed.</p><h3>Terraform Cleanup: Destroy Jenkins and EKS Infrastructure</h3><p>The previous pipeline removes application resources, but the Jenkins EC2 instance and EKS infrastructure must be destroyed separately using Terraform.</p><p>Navigate to the EKS Terraform directory and execute:</p><pre>terraform destroy -auto-approve</pre><p>This removes the EKS cluster and its associated infrastructure.</p><p>Navigate to the EC2 Terraform directory and execute the same command</p><p>This removes the Jenkins EC2 instance and its Terraform-managed resources.</p><h4>Verify AWS Resource Deletion</h4><p>Check the AWS Console to confirm:</p><ul><li>EKS cluster and node groups are deleted.</li><li>Worker EC2 instances are terminated.</li><li>Jenkins EC2 instance is terminated.</li><li>Terraform-managed security groups and associated resources are removed.</li></ul><p>EKS deletion may take approximately 10–15 minutes, depending on the resources involved.</p><h4>This completes the cleanup process and the project</h4><p>This end-to-end AWS DevOps project gave me practical experience with the complete application delivery lifecycle. I started by configuring AWS access and creating the Jenkins server using Terraform. I then integrated GitHub, SonarQube, NPM and Aqua Trivy into Jenkins to automate the build process. After creating the Docker image, I pushed it to Amazon ECR with build-specific tags. Next, I provisioned an Amazon EKS cluster using Terraform and created a separate Jenkins deployment pipeline. I configured Argo CD to deploy the Amazon Prime clone application from GitHub and used Helm to install Prometheus and Grafana.</p><p>Finally, I imported the Node Exporter dashboard into Grafana, verified the monitoring metrics and performed cleanup of the Kubernetes and AWS resources. The most important part of this project was understanding how the complete workflow operates and learning how to investigate problems when individual components do not work as expected.</p><p>Biggest takeaway from this project will be that successful DevOps workflow is not just about deploying an application. It is about automating infrastructure, integrating tools, maintaining visibility, managing versions, troubleshooting failures and cleaning up resources responsibly.</p><p>GitHub Repository:</p><p><a href="https://github.com/suman526/amazon-prime-devops-platform/tree/main">GitHub - suman526/amazon-prime-devops-platform: A production-style DevOps implementation of a streaming application using Terraform, AWS EKS, Jenkins, Docker, ArgoCD, Prometheus and Grafana.</a></p><p>LinkedIn:</p><p><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1/">Suman Yadav - Rajasthan, India | Professional Profile | LinkedIn</a></p><p>Thank you for reading!</p><p>If you are learning AWS and DevOps, I strongly recommend building an end-to-end project and implementing the complete workflow yourself. The practical integration and troubleshooting experience is extremely valuable.</p><p><strong>If you enjoyed this read, feel free to </strong><a href="https://buymeacoffee.com/sumanyadav"><strong>buy me a coffee </strong></a><strong>☕to support my writing!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae0f2ab152f9" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/building-an-end-to-end-devops-ci-cd-pipeline-for-an-amazon-prime-video-clone-using-aws-terraform-ae0f2ab152f9">Building an End-to-End DevOps CI/CD Pipeline for an Amazon Prime Video Clone Using AWS, Terraform…</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Docker to Kubernetes: A Real-World Two-Tier Flask + MySQL Deployment]]></title>
            <link>https://aws.plainenglish.io/from-docker-to-kubernetes-a-real-world-two-tier-flask-mysql-deployment-4ccd5b936899?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ccd5b936899</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Sun, 01 Feb 2026 10:32:07 GMT</pubDate>
            <atom:updated>2026-02-05T10:19:33.377Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jx_6kwBA7b4TIroTiDvv_A.png" /></figure><p>This blog documents a real-world journey of deploying a two-tier application — a Python Flask frontend with a MySQL database — moving from local Docker to Docker Compose and finally to a Kubernetes cluster.</p><p>It tackles the classic <em>“it works on my machine”</em> problem, which usually stems from environment inconsistencies like mismatched dependencies or configurations. Docker solves this by packaging the application and its dependencies into portable containers.</p><p>As the system grows, the blog demonstrates why Docker alone is not enough and how Kubernetes adds critical orchestration capabilities such as auto-healing and self-scaling. Overall, this serves as a practical guide to understanding how cloud-native systems fail — and how to rebuild them to be resilient.</p><h3>What Is Docker?</h3><p>Docker is an <strong>open-source containerization tool</strong> that allows you to:</p><ul><li>Package your application</li><li>Include all dependencies</li><li>Run it <strong>anywhere</strong>, regardless of OS</li></ul><h3>The Core Problem Docker Solves</h3><blockquote><em>“It works on my machine but not on the client’s machine.”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hvW4RfYYKLMB4cfv-FebqA.png" /></figure><p>Our development system might be: Windows, macOS. But production server might be: Ubuntu Linux. Docker solves this by creating a <strong>container (a virtual box)</strong> that includes: Operating System, Application code, Required libraries, Runtime commands</p><p>This box is called a <strong>Container</strong>.</p><h3>Docker Concepts</h3><h3>1. Dockerfile</h3><p>A <strong>Dockerfile</strong> is a text file where you define everything your application needs: Base OS, Language runtime, Libraries, Commands to run the app</p><h3>2. Docker Image</h3><p>Built <strong>from Dockerfile, </strong>Acts like a <strong>template</strong></p><h3>3. Docker Container</h3><p>Created <strong>from an image, </strong>A running instance of your application</p><h3>WE CAN SIMPLY UNDERSTAND IT AS</h3><ul><li><strong>Dockerfile</strong> → Assignment instructions</li><li><strong>Docker Image</strong> → Photocopy of topper’s assignment</li><li><strong>Docker Container</strong> → Student submitting the assignment</li></ul><h3>LETS START WITH PRACTICAL</h3><h3>Application (Flask + MySQL)</h3><ul><li>Written in <strong>Python Flask</strong></li><li>Exposes APIs</li><li>Stores messages in <strong>MySQL</strong></li><li>Runs on <strong>port 5000</strong></li></ul><p>Flask acts like a <strong>waiter</strong>:</p><ul><li>Client requests → Flask → Flask talks to database → Database responds → Flask → Client</li></ul><h3>Infrastructure Prerequisite: AWS EC2</h3><h3>Why AWS EC2?</h3><p>We need:</p><ul><li>Ubuntu server, Public IP, Linux environment</li></ul><p>EC2 gives us a <strong>virtual Ubuntu server</strong>.</p><p><strong>Launch EC2 Instance &amp; Connect to EC2 via SSH</strong></p><pre>chmod 400 your-key.pem<br>ssh -i your-key.pem ubuntu@&lt;public-ip&gt;</pre><p>Type yes when prompted. You are now inside the EC2 server.</p><h3>Install Docker on Ubuntu</h3><h4>Error Faced</h4><pre>sudo apt install docker.io</pre><p><em>Package has no installation candidate</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*kh5dpBPpD7Cf_QlVN-35-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*88TRDCvhVf7xV62MzvwuVQ.png" /></figure><pre>sudo apt update<br>sudo apt install -y docker.io</pre><h4>When you try to do docker ps, it will throw permission denied error.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfFqKvqRnjclZcaggxAmnQ.png" /></figure><h4>GRANT PERMISSION TO UBUNTU USER</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*QJAKKmkeF_rI0AxJBTe1YQ.png" /></figure><h4>Install Docker on Ubuntu</h4><h3>Containerizing the Flask Application</h3><h4>Flask Application Requirements</h4><blockquote>Runs on port 5000</blockquote><blockquote>Connects to MySQL using environment variables</blockquote><blockquote>Must bind to 0.0.0.0 inside containers</blockquote><blockquote>app.run(host=”0.0.0.0&quot;, port=5000)</blockquote><h3>Clone Application Code</h3><pre>git clone &lt;github-repo-url&gt;<br>cd two-tier-flask-app</pre><h3>Create Dockerfile</h3><pre># Use an official Python runtime as the base image<br>FROM python:3.9-slim<br><br># Set the working directory in the container<br>WORKDIR /app<br><br># install required packages for system<br># upgrade pakages and install mysql libraries and a client for running mysql <br># in the end of this cmd will remove the temporary files list created in this installation process <br>RUN apt-get update \<br>    &amp;&amp; apt-get upgrade -y \<br>    &amp;&amp; apt-get install -y gcc default-libmysqlclient-dev pkg-config \<br>    &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br># Copy the requirements file into the container<br>COPY requirements.txt .<br><br># Install app dependencies<br>RUN pip install mysqlclient<br>RUN pip install --no-cache-dir -r requirements.txt<br><br># Copy the rest of the application code from source to container (target)<br>COPY . .<br><br># Specify the command to run your application<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mQ-I9id8LuSi37UOu67bpA.png" /></figure><h3>Build Docker Image</h3><pre>docker build -t flaskapp .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p_bB5rZGT_V4cQ5UNpT8Uw.png" /></figure><p>Check images:</p><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*lK7MuQhKgK3BBGXA2rkOQw.png" /></figure><h3>Run Flask Container</h3><pre>docker run -d -p 5000:5000 flaskapp:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ixm81dcDwODfk7HfrGosQA.png" /></figure><h3>Open Port in AWS Security Group</h3><ul><li>Inbound Rule → Port: 5000 → Source: Anywhere</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oM5zmEHzpN2ZMp4uYpm1dw.png" /></figure><p>Access:</p><pre>http://&lt;public-ip&gt;:5000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*LueRvTXefL7kHbpuvQnCLg.png" /></figure><h3>MySQL Connection Error</h3><blockquote><em>Flask cannot connect to MySQL, b</em>ecause <strong>MySQL container does not exist yet</strong>. And also Flask and MySQL are isolated.</blockquote><h3>Create Docker Network</h3><pre>docker network create twotier</pre><p>Stop created containers:</p><pre>docker kill &lt;container-id&gt;</pre><p>Build the docker app</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*M3huReBGGY_VjogRHhj9KA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uWFDp6f0hG6G03rQIrch0g.png" /></figure><h3>Critical Error: Container builds but immediately exits</h3><p>This was the most important failure till now.</p><pre>docker ps</pre><p>➡ Shows nothing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1eextJCrhlMAwQ3JlPGT3w.png" /></figure><h4>What this error really means?</h4><p>Docker containers live as long as their main process runs. If the process crashes or exits due to an exception or fails during startup. Docker stops the container immediately.</p><h3>Root Cause (Flask + MySQL dependency failure)</h3><p>Flask was starting before MySQL was available, or with: missing database, wrong credentials, insufficient privileges</p><p>Flask crashed → container exited.</p><h4>1: MySQL is NOT running, start it explicitly:</h4><p>Run Container with Network + Environment Variables</p><pre>docker run -d \<br> — name mysql \<br> — network twotier \<br> -e MYSQL_ROOT_PASSWORD=admin \<br> -e MYSQL_DATABASE=mydb \<br> -e MYSQL_USER=admin \<br> -e MYSQL_PASSWORD=admin \<br> mysql:8</pre><p>Wait 20–30 seconds<br>MySQL initializes internal tables on first startup.</p><h4>2: Verify the database exists</h4><pre>docker exec -it mysql mysql -u admin -padmin -e “SHOW DATABASES;”</pre><p>Expected output:</p><pre>mydb<br>information_schema<br>mysql<br>performance_schema<br>sys</pre><p>If mydb is missing:</p><pre>docker exec -it mysql mysql -u root -padmin -e “CREATE DATABASE mydb;”</pre><h4>3: Grant MySQL user privileges (VERY IMPORTANT)</h4><pre>docker exec -it mysql mysql -u root -padmin</pre><p><strong>Inside MySQL:</strong></p><pre>GRANT ALL PRIVILEGES ON mydb.* TO ‘admin’@’%’;<br>FLUSH PRIVILEGES;<br>EXIT;</pre><p><strong>Without this step, Flask will still crash even if MySQL is running.</strong></p><h4>4: Remove stale Flask container (name conflict fix)</h4><pre>docker rm -f flaskapp</pre><p>Verify:</p><pre>docker ps -a</pre><p>The old container must be gone.</p><h4>5: Start Flask container correctly</h4><pre>docker run -d \<br> — name flaskapp \<br> — network twotier \<br> -e MYSQL_HOST=mysql \<br> -e MYSQL_USER=admin \<br> -e MYSQL_PASSWORD=admin \<br> -e MYSQL_DB=mydb \<br> -p 5000:5000 \<br> flaskapp:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*whNzQtuedHEUbajYWnIaFw.png" /></figure><p>Now:</p><pre>docker ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HgXCg7Coy0pC_EU5Npkwvw.png" /></figure><p>➡ Both containers are RUNNING</p><p>When a container exits immediately, do not rebuild blindly.<br>Always validate dependency services, databases, and permissions first.</p><p>Access:</p><pre>http://&lt;public-ip&gt;:5000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ODU97Ys0DEKJndMfoEXHA.png" /></figure><h3>Application Works</h3><ul><li>Messages insert successfully, Data stored in MySQL</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JVu6RYIp4lavp1dqwvo40Q.png" /></figure><p>LETS CHECK ON MYSQL CONTAINER</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWQy6oOv1Pp1ksaVnYWF0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*zzLHqy02Pxhyz8Q3ZdxBKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*1a-ev8PN4HIX1yghEt539g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*BvYOzhH4o58mOArLnVaYvA.png" /></figure><h3>Push Image to Docker Hub</h3><pre>docker login</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PeYI1Sm51cjKmQ4zGz5MoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*dm8XpFTtzr1oBCgheh9Wvg.png" /></figure><pre>docker tag flaskapp yourusername/flaskapp:latest<br>docker push yourusername/flaskapp:latest</pre><p>Now anyone can pull and run our app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fTODvvMhMd4ldZWEq2m2jA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0XKjdLGJE_BwIKmAJ35Gzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UC5nWwrbLXpAeDwp_7A6uw.png" /></figure><h3>Docker Compose</h3><h3>Why Docker Compose?</h3><p>Problems without Docker Compose:</p><ul><li>Multiple docker run commands, Manual networking, Hard to manage</li></ul><p>Solution: <strong>Docker Compose</strong></p><h4>Install Docker Compose</h4><pre>sudo apt install docker-compose</pre><h4>Create docker-compose.yml</h4><pre>version: “3.9”<br>services:<br> mysql:<br> image: mysql:5.7<br> restart: always<br> environment:<br> MYSQL_ROOT_PASSWORD: root<br> MYSQL_DATABASE: devops<br> volumes:<br> - mysql-data:/var/lib/mysql<br>flask-app:<br> image: sumany15/flaskapp:latest<br> depends_on:<br> - mysql<br> environment:<br> MYSQL_HOST: mysql<br> MYSQL_USER: root<br> MYSQL_PASSWORD: root<br> MYSQL_DB: devops<br> ports:<br> - &quot;5000:5000&quot;<br>volumes:<br> mysql-data:</pre><p>Docker Compose automatically creates a network, Services communicate via service names, localhost is not valid for inter-container communication</p><h3>Run with Docker Compose</h3><pre>docker-compose up -d</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKWAI-CCxmzvA7sCqy5Pjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QZyTbkZmfK7KQrMola4Rtw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvMQNJ6smkLimSZ85Ma3Iw.png" /></figure><h4><strong>TILL NOW WE ARE DONE WITH:</strong></h4><blockquote>Docker fundamentals → Docker errors &amp; fixes → Image creation → Container networking → Docker Hub publishing → Docker Compose orchestration</blockquote><h3>Kubernetes Deployment</h3><ul><li>Frontend / Backend → <strong>Flask Application</strong></li><li>Database → <strong>MySQL</strong></li><li>Both are to be deployed on a <strong>Kubernetes cluster</strong></li></ul><h3>Why Kubernetes When We Already Have Docker?</h3><ul><li>Docker containers <strong>crash</strong></li><li>Docker containers <strong>do not scale automatically</strong></li><li>Managing containers manually is painful</li></ul><p>Kubernetes solves this. Kubernetes does <strong>not replace Docker</strong>. Instead, it <strong>manages Docker containers</strong> for us.</p><h3>1. Pod — The Smallest Unit in Kubernetes</h3><ul><li>Applications still run inside <strong>containers, </strong>Containers run inside <strong>Pods</strong></li><li>Pod = a <em>home</em> for containers</li></ul><p>A Pod can contain</p><ul><li>Main application container, Helper / sidecar containers, Shared environment variables, Shared storage</li></ul><p>We scale Pods, not containers.</p><h3>2. Container Runtime Interface (CRI)</h3><p>Kubernetes does not directly use Docker. Instead:</p><ul><li>Kubernetes → CRI → containerd</li><li>Docker internally also uses containerd</li></ul><p>That’s why Kubernetes can run:</p><ul><li>Docker containers, containerd containers, other OCI-compliant containers</li></ul><h3>3. Deployment — Scaling &amp; Self-Healing</h3><p>Pods cannot scale automatically &amp; are not ideal for production</p><p>Solution: <strong>Deployment</strong></p><p>A <strong>Deployment</strong> defines the <em>desired state</em>:</p><ul><li>How many Pods should run, Which image to use, Environment variables, Restart policy</li></ul><p>Benefits:</p><ul><li>Auto-healing (crashed Pods restart), Scaling (1 → 10 → 100 replicas), Rolling updates</li></ul><h3>4. Service — Network Access</h3><p>Pods:</p><ul><li>Have dynamic IPs</li><li>Die and recreate</li><li>Cannot be accessed directly</li></ul><p>A <strong>Service</strong>:</p><ul><li>Groups Pods</li><li>Provides a stable IP / DNS</li><li>Exposes Pods internally or externally</li></ul><p>We can <strong>never access Pods directly</strong> — always via a Service.</p><h3>5. Persistent Volume (PV) &amp; Persistent Volume Claim (PVC)</h3><p>Databases need <strong>storage</strong>.</p><p>Containers are temporary can also lose data when restarted</p><p>Persistent Volumes:</p><ul><li>Connect Kubernetes Pods to disk storage</li><li>Keep data safe even if Pods crash</li></ul><h4>The Five Mandatory Kubernetes Objects</h4><p>To run a two-tier app, we need:</p><ol><li><strong>Pod</strong> — runs container</li><li><strong>Deployment</strong> — scaling &amp; healing</li><li><strong>Service</strong> — networking</li><li><strong>Persistent Volume (PV)</strong> — storage</li><li><strong>Persistent Volume Claim (PVC)</strong> — storage request</li></ol><p>That’s it.</p><h3>KUBERNETES CLUSTER SETUP</h3><p>To deploy our two-tier application, we first need a Kubernetes cluster. <br>This setup uses 1 master node and 2 worker nodes.</p><p>I have already explained the complete Kubernetes installation process in detail. <br>You can follow it from the link below:</p><p>🔗<a href="https://github.com/suman526/k8s-setup-for-ubuntu.git">https://github.com/suman526/k8s-setup-for-ubuntu.git</a></p><p>Once the cluster is ready, continue with the application deployment steps.</p><h3>Kubernetes Uses YAML (Manifest Files)</h3><p>Anything we will create in Kubernetes must be defined using <strong>YAML manifest files</strong>.</p><p>Examples:</p><ul><li>Pod → pod.yaml</li><li>Deployment → deployment.yaml</li><li>Service → service.yaml</li></ul><p>These files are also called <strong>manifest files</strong>.</p><h3>LETS START OUR DEPLOYMENT</h3><p>Clone the Repository</p><pre>git clone &lt;two-tier-flask-repo&gt;<br>cd two-tier-flask-app</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJgpcsl9nwwI7slxrr7FOQ.png" /></figure><pre>mkdir k8s<br>cd k8s</pre><h3>Deploying MySQL</h3><p>MySQL Deployment in Kubernetes</p><p>mysql-deployment.yml</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: mysql-deployment<br>  labels:<br>    app: mysql<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: mysql<br>  template:<br>    metadata:<br>      labels:<br>        app: mysql<br>    spec:<br>      containers:<br>        - name: mysql<br>          image: mysql:latest<br>          env:<br>            - name: MYSQL_ROOT_PASSWORD<br>              value: &quot;admin&quot;<br>            - name: MYSQL_DATABASE<br>              value: &quot;mydb&quot;<br>            - name: MYSQL_USER<br>              value: &quot;admin&quot;<br>            - name: MYSQL_PASSWORD<br>              value: &quot;admin&quot;<br>          ports:<br>            - containerPort: 3306</pre><p>MySQL Service</p><p>mysql-svc.yml</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: mysql<br>spec:<br>  selector:<br>    app: mysql<br>  ports:<br>    - port: 3306<br>      targetPort: 3306</pre><h3>Apply MySQL</h3><pre>kubectl apply -f mysql-deployment.yaml<br>kubectl apply -f mysql-service.yaml</pre><pre>kubectl get pods -w</pre><pre>mysql-deployment-xxxxx 1/1 Running</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYNIDVTNDj1w1X27gPIpvQ.png" /></figure><p>Get Linux shell inside MySQL pod</p><pre>kubectl get pods   #copy mysql pod name</pre><pre>kubectl exec -it mysql-pod — /bin/bash</pre><p>Then:</p><pre>mysql -u root -p</pre><p>Then enter password. Password here</p><pre>admin</pre><p>(we can use this as well) Instead of copying pod name every time:</p><pre>kubectl exec -it &lt;mysql-pod-name&gt; -- mysql -uroot -p</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYNIDVTNDj1w1X27gPIpvQ.png" /></figure><p>Grant FULL privileges to admin (we are currently giving full priviledge)</p><pre>CREATE USER IF NOT EXISTS ‘admin’@’%’ IDENTIFIED BY ‘admin’;<br>GRANT ALL PRIVILEGES ON *.* TO ‘admin’@’%’ WITH GRANT OPTION;<br>FLUSH PRIVILEGES;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*cWInPtQktiB46TS_oPR8Uw.png" /></figure><h3>Production-safe alternative</h3><p>Instead of full access:</p><pre>GRANT SELECT, INSERT, UPDATE, DELETE ON mydb.* TO &#39;admin&#39;@&#39;%&#39;;<br>FLUSH PRIVILEGES;</pre><h4>MySQL itself is healthy and running correctly</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F8X1R9ZoyvNcfTri8xFjiA.png" /></figure><h3>Deployment Manifest (Flask App)</h3><h4>Labels &amp; Selectors</h4><ul><li>Labels identify Pods</li><li>Selectors match Pods</li><li>Deployment manages Pods using labels</li></ul><p>two-tier-app-deployment.yml</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: two-tier-app<br>  labels:<br>    app: two-tier-app<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: two-tier-app<br>  template:<br>    metadata:<br>      labels:<br>        app: two-tier-app<br>    spec:<br>      containers:<br>        - name: two-tier-app<br>          image: sumany15/flaskapp:latest<br>          env:<br>            - name: MYSQL_HOST<br>              value: &quot;mysql&quot;          # this is your mysql&#39;s service clusture IP, Make sure to change it with yours<br>            - name: MYSQL_PASSWORD<br>              value: &quot;admin&quot;<br>            - name: MYSQL_USER<br>              value: &quot;root&quot;<br>            - name: MYSQL_DB<br>              value: &quot;mydb&quot;<br>          ports:<br>            - containerPort: 5000<br>          imagePullPolicy: Always</pre><p>Apply:</p><pre>kubectl apply -f two-tier-app-deployment.yml</pre><p>Check Pods:</p><pre>kubectl get pods</pre><h3>Exposing Application Using Service (NodePort)</h3><h4>Why Service?</h4><ul><li>Pods have no fixed IP</li><li>Service provides stable access</li></ul><h4><strong>NodePort: Manual External Access</strong></h4><p><strong>NodePort</strong> is a service type used to expose an application by opening a specific port on every worker node in the cluster. It is generally “free” from a service standpoint because it uses the existing IP and ports of your EC2 instances.</p><p>• <strong>How it Works:</strong> In this deployment, the Flask application is exposed using type: NodePort with a designated nodePort: 30004. This maps the internal container port (5000) to a port accessible on the host machine.</p><p>• <strong>Accessing the App:</strong> Users can reach the application via http://&lt;Worker-Node-IP&gt;:30004</p><p>two-tier-app-svc.yml (TYPE: NODEPORT)</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: two-tier-app-service<br>spec:<br>  selector:<br>    app: two-tier-app<br>  ports:<br>    - protocol: TCP<br>      port: 80<br>      targetPort: 5000<br>      nodePort: 30004<br>  type: NodePort</pre><p>Open port <strong>30004 </strong>in worker node security group.</p><p>Apply and check:</p><pre>kubectl apply -f two-tier-app-svc.yml<br>kubectl get svc</pre><p>Access app:</p><pre>http://&lt;worker-node-ip&gt;:30004</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZo-uhLqLXwmJaSk1bX_aQ.png" /></figure><h4><strong>LoadBalancer Service</strong> for our <strong>Flask application</strong></h4><p><strong>LoadBalancer: Cloud-Native Integration</strong></p><p>The <strong>LoadBalancer</strong> service is the standard way to expose a service to the internet in a cloud environment. Kubernetes communicates with AWS to automatically provision an Elastic Load Balancer (ELB). In a real-world scenario, the type: LoadBalancer command triggers an API call to the cloud provider. It typically incurs additional hourly costs from AWS for the provisioned ELB resource, along with data transfer fees.</p><p>• <strong>How it Works:</strong> When the service YAML is updated to type: LoadBalancer, Kubernetes facilitates the creation of an external entry point.</p><p>• <strong>Key Difference:</strong> While it still assigns a nodePort (for example, 30148) internally, it is designed to work with cloud providers to automatically route external traffic to the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*Tic_bs8BbOsk2XwOfXfD9A.png" /></figure><p>Apply and check:</p><pre>kubectl apply -f two-tier-app-svc.yml<br>kubectl get svc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RytOlYCnPsEAYn9WTdaOjA.png" /></figure><p>Now in our Flask service YAML, we have:</p><pre>type: LoadBalancer</pre><pre>kubectl get svc </pre><p>nodePort: 30148</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2B4d5VkVvzIJoj-nE9GGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_HEE9ZV09aWPpihrEN1HVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtWBjkkO9QoP1E8AjB09nQ.png" /></figure><p><strong>Lets check the database in action:</strong></p><p>Use kubectl get pods to ensure the MySQL deployment is in a Running state. Enter the database shell directly from the terminal using: kubectl exec -it &lt;mysql-pod-name&gt; -- mysql -u root -p or we can also enter with the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FI7POmCSl-R83eGC02rgEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*s1jklH5R3viAseaAcnYa9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kk-bLRRe8pJ3whUL0rYaTQ.png" /></figure><p><strong>WE HAVE SUCCESSFULLY DEPLOYED OUR APP ON KUBRNETES CLUSTER</strong></p><p>We have moved past the “it works on my machine” stage to achieve a setup that supports <strong>auto-healing and automatic scaling</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLHFs-tYPZsmtDxdnbpmPw.png" /></figure><p><strong>The Path Ahead: Moving to AWS EKS</strong></p><p>While our current setup uses a self-managed cluster on AWS EC2 instances, the next logical step in this evolution is to move toward a <strong>managed service</strong>,.</p><p><strong>Future plans for this project include deploying the entire two-tier application onto AWS EKS (Elastic Kubernetes Service).</strong> Transitioning to <strong>EKS</strong> will allow us to offload the management of the Kubernetes control plane to AWS, ensuring even higher availability and seamless integration with other cloud-native tools. This next phase will focus on automating the infrastructure even further, moving away from manual node setup to a fully managed, scalable cloud environment.</p><h3>GitHub Repository</h3><p>Complete source code are available on GitHub:<br><a href="https://github.com/suman526/two-tier-flask-app-for-k8s/tree/main">https://github.com/suman526/two-tier-flask-app-for-k8s/tree/main</a></p><p>THANK YOU!</p><p><strong>Let’s stay connected!</strong></p><p><strong>You can find me on LinkedIn🤝</strong></p><p><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1/"><strong>https://www.linkedin.com/in/suman-yadav-2972a71a1/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ccd5b936899" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/from-docker-to-kubernetes-a-real-world-two-tier-flask-mysql-deployment-4ccd5b936899">From Docker to Kubernetes: A Real-World Two-Tier Flask + MySQL Deployment</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FROM COMMIT TO CONTAINER: Complete CI/CD Pipeline Using GitHub, Jenkins, SonarQube, and Docker on…]]></title>
            <link>https://towardsaws.com/from-commit-to-container-complete-ci-cd-pipeline-using-github-jenkins-sonarqube-and-docker-on-c0e84be5ab8e?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0e84be5ab8e</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[sonarqube]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Fri, 02 Jan 2026 21:54:02 GMT</pubDate>
            <atom:updated>2026-01-11T14:36:46.855Z</atom:updated>
            <content:encoded><![CDATA[<h3>FROM COMMIT TO CONTAINER: Complete CI/CD Pipeline Using GitHub, Jenkins, SonarQube, and Docker on AWS EC2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAFMCAxpVXxFZjZZuEbBzQ.png" /></figure><p>For most DevOps engineers, the real goal is pretty simple: we want a clean, fully automated CI/CD pipeline that “just works.” Imagine pushing your code to GitHub and then sitting back while everything else happens automatically — the code is fetched, checked for issues, packaged, and deployed as a live application without you clicking a single extra button.</p><p>This blog walks you through building an automated <strong>CI/CD pipeline</strong> using <strong>AWS EC2, Jenkins, SonarQube, and Docker</strong>. We begin by provisioning three EC2 instances to host Jenkins, SonarQube, and the Docker deployment environment. The application code is pushed to GitHub, where a <strong>webhook</strong> automatically triggers Jenkins to pull the latest changes. The source code is then analyzed in SonarQube to detect bugs and security vulnerabilities. If the quality gates pass, the code is transferred to the Docker server, built into an image, and deployed as a running container. Finally, the deployment is verified by accessing the website through a web browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OjkRobZ_TDf2ZBWMmF8V7Q.png" /><figcaption>ARCHITECTURE</figcaption></figure><h3>REQUIREMENTS</h3><blockquote>We will use <strong>three EC2 instances</strong>:</blockquote><blockquote>Jenkins Server → CI/CD tool</blockquote><blockquote>SonarQube → ServerStatic code analysis</blockquote><blockquote>Docker Server → Application deployment</blockquote><blockquote>At the end, we will verify the deployment in a <strong>browser</strong>.</blockquote><h3>Laying the Foundation</h3><h3>Prepare Website Template</h3><p>Search <strong>Free CSS template → </strong>Download any HTML template → Extract it anywhere on desktop</p><p>This is the app we will deploy</p><h3>Create GitHub Repository</h3><p>Login to GitHub → Click <strong>New Repository → </strong>Example name: jenkins-sonarqube-docker → Upload your website template files → Commit changes</p><p>This repository will be used in Jenkins.</p><h3>Create Three EC2 Instances</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*WWSys46QQJAKMY9H8Pp4ww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_SzPv7QaQzFDgzFWSotjQA.png" /></figure><h3>1️⃣Building the Automation Engine️⃣</h3><h3>SSH into Jenkins Instance</h3><p>Copy public IP → open terminal</p><pre>ssh -i jenkins-sonarqube-docker.pem ubuntu@PUBLIC_IP</pre><p>If <strong>permission denied → </strong>Fix by running</p><pre>chmod 400 jenkins-sonarqube-docker.pem</pre><p>Then connect again.</p><h3>Install Jenkins</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*vQjkmP-xOj7icKx5BExOeQ.png" /><figcaption>LOGIN TO JENKINS INSTANCE</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*8uCsLFBu8VTlV5XK8hX35A.png" /><figcaption>Update server</figcaption></figure><p>Go to jenkins.io → Documentation → Installing Jenkins → Linux → Debian/Ubuntu</p><p>Copy installation commands and run them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*ZbCQkm281nAN-NnXopAnLg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*fpO-M80Lk2h7YB9y3ZzgyQ.png" /></figure><p>Start Jenkins and verify:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-XFwm0awzbQNwJB4WsgUQA.png" /></figure><h3>Allow Jenkins Port 8080</h3><p>Go to AWS EC2 console: Security → Security Groups → Edit inbound rules</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nz2Rx3XXaTtDUqi1ulzrHg.png" /></figure><h3>Unlock Jenkins and Install Plugins</h3><p>Visit in browser:</p><pre>http://PUBLIC_IP:8080</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VDxr_1LA8Pd5ZoJ6Y6imaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*P8Uflmvxz0i6JfFZwTXHdQ.png" /></figure><h4>Find admin password:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CK97wP-BlhwxlXldX1YV7A.png" /></figure><p>Paste → continue → Install suggested plugins</p><p>Create admin user → Finish setup.</p><h3>Create Jenkins Freestyle Project</h3><ol><li>Go to <strong>New Item → </strong>Name: jenkins-automated-pipeline</li><li>Select <strong>Freestyle project → </strong>Source Code Management → Git → Paste GitHub repo URL → Branch: main → Save</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*D4fg4qdy_q_poaNp2J4G6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*mimuE_TT8kGlLF6w6PyHrA.png" /></figure><h3>Building the Automation Engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*67NQRo_uHzHqZE0U0lp-tQ.png" /></figure><h3>Add GitHub Webhook Trigger</h3><p>Go to Project → Configure → Enable: GitHub hook trigger for GITScm polling</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*-v1JOyR7cyYmeLNweulPaQ.png" /></figure><p><strong>Now go to GitHub repo:</strong></p><p>Settings → Webhooks → Add webhook → Payload URL:</p><pre>http://JENKINS_PUBLIC_IP:8080/github-webhook/</pre><p>Select events: Push, Pull Request → Add webhook</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*37azTYh13gJ1CuVPw0KvuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*dTjQhffCxsYDxJRnt4TGIA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*n82oFgLpnqqN1Cfd88YLuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*2J6CZKzBLgt5TRt_BFhZ7g.png" /></figure><h3>Confirming the GitHub-to-Jenkins Connection</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*MGxBx6E1cyZTRPKgRCz4zA.png" /></figure><h3>Verify Webhook Automation</h3><p>Add file in GitHub:</p><pre>test.txt</pre><p>Commit file.</p><p>Go to Jenkins:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*TXIbK7GQBe9kSeT1YzKD5A.png" /></figure><h4>Build is triggered : Confirmed new file exists</h4><h3>2️⃣ Installing the Quality Gate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*gYM3QGwofIi8oFCisaQ0HA.png" /></figure><h3>Launch SonarQube Instance</h3><p>SSH into SonarQube server → Change hostname:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*OtSvBIuFu7c3u1hhYdH7bg.png" /></figure><p>Update system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*eYu2SC7Pw4rYshGC54FPAg.png" /></figure><p>Install Java 17 JRE:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*_rhnVnMbFXodTeT9K2m4yA.png" /></figure><p>Add a dedicated user for SonarQube</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*qRGpqo956ZtXUaAuBhNuNQ.png" /></figure><h3>Download SonarQube</h3><ol><li>Open sonarQube download page → Copy link → Use wget:</li></ol><pre>cd /opt<br>sudo wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-9.9.0.65466.zip</pre><p>Install unzip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*ZB2E023CA6Vol_NFH_YKMw.png" /></figure><p>Unzip:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*cXu1zaMSPxwFWt_PHGPnrg.png" /></figure><h4>Go to folder → bin → linux-x86–64</h4><p>Run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K25oqIIOumDgWTC82oYg3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*ooDTiHBtuCXNQMJmVl1NsA.png" /></figure><h3>Allow SonarQube Port 9000</h3><p>Security group inbound rule: Port 9000</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXJdEGPWKQE6yqnHS3Ptiw.png" /></figure><h4>Access in browser:</h4><pre>http://PUBLIC_IP:9000</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*Bl7zH64Jvalp6s7HiU17NQ.png" /></figure><p>Login default:</p><pre>username: admin<br>password: admin</pre><p>Change password.</p><h3>Create SonarQube Project and Token</h3><p>Inside SonarQube: Projects → Manually create project</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*SZtzk3rTLsB79h8_30jHyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*vQ1YjpPLq6abLQTMAm69YA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*vQ1YjpPLq6abLQTMAm69YA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*bg_F6bdhODYxXlOsDmalIQ.png" /></figure><p>Save: project key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*31-Hwznp_Eaeb7gmtMDCsA.png" /></figure><p>Save: token</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6GpjfV6RGhK3Gg1SqgcLg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*twIPdWerFXV4d3AtH94Bnw.png" /></figure><h4>Copy both to notepad.</h4><h3>3️⃣ Integrate SonarQube with Jenkins</h3><p>In Jenkins: Manage Jenkins → Plugin Manager → Available</p><p>Install:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yk27rkelogNKcvK5DqoKHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRNxHE7N36xJBh4rHsg_-A.png" /></figure><p>Then:</p><p>Manage Jenkins → Global Tool Configuration</p><p>Add:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ha6cvWXc654HPFnLVEo-Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*GFJE1HaDQjMhVKu_QU2zlA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D8Rwm91gHYcoC0vbQvrTmw.png" /></figure><p>Then:</p><p>Manage Jenkins → Configure System</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*eT9ObN5Kr5IBnao2B4vjHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afoUkOMJomriKQT1cJzN_A.png" /></figure><p>Add:</p><ul><li>SonarQube server URL</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*T5pO0mWGL9mUQILeDd5-Iw.png" /></figure><ul><li>Sonar token as Secret Text credential (copied while creating project in sonarqube)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0QFVLMkUYqbnQR-dEuRCqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yqC2jFxI_y6IvTV-EuHffw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rno8QGL622Y4OJs4647_Xw.png" /></figure><h3>Add Sonar Scanner Step in Pipeline</h3><p>In project → Configure:</p><p>Add build step:</p><pre>Execute SonarQube Scanner</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*eT9ObN5Kr5IBnao2B4vjHA.png" /></figure><p>Paste:</p><ul><li>project key, token id</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afoUkOMJomriKQT1cJzN_A.png" /></figure><h4>Build → SonarQube scans code successfully ✔</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5cVfliMKTxeH1h9KFdOvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pHNkQPHXfCWgd3yQhV7zMQ.png" /></figure><h3>4️⃣ Delivering the Final Product</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*elNPFdUj3Q5R_YqQj0KllA.png" /></figure><h3>️Configure Docker Server</h3><p>SSH into docker instance:</p><pre>ssh -i key_name.pem ubuntu@PUBLIC_IP</pre><p>Set hostname:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*RYct3cItfA4Zf3Sz2tYUVw.png" /></figure><p>Update:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*cM1Bw8ELosZJRuVz_Ckprg.png" /></figure><p>Install docker repository → keys → install Docker</p><p>Add user to docker group:</p><pre>sudo usermod -aG docker ubuntu<br>newgrp docker</pre><h4>Verify:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKl-EfwSuLit0u7G_6uHgQ.png" /></figure><h3>Enabling Jenkins to Control Docker</h3><p>From Jenkins user:</p><p>Trying SSH to docker → permission denied</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*SiMQfv9ooY_8cro4JZdJ6Q.png" /></figure><p>️Fixed it by: CREATE A NEW USER</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Bzqg7c2GnV1EdcoOczPPBg.png" /></figure><p>Editing sshd_config on docker server</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*75URGHojOGO_O4mS9mQfDA.png" /></figure><p>Enabling password authentication in the file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*5sg2mTsdZgqd2ZaiTAqSyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*ESOpn-7Z3bExHz7vkuALPw.png" /></figure><p>Restarting ssh service</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*WGok_b5iN_f31K_CyV0m4g.png" /></figure><h4>If still permission denied → Generate SSH key on Jenkins EC2</h4><p>Connect to Jenkins EC2 (where Jenkins is installed) and run:</p><pre>ssh-keygen -t rsa</pre><h4>Copy Jenkins SSH key to Docker EC2</h4><pre>ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@&lt;docker-ec2-ip&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ow4oYrm5fP2mVt8JAg9SKw.png" /></figure><p>This copies our Jenkins public key into the Docker EC2 authorized keys.</p><h4>If ssh-copy-id command errors</h4><ol><li>On <strong>Jenkins EC2</strong>: cat ~/.ssh/id_rsa.pub</li><li>Copy the output text</li><li>On <strong>Docker EC2</strong>: mkdir -p ~/.ssh nano ~/.ssh/authorized_keys</li></ol><p>Paste the copied text, save with <strong>Ctrl+O + Enter + Ctrl+X</strong></p><p>Fix permissions: chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*shNYLln0-8diKrwyIypxbw.png" /></figure><p>NOW TRY TO SSH:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*RWvorh9v4Igvet9JSBl-6w.png" /></figure><h4>SSH success 🎉</h4><h3>5️⃣ Add Docker Server to Jenkins</h3><ul><li><strong>Server Group Name:</strong> docker-server</li><li><strong>Server Name:</strong> any label like docker1</li><li><strong>Server IP:</strong> 13.xx.xx.xx (your Docker EC2 public IP)️</li><li><strong>SSH Port:</strong> 22</li><li><strong>User Name:</strong> ubuntu</li><li><strong>Password:</strong> <strong>empty</strong> (leave blank)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*543WbYCQPI0NScOrviHoWQ.png" /></figure><h4>Now add SSH key credentials:</h4><p><strong>Manage Jenkins → Credentials → System → Global</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEyBuQRYy4rTaFt7yvWheA.png" /></figure><ol><li><strong>Add Credentials → Kind:</strong> SSH Username with private key</li><li><strong>Username:</strong>ubuntu</li><li><strong>Private key:</strong> <em>Enter directly</em></li><li>Paste your <strong>.pem key contents</strong></li><li><strong>ID:</strong> for example docker-key</li><li>Save</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsinoWpjLwQ_a54LvBTLIA.png" /></figure><h4>NEXT</h4><p>Manage Jenkins → Plugins → Available</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*P9sNnMIe4UbQGeEiNsRSNg.png" /></figure><h3>After plugin install — configure job again</h3><p>Open your job → <strong>Configure → </strong>Scroll to <strong>Build Environment → Use SSH Agent</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*874o4-M2Kksxy9NvIm9HWA.png" /></figure><p>In <strong>Credentials</strong> dropdown select: docker-key</p><h4>Test connection with simple command</h4><p>In <strong>Build Steps → Execute shell / Remote shell</strong></p><pre>ssh -o StrictHostKeyChecking=no ubuntu@&lt;docker-ec2-ip&gt; “hostname &amp;&amp; touch jenkins_test_file.txt”</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K1MovS9_GINBK2eYxl0PgQ.png" /></figure><p>Save → Build Now.</p><h4>Then check on Docker EC2:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*Hm_rs99TB20ChXYznXNXBA.png" /></figure><h4>File successfully created on docker server.</h4><h3>6️⃣ Create Dockerfile in GitHub Repo</h3><p>Add Dockerfile:</p><pre>FROM nginx<br>COPY . /usr/share/nginx/html</pre><p>Commit → webhook builds pipeline automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kw4BX_KW5QziDSaYB51R6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cp5cljB2yKoQpZZ0g7whRg.png" /></figure><h3>Great — We’re now at the <strong>deployment</strong> stage</h3><h4>Jenkins → SSH to Docker EC2 → build Docker image → run container</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*CWkggssYQmHTtG0wKureaA.png" /></figure><h3>Transfer Website Files Using SCP</h3><p>In Jenkins build step:</p><pre>scp -R * ubuntu@DOCKER_IP:/home/ubuntu/website</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OvYZRYOOTCEimZubI4oEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGzhlsHIf6SFwo6rEvX82Q.png" /></figure><h3>Build Docker Image &amp; Run Container</h3><p>In Jenkins: <strong>Configure → Build Steps → Add build step → Execute shell</strong></p><pre>ssh -o StrictHostKeyChecking=no ubuntu@&lt;DOCKER_IP&gt; &quot;<br>cd /home/ubuntu/website &amp;&amp;<br>docker build -t mywebsite . &amp;&amp;<br>docker rm -f mywebsite || true &amp;&amp;<br>docker run -d -p 8085:80 --name onix-website mywebsite<br>&quot;</pre><p>Open port 8085 in EC2 security rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x-fvjJ8UHsLwnGMhcZ2aBA.png" /></figure><h3>Access app:</h3><pre>http://PUBLIC_IP:8085</pre><h3>!! Website worked !!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h37aNpxSV0VlEVhsPE7zWg.png" /><figcaption>container launched</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOIB8_t1vQnX7ddGmAwsXw.png" /></figure><h3>Final Pipeline Summary implemented in this project:</h3><blockquote>In short, this project shows how powerful a fully automated CI/CD pipeline can be. With GitHub, Jenkins, SonarQube, Docker, and AWS working together, a single code push can trigger everything from testing to deployment.</blockquote><blockquote>✔ GitHub Push: Code changes trigger the pipeline.</blockquote><blockquote>✓ Jenkins Pull: Jenkins automatically fetches the latest code.</blockquote><blockquote>✔ SonarQube Scan: Code quality is verified.</blockquote><blockquote>✔ Docker Deployment:Application is containerised and launched.</blockquote><blockquote>✔ Browser Access: The live website is successfully served.</blockquote><h3>GitHub Repository</h3><p>The complete source code is available on GitHub:<br><a href="https://github.com/suman526/jenkins-sonarqube-docker">https://github.com/suman526/jenkins-sonarqube-docker</a></p><p>THANK YOU!</p><p><strong>Let’s stay connected! You can find me on LinkedIn🤝</strong></p><p><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1/"><strong>https://www.linkedin.com/in/suman-yadav-2972a71a1/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0e84be5ab8e" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/from-commit-to-container-complete-ci-cd-pipeline-using-github-jenkins-sonarqube-and-docker-on-c0e84be5ab8e">FROM COMMIT TO CONTAINER: Complete CI/CD Pipeline Using GitHub, Jenkins, SonarQube, and Docker on…</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Daily Cloud Health Check Automation using AWS, Linux, Bash, and Cron]]></title>
            <link>https://towardsaws.com/daily-cloud-health-check-automation-using-aws-linux-bash-and-cron-680b28be3af5?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/680b28be3af5</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[cron]]></category>
            <category><![CDATA[bash]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Thu, 25 Dec 2025 17:25:45 GMT</pubDate>
            <atom:updated>2025-12-28T03:30:28.646Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rZ5XItjRALmLm9aQHjXZ0A.png" /></figure><p>In Cloud Operations, engineers perform routine health checks every day to ensure servers are stable and issues are detected early.</p><p>The goal of this project is to automate basic system checks on an EC2 instance, log system metrics such as CPU usage, memory usage, and disk utilization, and configure alerts using CloudWatch and SNS so that issues can be identified without manual monitoring.</p><h3>Tools &amp; Technologies Used</h3><blockquote>AWS EC2 — Virtual server in the cloud</blockquote><blockquote>Linux (Ubuntu) — Operating system on EC2</blockquote><blockquote>Bash Scripting — For automation</blockquote><blockquote>Cron — For scheduling tasks</blockquote><blockquote>AWS CloudWatch — Monitoring service</blockquote><blockquote>AWS SNS — Email notifications</blockquote><blockquote>Git &amp; GitHub — Version control and documentation</blockquote><h3>Architecture Overview</h3><p>This consists of an EC2 instance where a Bash script runs daily via cron.<br>The script collects system health metrics and stores them in log files.<br>CloudWatch monitors CPU utilization and triggers SNS email alerts if thresholds are breached.</p><h3>Launching an EC2 Instance</h3><p>An EC2 instance is required to simulate a cloud server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jwNsCC_3cBNI1o6qgRv7g.png" /></figure><p>Once the instance is running, connect to it using SSH.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*RXp14DlrsVdrcIOGk57GLA.png" /></figure><h3>Creating Project Directory</h3><p>After logging into the EC2 instance, a project directory is created to keep everything organized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*40RFkGDaJ5oAuVQykAJQHQ.png" /></figure><h3>Creating the Bash Health Check Script</h3><p>Now we create a Bash script that collects system health information.</p><pre>nano health-check.sh</pre><h4>Bash Script</h4><pre>#!/bin/bash<br><br>##########################################<br># AUTHOR : SUMAN YADAV<br># THIS IS  SCRIPT THAT CHECKS LINUX SERVER<br># VERSION : 0.0.1 <br>##########################################<br><br>#IT WILL GET DATE AND TIME --&gt; useful for logs and auditing.<br>DATE=$(date &#39;+%Y-%m-%d %H:%M:%S&#39;)<br><br>#SYSTEM HOSTNAME<br>HOSTNAME=$(hostname)<br><br>#DISK USAGE OF ROOT USER --&gt; df -h / → Shows disk usage of root filesystem,<br>#awk extracts the usage percentage, Used to detect disk full issues<br>DISK_USAGE=$(df -h / | awk &#39;NR==2 {print $5}&#39;)<br><br>#CPU USAGE --&gt; top → Shows CPU statistics, $2+$4 → User + system CPU usage,<br>#Used to detect high CPU load<br>CPU_LOAD=$(top -bn1 | grep &quot;Cpu(s)&quot; | awk &#39;{print $2+$4}&#39;)<br><br>#MEMORY USAGE--&gt; free -m → Shows memory usage<br>MEM_USAGE=$(free -m | awk &#39;NR==2{printf &quot;%.2f%%&quot;, $3*100/$2}&#39;)<br><br>#REPORT FILE PATH --&gt;location of the log file where health check results are going to stored<br>REPORT=&quot;/home/ubuntu/cloud-health-check/health_report.log&quot;<br><br>#REPORT DETAILS --&gt; Appending health check results to the log file<br>echo &quot;############################################&quot; &gt;&gt; $REPORT<br>echo &quot;Health Check Report - $DATE&quot; &gt;&gt; $REPORT<br>echo &quot;Server: $HOSTNAME&quot; &gt;&gt; $REPORT<br>echo &quot;CPU Usage: $CPU_LOAD %&quot; &gt;&gt; $REPORT<br>echo &quot;Memory Usage: $MEM_USAGE&quot; &gt;&gt; $REPORT<br>echo &quot;Disk Usage: $DISK_USAGE&quot; &gt;&gt; $REPORT<br>echo &quot;############################################&quot; &gt;&gt; $REPORT<br>echo &quot;&quot; &gt;&gt; $REPORT</pre><h3>Making Script Executable</h3><p>Linux requires execute permission to run scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*ilLLuT5jGU2kS59mG9T_2w.png" /></figure><h3>Testing Script Manually</h3><p>Before automation, always test manually:</p><pre>./health-check.sh</pre><p>Check logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*WrHNoah2vUTG8PMG7LM3zQ.png" /></figure><p>This confirms the script works correctly.</p><h3>Automating with Cron Job</h3><p>Cloud Ops engineers automate repetitive tasks using cron.</p><p>Open crontab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*Cu2CvPM1ki8wld7bIhuatg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oh4MecvS6LyoOFb3fRhYLQ.png" /></figure><pre># The script is scheduled using a cron job to run every day at 9 AM:<br>0 9 * * * /home/ubuntu/cloud-health-check/scripts/health_check.sh</pre><p>Save → <strong>CTRL+O</strong>, Enter<br>Exit → <strong>CTRL+X</strong></p><p>This ensures automated execution without manual intervention.</p><h3>Monitoring &amp; Alerts</h3><h4><strong>1. Enable CloudWatch Detailed Monitoring</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eov5QN-hO7xYpH72Uh9ZtA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*m_uEEcnc5vZHJCtG_H3LrQ.png" /></figure><p>Manual scripts monitor inside the server, but <strong>CloudWatch monitors from AWS side</strong>.</p><h3>Setting Up Email Alerts using AWS SNS (Simple Notification Service)</h3><p>Monitoring alone is not enough in Cloud Operations.<br>When something goes wrong, the system must <strong>notify the administrator automatically</strong>. This is where <strong>AWS SNS (Simple Notification Service)</strong> is used.</p><p>In this project, SNS is used to <strong>send email alerts</strong> whenever CPU usage crosses a defined threshold.</p><h4>What is AWS SNS?</h4><p>AWS SNS is a <strong>messaging service</strong> that sends notifications to: Email, SMS, Other AWS services. It will receive <strong>instant email alerts, </strong>avoid manual monitoring, enable <strong>proactive incident response</strong></p><p>This ensures issues are detected <strong>before users are impacted</strong>.</p><h4>Step-by-Step SNS Configuration</h4><h4>Create an SNS Topic</h4><p>Open <strong>AWS Console → </strong>Go to <strong>SNS → Create topic → </strong>Choose <strong>Standard → </strong>Topic name: instance_health_report → Create the topic</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-QR-j47q5pwuxrF4HI9a2w.png" /></figure><h4>Create Email Subscription</h4><p>Open the created SNS topic → Click <strong>Create subscription → </strong>Protocol: <strong>Email → </strong>Endpoint: <em>your email address → </em>Click <strong>Create subscription</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-WeXaeVl9mbYPrF-Q3mBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O23uKXBV7OkSQgQltf08-w.png" /></figure><p>Confirm subscription from your email inbox</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbkePKiZz2zSikSofsGihg.jpeg" /></figure><p>This step links SNS with your email.</p><h4>Connect CloudWatch Alarm to SNS</h4><p>After creating the SNS topic, it is to be linked to CloudWatch alarm.</p><ul><li>Go to <strong>CloudWatch → </strong>Alarms → Create alarm → Select metric → EC2 → Per-instance metrics → Choose <strong>CPUUtilization</strong></li><li>Condition: Threshold → <strong>Greater than 70% → </strong>Duration: <strong>5 minutes</strong></li><li>Notification: Select SNS topic → instance_health_report</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7XxcdvHVcWmOkkxSjtIjQw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emSl0eNXG4ZdA8b0mzppnA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7QrLqv2sMSg2jJPMsqcH-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*75z-KcXAW0NfwEaSosHBOA.png" /><figcaption>done!!</figcaption></figure><p>Now the flow becomes:</p><blockquote><strong><em>EC2 CPU spike → CloudWatch alarm → SNS → Email notification</em></strong></blockquote><h4>GitHub Repository</h4><p>Complete source code are available on GitHub:<br><a href="https://github.com/suman526/daily-cloud-health-check-aws">https://github.com/suman526/daily-cloud-health-check-aws</a></p><blockquote><strong>We come to the end of this article. The learnings we got is understanding daily responsibilities of Cloud Operations teams, automating routine health checks using Bash and cron, monitoring EC2 metrics using CloudWatch, setting up proactive alerting with SNS, importance of documentation and cleanup of cloud resources.</strong></blockquote><p>THANK YOU!</p><p><strong>Let’s stay connected!</strong></p><p><strong>You can find me on LinkedIn🤝</strong></p><p><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1/"><strong>https://www.linkedin.com/in/suman-yadav-2972a71a1/</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=680b28be3af5" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/daily-cloud-health-check-automation-using-aws-linux-bash-and-cron-680b28be3af5">Daily Cloud Health Check Automation using AWS, Linux, Bash, and Cron</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why So Many Websites Went Down — Cloudflare Outage]]></title>
            <link>https://suman15.medium.com/why-so-many-websites-went-down-cloudflare-outage-7dce447a6e9e?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dce447a6e9e</guid>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cdn]]></category>
            <category><![CDATA[outage]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Thu, 20 Nov 2025 10:09:44 GMT</pubDate>
            <atom:updated>2025-11-20T10:09:44.304Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why So Many Websites Went Down — Cloudflare Outage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C28VDn88CcHj80-QYIdvww.png" /></figure><p>Yesterday felt like the whole internet went to sleep at the same time. Apps wouldn’t open, websites refused to load, and even developers saw their tools break. If you’re wondering what caused this chaos, don’t worry — here’s the simplest explanation.</p><p><strong>Hello everyone, my name is Suman, and welcome to my blog.</strong></p><blockquote><strong>Yesterday, something strange happened on the internet.</strong><br><strong>Many big websites were not working:</strong></blockquote><blockquote><strong>Twitter (X), Canva, Udemy, Bet365, ChatGPT</strong></blockquote><blockquote><strong>Not only websites, many companies said their CI/CD pipelines also stopped working.</strong></blockquote><blockquote><strong>All of this happened because <em>CLOUDFLARE </em>had a major problem.</strong></blockquote><h3>🌐 What Is Cloudflare?</h3><p>Imagine you want to visit a website. You type the name, and your phone or laptop sends a request. Usually, your request does <strong>not</strong> go directly to the website.<br>First, it goes to <strong>Cloudflare</strong>. Cloudflare then forwards your request to the website and sends the response back to you. So Cloudflare is like a <strong>traffic controller</strong> and <strong>security guard</strong> for the internet.</p><p>Companies use Cloudflare because it gives them:</p><ul><li><strong>CDN (Content Delivery Network):</strong> makes websites load faster</li><li><strong>Security (WAF, DDoS protection):</strong> protects websites from attacks</li><li><strong>Load balancing:</strong> spreads traffic so websites don’t crash</li><li><strong>Caching:</strong> saves copies of pages to make loading faster</li></ul><p>Cloudflare is used by millions of websites.<br>So when Cloudflare has a problem, many websites break at the same time.</p><h3>🔥 What Happened Yesterday?</h3><p>Cloudflare said one of their <strong>internal services</strong> received <strong>“unusual traffic.”</strong></p><p>This unusual traffic caused that service to fail. And because Cloudflare sits in front of so many websites, the failure affected the whole world. We still don’t know what exact service was affected.</p><p>Cloudflare said:</p><ul><li>This does <strong>not</strong> look like a cyberattack</li><li>It looks like a <strong>traffic surge</strong> (a big sudden increase in requests)</li></ul><h3>📌 Surge vs DDoS Attack (Simple Explanation)</h3><h4>Surge (Normal traffic increase)</h4><p>Imagine Cloudflare normally gets 2 <strong>million requests per minute</strong>.<br>Suddenly it gets 2<strong>.5 million</strong>. If these extra requests are from <strong>real users</strong>, it is just a surge.</p><h4>DDoS Attack</h4><p>If the extra 500,000 requests come from <strong>hackers or bots</strong> trying to break Cloudflare on purpose — that is a DDoS attack. Cloudflare said today’s issue looks more like <strong>a surge</strong>, not an attack.</p><h3>🛠️ Why Did CI/CD Pipelines Also Break?</h3><p>This confused many people. Why did developers see errors in their pipelines?</p><p>Simple:</p><p>CI/CD pipelines often download software packages or dependencies.<br> For example:</p><ul><li>Java downloads dependencies from Maven/JFrog</li><li>Node downloads packages from npm</li><li>Python uses PyPI</li></ul><p>If these services use Cloudflare for security, then the pipeline also goes through Cloudflare. If Cloudflare is down → pipelines cannot download anything → the build fails. So even if your website was working, your CI/CD could still fail.</p><h3>Current Status</h3><p>Cloudflare posted an update on their status page:</p><ul><li>They applied a fix &amp; websites are now working again</li></ul><p>Cloudflare will soon share a <strong>full RCA (root cause analysis)</strong>.<br>Once that is released, I’ll make another blog explaining everything in detail.</p><h4>📚 Important Terms Explained for Beginners</h4><h4>CDN</h4><blockquote>Servers placed around the world to make websites load faster.</blockquote><h4>CI/CD</h4><blockquote>A system that automatically builds, tests, and deploys software.</blockquote><h4>WAF</h4><blockquote>A firewall that protects websites from harmful traffic.</blockquote><h4>DDoS Attack</h4><blockquote>A type of attack where hackers send too many requests to crash a service.</blockquote><h3>✅ Final Thoughts</h3><p>Yesterday’s outage shows how closely everything on the internet is connected. If one big company or a major service like Cloudflare goes down, many websites and tools go down with it.</p><p>Thank you for reading this.<br>If you feel I missed anything, please let me know — I’d love to hear from you too.</p><p>See you in the next blog. Take care!</p><blockquote><strong><em>Let’s stay connected!</em></strong></blockquote><blockquote><strong><em>You can find me on LinkedIn🤝</em></strong></blockquote><blockquote><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1/"><strong><em>https://www.linkedin.com/in/suman-yadav-2972a71a1/</em></strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dce447a6e9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hadoop Uploads the data serially or parallelly ?]]></title>
            <link>https://suman15.medium.com/hadoop-uploads-the-data-serially-or-parallelly-972420eb9ad1?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/972420eb9ad1</guid>
            <category><![CDATA[hadoop-cluster]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[tcpdump]]></category>
            <category><![CDATA[myths]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Fri, 23 Dec 2022 09:34:41 GMT</pubDate>
            <atom:updated>2022-12-23T09:35:12.337Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*bbD-MEHf2z8a7VWS-y40Vg.jpeg" /></figure><h3>Hadoop Uploads data serially or parallelly ?</h3><p><strong>What is Hadoop?</strong></p><p>Hadoop is an open-source framework that allows to store and process big data in a distributed environment across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.</p><h4>Let’s Research about a Myth of Hadoop</h4><p><strong>TASK DESCRIPTION:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wCKz-wQ0-S0evAdSm2KM8Q.png" /></figure><blockquote>LET’S TALK SOMETHING ABOUT OUR HINT:</blockquote><blockquote><strong><em>“tcpdump“ :</em></strong><em> It </em>is a packet sniffing and packet analyzing tool for a System Administrator to troubleshoot connectivity issues in Linux. It is used to capture, filter, and analyze network traffic such as TCP/IP packets going through your system. It is many times used as a security tool as well. It also gives us a option to save captured packets in a file for future analysis.</blockquote><p><strong>To install tcpdump package (Note: install it first, as we are going to use it later)</strong></p><p><em># yum install tcpdump</em></p><p><strong><em>Let’s begin the practical :</em></strong></p><h4>Step 1 : To configure hadoop cluster (either on AWS cloud or VM)</h4><p><a href="https://www.linkedin.com/posts/suman-yadav-2972a71a1_selfreflection-vimaldaga-righteducation-activity-6790682518891962368-UUMG?utm_source=share&amp;utm_medium=member_desktop">Suman Yadav on LinkedIn: #selfreflection #vimaldaga #righteducation #educationredefine #cluster...</a></p><p><strong>You can use either above step or below steps mentioned:</strong></p><blockquote>Configure one instance as NameNode, one as Client and remaining two as DataNodes.</blockquote><blockquote>2. Install jdk and hadoop package in all instances</blockquote><blockquote>3. Configure “hdfs-site.xml” and “core-site.xml” file in both datanodes and namenode. (Note: no need to configure “hdfs-site.xml” file in hadoop-client , only configure “core-site.xml” file)</blockquote><blockquote>4. Format the NameNode</blockquote><blockquote>5. Start the Hadoop daemon services in both DataNodes and NameNode and check by using <strong>“<em>jps</em>” </strong>command</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*k5y5yr6WoACxD2Co.png" /></figure><blockquote>6. Check Datanodes available to the Hadoop-Cluster by using command “<strong><em>hadoop dfsadmin -report”</em></strong></blockquote><p><strong>NOW OUR HADOOP CLUSTER IS READY !!</strong></p><h4>Step 2 → Upload the file</h4><ol><li>Hadoop-Client <em>uploads</em> a file to the Hadoop-Cluster by this command: <strong><em>hadoop fs -put &lt;file_name&gt; /</em></strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*zZeROtR_xmh_5hR9.png" /></figure><p>2. Check file in hadoop-cluster by using command:- <em>hadoop fs -ls /</em></p><h4>Step 3 → Checking of file</h4><ol><li>While uploading file run tcpdump command that we have installed previously in the datanode and namenode to check the transferred packets between client , master and slaves.</li></ol><p>2. To trace the data packets use port number 50010</p><p><strong><em># tcpdump -i eth0 port 50010</em></strong></p><blockquote><strong>Background Reality:</strong> Client is requesting NameNode to get the ip addresses of DataNodes. So Client is the one who directly uploads the data to the DataNodes and Master is replying by sending the network packets to the Client which contains the ip addresses of DataNodes.</blockquote><h4>OUTCOME:</h4><p><strong>Just after running the above command we will see:</strong></p><blockquote>Some data packets are recieving to the DataNodes</blockquote><blockquote>First some packets recieved by DataNode1.</blockquote><blockquote>After this, some packets recieved by DataNode 2 and then stops.</blockquote><blockquote>This process continues till whole file gets uploaded to the Cluster.</blockquote><h4>Thus, The Data Flow in data packets from client to datanodes is in the serial order and faster so, no velocity problem.</h4><p>We come to the end of this article. I hope I have thrown some light on to your knowledge on this hadoop myth.</p><p><strong>Thank you!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=972420eb9ad1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform Module]]></title>
            <link>https://suman15.medium.com/terraform-module-2c6c91e83071?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c6c91e83071</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[ebs-volume]]></category>
            <category><![CDATA[terraform-modules]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Wed, 13 Jul 2022 11:50:24 GMT</pubDate>
            <atom:updated>2022-07-14T07:12:02.738Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XwmnzEvlBYajlzBi8azpnw.png" /></figure><h3>What is terraform module ?</h3><p>A <em>Terraform module</em> is a collection of standard configuration files in a dedicated directory. Terraform modules encapsulate groups of resources dedicated to one task, reducing the amount of code you have to develop for similar infrastructure components.</p><p>Terraform module definition is a single or many .tf files stacked together in their own directory.</p><p>A typical module can look like this:</p><pre>.<br>├── main.tf<br>├── outputs.tf<br>├── README.md<br>└── variables.tf</pre><p>If we run Terraform in this directory, those configuration files would be considered a <strong>root module</strong>. It means that this configuration is the base of our operation, that we can expand further.</p><h3>Installation of Terraform</h3><p>We are going to install on one of the instance of AWS cloud:</p><p>→ <strong>Get the latest release version</strong></p><pre>TERRAFORM_VER=`curl -s <a href="https://api.github.com/repos/hashicorp/terraform/releases/latest">https://api.github.com/repos/hashicorp/terraform/releases/latest</a> |  grep tag_name | cut -d: -f2 | tr -d \&quot;\,\v | awk &#39;{$1=$1};1&#39;`</pre><p>→ <strong>To download the latest release of Terraform run the below command</strong></p><pre>wget <a href="https://releases.hashicorp.com/terraform/$%7BTERRAFORM_VER%7D/terraform_$%7BTERRAFORM_VER%7D_linux_amd64.zip">https://releases.hashicorp.com/terraform/${TERRAFORM_VER}/terraform_${TERRAFORM_VER}_linux_amd64.zip</a></pre><p>→ <strong>Extract the file</strong></p><pre>sudo yum -y install unzip<br>unzip terraform_${TERRAFORM_VER}_linux_amd64.zip</pre><p>→<strong> Move binary file to the /usr/local/bin directory</strong></p><pre>sudo mv terraform /usr/local/bin/</pre><h3>Creating a Terraform Module</h3><p>Here we are going to do the following things through terraform module:</p><blockquote><strong>DESCRIPTION :</strong></blockquote><blockquote><strong>1. Create a key pair</strong></blockquote><blockquote><strong>2. Create a SECURITY GROUP</strong></blockquote><blockquote><strong>3. Launch an EC2 instance using key pair &amp; security group</strong></blockquote><blockquote><strong>4. Create an EBS Volume</strong></blockquote><blockquote><strong>5. Attach EBS Volume to the EC2 instance</strong></blockquote><p>This is the same practical that I have performed through aws cli which you can read from below link:</p><p><a href="https://suman15.medium.com/accessing-of-aws-through-cli-5d4a7fd8d211">Accessing of AWS through CLI</a></p><h4><strong>Let’s head over our practical</strong></h4><p>Let’s start with the</p><h3><strong>→ main.tf</strong></h3><p>This is our main configuration file where we are going to define our resource definition. It will call our module, It contains some required variables which is already defined in source path.</p><pre>module &quot;aws_architecture&quot; {                <br>  source                        = &quot;github.com/kmsatyam/instance_ebs_volume_resources?ref=v0.0.1&quot;<br>  instance_type                 = var.instance_type<br>  availability_zone             = var.availability_zone<br>  vpc_security_group_ids        = var.vpc_security_group_ids<br>  key_name                      = var.key_name<br>  disk_name                     = var.disk_name<br>  ami_id                        = var.ami_id<br>  disk_size                     = var.disk_size}</pre><h3><strong>→ variable.tf</strong></h3><p>Define your variables in separate file which contains all the values defined into source</p><p>Input variables let you customize aspects of Terraform modules without altering the module’s own source code. This allows you to share modules across different Terraform configurations, making your module composable and reusable.</p><p>When you declare variables in the root module of your configuration, you can set their values using CLI options and environment variables. When you declare them in child modules, the calling module should pass values in the module block.</p><pre>variable &quot;ami_id&quot; {<br>	  type        = string<br>	  default     = &quot;ami-0447a12f28fddb066&quot;<br>	}<br>		variable &quot;instance_type&quot; {<br>	  type        = string<br>	  default     = &quot;t2.micro&quot;<br>	}<br>		variable &quot;availability_zone&quot; {<br>	  type        = string<br>	  default     = &quot;ap-south-1a&quot;<br>	}<br>		variable &quot;vpc_security_group_ids&quot; {<br>	  type        = list<br>	  default     = [&quot;default&quot;]<br>	}<br>		variable &quot;key_name&quot; {<br>	  type        = string<br>	  default = &quot;task37key&quot;<br>	}<br>		variable &quot;disk_size&quot; {<br>	  type        = number<br>	  default = 1<br>	}<br>		variable &quot;disk_name&quot; {<br>	  type        = string<br>	  default =  &quot;/dev/sdd&quot;<br>	}</pre><h3><strong>→ output.tf</strong></h3><p>Output values make information about your infrastructure available on the command line, and can expose information for other Terraform configurations to use. Output values are similar to return values in programming languages.</p><pre>output &quot;instance_type&quot; {<br>	  value = module.launching_os.instance_type<br>	}<br>		output &quot;disk_name&quot; {<br>	  value = module.launching_os.disk_name<br>	}<br>		output &quot;ami_id&quot; {<br>	  value = module.launching_os.image_id<br>	}<br>		output &quot;disk_size&quot; {<br>	  value = module.launching_os.disk_size<br>	}</pre><h4>→ <strong>Now our terraform code is ready, the first command we are going to execute is</strong></h4><ul><li><strong>terraform init:</strong> It’s going to download code for a provider(aws) that we will use.</li></ul><p>Then →</p><ul><li><strong>terraform plan:</strong> This command will tell what terraform does before making any changes.<br>1: (+ sign): Resource going to be created<br>2: (- sign): Resource going to be deleted<br>3: (~ sign): Resource going to be modified</li></ul><p>Then →</p><ul><li><strong>terraform apply:</strong> To apply the changes, run terraform apply command.</li></ul><p>Then we will be able to see aws ec2 with Ebs attched successfully with the help of Terraform. Terraform requires minimal effort than aws cli which we have used in the above link provided.</p><blockquote><em>SUCCESSFULLY DONE!</em></blockquote><blockquote><em>THANK YOU ALL FOR READING!</em></blockquote><blockquote>For more such articles, Stay Connected 😄</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c6c91e83071" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A MULTI-CLOUD SETUP OF K8S CLUSTER]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://suman15.medium.com/a-multi-cloud-setup-of-k8s-cluster-c89afe109796?source=rss-299760da75d8------2"><img src="https://cdn-images-1.medium.com/max/1366/1*fRSLnVs6aHIGjSuOuDeLWw.png" width="1366"></a></p><p class="medium-feed-snippet">What is Kubernetes?</p><p class="medium-feed-link"><a href="https://suman15.medium.com/a-multi-cloud-setup-of-k8s-cluster-c89afe109796?source=rss-299760da75d8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://suman15.medium.com/a-multi-cloud-setup-of-k8s-cluster-c89afe109796?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c89afe109796</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Wed, 13 Jul 2022 10:44:07 GMT</pubDate>
            <atom:updated>2022-07-13T10:44:07.167Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Automating K8s Cluster using Ansible]]></title>
            <link>https://suman15.medium.com/automating-k8s-cluster-using-ansible-5d25d39be078?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d25d39be078</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[wordpress]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Tue, 12 Jul 2022 06:34:14 GMT</pubDate>
            <atom:updated>2022-07-17T11:37:08.794Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-gGQl7ZxVnAZ7qpds3b9w.png" /></figure><h3>What is KUBERNETES?</h3><p>Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.</p><p>Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.</p><h3>What is ANSIBLE?</h3><p>Ansible is a software tool that provides simple but powerful automation for cross-platform computer support. It is primarily intended for IT professionals, who use it for application deployment, updates on workstations and servers, cloud provisioning, configuration management, intra-service orchestration, and nearly anything a systems administrator does on a weekly or daily basis. Ansible doesn’t depend on agent software and has no additional security infrastructure, so it’s easy to deploy.</p><h3>KUBERNETES CLUSTER :</h3><p>A Kubernetes cluster is a set of node machines for running containerized applications. If you’re running Kubernetes, you’re running a cluster.</p><p>Kubernetes clusters allow containers to run across multiple machines and environments: virtual, physical, cloud-based, and on-premises. Kubernetes containers are not restricted to a specific operating system, unlike virtual machines. Instead, they are able to share operating systems and run anywhere.</p><p>The <strong>master node</strong> controls the state of the cluster; for example, which applications are running and their corresponding container images. The master node is the origin for all task assignments. It coordinates processes such as:</p><ul><li><strong>Scheduling</strong> and scaling applications</li><li><strong>Maintaining</strong> a cluster’s state</li><li><strong>Implementing</strong> updates</li></ul><p>The <strong>worker nodes</strong> are the components that run these applications. Worker nodes perform tasks assigned by the master node. They can either be <a href="https://www.vmware.com/topics/glossary/content/virtual-machine.html">virtual machines</a> or physical computers, all operating as part of one system.</p><p>There must be a minimum of one master node and one worker node for a Kubernetes cluster to be operational. For production and staging, the cluster is distributed across multiple worker nodes. For testing, the components can all run on the same physical or virtual node.</p><p>A <strong>namespace</strong> is a way for a Kubernetes user to organize many different clusters within just one physical cluster. Namespaces enable users to divide cluster resources within the physical cluster among different teams via resource quotas. For this reason, they are ideal in situations involving complex projects or multiple teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*fY-SSQgUV_e_eA9o" /></figure><h3>LET’S JUMP TO OUR PRACTICAL</h3><p>TASK DESCRIPTION:</p><blockquote>Ansible Role to Configure K8S Multi Node Cluster over AWS Cloud. <br>🔅 Create Ansible Playbook to launch 3 AWS EC2 Instance <br>🔅 Create Ansible Playbook to configure Docker over those instances. <br>🔅 Create Playbook to configure K8S Master, K8S Worker Nodes on the above created EC2 Instances using kubeadm. <br>🔅 Also Upload all the YAML code over your GitHub Repository.</blockquote><p>AND</p><blockquote>🔅 Launch ec2-instances on AWS Cloud eg. for master and slave.</blockquote><blockquote>🔅 Create roles that will configure master node and slave node separately.</blockquote><blockquote>🔅 Launch a WordPress and MySQL database connected to it in the respective slaves.</blockquote><blockquote>🔅 Expose the WordPress pod and client able hit the WordPress IP with its respective port.</blockquote><p>So, we are combining both the tasks.</p><p><strong>Let’s begin,</strong></p><p>I have included all the steps of configuring Kubernetes cluster in the below video.</p><p>Please go through the video and configure it accordingly.</p><h3>🎥Video</h3><p><a href="https://drive.google.com/file/d/1FnKOKehs7EtGwXRAZm2ZPhT0Fivn6yY5/view?usp=drivesdk">k8s_cluster_on_aws.mp4</a></p><p>Kudos!! We have configured Kubernetes cluster successfully!!</p><p>So, our first practical got completed.</p><h4>NEXT</h4><p>Let&#39;s proceed to the next one --&gt;</p><p>After doing all the above steps now we have to Launch a WordPress and MySQL database connected to it in the respective slave and Expose the WordPress pod and client able hit the WordPress IP with its respective port.</p><p>Ok, Lets begin next part!!</p><p>Create a new role for wordpress using the below command:</p><pre>ansible-galaxy init wordpress_configuration</pre><p><strong>We are going to write role for setup of Wordpress and MySQL</strong></p><p>Go inside the file folder of wordpress role folder</p><p>To configure wordpress installation part inside wordpress.yml file</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: wordpress<br>  labels:<br>    app: wordpress<br>spec:<br>  ports:<br>    - port: 80<br>      nodePort: 30333<br>  selector:<br>    app: wordpress<br>    tier: frontend<br>  type: LoadBalancer<br>---<br>apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2<br>kind: Deployment<br>metadata:<br>  name: wordpress<br>  labels:<br>    app: wordpress<br>spec:<br>  selector:<br>    matchLabels:<br>      app: wordpress<br>      tier: frontend<br>  strategy:<br>    type: Recreate<br>  template:<br>    metadata:<br>      labels:<br>        app: wordpress<br>        tier: frontend<br>    spec:<br>      volumes:<br>      - name: wordpress-persistent-storage<br>        persistentVolumeClaim:<br>          claimName: wordpress-pv-claim<br>      containers:<br>      - image: wordpress:4.8-apache<br>        name: wordpress<br>        env:<br>        - name: WORDPRESS_DB_HOST<br>          value: wordpress-mysql<br>        - name: WORDPRESS_DB_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: mysqlsecret<br>              key: password<br>        ports:<br>        - containerPort: 80<br>          name: wordpress<br>        volumes:<br>        volumeMounts:<br>        - name: wordpress-persistent-storage<br>          mountPath: /var/www/html</pre><p>Next, create a Kubernetes pvc_wordpress.yml file</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>   name: wordpress-pv-claim<br>   labels:<br>        app: wordpress<br>        tier: frontend<br>spec:<br>   storageClassName: &quot;&quot;<br>   resources:<br>        requests:<br>             storage: 1Gi<br>   accessModes:<br>     - ReadWriteOnce<br>---<br><br>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: wordpress-pv<br>spec:<br>  storageClassName: &quot;&quot;<br>  capacity:<br>     storage: 1Gi<br>  accessModes:<br>    - ReadWriteOnce<br>  hostPath:<br>    path: /wordpressdata</pre><p>Next, create a Kubernetes mysql.yml file</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: wordpress-mysql<br>  labels:<br>    app: wordpress<br>spec:<br>  ports:<br>    - port: 3306<br>  selector:<br>    app: wordpress<br>    tier: mysql<br>  clusterIP: None<br>---<br>apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2<br>kind: Deployment<br>metadata:<br>  name: wordpress-mysql<br>  labels:<br>    app: wordpress<br>spec:<br>  selector:<br>    matchLabels:<br>      app: wordpress<br>      tier: mysql<br>  strategy:<br>    type: Recreate<br>  template:<br>    metadata:<br>      labels:<br>        app: wordpress<br>        tier: mysql<br>    spec:<br>      volumes:<br>      - name: mysql-persistent-storage<br>        persistentVolumeClaim:<br>          claimName: mysql-pv-claim<br>      containers:<br>      - image: mysql:5.6<br>        name: mysql<br>        env:<br>        - name: MYSQL_ROOT_PASSWORD<br>          valueFrom:<br>            secretKeyRef:<br>              name: mysqlsecret<br>              key: password<br>        - name: MYSQL_USER<br>          value: abcdefgh<br>        - name: MYSQL_DATABASE<br>          value: 123456789<br>        ports:<br>        - containerPort: 3306<br>          name: mysql<br>        volumeMounts:<br>        - name: mysql-persistent-storage<br>          mountPath: /var/lib/mysql</pre><p>Next, create a Kubernetes pvc_mysql.yml file</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>   name: mysql-pv-claim<br>   labels:<br>        app: wordpress<br>        tier: mysql<br>spec:<br>   storageClassName: &quot;&quot;<br>   resources:<br>        requests:<br>             storage: 1Gi<br>   accessModes:<br>     - ReadWriteOnce<br>---<br><br>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: msql-pv<br>spec:<br>  storageClassName: &quot;&quot;<br>  capacity:<br>     storage: 1Gi<br>  accessModes:<br>    - ReadWriteOnce<br>  hostPath:<br>    path: /mysqldata</pre><p>Now, create a secrete file “secret.yml” which will contain the password of MySQL database</p><pre>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: ABCHEF<br>data:<br>   password: xxxxxxxxxx</pre><p>Edit the main.yml file inside the tasks folder in the wordpress_configuration role:</p><pre>- name: Copying Wordpress and MySQL files Master Node<br>    copy:<br>        src: &quot;{{ item }}&quot;<br>        dest: /root/<br>    loop:<br>        - mysql.yml<br>        - pvc_mysql.yml<br>        - pvc_wordpress.yml<br>        - secret.yml<br>        - wordpress.yml<br><br>  - name: Creating directory over which MySQL container mounts the PersistentVolume at /var/lib/mysql.<br>    file:<br>        path: /mysqldata<br>        state: directory<br><br>  - name: Creating directory over which WordPress container mounts the PersistentVolume at /var/www/html.<br>    file:<br>        path: /wordpressdata<br>        state: directory<br><br><br>  - name: Setup of Wordpress and MySQL<br>    shell: &quot;kubectl create -f /root/{{ item }}&quot;<br>    loop:<br>        - mysql.yml<br>        - pvc_mysql.yml<br>        - pvc_wordpress.yml<br>        - wordpress.yml</pre><p>Now it’s finally the time to edit final-run.yml file inside kubernetes_cluster folder which we have setup for k8s configuration.</p><pre>- hosts: kubernetes_master<br>  gather_facts: no<br>  tasks:<br>    - name: Running Wordpress and Mysql<br>      include_role:<br>        name: wordpress_configuration</pre><p>use below commands to run your ansible playbook.</p><pre>ansible-playbook final-run.yml --ask-vault-pass</pre><p>Now once our pods are ready, then we can take the public ip of any node either master or slave. We will land to the Wordpress login page.</p><p>Here we moved over to the master IP on port 30333 and we can see the WordPress</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*kLxgQGBw0gdyuypWF9f5Ow.png" /></figure><p>Wordpress application is ready !!</p><p>Thank You!!</p><blockquote><strong>I hope you got an idea about the task</strong></blockquote><blockquote><strong>For more such articles</strong></blockquote><blockquote><strong>Stay Connected 😄</strong></blockquote><blockquote><strong>I’ll be grateful to have connections like you on </strong><a href="https://www.linkedin.com/in/suman-yadav-2972a71a1"><strong>Linkedln</strong></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d25d39be078" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Helm chart on any Technology]]></title>
            <link>https://suman15.medium.com/helm-chart-on-any-technology-3600e5e3ad3d?source=rss-299760da75d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/3600e5e3ad3d</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[helm-chart]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[web-server]]></category>
            <category><![CDATA[cluster]]></category>
            <dc:creator><![CDATA[suman15]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 07:55:50 GMT</pubDate>
            <atom:updated>2022-07-12T01:57:59.598Z</atom:updated>
            <content:encoded><![CDATA[<h3>Helm chart on any technology</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fhOLyxiD7WajUuqt.png" /></figure><h3>What is Helm?</h3><p>Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.</p><p>Helm is widely known as “the package manager for Kubernetes”. Although it presents itself like this, its scope goes way beyond that of a simple package manager. However, let’s start at the beginning.</p><h3><strong>Helm Chart</strong></h3><p>The path Helm took to solve this issue was to create Helm <strong>Charts</strong>. Each chart is a bundle with one or more Kubernetes manifests — a chart can have child charts and dependent charts as well.</p><p>This means that Helm installs the whole dependency tree of a project if you run the install command for the top-level chart. You just a single command to install your entire application, instead of listing the files to install via kubectl.</p><p>Charts allow you to version your manifest files too, just like we do with Node.js or any other package. This lets you install specific chart versions, which means keeping specific configurations for your infrastructure in the form of code.</p><h3>Why helm?</h3><p>Helm also keeps a release history of all deployed charts, so you can go back to a previous release if something went wrong.</p><p>Helm supports Kubernetes natively, which means you don’t have to write any complex syntax files or anything to start using Helm. Just drop your template files into a new chart and you’re good to go.</p><h3>How Does Helm Work?</h3><p>Helm and Kubernetes work like a client/server application. The Helm client pushes resources to the Kubernetes cluster. The server-side depends on the version: Helm 2 uses Tiller while Helm 3 got rid of Tiller and entirely relies on the Kubernetes API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*j8ey0II0E0y6Hv7o.png" /><figcaption>WORKING OF HELM CHART</figcaption></figure><h3>Prerequisites:</h3><blockquote>1. Kubernetes cluster should be configured</blockquote><blockquote>It is often advisable to install <strong>the most compatible version of the Kubernetes</strong> as supported by Helm. We should also install and configure the Kubernetes command-line tool kubectl<em>, enabling</em> us to work with our cluster efficiently.</blockquote><blockquote>2. Helm should be installed</blockquote><h4>Installing kubernetes</h4><blockquote>So, First we are going to install kubernetes</blockquote><blockquote>We are going to install from command line program kubectl installed on your Windows 10 computer and configured to work with a Kubernetes Cluster. The easiest way to do this is to install using the Windows package manager Chocolatey, choco install kubernetes-cli.</blockquote><p>Or you can use the below link to configure k8s on aws clould and perform the practical there also</p><p><a href="https://drive.google.com/file/d/1FnKOKehs7EtGwXRAZm2ZPhT0Fivn6yY5/view?usp=drivesdk">k8s_cluster_on_aws.mp4</a></p><h4>Installing Helm</h4><p>There are several ways to install Helm that are neatly described on the official install page on Helm. <strong>The quickest way to install helm on Windows is using Chocolaty</strong>, a package manager for Windows platforms.</p><p>Using Chocolaty, it’s a simple one-line command to install Helm:</p><pre>choco install kubernetes-helm</pre><p>This installs the Helm client locally. This also provides us with the Helm command-line tool that we’ll use to work with Helm in this tutorial.</p><p>Before proceeding further, we should <strong>ensure that the Kubernetes cluster is running</strong> and accessible using the <em>kubectl </em>command<strong><em>:</em></strong></p><pre>kubectl cluster-info</pre><p>We could initialize Helm through the Helm CLI using the command:</p><pre>helm init</pre><p>If you are starting with Helm 3, since there is no more Tiller server, <strong>it’s unnecessary to initialize Helm</strong>. In fact, this command has been removed. Consequently, the Helm state is created automatically when required.</p><h3>Let’s start</h3><p>Let’s quickly see the directory structure created for us:</p><pre>helmweb /<br>  Chart.yaml<br>  values.yaml<br>  templates /<br>  charts /<br>  .helmignore</pre><p>Let’s understand the relevance of these files and folders created for us:</p><ul><li><em>Chart.yaml</em>: This is the main file that contains the description of our chart</li><li><em>values.yaml</em>: this is the file that contains the default values for our chart</li><li><em>templates</em>: This is the directory where Kubernetes resources are defined as templates</li><li><em>charts</em>: This is an optional directory that may contain sub-charts</li><li><em>.helmignore</em>: This is where we can define patterns to ignore when packaging (similar in concept to .gitignore)</li></ul><h3>Creating a Chart</h3><p>Create one directory and generally, the name of the package (chart) will be equal to the directory name.</p><pre>mkdir helmweb</pre><p>The next step, of course, would be to create a new file with a name say chart.yaml:</p><ul><li>This is where you’ll put the information related to your chart. That includes the chart version, name, and description so you can find it if you publish it on an open repository. Also in this file you’ll be able to set external dependencies using the dependencies key.</li></ul><pre># Chart.yaml<br>apiVersion: v1<br>name: helmweb<br>description: A Helm chart for Kubernetes for launching webserver<br>type: application<br>version: 0.0.5<br>appVersion: 1.1.0</pre><h3>Creating Template</h3><pre># templates/deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:  <br>  creationTimestamp: null  <br>  labels:    <br>    app: mydeploy  <br>  name: mydeploy<br>spec:  <br>  replicas: {{ .Values.replicaCount }}  <br>  selector:    <br>    matchLabels:      <br>      app: mydeploy  <br>strategy: {}<br>template:<br>  metadata:<br>    creationTimestamp: null<br>    labels:<br>      app: mydeploy<br>  spec:<br>   containers:<br>   - image: suman526/webpage_helm<br>     name: webpage<br>     resources: {}<br>status: {}</pre><p>Now next file in the template folder is service.yaml</p><pre># helmweb/templates/service.yaml<br>apiVersion: v1<br>kind: Servicemetadata:<br>  creationTimestamp: null<br>  labels:<br>    app: mysvc<br>  name: mysvc<br>spec:<br>  ports:<br>  - name: 80-80<br>    port: {{ .Values.myservice.port }}<br>    protocol: TCP<br>    targetPort: 80<br>  selector:<br>    app: mysvc<br>  type: {{ .Values.myservice.type }}<br>status:  loadBalancer: {}</pre><h3>Creating Values files</h3><p>Like we saw before, this is the file that contains defaults for variables.</p><pre>replicaCount: 3<br>myservice: <br>      type: Nodeport<br>      port: 8080</pre><p>Now, our Helm chart ready!</p><p>We have to create a package for helm chart.</p><pre>mkdir charts/</pre><p>Now, run the following command to packages the chart and store it inside the charts/ directory.</p><pre>helm package helmweb -d charts/</pre><p>After successfully packaged chart and saved create an index.yaml file</p><h3>Creating an index.yaml file</h3><p>For every Helm repository, we must require an index.yaml file. The index.yaml file contains the information about the chart that is present inside the current repository/directory.</p><pre>helm repo index charts/</pre><pre>apiVersion: v1<br>entries:<br>  myweb:<br>  - apiVersion: v1<br>    appVersion: 1.1.0<br>    created: &quot;2022-06-09T17:23:26.9461245+05:30&quot;<br>    description: A Helm chart for Kubernetes for launching webserver<br>    digest: 1db457438057de8db20ed35f37fcb5d1dd997401a421673c3f71a94e39f519a3<br>    name: helmweb<br>    urls:<br>    - myweb-0.0.5.tgz<br>    version: 0.0.5<br>generated: &quot;2022-06-09T17:23:26.9461245+05:30&quot;</pre><h3>Push the chart to the GitHub public repository</h3><p>Here, push the packaged chart as well as the chart source code to the Github repository</p><p>For publishing the Helm chart on <a href="https://artifacthub.io/">artifacthub.io/</a>, we must require a URL where we had hosted our helm chart. One of the free and easy ways to host the charts is using GitHub pages. So, we will first host our charts using GitHub pages.</p><h3>Publish Helm chart to ArtifactHub</h3><p>Go to the <a href="https://artifacthub.io/">https://artifacthub.io/</a> and login to your account.</p><p>Click on add repository.</p><p>If the URL you had provided to the charts is correct then it will create a Helm repository.</p><p>For installing any helm chart, we first required to add the repository in which that chart is present.</p><p>Now, we can install our chart from this repository. As you known, the name of our chart is myweb.</p><pre>helm install web httpd-webserver/helmweb</pre><p><strong><em>Finally, we have successfully published our helm chart.</em></strong></p><p>Thank You for Reading!!!!!😀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3600e5e3ad3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>