<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thomas Puntillo on Medium]]></title>
        <description><![CDATA[Stories by Thomas Puntillo on Medium]]></description>
        <link>https://medium.com/@tom.puntillo?source=rss-154d0cadceae------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d7mQcFkCNZW2mAZegpEh3A.jpeg</url>
            <title>Stories by Thomas Puntillo on Medium</title>
            <link>https://medium.com/@tom.puntillo?source=rss-154d0cadceae------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:59:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tom.puntillo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a 2 tier application in Terraform using custom modules]]></title>
            <link>https://towardsaws.com/building-a-2-tier-application-in-terraform-using-custom-modules-9d599870afe2?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d599870afe2</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform-cloud]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[terraform-modules]]></category>
            <category><![CDATA[mysql]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 20 Aug 2023 02:05:57 GMT</pubDate>
            <atom:updated>2023-10-11T01:59:14.895Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYvB4Knt1GPFumn8FwMcyA.png" /></figure><p>Hello, and welcome! If you have found yourself here after reading some of my previous articles, welcome back! As you no doubt have guessed (based on the title of this article and my statements thus far), today we will be continuing our foray into Infrastructure as Code with Terraform. In today’s article, we will be building a custom 2-tier architecture. Furthermore, we will be using Terraform Cloud as a CI/CD tool for version control. We will be using custom modules to build out our code. Before we jump in, let’s first define some terms we’ll need to understand to follow along with this article.</p><p><strong><em>TERMINOLOGY:</em></strong></p><p><strong><em>Modules — </em></strong>In Terraform, a module is simply any directory containing .tf files. Any time you are using Terraform, you are utilizing at least one module, the root module. If you make any new directories within that module and fill the new directory with .tf files, that new directory is now a “<em>child</em>” module. A child module can output values which can be used as input variables to the parent (or root) module. By breaking your infrastructure into smaller modules, the code can be easily repeated and redeployed into new environments.</p><p><strong><em>Terraform Cloud — </em></strong>Terraform Cloud is an application that teams can use to better manage infrastructure that is contributed to by many individuals. We will be using Terraform cloud both as the remote backend for our statefile, but also as a CI/CD tool for version control. Not only does this give us a centralized place where we can review and manage our code, but it also allows us to easily roll back the entire infrastructure to the last known good configuration in case of any failed upgrade or emergency.</p><p>That should do for now in regards to terminology. Now, let’s take a look at out project objectives.</p><p><strong><em>OBJECTIVES</em></strong>:</p><ul><li>Create a highly available two-tier AWS architecture containing the following:</li><li>Custom VPC with:</li><li>2 Public Subnets for the Web Server Tier</li><li>2 Private Subnets for the RDS Tier</li><li>Appropriate route tables</li><li>Launch an EC2 Instance with your choice of webserver in each public web tier subnet (apache, NGINX, etc).</li><li>One RDS MySQL Instance (micro) in the private RDS subnets</li><li>Security Groups properly configured for needed resources ( web servers, RDS)</li><li>Deploy this using Terraform Cloud as a CI/CD tool to check your build.</li><li>Push your code to GitHub and include the link in your write up.</li><li>(optional but highly encouraged) Use custom modules to build out your code for repeatability.</li></ul><p><strong>NOTE: DO NOT FORGET TO SET YOUR ACCESS KEY, SECRET ACCESS KEY AND REGION ENVIRONMENT VARIABLES IN TERRAFORM CLOUD</strong></p><p>Ok, let’s get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*poB9VxgeXk8mtk8ukojocw.gif" /></figure><p>Before we begin writing out our code, we’ll want to set up our environment. Our objectives specify that we need to use Terraform Cloud as a CI/CD tool to check our build. In order to create a workspace with a version control workflow, we’ll need to have a GitHub repo to associate with the workflow. We can start by creating that. Go to your GitHub account and click on “Repositories”. Click on “New”. Create a repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nvnIBKjS2bOJHpV1bXMW_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAfkpNb8lnkW9AOjggKIgg.png" /></figure><p>With our repo created, clone it to your IDE. We can now create our Terraform Cloud Workspace (Note: You will already need to have a Terraform Cloud account. If you do not have one, create one before proceeding). In the cli, type “terraform login”. Say yes to get the url needed to grab the API token. Generate and copy the token. Paste it in the cli.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*FZZZDLh8MBRdxmoBdQRaZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*a5kWTfVwXXZ1vz2ULf-uWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HV88QtL6DXeKBRl_4_Xdug.png" /></figure><p>You’ll see the Terraform Cloud greeting in the cli.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*HwjTyx1Z9uaKTp_gYfNVPA.png" /></figure><p>In the browser, under “Projects and workspaces”, click “New”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gwc1UdxrOT0YCBOWyS5qkQ.png" /></figure><p>Select “Version control workflow”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VcW7CkkEd5uGS_T--vDmmg.png" /></figure><p>Select GitHub as the version control provider. Choose “Only select repositories” and choose the repo we created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a46Iw_DiobCG9KYN98JYqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*hUgyNdGdePoeAm36XYk7qg.png" /></figure><p>Select the repo again on the next page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hX0RJcFpWBfu8ZEocT-Mpg.png" /></figure><p>Click on “Create workspace”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qiy5_M6ARfuISJ4dxPqUxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zjTehi5MLwn-VrLBHfPRSg.png" /></figure><p>Ok, next we need to assign our AWS credentials to this workspace so it can actually deploy resources in AWS. You’ll need the Access_Key_ID and the Secret_Access_Key_ID of an IAM User with the appropriate permissions. In the Terraform Cloud UI, under “Projects and workspaces” click on “Settings”. Click on “Variable sets” and click “Create variable set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MxhKPQEMxNgZoWQcZywmbA.png" /></figure><p>Configure the settings and set the keys as environment variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vu6cnfefXZ5piPyzmhAADA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5i6Qr6DrFX1_qzotM8dFg.png" /></figure><p>Ok, great! Now we can begin writing our code. Let’s start by creating the “providers.tf” and the “terraform.tf” files for our root module. The reason being is because all child modules will depend on these. Notice in “providers.tf”, the “required_providers” have the AWS version set to at least v5.0. We’re declaring “us-east-1” as our region. In “terraform.tf”, we’re selecting the cloud workspace we just created as the space where the state will live.</p><p>Below are the gists for the root module “providers.tf” and “terraform.tf” files:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c220ac835091e492563b4576d506eac3/href">https://medium.com/media/c220ac835091e492563b4576d506eac3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3670d1c1d79a34883a58f7c5fdeb5f0/href">https://medium.com/media/c3670d1c1d79a34883a58f7c5fdeb5f0/href</a></iframe><p>With those 2 files created, let’s initialize our directory with a terraform init command.</p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*-J7x_Y7pmDZAYsgacBgfpw.png" /></figure><p>Now we can get to work on the meat and potatoes of our code. Remember, we’re building this out using custom modules. Let’s create a directory for each module. We’ll want one for vpc, ec2, security_groups and mysql.</p><pre>mkdir ec2 vpc security_groups mysql</pre><p>We’ll begin with the vpc module, as all other resources are dependent on there being a vpc for them to go in. Our objectives specify that our architecture will require 2 private subnets, 2 public subnets and the appropriate route tables. What it does not mention are the remaining resources we’ll need in order to make this vpc usable. Those resources are:</p><ul><li>An Internet Gateway</li><li>A NAT Gateway</li><li>An Elastic-IP</li><li>Route Table associate resource block for each subnet</li></ul><p>We’ll need to make sure those are taken into account in our code. In addition to the “main.tf” file, the vpc module will also need an “outputs.tf” to make certain values available to the other modules, such as the vpc_id and subnet_ids. Also, create a “variables.tf” file for this module.</p><p>(<strong>NOTE</strong>: Make sure the public subnets have the argument “map_public_ip_on_launch = true” in order to assure that any EC2 instance deployed to that subnet will be issued a public ip address).</p><p>Below are the gists for the files in the vpc module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3000030ca91080e8f5fb2925a440e587/href">https://medium.com/media/3000030ca91080e8f5fb2925a440e587/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2372c886faf2b69ebc8c51c1157a723d/href">https://medium.com/media/2372c886faf2b69ebc8c51c1157a723d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cac584e0b3d61932c2939d5f8961e7ba/href">https://medium.com/media/cac584e0b3d61932c2939d5f8961e7ba/href</a></iframe><p>Next up we’ll cd into the “security_groups” directory we made and work on that module. Again, in addition to our “main.tf” file, this module will need a “variables.tf” file and an “outputs.tf” file. The only input variable we need to set for this module is the “vpc_id”, which we will get from the associated value in the vpc module’s “outputs.tf” file.</p><p>In the security_groups module, the “outputs.tf” file will contain the id of each security group created to be used by the EC2 instances and the MySQL db instance. The ports we’ll leave open for the EC2 instances are 22, 80 and 443. The port we are going to leave open for the MySQL db instance is 3306, the default MySQL port.</p><p>Below are the gists for the files in the security_groups module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8239a747800a8086e6fbc312f84d247d/href">https://medium.com/media/8239a747800a8086e6fbc312f84d247d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3c83d848ea97e2794322f80af764e49/href">https://medium.com/media/c3c83d848ea97e2794322f80af764e49/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/907578af4933357b9c10ea13f51907b4/href">https://medium.com/media/907578af4933357b9c10ea13f51907b4/href</a></iframe><p>Next, we’ll cd into the ec2 directory and begin building that module. We’ll create a “variables.tf” file with variables for the ami, the instance type ,the public subnets to deploy them into (these values will be passed from the vpc module), the security groups to assign (these will be passed from the security_groups module) and the base64 encoded user data file. For the user data, we’ll have to write out a bash script and then run the base64 command on the script to obtain the encoded output. We’ll use the encoded output as the value in our user_data variable.</p><p>No “outputs.tf” file is necessary for this module.</p><p>Below are the gists for the files in the ec2 module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5e1b25aac0e123bdd1918ba9f49197b/href">https://medium.com/media/a5e1b25aac0e123bdd1918ba9f49197b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccf6a18471360390a04d207e6d586c24/href">https://medium.com/media/ccf6a18471360390a04d207e6d586c24/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23aba5351a9e76b796c3a25ae087f4d3/href">https://medium.com/media/23aba5351a9e76b796c3a25ae087f4d3/href</a></iframe><p>Finally, we move on to the mysql module. By far this module will have the largest “variables.tf” file. That is because I chose to use variables for the many requirements of an aws_db_instance.</p><p>(<strong>NOTE:</strong> In this code we set the password as a variable in plain text. This is not recommended for any real world scenario. It works in this code as a proof of concept only).</p><p>Below are the gists for the files in the mysql module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a14d72b48f89dbf30c497b5302e7dc6/href">https://medium.com/media/5a14d72b48f89dbf30c497b5302e7dc6/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07eb9f566cc4e29cceda313c2e98ca19/href">https://medium.com/media/07eb9f566cc4e29cceda313c2e98ca19/href</a></iframe><p>With the last of our modules created and the “terraform.tf” file and the “providers.tf” file already created in the root module, the only file left to write is our root modules “main.tf” file. This file will simply call the other 4 child modules to execute their code. Additionally, the modules will use each other’s output values as input variables for themselves where applicable.</p><p>Here is the gist to the root modules “main.tf” file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a924fb919ccc7338477ef425377f0e32/href">https://medium.com/media/a924fb919ccc7338477ef425377f0e32/href</a></iframe><p>There we go! We have all our code written and broken into custom modules for increased repeatability. Now we need to actually deploy our infrastructure. In my previous Terraform articles, after the code was written I’d typically run these commands in this order:</p><pre>terraform init<br>terraform fmt<br>terraform validate<br>terraform plan<br>terraform apply</pre><p>Since we are using Terraform Cloud as a CI/CD tool to check our build and employ version control, the “apply” part of our workflow will be initiated by a git push, rather than a terraform cli command.</p><p>Ok, let’r run the terraform init command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*JLtVNtUlR0S5QG2rWlVp7w.png" /><figcaption>This is typically where you would see Terraform initializing the modules. However, since this capture was done later as a part of documenting the project, they were already initialized</figcaption></figure><p>Next we’ll run Terraform fmt and terraform validate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*5-rVsZjDsbcnj1KVdh3dOg.png" /></figure><p>Notice now though, when we run the terraform plan command, it will tell us the plan is happening remotely. We will be provided with a link we can click on to be taken to the Terraform Cloud UI and watch the plan there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*zEhUEZ47CXvLdK6dSYynrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45jXLfZi8Xpcxc0WvAuB5w.png" /></figure><p>Ok, everything is looking good. Now, to initiate the actual terraform apply, we’ll use a git push command in our IDE. Once the branch is merged with the main branch in GitHub, That will trigger the plan for the apply. Like in the cli, after a successful plan, the apply must be explicitly confirmed in the Terraform Cloud UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*GTMMuQe4WaltE7jG_dlPSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhbxE2tOTPTmtJdvqwGNuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhtdyRj4LJL_p49Tp5pjUA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8ZC9CKpyJ_0J4oieSRQqQ.png" /><figcaption>Not recommended to ever self approve in a production environment</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KyJ6zNQvGULMdGlY1_rGlw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYlYehnL3ysqbIt7i5SZXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kz9Hn05r5wLObhK7Yp6Xfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*9HtQJKGhCZhVhjr9VwxO9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHHhNZ8rB7CtCrq_0rnEWg.png" /></figure><p>There you have it folks! We’ve just successfully deployed a 2 tier architecture using custom modules in Terraform. Let’s do some quick checks on our infrastructure to make sure everything is set up the way we want it to be. We’ll start by logging into the AWS console and looking at the vpc “Resource map”. Here we can verify that all the subnets were created, along with their route tables and route table associations. We’ll also see if the routes to the internet gateway and the NAT gateway are setup correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liWIvlHJS5a4rc9AfJR5ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxjf0nwQtl4xvH-7wwqnWg.png" /></figure><p>Next, in the AWS console, we’ll go to the EC2 dashboard and look at the running instances. We’ll select one of the ones we just created and we’ll check the security groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y8pEmfREXRO7VEXxjfU3DA.png" /></figure><p>Next, we’ll copy the public ip address and paste it into the URL bar of our browser. If our user_data script was a success, we’ll be taken to the NGINX landing page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0BZyaOWK-j1eYKvXnvyvow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RqRG38m-9HTiu6Dxoyl5_Q.png" /></figure><p>Boom! We got there!</p><p>Ok, lastly, we need to check the RDS MySQL instance. In the AWS console, navigate to the “RDS” dashboard and select “Databases” from the left-hand pane. Click on the newly created database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZYVfsiO8ZrVLQ6EwVCUH_g.png" /></figure><p>We want to review the endpoint as well as the port (the port should be 3306).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4sTARwti4npLyY4dEhTWng.png" /></figure><p>Our final check is to see if the MySQL instance can accept inbound traffic on port 3306 from the EC2 instances deployed in the public subnets. To do that go back to the EC2 instance where copied the public ip address and this time, select the “Connect” button in the top right corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ie4nQwvipqmEDR9jmqWbmQ.png" /></figure><p>On the next page, click on the “Connect” button again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*tG3QeU5J1UotRsxpRlVhDQ.png" /></figure><p>Now that we have an ssh connection to the EC2 instance, from here we can try to connect to the MySQL database. Issue the following command and enter the password when prompted:</p><pre>mysql -h &lt;mysql-endpoint&gt; -P 3306 -u &lt;username&gt; -p</pre><p>If successful, the return output will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qqctHsJm8VCqtLrU2Tmq0g.png" /></figure><p>And with these final checks, we can call our deployment a success! But, what comes next?</p><p>At the end of any terraform infrastructures lifecycle, comes the terraform destroy command. However, since we configured our architecture for cloud operations, we need to destroy it from Terraform Cloud. In the Terraform Cloud UI, under the workspace “Settings” tab, select “Destruction and Deletion”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0wfodp55IkcjFgztW8HX1w.png" /></figure><p>Then click on “Queue destroy plan” in order to delete the infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*R4wAOeRJ5d6wn0scepz2Qw.png" /></figure><p>Enter the workspace name and click “”Queue destroy plan”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*8UNFQZJFpm45BaAD0RkUuw.png" /></figure><p>After the destroy plan has successfully finished, confirm and apply the destroy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Aqzw_5jVr_cJbcjT1TwKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*DVDThnteNbBDzujxFX4xhQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*DuzrSaRSJwWJ-MN0fXI_Aw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNdT7y7USa1NoVaz4ZtOLw.png" /></figure><p>After the infrastructure has been successfully destroyed, if you wish, you can also delete the environment from Terraform Cloud.</p><p>And that wraps up our project for this article. To conclude, we deployed a 2-tier architecture with a web tier and with a database tier. We deployed our infrastructure using Terraform custom modules, making our code highly repeatable and easier to maintain. We deployed our infrastructure and ran some checks to verify the deployment was a complete success. Finally, we destroyed our architecture from the Terraform Cloud UI. Please keep an eye out for more educational articles from me. I hope you had as much fun reading this article as I had making it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d599870afe2" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/building-a-2-tier-application-in-terraform-using-custom-modules-9d599870afe2">Building a 2 tier application in Terraform using custom modules</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a highly available and fault tolerant Apache server autoscaling group with Terraform!]]></title>
            <link>https://medium.com/@tom.puntillo/deploying-a-highly-available-and-fault-tolerant-apache-server-autoscaling-group-with-terraform-4ca4f877dfff?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ca4f877dfff</guid>
            <category><![CDATA[autoscaling-group]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Mon, 07 Aug 2023 00:11:30 GMT</pubDate>
            <atom:updated>2023-08-07T00:11:30.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*maktL-0WT9IKlOhBhke0lg.png" /></figure><p>Hello! In today’s project, we are continuing our journey with Terraform that we started last week. In the previous write-up, we created what is known as a “monolithic”architecture, where all of our configuration code was in a single file. For today’s demonstration we are going to logically divide the code between some standard terraform configuration files so we can follow industry best-practices. This will help us keep our code clean and organized as well as giving the added benefit of easier maintenance in the future.</p><p>Now, let’s take a look at this week’s challenge!</p><p><strong><em>SCENARIO:</em></strong></p><p>An e-commerce company needs to handle a surge in traffic during the holiday season. The company wants to ensure that their website remains available and responsive to customers even during high traffic periods.</p><p>The project can be used to launch an Auto Scaling group that spans two subnets in the default VPC, ensuring high availability and fault tolerance. The Auto Scaling group will automatically scale up or down based on traffic, ensuring that the website remains responsive to customers at all times.</p><p>To ensure that the instances in the Auto Scaling group are secure, a security group is created that allows traffic from the internet and associates it with the instances. Additionally, a script is included in the user data to launch an Apache web server, ensuring that the website is available to customers.</p><p>To ensure that the Auto Scaling group has the appropriate capacity, it is set to have a minimum of two instances and a maximum of five. This ensures that the website can handle a surge in traffic during the holiday season, while also minimizing costs during periods of low traffic.</p><p>To verify that everything is working correctly, the public IP addresses of the two instances are checked. One of the instances is manually terminated to verify that another one spins up to meet the minimum requirement of two instances.</p><p>Finally, to ensure that the infrastructure is reliable and can be easily managed, an S3 bucket is created and set as the remote backend for Terraform. This ensures that the infrastructure is versioned and can be easily rolled back if necessary.</p><p>Be sure to use Variables where you feel appropriate to make your code as reusable as possible. Generally if it’s a value you think could change in the future or if it’s a value that’s repeated, then it should be a variable.</p><p><strong><em>OBJECTIVES:</em></strong></p><ol><li>Launch an Auto Scaling group that spans 2 subnets in your default vpc.</li><li>Create a security group that allows traffic from the internet and associate it with the Auto Scaling group instances.</li><li>Include a script in your user data to launch an apache webserver. The Auto Scaling group should have a min of 2 and max of 5.</li><li>To verify everything is working check the public ip addresses of the two instances. Manually terminate one of the instances to verify that another one spins up to meet the minimum requirement of 2 instances.</li><li>Create an S3 bucket and set it as your remote backend.</li></ol><p>Before we begin writing our code, let’s create the things that our code will be referencing. We’ll start with the S3 bucket.</p><p>In the AWS Console, navigate to S3 and click on “Create bucket”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QU5gr1ph6iNMfi7sDvFstA.png" /></figure><p>Make sure when creating your bucket to keep the default of “Block <em>all </em>public access” enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_ZOmOwuE-owffrpQGUzAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*mdIpZn0Y8APCZVfUfQJPNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*R-a7JBP9c8DQuva_rrDDAg.png" /></figure><p>Next, let’s go ahead and create an ssh key pair, in case we decide at any point we’d like to ssh into our instances. We’ll do that with the ssh-keygen command.</p><pre>ssh-keygen</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*aD5tGBrsQUHsnSKqGKf6GQ.png" /></figure><p>Next, let’s quickly write our user-data script to install Apache. Open up your favorite text editor. I’ll be using Vim. Input the following bash script:</p><pre>#!/bin/bash<br><br>sudo yum update -y<br>sudo yum upgrade -y<br>sudo yum install httpd -y<br>sudo systemctl enable httpd<br>sudo systemctl start httpd</pre><p>Save and close the file. Our launch template will require the user-data script to be base64 encoded. Let go ahead and do that now utilizing the following command:</p><pre>base64 -w0v/path/to/user-data-script /desired/path/to/encoded/file</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*NdnYhj8z8h3oeV2Op8iHrQ.png" /></figure><p>Now if we take a look into the newly created file with the base64 encoded user-data-script, you’ll see a coded string that you can now reference in your terraform code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kx_FaRsooQBxpQm6geojjQ.png" /></figure><p>Ok, with all that created, we can finally begin writing our code!</p><p>Let’s start by creating our terraform.tf file.This is where we will define our terraform block, which will configure our backend, ensuring the S3 bucket we created is being used for storing the .tfstate file. It also defines the terraform version our code will use and select which providers our code will require. Mine looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*mOVYIcJOeqS40Z8r8jUt9w.png" /></figure><p>Next, let’s create our providers.tf file. As you can see ours is pretty simple. We’re only defining a single provider, specifying the default region and the tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*28ZQfhrcUhqn6qab7T1a-g.png" /></figure><p>Now we’ll create our variables.tf file to define variables that are likely to be repeated or frequently changed. For our purposes, we’re only defining 3 variables. The “host_ip” variable, which denotes the public ip address of the machine running Terraform. The “user_data” variable, which contains the base64 encoded user_data script we made earlier. And finally, the “azs” variable, which defines a list of availability zones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lbRXTJe4ekn-uYvdIi2N7Q.png" /></figure><p>And now, most importantly, let’s create our main.tf file and start defining our resources. First up, our security group. Here, we’re setting 2 separate ingress rules. The first allows traffic on port 22 (SSH) coming from our terraform environments public ip address (defined using the “host_ip” variable). The second allows inbound traffic on port 80 (HTTP) from anywhere. Our egress rule sends traffic out on all ports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*fokE9nf4Ak-Qm-aATA-b9A.png" /></figure><p>Next up, we’ll create a key pair. Together with the ingress rule on port 22, this key pair will allow us to ssh into our instances if we need to. For the public key, use the id_rsa.pub key that was created when we ran the ssh-keygen command earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*Rp8IizOMrhJP3GJNxGe2pQ.png" /></figure><p>Our next resource we’ll create is the launch template that our EC2 instances in our autoscaling group are based on. Here we’re going to declare the image type (Amazon Linux 2), the shutdown behavior (terminate), the instance type (t2.micro), the key name (the key we created earlier is referenced), The security group (the security group we created earlier is referenced) and the user_data script (user_data script created earlier is referenced).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*NwqAtBfm3ea3_SwS0UACkw.png" /></figure><p>Lastly, the final resource we’ll declare is the autoscaling group. Here we’ve selected our availability zones (defined by the “azs” variable), set our capacity reservations and selected the launch template to base our instances on (launch template created earlier is referenced).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*kwhB6u6DAfXYMOiSj4fB0w.png" /></figure><p>And that’s all of our code! Now it’s time to put it into action. We’ll start by initializing our directory with the following command:</p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*hy9c-Cb46uaXr4HNI61k-g.png" /><figcaption>I had already run this command previously, but this is where you would see S3 get set as the backend and see the provider plug-ins get installed</figcaption></figure><p>Next, I’ll run the following 2 commands to format my code and to validate the syntax is correct:</p><pre>terraform fmt<br>terraform validate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*VfwlS4mhyzGblK-fZ34tsQ.png" /></figure><p>Next up is the planning stage. This will visualize the plan of action terraform will take to execute the environment.</p><pre>terraform plan</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ox13C2ERbM1gG0tsKVz1Tg.png" /></figure><p>And to actually deploy our environment, we’ll execute the terraform apply command.This will again show us a plan of action as well as prompt us to continue. Any response other than “yes’ will abort the apply.</p><pre>terraform apply</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*Oo1D0Lkh_D4c-tWkkqEHOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*IQ8AwGBxdAwSW8aegNCTdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*_XzUp_-UMlftRuKvJdEN7Q.png" /></figure><p>Excellent! Now that our environment has been deployed, let’s jump over to the AWS Console and take a look at some of our resources. Let’s start with the S3 bucket and take a look at our state file. Here we can see the details of our current deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EfK-1Hll1zeMfOrEVXhdDg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXrf7tee2VdF_h6eF6iQvQ.png" /></figure><p>Next let’s take a look at our running instances. Here we see the 3 EC2 instances we deployed (my 4th running EC2 is my AWS Cloud9 environment). We’ll copy the public ip address of one of them and paste it into our browser to see if we can reach the Apache landing page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MKASQ-QjrbVM3myeLP8uFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OMTMTY1HX3L16-sj3I9yHg.png" /></figure><p>We got there! Nice. Now the last thing to check is to manually terminate one of the instances and verify the autoscaling group replaces it. Let’s go back and take another look at our instances and select one to terminate. Take note of the instance id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNl-RwVh1BFhFVc5IXXz4Q.png" /></figure><p>Refresh the dashboard to see the instance is gone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vQ8uRQW_8iw_UOR4InP7ug.png" /></figure><p>Wait a few minutes and refresh it again and bam! There you have it! The terminated instance was replaced by our autoscaling group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQWeV_Ph3W2_Y4VrEUYVrQ.png" /></figure><p>That’s all it takes folks! Just over 100 lines of code and you can have a ready made, highly available and fault tolerant fleet of web servers ready to serve your customers!</p><p>Don’t forget to clean up your environment when you’re done with terraform destroy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*KOzPLjm6cHce8xsnFYhzfg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*mpF0ylguc3giZo-zyKTqrw.png" /></figure><p>Hopefully this demonstration has illustrated to you some of the in demand and highly valued benefits that Terraform has to offer. I hope you’ve had as much fun reading this article as I’ve had making it. See you next time!</p><p>Below are the secret GitHub Gists to the configuration files I used in this demonstration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f1f748e89441551f38a2e90e35776f9/href">https://medium.com/media/1f1f748e89441551f38a2e90e35776f9/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f75314c8846ccd8c3068ee143669ad8/href">https://medium.com/media/9f75314c8846ccd8c3068ee143669ad8/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9608461e5b02ee960249e22343a138a7/href">https://medium.com/media/9608461e5b02ee960249e22343a138a7/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f6d44688771029086fec061a7cec879/href">https://medium.com/media/8f6d44688771029086fec061a7cec879/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5eaa482c99e0875e9c57369cc3a802e8/href">https://medium.com/media/5eaa482c99e0875e9c57369cc3a802e8/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a37d70e921c96fc7d0e60039cc9c4cbe/href">https://medium.com/media/a37d70e921c96fc7d0e60039cc9c4cbe/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ca4f877dfff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Terraform to deploy a Jenkins build server with an S3 bucket for saving artifacts]]></title>
            <link>https://medium.com/@tom.puntillo/using-terraform-to-deploy-a-jenkins-build-server-with-an-s3-bucket-for-saving-artifacts-10b891748162?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/10b891748162</guid>
            <category><![CDATA[iac]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Mon, 31 Jul 2023 00:24:01 GMT</pubDate>
            <atom:updated>2023-07-31T00:24:01.947Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08Ktmrc2ZAoLeOn1Y2YYZg.png" /></figure><p>Hello! Do you spend your days deploying the same type of resources over and over? Do you find yourself moving from new environment to new environment only to have to build everything from scratch yet again? If so, you may want to seriously consider adopting Terraform, the world’s premier Infrastructure as Code tool that businesses are using to skyrocket their productivity and minimize their technical debt.</p><p>Like all IaC tools, Terraform significantly reduces the time required to build, modify and destroy environments as well as removes the risk of human error. Where Terraform truly stands apart though is the fact that it is cloud agnostic, meaning it is compatible with all the leading Cloud Providers. This, in addition to its ease of use and intuitive design, is what makes Terraform a titan in today’s modern landscape.</p><p>In order to get a practical idea of how this technology can be used, let’s take a look at this weeks project objectives.</p><p><strong><em>OBJECTIVES:</em></strong></p><ol><li>Deploy 1 EC2 Instances in your Default VPC.</li><li>Bootstrap the EC2 instance with a script that will install and start Jenkins. Review the official Jenkins Documentation for more information.</li><li>Create and assign a Security Group to the Jenkins Security Group that allows traffic on port 22 from your ip and allows traffic from port 8080.</li><li>Create a S3 bucket for your Jenkins Artifacts that is not open to the public.</li><li>Verify that you can reach your Jenkins install via port 8080 in your browser. Be sure to include a screenshot of the Jenkins login screen in your documentation.</li><li>Push your code to GitHub and include the link in your write up.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*duQB0-MD-57t0UK7Jy3UBg.jpeg" /><figcaption>Terraforming… get it?</figcaption></figure><p>I will be utilizing the AWS Cloud9 IDE during the course of this write-up, but feel free to use whichever IDE you like best.</p><p>I will be writing all my code in a single main.tf file with hardcoded data. This design is known as monolith and is generally not well suited to production environments, but for small proof of concept / throw away environments like this one, this design is fine. Just know that Terraform is capable of dividing the configuration among several files to help with organization and maintenance. As long as all the configuration files are in the same directory, Terraform is smart enough to put it all together. Aside from main.tf, other common configuration files are variable.tf, provider.tf and output.tf.</p><p>The first thing I’m going to do is create a key pair to use in my terraform code. I’ll generate this key-pair locally and eventually paste the public key into my main.tf file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*P1ov7VSF52vYaGd_tMnvOg.png" /></figure><p>Terraform language is declarative, meaning we must “declare” every aspect of our build. I am going to step through the code I’ve written block by block to show you exactly what this means.</p><p>The first thing we’ll need to add to our main.tf file is the terraform block and the provider block. This will select the cloud and the version of the cloud to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*mN16pSLln5UBbBAeDfoD_g.png" /></figure><p>Next, I’m going to define a variable called “aws_region”, setting the default region to us-east-1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*N3LIVy1IkAYwJ8wieeauig.png" /></figure><p>Now we’re going to begin declaring resources. First up is the bucket we need to create for the Jenkins artifacts. Public access to buckets is blocked by default, so we can simply just create it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*Wm6NMX3_EoMwnY8FGqjRaw.png" /></figure><p>Next we’ll create the security group. Our objectives specify that ports 22 and 8080 must be open in our inbound rules (called “ingress” in terraform language).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*AlzC1Ui2AMRd78cVcXRIow.png" /></figure><p>The next resource I wanted to declare was the aws_key_pair resource. I decided to define 2 additional variables to make this part easier. The variables are “key_name” and “public_key”. For the “public_key” variable, I copied the public key from the key pair I created earlier (~/.ssh/id_rsa_pub on the host machine) and pasted it into the “default” section of the variable. I then used “var.key_name” and “var.public_key” to create the aws_key_pair resource, “ssh-key”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*qKP-1mBGEzoAuA6vmL_w6A.png" /></figure><p>Up next is the only data block in this code. Data blocks differ from variable blocks in that, for data blocks, you select existing resource options to include in your code. Whereas in a variable block, you define your own options. Our data block is defining the AMI that will be selected for our EC2 Instance .</p><p>We’re going to use an Amazon Linux 2 image and we are going to use the “most recent” version of that image. The “values” section in the filter brackets show the “amzn2’ ami type, the “hvm-2.0” virtualization type and the “gp2” storage type. The “Owner” of the ami is Amazon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*O0saCMzBe4hQxrF3UulIiw.png" /></figure><p>The next several resource blocks all center around providing the EC2 Instance with the appropriate permissions to upload and remove objects from the S3 bucket we’re creating. First we are going to create an IAM Policy called “jenkins_policy” defining the permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*B2cGoodBSHJFwdR6re8-eg.png" /></figure><p>Then, we’re going to create an IAM Role called “jenkins_role”. The “AssumeRole” action has an “Allow” effect for the “ec2.amazon.com” service (only ec2 resource types can assume the role).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*8JoWka6JFPSFCRBzhMvHBg.png" /></figure><p>Next, we’re creating the IAM Instance Profile and assigning it to the “jenkins_role” Role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*RlTYvjyrMY6BwQLZpBK9CQ.png" /></figure><p>And then we complete the permissions setup by attaching the “jenkins_role” to the “jenkins_policy”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*2yeVvyjEuHgCJ8Hii3mStQ.png" /></figure><p>The final resource we are going to declare in this setup is the EC2 instance. Notice for the “ami”, we’re calling on the earlier defined data block and the “key_name” is calling on the aws_key_pair resource block. Additionally, we are including a user data script that will install and start Jenkins for us. (I also changed the permissions on the /var/lib/jenkins/secrets/ directory and the initialAdminPassword file to allow me to use that info at the Jenkins login screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*ggBmmDGhRSvRzkd3QxoqUA.png" /><figcaption>The steps to install jenkins can be found at the following link: <a href="https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/">https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/</a></figcaption></figure><p>And that’s it for our code! Now let’s put it to use. First, don’t forget to save your main.tf file! Once that’s saved, in your terminal, type the following command to initialize your directory:</p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*gHzOKPxlmsQ4aTidaLmiRw.png" /><figcaption>I had nothing to initialize here because I already ran this command during my practice run, but this is where you would see terraform downloading the required providers</figcaption></figure><p>Next, I’ll validate my code is written in the proper syntax:</p><pre>terraform validate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*XdrH5vu-Uog01G3vM5L2CA.png" /></figure><p>Now it is time to issue the terraform plan command. This, exactly as the name implies, will show us the plan of action that will be executed when deploying the environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CkXJvsei46XnlkJUwVBjhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JGnK0uvyKfbhJvAzL0M4QA.png" /></figure><p>Notice how it gives a total count of all resources that will be deployed as well as detailed accounts of each of those resources.</p><p>Up next is arguably the most important command of the all, terraform apply. This is the command that will actually deploy our infrastructure. Once deployed, Terraform will keep note of all the resources and their current state using the terraform.tfstate file. Once executed, terraform apply will require us to say “yes” when prompted (except when using the -auto-approve flag). If any other value other than “yes” is provided, the terraform apply command will be canceled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PPeMk5tLSdTr5DG0gHFM0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*2wya1v0Ko63NM9iRn6dHSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*FjwFJv1C8YPCsO2aR87ZNw.png" /></figure><p>We did it! We successfully deployed our cloud environment. Now we need to test to see if everything went according to plan.</p><p>First, let’s go to the AWS Console and grab the public ip address of our newly created EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3XFhk14jr-szxqOKKWcJKQ.png" /></figure><p>Then, we’ll paste the ip address into the browser url bar and follow it with “:8080” to ensure traffic is going over port 8080.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9J88CtBEpJPs3rOZJH78Lg.png" /></figure><p>So far so good! Let’s connect to this instance and pull the initialAdminPassword and use it to set up Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4TRyTmgo_9Ylx8FN89JVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*18FKZ77L_uuUoRWwUiapVw.png" /></figure><p>So far so good! Let’s connect to this instance and pull the initialAdminPassword and use it to set up Jenkins.</p><pre>cat /var/lib/jenkins/secrets/initialAdminPassword </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*jiBJwDPBBmhHitkCr8I_iw.png" /></figure><p>Paste it into the “Administrator Password” field in the browser and complete the Jenkins setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mPFZEakaUBnbGSzd5xHv3A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXMnycl0KvSVd--FFhW93g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*db2LmJukxcqfESB6o1i4Sg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-_3XRqo5iRwubfBKNhkKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1nbdqVNwFm2Zbizppk_vWA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rV29wqJfLku_B48wkIyxjg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VLXXQ2qL-cmwmR4hmMEPeQ.png" /></figure><p>Ok, Jenkins seems to be in working order. The final thing we need to verify is if our IAM Instance Profile is in fact allowing our EC2 instance to add and remove objects from our bucket.</p><p>In the AWS Console, navigate to S3 and click on “Buckets”. Select the bucket we created with our terraform code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qpMkBzJvTf6ZH19MiClloA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhQ5-wshyrAp18byadBfAg.png" /></figure><p>Ok, so, now we’re going to go back to our EC2 instance and create a file. We’ll then attempt to upload the file to our bucket. If our IAM Instance Profile is working like it should, we’ll be able to upload it without needing to provide any credentials. We’ll call our file “testfile.txt”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*UiVsmRm7VC-U1sUkLV6-vA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*-6ALhgEO7BgNZeUcoSah4A.png" /></figure><p>Go back to the console and copy the bucket name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NnQ7zDe1FBmtjteCI3bm3A.png" /></figure><p>Enter the following command:</p><pre>aws s3 cp testfile.txt s3://name-of-bucket</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*2b9lFkIc1pNCrCmz9WxlWg.png" /></figure><p>Now go back to the bucket in the AWS Console and check to see if your upload succeeded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hBgHH0ZEakCz8zNqr75Tug.png" /></figure><p>There it is! Let’s see if we can open it. Click on the object, then on the next page, click “open”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Evlx-NmxeVy5_jkYPsy9eA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*zYKLAKFT3zuVwpNC9xmNLw.png" /></figure><p>There you have it folks! Nothing to it!</p><p>As it stands, this was just a quick look at the opportunities made possible by using Terraform. Over the next few weeks I’ll be diving deeper into this technology and creating more articles to share!</p><p>Before we end things, don’t forget to tear down your environment with terraform destroy. Just like with terraform apply, you will need to say “yes” when prompted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*wk4c0BYF5lPbyXb_SN-gAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*MYI0NdfLw4a6Xdh3WLrH5Q.png" /></figure><p>Whoops! Looks like I forgot to empty my bucket first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qw5i5Y76Z18HDuwxYTboNA.png" /></figure><p>Let’s navigate back to the bucket and delete the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HP9DZ1IhEiziWw2jOgpGTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*5ywVOfYtIgaq2EDDPeGEIA.png" /></figure><p>Now let’s try terraform destroy again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*gyyL8PvU9Jd4837oWwIZPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*SiY5UTNKzStFeSAilyLbIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*kA81vSKeRptaC76m6LYqQg.png" /></figure><p>Below is the secret gist to the Terraform code I wrote for this article:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c19bbbcd526ffc15bc07db250f2f3e9/href">https://medium.com/media/7c19bbbcd526ffc15bc07db250f2f3e9/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10b891748162" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn How to Make Your Applications Highly Available with Docker Swarm]]></title>
            <link>https://blog.stackademic.com/learn-how-to-make-your-applications-highly-available-with-docker-swarm-bfa39b404670?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfa39b404670</guid>
            <category><![CDATA[docker-stack]]></category>
            <category><![CDATA[docker-swarm-manager]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-swarm]]></category>
            <category><![CDATA[yaml]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sat, 15 Jul 2023 21:44:00 GMT</pubDate>
            <atom:updated>2023-10-11T01:18:56.515Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Zyo9JLxYAVO5_2s__lMiNw.png" /></figure><p>Hello! Today I will be showing you all about Docker Swarm and demonstrating why businesses all over the world have started to incorporate it into their tech stack. If you have read my previous articles on docker, docker swarm will definitely get you excited.</p><p>If you haven’t already done so, I encourage you to read my previous articles on docker before continuing.</p><p><a href="https://medium.com/@tom.puntillo/all-hail-the-whale-running-containers-and-pushing-new-images-to-docker-hub-e428a1e6937c">https://medium.com/@tom.puntillo/all-hail-the-whale-running-containers-and-pushing-new-images-to-docker-hub-e428a1e6937c</a></p><p><a href="https://medium.com/@tom.puntillo/creating-a-jenkins-container-with-persistent-data-using-docker-compose-2a4a06c25793">https://medium.com/@tom.puntillo/creating-a-jenkins-container-with-persistent-data-using-docker-compose-2a4a06c25793</a></p><p>This week for terminology, we only have a single term to define</p><p><strong><em>TERMINOLOGY:</em></strong></p><p><strong><em>DOCKER SWARM — </em></strong>Docker swarm is a collection of computers or virtual machines bound together in a cluster. The cluster is divided between Manager Nodes that orchestrate and schedule tasks and Worker Nodes that carry out those tasks. (NOTE: Manager nodes can also execute tasks but worker nodes cannot dispatch tasks). A node may be switched between manager and worker node as needed. When utilizing docker swarm you can distribute services across nodes, making your applications highly available and extremely resilient.</p><p>And that’s all I have for you this week for new terms. Let’s jump right in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*a8gO7Eq2nNOOLm4L1G43zg.png" /></figure><p><strong>SCENARIO:</strong></p><p>Your company is new to using containers and has been having issues with containerized workloads failing for their global shipping application. The company is worried about additional potential data loss as they have no way to reschedule containers that are failing. The company has one week to update their infrastructure and leverage the use of Docker Swarm so that data can be restored for the containers that are no longer in a healthy state. Because they are not familiar with Docker Swarm, they will need a step-by-step guide on setting up a Swarm for their containerized workloads to assist with container orchestration.</p><p>‌The solution should include the use of a global service since the company’s global shipping application is having issues. Your Swarm should be able to launch the containers using a global service. Remember, a service is simply a group of containers of the same image that facilitates the scaling of applications. Your global shipping application will require the use of at least three networked hosts, which should be AWS EC2 Ubuntu 20.04 instances.</p><p><strong>FOUNDATIONAL:</strong></p><p>As the junior engineer, you are responsible for only configuring Docker Swarm and providing your team with a step-by-step guide of setting up the Swarm. You should assume that there are new team members that may not have Docker installed on their devices. Your swarm creation should include the following steps via your remote ssh terminal in VSCode:</p><ul><li>Installing docker on all hosts</li><li>Verifying that docker is in an active state</li><li>Change the hostname on each node to identify the master node and the two worker nodes</li><li>Validating the worker nodes all share the same security group as the master node</li><li>Be sure to add your ssh key to each node</li><li>Creating the Swarm using one master node and two worker nodes</li><li>Show your team members the status of the docker nodes</li></ul><p>The first thing we need to do for a swarm is to create the instances the swarm will be deployed to. These instances ought to be the same OS build and all instances MUST be in the same security group. The easiest way to accomplish this is to create a launch template for your Docker instances. This will ensure consistency through any instance spun up for the swarm.</p><p>Let’s start off by going to the EC2 dashboard and selecting “Security Groups” We will need to ensure we have all the proper ports opened in the “Inbound rules”.</p><p>Click on “Create security group” and provide the basic details. Be sure to use the default VPC for the region you are in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZlKn6xqdCZbKxn_lZZv4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QeWnEUzrJIUDb3UT_TpmjA.png" /></figure><p>In the “Inbound rules” section, click on “Add rule”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cs3U9XkG2jWYy0AdIF1mfw.png" /></figure><p>Add 3 rules. SSH on port 22 from your ip address, HTTP on port 80 from anywhere, and All traffic coming from your VPC CIDR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuJTsw_wN-WTBABSpf5UoQ.png" /></figure><p>(<strong>NOTE:</strong> You can find the VPC CIDR in the VPC Dashboard)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAwZgXKTOrK4ieE3UYN_kw.png" /></figure><p>Next, we’ll move onto creating the key pair. All of the nodes in the swarm must use the same key pair. Click on “Key Pair” in the left pane.</p><p>Click “Create key pair”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6nLZbvj3-eg7MNhls3NFQ.png" /></figure><p>Provide your key pair with a name and ensure the “.pem” option is selected. Click “Create key pair” and save it to your downloads folder (or any folder of your choosing).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*AaPpKrU_tiwYiIwtGljmZQ.png" /></figure><p><strong>HELPFUL HINT: </strong>Upload your key pair to one of your GitHub repos so it can easily be pulled to your swarm nodes later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-FVposEhM2E3ztj_1PvNA.png" /></figure><p>Now that we have our security group and our key pair created, we can create our launch template. In the left pane, click on “Launch Templates”. Click on “Create launch template”. Give your template a name and fill out all the required fields. The image we want to use is Ubuntu 20.04, the instance type to use is t2.micro and we’ll use the security group and key pair we just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EMDlL_cYGcJM6rRJBL7Mxw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*-sEUv8h9jVpXUf_eaCleww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*wQPVOtTbMHYTSSdGTkdOVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*Tbsd2kuIpTE-tSMt16SBBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*6MQwYfjjpR2crOZwGaWe8Q.png" /></figure><p>The final portion of the launch template is to provide the user data. This will run on the first launch of the instance only. We will use this to update the instances and add docker to them so it’s already on there when we spin them up. This must be written in the form of a bash script. You can use the “convenience” installation script found at <a href="https://docs.docker.com/engine/install/ubuntu/">https://docs.docker.com/engine/install/ubuntu/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k9Ykk7Vkwsfp-F0lqgusow.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b8RtKhKca4NF1FninmqCqQ.png" /></figure><pre>#!/bin/bash<br><br>apt update -y<br>apt upgrade -y<br>curl -fsSL https://get.docker.com -o get-docker.sh<br>sudo sh get-docker.sh</pre><p>Click on “Create launch template”.</p><p>Click on the link in the “Success” message.</p><p>Click on “Actions” followed by “Launch instance from template”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K72KYdPF0nVoplrnjB3HdQ.png" /></figure><p>Update the “Number of instances” field to the desired number. We will be launching 3 instances for this example. Click on “Launch instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dn3zPB_KnH0hgyqSx194hw.png" /></figure><p>Now that we have our 3 instances up and running, let’s edit the “Name” field to make them easily identifiable. Specify 1 manager node and 2 worker nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-0lcfPDHJTXTu8kMeGJHQ.png" /></figure><p>Ok, we’re halfway there!</p><p>Next, we’ll want to configure your IDE’s ssh config file to point towards what will be the manager node.</p><p>If you don’t already have a config file created, create one now. Open your IDE (I’m using VS Code) and navigate to your hidden ssh folder.</p><p>For Windows use “C:\Users\your_username\.ssh” and for Mac or Linux use “~/.ssh/”</p><p>Click on “File”, “New File” and name it “config”. Add the following content to your config file.</p><pre>Host &lt;Name you wish to give your connection&gt;<br>    HostName  &lt;Instance Public IP Address&gt;<br>    IdentityFile &lt;Path to your locally saved private key from the earlier key pair creation&gt;<br>    User ubuntu</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iLEnyFj8acIPALoosfuQtw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MTHhgKGw0FRr_DR2nfdh-Q.png" /></figure><p>Don’t forget to save!</p><p>Now that we have our config file created we can use it to ssh into our manager node. In VS Code, in the bottom left corner, click on the “Open a remote window” button. Click on “Connect to a host”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oqzCya8NJj7lPS6ur-BB6g.png" /></figure><p>Click on the name you gave your host in the config file. To keep it simple, I called mine “Docker”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOpsy25i-eRJDoeImkDo0Q.png" /></figure><p>This will open up a new window. Click “Continue”</p><p>You are now ssh’d into the manager node. We haven’t updated the hostname yet though so it still only shows as the instance’s private ip address.</p><p>To verify our user data script was effective, check if docker is installed using the following command:</p><pre>docker --version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*Kl1Voqyf_9membKE74uBvw.png" /></figure><p>There it is! We can take this to mean the user data script ran successfully and docker is installed and active.</p><p>Now, I always like to change the root password to something I can easily remember. Update the root password with the following command:</p><pre>sudo passwd</pre><p>(<strong>NOTE:</strong> The new password will not be displayed as you type. This is a Linux security feature. Rest assured your input is being recorded.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*DnF7cWa-inDRAhmGRhTGaQ.png" /></figure><p>Now with that change made we can update the hostname. To do so, use the following command:</p><pre>sudo hostnamectl set-hostname manager1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*P0cBampRH1JcAsM-A9sYVQ.png" /></figure><p>This change will not take effect until a reboot has taken place. Since we need to do that anyway, now is the perfect time to elevate our permissions so we don’t need to type “sudo” before every important command. To elevate your permissions, enter the following 3 commands:</p><pre>sudo usermod -aG sudo ubuntu<br>sudo usermod -aG adm ubuntu<br>sudo usermod -aG docker ubuntu</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*eNStaLScbm1hhVem2DVJgA.png" /></figure><p>With all those changes made we can now reboot the instance. Do so with the following command:</p><pre>sudo reboot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*AL6GoVDxNvpr2jsQNE8FRA.png" /></figure><p>Once reconnected, you’ll notice the hostname has now been updated to manager1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*uxTivZWFBHEvmKssFi2dPw.png" /></figure><p>Now we’ll need to set up the connection to our worker nodes. We’ll do that by creating another config file, this time, we’ll make it on manager1. Remember though, in order to connect to the other nodes via ssh, we’ll need a copy of the private key. Let’s go ahead and clone the git repository where we stored our private key to the manager node. To do so, you’ll need to first head over to your GitHub account and copy the clone URL. Be sure to copy the one under the HTTPS tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzA0QVb66R9YZRWRXkQ7eQ.png" /></figure><p>Back in your terminal, enter the following command:</p><pre>git clone &lt;url you just copied&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*vR2fefdcKzEZ7Ac6yoHngA.png" /></figure><p>Use the “ls” command to verify the git repo has been cloned</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*1Z_4pW598nB5NhHXPZK75A.png" /></figure><p>Now, copy the private key from wherever you saved it in the repo into the hidden ssh folder. Use the following command:</p><pre>cp &lt;path to private key&gt; ~/.ssh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*7VxRLy7o4kc68vVJ-nBQLQ.png" /></figure><p>Since this needs to be used securely we need to modify the permissions to the private key. To do so, use the following command:</p><pre>sudo chmod 600 ~/.ssh/&lt;private key&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*88dv8UcHlk8cWJDh0xMykg.png" /></figure><p>Change directories into .ssh.</p><p>Use the vim command to create and edit the config file. We’ll have 2 entries in this config file. One for node2 and one for node3. Also, for the config file(s) created on our instances, we’ll set the hostnames to the instances’ private ip addresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WhAyEjBYMUrukTUVxqa8Eg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJP_Ny6Tzmyii95aMoFY_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*Sam8fpRlVbquy_RlyMMxZg.png" /></figure><p>With the config file created we can now ssh into either of our worker nodes. Split your terminal into 3 terminals using the “Split terminal” option in VS Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pXusGOyPrKa8r9f2VyAfLA.png" /></figure><p>In the 2nd and 3rd terminals, ssh into node2 and node3 respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MYTkJtyeb7AeF-O1IWEVg.png" /></figure><p>Almost there!</p><p>Now that we have successfully remoted into both worker nodes, we are going to repeat the same steps on both of those we just ran on manager1.</p><p>• Check if docker is installed</p><p>• Update root passwd</p><p>• Set hostname</p><p>• Elevate permissions</p><p>• Reboot</p><p>• Clone the git repo</p><p>• Copy the private key into the ssh folder</p><p>• chmod the private key</p><p>• Create the ssh config file (Be sure to use the correct hostnames and private ip’s to connect to the other 2 instances, i.e. the hostnames and private ip’s of the 2 nodes you are not currently configuring)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSDRQqUQ81umOePECzn_Bw.png" /></figure><p>With all 3 of our nodes configured, the last thing to do is actually create the swarm. Luckily for us though it only takes one command per node. On your manager node, type the following command:</p><pre>docker swarm init --advertise-addr &lt;manager node private ip&gt;</pre><p>This will return the join command and token to provide to the worker nodes. Simply copy it and paste it into each worker node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xAPyMogxES0FndlTk91Rw.png" /></figure><p>To verify your swarm has been created, on the manager node, type the following command:</p><pre>docker node ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*I3Exc_guWFvyZj06Lm0pxQ.png" /></figure><p>And that’s it! You have successfully deployed a docker swarm and it is ready to start running global services!</p><p><strong>Advanced:</strong></p><p>As the lead engineer, you’ve reviewed the Swarm configuration and you are now ready to deploy your services on the Swarm. Consider the following steps as you deploy the service:</p><ul><li>Using only the CLI, ssh into the master node host and run the command to create your service using an Official image of your choice out of the following company preferred images:</li></ul><ol><li>nginx</li><li>apache</li><li>redis</li><li>python/alpine</li><li>ubuntu</li></ol><ul><li>Initially only launch 1 replica.</li><li>Run the commands to verify the service has been created and is running</li><li>Since it is a global application that needs to scale, Using the CLI, run the necessary commands to scale the service to 5 replicas.</li><li>Verify that the service has scaled</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*uVekx9e1D4qfPJHD3rlR_Q.jpeg" /></figure><p>A docker swarm is of no use unless it’s running some services. More than that though, where docker swarm truly shines is in its ability to scale those services up and down to meet demand. In this section I’ll show you how to do just that.</p><p>The service we are going to run is going to be nginx. In your manager node terminal, run the following command:</p><pre>docker service create --name nginx-service nginx</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*xmkpNoxJck_wB9EEl73TbQ.png" /></figure><p>In the output you will see an id, as usual. You will also see a countdown starting from 5 seconds, verifying the service(s). Once you see “verify: Service converged”, you know your service is up and running.</p><p>Verify with the following command:</p><pre>docker service ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*b6UdLu3nD01Ed1dv7mZBHw.png" /></figure><p>And you can follow that up with the docker ps command to view further details on the running service.</p><pre>docker service ps &lt;service name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*sm-4OB8ss3wi62ycfJPs0g.png" /><figcaption>(Apologies for the different ID number shown here. I somehow missed taking this screenshot while creating this tutorial and had to recrete the service again later to provide this screenshot. It is the only one with this service ID)</figcaption></figure><p>The output given from this command is extremely useful. Notice in the “Name” field, docker added a .1 to the end of the service name. That is because we only have 1 replica running. Also, you can see the node the service is running on as well as the desired and current state of the service.</p><p>Now it’s time to demonstrate how easy it is to scale your docker services. We are going to scale to 5 replicas. To do so, use the following command:</p><pre>docker service scale nginx-service=5</pre><p>You’ll see the same output as when you ran docker service create, except this time it has completed 5 tasks instead of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*xaHf0Zma0OMUx9yMNdI7Bw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*Anc9QQwTmD2kWXKw5_EMFA.png" /></figure><p>Again, we’ll run the docker service ls and the docker service ps nginx-service commands to view the additional replicas and details about them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*iu5R6dKf4vWEuUrDFWC-ow.png" /></figure><p>As you can see we are now running 5 replicas. Also, direct your attention back to the “Name” field under the ps command. Docker has added a numeral suffix for each replica of the service. Another thing to note is the “Node” field. As you can see, the replicas have been dispersed across our swarm.</p><p>It’s that simple folks! But as simple as that just was, it was still very much a manual process. Let’s learn how to automate this process</p><p><strong>COMPLEX:</strong></p><p>As the senior engineer, you are looking to make this deployment as seamless as possible, and you’d like to use a more complex approach. Consider the steps for the foundational and advanced sections. Note: You will have to do some research to ensure you have all files required for this deployment.</p><p>Deploy your application via a Stack to your Swarm</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2RfK-cTQkr9_aRRdVb3sVA.jpeg" /></figure><p>In a previous article I demonstrated the benefits of using a docker compose file for deploying containers. For docker swarm however, that same type of file is called a docker stack file (although it is still called with -c and –compose-file when deploying the stack.) Additionally, just as the docker stack file has replaced the docker compose file, the “run” command has evolved into “deploy”. You “run” a container, you “deploy” a stack.</p><p>Before we write our stack file though, let’s tear down our existing running services. We do that using the following command:</p><pre>docker service rm &lt;service name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*07ZezcS0KJUtUC-uHX5oog.png" /></figure><p>The output will spit out the name of the service. We can run the docker service ls command again for verification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*ZjEI8kFS4X6I193iDJeiFw.png" /></figure><p>Good! Now we can use the nginx-service name again without getting any conflicts.</p><p>We’ll start by making a new directory and cding into it. I’m making mine inside the git repo we cloned so I can push it back up to GitHub to reuse again anytime I need to, anywhere I am.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*PaWya56aqq4aUofN7ccJzg.png" /></figure><p>Create your docker stack file using the touch command. You can call this file anything you want. Mine is going to be called deploy-nginx.yml</p><pre>touch &lt;filename&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*mAg1wqMCbDUVVp5ENyGBgw.png" /></figure><p>Next, open the file in your IDE and edit it.</p><p>Your file should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*cAIEcHdUzsPR0-wKtz4KpQ.png" /></figure><p>Don’t forget to save!</p><p>Now, from the directory you have your stack file in, run this command:</p><pre>docker stack deploy -c &lt;stack-file.yml&gt; &lt;desired service name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lbWK8cPr--NgStsgnY7oSg.png" /></figure><p>And boom!. Look at that! We’ve got nginx running. And because it’s running on all nodes, you can visit the public ip of any of them and still hit the nginx landing page. Check it out!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*owS66hbLHWQ5hx0CliHyZA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*85727HiaW0Wnw5lXLlL0Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*UV664SP4al734rvVcJB6FQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ne0MCb51A8W7EqXYTZb4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*FJQPeWYWYyCiDTcgIfSYTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H2_IamtxI4psz4urWoPxEg.png" /></figure><p>And there you have it. We just deployed a highly available web app with a single command. How cool is that! After all, that’s what this whole thing has been about, high availability.</p><p>But wait, what happens if one of your running services goes down? What do you do then? Absolutely nothing. If a service goes down it will automatically be replaced. Watch what happens when I stop 1 of the containers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jogmNW53UXkXgdHFF35K4g.png" /></figure><p>Check out nginx-service_nginx.4. Let’s break this down. The service was stopped 6 seconds ago. A replacement was up and running less than 1 second ago. 5 seconds. That’s all it took. Wow! Also, notice how the node that particular replica was running on has changed too.</p><p>Docker will take care of everything, don’t you worry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*8BcsNXN9GQ8xprZ6X-_2Qg.png" /></figure><p>Ok, so we did a lot today. If you have made it this far give yourself a big pat on the back. You deserve it. As you can see there are huge and obvious benefits to using docker swarm. I hope you found this article informative and enjoyable and I hope you had as much fun reading this article as I had making it.</p><p>Before we end things don’t forget to clean up using docker stack rm.</p><pre>docker stack rm &lt;stack name&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*xRvh-JmKp9U1SzI0-D0y2A.png" /></figure><p>See you next time!</p><p>Below is the secret gist of my stack file</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abf654adef5b14bd1bfb5993ce222327/href">https://medium.com/media/abf654adef5b14bd1bfb5993ce222327/href</a></iframe><h3>Stackademic</h3><p><em>Thank you for reading until the end. Before you go:</em></p><ul><li><em>Please consider </em><strong><em>clapping</em></strong><em> and </em><strong><em>following</em></strong><em> the writer! 👏</em></li><li><em>Follow us on </em><a href="https://twitter.com/stackademichq"><strong><em>Twitter(X)</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/stackademic"><strong><em>LinkedIn</em></strong></a><em>, and </em><a href="https://www.youtube.com/c/stackademic"><strong><em>YouTube</em></strong></a><strong><em>.</em></strong></li><li><em>Visit </em><a href="http://stackademic.com/"><strong><em>Stackademic.com</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfa39b404670" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/learn-how-to-make-your-applications-highly-available-with-docker-swarm-bfa39b404670">Learn How to Make Your Applications Highly Available with Docker Swarm</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Jenkins container with persistent data using Docker Compose]]></title>
            <link>https://medium.com/@tom.puntillo/creating-a-jenkins-container-with-persistent-data-using-docker-compose-2a4a06c25793?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a4a06c25793</guid>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[yaml]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker-volume]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 09 Jul 2023 05:48:16 GMT</pubDate>
            <atom:updated>2023-09-12T18:06:55.679Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7d0ewjUDX_ZATq_hQse0g.png" /></figure><p>Hello there! I’m so glad you decided to read this article because today we’ll be discussing a tool that will change your life (assuming you’re creating containers all day lol). Today I will be giving a brief overview and tutorial of Docker Compose, an extremely popular tool used by DevOps and Cloud Engineers everywhere to simplify and automate the deployment of their containers. For our example we’re going to build a Jenkins container. As always, before we begin, I’ll be defining some of the terms used in this article.</p><p><strong><em>Docker Compose -</em></strong></p><p>Docker compose is a tool used to define and execute the deployment of a multi-container environment. It takes its instructions from the docker compose file. Written in YAML, the docker compose file lays out steps to build and run the containers and any attached networks or volumes. The real kicker is that once you have your docker compose file written, you can bring up or tear down entire environments using only a single command. Capable of handling many containers at once, it is the perfect tool to create, manage and tear down your microservice applications.</p><p><strong><em>Docker Volume -</em></strong></p><p>Simply put, a volume in docker is where you would store any data you wish to re-use at a later time. By their nature, containers are what’s known as ephemeral. Meaning that they are stateless and any data saved to a running instance is lost once that container is stopped. The way around this is to attach a volume to your container. This will allow you to store your data outside of the container itself, allowing it to be re-used again and again by any subsequent containers. Your data “persists” through the container’s shutdown.</p><p><strong><em>Jenkins -</em></strong></p><p>Jenkins is a Continuous Integration (CI) tool used by developers to build and test their applications. However, we will not be using it in such a way. We’re only going to use it to confirm the data stored in our volume persisted.</p><p>That’s all we have this week for new terms. Now, we’ll move on to defining our objectives.</p><p><strong><em>Objectives -</em></strong></p><p><strong>Foundational:</strong></p><p>Using the CLI</p><ol><li>Pull the Jenkins LTS image from Docker Hub.</li><li>Create a Docker volume for Jenkins to persist its data.</li><li>Run the container with the volume and correct port mapped to the container.</li><li>Retrieve the Admin password.</li><li>Login and create a new user.</li><li>Launch a new Jenkins container using the same volume and verify that the data has persisted by logging in as the newly created user.</li></ol><p><strong><em>PREREQUISITES:</em></strong></p><p><strong><em>DOCKER HUB ACCOUNT — </em></strong>Docker hub is the registry we will be pulling our official image from. You will need to have an account. If you do not already have one, you can create one for free by navigating to <a href="https://hub.docker.com/signup">https://hub.docker.com/signup</a>.</p><p><strong><em>LINUX SERVER WITH DOCKER INSTALLED— </em></strong>I will be using an EC2 Instance with Ubuntu 22.04 installed for this tutorial. You will need a similar setup to follow along.</p><p><strong><em>IDE OF YOUR CHOICE (Optional) -</em></strong></p><p>I will be using Visual Studio Code in this tutorial but feel free to use whatever IDE you like best. Optionally, you could use a built-in text editor instead.</p><p>Ok, now it’s time to get our hands dirty!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ElLV1ZA0DjLTV8jhSqoPg.jpeg" /></figure><p><strong><em>Step 1:</em></strong></p><p>Head to DockerHub and search for Jenkins</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSmkonII2JEYkrvXUHky3A.png" /></figure><p>Notice how this image is deprecated. Better not use that one. Head to the suggested image called jenkins/jenkins.</p><p>Copy the pull command on the right side of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdsZF9wkRtG-ahgJ349vPg.png" /></figure><p>Paste it in your terminal. The output should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*kBZOa_LoKKY4ALLSRQT9Rg.png" /></figure><p>You can verify image is pulled with docker image ls.</p><pre>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*HVztPMLgNcgfv4o37J1nTw.png" /></figure><p>Step one complete!</p><p><strong><em>Step 2:</em></strong></p><p>Now we need to add a volume. Head to Docker docs and search for “Volumes’’. Scroll down to the create volume example and copy it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2VMe5Py6Q5qSaLfsN7-TkQ.png" /></figure><p>Paste it into the terminal and replace “my-vol” with the name you wish to call your volume.</p><pre>docker volume create &lt;volume-name&gt;</pre><p>Run the docker volume ls command to verify the volume has been created.</p><pre>docker volume ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*ldiKouZ8iFGhtM4HRpwFpQ.png" /></figure><p>Step 2 down! Let’s keep it moving.</p><p><strong><em>Step 3:</em></strong></p><p>We need to figure out the correct port number to use. To do that, head back over to the jenkins/jenkins DockerHub page and click on the link for the documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxM7mkO6bbagnpW_tBWOew.png" /></figure><p>This will take you to the README file for the image. Right near the top of the file, copy the docker run command that specifies a volume. This example already provides us with the mountpoint of the volume inside the container, as well as the port numbers we need to use. We simply need to replace the info where needed and omit the unnecessary options. Change the name of the volume to whatever you called yours. Remove the tags from the image name at the end of the command. You can also remove -p 50000. This port is only needed when you are connecting agents, which we do not have to worry about for this project. The “-v” option is how we are telling docker to mount the volume to the container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WJ9O-3KxEymOcf7AssJhRg.png" /></figure><p>The cleaned up run command should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*qR5XotGf_fv6TAUibGh-bw.png" /></figure><pre>docker run -d -v week-17-vol:/var/jenkins_home -p 8080:8080 jenkins/jenkins</pre><p>Run the command and then run docker ps to verify the container is running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XP8TA8Z_9QOGXAar785sdQ.png" /></figure><p>Step 3 done! Keep knocking them out!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GHvlFIY6r2Q35R-8ZNywfg.jpeg" /></figure><p><strong><em>Step 4:</em></strong></p><p>Next we need to retrieve the admin password. We can easily do that by running the docker logs command and specifying the new container. You only need to type the first few characters of the Container ID into the command.</p><pre>docker logs &lt;container-id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BekUxnRm0fiGiBkC6-Gm-w.png" /></figure><p>Near the bottom of the output you will see the initial admin password displayed in plain-text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yineh-myw_7Or3eBe2O0dQ.png" /></figure><p>Copy that password and save it somewhere safe. We’ll need it in just a moment when we sign in.</p><p>Step 4 complete!</p><p><strong><em>Step 5:</em></strong></p><p>To login we’ll use our browser. More than likely in the search bar you can enter “localhost:8080”, but since I am working on an EC2 Instance, I’ll copy my public ip address and enter that into the search bar followed by “:8080”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0W9aUVUSYfVyMYro2tZZZQ.png" /></figure><p>Paste the initial admin password that you copied into the “Administrator password” field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UoGhEoHZ0Nxk_vgr78l9FA.png" /></figure><p>Now you’ll need to go through the initial setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4E79cYFzn6S8GjZv0X1lMw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*50XwszvWH-YtU2ojfdorJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WbrkoVDHbqg8dwu3CbUmBQ.png" /></figure><p>Once at the Dashboard, click on “Manage Jenkins”. Then click on “Users”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O_autEAU3TNB5l920oEzFQ.png" /></figure><p>Click on “Create User”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ddGYaalFVfISxsHEx4ba9w.png" /></figure><p>Create a new user. Be sure to save the login credentials because we will be using those in just a second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bu_CN35WZi_PqsxGdcKLEg.png" /></figure><p>Step 5 is finished! You’re nearly there!</p><p><strong><em>Step 6:</em></strong></p><p>First, let’s stop our running container. To do that we’ll use the docker container stop command.</p><pre>docker container stop &lt;container-id&gt;</pre><p>Run docker ps to verify the container is no longer running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A_d1njH7jah8rHM2nePA3g.png" /></figure><p>To test our volume and ensure that our data has persisted, we’ll create a new container using the same volume and try logging in as the newly created user. To do that, press the up arrow on the keyboard a few times until you are at the docker run command. All we are going to do is add a name to the command to differentiate between the 2 containers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V6E2H5Xb_Y8sxHj9xssfRw.png" /></figure><p>In the browser, refresh the tab for “localhost:8080” (or in my case, “public-ip:8080”). This will take you to the Jenkins login page. Login with the new user credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIKOrsVjo7Ifk5-eENdCIA.png" /></figure><p>As you can see, we are logged into the new user account, confirming our data has persisted due to the volume we created. Easy peezy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b4YfBuzkenHb9Jgt59UnPw.png" /></figure><p>That’s it for the foundational portion. Let’s move onto something a bit more advanced.</p><p><strong>ADVANCED<em>:</em></strong></p><p><strong><em>OBJECTIVES:</em></strong></p><ol><li>Instead use Docker Compose to do the steps in the Foundational.</li></ol><p>So now that we understand how to pull images and turn them into containers while simultaneously creating and attaching a volume, let’s take it one step further and do all that again, only this time we’ll utilize Docker Compose.</p><p>First we’ll run a docker system prune command to remove all unused containers, images, networks and we can also specify volumes.</p><pre>docker system prune --volumes </pre><p>Now let’s create a docker compose file and fill it with all the info we’ll need to get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7zZQeziU6vq94OTzRGG5Q.jpeg" /><figcaption>Composing. Get it? lol.</figcaption></figure><p>As you can see, the docker compose file specifies the same image, port numbers and volume as the docker run command we ran earlier. The main difference to point out is in the “services” block. Do you see where it says “new-jenkins-container”? That is the name we will be giving to the container when we run it. Inside the docker compose file, when you see “services”, think “container name”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*rzzIBC2sWQJub3hEC3mEAw.png" /></figure><p>Save the file.</p><p>Now we’ll launch the build and run process and expose the correct port numbers and attach the correct volume. We’ll do this all with a single command, “docker compose up”</p><pre>docker compose up</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6K32WKJMmsxAQTg_niocEg.png" /></figure><p>I restarted my EC2 instance while writing this article, so I need to go back and grab my public ip address again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*axqv97wmb2NY2zciSlbsdw.png" /></figure><p>Now we’ll go through the Jenkins setup one more time and create a new_user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uEHRIdl7PU0ir4DARM7Hg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuskb9uSHFBUWBkqDTS99A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDFpEY995u1_JskIwdqLkg.png" /></figure><p>Back in the terminal, stop the container with ctrl+c and then type “docker compose down”.</p><pre>docker compose down</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xMEFS0CN4WG5BnAYcJlZAA.png" /></figure><p>Now we’ll create a 2nd docker compose file. Copy the contents of the original docker compose file and paste them into the 2nd docker compose file. Mine is called docker-compose-new_user.yml.</p><p>Update the services block to change the name of the container to “new_user-jenkins-container”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TW_We7_m1hqzX1utt1KiTw.png" /></figure><p>Save the file. This time when we run the docker compose up command, we’ll utilize the “-f” flag to specify which file to use as the docker compose file. We’ll point the flag at docker-compose-new_user.yml.</p><pre>docker compose -f docker-compose-new_user.yml up</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPH7ZlhfUvXOVVzVJ2SDdQ.png" /></figure><p>And just as before, we’re taken to the Jenkins login screen. Try logging in with the “new_user” credentials you made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y5uhM7aGkpSTCVXtHxqT6Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JfuIzMPyEn65sxUhYA--Nw.png" /></figure><p>Success! The fact that you can log in as new_user shows us our data persisted in the volume, even when attached to a different container.</p><p>And that’s all there is to it! If you made it this far give yourself a big pat on the back. I hope this tutorial demonstrates the increased efficiency and control made possible by docker compose. Before we end though we need to clean up. Time to tear it all down again! After stopping the container with ctrl+c run the docker compose down command on last time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*fMRsoN_3go7B2vBnETplDQ.png" /></figure><p>I hope you had as much fun reading this article as I had making it. Until next time… see ya!</p><p><strong><em>GitHub Gists:</em></strong></p><p>Below are the secret gists to my 2 docker compose files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53023f290b03ec8578fd943048ec6ea0/href">https://medium.com/media/53023f290b03ec8578fd943048ec6ea0/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/76eeb5e0500fab3b6e291e70ce28ca82/href">https://medium.com/media/76eeb5e0500fab3b6e291e70ce28ca82/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a4a06c25793" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All hail the whale! Running containers and pushing new images to Docker Hub.]]></title>
            <link>https://medium.com/@tom.puntillo/all-hail-the-whale-running-containers-and-pushing-new-images-to-docker-hub-e428a1e6937c?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/e428a1e6937c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[dockerhub]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 02 Jul 2023 04:32:16 GMT</pubDate>
            <atom:updated>2023-07-02T04:32:16.369Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e2VJIbbuhQyz4nE_fC2P5Q.png" /></figure><p>That’s right! This week we’re diving head first into Docker and not looking back! Docker is more popular than ever and companies all over the world are looking for engineers with the skills to build and run their applications. In this article I will show you how to run an Ubuntu container running an Apache server with a custom webpage. Come watch how easy it is to get started with containers. I will be working off of an Ubuntu 22.04 server to complete these steps</p><p>But before we get started, let’s get some key terms defined.</p><p><strong><em>TERMINOLOGY:</em></strong></p><p><strong><em>IMAGE — </em></strong>Essentially, docker images are the files of an application needed for a container runtime (source code, binaries, meta-data, etc…). They are the basis for containers.</p><p><strong><em>CONTAINER — </em></strong>In the simplest terms, a container is an isolated process running an application on a server. They are running instances of images. Since all the cpu and memory resources are present within the runtime, this prevents containers from competing for resources.</p><p><strong><em>DOCKER — </em></strong>Docker is the world’s leading containerization platform. It is split into 3 basic components. The Client, The Docker_Host and the Registry.</p><p><strong><em>DOCKER OFFICIAL IMAGE — </em></strong>An official docker image is an image that has been built by the engineers at Docker. It can be identified by the “Docker Official Image” ribbon. These images can be trusted as stable.</p><p>That’s all we have for terminology this week. Now let’s discuss the prerequisites necessary to follow along with this article</p><p><strong><em>PREREQUISITES:</em></strong></p><p><strong><em>DOCKER HUB ACCOUNT — </em></strong>Docker hub is the registry we will be pulling our official image from. You will need to have an account. If you do not already have one, you can create one for free by navigating to <a href="https://hub.docker.com/signup">https://hub.docker.com/signup</a>.</p><p><strong><em>LINUX SERVER — </em></strong>I will be using an EC2 Instance with Ubuntu 22.04 installed for this tutorial. You will need a similar setup to follow along.</p><p><strong><em>OBJECTIVES:</em></strong></p><p>1.Run a Docker Ubuntu container running detached and on port 80</p><p>2. Using BASH shell update all packages within the container .</p><p>3. Using BASH install Apache2 on the Ubuntu container.</p><p>4. Add a custom webpage.</p><p>5. Verify you can reach the webserver from your browser.</p><p>Let’s dive right in!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*92YRsbjmZBPfI752cgi9Yg.jpeg" /></figure><p><strong>Step 1:</strong></p><p>The first thing we’ll need to do is install docker on our machine. To do that ssh into your linux server and type in the following command:</p><pre>curl -fsSL https://get.docker.com -o get-docker.sh<br>sudo sh get-docker.sh</pre><p>This will download and install the latest version of the docker onto your system. The output should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*_IfraJ1IwpGA_Kz4rw8rFA.png" /></figure><p>Now that we have docker installed, we can head over to <a href="https://hub.docker.com/">https://hub.docker.com/</a> to search for the official docker image for Ubuntu. To find it, simply enter “Ubuntu” into the search bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wd3kvYpnWHMV-osBcpPGXA.png" /></figure><p>This will take you to a list of Ubuntu images. We are going to select the top option called simply, “ubuntu”, which is a Docker Official Image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*RcJmNMZRETd4h91OJ8znXQ.png" /></figure><p>To run the container in detached mode over port 80, run the following command in your terminal:</p><pre>docker container run -it -d -p 80:80 ubuntu</pre><p>The output should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*Z9JkjCPs26aw10_ZfFOD6A.png" /></figure><p>You can check the status with the docker ps command as well.</p><pre>docker ps</pre><p>Step 1 complete!!! Now on to step 2.</p><p><strong>Step 2:</strong></p><p>The next thing we need to do is to enter the container with a BASH shell and upgrade all the packages. We do that with the following command:</p><pre>docker exec -it &lt;CONTAINER ID&gt; bash</pre><p>Since this is an Ubuntu image, we’ll use the apt package manager to update all the packages. Enter the following command:</p><pre>apt update -y</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*-9NC_5ZPg51Rjl3UGoU5bA.png" /></figure><p>Step 2 is done!!!</p><p><strong>Step 3:</strong></p><p>Moving right along, the next thing we’re going to do is install Apache. Please enter the following command:</p><pre>apt install apache2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fIimIp6LfLJvA73hpSnGSQ.png" /></figure><p>Step 3 complete!!!</p><p><strong>Step 4:</strong></p><p>Step 4: The next thing we need to do is to add a custom webpage. Luckily for us, Apache comes with a default index.html page located at /var/www/html/index.html. Let’s open that file in a text editor and make some changes to it.</p><p>First, we’ll need to install nano.</p><pre>apt install nano</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vv6oF_8yBDP1OEZ4HdGesg.png" /></figure><p>Now, using nano, we’ll edit the index.html file to give a custom message. Type the following command:</p><pre>nano index.html</pre><p>Scroll down until you reach the paragraph section and enter your custom message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95pGkDrlmwLbJ_WFJNC2CQ.png" /></figure><p>Step 4 done!!! Only 1 more to go!!!</p><p><strong>Step 5:</strong></p><p>Before we can test this out in our browser we need to enable and start the apache2 service if it is not already running. To do that, enter the following commands:</p><pre>service apache2 status<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*gAThDw0rGewLxPOickCPgw.png" /></figure><pre>service apache2 start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4tJxrXvQkhHsIK4UIVmuwg.png" /></figure><p>The final step in our objectives is to test if the Apache server is available from a browser. We’ll need to grab the host machine’s public ip address. I am running mine on an EC2 instance, so I’ll just copy mine from the AWS Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/178/1*fR_xjH1kQiqTZ3ViR-yN4A.png" /></figure><p>And now for the moment of truth…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBfxd04Hy8lcqK3QwNayqQ.png" /></figure><p>We did it!!!</p><p>So there you have it. I knew we could get through it.</p><p><strong><em>ADVANCED:</em></strong></p><p>Now that we’ve verified that we can access our custom webpage through the browser, let’s go ahead and turn this into a new image!</p><p><strong><em>ADVANCED OBJECTIVES:</em></strong></p><p>1. Push to Docker Hub.</p><p>2. Pull your new image and run on a different port than 80 while still mapping to 80 on the container. Verify you can reach the webserver using the updated port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABnbtkTf6yY6ey77xD0k6g.png" /></figure><p><strong>Step 1:</strong></p><p>Before we can push anything we first need to create a new image from our changes. To do that we use the docker commit command and provide a new name. If you’re still inside the container with a BASH shell, type “exit” to exit the container and get back to your terminal. Type the following command:</p><pre>docker container commit &lt;Container ID&gt; &lt;New_Image_Name&gt;</pre><p>You should receive a sha256 hash as output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*W0TFfdTfkTZC1qvoCZO1VQ.png" /></figure><p>You can verify the new image has been created by listing the images.</p><pre>docker image ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*jIfR5jImyWuzDGYl05RNIg.png" /></figure><p>Next, we’ll need to go back to Docker Hub and create a repository to push to. At your Docker Hub homepage, click the button at the top of the page labeled “Repositories”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zqzc6Xfl41ddSoES-g5PQ.png" /></figure><p>Click on “Create repository”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wz2vA4ZaDcz6Q_gQBGnVoQ.png" /></figure><p>Leave the namespace as defaulted to your username and provide a name for the new repository. Make sure the visibility is set to “public”. Click “Create”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqnysKbQh4BLhfVkIxuHcQ.png" /></figure><p>The next page displays the newly created repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5G26Zx_szttJt3oa1ayaQ.png" /></figure><p>Back in the terminal, list the docker images so the newly created image’s details are displayed on screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*_cFolhnuWIZ10XfBWo_BXg.png" /></figure><p>Next we need to tag our image. To do that we’ll use the docker tag command.</p><pre>docker tag [Source_Repo]:[Tag] [Target_Repo]:[Tag]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*gPt9TBTN9WmcM2GX2RwhAg.png" /></figure><p>List the containers again to see the tag. Also, notice how the repositories for the image we created as well as the one we tagged share the same Image ID.</p><p>Next use the following command to login to Docker Hub. You will be prompted for your username and password. (Note: since this is a linux instance, when you type your password, it will look like nothing is happening. Your password is being typed). If successful, you will see a “Login Succeeded” message.</p><pre>docker login docker.io</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3MjUdFR3_br5-kpWfjIWzw.png" /></figure><p>Go back to the repo you created on docker hub and copy the docker push command into your clipboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fsOOFv1QkMXhcfJ-XFLuHg.png" /></figure><p>Paste the command into your terminal, replacing “tagname” with whatever tag you gave the new image. Your output should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*9UDXFX3K_hD3FhWPW6G-Sw.png" /></figure><p>Next, go back to your new repository in Docker Hub and refresh the page. You should now see the image you just pushed under “Tags”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72nmeFHATVJKeXvhaPS5IA.png" /></figure><p>That’s it for step 1!!</p><p><strong>Step 2:</strong></p><p>Now, let’s try pulling this new image, specifying a different port other than 80.</p><p>To do that, we’ll run the same docker run command we used earlier, only this time, we’ll substitute the port number and image name to suit our needs.</p><pre>docker container run -it -d -p 8080:80 tompuntillo/week-16-repo:latest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*lBXuF83MMi6jeBrMaW9oaw.png" /></figure><p>Run another docker ps command to verify the container is up and running.</p><pre>docker ps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HZQTHquaOCRka64Jr7KX_A.png" /></figure><p>Just like last time, we’ll enter the container in a bash terminal to start the Apache service.</p><pre>docker exec -it ef8c617b7c08 bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UPaLxYnoIppQGFsoMmb6w.png" /></figure><p>I restarted the EC2 instance my containers are running on during the course of writing this article, so I’ll go back to the AWS console and copy the new public ip address for this instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wou2Wa5HAT1Tbl4DARF3w.png" /></figure><p>Ok, let’s try putting the public ip into a browser and add port 8080 to the end of the address.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_FEQbMQuGAdWzZB8ZqONdQ.png" /></figure><p>Hmmm. That didn’t work. The Apache server took too long to respond when trying to use port 8080. Let me go check the inbound rules for the security group my EC2 instance belongs to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmCAOkOFFjiRYoREVY9zIA.png" /></figure><p>Just as I suspected. We’re only allowing inbound HTTP traffic over port 80. We’ll need to edit our rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WLuCwCySz3ntk6krd8VUxQ.png" /></figure><p>Ok, let’s try that again…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aP3z9imfTWN9-tJtMpSJWA.png" /></figure><p>It works! Let’s celebrate!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o9o54Kt8k9-MjZMlbHAy7Q.jpeg" /></figure><p>And that’s all there is to it! We now have 2 separate containers, listening on 2 separate ports and both serving up the same custom webpage.</p><p>Hopefully this article illustrates to you how quickly you can get applications up and running when utilizing Docker. I hope you found this engaging and informative and I hope you had as much fun reading it as I had making it. See you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e428a1e6937c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automate Sending Messages to Your SQS Queue Using Boto3, Lambda, and Amazon API Gateway]]></title>
            <link>https://medium.com/@tom.puntillo/automate-sending-messages-to-your-sqs-queue-using-boto3-lambda-and-amazon-api-gateway-57641dd7bb1d?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/57641dd7bb1d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[sqs]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 25 Jun 2023 05:21:32 GMT</pubDate>
            <atom:updated>2023-09-07T14:36:49.632Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*tqyguZdV20i-ESNtBlMr1Q.png" /></figure><p>Hello! In this week’s project we tackle some automation. Using boto3, we’ll create an SQS queue and then we’ll create a lambda function and set up an HTTP API using Amazon API Gateway. Before we begin however, let’s go over some terminology we’ll be using</p><p><strong><em>TERMINOLOGY:</em></strong></p><p><strong><em>SQS — </em></strong>Simple Queue Service is an AWS service that allows you to build a queue to store your applications’ important messages. The queue can be polled for messages manually or at set durations.</p><p><strong><em>LAMBDA </em></strong>— Lambda is an AWS computer service that operates on a pay as you go model. Lambda functions can be written in most modern programming languages and is a common tool used in automation as it is highly configurable and can interact with many other AWS services.</p><p><strong><em>HTTP API — </em></strong>an HTTP API is an API endpoint that we will expose to our customers. API’s handle the exchange of information between services. Once integrated with the lambda function, any time the Invoke URL is hit, the lambda function is automatically triggered.</p><p>That’s all the contextual information you’ll need to understand to follow along with this project.</p><p>Let’s take a look at this week’s objectives</p><p><strong><em>SCENARIO:</em></strong></p><p>A company wants to create a system that can track customer orders and send notifications when orders have been shipped. They want to use AWS services to build the system. They have decided to use SQS, Lambda, and Python for the project.</p><p>All code should be inline commented.</p><ol><li>Create a Standard SQS Queue using Python: This involves using the Boto3 library (AWS SDK for Python) to programmatically create an SQS queue.</li><li>Create a Lambda function in the console with a Python 3.7 or higher runtime: In the AWS Lambda console, create a new function and configure it to use Python 3.7 or a higher version.</li><li>Modify the Lambda to send a message to the SQS queue: Update the Lambda function code to send a message to the SQS queue. You can use the current time or a random number as the message content. Make sure to use the Boto3 library to interact with SQS.</li><li>Create an API Gateway HTTP API type trigger: Set up an HTTP API trigger for the Lambda function in the API Gateway console. This allows you to invoke the Lambda function through an API endpoint.</li><li>Test the trigger to verify the message was sent: Use a tool like curl or a web browser to send a request to the API endpoint and verify that the Lambda function sends a message to the SQS queue.</li><li>Push your code to GitHub and include the link in your write-up: Share the code you wrote for the Lambda function, API Gateway configuration, and any other relevant files by hosting them on GitHub. Include the GitHub link in your write-up or documentation.</li></ol><p>Below is a link to the video tutorial. Please join me as I show you how to complete the outlined objectives:</p><p><a href="https://www.youtube.com/watch?v=RgH7eHA-SZI">https://www.youtube.com/watch?v=RgH7eHA-SZI</a></p><p><strong><em>GITHUB GISTS</em></strong></p><p>As specified in the requirements above, here are the secret gists of the code I wrote to complete the objectives in this project:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81f928bb8ca1fff19863dc79497fa8f3/href">https://medium.com/media/81f928bb8ca1fff19863dc79497fa8f3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bd22972ac21b63254b7e97ad64130cb/href">https://medium.com/media/9bd22972ac21b63254b7e97ad64130cb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57641dd7bb1d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently Halting EC2 Instances with Boto3 and Python]]></title>
            <link>https://aws.plainenglish.io/stopping-your-running-ec2-instances-in-your-dev-environment-using-boto3-30575605c9d9?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/30575605c9d9</guid>
            <category><![CDATA[scripting]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 18 Jun 2023 17:30:38 GMT</pubDate>
            <atom:updated>2023-10-10T04:21:22.443Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdecrhaX5XNR-cdoQUPATw.jpeg" /></figure><p>Hello! This week I’ll cover how you can stop your running ec2 instances directly from python using boto3.</p><p><strong><em>What is boto3?</em></strong></p><p>Boto3 is the python module that allows you to work programmatically with your aws environment. Today, we’ll be working from an Cloud9 instance, which, while an aws service, does not come pre-installed with boto3. That being the case I will cover how to add it to your Cloud9 environment. I will not cover how to create a Cloud9 environment or how to configure it to use your GitHub account as I have covered that in a previous article which I will link here:</p><p><a href="https://medium.com/@tom.puntillo/generating-a-list-of-dictionaries-in-python-is-easier-than-you-think-fb44f931a0f9">https://medium.com/@tom.puntillo/generating-a-list-of-dictionaries-in-python-is-easier-than-you-think-fb44f931a0f9</a></p><p>Let’s take a look at this week’s scenario.</p><p><strong><em>SCENARIO:</em></strong></p><p>Scenario: Our DevOps engineering team often uses a development lab to test releases of our application. The Managers are complaining about the rising cost of our development lab and need to save money by stopping our (for this example) 3 ec2 instances after all engineers are clocked out.</p><p>Create a Python script that you can run that will stop all instances. Push your code to GitHub and include the link in your write up.</p><p>Ok, let’s get started!</p><p>First thing, we’ll need to install the boto3 module into your Cloud9 environment. To do that, we’ll follow the steps listed in the boto3 documentation, linked here:</p><p><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html">https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html</a></p><p>In your Cloud9 terminal, type:</p><pre>sudo pip install boto3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WY85bLq2vqbvcf35G4VzJQ.png" /></figure><p>After boto3 successfully gets installed, it needs to be configured with your aws account credentials. You’ll need to provide the access key and a secret access key of a profile that has sufficient right to perform the required actions. If you do not already have those access credentials provisioned, you can do so by following the steps provided in the linked aws documentation:</p><p><a href="https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html">https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html</a></p><p>In the terminal, type the following command:</p><pre>aws configure</pre><p>Next, provide the required credentials:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*LRkmrD_faqg7gXAEmH-9-Q.png" /></figure><p>Optionally, you can select a default region.</p><p>Now we’re ready to get coding! The rest of the example will proceed under the assumption you have already cloned your desired repository to the environment. Before we can begin we need to open a new python file and create a new branch to work from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*LGcwTuawx4eqy2b7xeS0Ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*vo6VL8EH2DKP7BX4YOETpg.png" /></figure><p>We’ll begin our code by importing the boto3 module we installed. We will also take this time to set our boto3.client variable to ‘ec2’.</p><pre>import boto3<br><br>ec2 = boto3.client(&#39;ec2&#39;)</pre><p>Our scenario objective tells us we need to shut down all running instances. If you have several ec2 instances already provisioned now is a great time to get them started so we can demonstrate the code later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J23RVzx2dhTh8kC1QTAz4Q.png" /></figure><p>Before we can know which instances need to be turned off, we need to know which instances are running. Hit “cmd + f” to open a search bar and look up “describe_i”. The bottom option will be “describe_instances”. We can open that up to see how to get the info we need. As you can see there is quite a lot of info we can get back. For now though, let’s just copy the example script and modify it to our needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*BzwJ9DfD-KMExg465GwOJw.png" /></figure><pre>import boto3<br><br>ec2 = boto3.client(&#39;ec2&#39;)<br><br>response = ec2.describe_instances()<br><br>print(response)</pre><p>As you can see, we erased all the content between ‘()’ and changed the “client” variable to ‘ec2’. Let’s save our file and run it to see what we get back. Be sure to save the file in the correct location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*eQO6YGIB7pqUTrqq80JlkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*utVy1_ZKcSLFUszXDrHQLg.png" /></figure><p>This response has returned a large amount of data which is difficult to read in its current format. We can copy the output and paste it into a json formatter to clean it up and make it easier to read. Paste it in and hit “process”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIJEDKGCPy1iSfVH47OcKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JBo2Pe0jod2PMVIMz7JdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oFvOlEsuc5K9VfwxcqYBsQ.png" /></figure><p>Now that the output has been reformatted into a proper json output it is much easier to read. As you can see now, the response returned a list of dictionaries called “Reservations”, which in turn contains a list of dictionaries called “Instances”. The “Instances” dictionary is what we will iterate to find the information we need. We will need to parse the data for “InstanceId” and “State”and “Name”.</p><p>If we use nested “for” loops and “if” statements, we can print out a list of all the current running instances in the AWS account.</p><pre>for reservation in response[&quot;Reservations&quot;]:<br>        for instance in reservation[&quot;Instances&quot;]:<br>            if instance[&quot;State&quot;][&quot;Name&quot;] == &#39;running&#39;:<br>               print(instance[&quot;InstanceId&quot;], instance[&quot;State&quot;][&quot;Name&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*5g1Kj1F09OKwB4uv2JFfWw.png" /></figure><p>We can use this parsed data to target the instances we want to stop. Let’s go back to the documentation now and do a search for “stop”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FNRLULA-dv9yOwspBkXW9g.png" /></figure><p>If we do a search for “required”, we see that this function has a required parameter named “InstanceIds”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87bmjbhhYcCYTZskBJ3zug.png" /></figure><p>Let’s scroll to the bottom of the document and copy the example again. We’ll paste it at the top of our script and define it as a function to be used in our code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*pduNrlT77A2gbcylOoelug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*DeqmgGOW2Ogyi1CRHPhHbg.png" /></figure><p>By the looks of it, we can simply plug in the function at the bottom of the nested “for” loop and “if” statement, replacing the “print” statement. Doing this allows us to use the “InstanceId” argument to parse the instance identity data to fulfill the functions requirement.</p><pre>for reservation in response[&quot;Reservations&quot;]:<br>        for instance in reservation[&quot;Instances&quot;]:<br>            if instance[&quot;State&quot;][&quot;Name&quot;] == &#39;running&#39;:<br>                    stop_instances(ec2, instance[&quot;InstanceId&quot;])</pre><p>But before we do that we better enter another variable to assign to the Cloud9 instance and set it to the variable “ ide_id”. Remember, Cloud9 environments run on ec2 instances and we don’t want ours shutting down when we run the command.</p><pre>ide_id = &#39;your-instance-id-here&#39;</pre><p>We’ll plug that in outside of our for loop.</p><p>Now we can plug in another “if” statement right above the “stop” statement, preventing our instance running the Cloud9 environment from shutting down when the script is run. With the stop function added, our code should look like this:</p><pre>import boto3<br><br>def stop_instances(client, instance_id):<br>    response = ec2.stop_instances(<br>        InstanceIds=[<br>            instance_id<br>            ],<br>        )<br><br>ec2 = boto3.client(&#39;ec2&#39;)<br><br>response = ec2.describe_instances()<br><br>ide_id = &#39;i-0cd54c8eee80de55c&#39;<br><br>for reservation in response[&quot;Reservations&quot;]:<br>        for instance in reservation[&quot;Instances&quot;]:<br>            if instance[&quot;State&quot;][&quot;Name&quot;] == &#39;running&#39;:<br>                if instance[&quot;InstanceId&quot;] != ide_id:<br>                    stop_instances(ec2, instance[&quot;InstanceId&quot;])</pre><p>Now it’s time to test! Let’s run our script and see what happens. I’m also renaming the file “stop_running_instances” so it is accurate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*hSlLOGfZhfRFl5nbMS-4aw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4BMEpaFkwBEOrVbxt72sjw.png" /></figure><p>It worked! That is exactly the result we wanted to see. Every running instance aside from our Cloud9 environment is stopping.</p><p>Now it’s time to push our code to GitHub. However, I won’t cover how to push code to GitHub as I’ve already covered it in the article I linked earlier. I will however provide a link to the GitHub Gist.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab2f8e6e0c53460920d5d152a5c0d7d3/href">https://medium.com/media/ab2f8e6e0c53460920d5d152a5c0d7d3/href</a></iframe><p>And that’s all there is to it! See, not so bad. You can now stop running ec2 instances in your environment with python using boto3.</p><p>We have completed the steps to satisfy the foundational requirements. Let’s take a look at what the advanced portion of the scenario objectives are.</p><p>ADVANCED</p><p>Scenario: We want to ensure that only our Development instances are stopped to make sure nothing in Production is accidentally stopped. Add logic to your script that only stops running instances that have the Environment: Dev tag.</p><p>Ok, we’ll need to add some tags to our existing instances to make this work. To Do so, we can head over to the ec2 dashboard in the aws console and add them there. First,select an existing instance and click on the “Actions” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJ-LfHErkb3gWOr9ugPAow.png" /></figure><p>Select “Instance settings” and “Manage tags”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5m7Ar81LIBiU1ueM0w5uzw.png" /></figure><p>Select “Add new tag” and enter your tags Key and Value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*Cgp76xwZVwmkFozXPH5RDA.png" /></figure><p>I have added the “Environment : Prod” tag to 3 of my instances and the “Environment : Dev” tag to 3 others. Now that we have our tags added, let’s run a “describe_instances()” function and process it in the json formatter to verify we can see our newly added tags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*X4ANxK0owOm649fBU5HMWA.png" /></figure><p>Ok, so now we are ready to begin. Let’s take another look at the foundational script we just wrote and see how we can add the tags to the iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*OK3w8F5c56uCMYdVTTPAQQ.png" /></figure><p>Luckily for us, adding the tags into our script will only require 2 extra lines of code. Inside the “kill_all” function I wrote, I added another “for” loop to iterate over “Tags” and below that, another “if” statement to check if the desired Key and Value exist inside “Tags”. The updated “kill_all” function should now read as such:</p><pre>def kill_all():<br>    for reservation in response[&quot;Reservations&quot;]:<br>        for instance in reservation[&quot;Instances&quot;]:<br>            if instance[&quot;State&quot;][&quot;Name&quot;] == &#39;running&#39;:<br>                for tag in instance[&quot;Tags&quot;]:<br>                    if tag[&quot;Key&quot;] == &#39;Environment&#39; and tag[&quot;Value&quot;] == &#39;Dev&#39;:<br>                        if instance[&quot;InstanceId&quot;] != instance_id:<br>                            stop_instances(ec2, instance[&quot;InstanceId&quot;])</pre><p>Now it’s time to test again! Let’s verify we have all the instances we added tags to in a “running” state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AjMJtGmazcjxE42ynSKNyA.png" /></figure><p>Let’s run our new script and see if it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SZl9WUCThInV6gkk9765Vg.png" /></figure><p>And it does! Excellenct! As you can see, only the running instances with the “Environment : Dev” tag have been stopped and the rest are still running. We have satisfied the requirements for the advanced portion of this project. Again, as in the foundational section, I will not cover how to push the code to GitHub. Please refer to my previous article if you need that information. Below is the link to the GitHub Gist of the advanced portion code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/482c39893e35fa165685957ad26a28c4/href">https://medium.com/media/482c39893e35fa165685957ad26a28c4/href</a></iframe><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30575605c9d9" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/stopping-your-running-ec2-instances-in-your-dev-environment-using-boto3-30575605c9d9">Efficiently Halting EC2 Instances with Boto3 and Python</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating a list of dictionaries in Python is easier than you think!]]></title>
            <link>https://python.plainenglish.io/generating-a-list-of-dictionaries-in-python-is-easier-than-you-think-fb44f931a0f9?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb44f931a0f9</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[pyrhon]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Sun, 11 Jun 2023 15:29:25 GMT</pubDate>
            <atom:updated>2023-10-11T15:26:29.475Z</atom:updated>
            <content:encoded><![CDATA[<h3>Generating a List of Dictionaries in Python is Easier than You Think</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I5cgzITT8-D6FD--UJKRMQ.jpeg" /></figure><p>Hello! Today we will be talking about Python lists and dictionaries and how they can be used to make your coding more structured and easier to use. As always we’ll begin with a high level overview of the concepts at play and some terminology.</p><p><strong><em>TERMINOLOGY:</em></strong></p><p><strong><em>LISTS — </em></strong>A list in Python is an unordered collection of data. Each element of a list is separated by a comma. Elements of a list do not necessarily need to be of the same data type, meaning you can mix strings, integers, floats and so on.</p><p><strong><em>DICTIONARIES — </em></strong>Dictionaries are a collection of key:value pairs that are separated by commas. Dictionaries can be used to store attributes of an object.</p><p><strong><em>CLOUD9 — </em></strong>Cloud9 is an AWS based IDE, which we will be using to write our code.</p><p><strong><em>FOR LOOP — </em></strong>A for loop in Python is a programming concept in which you can instruct your code to iterate over a specified object a specified number of times. By doing this we can assign or modify certain values to the contents of the object as the loop encounters them. This may sound confusing now but it will make much more sense shortly when you see it put to use in the coding example.</p><p>That’s enough terminology for now.</p><p>Now let’s take a look at the objective we will be trying to complete.</p><p><strong><em>SCENARIO:</em></strong></p><p><strong><em>FOUNDATIONAL:</em></strong></p><p>Create a script that generates a list of dictionaries about files in the working directory. Then print the list.</p><p>Push your code to GitHub and include the link in your write up.</p><p>Before we can get started, we’ll need to set up our Cloud9 environement. To do so, log into you AWS account and navigate to Cloud9 using the search bar. Once there, click on the “Create environment” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUXJdV0CSjkvHeAxIIXNCA.png" /></figure><p>Cloud 9 environments are hosted on EC2 instances. On the next page, provide your EC2 instance with a name. Keep all the defaults selected and click “Create”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtWIVeyFY3o6KUgyV5-dBg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtWIVeyFY3o6KUgyV5-dBg.png" /></figure><p>On the next page you will see a notification that your environment is being created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zu1hoKYivaIL-u_mVHeytw.png" /></figure><p>If we head over to our EC2 instances page, we can see a newly created instance with the name we provided for the Cloud9 environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19c-6-a3DuOr3y3MSIaScQ.png" /></figure><p>Back on our Cloud9 environments page, let’s click on “Open” to get into our IDE!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gHuowCZy4217jNzmKA2G0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kawtWPqxJ5ZK8f8upkKSuA.png" /></figure><p>Now that we have our Cloud9 environment up and running, we need to clone a repository to it to get working on our code. Let’s head over to GitHub and grab the URL of the repo you’d like to clone.</p><p>(NOTE: I already have a repository I’ve been working with but if you don’t you will need to initialize one before you can begin.)</p><p>My repository is called “blue-python-repo”. I’m going to highlight and copy the URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7yiLht6itR91Gz77bKap-w.png" /></figure><p>Back in the Cloud9 environment, click on the source control button and select “Clone Repository”. Enter the copied URL into the search bar and select the “clone from GitHub” prompt that pops up. Then select the default repository location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AE8t71_F5q2FpsxIU7_TpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jzdBFMGRr95ufdQysSqVXQ.png" /></figure><p>rom here, you will now see your newly cloned repo in the left pane of your Cloud9 environment. If you cloned an existing repo and did not initialize a new one, you can see all of your repos files if you click the dropdown arrow next to your repository name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DzxxAOL6BLne97Hxg0Ik3w.png" /></figure><p>Now that we have our Cloud9 environment setup and our repository cloned… let’s get coding!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PXlsa44LJxHGhv1QYRtsYA.jpeg" /></figure><p>The first thing we need to do is create a new branch. To do so, on the bottom left of the screen, click the branch icon that says “main”. This is signifying you are on the main branch of the local repository. Click “Create new branch”. Enter a branch name hit enter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYSd69Kc5BhyD8wuRQmKZw.png" /></figure><p>You can now see the name of the new branch in the bottom left of the screen. Additionally, in the bash terminal, if you cd into the cloned repo, you can also view which branch you are currently working on there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9fxRMlLNOjG2cYEZB8NChg.png" /></figure><p>On our new branch we want to create a new file to work with. To do so, click on File&gt;&gt;New from template&gt;&gt;Python File.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_aC1IYD0MwjDxbeKRMtcw.png" /></figure><p>This will open an untitled1.py file. From here, click on File&gt;&gt;Save as and provide a new name for the file. Be sure to leave the “.py” extension on the end of the filename. Confirm you have your repository selected as the save folder. Click “save”.Clear out the pre populated code and you’re ready to begin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCYExHvjkNnXEyYr0UCbQA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jWayjeD5-R6fShQqdrjsmg.png" /></figure><p>Our goal is to generate a list of dictionaries where each key:value pair represents an attribute of a given file. To start, we’ll need to import the os module. This will allow us to interact directly with the operating system of the EC2 instance and give us access to the files we need.</p><pre>import os</pre><p>Next, we need to create an empty list called file_list. This list will not remain empty for long.</p><pre>import os<br><br>file_list = []<br></pre><p>After that we initialize our for loop. This is where the logic of the script gets executed. Let’s take a look and break down what’s happening.</p><pre>import os<br><br>file_list = []<br><br>for name in os.listdir():<br>    size = os.stat(name).st_size<br>    <br>    file_dict = {<br>        &#39;Name&#39; : name,<br>        &#39;Size&#39; : size<br>    }<br>    <br>    file_list.append(file_dict)<br>    <br>print(file_list)</pre><p>The os.listdir() function returns a list of the files in the current working directory, very similarly to how the linux ls command functions. Here,we are iterating through the current working directory and setting the name variable to the name of the file being iterated. Inside the loop, we also set the size variable to equal the size of the iterated file in bytes. We accomplish this by using the os.stat(name).st_size function. As you can see we are calling on the name variable to select the file.</p><p>Further down in the loop we initialize a dictionary called file_dict. The dictionary has 2 key:value pairs. The first key, ‘Name’, gets passed the name value and the second key, ‘Size’, gets passed the size value.</p><p>Even further down in the loop, we append the newly created dictionary to the end of the list called file_list. This process gets repeated for every file in the current working directory until all files have been iterated.</p><p>After we break out of the for loop, we print file_list. We now have a list of dictionaries describing attributes of the files in our current working directory.</p><p>Now, go ahead and run your code and you should see the list of dictionaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ydZJ327TVtPRxv8l6hI3hw.png" /></figure><p>And there you have it! Simpler than you thought, wasn’t it? Well done on getting this far. But we’re not done yet. Writing the code is only half the battle. Next we need to push our code to the remote repo. To do that, go ahead and select the bash terminal in your Cloud9 environment. Write git push and hit enter.</p><p>This should give you a fatal error as there is no upstream branch on the remote repo for the branch you’ve been working on. That’s ok though because the suggested command to set the upstream branch gets returned in the output as well. Go ahead and copy the command and enter it back into the terminal. You will be prompted for a username and password. Enter your GitHub username and a valid personal access token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rVRW4pg1eyQXwyduguuUPg.png" /></figure><p>If you need a personal access token, log into your GitHub account. Click on the profile icon and select settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-oX7UNjmnXKjCyzOdko5g.png" /></figure><p>In the left hand pane click on developer settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UFKU8FP67pBJ9K_Q7i6-A.png" /></figure><p>Click on “Personal Access Tokens” &gt;&gt; “Tokens Classic”. Click “Generate New Token”&gt;&gt; “Generate New Token Classic”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXxVTF5z25bfbH5DNhkJpg.png" /></figure><p>Enter your password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*KyxYHezwAzRhKC1L31NzPQ.png" /></figure><p>Provide a note and expiration date. Set the scope to “repo”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rjGOD-ePs9mt_d7OUkjiBw.png" /></figure><p>Click “Generate Token”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1r2d4A9NstkRN_awurxWA.png" /></figure><p>This is the only time you’ll be able to copy down your token so be sure to do so and store it somewhere safe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*juii2rLHSsruog21MkV6mA.png" /></figure><p>Remember, since this is a linux system, when you paste the token into the terminal, it will look blank but it will be in there. Hit enter. You should see something similar to the following output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lzHrbYUW-B6dKKmefN-g6w.png" /></figure><p>Click on the source control button and click the + sign. The + will turn into -. Type a message in the message field and commit the changes by pressing “Ctrl + enter”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*Ui_oMm9k6WKD-4XeziFABw.png" /></figure><p>Oops! It looks like we forgot to configure our username and email in git. Let’s do that now! In the bash terminal, enter the following:</p><pre> git config --global user.name &quot;USERNAME&quot;<br> git config --global user.email &quot;user@email.com&quot;</pre><p>Substitute your own username and email address.</p><p>Now let’s try that again. Click on the message field and hit “ Ctrl + enter”. You should see the files disappear. Next click on the cloud icon at the bottom left of the screen. If prompted, enter your GitHub username and the personal access token again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWDU36ewR_YHynweO6SPow.png" /></figure><p>Next, back to GitHub, go to your repository. Click on the big green “Compare &amp; pull request” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30bj5hwJZsHMCQloaf5yMw.png" /></figure><p>Click “Create pull request”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E2j1pEo3a1-6GRm93L39Vw.png" /></figure><p>Self approve and click on “Merge pull request” amd “Confirm merge”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-65Ls49wpANgRMy2De9qhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*b5kcwOMOgOt-M1hVz0oFgA.png" /></figure><p>Click on “Delete branch”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*73DNkTKsX0VN70R9jXt7kQ.png" /></figure><p>Back on Cloud9, click on the branch icon in the bottom left of the screen and then select “main” to switch back to the main branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x1b6rVJ4P9QtEAqyhh09fw.png" /></figure><p>Then click on the refresh icon on the bottom to update against the remote branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*F9F3N_hqwzxklVrUy7u6dg.png" /></figure><p>Then click on the refresh icon on the bottom to update against the remote branch.</p><p>And there you have it! We’ve completed our objective. We created a list of dictionaries describing file attributes in the current working directory and we’ve pushed our code back up to GitHub. All in a day’s work!</p><p>Here is the final version of the code written for this demo.</p><p><a href="https://gist.github.com/tom-puntillo/46b80a02282b1bbe47eea9c0f04b6980">https://gist.github.com/tom-puntillo/46b80a02282b1bbe47eea9c0f04b6980</a></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb44f931a0f9" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/generating-a-list-of-dictionaries-in-python-is-easier-than-you-think-fb44f931a0f9">Generating a list of dictionaries in Python is easier than you think!</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make your life easier by utilizing a 3-tier architecture for your web applications.]]></title>
            <link>https://medium.com/@tom.puntillo/make-your-life-easier-by-utilizing-a-3-tier-architecture-for-your-web-applications-52669dc2f2e8?source=rss-154d0cadceae------2</link>
            <guid isPermaLink="false">https://medium.com/p/52669dc2f2e8</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[application-load-balancer]]></category>
            <category><![CDATA[auto-scaling-groups]]></category>
            <category><![CDATA[sms]]></category>
            <category><![CDATA[3-tier-architecture]]></category>
            <dc:creator><![CDATA[Thomas Puntillo]]></dc:creator>
            <pubDate>Mon, 15 May 2023 15:41:13 GMT</pubDate>
            <atom:updated>2023-05-15T15:41:13.356Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eoN9MbqWMubBzoDPXg25xA.png" /></figure><p>Have you ever wondered how your favorite web applications provide you with their services? How Netflix serves up your favorite movies and shows or how Amazon brings an item from your computer screen to your front door? The answer is simple (not really). Cloud Engineers are taking advantage of what’s known as a 3-Tier Architecture. In essence it’s the practice of logically separating the different parts of your web apps into 3 distinct tiers. They do this in order to improve performance, lower costs and make the application easier to update and maintain. By decoupling the different components of the application, they ensure that if one part were to fail, the other parts would remain unaffected. This in turn allows for high availability.</p><p><strong><em>The Web Tier — </em></strong>Also known as the “Presentation Layer”, this is where the customer will interact with the application. This is where you would queue up a movie or put an item in your cart.</p><p><strong><em>The Application Tier — </em></strong>Next up we have the application layer. This is where all of the business logic of your app gets processed. Think things like payment transactions for an online purchase.</p><p><strong><em>The Database Tier — </em></strong>Lastly we have the database tier. This is where all of the application data is stored.</p><p>See… simple. At least the concept is simple to understand. To follow along with this article though, we’ll need to cover some additional terminology.</p><p><strong>Terminology:</strong></p><p><strong><em>VPC — </em></strong>Virtual Private Cloud is your own little slice of the aws cloud platform. This is the environment you will build your applications in.</p><p><strong><em>Subnet — </em></strong>A subnet is a smaller portion of the VPC’s available IP addresses which have been logically segmented from the rest of the IP addresses. A subnet can be either private or public</p><p><strong><em>Route Table — </em></strong>A route table needs to be explicitly associated with one or more subnets. Once associated, the route table is responsible for defining the flow of outgoing traffic.</p><p><strong><em>Internet Gateway — </em></strong>This is your connection to the public internet. IGW’s need to be attached to a VPC. Once they are, you can use it as a target for a route table for any traffic destined for the internet.</p><p><strong><em>NAT Gateway — </em></strong>A NAT Gateway provides public internet access to a private subnet. The way this is done is by placing it in a public subnet and then adding it to a route table associated with a private subnet. NAT Gateways are used for outbound traffic, which is why they are added to route tables.</p><p><strong><em>AWS Systems Manager — </em></strong>AWS Systems Manager (SSM) is a tool you can use to securely connect to your AWS resources.</p><p><strong><em>Elastic IP — </em></strong>An Elastic IP address is a public IP address that gets allocated from Amazon’s public IP address pool for your account.</p><p><strong><em>Security Group — </em></strong>Security groups are used to define the flow of incoming traffic. For example, you could set an inbound rule to restrict SSH over port 22 to a specific group of IP addresses.</p><p><strong><em>Load Balancer — </em></strong>A load balancer evenly distributes the incoming traffic to the instances of all subnets it is associated with.</p><p><strong><em>Target Group — </em></strong>A target group defines the port your load balancer will be listening on for incoming traffic. For example, If I create an Instance type target group listening for HTTP traffic on port 80, then add that target group to a load balancer, any incoming traffic that meets that condition can be forwarded to an instance that is in one of the subnets behind the load balancer.</p><p><strong><em>Auto Scaling Group — </em></strong>An auto scaling group is a group of EC2 instances that can be scaled up or down horizontally to meet the compute demand of your application.</p><p><strong><em>Launch Template — </em></strong>A launch template is the blueprint for the auto scaling group. It defines things like which ami to use as well as what kind of instance type.</p><p>I think that just about covers it as far as terminology goes. Now, let’s get into the good stuff!</p><p>For this week’s project, we’ll be building out the 3-tier architecture we described above. The diagram at the top of this article illustrates the layout of our environment.</p><p><strong>Objectives:</strong></p><p><strong>Web Tier</strong></p><ol><li>2 public subnets</li><li>Minimum of 2 EC2 instances with an OS of your choice (free tier) in an Auto Scaling Group.</li><li>EC2 Web Server Security Group allowing inbound permission from the internet.</li><li>Boot strap static web page or create a custom AMI that already includes the static web page.</li><li>Create a public route table and associate the 2 public subnets.</li></ol><p><strong>Application Tier</strong></p><ol><li>2 private subnets</li><li>Minimum of 2 EC2 instances with an OS of your choice (free tier) in an Auto Scaling Group.</li><li>EC2 Application Server Security Group allowing inbound permission from the Web Server Security Group.</li><li>Associate with private route table.<br>Note: This is not a true application tier as we don’t have any provided code to run on the EC2 instances.</li></ol><p><strong>Database Tier</strong></p><ol><li>Use a free Tier MySql RDS Database.</li><li>The Database Security Group should allow inbound traffic for MySQL from the Application Server Security Group.</li><li>2 private subnets.</li><li>Associate with private route table.<br>Note: No need to use Multi-AZ but be sure to document how you would add it</li><li>Remember that when diagraming this tier you are only creating one RDS instance in one subnet even though you are creating two subnets. If you use multi-az or a read replica then and only then should you have a second instance. Make sure to label accordingly.</li></ol><p>Considerations:</p><ol><li>You will need to configure a public route table along with private route tables.</li><li>You will need a Nat Gateway in one of your public subnets to allow your instances in private subnets to update packages and patches. Make sure to include this in the private route table.</li><li>In order to reach your EC2 instances in the private subnets, you will either need a bastion host in the public subnet or need to use SSM. You will need to research into both to see which works better for your use case.</li><li>Plan out your CIDRs before you create your VPC and Subnets</li></ol><p><strong>ADVANCED</strong></p><p>(DO NOT ATTEMPT UNTIL YOU HAVE DONE THE FOUNDATIONAL PROJECT)</p><p>Application Load Balancers should be used for the Web Tiers with the same Security Group considerations. Example The ALB should allow HTTP traffic from 0.0.0.0/0 and the Web Server Security group should allow traffic only from the ALB. Don’t forget to create Target Groups and enable Health Checks.</p><p><strong>Verification and Deliverables:</strong></p><ol><li>Make sure you can access the web tier web page from the internet.</li><li>Verify that you can ping the application tier from the web tier by running the ping command from an EC2 instance in the web tier.</li></ol><p>Let’s get building!</p><p>To watch my video tutorial on how to build out this 3-tier architecture please visit the link below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfLuOD6IugL4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfLuOD6IugL4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfLuOD6IugL4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/820f95a162f4064dbc26b31163493a0f/href">https://medium.com/media/820f95a162f4064dbc26b31163493a0f/href</a></iframe><p>(Note: The length of this video was longer than I anticipated it would be when making it so feel free to adjust the playback speed while you watch).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52669dc2f2e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>