<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stephen Cleary on Medium]]></title>
        <description><![CDATA[Stories by Stephen Cleary on Medium]]></description>
        <link>https://medium.com/@stepencleary?source=rss-4b182c65f36b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CGULfkntFaHI7vMyhzqztg.jpeg</url>
            <title>Stories by Stephen Cleary on Medium</title>
            <link>https://medium.com/@stepencleary?source=rss-4b182c65f36b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:50:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stepencleary/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using the Serverless Framework to Deploy Faster, Easier, and More Consistently]]></title>
            <link>https://medium.com/protagona/using-the-serverless-framework-to-deploy-faster-easier-and-more-consistently-af02eed7df93?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/af02eed7df93</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Wed, 22 Nov 2023 18:31:13 GMT</pubDate>
            <atom:updated>2024-02-19T20:42:33.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*0WdgfVTlnfM3QQtkp-g0ag.png" /></figure><p>I’m ecstatic to be back with another electrifying writeup (pun intended). Today I will be seamlessly walking you through a simple workflow with an HTTP API, a Lambda function, and DynamoDB.</p><p>You’ve been hearing the buzzword “serverless” for years, well now is the time to finally try it out for yourself.</p><p>We don’t want to just go in the console and configure all of these services; It’s 2023 and we strive to build our infrastructure efficiently.</p><p>By using modern tools such as the Serverless Framework and Gitlab CI, we can reduce our time to market by releasing changes efficiently, systematically, and with confidence.</p><p>Let’s take a look at some of the services that we will be working with today!</p><p><strong><em>What is an HTTP API?</em></strong></p><p>An HTTP API communicates between two systems using the HTTP protocol; HTTP APIs make endpoints available as API gateways, allowing HTTP queries to connect to a server. You’re using APIs every day you just may not realize it.</p><p><strong><em>What is a Lambda function?</em></strong></p><p>Lambda is<strong> </strong>a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. These events may include changes in state or an update, such as buying plane tickets online.</p><p><strong><em>What is DynamoDB?</em></strong></p><p>DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB allows you to store, retrieve, and update data items in tables. It uses a primary key as a unique identifier for each item in a table, and secondary indexes to provide more flexibility when querying.</p><p><strong><em>What is the Serverless Framework ?</em></strong></p><p>The Serverless Framework is a free open-source web framework written in Node.js and designed specifically for building serverless applications with AWS Lambda. You can now use the Serverless Framework with a wide range of other cloud providers and services allowing you to deploy common resources quickly and efficiently.</p><p><strong><em>What is GitLab?</em></strong></p><p>GitLab is a platform that provides tools for version control and project collaboration offering features for source code management, continuous integration, continuous delivery (CI/CD), issue tracking, code review, and more.</p><p>Now that you are all ramped up on some of the services and tools we will be working with let&#39;s take a gander at some of the things we will need to be successful in this project.</p><p><strong>Project:</strong></p><ul><li>Find a free HTTP API and configure a Lambda Function and a DynamoDB table while utilizing the Serverless Framework</li><li>Use Python to write the Lambda Function and ensure you can capture the data sent to DynamoDB through the HTTP API</li><li>Implement a CICD pipeline and ensure that new pushes automatically trigger and build the services correctly.</li></ul><p><strong>Prerequisites:</strong></p><ul><li>An AWS account</li><li>An IDE (I will be using VS Code)</li><li>Basic AWS, Git, GitLab and Python knowledge</li><li>The will and desire to learn and get all the other stuff you don’t know or have</li></ul><p>Alright, you’re officially up to speed. In 3,2,1… Take off!</p><p><strong>Step 1: Finding a free API</strong></p><p>Obviously we need to get data into our DynamoDB table and what better way to do that other than finding a free API that will give us live data that we can ingest.</p><p>I shopped around for quite a bit and I found that Tequila (Kiwi) had a good flight search API that was perfect to use because who doesn’t like a good vacation! We will need to sign up for the program then receive an API key so that we can have access to the data.</p><p>Head on over to the <a href="https://tequila.kiwi.com/portal/login">Tequila website</a> and create and account (Don’t worry it’s free!)</p><p>Once you have the account all set up we are going to need to locate the API key that we will use in our Python code.</p><p>Click on My Company and then Create Solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h7nkDT1fRFQ7QT9pso8glA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*W9P6-IVzzx0eGQOTWYIfvw.png" /></figure><p>Then select One-way and Return and Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*FfHmwCxnkgCm8UtFfV8dUw.png" /></figure><p>Enter and name and then click Create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*cg9nSK0NOe49R2YpuNlG6A.png" /></figure><p>Your new solution will be created go ahead and click on the solution.</p><p>Once there you will see your API key. We will be using this in our Python code at a later point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*rbyJSt0lFkUHU-xv2PjiPw.png" /></figure><p>Off to a great start! I promise the boring administrational stuff is over.</p><p><strong>Step 2: Setting up a project with the Serverless Framework</strong></p><p>Now is when the fun really begins and you get to witness the power of the Serverless Framework. To begin we will be making sure that our local machine has Node.js installed. If you don’t already have it you can find it <a href="https://nodejs.org/en">here</a>.</p><p>Once we are all set up with Node.js we can move over into our IDE in my case <a href="https://code.visualstudio.com/">VSCode</a>.</p><p>I’ll be building in my Documents directory feel free to build where you see fit.</p><p>We need install a serverless module to get us started.</p><p><strong><em>Enter the following command:</em></strong></p><pre>npm install -g serverless</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7gqRFaFd0BWDspbml4QvA.png" /></figure><p>Alrighty, now we will need to get a project started.</p><p><strong><em>Enter the following command:</em></strong></p><pre>serverless</pre><p>You will be prompted with a list of available projects to choose from. We will obviously be working with the AWS — Python — HTTP API project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sh9rApZFzvWvvG0coozrYg.png" /></figure><p>The next prompt will be asking for the name of the project, get creative, have some fun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DalbgoTjTGC8JfPFuVa95w.png" /></figure><p>The serverless command will prompt you as a new user to do the following.</p><ul><li>Create a new project</li><li>Configure your <a href="https://serverless.com/framework/docs/providers/aws/guide/credentials/">AWS credentials</a></li><li>Set up a free Serverless Framework account with additional features</li></ul><p>Once you have everything configured lets add the service to your newly created account and organization.</p><p>Select create a new app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bLjHkTctefoVz8wY31NpLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZJXOnr8gKNPKf9tyT-UOVA.png" /></figure><p>You will be prompted with the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YMpMc21JN8575rVI-WlM-w.png" /></figure><p>Enter n. We will deploy this later.</p><p>If you look in your directory you will now conveniently see some files that may look familiar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3EjZsQvSNnR812mR1uZVyA.png" /></figure><p>It created the files for our lambda function, a README.md, a gitignore and the serverless.yml which will deploy our infrastructure.</p><p>We need to add a plugin that will take care of some of the pesky dependencies that we may need for our lambda function to work properly.</p><p><strong><em>Enter the following command:</em></strong></p><pre>sls plugin install -n serverless-python-requirements</pre><p><strong>Note: Make sure you are in the projects directory.</strong></p><p>Again, we have some more useful files and dependencies created for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8OpLQ8KbEWn5flpm-U54Fg.png" /></figure><p>Great work! Getting closer to deployment!</p><p><strong>Step 3: Configure necessary files</strong></p><p>In this portion of the tutorial we will be working on getting all of the necessary information into our newly created files. As this tutorial is more about witnessing the power of the Serverless Framework I will provide everything you need to get started. Feel free to experiment on your own.</p><p>We will start with our Lambda function. This will be the file named handler.py.</p><p>Open the file in your IDE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NsT28AT8kPIz3D5no1D-KQ.png" /></figure><p>You will see sample code as shown above.</p><p>Replace it with the following.</p><pre>import json<br>import boto3<br>import requests<br>import uuid<br>import datetime<br>from boto3.dynamodb.conditions import Key<br>import os<br><br># Initialize the AWS SDK and DynamoDB table<br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br><br>def extract_flight_details(item):<br>    flyFrom = item.get(&#39;flyFrom&#39;, &#39;&#39;)<br>    flyTo = item.get(&#39;flyTo&#39;, &#39;&#39;)<br>    duration = item.get(&#39;duration&#39;, &#39;&#39;)<br>    price = item.get(&#39;price&#39;, &#39;&#39;)<br><br>    return {<br>        &#39;flyFrom&#39;: flyFrom,<br>        &#39;flyTo&#39;: flyTo,<br>        &#39;duration&#39;: duration,<br>        &#39;price&#39;: price<br>    }<br><br>def query_flight_data(table):<br>    try:<br>        response = table.scan()<br>        items = response[&#39;Items&#39;]<br>        flight_details = [extract_flight_details(item) for item in items]<br><br>        return {<br>            &#39;statusCode&#39;: 200,<br>            &#39;body&#39;: {<br>                &#39;message&#39;: &#39;Flight data retrieved successfully&#39;,<br>                &#39;data&#39;: flight_details<br>            }<br>        }<br>    except Exception as e:<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;body&#39;: {<br>                &#39;message&#39;: &#39;An error occurred while retrieving flight data&#39;,<br>                &#39;error&#39;: str(e)<br>            }<br>        }<br><br>def lambda_handler(event, context):<br>    # Retrieve the DynamoDB table name from the event object<br>    dynamodb_table = os.environ.get(&#39;DYNAMODB_CUSTOMER_TABLE&#39;)<br><br>    # Create a DynamoDB table object<br>    table = dynamodb.Table(dynamodb_table)<br><br>    # Check if this is a request to query flight data<br>    if event.get(&#39;action&#39;) == &#39;query_flight_data&#39;:<br>        return query_flight_data(table)<br><br>    api_key = &#39;*******ENTER YOUR TEQUILA API KEY********&#39;<br>    endpoint = &#39;https://tequila-api.kiwi.com/v2/search&#39;<br><br>    <br>    current_timestamp = datetime.datetime.now().isoformat()<br><br>    params = {<br>        &#39;fly_from&#39;: event.get(&#39;fly_from&#39;, &#39;ATW&#39;),<br>        &#39;fly_to&#39;: event.get(&#39;fly_to&#39;, &#39;AUS&#39;),<br>        &#39;date_from&#39;: event.get(&#39;date_from&#39;, &#39;2023-11-16&#39;),<br>        &#39;date_to&#39;: event.get(&#39;date_to&#39;, &#39;2024-01-01&#39;),<br>        &#39;adults&#39;: int(event.get(&#39;adults&#39;, 1)),<br>        &#39;currency&#39;: event.get(&#39;currency&#39;, &#39;USD&#39;),<br>        &#39;partner_market&#39;: event.get(&#39;partner_market&#39;, &#39;us&#39;),<br>        &#39;sort&#39;: &#39;price&#39;,<br>        &#39;limit&#39;: 5<br>    }<br><br>    headers = {<br>        &#39;apikey&#39;: api_key,<br>    }<br><br>    try:<br>        # GET request to the Kiwi.com API<br>        response = requests.get(endpoint, params=params, headers=headers)<br><br>        # Check if the request was successful<br>        if response.status_code == 200:<br>            # Extract flight details from the API response<br>            data = response.json()<br>            if &quot;data&quot; in data:<br>                flight_data = data[&quot;data&quot;]<br>                flight_details = [extract_flight_details(item) for item in flight_data]<br>                if flight_details:<br>                    # Create formatted flight details list<br>                    formatted_flight_details = []<br>                    for index, flight_detail in enumerate(flight_details):<br>                        search_id = str(uuid.uuid4())<br>                        table.put_item(<br>                            Item={<br>                                &#39;search_id&#39;: search_id,<br>                                &#39;fly_from&#39;: flight_detail[&quot;flyFrom&quot;],<br>                                &#39;fly_to&#39;: flight_detail[&quot;flyTo&quot;],<br>                                &#39;duration&#39;: flight_detail[&quot;duration&quot;],<br>                                &#39;price&#39;: flight_detail[&quot;price&quot;],<br>                                &#39;timestamp&#39;: current_timestamp,<br>                                &#39;item_index&#39;: index  # Add an index to distinguish between items<br>                            }<br>                        )<br>                        # Append to formatted_flight_details<br>                        formatted_flight_details.append({<br>                            &#39;search_id&#39;: search_id,<br>                            &#39;fly_from&#39;: flight_detail[&quot;flyFrom&quot;],<br>                            &#39;fly_to&#39;: flight_detail[&quot;flyTo&quot;],<br>                            &#39;duration&#39;: flight_detail[&quot;duration&quot;],<br>                            &#39;price&#39;: flight_detail[&quot;price&quot;],<br>                            &#39;timestamp&#39;: current_timestamp,<br>                            &#39;item_index&#39;: index<br>                        })<br><br>                    <br>                    return {<br>                        &#39;statusCode&#39;: 200,<br>                        &#39;body&#39;: json.dumps({<br>                            &#39;message&#39;: &#39;Flight data retrieved and stored in DynamoDB successfully&#39;,<br>                            &#39;formatted_flight_details&#39;: formatted_flight_details<br>                        })<br>                    }<br>                else:<br>                    return {<br>                        &#39;statusCode&#39;: 500,<br>                        &#39;body&#39;: json.dumps({<br>                            &#39;message&#39;: &#39;Failed to extract flight details from the API response&#39;,<br>                            &#39;formatted_flight_details&#39;: []<br>                        })<br>                    }<br>            else:<br>                return {<br>                    &#39;statusCode&#39;: 500,<br>                    &#39;body&#39;: json.dumps({<br>                        &#39;message&#39;: &#39;No &quot;data&quot; field in API response&#39;,<br>                        &#39;formatted_flight_details&#39;: []<br>                    })<br>                }<br>        else:<br>            return {<br>                &#39;statusCode&#39;: response.status_code,<br>                &#39;body&#39;: json.dumps({<br>                    &#39;message&#39;: f&quot;Request failed with status code: {response.status_code}&quot;,<br>                    &#39;formatted_flight_details&#39;: []<br>                })<br>            }<br>    except Exception as e:<br>        return {<br>            &#39;statusCode&#39;: 500,<br>            &#39;body&#39;: json.dumps({<br>                &#39;message&#39;: &#39;An error occurred&#39;,<br>                &#39;error&#39;: str(e),<br>                &#39;formatted_flight_details&#39;: []<br>            })<br>        }</pre><p>Remember the API key that we created at the beginning of the tutorial? Now is the time! Go ahead and grab your API key and insert it in this line in the code block above.</p><pre>api_key = &#39;*******ENTER YOUR TEQUILA API KEY********&#39;<br></pre><p>Then save the changes.</p><p>Now we can look at the serverless.yml file which will define your functions, the events that trigger them, and the AWS resources to deploy.</p><p>So we need to define our infrastructure as well as the necessary permissions here.</p><p>Open the serverless.yml file.</p><p>Replace it with the following.</p><pre>org: steve0cleary<br>app: serverless-onboarding<br>service: aws-python-http-api-project<br>frameworkVersion: &#39;3&#39;<br><br>provider:<br>  name: aws<br>  runtime: python3.8<br>  httpApi:<br>    cors: true<br>  environment:<br>    DYNAMODB_CUSTOMER_TABLE: ${self:service}-customerTable-${sls:stage}<br>  iam:<br>    role:<br>      statements:<br>        - Effect: &#39;Allow&#39;<br>          Action:<br>            - &#39;dynamodb:PutItem*&#39;<br>            - &#39;dynamodb:Get*&#39;<br>            - &#39;dynamodb:Scan*&#39;<br>            - &#39;dynamodb:UpdateItem&#39;<br>            - &#39;dynamodb:DeleteItem&#39;<br>          Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:table/${self:service}-customerTable-${sls:stage}<br><br>functions:<br>  kiwi:<br>    handler: handler.lambda_handler<br>    events:<br>      - httpApi:<br>          path: /<br>          method: get<br><br>          <br>resources:<br>  Resources:<br>    CustomerTable:<br>      Type: AWS::DynamoDB::Table<br>      Properties:<br>        AttributeDefinitions:<br>          - AttributeName: search_id<br>            AttributeType: S<br>        BillingMode: PAY_PER_REQUEST<br>        KeySchema:<br>          - AttributeName: search_id<br>            KeyType: HASH<br>        TableName: ${self:service}-customerTable-${sls:stage}<br><br>plugins:<br>  - serverless-python-requirements</pre><p>Save the changes.</p><p>One more thing!</p><p>When we finally push our code to GitLab we won’t need the node_modules as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEG6Mc-vQ3KdDHzqnu2AGQ.png" /></figure><p>Add this line to your .gitignore file to keep them from being synced.</p><pre>*node_modules/*</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfBfVY0VpwkiELMRsz8LmQ.png" /></figure><p>Then save the changes.</p><p>Nice one! Moving forward or sideways, whatever you prefer.</p><p><strong>Step 4: Configuring and pushing to GitLab</strong></p><p>As I had mentioned in the beginning of this article we will be configuring a CICD pipeline so that our future deployments will be automated and built efficiently.</p><p>If you don’t have a GitLab account you will need to sign up for one <a href="https://gitlab.com/users/sign_up"><strong>here</strong></a>.</p><p>The first thing that we are going to want to do is to create a project.</p><p>There will be a sidebar on the left.</p><ol><li>Click the ‘+’ symbol and New project/repository</li><li>Click on the Create a blank project option</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7TsdfHkZ0TEP4QBCEPQeQ.png" /></figure><p>You will be prompted with the following as shown below.</p><ol><li>Give your project a name</li><li>Choose the location to create the project</li><li>Make your repo public or private by selecting the correct box in the visibility selection</li><li>Uncheck the Initialize repository with a README box</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZeshMm5zg-Cqq4LtzlhvPg.png" /></figure><p>Perfect. You will then be prompted with the following page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xtGibNfwM5RsG1vg0Qt0Hg.png" /></figure><p>We will first need to configure our access tokens however before we can move forward.</p><p>Over to the left on your sidebar make sure you are in the correct project. Click ‘Settings’ then ‘Access Tokens’ as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*XmRAzCzA4CEToi93TJdvfA.png" /></figure><p>On the next page click Add a new token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WtXWoEmUQtwFiIdyVJ-okQ.png" /></figure><p>Next, give the token a name, an adequate expiration date, and select the appropriate permissions.</p><p><strong><em>Note: For this demo, I will be granting full permissions. It’s advised to understand the scope of these permissions before doing so in production.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVmUsUQR8cRxYWTHzFqIbw.png" /></figure><p>You’ll be prompted with your access code. Take a sneak peak and make sure to save it somewhere safe, we will be using it shortly.</p><p><strong><em>Note: You won&#39;t be able to access this again. If you fail to save it you’ll need to follow the steps again.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aN3KHRyyVj_WKsHXNYgzHQ.png" /></figure><p>Navigate back to your IDE we’re going to Git it going!</p><p>If you don’t already have Git installed you’ll need to do so here.</p><p><a href="https://git-scm.com/">Git</a></p><p>Great, let’s Git to it. Last one, I promise!</p><p><strong>Note: Make sure you are working in the correct directory.</strong></p><p><strong><em>Enter the following commands:</em></strong></p><pre>git config --global user.name &quot;your name&quot;<br>git config --global user.email &quot;email@gmail.com&quot;<br>cd &lt;your_folder&gt;<br>git init --initial-branch=main<br>git remote add origin git@gitlab.com:youraccount/project.git<br>git add .<br>git commit -m &quot;Initial commit&quot;<br>git push --set-upstream origin main</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NxxYQDuOXfeTWBWDMGNuhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*C_ljQFrzG565Fm1wgHsKYw.png" /></figure><p>Great, lets take care of one more thing! Our project depends on libraries and other dependencies so well need to create a requirements.txt.</p><p><strong><em>Enter the following commands:</em></strong></p><pre>python3 -m venv .venv<br>source .venv/bin/activate <br>pip3 install boto3 requests uuid datetime os<br>pip freeze &gt;&gt; requirements.txt </pre><p>If done correctly you will see that you have a new file created named requirements.txt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-SImCCvJjDQZW5sxfrcrZw.png" /></figure><p><strong><em>Note: Commands will vary depending on OS and other factors.</em></strong></p><p>Now we need to add the file and push it to out GitLab repo.</p><p><strong><em>Enter the following commands:</em></strong></p><pre>git add requirements.txt <br>git commit -m &quot;adding requirements.txt<br>git push --set-upstream origin main </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*E61dhitP1OBjx7sObIpUWw.png" /></figure><p><strong>Step 5: Creating the CICD pipeline</strong></p><p>We have all of our necessary files in our repository except for one and probably the most important one. To use GitLab we need to create and configure our .gitlab-ci.yml file.</p><p>The gitlab-ci.yml file is holds all of the scripts we want to run, configuration files and templates you want to include, dependencies and caches, whether you want to run the scripts automatically or trigger any of them manually, location to deploy your application to.</p><p>It’s, well, important!</p><p>At your repositories landing page click on Setup CI/CD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPYGQUq9qqsvRHwsuH9Cbg.png" /></figure><p>Then click Configure pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s9Or33ZhWcayrtfdsmw2dA.png" /></figure><p>You’ll come to this page below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X4eG6APZzq8CVT_IjMxPIw.png" /></figure><p>Remove the YAML and copy and paste what I have provided below in this section:</p><pre>image: amazonlinux:2<br>stages:          # List of stages for jobs, and their order of execution<br>  - build<br>  - test<br>  - deploy<br><br>build-job:       # This job runs in the build stage, which runs first.<br>  stage: build<br>  script:<br>    - echo &quot;Compiling the code...&quot;<br>    - echo &quot;Compile complete.&quot;<br><br>before_script:<br>    - yum update -y<br>    - yum install gcc-c++ make -y<br>    - yum install https://rpm.nodesource.com/pub_16.x/nodistro/repo/nodesource-release-nodistro-1.noarch.rpm -y<br>    - yum install nodejs -y --setopt=nodesource-nodejs.module_hotfixes=1 <br>    - amazon-linux-extras enable python3.8<br>    - yum install python3.8 -y<br>    - npm install<br>    - npm install -g serverless<br><br><br>deploy-job:      # This job runs in the deploy stage.<br>  stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.<br>  environment: production<br>  script:<br>    - serverless deploy<br></pre><p>Then commit the changes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_oDZ7DP-d7cPSUVYj1lsig.png" /></figure><p>As soon as you commit, the pipeline has started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kX6HPnRcqpHyttmpFStitQ.png" /></figure><p>If you click on view pipeline we can see the two stages that we have configured in the .gitlab-ci.yml file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1i_sdXAgmKQKo32F68R_vw.png" /></figure><p>This is going to fail as we don’t haven’t set our access keys for AWS and Serverless Framework.</p><p>We will need to pass our access key, secret access key, our region, and our Serverless access key to GitLab so that it can authenticate with our credentials.</p><p><strong>DO NOT HARDCODE ANY CREDENTIALS!</strong></p><p>Instead, set environment variables from within the GitLab platform.</p><p>Click the settings drop-down tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8Wu-0ZhIynEbm842shw9w.png" /></figure><p>Then select CI/CDand expand the variables section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UkIJEzQxUx7xbV2tFg9tzA.png" /></figure><p>Here is where we can add our variables. Click Add variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6hKO3pjJNLQpLQZELij4w.png" /></figure><p>We will be adding three variables. Below are the keys we will be using.</p><pre>    KEYS                       VALUES <br>AWS_ACCESS_KEY_ID        &lt;AWS ACCESS KEY ID&gt;<br>AWS_SECRET_ACCESS_KEY    &lt;AWSSECRET ACCESS KEY ID&gt;<br>SERVERLESS_ACCESS_KEY    &lt;Serverless ACCESS KEY ID&gt;</pre><p>If you are unsure what your values are you will need to create the access keys in AWS IAM. I will walk you through it quickly.</p><p>If you already have access keys stored you can skip over this part.</p><p>Navigate to the IAM dashboard then select Users on the left and click on the user that you want to have access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*wpGq5mr1j9A7aJ50.png" /></figure><p>Click the “Security credentials” tab and then Create access key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*c_OIqyZIojPAt9BI.png" /></figure><p>Click other and next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*u2EICm65RbSVFqeW.png" /></figure><p>Enter a description and click Create access keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*vWtPqS-m0ZpESFEn.png" /></figure><p>On the next page, you will have your keys presented. This will be what you enter into the value section when setting the variables in GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Ok9KyxyXHT7iXh80.png" /></figure><p>I recommend downloading the .csv file. You will not see the secret access key again once you leave this page. Alright, back to setting the variables in GitLab.</p><p>After you expand the variables section click on Add variable once again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*OTAG1S-RdQZbFTGS.png" /></figure><p>Then enter the key and the value which you have just created in AWS. Make sure and check the mask variable box as well. Then click Add variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*PMXyGpVCwGgddKPT.png" /></figure><p>Follow the same step for the AWS_SECRET_ACCESS_KEYthen we just need to get the SERVERLESS_ACCESS_KEY from our <a href="https://www.serverless.com/framework/docs/guides/dashboard">dashboard</a> and enter it.</p><p>Once there click on the gear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RB4A57WHpSt_wh2rYHwndA.png" /></figure><p>Then select Add .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UN6YAptzMfESe60e752NQ.png" /></figure><p>Give your key a name (anything will do).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEyEY0bcoo8MB6MkDYty_A.png" /></figure><p>You’ll be prompted with your access key. Now we need to input that into our GitLab account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mTTvwusvsZMhxwn0iYWdQA.png" /></figure><p>If done correctly you should have three variables for your CICD pipeline to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSB5w23_aWkEI975AH-uDQ.png" /></figure><p>Let’s run the pipeline again manually.</p><p>From the sidebar click Pipelines and Run Pipeline .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_RmLRvHcTsWGmauRse13g.png" /></figure><p>On the next page click “Run pipeline” again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wET9-scFD_Zyjtmp9-f3XQ.png" /></figure><p>When the build-job is finished let’s take a look at the deploy-job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7utRjc6J0VSk7rUUrsJkqA.png" /></figure><p>Click deploy-job.</p><p>Here we can see the deployment live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgrs6VCj32NOZFHI6109VA.png" /></figure><p>And as a wise Frenchman said, “voila”.</p><p>As you can see the job deployed successfully and here we have our endpoint.</p><p><strong><em>Click on the underlined endpoint.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q_fdhw8EnrHCs2gzjRquuA.png" /></figure><p>Here we have the data that we are requesting from the Kiwi Api!</p><p>Let’s go back to the console to verify that we have deployed the necessary resources.</p><p><strong>Step 6: Verify in AWS console</strong></p><p>This is were we see the real power of using the Serverless Framework as well as implementing CICD pipeline. When you implement this practice you are not only saving time clicking around in the console you have reusable infrastructure that can be deployed in a matter of minutes.</p><p>Head on over to the API Gateway landing page by searching for API Gateway in the AWS management console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KPIBE0XMIJiAvvDYwETmfw.png" /></figure><p>Then select the API Gateway that you created.</p><p>Once on your API Gateway page you will see a familiar URL!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vjaxhjb_hLs3FzFjbJgWLw.png" /></figure><p>Go ahead and invoke the URL by clicking it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2Ur_QW2vosghMbHbAIZfQ.png" /></figure><p>And as the Englishman once said, “ta-da”!</p><p>Now let’s look at our Lambda Function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X6-_x-guUWjhdf86nP0sfw.png" /></figure><p>Click on your functions name.</p><p>As you can see below we have created our Lambda Function as well as an API gateway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7DZ3EFVs-FbUPO2TEdzug.png" /></figure><p>Great! Let&#39;s test our solution.</p><p>If you click on the test tab as shown below and create a new test even this will allow us to manually test our function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dvLfIx8pqOsachywSbbZFg.png" /></figure><p>Then click the orange test button.</p><p>Again, we have the results we are looking for!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ny8ceCqym5zVUIaqZpDm4g.png" /></figure><p>Just one last thing to check: If you remember correctly we created a DynamoDB table in our serverless.yml. Let’s make sure that we are storing our data as intended.</p><p>Navigate to the DynamoDB landing page.</p><p>Select tables and then the table that you created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysUKIiedg74Hatww_uo_Pg.png" /></figure><p>Click explore table items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kO8g9xF-gXUGiUY9ZH80Bg.png" /></figure><p>There you should see the data that you have ingested from the Kiwi API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YffQU6yhnLoxnqnbeInyOg.png" /></figure><p>Now you can see for yourself the benefits of using the Serverless Framework and implementing a CICD pipeline. By utilizing some of these tools you will highly improve your team’s life and create business value!</p><p><strong><em>Note: Make sure and destroy all infrastructure that you created in AWS.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*54T1j3svwsIAPsZ_xNBj7g.png" /></figure><p>As always, feel free to join us on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation and help you learn more about the exciting world of cloud computing.</p><p>Thank you for reading, and I look forward to hearing from you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af02eed7df93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/using-the-serverless-framework-to-deploy-faster-easier-and-more-consistently-af02eed7df93">Using the Serverless Framework to Deploy Faster, Easier, and More Consistently</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building AWS Infrastructure with a GitLab CICD Pipeline and Terraform]]></title>
            <link>https://medium.com/protagona/building-aws-infrastructure-with-a-gitlab-cicd-pipeline-and-terraform-621a504df02e?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/621a504df02e</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Thu, 14 Sep 2023 21:20:21 GMT</pubDate>
            <atom:updated>2023-09-14T21:20:21.868Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JvNfv1BtSMjykOLztlcyw.png" /></figure><p>If you’re reading this article you’ve probably heard of the many benefits of using Infrastructure as Code. Using IaC allows us to develop our infrastructure faster, cheaper, and more consistently. What’s not to love about that?</p><p>It gets even better though! With cultural shifts like GitOps and DevOps, we can sweep the days of “ClickOps” aside. With these revolutionary practices we are able to not only avoid making changes in the console but establish a workflow that promotes increased operational efficiency.</p><p>In this tutorial I will be walking you through creating a GitLab CICD pipeline and using Terraform as our IaC tool.</p><p>If you’re anything like me and you love infrastructure and automating everything you can get your hands on, this article is for you.</p><p><strong>What is Terraform?</strong></p><p>Terraform is one of the leading Infrastructure as Code tools on the market today. What makes Terraform so popular is the fact that it is cloud-agnostic meaning that it will work with AWS, Azure, GCP, and other providers. In short, Terraform automates the process of creating the same resources over and over again.</p><p><strong>What is GitLab?</strong></p><p>GitLab is a platform that provides tools for version control and project collaboration. It offers features for source code management, continuous integration, continuous delivery (CI/CD), issue tracking, code review, and more.</p><p><strong>Pre-Requisites</strong></p><ul><li>GitLab account</li><li>Terraform Code (HCL)</li><li>AWS Account</li><li>IDE ( I will be using Cloud 9)</li></ul><p>Now that you know what you’re working with and understand what tools you need, let’s get started!</p><p><strong>Step 1: Create a Project and Push to GitLab Repository</strong></p><p>Like every great team or project, we need to have a legendary name. I’m going to go with “Freedom” as this project will release us from some of the toil that may be involved in a poor production environment.</p><p>During your account creation you will be prompted with the following form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b068ELgOYw29sduJFARhQg.png" /></figure><p>Click “Create project” after you’ve come up with your own legendary name!</p><p>Follow the prompts until you arrive at the GitLab landing page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oPAOxLjQE1Zm0NLGDpNlCA.png" /></figure><p>From here, we need to get our code into the GitLab repository.</p><p>To do this we will need to initialize the repository on our local machine using git.</p><p><strong><em>Enter the following commands:</em></strong></p><pre>git add .<br><br>git commit -m &quot;add comment&quot;</pre><p>To effectively push and pull between our local Git repo and our repo in Gitlab we will need add Gitlab as the remote source.</p><p><strong><em>Enter the following command:</em></strong></p><pre>git remote add gitlab &lt;https://github.com/ENTER PROJECT URL</pre><p>Next, we will upload our existing project into GitLab by pushing it.</p><p><strong><em>Enter the following command:</em></strong></p><pre>git push &lt;https://github.com/ENTER PROJECT URL&gt;</pre><p><strong>Permission denied!</strong></p><p>Let’s confirm the repository exists and we have the permissions configured correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjlOJYAN6s8Z1kcW91LEVg.png" /></figure><p>The repository has been configured correctly as shown above with the main branch.</p><p>Navigate to the “Project overview” tab on the left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRR_QbKlwHuqaiAFc1FEDg.png" /></figure><p>As you can see from the above photo, GitLab is screaming at us to get some authentication methods set up.</p><p>Since my last article involved setting up SSH keys with AWS CodeCommit and a local machine we will mix it up a little and configure our authentication using personal access tokens.</p><p>At the top left of the Gitlab landing page you will see your avatar. Go ahead and click on the avatar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6Y9LG7mW38MCqpQWgftXw.png" /></figure><p>Next, click “Edit profile”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N1GNsdVcZR5mbHStMTOvCA.png" /></figure><p>Then access tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qb7pwIMgq-u4yoBeJZsdog.png" /></figure><p>Then add new token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_YQ7e7PFpIsI_jC2IKFLg.png" /></figure><p>Next, give the token a name, an adequate expiration date, and select the appropriate permissions.</p><p><strong><em>Note: For this demo, I will be granting full permissions. It’s advised to understand the scope of these permissions before doing so in production.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EDZZ76h6T3KBiU9s4bdNyw.png" /></figure><p>Your personal access token will then be generated. Make sure and save this token somewhere secure as you will not be able to view it again after its initial creation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3dio7L8bKwyP07hJE_lOug.png" /></figure><p>Now that we have our personal access token configured we can now push our code into the GitLab repo.</p><p>You will be prompted for your username as well as the password. Your username can be found under the account tab on the left and under the change username section as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5N6eve5a8xtpNbL5gSxXw.png" /></figure><p>The password will be your personal access token that you just generated moments ago. Hopefully, you saved it! If not, no worries, just create a new one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*mjBM0ah1jLZ2bk8c7c--mQ.png" /></figure><p>Head on over to your projects and select the correct one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wm-0yZA0IEaq_yORwlYTtw.png" /></figure><p>As you can see in the below photo, all the files from our local Git repo have been pushed to the GitLab repo that we just created!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rM5zw1vLoGUopzbUOFCJEA.png" /></figure><p>Nice work!</p><p><strong>Step 2: Authenticate with AWS</strong></p><p>As we will be deploying all of our infrastructure into AWS we will need to pass our access key, secret access key, and our region to GitLab so that it can authenticate with our credentials. If we hardcoded these credentials it would be very easy for someone to obtain.</p><p><strong>DO NOT DO THIS!</strong></p><p>Instead, set environment variables from within the GitLab platform.</p><p>Click the settings drop-down tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f33fMZBAcGu1wJL2PH9npg.png" /></figure><p>Next, click “CICD” and expand the variables section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FKzRuq4FunBesOaVj0GlNA.png" /></figure><p>We will be adding three variables below are the keys we will be using.</p><pre>     KEYS                   VALUES <br>AWS_ACCESS_KEY_ID        &lt;ACCESS KEY ID&gt;<br>AWS_SECRET_ACCESS_KEY    &lt;SECRET ACESS KEY ID&gt;<br>AWS_DEFAULT_REGION       &lt;DEFAULT REGION YOU ARE DEPLOYING TO&gt;<br></pre><p>If you are unsure what your values are you will need to create the access keys in AWS IAM. I will walk you through it quickly.</p><p>If you already have access keys stored you can skip over this part.</p><p>Navigate to the IAM dashboard then select “Users” on the left and click on the user that you want to have access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bEciWmlsgf_rmBbo3iaEBw.png" /></figure><p>Click the “Security credentials” tab and then Create access key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R9WFAoxTFYi2b402ZIfgqw.png" /></figure><p>Click other and next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HsXw1Eb9nBMA-Qa5UN-IXA.png" /></figure><p>Enter a description and click “Create access keys”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJd34_JPTAUnchU6pgM8cg.png" /></figure><p>On the next page, you will have your keys presented. This will be what you enter into the value section when setting the variables in GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBdWg2_TjaORmnxc0M90zA.png" /></figure><p>I recommend downloading the .csv file. You will not see the secret access key again once you leave this page.</p><p>Alright, back to setting the variables in GitLab.</p><p>After you expand the variables section click on add variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hM0qYJPIJ52BSbKecdS2tw.png" /></figure><p>Then enter the key and the value which you have just created in AWS. Make sure and check the mask variable box as well. Then click add variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*BWtxv7CMasnwG_L-ySk0Lw.png" /></figure><p>We will repeat this process for the other two variables that we need to set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2I11gA16nBNcVeHRozfN7w.png" /></figure><p>Now onto the fun stuff, building out the pipeline! Freedom is upon us!</p><p><strong>Step 3: Building out the CICD Pipeline</strong></p><p>This section is what we are really here for. We will be building out our CICD pipeline. This will allow us to make infrastructure changes with just a few strokes of the keyboard.</p><p>To use GitLab CICD there are two requirements. One of them is to have our application code hosted in a Git Repository. We have already accomplished this when we pushed our code from our local machine into the repository hosted in GitLab.</p><p>The other requirement is that we have a .gitlab-ci.yml file located in the root of the repository.</p><p><strong>What is a .gitlab-ci.yml file?</strong></p><p>Basically, this is a file that contains the work that the pipeline needs to perform. It contains additional configuration files and templates, dependencies, caches, and commands of the code that will be running in the CICD Pipeline.</p><p>For example, you might need to make sure that Terraform is installed before running all of the HCL code. Otherwise, it would fail.</p><p>Let’s add the .gitlab-ci.yml file to our GitLab repository so we can automate our workflow already!</p><p>Click on the “Code” dropdown tab and then click “Repository”. From there we can add a new file to the main branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKtqVbRMuLKG0Uq85ZS2iw.png" /></figure><p>Next, we will add the file and its contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJj0WIoyWS6hRn9FmqpmBA.png" /></figure><p>A few things to note here. We need to make sure that the file is exactly .gitlab-ci.yml otherwise it will not work.</p><p>There are also templates available to use for your convenience. I personally used a Terraform template here but it didn’t work. You more than likely will need to find your own solutions through trial and error and documentation.</p><p>We will be committing directly to main. This is not a great idea in production for obvious reasons. Consider using a branching strategy.</p><p>I will leave my code here as well.</p><pre>stages:          # List of stages for jobs, and their order of execution<br>  - build<br>  - test<br>  - deploy<br><br>build-job:       # This job runs in the build stage, which runs first.<br>  stage: build<br>  script:<br>    - echo &quot;Compiling the code...&quot;<br>    - echo &quot;Compile complete.&quot;<br><br>unit-test-job:   # This job runs in the test stage.<br>  stage: test    # It only starts when the job in the build stage completes successfully.<br>  script:<br>    - echo &quot;Running unit tests... This will take about 60 seconds.&quot;<br>    - sleep 60<br>    - echo &quot;Code coverage is 90%&quot;<br><br>lint-test-job:   # This job also runs in the test stage.<br>  stage: test    # It can run at the same time as unit-test-job (in parallel).<br>  script:<br>    - echo &quot;Linting code... This will take about 10 seconds.&quot;<br>    - sleep 10<br>    - echo &quot;No lint issues found.&quot;  <br><br>deploy-job:      # This job runs in the deploy stage.<br>  stage: deploy<br>  script:<br>    - apt-get update -y  # Update the package manager (for Debian/Ubuntu)<br>    - apt-get install -y unzip wget  # Install necessary tools<br>    - wget https://releases.hashicorp.com/terraform/1.1.0/terraform_1.1.0_linux_amd64.zip<br>    - unzip terraform_1.1.0_linux_amd64.zip<br>    - mv terraform /usr/local/bin/<br>    - terraform init<br>    - export TF_VAR_candidate=stephen <br>    - echo &quot;stephen&quot; <br>    - terraform apply -auto-approve<br>  only:<br>    - main  <br>  environment:<br>    name: production</pre><p>When you are ready click “Commit changes”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5WVDHY6xX2WZcm3cJekUnw.png" /></figure><p>Quickly navigate to the build drop-down tab in your project and click on “Pipelines”. You will see at the top that the pipeline is in fact running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JQoqg5XUrklpQbFmmm_Urg.png" /></figure><p>We can click on the pipeline to further investigate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v2hqJr1xnNLjNr70d9NdLQ.png" /></figure><p>Here we can see the different stages of the build being completed or failing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxeF-4HYXrz6ywXn3yQvSA.png" /></figure><p>Let’s go a little bit further and watch the build in action. Click one of the stages I’ll be showing the deploy job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1EQup4w8Rjhp0MGhsGQzw.png" /></figure><p>Now, as you can see, you can actually watch what is being performed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*Hig92KRVsjfLq4ZUXh7GKg.png" /></figure><p>And if all goes correctly….</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*JiX-EBBFr6gThlQV7-K0Qw.png" /></figure><p>Job succeeded!</p><p>Don’t worry if this doesn’t go as smoothly with your code. This was all trial and error for me. When you finally get this message, you’re gonna be glad you stuck with it and persevered, trust me!</p><p><strong>Step 4: Verify in AWS Console</strong></p><p>If you’ve worked in IT for any amount of time you’ll know how utterly important it is to verify your work. In this step, we are going to go into the AWS console and make sure that our infrastructure is set up correctly and in working order.</p><p>The main purpose of the HCL(Terraform) code was to create an environment that would allow a Lambda function to retrieve a random quote and store it in a DynamoDB table.</p><p>Let’s first check to make sure that the Lambda function is working correctly by running a test.</p><p>I’ll head over to the Lambda dashboard in AWS.</p><p>Click “Functions”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4opvvu_7L9ua5iZV9ZRGqw.png" /></figure><p>Select the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpR4_MXu5Hw8760HtGFCJw.png" /></figure><p>Then test the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9H3YETuAS4hmZEVmFwmj5g.png" /></figure><p>Perfect, we got the response we were looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sOthSaiUKafVjAQfk0UqxQ.png" /></figure><p>Now, let’s check the DynamoDB table to make sure that we have the items.</p><p>Head to the DynamoDB dashboard.</p><p>Select tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*puFqTqDw6joGxyZHcblAJQ.png" /></figure><p>Click on the appropriate table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jSeLTFkxpoc5AnC7Ix71yg.png" /></figure><p>Explore table items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1oUh_hUPacHH0XnPCzTdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3P7z4jeIgH6IvJhNlYX7sg.png" /></figure><p>There we have it! A few quotes that we were expecting from the random quote generator.</p><p><strong>Step 5: Testing the CICD Pipeline Automation</strong></p><p>This whole article has been about gaining freedom by incorporating cultural shifts internally and building automated systems to make our lives easier.</p><p>We need to test the repository to make sure that when a developer pushes new code we can trigger the pipeline and intern create resources or whatever changes that are implemented in the commit.</p><p>Let’s head back to our local machine make some changes in the main.tf and try to trigger the CICD Pipeline.</p><p>We are going to add a new S3 bucket and push it to GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*KwML6NmWnkj9GQwH-NzZaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*bUKRZVI0CKCejeAQSuYp7g.png" /></figure><p>As you can see, we have a new commit in our repository in GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f9-_tD03VjhQYjmsrmxuSQ.png" /></figure><p>However, if we look at our pipeline it’s not running. But why?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*95-89fKUVjgUyzaSkZJF2w.png" /></figure><p>GitLab does not run the build straight out of the box there are various methods that you can use to implement the changes that you think are a good fit for your organization.</p><p>In the next step, we are going to create a pipeline schedule. This will trigger the pipeline to run at a given time. You can think of it like a cronjob.</p><p><strong><em>Note: You can also trigger pipelines manually or via API methods with GitLab. See the documentation </em></strong><a href="https://docs.gitlab.com/ee/ci/triggers/index.html"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><p><strong>Step 6: Create a Pipeline Schedule</strong></p><p>Now, as I mentioned in the previous section there are a few ways that you can automate your pipeline build. In this section, I will be walking you through creating a schedule for your pipeline to run. This will run the pipeline at your specified interval.</p><p>Back in the GitLab UI, click the “Build” drop-down tab and click “Pipeline schedules”.</p><p>Then we can proceed by clicking “New Schedule”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vVujJIHnidV1ZERYWa65hg.png" /></figure><p>Enter the following information and create the appropriate schedule that works best for you. If you are unfamiliar with cron schedule expressions you can find a calculator <a href="https://crontab.guru/#35_10_*_*_*">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2zaVgQlrBnz-hCUIua4pcw.png" /></figure><p>Click “Save pipeline schedule”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Dh4bmhzFCy0B9ETyPHoIg.png" /></figure><p>Great! Remember that we have already committed the infrastructure change to the Gitlab repo. If you remember we added an s3 bucket. Now we will wait and see if the job runs successfully and if we have a new s3 bucket in our AWS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KBQWepYjLcq5JGo-ihEdpg.png" /></figure><p>The pipeline is running as expected, let’s wait and verify it passes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHg90NaOwG9GBvL454tYUw.png" /></figure><p>Everything worked as planned, now the moment of truth. Did the scheduled pipeline create our new s3 bucket?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pc735clbRYsWaiPxQFshjg.png" /></figure><p>It sure did!</p><p>Now you can see the power of running scheduled jobs and implementing a CICD pipeline. By learning some of these tools you will highly improve your team&#39;s life!</p><p>Until next time!</p><p><strong><em>Note: Make sure and destroy all infrastructure that you created in AWS.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lYM_7xFybpNsVyMwkTGMZQ.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation and help you learn more about the exciting world of cloud computing.</p><p>Thank you for reading, and I look forward to hearing from you soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621a504df02e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/protagona/building-aws-infrastructure-with-a-gitlab-cicd-pipeline-and-terraform-621a504df02e">Building AWS Infrastructure with a GitLab CICD Pipeline and Terraform</a> was originally published in <a href="https://medium.com/protagona">Protagona</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using SSH to Clone an AWS CodeCommit Repository]]></title>
            <link>https://towardsaws.com/using-ssh-to-clone-an-aws-codecommit-repository-d244f390cfca?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d244f390cfca</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[ssh]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Tue, 01 Aug 2023 23:23:45 GMT</pubDate>
            <atom:updated>2023-08-03T06:38:33.734Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBMaKw956Ailuj-1gEd93A.png" /></figure><h3><strong>The Scenario</strong></h3><p>You’ve just been given the new repositories&#39; details in CodeCommit and you’re ready to start slamming away at making vital changes to the infrastructure that your team needs.</p><p>You find the “Clone URL” and “HTTPS” button and copy the URL.</p><p>You seamlessly enter it into the command line and BAM, it doesn’t work!</p><p>So what’s next do we dive deeper into the reason and dig for the cause? Or do we find a trusty workaround?</p><p>In this Tutorial, I will be walking you through setting up SSH with CodeCommit!</p><p><strong>Step:1 Create a CodeCommit Repository</strong></p><p>First things first, we need a repository to clone. Navigate to your AWS console and search for “Code Commit”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBl1dcGIntmr5_Yg60ppYA.png" /></figure><p>Next, click the orange “Create repository” button.</p><p>When the next page has loaded go ahead and give a name to your repository. You can also add a description (optional).</p><p>Then click “Create”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*co-uY7cCTbuWnB59cUcDCQ.png" /></figure><p>Next, we will add a file so that our Repo isn’t empty. Go ahead and click “Create a file”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLpGe9ko2dync-18nTKy7Q.png" /></figure><p>In the next step, I’m going to use terraform code to create a file but feel free to create any normal file. I’ll leave the Terraform code below as well for your use.</p><pre>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}<br><br>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}<br><br>terraform {<br>  cloud {<br>    organization = &quot;two_tier&quot;  # enter you organization name here<br>    <br>    workspaces {<br>      name = &quot;two-tier-dev&quot;    # enter you workspaces name here<br>    }<br>  }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7b4irAn0xtrwDy7fGSCS1w.png" /></figure><p>Now, we just need to commit our changes, click “Commit changes”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EPmQ8ZOXipz4mcAUo85Vw.png" /></figure><p>As you can see we now have a repository with a file, Great! Moving forward!</p><p><strong>Step 2: Configuring SSH Keys</strong></p><p>Now remember from the scenario that HTTPS wasn’t working and we needed a workaround. In this section, I will discuss using SSH and how to configure it with CodeCommit.</p><p>Grab the URL for SSH.</p><p><strong><em>Enter the following command:</em></strong></p><pre>git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/Terraform_Project4</pre><p>You will get the following error or something similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4yYEGBuyo9666CibmD5ROg.png" /></figure><p>Hmm, Permission denied (publickey).</p><p>Let’s take a look at how to resolve this.</p><p><strong><em>Enter the following command into your terminal:</em></strong></p><pre> ssh-keygen</pre><p>Then, press enter to skip the input of the next three lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1JRv1a-MBZJeGX-E01iY-w.png" /></figure><p>If done correctly you should have an output as shown above.</p><p>I know what you’re thinking….What just happened? Well, we created two key files, both a public and private key file. This is what we will use to authenticate with CodeCommit.</p><p>We need to view the contents of the public key file.</p><p><strong><em>Enter the following command:</em></strong></p><pre>cat ~/.ssh/id_rsa.pub</pre><p>If done correctly you should see a similar output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpUrmle7sUTV1bUP2XKhyA.png" /></figure><p>Copy this, we are going to need it!</p><p>Now we need to go into the IAM dashboard and make some changes to our user’s permission.</p><p>Select the user that you want to make the changes for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s4RLIaaceXEpRkoKMmas9w.png" /></figure><p>On the user’s page, click the Security credentials tab and then Upload SSH public key. This is where you will be pasting the public key that you copied from the command line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rymXtxQDth5U-rVMmOf2eQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O00u4yJU4Vrq7S3f99GgYg.png" /></figure><p>Then copy the SSH Key ID as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpAbolzdn0Zh8Lmjmf6WOg.png" /></figure><p>Back on your local machine, we need to make some changes.</p><p>We need to add a config file to the ~/.ssh directory and add some information.</p><p>Navigate to the .ssh directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*quLD-aIE4uRJFKTaEVDyfA.png" /></figure><p><strong><em>Enter the following command:</em></strong></p><pre>nano config</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0ljpAmFnqBeiMS-gRidxw.png" /></figure><p><strong><em>Enter the following into the text editor:</em></strong></p><pre>Host git-codecommit.*.amazonaws.com<br>  User &lt;SSH Key ID&gt;<br>  IdentityFile ~/.ssh/id_rsa</pre><p>Then save and exit.</p><p>Great now we need to adjust the permissions on the file.</p><p><strong><em>Enter the following command:</em></strong></p><pre>chmod 600 config</pre><p>Perfect, now we can test the configuration!</p><p><strong><em>Enter the following command:</em></strong></p><pre>ssh git-codecommit.us-east-2.amazonaws.com</pre><p>Enter yes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KdDcfYdH7XQNLJrIJs9AAg.png" /></figure><p>If done correctly you will see that you have successfully authenticated over SSH.</p><p>Now we can clone the repo!</p><p><strong><em>Enter the following command to clone the repo:</em></strong></p><pre> git clone &lt;SSH URL&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTekaaMq3aWXmMqcM8v8kQ.png" /></figure><p>Well done! Now, let’s get to those infrastructure changes! 😎</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*YziUMpboJCpfmfXk.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation and help you learn more about the exciting world of cloud computing.</p><p>Thank you for reading, and I look forward to hearing from you soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d244f390cfca" width="1" height="1" alt=""><hr><p><a href="https://towardsaws.com/using-ssh-to-clone-an-aws-codecommit-repository-d244f390cfca">Using SSH to Clone an AWS CodeCommit Repository</a> was originally published in <a href="https://towardsaws.com/">Towards AWS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Two-Tier Architecture in AWS using Terraform]]></title>
            <link>https://aws.plainenglish.io/building-a-two-tier-architecture-in-aws-using-terraform-77e0815bb495?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/77e0815bb495</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Wed, 19 Apr 2023 11:53:04 GMT</pubDate>
            <atom:updated>2023-04-20T14:14:36.547Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Os_acqgU7dyBsjSajY3VUw.png" /></figure><p>In this project, we will be working with AWS and Terraform some of my favorite tools thus far on my cloud engineering journey. If you’re anything like me and you love infrastructure and automating everything you can get your hands on, this article is for you.</p><p><strong>What is Terraform?</strong></p><p>Terraform is one of the leading Infrastructure as Code tools on the market today. What makes Terraform so popular is the fact that it is cloud-agnostic meaning that it will work with AWS, Azure, GCP, and others. In short, Terraform automates the process of creating and managing resources.</p><p><strong>What is a two-tier architecture?</strong></p><p>A two-tier architecture contains two tiers, one being the web tier which will have your web server running on it, and the interface that will interact with the users and process the input. The database tier is responsible for processing and storing the data. The two tiers will then communicate over the network exchanging said data.</p><p>This architecture is usually good for small to medium-sized applications that don’t need a massive infrastructure to support it.</p><p><strong>Pre-Requisites</strong></p><ul><li>An AWS account</li><li>An IDE (preferably Cloud 9)</li><li>Basic knowledge of Linux commands</li><li>Basic AWS knowledge</li><li>Terraform basics</li><li>Github account</li><li>Terraform Cloud account</li></ul><p><strong>Tasks</strong></p><ul><li>Create a custom VPC with 2 public and 2 private subnets</li><li>Create and configure the appropriate route tables</li><li>Launch an EC2 instance in each public subnet</li><li>Launch one RDS instance in a public subnet</li><li>Create and configure security groups for resources</li><li>Use a CI/CD tool for your build</li><li>Push your code to GitHub</li></ul><p><em>If that’s not enough, you can always add more!</em></p><p><strong>Step 1: Setting up a development environment</strong></p><p>For the simplicity of this demo, I will be using Cloud 9 as a development environment. It is an IDE that is offered by Amazon and is also running Linux which will make all the commands that I mention in this tutorial Linux based.</p><p>So, if you decide to use an alternate code editor on your local machine of choice, just note that the commands will vary depending on the operating system and code editor that you choose.</p><p>Follow this <a href="https://docs.aws.amazon.com/cloud9/latest/user-guide/setup-express.html">documentation</a> to get started with Cloud 9.</p><p>Now that we have an environment to write the code for our Terraform files, we will need to create a repository for the files to live in.</p><p><em>To create a directory enter the following command:</em></p><pre>mkdir &lt;name of directory&gt;</pre><p>To change into the newly created directory enter the following command:</p><pre>cd &lt;name of directory&gt;</pre><p>Great, now that we have a directory to work in we can start creating our infrastructure using Terraform.</p><p><strong>Step 2: Setting up Terraform Cloud</strong></p><p>As mentioned in the pre-requisites you will need to have a Terraform cloud account for this project. It is free and you can set up an account <a href="https://app.terraform.io/public/signup/account">here</a>.</p><p>Terraform Cloud will be acting as our remote backend so it is crucial that we use it. This means that it will be keeping track of our state so that Terraform is aware of what is being created and what resources are being managed.</p><p><strong><em>To log in enter the following command:</em></strong></p><pre>terraform login</pre><p>Terraform will now request an API token for you to use. Enter “yes.”</p><p>Open the link as shown in the photo below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*zVOVzI-CjnJuDMcHMchCng.png" /></figure><p>Give the token a description.</p><p>Copy the token into the terminal and press “enter.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*l8bw2b18l3MUar9pIa1faQ.png" /></figure><p>If done correctly you should see the Terraform logo on your screen with directions on how to use Terraform Cloud. Feel free to check this out if you have time.</p><p>Next, we can create an organization back in the Terraform Cloud console.</p><p>To do so navigate to the organization page and then click “create an organization.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*jC_F8nc8QWrd6pFi2ez6xA.png" /></figure><p>After your organization has been created successfully create a workspace as well. Workspaces allow you to deploy resources quickly in different environments. For example, you can have a workspace for Development, Testing, and Production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*S8JIU0wzcOUrBe9ZhUcUZA.png" /></figure><p>Select the “CLI-driven workflow” and create the workspace by following the prompts.</p><p>In order to communicate with AWS to create resources we will need to set the access key and secret access key. If you don’t have the keys configured yet you can do so in IAM with the user that you will be working with under the “security credentials tab.”</p><p>Once you have your access keys ready, find the variable sets tab in your organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zn5p37kQQoKxwklfDnR8yQ.png" /></figure><p>Then click the “Create variable set” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*NIayrkSB8A3pELtJo-m0vg.png" /></figure><p>Enter the necessary information and add the variables for AWS to Terraform Cloud.</p><p>Then click “create variable set.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*tLNEcvqDIGPQ4AEZlmE1ZQ.png" /></figure><p>That’s it! On to the next task.</p><p><strong>Step 3: Configuring providers and remote backend</strong></p><p>There are many providers that you can use in Terraform. Today we will mainly be focusing on AWS. A provider is a plug-in that lets Terraform manage an external API.</p><p>By using Terraform providers, infrastructure teams can write a single configuration file that defines the desired state of their entire infrastructure, regardless of the underlying system, and then use Terraform to manage it in a consistent, repeatable manner.</p><p>In this section, we will name our providers as well as version them so that we are using our desired version of the provider.</p><p>We will also set up our remote backend to house our state files which is why we set up our Terraform Cloud organization previously.</p><p>First, we will need to create a file.</p><p><strong><em>To create a file for our providers and remote backend enter the following command:</em></strong></p><pre>touch providers.tf</pre><p>Below I will add my code. Feel free to use it if you are following this demo if you need other versions you will need to make the adjustments accordingly.</p><pre><br>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}<br><br>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}<br><br>terraform {<br>  cloud {<br>    organization = &quot;two_tier&quot;  # enter you organization name here<br>    <br>    workspaces {<br>      name = &quot;two-tier-dev&quot;    # enter you workspaces name here<br>    }<br>  }<br>}<br><br></pre><p><strong><em>Enter the following commands:</em></strong></p><pre>terraform init<br>terraform apply </pre><p>If you look into the state tab to the left on the workspaces page you can see that Terraform Cloud is now tracking the state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IrT5YBVplROtmFYCJk0u6g.png" /></figure><p><strong>Step 4: Networking</strong></p><p>We will be creating a file with all of the necessary resources that we need to configure our network. This includes the VPC, subnets, route tables, and route table associations.</p><p><strong><em>Enter the following command to create a file:</em></strong></p><pre>touch networking.tf</pre><p>Copy the following code into the file and save it.</p><pre>###################################################################<br>#        VPC<br>##################################################################<br>resource &quot;aws_vpc&quot; &quot;vpc&quot; {<br>  cidr_block = var.vpc_cidr<br>  tags = {<br>    Name = &quot;vpc&quot;<br>  }<br>}<br>###################################################################<br>#        Internet Gateway<br>##################################################################<br>resource &quot;aws_internet_gateway&quot; &quot;internet_gateway&quot; {<br>  vpc_id = aws_vpc.vpc.id<br>  tags = {<br>    Name = &quot;igw&quot;<br>  }<br>}<br><br>###################################################################<br>#        SUBNETS<br>##################################################################<br><br>data &quot;aws_availability_zones&quot; &quot;available_az&quot; {}<br><br>resource &quot;aws_subnet&quot; &quot;public_subnet1&quot; {<br><br>  vpc_id                  = aws_vpc.vpc.id<br>  cidr_block              = var.subnet1_cidr<br>  map_public_ip_on_launch = true<br>  availability_zone       = data.aws_availability_zones.available_az.names[0]<br>  depends_on              = [aws_vpc.vpc]<br><br>  tags = {<br>    Name = &quot;public_subnet1&quot;<br>  }<br>}<br><br>resource &quot;aws_subnet&quot; &quot;public_subnet2&quot; {<br><br>  vpc_id                  = aws_vpc.vpc.id<br>  cidr_block              = var.subnet2_cidr<br>  map_public_ip_on_launch = true<br>  availability_zone       = data.aws_availability_zones.available_az.names[1]<br>  depends_on              = [aws_vpc.vpc]<br><br>  tags = {<br>    Name = &quot;public_subnet2&quot;<br>  }<br>}<br><br>resource &quot;aws_subnet&quot; &quot;private_subnet1&quot; {<br><br>  vpc_id                  = aws_vpc.vpc.id<br>  cidr_block              = var.subnet3_cidr<br>  map_public_ip_on_launch = true<br>  availability_zone       = data.aws_availability_zones.available_az.names[0]<br>  depends_on              = [aws_vpc.vpc]<br><br>  tags = {<br>    Name = &quot;private_subnet1&quot;<br>  }<br>}<br><br>resource &quot;aws_subnet&quot; &quot;private_subnet2&quot; {<br><br>  vpc_id                  = aws_vpc.vpc.id<br>  cidr_block              = var.subnet4_cidr<br>  map_public_ip_on_launch = true<br>  availability_zone       = data.aws_availability_zones.available_az.names[1]<br>  depends_on              = [aws_vpc.vpc]<br><br>  tags = {<br>    Name = &quot;private_subnet2&quot;<br>  }<br>}<br><br>##############################################################################<br>#                   Route Tables<br>##############################################################################<br><br>resource &quot;aws_route_table&quot; &quot;public_route_table&quot; {<br>  vpc_id = aws_vpc.vpc.id<br><br>  route {<br>    cidr_block = var.route_table_cidr<br>    gateway_id = aws_internet_gateway.internet_gateway.id<br>  }<br><br>  tags = {<br>    Name = &quot;public_route_table&quot;<br>  }<br>}<br><br>resource &quot;aws_route_table_association&quot; &quot;public_subnet1_association&quot; {<br>  subnet_id      = aws_subnet.public_subnet1.id<br>  route_table_id = aws_route_table.public_route_table.id<br>}<br><br>resource &quot;aws_route_table_association&quot; &quot;public_subnet2_association&quot; {<br>  subnet_id      = aws_subnet.public_subnet2.id<br>  route_table_id = aws_route_table.public_route_table.id<br>}</pre><p>Notice that some of the values have variables mentioned in them. We will need to create an alternative file to store our variables in. There are many ways to organize yourself with Terraform and this may not be the best practice to store everything in one file, especially if you want to move to using modules in the future. That’s a story for another day however!</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch variables.tf</pre><p>Copy and paste the code from below into the newly created file.</p><pre>########################################################################<br>#   VPC Variables<br>########################################################################<br><br>variable &quot;vpc_cidr&quot; {<br>  type    = string<br>  default = &quot;10.0.0.0/16&quot;<br>}<br><br>###################################################################<br>#    Subnet Variables <br>##################################################################<br><br><br>variable &quot;subnet1_cidr&quot; {<br>  type        = string<br>  description = &quot;Subnet 1 cidr block&quot;<br>  default     = &quot;10.0.1.0/24&quot;<br>}<br><br>variable &quot;subnet2_cidr&quot; {<br>  type        = string<br>  description = &quot;Subnet 2 cidr block&quot;<br>  default     = &quot;10.0.2.0/24&quot;<br>}<br><br>variable &quot;subnet3_cidr&quot; {<br>  type        = string<br>  description = &quot;Subnet 3 cidr block&quot;<br>  default     = &quot;10.0.3.0/24&quot;<br>}<br><br>variable &quot;subnet4_cidr&quot; {<br>  type        = string<br>  description = &quot;Subnet 4 cidr block&quot;<br>  default     = &quot;10.0.4.0/24&quot;<br>}<br><br>##############################################################################<br>#  Route Tables Variables<br>##############################################################################<br><br>variable &quot;route_table_cidr&quot; {<br>  type        = string<br>  description = &quot;cidr block for public route table&quot;<br>  default     = &quot;0.0.0.0/0&quot;<br>}</pre><p><strong><em>NOTE: Make sure to check your work by using the “terraform apply” command. I will not be applying everything until the end for demonstration purposes.</em></strong></p><p><strong>Step 5: Web Servers</strong></p><p>We will be focusing on getting the web servers ready for launch in this section. If you remember from the task we will be creating two EC2 instances in different availability zones. This will allow us to have high availability in case one server happens to go down, we have a backup in an alternate availability zone.</p><p>As usual, we will create a file to keep our EC2 instance code in.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch ec2_instances.tf</pre><p>Copy the following code into the file.</p><pre>resource &quot;aws_instance&quot; &quot;instance1&quot; {<br>  ami                    = var.ami_id<br>  instance_type          = var.instance_type<br>  vpc_security_group_ids = [aws_security_group.web_server_sg.id]<br>  user_data              = file(&quot;apache.sh&quot;)<br>  subnet_id              = aws_subnet.public_subnet1.id<br>  key_name               = &quot;two_tier&quot;<br><br>  tags = {<br>    Name = &quot;instance1&quot;<br>  }<br>}<br><br>resource &quot;aws_instance&quot; &quot;instance2&quot; {<br>  ami                    = var.ami_id<br>  instance_type          = var.instance_type<br>  vpc_security_group_ids = [aws_security_group.web_server_sg.id]<br>  user_data              = file(&quot;apache.sh&quot;)<br>  subnet_id              = aws_subnet.public_subnet2.id<br><br>  tags = {<br>    Name = &quot;instance2&quot;<br>  }<br>}</pre><p>Then copy the following code into the “variables.tf” file that we created. You can also create separate files for each of your resources as well if you wish.</p><pre>################################################################################<br>#                   EC2 Instance Variables<br>################################################################################<br><br>variable &quot;ami_id&quot; {<br>  type        = string<br>  description = &quot;var for AMI ID&quot;<br>  default     = &quot;ami-069aabeee6f53e7bf&quot;<br>}<br><br>variable &quot;instance_type&quot; {<br>  type        = string<br>  description = &quot;var for instance type&quot;<br>  default     = &quot;t2.micro&quot;<br>}<br></pre><p>Notice in the “ec2_instances.tf” file under the user data field, we are pointing to a script. We will need to create a file and add the script.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch apache.sh</pre><p>Now copy the code below into the file and save it.</p><pre>#!/bin/bash<br><br># update packages<br>sudo yum update<br><br># apache install, enable, and status check<br>sudo yum -y install httpd<br>sudo systemctl start httpd<br>sudo systemctl enable httpd<br>sudo systemctl status httpd</pre><p>This will run a script at startup installing and starting an apache web server.</p><p>That’s all for the EC2 instances.</p><p><strong>Step 6: Database</strong></p><p>We will be using an RDS database in today’s demo. Amazon Relational Database Service (RDS) is managed SQL database server offered by AWS. It will be processing and storing the data from the web tier.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch database.tf</pre><p>Copy the following code into the file.</p><pre><br>resource &quot;aws_db_subnet_group&quot; &quot;subnet_group&quot; {<br>  name       = &quot;subnet_group&quot;<br>  subnet_ids = [aws_subnet.private_subnet2.id, aws_subnet.private_subnet1.id]<br><br>  tags = {<br>    Name = &quot;My DB subnet group&quot;<br>  }<br>}<br><br><br>resource &quot;aws_db_instance&quot; &quot;instance_db&quot; {<br>  allocated_storage    = 10<br>  db_name              = &quot;instance_db&quot;<br>  engine               = &quot;mysql&quot;<br>  engine_version       = &quot;5.7&quot;<br>  instance_class       = &quot;db.t2.micro&quot;<br>  username             = &quot;admin&quot;    <br>  password             = &quot;password&quot;<br>  skip_final_snapshot  = true<br>  publicly_accessible = false<br>  vpc_security_group_ids = [aws_security_group.database_sg.id]<br>  availability_zone      = data.aws_availability_zones.available_az.names[0]<br>  db_subnet_group_name = &quot;subnet_group&quot;<br>  port = &quot;3306&quot;<br>  <br>  }</pre><p><strong>NOTE: Notice that the password and username are in plain text. There are a variety of different ways to work around this but for the demos purpose, we will keep them as is.</strong></p><p><strong>Step 6: Security Groups</strong></p><p>If you are not familiar with what security groups are in AWS, you can think of them as firewalls. In this section, we will create a security group for both our EC2 instances and our database.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch security_groups.tf</pre><p>Copy the following code into the file then save.</p><pre>#security group for launch_template<br>#allowing http and ssh<br>resource &quot;aws_security_group&quot; &quot;web_server_sg&quot; {<br>  name        = &quot;web_server_sg&quot;<br>  description = &quot;Allow traffic&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br><br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br>}<br><br>resource &quot;aws_security_group&quot; &quot;database_sg&quot; {<br>  name        = &quot;database_sg&quot;<br>  description = &quot;Allow traffic&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br>  ingress {<br>    from_port       = 3306<br>    to_port         = 3306<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.web_server_sg.id]<br>    <br>    <br>   <br>  }<br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [var.cidr_open]<br><br>  }<br>}<br><br></pre><p>This will allow specific traffic to reach our intended resources. You can also make the code more efficient by adding some of your own environment variables with the code in this block.</p><p><strong>Step 7: Apply Infrastructure</strong></p><p>You may have already added some of the Infrastructure by entering the “terraform apply” command and that’s a good thing. As I mentioned previously it’s best to build as you go to make troubleshooting easier when it arrives and trust me, it will.</p><p>In this section, we are going to run the code from the Terraform Cloud console just so you can see how it looks there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QFYTJlOHlJRhLddXhgnzaw.png" /></figure><p>In the actions tab select “Start new run”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*krHFs8qfx-XQx5lYtNykpw.png" /></figure><p>You’ll be prompted with a full plan of all the resources that you plan on applying to your AWS account.</p><p>Click “Confirm &amp; Apply.” Then add a description.</p><p>Once your resources have all successfully been created you can check them in the console and make sure everything looks correct. I had the database fail to apply but was successful on the second attempt.</p><p><strong>Step 8: Verify</strong></p><p>If there is anything that I’ve learned in IT, is always verify, and never assume. Sure everything looks good but can we access it? Are the right resources running? In this section, we will check to see if we can access our web server and our database.</p><p>Navigate to the EC2 dashboard and look for the instances that we just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*d0vTv36d6YTyHKEFif-AKQ.png" /></figure><p>Enter the Public IP address in your browser.</p><pre>PublicIP:80</pre><p>If done successfully you should see the following page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F--8qzRQk0zU4g5FCRHgUQ.png" /></figure><p>Now, let’s check the connection to the database. You will need to install mariadb onto the instance in the web tier. You can find documentation <a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToInstance.html">here</a>.</p><p><strong><em>To install mariadb enter the following command:</em></strong></p><pre>sudo yum install mariadb</pre><p>Once that is finished we will need to find the RDS endpoint in the console. Navigate to the RDS dashboard and select the database that we created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s83eGBmmj5wnORWbLOO7Gg.png" /></figure><p>Click on the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*MpBM-GsBXjfPbWwR_LuTzw.png" /></figure><p>There you will find the endpoint you need for the next command.</p><p><strong><em>Enter the following command to log into the database:</em></strong></p><pre>mysql -h &lt;mysql–instance1.123456789012.us-east-1.rds.amazonaws.com&gt; -P 3306 -u &lt;username&gt; -p</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6200dzqG4oc7UuCxMM0MA.png" /></figure><p>We have successfully logged into the RDS server from our web tier. Congratulations, we are almost there!</p><p><strong>Step 9: Version control</strong></p><p>One of our tasks was to deploy our resources using version control. In this demo, we will be using GitHub to accomplish this. We will need to create a new repository in Github. If you don’t know how to create a repository in GitHub you can find the information in an article that I wrote previously <a href="https://medium.com/towards-aws/from-fork-to-push-a-funky-guide-to-github-f1894bb50693">here</a>.</p><p>Next, we will need to create a connection between Terraform Cloud and our newly created repository.</p><p>Navigate to the setting in the workspace and look for the “version control” tab. Then click “connect to version control.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUXUoXY7wwwRtJsr_SSAWg.png" /></figure><p>On the next page select “GitHub” as the version control.</p><p>You may need to disable your pop-up blocker if you have one configured. If you are like me and hate ads this will more than likely be the case.</p><p>Follow the prompts and make sure that you select the “Only select repositories” button and select the repository that you just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*7xk4lF2-UYCJXkhpW0D8MA.png" /></figure><p>Click “Install”</p><p>Then configure as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1Cx98Op45_dkpZ5JiTxLg.png" /></figure><p>The next step is to push all your code to the GitHub repo that you created. I had some issues with this and eventually just created a new IDE and reconfigured git and pushed the files over.</p><p>I then ran into the problem of Terraform Cloud attempting to apply the changes right away so I needed to change the settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*6c3SKU4JH5dLqeIzvn5StQ.png" /></figure><p>If you run into an issue as well make sure and change to the “manual apply” button until you get all of your code into the repository.</p><p>Now, let’s add a resource and see if it triggers an action.</p><p>Enter the following code into the “ec2_instances.tf” file. Save and then push to Github.</p><pre>resource &quot;aws_instance&quot; &quot;instance3&quot; {<br>  ami                    = var.ami_id<br>  instance_type          = var.instance_type<br>  vpc_security_group_ids = [aws_security_group.web_server_sg.id]<br>  user_data              = file(&quot;apache.sh&quot;)<br>  subnet_id              = aws_subnet.public_subnet2.id<br>  <br><br>  tags = {<br>    Name = &quot;instance3&quot;<br>  }<br>}</pre><p>Once you merge the branch into main you will notice the status change in Terraform Cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*YEqt9XPdVZ43Wb41jnfYXA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*c1OTdtUJP8FzX2XVedMZgA.png" /></figure><p>Let’s check our console to make sure that the instance has been launched.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJdo5nlFwLud6f9sXcWPkw.png" /></figure><p>There you have it. Hopefully, you can see the power of CI/CD pipelines here. We added a resource block to our code and it automatically applied the changes.</p><p><strong>Step 10: Destroy</strong></p><p>A sad but necessary step is to destroy all the resources that we worked so hard to create. Navigate to the workspace settings page in Terraform Cloud and to the “Destruction and Deletion” tab in the Workspace settings. Follow the prompts and queue a destroy. The resources will then start to destroy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*7-IAnFCH8fmUgUSPCUv93Q.png" /></figure><p>Once the resources have been deleted successfully go back to the console just to confirm.</p><p>Now that was a project!</p><p>I hope you enjoyed it and learned something new I had many issues along the way but also learned some valuable lessons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWLpfe9VwMhR-O0xDnkefg.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation and help you learn more about the exciting world of cloud computing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77e0815bb495" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/building-a-two-tier-architecture-in-aws-using-terraform-77e0815bb495">Building a Two-Tier Architecture in AWS using Terraform</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configuring a Highly Available Infrastructure in AWS using Terraform]]></title>
            <link>https://faun.pub/configuring-a-highly-available-infrastructure-in-aws-using-terraform-2fc9dbb519b6?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fc9dbb519b6</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Fri, 07 Apr 2023 12:05:49 GMT</pubDate>
            <atom:updated>2023-04-11T14:26:23.128Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4dh20tQPOzriuu8KAZe8iw.png" /></figure><p>Welcome back I hope you are ready for another fun-filled tutorial with Terraform. You’ve heard the buzzwords “fault-tolerant” and “highly available” over and over, now it’s time to finally build it. Follow along and let’s put a plan to action using Infrastructure as Code with Terraform.</p><p><strong>What is Terraform?</strong></p><p>Terraform is one of the leading Infrastructure as Code tools on the market today. What makes Terraform so popular is the fact that it is cloud-agnostic meaning that it will work with AWS, Azure, GCP, and others. In short, Terraform automates the process of creating and managing resources.</p><p><strong>What is an Auto-scaling group?</strong></p><p>An AWS Auto Scaling Group is a service provided by Amazon Web Services (AWS) that automatically scales up or down a group of EC2 instances based on demand.</p><p>Auto Scaling Groups can help maintain the availability and performance of your applications while reducing costs by only running the required number of instances.</p><p>Auto Scaling Groups work by defining a minimum and maximum number of instances in a group, and then scaling the number of instances based on the demand. When demand is low, instances are removed to save costs, and when demand is high, instances are added to maintain performance.</p><p>What is an Application Load Balancer?</p><p>An Application Load Balancer (ALB) is a type of load balancer that distributes incoming traffic across multiple targets such as EC2 instances, containers, IP addresses, and Lambda functions within a single Availability Zone or multiple Availability Zones.</p><p><strong>Pre-Requisites</strong></p><ul><li>An AWS account</li><li>An IDE (preferably Cloud 9)</li><li>Basic knowledge of Linux commands</li><li>Basic AWS knowledge</li></ul><p><strong>Tasks</strong></p><ul><li>Create an S3 bucket and set it as your remote backend</li><li>Launch an Auto Scaling Group that spans 2 subnets in your default VPC</li><li>Include a script in your user data to launch an Apache web server. The Auto Scaling group should have a min of 2 and a max of 5</li><li>Add an ALB in front of the Auto Scaling group</li><li>Create a security group for the ALB that allows traffic from the internet and associate it with the ALB</li><li>Configure the Auto Scaling group security group to allow only traffic from the ALB</li><li>Output the public DNS name of the ALB and verify you are able to reach your web servers from your browser</li></ul><p><em>Great, you’re ready to get started!</em></p><p><strong>Step 1: Setting up a development environment</strong></p><p>For the simplicity of this demo, I will be using Cloud 9 as a development environment. It is an IDE that is offered by Amazon and is also running Linux which will make all the commands that I mention in this tutorial Linux based.</p><p>So, if you decide to use an alternate code editor on your local machine of choice, just note that the commands will vary depending on the operating system and code editor that you choose.</p><p>Follow this <a href="https://docs.aws.amazon.com/cloud9/latest/user-guide/setup-express.html">documentation</a> to get started with Cloud 9.</p><p>To start working with AWS from your IDE you will need to provide access keys so that you can communicate with AWS. You can decide which route you want to go by reading the documentation <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs">here</a>.</p><p>I find the simplest way is just to use the export command locally.</p><p>Now that we have an environment to write the code for our Terraform files, we will need to create a repository for the files to live in.</p><p><strong><em>To create a directory enter the following command:</em></strong></p><pre>mkdir &lt;name of directory&gt;</pre><p>To change into the newly created directory enter the following command:</p><pre>cd &lt;name of directory&gt;</pre><p>Great, now that we have a directory to work in we can start creating our infrastructure using Terraform.</p><p><strong>Step 2: Naming our Providers</strong></p><p>There are many providers that you can use in Terraform. Today we will mainly be focusing on AWS. A provider is a plug-in that lets Terraform manage an external API.</p><p>By using Terraform providers, infrastructure teams can write a single configuration file that defines the desired state of their entire infrastructure, regardless of the underlying system, and then use Terraform to manage it in a consistent, repeatable manner.</p><p>In this section, we will name our providers as well as version them so that we are using our desired version of the provider.</p><p>First, we will need to create a file.</p><p><strong><em>To create a file for our providers enter the following command:</em></strong></p><pre>touch providers.tf</pre><p>Below I will add my code. Feel free to use it if you are following this demo if you need other versions you will need to make the adjustments accordingly.</p><pre>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}<br><br>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}</pre><p><strong><em>Enter the following command:</em></strong></p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5L85H9kA6yO1Su6ijCB03g.png" /></figure><p><strong>Note: We will need to add one in a later step I will explain why there.</strong></p><p><strong>Step 3: Set an S3 bucket as the remote backend</strong></p><p>By default, Terraform stores its state in a state file which is stored locally. This is how Terraform knows what it is managing and what changes are being made to the state and so on.</p><p>The issue with the files being stored locally however is that no one from your team can easily access the state file so things tend to become difficult very quickly when working on projects as a team.</p><p>For further information check the documentation <a href="https://developer.hashicorp.com/terraform/language/settings/backends/s3">here</a>.</p><p>Let’s create a file for our code.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch s3.tf</pre><p>Now add the HCL code to the file so that we can create our s3 bucket.</p><pre>#creating a s3 bucket <br>#enter a unique bucket name here<br>resource &quot;aws_s3_bucket&quot; &quot;project_s3&quot; {<br>  bucket = &quot;testbucket112286699&quot; <br>}<br><br>#attaching a bucket policy to give terraform permissions<br>resource &quot;aws_s3_bucket_policy&quot; &quot;bucket_policy&quot; {<br>  bucket = &quot;testbucket112286699&quot;<br>  policy = file(&quot;policy.json&quot;)<br>}<br><br>#enabeling bucket versioning for recovery<br>resource &quot;aws_s3_bucket_versioning&quot; &quot;project_versioning&quot; {<br>  bucket = aws_s3_bucket.project_s3.id<br>  versioning_configuration {<br>    status = &quot;Enabled&quot;<br>  }<br>}<br><br>#setting the acl of the bucket to private<br>resource &quot;aws_s3_bucket_acl&quot; &quot;project_acl&quot; {<br>  bucket = aws_s3_bucket.project_s3.id<br>  acl    = &quot;private&quot;<br>}</pre><p>Enter the following command:</p><pre>terraform apply</pre><p>Each of the blocks of code are conveniently commented for you to understand what they are doing. If you want to look more into the Terraform documentation for creating an S3 bucket, you can find it <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket">here</a>.</p><p>In the second block of code, you will notice that it is referencing a json policy. These are the permissions that Terraform will need to perform specific actions to the bucket.</p><p>First, we will need to create a file to reference the policy.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch policy.json</pre><p>Now copy the policy into the file and save.</p><pre><br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: &quot;*&quot;,<br>      &quot;Action&quot;: &quot;s3:ListBucket&quot;,<br>      &quot;Resource&quot;: &quot;arn:aws:s3:::****YOUR_BUCKET_NAME*****&quot;<br>    },<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: &quot;*&quot;,<br>      &quot;Action&quot;: [&quot;s3:GetObject&quot;, &quot;s3:PutObject&quot;, &quot;s3:DeleteObject&quot;],<br>      &quot;Resource&quot;: &quot;arn:aws:s3:::****YOUR_BUCKET_NAME*****/dev/terraform.tfstate&quot;<br>   }<br>  ]<br>}</pre><p>If you look at the line of code with “Principle” stated this is basically the IAM user that will have the privileges. For ease of this demonstration, I have put the “ * ” which means everyone will have access. This is not recommended in a production environment and I recommend replacing it with your ARN.</p><p>Now that we have the file saved when we run the command Terraform will automatically look for the “policy.json” file and use it to grant permissions.</p><p>We need to add one more file so that our configuration files can use the variables in the file.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch variables.tf</pre><p>This will house all the variables that we will be using throughout the project.</p><p>Copy the following code into the variables.tf file and save.</p><pre>variable &quot;vpc_id&quot; {<br>  type        = string<br>  description = &quot;var for VPC ID&quot;<br>  default     = &quot;*****Enter your VPC ID here********&quot;<br>}<br><br>variable &quot;cidr_open&quot; {<br>  type        = string<br>  description = &quot;CIDR block to allow traffic from anywhere&quot;<br>  default     = &quot;0.0.0.0/0&quot;<br>}<br><br>variable &quot;cidr_vpc&quot; {<br>  type        = string<br>  description = &quot;CIDR block to allow traffic from local machine&quot;<br>  default     = &quot;*****Enter your default VPC CIDR block**********&quot;<br>}<br><br><br>variable &quot;instance_type&quot; {<br>  type        = string<br>  description = &quot;var for instance type&quot;<br>  default     = &quot;t2.micro&quot;<br>}<br><br>variable &quot;ami_id&quot; {<br>  type        = string<br>  description = &quot;var for AMI ID&quot;<br>  default     = &quot;ami-04581fbf744a7d11f&quot;<br>}<br><br>variable &quot;availability_zone_a&quot; {<br>  type    = string<br>  default = &quot;us-east-1a&quot;<br>}<br><br>variable &quot;availability_zone_b&quot; {<br>  type    = string<br>  default = &quot;us-east-1b&quot;<br>}</pre><p>Great without this file Terraform can’t find some of the variables stated in the configuration.</p><p>Let’s go ahead and run this code and see if we can get our remote backend set up.</p><p>With Terraform there is a core workflow that we will follow for each of our deployments. Some listed are optional.</p><pre>terraform init     #initiates the directory<br>terraform fmt      #formats the code <br>terraform validate #validates the code is correct in terms of syntax<br>terraform plan     #provides a plan for the build<br>terraform apply    #creates the desired infrastructure<br>terraform destroy  #destroys all infrastructure </pre><p>Make sure you are in the directory that you previously created to work in and that they are saved and let’s start.</p><p><strong><em>Enter the following command:</em></strong></p><pre>terraform apply</pre><p>You may need to run this twice if you get an error that the bucket does not exist.</p><p>Ok, we should have our newly created remote backend with our state file in our S3 bucket. Let’s check the console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*zExe761nX7ICslDWkPw8PA.png" /></figure><p>We have a bucket but no file hmm……</p><p>This is because we need to add the S3 bucket as the backend in the provider.tf file but you can only do this once the bucket has been created.</p><p><strong>Note: You can just create the S3 bucket in the console or via the CLI to save the headache, but what fun is that!</strong></p><p>Head back to the “providers.tf” file and add the following code.</p><pre>#creating up a remote state backend in s3<br>terraform {<br>  backend &quot;s3&quot; {<br>    bucket = &quot;****YOUR_BUCKET_NAME*****&quot;<br>    key    = &quot;dev/terraform.tfstate&quot;<br>    region = &quot;us-east-1&quot;<br>  }<br>}</pre><p>Then save it.</p><p>Enter the following command:</p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*D7GpPHSBMib33syPKVLKQw.png" /></figure><p>You will then be prompted to migrate the state to the s3 bucket you can enter “yes”.</p><p>Now for confirmation in the S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*TVLW2IChlpAMhfiUutlTwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*c2NZ3dOuLHYV1xFSFsnpyg.png" /></figure><p>Great, we now have a remote backend. This will store our state and we can share it with a remote team.</p><p><strong><em>Step 4: Launch an Auto-Scaling Group and Load Balancer</em></strong></p><p>The whole purpose of this article is to get some web servers up and running and make sure that they are highly available as well as fault tolerant. In this section, we will do just that by using Terraform to deploy all the necessary components.</p><p>First we will need a file for the configuration of the Auto Scaling Group and the Load Balancer.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch acg_alb.tf</pre><p>Copy the code below into the file.</p><pre>#creating a launch template with ubuntu ami and script for install<br>resource &quot;aws_launch_template&quot; &quot;launch_template&quot; {<br>  name                   = &quot;launch_template&quot;<br>  image_id               = var.ami_id<br>  instance_type          = var.instance_type<br>  user_data              = filebase64(&quot;apache.sh&quot;)<br>}<br><br>#creating an auto scaling group<br>resource &quot;aws_autoscaling_group&quot; &quot;asg_group&quot; {<br>  name               = &quot;asg_group&quot;<br>  min_size           = 2<br>  max_size           = 5<br>  availability_zones = [var.availability_zone_a, var.availability_zone_b]<br>  target_group_arns = [aws_lb_target_group.alb-targetgroup.arn]<br><br><br>  launch_template {<br>    id = aws_launch_template.launch_template.id <br>    version = &quot;$Latest&quot;<br>  }<br>}<br><br>#creating an application load balancer<br>resource &quot;aws_lb&quot; &quot;alb-project&quot; {<br>  name               = &quot;alb-project&quot;<br>  internal           = false<br>  load_balancer_type = &quot;application&quot;<br>  subnets            = [&quot;subnet-026b3cb418d28e844&quot;, &quot;subnet-0f9cfcd126a7dcfa7&quot;]<br><br>}<br><br>#configuring and creating the listener for the lb<br>resource &quot;aws_lb_listener&quot; &quot;front_end&quot; {<br>  load_balancer_arn = aws_lb.alb-project.arn<br>  port              = 80<br>  protocol          = &quot;HTTP&quot;<br><br>  default_action {<br>    type             = &quot;forward&quot;<br>    target_group_arn = aws_lb_target_group.alb-targetgroup.arn<br>  }<br>}<br><br><br>#creating the alb target group<br>resource &quot;aws_lb_target_group&quot; &quot;alb-targetgroup&quot; {<br>  name        = &quot;alb-targetgroup&quot;<br>  port        = 80<br>  protocol    = &quot;HTTP&quot;<br>  vpc_id      = var.vpc_id<br>}</pre><p>Notice that in the launch template resource, we are pointing to a script. We will need to create a file and add the script.</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch apache.sh</pre><p>Now copy the code below into the file.</p><pre>#!/bin/bash<br><br># update packages<br>sudo yum update<br><br># apache install, enable, and status check<br>sudo yum -y install httpd<br>sudo systemctl start httpd<br>sudo systemctl enable httpd<br>sudo systemctl status httpd</pre><p>This will now install Apache and enable it on our servers. Save all the files that we just added to our directory.</p><p><strong><em>Enter the following commands:</em></strong></p><pre>terraform fmt<br>terraform validate<br>terraform plan<br>terraform apply<br></pre><p>When asked to enter a value after entering “terraform apply” enter yes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*e7_oMQ2An2CcecyA9heN9w.png" /></figure><p>If successful your resources will be added.</p><p><strong><em>Note: We will need to come back and add security groups to some of these resources.</em></strong></p><p><strong>Step 5: Create the Security Groups</strong></p><p>If you are not familiar with what security groups are in AWS, you can think of them as firewalls. In this section, we will create a security group for both our load balancer and the instances that are being spun up from our launch template.</p><p>Enter the following command to create two files:</p><pre>touch sg_alb.tf<br>touch sg_lt.tf</pre><p>Enter the code below in the “sg_alb.tf” file and save it.</p><pre>#security group for the application load balancer<br>#allowing http and https<br>resource &quot;aws_security_group&quot; &quot;sg_alb&quot; {<br>  name        = &quot;sg_alb&quot;<br>  description = &quot;Allow traffic&quot;<br>  vpc_id      = var.vpc_id<br><br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  ingress {<br>    description = &quot;https&quot;<br>    from_port   = 443<br>    to_port     = 443<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br>   egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br>}</pre><p>Enter the code below into the “sg_lt.tf” file and save it.</p><pre>#security group for launch_template<br>#allowing http and ssh<br>resource &quot;aws_security_group&quot; &quot;sg_lt&quot; {<br>  name        = &quot;sg_lt&quot;<br>  description = &quot;Allow traffic&quot;<br>  vpc_id      = var.vpc_id<br><br><br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_vpc]<br>  }<br><br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br>}</pre><p>You will need to change the variable in the “variable.tf” file to reflect the CIDR block of your default VPC. This will allow traffic only from within your VPC to communicate with the instances.</p><p>After the files have been saved we can apply them to the configuration.</p><p><strong><em>Enter the following commands:</em></strong></p><pre>terraform plan<br>terraform apply<br></pre><p>Enter “yes”. Then, if done successfully you should see that the two security groups have been added.</p><p>Now, we need to go back and add the security groups to the launch template and the load balancer. This way we will have access to the servers. we couldn’t add them before because we didn’t have the security groups configured yet.</p><p>Navigate to the “acg_alb.tf” file.</p><p><strong><em>Add the following line to the launch template:</em></strong></p><pre>vpc_security_group_ids = [aws_security_group.sg_lt.id]</pre><p><strong><em>Add the following line to the application load balancer:</em></strong></p><pre>security_groups    = [aws_security_group.sg_alb.id]</pre><p><strong><em>Enter the following commands:</em></strong></p><pre>terraform fmt<br>terraform validate<br>terraform plan<br>terraform apply</pre><p>We should have added to two resources if done successfully.</p><p><strong>Step 6: Outputs</strong></p><p>When using we sometimes need to check the configuration of resources but it gets a little bit annoying to go back to the console every time to gather the information that we need to test.</p><p>We can use an output block in our configuration which will give us our desired information right to our terminal after the command has been completed.</p><p><strong><em>To create an output file enter the following command:</em></strong></p><pre>touch outputs.tf</pre><p>Copy the code below into the file and save it.</p><pre>#output block that will give us alb dns name to verify connection<br>output &quot;alb_public_url&quot; {<br>  description = &quot;Public URL for Application Load Balancer&quot;<br>  value       = aws_lb.alb-project.dns_name<br>}</pre><p>This code will give us the public URL for our application load balancer.</p><p><strong><em>Enter the following command:</em></strong></p><pre>terraform apply</pre><p>Enter yes for a value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*IiofmD2hUZlU9hZkYTBkgA.png" /></figure><p>How convenient is that!</p><p><strong>Step 7: Testing</strong></p><p>Now, for the part we have all been waiting for, here we are going to test the resources to make sure that we can access them as expected and that they are highly available.</p><p>Grab the URL from the terminal at the end of our last apply command and put it in an incognito window in your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yp7L7-yQKMrV1egJGyQ1sQ.png" /></figure><p>Great, our server is up and running and we can access them through the load balancer.</p><p>Now, we need to check one more thing. Let’s terminate one of the instances in the auto- scaling group and check to see that another one is launched shortly after. We configured the group so that there are a minimum of two running at all times.</p><p>Navigate to the EC2 dashboard where the instances are running and stop one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n_METRHJTslOiT9LpfcPoQ.png" /></figure><p>Once you have stopped the instance go ahead and refresh the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8vL2rC4cbfrh9OMHRthwAA.png" /></figure><p>As you can see we just have one running at the moment. Keep refreshing until you see another instance appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JWRTyKjJ9yIVC5r8mjXYgg.png" /></figure><p>There we have it that’s confirmation that the auto-scaling group is in working order. Looks like we have completed and verified all tasks from our list, great job!</p><p>I hope you enjoyed this tutorial and had fun experiencing the power of Terraform. I know I did!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_v2xJ855X-fLtVkW6_HOw.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation and help you learn more about the exciting world of cloud computing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*HtjY9aXfkHknIs7n.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fc9dbb519b6" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/configuring-a-highly-available-infrastructure-in-aws-using-terraform-2fc9dbb519b6">Configuring a Highly Available Infrastructure in AWS using Terraform</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraforming  Jenkins on AWS]]></title>
            <link>https://aws.plainenglish.io/terraforming-jenkins-on-aws-e755e53a2899?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e755e53a2899</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 09:14:00 GMT</pubDate>
            <atom:updated>2023-04-05T05:18:18.862Z</atom:updated>
            <content:encoded><![CDATA[<h3>Terraforming Jenkins on AWS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pd70bKtG39iY9P3XVwiGxA.png" /></figure><p>Welcome back, in this short tutorial we will explore how to use Terraform to set up a Jenkins server on AWS. I will walk through all the necessary steps and configurations you need to get started. Setting up a Jenkins server can be a daunting task for the first time, so if you’re looking to streamline your software development process and deploy Jenkins on AWS, you’re in the right place.</p><p><strong>What is Terraform?</strong></p><p>Terraform is one of the leading Infrastructure as Code tools on the market today. What makes Terraform so popular is the fact that it is cloud-agnostic meaning that it will work with AWS, Azure, GCP, and others. In short, Terraform automates the process of creating and managing resources.</p><p><strong>What is Jenkins?</strong></p><p>Jenkins is a highly customizable automation server that is used for building, testing, and deploying projects. Jenkins focuses on continuous integration and continuous deployment which allows developers to frequently merge their code into a shared repository and have it automatically tested and deployed.</p><p>Now that you have a general understanding of some of the DevOps tools we will be working with, let’s take a look at some prerequisites and tasks for this walk-through.</p><p><strong>Pre-Requisites</strong></p><ul><li>An AWS account</li><li>An IDE (preferably Cloud 9)</li><li>Basic knowledge of Linux commands</li></ul><p><strong>Tasks</strong></p><ul><li>Deploy 1 EC2 Instance in your Default VPC.</li><li>Bootstrap the EC2 instance with a script that will install and start Jenkins.</li><li>Create and assign a Security Group to the Jenkins EC2 that allows traffic on port 22 from your ip and allows traffic from port 8080.</li><li>Create an S3 bucket for your Jenkins Artifacts that is not open to the public.</li><li>Verify that you can reach Jenkins via port 8080 in your browser.</li></ul><p><em>Now that you’re up to speed, lets go!</em></p><p><strong>Step 1: Setting up a development environment</strong></p><p>For the simplicity of this demo, I will be using Cloud 9 as a development environment. It is an IDE that is offered by Amazon and is also running Linux which will make all the commands that I mention in this tutorial Linux based.</p><p>So, if you decide to use an alternate code editor on your local machine of choice, just note that the commands will vary depending on the operating system and code editor that you choose.</p><p>Follow this <a href="https://docs.aws.amazon.com/cloud9/latest/user-guide/setup-express.html">documentation</a> to get started with Cloud 9.</p><p>Now that we have an environment to write the code for our Terraform files, we will need to create a repository for the files to live in.</p><p><strong><em>To create a directory enter the following command:</em></strong></p><pre>mkdir &lt;name of directory&gt;</pre><p>To change into the newly created directory enter the following command:</p><pre>cd &lt;name of directory&gt;</pre><p>Great, now that we have a directory to work in we can start creating our infrastructure using Terraform.</p><p><strong>Step 2: Naming our providers</strong></p><p>There are many providers that you can use in Terraform. Today we will mainly be focusing on Terraform and AWS. A provider is a plug-in that lets Terraform manage an external API.</p><p>In this section, we will name our providers as well as version them so that we are using our desired version of the provider.</p><p>First, we will need to create a file.</p><p><strong><em>To create a file for our providers enter the following command:</em></strong></p><pre>touch providers.tf</pre><p>Below I will add my code. Feel free to use it if you are following this demo if you need other versions you will need to make the adjustments accordingly.</p><pre>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}</pre><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;~&gt; 4.0&quot;<br>    }<br>  }<br>}</pre><p>You will also need to provide access key so that you can communicate with AWS you can decide which route you want to go by reading the documentation <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs">here</a>.</p><p><strong>Step 3. Creating our infrastructure</strong></p><p>Normally when building out our Terraform projects most of our configurations will be in the main.tf file. This will house most of our instances, networking, resources, etc…</p><p><strong><em>To create a file enter the following command:</em></strong></p><pre>touch main.tf</pre><p>This will be the file where our configurations will go for today’s demo.</p><p>Below is the code for our infrastructure that we will be deployed along with other resources.</p><pre>resource &quot;aws_instance&quot; &quot;jenkins&quot; {<br>  ami                    = var.ami_id<br>  instance_type          = var.instance_type<br>  vpc_security_group_ids = [aws_security_group.jenkins-sg.id]<br>  user_data              = file(&quot;jenkins.sh&quot;)<br>  tags = {<br>    Name = &quot;Jenkins_Instance&quot;<br>  }<br>}<br><br>resource &quot;aws_security_group&quot; &quot;jenkins-sg&quot; {<br>  name        = &quot;jenkins-sg&quot;<br>  description = &quot;Allow traffic&quot;<br>  vpc_id      = var.vpc_id<br><br>  ingress {<br>    description = &quot;allow traffic&quot;<br>    from_port   = 8080<br>    to_port     = 8080<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_open]<br>  }<br><br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.cidr_my]<br>  }<br><br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [var.cidr_open]<br><br>  }<br><br>  tags = {<br>    Name = &quot;jenkins-sg&quot;<br>  }<br>}<br><br>resource &quot;aws_s3_bucket&quot; &quot;jenkinsbucket&quot; {<br>  bucket = var.s3_bucket<br>}<br><br>resource &quot;aws_s3_bucket_acl&quot; &quot;jenkinsbucket_acl&quot; {<br>  bucket = var.s3_bucket<br>  acl    = &quot;private&quot;<br>}</pre><p>In the above code block, you will notice that some of the values are in fact variables. Variables are a great way to specify given values for the resources that we deploy. It also allows our configuration file to be more re-usable and flexible.</p><p>In a later step we will add another file that will contain the variables used throughout our configuration file.</p><p>If you take a look at the “user_data” field in the resource that creates our EC2 instance you will notice that it mentions a file. This file contains a script that will install Jenkins at startup. You will need to create a file in the directory we are working in.</p><p>To create a file for the script enter the following command:</p><pre>touch &lt;name of file&gt;</pre><p>Then add the code below into the file.</p><pre>  #!/bin/bash<br>  sudo yum update -y <br>  sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo<br>  sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key<br>  sudo yum upgrade <br>  sudo amazon-linux-extras install java-openjdk11 -y<br>  sudo yum install jenkins -y <br>  sudo systemctl enable jenkins<br>  sudo systemctl start jenkins</pre><p><strong>Step 4: Naming our variables</strong></p><p>In this section, we will add another file to our directory that will contain the variables that were stated in our “main.tf” file. Terraform will look for the variables in the directory that we deploy from.</p><p><strong><em>To create a file for the variables enter the following command:</em></strong></p><pre>touch variables.tf</pre><p>Then copy the text from the code block below.</p><pre>variable &quot;vpc_id&quot; {<br>  type        = string<br>  description = &quot;var for VPC ID&quot;<br>  default     = &quot;enter your VPC ID&quot;<br>}<br><br>variable &quot;cidr_open&quot; {<br>  type        = string<br>  description = &quot;CIDR block to allow traffic from anywhere&quot;<br>  default     = &quot;0.0.0.0/0&quot;<br>}<br><br>variable &quot;cidr_my&quot; {<br>  type        = string<br>  description = &quot;CIDR block to allow traffic from local machine&quot;<br>  default     = &quot;enter your ip cidr block&quot;<br>}<br><br>variable &quot;s3_bucket&quot; {<br>  type        = string<br>  description = &quot;S3 bucket&quot;<br>  default     = &quot;enter unique bucket name&quot;<br>}<br><br>variable &quot;instance_type&quot; {<br>  type        = string<br>  description = &quot;var for instance type&quot;<br>  default     = &quot;t2.micro&quot;<br>}<br><br>variable &quot;ami_id&quot; {<br>  type        = string<br>  description = &quot;var for AMI ID&quot;<br>  default     = &quot;enter the ami of the instance you wish to use&quot;<br>}</pre><p>You will need to make some adjustments to the file above according to what your environment as well as what instance you want to use.</p><p><strong><em>Note: The script above is for RedHat products so it is suggested that you use an Amazon 2 “free-tier” image.</em></strong></p><p><strong>Step 5: Deploying into AWS</strong></p><p>In the section, we will use the Terraform workflow step-by-step to make sure that everything is in working order and ready to deploy to AWS. Once everything is created Terraform makes it very simple for us to create our infrastructure.</p><p>Make sure that you are still in the directory that we created our files in.</p><p><strong><em>Enter the following command:</em></strong></p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*7WiVlcFR_NmnDwmBOixxwA.png" /></figure><p>This command will initialize our backend as well as any other plugins that may be necessary. This should be the first command that we run when using a new directory or when copying files over.</p><p><strong><em>Enter the following command to check the format:</em></strong></p><pre>terraform fmt</pre><p>One thing to note here is that the file must be saved with all of the most recent changes.</p><p>This is one thing that I love about Terraform! No more copy and pasting code back and forth between your editor and a validator.</p><p><strong><em>Enter the following command to validate:</em></strong></p><pre>terraform validate</pre><p>The validate runs checks that verify whether a configuration is syntactically valid and internally consistent. It is useful for general verification of attribute names and value types.</p><p><strong><em>Enter the following command:</em></strong></p><pre>terrafrom plan</pre><p>This command will show you the resources and configurations that will be executed when you run the “terrafrom apply” command. This stage will also show you errors but sometimes it’s not an end all be all.</p><p>If everything looks good and there are no errors that need to be fixed, there is just one more step.</p><p><strong><em>Enter the following command to apply configuration files:</em></strong></p><pre>terraform apply</pre><p>Then enter “yes”.</p><p>It will take a moment for everything to be deployed to AWS.</p><p>Almost there, one more step left!</p><p><strong><em>Step 6: Verify</em></strong></p><p>In this final step, we will make sure that all of our resources were deployed properly. We need to check the Amazon S3 bucket that we created as well as our Jenkins server.</p><p>First we will verify that we can reach our Jenkins server via port 8080 from our browser. Navigate to the EC2 dashboard and find the instance that we created. We will need the public IP address.</p><p>Then check in your browser to make sure that the server is accessible.</p><p>Enter the IP address into the browser as follows:</p><pre>instance_IP:8080</pre><p>This will allows to check on port 8080. This is how we configured it in our security groups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*zxH-7KoKuug8fJgTpz4yZg.png" /></figure><p>If you can see the screen above then you have successfully initiated a Jenkins server with the correct Security Groups.</p><p><strong><em>Note: You must make sure that the instance is fully initialized and has completed. I got stuck for longer than I would like to admit here. Be patient!</em></strong></p><p>Now, we just need to check the S3 bucket. Head on over to the S3 service so we can make sure that the bucket has been configured properly.</p><p>Find the bucket that you have created and find the permissions tab and click on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*tYvbbIT3kiMCUkucBAl4MA.png" /></figure><p>If you can see that the bucket has been created and the permissions are correct, then the mission is accomplished.</p><p><strong>Make sure you delete all of the resources that we created.</strong></p><p>Enter the following command to destroy the resources:</p><pre>terraform destroy</pre><p>Then enter “yes”.</p><p>Just like that everything is gone! Now you see why Terraform is such an incredible tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*wQXNRHe2p7sHA5BF.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation, and help you learn more about the exciting world of cloud computing.</p><p>Thank you for reading, and I look forward to hearing from you soon.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>.</em></p><p><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>, </em><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><em>, and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><p><strong><em>Interested in scaling your software startup</em></strong><em>? Check out </em><a href="https://circuit.ooo/?utm=publication-post-cta"><strong><em>Circuit</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e755e53a2899" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/terraforming-jenkins-on-aws-e755e53a2899">Terraforming  Jenkins on AWS</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Persistence with Kubernetes]]></title>
            <link>https://medium.com/@stepencleary/persistence-with-kubernetes-46f039d9a2ad?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/46f039d9a2ad</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Fri, 17 Mar 2023 12:08:46 GMT</pubDate>
            <atom:updated>2023-03-24T04:20:51.222Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ardBqjgXN6mMWDG48e1-vQ.png" /></figure><p>I’m back with another Kubernetes project this week as persistence is key. I will be showcasing a variety of workloads in this short demo, such as deployments, services, configmaps, persistent volumes, and persistent volume claims. If you&#39;re like me you’re probably eager to get started so let’s dig in!</p><p><strong>What is Kubernetes?</strong></p><p>Kubernetes is an open-source platform that automates the deployment, scaling, and management of containerized applications. It allows you to deploy and manage your applications with ease. With Kubernetes, you can easily manage and scale your applications, perform rolling updates, and ensure high availability.</p><p><strong>What are Deployments?</strong></p><p>A deployment is a higher-level resource that provides declarative updates and scaling for a set of replica pods. It specifies the desired state for the replica pods, and Kubernetes takes care of creating, updating, and deleting the necessary pods to match that state. Deployments allow you to manage your application’s lifecycle with ease, from rolling out updates to rolling back to a previous version.</p><p><strong>What are Services?</strong></p><p>In Kubernetes, a service is an abstract way to expose a set of replica pods running in your cluster as a network service. It provides a stable IP address and DNS name that can be used to access the replicas, even as they are scaled up or down, or moved between nodes.</p><p><strong>What are ConfigMap files?</strong></p><p>ConfigMap files are used to store configuration data that is needed by applications running in containers, such as environment variables, command-line arguments, configuration files, and other settings. By using ConfigMaps, you can separate the configuration data from the container images, making it easier to manage and update your application’s configuration without having to rebuild a container image.</p><p>Now that you have a general understanding of some of the workloads we will be working with, let’s take a look at some prerequisites and goals for this walk-through.</p><p><strong>Pre-Requisites</strong></p><ul><li>Basic knowledge of Kubernetes</li><li>Familiarity with AWS and an AWS account</li><li>Linux basics</li></ul><p><strong>Tasks:</strong></p><ul><li>Create two deployments with 2 pods each running Nginx images</li><li>Include a ConfigMap that points to a custom HTML page (for both deployments)</li><li>Create a Service that points to both Deployments</li><li>Validate the custom HTML pages using the curl command</li><li>Create a persistent storage manifest that gets 250MB of disk space on the host computer in the /temp/k8s directory</li><li>Create a corresponding persistent storage claim manifest file.</li></ul><p>Let’s get after it!</p><p><strong>Step 1: Setting up a Kubernetes environment</strong></p><p>Kubernetes setup can be a job in and of itself so instead of struggling with all sorts of different environments and installs we are just going to keep it as simple as possible so we can start with the project.</p><p>In a previous article, I gave step-by-step instructions to set up an environment using Cloud 9 in the AWS console as well as using Rancher k3s.</p><p>You can find the article <a href="https://medium.com/nerd-for-tech/kubernetes-the-easy-way-b1bc97731749">here</a> and follow the set of instructions for the installation.</p><p><strong>Step 2: Create two ConfigMap files</strong></p><p>Here we will create and deploy two ConfigMap files to our server. If you remember from the tasks section we need to create two separate HTML pages for our test. To do this we will use ConfigMap files.</p><p>ConfigMap files allow us to separate the configuration data so that it can be easily updated or changed. Especially if we have the whole process automated within a CI/CD pipeline.</p><p>First thing is first, we need to create a directory for our files to live in.</p><p><strong>To make a directory enter the following command:</strong></p><pre>mkdir &lt;directory name&gt;</pre><p>Then change to the newly created directory.</p><p><strong>To change directories enter the following command:</strong></p><pre>cd &lt;name of directory&gt;</pre><p>Now, we will need to create a file and save the code in the file. I will be using nano to do this as it’s the most user-friendly and we are trying to do this the easy way!</p><p><strong>To open Nano enter the following command:</strong></p><pre>nano &lt;name of file&gt;</pre><pre>---<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: configmap1<br>data:<br>  index.html: |<br>    &lt;html&gt;<br>      &lt;head&gt;<br>        &lt;title&gt;This is Deployment 1&lt;/title&gt;<br>      &lt;/head&gt;<br>      &lt;body&gt;<br>        &lt;h1&gt;<br>      &lt;/body&gt;<br>    &lt;/html&gt;</pre><p>You will be prompted with the text editor. Go ahead a copy the code from above into the window then press “Ctrl + X”, then “Y” and finally “enter.”</p><p>Now for the second file. Repeat the process from above.</p><pre>---<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: configmap2<br>data:<br>  index.html: |<br>    &lt;html&gt;<br>      &lt;head&gt;<br>        &lt;title&gt;This is Deployment 2&lt;/title&gt;<br>      &lt;/head&gt;<br>      &lt;body&gt;<br>        &lt;h1&gt;<br>      &lt;/body&gt;<br>    &lt;/html&gt;</pre><p>Great we have the YAML files written and on our server, time to deploy them.</p><p><strong><em>To deploy the ConfigMap files enter the following command:</em></strong></p><pre>sudo kubectl apply -f &lt;name of your file&gt;.yml</pre><p>Repeat the process for the second file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*-1gnTOQDMh5LA9-3Wa-Sww.png" /></figure><p>There you have it we can check that off the list!</p><p><strong>Step 3: Create the Deployments</strong></p><p>For this section, we will be creating multiple deployments. A deployment will set the stage for our pods. We can expose our ports, set our replicas, including our container image, as well as mount our volumes.</p><p>Just like in the previous step we will need to create a file in our directory for both deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*aOo3JLlm2oU8obnETr5sqQ.png" /></figure><p>Now that we have our files created we will once again copy the code provided below into each file. You can tell the difference by the ConfigMap name located at the bottom of the file.</p><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx<br>  labels: <br>    app: nginx<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>        - name: nginx<br>          image: nginx:latest<br>          ports:<br>          - containerPort: 80<br>          volumeMounts:<br>            - name: custom-html<br>            mountPath: /usr/share/nginx/html<br>      volumes:<br>        - name: custom-html<br>          configMap:<br>            name: configmap1</pre><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx<br>  labels: <br>    app: nginx<br>spec:<br>  replicas: 2<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx2<br>    spec:<br>      containers:<br>        - name: nginx<br>          image: nginx:latest<br>          ports:<br>          - containerPort: 80<br>          volumeMounts:<br>            - name: custom-html<br>              mountPath: /usr/share/nginx/html<br>      volumes:<br>        - name: custom-html<br>          configMap:<br>            name: configmap2</pre><p>After the code is safe and secure in your newly created files. Let’s go ahead and deploy them.</p><p><strong><em>To deploy the Deployment files enter the following command:</em></strong></p><pre>sudo kubectl apply -f &lt;name of your file&gt;.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*owMQONJKY-Bj2zOOHlTDYw.png" /></figure><p>You thought we would be doing this without errors. Think again! Looks like we are missing a “-” in our code. Let’s take a look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*YSawiCLA-HUO3X5EDGWSXA.png" /></figure><p>I was wrong it was an indentation on the mountPath line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*KNu0IDFCmQcbfe3KhDcAEg.png" /></figure><p>Now that we have that fixed, try and deploy the Deployment file again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*XJwBaDKE19qTFRa-PVja0Q.png" /></figure><p>Perfect. We are in business!</p><p>Now, repeat the process for the second deployment file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*OV-wcws3FCmRKYGAaOvupg.png" /></figure><p>Alright, we have our deployments all set up. Let’s go ahead and check them just to make sure.</p><pre>sudo kubectl get pods -o wide</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*yDFWOTY5_ioWHD1XJfpXrg.png" /></figure><p>Perfect!</p><p><strong>Step 4: Create a service</strong></p><p>If you can recall back to the start of the article I mention that we need to create a service that points to both deployments. We will need to create a type of service that we can use to communicate.</p><p>Just like before create a file and then edit the file using Nano.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: nginx-service<br>spec:<br>  selector:<br>    app: nginx<br>  ports:<br>    - name: http<br>      protocol: TCP<br>      port: 80<br>      targetPort: 80<br>  type: NodePort</pre><p><strong><em>To apply the service enter the following command:</em></strong></p><pre>sudo kubectl apply -f &lt;name of your file&gt;.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*zfAAP1BD8RhzFXba56iJUg.png" /></figure><p><strong><em>Enter the following command check the service:</em></strong></p><pre>sudo kubectl get service</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*IARx9LrekSKYDkYOvxhtBg.png" /></figure><p>We can use the cluster IP and the port that we used to check for the custom HTML files using the curl command.</p><p><strong><em>To verify the HTML enter the following command:</em></strong></p><pre>curl &lt;local ip&gt;:&lt;port&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*CoqgGBbMwu7aihmX9k58ww.png" /></figure><p><strong>Note: Sometimes it takes time for the HTML to change give it a few tries. I spent far too long troubleshooting and it was actually working the whole time.</strong></p><p><strong>Step 5: Create a persistent volume</strong></p><p>In this section, we will create a HostPath persistent volume. Kubernetes supports hostPath for development and testing on a single-node cluster.</p><p>A persistent volume HostPath is a persistent volume that uses a directory on the host’s filesystem as its storage location. You specify the path to the directory when creating the persistent volume. Kubernetes will then use that directory as the source for the volume, and mount it into the Pod’s filesystem.</p><p>Below is the code for the persistent volume. The path will be “/temp/k8s” as specified in the task section along with 250 mb of storage.</p><pre>apiVersion: v1<br>kind: PersistentVolume<br>metadata:<br>  name: my-pv-volume<br>  labels:<br>spec:<br>  capacity:<br>    storage: 250Mi<br>  accessModes:<br>    - ReadWriteOnce<br>  hostPath:<br>    path: &quot;/temp/k8s&quot;</pre><p>The next step is to create a “PersistentVolumeClaim.” Pods use “PersistentVolumeClaims” to request physical storage.</p><pre>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: my-pv-claim<br>spec:<br>  accessModes:<br>    - ReadWriteOnce<br>  resources:<br>    requests:<br>      storage: 250Mi</pre><p>Let’s go ahead and apply them.</p><p><strong><em>To apply the files enter the following command:</em></strong></p><pre>kubectl apply -f &lt;file name.yml&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*MPHTHI6thHEpsYMm2-SVsw.png" /></figure><p>Both the volume and the claim have been created.</p><p><strong><em>To check the volume enter the following command:</em></strong></p><pre>kubectl get pv &lt;name of volume&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*8onsZSG6kNIEDOChgoyh_g.png" /></figure><p>That’s it if you take a look at the tasks that we had for the day we have checked them off one by one!</p><p>Great job! You’re on the way to becoming a Kubernetes master!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*kwTHvzZaiV89V5Xk.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation, and help you learn more about the exciting world of cloud computing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46f039d9a2ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes the Easy Way]]></title>
            <link>https://medium.com/@stepencleary/kubernetes-the-easy-way-b1bc97731749?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1bc97731749</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Tue, 14 Mar 2023 21:51:31 GMT</pubDate>
            <atom:updated>2023-03-16T02:46:55.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmURRTDQq4LE7owTI0OLyA.png" /></figure><p>In this extremely easy-to-follow tutorial, I will be walking you through how you can get started with the infamous Kubernetes. Let’s face it, Kubernetes is a beast within itself and can be a daunting task just to get your environment configured and ready to go. We will be setting up a single node cluster in AWS using Cloud9 and Rancher K3s.</p><p><strong>What is Kubernetes?</strong></p><p>Kubernetes is an open-source platform that automates the deployment, scaling, and management of containerized applications. It allows you to deploy and manage your applications with ease, regardless if they are running on-premises, in the cloud, or hybrid. With Kubernetes, you can easily manage and scale your applications, perform rolling updates, and ensure high availability.</p><p><strong>What is Rancher k3s?</strong></p><p>Rancher k3s is a lightweight, certified Kubernetes distribution designed for resource-constrained environments such as edge computing, IoT, and ARM-based systems. It is a fully compliant Kubernetes distribution that is packaged in a single binary of less than 50MB, making it easy to install and operate. With k3s, you get all the benefits of Kubernetes such as container orchestration, automatic scaling, and self-healing, but with a smaller footprint and simplified operations.</p><p>Now that you have a general understanding of some of the tools we will be working with let’s take a look at some prerequisites and goals for this tutorial.</p><p><strong>Pre-Requisites</strong></p><ul><li>Basic knowledge of Kubernetes</li><li>An understanding of Docker containers and basic concepts</li><li>Familiarity with AWS and an AWS account</li><li>Linux basics</li></ul><p>Tasks:</p><ul><li>Use AWS to create a Single node cluster</li><li>Configure a Redis image with 4 replicas</li><li>Configure an Apache image with 10 replicas</li><li>Configure a Postgres image with 1 replica</li></ul><p>Great! You’re now equipped with all you need to get started with Kubernetes! Let’s go!</p><p><strong>Step 1: Create a Cloud 9 Environment</strong></p><p>As you may have already guessed this first step is all about getting our environment set up. We will be using Cloud 9 and K3s to make it as simple as possible. Kubernetes can be very time-consuming to set up with obstacles that may interfere with doing what you really want to do, which is to learn Kubernetes.</p><p>Head over to the AWS console and navigate to the Cloud 9 landing page.</p><p>Once there we are going to want to click the orange “Create environment” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*1mTXx3q5Vj8ke7HeivSGYg.png" /></figure><p>On the next page, you will be prompted to fill out some information. Go ahead and give your environment a name and description along with selecting “New EC2 instance.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*UbC8QxaFTuxpnXSx87t3Ow.png" /></figure><p>Select the options for the EC2 instance as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*z6Ur4qqIg9SyNGnULerSWw.png" /></figure><p>For the Network settings, we will be selecting the “Secure Shell” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*1HEcuvCp7q46e9UHoiTSbQ.png" /></figure><p>Then, go ahead and click the orange “Create” button at the bottom right.</p><p>After clicking the create button the environment will start to spin up, this may take a few minutes to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*4RY4ZUxMH4HG-4090d0h6w.png" /></figure><p>Once finished click on “Open” from the environment page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U-a1f-z2cXWdbgau4vx7uw.png" /></figure><p>There you will see your development environment. At the bottom of the screen, you can see the terminal that we will be using in today’s tutorial.</p><p>Great job! On to the fun stuff!</p><p><strong>Step 2: Installing packages</strong></p><p>Here we will be updating and installing all packages that we will need to get Kubernetes up and running. There are just a few short steps, which is why the title of this article is “Kubernetes The Easy Way.”</p><p><strong>To update packages enter the following commands:</strong></p><pre>sudo apt-get update &amp;&amp; sudo apt-get upgrade -y</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*pSeh2O7wRJH1Xa0_IwRzkA.png" /></figure><p>Next, we will want to get K3s installed. This will allow us to run Kubernetes commands from within the terminal.</p><p><strong>To install K3s enter the following command:</strong></p><pre>curl https://get.k3s.io | sh -</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*IxA-oxqGfLCksUSR3B3-wg.png" /></figure><p>There you go, we are all set. That was easy, right? For any further information regarding K3s you can find the documentation <a href="https://docs.k3s.io/">here</a>.</p><p><strong>Step 3: Creating deployments</strong></p><p>As I had mentioned before we have three deployments. I have written them in three separate YAML files. You may choose to do it differently but I will be providing you with the code either way.</p><p>First, we will start with the Apache containers.</p><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: lb-service<br>  labels:<br>    app: apache-server<br>spec:<br>  type: LoadBalancer<br>  ports:<br>    - port: 80<br>  selector:<br>    app: apache-server<br><br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: apache-server<br>spec:<br>  replicas: 10<br>  selector:<br>    matchLabels:<br>      app: apache-server<br>  template:<br>    metadata:<br>      labels:<br>        app: apache-server<br>    spec:<br>      containers:<br>        - name: apache-server<br>          image: httpd:latest</pre><p>We will need to transfer the above code to our environment here using a local text editor that is already installed. Along with that, we will need a directory for the files to live in.</p><p><strong>To make a directory enter the following command:</strong></p><pre>mkdir &lt;directory name&gt;</pre><p>Then change to the newly created directory.</p><p><strong>To change directories enter the following command:</strong></p><pre>cd &lt;name of directory&gt;</pre><p>Now we will need to create a file and save the code in the file. I will be using nano to do this as it’s the most user-friendly and we are trying to do this the easy way!</p><p><strong>To open Nano enter the following command:</strong></p><pre>nano &lt;name of file&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*3jrHod7Ap9jthJ6RssE9lQ.png" /></figure><p>You will be prompted with the text-editor. Go ahead a copy the code from above into the window then press “Ctrl + X”, then “Y” and finally “enter.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*9E-gvbqheGop6_U8ylvvOg.png" /></figure><p>As you can see we have now created a Yaml file in our new directory. We need to do this with two other files. Repeat the above steps for the next two files. I will post the code below.</p><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: redis-0<br>  labels:<br>    app: redis<br>spec:<br>  type: LoadBalancer<br>  ports:<br>    - port: 7000<br>  selector:<br>    app: redis<br><br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: redis<br>spec:<br>  replicas: 4<br>  selector:<br>    matchLabels:<br>      app: redis<br>  template:<br>    metadata:<br>      labels:<br>        app: redis<br>    spec:<br>      containers:<br>        - name: redis<br>          image: redis:latest</pre><pre>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: postgres<br>  labels:<br>    app: postgres<br>spec:<br>  type: NodePort<br>  ports:<br>    - port: 5423<br>  selector:<br>    app: postgres<br><br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: postgres<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: postgres<br>  template:<br>    metadata:<br>      labels:<br>        app: postgres<br>    spec:<br>      containers:<br>        - name: postgres<br>          image: postgres:latest<br>          env:<br>            - name: POSTGRES_PASSWORD<br>              value: passwd123</pre><p>Below you can see that I now have three files for each deployment and service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*3W2qNWo7yd376HKwK719Uw.png" /></figure><p>Almost there! We are now going to create the pods for each deployment.</p><p><strong>Enter the following command:</strong></p><pre>sudo kubectl apply -f &lt;name of your file&gt;.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*Ze5wb1njvZCyi9w8dOef4Q.png" /></figure><p>We have now created the Apache pods. Repeat the step for the other two files that we have in the directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*iUwcD6dwFBLeCVpunmvpeg.png" /></figure><p>Wow, that was smooth, no bugs in our code.</p><p>That’s because I debugged it prior to writing this article. I did have a few issues with my code, to begin with. Yaml is very picky about indentation so keep that in mind. If you decide to write your own code is suggested you use a <a href="https://www.yamllint.com/">yaml formatter</a>.</p><p>Let’s check our running pods.</p><p><strong>To check pods enter the following command:</strong></p><pre>sudo kubectl get pods</pre><p>If you get an error make sure you have permission and are using the “sudo” command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*xeHCghhCFpHlkd9Cc3qavw.png" /></figure><p>Great all of our pods are up and running!</p><p>Remember they are all running on one node. This isn’t exactly how we want to run containers, but it’s a great start to learning the basics of Kubernetes and it’s really easy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8x6732ziLPyepyphpl__lQ.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation, and help you learn more about the exciting world of cloud computing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1bc97731749" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS to Create a Docker Swarm]]></title>
            <link>https://aws.plainenglish.io/using-aws-to-create-a-docker-swarm-b48731c850c?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b48731c850c</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 10:35:13 GMT</pubDate>
            <atom:updated>2024-09-09T05:26:13.989Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using AWS to Create a Docker Swarm Cluster</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y97mRQYX1EKWGIHJKWhgzA.png" /></figure><p>In this tutorial, we’ll explore how to create a Docker Swarm cluster in AWS, step-by-step. One of the most popular approaches for managing containers in AWS is by using Docker Swarm, which is a native clustering tool.</p><p><strong>What is Docker Swarm?</strong></p><p>Docker Swarm is a tool that allows you to manage multiple Docker containers as a single cluster, making it easier to deploy and scale applications.</p><p>With Docker Swarm, you can distribute containerized applications across multiple nodes and ensure that they are running efficiently and reliably. It provides features such as load balancing, rolling updates, and service discovery, making it a popular choice for managing containerized applications in production environments.</p><p>Overall, Docker Swarm simplifies the process of managing large-scale container deployments and helps ensure the availability and scalability of your applications.</p><p>Great, now that you are up to speed, let’s take a look at the project.</p><p><strong>Tasks:</strong></p><ul><li>Use AWS to create a Docker Swarm that consists of one manager and three worker nodes</li><li>Create one manager node</li><li>Create a service based on the Redis docker image with 4 replicas</li><li>Create a service based on the Apache docker image with 10 replicas</li><li>Create a service based on the Postgres docker image with 1 replica</li></ul><p><strong>Prerequisites:</strong></p><ul><li>General Linux knowledge</li><li>Basic knowledge of Docker and containers</li><li>An AWS account with sufficient permissions</li></ul><blockquote><em>💡</em> <strong>Learn how to efficiently store one million small objects to S3 for your stock analytics app with AWS Athena integration:</strong></blockquote><p><a href="https://differ.blog/p/how-to-write-one-million-small-objects-to-s3-c69e8c">How to write one million small objects to S3</a></p><blockquote><em>👉</em> To read more such acrticles, <a href="https://differ.blog/"><strong>sign up for free on Differ</strong></a>.</blockquote><p><strong>Step 1: Configure four instances</strong></p><p>First up we will need to configure four instances that we can use to deploy our swarm. This will consist of one manager node and three worker nodes as mentioned above.</p><p>To accomplish this task, navigate to the AWS console. Then head on over to the EC2 landing page where we can create our instances.</p><p>Once you arrive at the EC2 dashboard click on the orange “Launch instance” button.</p><p>We will be creating four instances. To the right, at the top of the page, increase the number of instances to four.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9weocuOeLs6nnlssXP4vw.png" /></figure><p>We will be leaving the name section blank and assigning the names once the instances have been configured.</p><p>Next, we can select the AMI and the instance type. In today’s demo, we will be using free trier resources. You may need to scale up if your workload is heavy, but for our project, the free-tier will suffice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dhIzerPNvSmj_Wk_gL9N5g.png" /></figure><p>Then, create or select a key pair. You will want to keep this safe if you are going to be using SSH to access your server.</p><p>Now, it’s time to set up our security group. To edit the security group click on “edit” in the network settings bar.</p><p>The security group functions as the firewall for the instances. We will need to open up ports as followed to allow the cluster to communicate with each other, as well as to allow us to access the instances.</p><p>Add the following security rules:</p><ul><li>TCP port 2376 for secure Docker client communication. This port is required for Docker Machine to work. Docker Machine is used to orchestrate Docker hosts.</li><li>TCP port 2377. This port is used for communication between the nodes of a Docker Swarm or cluster. It only needs to be opened on manager nodes.</li><li>TCP and UDP port 7946 for communication among nodes (container network discovery).</li><li>UDP port 4789 for overlay network traffic (container ingress networking).</li><li>TCP port 22 to SSH into our instances remotely</li></ul><p>If you are interested in learning more about the following ports, I’ve provided a link <a href="https://www.digitalocean.com/community/tutorials/how-to-configure-the-linux-firewall-for-docker-swarm-on-ubuntu-16-04">here</a>.</p><p>Next, we open the advanced details carrot so that we can enter user data for our instances.</p><p>Since docker does not come pre-installed with our instances we would have to manually install docker on each machine. We can enter a script into the user data field so that it will run when the instance is spinning up. This way docker will be installed once we launch the instance.</p><p><strong>Enter the following script into the user data field:</strong></p><pre>#!/bin/bash<br>sudo yum update<br>sudo yum -y install docker<br>service docker start<br>usermod -a -G docker ec2-user<br>chkconfig docker on<br>pip3 install docker-compose</pre><p>Once ready you can click the orange “Launch instance” button.</p><p><strong><em>Note: You will need to reboot all machines after the initial startup for all the commands to take effect!</em></strong></p><p>Since each node(ec2 instance) will have different roles we will need to organize them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oh9-sgIHcIKVNWWZCgvAoQ.png" /></figure><p>We will have one master node and three other nodes running our specific services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r0MKTtGVWYpJ1dljhWp2Cw.png" /></figure><p><strong><em>Note: Generally you’ll want to have more than one master node for backup but since this is a demo it’s acceptable. Check the documentation </em></strong><a href="https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/"><strong><em>here</em></strong></a><strong><em>.</em></strong></p><p><strong>Step 2: Configure the master and worker nodes</strong></p><p>In this step, we will get our master node set up in our cluster. Your master node will handle cluster management tasks such as maintaining the cluster state, scheduling services and serving swarm mode HTTP API endpoints.</p><p>Navigate to your worker node that we had labeled in the previous step. We did open up port 22 for ssh but for simplicity’s sake, I will be using EC2 instance connect through the console. Start the EC2 instance and then click “Connect.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*6ja8S9V-EuK2lIG-uGiy7A.png" /></figure><p>Then make sure you are under the EC2 instance connect tab and click “connect.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*0c7GjjRMZLXiv_Whp32Ipw.png" /></figure><p>Now that you are in the shell of the instance we will initialize swarm mode.</p><p><strong>Enter the following command to configure the master node:</strong></p><pre>docker swarm init --advertise-addr &lt;private ip of master node&gt; </pre><p><strong><em>Note: Make sure to use a private static IP address above. I made the mistake of using a dynamic public IP and this will change after reboot causing the worker nodes to be unable to connect with the master.</em></strong></p><p>You should get the following output from the command. Make sure you copy the command it suggests to add a worker to a swarm. We will be using this in the next step and need to use the token provided.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_BtMS6_ysiyUN1fXLlQug.png" /></figure><p>Now for the worker nodes. Go ahead and open three EC2 instance connect sessions we will need to join all three to the master node using the command from above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ra0q1jkxg1R_KxsLlGJkuw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHpQ1pCj7IEGNX45rZARdw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8XNJKhVAzq-srqlFp9csug.png" /></figure><p>Let’s check our manager node to make sure that the cluster has been configured correctly.</p><p><strong>Enter the following command to list nodes:</strong></p><pre>docker node ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ny5-nrT8v50kB9_YBMfJSw.png" /></figure><p>You can see above and confirm that there is one manager/leader node and three workers.</p><p>Great our cluster is configured!</p><p><strong>Step 3: Configuring the services</strong></p><p>At the beginning of the project, we were tasked with setting up three services within the cluster. In this section, we will create the services from the manager node. We will just be using the default latest in this tutorial that will be pulled from Dockerhub</p><p><strong>To converge the Redis service enter the following command:</strong></p><pre>docker service create --name &lt;name of service&gt; --replicas &lt;# of replicas&gt; &lt;name of image&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*s2_eLnr_3nEBYNMeRB9v1w.png" /></figure><p><strong>To converge the Apache service enter the following command:</strong></p><pre>docker service create --name &lt;name of service&gt; --replicas &lt;# of replicas&gt; &lt;name of image&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*IhlnyteHm_fhADwL32Ll4w.png" /></figure><p>Great one left!</p><p>When trying to configure the Postgres services I ran into a small issue. I couldn’t get the task to run successfully. After some research, I found that you need to create a docker secret from within the command line for it to work properly.</p><p>If you’re interested in reading more you can find the documentation <a href="https://hub.docker.com/_/postgres">here.</a></p><p><strong>Enter the following command to converge the Postgres service:</strong></p><pre>docker service create --name postgres_service --replicas 1 -e POSTGRES_PASSWORD=mypasswd postgres</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*obY0qcE9znJKJFnDoCZDog.png" /></figure><p>Our services are now set up and running on our manager node. It’s always best to verify though.</p><p><strong>To check the services enter the following command:</strong></p><pre>docker service ls </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLnuFgCGDfBdtDtoGNLlIA.png" /></figure><p>As you can see, we have all of our services up and running on our worker nodes!</p><p><strong>Step 4: (optional) Using a Stack</strong></p><p>In this section, I will go through an alternative way to configure the master and worker nodes with the correct services. Docker stacks can be used to manage the orchestration of Docker containers across multiple nodes.</p><p>First, we need to remove the services on the master node that we had configured previously. We will be using identical services for demonstration purposes.</p><p><strong>To remove services enter the following command:</strong></p><pre>docker service rm &lt;service name&gt;</pre><p>Once you have all the services removed we will create the stack using a Docker compose file and YAML.</p><p>First, we need create a directory on the instance of the manager node.</p><p><strong>To create a directory enter the following command:</strong></p><pre>mkdir &lt;directory name&gt;</pre><p>Then, let’s open up a text editor so that we can create the file for the Docker stack.</p><p><strong>Enter the following command to open a text editor.</strong></p><pre>nano &lt;name of file&gt;</pre><p>You can copy and paste the text below if you are following along or you can create your own. If you creating your own YAML file you can find additional documentation <a href="https://docs.docker.com/engine/reference/commandline/service_create/#constraint">here</a>.</p><pre>version: &quot;3&quot;<br>  <br>services:<br><br>  web_app:<br>    image: httpd:latest<br>    deploy: <br>      replicas: 10<br>      placement:<br>        constraints:<br>          - node.role==worker<br>  <br>  data_base:<br>    image: redis:latest<br>    deploy: <br>      replicas: 4<br>      placement:<br>        constraints:<br>          - node.role==worker<br><br>  data_base2:<br>    image: postgres:latest<br>    environment:<br>      POSTGRES_PASSWORD: mypasswd<br>    deploy: <br>      replicas: 1<br>      placement:<br>        constraints:<br>          - node.role==worker</pre><p>Once your file is ready let’s go ahead and test it. Make sure you are in the directory where the file is located when you run the command.</p><p><strong>Enter the following command to deploy the stack:</strong></p><pre>docker stack deploy --compose-file &lt;name of file&gt;&lt;name of stack&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wc-3RpWxYhgyT51emeAh-w.png" /></figure><p>At first glance, it looks like our services have been deployed. Let’s go ahead and double-check.</p><p><strong>To confirm the services have been deployed enter the following command:</strong></p><pre>docker service ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sf65xkxPVh9OQY-HWxgr3g.png" /></figure><p>As you can see in the photo above the correct amount of replicas and services are running.</p><p>Let’s also make sure that no stacks are running on the manager node.</p><p><strong>Enter the following command:</strong></p><pre>docker stack ps &lt;name of stack&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vm32JbsJ5BTCwnT4ATd33A.png" /></figure><p>As you can see everything has worked as planned with all services running on the correct nodes.</p><p><strong>Make sure and clean up all your nodes and services!</strong></p><p>I hope you enjoyed the optional way of configuring services with Docker Swarm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3pwonK1PixSRtOPA1YxeQ.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation, and help you learn more about the exciting world of cloud computing.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b48731c850c" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/using-aws-to-create-a-docker-swarm-b48731c850c">Using AWS to Create a Docker Swarm</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Docker Containers with Bind Mounts]]></title>
            <link>https://faun.pub/building-docker-containers-with-bind-mounts-ba889bf03eb6?source=rss-4b182c65f36b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba889bf03eb6</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Stephen Cleary]]></dc:creator>
            <pubDate>Mon, 06 Mar 2023 13:33:58 GMT</pubDate>
            <atom:updated>2023-03-28T11:56:49.660Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploying Docker Containers with Bind Mounts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_dG5lHupBmsuBGcbME-DQ.png" /></figure><p>Welcome back, everyone! I’m here with another fun project for you to try out. In this week’s demo, we will be working with the ever-popular Docker. If you’re interested in streamlining your development cycle, this one&#39;s for you!</p><p><strong>What is Docker?</strong></p><p>Docker is a containerization platform that allows developers to easily create, deploy, and run applications in isolated environments called “containers.” These containers bundle the necessary dependencies and libraries, making it easier to ensure that applications run consistently across different machines and environments.</p><p>In short, Docker ensures a consistent isolated environment for whatever application you need to run.</p><p><strong>What is DockerHub?</strong></p><p>DockerHub is a hosted repository service provided by Docker for finding and sharing container images. We can search easily for official containers and environments to build off of.</p><p>Now that you have a general understanding of what Docker and DockerHub are and what business problems they solve, let’s dive into the project!</p><p><strong>Tasks:</strong></p><ul><li>Build Dockerfile for Boto3 / Python</li><li>Use the Python official image from Dockerhub</li><li>Download any three repositories to your local host</li><li>Create three Python containers</li><li>Each container should have a bind mount to one of the repositories directories</li><li>Log into each container and verify access to each repo directory</li></ul><p><strong>Prerequisites:</strong></p><ul><li>An IDE (I will be using Cloud 9)</li><li><a href="https://hub.docker.com/signup">A Docker Hub Account</a></li><li>Basic Linux skills</li><li><a href="https://docs.docker.com/get-docker/">Docker Installed</a></li></ul><p>Great, now that you are well-informed as to what we are going to get into, let’s get started!</p><p><strong>Step 1: Build a Dockerfile</strong></p><p>As you may have already guessed from the title of this step, we are going to be building out a Dockerfile. A Dockerfile is just a text document that contains all the commands to assemble an image. Go ahead and open a text editor of your choice and we can start building.</p><p>The first thing we are going to want to do after we get an IDE all set up is to head on over to <a href="https://hub.docker.com/">DockerHub</a> to get some information for our Dockerfile.</p><p>Since we are going to be using a Python Image to build off of, we will need the tag to reference in our file.</p><p>Simply search for “Python” and click on the official image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*cWvc_YA-nmSHVj7k2lOMpw.png" /></figure><p>For simplicity’s sake, I will be using the “alpine3:16” image to build off of. Just note that you are free to use whatever version you would like. You’ll just need to make adjustments to the Dockerfile accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*HTnlhYdGixJQaAL3a_gYfg.png" /></figure><p>Remember that this image will also need to have boto3 installed so we will need to install this on top of the previous image as well.</p><p>Below is a simple Dockerfile that we can use to build our new image for development purposes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4fb22b9040fc8ecb15a16115effe802/href">https://medium.com/media/e4fb22b9040fc8ecb15a16115effe802/href</a></iframe><p>We are going to need to store this file in a directory so let’s head over to the command line and create one.</p><p><strong><em>To create a new directory enter the following command:</em></strong></p><pre>mkdir &lt;name of directory&gt;</pre><p>Now, let’s change directories.</p><p><strong><em>To change directories enter the following command:</em></strong></p><pre>cd &lt;name of newly created directory&gt;</pre><p>Once there we are going to use nano to save the Dockerfile to this directory. Make sure you are in the correct directory.</p><p>To open nano enter the following command:</p><pre>nano Dockerfile</pre><p>Then take the text from the Dockerfile that you have created or mine if you are following along and we will paste it in and save the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*AgBzn86fr22QwcKAB28qqw.png" /></figure><p>To save the file press “ctrl+x” then “y” and finally press “enter.”</p><p>Let’s double-check to make sure we saved the file and named it properly. A Dockerfile can only be named Dockerfile with no extensions, just keep that in mind.</p><p><strong><em>To check the files in the directory enter the following command:</em></strong></p><pre>ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*TSW9nVtGJZ2gd0s6YwQP4w.png" /></figure><p>As you can see, everything is in order.</p><p>Now that we have the file we need to build our custom image with the Dockerfile that we have created. This will build our specifications on top of the image that we entered into the Dockerfile.</p><p><strong><em>Enter the following command to build the image:</em></strong></p><pre>docker build -t &lt;name of image&gt; .</pre><p>If done correctly you should see the image building according to the Dockerfile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*UsF8ukDGdlc2QiMH08hpwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*v8jRKxqS_2WFoSrQZGsxwg.png" /></figure><p>Check to make sure that your image was created successfully with the correct tag.</p><p><strong><em>To list images enter the following command:</em></strong></p><pre>docker images</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*vB0e4TDN0iMDbOeOZd5e0w.png" /></figure><p>There you have it, a brand new image!</p><p><strong>Step 2: Clone three repositories to our local machine</strong></p><p>If you take a look back at the tasks for today, we need to clone 3 repositories so that we can eventually bind them to the containers in a later step. If you don’t have a GitHub account feel free to clone my public repositories<a href="https://github.com/steve0c"> here</a>.</p><p><strong><em>To clone the repositories enter the following command:</em></strong></p><pre>git clone &lt;url of GitHub repository&gt;</pre><p>After you have cloned the three repositories go ahead and make sure they are in your directory.</p><p>As you can see in the photo below we have three repositories in our working directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*LsHrNp_-mOzk3az42Gb06g.png" /></figure><p>Mission complete!</p><p><strong>Step 3: Create three containers and bind mount repositories</strong></p><p>This is where it starts to get good. In this step, we are going to start up our containers as well as bind mount our repositories to the container. This will all be done in one command.</p><p>To create a container enter the following command:</p><pre>docker run -d -it --name &lt;name of container&gt; --mount type=bind,source=$(pwd),target=/home/ &lt;image name&gt;:&lt;image tag&gt;</pre><p>Below is what I used to run in my environment. Just note that yours will be different according to your file names and or paths.</p><pre>docker run -d -it --name cloudformation --mount type=bind,source=$(pwd),target=/home/ python_boto3_image:latest</pre><p>If you want to look further into the ways that you can bind mount a volume or what some of the options are, you can check the official Docker documentation<a href="https://docs.docker.com/storage/bind-mounts/"> here</a>.</p><p><strong>Note: You must be in the directory that you want to bind if using the “$(pwd)” variable.</strong></p><p>Otherwise, enter the full path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*QDRel1hyTXfV_IEJ_hqYwQ.png" /></figure><p>Great, we have three containers that have a bind mount to the GitHub repositories that we cloned to our local machine.</p><p><strong><em>To check the status of our containers enter the following command:</em></strong></p><pre>docker container ls </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*OfmSep8sbIA0OFzrGWjJFw.png" /></figure><p>If you see the three containers that you have created here, you&#39;re all set! Great job!</p><p><strong>Step 4: Verify</strong></p><p>In this section, we are going to enter into our containers and check that the bind mounts that we configured are in fact working. Since the Alpine image doesn’t have Bash installed we will have to use another shell.</p><p><strong><em>Enter the following commands to confirm:</em></strong></p><pre>docker exec -it &lt;name of container&gt; bin/sh</pre><pre>cd /home/</pre><pre>ls</pre><p>To look further into the “exec” command you can check the documentation <a href="https://docs.docker.com/engine/reference/commandline/exec/">here</a>.</p><p>Since we binded the repository to the home directory in the container we can list the files in that location to confirm that they there were binded correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*Nis9auIdnzIyaug1uzVFNg.png" /></figure><p>In the above photo, you can see that we have two files from our directory located in our container!</p><p>Repeat this process for the other two containers to verify that everything is in working order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*xln4iQqSSUxtFrchNjS1eA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*xln4iQqSSUxtFrchNjS1eA.png" /></figure><p>Fantastic everything is working as planned!</p><p><strong>Step 5: Delete Containers and Images</strong></p><p>Now that we have built and tested our lab environment let’s go ahead and clean everything up.</p><p><strong><em>To list containers enter the following command:</em></strong></p><pre>docker container ls </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*BjLIK2DV4byHrgPwxn3LwQ.png" /></figure><p><strong><em>To delete the containers enter the following command:</em></strong></p><pre>docker rm -f &lt;first three characters of container id&gt;&lt;first three characters of container id&gt;&lt;first three characters of container id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*7ivDlkzO1gJR7K1jokvXTw.png" /></figure><p>Great we clean up our containers. Now, we can remove our images.</p><p><strong><em>To list images enter the following command:</em></strong></p><pre>docker images </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*cSFH_pZczVhpItar8uDwsg.png" /></figure><p><strong><em>To delete images enter the following command:</em></strong></p><pre>docker rmi -f &lt;first three characters of image id&gt;&lt;first three characters of image id&gt;&lt;first three characters of image id&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*yytoDWrXECJWN8eOzLRziQ.png" /></figure><p>That’s it! I hope you had a whale of a good time with this project on Docker and you can see the true power of containers firsthand!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wKSvZBJBYAMGlyDt8id_w.png" /></figure><p>As always, feel free to join me on this journey of discovery. Whether you come back to this blog, follow me on <a href="https://www.linkedin.com/in/stephencleary-devops/">social media</a>, or reach out to me directly, I would love to continue the conversation, and help you learn more about the exciting world of cloud computing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*aDhYDdTU4nzsOBrt.png" /></figure><h4>👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇</h4><h4>🚀<a href="http://from.faun.to/r/8zxxd">Join FAUN Developer Community &amp; Get Similar Stories in your Inbox Each Week</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba889bf03eb6" width="1" height="1" alt=""><hr><p><a href="https://faun.pub/building-docker-containers-with-bind-mounts-ba889bf03eb6">Building Docker Containers with Bind Mounts</a> was originally published in <a href="https://faun.pub/">FAUN.dev() 🐾</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>