<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Artem Nosulchik on Medium]]></title>
        <description><![CDATA[Stories by Artem Nosulchik on Medium]]></description>
        <link>https://medium.com/@anosulchik?source=rss-19cc4534363------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2g_GBge6Y2JQbb34fPCOVA@2x.jpeg</url>
            <title>Stories by Artem Nosulchik on Medium</title>
            <link>https://medium.com/@anosulchik?source=rss-19cc4534363------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:39:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anosulchik/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #7: AWS ECS insights and troubleshooting]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-7-aws-ecs-insights-and-troubleshooting-13ed1bfdf1c4?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/13ed1bfdf1c4</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[ces]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Tue, 09 Jan 2018 17:14:21 GMT</pubDate>
            <atom:updated>2018-01-09T17:20:58.409Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of our AWS/Terraform Workshops series that explores our vision for Service Oriented Architecture (SOA) and closely examines AWS Simple Storage Service, Terraform Remote State, and Identity Access Management. To learn more, check out our </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="https://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*AhDfyWmS22Cp4vhm83M4bw.png" /></figure><h3>Prerequisites</h3><ul><li><strong>Go through Workshops </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>#1</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l"><strong>#2</strong></a><strong>, </strong><a href="https://medium.com/@anosulchik/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f#.7tlpeayrd"><strong>#3</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7"><strong>#5</strong></a><strong> and </strong><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71"><strong>#6</strong></a></li><li><a href="https://docs.docker.com/engine/userguide/intro/"><strong>Docker Introduction</strong></a></li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html"><strong>AWS EC2 Container Service Introduction</strong></a></li></ul><p>This workshop covers two ECS service insights: troubleshooting of applications running in ECS and deployment into ECS with zero downtime. Unlike previous workshops there’s no preface section here, it’s done intentionally — please just move forward to the next section.</p><h3>Hands On</h3><h4>Troubleshooting EC2 instance not registering in ECS cluster</h4><p>ECS agent allows container instances to connect to your cluster. ECS agent is a container that is included in the Amazon ECS-optimized AMI, and starts on instance boot by default.</p><p>ECS agent container stores configuration in <strong>/etc/ecs/ecs.config</strong> on host machine and logs to <strong>/var/log/ecs</strong> directory. You can also use command to see its logs:</p><pre>docker logs &lt;container id&gt;</pre><p>In most of the cases ECS agent’s configuration in /etc/ecs/ecs.config only includes name of ECS cluster to join.</p><p>Here are top 3 reasons why ECS agent may fail to register instance in ECS cluster:</p><pre><strong>Missing IAM permissions for ECS agent.</strong></pre><pre><strong>Troubleshooting</strong>: check ECS agent logs in /var/log/ecs dir.</pre><pre><strong>Reason</strong>: ECS agent communicates with ECS service and sends API calls to it e.g. to register EC2 instance in cluster. In case IAM role for EC2 instance that runs ECS agent doesn’t have required permissions those API calls will fail with 403.</pre><pre><strong>Solution</strong>: Configure IAM role for EC2 instance with minimum required permissions for ECS:</pre><pre>&quot;Action&quot;: [<br>  &quot;ecs:RegisterContainerInstance&quot;,<br>  &quot;ecs:DeregisterContainerInstance&quot;,<br>  &quot;ecs:SubmitContainerStateChange&quot;,<br>  &quot;ecs:SubmitTaskStateChange&quot;,<br>  &quot;ecs:List*&quot;,<br>  &quot;ecs:*Poll*&quot;<br>]</pre><pre><strong>ECS cluster name is missing in agent’s configuration</strong>.</pre><pre><strong>Troubleshooting</strong>: check ECS agent logs, check its config.</pre><pre><strong>Reason</strong>: If ECS agent cannot read its config file or if config file is empty — ECS agent will register EC2 instance in ECS cluster named ‘default’.</pre><pre><strong>Solution</strong>: Update config file with proper ECS cluster name before agent is started. Instance cannot be re-registered(moved) to another ECS cluster. It has to be replaced with new instance with properly configured ECS agent.</pre><pre><strong>Docker or ECS agent container is missing in AMI</strong></pre><pre><strong>Troubleshooting</strong>: SSH into the instance, check for docker and list of running containers.</pre><pre><strong>Reason</strong>: in case you’re using custom AMI for your EC2 instances it may not come with docker or ECS agent, or wrong AMI is used.</pre><pre><strong>Solution</strong>: add missing components (docker and ECS agent container) to AMI or use ECS-optimized AMI.</pre><p>Now let’s troubleshoot the problem:</p><pre><strong>Use suggested Terraform configuration to create ECS cluster, register EC2 instance in this cluster. Find out why EC2 instance doesn&#39;t register in ECS cluster and fix it.</strong></pre><pre>1. Go to <strong>w7</strong> directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>2. Complete terraform configuration to create ECS cluster and EC2 instance. <strong>Keep remaining resources in terraform configuration files commented out.</strong></pre><pre>3. Add your SSH key into user-data.txt file.</pre><pre>4. Apply terraform configuration.</pre><pre>5. Go to ECS web console to make sure that container instance is registered there or not.</pre><pre>6. Troubleshoot why EC2 instance isn’t registered in ECS cluster and fix the problem.</pre><h4>Troubleshoot ECS task failing in ECS service not attached to ELB</h4><p>ECS task definition is JSON data that contains configurations of all containers which are required to run your application including image names, required resources, links between containers etc.</p><p>Top reasons why ECS tasks may fail in ECS service not attached to ELB:</p><pre><strong>Mistake in ECS task definition e.g. missing docker image or requested more resources than available in ECS cluster.</strong></pre><pre><strong>Troubleshooting</strong>: ECS web console — see stopped task status for ECS service, it includes the reason why it was stopped e.g. cannot pull container.</pre><pre><strong>Reason</strong>: we’re humans and may make mistakes in container definitions e.g. a typo in docker image name etc.</pre><pre><strong>Solution</strong>: Examine source of the problem and fix it in task definition.</pre><pre><strong>ECS cluster infrastructure problems.</strong></pre><pre><strong>Troubleshooting</strong>: ECS web console — see stopped tasks reasons, see application’s logs for errors, change logging severity if necessary to identify why application in container or container fails to start.</pre><pre><strong>Reason</strong>: there might be no access to smartling docker registry or to any other resource service might rely on, e.g. database.</pre><pre><strong>Solution</strong>: Adjust security groups and/or makes other changes to infrastructure to eliminate source of the problem.</pre><pre><strong>Crashing application running in container</strong>.</pre><pre><strong>Troubleshooting</strong>: ECS console and application container logs.</pre><pre><strong>Reason</strong>: application exits unexpectedly e.g. due to bug in code or corrupt docker image.</pre><pre><strong>Solution</strong>: identify and fix the problem: create new docker image with the bugs fixed etc.</pre><pre><strong>Missing IAM permissions to start tasks by ECS agent.</strong></pre><pre><strong>Troubleshooting</strong>: ECS agent logs.</pre><pre><strong>Reason</strong>: ECS agent must have enough permissions to start/stop ECS tasks.</pre><pre><strong>Solution</strong>: Fix IAM permissions for IAM role of container instance.</pre><p>Let’s troubleshoot and fix failing ECS service:</p><pre>Create ECS service to run sample application. <strong>Do not attach ELB to ECS service at this step.</strong> (troubleshoot container failing to start after deployment).</pre><pre>1. Uncomment resources in file <em>ecs_service.tf</em>, complete terraform configuration. Keep resources in <em>ecs_service_elb.tf</em> file commented out.</pre><pre>2. Make sure you specified docker image <strong>anosulchik/workshop-sample-application:v1.0</strong> in task definition.</pre><pre>3. Apply terraform configuration.</pre><pre>4. Go to ECS web console, choose ECS cluster and ECS service, check pending, running and stopped ECS tasks.</pre><pre>5. Troubleshoot why tasks fail to start.</pre><pre>6. Fix the problem, make changes into terraform config files.</pre><pre>7. Apply terraform configuration changes.</pre><pre>8. Go to ECS console to see ECS tasks running.</pre><h4>Troubleshoot ECS task failing in ECS service attached to ELB</h4><p>ELB attached to ECS service automatically checks the health of the tasks in your service. If it finds an unhealthy task, it stops sending traffic to the instance and reroutes traffic to healthy instances. Then ECS stops unhealthy task and starts another instance of that task.</p><p>Top reasons of failing ECS tasks in ECS attached to ELB:</p><pre><strong>Application running in container fails to respond to ELB health checks.</strong></pre><pre><strong>Troubleshooting</strong>: ECS service Events in web console.</pre><pre><strong>Reason</strong>: bug in application code, misconfigured health checks or port mapping, no access from ELB to EC2 instance.</pre><pre><strong>Solution</strong>: eliminate reason of the problem e.g. fix EC2 security group, make changes to ELB health check config etc.</pre><p>Go ahead and troubleshoot why ECS service doesn’t stabilize:</p><pre>Create ECS service with ELB attached (troubleshoot flapping container -- it’s started and then stopped by ECS scheduler). <strong>You should have EC2 instance registered in ELB with status InService.</strong></pre><pre>1. Uncomment resources in <strong>ecs_service_elb.tf</strong> file.</pre><pre>2. Finish configuration to create ECS service attached to ELB.</pre><pre>3. Do &quot;terraform plan&quot; and &quot;terraform apply&quot;.</pre><pre>4. Go to EC2 web console, find your ELB and see instance’s state.</pre><pre>5. Go to ECS web console to see what happens in newly created ECS service -- tasks are constantly restarted. Find out the reason of this happening.</pre><pre>6. Once the problem is fixed you should see EC2 instance in ELB in InService state. Sample application should be accessible via HTTP - open ELB&#39;s address in your browser.</pre><h4>How to track deployments into ECS?</h4><p>There are three deployment statuses of ECS service:</p><ul><li>PRIMARY — the most recent deployment</li><li>ACTIVE — previous deployments that still have tasks running, which are being replaced with the PRIMARY deployment</li><li>INACTIVE — deployments that have been completely replaced</li></ul><pre>1. Go to ECS service web console, choose ECS service and go to Deployments tab - you should see 1 deployment in state PRIMARY.</pre><pre>2. Change container definition - change allocated memory e.g. from 100 to 128.</pre><pre>3. Run terraform plan and terraform apply.</pre><pre>4. Return to ECS service console, choose Deployments tab in ECS service - you should see 2 deployments - one of them is ACTIVE and another is PRIMARY.</pre><p>Here in Smartling, we’re using Lambda function to detect anomalies in ECS services like failed and/or stuck deployments. You may look into it in optional hands on step of this workshop (see Bonus Track section below).</p><h4>Deployments into ECS with zero downtime</h4><p>ECS service deployment configuration controls how many ECS tasks may run during the deployment and the ordering of stopping and starting tasks:</p><ol><li><strong>minimumHealthyPercent</strong> (the lower limit of the number of running tasks that must remain running in a service during a deployment).</li><li><strong>maximumPercent</strong> (the upper limit of the number of running tasks that must remain running in a service during a deployment).</li></ol><p>Default ECS deployment configuration has minimumHealthyPercent = 100% that means that your ECS cluster’s capacity must be enough to start new version of application while still running old version.</p><pre>1. Configure ECS service to have deployments with zero downtime:</pre><pre>a. Change setup of ASG: change min = max = 2 to add one more instance into ECS cluster.</pre><pre>b. Change desired_count in ECS service (one attached to ELB) from 1 to 2.</pre><pre>c. Run terraform plan and terraform apply.</pre><pre>d. Wait until ELB has two instances InService. It may take 3-4 minutes for second instance to bootup and run tasks.</pre><pre>e. SSH into one of instances and start simple uptime checker for our sample service:</pre><pre>while true; do curl -m 5 -sko/dev/null &quot;http://<strong>&lt;elb_dns_name&gt;</strong>:80/&quot; -w &quot;%{http_code} %{time_appconnect} %{time_connect} %{time_total}\n&quot; 2&gt;&amp;1;sleep 1; done</pre><pre>Where elb_dns_name is DNS name of ELB that is attached to ECS service. You can find it in terraform.tfstate or in AWS web console.</pre><pre>If service is up and running you should see smth like this:</pre><pre>200 0.000 0.013 0.017<br>200 0.000 0.005 0.008<br>200 0.000 0.005 0.007<br>200 0.000 0.005 0.007</pre><pre>f. Change docker image from <strong>anosulchik/workshop-sample-application:v1.0</strong> to <strong>anosulchik/workshop-sample-application:v2.0</strong> in containers definition.</pre><pre>g. terraform plan, terraform apply.</pre><pre>h. Review output of uptime checker - you should see 503 returned by ELB that means there’s downtime during deployment.</pre><pre>i. Make changes into terraform configuration to deploy version <strong>v1.0</strong> again with zero downtime. It’s required to make changes into ELB and ECS service resources.</pre><pre>k. terraform plan, terraform apply.</pre><pre>l. Check uptime checker’s output to make sure there was no downtime for service during deployment.</pre><h4>Bonus Track</h4><pre><strong>Automate detection of failed ECS deployments.</strong></pre><pre>1. Uncomment <strong>lambda.tf</strong> file, finish configuration to create lambda function that detects failed deploys into ECS service attached to ELB (deployed in previous step on this workshop).</pre><pre>2. Make intentional mistake in task definition - specify non-existent docker image.</pre><pre>3. Subscribe your email to SNS topic that is used by Lambda to send notifications.</pre><pre>4. Run terraform plan, terraform apply.</pre><pre>5. You should get email with notification about failed deployment as soon as Lambda will be executed by CloudWatch event rule.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13ed1bfdf1c4" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-7-aws-ecs-insights-and-troubleshooting-13ed1bfdf1c4">AWS/Terraform Workshop #7: AWS ECS insights and troubleshooting</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e60686c1f71</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[terrorism]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Mon, 22 May 2017 20:20:13 GMT</pubDate>
            <atom:updated>2017-05-23T15:14:59.709Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of our AWS/Terraform Workshops series that explores our vision for Service Oriented Architecture (SOA), and closely examines AWS Simple Storage Service, Terraform Remote State, and Identity Access Management. To learn more, check out our </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="https://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*8r2bLv8hJq_Kg1N7IEfiQA.png" /></figure><h3>Prerequisites</h3><ul><li><strong>Go through workshops </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>#1</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l"><strong>#2</strong></a><strong>, </strong><a href="https://medium.com/@anosulchik/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f#.7tlpeayrd"><strong>#3</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7"><strong>#5</strong></a></li><li><a href="https://docs.docker.com/engine/userguide/intro/"><strong>Docker Introduction</strong></a></li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html"><strong>AWS EC2 Container Service Introduction</strong></a></li></ul><h3>Preface</h3><h4>AWS EC2 Container Service (Amazon ECS)</h4><p>Amazon ECS is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. It lets you launch and stop container-based applications with simple API calls, allows you to get the state of your cluster from a centralized service, and gives you access to many familiar Amazon EC2 features.</p><ul><li><strong>Cluster</strong>: A logical grouping of container instances on which you can place tasks.</li><li><strong>Container instance</strong>: EC2 instance that is running the Amazon ECS agent and has been registered into a cluster.</li><li><strong>Task definition</strong>: A description of an application that contains one or more container definitions.</li><li><strong>Task</strong>: An instantiation of a task definition that is running on a container instance.</li><li><strong>Service</strong>: ECS service allows you to run a specified number of tasks simultaneously. If any of the tasks fails or stops for any reason (e.g. OOM), ECS service scheduler re-launches the failed task.</li></ul><p>It’s possible to attach <strong>Elastic Load Balancer (ELB) </strong>to ECS service to distribute traffic across the multiple instances of the service.</p><p>You can <strong>update a running service</strong> to change the number of tasks (desired count) and/or definition for the tasks in ECS service.</p><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_clusters.html">AWS ECS Clusters</a></li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html">AWS ECS Container Instances</a></li><li><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html">AWS ECS Services</a></li></ul><h4>AWS Lambda for synchronization of Auto Scale group and ECS</h4><p>In this workshop, we will setup AWS Lambda function that will sync a desired count of tasks in ECS service depending on the number of EC2 instances in the Auto Scaling group. This is now natively supported by AWS, therefore, mentioning the Lambda function may be used for learning purposes.</p><p><em>Note: We recommend using </em><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html"><strong><em>AWS ECS service Auto Scaling</em></strong></a><em> in production environments instead of the Lambda function presented in this workshop.</em></p><h3>Hands On</h3><h4>1. Create EC2 instance with IAM role with full access to AWS.</h4><pre><em>Note: there are no intentional mistakes or confusions at this step. But next sections in this workshop still contain some so you&#39;ll need to fix them.</em></pre><pre>a. Go to w6/workshop_part_1 directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>b. Configure terraform to create IAM role, security group and EC2 instance.</pre><pre>c. Make sure that user data for EC2 instance contains your SSH key.</pre><pre>d. Apply terraform configuration:</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre>e. Login to newly created EC2 instance via SSH.</pre><pre>f. Checkout git repository with terraform workshops.</pre><pre>$ git clone <a href="https://github.com/Smartling/aws-terraform-workshops">https://github.com/Smartling/aws-terraform-workshops</a><br>$ cd w6/workshop_part_2</pre><h4>2. Create ASG and ECS cluster with EC2 instance registered in it.</h4><p><strong>ECS cluster</strong> is a logical grouping of container instances on which you can place tasks.</p><p><strong>Container instance</strong>: EC2 instance that is running the Amazon ECS agent and has been registered into a cluster. Instances of different types can be included into a cluster. Container instance can be a member of one cluster at a time. Type of container instance determines resources available within the cluster.</p><p><strong>ECS agent</strong> makes API calls to ECS service on your behalf, therefore container instances must be created in an instance profile that opens the required permissions.</p><pre><em>Note: Credentials for Terraform are not required for these steps as “terraform apply” will be executed on EC2 instance which has full access to AWS.</em></pre><pre>a. Go to w6/workshop_part_2 directory at newly created EC2 instance and configure terraform to create ECS cluster. You may use <strong>mcedit</strong>, <strong>vim</strong> or <strong>nano</strong> command line text editors.</pre><pre>b. Apply terraform configuration:</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre><em>Note: At this step you should run terraform in w6/workshop_part_2 directory at newly created EC2 instance.</em></pre><pre>c. Go to AWS ECS console to see newly created ECS cluster.</pre><pre>d. Configure terraform to create Auto Scaling Group and instances registered in this ECS cluster.</pre><pre>e. Check AWS ECS web console to see new container instance.</pre><h4>3. Create ECS service with ELB attached to it.</h4><p><strong>ECS service</strong> allows you to simultaneously run and maintain a specified number of instances of a task definition simultaneously. If any of the tasks fails or stops for any reason (e.g. crash, OOM), ECS service scheduler launches another instance of task definition to replace it and maintain the desired count of tasks in the service. Number of tasks running in ECS service is specified in its parameter <strong>desired_count</strong>.</p><p>If attached to ECS service <strong>Elastic Load Balancer</strong> (ELB), it distributes traffic across the tasks that are associated with ECS service. ECS and ELB take care of adding and removing container instances into/from balancing, without downtime for the application running in containers. ECS can have one ELB attached.</p><p>In order to attach the ELB to ECS service you must specify <strong>IAM role</strong> that will allow ECS service to communicate with ELB on your behalf.</p><p>After ECS service is created, container port, container name, ELB specified in ECS service configuration cannot be changed. In order to change them you must re-create ECS service.</p><p><strong>ECS task definition</strong> contains information about containers which should be started at container instances. This information includes: docker images, CPU, and memory available for containers, links between containers, volumes shared (or not) by containers, ports mapping between containers and container instance, and container configuration environment variables. Task definition structure:</p><pre>{<br> “family”: “myservice”, <br> “containerDefinitions”: [<br> {<br>   “name”: “myservice-container-one”, <br>   “image”: “myservice-image:latest”, <br>   “cpu”: 100, <br>   “memory”: 512, <br>   “portMappings”: [<br>   { “containerPort”: 8080, “hostPort”: 80, “protocol”: “tcp” }<br>   …<br>   ],<br>   “environment”: [ { “name”: “variable1”, “value”: “value1” } ],<br>   …<br>  }<br>  …<br>}</pre><p><strong>ignore_changes</strong> is meta-parameter in Terraform that customizes how diffs are evaluated for resources, allowing individual attributes to be ignored through changes. For example, terraform will not apply desired_count specified in ECS service configuration if ECS service already exists.</p><pre>a. Configure terraform to create sample ECS service with ELB.</pre><pre>  i. Notice ignore_changes configuration in ECS service definition.</pre><pre>  ii. Apply Terraform configuration.</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre>b. Use ELB DNS name to open sample service in browser: check events for ECS service in AWS web console to see why tasks aren’t starting at container instance.</pre><pre>c. Adjust terraform configuration and apply it to fix the issue.</pre><h4>4. Initiate deployment to ECS service.</h4><p>You can <strong>update a running ECS service</strong> to change the number of tasks that are maintained by a service (desired_count) or decide which task definition is used by the tasks.</p><p><strong>ECS service deployment configuration</strong> is an optional parameter for ECS service that controls how many tasks run during the deployment and the ordering of stopping and starting tasks: <strong>minimumHealthyPercent</strong> (the lower limit of the number of running tasks that must remain running in a service during a deployment) and <strong>maximumPercent</strong> (the upper limit of the number of running tasks that must remain running in a service during a deployment).</p><p>Default ECS deployment configuration has minimumHealthyPercent = 100% that means that your ECS cluster capacity must have enough container instances to start a new version of the application while still running the old version.</p><pre>a. Change docker image name from <strong>anosulchik/workshop-sample-application:v1.0 </strong>to <strong>anosulchik/workshop-sample-application:v2.0 </strong>in containers.txt that defines containers in ECS task.</pre><pre>b. Apply Terraform configuration.</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre>c. Use ELB DNS name to open sample service in browser: You should see the new version v2.0 of the sample application here.</pre><h4>5. Create Lambda function to sync ASG desired EC2 instances capacity with ECS desired tasks count.</h4><pre>a. Configure terraform to bind SNS topic used by ASG to send notifications to trigger Lambda function.</pre><pre>b. Apply Terraform configuration.</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre>c. Uncomment resource in lambda.tf and apply terraform configuration.</pre><pre>d. Change min_size = max_size = 2 in ASG configuration.</pre><pre>e. Check Lambda function’s logs and ECS service desired count value in web console (you should see desired_count = 2 here applied by Lambda).</pre><pre>f. Change min_size = max_size = 1 in ASG configuration.</pre><h4>6. Destroy AWS resources created in this workshop.</h4><pre><em>Note: Order is extremely important here — run Terraform destroy at EC2 instance first, then locally.</em></pre><pre>a. Run “terraform destroy” command on EC2 instance where you logged into via SSH, in the <strong>w6/workshop_part_1</strong> directory.</pre><pre>b. Run “terraform destroy” on your computer in <strong>w6/workshop_part_2</strong> directory.</pre><p>Introduction:</p><p><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></p><p><strong>Workshop #1:</strong></p><p><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a></p><p><strong>Workshop #2:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a></p><p><strong>Workshop #3:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a></p><p><strong>Workshop #4:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a></p><p><strong>Workshop #5:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e60686c1f71" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #5: AWS Lambda Functions]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/9af79af996b7</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Mon, 27 Mar 2017 15:08:14 GMT</pubDate>
            <atom:updated>2017-05-23T07:31:05.080Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of our AWS/Terraform Workshops series that explores our vision for Service Oriented Architecture (SOA) and closely examines AWS Simple Storage Service, Terraform Remote State, and Identity Access Management. To learn more, check out our </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="https://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*k6KoP-bsw86ru1q_7r2h8w.png" /></figure><h3>Prerequisites</h3><ul><li><strong>Go through Workshop </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>#1</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l"><strong>#2</strong></a><strong>, </strong><a href="https://medium.com/@anosulchik/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f#.7tlpeayrd"><strong>#3</strong></a><strong> and </strong><a href="https://medium.com/@anosulchik/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1#.9bjezbydl"><strong>#4</strong></a></li><li><a href="http://docs.aws.amazon.com/lambda/latest/dg/welcome.html"><strong>Amazon AWS Lambda Introduction</strong></a></li></ul><h3>Preface</h3><p><strong>AWS Lambda</strong> is a compute service where you can upload your code and the application will run it on your behalf using AWS infrastructure. Once code is uploaded as a Lambda function, AWS will take care of provisioning and managing the servers which will be used to run it. There are three languages currently supported: <strong>Node.js, Python 2.7 </strong>and<strong> Java</strong>.</p><p>A common application of Lambda is for event-driven compute service where code is executed in response to events, such as changes to data in an S3 bucket, message in SNS topic, CloudWatch event etc.</p><p>Lambda function consists of code, associated dependencies and configuration. In configuration you specify how much <strong>memory</strong> should be allocated, the <strong>execution timeout, </strong>and the <strong>IAM role</strong> that Lambda will assume to execute code on your behalf. For Lambda function it is also required to specify a <strong>handler</strong> (that is, a filename and name of method/function in your code) where AWS Lambda can begin executing your code.</p><p><em>Notice: CPU power cannot be specified in Lambda configuration. AWS Lambda allocates it proportionally to allocated memory.</em></p><p><em>Notice:</em><strong><em> 1536 MB</em></strong><em> is the maximum memory that can be allocated to Lambda function, with a max execution duration of </em><strong><em>300 seconds</em></strong><em>. More limits can be found </em><a href="http://docs.aws.amazon.com/lambda/latest/dg/limits.html"><em>here</em></a><em>.</em></p><p><strong>Event sources</strong> publish events that cause the Lambda function to be invoked. You associate an event source with your Lambda function using an <strong>event source mapping</strong>. List of event sources supported by AWS Lambda (push invocation model) includes: <strong>SNS</strong> (when you push a new message to an Amazon SNS topic, it can trigger a Lambda function), <strong>scheduled events </strong>(you can set up AWS Lambda to invoke your code on a regular, scheduled basis using the schedule event capability in CloudWatch), and <strong>S3</strong> (you can configure notification on an Amazon S3 bucket to publish bucket events, such as when objects are created or deleted, to AWS Lambda and invoke a Lambda function to respond to these events).</p><p><em>Notice: </em><strong><em>Scheduled event sources</em></strong><em> are defined as CloudWatch (event rules specify expression for schedules).</em></p><p><em>Notice: Scheduled CloudWatch event sources support a minimum interval of </em><strong><em>5 minutes</em></strong><em>. If you wish to trigger Lambda function faster you’ll need to write your own scheduler so that it will invoke Lambda with the desired frequency.</em></p><p>In addition to invoking Lambda functions using event sources, you can also invoke your Lambda function over HTTPS. You can do this by defining a custom REST API and endpoint using Amazon API Gateway. This is beyond the scope of this workshop , and you can find <a href="http://aws.amazon.com/api-gateway/">more info here</a>.</p><p><strong>Execution role</strong> — IAM role must grant the permissions that your Lambda function needs (e.g. read objects in S3, read messages in SNS, permissions to modify AWS resources according to code).</p><p><em>Notice: Execution role specifies permissions for Lambda function itself but not permissions for those entities which trigger Lambda function. So in order to make it possible for S3, SNS or CloudWatch to trigger Lambda function you should use</em><a href="http://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html"><em> </em><strong><em>AddPermission</em></strong></a><em> API call to add those permissions (or use terraform</em><a href="https://www.terraform.io/docs/providers/aws/r/lambda_permission.html"><em> aws_lambda_permission</em></a><em> resource instead).</em></p><p><em>Notice: IAM role specified as execution role for Lambda function must include AWS Lambda service in its </em><strong><em>trust policy</em></strong><em> to allow the function to assume this role.</em></p><p>Lambda function <strong>execution metrics and logs</strong>. AWS Lambda automatically monitors Lambda functions on your behalf, reporting metrics through Amazon CloudWatch. To help you troubleshoot failures in a function, Lambda logs all requests handled by your function and also automatically stores logs generated by your code through Amazon CloudWatch Logs.</p><p><em>Notice: In order to allow Lambda function to create logs IAM role specified in function’s configuration must have corresponding permissions to CloudWatch.</em></p><h3>Hands On</h3><pre>1. Go to w5 directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>2. Create Autoscaling group (ASG), attach ELB to ASG.</pre><pre>a. Finish incomplete terraform configuration and <strong>be prepared to fix mistakes</strong>.<br>  <br>  b. Attach ELB to ASG (do not enable ELB checks for ASG, keep default EC2).</pre><pre>c. terraform plan, terraform apply:</pre><pre>$ aws-profile yourteam-dev terraform plan<br>$ aws-profile yourteam-dev terraform apply</pre><pre>d. Check AWS resources created in this step.</pre><pre>e. Make sure ELB DNS name can be opened with browser -- it should show nginx welcome page.</pre><pre>3. Create SNS topic, subscribe your email to it.</pre><pre>4. Create Lambda function for monitoring nginx behind ELB, it will send check the results to SNS (and to your mailbox).</pre><pre>a. Finish incomplete terraform configuration to create Lambda function triggered by CloudWatch events every 5 minutes.</pre><pre>b. terraform plan, terraform apply.</pre><pre>c. Go to AWS Lambda console and change check <strong>URL and SNS</strong> in Lambda function’s code:</pre><pre><em>Note: Make sure you specified actual DNS name of your ELB (it can be found in AWS web console or in tfstate file)</em></pre><pre>d. Try to trigger Lambda function in console manually, consider what should be changed in case lambda execution fails.</pre><pre>e. Check lambda function execution logs in CloudWatch.</pre><pre>5. Simulate service outage by stopping nginx at instance in ASG.</pre><pre>a. Options:</pre><pre>i. Go to instance via SSH and run ‘sudo service docker stop’ command to shutdown container with nginx.</pre><pre>ii. Adjust rules in ec2 security group in terraform configuration to prevent ELB from communicating with instance.</pre><pre>b. Check that ELB doesn’t show nginx welcome page in browser anymore.</pre><pre>c. Wait until Lambda function is executed by Cloudwatch schedule.<br>Make sure your received email about failed web check to your mailbox.</pre><pre>6. <strong>Destroy aws resources by corresponding terraform command.</strong></pre><pre>(optional) 7. Change code of Lambda function to send custom metrics into CloudWatch e.g. <strong>1</strong> when web check succeeded and <strong>0</strong> when it failed.</pre><p><strong>Introductory story:</strong></p><p><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></p><p><strong>Series of workshops:</strong></p><ul><li><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9af79af996b7" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/71c0d7512bf1</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[workshop]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Fri, 10 Feb 2017 17:40:59 GMT</pubDate>
            <atom:updated>2017-05-25T15:41:03.172Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of our AWS/Terraform Workshops series that explores our vision for Service Oriented Architecture (SOA) and closely examines AWS Simple Storage Service, Terraform Remote State, and Identity Access Management. To learn more, check out our </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="https://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*s8mwAOWdb2choX4pj9ozvw.png" /></figure><h3>Prerequisites</h3><ul><li><strong>Go through Workshops </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>#1</strong></a><strong>, </strong><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l"><strong>#2</strong></a><strong>, </strong><a href="https://medium.com/@anosulchik/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f#.7tlpeayrd"><strong>#3</strong></a></li><li><a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html"><strong>AWS S3 Introduction</strong></a></li><li><a href="https://www.terraform.io/docs/commands/remote-config.html"><strong>Terraform Remote State</strong></a></li><li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html"><strong>AWS IAM Introduction</strong></a></li><li><a href="https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins"><strong>Jenkins Introduction</strong></a></li></ul><h3>Preface</h3><h4>AWS Simple Storage Service (S3)</h4><p>Amazon S3 stores data as objects within buckets. An object consists of a file and any metadata that describes that file.</p><p><strong>Buckets</strong> are the containers for objects and there can be multiple buckets. You can control access to each bucket (i.e. who can create, delete, and list objects in that bucket), view access logs for each bucket and its objects, and choose the geographical region where Amazon S3 will store the bucket and its contents.</p><p><strong>Permissions for buckets and objects . </strong>You can specify permissions and attach resource-based policies to specific buckets and objects to determine which parts of aws infrastructure can have access to the resources. It’s possible to control objects creation, updates, deletes and lists. By default S3 resources are private and are available only to the resource owner (or the entity that created bucket and/or object).</p><p><strong>Object versioning.</strong> When turned on, object versioning enables you to maintain multiple versions of an object in one bucket, for example, my-image.jpg (version 111111) and my-image.jpg (version 222222). You might want to enable versioning to protect yourself from unintended overwrites and deletions, or to archive objects so that you can retrieve previous versions of them later on.</p><p><strong>S3 access logs.</strong> In order to track requests for access to your bucket, you can enable access logging. Each access log record provides details about a single access request, such as the requester, bucket name, request time, request action, response status, and error code, if any.</p><p>In order to enable logging you must specify the name of the S3 bucket that will store logs from another S3 bucket (that you enabled logging for).</p><p><strong>Read more:</strong></p><ul><li><a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html">Working with AWS S3 buckets</a></li><li><a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html">AWS S3 object versioning</a></li><li><a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html">AWS S3 access logs</a></li></ul><h4>Terraform Remote State</h4><p>By default, Terraform persists its state only to a local disk. When remote state storage is enabled, Terraform will automatically fetch the latest state from the remote server when necessary and if any updates are made, the newest state is persisted back to the remote server. In this mode, users do not need to durably store the state using version control or shared storage.</p><p>One of backends supported by Terraform for remote state is <strong>AWS</strong> <strong>S3</strong>.</p><p>Remote state gives you more than just easier version control and safer storage. It also allows you to delegate the <a href="https://www.terraform.io/docs/configuration/outputs.html">outputs</a> to other teams. This allows your infrastructure to be broken down more easily into components that multiple teams can access. Put another way, remote state also allows teams to share infrastructure resources in a read-only way.</p><p><strong>Read more:</strong></p><ul><li><a href="https://www.terraform.io/docs/state/remote/s3.html">Terraform Remote State S3 backend</a></li></ul><h4>AWS Identity Access Management (IAM)</h4><p>Identity and Access Management service helps to control access to AWS resources , including who can access (authentication) and what resources they can use and in what ways (authorization).</p><p><strong>Permissions and policies.</strong> You grant permissions by creating a policy, which is a document that lists the actions that a user can perform and the resources that the actions can affect. Example policy document:</p><pre>{ <br> &quot;Version&quot;: &quot;2012-10-17&quot;, <br> &quot;Statement&quot;: { <br>   &quot;Effect&quot;: &quot;Allow&quot;, <br>   &quot;Action&quot;: &quot;s3:*&quot;, <br>   &quot;Resource&quot;: &quot;arn:aws:s3:us-east-1::mybucket/*&quot; <br> } <br>}</pre><p><strong>Users, groups, roles, instance profiles.</strong> When you attach the policy to a user , that user gets permissions specified in the policy document. A policy attached to a group is applied to all users that are included in this group.</p><p><strong>IAM roles. </strong>An IAM role is similar to a user, in that it is an AWS identity with permission policies that determines what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it. Also, a role does not have any credentials (password or access keys) associated with it. Instead, if a user is assigned to a role, access keys are created dynamically and provided to the user.</p><p><strong>Instance profiles. </strong>An instance profile is a container for an IAM role that you can use to pass role information to an EC2 instance when the instance starts.</p><p><strong>Read more:</strong></p><ul><li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions.html">Overview of IAM permissions</a></li><li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html">Overview of IAM policies</a></li><li><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/id.html">Identities (Users, Groups, and Roles)</a></li></ul><h3>Hands On</h3><pre>1. Go to w4 directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>2. <strong>Create S3 bucket</strong> for terraform remote state:</pre><pre>  a. cd remote_state, edit file <strong>s3.tf</strong></pre><pre>  b. Define S3 bucket in terraform configuration (make sure versioning is enabled for it).</pre><pre>  c. Apply terraform configuration:</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre>Note: Names of S3 buckets must be unique with AWS S3 service so if anyone already took your bucket name, just use something like <strong>mybucket-w4-workshop-yourname</strong>.</pre><pre>  d. cd ../jenkins</pre><pre>  e. Configure terraform to use newly created S3 bucket as a remote state:</pre><pre>$ terraform remote config -backend=S3 -backend-config=&quot;bucket=<strong>mybucket-w4-workshop</strong>&quot; -backend-config=&quot;key=/terraform.tfstate&quot; -backend-config=&quot;region=us-east-1&quot;</pre><pre>  f. Check <strong>.terraform/terraform.tfstate</strong> file – you should see remote config section there e.g.:</pre><pre>$ cat .terraform/terraform.tfstate<br>...<br>&quot;remote&quot;: {<br>        &quot;type&quot;: &quot;s3&quot;,<br>        &quot;config&quot;: {<br>            &quot;bucket&quot;: &quot;mybucket-w4-workshop&quot;,<br>            &quot;key&quot;: &quot;/terraform.tfstate&quot;,<br>            &quot;region&quot;: &quot;us-east-1&quot;<br>        }<br>    },<br>...</pre><pre>2. <strong>Deploy Jenkins</strong> using terraform.</pre><pre>  a. Define missing resources in terraform configuration according to comments <strong>*.tf</strong> files</pre><pre>  b. Make sure user-data for Jenkins ec2 instance contains your public SSH key terraform plan.</pre><pre>  c. terraform plan and apply</pre><pre>  d. Get public IP address of instance that hosts Jenkins and open <em>http://&lt;ip address&gt;:8000</em> is browser. You&#39;ll be asked to provide password that can be obtained at Ec2 instance so you&#39;ll need to access it via SSH.</pre><pre><em>Note: it takes about 4 minutes for Jenkins to bootstrap before it shows welcome/installation page.</em></pre><pre>  e. Follow instructions on the screen to install Jenkins.</pre><pre>3. <strong>Configure Jenkins</strong> to build, test and deploy sample project.</pre><pre>  a. Go to “Manage Jenkins → Manage Plugins” section and install “Git plugin” (use search bar with exact name of plugin).</pre><pre>  b. Make sure you choose to restart Jenkins during the installation.</pre><pre>  c. Create new Project:</pre><pre>     i. Press “create new jobs” at Jenkins, specify Project name, select Freestyle project.<br>    <br>     ii. Configure Job with Git repository of sample project (choose some open source project at Github and configure checkout step for it).</pre><pre>     iii. Save changes.</pre><pre>     iiii. Go to newly created Project and press ‘Build Now’ -- look around and check that the build was successful (find out where build’s log can be found).</pre><pre>  d. Add build actions for the project.</pre><pre>     i. Go to Project&#39;s configuration.</pre><pre>     ii. Add one or more build steps (choose Execute Shell and add sample commands like &quot;ls -la&quot;, &quot;pwd&quot;, &quot;date&quot; etc.).</pre><pre>4. <strong>Destroy AWS resources</strong> using ‘terraform destroy’ command.</pre><pre>$ cd jenkins<br>$ terraform destroy<br> <br>$ cd ../remote_state<br>$ terraform destroy</pre><pre><em>Note: terraform can&#39;t delete S3 bucket because it’s not empty so you may need to go to S3 web console and delete all files and all their versions for remote tfstate file.</em></pre><p><strong>Introductory story:</strong></p><p><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></p><p><strong>Series of workshops:</strong></p><ul><li><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a></li></ul><p>Did you find our workshops useful? Click the 💙 below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71c0d7512bf1" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #3: ELB, SNS, AutoScaling]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/210e2337057f</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[elb]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Mon, 06 Feb 2017 19:42:51 GMT</pubDate>
            <atom:updated>2017-05-23T07:34:34.944Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is part of AWS/Terraform Workshops series that we share with you along with our vision of Service Oriented Architecture (SOA). Check out </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="http://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ckd0A4Km-DfYw0s5IF0OKg.png" /></figure><h3>Prerequisites</h3><ul><li><strong>Go through Workshops </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>#1</strong></a><strong> and </strong><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l"><strong>#2</strong></a></li><li><a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html"><strong>AWS Elastic Load Balancers (ELB)</strong></a></li><li><a href="http://docs.aws.amazon.com/sns/latest/dg/welcome.html"><strong>AWS Simple Notification Service (SNS)</strong></a></li></ul><h3>Preface</h3><h4>AWS Elastic Load Balancing (ELB)</h4><p><strong>AWS ELB</strong> automatically distributes incoming application traffic across multiple Amazon <a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima">EC2 instances</a>. It detects unhealthy instances and reroutes traffic to healthy instances until the unhealthy instances have been restored. Elastic Load Balancing automatically scales its request handling capacity in response to incoming traffic.</p><p><strong>Health checks</strong>: To discover availability of your EC2 instances, the load balancer periodically sends pings, attempts connections, or sends requests to test the EC2 instances. These tests are called health checks. The status of the instances that are healthy at the time of the health check is <strong>InService</strong>. The status of any instances that are unhealthy at the time of the health check is <strong>OutOfService</strong>. The load balancer performs health checks on all registered instances, whether the instance is in a healthy state or an unhealthy state.</p><p><strong>ELB Listener </strong>defines frontend and backend protocol/port for proxy connections. In case frontend protocol it HTTPS you will need to specify SSL certificate and ciphers.</p><p><strong>ELB Security Group </strong>acts as a firewall that controls the traffic allowed to and from one or more ELBs or instances.</p><p><strong>Additional ELB configurations:</strong></p><ul><li>To ensure that the load balancer stops sending requests to instances that are de-registering or unhealthy, while keeping the existing connections open, use <strong>connection draining</strong>. This enables the load balancer to complete in-flight requests made to instances that are de-registering or unhealthy.</li><li>By default, your load balancer distributes incoming requests evenly across its enabled <a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l">Availability Zones</a>. To ensure that your load balancer distributes incoming requests evenly across all back-end instances, regardless of the Availability Zone that they are in, enable<strong> cross-zone load balancing</strong>.</li><li>Elastic Load Balancing provides <strong>access logs</strong> that capture detailed information about all requests sent to your load balancer. Each log contains information such as the time the request was received, the client’s IP address, latencies, request paths, and server responses. You can use these access logs to analyze traffic patterns and to troubleshoot issues.</li></ul><p><strong>Auto Scaling (ASG) integrates with Elastic Load Balancing</strong> to enable you to attach one or more load balancers to an existing Auto Scaling group. After you attach the load balancer, it automatically registers the instances in the group and distributes incoming traffic across the instances.</p><p>By default, an Auto Scaling group determines the health state of each instance by periodically checking the results of the<strong> EC2 instance status checks</strong>. If an instance fails the EC2 instance status checks, Auto Scaling marks the instance as unhealthy and replaces the instance. However, if you have attached one or more load balancers to your Auto Scaling group and the instance fails the <strong>Elastic Load Balancing health checks</strong>, Auto Scaling does not replace the instance by default. You can configure your Auto Scaling group to use both EC2 instance status checks and Elastic Load Balancing health checks to determine the health status of your instances.</p><p>If <strong>connection draining</strong> is enabled for your load balancer, Auto Scaling waits for the in-flight requests to complete or for the maximum timeout to expire, whichever comes first, before terminating instances due to a scaling event or health check replacement.</p><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/what-is-load-balancing.html">What is Elastic Load Balancing?</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/attach-load-balancer-asg.html">Attaching a Load Balancer to Your Auto Scaling Group</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/as-add-elb-healthcheck.html">Adding Health Checks to Your Auto Scaling Group</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/healthcheck.html">Health Checks for Auto Scaling Instances</a></li><li><a href="http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-conn-drain.html">Connection Draining for ELB</a></li></ul><h4>AWS Simple Notification Service</h4><p><strong>AWS SNS </strong>is a web service that coordinates and manages the delivery or sending of messages to subscribing endpoints or clients. In Amazon SNS, there are two types of clients — publishers and subscribers — also referred to as <strong>producers</strong> and <strong>consumers</strong>. Publishers communicate asynchronously with subscribers by producing and sending a message to a topic, which is a logical access point and communication channel. Subscribers (i.e., web servers, email addresses etc) consume or receive the message or notification over one of the supported protocols (HTTP/S, email) when they are subscribed to the topic.</p><p>SNS is common way to enable communication between AWS components and part of infrastructure placed outside AWS, for example:</p><ul><li>You can configure <a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l">AWS CloudWatch</a> alarm to send ALARM action to SNS and subscribe to it via email to get notifications.</li><li>ASG dynamic scaling requires CloudWatch alarm to trigger <a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17#.jvz251u8l">ASG</a> scale policies, communication between CloudWatch and ASG is done via SNS.</li><li>Using SNS you can trigger AWS Lambda functions (covered in next workshops).</li></ul><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/sns/latest/dg/welcome.html">What is Amazon Simple Notification Service?</a></li></ul><h3>Hands On</h3><pre>1. Go to w3 directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>2. Specify actual IDs of <strong>AWS VPC</strong>,<strong> Subnet </strong>and<strong> Availability Zone</strong> into <em>terraform.tfvars</em><strong> </strong>file.</pre><pre><em>Note: Follow instructions in Hands On section of Workshop #2 or just copy terraform.tfvars file from it.</em></pre><pre>3. Add your public SSH key to <em>user-data.txt</em> file.</pre><pre>4. <strong>Create Autoscaling Group</strong>:</pre><pre>  a. Configure security group for instances in ASG to accept incoming connections via 22 and 80 TCP ports from 0.0.0.0/0.</pre><pre><em>Note: This is generally bad idea from security perspective to open access to your resources via 22 port from anywhere -- please avoid such setup in configurations other than workshop.</em></pre><pre>  b. Add missing arguments for ASG in <em>autoscaling.tf</em><strong> </strong>file:</pre><pre>    i. Min instances limit = 2, max instances = 2<br>    ii. Instance type t2.nano<br>    iii. Add references between AWS resources: attach ASG launch configuration to autoscaling group etc.</pre><pre><em>Note: Be prepared for mistakes made in terraform configuration intentionally – just fix them.</em></pre><pre>  c. Apply terraform configuration.</pre><pre>  d. Check newly created AWS resources in AWS web console.</pre><pre>5. <strong>Create AWS ELB and attach it to ASG</strong> created before.</pre><pre>  a. Uncomment resources in <strong>elb.tf </strong>and add finish configuration.</pre><pre>  b. Configure ELB Listener to accept HTTP connection on port 80 and forward them to port 80.</pre><pre>  c. Enable connection draining.</pre><pre>  d. Configure ELB health checks:</pre><pre>    i. Ping Target: &quot;HTTP:80/&quot;</pre><pre>    ii. Healthy threshold = 3</pre><pre>    iii. Unhealthy threshold = 3</pre><pre>    iiii. Timeout = 2</pre><pre>    iiiii. Interval = 5</pre><pre>  e. Put ELB in the same Security Group with instances in ASG.</pre><pre>  f. Apply Terraform configuration.</pre><pre>  g. Attach  ELB to ASG:</pre><pre>    i. Update ASG configuration in terraform<br>    ii. Configure ASG to use ELB metrics instead of EC2</pre><pre>  h. Find ELB endpoint and open it in browser – you should see nginx welcome page.</pre><pre>6. <strong>Create SNS topic</strong> to receive ASG scaling notifications to email.</pre><pre>  a. Uncomment SNS topic resource in <strong>sns.tf </strong>file.</pre><pre>  b. Apply terraform configuration</pre><pre>  c. Go to AWS SNS web console, find newly created SNS topic and create subscription to your email address.</pre><pre>  d. Update ASG configuration to send its scaling events to SNS topic</pre><pre>  e. Apply terraform configuration.</pre><pre>  f. Login to one of ec2 instances via SSH and stop docker &quot;sudo service docker stop&quot;. ASG will detect that instance in unhealthy (as it doesn&#39;t reply to health checks), will terminate it and will create new.</pre><pre>  g. Make sure you received notification from ASG to your email.</pre><pre>7. Destroy AWS resources.</pre><p><strong>Introductory story:</strong></p><p><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></p><p><strong>Workshop #1:</strong></p><p><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a></p><p><strong>Workshop #2:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a></p><p><strong>Workshop #4:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a></p><p><strong>Workshop #5:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a></p><p><strong>Workshop #6:</strong></p><p><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a></p><p>Did you find our workshops useful? Click the 💙 below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=210e2337057f" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch]]></title>
            <link>https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/12ee08c17</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[workshop]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Mon, 23 Jan 2017 15:01:07 GMT</pubDate>
            <atom:updated>2017-05-23T07:35:04.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*OQ3Ibe03GpJ8RePln274Wg.png" /></figure><p><em>This post is part of AWS/Terraform Workshops series that we share with you along with our vision of Service Oriented Architecture (SOA). Check out </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.g3mlnlwsr"><em>introductory workshop</em></a><em> and new posts at </em><a href="http://tech.smartling.com/"><em>Smartling Engineering Blog</em></a><em>.</em></p><h3>Prerequisites</h3><ul><li><strong>Go through </strong><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.3ikhwxima"><strong>Workshop #1</strong></a></li><li><strong>Amazon AWS EC2 Networking</strong>: <br><a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">Regions and Availability Zones</a><br><a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html">Virtual Private Cloud (VPC) and Subnets</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html"><strong>AWS Auto Scaling Groups</strong></a></li><li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html"><strong>AWS CloudWatch</strong></a></li></ul><h3>Preface</h3><h4>AWS EC2 Networking Concepts</h4><p>Amazon EC2 is hosted in multiple locations world-wide composed of <strong>Regions</strong> and <strong>Availability Zones</strong> (AZs). Each region is a separate geographic area that has multiple, isolated locations known as Availability Zones. Amazon EC2 provides you the ability to place resources, such as EC2 instances and data in multiple locations. Resources aren’t replicated across regions unless you do so specifically.</p><p>By placing your resources in <strong>multiple AZs</strong> you increase probability that failures which may occur on Amazon side won’t cause complete outage of your service but just a temporary drop in its capacity and most probably slowdown in its performance. Amazon data centers provide high speed connections between AZs but you should keep in mind that usually inter-AZ communications <strong>adds about 1 ms latency</strong> in comparison to communications within the same AZ.</p><p>Amazon <strong>Virtual Private Cloud (VPC)</strong> enables you to launch AWS resources in a virtual network that by default is isolated from other networks. VPC in AWS can be configured with IP address range, routing tables, subnets, network gateways and security settings like ACLs. In Smartling’s SOA tech stack VPC comes preconfigured so developers shouldn’t worry about networking configurations like IP addressing scheme, routing etc. At the same time the possibility to make changes into AWS networking configuration is open to service owners.</p><p><strong>Subnet</strong> is a range of IP addresses in your VPC so you can launch AWS resources into a subnet that you select. <strong>Each subnet resides within one AZ and cannot span zones.</strong> There are <strong>public </strong>and<strong> private subnets</strong>. Public subnets make it possible to make AWS resources like EC2 instances reachable from Internet. Private subnets are used for those resources which should not be exposed to Internet. Most of our SOA components reside in private subnets for security reasons.</p><p><strong>EC2 security group</strong> acts as a virtual firewall that controls the traffic for one or more EC2 instances. When you launch an instance, you associate one or more security groups with the instance. You can add rules to each security group that allow <strong>traffic to </strong>or<strong> from</strong> its associated instances. You can modify the rules for a security group at any time — the new rules are automatically applied to all instances that are associated with the security group.</p><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">Regions and Availability Zones</a></li><li><a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Introduction.html">VPC Introduction</a></li><li><a href="http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Subnets.html">VPCs and Subnets</a></li><li><a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html">Amazon EC2 Security Groups for Linux Instances</a></li></ul><h4>AWS CloudWatch</h4><p>Amazon CloudWatch monitors your AWS resources and the applications. You can use CloudWatch to collect and track <strong>metrics</strong>, which are the variables you want to measure for your resources and applications like CPU load, network traffic IO etc. <strong>CloudWatch</strong> <strong>alarms</strong> send notifications or automatically make changes to the resources you are monitoring based on rules that you define, for example trigger Autoscaling Group to scale.</p><p>A <strong>metric</strong> is the fundamental concept in CloudWatch. It represents a <strong>time-ordered set of data points</strong> that are published to CloudWatch. These data points can be either your custom metrics or metrics from services in AWS. You can retrieve statistics about those data points as an ordered set of time-series data. Think of a metric as a variable to monitor, and the data points represent the values of that variable over time. For example, the CPU usage of a particular Amazon EC2 instance is one metric, and the latency of an Elastic Load Balancing load balancer is another.</p><p>CloudWatch <strong>namespaces</strong> are containers for metrics. Metrics in different namespaces are isolated from each other, so that metrics from different applications are not mistakenly aggregated into the same statistics. Example namespaces AWS/EC2 for EC2 instances metrics and AWS/ELB for Elastic Load Balancers metrics.</p><p>A <strong>dimension</strong> is a name/value pair that helps you to uniquely identify a metric. Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics. Dimensions help you design a structure for your statistics plan. Because dimensions are part of the unique identifier for a metric, whenever you add a unique name/value pair to one of your metrics, you are creating a new metric. Examples:</p><ul><li><strong>AutoScalingGroupName</strong> (this dimension filters the data you request for all instances in a specified capacity group e.g. total CPU load within ASG)</li><li><strong>InstanceId</strong> (this dimension filters the data you request for the identified instance only)</li></ul><p>Cloudwatch <strong>alarm</strong> watches a single metric over a time period you specify, and performs one or more actions based on the value of the metric relative to a given threshold over a number of time periods. CloudWatch alarms will not invoke actions simply because they are in a particular state, the state must have changed and been maintained for a specified number of periods. After an alarm invokes an action due to a change in state, its subsequent behavior depends on the type of action that you have associated with the alarm. For Auto Scaling policy notifications, the alarm continues to invoke the action for every period that the alarm remains in the new state.</p><p>An alarm has three possible <strong>states</strong>:</p><ul><li><strong>OK</strong>: The metric is within the defined threshold</li><li><strong>ALARM</strong>: The metric is outside of the defined threshold</li><li><strong>INSUFFICIENT_DATA</strong>: The alarm has just started, the metric is not available, or not enough data is available for the metric to determine the alarm state</li></ul><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html">What is Amazon CloudWatch?</a></li><li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html">Amazon CloudWatch Concepts</a></li><li><a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html">Creating Amazon CloudWatch Alarms</a></li></ul><h4>AWS Autoscaling Groups</h4><p><strong>Auto Scaling</strong> helps you ensure that you have the correct number of EC2 instances available to handle the load for your application. You can create collections of EC2 instances called <strong>Auto Scaling Groups (ASG)</strong>. In each ASG you should specify minimum number of instances to ensure that your ASG never goes below this size. The same with maximum number of instances. If you specify <strong>desired capacity</strong>, either when you create the group or at any time thereafter, ASG ensures that your group has this many instances.</p><p><strong>ASG launch configuration</strong> is a template that an ASG uses to launch EC2 instances. When you create a launch configuration, you specify information for the instances such as the ID of the Amazon Machine Image (<strong>AMI</strong>), the instance type, security groups etc. <strong>Changes into launch configuration doesn’t trigger ASG to recreate existing instances with new template so you should rotate instances by your own.</strong></p><p>Auto Scaling provides several ways for you to <strong>scale your ASG</strong>:</p><ul><li><strong>Maintain current instance levels at all times</strong> (to maintain the current instance levels, Auto Scaling performs a periodic health check on running instances within ASG).</li><li><strong>Manual scaling</strong>. You only need to specify the change in the maximum, minimum, or desired capacity of your Auto Scaling group.</li><li><strong>Scale based on a schedule</strong>. Scaling by schedule means that scaling actions are performed automatically as a function of time and date.</li><li><strong>Scale based on demand</strong>. A more advanced way to scale your resources, scaling by policy, lets you define parameters that control the Auto Scaling process. For example, you can create a policy that calls for enlarging your fleet of EC2 instances whenever the average CPU utilization rate stays above ninety percent for fifteen minutes.</li></ul><p>Ondemand based scaling works in conjunction with<strong> AWS CloudWatch</strong> that collects metrics and trigger scaling process.</p><p>The Auto Scaling <strong>cooldown</strong> period is a configurable setting for your Auto Scaling group that helps to ensure that Auto Scaling doesn’t launch or terminate additional instances before the previous scaling activity takes effect. After the Auto Scaling group dynamically scales using a simple scaling policy, Auto Scaling waits for the cooldown period to complete before resuming scaling activities.</p><p>Auto Scaling enables you to put an instance that is in the InService state into the <strong>Standby state</strong>, update or troubleshoot the instance, and then return the instance to service. Instances that are on standby are still part of the Auto Scaling group, but they do not actively handle application traffic.</p><p>Auto Scaling enables you to <strong>suspend and then resume one or more of the Auto Scaling processes</strong> in your Auto Scaling group. This can be very useful when you want to investigate a configuration problem or other issue with your web application and then make changes to your application, without triggering the Auto Scaling process.</p><p>Read more:</p><ul><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/LaunchConfiguration.html">Launch Configurations</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/scaling_plan.html">Scaling the Size of Your Auto Scaling Group</a></li><li><a href="http://docs.aws.amazon.com/autoscaling/latest/userguide/CHAP_Troubleshooting.html">Troubleshooting Auto Scaling</a></li></ul><h3>Hands On</h3><pre>1. Login to <a href="https://console.aws.amazon.com/console/home?region=us-east-1#">AWS console</a>, switch to proper AWS account (if you are using multiple accounts) and go to <a href="https://console.aws.amazon.com/vpc/home?region=us-east-1">VPC Management Console</a>.</pre><pre>  a. Go to VPC section, select <strong>Your VPCs</strong> section and write down VPC ID.</pre><pre>  b. Go to <strong>Subnets </strong>section, choose private subnet and write down its ID as well as <strong>Availability Zone</strong> that it&#39;s bound to.</pre><pre><em>Note: In Smartling we&#39;re marking subnets with AWS tags so that it&#39;s easy to identify private and public subnets without digging into routing tables they are associated with.</em></pre><pre>3. Specify collected data in terraform configuration:</pre><pre>  a. Go to w2 directory in cloned <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> git repository.</pre><pre>  b. Edit file <em>terraform.tfvars</em>: specify VPC ID, Subnet ID and AZ, for example:</pre><pre>$ cat terraform.tfvars<br>vpc_id = &quot;vpc-1234567&quot;<br>subnet_id = &quot;subnet-1234567&quot;<br>availability_zone_id = &quot;us-east-1c&quot;</pre><pre>4. Follow terraform documentation for ASG and comments in <em>autoscaling.tf</em> file to complete Auto Scaling Group and Launch Configuration.</pre><pre>  a. Add missing names for terraform resources.</pre><pre>  b. Configure launch configuration to create one <strong>t2.micro</strong> instance in security group that is created in <strong>ec2.tf </strong>file.</pre><pre>  c. Set min_size = 1 and max_size = 3 in ASG, cooldown 60 seconds.</pre><pre>  d. <strong>Make sure user-data for EC2 instances in ASG contains your public SSH key.</strong></pre><pre>  e. Apply terraform configuration:</pre><pre>$ terraform plan<br>$ terraform apply</pre><pre><em>Note #1: always run terraform plan before apply and examine what is actually terraform is going to change/create/delete in AWS.</em></pre><pre><em>Note #2: you need to configure terraform with your AWS credentials here. There&#39;re multiple ways to do it and you can find one in our  </em><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c#.mwapdmllh"><em>AWS/Terraform Workshop #1</em></a><em>.</em></pre><pre>  f. Check newly created ASG in <a href="https://console.aws.amazon.com/ec2/v2/home?region=us-east-1">AWS EC2 Management Console</a>. You should see Auto Scaling group, Launch Configuration and EC2 instance created by ASG.</pre><pre>5. Uncomment code in terraform configuration files to create CloudWatch (CW) alarm to trigger ASG scaling up policy if total CPU load in ASG is more than 40%.</pre><pre>  a. Enable <strong>EC2 detailed monitoring</strong> for EC2 instances in ASG so that CloudWatch will collect metrics every 1 minute (<em>Hint: see docs for launch config terraform resource</em>).</pre><pre>  b. Configure CW alarm to add +1 instance if CPU load in ASG is more than 40%, cooldown = 60.</pre><pre>  c. Use CW alarm ARN to reference it in template.</pre><pre>  d. Apply terraform configuration.</pre><pre>  e. Check CW alarm in AWS web console.</pre><pre>6. Enable scaling protection for EC2 instance.</pre><pre>  a. Find your autoscaling group in <a href="https://console.aws.amazon.com/ec2/autoscaling/home?region=us-east-1#AutoScalingGroups:">AWS EC2 Management Console</a> and go to &quot;Instances&quot; tab.</pre><pre>  b. Select and instance, click on &quot;Actions-&gt;Instance protection-&gt;Set scale in protection&quot;.</pre><pre>7. Generate CPU load to trigger CloudWatch alarm and ASG scaling up process.</pre><pre>  a. Login to EC2 instance via SSH and run the following commands:</pre><pre>$ dd if=/dev/urandom bs=1M count=200096 | gzip -9 |gzip -9 | gzip -9 &gt;/dev/null</pre><pre>  b. Review ASG events in AWS web console, see +2 instances in 2 minutes. You can see it in Activity History tab for ASG.</pre><pre>8. Add scale down policy to remove 1x Ec2 instance in case CPU load in ASG is less than 35%.</pre><pre>  a. Create new CW alarm that will trigger scale down ASG policy.</pre><pre>  b. Apply terraform configuration.</pre><pre>9. Watch scaling activity for Auto Scaling Group.</pre><pre>10. Destroy AWS resources.</pre><pre>  a. <strong>Disable protection for instance in AWS web console.</strong></pre><pre>  b. Run destroy command:</pre><pre>$ terraform destroy</pre><pre><em>Note: It will take slightly more time to terminate all resources in AWS than in previous workshop.</em></pre><p><strong>Introductory story:</strong></p><p><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></p><p><strong>Series of workshops:</strong></p><ul><li><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12ee08c17" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Terraform and AWS EC2, first steps. Workshop #1]]></title>
            <link>https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/e38607f0fd4c</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Tue, 06 Dec 2016 20:56:50 GMT</pubDate>
            <atom:updated>2017-05-23T07:35:34.413Z</atom:updated>
            <content:encoded><![CDATA[<p>Getting started with Terraform and AWS EC2, first steps. Workshop #1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*Pnu3DVRvV_wGMhw12nkGGw.png" /></figure><p>Here at Smartling, we’re implementing Service Oriented Architecture (SOA). According to our vision of SOA and service ownership every tech team owns and runs services in Amazon Web Services (AWS). Developers from each team have their own AWS account and can launch, support, deploy and maintain AWS infrastructure for services they own using Terraform and a set of templates.</p><p>Our developers passed a set of AWS/Terraform workshops covering AWS basics like Elastic Compute (EC2), Elastic Block Storage (EBS), Elastic Load Balancing (ELB) as well as Terraform syntax and best practices in this field.</p><p>Now we’re sharing those workshops with you. In this post you’ll find our introductory workshop, covering Terraform and AWS EC2.</p><h3>Prerequisites</h3><ul><li><strong>Amazon AWS</strong>: <br><a href="http://searchaws.techtarget.com/feature/Amazon-Elastic-Compute-Cloud-features-explained">Read introduction to AWS</a><br><a href="http://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_how-users-sign-in.html">Get access to your AWS account</a><br><a href="http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html">Get your AWS access and secret keys</a></li><li><strong>Terraform</strong>:<br><a href="http://blog.scottlowe.org/2015/11/25/intro-to-terraform/">Read introduction to Terraform</a><br><a href="https://www.terraform.io/intro/getting-started/install.html">Install latest version Terraform</a></li><li><strong>Other</strong>:<br>Clone <a href="https://github.com/Smartling/aws-terraform-workshops">Smartling/aws-terraform-workshops</a> Git repository <br><a href="https://confluence.atlassian.com/bitbucketserver/creating-ssh-keys-776639788.html">Get ready your public SSH key and SSH client</a></li></ul><h3>Preface</h3><h4>EC2</h4><p>EC2 stands for <strong>Elastic Compute Cloud</strong> — service that provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using EC2 you can launch virtual servers, setup networking and security for them, attach storage, public IP addresses or associated domain names. EC2 lets you to scale up or down computing resources depending on changes in requirements or spikes in popularity, reducing your need to forecast traffic.</p><p>When you launch an instance, you associated it with one or more<strong> security groups:</strong> virtual firewalls enforcing a set of rules that control the traffic to and from instances. AMI (<strong>Amazon Machine Image</strong>) is a template that contains a software configuration, including an operating system and packages. Using AMI you can launch an instance that contains a copy of the AMI running as a virtual server in AWS cloud.</p><p>Locations for running EC2 instances are composed of <strong>regions</strong> and <strong>availability zones</strong>. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones. Amazon EC2 provides you the ability to place resources, such as instances, and data in multiple locations.</p><p>EBS (<strong>Elastic Block Store</strong>) provides block level storage volumes for use with EC2 instances. You can create highly reliable EBS volumes and attach them to any running instance that is in the same Availability Zone. EBS volumes that are attached to an EC2 instance are exposed as storage volumes that persist independently from the life of the instance.</p><p>On-demand EC2 instances let you pay by the hour with no long-term commitments. Price varies considerably, based on <strong>instance type</strong> and attached storage. Internet traffic, public IP addresses, load balancing and detailed monitoring resources are paid separately. There are options to reduce costs by upfront payments like instances reservations.</p><p><strong>Read more:</strong></p><ul><li><a href="http://searchaws.techtarget.com/feature/Amazon-Elastic-Compute-Cloud-features-explained">AWS EC2 Cloud Features Explained</a></li><li><a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html">AWS EC2 Concepts</a></li></ul><h4>Terraform</h4><p>Terraform is <strong>command line tool</strong> for building, changing, and versioning infrastructure, it supports popular cloud providers including Amazon AWS. <strong>Configuration files</strong> describe the infrastructure components needed to run a single application, for example, an EC2 instance with an attached EBS volume.</p><p>Terraform generates an <strong>execution plan,</strong> describing what it will do to reach the desired state, then executes it to build the described infrastructure. As the configuration changes, Terraform is able to determine what has changed and create incremental execution plans which can be applied. The actual state of AWS resources managed by terraform are stored in a <strong>terraform.tfstate</strong> file that is created after first run of terraform.</p><p><strong>Key features of Terraform</strong>:</p><ul><li><strong>Infrastructure as Code</strong>: Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used.</li><li><strong>Execution Plans</strong>: Terraform has a “planning” step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure.</li><li><strong>Resource Graph</strong>: Terraform builds a graph of all your resources in order to create or modify non-dependent resources. Because of this, Terraform builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure.</li><li><strong>Change Automation</strong>: Complex changesets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors.</li></ul><p>Example:</p><pre>resource &quot;aws_instance&quot; &quot;web&quot; {<br>  ami = &quot;ami-12345678&quot;<br>  instance_type = &quot;t1.micro&quot;<br>  tags {<br>    Name = &quot;HelloWorld&quot;<br>  }<br>  security_groups = [ &quot;${aws_security_group.my_security_group.id}&quot; ]<br>}</pre><p><strong>aws_instance</strong> is <strong>TYPE</strong> of resource to be created. In this example it’s an AWS EC2 instance</p><p><strong>web</strong> is the <strong>NAME</strong> of a resource that is addressable within terraform configuration file.</p><p><strong>Read more:</strong></p><ul><li><a href="https://www.terraform.io/docs/configuration/syntax.html">Terraform configuration syntax</a></li><li><a href="https://www.terraform.io/docs/configuration/resources.html">Terraform resources</a></li></ul><h3>Hands on</h3><pre>1. Login to your AWS account console.<br>2. Clone git repository with workshop data, go to <strong>w1</strong> directory:</pre><pre>$ git clone <a href="https://github.com/Smartling/aws-terraform-workshops.git">https://github.com/Smartling/aws-terraform-workshops.git</a><br>$ cd aws-terraform-workshops/w1</pre><pre>3. Configure Terraform with AWS credentials (see pre-requisites) in <strong>creds.tf</strong> configuration file:</pre><pre>$ cat creds.tf<br>provider &quot;aws&quot; {<br>  access_key = &quot;THISISEXAMPLETHISISEXAMPLE&quot;<br>  secret_key = “THISISEXAMPLE/kToJ5qUtCpxr/THISISEXAMPLE&quot;<br>  region = &quot;us-east-1&quot;<br>}</pre><pre>4. Follow terraform documentation for EC2 instance and comments in ec2.tf to complete configuration:</pre><pre>   4.1. Go to <a href="https://console.aws.amazon.com/vpc/?#">AWS VPC console</a>, write down VPC ID and subnet ID which are required to complete configuration.</pre><pre><strong>Please notice that VPC and subnets are covered in details in the next workshops but they are still required to finish EC2 configuration.</strong></pre><pre>   4.2. Use t2.nano as instance type for EC2 instance.</pre><pre>   4.3. You should specify names for AWS resources as well as missing configuration parameters.</pre><pre>   4.4. In this workshop we need to create EC2 instance in its own security group, see documentation <a href="https://www.terraform.io/docs/providers/aws/r/security_group.html">here</a> and <a href="https://www.terraform.io/docs/providers/aws/r/instance.html">here</a>.</pre><pre>   4.5. Run <strong>terraform plan</strong> to make sure configuration is ready to be applied.</pre><pre>   4.6. Run <strong>terraform apply</strong> to actually create AWS resources: EC2 security group and EC2 instance.</pre><pre>5. Go to AWS console and find newly created EC2 instance and security group.</pre><pre>6. Open <strong>terraform.tfstate</strong> to examine its structure and newly created AWS resources. <em>Please don&#39;t make any changes into this file. It&#39;s managed by terraform so manual changes into this file may break things up.</em></pre><pre>7. Modify EC2 instance type:</pre><pre>   7.1. Change EC2 instance type from t2.nano to t2.micro.</pre><pre>   7.2. Run <strong>terraform plan </strong>and then <strong>terraform apply </strong>to actually apply changes.</pre><pre>   7.3. Check changes in AWS EC2 web console.</pre><pre>8. Add your SSH key to EC2 instance and access it via SSH.</pre><pre>   8.1. Uncomment user_data parameter in terraform config.</pre><pre>   8.2. Replace example SSH key with <strong>your public SSH key</strong> to <em>shared/user-data.txt</em> file:</pre><pre>cd ../../<br>$ cat shared/user-data.txt<br>#!/bin/bash<br>   <br>mkdir -p /home/ec2-user/.ssh<br>cat &lt;&lt;FILE &gt; /home/ec2-user/.ssh/authorized_keys<br>ssh-rsa AAAAEXAMPLEyc2EAAAADAQABAAABAQCxz1G2vfqCabgFNZiL/timcrISitT4ShZP0iB4G1F+tFRM7to3CstEbS9TFeZwJdKeKLJoGsB5mMueqQb34lVt+ieodNKn8vMjTqv62W8YLqhRavnJ7bTGqGxNhAuLZJdEXAMPLEgywFwQjKYIVQt0SeB0XXLgAUIp+FS7MVyywDdViLqHWexxFN9Nrd6nPAj0fLV9DRIwe7nRccj+R4HvGIC7rQ060QCDCssYiZT/FVihNcPfohQA1JlNGao/lXLkSivwtl0pEDECyzs2KULS+9mc5Bz0Ap1Liskoa5V9umz8LhA9WLqNaCtt6fWQurPAd5lpEXAMPLE user@host<br>FILE</pre><pre>chown ec2-user.ec2-user /home/ec2-user/.ssh/authorized_keys<br>chmod 400 /home/ec2-user/.ssh/authorized_keys<br>yum -y erase docker</pre><pre>   8.3. Apply configuration changes.<br>   8.4. Login to newly created EC2 instance via SSH.<br>   8.5. Run commands <strong>uptime</strong>, <strong>top</strong>, <strong>uname -a</strong> on EC2 instance.</pre><pre>9. Run <strong>terraform destroy </strong>to delete AWS resources which were created during this workshop.</pre><pre><strong>Hint</strong>: in case you got stuck during this workshop you can check <em>answers</em> directory in <a href="https://github.com/Smartling/aws-terraform-workshops.git">aws-terraform-workshops</a> git repository.</pre><p>Read related article with root ideas of DevOps culture!</p><p><strong>Introductory stories:</strong></p><ul><li><a href="https://tech.smartling.com/devops-as-a-culture-set-of-smartling-workshops-6124268d2179">DevOps as a Culture — set of Smartling workshops</a></li><li><a href="https://tech.smartling.com/managed-services-killed-devops-27f169ff4be2">Managed services killed DevOps</a></li></ul><p><strong>Series of workshops:</strong></p><ul><li><a href="https://tech.smartling.com/aws-terraform-workshop-2-ec2-networking-autoscaling-groups-cloudwatch-12ee08c17">AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-3-elb-sns-autoscaling-210e2337057f">AWS/Terraform Workshop #3: ELB, SNS, AutoScaling</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-4-s3-iam-terraform-remote-state-jenkins-71c0d7512bf1">AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-5-aws-lambda-functions-9af79af996b7">AWS/Terraform Workshop #5: AWS Lambda Functions</a></li><li><a href="https://tech.smartling.com/aws-terraform-workshop-6-ec2-container-service-aws-lambda-9e60686c1f71">AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e38607f0fd4c" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/getting-started-with-terraform-and-aws-ec2-first-steps-workshop-1-e38607f0fd4c">Getting started with Terraform and AWS EC2, first steps. Workshop #1</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logs collection from AWS Elasticbeanstalk to Splunk]]></title>
            <link>https://tech.smartling.com/logs-collection-from-aws-elasticbeanstalk-splunk-7edd0348bc96?source=rss-19cc4534363------2</link>
            <guid isPermaLink="false">https://medium.com/p/7edd0348bc96</guid>
            <category><![CDATA[soa]]></category>
            <category><![CDATA[elastic-beanstalk]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[splunk]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Artem Nosulchik]]></dc:creator>
            <pubDate>Mon, 27 Jul 2015 15:34:51 GMT</pubDate>
            <atom:updated>2016-08-17T14:01:47.752Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is AWS Elasticbeanstalk</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*2IwaUJTqyQ4UMtAHyo8GQA.jpeg" /></figure><p><a href="http://aws.amazon.com/documentation/elastic-beanstalk/">AWS Elasticbeanstalk</a> is an AWS service that comes as Platform as a Service (PaaS). It provides a standardised way to manage and deploy applications in the <a href="http://aws.amazon.com/">AWS cloud</a>, so its primary purpose is to reduce management complexity without restricting control. In general, you just upload application and AWS Elasticbeanstalk takes care of the environment for the application, interconnection between required AWS resources, and provides monitoring and maintenance tools.</p><p>When Smartling adopted Service Oriented Architecture (SOA) we chose AWS Elasticbeanstalk as one of the options for Java based services — it provides teams with a reliable and scalable/redundant platform that is easy to use. Teams can manage their own infrastructure, do capacity planning and provisioning, and define deployments and monitoring. The whole point of SOA is ownership — when teams own not just code, but service infrastructure and does its own maintenance as well — this is an essential part of doing DevOps.</p><h4>Logs collection in AWS Elasticbeanstalk</h4><p>Once we decided to go with AWS Elasticbeanstalk the first problem that had to be solved was logs collection and aggregation so that those logs would be available for service team owners in near real time. AWS logs shipping is not great, so we decided to use <a href="http://www.splunk.com/">Splunk</a>.</p><p>In a nutshell, we needed to install the <a href="http://www.splunk.com/en_us/download/universal-forwarder.html">Splunk Universal Forwarder</a> to instances in AWS Elasticbeanstalk and configure them to collect the required logs and forward them to <a href="http://www.splunk.com/en_us/products/splunk-enterprise.html">Splunk Enterprise</a> for consolidation.</p><p>This is where <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html">ebextensions</a> is useful.</p><p><strong>1.</strong> Configure your AWS Elasticbeanstalk environment with the following environment variables:</p><pre>APPLICATION_NAME — name of an application as it will be shown in Splunk</pre><pre>SPLUNK_FORWARDER_RPM_DOWNLOAD_URL — URL from where to download Splunk forwarder</pre><pre>SPLUNK_SERVER_HOST — Splunk server host that will access logs from splunk forwarders</pre><p>These can set these variable using <a href="https://console.aws.amazon.com/elasticbeanstalk/">AWS console</a> or <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#customize-containers-format-options">AWS Elasticbeanstalk configuration files</a>.</p><p>Example:</p><pre>APPLICATION_NAME = MyJavaApp-EB</pre><pre>SPLUNK_FORWARDER_RPM_DOWNLOAD_URL = <a href="https://download.splunk.com/products/splunk/releases/6.2.2/universalforwarder/linux/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm">https://download.splunk.com/products/splunk/releases/6.2.2/universalforwarder/linux/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm</a> </pre><pre>SPLUNK_SERVER_HOST = 10.20.30.40</pre><p><strong>2.</strong> Add<em> .ebextensions</em> directory into the source bundle (usually zip file) of your application that is going to be deployed to AWS Elasticbeanstalk. Create <em>101splunk-fowarder.config</em> in <em>.ebextensions</em> with the following contents:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57f8e90bb2ecab142f03cc89da26148d/href">https://medium.com/media/57f8e90bb2ecab142f03cc89da26148d/href</a></iframe><p><strong>Warning #1:</strong> This ebextension was tested with Amazon Linux. If other Linux is used, e.g. Ubuntu, you must alter it, accordingly.</p><p><strong>Warning #2:</strong> Configuration expects application to write logs to /var/log/tomcat8. Ebextension will need to be changed if application logs are located somewhere else.</p><p><strong>3.</strong> Deploy source bundle containing <em>.ebextensions/101splunk-fowarder.config</em>. During the deploy process, Splunk Fowarder will be installed, configured and started on every Elasticbeanstalk EC2 instance</p><p>Here are some Splunk query examples, note APPLICATION_NAME variable value (MyJavaApp-EB) is a part of search query:</p><pre>host=&quot;MyJavaApp-EB/*&quot;</pre><p>Splunk query to see only one log file at particular EC2 instance in AWS Elasticbeanstalk will be:</p><pre>host=&quot;MyJavaApp-EB/i-1234567&quot; source=&quot;/var/log/tomcat8/catalina.out&quot;</pre><h4>Logs collection from Docker in AWS Elasticbeanstalk</h4><p>For applications running in ElasticBeanstalk under Docker there is a way to have log collection with Splunk as well.</p><p><strong>1. </strong>Add <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_image.html">Dockerrun.aws.json</a> to source bundle:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53ed9dce69e12feed2d9645e9b0ee5fb/href">https://medium.com/media/53ed9dce69e12feed2d9645e9b0ee5fb/href</a></iframe><p>This file instructs Docker to map a volume from an EC2 instance to Docker container.</p><p><strong>2.</strong> Configure application in Docker to store logs into <em>/var/log/myapplicationlogs</em> directory and prepare logs directory for it. Here’s an example of a <em>Dockerfile</em> section for this:</p><pre>...<br>RUN mkdir /var/log/myapplicationlogs<br>RUN chown jboss.jboss /var/log/myapplicationlogs<br>...<br>ADD logging.properties /opt/jboss/standalone/configuration<br>...</pre><p><strong>3.</strong> Add <em>/var/log/myapplicationlogs</em> directory to Splunk Forwarder in <em>.ebextensinons/101splunk-fowarder.config</em> (see it above) and put this config into source bundle:</p><pre>/opt/splunkforwarder/bin/splunk add monitor &quot;/var/log/myapplicationlogs/*.log&quot; -hostname &quot;$splunk_logs_hostname&quot; -sourcetype log4j</pre><p><strong>4.</strong> Deploy an application using source bundle containing <em>.ebextensinons/101splunk-fowarder.config</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7edd0348bc96" width="1" height="1" alt=""><hr><p><a href="https://tech.smartling.com/logs-collection-from-aws-elasticbeanstalk-splunk-7edd0348bc96">Logs collection from AWS Elasticbeanstalk to Splunk</a> was originally published in <a href="https://tech.smartling.com/">Universal Language</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>