<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aalok Trivedi on Medium]]></title>
        <description><![CDATA[Stories by Aalok Trivedi on Medium]]></description>
        <link>https://medium.com/@aaloktrivedi?source=rss-fda853e0723e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*r5H_J2juE65JoHkkbmNnjA.png</url>
            <title>Stories by Aalok Trivedi on Medium</title>
            <link>https://medium.com/@aaloktrivedi?source=rss-fda853e0723e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:08:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aaloktrivedi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Creating a CI/CD pipeline with Terraform Cloud to deploy WordPress application infrastructure: Part…]]></title>
            <link>https://medium.com/@aaloktrivedi/creating-a-ci-cd-pipeline-with-terraform-cloud-to-deploy-wordpress-application-infrastructure-part-da8a549e7406?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/da8a549e7406</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[wordpress]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Sat, 22 Apr 2023 20:28:34 GMT</pubDate>
            <atom:updated>2023-04-22T20:28:34.935Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3HM_e1GaQ_gu1YH4dHh1g.png" /></figure><h3>Creating a CI/CD pipeline with Terraform Cloud to deploy WordPress application infrastructure: Part 2</h3><h3>Onward!</h3><p><a href="https://medium.com/@aaloktrivedi/creating-a-ci-cd-pipeline-with-terraform-cloud-to-deploy-wordpress-application-infrastructure-part-8f1cb9f4f022">In Part 1</a>, we used our Terraform Cloud CI/CD pipeline to deploy the networking base for our WordPress application infrastructure. In Part 2, we’ll continue with our CI/CD pipeline and focus on creating the RDS database and Auto Scaling Group (ASG) with WordPress installed.</p><p>Again, if you need a refresher, here are some past articles that explain Terraform basics in a bit more detail:</p><ul><li><a href="https://medium.com/@aaloktrivedi/automating-with-infrastructure-as-code-deploying-a-jenkins-server-with-terraform-e32f7cc79fe5">Automating with infrastructure as code: Deploying a Jenkins server with Terraform</a></li><li><a href="https://medium.com/@aaloktrivedi/configuring-a-terraform-remote-backend-with-s3-and-dynamodb-ebcefa8432ea">Configuring a Terraform remote backend with S3 and DynamoDB</a></li><li><a href="https://medium.com/@aaloktrivedi/deploying-application-infrastructure-to-dynamically-scale-and-handle-high-traffic-with-aws-9d92b43979cd">Deploying application infrastructure to dynamically scale and handle high traffic with AWS &amp; Terraform</a></li></ul><h4>What we’re building</h4><ol><li>An <strong>RDS database</strong> that will connect to our WordPress instances.</li><li>An <strong>Application Load Balancer (ALB)</strong> to distribute traffic to the ASG.</li><li>An <strong>Auto Scaling group (ASG)</strong> with WordPress installed on instances.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7FdSJGNzt7qpx4hUJj_uA.png" /></figure><h4>GitHub repo</h4><p><a href="https://github.com/aalokt89/terraform_wordpress_dev">Here is my GitHub repo if you want to follow along ➡️</a></p><h3>🚀 🚀 🚀 🚀 🚀 Let’s get started!</h3><h3>Create the security groups</h3><p>First, we’ll create the necessary security groups, so each component allows the proper inbound and outbound traffic.</p><p>Let’s create a new branch named rds-database.</p><pre>git checkout -b &quot;rds-database&quot;</pre><ol><li>SSH security group:<br><strong>Inbound:</strong> SSH traffic on port 22</li><li>ALB security group: <br><strong>Inbound:</strong> HTTP/HTTPS traffic on ports 80 and 443 | <strong>Outbound: </strong>All</li><li>Database security group:<br><strong>Inbound: </strong>Traffic on port 3306 from the ASG security group</li><li>ASG instance security group:<br><strong>Inbound: </strong>HTTP/HTTPS traffic from ALB security group | <strong>Outbound: </strong>All</li></ol><pre>#--variables.tf--<br><br># Security groups<br>#----------------------------------------<br>variable &quot;sg_timeout&quot; {<br>  type        = string<br>  description = &quot;Security group delete timeout&quot;<br>  default     = &quot;5m&quot;<br>}<br>variable &quot;ssh_cidr&quot; {<br>  type        = string<br>  description = &quot;ssh cidrs&quot;<br>  default     = &quot;0.0.0.0/0&quot;<br>}<br>variable &quot;db_port&quot; {<br>  type        = number<br>  description = &quot;wordpress db port&quot;<br>  default     = 3306<br>}</pre><h4>SSH security group</h4><pre>#--security_groups.tf--<br><br># Create security groups<br>#----------------------------------------------------<br><br># ssh<br>resource &quot;aws_security_group&quot; &quot;ssh_sg&quot; {<br>  name        = &quot;${local.name_prefix}-ssh-sg&quot;<br>  description = &quot;Allow ssh.&quot;<br>  vpc_id      = module.network.vpc_id<br><br>  ingress {<br>    description = &quot;ssh&quot;<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.ssh_cidr]<br>  }<br><br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.sg_timeout<br>  }<br><br>  tags = {<br>    Name = &quot;${local.name_prefix}-ssh-sg&quot;<br>  }<br>}</pre><blockquote><strong>Note:</strong> You generally want to lock down the inbound SSH CIDRs to approved IP addresses/ranges, but for this demo, we’ll allow SSH from ‘all traffic.’</blockquote><h4>ALB security group</h4><pre>#--security_groups.tf--<br><br>#http/https for ALB<br>resource &quot;aws_security_group&quot; &quot;web_access_sg&quot; {<br>  name        = &quot;${local.name_prefix}-web-access-sg&quot;<br>  description = &quot;http/https access attached to ALB.&quot;<br>  vpc_id      = module.network.vpc_id<br><br>  # http<br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  # https<br>  ingress {<br>    description = &quot;https&quot;<br>    from_port   = 443<br>    to_port     = 443<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }<br>  # all<br>  egress {<br>    from_port        = 0<br>    to_port          = 0<br>    protocol         = &quot;-1&quot;<br>    cidr_blocks      = [&quot;0.0.0.0/0&quot;]<br>    ipv6_cidr_blocks = [&quot;::/0&quot;]<br>  }<br>  lifecycle {<br>    create_before_destroy = true<br>  }<br>  timeouts {<br>    delete = var.sg_timeout<br>  }<br>  tags = {<br>    Name = &quot;${local.name_prefix}-web-access-sg&quot;<br>  }<br>}</pre><h4>RDS database security group</h4><pre>#--security_groups.tf--<br><br># wordpress rds sg<br>resource &quot;aws_security_group&quot; &quot;wordpress_rds&quot; {<br>  name        = &quot;${local.name_prefix}-wordpress-rds-sg&quot;<br>  description = &quot;Allows inbound traffic from wordpress ec2s. Attached to rds&quot;<br>  vpc_id      = module.network.vpc_id<br><br>  # to ec2<br>  ingress {<br>    description     = &quot;traffic from ec2 sg&quot;<br>    from_port       = var.db_port<br>    to_port         = var.db_port<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.wordpress_ec2.id]<br>  }<br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.sg_timeout<br>  }<br><br>  tags = {<br>    Name = &quot;${local.name_prefix}-wordpress-rds-sg&quot;<br>  }<br>}</pre><h4>ASG instance security group</h4><pre>#--security_groups.tf--<br><br># wordpress ec2 sg<br>resource &quot;aws_security_group&quot; &quot;wordpress_ec2_sg&quot; {<br>  name        = &quot;${local.name_prefix}-wordpress-ec2-sg&quot;<br>  description = &quot;Allows inbound http/https traffic from ALB sg and all outboound traffic. Attached to ec2&quot;<br>  vpc_id      = module.network.vpc_id<br><br>  # http<br>  ingress {<br>    description     = &quot;http from ALB&quot;<br>    from_port       = 80<br>    to_port         = 80<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.web_access_sg.id]<br>  }<br>  # https<br>  ingress {<br>    description     = &quot;https from ALB&quot;<br>    from_port       = 443<br>    to_port         = 443<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.web_access_sg.id]<br>  }<br>  # all<br>  egress {<br>    from_port        = 0<br>    to_port          = 0<br>    protocol         = &quot;-1&quot;<br>    cidr_blocks      = [&quot;0.0.0.0/0&quot;]<br>    ipv6_cidr_blocks = [&quot;::/0&quot;]<br>  }<br><br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.sg_timeout<br>  }<br><br>  tags = {<br>    Name = &quot;${local.name_prefix}-wordpress-ec2-sg&quot;<br>  }<br>}</pre><h3>Create the RDS database</h3><h4>Why use RDS?</h4><p>For WordPress to work properly, it needs a MySQL database to store all of its important information and website content. In most cases, when WordPress is installed, a MySQL DB is created directly on the same server. This creates two problems: Fault tolerance and maintenance.</p><p>If the server goes down, so does the database and all of its data. By decoupling the WordPress server and the database, we’re able to provide fault tolerance and ensure the data is protected.</p><p>RDS is a fully managed service and provides automated backups, updates, and patches, which makes maintaining and managing the database incredibly easy.</p><h4>Terraform Cloud variables</h4><p>When we create the database, we’ll need three key pieces of information: db_name, username, and password. Since this is sensitive data, we never want to hardcode it into our configurations — even as variables in our <em>variables.tf</em> file. We can store these as workspace variable <em>secrets </em>in Terraform Cloud.</p><p>Let’s create three new <em>Terraform </em>variables and mark the db_username and db_password as <em>sensitive.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6B3OyggouL3tbRjRayVjfw.png" /></figure><h4>Create the RDS resource</h4><p>It’s time to create the RDS resource with a MySQL engine. Even though we created the db_name, db_username, and db_password variables in Terraform Cloud, we still need to establish them locally. We’ll mark them as sensitive and won’t provide a default value.</p><p>We’ll also create a db_subnet_group to place the RDS in the private subnets.</p><pre>#--variables.tf--<br><br># RDS wordpress<br>#----------------------------------------<br>variable &quot;db_allocated_storage&quot; {<br>  type        = number<br>  description = &quot;db allocated storage&quot;<br>  default     = 20<br>}<br>variable &quot;db_max_allocated_storage&quot; {<br>  type        = number<br>  description = &quot;wordpress db max allocated storage&quot;<br>  default     = 500<br>}<br>variable &quot;db_name&quot; {<br>  type        = string<br>  description = &quot;wordpress db name&quot;<br>}<br>variable &quot;db_instance_class&quot; {<br>  type        = string<br>  description = &quot;wordpress db instance type&quot;<br>  default     = &quot;db.t2.micro&quot;<br>}<br>variable &quot;db_username&quot; {<br>  type        = string<br>  description = &quot;wordpress db username&quot;<br>  sensitive   = true<br>}<br>variable &quot;db_password&quot; {<br>  type        = string<br>  description = &quot;wordpress db password&quot;<br>  sensitive   = true<br>}<br>variable &quot;db_engine&quot; {<br>  type        = string<br>  description = &quot;wordpress db engine&quot;<br>  default     = &quot;mysql&quot;<br>}<br>variable &quot;db_engine_version&quot; {<br>  type        = string<br>  description = &quot;wordpress db engine version&quot;<br>  default     = &quot;8.0.32&quot;<br>}<br>variable &quot;db_publicly_accessible&quot; {<br>  type        = bool<br>  description = &quot;wordpress db publicly accessible&quot;<br>  default     = false<br>}<br>variable &quot;db_deletion_protection&quot; {<br>  type        = bool<br>  description = &quot;wordpress db deletion protection&quot;<br>  default     = false<br>}<br>variable &quot;db_skip_final_snapshot&quot; {<br>  type        = bool<br>  description = &quot;skip final snapshot&quot;<br>  default     = true<br>}</pre><pre>#--database.tf--<br><br># Create RDS<br>#----------------------------------------------------<br><br>resource &quot;aws_db_instance&quot; &quot;wordpress&quot; {<br>  allocated_storage         = var.db_allocated_storage<br>  max_allocated_storage     = var.db_max_allocated_storage<br>  db_name                   = var.db_name<br>  engine                    = var.db_engine<br>  engine_version            = var.db_engine_version<br>  instance_class            = var.db_instance_class<br>  username                  = var.db_username<br>  password                  = var.db_password<br>  availability_zone         = local.azs[0]<br>  publicly_accessible       = var.db_publicly_accessible<br>  deletion_protection       = var.db_deletion_protection<br>  db_subnet_group_name      = aws_db_subnet_group.wordpress.name<br>  vpc_security_group_ids    = [aws_security_group.wordpress_rds.id]<br>  skip_final_snapshot       = var.db_skip_final_snapshot<br><br>  tags = {<br>    &quot;Name&quot; = &quot;${local.name_prefix}-${var.db_name}-db&quot;<br>  }<br>}<br><br>resource &quot;aws_db_subnet_group&quot; &quot;wordpress&quot; {<br>  name       = &quot;${var.app_name}-db-subnet-group&quot;<br>  subnet_ids = module.network.private_subnets<br><br>  tags = {<br>    Name = &quot;${local.name_prefix}-db-subnet-group&quot;<br>  }<br>}</pre><h4>Outputs</h4><p>To connect WordPress to the database, we need the RDS endpoint address. We’ll create a new output block that will output the endpoint for later use.</p><pre>#--outputs.tf--<br><br>output &quot;wordpress_db_endpoint&quot; {<br>  value = aws_db_instance.wordpress.address<br>}</pre><h4>Commit, push, pull request</h4><p>Alright! The security groups and database are configured, so let’s <strong>commit</strong> and <strong>push</strong> our code to our ‘rds’ branch. We can then create a pull request so another code owner can review and approve our code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJAgsDYSV2mGpEPBOM4gwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mxS2o2MYINknwZgUBohe5w.png" /><figcaption>Create pull request</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNV-uZ_0OZg5m0OOsSEm1w.png" /><figcaption>Review new/changed resources</figcaption></figure><h4>Approve/Merge to main branch</h4><p>Everything looks good so we’re going to approve the request and merge the branch into main (Obviously, we would never approve our own requests, but let’s just pretend we’re another code owner).</p><p>Remember, once we merge, the plan will automatically be applied and deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ngJzgY8vZY8m9FW09-wM4A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pi6VQvWaNu8y3QVNpi8ntA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KfGytpXq-zKuDwGXHmn-4w.png" /></figure><p>Woo! Our WordPress database has been successfully deployed.</p><h3>Create the ALB</h3><p>Now, it’s time to create the ALB that will listen for HTTP and distribute the traffic to our eventual ASG WordPress servers.</p><p>Let’s create a new branch called wordpress-server-creation</p><pre>git checkout -b &quot;wordpress-server-creation&quot;</pre><h4>What we’ll need</h4><ol><li>An <strong>ALB</strong> resource.</li><li>A <strong>target group</strong> resource that targets the ASG instances.</li><li>A <strong>listener</strong> resource to listen for HTTP on port 80.</li></ol><h4>ALB</h4><p>In a new <em>load_balancer.tf</em> file, let’s create the ALB. We’ll attach the ALB to the public subnets and attach the web_access_sg security group so it can get inbound HTTP traffic.</p><pre>#--variables.tf--<br><br># ALB<br>#----------------------------------------<br>variable &quot;web_alb_name&quot; {<br>  type        = string<br>  description = &quot;alb name&quot;<br>  default     = &quot;web-alb&quot;<br>}<br>variable &quot;web_alb_type&quot; {<br>  type        = string<br>  description = &quot;alb type&quot;<br>  default     = &quot;application&quot;<br>}<br>variable &quot;web_alb_internal&quot; {<br>  type        = bool<br>  description = &quot;internal facing&quot;<br>  default     = false</pre><pre>#--load_balancer.tf--<br><br># Create ALB, listener, and target groups<br>#----------------------------------------------------<br><br># ALB<br>resource &quot;aws_lb&quot; &quot;web_alb&quot; {<br>  name               = &quot;${local.name_prefix}-${var.web_alb_name}&quot;<br>  internal           = var.web_alb_internal<br>  load_balancer_type = var.web_alb_type<br>  security_groups    = [aws_security_group.web_access_sg.id]<br>  subnets            = module.network.public_subnets<br><br>  tags = {<br>    Name = &quot;${local.name_prefix}-${var.web_alb_name}&quot;<br>  }<br>}</pre><h4>Target group</h4><p>Next, we need to create a target group that we’ll later apply to our WordPress server instances.</p><pre>#--variables.tf--<br><br>#target group<br>variable &quot;web_alb_tg_http_name&quot; {<br>  type        = string<br>  description = &quot;target group name&quot;<br>  default     = &quot;wp-server-tg-http&quot;<br>}<br>variable &quot;web_alb_tg_http_protocol&quot; {<br>  type        = string<br>  description = &quot;target group protocol&quot;<br>  default     = &quot;HTTP&quot;<br>}<br>variable &quot;web_alb_tg_http_port&quot; {<br>  type        = number<br>  description = &quot;target group port&quot;<br>  default     = 80<br>}<br>variable &quot;web_alb_tg_http_type&quot; {<br>  type        = string<br>  description = &quot;target group type&quot;<br>  default     = &quot;instance&quot;<br>}</pre><pre>#--load_balancer.tf--<br><br># ALB target groups <br>#----------------------------------------<br># HTTP<br>resource &quot;aws_lb_target_group&quot; &quot;web_alb_tg_http&quot; {<br>  depends_on = [aws_lb.web_alb]<br>  name       = &quot;${var.web_alb_name}-${var.web_alb_tg_http_name}&quot;<br>  port       = var.web_alb_tg_http_port<br>  protocol   = var.web_alb_tg_http_protocol<br>  vpc_id     = module.network.vpc_id<br><br>  tags = {<br>    Name = &quot;${var.web_alb_name}-${var.web_alb_tg_http_name}&quot;<br>  }<br>}</pre><h4>Listener</h4><p>Finally, we’ll create an HTTP listener, attach the target group, and attach the ALB.</p><pre>#--variables.tf--<br><br>#listener<br>variable &quot;web_alb_listener_http_protocol&quot; {<br>  type        = string<br>  description = &quot;http listener protocol&quot;<br>  default     = &quot;HTTP&quot;<br>}<br>variable &quot;web_alb_listener_http_port&quot; {<br>  type        = number<br>  description = &quot;http listener port&quot;<br>  default     = 80<br>}<br>variable &quot;web_alb_listener_http_action_type&quot; {<br>  type        = string<br>  description = &quot;http listener action type&quot;<br>  default     = &quot;forward&quot;<br>}</pre><pre>#--load_balancer.tf--<br><br># ALB listeners<br>#----------------------------------------<br># HTTP<br>resource &quot;aws_lb_listener&quot; &quot;web_alb_listener_http&quot; {<br>  load_balancer_arn = aws_lb.web_alb.arn<br>  port              = var.web_alb_listener_http_port<br>  protocol          = var.web_alb_listener_http_protocol<br><br>  default_action {<br>    type             = var.web_alb_listener_http_action_type<br>    target_group_arn = aws_lb_target_group.web_alb_tg_http.arn<br>  }<br>}</pre><h4>Outputs</h4><p>We’ll need the ALB’s DNS name to access our WordPress application, so let’s output the dns_name.</p><pre>#--outputs.tf--<br><br>output &quot;alb_dns&quot; {<br>  description = &quot;ALB public DNS&quot;<br>  value       = &quot;http://${aws_lb.web_alb.dns_name}&quot;<br>}</pre><p>Commit and push the changes to our current working branch.</p><h3>Create the ASG</h3><p>Final step! Now we have to create the ASG that will provision the WordPress servers.</p><h4>What we’ll need</h4><ol><li>An <strong>ASG</strong> resource.</li><li>A <strong>launch template </strong>resource to define what kind of EC2 instances the ASG will deploy.</li><li>A <strong>bash script</strong> that will install WordPress.</li><li>A <strong>scaling policy</strong> that dictates how the instances will be scaled.</li></ol><h4>ASG</h4><p>Similar to the VPC and network, we’re going to use an <a href="https://registry.terraform.io/modules/terraform-aws-modules/autoscaling/aws/latest">Auto Scaling Group module from the Terraform Registry</a>. This module will make things a bit easier and create the ASG, launch template, and scaling policy all in one go.</p><p>Our WordPress servers are going to use <strong>t2.micro </strong>instances with the <strong>Amazon Linux 2 AMI</strong>. Since AMI IDs change from region to region, it’s best to get the ID through a <strong>data lookup</strong>, instead of hardcoding. Using a data block, we can get the AMI ID by filtering by ‘name.’ The output will give us the proper ID based on our current region.</p><pre>#--auto_scaling.tf--<br><br># Get amazon linux 2 ami<br>data &quot;aws_ami&quot; &quot;amazon_linux2&quot; {<br>  most_recent = true<br>  owners      = [&quot;amazon&quot;]<br><br>  filter {<br>    name   = &quot;name&quot;<br>    values = [&quot;amzn2-ami-hvm-*-x86_64-gp2&quot;]<br>  }<br>}</pre><p>Let’s first define the inputs for the ASG. We want a min_size of 2 and a max_size of 4. We also need to input the target group ARNs from the ALB.</p><pre>#--variables.tf--<br><br># WordPress ASG<br>#----------------------------------------<br>variable &quot;wordpress_asg_min_size&quot; {<br>  type        = number<br>  description = &quot;asg min size&quot;<br>  default     = 2<br>}<br>variable &quot;wordpress_asg_max_size&quot; {<br>  type        = number<br>  description = &quot;asg max size&quot;<br>  default     = 4<br>}<br>variable &quot;wordpress_asg_health_check_type&quot; {<br>  type        = string<br>  description = &quot;asg health check type &quot;<br>  default     = &quot;ELB&quot;<br>}</pre><pre>#--auto_scaling.tf--<br><br># Autoscaling group<br>#----------------------------------------<br>module &quot;wordpress_asg&quot; {<br>  depends_on = [aws_db_instance.wordpress]<br>  source     = &quot;terraform-aws-modules/autoscaling/aws&quot;<br>  version    = &quot;6.9.0&quot;<br><br>  name                = &quot;${local.name_prefix}-${var.wordpress_server_name}-asg&quot;<br>  min_size            = var.wordpress_asg_min_size<br>  max_size            = var.wordpress_asg_max_size<br>  health_check_type   = var.wordpress_asg_health_check_type<br>  vpc_zone_identifier = module.network.public_subnets<br>  target_group_arns   = [aws_lb_target_group.web_alb_tg_http.arn]<br><br><br>}</pre><h4>Launch template</h4><p>Next, we need to define the inputs for the launch template.</p><pre>#--variables.tf--<br><br>#Launch template<br>variable &quot;wordpress_server_name&quot; {<br>  type        = string<br>  description = &quot;wordpress template name&quot;<br>  default     = &quot;wordpress-server&quot;<br>}<br>variable &quot;wordpress_template_instance_type&quot; {<br>  type        = string<br>  description = &quot;wordpress server type&quot;<br>  default     = &quot;t2.micro&quot;<br>}<br>variable &quot;key_name&quot; {<br>  type        = string<br>  description = &quot;wordpress server key pair&quot;<br>  sensitive   = true<br>}<br>variable &quot;wordpress_template_enable_monitoring&quot; {<br>  type        = bool<br>  description = &quot;wordpress server enable monitoring&quot;<br>  default     = true<br>}</pre><pre>#--auto_scaling.tf--<br><br># Autoscaling group<br>#----------------------------------------<br>module &quot;wordpress_asg&quot; {<br>  depends_on = [aws_db_instance.wordpress]<br>  source     = &quot;terraform-aws-modules/autoscaling/aws&quot;<br>  version    = &quot;6.9.0&quot;<br><br>  ...<br>  <br>  ...<br><br>  # Launch template<br>  launch_template_name        = &quot;${var.wordpress_server_name}-template&quot;<br>  launch_template_description = &quot;WordPress instances&quot;<br><br>  image_id          = data.aws_ami.amazon_linux2.id<br>  instance_type     = var.wordpress_template_instance_type<br>  enable_monitoring = var.wordpress_template_enable_monitoring<br>  security_groups   = [aws_security_group.ssh_sg.id, aws_security_group.wordpress_ec2_sg.id]<br>  key_name          = var.key_name<br><br>  user_data = &quot;&quot;<br><br>}</pre><p>Remember, we need to add the SSH and WordPress security groups to the template. It’s also a good idea to add the instance key pair name to the workspace variables. We’ll leave the user_data empty for now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRcQkoezxEzL9jNjpAr6Aw.png" /></figure><h4>WordPress script</h4><p>We need to install WordPress on our servers, so let’s create a bash script to input as the user_data. But before we do, we need to store some more secrets as workspace variables.</p><p>Along with the db_username and db_password, we need to create a username, password, and email for the initial WordPress account. So let’s store those as terraform variables and mark the password and email as <em>sensitive.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VKQAqfy4Na9Mwedd-a4XQ.png" /></figure><p>Let’s also declare those variables in the <em>variables.tf</em> file without any default values, so we can call them inside the configurations.</p><pre>#--variables.tf<br><br>#Wordpress account<br>variable &quot;wp_username&quot; {<br>  type        = string<br>  description = &quot;wordpress username&quot;<br>  sensitive   = true<br>}<br>variable &quot;wp_password&quot; {<br>  type        = string<br>  description = &quot;wordpress password&quot;<br>  sensitive   = true<br>}<br>variable &quot;wp_email&quot; {<br>  type        = string<br>  description = &quot;wordpress email&quot;<br>  sensitive   = true<br>}</pre><p>But how are we going to use these terraform variables in our bash script? Good question! We can make use of the template_file data resource to insert variables or outputs from other resources into external files.</p><p>First, let’s create a new file called <em>install_wordpress.tpl. </em>The .tpl extension designates a template file.</p><p>Next, let’s create the template data source in the <em>auto_scaling.tf </em>file. We’ll define the template file we want to populate (the <em>install_wordpress.tpl</em>) and designate the proper variables we need.</p><pre>#--auto_scaling.tf--<br><br># user data wordpress script template<br>data &quot;template_file&quot; &quot;user_data&quot; {<br>  template = file(&quot;${path.module}/install_wordpress.tpl&quot;)<br>  vars = {<br>    db_username = var.db_username<br>    db_password = var.db_password<br>    db_name     = var.db_name<br>    db_host     = aws_db_instance.wordpress.address<br><br>    wp_username = var.wp_username<br>    wp_password = var.wp_password<br>    wp_email    = var.wp_email<br>    wp_url      = aws_lb.web_alb.dns_name<br>    wp_title    = var.app_name<br>  }<br>}</pre><p>Now let&#39;s build the script!</p><p>We’ll start by establishing our variables again and attach them to the <strong>template vars.</strong></p><pre>#--install_wordpress.tpl--<br><br>#!/bin/bash<br><br>#template vars<br>db_username=${db_username}<br>db_password=${db_password}<br>db_name=${db_name}<br>db_host=${db_host}<br><br>wp_username=${wp_username}<br>wp_password=${wp_password}<br>wp_email=${wp_email}<br>wp_url=${wp_url}<br>wp_title=${wp_title}</pre><p>Next, we’ll run some initial package updates and install Apache, PHP, and MySQL.</p><pre>#--install_wordpress.tpl--<br><br>sudo yum update -y<br>sudo yum install -y httpd mysql<br>sudo systemctl start httpd<br>sudo systemctl enable httpd<br><br>sudo amazon-linux-extras install -y php7.4<br>sudo amazon-linux-extras enable php7.4<br>sudo yum clean metadata<br>sudo yum install php-cli php-pdo php-fpm php-json php-mysqlnd</pre><p>Then we need to add the ec2-user to the apache group and change ownership/permissions of the var/www directories recursive.</p><pre>#--install_wordpress.tpl--<br><br>#Change OWNER and permission of directory /var/www<br>sudo usermod -a -G apache ec2-user<br>sudo chown -R ec2-user:apache /var/www<br>sudo find /var/www -type d -exec chmod 2775 {} \;<br>sudo find /var/www -type f -exec chmod 0664 {} \;</pre><p>We’ll install WordPress through their CLI, so we need to download and install the CLI.</p><pre>#--install_wordpress.tpl--<br><br># Download and install the WordPress CLI<br>sudo curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar<br>sudo chmod +x wp-cli.phar<br>sudo mv wp-cli.phar /usr/local/bin/wp</pre><p>We’ll create the wp-config file through the WordPress CLI and enter the required credentials (db_name, db_user, db_password, db_host)</p><pre>#--install_wordpress.tpl--<br><br># Create the WordPress configuration file<br>cd /var/www/html<br>wp core download --allow-root<br>wp config create --dbname=$db_name --dbuser=$db_user --dbpass=$db_password --dbhost=$db_host --allow-root</pre><p>Then install WordPress using the wp_username, wp_password, wp_email, and ALB dns_name.</p><pre>#--install_wordpress.tpl--<br><br># Install WordPress<br>wp core install --url=$wp_url --title=$wp_title --admin_user=$wp_username --admin_password=$wp_password --admin_email=$wp_email --allow-root</pre><p>Finally, we’ll set appropriate permissions for the main directory and restart apache.</p><pre>#--install_wordpress.tpl--<br><br># Set the appropriate permissions<br>sudo chown -R apache:apache /var/www/html<br>sudo chmod -R 755 /var/www/html<br>sudo systemctl restart httpd.service</pre><p>Back in our ASG module block, enter the template data source for the user_data.</p><pre>#--auto_scaling.tf--<br><br>module &quot;wordpress_asg&quot; {<br>  <br>  ...<br> <br>  ...<br><br># Launch template<br>  ...<br><br>  ...<br><br>  user_data = base64encode(data.template_file.user_data.rendered)<br><br>}</pre><h4>Scaling policy</h4><p>Lastly, we’ll create a scaling policy that tells the ASG to scale up instances once the average CPU usage goes above 75%</p><pre>#--variables.tf<br><br>#Scaling policy<br>variable &quot;wordpress_asg_scaling_policy_type&quot; {<br>  type        = string<br>  description = &quot;wordpress asg scalng policy type&quot;<br>  default     = &quot;TargetTrackingScaling&quot;<br>}<br>variable &quot;wordpress_asg_scaling_policy_metric_type&quot; {<br>  type        = string<br>  description = &quot;wordpress asg scalng policy metric type&quot;<br>  default     = &quot;ASGAverageCPUUtilization&quot;<br>}<br>variable &quot;wordpress_asg_scaling_policy_target_value&quot; {<br>  type        = number<br>  description = &quot;wordpress asg scalng policy target value&quot;<br>  default     = 75.0<br>}</pre><pre>#--auto_scaling.tf--<br><br>module &quot;wordpress_asg&quot; {<br>  <br>  ...<br> <br>  ...<br><br># Launch template<br>  ...<br><br>  ...<br><br>  scaling_policies = {<br>    avg-cpu-policy-greater-than-75 = {<br>      policy_type = var.wordpress_asg_scaling_policy_type<br>      target_tracking_configuration = {<br>        predefined_metric_specification = {<br>          predefined_metric_type = var.wordpress_asg_scaling_policy_metric_type<br>        }<br>        target_value = var.wordpress_asg_scaling_policy_target_value<br>      }<br>    }<br>  }<br>}</pre><h4>Done!</h4><p>Since we created a new module resource, we need to run another terraform init so Terraform recognizes it.</p><h3>Final commit, push, merge</h3><p>Our entire configuration is finished! Let’s commit/push the code to our working branch and create a pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOyI0mHUjyWFcGatWo149Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJ-bEjjl-1SDw_LKtNsT3Q.png" /></figure><p>And if we go to the ALB DNS…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QJaJ5tVVChh8rnT4rwzf1w.png" /></figure><p>Let’s go!!! WordPress has successfully installed and connected to the RDS database!</p><h4>Confirm database</h4><p>We got WordPress to install, but let’s just confirm it can connect to the database by SSH-ing into one of the instances.</p><p>We can get one of the instance’s public IP addresses and SSH into it.</p><p>Then, we can run a mysql command to connect to RDS. Use the db_host, db_name, db_username, and db_password.</p><pre>mysql -h &lt;DB_HOSTNAME&gt; -P 3306 -u &lt;DB_USERNAME&gt; -p</pre><p>Switch to our woordpress database.</p><pre>MySQL [(none)]&gt; USE wordpress</pre><p>And then list the tables</p><pre>MySQL [wordpress]&gt; SHOW TABLES;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8x9Vs8IaOcla0vMZNHo3Mg.png" /></figure><h3>Success!</h3><p>We’ve successfully used a CI/CD process to deploy a highly-available, 2-tier architecture for a WordPress application!</p><p>We can see the power of Terraform now, right? An entire architecture configured with a click of a button. Now that we have this environment set up, we can easily reuse and reconfigure the Terraform structure how we see fit.</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><p>Feel free to connect with me on <a href="https://www.linkedin.com/in/aalok-trivedi">LinkedIn</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da8a549e7406" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a CI/CD pipeline with Terraform Cloud to deploy WordPress application infrastructure…]]></title>
            <link>https://medium.com/@aaloktrivedi/creating-a-ci-cd-pipeline-with-terraform-cloud-to-deploy-wordpress-application-infrastructure-part-8f1cb9f4f022?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f1cb9f4f022</guid>
            <category><![CDATA[terraform-cloud]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Wed, 19 Apr 2023 03:25:45 GMT</pubDate>
            <atom:updated>2023-04-21T01:49:56.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MvuIhiyFPs9um4YcIKGggQ.png" /></figure><h3>Creating a CI/CD pipeline with Terraform Cloud to deploy WordPress application infrastructure: Part 1</h3><h3>Intro</h3><p>Over the last couple of articles, I’ve delved deeper and deeper into Terraform and learned ways to implement Infrastructure as Code (IaC) more efficiently. If you haven&#39;t already, I highly recommend reviewing my past articles, as I go over the basics of Terraform in more detail.</p><ul><li><a href="https://medium.com/@aaloktrivedi/automating-with-infrastructure-as-code-deploying-a-jenkins-server-with-terraform-e32f7cc79fe5">Automating with infrastructure as code: Deploying a Jenkins server with Terraform</a></li><li><a href="https://medium.com/@aaloktrivedi/configuring-a-terraform-remote-backend-with-s3-and-dynamodb-ebcefa8432ea">Configuring a Terraform remote backend with S3 and DynamoDB</a></li><li><a href="https://medium.com/@aaloktrivedi/deploying-application-infrastructure-to-dynamically-scale-and-handle-high-traffic-with-aws-9d92b43979cd">Deploying application infrastructure to dynamically scale and handle high traffic with AWS &amp; Terraform</a></li></ul><p>As I continue my DevOps journey, I want to gain more insight and experience with <strong>Continuous Integration/Continuous Deployment (CI/CD)</strong> to simulate an application environment as best as I can.</p><p>In this article, we will use a CI/CD process to build and deploy a WordPress application infrastructure through Terraform.</p><h4>What is CI/CD?</h4><p>CI/CD is one of the core methods of development that allows organizations to build and ship software quickly and efficiently. By adding automation to the build and deploy process, we are able to continuously deliver code, test, and deploy, ensuring a constant flow of app features, upgrades, and bug fixes to customers. As DevOps engineers, it’s crucial to become familiar with CI/CD and constantly find ways to improve upon the deployment process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQuXd_tIbk-d0qkQGJZg8g.png" /><figcaption><a href="https://www.synopsys.com/glossary/what-is-cicd.html">https://www.synopsys.com/glossary/what-is-cicd.html</a></figcaption></figure><h4>What we’re building</h4><p>We are going to use GitHub and Terraform Cloud to deploy a highly-available WordPress application infrastructure through a simple CI/CD process.</p><p>The infrastructure includes:</p><ol><li>A VPC with two (2) public and private subnets.</li><li>An Auto Scaling Group (ASG) that provisions EC2 instances with WordPress installed.</li><li>An RDS MySQL database for WordPress to store data.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7FdSJGNzt7qpx4hUJj_uA.png" /></figure><h4>Terraform Cloud</h4><p>Although there are tons of CI/CD tools out there (GitLab, Circle CI, AWS Code Build/Code Deploy/Code Pipeline, etc…), we’re going to use GitHub and Terraform Cloud, as they provide the least amount of setup and a great introduction to CI/CD.</p><p><a href="https://www.terraform.io/">Terraform Cloud</a> allows us to easily manage resources, state, secrets, and deployments all in one platform. It can even integrate with other CI/CD tools to fit your organization’s existing processes.</p><h4>Prerequisites</h4><ol><li>An AWS account with IAM user access.</li><li>A <a href="https://github.com/">GitHub</a> and <a href="https://www.terraform.io/">Terraform Cloud</a> account.</li><li>Foundational knowledge of Linux file systems and commands.</li><li>Foundational knowledge of AWS resources, such as VPCs, subnets, security groups, Auto Scaling Groups, RDS, etc…</li><li>Foundational knowledge of Terraform basics.</li><li>Access to a terminal/command line tool.</li><li>An IDE, such as VS Code or Cloud9.</li></ol><h4>GitHub repo</h4><p><a href="https://github.com/aalokt89/terraform_wordpress_dev">Here is my GitHub repo if you want to follow along ➡️</a></p><h3>🚀 🚀 🚀 🚀 🚀 Let’s get started!</h3><h3>Configure GitHub &amp; Terraform Cloud</h3><h4>Create a repository</h4><p>Before we create our pipeline, we need to create a repository where we can store and version control our code. For this scenario, we’ll use <a href="https://github.com/">GitHub</a> and call the repo terraform_wordpress_dev. The repo will later be connected to our Terraform Cloud <strong>workspace</strong>, which allows the deployment to be triggered by pull or merge requests.</p><blockquote><strong>.gitignore: </strong>It’s important to have a .gitignore file so we don’t upload unnecessary .terraform and .tfstate files. Luckily, GitHub has a nice Terraform .gitignore template, so be sure to include that in the repo.</blockquote><p>We’ll also enable some standard branch protection rules so proper merge and pull request protocols are set in place (<em>Repo settings &gt; Branches</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvwYAS33y8wGfV0WW6vaSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-B_36R4bPyg9_FH5ZBnLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gory7zky410yjZ3tRSFWOw.png" /></figure><h4>Create a project &amp; workspace</h4><p>After creating a Terraform Cloud account, we’ll create a new <strong>project</strong> called <strong><em>Wordpress App</em></strong><em> </em>and a new <strong>workspace </strong>called <strong><em>wordpress_app_us_east_1_DEV.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*TfGxYJp-3vwOzW3bfyrDsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hSsayFp03qf-Sz0m0yZ96A.png" /></figure><p>Workspaces are like mini environments that allow teams to separate application flows and stages. We can use workspaces to separate our application into tiers, such as ‘<em>networking</em>,’ ‘<em>database</em>,’ ‘<em>frontend,’ </em>etc… or stages, such as ‘<em>dev</em>,’ ‘<em>QA</em>,’ and ‘<em>prod</em>.’</p><p>We’ll create a version control (VCS) workflow and connect our GitHub repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LAAVS0Sv5WQhPeH1qexrdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BV2Li-KDtBxwEnLq-_kUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p07m9sAtbNUj6HbAsihjUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ksd8q6JAgejfgaXstSWWiA.png" /></figure><p>If we expand the <em>Advanced options</em>, we’ll see more options on how our VCS can control deployments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eVeAs-4gBuT9GumDxODTVw.png" /></figure><p>We’ll select <em>Manual apply</em> to add an extra layer of protection before we let Terraform Cloud run and apply our code. This option means we’ll (more likely a higher-up/senior) have to manually approve and apply the planned deployment From Terraform Cloud.</p><p>We’ll see this in action later on and see what happens for each option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tdg9_LcXFCT9jeOzJK2x6A.png" /></figure><h4>Add workspace variables</h4><p>Terraform Cloud allows us to create workspace-specific terraform and environment variables to use in our deployments. These variables can also be shared across multiple workspaces.</p><p>We’ll use workspace <strong>Environment</strong> <strong>variables</strong> to store our AWS credentials, such as our AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Be sure to mark them as S<em>ensitive.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*feyEg1ggS7h414_wlHIEeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3w3hqQUNGxxuQi7-yqfWRw.png" /></figure><p>Alright! Our CI/CD pipeline is set up. Now we have to write the code for our WordPress app infrastructure!</p><h3>Create the network base</h3><p>Let’s start by creating the networking base for our app, which includes:</p><ol><li>A VPC with a CIDR block of 10.0.0.0/16.</li><li>Two (2) public and private subnets.</li><li>Public and private route tables.</li></ol><p>First, let’s create a new branch, called ‘<strong><em>networking’. </em></strong>Self-explanatory, but we should never ever be working on or committing to the <em>main</em> branch.</p><pre>$ git checkout -b networking</pre><h4>Login</h4><p>To use Terraform Cloud in the CLI, we first need to log in and generate an API token. Run the terraform login command in the terminal and follow the directions. We shouold be autmmatically taken to Terraform Cloud too generate the token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y06dI47RZcfRwleKTRE-sg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wmN1qFJ1ZTRL5f9mM7f3sw.png" /></figure><h4>Remote backend &amp; providers</h4><p>Since we’re using Terraform Cloud as our CI/CD tool, we can also use it as our remote backend to control the state. To configure the backend, we’ll need the organization and workspace name we set in our account.</p><pre>terraform {<br>  cloud {<br>    organization = &quot;ORGANIZATION NAME&quot;<br>    workspaces {<br>      name = &quot;wordpress_app_us_east_1_DEV&quot;<br>    }<br>  }<br>}</pre><p>We’ll also configure the AWS provider to assign the region and default tags. We can even add validation to variables to ensure proper values are used.</p><pre>#--variables.tf--<br><br># environment vars<br>#---------------------------------------<br>variable &quot;environment&quot; {<br>  type    = string<br>  default = &quot;DEV&quot;<br><br>  validation {<br>    condition     = contains([&quot;DEV&quot;, &quot;QA&quot;, &quot;PROD&quot;], upper(var.environment))<br>    error_message = &quot;The &#39;environment&#39; tag must be one of &#39;DEV&#39;,&#39;QA&#39;, or &#39;PROD&#39;.&quot;<br>  }<br>  validation {<br>    condition     = upper(var.environment) == var.environment<br>    error_message = &quot;The &#39;environment&#39; tag must be in all in uppercase.&quot;<br>  }<br>}<br>variable &quot;aws_region&quot; {<br>  type        = string<br>  description = &quot;AWS region&quot;<br>  default     = &quot;us-east-1&quot;<br>}<br>variable &quot;app_name&quot; {<br>  type        = string<br>  description = &quot;application name&quot;<br>  default     = &quot;wordpressApp&quot;<br>}<br><br>variable &quot;az_count&quot; {<br>  type        = number<br>  description = &quot;Number of availability zones to use&quot;<br>  default     = 2<br>}</pre><pre>#--providers.tf--<br><br>terraform {<br>  cloud {<br>    organization = &quot;ORGANIZATION NAME&quot;<br>    workspaces {<br>      name = &quot;wordpress_app_us_east_1_DEV&quot;<br>    }<br>  }<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;4.60.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>  default_tags {<br>    tags = {<br>      App         = var.app_name<br>      Environment = var.environment<br>      Terraform   = &quot;True&quot;<br>    }<br>  }<br>}</pre><p>Let’s run terraform init to initialize the local directory and configure the backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_hRgeJF68t09HiKw8vO3A.png" /></figure><p>Let’s also run a terraform apply just to initialize the state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BlfYLDWxgtfRiK5F_ye-g.png" /></figure><p>Whoops, Just kidding! Because we connected our repo to our Terraform Cloud CI/CD workspace, it will not let us apply our configuration. That’s a good thing! Being able to apply locally kind of defeats the purpose of automation, doesn&#39;t it? Everything should run through the proper CI/CD workflow and be applied only when we get the proper approval. We won&#39;t apply anything, just yet.</p><h4>Locals</h4><p>In a new <em>network.tf </em>file, let’s establish some <strong>locals.</strong></p><p>Locals are useful to use when you want to give the result of an expression and then re-use that result throughout your configuration. Some use cases for using locals are repeatable naming prefixes/conventions, common tags, and extracting <strong>data</strong> source information.</p><p>For our locals, we’ll establish a naming prefix (app_name-environment) and the names of the Availability Zones we want to use (us-east-1a, us-east-1b)</p><pre>#--network.tf--<br><br># Retrieve the list of AZs in the current AWS region<br>data &quot;aws_availability_zones&quot; &quot;available&quot; {}<br><br>locals {<br>  name_prefix = lower(&quot;${var.app_name}-${var.environment}&quot;)<br>  azs         = slice(data.aws_availability_zones.available.names, 0, var.az_count)<br>}</pre><h4>VPC module</h4><p>Hey, remember, for my last article, we had to manually create a VPC, Internet Gateway, public/private subnets, NAT Gateway, Elastic IP, public/private route tables, and public/private route table associations…? Yeah, we don’t have to do that anymore…</p><p>This time, we’ll use the power of <strong>modules </strong>to quickly create multiple resources or common settings all in one go.</p><p>Modules provide the greatest amount of flexibility and reusability of common configurations across multiple projects without having to create each resource from scratch. Instead of creating a custom module for the network, we’ll use a great <a href="https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest">VPC module</a> from the Terraform Registry. This VPC module allows us to create a VPC, subnets, Internet Gateway, NAT, and much more, all in one go!</p><p>I’ll admit, modules are a bit of a learning curve, and it does take a ton of pouring through the documentation to figure out all the options/inputs (there’s a lot…), but they are SUPER powerful once you get the hang of it.</p><p>In our <em>network.tf </em>file, we’ll call the vpc module with a <strong>module block.</strong></p><pre>#--network.tf--<br><br>module &quot;network&quot; {<br>  source                  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version                 = &quot;4.0.1&quot;<br>}</pre><p>The source argument defines where the module lives. If it’s a remote module, the source can be a Terraform Registry path, a Git repo link, etc… If it’s a local module, it will point to the module path within the main directory.</p><p>Here’s the full VPC module configuration we’ll use:</p><pre>#--network.tf--<br><br>module &quot;network&quot; {<br>  source               = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version              = &quot;4.0.1&quot;<br>  name                 = local.name_prefix<br>  cidr                 = var.vpc_cidr<br>  enable_dns_hostnames = var.enable_dns_hostnames<br><br>  azs = local.azs<br><br>  # Public subnets<br>  public_subnets = [for key, value in local.azs : cidrsubnet(var.vpc_cidr, var.public_newbits, key + var.public_newnum)]<br>  public_subnet_tags = {<br>    &quot;Tier&quot; = &quot;Web&quot;<br>  }<br>  map_public_ip_on_launch = var.map_public_ip_on_launch<br><br>  # Private subnets<br>  private_subnets = [for key, value in local.azs : cidrsubnet(var.vpc_cidr, var.private_newbits, key + var.private_newnum)]<br>  private_subnet_tags = {<br>    &quot;Tier&quot; = &quot;Database&quot;<br>  }<br><br>  # NAT<br>  enable_nat_gateway     = var.enable_nat_gateway<br>}</pre><pre>#--variables.tf--<br><br># VPC<br>#----------------------------------------<br>variable &quot;vpc_name&quot; {<br>  type        = string<br>  description = &quot;VPC name&quot;<br>  default     = &quot;vpc&quot;<br>}<br>variable &quot;vpc_cidr&quot; {<br>  type        = string<br>  description = &quot;VPC cidr&quot;<br>  default     = &quot;10.0.0.0/16&quot;<br>}<br>variable &quot;enable_dns_hostnames&quot; {<br>  type        = bool<br>  description = &quot;enable dns hostnames&quot;<br>  default     = true<br>}<br><br># Subnets<br>#----------------------------------------<br><br># Public<br>variable &quot;map_public_ip_on_launch&quot; {<br>  type        = bool<br>  description = &quot;enable auto-assign ipv4&quot;<br>  default     = true<br>}<br>variable &quot;public_newbits&quot; {<br>  type        = number<br>  description = &quot;number to add for public_subnet &#39;newbits&#39; cidrsubnet() function&quot;<br>  default     = 8<br>}<br>variable &quot;public_newnum&quot; {<br>  type        = number<br>  description = &quot;number to add for public_subnet &#39;newnum&#39; cidrsubnet() function&quot;<br>  default     = 100<br>}<br><br># Private<br>variable &quot;private_newbits&quot; {<br>  type        = number<br>  description = &quot;number to add for private_subnet &#39;newbits&#39; cidrsubnet() function&quot;<br>  default     = 8<br>}<br>variable &quot;private_newnum&quot; {<br>  type        = number<br>  description = &quot;number to add for private_subnet &#39;newnum&#39; cidrsubnet() function&quot;<br>  default     = 4<br>}<br><br># NAT gateway<br>#----------------------------------------<br>variable &quot;enable_nat_gateway&quot; {<br>  type        = bool<br>  description = &quot;enable NAT gateway&quot;<br>  default     = false<br>}<br>variable &quot;single_nat_gateway&quot; {<br>  type        = bool<br>  description = &quot;enable single NAT&quot;<br>  default     = false<br>}<br>variable &quot;one_nat_gateway_per_az&quot; {<br>  type        = bool<br>  description = &quot;enable NAT gateway in each AZ&quot;<br>  default     = false<br>}</pre><p>Let’s break this down a bit:</p><ul><li><strong>Subnets: </strong>for key, value in local.azs : cidrsubnet(local.vpc_cidr, var.public_newbits, key + var.public_newnum)<br>The module takes a public_subnets and private_subnets input, where we can iterate over our established AZs (us-east-1a &amp; us-east-1b) and create a public/private subnet for each zone. We can also use the cidrsubnet() function to assign the CIDR block for each.</li><li><strong>NAT Gateway:</strong> We don’t need a NAT for the RDS database, so we’ll set enable_nat_gateway to false.</li></ul><p>The beauty of this module is it will automatically create all the proper public/private routes tables, routes to the IGW and/or NAT, and route associations.</p><h4>Outputs</h4><p>Now that we have the networking module in place, let’s create a new <strong><em>outputs.tf</em></strong> file and output some resource information, such as the VPC ID and subnet IDs.</p><pre>#--outputs.tf--<br><br>output &quot;vpc_id&quot; {<br>  value = module.network.vpc_id<br>}<br><br>output &quot;public_subnets&quot; {<br>  value = module.network.public_subnets<br>}<br>output &quot;private_subnets&quot; {<br>  value = module.network.private_subnets<br>}</pre><h4>Plan</h4><p>We can’t apply the configuration, but we can still locally run a terraform plan to make sure the proper resources are created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nyf-E3MJDoLbKhMCDCHbaw.png" /><figcaption>VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRJS2dUBY-ZPbfJAMfARNg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tohlVDoGVkF_hRgWbxnJNw.png" /><figcaption>Public subnets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kQp1ILUt-yYL-Aa_fyt_6A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZDBpvutEW7NUGho0NreFzA.png" /><figcaption>Private subnets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OXMBJ1f4uxm2F4o-Lry72A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LI5gRjTPG9tikBpwzjQfBQ.png" /><figcaption>Route tables &amp; associations</figcaption></figure><p>Yeah… Hell yeah!</p><h4>Commit &amp; Push</h4><p>Now that we’ve completed the networking base of the app, let’s <strong>commit</strong> and <strong>push</strong> the code to the <strong>networking</strong> branch.</p><pre>git commit -a -m &quot;Create networking base.&quot;<br><br>git push</pre><h3>Create a pull request</h3><p>If we go back to our GitHub repo, we can create a <strong>pull request </strong>to allow someone from our team to review and approve our code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ddx8xD2WX6STC65Ez2mk3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0Fnv0P4I8zrXDKylL7i_A.png" /></figure><h4>Review <strong>speculative plan</strong></h4><p>Once a pull request has been made, Terraform Cloud will create a <strong>speculative plan</strong> so the reviewer can look over the planned resources and approve the pull request or require changes.</p><p>Notice how GitHub will create a ‘check’ for the Terraform plan. If we click on the <em>Details </em>link, we’ll be taken to the plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Rt1nmsLhXlHu7-74NMd9A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*roksMiwGlLSJQTQH03ko3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l6iOBsOt9sE1z1RbmlwP4w.png" /></figure><p>Here, the reviewer can look over the plan and add any comments, if necessary. If the plan fails, the ‘check’ will also fail, and we won’t be able to merge the branch.</p><blockquote><strong>NOTE:</strong> By nature of the GitHub flow, the one who made the pull request <strong>can’t</strong> be the <strong><em>reviewer</em></strong>, so we’ll have to pretend we’re the <em>reviewer,</em> commit cardinal sin, and approve our own requests. Just know, that we should be recieving all green checks before merging our branch.</blockquote><h3>Merge the networking branch to main</h3><p>After reviewing the code and plan, we’ve decided everything is good to go. We’re ready to <strong>merge</strong> our networking branch with the main branch and apply the Terraform plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRRWn7DAsqKG9dNLW2vDIQ.png" /><figcaption>Again, just for this demo, we’ll commit cardinal sin and merge our own requests. Shhhhhhh</figcaption></figure><h4>Review &amp; apply</h4><p>Once we’ve merged the branch into main, a new Terraform ‘<em>Run</em>’ will be triggered. Since we chose ‘<em>Manual apply</em>’<em> </em>in our workspace settings, we’ll still need to go into Terraform Cloud and manually apply the plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6uJvYuPObgOmJ0tdI6uqig.png" /></figure><p>If everything looks good, click on <em>Apply</em> and add a comment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZnpJf1B-uRh0bNxdZ8J7uA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0oxxkKoeJeI0EhqSTobog.png" /></figure><p>We should see a live view of all the resources being created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGn3_oRXFj6YoMcEVIrxog.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9iPJ-dJDxlJYrzetoOW1ww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rO46GJHhAlb28j5xa88QdQ.png" /><figcaption>VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FvNyjLmNt2ZAsYpSR0t7LQ.png" /><figcaption>Subnets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQ0wHiX4I3nSDUmt7hGsyg.png" /><figcaption>Route Tables</figcaption></figure><h4>Change settings to ‘Auto apply’</h4><p>Now let’s see what happens if we change the ‘<em>Apply method’</em> setting to ‘<em>Auto apply.</em>’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5AGqvBUOIq6pyql4kGskw.png" /></figure><p>Let’s create a new branch and make small change and adjust the <strong>private_newnum</strong> variable from <strong>4</strong> to <strong>8</strong>.</p><pre>git checkout -b &quot;private-cidr-fix&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qj8-A5WsbjCt5NpSgDEw3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AO5W0fJge767dK1_-Xu9WQ.png" /><figcaption>Result should be a CIDR change from “10.0.4.0/24” | “10.0.5.0/24” → “10.0.8.0/24” | “10.0.9.0/24”</figcaption></figure><p>Commit the changes, push to the branch, and create a new pull request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QWuVStvnNHrndbsFOHS3IA.png" /></figure><p>Once the plan has succeeded, review the plan and merge to the main branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IGcgJ7ohmyx6xCFafCY_jA.png" /></figure><p>Before, we had to manually apply the plan. Now, the plan will automatically apply when we merge to main! Since we’re now automating the <em>apply</em> stage, we have to take extra care when reviewing and approving someone’s code (Again, in the real world, this will probably done by a senior team member).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1Sk0rYjKq-LmCtv4loygQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77vuJ8Af87o2HyIQsFZKXg.png" /></figure><h3>Success!</h3><p>Woo! We’ve just used our first CI/CD pipeline to deploy a networking base for our application!</p><p>That’s all for now. In Part 2, we’ll continue on and configure the ASG to provision the instances so they have WordPress properly installed and connected to an RDS database.</p><p>Stay tuned!</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><p>Feel free to connect with me on <a href="https://www.linkedin.com/in/aalok-trivedi">LinkedIn</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f1cb9f4f022" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying application infrastructure to dynamically scale and handle high traffic with AWS &…]]></title>
            <link>https://medium.com/@aaloktrivedi/deploying-application-infrastructure-to-dynamically-scale-and-handle-high-traffic-with-aws-9d92b43979cd?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d92b43979cd</guid>
            <category><![CDATA[autoscaling]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Fri, 07 Apr 2023 20:56:26 GMT</pubDate>
            <atom:updated>2023-04-07T22:26:46.729Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TW3D8uUCJQRmrjYKDMvbNw.png" /></figure><h3>Deploying application infrastructure to dynamically scale and handle high traffic with AWS &amp; Terraform</h3><h3>Intro</h3><p>Infrastructure as Code (IaC) is increasingly becoming a necessity for lightning quick resource deployment, easy maintenance, and high flexibility and modularity. Terraform, one of the leading open-source IaC tools, allows for cloud-agnostic infrastructure configuration with ease. In this article, I’ll show you just how easy it is to use Terraform and AWS to configure application infrastructure that responds to high demand.</p><h4>Scenario</h4><p>Our e-commerce web application, <em>Shoppr</em>, is having a major sales event in a couple of weeks. The engineering team wants to be prepared for the anticipated increase in traffic and use Terraform to restructure parts of the infrastructure to manage AWS resource configurations.</p><p>The application needs to be highly available and handle times of high AND low traffic without a ton of server overhead and maintenance.</p><blockquote><strong>Prior reading:</strong> If you haven’t already, I highly recommend reviewing my last article about <a href="https://medium.com/@aaloktrivedi/automating-with-infrastructure-as-code-deploying-a-jenkins-server-with-terraform-e32f7cc79fe5">deploying a Jenkins server with Terraform</a>, as I go over many of the basic Terraform concepts in more detail. Many of the steps we’ll take are very similar and repeatable.</blockquote><h4>What we’re building</h4><ol><li>A remote backend for Terraform to store <strong>state </strong>in S3.</li><li>A <strong>VPC</strong> with two (2) <strong>public</strong> and two (2) <strong>private subnets </strong>in two (2) <strong>availability zones</strong>.</li><li>An<strong> Application Load Balancer (ALB)</strong> deployed in the public subnets that distributes traffic to the ASG.</li><li>An <strong>Auto Scaling Group (ASG)</strong> that deploys web server EC2 instances in the private subnets and dynamically scales based on demand.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KOteBut1ZeJVNEcZJUq0zQ.png" /></figure><h4>Prerequisites</h4><ol><li>An AWS account with IAM user access.</li><li>Foundational knowledge of Linux systems and commands.</li><li>Foundational knowledge of AWS resources, such as VPCs, subnets, security groups, Auto Scaling Groups, S3, etc…</li><li>Foundational knowledge of Terraform basics.</li><li>Access to a command line tool.</li><li>An IDE, such as VS Code or Cloud9.</li></ol><h4>GitHub repo</h4><p><a href="https://github.com/aalokt89/terraform_auto_scaling_group">Here is a link to my repo if you want to follow along ➡️</a></p><h3>🚀 🚀 🚀 🚀 🚀 Let’s get started!</h3><h3>Set up a remote backend</h3><p>By default, Terraform creates the backend locally and stores <em>state </em>in our working directory. This can cause issues when trying to work with team members because they won’t have access to the most current state file. However, we can create a <strong>remote backend</strong> with S3 and DynamoDB so Terraform can control the state remotely and securely in the cloud.</p><p>I’ve detailed <a href="https://medium.com/@aaloktrivedi/configuring-a-terraform-remote-backend-with-s3-and-dynamodb-ebcefa8432ea">how to create a remote backend</a> in a previous article, so I highly recommend taking these steps before moving forward.</p><h3>Configure the providers</h3><p>After the remote backend has been configured, let’s configure Terraform and establish the providers we’ll need (AWS).</p><p>All of our variables will be stored in a <em>variables.tf</em> file.</p><pre>#--variables.tf--<br><br># environment vars<br>#---------------------------------------<br>variable &quot;environment&quot; {<br>  type    = string<br>  default = &quot;Dev&quot;<br>}<br><br>variable &quot;aws_region&quot; {<br>  type    = string<br>  default = &quot;us-east-1&quot;<br>}<br># naming vars<br>#---------------------------------------<br>variable &quot;app_name&quot; {<br>  type        = string<br>  description = &quot;app name prefix for naming&quot;<br>  default     = &quot;shoppr&quot;<br>}</pre><p>In a <em>providers.tf</em> file, we’ll configure the backend, required providers, and AWS provider. We’ll also get all the Availability Zones in our region (us-east-1) with a data block.</p><pre>#--providers.tf--<br><br>terraform {<br>  backend &quot;s3&quot; {<br>    bucket         = &quot;shoppr-remote-backend&quot;<br>    key            = &quot;global/s3/terraform.tfstate&quot;<br>    region         = &quot;us-east-1&quot;<br>    dynamodb_table = &quot;shoppr-tfstate-locking&quot;<br>    encrypt        = true<br>  }<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;4.60.0&quot;<br>    }<br>    random = {<br>      source  = &quot;hashicorp/random&quot;<br>      version = &quot;3.4.3&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = var.aws_region<br>  default_tags {<br>    tags = {<br>      App         = var.app_name<br>      Environment = var.environment<br>      Terraform   = &quot;True&quot;<br>    }<br>  }<br>}<br><br># Retrieve the list of AZs in the current AWS region<br>data &quot;aws_availability_zones&quot; &quot;available&quot; {}<br>data &quot;aws_region&quot; &quot;current&quot; {}</pre><h4>Initialize the directory</h4><p>We need to run terraform init to initialize the backend and providers, and prepare the working directory for Terraform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8s4xLXHJANtvP2Rt0zOeVA.png" /></figure><blockquote><strong>NOTE: </strong>For organization purposes, we’ll put all the variables in a single variables.tf file, and all the resources in their own .tf files (vpc.tf, subnets.tf, route_tables.tf, etc…).</blockquote><h3>Create the network base</h3><h4>What we‘ll need</h4><ol><li>A VPC.</li><li>Two (2) public and two (2) private subnets.</li><li>An Internet Gateway.</li><li>A NAT Gateway.</li><li>A public and private route table.</li></ol><h4>VPC</h4><pre>#--variables.tf--<br><br># vpc vars<br>#----------------------------------------<br>variable &quot;vpc_cidr&quot; {<br>  type        = string<br>  description = &quot;VPC cidr block&quot;<br>  default     = &quot;10.0.0.0/16&quot;<br>}<br><br>variable &quot;enable_dns_hostnames&quot; {<br>  type        = bool<br>  description = &quot;enable dns hostnames&quot;<br>  default     = true<br>}</pre><pre>#--vpc.tf--<br><br># Define the VPC<br>#----------------------------------------------------<br>resource &quot;aws_vpc&quot; &quot;vpc&quot; {<br>  cidr_block           = var.vpc_cidr<br>  enable_dns_hostnames = var.enable_dns_hostnames<br><br>  tags = {<br>    Name = &quot;${var.app_name}-vpc&quot;<br>  }<br>}</pre><h4>Subnets</h4><pre>#--variables.tf--<br><br># public subnet vars<br>#----------------------------------------<br>variable &quot;public_subnets&quot; {<br>  default = {<br>    &quot;public-subnet-1&quot; = 0<br>    &quot;public-subnet-2&quot; = 1<br>  }<br>}<br>variable &quot;auto_ipv4&quot; {<br>  type        = bool<br>  description = &quot;enable auto-assign ipv4&quot;<br>  default     = true<br>}<br><br># private subnet vars<br>#----------------------------------------<br>variable &quot;private_subnets&quot; {<br>  default = {<br>    &quot;private-subnet-1&quot; = 0<br>    &quot;private-subnet-2&quot; = 1<br>  }<br>}</pre><pre>#--subnets.tf--<br><br># deploy subnets<br>#----------------------------------------------------<br># deploy the public subnets<br>resource &quot;aws_subnet&quot; &quot;public_subnets&quot; {<br>  for_each          = var.public_subnets<br>  vpc_id            = aws_vpc.vpc.id<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value + 100)<br>  availability_zone = tolist(data.aws_availability_zones.available.names)[each.value]<br><br>  map_public_ip_on_launch = var.auto_ipv4<br><br>  tags = {<br>    Name = &quot;${var.app_name}-${each.key}&quot;<br>    Subnet = &quot;Public&quot;<br><br>  }<br>}<br><br># deploy the private subnets<br>resource &quot;aws_subnet&quot; &quot;private_subnets&quot; {<br>  for_each          = var.private_subnets<br>  vpc_id            = aws_vpc.vpc.id<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value + 1)<br>  availability_zone = tolist(data.aws_availability_zones.available.names)[each.value]<br><br>  tags = {<br>    Name = &quot;${var.app_name}-${each.key}&quot;<br>    Subnet = &quot;Private&quot;<br>  }<br>}</pre><p>We’ll iterate over the subnet vars using the for_each key so we can create multiple resources from one declaration.</p><p>The cidrsubnet() <a href="https://developer.hashicorp.com/terraform/language/functions/cidrsubnet">function</a> dynamically generates the CIDR blocks within the VPC CIDR, and we can use the AZ data from the earlier data block to place them in their respective AZs.</p><ul><li>public-subnet-1: 10.0.100.0/24 / private-subnet-1: 10.0.1.0/24= us-east-1a</li><li>public-subnet-2: 10.0.101.0/24 / private-subnet-2: 10.0.2.0/24= us-east-1b</li></ul><h4>Plan &amp; apply</h4><p>Let’s terraform plan and terraform apply what we have so far.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hA2SMsant8PgOmNx_VMomQ.png" /><figcaption>VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NSWdcq94jXDR3M5vj7ezwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXOH4G3tg-LZ2TVZs9UtqA.png" /><figcaption>Public subnets</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0gM_MNrty8_l9GaMjDMuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFjCGcuOMH08yOmPQ9leRA.png" /><figcaption>Private subnets</figcaption></figure><h4>Internet gateway (IGW) &amp; NAT gateway</h4><p>We’ll create an <strong>IGW</strong> and <strong>NAT</strong> so instances in the public and private subnets can access the internet.</p><pre>#--gateways.tf--<br><br># Create Internet Gateway<br>#----------------------------------------------------<br>resource &quot;aws_internet_gateway&quot; &quot;igw&quot; {<br>  vpc_id = aws_vpc.vpc.id<br>  tags = {<br>    Name = &quot;${var.app_name}-igw&quot;<br>  }<br>}</pre><p>The NAT needs an <strong>elastic IP</strong>, so we’ll create a resource for the eip and allocate the id to the NAT. We’ll also place it in one of the public subnets.</p><pre>#--gateways.tf--<br><br># Create NAT Gateway<br>#----------------------------------------------------<br><br># Create elastic IP<br>resource &quot;aws_eip&quot; &quot;eip&quot; {<br>  vpc = true<br>}<br><br>resource &quot;aws_nat_gateway&quot; &quot;nat_gw&quot; {<br>  depends_on    = [aws_internet_gateway.igw]<br>  allocation_id = aws_eip.eip.id<br>  subnet_id     = aws_subnet.public_subnets[&quot;public-subnet-1&quot;].id<br><br>  tags = {<br>    Name = &quot;${var.app_name}-nat&quot;<br>  }<br>}</pre><h4>Public and private route tables</h4><p>Now let’s create our route tables and route associations. We’ll need a public and private route table and associate them with the public and private subnets.</p><p>We’ll use the VPC’s <strong>default route table</strong> as the public_rt and add a <strong>route</strong> to direct all traffic to the IGW.</p><pre>#--route_tables.tf--<br><br># Edit default route table for public subnets<br>#----------------------------------------------------<br># Public rt<br>resource &quot;aws_default_route_table&quot; &quot;public_rt&quot; {<br>  default_route_table_id = aws_vpc.vpc.default_route_table_id<br><br>  route {<br>    cidr_block = var.all_traffic<br>    gateway_id = aws_internet_gateway.igw.id<br>  }<br>  tags = {<br>    Name = &quot;${var.app_name}-public-rt&quot;<br>  }<br>}<br><br># Public rt assoc.<br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  depends_on     = [aws_subnet.public_subnets]<br>  route_table_id = aws_default_route_table.public_rt.id<br>  for_each       = aws_subnet.public_subnets<br>  subnet_id      = each.value.id<br>}</pre><p>We’ll create a new private_rt and add a <strong>route</strong> to direct all traffic to the NAT.</p><pre>#--route_tables.tf--<br><br># Route table for private subnets<br>#----------------------------------------------------<br># Private rt<br>resource &quot;aws_route_table&quot; &quot;private_rt&quot; {<br>  vpc_id = aws_vpc.vpc.id<br><br>  route {<br>    cidr_block     = var.all_traffic<br>    nat_gateway_id = aws_nat_gateway.nat_gw.id<br>  }<br>  tags = {<br>    Name = &quot;${var.app_name}-private-rt&quot;<br>  }<br>}<br><br># Private rt assoc.<br>resource &quot;aws_route_table_association&quot; &quot;private&quot; {<br>  depends_on     = [aws_subnet.private_subnets]<br>  route_table_id = aws_route_table.private_rt.id<br>  for_each       = aws_subnet.private_subnets<br>  subnet_id      = each.value.id<br>}</pre><h4>Plan &amp; apply</h4><p>Let’s plan and apply route tables and associations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EbzwZbaj_8NRYdmAuTBRZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KcHfyHSUUfDA4qlLqT4xmw.png" /><figcaption>Public and private route tables.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEs9iVvmcjGRkNEL_hHhWg.png" /><figcaption>Public and private route table associations.</figcaption></figure><h3>Create the security groups</h3><p>We’ll create three security groups to allow proper inbound/outbound traffic:</p><ol><li>One to allow for SSH.</li><li>One to allow inbound HTTP/HTTPS traffic (we’ll attach this to the ALB).</li><li>One to allow inbound traffic from the ALB (we’ll attach this to the ASG instances).</li></ol><h4>SSH security group</h4><pre>#--variables.tf--<br><br># security group vars<br>#----------------------------------------<br>variable &quot;ssh_sg&quot; {<br>  type        = map(any)<br>  description = &quot;ssh security group vars&quot;<br>  default = {<br>    &quot;cidr_block&quot;            = &quot;0.0.0.0/0&quot;<br>    &quot;timeout_delete&quot;        = &quot;2m&quot;<br>  }<br>}</pre><pre>#--security_groups.tf--<br><br># ssh security group<br>#----------------------------------------------------<br>resource &quot;aws_security_group&quot; &quot;ssh_sg&quot; {<br>  name        = &quot;${var.app_name}-ssh-sg&quot;<br>  description = &quot;Allow ssh&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br>  ingress {<br>    description = &quot;ssh from IP&quot;<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.ssh_sg[&quot;cidr_block&quot;]]<br>  }<br><br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.ssh_sg[&quot;timeout_delete&quot;]<br>  }<br><br>  tags = {<br>    Name = &quot;${var.app_name}-ssh-sg&quot;<br>  }<br>}</pre><p>Because resources are highly dependent on security groups, AWS will not allow us to delete them if they have an association. This can cause deletion issues when Terraform tries to change or delete security groups. The create_before_destroy argument tells Terraform to create the new/updated group <strong>before</strong> destroying the old one to ensure any associations with resources are not broken. for more on this topic, visit the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group#recreating-a-security-group">official documentation</a>.</p><blockquote><strong>Disclaimer: </strong>Generally, you want to lock down the SSH IP address to an authorized CIDR range or secure address, but for demo purposes, we’ll leave it to “<em>anywhere</em>” or your local machine’s &quot;&lt;public_IP&gt;/32&quot;.</blockquote><h4>Web access security group</h4><p>We need to allow inbound HTTP/HTTPS traffic to the ALB.</p><pre>variable &quot;web_access_sg&quot; {<br>  type        = map(any)<br>  description = &quot;web access security group vars &quot;<br>  default = {<br>    &quot;timeout_delete&quot; = &quot;2m&quot;<br>  }<br>}</pre><pre>#--security_groups.tf--<br><br># Web access security group<br>#----------------------------------------------------<br>resource &quot;aws_security_group&quot; &quot;web_access_sg&quot; {<br>  name        = &quot;${var.app_name}-web-access-sg&quot;<br>  description = &quot;Allow http/https traffic to alb&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br>  # http<br>  ingress {<br>    description = &quot;http&quot;<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.all_traffic]<br>  }<br>  # https<br>  ingress {<br>    description = &quot;https&quot;<br>    from_port   = 443<br>    to_port     = 443<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.all_traffic]<br>  }<br>  #all outbound traffic<br>  egress {<br>    from_port        = 0<br>    to_port          = 0<br>    protocol         = &quot;-1&quot;<br>    cidr_blocks      = [var.all_traffic]<br>    ipv6_cidr_blocks = [&quot;::/0&quot;]<br>  }<br>  <br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.web_access_sg[&quot;timeout_delete&quot;]<br>  }<br><br><br>  tags = {<br>    Name = &quot;${var.app_name}-web-access-sg&quot;<br>  }<br>}</pre><h4>ASG instance security group</h4><p>We need to allow inbound HTTP/HTTPS traffic from the ALB to our instances. Instead of a cidr_block, we’ll use security_groups as the source and input the web_access_sg ID.</p><pre>resource &quot;aws_security_group&quot; &quot;alb_access_sg&quot; {<br>  name        = &quot;${var.app_name}-alb-access-sg&quot;<br>  description = &quot;Allow http traffic from ALB&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br>  # http<br>  ingress {<br>    description     = &quot;allow traffic from alb sg&quot;<br>    from_port       = 80<br>    to_port         = 80<br>    protocol        = &quot;tcp&quot;<br>    security_groups = [aws_security_group.web_access_sg.id]<br>  }<br><br>  egress {<br>    from_port        = 0<br>    to_port          = 0<br>    protocol         = &quot;-1&quot;<br>    cidr_blocks      = [var.all_traffic]<br>    ipv6_cidr_blocks = [&quot;::/0&quot;]<br>  }<br><br>  lifecycle {<br>    create_before_destroy = true<br>  }<br><br>  timeouts {<br>    delete = var.alb_access_sg[&quot;timeout_delete&quot;]<br>  }<br><br>  tags = {<br>    Name = &quot;${var.app_name}-alb-access-sg&quot;<br>  }<br>}</pre><h4>Plan &amp; apply</h4><p>You know the deal…</p><h3>Create the Application Load Balancer (ALB)</h3><p>Before we create our ASG, let’s create the ALB.</p><p>In times of high traffic volume, The ASG will scale our web servers based on demand, but we need a way to efficiently distribute incoming traffic to the proper targets (in our case, the web server EC2 instances).</p><h4>What we’ll need</h4><ol><li>An <strong>ALB </strong>to distribute the traffic among the web servers.</li><li>A <strong>target group</strong> that targets our web server EC2 instances.</li><li>A <strong>listener</strong> to listen for HTTP traffic on port 80 so the ALB can direct traffic to the target group.</li></ol><h4>ALB</h4><p>First, let’s create the ALB resource</p><pre>#--variables.tf--<br><br># ALB vars<br>#----------------------------------------<br>variable &quot;load_balancer_name&quot; {<br>  type        = string<br>  description = &quot;load balancer name&quot;<br>  default     = &quot;web-alb&quot;<br>}<br><br>variable &quot;load_balancer_internal&quot; {<br>  type        = bool<br>  description = &quot;Is looad balancer internal facing?&quot;<br>  default     = false<br>}<br><br>variable &quot;load_balancer_type&quot; {<br>  type        = string<br>  description = &quot;load balancer&quot;<br>  default     = &quot;application&quot;<br>}</pre><pre>#--load_balancer.tf--<br><br># ALB<br>#----------------------------------------<br>resource &quot;aws_lb&quot; &quot;web_alb&quot; {<br>  name               = &quot;${var.app_name}-${var.load_balancer_name}&quot;<br>  internal           = var.load_balancer_internal<br>  load_balancer_type = var.load_balancer_type<br>  security_groups    = [aws_security_group.web_access_sg.id]<br>  subnets            = [for subnet in aws_subnet.public_subnets : subnet.id]<br><br>  tags = {<br>    Name = &quot;${var.app_name}-${var.load_balancer_name}&quot;<br>  }<br>}</pre><p>We want the ALB to be internet-facing, so we’ll set the internal argument to false. We’ll also attach the web_access_sg security group and iterate over the aws_subnet resource to place the ALB in the <strong>public subnets</strong>.</p><h4>Target group</h4><p>We need a target group to point traffic to the ASG web server instances. The aws_lb_target_group resource needs to direct <strong>HTTP</strong> traffic on port <strong>80</strong> in our VPC.</p><pre>#--variables.tf--<br><br># ALB target group vars<br>#----------------------------------------<br>variable &quot;web_alb_tg_http&quot; {<br>  type        = map(any)<br>  description = &quot;http target group vars&quot;<br>  default = {<br>    &quot;name&quot;     = &quot;web-server-tg-http&quot;<br>    &quot;port&quot;     = 80<br>    &quot;protocol&quot; = &quot;HTTP&quot;<br>  }<br>}</pre><pre>#--load_balancer.tf--<br><br># ALB target group<br>#----------------------------------------<br># HTTP<br>resource &quot;aws_lb_target_group&quot; &quot;web_alb_tg_http&quot; {<br>  depends_on = [aws_lb.web_alb]<br>  name       = &quot;${var.app_name}-${var.web_alb_tg_http[&quot;name&quot;]}&quot;<br>  port       = var.web_alb_tg_http[&quot;port&quot;]<br>  protocol   = var.web_alb_tg_http[&quot;protocol&quot;]<br>  vpc_id     = aws_vpc.vpc.id<br>}</pre><h4>ALB listener</h4><p>Lastly, we need a listener to detect <strong>HTTP</strong> traffic on port <strong>80</strong> and connect it to the <strong>target group</strong> and <strong>ALB</strong>. We’ll attach the listener to the ALB and <strong>forward</strong> the traffic to the t<strong>arget group</strong>.</p><pre>#--variables.tf--<br><br># ALB listener vars<br>#----------------------------------------<br>variable &quot;web_alb_listener_http&quot; {<br>  type        = map(any)<br>  description = &quot;http listener vars&quot;<br>  default = {<br>    &quot;port&quot;        = 80<br>    &quot;protocol&quot;    = &quot;HTTP&quot;<br>    &quot;action_type&quot; = &quot;forward&quot;<br>  }<br>}</pre><pre>#--load_balancer.tf--<br><br># ALB listeners<br>#----------------------------------------<br># HTTP<br>resource &quot;aws_lb_listener&quot; &quot;web_alb_listener_http&quot; {<br>  load_balancer_arn = aws_lb.web_alb.arn<br>  port              = var.web_alb_listener_http[&quot;port&quot;]<br>  protocol          = var.web_alb_listener_http[&quot;protocol&quot;]<br><br>  default_action {<br>    type             = var.web_alb_listener_http[&quot;action_type&quot;]<br>    target_group_arn = aws_lb_target_group.web_alb_tg_http.arn<br>  }<br>}</pre><blockquote><strong>Note:</strong> If you have HTTPS certificates and want to allow HTTPS traffic, you’ll need an additional target group and listener that listens for HTTPS on port 443.</blockquote><h4>Plan &amp; apply</h4><p>Plan and apply the resources. We’ll test the ALB when we create our ASG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Owxx4r11O395mp74J9xCnA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BrF20jnkPCbzNXp0QyrGgA.png" /><figcaption>Target group and listener</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ya_D0yuVJaS77hYdiIS5MQ.png" /><figcaption>ALB</figcaption></figure><h3>Create the EC2 launch template</h3><p>Alright! Our network base, security groups, and ALB are configured and deployed, so let’s set up our web servers. Before we create our ASG, we need a template that will define the type of EC2 instances to launch.</p><p>We’ll be using <strong>Amazon Linux 2</strong>, <strong>t2-micro</strong> instances bootstrapped with <strong>Apache</strong>.</p><h4>Apache script</h4><p>Let’s create a separate install_httpd.sh file that contains our script to install an Apache web server on each instance (<em>the index.html code is optional. The designer in me can’t resist being extra fancy</em>).</p><pre>#!/bin/bash<br>sudo yum update -y<br>sudo yum install -y httpd &amp;&amp; sudo yum enable httpd<br>sudo yum start httpd<br><br>sudo echo &#39;&lt;!DOCTYPE html&gt;<br>&lt;html lang=&quot;en&quot;&gt;<br> &lt;head&gt;<br>  &lt;meta charset=&quot;utf-8&quot; /&gt;<br>  &lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;<br>  &lt;title&gt;A Basic HTML5 Template&lt;/title&gt;<br>  &lt;link rel=&quot;preconnect&quot; href=&quot;https://fonts.googleapis.com&quot; /&gt;<br>  &lt;link rel=&quot;preconnect&quot; href=&quot;https://fonts.gstatic.com&quot; crossorigin /&gt;<br>  &lt;link<br>   href=&quot;https://fonts.googleapis.com/css2?family=Inter:wght@400;500;700;800&amp;display=swap&quot;<br>   rel=&quot;stylesheet&quot;<br>  /&gt;<br>  &lt;link rel=&quot;stylesheet&quot; href=&quot;css/styles.css?v=1.0&quot; /&gt;<br> &lt;/head&gt;<br> &lt;body&gt;<br>  &lt;div class=&quot;wrapper&quot;&gt;<br>   &lt;div class=&quot;container&quot;&gt;<br>    &lt;h1&gt;Welcome! An Apache web server has been started successfully.&lt;/h1&gt;<br>    &lt;p&gt;Replace this with your own index.html file in /var/www/html.&lt;/p&gt;<br>   &lt;/div&gt;<br>  &lt;/div&gt;<br> &lt;/body&gt;<br>&lt;/html&gt;<br>&lt;style&gt;<br> body {<br>  background-color: #34333d;<br>  display: flex;<br>  align-items: center;<br>  justify-content: center;<br>  font-family: Inter;<br>  padding-top: 128px;<br> }<br> .container {<br>  box-sizing: border-box;<br>  width: 741px;<br>  height: 449px;<br>  display: flex;<br>  flex-direction: column;<br>  justify-content: center;<br>  align-items: flex-start;<br>  padding: 48px 48px 48px 48px;<br>  box-shadow: 0px 1px 32px 11px rgba(38, 37, 44, 0.49);<br>  background-color: #5d5b6b;<br>  overflow: hidden;<br>  align-content: flex-start;<br>  flex-wrap: nowrap;<br>  gap: 24;<br>  border-radius: 24px;<br> }<br> .container h1 {<br>  flex-shrink: 0;<br>  width: 100%;<br>  height: auto; /* 144px */<br>  position: relative;<br>  color: #ffffff;<br>  line-height: 1.2;<br>  font-size: 40px;<br> }<br> .container p {<br>  position: relative;<br>  color: #ffffff;<br>  line-height: 1.2;<br>  font-size: 18px;<br> }<br>&lt;/style&gt;<br>&#39; &gt; /var/www/html/index.html</pre><h4>Launch template</h4><p>The aws_launch_template resource takes in most of the same arguments as a regular aws_instance. We’ll create vars for each of the necessary arguments and insert them into the resource. Don’t forget to add <strong>both</strong> the ssh_sg and alb_access_sg security group IDs.</p><pre>#--variables.tf--<br><br># ec2 vars<br>#----------------------------------------<br>variable &quot;web_server_name&quot; {<br>  type    = string<br>  default = &quot;web-server&quot;<br>}<br>variable &quot;web_server_ami&quot; {<br>  type        = string<br>  description = &quot;Instance AMI: Amazon Linux 2&quot;<br>  default     = &quot;ami-04581fbf744a7d11f&quot;<br>}<br>variable &quot;web_server_type&quot; {<br>  type    = string<br>  default = &quot;t2.micro&quot;<br>}<br><br>variable &quot;key_pair&quot; {<br>  type        = string<br>  description = &quot;ec2 key pair&quot;<br>  default     = &quot;YOUR_KEY_PAIR&quot;<br>}<br><br>variable &quot;user_data_file&quot; {<br>  type        = string<br>  description = &quot;user data file path&quot;<br>  default     = &quot;install_httpd.sh&quot;<br>}</pre><pre>#--ec2_launch_template.tf--<br><br># EC2 web launch template<br>#----------------------------------------<br>resource &quot;aws_launch_template&quot; &quot;web_server&quot; {<br>  name                   = &quot;${var.app_name}-${var.web_server_name}-template&quot;<br>  image_id               = var.web_server_ami<br>  instance_type          = var.web_server_type<br>  user_data              = filebase64(&quot;${path.module}/${var.user_data_file}&quot;)<br>  key_name               = var.key_pair<br>  vpc_security_group_ids = [aws_security_group.ssh_sg.id, aws_security_group.alb_access_sg.id]<br><br>  tags = {<br>    &quot;Name&quot; = &quot;${var.app_name}-${var.web_server_name}-template&quot;<br>  }<br>}</pre><h4>Plan &amp; apply</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MY7q_EEXr8maEV7PqaIvHg.png" /></figure><h3>Create Auto Scaling Group (ASG)</h3><p>Now that our launch template is complete, we can create our ASG to launch our web server instances and have it scale based on demand.</p><h4>ASG</h4><p>Our ASG will have a min_sizeof <strong>2</strong>, and max_size of <strong>5</strong>.</p><pre>#--variables.tf--<br><br># ASG vars<br>#----------------------------------------<br>variable &quot;web_asg_capacity&quot; {<br>  type        = map(any)<br>  description = &quot;min, max, and desired instance capacity&quot;<br>  default = {<br>    &quot;min&quot;     = 2<br>    &quot;max&quot;     = 5<br>  }<br>}<br>variable &quot;web_asg_health_check_type&quot; {<br>  type        = string<br>  description = &quot;health check type&quot;<br>  default     = &quot;ELB&quot;<br>}</pre><pre>#--auto_scaling_group.tf--<br><br># Auto scaling group<br>#----------------------------------------<br>resource &quot;aws_autoscaling_group&quot; &quot;web_server_asg&quot; {<br>  name                = &quot;${var.app_name}-web-server-asg&quot;<br>  min_size            = var.web_asg_capacity[&quot;min&quot;]<br>  max_size            = var.web_asg_capacity[&quot;max&quot;]<br>  vpc_zone_identifier = [for subnet in aws_subnet.private_subnets : subnet.id]<br>  target_group_arns   = [aws_lb_target_group.web_alb_tg_http.arn]<br>  health_check_type   = var.web_asg_health_check_type<br><br>  launch_template {<br>    id      = aws_launch_template.web_server.id<br>    version = &quot;$Latest&quot;<br>  }<br><br>  tag {<br>    key                 = &quot;Name&quot;<br>    value               = &quot;${var.app_name}-${var.web_server_name}-asg&quot;<br>    propagate_at_launch = true<br>  }<br>}</pre><p>The vpc_zone_indentifier takes a list of subnet IDs for the ASG to deploy instances into. We can iterate over our aws_subnet.private_subnets to get the ASG to span across many AZs and place instances in those subnets.</p><p>Don’t forget to add the target_group_arns from the ALB target group.</p><p>We’ll also assign our <strong>web-server</strong> launch-template to the ASG.</p><h4>Scaling policy</h4><p>Lastly, let’s add a <strong>scaling policy </strong>to determine how to scale the web servers. We want to scale up instances when the <strong>average CPU utilization</strong> goes above <strong>75%.</strong></p><ul><li>policy_type = “TargetTrackingScaling”</li><li>metric_type = “ASGAverageCPUUtilization”</li><li>target_value = 75.0</li></ul><pre>#--variables.tf--<br><br>variable &quot;web_asg_scaling_policy&quot; {<br>  type        = map(any)<br>  description = &quot;scaling policy&quot;<br>  default = {<br>    &quot;policy_type&quot;  = &quot;TargetTrackingScaling&quot;<br>    &quot;metric_type&quot;  = &quot;ASGAverageCPUUtilization&quot;<br>    &quot;target_value&quot; = 75.0<br>  }<br>}</pre><pre>#--auto_scaling_group.tf--<br><br># Scaling policy<br>resource &quot;aws_autoscaling_policy&quot; &quot;web_server_asg&quot; {<br>  depends_on             = [aws_autoscaling_group.web_server_asg]<br>  name                   = &quot;${var.app_name}-asg-scaling-policy&quot;<br>  policy_type            = var.web_asg_scaling_policy[&quot;policy_type&quot;]<br>  autoscaling_group_name = aws_autoscaling_group.web_server_asg.name<br><br>  target_tracking_configuration {<br>    predefined_metric_specification {<br>      predefined_metric_type = var.web_asg_scaling_policy[&quot;metric_type&quot;]<br>    }<br><br>    target_value = var.web_asg_scaling_policy[&quot;target_value&quot;]<br>  }<br>}</pre><h4>Plan &amp; apply</h4><p>Let’s deploy our ASG with terraform plan and terraform apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUFwPc2ZTSrsZ07GDYM0mA.png" /></figure><h3>Testing!</h3><p>We’ve got all of our resources deployed, so let’s test if it works!</p><p>To test if the ALB is properly distributing traffic to the ASG instances, we need the ALB dns_name. We can either get this from the Console… OR… we can directly <strong>output</strong> this information to the CLI.</p><h4>Outputs</h4><p>An <strong>output block</strong> allows us to output useful resource information such as IDs, names, ARNs, etc… when deployed. This is super useful for creating modules or just getting common info.</p><p>In a new <em>outputs.tf </em>file, we’ll output the public dns_name of the ALB. We can even use string interpolation to create a clickable link!</p><pre>#--outputs.tf--<br><br>output &quot;alb_dns&quot; {<br>  description = &quot;ALB public DNS&quot;<br>  value       = &quot;http://${aws_lb.web_alb.dns_name}&quot;<br>}</pre><p>Let’s also output our VPC and subnet IDs.</p><pre>#--outputs.tf--<br><br>output &quot;vpc_id&quot; {<br>  description = &quot;VPC ID&quot;<br>  value       = aws_vpc.vpc.id<br>}<br><br>output &quot;public_subnets&quot; {<br>  description = &quot;Public subnets&quot;<br>  value       = [for subnet in aws_subnet.public_subnets : subnet.id]<br>}<br><br>output &quot;private_subnets&quot; {<br>  description = &quot;Private subnets&quot;<br>  value       = [for subnet in aws_subnet.private_subnets : subnet.id]<br>}</pre><p>If we run a terraform apply we should see all of our outputs in the CLI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQQpHR3-Qd51EpvRv9M7Bw.png" /></figure><p>Click the link or paste the address into a browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QIEogSc1_JSzQm4P0XRogw.png" /></figure><h3><strong>Success</strong>!</h3><p>Our Shoppr application now has the proper infrastructure to scale up to handle high traffic volume AND to scale back down to save costs in times of low volume.</p><p>Because our infrastructure is configured in Terraform, maintaining and updating the application is now a breeze!</p><p><a href="https://github.com/aalokt89/terraform_auto_scaling_group/blob/main/gateways.tf">Here is the GitHub repo again.</a></p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><p>Feel free to connect with me on <a href="https://www.linkedin.com/in/aalok-trivedi">LinkedIn</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d92b43979cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configuring a Terraform remote backend with S3 and DynamoDB]]></title>
            <link>https://medium.com/@aaloktrivedi/configuring-a-terraform-remote-backend-with-s3-and-dynamodb-ebcefa8432ea?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebcefa8432ea</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 00:10:11 GMT</pubDate>
            <atom:updated>2023-04-11T20:08:30.279Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TNRtnqGPbVR6mamoGGQT7w.png" /></figure><h3>Intro</h3><p>By default, Terraform controls the state locally by storing the terraform.tfstate file in our working directory. Because the state file stores all of the environment information, such as IP addresses, credentials, keys, etc…, we should avoid uploading this file to GitHub or any CI/CD tool for anyone to access; however, not sharing the most current .tfstate can cause issues when collaborating with team members. When working on the same Terraform files, team members need the most current <em>state</em> to avoid resource conflicts.</p><p>Using a <strong>remote backend</strong> allows Terraform to control the .tfstate remotely and store secrets and credentials. Terraform has its own remote backend platform called <a href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/aws-remote">Terraform cloud</a>, but we can also create one within AWS through an S3 bucket and DynamoDB table.</p><p>Let’s create a Terraform remote backend for our e-commerce app, <strong>Shoppr!</strong></p><h4>What we’ll build</h4><ol><li>An S3 bucket to store the terraform.tfstate file.</li><li>A DynamoDB table to create a locking mechanism</li><li>Configure Terraform to use the above resources as the backend state controller.</li></ol><h4>Prerequisites</h4><ol><li>An AWS account with IAM user access.</li><li>Terraform installed and foundational knowledge of Terraform basics.</li><li>Access to a command line tool.</li><li>An IDE, such as VS Code or Cloud9.</li></ol><h3>🚀 🚀 🚀 Let’s get started!</h3><h3>Create the S3 bucket</h3><p>In the AWS Management Console (or through the AWS CLI), create a new bucket and give it a unique name. I’m naming mine shoppr-tfstate-bucket.</p><ul><li><strong>Region: </strong>US-East-1</li><li><strong>Block all public access: </strong>True</li><li><strong>Enable bucket versioning:</strong> True</li><li><strong>Enable default encryption:</strong> True</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCUwUcOOg06WApw4XLzJvw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_X6w31zu76YxPK2prjZ04Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xHdSBwls28angDaxeEjYg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69DSbZ1t7YpazIVXdlZkbA.png" /></figure><h4>Add a new bucket policy</h4><p>We need to create a bucket policy to <strong>allow</strong> Terraform GetObject, Putbject, and ListBucket actions.</p><p>Additionally, we also have to make sure the bucket cannot be deleted, so we need to <strong>deny</strong> DeleteBucket actions.</p><p>The path we’ll use to store the key is /global/s3/terraform.tfstate.</p><p>Navigate to <em>Permissions &gt; Bucket policy &gt; Edit </em>and create a new policy with the ‘<em>Policy generator.’</em></p><p>Statement 1:</p><ul><li><strong>Effect:</strong> Allow</li><li><strong>Principal:</strong> authorized user/group/account ARNs</li><li><strong>Service: </strong>S3</li><li><strong>Actions:</strong> DeleteBucket</li><li><strong>ARN:</strong> The bucket ARN, bucket key path</li></ul><p>Statement 2 :</p><ul><li><strong>Effect:</strong> Deny</li><li><strong>Principal:</strong> authorized user/group/account ARNs</li><li><strong>Service: </strong>S3</li><li><strong>Actions:</strong> DeleteBucket</li><li><strong>ARN:</strong> The bucket ARN, bucket key path</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIMbin2RbLgFKEt1Tu5yLg.png" /></figure><p>Add the statements and then ‘<em>Generate Policy.</em>’ We should get something like this:</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Id&quot;: &quot;Policy1680820022117&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;Stmt1680819929502&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::&lt;account&gt;:user/USER&quot;<br>            },<br>            &quot;Action&quot;: [<br>                &quot;s3:DeleteObject&quot;,<br>                &quot;s3:GetObject&quot;,<br>                &quot;s3:ListBucket&quot;,<br>                &quot;s3:PutObject&quot;<br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:s3:::S3-BUCKET-NAME&quot;,<br>                &quot;arn:aws:s3:::S3-BUCKET-NAME/global/s3/terraform.tfstate&quot;<br>            ]<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;Stmt1680819959950&quot;,<br>            &quot;Effect&quot;: &quot;Deny&quot;,<br>            &quot;Principal&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: &quot;s3:DeleteBucket&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::S3-BUCKET-NAME&quot;<br>        }<br>    ]<br>}</pre><p>Copy and paste the JSON into the Policy block and save the changes.</p><h3>Create the DynamoDB table</h3><p>To avoid resource conflicts, we can use a DynamoDB table as a locking mechanism, so only one user can make changes to the state file at any given time. I will name my table shoppr-tfstate-locking.</p><ul><li><strong>Partition key:</strong> “LockID” (<em>It needs to be this exact name</em>)</li><li><strong>Read/Write capacity settings: </strong>On-demand</li><li><strong>Deletion protection:</strong> True</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fsdWem7qimERZEEb849_xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3sPU-Rm3ZkIzyfgI2ES8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5nywvY-RvMLXDGF8mSCtMw.png" /></figure><h3>Declare the backend</h3><p>Now that we have our S3 bucket and DynamoDB table set up, we need to point Terraform to the backend resources.</p><p>Let’s create a new file named <em>providers.tf </em>and create a <strong>terraform configuration block.</strong></p><pre>terraform {<br>  backend &quot;s3&quot; {<br>    bucket         = &quot;shoppr-remote-backend&quot;<br>    key            = &quot;global/s3/terraform.tfstate&quot;<br>    region         = &quot;us-east-1&quot;<br>    dynamodb_table = &quot;shoppr-tfstate-locking&quot;<br>    encrypt        = true<br>  }<br>}</pre><p>We’ll declare “s3” as the backend source and define the bucket, key(the location to store the <em>terraform.tfstate</em> file within the bucket), region, dynamodb_table, and ecrypt option.</p><p>If we then run the terraform init command, we’ll get confirmation that the backend is configured and ready to go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGRJps_KE0F_14iToKWU-g.png" /></figure><p>Now, let’s run terraform apply to initiate the state and test the backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-THVlKJv7br5ekmwEI-1g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7R2AgE43aROwarbknRW7aw.png" /><figcaption>S3 bucket</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pB7fH318vS2cOmCOVhiuFg.png" /><figcaption>DynamoDB table</figcaption></figure><h3>Success!</h3><p>We’ve successfully configured a remote backend for our Terraform application infrastructure! We’re ready to start creating our resources and collaborating with our team.</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebcefa8432ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating with infrastructure as code: Deploying a Jenkins server with Terraform]]></title>
            <link>https://medium.com/@aaloktrivedi/automating-with-infrastructure-as-code-deploying-a-jenkins-server-with-terraform-e32f7cc79fe5?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e32f7cc79fe5</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Sun, 02 Apr 2023 19:58:27 GMT</pubDate>
            <atom:updated>2023-04-05T19:30:31.774Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhBmdDD-scJLUYFpGmVl_w.png" /></figure><h3>intro</h3><p>Hey, remember when we had to manually create a VPC, subnets, route tables, route table associations, security groups, and EC2 instances, one by one?… Then we had to manually delete them all, one by one?…</p><p>…Yeah, we don&#39;t have to do that anymore.</p><h4>Why Infrastructure as Code &amp; Terraform?</h4><p>While there are many Infrastructure as Code (IaC) tools out there, such as AWS’ CloudFormation and Azure’s ARM. Terraform is cloud-agnostic, so we can configure deployments to any cloud platform.</p><p>IaC allows us to remove manual steps and prevent human error while providing capabilities to scale quickly and collaborate.</p><h4>Scenario</h4><p>Your company, <em>Brainiac</em>, wants to start migrating its application to AWS and use Terraform to maintain and automate the infrastructure. They want to start using Jenkins as their CI/CD tool, so they need to launch a Jenkins server to begin the process.</p><h4>What we’ll be building</h4><ol><li>A VPC, two public and private subnets, an internet gateway, and route table.</li><li>An EC2 instance that will run Jenkins.</li><li>An S3 bucket for Jenkins to store artifacts.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJgn7IUt-krBifqdDZwKWQ.png" /></figure><blockquote><strong>Disclaimer:</strong> Although we’re starting up a Jenkins server, this a not a ‘how-to’ for Jenkins. This is simply a way to use Terraform to deploy resources; one of which is an EC2 instance with Jenkins installed.</blockquote><h4>Prerequisites</h4><ol><li>An AWS account with IAM user access.</li><li>Foundational knowledge of Linux systems and commands.</li><li>Foundational knowledge of AWS networks (VPCs, subnets, security groups), EC2 instances, IAM, and S3.</li><li>AWS CLI installed and configured.</li><li>Access to a command line tool.</li><li>An IDE, such as VS Code or Cloud9.</li></ol><h4>GitHub repo</h4><p>Here is a link to my <a href="https://github.com/aalokt89/Terraform_Jenkins">GitHub repo</a> for this project.</p><h3>Let’s get started!</h3><h3>Install Terraform</h3><p>If you haven&#39;t already done so, install Terraform however you see fit for your OS/environment. You can find detailed instructions in the <a href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli">Terraform documentation</a>.</p><h4>File setup</h4><p>I will be using VS Code as my IDE. If you want to make your life easier, install the official Hashicorp Terraform extension. This will help with autocompletion and syntax highlighting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OSUwlPZ1GVtw2fTDM5Vp6Q.png" /></figure><p>In our working directory, let’s create three files, named <em>providers.tf, main.tf, and variables.tf. </em>Make sure each file has the ‘.tf’ extension, so the system knows these are Terraform files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*H9DUO7ZKjQCY7e-j1VaiGg.png" /></figure><h3>Step 1: Establish our providers and credentials</h3><p>Alright, we’re ready to start terraforming! For Terraform to access the correct resources and data, we need to establish a <strong>provider</strong> — in our case, AWS.</p><p>Providers are a logical abstraction of an upstream API. They are essentially a method that allows Terraform to understand API interactions and expose resources. We can see a full list of official providers in the <a href="https://registry.terraform.io/browse/providers">Terraform Registry</a>.</p><p>If we go to the AWS page in the registry and ‘<em>Use Provider,</em>’<em> </em>it will give us the configuration code to define it as a provider.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Qg78VeBMWsLFRlinT1GJA.png" /></figure><p>We’ll paste this code into the <em>providers.tf </em>file and define the region we want to deploy our resources (I will be using “us-east-1”).</p><pre>terraform {<br>  required_providers {<br>    aws = {<br>      source  = &quot;hashicorp/aws&quot;<br>      version = &quot;4.60.0&quot;<br>    }<br>  }<br>}<br><br>provider &quot;aws&quot; {<br>  region = &quot;us-east-1&quot;<br>}</pre><h4>Using variables</h4><p>One of the biggest benefits of Terraform is its flexibility and modularity. It’s best practice to use variables whenever possible and prevent hardcoding to maximize the reusability of the configurations.</p><p>We’ll store all of our variables in the <em>variables.tf</em> file we created earlier. To create a variable, we need a <strong>variable block.</strong></p><pre>variable &quot;var_name&quot; {<br>  type        = string<br>  description = &quot;description that describes the variable (optional)&quot;<br>  default     = &quot;var_value&quot;<br>}</pre><p>Let’s create one for the region for our AWS provider.</p><pre>variable &quot;aws_region&quot; {<br>  type    = string<br>  default = &quot;us-east-1&quot;<br>}</pre><p>Now, let’s call that variable back in our provider (in the <em>providers.tf </em>file).</p><pre>provider &quot;aws&quot; {<br>  region = var.aws_region<br>}</pre><p>Terraform automatically knows to look for the var in the <em>variables.tf</em> file. We’ll go back and forth between the <em>variables.tf</em> and the <em>main.tf </em>files as we continue with the tasks.</p><blockquote><strong>Formatting tip:</strong> You can run terraform fmt in the terminal, and it will automatically format your files so the keys and values are nicely aligned. Make sure you’re in the working directory. If you’re using the Terraform VS Code extension, it should auto-format whenever you save.</blockquote><h4>Export AWS credentials</h4><p>To access AWS resources, we need to export our credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) as environment variables. In the terminal export your keys.</p><pre>#Linux/Mac/Bash<br>export AWS_ACCESS_KEY_ID=&quot;&lt;YOUR ACCESS KEY&gt;&quot;<br>export AWS_SECRET_ACCESS_KEY=&quot;&lt;YOUR SECRET KEY&gt;&quot;<br><br>#Windows PowerShell<br>$Env:AWS_ACCESS_KEY_ID=&quot;&lt;YOUR ACCESS KEY&gt;&quot;<br>$Env:AWS_SECRET_ACCESS_KEY=&quot;&lt;YOUR SECRET KEY&gt;&quot;<br><br>#Default Windows<br>setx AWS_ACCESS_KEY_ID &lt;YOUR ACCESS KEY&gt;<br>setx AWS_SECRET_ACCESS_KEY &lt;YOUR SECRET KEY&gt;</pre><h4>Initialize Terraform</h4><p>We’re ready to init Terraform. The init is a core command that prepares the working directory for Terraform and initializes the providers. Each time a provider is added/edited or the main working directory is changed, we need to re-initialize Terraform.</p><p>In the terminal, under the working directory, run the init command.</p><pre>terraform init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ay5Nc5Nhvrl0Bb4DacQghw.png" /></figure><p>Perfect! We’re ready to build our infrastructure!</p><p>Let’s also apply our Terraform code. The apply is another core command that will deploy any resources we’ve defined. We don’t have any resources, but we’ll run it anyway for good practice. Run the command and enter yes to confirm</p><pre>terraform apply</pre><h4>Store state and credentials in Terraform Cloud (optional)</h4><p>Once we apply our code, notice Terraform creates a <strong><em>terraform.tfstate</em></strong> file. This is a very important file that keeps track of each ‘<em>apply’</em> state. Every time we run the apply command, the state is updated and all of the environment information, such as resources, ids, IP addresses, credentials, etc… are stored. Can we see how this can become a security risk?</p><p>We’re building this locally, so it’s not a major issue for now, but if we’re using GitHub or a CI/CD pipeline, we don’t want this file out in the open, so make sure it’s included in the .gitignore file (GitHub has a Terraform template available).</p><p>Another way to keep this information safe is to use a <strong>remote backend</strong>, such as <strong>Terraform Cloud</strong>. Terraform Cloud controls the state remotely and can even store credentials, such as our access keys.</p><p>This step isn’t necessary to complete the rest of the tasks, but it’s highly recommended (and free!). For a full walkthrough, follow the tutorial on their <a href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/aws-remote">official site</a>.</p><h3>Step 2: Create the network infrastructure</h3><p>We’re ready to create our network infrastructure that will house our Jenkins server.</p><p>Our network will consist of:</p><ol><li>A VPC.</li><li>Two (2) public and two (2) private subnets.</li><li>An internet gateway.</li><li>A public route table.</li></ol><h4>Availability zones</h4><p>First, in the <em>main.tf </em>file, we need to retrieve the availability zones in our region (us-east-1) using a <strong>data block</strong>. We’ll use this data when we create the subnets.</p><pre># Retrieve the list of AZs in the current AWS region<br>data &quot;aws_availability_zones&quot; &quot;available&quot; {}<br>data &quot;aws_region&quot; &quot;current&quot; {}</pre><h4>VPC</h4><p>First, let’s declare some variables we’ll use for the VPC (<em>variables.tf</em>).</p><pre># naming vars<br>#---------------------------------------<br>variable &quot;app_name&quot; {<br>  type        = string<br>  description = &quot;app name prefix for naming&quot;<br>  default     = &quot;brainiac&quot;<br>}<br><br># vpc vars<br>#----------------------------------------<br>variable &quot;vpc_cidr&quot; {<br>  type        = string<br>  description = &quot;VPC cidr block&quot;<br>  default     = &quot;10.0.0.0/16&quot;<br>}<br><br>variable &quot;enable_dns_hostnames&quot; {<br>  type        = bool<br>  description = &quot;enable dns hostnames&quot;<br>  default     = true<br>}<br><br># common cidrs<br>#----------------------------------------<br>variable &quot;all_traffic&quot; {<br>  type        = string<br>  description = &quot;all traffic&quot;<br>  default     = &quot;0.0.0.0/0&quot;<br>}</pre><p>The first variable stores the application name “<em>brainiac</em>,” so we can use it to consistently and dynamically name our resources.</p><p>The second stores the CIDR block we’ll use for the VPC. We can also store common CIDRs, so we don&#39;t have to type it all out repeatedly.</p><p>Now, in the <em>main.tf </em>file (all of our <strong><em>resources</em></strong> will be in this file), we’ll create a <strong>resource block</strong> for the VPC and assign the variables to their appropriate keys.</p><pre># Define the VPC<br>#----------------------------------------------------<br>resource &quot;aws_vpc&quot; &quot;vpc&quot; {<br>  cidr_block           = var.vpc_cidr<br>  enable_dns_hostnames = var.enable_dns_hostnames<br><br>  tags = {<br>    Name        = &quot;${var.app_name}_vpc&quot;<br>    Environment = var.environment<br>    Terraform   = &quot;true&quot;<br>  }<br>}</pre><p>Let’s break this down a bit</p><p>resource &quot;aws_vpc&quot; &quot;vpc&quot;:<br>We are creating a new VPC resource from the AWS provider → &quot;aws_vpc&quot;. Then we give it a name → &quot;vpc&quot;. This can be anything, but keep it logical. It’s important to know that this ‘name’ is only for the Terraform object, NOT the actual VPC resource name.</p><p>Inside the resource block, we define the cidr_block and assign it the variable we created.</p><p>Under tags, we’ll give it a Name using our app_name variable (this will ultimately give the VPC the name of <em>brainiac_vpc</em>) and some other tags. Whatever is undefined will end up using the default settings.</p><h4>Subnets</h4><p>We’re creating two public and two private subnets, so let’s declare some more variables.</p><pre># private subnet vars<br>#----------------------------------------<br>variable &quot;private_subnets&quot; {<br>  default = {<br>    &quot;private_subnet_1&quot; = 0<br>    &quot;private_subnet_2&quot; = 1<br>  }<br>}<br><br># public subnet vars<br>#----------------------------------------<br>variable &quot;public_subnets&quot; {<br>  default = {<br>    &quot;public_subnet_1&quot; = 0<br>    &quot;public_subnet_2&quot; = 1<br>  }<br>}<br><br>variable &quot;auto_ipv4&quot; {<br>  type        = bool<br>  description = &quot;enable auto-assign ipv4&quot;<br>  default     = true<br>}</pre><p>For both subnets, we’re creating a list with multiple values. Essentially, we’re defining the names of our subnets and giving each an integer value. It seems strange, but it will make sense in the next step.</p><p>Now to create the subnet resources (do you see a pattern?).</p><pre># deploy the public subnets<br>resource &quot;aws_subnet&quot; &quot;public_subnets&quot; {<br>  vpc_id            = aws_vpc.vpc.id<br>  for_each          = var.public_subnets<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value + 100)<br>  availability_zone = tolist(data.aws_availability_zones.available.names)[each.value]<br><br>  map_public_ip_on_launch = var.auto_ipv4<br><br>  tags = {<br>    Name      = &quot;${var.app_name}_${each.key + 1}&quot;<br>    Environment = var.environment<br>    Terraform = &quot;true&quot;<br>  }<br>}</pre><p>vpc_id = aws_vpc.vpc.id:<br>We need to attach the subnets to the VPC, but we don’t have the VPC ID… <br>No worries! We can still reference other resources and get the ‘planned’ VPC ID from the resource we created. When Terraform deploys the VPC resource, it will get the ID and use it for the subnet resources.</p><p>for_each = var.public_subnets:<br>If you’re familiar with any programming language, you might be familiar with iterators or for-loops. For-loops allow us to iterate over a list of values programmatically. We can do the same with Terraform using the for_each key and assigning it our public_subnets to establish what we’re going to iterate over.</p><p>cidr_block = cidrsubnet(var.vpc_cidr, 8, each.value + 100):<br>Terraform has helper functions to make our lives easier. One of which, is the cidrsubnet() function. This function calculates and generates a subnet address within a given IP network address prefix.</p><ul><li>We give it the initial CIDR block for our VPC 10.0.0.0/16</li><li>Then the number of ‘newbits’: 8 = 10.0.0.0/16 → 10.0.0.0/24</li><li>Finally, the number we want to add to the initial address prefix. Here, we’re iterating over the var.public_subnets to get each key <strong>value </strong>(0 and 1) add 100.</li><li>&quot;public_subnet_1&quot;=0 → +100 → 10.0.100.0/24<br>&quot;public_subnet_2&quot;=1 → +100 → 10.0.101.0/24</li></ul><p>For more details on the cidersubnet() function, <a href="https://developer.hashicorp.com/terraform/language/functions/cidrsubnet">visit the official documentation</a>.</p><p>availability_zone = tolist(data.aws_availability_zones.available.names)[each.value]:<br>We can map each public subnet to its own AZ from the data block we created earlier:</p><ul><li>The tolist() helper function can be used to convert the AZ data we retrieved into a list. We can then iterate over each of the values in var.public_subnets (0 and 1) to map each AZ in the list to a subnet.</li><li>tolist(data.aws_availability_zones.available.names)[0] = us-east-1a</li><li>tolist(data.aws_availability_zones.available.names)[1] = us-east-1b</li></ul><p>We’ll do the same for the private subnets</p><pre># deploy subnets<br>#----------------------------------------------------<br># deploy the private subnets<br>resource &quot;aws_subnet&quot; &quot;private_subnets&quot; {<br>  for_each          = var.private_subnets<br>  vpc_id            = aws_vpc.vpc.id<br>  cidr_block        = cidrsubnet(var.vpc_cidr, 8, each.value + 1)<br>  availability_zone = tolist(data.aws_availability_zones.available.names)[each.value]<br><br>  tags = {<br>    Name      = &quot;${var.app_name}_${each.key + 1}&quot;<br>    Environment = var.environment<br>    Terraform = &quot;true&quot;<br>  }<br>}</pre><h4>Plan &amp; apply</h4><p>Phew! That was a lot to take in, so let’s take a break from resource creation and deploy what we have so far.</p><p>If we run terraform plan, Terraform will essentially create a ‘dry-run’ of our deployment and present the resources about to be created (without actually creating them). This is a great way to double-check our work, check for errors, and validate our deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h83BfxWUbJC1VuFbEc7I2Q.png" /><figcaption>VPC</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHBdGKyOzQMxPs3hF-OC9Q.png" /><figcaption>Public subnets</figcaption></figure><p>Everything looks great! Let’s deploy our VPC and subnets with the terraform apply command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPhY6T16jA81KE6wXrqDbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDjuSuXyiMxYQlp9nRIjHA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G5rkv2PaABW_VUXheQB0IQ.png" /></figure><p>Starting to see the power of Terraform? Alright, let’s plow through and create our internet gateway and public route table.</p><h4>Internet gateway</h4><p>We’ll need an internet gateway for our public subnets, so let’s create a new resource.</p><pre># Create Internet Gateway<br>#----------------------------------------------------<br>resource &quot;aws_internet_gateway&quot; &quot;internet_gateway&quot; {<br>  vpc_id = aws_vpc.vpc.id<br>  tags = {<br>    Name = &quot;${var.app_name}_igw&quot;<br>    Environment = var.environment<br>    Terraform = &quot;true&quot;<br>  }<br>}</pre><p>Again, we’ll reference the VPC resource to get the ID and use the var.app_name variable.</p><h4>Route table</h4><p>Instead of creating a new route table, we can use the table automatically created by our VPC — known as the aws_default_route_table — and add a route to direct all traffic from the public subnets to the internet gateway.</p><pre>#Edit default route table for public subnets<br>#----------------------------------------------------<br>resource &quot;aws_default_route_table&quot; &quot;public_route_table&quot; {<br>  default_route_table_id = aws_vpc.vpc.default_route_table_id<br><br>  route {<br>    cidr_block = var.all_traffic<br>    gateway_id = aws_internet_gateway.internet_gateway.id<br>  }<br>  tags = {<br>    Name      = &quot;${var.app_name}_public_rt&quot;<br>    Environment = var.environment<br>    Terraform = &quot;true&quot;<br>  }<br>}</pre><h4>Route table associations</h4><p>Lastly, we need to associate the public subnets with the public route table.</p><pre>#Create route table associations<br>resource &quot;aws_route_table_association&quot; &quot;public&quot; {<br>  depends_on     = [aws_subnet.public_subnets]<br>  route_table_id = aws_default_route_table.public_route_table.id<br>  for_each       = aws_subnet.public_subnets<br>  subnet_id      = each.value.id<br>}</pre><p>Here, we can use the depends_on key so Terraform knows to create the subnets before it creates the association.</p><p>Again, we can iterate over all of the subnets, so we don’t have to individually create association resources for each subnet.</p><h4>Plan &amp; apply</h4><p>Let’s apply these resources by running terraform apply again. Notice how Terraform will detect changes/additions and only apply those changes, rather than redeploying every resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0BtYSt73NqLI1KIngzelg.png" /></figure><p>Fantastic! We’ve deployed the networking base for our Jenkins infrastructure.</p><h3>Step 4: Create the Jenkins server</h3><p>Now that our networking base has been deployed, let’s create the EC2 instance that will run as the Jenkins server.</p><p>Our Jenkins server will need:</p><ol><li>A security group.</li><li>An Amazon EC2 instance.</li><li>A script to install Jenkins.</li></ol><h4>Security group</h4><p>Before we create our instance, let’s set up the security group. For inbound rules (ingress), we’ll need SSH access from “our IP address” and “all traffic” on port 8080, which is the default port used by Jenkins. We’ll allow all outbound (egress) traffic.</p><p>Let’s create a variable for our IP Address.*</p><pre># security group vars<br>#----------------------------------------<br>variable &quot;ssh_location&quot; {<br>  type        = string<br>  description = &quot;My IP address&quot;<br>  default     = &quot;YOUR_PUBLIC_IPV4_ADDRESS/32&quot;<br>}</pre><blockquote><strong>Disclaimer:</strong> This is for demo purposes only. In an actual production environment, we should not be importing our local IP address like this, as this is not secure and always changing. Normally, there would be a CIDR range of approoved addresses to SSH from. Make sure you remove your IP address when uploading your files to an open repo like Github.</blockquote><p>And now for the security group resource.</p><pre># # deploy security groups<br>#----------------------------------------------------<br>resource &quot;aws_security_group&quot; &quot;jenkins_sg&quot; {<br>  name        = &quot;${var.jenkins_server_name}_sg&quot;<br>  description = &quot;Allow ssh and traffic on port 8080&quot;<br>  vpc_id      = aws_vpc.vpc.id<br><br>  # ssh<br>  ingress {<br>    description = &quot;ssh from IP&quot;<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.ssh_location]<br>  }<br>  # jenkins port 8080<br>  ingress {<br>    description = &quot;jenkins default port&quot;<br>    from_port   = 8080<br>    to_port     = 8080<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [var.all_traffic]<br>  }<br>  # allow all outbound traffic<br>  egress {<br>    from_port        = 0<br>    to_port          = 0<br>    protocol         = &quot;-1&quot;<br>    cidr_blocks      = [var.all_traffic]<br>    ipv6_cidr_blocks = [&quot;::/0&quot;]<br>  }<br><br>  tags = {<br>    Name      = &quot;${var.app_name}_${var.jenkins_server_name}_sg&quot;<br>    Environment = var.environment<br>    Terraform = &quot;true&quot;<br>  }<br>}</pre><h4>EC2 instance</h4><p>Let’s declare some variables for the instance that will store the name, AMI, type, key pair, and user data.</p><pre># ec2 vars<br>#----------------------------------------<br>variable &quot;jenkins_server_name&quot; {<br>  type    = string<br>  default = &quot;jenkins_server&quot;<br>}<br>variable &quot;jenkins_server_ami&quot; {<br>  type        = string<br>  description = &quot;Instance AMI: Amazon Linux 2&quot;<br>  default     = &quot;ami-04581fbf744a7d11f&quot;<br>}<br>variable &quot;jenkins_server_type&quot; {<br>  type    = string<br>  default = &quot;t2.micro&quot;<br>}<br><br>variable &quot;key_pair&quot; {<br>  type        = string<br>  description = &quot;ec2 key pair&quot;<br>  default     = &quot;YOUR_KEY_PAIR&quot;<br>}<br><br>variable &quot;user_data_file&quot; {<br>  type        = string<br>  description = &quot;user data file name&quot;<br>  default     = &quot;install_jenkins.sh&quot;<br>}</pre><p>Before we create the EC2 resource, we need to bootstrap the instance to install and launch Jenkins. We’ll do this by creating a shell script and inserting it as the instance’s user_data.</p><p>Create a new file named install_jenkins.sh and insert this script:</p><pre>#!/bin/bash<br>#Update all packages<br>sudo yum update -y <br><br>#Get the latest stable version of jenkins and import the key<br>sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo<br>sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key<br><br>#Install java first. Then Jenkins<br>sudo amazon-linux-extras install java-openjdk11 -y &amp;&amp; sudo yum install jenkins -y<br>sudo systemctl daemon-reload<br><br>#Enable and start Jenkins<br>sudo systemctl enable jenkins<br>sudo systemctl start jenkins</pre><blockquote><strong>NOTE</strong>: I’ve had some trouble getting the Jenkins key to import properly, so if the Jenkins doesn’t install properly, try this URL instead: https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key</blockquote><p>Now, let’s create the EC2 resource &quot;aws_instance” and name it &quot;jenkins_server&quot;. We’ll place the instance in public_subnet_1 and point to our script file for the user_data.</p><pre># deploy ec2 instance<br>#----------------------------------------------------<br>resource &quot;aws_instance&quot; &quot;jenkins_server&quot; {<br>  ami                    = var.jenkins_server_ami<br>  instance_type          = var.jenkins_server_type<br>  subnet_id              = aws_subnet.public_subnets[&quot;public_subnet_1&quot;].id<br>  vpc_security_group_ids = [aws_security_group.jenkins_sg.id]<br>  key_name               = var.key_pair<br>  user_data              = file(&quot;${path.module}/${var.user_data_file}&quot;)<br><br>  tags = {<br>    Name        = &quot;${var.app_name}_${var.jenkins_server_name}&quot;<br>    Environment = var.environment<br>    Terraform   = true<br>  }<br>}</pre><h4>Plan &amp; deploy</h4><p>Let&#39;s deploy these two resources using terraform apply and make sure we can reach Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLJWH7H3gej7cb4tZsurxA.png" /></figure><p>In a web browser, head to the instance&#39;s public IP address and port 8080 in INSTANCE_PUBLIC_IP:8080.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVjKDC_wmHXVDuSuOJG8KQ.png" /></figure><p>Success! If you can’t reach Jenkins, you might need to give it a minute or two to fully initialize.</p><p>From here, we can SSH into the server, get the admin password, and continue with the setup process.</p><h3>Step 5: Create the S3 bucket</h3><p>Our Jenkins server is set up, so let’s create an S3 bucket so Jenkins can store artifacts.</p><p>Let’s declare some variables for the bucket name and settings to make our bucket and objects private.</p><pre># S3 vars<br>#----------------------------------------<br>variable &quot;s3_name&quot; {<br>  type    = string<br>  default = &quot;jenkins-artifacts&quot;<br>}<br><br># S3 private<br>variable &quot;block_public_acls&quot; {<br>  type    = bool<br>  default = true<br>}<br>variable &quot;block_public_policy&quot; {<br>  type    = bool<br>  default = true<br>}<br>variable &quot;ignore_public_acls&quot; {<br>  type    = bool<br>  default = true<br>}<br>variable &quot;restrict_public_buckets&quot; {<br>  type    = bool<br>  default = true<br>}</pre><blockquote><strong>Tip: </strong>Save yourself 30 minutes of head-smashing and review the <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html">S3 bucket name rules</a> so you don’t go insane from errors due to AWS’s inconsistent naming conventions…</blockquote><h4>Randomize bucket name</h4><p>Since S3 buckets require a unique global namespace, we can either hope and pray the name we give is unique… OR we can make use of Terraform’s random provider and use the random_id resource to create a randomized alphanumeric suffix.</p><p>In the <em>main.tf</em> file, create a new resource.</p><pre># deploy S3 bucket<br>#----------------------------------------------------<br># random alphanum<br>resource &quot;random_id&quot; &quot;randomize&quot; {<br>  byte_length = 8<br>}</pre><p>Although we’re not explicitly declaring the new randomprovider, we still need to re-initialize Terraform. Rerun terraform init so Terraform knows to grab the provider.</p><p>Now, we can create a new S3 bucket resource and give it a unique name.</p><pre>resource &quot;aws_s3_bucket&quot; &quot;jenkins_artifacts_s3&quot; {<br>  bucket = &quot;${var.app_name}-${var.s3_name}-${random_id.randomize.hex}&quot;<br><br>  tags = {<br>    Name        = &quot;${var.app_name}_${var.s3_name}_s3&quot;<br>    Environment = var.environment<br>    Terraform   = true<br>  }<br>}</pre><p>We also need to create make sure we set the bucket and objects to “private.”</p><pre># set s3 to private<br>resource &quot;aws_s3_bucket_public_access_block&quot; &quot;s3_private_access&quot; {<br>  bucket                  = aws_s3_bucket.jenkins_artifacts_s3.id<br>  block_public_acls       = var.block_public_acls<br>  block_public_policy     = var.block_public_policy<br>  ignore_public_acls      = var.ignore_public_acls<br>  restrict_public_buckets = var.restrict_public_buckets<br>}</pre><h4>Plan &amp; apply</h4><p>You know the deal… terraform apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AfgAwZRHioLGwEnctaz1vw.png" /></figure><h3>Step 6: Create permissions</h3><p>Home stretch! The last step is to give our Jenkins server permissions to access the S3 bucket.</p><h4>What we’ll need</h4><ol><li>An <strong>IAM role</strong> for the Jenkins server.</li><li>An <strong>IAM policy</strong> that allows the Jenkins server to Putbject, GetObject, and ListBucket in our S3 bucket.</li><li>An <strong>IAM policy attachment</strong> to attach the policy to the role.</li><li>An <strong>IAM instance profile</strong> to attach the role to the Jenkins server.</li></ol><h4>IAM role</h4><p>We need to create a role for the EC2 instance to assume, so let’s create an aws_iam_role resource.</p><p>First, a variable for the role name.</p><pre># IAM vars<br>#----------------------------------------<br># role name<br>variable &quot;iam_role_name&quot; {<br>  type        = string<br>  description = &quot;IAM role name&quot;<br>  default     = &quot;jenikins_s3_role&quot;<br>}</pre><p>And then the resource. We can use the jsonencode() helper function to format the output into JSON.</p><pre># create IAM role<br>#----------------------------------------------------<br>resource &quot;aws_iam_role&quot; &quot;jenkins_s3_role&quot; {<br>  name = var.iam_role_name<br>  assume_role_policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Effect    = &quot;Allow&quot;<br>        Principal = { Service = &quot;ec2.amazonaws.com&quot; }<br>        Action    = &quot;sts:AssumeRole&quot;<br>      }<br>    ]<br>  })<br>}</pre><h4>IAM policy</h4><p>The IAM policy should allow the Jenkins server to perform Putbject, GetObject, and ListBucket actions on our s3 bucket.</p><p>Let’s create variables that will store the policy name, actions, and resource type.</p><pre># policy name<br>variable &quot;iam_policy_name&quot; {<br>  type        = string<br>  description = &quot;IAM policy name&quot;<br>  default     = &quot;jenikins_s3_policy&quot;<br>}<br><br># policy resource actions<br>variable &quot;iam_actions&quot; {<br>  type        = list(string)<br>  description = &quot;actions allowed by Jenkins server&quot;<br>  default = [<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:ListBucket&quot;<br>  ]<br>}<br><br># resource type/prefix<br>variable &quot;iam_resource_type&quot; {<br>  type        = string<br>  description = &quot;IAM policy resource type&quot;<br>  default     = &quot;arn:aws:s3:::&quot;<br>}</pre><p>And now, the resource. Notice how we can combine the var.iam_resource_type variable with the output of the S3 bucket name.</p><pre># create IAM policy<br>#----------------------------------------------------<br>resource &quot;aws_iam_policy&quot; &quot;jenkins_s3_policy&quot; {<br>  name = var.iam_policy_name<br>  policy = jsonencode({<br>    Version = &quot;2012-10-17&quot;<br>    Statement = [<br>      {<br>        Effect = &quot;Allow&quot;<br>        Action = var.iam_actions<br>        Resource = [<br>          &quot;${var.iam_resource_type}${aws_s3_bucket.jenkins_artifacts_s3.bucket}&quot;<br>          &quot;${var.iam_resource_type}${aws_s3_bucket.jenkins_artifacts_s3.bucket}/*&quot;<br>        ]<br>      }<br>    ]<br><br>  })<br>}</pre><h4>IAM policy attachment</h4><p>Next, we need to attach the <strong>policy</strong> to the <strong>role</strong>.</p><pre># create IAM policy attachment<br>#----------------------------------------------------<br>resource &quot;aws_iam_role_policy_attachment&quot; &quot;jenkins_s3_policy_attachment&quot; {<br>  role       = aws_iam_role.jenkins_s3_role.name<br>  policy_arn = aws_iam_policy.jenkins_s3_policy.arn<br>}</pre><h4>IAM instance profile</h4><p>Finally, we need to create an instance profile so we can attach the <strong>role </strong>to the Jenkins server.</p><p>Variable:</p><pre># instance profile name<br>variable &quot;iam_instance_profile_name&quot; {<br>  type        = string<br>  description = &quot;instance profile name&quot;<br>  default     = &quot;jenkins_s3_instance_profile&quot;<br>}</pre><p>Resource:</p><pre># create IAM instance profile<br>#----------------------------------------------------<br>resource &quot;aws_iam_instance_profile&quot; &quot;jenkins_s3_instance_profile&quot; {<br>  name = var.iam_instance_profile_name<br>  role = aws_iam_role.jenkins_s3_role.name<br>}</pre><p>Remember to go back and add the iam_instance_profile to the Jenkins EC2 instance resource.</p><pre># deploy ec2 instance<br>#----------------------------------------------------<br>resource &quot;aws_instance&quot; &quot;jenkins_server&quot; {<br>  ami                    = var.jenkins_server_ami<br>  instance_type          = var.jenkins_server_type<br>  subnet_id              = aws_subnet.public_subnets[&quot;public_subnet_1&quot;].id<br>  vpc_security_group_ids = [aws_security_group.jenkins_sg.id]<br>  key_name               = var.key_pair<br>  user_data              = file(&quot;${path.module}/${var.user_data_file}&quot;)<br>  iam_instance_profile   = aws_iam_instance_profile.jenkins_s3_instance_profile.name<br><br>  tags = {<br>    Name        = &quot;${var.app_name}_${var.jenkins_server_name}&quot;<br>    Environment = var.environment<br>    Terraform   = true<br>  }<br>}</pre><h3>Step 7: Output resource information</h3><p>Last step! Once Terraform applies our resources, we can tell it to output certain several helpful resource information, such as IDs, resource ARNS, public IPs, etc…</p><p>Let’s <strong>output </strong>some info so we can verify the Jenkins server has proper access to the S3 bucket. In a new file, named outputs.tf, we’ll create <strong>output blocks </strong>for the Jenkins server public IPv4 address, S3 bucket name, and S3 bucket ARN.</p><pre>output &quot;jenkins_server_public_ip&quot; {<br>  description = &quot;publiic ip address for Jenkins server &quot;<br>  value       = aws_instance.jenkins_server.public_ip<br>}<br><br>output &quot;s3_bucket_name&quot; {<br>  description = &quot;Jenkins S3 bucket name&quot;<br>  value       = aws_s3_bucket.jenkins_artifacts_s3.bucket<br>}<br><br>output &quot;s3_bucket_arn&quot; {<br>  description = &quot;Jenkins S3 bucket arn&quot;<br>  value       = aws_s3_bucket.jenkins_artifacts_s3.arn<br>}</pre><p>When the deployment is applied, Terraform will output the information for us to use. Run another terraform init and terraform apply.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yehR-4_BDQDfVJUA7A4ktg.png" /></figure><h3>Step 8: Verify Jenkins server → S3 access</h3><p>Now that we have the correct IAM role and outputs, we need to verify that the Jenkins server can upload to the S3 bucket. We’ll test this through the AWS CLI.</p><p>First, let’s SSH into the Jenkins server and create a test file touch file.txt, and then use the AWS CLI to upload the file.</p><pre>aws s3api put-object --bucket YOUR-BUCKET-NAME --key file.txt --body file.txt</pre><p>The --key refers to the key name we’ll give the object. The --body refers to the local file path to the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vxvCx0CliiEfvEiWOXLC_A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfrAXbzfzLOStpX4g8IKHw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WMxQcI460HzPku2_gPNhZQ.png" /></figure><h3>Success!</h3><p>Congrats! We’ve successfully deployed an infrastructure for a Jenkins server and connected it to an S3 bucket using Terraform! It may have seemed like a lot of work upfront, but now, we can reuse this as a template for future deployments.</p><h4>Destroy resources</h4><p>As quickly as it was to create the resources, it’s just as quick to tear everything down. We can run terraform destroy and confirm the resources to be destroyed.</p><blockquote>You may have to manually delete the S3 bucket because it has an object. Alternatively, you can set force_destroy = true on the S3 bucket resource.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o62kzPgW-LVTvoaUWMz2Lg.png" /></figure><p>Here is the <a href="https://github.com/aalokt89/Terraform_Jenkins">Github repo again</a>.</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e32f7cc79fe5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Controlling Kubernetes deployments with ConfigMaps and ingress]]></title>
            <link>https://medium.com/@aaloktrivedi/controlling-kubernetes-deployments-with-configmaps-and-ingress-3cd03be082e5?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd03be082e5</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 04:00:57 GMT</pubDate>
            <atom:updated>2023-03-20T04:00:57.462Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ycoPYsisaK6u1S3_028wOA.png" /></figure><h3>Intro</h3><p>In my last <a href="https://medium.com/@aaloktrivedi/using-kubernetes-to-deploy-a-3-tier-containerized-application-infrastructure-9fbbbbc85ff6">article</a>, we created used Kubernetes to deploy a 3-tier application architecture. If you need a review of Kubernetes clusters, services, and deployments using YAML files, I advise a brief review.</p><p>One of the main philosophies of containerization is portability and being able to move your application development from environment to environment without a bunch of overhead and breakage. It should just simply work on any machine/environment!</p><p>One way to compartmentalize is through ConfigMaps. With ConfigMaps, we can store environment-specific or user-specific data that is then referenced by pods at the time of deployment. We can easily update or update this info without having to redeploy the application.</p><h4>Scenario</h4><p>We have an application with several microservices and the company wants to add two more (Campaigns, and Orders). We are tasked to set up the frontend architecture for these services.</p><p>The company wants an easy way to update and edit the landing pages for both services without any heavy lifting or redeployment of the application. They also require a routing solution so each service is easily reached by a named path (/campaigns, /orders).</p><h4>What we’re building</h4><ol><li>Two (2) <strong>configMaps</strong> that will hold HTML data.</li><li>Two (2) <strong>Apache services and deployments</strong> for our application’s <strong>Campaigns</strong> and <strong>Orders</strong> microservices that display HTML from the configMap.</li><li>An <strong>ingress</strong> rule that will provide load balancing and URL routing to both deployments.</li></ol><h4>Prerequisites</h4><ol><li>Foundational knowledge of Docker and Kubernetes clusters, services, and deployments.</li><li>Basic knowledge of Linux file systems and commands.</li><li>Minikube and Hyperkit installed.</li><li>Access to a command line terminal.</li><li>An IDE, such as VS Code.</li></ol><h3>Let’s get started!</h3><h3>Create a local environment</h3><h4>Start a cluster</h4><p>I will be using <a href="https://minikube.sigs.k8s.io/docs/start/">Minikube</a> as my local Kubernetes environment and <a href="https://minikube.sigs.k8s.io/docs/drivers/hyperkit/">Hyperkit</a> as the VM driver.</p><p>Let’s start by creating our cluster with 1 master/control plane node and 2 worker nodes.</p><pre>minikube start --driver hyperkit -n 3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Apae102Dc-6tox9d8evHkw.png" /></figure><p>We’ll also add a taint to the master node to ensure no pods get assigned to it.</p><pre>kubectl taint nodes minikube master=true:NoSchedule</pre><h4>Enable ingress</h4><p>We’ll need ingress capabilities for later use, so let’s enable it from Minikube’s addons.</p><pre>minikube addons enable ingress</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IAgeFKFBgoeIkHRpuEyO7w.png" /></figure><h3>Create the ConfigMap</h3><h4>What is a configMap?</h4><p>A ConfigMap is an API object used to store non-confidential data in key-value pairs. We can use ConfigMaps to store information such as environment variables, command-line arguments, or configuration files that are then used by pods and containers. It’s an efficient way to decouple environment-specific information and keep our applications light and portable.</p><p>We’re going to demonstrate how ConfigMaps can also be used as mounted volumes to deliver files to containers by replacing the index.html for our two deployments.</p><p>The major benefit of using a ConfigMap in this manner is it decouples configurations and content from the container and image. We can update the file and the ConfigMap as needed without having to rebuild container images or redeploy the application.</p><h4>ConfigMap from file</h4><p>In our working directory on our local machine, create new directories for <em>Campaigns </em>and <em>Orders. </em>Inside each directory, create a new index.html file. Input whatever HTML you want (I just did a simple &lt;h1&gt; Your Campaigns &lt;/h1&gt;/&lt;h1&gt; Your Orders &lt;/h1&gt;).</p><p>In the terminal, we’ll create two ConfigMaps from both index.html files.</p><pre>kubectl create configmap campaigns-config --from-file &lt;your_file_path&gt;</pre><pre>kubectl create configmap orders-config --from-file &lt;your_file_path&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pb7PKWmDt2pF1TAvPTklEQ.png" /></figure><p>Let’s get the output of one of the config manifests by running kubectl get configmaps campaigns-config -o yaml.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6fxf8gKHpHLTQk6gmrilwg.png" /></figure><p>As we can see, the ConfigMap imports the HTML file and creates index.html as a <em>key</em> and the content as the <em>value. </em>We’ll use this data to mount the file to containers.</p><h3>Create the services &amp; deployments</h3><p>Our cluster and configs are set up, so let’s create deployments for our two microservices: Campaigns and Orders.</p><h4>Campaigns</h4><p>Create a new YAML file called campaigns-deployment.yml. We’ll start by creating a simple NodePort service on port 8000.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: campaigns-service<br>spec:<br>  type: NodePort<br>  ports:<br>    - protocol: TCP<br>      port: 8000<br>      targetPort: 80<br>  selector:<br>    app: campaigns<br>---</pre><p>In the same file, underneath the service, define the deployment.</p><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: campaigns<br>spec:<br>  selector:<br>    matchLabels:<br>      app: campaigns<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        app: campaigns<br>    spec:<br>      containers:<br>        - name: campaigns<br>          image: httpd:alpine3.17<br>          ports:<br>            - containerPort: 8000<br>          volumeMounts:<br>            - name: campaigns-index #refers to volume name<br>              mountPath: /usr/local/apache2/htdocs/index.html #path to where to we mount the configmap<br>              subPath: index.html #file to put data into. Make sure it&#39;s the same as the &#39;key&#39; and &#39;path&#39;.<br>      volumes: #volume that maps to the configmap<br>        - name: campaigns-index<br>          configMap: <br>            name: campaigns-config<br>            items:<br>              - key: index.html<br>                path: index.html</pre><p>We have to create a new volume at the template level that maps to the campaigns-config. Under items, we define the keys we want the volume to get data from.</p><p>Once we define the volume, we need to assign it to the container as a volumeMount. In the mountPath and subPath, we define the path where we want the index.html page to be mounted. In this case, the path Apache creates for the HTML files.</p><h4>Orders</h4><p>We’ll do the same for the <em>Orders </em>service and deployment and map it to the orders-config.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: orders-service<br>spec:<br>  type: NodePort<br>  ports:<br>    - protocol: TCP<br>      port: 8080<br>      targetPort: 80<br>  selector:<br>    app: orders<br>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: orders<br>spec:<br>  selector:<br>    matchLabels:<br>      app: orders<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        app: orders<br>    spec:<br>      containers:<br>        - name: orders<br>          image: httpd:alpine3.17<br>          ports:<br>            - containerPort: 8000<br>          volumeMounts:<br>            - name: orders-index #refers to volume name<br>              mountPath: /usr/local/apache2/htdocs/index.html #path to where to we mount the configmap<br>              subPath: index.html #file to put data into. Make sure it&#39;s the same as the &#39;key&#39; and &#39;path&#39;.<br>      volumes: #volume that maps to the configmap<br>        - name: orders-index<br>          configMap: <br>            name: orders-config<br>            items:<br>              - key: index.html<br>                path: index.html</pre><p>Apply both deployments to our cluster:</p><pre>kubectl apply -f campaigns_deployment.yml</pre><pre>kubectl apply -f orders_deployment.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ievx0kthZQGk6CoBEhZffg.png" /></figure><h4>Test deployments</h4><p>Let’s check if our deployments and services are running correctly by running a kubectl get all command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyHYP-L0X6bmZminqop5nA.png" /></figure><p>Great! Let’s also make sure our ConfigMap mounts worked by going to our service IPs in the browser. We can easily get each service’s url by running minikube service &lt;service-name&gt; --url.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0i-dT7WG1K0alMhoSrVi8w.png" /></figure><p>Perfect! Our ConfigMaps work! We’re ready to create our ingress for easy routing.</p><h3>Create the ingress rules</h3><p>Now that our deployments are in place, we want to be able to access them from the internet and provide named routing. We can do this through an <strong>ingress rule.</strong></p><h4>What is ingress?</h4><p>Ingress provides exposure to HTTP/HTTPS routes from outside the cluster to services within the cluster. We can use an ingress for load balancing, SSL/TLS termination, and giving services externally-reachable URLs. With a fanout configuration like below, we’re able to route traffic to our services from a single IP address and create named paths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M-Sk5lLh8Cz7muVqGcszNA.png" /></figure><p>For an ingress to work, the cluster must have an ingress controller. Some of the popular controllers are by <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller#readme">AWS</a>, <a href="https://git.k8s.io/ingress-gce/README.md#readme">GCE</a>, and <a href="https://github.com/kubernetes/ingress-nginx/blob/main/README.md#readme">nginx</a>. Since our cluster is local, we’ll use the nginx controller.</p><p>For more info on ingress, visit the <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">official Kubernetes documentation</a>.</p><h4>Create the ingress manifest</h4><p>In a new YAML file called ingress_rules.yml, we’ll define our ingress rules.</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: page-ingress<br>  annotations:<br>    nginx.ingress.kubernetes.io/rewrite-target: /$1 #nginx ingress controller<br>spec:<br>  rules:<br>     - http:<br>        paths:<br>          - path: /campaigns #the external path we want for our service<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: campaigns-service #the service we should point to<br>                port:<br>                  number: 8000 <br>          - path: /orders<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: orders-service<br>                port:<br>                  number: 8080</pre><p>Let’s break this down a bit.</p><p>Under metadata, we have a section called annotations. This is where we refer to the nginx ingress controller.</p><p>Ingress rules and three main aspects:</p><ul><li>host(optional): we can define a configured host or DNS for our traffic to route to (myapp.com, campaigns.myapp.com, orders.myapp.com, etc…). Since we don’t have a host, we can leave this out and just use the IP address we’re given.</li><li>paths: Before, with just a simple NodePort, we had to know the service’s port to access the page. Now, we can define the paths we want each service to route to for easy access.</li><li>backend: Defines the cluster services and ports the ingress should route traffic to. In our case, the campaigns-service and orders-service.</li></ul><h4>Apply the ingress</h4><p>Now we can apply the ingress to our cluster.</p><pre>kubectl apply -f ingress_rules.yml</pre><p>If we kubectle get ingresses, we’ll see the external IP attached to the ingress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AybQrBuViXJ61vGJjoVLaQ.png" /><figcaption>If the IP address doesn’t show up, give it a minute and run the `get` command again.</figcaption></figure><p>Let’s see if the ingress worked! In your browser, go to both services using the ingress paths: ip_address/campaigns | ip_address/orders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpxQB1mxo-dz26HRALs39w.png" /></figure><p>Perfect! We now have efficient routing for both of our app services.</p><h3>Success!</h3><p>Congrats! We’ve successfully created an application architecture that uses ConfigMap data to dynamically display content and ingress rules to provide load balancing and path-based routing to our application services.</p><p>The beauty of Kubernetes and containerization is being able to easily decouple/isolate different application services and tasks for easy management. Using ConfigMaps and Ingress is just one of the countless methods to make our applications more efficient, manageable, and portable.</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd03be082e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Kubernetes to deploy a 3-tier containerized application infrastructure]]></title>
            <link>https://medium.com/@aaloktrivedi/using-kubernetes-to-deploy-a-3-tier-containerized-application-infrastructure-9fbbbbc85ff6?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fbbbbc85ff6</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Sun, 12 Mar 2023 22:33:21 GMT</pubDate>
            <atom:updated>2023-03-12T22:33:21.802Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fXtUA0ZN8SRSSeiwgL6gfA.png" /></figure><h3>Intro</h3><p>In my last article, we discussed containerized application orchestration using Docker Swarm. Now we’re going to deploy that same architecture with another leading orchestration platform: Kubernetes.</p><p>If you need a refresher, review the <a href="https://medium.com/@aaloktrivedi/automating-scaling-application-infrastructure-with-docker-swarm-and-aws-197e7cb326f9">previous article</a>.</p><h4>What we’re building</h4><ol><li>A Kubernetes environment with one (1) master/control plane node and three (3) worker nodes, using minikube.</li><li>A Kubernetes application deployment that will deploy a 3-tier application architecture (Web/Apache, Node.js, and Postgres DB).</li></ol><blockquote><strong>Note:</strong> We won’t have any application source code in this demo. We’ll focus more on the architecture and infrastructure of the application.</blockquote><h4>Prerequisites</h4><ul><li>Review of the last <a href="https://medium.com/@aaloktrivedi/automating-scaling-application-infrastructure-with-docker-swarm-and-aws-197e7cb326f9">article</a> about Docker Swarm.</li><li>Familiarity with Linux systems and commands.</li><li>Familiarity with Docker containers and orchestration methods, such as Docker Compose and Swarm.</li><li>Docker installed on your system.</li><li>Access to a command line tool.</li></ul><h3>What is Kubernetes?</h3><p>Similar to Docker Swarm, Kubernetes is one of the leading container orchestration tools, but for larger and more complex application workloads. It offers much more flexibility, scalability, reliability, and specificity on how to deploy workloads.</p><p>So…Kubernetes is better than Docker Swarm, right?</p><p>Not necessarily! Although Docker Swarm is more lightweight, it’s often considered to be easier to use and scale. Like anything in the tech world, it really depends on your use case and what you&#39;re trying to build.</p><h4>Some key differences</h4><p>Kubernetes adds a layer of abstraction on top of Docker, so it has its own set of APIs to create networks, containers, and volumes. Like docker, we can access the API engine through the kubectl CLI.</p><p><strong>Pods:</strong><br>Pods are the basic unit of deployment. They are pretty much like services/containers in Swarm, but with a layer of abstraction on top (you’ll hear a lot of this). You can have multiple pods within a node and multiple containers within a pod. It can be a little confusing, but just know, when we mention ‘pods,’ think ‘containers.’</p><p><strong>Services:</strong><br>In Swarm, a service referred to container/task definitions and deployments. In Kubernetes, a <em>service</em> refers to a network and stable endpoint/address for pods to either communicate internally or with outside services. There are three main types of services: <strong>ClusterIP</strong>, <strong>NodePort</strong>, and <strong>LoadBalancer</strong>.</p><h3>Let’s get started!</h3><h3>Install minikube and Hyperkit</h3><p>Before start building our containerized application, we need to create a local dev environment. <a href="https://minikube.sigs.k8s.io/docs/"><strong>minikube</strong></a><strong> </strong>is a great tool that allows us to implement multi-node Kubernetes clusters locally on our machines and comes with <strong>kubectl</strong> already installed.</p><p><a href="https://minikube.sigs.k8s.io/docs/start/">Install minikube</a> on your system however you see fit, but for me (macOS), the <a href="https://brew.sh/">Homebrew</a> package manager was the most straightforward method and also allows us to install the virtual machine environment we’ll need.</p><pre>brew install minikube</pre><p>To run a multi-node cluster locally, minikube needs a virtual machine driver, so we need to install a VM tool, such as Hyperkit, VirtualBox, or Parallels. I’ll be using Hyperkit, but feel free to choose your own.</p><pre>brew install hyperkit</pre><h4>Cluster setup</h4><p>If you remember from Docker Swarm, our container workloads run on ‘nodes,’ which can be physical or virtual machines/servers that provide high availability and fault tolerance. In Swarm, nodes are assigned roles of <em>manager</em> or <em>worker. </em>The principle is the same in Kubernetes, but nodes are split into roles of the <em>control plane — </em>which contain all the <em>master</em> nodes — and <em>workers.</em> Let’s set up our cluster in minikube:</p><ol><li>One (1) master/control plane node.</li><li>Three (3) worker nodes.</li></ol><p>By default, minikube will assign our local machine as a master/control plane node and all others as workers.</p><pre>minikube start --driver hyperkit --nodes 4</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AItExUAtVvGB-NWmcDB2g.png" /></figure><p>Now, we can use kubectl commands to get our nodes</p><pre>kubectl get nodes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etBI7MLH68QQn2RheYSvyw.png" /></figure><h4>Restrict pod assignments</h4><p>We only want pods to be scheduled on worker nodes, so we need to add special instructions to the control plane in the form of a <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/"><strong>taint</strong></a>. A Taint allows a node to repel a set of pods by telling the scheduler not to schedule any pods unless they have a matching <strong>toleration.</strong></p><pre>kubectl taint nodes node_name key1=value1:NoSchedule </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZMNWbhY4RJ6zm1MaZONjyw.png" /></figure><h3>Deploy the frontend web tier</h3><p>Similar to Docker Compose, an application’s entire infrastructure can be deployed using a single YAML file; however, Kubernetes YAML files can be a little more involved and detailed. Although a single YAML file can be used, the deployment configurations might be easier to manage if we break them out into a few organized files.</p><p>We’ll start with the frontend web tier and deploy our pods that contain our Apache web servers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V7LWxshqY8AloNrOtMWDpA.png" /><figcaption>I’ve organized my ‘kube_app’ application directory into ‘frontend,’ ‘backend,’ and ‘database’ tiers.</figcaption></figure><h4>Frontend web Service</h4><p>We’ve been working with Docker Swarm so, you might be familiar with the term, ‘<em>services</em>.’ In Docker, a service is the image or image type run on a container, such as Apache, NGINX, Node, Postgres, etc… It is essentially a description and role of a container/task that will be deployed in a cluster.</p><p>A Kubernetes <em>service</em> refers to a network and stable endpoint/address for pods to either communicate internally or with outside services. There are three main types of services: <strong>ClusterIP</strong>, <strong>NodePort</strong>, and <strong>LoadBalancer</strong>.</p><p>A <strong>ClusterIP</strong> creates virtual IPs to allow pods to communicate with each other. A default ClusterIP is given to our cluster, so the control plane and worker nodes can communicate with each other.</p><pre>kubectl get services</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3qMnbCGcqfEAyOV7F1EJPA.png" /></figure><p>A <strong>NodePort</strong> allows the control plane to expose and allocate specific ports for outside access. This is the type of service we’ll need to access our website on port 80.</p><p>Create a new YAML file named frontend-kube-app.yml.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: web-httpd-service<br>spec:<br>  type: NodePort<br>  ports:<br>    - protocol: TCP<br>      port: 80<br>      targetPort: 80<br>  selector:<br>    app: web-httpd<br>---</pre><p>Each Kubernetes <em>object, </em>or <em>kind,</em> refers to a certain API group, so it’s important to know which apiVersion we should point to. An easy way to find this is to look at the api-resources and find the version of the object.</p><pre>kubectl api-resources</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezaK9YAo2EDTDbOM9H8Qew.png" /></figure><p>Under metadata, we can give our service a name (required).</p><p>We specify the type as Nodeport and expose port 80.</p><p>The concept of selector and labels are very important. The service uses the selector to allow traffic to pods with matching labels.</p><p>The --- indicates a break and the end of the object.</p><blockquote><strong>Tip:</strong> Instead of trying to go back and forth between documentation, you can get details about each section of an object from the explain command. For example, you can use dot-notation to drill down on specific areas, such askubectl explain service.spec and it will give you an explanation of each option that falls under that category.</blockquote><h4>Frontend web deployment</h4><p>Now we’re ready to create a <em>Deployment! </em>Deployments are more akin to Docker <em>services</em>. It’s the set of instructions and specifications for the pods including the image, number of replicas, ports, restart policies, etc…</p><p>In the same file, under the service object:</p><pre>---<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: web-httpd-deployment<br>spec:<br>  selector:<br>    matchLabels:<br>      app: web-httpd<br>  replicas: 10<br>  template:<br>    metadata:<br>      labels:<br>        app: web-httpd<br>    spec:<br>      containers:<br>        - name: web-httpd<br>          image: httpd:2.4.55<br>          ports:<br>            - containerPort: 80</pre><p>We’re keeping this pretty simple, but you can see how more detailed specs can make this a little more complicated.</p><p>As we can see, pod deployments are under the kind: <em>Deployment</em>, with its own API Group. We should also notice the selector section where we&#39;ve given it a matchlabel of the same name as the <em>web-httpd-service </em>selector, so they know to connect to each other.</p><p>Under template, we can provide the container information, such as the name, image, and port.</p><h4>Deploy the pods and service</h4><p>We’re ready to deploy our first pods! This is the easy part. All we have to do is apply the configuration and point to our YAML file.</p><pre>kubectl apply -f ./frontend/frontend-kube-app.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q1hayh8l-G5-eIv0lJrQdg.png" /></figure><p>We can run kubectl get all -o wide to get all the service and pod details for our cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*leWaTAEJaLouarX7SI_B4Q.png" /></figure><p>Alright! All ten of our pods are up and running, and we can see that none of them have been distributed on the master/control plane node.</p><p>We also see our NodePort service and its port mapping. Let’s see if we can access the web server by going to any of our node&#39;s IP addresses with the NodePort port (the number on the right). We can get a node&#39;s IP by running kubectl get nodes -o wide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THqyGdoBNy8Myf6RoRXraA.png" /></figure><p>It works! We’ve just deployed our first Kubernetes pods to our cluster!</p><h3>Deploy the backend application tier</h3><p>Now that we have our first tier, out of the way, it’s pretty much rinse-and-repeat — but with varying specs to fit your needs.</p><p>Since we don’t have any application source code or need to connect anything at this point, we’ll just create a simple Deployment for the app backend. In the backend directory, create a new YAML file called backend-kube-app.yml.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nodejs-app-deployment<br>spec:<br>  selector:<br>    matchLabels:<br>      app: nodejs-app<br>  replicas: 4<br>  template:<br>    metadata:<br>      labels:<br>        app: nodejs-app<br>    spec:<br>      containers:<br>        - name: nodejs-app<br>          image: node:19-alpine3.16<br>          command: [&quot;sleep&quot;, &quot;100000&quot;]</pre><p>We’ll enter a command of sleep for testing purposes to keep the container open and running.</p><p>Run the apply command again and get the results.</p><pre>kubectl apply -f ./backend/backend-kube-app.yml</pre><pre>kubectl get all -o wide</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E4bPKnK-GkGp4xa3huIkzg.png" /></figure><p>Great! 4/4 pods we provisioned are running successfully.</p><h3>Deploy the backend database tier</h3><p>We’re ready for the final phase, and deploy the database (Postgres) for our app. The database can be a little more involved so let’s break down what we’ll need:</p><ol><li>A ConfigMap to store secrets such as usernames and passwords.</li><li>A PersistantVolume and PersistantVolumeClaim to define and allocate storage for the A database.</li><li>A Service to connect the database</li><li>A Deployment to define and deploy the database pod to our cluster.</li></ol><h4>Create a ConfigMap</h4><p>It’s always best practice to separate data from code. A good way to do this in development is through a ConfigMap. Kubernetes has its own API group for this purpose.</p><p>In the database directory, create a new YAML file called postgres-config.yml</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: postgres-config<br>  labels:<br>    app: postgres-db<br>data:<br>  POSTGRES_DB: postgresdb<br>  POSTGRES_USER: admin<br>  POSTGRES_PASSWORD: mypass</pre><p>Here, we’ll store environment variables such as the DB name, user, and password. Again, take special notice of the labels, since this is what our Deployment will look for to connect to the config.</p><p>Apply the ConfigMap to the cluster.</p><pre>kubectl apply -f ./database/postgres-config.yml</pre><h4>Create a PersistantVolume &amp; PersistantVolumeClaim</h4><p>Since pods are ephemeral — meaning once they’re gone, so is all the data they carried — we’ll need persistent storage in place for our database. Here, we’ll define the capacity, access, and the hostPath of the volume.</p><p>Once we create the volume, we need a PersistantVolumeClaim, which defines how the users request and consume PV resources.</p><p>Create a new YAML file named postgres-pvc-pv.yml.</p><pre>kind: PersistentVolume<br>apiVersion: v1<br>metadata:<br>  name: postgres-pv-volume  # Sets PV&#39;s name<br>  labels:<br>    type: local  # Sets PV&#39;s type to local<br>    app: postgres-db<br>spec:<br>  storageClassName: manual<br>  capacity:<br>    storage: 5Gi # Sets PV Volume<br>  accessModes:<br>    - ReadWriteMany<br>  hostPath:<br>    path: &quot;/mnt/data&quot;<br>---<br>kind: PersistentVolumeClaim<br>apiVersion: v1<br>metadata:<br>  name: postgres-pv-claim  # Sets name of PVC<br>  labels:<br>    app: postgres-db<br>spec:<br>  storageClassName: manual<br>  accessModes:<br>    - ReadWriteMany  # Sets read and write access<br>  resources:<br>    requests:<br>      storage: 5Gi  # Sets volume size</pre><p>Apply the PV and PVC to the cluster.</p><pre>kubectl apply -f ./database/postgres-pvc-pv.yml</pre><h4>Create the Service</h4><p>For now, we’ll just create a simple NodePort service and expose port 5432.</p><p>Create a new YAML file named database-kube-app.yml.</p><pre>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: postgres # Sets service name<br>  labels:<br>    app: postgres-db # Labels and Selectors<br>spec:<br>  type: NodePort # Sets service type<br>  ports:<br>    - port: 5432 # Sets port to run the postgres application<br>  selector:<br>    app: postgres-db<br>---</pre><h4>Create the Deployment</h4><p>Finally, we’ll create the deployment options for our Postgres database (in the same file).</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: postgres-db-deployment  # Sets Deployment name<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: postgres-db<br>  template:<br>    metadata:<br>      labels:<br>        app: postgres-db<br>    spec:<br>      containers:<br>        - name: postgres-db<br>          image: postgres:10.1 # Sets Image<br>          imagePullPolicy: &quot;IfNotPresent&quot;<br>          ports:<br>            - containerPort: 5432  # Exposes container port<br>          envFrom:<br>            - configMapRef: # Maps env variable from ConfigMap<br>                name: postgres-config<br>          volumeMounts:<br>            - mountPath: /var/lib/postgresql/data<br>              name: postgres-volume<br>      volumes:<br>        - name: postgres-volume<br>          persistentVolumeClaim:<br>            claimName: postgres-pv-claim # Maps claim from PersistantVolumeClaim</pre><p>We can map our environment variables envFrom and volumes from our config and PVC (this is why names and labels are important!).</p><p>Apply the service and Deployment to the cluster.</p><pre>kubectl apply -f ./database/postgres-pvc-pv.yml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qj_OPtSgrSwrffBjrTC_cA.png" /></figure><h4>Test database connection</h4><p>For good measure, let’s test and see if we can connect to the database.</p><pre>kubectl exec -it [pod-name] --  psql -h localhost -U admin --password -p 5432 postgresdb</pre><p>Enter the DB password and type \l to list all the databases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*97W-mNDcgUVZ7MMetO3Bjw.png" /></figure><p>And we’re in!</p><h3>Success!</h3><p>Congrats! We’ve just successfully deployed a highly available and fault-tolerant application infrastructure with Kubernetes! The beauty of Kubernetes, and containerized applications in general, is we can package this up and import it into any environment with minimal to no changes.</p><h4>Cleanup</h4><p>Once we’re done, we can run minikube stop to stop the environment, or minikube delete to delete all resources.</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fbbbbc85ff6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating & scaling application infrastructure with Docker Swarm and AWS]]></title>
            <link>https://medium.com/@aaloktrivedi/automating-scaling-application-infrastructure-with-docker-swarm-and-aws-197e7cb326f9?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/197e7cb326f9</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[docker-swarm]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Thu, 09 Mar 2023 12:01:34 GMT</pubDate>
            <atom:updated>2023-03-09T21:46:10.038Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXuVdWczd5WLu4FQJjRS3Q.png" /></figure><h3>Intro</h3><p>In my last article, we discussed how quick and efficient it is to create flexible application environments with Docker Compose. But what if we wanted to scale our containerized application architecture, add high availability, and provide fault tolerance without the headache and bloat of management overhead? In comes <strong>Docker Swarm!</strong></p><p>Swarm is one of the leading tools in cluster orchestration &amp; management and makes scaling applications and infrastructure a breeze. By taking advantage of decentralized roles and built-in load balancing, Swarm is able to leverage high availability and provide ultra-fast deployment with minimal overhead.</p><p>Too good to be true? Well, let&#39;s build something and find out!</p><h4>What we’re building</h4><ol><li>A Docker Swarm environment with one (1) manager node and three (3) worker nodes.</li><li>A Docker Stack that will deploy a 3-tier application architecture with three (3) Docker services (Web/Apache, Node.js, and Postgres).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OuFUMqGYdyGrUTUR83QXg.png" /></figure><h4>Prerequisites</h4><ol><li>An AWS account with IAM user access.</li><li>A Docker Hub account.</li><li>Familiarity with basic Docker concepts, Docker Compose, and CLI commands.</li><li>Familiarity with Amazon EC2 instances and the AWS Management Console.</li><li>Familiarity with Linux file systems and commands</li><li>Access to a command line tool.</li><li>An IDE, such as VS Code.</li></ol><h4>Key concepts</h4><p><strong>Node</strong><br>A node is a physical or virtual machine that hosts our application and can take on different roles, such as a <strong>manager</strong> or <strong>worker. </strong>A manager interprets the services and distributes tasks to all the worker nodes. A worker runs the tasks, given by a manager node.</p><p><strong>Task<br></strong>A task is simply a Docker container and the set of instructions or commands to be run on that container.</p><h3>Let’s get started!</h3><h3>Step 1: Groundwork</h3><p>Before we dive into the nitty gritty of Docker Swarm, we need to lay some runway and set up servers/machines that will host our Swarm nodes. We’ll be using Amazon EC2 instances, but we can also use virtual and local machines as well.</p><h4><strong>What we need:</strong></h4><ol><li>One (1) EC2 instance with a node role of <strong><em>manager</em></strong>.</li><li>Three (3) EC2 instances with a node role of <strong><em>worker</em></strong>.</li></ol><h4>Security groups</h4><p>First, let’s work a little backward and create our security groups for both the manager and worker nodes.</p><p>To ensure all the nodes can communicate with each other, Docker Swarm needs access to specific ports:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOQJZ1Ud0hAB_OnOL3iVLA.png" /><figcaption>For more details, visit: <a href="https://www.bretfisher.com/docker-swarm-firewall-ports/">https://www.bretfisher.com/docker-swarm-firewall-ports/</a></figcaption></figure><p>So, let’s create two security groups: One for the manager (<strong>swarm_app_mgr_sg</strong>), and one for the workers (<strong>swarm_app_wkr_sg</strong>).</p><blockquote>Just create the groups first, and then go back to add the inbound rules for both. We need the source for each rule set to the mgr/wkr security groups, and AWS won’t let you add the source to an sg after-the-fact.</blockquote><p><strong>Manager security group inbound rules:</strong></p><ul><li>SSH | TCP | Port: 22 | Source: My IP</li><li>HTTP | TCP | Port: 80 | Source: IPv4 anywhere</li><li>HTTPS | TCP | Port: 443 | Source: IPv4 anywhere</li><li>Custom TCP | TCP | Port: 8080 | Source: IPv4 anywhere</li><li>Custom TCP | TCP | Port: 2377 | Source: swarm_app_wkr_sg</li><li>Custom TCP | TCP | Port: 7946 | Source: swarm_app_wkr_sg</li><li>Custom UDP | UDP | Port: 7946 | Source: swarm_app_wkr_sg</li><li>Custom UDP | UDP | Port: 4789 | Source: swarm_app_wkr_sg</li></ul><p><strong>Worker security group inbound rules:</strong></p><ul><li>SSH | TCP | Port: 22 | Source: My IP</li><li>HTTP | TCP | Port: 80 | Source: IPv4 anywhere</li><li>HTTPS | TCP | Port: 443 | Source: IPv4 anywhere</li><li>Custom TCP | TCP | Port: 7946 | Source: swarm_app_mgr_sg</li><li>Custom UDP | UDP | Port: 7946 | Source: swarm_app_mgr_sg</li><li>Custom UDP | UDP | Port: 4789 | Source: swarm_app_mgr_sg</li></ul><h4>Create the manager node server</h4><p>If Docker Swarm is the orchestra, the manager nodes are the conductors and administrators. It’s the manager’s job to control/distribute services and tasks to the worker nodes. A manager can also add/remove and promote/demote nodes based on need. Think of it as the administrative control center of your application environment.</p><p>It’s generally good practice to have multiple managers for high availability, but for now, one will do.</p><p>To save ourselves some repetition, we’ll create a new EC2 <em>Launch Template </em>(<strong>swarm_node_template</strong>)<em> </em>to easily apply the same settings to each server/node we need to spin up.</p><p><strong>Launch template settings:</strong></p><ul><li><strong>AMI:</strong> Amazon Linux 2.</li><li><strong>Type: </strong>t2.micro (Free tier, but feel free to choose whatever fits your application requirements).</li><li><strong>Key pair: </strong>Choose/create a key pair.</li><li><strong>Network:</strong> We can set these when we launch our instances.</li><li><strong>Advanced Details User data:</strong> This is where we need to add a bash script to make sure Docker and Docker Compose are installed, enabled, and started when the instances are launched. Copy and paste this script into the <em>User details</em> section.</li></ul><pre>sudo yum update -y<br>sudo yum install -y docker<br><br>#install docker compose<br>sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose<br>sudo chmod +x /usr/local/bin/docker-compose<br><br>sudo systemctl enable docker<br>sudo systemctl start docker</pre><p>Done! Let’s launch an instance based on this template and name it <strong>swarm_app_mgr. </strong>Remember to attach the <strong>swarm_app_mgr_sg </strong>security group! We can use the default VPC and subnet.</p><h4>Create the worker node servers</h4><p>We can use the same launch template for the worker machines to launch three instances with the <strong>swarm_app_wkr_sg </strong>security group and name them <strong>swarm_app_wkr1, swarm_app_wkr2, </strong>and<strong> swarm_app_wkr3</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uso7yvN6NQhzvfoQopbnfQ.png" /></figure><p>Perfect! We’re ready to Swarm!</p><h3>Step 2: Create the Swarm</h3><p>It’s time to create our Swarm environment and assign each EC2 instance its node role.</p><h4>Assign the manager</h4><p>First, let’s SSH into the instance we designated as our manager node and make sure Docker and Docker Compose have been successfully installed by running docker --version and docker-compose --version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*No2VxZ-h0CmX0jW3RoKVvg.png" /></figure><blockquote>Note: Log in with your Docker Hub username and password sudo docker login</blockquote><p>To create a new swarm and assign this server as the manager, run this command:</p><pre>sudo docker swarm init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_6ZYrIwNyfqxGaa254zq-Q.png" /></figure><p>We now have a manager node and directions to create new manager/worker nodes!</p><p>Notice that the IP address is the instance’s private IP address. This is important because a manager node needs a static IP for the worker nodes to connect to. Copy that first command.</p><blockquote>In this case, Swarm automatically detected the right IP, but sometimes it might direct you to ‘advertise’ an IP. If that happens you&#39;ll need to run the init command again with an additional --advertise-addr &lt;YOUR_IP_ADDRESS&gt; option.</blockquote><h4>Assign the workers</h4><p>With the command copied, let’s open three more terminal windows and SSH into each instance we designated as the worker nodes. Paste the command we were given from the manager. We should get confirmation of each node’s addition to the swarm.</p><pre>sudo docker swarm join --token &lt;TOKEN&gt; &lt;MANAGER_IP_ADDRESS&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LNItrUWX0aWRiS3oXHcdLA.png" /></figure><p>In the Manager terminal, we can run sudo docker node ls to see all of our nodes and their statuses. (the * indicates the node we’re currently on).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rETiZFWYXKAP4csN3oWnYQ.png" /></figure><blockquote>If we try this same command in a Worker we’ll get an error message:<br>Error response from daemon: This node is not a swarm manager. Worker nodes can’t be used to view or modify cluster state. Please run this command on a manager node or promote the current node to a manager.</blockquote><blockquote>This is by design. All of our admin and orchestration should be done only from the Manager nodes.</blockquote><blockquote><strong>Optional bonus:</strong> I decided to change the hostnames for each node to make it easier to recognize what-is-what. It’s not necessary, but <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-hostname.html">here&#39;s how to do it, if you’re interested</a>.</blockquote><p>We won’t really need the other worker terminals anymore, so we can close or minimize them.</p><p>And, with that, our swarm nodes are ready for action!</p><h3>Step 3: Create the Swarm stack</h3><p>Our base environment is set, but we must now define and deploy the application architecture. Our application will be set up in three tiers:</p><ol><li><strong>Frontend/web tier:</strong> powered by Apache web servers<br>(<strong>Replicas: 10</strong>).</li><li><strong>Backend/Application tier:</strong> powered by Node.js<br>(<strong>Replicas: 4</strong>).</li><li><strong>Backend/Database tier:</strong> powered by Postgres<br>(<strong>Replicas: 1</strong>).</li></ol><blockquote>Note: We won’t have any actual application source code to deploy, but the process would be the same.</blockquote><p>The beauty of Swarm is that it’s a declarative process, meaning all we have to do is define our resources and the end result. Swarm takes care of the rest and figures out what it needs to do to get to that result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OuFUMqGYdyGrUTUR83QXg.png" /></figure><p>We define all the nodes, services, images, volumes, and deployment options; and Swarm takes care of load balancing and distributing the services/containers amongst the available nodes.</p><p>If a <strong>task</strong> (a service instance/container) fails, Swarm will automatically shut it down and create a new one to replace it to fulfill the total number of replicas we declared.</p><p>If a node fails or becomes unavailable, Swarm will redistribute the required services and containers amongst the available nodes. This is what makes Swarm so highly available and fault-tolerant!</p><h4>Docker Compose</h4><p>Of course, we can individually create services and tasks using Docker CLI commands, but dear god, that sounds tedious and time-consuming. If you’re familiar with Docker Compose, we can create a <strong>Swarm Stack </strong>to<strong> </strong>orchestrate our entire architecture with one file and one command!</p><p>Our stack will be built from a docker-compose.yml file. Everything is the same as Docker Compose, such as defining services, ports, volumes, and networks; however, a stack additionally allows us to define deployment options. We can dictate how many replicas each service needs, placement constraints, restart policies, rollback policies, and much more. For a complete list of options, visit the <a href="https://docs.docker.com/compose/compose-file/deploy/">official documentation</a>.</p><p>Enough preamble. Let’s build!</p><p>While still SSH-ed in your manager node, create a new docker-compose.yml file or clone a Github repo (this is what I will be doing), with the compose file in it.</p><p>As a reminder, this is the basic structure of the compose file. We have four services* and two bridge networks for the frontend and backend.</p><pre>version: &quot;3.8&quot;<br><br>services:<br>  web:<br>  <br>  node:<br><br>  db:<br>  <br>  visualizer:<br><br>networks:<br>  frontend:<br>    driver: bridge<br>  backend: <br>    driver: bridge <br><br>volumes:</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d75b6248425815a492c63a392926f4fd/href">https://medium.com/media/d75b6248425815a492c63a392926f4fd/href</a></iframe><blockquote>*: I’ve included a fourth service called visualizer, which is a really great dev tool visualizes the landscape and health of services across all nodes. You’ll see!</blockquote><p>Since we’re only focused on the architecture, and not the actual application code, I’ve left my Compose file fairly simple. Feel free to add to your file and make it as robust as you want for your application needs.</p><p>For <strong>deploy</strong>, replicas is the only required option. We’ve also declared placement constraints on each service so they only get distributed to worker nodes. If we didn’t have this declaration, tasks would also be distributed to the manager node.</p><p>Our Compose file is ready!</p><h3>Step 4: Deploy the Stack</h3><p>Ready to see how easy it is to deploy a Swarm Stack?</p><p>In the manager node shell, we can deploy by using the docker stack delpoy command with a -c to indicate a Compose file. Then add your compose file and the name of your stack (I named it swarm_app).</p><pre>sudo docker stack deploy -c docker-compose.yml stack_name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_78yCzsybKpDwNa_VEQF5g.png" /><figcaption>Networks and services are being created!</figcaption></figure><p>And just like that our application is deployed!… well, for development, at least. Cool, right?!</p><p>For an overview of services in our stack, we can run:</p><pre>sudo docker stack services stack_name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SJZNHCFVJ2h0FCdTTJ5jDQ.png" /><figcaption>Woo! all of our replicas are running!</figcaption></figure><p>For a more detailed view of all of our tasks/containers, we can run:</p><pre>sudo docker stack ps stack_name</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7bm5M1HJ-H23sLn5DbugQ.png" /></figure><p>In the highlighted section, we can see Swarm at work, live! It looks like some of the <strong>node.js </strong>service tasks shut down (probably because of the ‘sleep’ command), but were immediately replaced by a running task!</p><p>If we go to any of our swarm node’s public IP addresses, we should see our Apache website live.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PIDUdu6nUnR_U4yqG5L9kQ.png" /></figure><h4>Visualizer</h4><p>Now for the really cool part! Remember that surprise fourth <strong>visualizer</strong> service we deployed on our manager node and set to port 8080? Let’s go to it! manager_node_public_IP:8080.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilnYDwcoWHR9ecvHo4torQ.png" /></figure><p>Beautiful! I come from a UX/UI background, so this dashboard is super helpful in understanding what’s happening. The visualizer gives us a live overview of our containers spread throughout the worker nodes.</p><p>If we want, we can simulate a node failure. In the AWS Management Console, stop one of the worker node servers and head back to the visualizer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6r4_xG3C5tN9ghGcgb7OuA.png" /></figure><p>Because one of the nodes became unavailable, Swarm redistributed the tasks to the other available nodes while still maintaining the same number of replicas for each service!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FrmP9lWX54hJ60%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FrmP9lWX54hJ60%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FrmP9lWX54hJ60%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="245" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0904ee0e348a0601caa1360f4c5247c/href">https://medium.com/media/b0904ee0e348a0601caa1360f4c5247c/href</a></iframe><h3>Congrats!</h3><p>We’ve successfully orchestrated and deployed a highly available, fault-tolerant web application architecture with Docker Swarm!</p><h3>Thank you</h3><p>Thank you for following me on my cloud engineering journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=197e7cb326f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an AWS Boto3 Python Environment with Docker Compose]]></title>
            <link>https://medium.com/@aaloktrivedi/building-an-aws-boto3-python-environment-with-docker-compose-a9984bb6cbdc?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9984bb6cbdc</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Mon, 06 Mar 2023 12:02:43 GMT</pubDate>
            <atom:updated>2023-03-06T22:03:18.365Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*osePNN4wXQSW0jkiCRav5A.png" /></figure><h3>Intro</h3><h4>Docker Docker Docker!</h4><p>By now, we should all know Docker is the hottest new movement in the tech world. It’s undeniable the impact it’s having on bringing efficiency to CI/CD and the ease of building, sharing, and deploying applications on any scale or environment. I’m going to show you just how easy it can make your life.</p><h4>What we‘re building</h4><p>A Boto3, Python environment with isolated Dev and Prod containers. To build this, we will need:</p><ol><li>Three (3) Dockerfiles to build three (3) docker images.</li><li>Three (3) Docker containers. One (1) is designated for <strong>development</strong> and two (2) for <strong>production</strong>.</li><li>Two virtual networks for development and production. The development container should not be able to access/communicate with production.</li><li>A Docker Compose file that will orchestrate all of the above with one command.</li></ol><h4>Prerequisites</h4><ol><li>A Docker Hub account.</li><li>Familiarity with Linux file systems and commands</li><li>Basic knowledge of Docker concepts, such as containers, images, and CLI commands.</li><li>Access to a command line tool.</li><li>An IDE (VS Code/Cloud9).</li></ol><h3>Let’s get started!</h3><h3>Create a Docker Hub account &amp; install Docker</h3><h4>Create an account</h4><p>The first step is to create an account with <a href="https://docs.docker.com/engine/install/">Docker Hub</a>. Docker Hub is an open-source platform where we can manage all of our Docker applications and resources, such as images. When we start creating our containers and need to define the image/OS we want to run on, Docker will pull the images from Docker Hub. We can use reliable, official images such as Centos, Ubuntu, NGINX, Alpine, etc.. right out of the box, OR build our own on top of any of these images and upload it to the Hub for easy access and sharing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUBVzty8NhV0zfZOXqgqDQ.png" /></figure><h4>Install Docker</h4><p>Whether on Mac, Windows, or Linux, the easiest way to install Docker on our local machine is through the <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop app</a>. Just like any other application, go through the guided installation process, and log in with your Docker Hub username/password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*72If_RcHL78OjlKRbVOhjg.png" /></figure><p>Docker Desktop is a super handy tool that lets you start/stop Docker and manage all your containers, images, and volumes. The app also comes with Docker Compose already installed.</p><blockquote>Note: If you want to install Docker on a Linux server, install the Docker engine via the command line. You can find installation guides in the <a href="https://docs.docker.com/engine/install/">official Docker documentation</a>.</blockquote><h4>VS Code Extensions</h4><p>If you’re using VS Code, I highly recommend installing these extensions to help with Dockerfile/Docker Compose file recognition and syntax highlighting:</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker">Docker</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers">Dev Containers</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml">YAML</a></li></ul><p>After Docker has been installed on the host machine, make sure it&#39;s running (The logo at the bottom of the app should be green) and run the docker --version command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHU26mlQdsJ1wTRDW_npCw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXzG6sJ3Wx9jbmrUMYQsNQ.png" /></figure><p>Great! Docker is up and running!</p><h3>File Setup</h3><p>Before we get to the fun stuff, let’s make sure we have our house in order and create the proper setup. Of course, you can set up your structure however you’re used to working, but this is just how I’ve set up mine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mw8zlBUEFO478VJdp-Rug.png" /></figure><p>I’ve created a folder called boto3_env that will serve as the root host directory (This is also a cloned repo from Github). Inside are three subdirectories: <em>development, productionA, and productionB</em>.</p><p>In the <em>development</em> subdirectory, I’ve just added some AWS Boto3-related python scripts.</p><p>Don’t worry about the <em>Dockerfiles</em> and <em>docker-compose</em> files just yet. We’ll create those as we go along.</p><h4>Requirements file (optional)</h4><p>To make sure the Docker images have all the python libraries we need, we can package the dependencies in a single <em>requirements.txt</em> file. Then, we can point the Dockerfile to this file so they will be installed on the Docker image.</p><p>To create this file (make sure you have python and pip installed on your machine), cd into your root directory (in my case, <em>boto_env</em>) and run this command:</p><pre>pip freeze &gt; requirements.text</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*naQySe76rZxOJLMEqMXoSw.png" /></figure><p>Perfect! As you can see, Boto3 is included in this file.</p><h3>Create the Images</h3><p>We’re now ready to create the images! Remember, a Docker image is essentially a collection of the application’s root file systems, dependencies, binaries, and the metadata of how to run it.</p><p>We can use one of the hundreds of official images from Docker Hub, but they may not have the dependencies and extra goodies we need to run our application. The beauty of Docker is we can build upon these existing images and add files, packages, and a set of instructions to fit our application needs.</p><p>This special set of instructions comes from a <em>Dockerfile. </em>We’re going to create three separate Dockerfiles to create images for our development and production environments.</p><p>Dockerfiles have a series of commands (layers) to, essentially, tell Docker how to build the image. Let’s go over a few of the common commands:</p><ul><li>FROM: The base image (i.e. Ubuntu, Alpine, Centos, NGINX, etc…).</li><li>RUN: A series of shell commands (i.e. installing/updating packages, creating directories, etc…) that will run at build-time.</li><li>COPY/ADD: Copying files from the host directory to the image/container.</li><li>WORKDIR: Creating/changing the working directory of the image/container.</li><li>CMD: The runtime command(s) that will be executed when a container is started (i.e. starting a webserver/app).</li></ul><h4>Development environment image</h4><p>In our <em>development</em> host directory, let’s create a new file named <em>Dockerfile. </em>No extension is needed. Docker knows to look for this filename and will execute the image build.</p><p>Let’s start by defining the base image we’ll be building upon. Since we’re building a Python environment, it makes sense to use the convenient, official Python image on Docker Hub.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nXiGPcuWJswLib8Se0Zfrw.png" /></figure><pre>FROM python:3.9-alpine3.17</pre><p>I’m choosing a Python image built on Alpine, a super lightweight Linux distro (only 5 MB!), but it’s up to you which version to install. You can find all the versions in the <em>tags</em> section of the image&#39;s details page.</p><blockquote>Note: if we just use python, Docker will automatically pull the <strong>latest</strong> version, which may not be ideal. It’s best practice to define a specific version to mitigate failures and make debugging easier.</blockquote><p>Docker images are usually stripped down to keep them lightweight, so they might not come with the basic packages we’ve come to expect. We’ll RUN a few commands to run an update and install a couple of packages we need like curl, bash, git, unzip, and iputils.</p><pre>RUN apk update \<br>    &amp;&amp; apk add bash \<br>    &amp;&amp; apk add curl zip git unzip iputils</pre><p>We can separate commands with a \ and use the &amp;&amp; operator to make sure the previous command runs successfully before executing the next one (Note that the commands will differ from one image to the next depending on the base Linux distro: yum, apt-get, apk, npm, etc…).</p><p>Next, we’ll run a command to install the AWS CLI</p><pre>RUN curl &quot;https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip&quot; -o &quot;awscliv2.zip&quot; \<br>    &amp;&amp; unzip awscliv2.zip \<br>    &amp;&amp; ./aws/install -i /usr/local/aws-cli -b /usr/local/bin \<br>    &amp;&amp; rm awscliv2.zip</pre><blockquote>Technically, you can put all of these commands under a single RUN layer, but I’ve had difficulty getting it to work that way. Also, the Docker CLI gives great feedback on errors and exactly which layer they occurred in, so it can be helpful to separate them. But be careful, Adding a lot of layers can increase the image size and build time.</blockquote><p>Now, we want to create/change the image’s working directory and copy the requirements.txt from the host directory to the image.</p><p>COPY &lt;host_directory_filepath&gt; &lt;image_directory_path&gt;</p><pre>WORKDIR /home/app<br><br>COPY requirements.txt .</pre><p>In this case, the . indicates the current working directory (which is now /home/app).</p><p>We’ll create another RUN layer to use pip to install the dependencies from the requirements.txt. And then copy all of our files from our <em>development</em> folder to the image’s working directory.</p><pre>RUN pip install -r requirements.txt<br><br>COPY ./development .</pre><p>Finally, we need to create a CMD layer. This layer will run the defined command(s) at runtime when a container is created/started. This is mostly used to run processes to start an application. However, our environment doesn&#39;t really have any processes to ‘start,’ so we need to include this command to keep the container running when started. If we don&#39;t include this, the container will immediately stop after it’s started.</p><pre>CMD [&quot;tail&quot;, &quot;-f&quot;, &quot;/dev/null&quot;]</pre><p>For more info on this topic, <a href="https://devopscube.com/keep-docker-container-running/">this article is very helpful</a>.</p><p>For more info on Dockerfile, visit the <a href="https://docs.docker.com/engine/reference/builder/">official documentation</a>.</p><p>Here is the full Dockerfile:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69b741178ce92692327dd7bac7935204/href">https://medium.com/media/69b741178ce92692327dd7bac7935204/href</a></iframe><p>Alright, let’s test to see if the image works as expected. In the command line, make sure we’re in our root boto3_env directory and use a docker image build command to build the image. We’ll call it test-dev.</p><pre>docker image build -t test-dev -f ./development/Dockerfile .</pre><p>Since the Dockerfile is not in the root folder, we need to specify the file path with the -f tag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q43HEA1cYNu9YqLTQmpgrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rOZSsnGu1vy8eTv6pK_w4w.png" /></figure><p>Woo! The Docker CLI provides great progress feedback, so we clearly know what’s happening and when. As you can see, each layer becomes a ‘step’ in the build process.</p><p>Now let’s test the image by creating a container from this image and connecting to it to see the file system. We should see all of our files and dependencies properly copied over.</p><pre>docker container run -d --name dev test_dev</pre><pre>docker exec -it dev bash</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBQMMwAFOzeZ44g-uUTJTg.png" /></figure><p>Let’s go! Now we’re cookin’ with gas! The working directory is properly set as /home/app and all of the files from the host development folder are ported over.</p><p>And if we go to the container’s root, we can see the AWS CLI has been installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqL0TkFjlxip657_sn9KGA.png" /></figure><h4>Production environment images</h4><p>Now, we can create the two production environment images. The steps are largely the same, but instead of copying files over, we’ll clone a GitHub repo straight to the image.</p><p>In the <em>productionA</em> host directory, create another <em>Dockerfile.</em></p><pre>FROM python:3.9-alpine3.17<br>#update, install curl, git, zip/unzip<br><br>RUN apk update \<br>    &amp;&amp; apk add bash \<br>    &amp;&amp; apk add curl zip git unzip iputils<br><br>#install aws cli<br>RUN curl &quot;https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip&quot; -o &quot;awscliv2.zip&quot; \<br>    &amp;&amp; unzip awscliv2.zip \<br>    &amp;&amp; ./aws/install -i /usr/local/aws-cli -b /usr/local/bin \<br>    &amp;&amp; rm awscliv2.zip<br><br>WORKDIR /home/app<br><br>COPY requirements.txt .<br><br>RUN pip install -r requirements.txt \<br>    &amp;&amp; git clone https://github.com/aalokt89/LUIT_python<br><br>CMD [&quot;tail&quot;, &quot;-f&quot;, &quot;/dev/null&quot;]</pre><p>The file is nearly identical; however, instead of copying over host files, we’re going to clone a GitHub repo into the working directory (choose any repo for demo purposes).</p><p>Let’s build the image! Remember to specify the proper Dockerfile path.</p><pre>docker image build -t test_prod_a -f ./productionA/Dockerfile .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xKKXqSpDvMYOdATUFrbuZA.png" /></figure><p>And, again, we’ll create a container with this image and connect to make sure it worked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M9Xi_oJWSgFRWjNmVs6vNw.png" /></figure><p>Fantastic! Repeat this step for the final image for the productionB environment (with a different GitHub repo).</p><h3>Orchestrate and automate!</h3><p>We’ve seen how easy it is to create images with Dockerfiles, but what if we had to create multiple containers (sometimes even dozens)? Manually creating them from the CLI sounds like a nightmare. In comes Docker Compose!</p><p>Docker Compose allows us to create multi-container application environments with one command. No, seriously… ONE COMMAND.</p><p>The way we do this is by creating a docker-compose.yml file. It’s pretty much a config file — written in YAML — that lets Docker know what containers to create, with what images, networks, volumes, environment variables, etc… in a clean and organized way… and super fast</p><h4>Create the Compose file</h4><p>In your host root directory (boto3-env), create a file named docker-compose.yml. It has to have this name.</p><p>The Docker Compose file follows this basic structure:</p><pre>Version: &quot;3&quot;<br><br>services:<br>  service_name1:<br>    image: image_name<br>    ports: 3000:80<br>    volumes:<br><br>  service_name2:<br>    image: image_name<br>  <br>  service_name2:<br>    image: image_name<br><br>volumes:<br>  volume1:<br>  volume2:<br><br>networks:<br>  network1:<br>  network2:<br></pre><p>Not every attribute is required, but you want at least the service_name and image.</p><p>let’s define each of our services (containers), and set each attribute as needed. Since we’re building new images with Dockerfiles, the syntax will be a little different.</p><h4>Service 1: dev</h4><p>We’ll start with the container that will host our ‘dev’ environment.</p><pre>Version: &quot;3&quot;<br><br>services:<br>  dev:<br>    build:<br>      context: .<br>      dockerfile: ./development/Dockerfile<br>    image: boto3_dev<br>    environment:<br>      ENVIRONMENT: dev<br>    volumes:<br>      - type: bind<br>        source: ./development<br>        target: /home/app<br>    networks:<br>      - dev_net<br><br>networks:<br>  dev_net:<br>  prod_net:</pre><p>Let’s break this down a bit so we understand what’s happening here.</p><p>dev: This is the name we’ll give our container.</p><p>build: Since we’re building a new image from a Dockerfile, we need to specify the context, which sets the directory we’re starting from (we’re starting from our root host directory, so we define it as .), and the Dockerfile file path. If we were creating the container from an already existing image, then we would not need this build attribute.</p><p>image: The name we want to give the image (or the name and version of an existing image).</p><p>environment: (<em>Optional</em>) We can give containers environment variables, so we’ll give this one a ‘dev’ variable.</p><p>volumes: (<em>Optional</em>) Here, we are defining a <em>bind-mount</em> for our dev container. Although we copied everything from the host development directory when creating the image, we would need to rebuild the image every time to update the content. Bind-mounting is a great way to get persistent data at the container level. Once we’ve selected the source (host) and target destination, every time we update a file or directory on the host, it will automatically update in the container! Magic! This is great for Dev environments because you can work locally and know that your changes will be reflected in the container.</p><p>networks: (<em>Optional</em>) Docker creates virtual networks, to which our containers will be attached. If no network is defined, all the containers will be under a default bridge network. Remember, we don’t want the dev container to be able to access/communicate with the prod containers, so we need to create separate networks for both dev and prod. These networks have to be defined at the top level networks section, and then referenced at the service level. We can configure the IP, specific subnets, and much more if we want, but the DNS name is enough for now. Docker will create the rest for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lo1EKMOKdhxdG4uds8nxww.png" /></figure><h4>Service 2 &amp;3: Prod</h4><p>Now we can do the same for the two production containers, except we won’t be bind-mounting them.</p><pre>prodA:<br>  build:<br>    context: .<br>    dockerfile: ./productionA/Dockerfile<br>  image: boto3_prod_a<br>  environment:<br>    ENVIRONMENT: prod<br>  networks:<br>    - prod_net<br><br>prodB:<br>  build: <br>    context: .<br>    dockerfile: ./productionB/Dockerfile<br>  image: boto3_prod_b<br>  environment:<br>    ENVIRONMENT: prod<br>  networks:<br>    - prod_net</pre><h4>Orchestrate!</h4><p>Here is the full docker-compose yaml file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02ffb8f7118b165ff0ea73b59241572d/href">https://medium.com/media/02ffb8f7118b165ff0ea73b59241572d/href</a></iframe><p>For more info on Docker Compose, visit the <a href="https://docs.docker.com/compose/compose-file/">official documentation</a>.</p><p>To run the docker-compose.yml we can run:</p><pre>docker compose up -d</pre><p>…and hope for no errors… 🤞🏾🤞🏾🤞🏾🤞🏾🤞🏾</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Q92wdG8XyGv2wdfeoS_Hw.png" /></figure><p>Black Magic!!! We can see by the last few lines, our two networks and three containers are up and running.</p><p>Run a docker container ls to see all our running containers and let’s do a final network test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5axO-kPcOuw_Ymnmojeotw.png" /></figure><p>First, let’s make sure our Dev container can’t communicate (ping) with our Prod containers.</p><p>We’ll run a ping from the Dev container using the exec command (we can write out the entire name or just the first three numbers/letters of the ID, but you do need the full name for the destination container):</p><pre>docker exec -it 552 ping boto3_env-prodA-1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kd4PDpaqq1pxbzsCe5_xTQ.png" /></figure><p>Perfect. Now let’s ping the Prod containers to each other to make sure they CAN communicate with each other.</p><pre>docker exec -it a24 ping boto3_env-prodB-1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vYh-U5UGP7wkeijRigpZKw.png" /></figure><h3>Success!</h3><p>Congrats! We’ve successfully deployed a Boto3 Python Docker environment from a single Docker Compose file! Just imagine the power this method can have in a more complex Dev-to-Production environment. Think of how much time, money, and resources can be saved.</p><h3>Thank you</h3><p>Thank you for following me on my cloud computing journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9984bb6cbdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Lambda to send and receive messages from SQS and SNS: Part 2]]></title>
            <link>https://medium.com/@aaloktrivedi/integrating-lambda-to-send-and-receive-messages-from-sqs-and-sns-part-2-6595133dd1a3?source=rss-fda853e0723e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6595133dd1a3</guid>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Aalok Trivedi]]></dc:creator>
            <pubDate>Mon, 27 Feb 2023 22:36:18 GMT</pubDate>
            <atom:updated>2023-02-27T22:36:18.707Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ft_FlLNizo6id1ZZKw-XoA.png" /></figure><h3>Intro</h3><p>In Part 1, we built a process that sends an API request to get the current datetime or random UUID and sends that message to an SQS queue for further processing. Now it’s time to process the messages from the queue, send an SNS notification, and store the data to a DynamoDB table.</p><p>If you need a refresher, here is the link to <a href="https://medium.com/@aaloktrivedi/integrating-lambda-to-send-and-receive-messages-from-sqs-and-sns-part-1-5b359f0f1678">Part 1</a>.</p><h4>What we’ll build</h4><ol><li>An SNS topic with an email subscription.</li><li>A DynamboDB table.</li><li>A Lambda function that will retrieve messages from the SQS queue, send them to the SNS topic, and write the data to the DynamoDB table.</li></ol><h3>Let&#39;s get started!</h3><h3>Groundwork</h3><p>Similar to Part 1, we have to start with a little groundwork and make sure everything is in place before we start connecting resources together.</p><h4>Create the SNS topic</h4><p>Let’s create the SNS topic that will receive our message from the Lambda and send an email. In a new python file, named createSNS.py, import Boto3, create an SNS resource, and define a new function.</p><pre>import boto3<br><br>sns = boto3.resource(&#39;sns&#39;, region_name=&#39;us-east-1&#39;)<br><br>def createSNS(topicName):<br>    try:<br>        topic = sns.create_topic(Name=topicName)<br>        print(topic.arn)<br><br>    except Exception as e:<br>        print(e)<br><br>#call the function<br>createSNS(&#39;datetime-uuid-topic&#39;)</pre><p>We can create a new topic with the .create_topic() method and pass in a Name. We will need the topic’s ARN, to send the message in the Lambda function later on, so let’s print that out and keep it handy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6hxNK_3vNTK419SXS9w1Q.png" /></figure><p>Next, we’ll need to create a <strong>subscription — </strong>a destination for our topic to send the data. In this case, we’re going to use an email protocol. In the SNS Console, navigate to the topic and add a subscription.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bmmiLGqDPQJG6IAd6vV65A.png" /></figure><p>Add the email address we want the topic to send the message. We’ll need to confirm the subscription from the email address (Check the spam folder).</p><h4>Create the Lambda</h4><p>Like last time, create a new Lambda and name it retrieveDatetime_UUID and let it create a new default execution role. We should also increase the function’s timeout to 30 seconds.</p><h4>Create the DynamoDB table</h4><p>Next, we’ll create the table where we’ll store our message data. In the Console, create a new table, named ‘messages.’ We’ll give it a <strong>partition key</strong> of message-id, and a <strong>sort key</strong> of timestamp. We’ll keep the defaults for the rest of the settings.</p><h4>Add permissions</h4><p>Finally, the Lambda will need permissions to interact with the SQS queue, SNS topic, and DynamoDB table. Just as in Part 1, let&#39;s navigate to the Lambda’s execution role and edit the policy to include the following actions:</p><ul><li>SQS: “sqs:DeleteMessage”, “sqs:Get*”, “sqs:ReceiveMessage”</li><li>SNS: “sns:Publish”</li><li>DynamoDB: “dynamodb:BatchWriteItem”, “dynamodb:PutItem”</li></ul><p>Make sure you include the ARN of each resource.</p><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor0&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;sqs:DeleteMessage&quot;,<br>                &quot;sqs:Get*&quot;,<br>                &quot;sqs:ReceiveMessage&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;,<br>                &quot;sns:Publish&quot;,<br>                &quot;dynamodb:BatchWriteItem&quot;,<br>                &quot;dynamodb:PutItem&quot;,<br>                <br>            ],<br>            &quot;Resource&quot;: [<br>                &quot;arn:aws:sns:us-east-1:054301730155:dateTime-uuid-topic&quot;,<br>                &quot;arn:aws:sqs:us-east-1:054301730155:DateTime_UUID_sqs&quot;,<br>                &quot;arn:aws:logs:us-east-1:054301730155:log-group:/aws/lambda/retrieveDatetime_UUID_message:*&quot;,<br>                &quot;arn:aws:dynamodb:us-east-1:054301730155:table/messages&quot;<br>            ]<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;VisualEditor1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: &quot;logs:CreateLogGroup&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:logs:us-east-1:054301730155:*&quot;<br>        }<br>    ]<br>}</pre><h3>Write the Lambda function</h3><p>Alright, back to the fun part!</p><h4>Add an SQS trigger</h4><p>Let’s start by adding our SQS trigger through the visualizer and choose our SQS queue from Part 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rqv-a0-uuFUGL0zcPONBVw.png" /></figure><h4>Environment variables</h4><p>Now let’s add a few environment variables.</p><ul><li>QUEUE_NAME: the name of our SQS queue.</li><li>TOPIC_ARN: the ARN of our SNS topic.</li><li>TABLE_NAME: the name of our DynamoDB table.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35PYQRgWC3mUyEjLQjCdaw.png" /></figure><h4>The code</h4><p>As always, let’s import our libraries and establish our environment variables, SNS client, and DynamoDB resource.</p><pre>import boto3<br>import os<br>import json<br>from datetime import datetime<br><br>#environment variables<br>QUEUE_NAME = os.environ[&quot;QUEUE_NAME&quot;]<br>TOPIC_ARN = os.environ[&quot;TOPIC_ARN&quot;]<br>TABLE_NAME = os.environ[&quot;TABLE_NAME&quot;]<br><br>sns = boto3.client(&#39;sns&#39;)<br>dynamodb = boto3.resource(&#39;dynamodb&#39;)</pre><p>Inside the lambda_handler, let’s capture the eventual SQS event.</p><pre>def lambda_handler(event, context):<br>   <br>    messages = event[&#39;Records&#39;]</pre><p>This is the structure of an SQS event response, so we’ll need to iterate over the ‘Records’ to extract the data. The two main keys we’ll need are “messageId,” and “body.” The “body” is where we’ll see the message string we sent from the sendDatetime_UUID Lambda.</p><pre>{<br>  &quot;Records&quot;: [<br>    {<br>      &quot;messageId&quot;: &quot;059f36b4-87a3-44ab-83d2-661975830a7d&quot;,<br>      &quot;receiptHandle&quot;: &quot;AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...&quot;,<br>      &quot;body&quot;: &quot;Test message.&quot;,<br>      &quot;attributes&quot;: {<br>        &quot;ApproximateReceiveCount&quot;: &quot;1&quot;,<br>        &quot;SentTimestamp&quot;: &quot;1545082649183&quot;,<br>        &quot;SenderId&quot;: &quot;AIDAIENQZJOLO23YVJ4VO&quot;,<br>        &quot;ApproximateFirstReceiveTimestamp&quot;: &quot;1545082649185&quot;<br>      },<br>      &quot;messageAttributes&quot;: {},<br>      &quot;md5OfBody&quot;: &quot;e4e68fb7bd0e697a0ae8f1bb342846b3&quot;,<br>      &quot;eventSource&quot;: &quot;aws:sqs&quot;,<br>      &quot;eventSourceARN&quot;: &quot;arn:aws:sqs:us-east-2:123456789012:my-queue&quot;,<br>      &quot;awsRegion&quot;: &quot;us-east-2&quot;<br>    },<br>    {<br>      &quot;messageId&quot;: &quot;2e1424d4-f796-459a-8184-9c92662be6da&quot;,<br>      &quot;receiptHandle&quot;: &quot;AQEBzWwaftRI0KuVm4tP+/7q1rGgNqicHq...&quot;,<br>      &quot;body&quot;: &quot;Test message.&quot;,<br>      &quot;attributes&quot;: {<br>        &quot;ApproximateReceiveCount&quot;: &quot;1&quot;,<br>        &quot;SentTimestamp&quot;: &quot;1545082650636&quot;,<br>        &quot;SenderId&quot;: &quot;AIDAIENQZJOLO23YVJ4VO&quot;,<br>        &quot;ApproximateFirstReceiveTimestamp&quot;: &quot;1545082650649&quot;<br>      },<br>      &quot;messageAttributes&quot;: {},<br>      &quot;md5OfBody&quot;: &quot;e4e68fb7bd0e697a0ae8f1bb342846b3&quot;,<br>      &quot;eventSource&quot;: &quot;aws:sqs&quot;,<br>      &quot;eventSourceARN&quot;: &quot;arn:aws:sqs:us-east-2:123456789012:my-queue&quot;,<br>      &quot;awsRegion&quot;: &quot;us-east-2&quot;<br>    }<br>  ]<br>}</pre><p>Create a new Test event with the above JSON, so we can test if the messages are coming through.</p><p>Let’s iterate over the Records to capture the data we need in a dictionary along with a timestamp.</p><p>Remember, the partition and sort keys we created for our table aremessage-id and timestamp.</p><pre>for message in messages<br>    messageData = {<br>       &#39;message-id&#39;: message[&#39;messageId&#39;],<br>       &#39;body&#39;: message[&#39;body&#39;],<br>       &#39;timestamp&#39;: datetime.now().isoformat()<br>    }</pre><h4>Publish to SNS topic</h4><p>We have our message data stored, so let’s publish it to our topic. With the .publish() method, we need to pass in a TopicArn, Message, and an optional Subject (The first two are required). pass in the ARN we set as an environment variable and a JSON of our message data. We’ll also print out the result to the console.</p><pre>try:<br>    # publish message to sns<br>    snsResponse = sns.publish(<br>        TopicArn=TOPIC_ARN,<br>        Message=json.dumps(messageData, indent=4),<br>        Subject=&#39;DATETIIME_UUID_MESSAGE&#39;<br>    )<br>    print(f&quot;Message sent to SNS: {json.dumps(messageData, indent=4)}&quot;)<br><br>except Exception as e:<br>    print(e)</pre><p>Let’s test it and check our email. We can either use the Test event or trigger it from our API from Part 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oexKSdy-gcPxaW9I3nVDyw.png" /><figcaption>SNS sent from Test</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MonVUL6182-YnZOqSXYS3Q.png" /><figcaption>SNS sent from API/first Lambda</figcaption></figure><p>Success! The SNS publisher is working!</p><h4>Write to DynamoDB table</h4><p>Great! The Lambda is successfully receiving messages from SQS and sending the data to an email via SNS. Now, it&#39;s time to store the data in a table.</p><p>Below the SNS publish code, create a Table sub-resource with our TABLE_NAME environment variable.</p><pre>#write to dynamodb<br>table = dynamodb.Table(TABLE_NAME)<br><br></pre><p>With the .put_item() method, we can pass in our attributes in the Item argument. The Item expects a dictionary of key-values, so we can just pass in the messageData. We’ll also print out a confirmation to the console and return the response.</p><pre>dynamodbResponse = table.put_item(Item = messageData)<br>print(f&quot;Message sent to DynamoDB: {json.dumps(dynamodbResponse, indent=4)}&quot;)<br><br>return json.dumps(dynamodbResponse, indent=4)</pre><p>Let’s test it with our Test event and API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mA1S93oct8a64RNEdSg7zQ.png" /><figcaption>Test execution result</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xEdwgxkE3mAxVnijpCNPlg.png" /><figcaption>Log results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IfV0s7Zjye9qJlQrBCnZoA.png" /><figcaption>DynamoDB table</figcaption></figure><p>Here is the full Lambda code:</p><pre>import boto3<br>import os<br>import json<br>from datetime import datetime<br><br>#environment variables<br>QUEUE_NAME = os.environ[&quot;QUEUE_NAME&quot;]<br>TOPIC_ARN = os.environ[&quot;TOPIC_ARN&quot;]<br>TABLE_NAME = os.environ[&quot;TABLE_NAME&quot;]<br><br>sns = boto3.client(&#39;sns&#39;)<br>dynamodb = boto3.resource(&#39;dynamodb&#39;)<br><br>def lambda_handler(event, context):<br>   <br>    messages = event[&#39;Records&#39;]<br><br>    for message in messages:<br>        messageData = {<br>           &#39;message-id&#39;: message[&#39;messageId&#39;],<br>           &#39;body&#39;: message[&#39;body&#39;],<br>           &#39;timestamp&#39;: datetime.now().isoformat()<br>        }<br>    <br>        try:<br>            # publish message to sns<br>            snsResponse = sns.publish(<br>                TopicArn=TOPIC_ARN,<br>                Message=json.dumps(messageData, indent=4),<br>                Subject=&#39;DATETIIME_UUID_MESSAGE&#39;<br>            )<br>            print(f&quot;Message sent to SNS: {json.dumps(messageData, indent=4)}&quot;)<br>            <br>            #write to dynamodb<br>            table = dynamodb.Table(TABLE_NAME)<br>            <br>            dynamodbResponse = table.put_item(Item = messageData)<br>            print(f&quot;Message sent to DynamoDB: {json.dumps(dynamodbResponse, indent=4)}&quot;)<br>        <br>            return dynamodbResponse<br>    <br>        except Exception as e:<br>            print(e)</pre><h3>Success!</h3><p>Congrats! We just created a data processing solution going from an API &gt;&gt; Lambda 1 &gt;&gt; SQS &gt;&gt; Lambda 2 &gt;&gt; SNS/DynamoDB!</p><p>Again, no, this isn’t the most efficient way to handle data processing, BUT it’s a great representation of how data can move and interact from one service to another. And we did it without the need of managing any servers!</p><p>This is just the tip of the iceberg Lambda’s capabilities.</p><h3>Thank you</h3><p>Thank you for following me on my cloud computing journey. I hope this article was helpful and informative. Please give me a like &amp; follow as I continue my journey, and I will share more articles like this!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6595133dd1a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>