<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jan Agulan on Medium]]></title>
        <description><![CDATA[Stories by Jan Agulan on Medium]]></description>
        <link>https://medium.com/@itsjan?source=rss-774f2d5d432e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*m5j3c9Jd7Em2fZpo-9dSsQ.jpeg</url>
            <title>Stories by Jan Agulan on Medium</title>
            <link>https://medium.com/@itsjan?source=rss-774f2d5d432e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:49:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itsjan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An implementation of on-demand feature environments on AWS using Terraform, Terragrunt, and GitHub…]]></title>
            <link>https://itsjan.medium.com/an-implementation-of-on-demand-feature-environments-on-aws-using-terraform-terragrunt-and-github-ad4951637fd3?source=rss-774f2d5d432e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad4951637fd3</guid>
            <category><![CDATA[continuous-delivery]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Jan Agulan]]></dc:creator>
            <pubDate>Tue, 01 Apr 2025 20:26:48 GMT</pubDate>
            <atom:updated>2025-04-01T20:26:48.820Z</atom:updated>
            <content:encoded><![CDATA[<h3>An implementation of on-demand feature environments on AWS using Terraform, Terragrunt, and GitHub Actions</h3><h3>Introduction</h3><p>A common setup in the development lifecycle is having a shared staging environment where features are combined together for QA, before eventually promoting it to production. In some cases, inefficient release processes block developers from promoting their changes to these shared environments. This leads to longer feedback cycles.</p><p>The nature of shared environments mean that the changes are not isolated. From conflicting changes to inconsistent configurations from the developer’s local environment, leading to the situation where we hear the dreaded “it works on my machine!”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*Ao_CfovpOVNOoO2_gLEfQw.png" /></figure><p>Or worse 😅</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*C3iSjxrvj5edTn_ucSxHbg.jpeg" /><figcaption>(Last meme for this article, I promise)</figcaption></figure><p>This is where on-demand environments come into the picture. Developers can test their changes in their own environment and hand exactly the same environment to QA or product owners for testing and receive immediate feedback.</p><p>Arun wrote a nice article on <a href="https://medium.com/techbeatly/what-are-on-demand-feature-environments-982eaa96fdd5">what on-demand feature environments are and its advantages</a>, I suggest you check it out.</p><p>In this article, I’ll walk you through how you can build an on-demand feature environment setup on AWS using Terraform, Terragrunt, and GitHub Actions. This approach ensures isolated, temporary environments for testing new features without disrupting existing workflows.</p><h3>Prerequisites</h3><ul><li><strong>AWS Account</strong> with permissions to provision infrastructure.</li><li><strong>Terraform </strong>and<strong> Terragrunt </strong>installed on your local machine or CI/CD runner.</li><li><strong>GitHub Repository</strong> with access to configure workflows.</li><li>Basic familiarity with Terraform, Terragrunt, and AWS services like VPCs, ECS, ALBs, SQS, and ALB.</li></ul><h3>An example AWS Infrastructure</h3><p>Throughout this article, we will refer to the architecture below that relates to what we can call the <em>Easy Peasy App</em>. In a nutshell, users access it through a load balancer. It runs on ECS and listens to messages via an SQS queue. It performs processing before persisting data to an S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3foz5h1EpwFZYL4qH8NNw.png" /><figcaption>Easy Peasy App Architecture</figcaption></figure><h3>Why Terraform and Terragrunt?</h3><p>We can all agree it is much easier to manage cloud resources declaratively with code. <a href="https://www.terraform.io/">Terraform</a> is an industry standard for Infrastructure as Code (IAC).</p><p>In a typical setup, multiple environments exist such as <strong>staging </strong>and <strong>production</strong>. Introducing another environment like <strong>on-demand</strong> environments add more complexity. This is where <a href="https://www.env0.com/blog/terragrunt#:~:text=Terragrunt%20can%20be%20used%20to,Terragrunt%20configuration%20in%20a%20terragrunt.">Terragrunt</a> comes in. Terragrunt simplifies this by reducing duplication by allowing shared configurations, therefore improving maintainability.</p><p>With a setup that includes <strong>test, production</strong> and <strong>ondemand</strong>, your infrastructure directory could be structured as follows:</p><pre>infrastructure/<br>│── environments/<br>│   ├── base-infra/<br>│   │   ├── production/<br>│   │   │   └── terragrunt.hcl<br>│   │   ├── test/<br>│   │   │   └── terragrunt.hcl<br>│   │   ├── on-demand/<br>│   │       └── terragrunt.hcl<br>│   ├── service-infra/<br>│   │   ├── production/<br>│   │   │   └── terragrunt.hcl<br>│   │   ├── test/<br>│   │   │   └── terragrunt.hcl<br>│   │   ├── on-demand/<br>│   │       └── terragrunt.hcl<br>│── modules/<br>│   ├── base-infra/<br>│   ├── service-infra/<br>│── terragrunt.hcl</pre><h3>Breaking Down the Infrastructure Modules</h3><p>To better understand the <strong>service-infra</strong> and <strong>base-infra</strong> modules, let’s explore why separating them makes sense.</p><h4>Why have multiple modules?</h4><p>A modular approach improves maintainability. In the context of ondemand environments especially, some resources can be considered to be long-lived, while others are short-lived.</p><p>In the case of <strong>Easy Peasy</strong>, the service-infra module depends on the base-infra module. This separation ensures that:</p><ul><li><strong>Persistent resources</strong> (networking and shared infrastructure) are created once and reused.</li><li><strong>Ephemeral resources</strong> (feature-specific services) are provisioned dynamically and torn down as needed.</li></ul><h4>Short-lived vs Long-lived resources</h4><p>Since on-demand feature environments are designed to be temporary, we need to carefully decide which resources should persist across environments and which should be recreated.</p><h4>Long-lived resources (base-infra)</h4><p>These resources are <strong>shared across multiple feature environments</strong> and do not need to be created each time. For example:</p><ul><li><strong>VPC: </strong>The networking setup remains unchanged.</li><li><strong>External ALB:</strong> can route traffic to multiple feature environments.</li><li><strong>IAM Roles and Policies: </strong>predefined roles can also be used and avoid redundant configuration.</li></ul><h4>Short-lived resources (service-infra)</h4><p>These resources are <strong>unique to each feature environment</strong> and should be created and destroyed dynamically:</p><ul><li><strong>ECS Services: </strong>Each feature environment gets its own containerized service.</li><li><strong>SQS Queues: </strong>Ensures isolated message processing for each environment.</li><li><strong>S3 Buckets:</strong> feature-specific storage that is removed after testing.</li></ul><pre>modules/<br>│── base-infra/<br>│   ├── external_alb.tf<br>│   ├── outputs.tf<br>│   ├── sns.tf<br>│   ├── variables.tf<br>│   ├── vpc.tf<br>│── service-infra/<br>│   ├── alb_target_group.tf<br>│   ├── ecs.tf<br>│   ├── ecs_task_definition.tf<br>│   ├── queues.tf<br>│   ├── s3.tf</pre><p>With this structure, <strong>base-infra</strong> remains <strong>static</strong> and is deployed once (or at least not as modified as often), while <strong>service-infra</strong> is dynamically provisioned for each on-demand feature environment.</p><p>For <strong>production</strong> and <strong>test</strong> environments, the same idea applies: the <strong>base-infra</strong> remains largely unchanged, while most updates happen within <strong>service-infra</strong>. However, the most important distinction I can think of is that instead of creating new resources for these environments, we <strong>replace or update existing ones</strong>, thanks to how Terraform manages state. Speaking of states, I will talk in more detail about Terraform state management in later sections.</p><h3>Defining resources in base-infra</h3><p>To illustrate how to structure <strong>base-infra</strong>, let’s take two examples: an <strong>external ALB</strong> and the <strong>VPC</strong>. Setting up these resources involves three key steps:</p><ol><li><strong>Define the resource: </strong>Declare the infrastructure component in Terraform.</li><li><strong>Expose outputs: </strong>Create output variables for any values that need to be referenced by service-infra.</li><li><strong>Reference outputs in </strong>service-infra: Use the exported outputs in dependent modules.</li></ol><p>By following this approach, service-infra can dynamically retrieve values from base-infra, making it reusable and as a result reduces duplication.</p><h4>Creating the External ALB</h4><p>/infrastructure/modules/base-infra/external_alb.tf</p><pre>module &quot;external_alb&quot; {<br>  source  = &quot;terraform-aws-modules/alb/aws&quot;<br>  version = &quot;9.15.0&quot;<br><br>  name    = &quot;external-alb&quot;<br>  vpc_id  = module.vpc.vpc_id<br>  subnets = module.vpc.public_subnets<br><br>  # Security Group<br>  security_group_ingress_rules = {<br>    all_http_ipv4 = {<br>      from_port   = 80<br>      to_port     = 80<br>      ip_protocol = &quot;tcp&quot;<br>      description = &quot;Allow HTTP public ingress access.&quot;<br>      cidr_ipv4   = &quot;0.0.0.0/0&quot;<br>    }<br>    all_https_ipv4 = {<br>      from_port   = 443<br>      to_port     = 443<br>      ip_protocol = &quot;tcp&quot;<br>      description = &quot;Allow HTTPS public ingress access.&quot;<br>      cidr_ipv4   = &quot;0.0.0.0/0&quot;<br>    }<br>  }<br>  security_group_egress_rules = {<br>    all_ipv4 = {<br>      ip_protocol = &quot;-1&quot;<br>      cidr_ipv4   = &quot;0.0.0.0/0&quot;<br>    }<br>  }<br><br>  listeners = {<br>    ex-http-https-redirect = {<br>      port     = 80<br>      protocol = &quot;HTTP&quot;<br>      redirect = {<br>        port        = &quot;443&quot;<br>        protocol    = &quot;HTTPS&quot;<br>        status_code = &quot;HTTP_301&quot;<br>      }<br>    }<br>    ex-https = {<br>      port            = 443<br>      protocol        = &quot;HTTPS&quot;<br>      certificate_arn = aws_acm_certificate.wildcard.arn<br><br>      fixed_response = {<br>        content_type = &quot;text/plain&quot;<br>        message_body = &quot;404 Not Found&quot;<br>        status_code  = &quot;404&quot;<br>      }<br>    }<br>  }<br>}</pre><h4>Creating the VPC</h4><p>/infrastructure/modules/base-infra/vpc.tf</p><pre>module &quot;vpc&quot; {<br>  source  = &quot;terraform-aws-modules/vpc/aws&quot;<br>  version = &quot;5.19.0&quot;<br><br>  name = &quot;vpc-easy-peasy-${var.env}&quot;<br><br>  cidr = var.vpc_cidr<br><br>  azs             = var.vpc_azs<br>  private_subnets = var.vpc_private_subnets<br>  public_subnets  = var.vpc_public_subnets<br><br>  enable_nat_gateway = true<br>  single_nat_gateway = var.single_nat_gateway<br>}</pre><h4>Creating the Outputs</h4><p>/infrastructure/modules/base-infra/outputs.tf</p><pre>output &quot;external_alb&quot; {<br>  value = module.external_alb<br>}<br><br>output &quot;vpc_id&quot; {<br>  value = module.vpc.vpc_id<br>}<br><br>output &quot;private_subnets&quot; {<br>  value = module.vpc.private_subnets<br>}</pre><h3>Defining resources in service-infra</h3><p>For resources in service-infra that depend on base-infra, you’ll need to <strong>consume the outputs</strong> from base-infra by passing them as inputs in service-infra. This process involves three steps too:</p><ol><li><strong>Define variables: </strong>Create variables to hold the output values from base-infra.</li><li><strong>Pass inputs in </strong>terragrunt.hcl<strong>: </strong>Reference the base-infra outputs and provide them as inputs.</li><li><strong>Use the variables in resource definitions — </strong>Incorporate the referenced values when defining resources in service-infra.</li></ol><h4>Create variables to hold the outputs</h4><p>/infrastructure/modules/service-infra/variables.tf</p><pre>variable &quot;external_alb&quot; {<br>  type = object({<br>    arn_suffix = string<br>    dns_name   = string<br>    zone_id    = string<br>    listeners = map(object({<br>      arn = string<br>    }))<br>    security_group_id = string<br>  })<br>}<br><br>variable &quot;vpc_id&quot; {<br>  type = string<br>}<br><br>variable &quot;private_subnets&quot; {<br>  type = list(string)<br>}</pre><h4>Consume the outputs from base-infra as inputs in service-infra</h4><p>/infrastructure/environments/service-infra/{environment}/terragrunt.hcl</p><pre>inputs = {<br>  external_alb    = dependency.base_infra.outputs.external_alb<br>  vpc_id          = dependency.base_infra.outputs.vpc_id<br>  private_subnets = dependency.base_infra.outputs.private_subnets<br>}</pre><h4>Use the variables to create the resources in service-infra</h4><p>To define an ALB target group in service-infra:</p><p>/infrastructure/modules/service-infra/alb.tf</p><pre>resource &quot;aws_alb_target_group&quot; &quot;ecs_service&quot; {<br>  name                 = local.prefix_app_name<br>  port                 = 80<br>  protocol             = &quot;HTTP&quot;<br>  vpc_id               = var.vpc_id<br>  target_type          = &quot;ip&quot;<br>  deregistration_delay = 30<br><br>  health_check {<br>    path              = &quot;/actuator/health&quot;<br>    interval          = 5<br>    timeout           = 4<br>    healthy_threshold = 2<br>  }<br>}<br><br>resource &quot;aws_route53_record&quot; &quot;this&quot; {<br>  zone_id = var.route_53_zone_id<br>  name    = local.route53_record_name<br>  type    = &quot;A&quot;<br><br>  alias {<br>    name                   = var.external_alb.dns_name<br>    zone_id                = var.external_alb.zone_id<br>    evaluate_target_health = false<br>  }<br>}<br><br>resource &quot;aws_alb_listener_rule&quot; &quot;ecs_service&quot; {<br>  listener_arn = var.external_alb.listeners[&quot;ex-https&quot;].arn<br><br>  action {<br>    type             = &quot;forward&quot;<br>    target_group_arn = aws_alb_target_group.ecs_service.arn<br>  }<br><br>  condition {<br>    host_header {<br>      values = [aws_route53_record.this.fqdn]<br>    }<br>  }<br>}</pre><p>Notice how the variable external_albis used to create the resource aws_route_53_record.</p><p>To create an ECS service:</p><p>/infrastructure/modules/service-infra/ecs_service.tf</p><pre>resource &quot;aws_ecs_service&quot; &quot;app&quot; {<br>  name            = local.prefix_app_name<br>  cluster         = aws_ecs_cluster.this.arn<br>  task_definition = aws_ecs_task_definition.app.arn<br>  launch_type                        = &quot;FARGATE&quot;<br>  propagate_tags                     = &quot;SERVICE&quot;<br><br>  network_configuration {<br>    security_groups = [aws_security_group.ecs_task.id]<br>    subnets         = var.private_subnets<br>  }<br><br>  load_balancer {<br>    target_group_arn = aws_alb_target_group.ecs_service.arn<br>    container_name   = local.prefix_app_name<br>    container_port   = var.port<br>  }<br><br>  lifecycle {<br>    # desired_count changes must be ignored because we are using autoscaling<br>    ignore_changes = [desired_count]<br>  }<br>}<br><br>resource &quot;aws_alb_target_group&quot; &quot;ecs_service&quot; {<br>  name                 = local.prefix_app_name<br>  port                 = 80<br>  protocol             = &quot;HTTP&quot;<br>  vpc_id               = var.vpc_id<br>  target_type          = &quot;ip&quot;<br>  deregistration_delay = 30<br>}<br><br>resource &quot;aws_route53_record&quot; &quot;this&quot; {<br>  zone_id = var.route_53_zone_id<br>  name    = local.route53_record_name<br>  type    = &quot;A&quot;<br><br>  alias {<br>    name                   = var.external_alb.dns_name<br>    zone_id                = var.external_alb.zone_id<br>    evaluate_target_health = false<br>  }<br>}<br><br>resource &quot;aws_alb_listener_rule&quot; &quot;ecs_service&quot; {<br>  listener_arn = var.external_alb.listeners[&quot;ex-https&quot;].arn<br><br>  action {<br>    type             = &quot;forward&quot;<br>    target_group_arn = aws_alb_target_group.ecs_service.arn<br>  }<br><br>  condition {<br>    host_header {<br>      values = [aws_route53_record.this.fqdn]<br>    }<br>  }<br>}<br><br>resource &quot;aws_security_group&quot; &quot;ecs_task&quot; {<br>  name        = &quot;${var.app_name}-ecs-task&quot;<br>  description = &quot;Allow inbound access from the ALB only.&quot;<br>  vpc_id      = var.vpc_id<br><br>  tags = {<br>    Name = &quot;${var.app_name}-ecs-task&quot;<br>  }<br>}</pre><p>Notice how the external_alb, vpc_id, and private_subnetswere used to create the aws_ecs_service and aws_alb_listener_rule.</p><p>In the next section, let’s have a quick recap on how Terraform manages resource states and how we can utilize it to create on-demand environments.</p><h3>Terraform state management</h3><p>Terraform maintains a state file to keep track of the real-world infrastructure it manages. This file maps your code configurations to the actual resources which tell Terraform to perform operations like creation, updates, and deletion.</p><p>The recommended practice for managing state files is to use terraform’s built-in support for remote backends. In AWS, S3 is the recommended solution.</p><p>An example remote state configuration looks like this:</p><pre>remote_state {<br>  backend = &quot;s3&quot;<br><br>  config = {<br>    bucket = &quot;infrastructure-${local.env}&quot;<br>    region = local.region<br>    key    = &quot;terraform-states/service-infra.tfstate&quot;<br>  }<br><br>}</pre><h4>A separate state for each feature</h4><p>For <strong>on-demand feature environments</strong>, it’s critical to isolate the state for each feature branch. This ensures that resources created for one feature don’t interfere with another and allows them to be safely created and destroyed in parallel.</p><p>To achieve this, you can dynamically generate the key for the state file by including a feature-specific prefix or branch identifier. For example, by using local.env_prefix, your terragrunt.hcl might look like this:</p><p>/infrastructure/service-infra/terragrunt.hcl</p><pre>remote_state {<br>  backend = &quot;s3&quot;<br><br>  config = {<br>    bucket = &quot;infrastructure-${local.env}&quot;<br>    region = local.region<br>    key    = &quot;terraform-states/${local.env_prefix}service-infra.tfstate&quot;<br>  }<br><br>}</pre><p>Using this approach:</p><ul><li>A <strong>unique Terraform state file</strong> is created for each feature environment.</li><li><strong>State isolation</strong> prevents conflicts during concurrent feature deployments.</li><li>Resources can be <strong>cleaned up safely</strong> when the environment is torn down.</li></ul><h4>How do we manage resources that already exist?</h4><p>Earlier, we discussed how splitting infrastructure into modules such as base-infra and service-infra help us manage which resources are created once and which are provisioned dynamically.</p><p>In the case of the <strong>Easy Peasy App</strong> which runs on ECS, the application requires <strong>security groups</strong>. Logically, you might think to define the security groups within service-infra since they are tied to the ECS service. However, there&#39;s a catch.</p><p>With our setup where each feature environment uses a <strong>unique Terraform state file</strong> (e.g., by prefixing the state key), any resource defined in service-infra will be <strong>recreated for each environment</strong>. That means Terraform will attempt to create <strong>new security groups</strong> every time, which will not work since the command will fail since the resource with the same name already exists.</p><p>For example, in the ECS service definition, the security groups can be defined in this way:</p><p>/infrastructure/modules/service-infra/ecs_service.tf</p><pre>resource &quot;aws_security_group&quot; &quot;ecs_task&quot; {<br>  name        = &quot;${var.app_name}-ecs-task&quot;<br>  description = &quot;Allow inbound access from the ALB only.&quot;<br>  vpc_id      = var.vpc_id<br><br>  tags = {<br>    Name = &quot;${var.app_name}-ecs-task&quot;<br>  }<br>}<br><br>resource &quot;aws_vpc_security_group_ingress_rule&quot; &quot;allow_alb&quot; {<br>  security_group_id = aws_security_group.ecs_task.id<br><br>  description                  = &quot;Allow all inbound access from the ALB.&quot;<br>  ip_protocol                  = &quot;tcp&quot;<br>  from_port                    = var.port<br>  to_port                      = var.port<br>  referenced_security_group_id = var.external_alb.security_group_id<br>}<br><br>resource &quot;aws_vpc_security_group_egress_rule&quot; &quot;internet&quot; {<br>  security_group_id = aws_security_group.ecs_task.id<br><br>  description = &quot;Allow all outbound access.&quot;<br>  ip_protocol = &quot;-1&quot;<br>  from_port   = -1<br>  to_port     = -1<br>  cidr_ipv4   = &quot;0.0.0.0/0&quot;<br>}</pre><p>To prevent Terraform from trying to recreate these existing resources, you can <strong>pre-import them into the state</strong> before doing the actual deployment. This way, Terraform recognizes them as already managed and will not attempt to recreate them.</p><p>Since we’re using Terragrunt, the process is simple. Here are example commands to import the ECS-related security group and associated rules:</p><pre>terragrunt import aws_security_group.ecs_task sg-idxxxx<br>terragrunt import aws_vpc_security_group_ingress_rule.allow_alb sgr-idxxxx<br>terragrunt import aws_vpc_security_group_egress_rule.internet sgr-idxxxx</pre><p>These commands associate the existing AWS resources with Terraform resources defined in your service-infra code, ensuring they’re tracked without being recreated.</p><p>In the next few sections, I’ll walk you through:</p><ul><li>The <strong>sequence</strong> in which these imports fit into the overall deployment flow, and</li><li>An <strong>example script</strong> to automate importing multiple resources across environments.</li></ul><h4>What about resources that should be created for each feature environment?</h4><p>In previous sections, we discussed avoiding the recreation of shared resources. But what about the resources we <strong>do want</strong> to provision <strong>independently for each feature environment</strong>?</p><p>For the <strong>Easy Peasy App</strong>, let’s say we want each feature branch to have its <strong>own SQS queue</strong> and <strong>dedicated S3 bucket</strong>.</p><p>To achieve this, we can dynamically name resources using a <strong>prefix</strong>, such as the <strong>branch name</strong> or any other unique environment identifier.</p><p>Using SQS as an example, here’s how you can configure this in your terragrunt.hcl for the on-demand environment:</p><p>/infrastructure/service-infra/ondemand/terragrunt.hcl</p><pre>locals {<br>  env = basename(get_terragrunt_dir())<br>  env_prefix = local.env == &quot;ondemand&quot; ? &quot;${run_cmd(&quot;git&quot;, &quot;rev-parse&quot;, &quot;--abbrev-ref&quot;, &quot;HEAD&quot;)}-&quot; : &quot;&quot;<br>}<br><br>inputs = {<br>  env_prefix = local.env_prefix<br>  queue_name = &quot;queue-name&quot;<br>}</pre><p>And then for queues.tf, you can implement something like this:</p><p>/infrastructure/modules/service-infra/queues.tf</p><pre>resource &quot;aws_sqs_queue&quot; &quot;input_queue&quot; {<br>  name                      = &quot;${var.env_prefix}${var.queue_name}&quot;<br>  delay_seconds             = 90<br>  max_message_size          = 2048<br>  message_retention_seconds = 86400<br>  receive_wait_time_seconds = 10<br>  redrive_policy = jsonencode({<br>    deadLetterTargetArn = aws_sqs_queue.terraform_queue_deadletter.arn<br>    maxReceiveCount     = 3<br>  })<br><br>}</pre><h4>What about resource teardown?</h4><p>When running terragrunt destroy, all resources tracked in the Terraform state will be <strong>fully destroyed</strong>. This is expected behavior for on-demand feature environments, which are designed to be temporary.</p><p>However, if your setup includes <strong>shared or reused resources</strong> like security groups, VPCs, or IAM roles that are intentionally reused across environments, you’ll need to take care to <strong>remove them from the state</strong> before executing the destroy operation. Otherwise, Terraform will attempt to delete them along with the rest of the environment.</p><p>To avoid accidentally destroying shared infrastructure:</p><ul><li><strong>Import shared resources</strong> into the state only when needed.</li><li><strong>Manually remove</strong> them from the state using terragrunt state rm before running terragrunt destroy.</li></ul><p>This way, you can safely tear down environment-specific resources without impacting anything shared.</p><h3>Github Actions Workflow</h3><p>One of the major advantages of on-demand environments is the flexibility to <strong>spin them up at any time. F</strong>or example, based on a specific branch. A common approach is to trigger environment creation <strong>manually</strong> for any feature branch using GitHub Actions.</p><p>In GitHub, you can enable manual triggering by adding a workflow_dispatch event to your workflow definition.</p><p>/.github/workflows/create-ondemand-env-pipeline.yml</p><pre><br>on:<br>  workflow_dispatch:</pre><p>This setup allows you to go to the <strong>Actions</strong> tab in your GitHub repository, select the <strong>“Run workflow”</strong> button, and choose the target branch you want to deploy from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IvqEgBot8LwugZJXKYCLWw.png" /></figure><p>This manual trigger is especially useful when:</p><ul><li>You want to deploy a specific feature branch for QA or testing.</li><li>You want more control over when environments are spun up (vs. automatic PR-based deployment).</li><li>You’re still refining your automation and prefer semi-manual control.</li></ul><p>Of course, it goes without saying that since service-infra depends on outputs from base-infra, the resources in base-infra must be created as a prerequisite.</p><p>Now that we’ve covered how to structure separate modules, manage Terraform state, and handle both reusable and per-feature resources, let’s move on to the next section for a high-level overview of how everything fits together.</p><h4>On-demand environment creation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNfB4wyp2iXcZFJFBcAH2Q.png" /></figure><p>The GHA workflow for deploying the service-infra will look something like this:</p><p>/.github/workflows/create-ondemand-env-pipeline.yml</p><pre>name: Deploy on demand environment<br><br>on:<br>  workflow_dispatch:<br><br>jobs:<br>  import-resources:<br>    name: Import resources to state<br>    permissions:<br>      id-token: write<br>      issues: write<br>      contents: read<br>    runs-on: ubuntu-latest-8core<br>    steps:<br>      - name: checkout<br>        uses: actions/checkout@v4<br><br>      - name: Configure AWS credentials<br>        uses: aws-actions/configure-aws-credentials@v4<br>        with:<br>          aws-region: eu-central-1<br>          role-to-assume: ${{ role-to-assume }}<br>          role-session-name: &quot;${{ role-session-name }}&quot;<br>          role-duration-seconds: 900<br><br>      - name: Set up Terraform<br>        uses: hashicorp/setup-terraform@v3<br>        with:<br>          terraform_version: ${{ vars.TF_VERSION }}<br><br>      - name: Setup Terragrunt<br>        uses: autero1/action-terragrunt@v3<br>        with:<br>          terragrunt-version: ${{ vars.TG_VERSION }}<br>          token: ${{ secrets.GITHUB_TOKEN }}<br><br>      - name: Import resources<br>        continue-on-error: true # if the resources already exist, it&#39;s fine. It only means the environment for this branch was already created.<br>        env:<br>          BRANCH_NAME: ${{ github.ref_name }}<br>        run: ./scripts/infrastructure/env/ondemand/import-resources.sh    <br><br>  build:<br>    # your steps to build the service<br><br>  deploy-service-infra:<br>    # your steps to deploy the service</pre><p>I mentioned earlier that you can use a script to automatically import the resources to the state before running the deployment. The <strong>import-resources.sh</strong> script is:</p><p>/scripts/infrastructure/env/ondemand/import-resources.sh</p><pre>#!/bin/bash<br><br># List of Terraform resource addresses to import from SSM Parameter Store<br>resource_addresses=(<br>  &quot;aws_security_group.ecs_task&quot;<br>  &quot;aws_iam_role.task_execution&quot;<br>  &quot;aws_iam_policy.task_execution_extended&quot;<br>  &quot;aws_iam_role.task&quot;<br>  &quot;aws_vpc_security_group_ingress_rule.allow_alb&quot;<br>  &quot;aws_vpc_security_group_egress_rule.internet&quot;<br>  # Add more Terraform resources to import here<br>)<br><br># Path to the Terragrunt module directory<br>terragrunt_dir=&quot;./infrastructure/service-infra/ondemand/&quot;<br><br># Prefix for SSM parameter names<br>ssm_prefix=&quot;/ondemand_env&quot;<br><br># Iterate over each resource and import if the ID is found in SSM<br>for resource_address in &quot;${resource_addresses[@]}&quot;; do<br>  ssm_param_name=&quot;$ssm_prefix/$resource_address/id&quot;<br>  resource_id=$(aws ssm get-parameter --name &quot;$ssm_param_name&quot; --query &quot;Parameter.Value&quot; --output text 2&gt;/dev/null)<br><br>  if [ -n &quot;$resource_id&quot; ]; then<br>    echo &quot;Importing resource: $resource_address with ID: $resource_id&quot;<br>    terragrunt import --terragrunt-working-dir &quot;$terragrunt_dir&quot; &quot;$resource_address&quot; &quot;$resource_id&quot;<br>  else<br>    echo &quot;Resource ID not found in SSM for: $resource_address&quot;<br>    echo &quot;Skipping import. This resource may need to be created and saved later.&quot;<br>  fi<br>done</pre><h4>On-demand environment teardown</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aa5zjNNeDSadnmB1ahvk9g.png" /></figure><p>When it’s time to tear down an environment, it’s important to <strong>remove any shared or reusable resources from the Terraform state</strong> before running the destroy command. This ensures those resources are not deleted and remain available for reuse in future environments.</p><p>As shown in the workflow above, the <strong>resource IDs of reusable components</strong> can be stored in <strong>AWS SSM Parameter Store</strong>, making it easy to reference and re-import them when needed.</p><p>This entire process can be encapsulated into a script to automate:</p><ul><li>Determine whether to save the resourceIDs in SSM</li><li>Removing them from the state before destruction</li></ul><p>/scripts/infrastructure/env/ondemand/destroy-resources.sh</p><pre>#!/bin/bash<br><br># Define Terraform resource addresses<br>tf_resources=(<br>  &quot;aws_security_group.ecs_task&quot;<br>  &quot;aws_cloudwatch_log_group.app&quot;<br>  &quot;aws_iam_role.task_execution&quot;<br>  &quot;aws_iam_policy.task_execution_extended&quot;<br>  &quot;aws_iam_role.task&quot;<br>  &quot;aws_vpc_security_group_ingress_rule.allow_alb&quot;<br>  &quot;aws_vpc_security_group_egress_rule.internet&quot;<br>  # Add more resources as needed<br>)<br><br># Path to the Terragrunt working directory<br>terragrunt_dir=&quot;./infrastructure/environments/service-infra/ondemand/&quot;<br><br># Fetch Terraform state in JSON format<br>tf_state_json=$(terragrunt show -json --terragrunt-working-dir &quot;$terragrunt_dir&quot; --terragrunt-forward-tf-stdout)<br><br># Remove first line (if it&#39;s a warning or empty line)<br>tf_state_json_cleaned=$(echo &quot;$tf_state_json&quot; | tail -n +2)<br><br># Process each resource<br>for tf_resource in &quot;${tf_resources[@]}&quot;; do<br>  tf_resource_id=$(jq -r --arg res_addr &quot;$tf_resource&quot; &#39;.values.root_module.resources[] | select(.address == $res_addr) | .values.id&#39; &lt;&lt;&lt; &quot;$tf_state_json_cleaned&quot;)<br><br>  if [ -n &quot;$tf_resource_id&quot; ]; then<br>    echo &quot;Resource: $tf_resource, ID: $tf_resource_id&quot;<br><br>    ssm_param_name=&quot;/ondemand_env/$tf_resource/id&quot;<br>    existing_ssm_value=$(aws ssm get-parameter --name &quot;$ssm_param_name&quot; --query &quot;Parameter.Value&quot; --output text 2&gt;/dev/null)<br><br>    if [ &quot;$existing_ssm_value&quot; == &quot;$tf_resource_id&quot; ]; then<br>      echo &quot;Parameter $ssm_param_name already exists with the same value. Skipping.&quot;<br>    else<br>      # Store ID in SSM<br>      aws ssm put-parameter --name &quot;$ssm_param_name&quot; --type String --value &quot;$tf_resource_id&quot; --overwrite<br>    fi<br><br>    # Remove resource from Terraform state<br>    terragrunt state rm --terragrunt-working-dir &quot;$terragrunt_dir&quot; &quot;$tf_resource&quot;<br>  fi<br>done</pre><h3>Conclusion</h3><p>Implementing on-demand feature environments with Terraform, Terragrunt, AWS, and GitHub Actions is one way to improve development workflows, reduce bottlenecks, and get earlier feedback in the devlopment lifecycle.</p><p>Throughout this article, we explored:</p><ul><li><strong>How to structure infrastructure</strong> using base-infra and service-infra modules for better reusability and maintainability.</li><li>The importance of <strong>remote state management</strong>, including how to isolate environments with dynamic state keys and import shared resources to avoid duplication.</li><li>How to integrate <strong>GitHub Actions</strong> to automate environment provisioning and destruction — enabling teams to deploy and validate features with minimal friction.</li></ul><p>Like any solution, this approach comes with trade-offs and caveats that should be carefully considered before implementation.</p><h3>Caveats</h3><h4>Resource Naming Constraints</h4><p>If you’re using the <strong>Git branch name as a prefix</strong>, be aware that many AWS services impose length limits on resource names. Long or complex branch names (e.g., feature/this-is-a-really-long-branch-name) can lead to errors during resource creation.</p><p>To work around this, some workarounds are:</p><ul><li><strong>Truncate or sanitize branch names</strong> before using them in resource identifiers.</li><li>Define a consistent, predictable <strong>naming convention</strong> (e.g., using a short hash or have branch naming conventions).</li></ul><h4>Cost Management</h4><p>On-demand environments can become <strong>expensive</strong> if left running unnecessarily. It’s critical to:</p><ul><li>Implement <strong>automated teardown</strong> mechanisms after PRs are merged or closed.</li><li>Regularly clean up <strong>idle environments</strong>, possibly with a TTL-based cleanup script or Lambda function.</li></ul><p>Failing to do so can lead to accumulating unused compute, storage, and networking resources — driving up your AWS bill.</p><h4>Manual State Management</h4><p>While Terraform advises against <strong>manual state manipulation</strong>, in controlled workflows it can be a practical necessity. In our case:</p><ul><li>base-infra is stable and rarely changes.</li><li>All changes follow <strong>standardized, version-controlled processes</strong>.</li><li>We’re using <strong>S3 as the remote backend</strong>, giving us the ability to <strong>audit, version, and restore</strong> state files if needed.</li></ul><p>With this disciplined approach, the risks of manual state handling are significantly reduced.</p><h4>Managing Complex Resources (e.g., Databases)</h4><p>Some resources — particularly <strong>databases</strong> — are trickier to manage in short-lived environments. Creating a full RDS instance per feature branch is often overkill in terms of both <strong>cost</strong> and <strong>provisioning time</strong>.</p><p>Alternative approaches include:</p><ul><li>Reusing a <strong>shared test database</strong>, while isolating test data at the application level.</li><li>Spinning up lightweight <strong>containerized databases</strong> (e.g., Postgres via Docker) for feature environments when full isolation is required.</li><li>Employing <strong>mock or stubbed services</strong> for features that don’t require full data persistence.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad4951637fd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cost Savings: Our journey migrating to self-hosted runners from GitHub runners]]></title>
            <link>https://itsjan.medium.com/cost-savings-our-journey-migrating-to-self-hosted-runners-from-github-runners-53e3e3f1df23?source=rss-774f2d5d432e------2</link>
            <guid isPermaLink="false">https://medium.com/p/53e3e3f1df23</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <dc:creator><![CDATA[Jan Agulan]]></dc:creator>
            <pubDate>Thu, 27 Feb 2025 07:23:23 GMT</pubDate>
            <atom:updated>2025-02-27T14:53:04.576Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cost Savings: A journey migrating to self-hosted runners from GitHub-hosted runners</h3><h3>Introduction</h3><p>A few months ago, I migrated GitHub actions pipelines to self-hosted runners. From a cost-saving perspective, I’d like to share why you should consider migrating.</p><h4>Why Self-Hosted Runners?</h4><p>Our company’s infrastructure is already hosted in AWS. There are some scenarios where CI tests (running on GitHub actions) had to make requests to private endpoints within our AWS infrastructure. This provided the initial need to have self-hosted runners to begin with.</p><h4>What this article covers (and what it doesn’t)</h4><p>This is not a step-by-step guide on how to create and configure self-hosted runners. Instead, I’ll focus on costs and possible savings, helping you decide whether migrating makes sense for your use case.</p><p>Although memory, storage, and networking clearly add to the costs, the biggest cost driver is job execution time. So, this is where I’ll focus my analysis.</p><p>At the end of the article, you will see a cost comparison and potential cost savings.</p><h3>Understanding GitHub Actions Billing</h3><p>Before diving into cost comparisons, it’s important to understand how Github Actions pricing works. <a href="https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions">This article</a> summarizes how billing works for GitHub actions.</p><p>Here are some key points for GitHub billing:</p><ul><li>Actions are billed on a per-minute basis, where the price varies depending on the runner’s vCPU.</li><li>GitHub rounds the partial minutes each job uses to the nearest whole minute. This means an action that runs for 1 minute and 1 second gets billed at 2 minutes.</li></ul><p>You can find the full list of the GitHub runners and their respective prices <a href="https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates-for-x64-powered-larger-runners">here</a>.</p><h3>Self-hosted Runners Pricing</h3><p>The pricing for the self-hosted runners my team used are based on EC2 spot pricing. There is no fixed price for this kind of instances. The cost fluctuates based on real-time market demand for spot instances. With this, the price that is in effect is the price on the “spot”, or the time period the instances are running for. To learn more about EC2 Spot Instances pricing, you can refer to <a href="https://aws.amazon.com/ec2/spot/pricing/">this article from AWS</a>.</p><h4>How do the self-hosted runners work?</h4><p>We forked a terraform module (<a href="https://github.com/github-aws-runners/terraform-aws-github-runner">terraform-aws-github-runner</a>) that created the required infrastructure needed to run the GitHub actions. Behind the scenes, these were using auto-scaling runners on AWS spot instances.</p><p>Extensive documentation is available on their <a href="https://github-aws-runners.github.io/terraform-aws-github-runner/">GitHub Pages Docs site</a>, for those that are curious and want to use something similar.</p><h4>Runner Types and Scaling Strategy</h4><p>For our use case, we configured 2 types of self-hosted runners:</p><ul><li><strong>Regular runners</strong>: This uses cheaper, lower-capacity instance types for standard workloads.</li><li><strong>Powerful runners:</strong> This utilizes instances with more vCPUs, cores, and threads per core for resource-intensive jobs.</li></ul><p>When an action is triggered (i.e., a test is ran, or a deployment is initiated), a spot instance will be created depending on the available instance type. By default, the <a href="https://github.com/philips-labs/terraform-aws-github-runner/blob/45fd1cba51e754772cecf8c0ee25306d1b843097/variables.tf#L474-L482">allocation strategy</a> is set to “lowest-price”, ensuring we always get the most cost-effective instance at any given time.</p><h3>Instance Types</h3><p>To make a better comparison of costs, I listed down the instance types that we used and their vCPUs. For a full list of supported CPU options for EC2 instance types, you can refer to <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html">this article</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A3_0G4o0GRsmETVXaUC4bg.png" /><figcaption>Self-hosted runners instance types</figcaption></figure><p>For GitHub, we were using the following runners:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*knDIq454BdU9eoxao875oA.png" /><figcaption>GitHub runners instance types</figcaption></figure><h3>Cost Comparison</h3><h4>GitHub Actions</h4><p>In GitHub, you can export the total costs per month, and see a detailed breakdown of how long the actions ran for, and it’s corresponding costs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDpAANKLxLGSBkMjU_0-bQ.png" /><figcaption>A sample GitHub actions usage export</figcaption></figure><p>For the purposes of easier comparison, we can get the cost per hour for the types of the used GitHub runners:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vr-Qa6Q535Cf3zt-3sQdag.png" /><figcaption>Per-hour rates for GitHub runners</figcaption></figure><h4>AWS EC2 Spot</h4><p>As mentioned earlier, EC2 spot pricing is dynamic. It depends on the time period, instance type, and region where the instance was created among other things.</p><p>To estimate costs, we can use historical prices from previous months. You can find the EC2 spot pricing history here: <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-history.html">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances-history.html</a></p><p>Below, we can review the spot price trends for the past 3 months. We can then use the most expensive type for each self-hosted runner for the purpose of comparison later.</p><h4>Regular Runner</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AujwYkAOZz1mP-L1IRltzQ.png" /><figcaption>t3a.xlarge pricing per hour</figcaption></figure><h4>Powerful Runner</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0jDM9k7KqFUaajDl_ffQQ.png" /><figcaption>c5.4xlarge pricing per hour</figcaption></figure><p>So, putting this in a table using the most expensive average hourly cost in the most expensive region, we can use these values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m4hpv7iXOTDN0jAzMQFKOw.png" /><figcaption>Hourly cost of the most expensive EC2 instance type for each type of runner</figcaption></figure><p>Looking at the hourly cost of spot instances, the difference is very noticeable when compared to GitHub runners.</p><p>However, let’s take it a step further. Instead of only comparing hourly rates, let’s introduce some arbitrary execution times for each runner type. This will give us a more practical cost comparison by simulating real-world CI/CD workloads.</p><h4>Example Execution and Costs</h4><h4>GitHub Runners</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1n3oKuoeBT7A7FcOBkkmKA.png" /><figcaption>Example execution and costs using GitHub runners</figcaption></figure><h4>Self-Hosted Runners</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cC9c8b8xutAWYGyyiIXIxw.png" /><figcaption>Using the same execution time, and using the average hourly cost for self-hosted runners</figcaption></figure><h4>Potential Cost Savings</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESh4Q-R2qxNI0LUMTQA-xg.png" /><figcaption>Potential cost savings</figcaption></figure><h3>Conclusion</h3><p>With around <strong>85% savings</strong> based on our comparison, migrating to self-hosted runners is worth considering, especially if you already have the infrastructure that can support it. In our case, most of our infrastructure were already hosted in AWS.</p><p>Another takeaway is that even more powerful EC2 instances often cost cheaper than GitHub’s standard runners. This means you can upgrade to better hardware without increasing costs. In some cases, you might even achieve additional savings.</p><h4>Performance Considerations</h4><p>From an execution time perspective, I did not observe a significant increase or decrease in the total time of our builds, provided that we used the correct self-hosted runner type. This suggests that with the right configuration, the performance remains stable.</p><h4>Challenges and Trade-offs</h4><ul><li><strong>Initial setup and management overhead: </strong>While we leveraged an open-source terraform module, there was still some initial setup needed. Unlike GitHub-hosted runners which require no maintenance, self-hosted runners need ongoing monitoring and management.</li><li><strong>Security considerations: </strong>Since self-hosted runners interact with AWS services (ECR, RDS, etc), you need to manage permissions and security policies yourself.</li><li><strong>The nature of spot instances: </strong>Spot instances by nature can take some time to spin up. They can also be reclaimed at any time by AWS with only a few minutes notice. This means your CI/CD jobs can be interrupted mid-run.</li></ul><h4>Final Thoughts</h4><p>If cost savings are a priority and you have the infrastructure and resources to manage self-hosted runners, then migrating is definitely worth considering. As with all solutions, trade-offs need to be evaluated to ensure that they align with your needs. With a potential 85% cost savings, maybe you can get your manager to get you a good coffee for a couple of months?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53e3e3f1df23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>