In traditional server-based systems, gradual rollouts and canary deployments are standard practice. In serverless systems, however, we too often treat infrastructure as magic black boxes rather than critical, complex systems that deserve the same rigor. Even with extensive pre-production testing, this leaves us exposed to unexpected outages.

Gradual rollouts are not a “nice-to-have” — they’re essential, even for serverless.

Why gradual rollouts matter to us

Runa embarked on its serverless journey nearly 3 years ago, adopting serverless technologies to power our next-generation global payout platform. The platform needs to be fast, but reliability is non-negotiable.

At Runa, this problem is not theoretical. We use Step Functions extensively across our platform, and they sit on the path of many of our most important production workflows. That scale is what made rollout safety impossible to ignore. Engineers ship changes several times a day. Every release introduces risk, and without guardrails, that risk quickly compounds. To maintain high deployment velocity without compromising availability, we needed a deployment mechanism that ensured:

• Smooth end-user experience during releases
• Real-time monitoring and automated rollback capabilities

AWS already gives Lambda users a familiar deployment model via CodeDeploy, with built-in traffic shifting, and automated rollback. We wanted the equivalent operational model for Step Functions in the Terraform-driven environment we actually run. That is what led us to build Tweety.

Defining Requirements

Before designing the rollout mechanism, we aligned on a small set of baseline requirements. These were not aspirational goals. They were constraints driven by how we build and operate production systems:

• Terraform Compatibility
  Our infrastructure is managed via Terraform. The rollout solution had to integrate seamlessly, leveraging the tools our engineers already use.
• Automation and Guardrails
  Rollout and rollback processes had to be fully automated with built-in safeguards to prevent accidental missteps.
• Simplicity for Engineers
  The underlying implementation could be complex, but interacting with it should not be. From an engineer’s perspective, deploying with gradual rollouts needed to feel no different from a standard release.

Planning for a rollout

Before we could even start thinking about gradual releases, we needed two critical capabilities:

• The ability to version your changes
• The ability to balance traffic between the versions

A gradual rollout only works if we can precisely control which version of a Step Function is executed at any given time. AWS Step Functions already support versioning (publishing a version creates an immutable snapshot of the state machine definition). Using aliases, we can then control which version is invoked and gradually shift traffic between versions.

In AWS, invoking a specific version is achieved using qualified ARNs, which explicitly reference either a published version or an alias.

For example:

• Unqualified ARN — always points to the latest version:

arn:aws:states:us-east-1:123456789012:stateMachine:OrderProcessor

• Version Qualified ARN — explicitly points to a published version:

arn:aws:states:us-east-1:123456789012:stateMachine:OrderProcessor:3

• Alias Qualified ARN — points to an alias that can route traffic between versions:

arn:aws:states:us-east-1:123456789012:stateMachine:OrderProcessor:prod

In order to ensure stability during the rollout we must use qualified ARNs across our infrastructure.

Only shift traffic at the entry point

Because Step Functions execute downstream services as part of a single workflow, traffic shifting can only happen at the entry point, which is the alias of the top-level state machine being invoked. Any child resources, such as nested Step Functions or Lambdas, must be version pinned.

If we attempt to shift traffic for downstream resources independently, we risk creating version mismatches when the contract between versions changes. For example, a new state machine may invoke an older Lambda with an incompatible input, or an older state machine may call a newer Lambda expecting fields that are no longer present. These mismatches can lead to runtime errors or partial outages that are hard to detect during rollout.

The diagram below shows an example of an incorrect rollout, where traffic is shifted independently across state machines and downstream Lambdas:

Version pin downstream dependencies

To ensure that changes to child resources are rolled out gradually and safely, we made version pinning mandatory for all downstream resources. This prevents changes from reaching production immediately and ensures that each Step Function version executes against a consistent set of dependencies.

Version pinning also plays a key role in driving the gradual rollout process as new versions of downstream resources are introduced and referenced by a new state machine version.

IAM resources are not versioned

This is easy to overlook during rollouts and has caused real production failures for us in the past.

While Step Functions can be versioned and aliased, IAM roles and policies have no built-in versioning. All versions of a Step Function share the same IAM configuration.

This has an important implication. Any IAM change must remain fully backward compatible with all Step Function versions that are still receiving traffic. Removing permissions, even if they are no longer required by the latest version, can cause older executions to fail with AccessDenied errors during a rollout.

Once we were able to version workflows and shift traffic safely, we formalised 3 non-negotiable rules:

1. Pin all downstream resources
2. Shift traffic only at the entry point
3. Keep IAM changes backward compatible

Violating any of these rules risks breaking production in the middle of a rollout.

Meet Tweety

To orchestrate the release process, we built Tweety, our in-house release manager implemented as a Step Function. The name is inspired by the Looney Tunes canary, reflecting Tweety’s role as an early warning system during deployments.

The idea behind Tweety is deliberately simple. If you have used Lambda canary deployments via CodeDeploy before, the model will feel familiar: publish an immutable version, shift traffic gradually, monitor health, and roll back automatically when signals degrade.

Tweety brings that operating model to Step Functions, packaged in a way that fits our Terraform workflows, our dependency-pinning rules, and the realities of rolling out orchestration logic safely.

It handles rollout between versions, monitors health checks, and automatically rolls back if alarms are triggered:

To kick off the gradual rollout process, we invoke Tweety whenever a Step Function version changes. The invocation is triggered from Terraform using the AWS CLI, with user-defined inputs such as traffic shift percentage, rollout interval, and monitoring thresholds:

resource "null_resource" "run_gradual_rollout" {
  triggers = {
    sfn_version = var.version_arn
  }

provisioner "local-exec" {
    # Only trigger deployment if the state machine version is not equal to 1
    command = <<EOT
      if [ ${var.version_arn} != ${local.current_version_arn} ]; then 
        aws stepfunctions start-execution \\
          --state-machine-arn ${var.gradual_rollout_stepfunction} \\
          --input '${jsonencode(local.rollout_payload)}'
      fi
    EOT
  }
}

Once the rollout is initiated, Tweety runs two jobs in parallel:

• Traffic management This job determines the next traffic weight and updates the Step Function alias accordingly.
• Health checks In parallel, the Health check job continuously evaluates system metrics against team-defined thresholds. If any degradation is detected, it stops the rollout, initiates a rollback, and alerts the team that the release has failed.

Building our own gradual rollout process allowed us to fully customise it to our needs, from Slack notifications to support for non-standard deployment strategies.

DevX as a functional requirement

Ease of adoption was a first-class concern when designing the rollout solution. Gradual rollouts had to be simple to adopt and should not require engineers to understand the internal workings of the rollout engine. To achieve this, we wrapped the required resources in a Terraform module that hides the complexity and manages the gradual rollout process end to end.

From an engineer’s perspective, deploying with gradual rollouts feels no different from any other change, except that it is safer. Engineers provide a small set of inputs to a Terraform module, and the rest is handled automatically: versioning, alias shifts, monitoring, rollback, and Slack updates.

The visual execution graphs and built-in logging provided by Step Functions also help engineers better understand the deployment process and simplify troubleshooting when things go wrong.

Closing thoughts

Once the gradual rollout mechanism was in place, it was quickly adopted across our engineering teams. A key enabler was the set of custom Terraform modules we built, which made safe rollouts easy to apply and operate consistently.

By adding safety guardrails to Step Functions, we enabled engineers to move fast and deploy with confidence, even during peak season, without putting production stability at risk.

While AWS provides powerful primitives such as Step Function versions and aliases, building a safe rollout workflow still requires stitching those pieces together with operational guardrails. Tweety is our attempt to codify those best practices into a reusable tool.

We are now open-sourcing the Terraform module behind Tweety so other teams can adopt and adapt the same approach in their own AWS environments.

To get started, you can find our open-source module here: https://registry.terraform.io/modules/wegift/tweety-sfn/aws/latest

Stop Breaking Production: A Step Function Rollout Strategy That Works was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Just like for a lot of software businesses out there, most of Runa’s services are based on a Cloud-based microservices design with AWS being our exclusive cloud provider. To build systems capable of processing hundreds of thousands of events in seconds in a reliable manner, we have additionally adopted the approach to favour serverless AWS services in our architecture.

In a serverless model, developers can build and deploy applications without having to manage the underlying infrastructure. The cloud provider will dynamically allocate resources based on usage, allowing our teams to develop and deploy faster while focusing on business logic.

AWS offers various serverless services across different domains, including API management (API Gateway), compute (Lambda, App Runner), event-driven integration (EventBridge, Step Functions, SQS, and SNS), and data storage (S3, DynamoDB, and Aurora Serverless).

As part of the Billing team, a sub-team of engineering, responsible for all things related to financial reconciliation and invoicing of customers and brands we work with, I have worked extensively with most of these over the past months at Runa.

Although on paper, these services might seem like a perfect solution to most problems, and they generally do provide a great developer experience, it’s worth noting that it hasn’t always been smooth sailing. There are trade-offs and pitfalls to navigate along the way.

The Common Challenges With Being Distributed Over Many Cloud Services

It is well-known that microservices increase the operational overhead and complexity of the system, making it more challenging for developers to follow and work with. This is no different in the Cloud, although AWS’s offering of supporting tools, combined with our preference for serverless, makes it significantly easier.

One of the main challenges with having the system components broken down into various Cloud services is ensuring adequate observability and monitoring — tracing a request end-to-end across a Step Function, which might invoke a Lambda, which in turn interacts with other services… although some supporting AWS services such as X-Ray and CloudWatch assist with logging and tracing these also have to be integrated with our internal observability framework and third-party tools like Datadog (and how we go about it deserves its own article 🙂) to achieve a reliable and comprehensive alerting and monitoring system as well as more manageable debugging of issues.

Some Stumbling Blocks Associated With Specific Services

Over the years, we have also encountered some limitations and complications linked to specific AWS services. For example, some commonly occurring issues with “phantom” requests in API Gateway and how we addressed them were described by my colleague in this article.

Last year, we were forced to migrate our Quantum Ledger Database (QLDB) to another service, as AWS announced its deprecation, with the shutdown scheduled for July 31, 2025. As a managed ledger database service, it offered a cryptographically verifiable transaction log, making it a perfect choice where the immutability of records and audit functionality were necessary. Apart from serverless support, QLDB offered integration with our current infrastructure (Step Functions, API Gateway) and fit our data model for the payouts system, in which a record is stored in a document format with support for nested objects and document history out of the box. To obtain a permanent, immutable history of all record changes for audit purposes, which QLDB offered with a different database service, those transaction logs can still be tracked and permanently stored, but only with the support of other services, for example, CloudTrail or S3.

Serverless Favorites

At Runa, one of the main domains in our infrastructure where the serverless services’ dominance is particularly visible is computing and workflow orchestration. Lambdas and Step Functions form the backbone of our infrastructure and underpin many of our services. They are used to power everything from real-time API calls to scheduled background processes and are widely adopted by all our teams. Lambdas let us run lightweight, event-driven workloads on demand, while Step Functions orchestrate more complex workflows across multiple services. Since each service deserves a closer look, this article will focus solely on AWS Lambda.

Lambdas are a popular choice when you want to run code without having to worry about how it will be run — AWS will provision and manage servers for you, and even automatically scale based on traffic demand. Additionally, you only pay for the amount of time the code is running (the charges are based on the number of requests and the time it takes to run them).

Lambda natively connects to AWS event sources (SNS, SQS, API Gateway, EventBridge, etc.), making it an ideal component for irregular or bursty workloads. These services act as managed ingress points that absorb traffic spikes, buffer events, and invoke Lambda functions only when work is available. As a result, there is no need to provision or scale compute capacity in advance: Lambda is invoked reactively in response to events, and automatically scales concurrency to match the incoming workload. Some of our workloads run on an infrequent schedule and may consume fluctuating amounts of resources — for example, our invoicing cycle runs only once a month on the first day. During the billing cycle, a large amount of resources will be consumed due to a high volume of data to be processed, but Lambda functions will scale automatically to the workload, which makes it a perfect choice since we can run these processes without having idle infrastructure outside of that time.

Lambda’s features help us to create a system that is not only more maintainable but also resilient — the function automatically retries failed asynchronous invocations and provides high availability out of the box. When integrated with durable event buses such as SNS and SQS, Lambdas can offer a decoupled and fault-tolerant workflow — for example, our invoicing system contains several Lambdas which consume and produce events that get processed along the pipeline with queues and topics keeping the services decoupled. Each Lambda is responsible for its own logic, as AWS best practices recommend writing single-purpose functions. This separation offers additional resilience under unpredictable loads; for example, a queue can act as a buffer if the downstream Lambda is experiencing a temporary slowness in processing the events.

Things to Consider When Working With Lambdas

It is worth mentioning that there are also some well-known drawbacks to using Lambdas. These include vendor lock-in with heavy reliance on AWS-managed services (something which is a lesser concern for us, but can be tackled with hexagonal architectures and standard runtimes) or execution and resource limits of the Lambdas, which need to be considered when developing a service. AWS limits the executions to 15 minutes per invocation, which makes it unsuitable for long-running jobs unless they are broken down into subtasks and split between multiple Lambdas or a Step Function coordinating each task separately. Additionally, as there are resource caps on Lambda memory and ephemeral disk storage, they might not be a suitable option for resource-intensive tasks.

Another well-known problem with Lambdas is what’s known as a cold start, which occurs when a function hasn’t been used recently and the first invocation requires AWS to spin up a new execution environment. This adds additional latency to the first request and can be particularly noticeable for infrequently used, latency-sensitive paths. In practice, however, this impact can be reduced significantly. For critical APIs, provisioned concurrency allows us to keep execution environments warm and predictable. Provisioned concurrency means that (if provisioned to the right scale) we always have a lambda that is warm and ready to compute. It is great for:

- When we have a critical but low traffic endpoint that requires high performance
- When we have a lambda that’s package size results in significant startup time

We can also mitigate cold starts by keeping deployment packages small, avoiding unnecessary dependencies, choosing lightweight runtimes for hot paths, and deferring expensive initialisation logic until it is actually needed. For workloads where some startup latency is acceptable, like in our invoicing lambdas, cold starts become largely irrelevant, allowing us to benefit from Lambda’s scaling and cost model without compromising user experience.

Our biggest concern in the example of the invoicing workflow was implementing an acceptable and reliable observability of that system. Having multiple distributed Lambdas, which additionally interact with other services, adds to the complexity and makes it harder to monitor and debug. We were able to overcome these challenges using centralised monitoring, which integrates AWS services such as X-ray, third-party tools (Datadog) and in-house solutions.

Final Thoughts

While AWS services have enabled us to develop and scale quickly, our journey has also revealed the challenges that come with depending on a constantly evolving cloud ecosystem. From troubleshooting service-specific issues to navigating the forced deprecation of QLDB, we’ve repeatedly had to adapt our systems and revisit design choices. Operating across many cloud services amplifies this complexity, making flexibility and resilience in system design essential to absorb change without disrupting the platform.

Despite these challenges, this approach has been the right one for us: managed services have allowed our teams to focus on product and business logic rather than infrastructure concerns, while still supporting rapid growth and highly variable workloads. By deliberately designing for loose coupling and clear boundaries between services, we’ve been able to evolve or replace underlying components when necessary without slowing down delivery.

These experiences highlight an important reality of building serverless platforms at scale: the advantages of managed services come with trade-offs, including limited control and potential service discontinuity. For us, the key lesson has been to build with flexibility in mind — designing integrations and data flows in a way that allows us to pivot when services change, without disrupting product delivery.

Within that context, AWS Lambda remains the engineering team’s compute favourite. It enables us to move faster while absorbing a significant amount of operational burden through automatic scaling, per-invocation billing, native retries for asynchronous invocations, and easy integration with AWS event sources. This makes it particularly well-suited for event-driven pipelines that must handle large, unpredictable bursts — such as month-end invoice processing — without provisioning or maintaining idle infrastructure. Lambdas are not a silver bullet, but for us, their limitations are well understood and manageable when paired with deliberate architectural and operational practices.

The Pains of the Cloud: Deprecated Services and why Lambdas are King was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Terraform recently added the datadog_workflow_automation resource to its Datadog registry. This new resource enables teams to provision and manage Datadog workflows using Terraform, giving teams the ease and flexibility of infrastructure-as-code for observability automation.

At Runa, we have found workflows to be especially useful in operations management, alert optimization, and incident flow automation. However, managing these workflows solely via the Datadog UI quickly becomes difficult. Changes are hard to track, reviews are not centralized, and small edits can easily get lost. Combined with the challenge of maintaining workflows that serve multiple teams with different priorities, these gaps become painful and error-prone.

This why I have spent the past few months migrating two of our existing operations management workflows to Terraform. In this post, I will take you through the key technical challenges, the solutions, and the pitfalls to avoid when adopting Terraform for Datadog workflows.

Key Technical Challenges and Solutions

Complex JSON Structure Management

The most significant challenge was managing the complex JSON structure required for the spec_json parameter. Datadog workflows have intricate step definitions with nested parameters, outbound edges, and display configurations.

Solution: Export the workflow from the Datadog console as Terraform code.

The Datadog console provides an “Export” option for existing workflows that outputs a complete Terraform representation of the workflow. This gives you the exact JSON format that Datadog expects, minimizing trial-and-error. From there, you can refactor the spec_json using jsonencode() and Terraform locals to make it more modular and maintainable.

Steps to export:

1. Go to your workflow in the Datadog console.
2. Click on the Export option in the menu bar.
3. Select the Terraform option.
4. Copy the generated spec_json content.

Dynamic Step References and Dependencies

One of the trickiest aspects was handling dynamic references between steps that are conditionally included. For example, one workflow includes a call to another workflow only when a specific criterion is met, which means that the outbound edges (the “links” between steps) must reference steps that may or may not exist.

Solution: Use Terraform locals to define conditional outbound edge maps. This approach allows you to model branching logic without duplicating JSON structures.

locals {
  check_monitor_outbound_edges = var.enable_notifications ? [
    {
      "nextStepName" : "SendSlackNotification",
      "branchName" : "true"
    }
  ] : []
}

{
          "name" : "CheckMonitorStatus",
          "actionId" : "com.datadoghq.core.if",
          "parameters" : [ ... ],
          "outboundEdges" : local.check_monitor_outbound_edges,
        }

AWS Services Integration and Connection Management

Both workflows integrate with multiple AWS services, which require authentication. On the Datadog console, AWS connection is easily managed by creating an appropriate connection for each environment. However, Terraform is environment agnostic, so you must supply the AWS connection for the correct environment being deployed.

Solution: Maintain a mapping of pre-existing environment-specific AWS connections:

locals {
  aws_connection_map = {
    "sandbox"    = { connection_id : "12345", label : "SAMPLE_CONNECTION_1" }
    "dev"    = { connection_id : "67890", label : "SAMPLE_CONNECTION_2" }
  }
}

Dynamic workflow naming

Not a challenge, but worth mentioning to avoid all provisioned workflows having the same name.

Solution: Use Terraform variables to parameterize workflow names

resource "datadog_workflow_automation" "my_workflow" {
  name        = "My-workflow-${var.team_name}-${var.env}"
}

JavaScript Code in Workflow Steps

Many workflow steps include JavaScript code for data transformation. Managing this code within Terraform strings can quickly become messy and hard to read.

Best Practices:

• Write and test your code directly in the Datadog workflow JavaScript step editor or any IDE.
• Maintain proper indentation and commenting.
• Export the working code as JSON from Datadog or IDE once validated.

{
  "name" : "ParseOutput",
  "actionId" : "com.datadoghq.datatransformation.func",
  "parameters" : [
    {
      "name" : "script",
      "value" : "// parse JSON string \nlet parsedFaults = $.Steps.DescribeOutput;\nlet trigger_monitor_url = $.Source.monitor.url;\n\n ... // rest of logic here"
    }
  ]
}

Complete Example: Putting It All Together

Here’s a sample workflow that demonstrates the patterns discussed above in a complete workflow.

locals {
  # Create conditional notification steps
  notification_steps = var.enable_notifications ? [
    {
      "name" : "SendSlackNotification",
      "actionId" : "com.datadoghq.slack.send_simple_message",
      "parameters" : [
        {
          "name" : "teamId",
          "value" : "MYSAMPLETEAM"
        },
        {
          "name" : "channel",
          "value" : "#alerts"
        },
        {
          "name" : "text",
          "value" : "Alert triggered: {{ Source.monitor.name }}"
        }
      ],
      "display" : {
        "bounds" : {
          "x" : 0,
          "y" : 432
        }
      }
    }
  ] : []

  # Create conditional outbound edges 
  # If notifications are enabled, route to notification step; otherwise, workflow ends
  check_monitor_outbound_edges = var.enable_notifications ? [
    {
      "nextStepName" : "SendSlackNotification",
      "branchName" : "true"
    }
  ] : []

  sample_workflow_name = "Sample-workflow-${var.team_name}"
  aws_connection_map = {
    # AWS connection IDs for different environments. Already created on datadog console
    "sandbox" = { connection_id : "abcd", label : "MY_SAMPLE_AWS_CONNECTION_1" }
    "dev"    = { connection_id : "abcdefg", label : "MY_SAMPLE_AWS_CONNECTION_2" }
  }
  aws_connection_id    =  local.aws_connection_map["sandbox"].connection_id
  aws_connection_label = local.aws_connection_map["sandbox"].label
}

variable "enable_notifications" {
  type        = bool
  description = "Enable Slack notifications in the workflow"
  default     = false
}

resource "datadog_workflow_automation" "sample_workflow" {
  name        = local.sample_workflow_name
  description = "Sample workflow demonstrating Terraform patterns for Datadog workflow automation"
  tags        = var.workflow_tags
  published   = true

  spec_json = jsonencode(
    {
      "triggers" : [
        {
          "startStepNames" : [
            "CheckMonitorStatus"
          ],
          "monitorTrigger" : {}
        }
      ],
      "steps" : concat([
        {
          "name" : "CheckMonitorStatus",
          "actionId" : "com.datadoghq.core.if",
          "parameters" : [
            {
              "name" : "joinOperator",
              "value" : "or"
            },
            {
              "name" : "conditions",
              "value" : [
                {
                  "comparisonOperator" : "eq",
                  "leftValue" : "{{ Source.monitor.overall_state }}",
                  "rightValue" : "Alert"
                },
                {
                  "comparisonOperator" : "eq",
                  "leftValue" : "{{ Source.monitor.overall_state }}",
                  "rightValue" : "Warn"
                }
              ]
            }
          ],
          "outboundEdges" : local.check_monitor_outbound_edges,
          "display" : {
            "bounds" : {
              "x" : 0,
              "y" : 0
            }
          }
        }
      ], local.notification_steps),
      "handle" : local.sample_workflow_name,
      "connectionEnvs" : [
        {
          "env" : "default",
          "connections" : [
            {
              "connectionId" : local.aws_connection_id,
              "label" : local.aws_connection_label
            }
          ]
        }
      ],
      "inputSchema" : {
        "parameters" : [
          {
            "name" : "MonitorURL",
            "type" : "STRING",
            "defaultValue" : "sample_workflow"
          }
        ]
      }
    }
  )
}

Critical Pitfalls to Avoid

1. Permissions & Ownership

Pitfall: Workflows created via Terraform are owned by Terraform, not by individual users. Standard-role users cannot publish or edit them directly. While this is not necessarily a problem, it could become a blocker in some situations.

Impact: Teams may be unable to manage workflows after they’re deployed.

Solution: Have an admin grant users permission to publish/unpublish workflows.

2. Step Function Limitations

Pitfall: Datadog workflows currently have limited capabilities with Express Step Functions.

Impact: Workflow may fail on step functions-related steps.

Solution: Prefer Standard Step Functions and document this limitation for your team.

3. Datastore Management

Pitfall: Datastores cannot be provisioned via Terraform, and workflows need explicit permissions to access them.

Impact: Workflow may fail due to missing datastore access.

Solution:

• Create your datastore in the Datadog console and reference it by ID in Terraform.
• Grant Terraform a Manager role in the datastore:

1. Go to the Datastore page in Datadog.
2. Click the Settings icon.
3. Select Edit Permissions
4. Add Terraform with Manager role

4. AWS Connection Management

Pitfall: AWS connections are environment-specific and must be pre-created in Datadog.

Impact: Workflow will fail if connections are missing or incorrect.

Solution:

• Create AWS connections for each environment in the Datadog console and reference them by ID in Terraform.

1. Click Actions in the Datadog sidebar.
2. Select Connections
3. Click on the + New Connection button at the top of the connections page.
4. Select AWS from the list of possible integrations.
5. Complete the form, then copy the generated IAM policy statement (the statement only shows at creation of the connection).
6. In the AWS console, create a new role and attach the generated policy statement to it.

Benefits Achieved

1. Team Autonomy: Each team can own and modify their workflow variations independently.
2. Environment Consistency: Identical workflows can be deployed across environments with environment-specific configurations.
3. Infrastructure as Code: Workflows are now part of the infrastructure, enabling automated deployments and rollbacks.
4. Documentation: The Terraform code serves as living documentation of the workflow logic.

Terraform support for Datadog workflows is a game-changer. It brings version control, consistency, and automation to what used to be a manual, UI-driven process. While the JSON structures can be complicated to set up, the long-term benefits outweigh the initial setup cost.

Further Reading

For more details on Datadog workflows, here are some resources:

• Datadog Workflow Automation Terraform Resource
• Terraform Datadog Provider Documentation
• Datadog Workflow Automation Guide

Terraforming Datadog Workflows was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

When we talk about GitLab pipelines, we usually think of continuous integrations and deployments — jobs that runs tests, builds artifacts, deploy services. Pipelines are essentially automated tasks that run in response to an event or condition, whether that’s a merge request, a push to a branch, or even a defined schedule.

Once we look beyond continuous integrations and deployments, pipelines open up an interesting world of possibilities. With a little creativity, they can become your productivity buddy and can keep team workflows in top shape.

At Runa, we gave a creative sparkle to scheduled pipelines and came up with the Runa Review Bot; a friendly reminder that sends periodic summaries of all open merge requests from our team to our Slack channel.

The problem we wanted to solve

• The Merge Requests(MR) sits idle until the next stand-up, or when the author finally drops a “Hey, can someone review my MR?” reminder.
• Reviewers open a large MR late in the day (say, at 6 PM) and instantly regret it. I have had a lot of such experiences, and I am sure you do too.
• Someone’s waiting on a code review but does not want to nag people on Slack.

These overall result in delays, context-switching, and often means reviews happen later than they should on something that could have been merged sooner.

Why MR Size Matters?

Not all MRs are created equal. By showing additions and deletions counts in the Slack summary, we give reviewers the choice.

Developers can get a sense of what’s coming and whether they have the bandwidth to review it, without even opening the MR. The size of an MR is not necessarily linked to the time taken to review it. It is more about helping each other plan better and making sure no one gets caught off guard.

At Runa, we generally prefer small MR’s. We believe it reduce the cognitive load for reviewers, making it easier to spot issues early and provide meaningful feedback. For authors, smaller MRs are easier to test, revert and/or merge, resulting in better review cycles and quality.

The Solution we built — The Runa Review Bot

Instead of manually pinging for review on Slack, we built a Gitlab pipeline schedule that runs twice a day — once in the morning and once in the late afternoon.

Here’s What it Does:

1. Scans for open MRs in our repositories.
2. Compiles a friendly Slack message with:

• MR title, link and author.
• Short description of what the MR is about.
• MR size (additions + deletions + no. of files changed), so reviewers can decide if they have time to tackle it now or later.

3. Finally, it sends the summary to the team’s Slack channel, or DM reviewers directly using Slack webhooks.

The scheduled pipeline triggers a Python script twice a day. Under the hood, the script fetches open merge requests, compiles details and posts a message to Slack.

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
      when: always
stages:
  - review-reminder
send_open_merge_requests_to_slack:
  stage: review-reminder
  variables:
    TEAM_NAME: $TEAM_NAME
    GITLAB_TOKEN: $GITLAB_TOKEN
    SLACK_CHANNEL_WEBHOOK_URL: $SLACK_CHANNEL_WEBHOOK_URL
    TEAM_MEMBER_IDS: $TEAM_MEMBER_IDS
  script:
    - cd runa_review_bot
    - poetry install
    - poetry run python main.py

Pseudocode:

BEGIN
  Load environment variables: TEAM_NAME, GITLAB_TOKEN, SLACK_WEBHOOK, TEAM_MEMBER_IDS
  Set BASE_URL ← "<https://gitlab.com/api/v4>"
  FUNCTION get_diff_stat(project_id, mr_id):
    Fetch diffs from GitLab
    RETURN additions, deletions, files_changed
  FUNCTION get_open_mrs(user_ids):
    all_mrs ← []
    FOR each user_id IN user_ids:
      Fetch open, non-draft MRs for user_id
      FOR each MR:
        diff_stat ← get_diff_stat(project_id, mr_id)
        IF MR can be merged AND no conflicts:
          Add MR with diff_stat to all_mrs
    RETURN all_mrs
  FUNCTION format_for_slack(mrs):
    RETURN formatted message with MR title, author, link, and size
  FUNCTION send_to_slack(message):
    POST message to SLACK_WEBHOOK
  FUNCTION main():
    mrs ← get_open_mrs(TEAM_MEMBER_IDS)
    IF mrs not empty:
      send_to_slack(format_for_slack(mrs))
    ELSE:
      PRINT "No open merge requests."
  main()
END

Workaround — for when you do not want your MR to be reviewed yet!

Well, mark it as a draft. Our review bot filters out draft MRs-we know sometimes you’re not ready for feedback, and you might want to hide your “not-so-proud-yet” commits.

What’s Improved?

• Better productivity — Reviewers can plan their day around MR sizes.
• Less Slack noise — No need for repeated manual pings.
• Lower friction — Hiding unfinished MRs as drafts keeps the bot from spamming half-baked work.

Behavioral Observations

• Spikes at 9:30 AM and 4:30 PM — correspond exactly to the bot’s scheduled Slack posts.
• Immediate approvals after notifications show the bot boosted MR visibility.

Final Thoughts

Scheduled pipelines aren’t just for shipping code — they can be communication tools too. By automating MR reminders, we’ve freed up mental bandwidth and reduced the “Did anyone see my MR?” chatter.

And yes, if you’re feeling cheeky, you can still hide your giant “work in progress” refactors by marking them as drafts. The bot won’t know. 😉

Beyond Deployments: Creative Uses for GitLab Scheduled Pipelines was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

It was two years into my engineering career and I had just started a new job when I first heard the term design patterns. My brain immediately pictured fabric swatches and tailor’s chalk — patterns that help cut clothes into shape. Blame it on my aversion to certain writing styles or my less-than-stellar attention span in CSC 205 lectures back in university. Whatever the reason, I was painfully unaware at the time and had to familiarize myself with the topic fast.

A quick online search yielded some interesting articles which helped a little, but the real progress happened on the job. I had to not only learn about design patterns but also develop the skill of deciding which pattern worked best for which problem.

This article breaks down what design patterns are, the categories, and how I choose the right one to solve real problems. My goal? To give you a practical lens for your next coding challenge — so when it’s time to decide, you won’t just guess. You’ll know.

Let’s dive in! What is a design pattern?

A design pattern is a typical solution to a commonly recurring problem.

In software design, we tend to get the same problems in slightly different forms. A design pattern provides us with a blueprint that can be used to solve a class of problems in a maintainable and consistent manner.

The benefit of design patterns is that they have been tried, tested and documented by other engineers. Which means that you can focus your energy on other aspects of your system without having to figure out from scratch a problem that has been solved before.

Design patterns generally fall into three categories: Creational, Structural and Behavioural.

• Creational design patterns help to solve problems that involve creating objects (classes, methods, etc) in a flexible manner i.e at runtime

Creational pattern example — Factory method

# Selecting a payment processor 
class PaymentProcessor:
    def pay(self, amount):
        raise NotImplementedError()

class PayPalProcessor(PaymentProcessor):
    def pay(self, amount):
        print(f"Paid {amount} with PayPal")

class StripeProcessor(PaymentProcessor):
    def pay(self, amount):
        print(f"Paid {amount} with Stripe")

class PaymentFactory:
    @staticmethod
    def get_processor(method):
        if method == "paypal":
            return PayPalProcessor()
        elif method == "stripe":
            return StripeProcessor()
        else:
            raise ValueError("Unknown payment method")


processor = PaymentFactory.get_processor("paypal")
processor.pay(100)

• Structural design patterns help to solve problems that involve composing multiple objects into larger, more complex structures.

Structural Pattern Example — Adapter method

# Adapting an old logging library to a new interface
class OldLogger:
    def log_message(self, msg):
        print(f"[OLD LOG] {msg}")

class LoggerAdapter:
    def __init__(self, old_logger):
        self.old_logger = old_logger

    def log(self, message):  # new interface
        self.old_logger.log_message(message)

# Client code
old_logger = OldLogger()
logger = LoggerAdapter(old_logger)
logger.log("Adapter makes old code usable!")

• Behavioural design patterns help to solve problems of communication and responsibility sharing between objects.

Behavioral Pattern Example — Command pattern

class Command:
    def execute(self):
        pass

class SayHello(Command):
    def execute(self):
        print("Hello, World!")

class SayGoodbye(Command):
    def execute(self):
        print("Goodbye, World!")

class Greeting:
    def __init__(self):
        self.commands = {}
    def register_command(self, name, command):
        self.commands[name] = command
    def greet(self, name):
        if name in self.commands:
            self.commands[name].execute()
        else:
            print("No such command registered.")

greeting = Greeting()
greeting.register_command("hello", SayHello())
greeting.register_command("goodbye", SayGoodbye())
greeting.greet("hello")

The first step in choosing a design pattern is to understand the problem at hand.

You have to think, what exactly is the problem I am trying to solve here, in the coding sense of it?

Imagine, for instance, that you are building an app to remotely control your home automation system. The remote control has several buttons that each perform different tasks such as: turning lights on, opening the garage door, playing music, etc. What kind of problem is that? Is it about deciding what object to use to perform a task? Is it about creating one giant object made up of many smaller objects? Or is it about how different parts of the system communicate with each other?

Let’s dig deeper. What kind of design problem is this? Let’s test it against the three categories of patterns.

First, is this a creational problem? Creational design patterns solve the problem of flexible creation of objects. The problem in this case is not how to create the devices, we are struggling with triggering actions on them. So, this is not a creational problem.

Next, we need to ask if this problem can be solved by having one giant complex object that has different smaller objects. If our remote control needs to assemble several devices into one object or if the different buttons need to share internal connections, this would be a good fit but each button is independent in this case, there is no composition of objects hence, this is not a problem for the structural design pattern.

Finally, could this be a problem of communication between objects and assigning responsibilities? The remote buttons need a way to request that a certain home device perform an action, without knowing the details of how that action is performed. We are looking for a pattern that separates the action of the button “invoking a request” from the action of “executing the request” so that they can be generally independent. This seems to be a problem of communication between objects which is definitely a behavioural pattern problem.

Now, let us look for a specific behavioural pattern that best solves this problem.

There are lots of behavioural patterns which lead us to the next challenge. How to know which behavioural design pattern is best suited for this problem?

Let’s restate our remote control problem. We want our remote control button to send a request to a device which would execute the request. For the device to successfully execute the request, the request has to contain all necessary information.

Of all the different behavioural patterns, the command pattern stands out for being the closest in definition to this problem.

The command pattern allows us to turn a request into a stand-alone object that contains all the information needed to perform an action. This object can be passed along between two different objects or queued for later execution, or used to trigger an event. This standalone object is called a command.

And there we have it, the right pattern for solving the problem! Of course, not every design problem is this easy to classify, but thinking through the categories and matching the problem to each pattern’s definition is a reliable way to make the right choice.

Now that we’ve found the right pattern, let’s have a look at a simple implementation of the command pattern for our home remote control app.

from abc import ABC, abstractmethod

# Command Interface – Defines the execute() method that all commands must implement.
class Command(ABC):
    @abstractmethod
    def execute(self):
        pass


# Receiver – The actual device that performs the work (Light, Blinds, etc.).
class Light:
    def on(self):
        print("Light is ON")

    def off(self):
        print("Light is OFF")


class Blinds:
    def close(self):
        print("Blinds are CLOSED")

    def open(self):
        print("Blinds are OPEN")


# Concrete Commands – Implement specific actions (turn lights on, close blinds).
class LightOnCommand(Command):
    def __init__(self, light):
        self.light = light

    def execute(self):
        self.light.on()


class BlindsCloseCommand(Command):
    def __init__(self, blinds):
        self.blinds = blinds

    def execute(self):
        self.blinds.close()


# Invoker – The remote control that triggers the commands.
class RemoteControl:
    def __init__(self):
        self.buttons = {}

    def set_command(self, button_name, command):
        self.buttons[button_name] = command

    def press(self, button_name):
        if button_name in self.buttons:
            self.buttons[button_name].execute()
        else:
            print("No command assigned to this button.")


# Client – The setup code that binds buttons to commands.
if __name__ == "__main__":
    # Receivers
    living_room_light = Light()
    blinds = Blinds()

    # Commands
    light_on = LightOnCommand(living_room_light)
    blinds_close = BlindsCloseCommand(blinds)

    # Remote setup
    remote = RemoteControl()
    remote.set_command("A", light_on)
    remote.set_command("B", blinds_close)

    # Press buttons
    remote.press("A")  # Output: Light is ON
    remote.press("B")  # Output: Blinds are CLOSED

Choosing the right behavioral pattern isn’t about memorizing definitions — it’s about understanding what each one unlocks. Once you recognize the shape of the problem, the right solution starts to reveal itself. The more you lean into that mindset, the more intuitive these decisions become.

Further Reading

If you’d like to explore design patterns in more detail, here are some excellent resources:

• What’s a design pattern?
• Design Patterns Tutorial.

Here’s a reliable method to choose the right design pattern was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Before joining Runa, my development world revolved around .NET. Every company I’d worked for used a similar architecture, tools, and “ways of working”.

Over the years, I became very comfortable in that environment — I knew the frameworks, the patterns, and where the tricky bugs usually hid. But that familiarity also meant fewer surprises, and I was ready for something that would push me to learn again.

Then I joined Runa — a team made up of people from all kinds of technology backgrounds, bringing different perspectives and ways of thinking. In practice, our architecture is firmly rooted in AWS and serverless, but the diversity of experience on the team meant I’d be learning a lot of new approaches.

On my very first day, everything was new. And I decided to lean into that.

I was offered a choice between a Windows or MacOS machine. I could have stayed in my comfort zone since I had only used Windows professionally… but instead I went with MacOS. If I was going to shake things up, I might as well go all in.

That meant learning hotkeys, window management, and a whole new interface from scratch. Luckily, my colleagues jumped in with tips, shortcuts, and “must-know” tricks that made the transition a little less bumpy.

Little did I know, that same “lean into the new” mindset would define my first three months.

First impressions and feelings

In the first few days, I was hit with terms like serverless, lambdas, step functions, API Gateway 🤯 — all completely new to me.

It was exciting, but also a little overwhelming — a mix of new tools, rapid learning, and the occasional “Is this job for me? Am I a good developer?” moment.

What kept me moving forward was Runa’s well-structured onboarding plan. It gave me milestones to hit, resources to explore, and documentation that made the learning curve feel less like a cliff.

I still remember closing my first ticket after just one week.

It was a low-priority task — just a couple of lines of simple code for a bug fix — but it felt incredibly satisfying to contribute something tangible to the team so early on ✅.

After that first win, I knew I could keep making progress — but I also knew I’d get there much faster by learning directly from my teammates. That’s where pair programming came in.

Pair Programming

With many new domain specific acronyms, tools, and even a brand-new IDE to get used to, I quickly realised I could spend days just trying to figure things out on my own.

Instead of reinventing the wheel, I started pairing with different teammates.

Each session was like a guided tour of the codebase — seeing how others approached problems, exploring different areas of the project and architecture, and walking through diagrams of the system.

This wasn’t just about learning shortcuts; it gave me a deeper understanding of how things fit together and how I could start making meaningful contributions.

One pairing session in particular stands out. We have two repositories with almost identical names 🙈. I had been quietly wondering for some time which was used for what. My teammate walked me through the difference, what they are used for and when we should make changes to them. It was such a simple explanation, but it instantly cleared up a big source of confusion and made the codebase feel a lot less intimidating.

That kind of teamwork not only saved me time, but I could now see more clearly how everything fit together and how I could start contributing effectively.

As I got more familiar with the codebase, I also started noticing how we made technical decisions — and that’s when I was introduced to something completely new for me: technical decision documents.

Technical Decision Documents

In previous companies, technical decisions often happened in quick calls or came as top-down instructions from someone in a more senior position. We’d agree on a solution, implement it, and move on — but rarely did we document the why behind those choices.

At Runa, it works differently. Every significant technical decision is captured in a decision document. It’s a structured write-up where we outline the problem, propose different solutions, list the pros and cons of each, and record the reasoning behind the final choice.

At first, it felt like extra work — until I wrote a few of them.

I realised how valuable they are especially when you’re new and trying to understand why something was built a certain way. You can trace the thinking that led to a particular solution without having to dig through endless chats or schedule extra meetings.

My first decision doc was… rough 😅. The structure was sloppy, I missed some important details, and it took several rounds of feedback before it was finally accepted. But that back-and-forth forced me to fully understand the problem I was solving. By the time it was accepted, I knew exactly what to build — and I could jump straight into implementing my tickets without hesitation.

But even with all that documentation, there were moments when I needed answers right away — and that’s when AI became one of my onboarding accelerators.

And, of course, AI

Everyone’s talking about AI these days — how powerful it is, how risky it might be, whether it will replace developers.

For me, its biggest impact during onboarding was much simpler: it helped me understand the codebase faster.

When I was stuck trying to figure out how a feature worked, I could ask AI to walk me through it — from where it was built, to where it was used, to how it was tested.

Having that map in my head made it much easier to extend features or fix issues without feeling like I was poking around in the dark.

I also found AI surprisingly useful when working on automated tests 🤖✨. It could suggest a rough starting point, which I could then refine and adapt to fit the project.

All these tools and habits I’d picked up in my first weeks — from decision docs to pairing and even AI — were about to be tested when I joined my first large-scale initiative: a design sprint.

Design sprints and POCs

At Runa, when we face bigger, more open-ended challenges, we tackle them with design sprints. We split into smaller groups, spend a few days exploring possible solutions, and build quick POCs (proofs of concept) to see how those ideas would work in practice.

These sprints aren’t just about coding fast — they’re about experimenting, validating assumptions, and making sure we choose the right direction before committing to a full build. Decision documents play a big role here too, capturing our thinking so it’s clear why we moved forward with one idea over another.

Recently, we kicked off an initiative to solve a specific problem, and we gave ourselves the time and space to experiment. We came up with three very different solutions — not just in code, but also in the infrastructure they required. While building the POCs, we changed our minds multiple times, each time spotting where that solution might fail, become overly complex for our use case, or simply not deliver the value we expected.

It was a little exhausting to switch directions so often, but every change taught us something new. Instead of clinging to ideas, we treated each discarded POC as a step toward finding the right approach.

Working on these POCs turned out to be one of the most valuable learning experiences of my first three months. I could try out ideas in a low-risk environment, get quick feedback, and see firsthand how we make decisions that shape the future of the product.

Key Takeaways

Looking back, my first three months at Runa were a mix of steep learning curves, small wins, and a lot of “aha” moments.

I joined from a completely different tech background, learned an entirely new stack, and went from being overwhelmed by acronyms to shipping meaningful work.

Pair programming sessions, technical decision documents, AI assistance, and design sprints all played their part in helping me get there.

If I had to sum up my onboarding into a few lessons, they’d be:

• Lean into the unknown. From a completely new tech stack to tools I’d never touched, every unfamiliar step helped me grow.
• Ask, pair, and learn from others. Every teammate sees the system from a different angle, and that perspective is gold when you’re new.
• Don’t get attached to your first idea. The best solution often comes after you’ve explored (and thrown away) a few others.
• Write things down. Decision docs aren’t just for the team — they’re a gift to your future self.
• Use the tools that help you learn faster. Whether it’s AI or just good documentation, find your accelerators.

Three months ago, I was wondering if I could keep up — now I’m contributing, learning, and excited for what’s next.

The Discomfort Zone: Onboarding Into an Unfamiliar Tech Stack was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Runa, we’re always striving to build the most performant products possible. I work on the Ledger team, a sub-team of engineering. Our focus is to enable each engineering team to transfer money with speed, accuracy, and ease.

Our approach to solving this problem is to create an interface over our core ledger. This prevents our other teams from getting bogged down with the complexities of this source of truth. This has allowed them to move with speed and efficiency while keeping the core ledger accurate and intact.

We achieve this through a set of APIs. They make up the Payment Intents service. In simple terms, they are like your bank’s concept of authorizing and settling a transaction.

This service is integral to our platform’s functionality, as fundamentally, the purpose of Runa is to move value with ease. Two key requirements, besides correctness, were high reliability (> 99.999%) and a fast response time.

We are plagued by what we call “phantom” requests. These are requests that AWS API Gateway receives, but never reach the resource, or return. AWS API Gateway has an SLA of just 99.95%, meaning that for a large volume endpoint, you are almost guaranteed to have requests getting lost in the network.

Below is a high level diagram of our architectural components. We often see the request from the API Gateway never reaching the Step Function, resulting in a timed-out request.

For an example of how the various SLA’s of AWS components can be compounded, see this post: Yan Cui — Compounding SLA’s

We used the below methodologies to take our API’s from 99.99% to 99.997% (99.999% after discounting the phantom errors out of our control) across 4 million requests.

Observability

Everything here starts with observability. Observability encompasses all of our performance metrics, logging and incident management.

We must use a data-driven, scientific approach to improve performance. Often, things thought to improve performance do not, and vice versa. Without these starting benchmarks you have no indication of whether your change has had a positive or negative effect.

At a high level, the two metrics we care about for these benchmarks are p95 response time, and your 4xx/5xx error code rate. The server error/timeout reasoning is obvious. However, 4xx errors have nuances. Is your authoriser failing? Is a resource that should exist missing? Are you hitting rate limits? These should be tackled on a case-by-case basis.

Now, what tools do we use to get our benchmarks?

Performance Testing

To get where you need to be, you need to know what your starting point is. To do this, you’ll need to implement a performance test; it will give you a baseline for the responsiveness and stability under certain workloads. It is the basis for evaluating any changes we have made, to see if it has actually improved things! Without this you could worsen response time, or increased errors, which only occur under certain workloads.

At Runa, we have used Artillery extensively to perform our load testing. It’s a very simple tool to set up and use. It enables you to achieve distributed load testing and provides you with a breakdown of the stats: response time, error codes etc.

It provides you with excellent visualisations of these metrics, and allows you to compare previous runs.

We used Artillery to measure all of the incremental changes to improve response time and reliability of our Payment Intents system. It highlighted multiple race condition issues and bottlenecks that we were not aware of until extensively testing.

Visualisation

Artillery gives us multiple snapshots of performance, but it does not show us a long term experience of our API users.

For this, at Runa, we have 2 options, Datadog, or AWS CloudWatch. We use Datadog for our longer term dashboards, but whilst troubleshooting ad-hoc issues or incidents, we lean on CloudWatch.

Summary

The two tools above should give you an idea of your system’s performance. Every system has its failure and slow points, and diagnosing them accurately can be a challenge. We’ll address this challenge in the next section.

Diagnosis Tooling

Tracing

Tracing gives us detailed execution diagnostics. It shows each component of an operation, from the gateway to the database. It can help you identify points of failure, slow components, and give you a better idea about your components’ topology.

Simply enable AWS X-Ray tracing on all your components; e.g Api Gateway resource, step function, lambda, etc.

With a few lines of configuration you can get a detailed map of the service, like below, with detailed execution durations.

You can then use trace map analytics to aggregate and analyze these durations, giving you a detailed view of each component’s performance. This is essential to ensure you get the full picture of the response time of the component, and not just one offs.

Logs Insights

If you use Lambda’s in your service, Logs Insights can help extract metrics that effect performance:

filter @type = “REPORT” 
| stats 
 count(@type) as countInvocations , 
 count(@initDuration) as countColdStarts ,     (count(@initDuration)/count(@type))*100 as percentageColdStarts,
 max(@initDuration) as maxColdStartTime,
 avg(@duration) as averageDuration,
 max(@duration) as maxDuration,
 min(@duration) as minDuration,
 avg(@maxMemoryUsed) as averageMemoryUsed,
 max(@memorySize) as memoryAllocated,  (avg(@maxMemoryUsed)/max(@memorySize))*100 as percentageMemoryUsed 
by bin(1h) as timeFrame

You can generate insights like the maximum cold boot duration for your lambda over time, shown in the graph below. Combining this with the number of cold boots during that period provides a solid basis for understanding what affects your response time.

Summary

These tools help us figure out what’s slow or unreliable. With these baselines, we can take a scientific approach to tweak, test, and iterate on improvements.

Lessons Learned

Avoid lambda layers where possible

Sharing code with Lambda Layers has been advocated as a simple way to reduce the amount of code you’re deploying, and improve cold boot duration. This is a common fallacy that is explained in far more detail in this article. You should read this separately, but the headlines are as follows:

• Lambda Layers do not improve or reduce cold boot duration
• The size of the code you load is the sole biggest factor in cold start time. It doesn’t matter if it’s in your lambda or a layer.
• Lambda layers can be a significant pain in the development process, as it means you will have “automagically” appearing dependencies in your lambda deployment. Debugging these is a PITA.
• Lambda layers do not support semantic versioning, just a LATEST and incremental versions. This means that if you make a major version change that affects multiple lambdas, all of them must be ready to handle this change at the same time.
• Dependency collisions are more likely using Lambda Layers.

It’s also worth noting that the Datadog extension, which enables APM traces, can increase cold boot duration, as discussed in this article: https://medium.com/@adtanasa/the-latency-overhead-of-application-performance-monitoring-with-aws-lambda-extensions-896582da9b51

Datadog has recently released an updated version of this layer, reducing the cold start impact of APM from 800ms to 400ms. However, this is still a significant delay, which is why the ledger team has chosen to use AWS Xray tracing for our critical low-latency lambdas where necessary.

Improve your cold start duration

Package size

The sole largest contributor to cold start duration is the size of your zip file. There is little way round decreasing your cold start duration without addressing this. The graph below shows the effect of package size on cold start durations.

Steps you can take to address this:

• Check for all non-essential dependencies and remove where possible.
• Split up your internal dependencies into smaller modules that can be selectively bundled.
• Use only the sub-parts of dependencies that you need, e.g @aws-sdk → @aws-sdk/dynamodb/client.
• Move any dependencies required for development/testing to dev-dependencies.
• Major improvements can be seen by using an adapted script that trims all non-essential files from node_modules. You can find a version of this script here: https://github.com/tuananh/node-prune/blob/develop/prune.sh. We used this script to reduce our node modules size from ~14mb to ~400kb.

We recommend using https://pnpm.io/ which ensures your lambdas are packaged only with what they need, reducing your bundle size dramatically.

These steps cut our P95 time by around 60ms for our critical endpoints — nearly a 20% reduction. This improvement came entirely from faster cold boot durations, which means bursty workloads are much less affected by cold starts. You can test these bursts of request volume in artillery by configuring different phases with quick ramp-up times.

Keep your lambda outside of VPCs where possible

VPCs are necessary for accessing certain resources like Redis and Postgres clusters. In your service, it might be tempting to put all your lambdas inside the VPC for easier development, but this approach increases your cold start duration.

Of course this is not always achievable, but be conscious of this!

Use AWS Lambda Power Tuning

https://github.com/alexcasalboni/aws-lambda-power-tuning

AWS Power Tuning is a tool that you can use to optimise the amount of memory that your Lambda should be configured to use.

Memory in AWS Lambda’s is your 1 stop shop lever for increasing CPU, RAM, and network priority. Up to a point, it can yield big performance gains. See the graph below for the results of one of our Lambdas. We saved 50ms (33%!) off of our initialisation duration for a single config variable.

The graph above shows a sweet spot at 4096 (circled). There, invocation time and cost converge.

In our case, it’s not worth going much higher. The effect on performance improvements is minimal — it’s suspected that higher memory takes longer for AWS to allocate to a point.

Also worth knowing, is that the cost graph isn’t strictly true. It does not account for the decreased execution duration, below is an example from AWS over 1000 executions.

Initialise clients outside of the Lambda handler

A crucial step in improving your Lambda’s performance is to reuse clients like Redis or DynamoDB between requests. To do this, make sure to initialize them outside of the Lambda handler.

This is because AWS Lambda uses a concept called “execution context” to improve performance. When a Lambda function is invoked, AWS Lambda creates an execution context to run the function. This context is maintained for some time in anticipation of another function invocation.

For example, the simple change below saved us 80ms off our p95 response time for this endpoint.

This change will yield a small increase in your initialisation duration the first time a Lambda is booted, but the subsequent requests will benefit greatly.

Favour direct integrations over generic calls from code

Where possible, using a step function direct integration is preferred, rather than integrating via Lambda code & boto/aws-sdk. This approach reduces latency and the duration of Lambda invocations, ultimately decreasing overall cost.

For simple endpoints, using a Lambda function is often sufficient. However, when you need to orchestrate multiple steps, components, and service calls, it’s highly recommended to wrap your operation in a Step Function. AWS Step Functions give you out of the box observability, flexibility in extending your service, and, most importantly, leading on to the next point: native error handling and retries.

Retry, retry, retry

This was learnt the hard way by the Ledger team, when we first released an early version of our Payment Intents service, we were being paged more than 20 times per week. All points of failure that could fail, did. Even retrieving secrets from SecretsManager!

To fix this, the team had to put in a lot of effort, adding retries to everything and making all calls that create a resource idempotent. As mentioned earlier, for instance, API Gateway’s SLA is only 99.95%, but since our client teams need to meet their 4 9’s SLOs, we had to achieve at least 5 9's.

This means that any operation involving a network call — whether it’s an API Gateway invocation, retrieving secrets, or publishing events — must be assumed to potentially fail, so you need to catch and retry it. Additionally, your services should be designed to be idempotent, allowing reliant API consumers to safely retry as well.

Using this methodology, we went from more than 20 errors a week, down to 0 errors over a 10 day period.

Step Function retries are much more robust and easier to implement than building these features into custom Lambda functions. We highly recommend this approach for a more reliable and observable service.

Our biggest takeaway from earlier versions was the importance of this approach, and we’ve made heavy use of it in our latest implementation to handle flaky network requests. Anywhere there’s a network call, it can break, and it will.

When setting up retries, be thoughtful about your configuration. Consider what you’re retrying and what you’re hitting. For example, we retry Redis operations five times in five milliseconds, but when dealing with legacy systems, we use a more gradual approach with exponential backoff. Each case is unique.

Can you do it asynchronously?

We can’t avoid the fact that our operations are slow and take a long time due to various reasons, such as relying on legacy services, third-party processors, or other slow dependencies.

Since these operations are critical but not time-sensitive and are required to generate a response to the client, they should be completed asynchronously. This can be done by invoking a step function asynchronously or using one of the many message brokers offered by AWS. You can find a comparison of their latencies for put and delivery here: https://lucvandonkersgoed.com/2022/09/06/serverless-messaging-latency-compared/

The key is to keep critical operations simple and do as little work as possible before sending a response to the client.

By taking slow operations out of the critical path, we’ve significantly reduced our P95 latencies.

Use the correct component to store your configuration

This article provides more in-depth information than I will cover here: https://aaronstuyvenberg.com/posts/ultimate-lambda-secrets-guide

In summary, for configurations that don’t require frequent updates and aren’t sensitive, store them in environment variables instead of Parameter Store — it’s much faster and encrypted at rest.

For critical secrets that need quick retrieval, use Secrets Manager instead of Parameter Store secure strings.

Also, remember to use the retry policy mentioned earlier when fetching secrets in a Step Function. If you’re using secrets in a Lambda, consider retrieving them during initialization, outside of the handler, like we did with clients.

A note on provisioned concurrency

Provisioned concurrency, in simple terms, means having a set number of Lambdas always running, which avoids the cost of cold starts.

When configuring your Lambda, it can be tempting to use provisioned concurrency as the default option. However, this can be very expensive. For instance, a 4096mb Lambda with a provisioned concurrency of 10 would cost over £300 a month. If you have multiple Lambdas, the costs add up quickly.

If your Lambdas are serving a high volume of requests, you don’t really need provisioned concurrency — Lambdas stay active after the current execution for an unspecified amount of time, however with a high throughput you will almost definitely end up re-using already warm instances.

The CloudWatch metric ProvisionedConcurrencySpilloverInvocations can be a good indicator of whether your provisioned concurrency is not being used and you could afford to scale it back.

Where possible use resource based IAM policies

Using resource-based IAM policies instead of identity/role-based ones can significantly reduce the response time between your API gateway and compute resource, such as Step Functions or Lambdas. This simple change can produce dramatic results. For more details, see this excellent article: https://medium.com/@lancers/low-hanging-fruit-to-reduce-api-gateway-to-lambda-latency-8109451e44d6

Summary

• Start with your benchmarks, you don’t know if you’ve improved things if you don’t know where you started.
• Tracing gives you highly detailed diagnostic capabilities.
• Logs insights can give you deeper insight to your Lambda init, cold start durations, and memory usage.
• Think about your overall Lambda size & whether it’s in a due to the impact on response time.
• Initialise clients so that they can be reused across requests.
• Step functions orchestrate even small operations, thanks to their retry and direct integration capabilities.
• If it can be async, it should be async.
• Use resource based IAM policies.

Recommended resources

• AWS Well Architected — build secure, high-performing, resilient, and efficient infrastructure: https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&wa-lens-whitepapers.sort-order=desc&wa-guidance-whitepapers.sort-by=item.additionalFields.sortDate&wa-guidance-whitepapers.sort-order=desc
• Use direct service integrations to optimize your architecture: https://aws.amazon.com/blogs/architecture/use-direct-service-integrations-to-optimize-your-architecture/
• What are lambda extensions and why do they matter? https://lumigo.io/blog/aws-lambda-extensions-what-are-they-and-why-do-they-matter/
• Cold start time analysis: https://medium.com/@r_smith/aws-lambda-memory-duration-and-cold-start-time-analysis-bfcd78d2e58a
• Improving Lambda cold starts: https://www.youtube.com/watch?v=IHFJ-0_3cXY
• Improve performance of services by avoiding unnecessary parameter retrieval: https://aaronstuyvenberg.com/posts/ultimate-lambda-secrets-guide
• Low hanging fruit to reduce AWS API Gateway Latency: https://medium.com/@lancers/low-hanging-fruit-to-reduce-api-gateway-to-lambda-latency-8109451e44d6

Fast, Reliable Serverless: From 50 minutes to 50 seconds a year of downtime on AWS was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you ready to embark on a journey from npm to pnpm? If you’ve been contemplating this migration, it’s essential to understand that the process might vary depending on the complexity of your project. I have read many articles on migration from npm to pnpm but it is mostly surface-level guidance. Let’s dive into some important decisions and share experiences that can make this transition smoother.

1. Choosing the Right Time for Migration ⏰

The first and most crucial decision you need to make is to select the perfect time for the migration. If your project is large and involves multiple developers, it’s best to plan the migration during a time of reduced development activity or during a dedicated sprint. Why?

• Folder structure changes during migration can lead to conflicts if parallel development is ongoing.
• Even if you use `pnpm import` to generate a `pnpm-lock.yaml` from another package manager’s lockfile, issues may still arise. Ensuring all dependencies are the same can be tricky.
• You may need time to handle regressions and have a backup plan if things go wrong.

Choosing the right time for migration will make your life easier, reduce pressure, and minimise potential bugs.

2. Understanding the Root Cause and Deep Diving into Issues 🌱

During migration, you might encounter unexpected problems, such as unit tests failing even when all dependencies and libraries seem fine. In such situations, it’s essential to resist the temptation to simply fix the failing tests. Instead, take the time to investigate the root cause.

For example, I faced unit test failures after migrating. After some research, I discovered that an incompatible node version in my updated `pnpm` version was affecting the reactivity of Vue.js. Fixing the tests alone wouldn’t have addressed the real problem, and the actual features of the application wouldn’t have worked correctly either. Upgrading the Vue.js library along with `pnpm` resolved the issue and improved overall performance.

3. Proper Regression Testing with Backup Plans 🔄

Once you are confident with the migration through your own testing, it’s time for comprehensive regression testing with a solid backup plan.

Changing the package manager can introduce issues here and there, due to slight differences in the way each package manager operates. Instead of merging your migration branch into the main branch immediately, set up a sandbox environment and point it to the migration branch. Observe and test it over a specific timeframe, similar to your production setup. This way, you can easily revert to the current main branch if any unexpected problems arise.

Although maintaining two branches might require extra effort, it will be worth it in the long run if you’ve chosen the right time for migration as advised earlier.

4. And more…

In real-world scenarios, you may also need to address related components such as Storybook, TypeScript, and CI/CD configurations. However, the three decisions above are the most important to think about before starting your journey from npm to pnpm.

Conclusion 🚀

Migrating from npm to pnpm is indeed an adventure, and you’re right to expect more than just surface-level guidance. Understanding the complexity of your project, choosing the right time for migration, investigating compatibility issues, conducting comprehensive testing, and considering the full scope are all critical steps that contribute to a successful migration.

Don’t be discouraged by potential challenges; instead, embrace the journey with a proactive and thorough approach. By doing so, you’ll gain valuable experience and insights that will help you navigate future migrations with confidence. Happy migrating!

Migrating from npm to pnpm: A Journey of Decisions and Experiences was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Runa’s Treasury team, our mission is to make the flow, accounting, and reporting of financial value seamless and efficient for our customers with 100% confidence at scale. We are excited to announce a new FX service to provide an effortless and instant currency exchange experience on the platform.

Runa FX delivers real-time currency exchange using AWS technologies. The long-term vision of this service is to provide customers with the ability to place orders in their base currency to buy assets in different currencies while conversions are done on the fly behind the scenes. Therefore, our service needs to be scalable and highly available to handle thousands of transactions per second.

Let’s dive into some of the big-bang decisions we have made to design this service with a low-code approach.

Database

Choosing the correct database was critical in our design to ensure low latency and high throughput. The main database operation was to retrieve quotes by different parameters. We initially considered DynamoDB and AuroraDB because they align with Runa’s serverless architecture vision. However, after weighing the options, we opted for DynamoDB, as it met our requirements for low latency, high availability, and scalability and did not require ACID transactions.

Another consideration was DocumentDB, which offered flexibility with schema and queries. It is a fully managed database with minimal configuration and is highly available. Choosing this approach would encourage us to fail fast as we can always redesign the schemas per new requirements whereas in DynamoDB it is critical to get the design right from the start. However, a deeper analysis revealed that DocumentDB’s scalability was limited compared to DynamoDB for our use case. In DocumentDB, the primary database instance handles both read and write operations, which could become a bottleneck for our write-heavy service.

Ultimately, DynamoDB proved to be the best fit for our FX service, offering low latency and high scalability, while ensuring the data was highly available.

Serverless orchestration

Our team’s goal from the start was to create a new and independent FX service from scratch. We considered using Lambdas in conjunction with Simple Notification Service (SNS), Simple Queue Service (SQS), or AWS Step Functions to manage workflows between components. AWS Step Functions emerged as the clear choice for our solution because our use case did not require a complex event-driven architecture. Instead, our priority was to have a simple and sequential design that was easy to debug. Our previous experience showed us that managing a serverless system with too many Lambdas can be challenging and difficult to scale.

The power of Step Functions lies in its ability to act as a ”function orchestrator” that connects the different steps of our application. The steps can handle tasks such as decision-making, retries, parallel processes, error handling, and timeouts, which results in less code to write and maintain. Step Functions also offer direct integrations with other AWS services such as DynamoDB, S3, EC2, and Batch, allowing us to reduce the number of Lambdas we need and improving the overall latency of the FX service.

In the diagram below, we can see an example of our step function setup for exchanging currencies. It is responsible for getting a quote from a provider, performing calculations, and storing the result in our database. The ability to visualise the steps allows our team to effectively debug, monitor, collaborate, and optimise because each transition can be reviewed in more detail without diving into any code. At each step, we directly integrate with another AWS service, perform data transformations, handle errors, and set up retry logic that can be easily configured.

Deployment

The primary cloud service provider for Runa is AWS, and different teams and repositories use different Infrastructure as Code (IaC) tools for development and deployment. The four tools in active use are Terraform, SAM, Cloudformation, and the AWS CDK. We used this opportunity to review our existing deployment tools and make a change if needed. After, careful consideration, we went with Terraform for various reasons.

One of the most appealing factors was its declarative language (Hashicorp Configuration Language), which makes development easier, and its ability to use modules, making it easier to reuse code, read, and maintain. Terraform also has a wider range of providers available, which was crucial in choosing it over the other IaC tools.

Additionally, our team considered the rollback mechanism, drift detection, and deployment security when making this decision. While the AWS CDK provides a rollback mechanism, Terraform’s advantages in these other areas made it the preferred choice for this project.

Summary

In the end, our team has achieved a low-code solution with only 300 lines of Python code. We utilised the AWS serverless stack to do most of the work for us, eliminating the maintenance and operation hassle that comes with writing code. To tie everything together we used Terraform to create, manage and deploy all of our resources. This low-code solution has several advantages, including reducing the time and resources required for development and maintenance, improving scalability and reliability, and enabling faster iteration and innovation. You can find the final FX architecture below.

Why You Should Consider a Low-Code Approach was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m Sydney, a Talent Partner here at WeGift. I joined 6 months ago as hire number 2 in the Talent Team. If I’m being honest, I was nervous to get started. Fintech and payments felt slightly out of my comfort zone and something I could barely begin to explain. But I was excited. I could see that the product being built was innovative, it was smart and it was solving a problem; a problem I understood.

And I was right to be excited. In just 6 months, I’ve been promoted from Talent Sourcer to Talent Partner, and experienced huge professional growth. WeGift was only my second role in the Talent space and I originally joined as a Talent Sourcer, a similar role to my previous experience. As a Talent Sourcer, I was primarily responsible for pooling candidates, outreaching and first screening calls, before passing them over to a Talent Partner to be progressed through the process. At WeGift, however, I was encouraged to be involved in all stages of the recruitment process, starting by shadowing everything from technical screens through to presenting insights in reporting syncs. I was supported in decision making and actively given opportunities to receive and challenge feedback. Learning through osmosis was valuable for seeing the autonomy the teams have in their roles, with the freedom to be creative in their approach to solving problems and, although not always right, seeing how they owned, digested, reflected on mistakes created a comfortable environment for me to do the same. This enabled me to upskill quickly, and the transition to Talent Partner was a natural step forward.

As I look back on my journey so far, I’ve reflected on the skills that I’ve learnt which have contributed to me becoming an effective Talent Partner.

Candidate Experience is Key

We’re in an incredibly competitive market, even with what appears to be a difficult economic time on the horizon, and we find ourselves competing with equally exciting businesses who are looking to secure the same top talent. By providing a candidate experience that’s both seamless and supportive, without compromising on time, we are able to attract and retain high quality candidates, while successfully competing with these other high growth businesses. Being committed to building high performing teams by hiring great people contributes to the success of candidate referrals. Great people know great people, and a positive candidate experience gives the best possible chance of being an employer of choice for world class talent. Being a Talent Partner gives the opportunity to consistently improve candidate experience, a metric which should be closely monitored. For example, initial calls should be a two way discovery. Of course, we want to find out as much as possible about a candidate’s background, experience and motivations, but equally there must be adequate time for the candidate to ask questions and for us to share the mission and exciting future plans of the business. If it’s looking to be a good fit, a full overview of the process and each stage is given, including who they’ll be meeting and what to expect.

Build Trust with Hiring Managers

Hiring Managers are one of the most important stakeholders of a Talent Partner and the relationship that is built with them will contribute to the success of hiring. Keep them updated, organise a recurring sync meeting every other day to run through pipeline health, candidates in process, market insights and reporting metrics. Be honest with them when facing blockers with a role, present the opportunity for their support in order to work on ways to overcome them collaboratively. It’s building this trust that enables you to adapt to changing circumstances, set and commit to expectations, be honest about challenges and feel empowered to make decisions.

Actively seek learning opportunities

We are all always learning, evolving in our careers and developing skills. Coaching from a Manager, prepared resources and learning sessions are great ways to gain skills, but taking accountability for personal learning can accelerate your development. Committing to new things can be a great way to learn. The thought of doing something for the first time can be nerve wracking, but with support from peers, when given the opportunity, go for it. Take a risk, but evaluate the learnings, whether successful or not. Ask lots of questions, proactively search for different resources and expect to make mistakes — just learn from them quickly.

Embrace challenges, they promote creativity

Challenges happen on a daily basis, they keep our roles exciting and ensure we continue to learn. For example, most of us in Talent will have managed a tough role. This could be anything from niche domain expertise to specific industry experience, and after running multiple extended ‘boolean’ searches with little rewards, it can become frustrating. However, this creates an opportunity for us to think more creatively about our sourcing methods. Market mapping is a great way to extend the candidate pool, identify and engage with top talent who may not otherwise surface in searches. Attending events and listening to different seminars helps to expand networks, while also providing alternative ways to uncover and attract talent. There are a number of different groups and communities full of top professionals, so joining these and immersing into the content and conversations they provide can yield great results, both from a hiring and learning perspective.

Consistently look for improvements in the hiring process

Often recruitment processes are predefined, but that does not mean that they cannot be improved. Aim to build a process that enables you to move fast and reduce time to hire, while also allowing candidate capabilities to be effectively measured. And continually iterate this. Data provides valuable insights into the success and potential blockers of the hiring process. Utilise passthrough data to evaluate which stages are bottlenecks, seek a greater insight into the reasoning and consider altering the process to resolve this. As well as internally assessing the hiring processes, external feedback is hugely valuable too. Building a strong working relationship with your candidates enables you to gain true and honest feedback about the hiring process, which can drive meaningful change.

In summary, my transition from Talent Sourcer to Talent Partner is something I’m incredibly proud of, but I recognise that I am still very early into my talent career. I’m excited for the future, looking forward to what I can learn and the impact I can make. I hope to reflect in the upcoming years and add the above learnings, striving to be the most effective Talent Partner I can be. I’ve highlighted what I believe makes a great Talent Partner and I’d love to hear your thoughts or ideas in the comments!

If you’re looking to work alongside a team of super talented individuals and join us on our mission to build the future of digital payouts, check out our careers page!

From Talent Sourcer to Talent Partner: 5 things I’ve learnt along the way was originally published in Engineering at Runa on Medium, where people are continuing the conversation by highlighting and responding to this story.