As a member of Glasswall’s SRE team, the reliability of our products is my number one concern. Reliability of our Rebuild for Email product is especially important, as it’s responsible for ensuring our customers never receive any unsafe and untrusted email attachments. Our system works as a mail transfer agent that scans and remediates email attachments with Glasswall’s d-FIRST system, a form of content disarm and reconstruction technology.

Ensuring the reliability of such a system is no small feat. We have set up a full monitoring suite, integrated with Datadog and PagerDuty to make sure we can detect any abnormalities in the system within minutes of occurrence, but this monitoring depends on email flowing through the system. When a problem occurs it can mean customers will observe delays in mail delivery — which is a bad thing for business when prompt communication is everything in our current fast-paced environment. So, how can we detect problems and delays in the system before they affect any customer?

In this post I will describe a system we have designed and implemented that sends synthetic emails through the system at a high rate, in order to trigger our monitoring for problems as early as theoretically possible, and ensure we can remediate any problems before they start affecting our customers.

Background

We’re an SRE team, so we try to follow Google’s ‘scripture’ as often as we can. In a nutshell, this means we define ‘service level objectives’ (SLOs) for the products we manage. These SLOs are based on specific metrics for a system, and are backed by hard data. Our SLOs for Rebuild for Email can be found on this page. In this post we’re focusing on the mail latency metrics.

If mail starts getting delayed, then we start eating into our ‘error budget’ for a particular SLO. If we use up more than 100% of the error budget, then we have broken the SLO. Ideally we don’t want this to happen, so we should be alerted that we’re burning the error budget very soon after it starts happening.

Google’s SRE workbook states that artificial traffic can be useful for detecting issues before they affect your end users:

A system can synthesize user activity to check for potential errors and high-latency requests. In the absence of real users, your monitoring system can detect synthetic errors and requests, so your on-call engineers can respond to issues before they impact too many actual users.

That sounds like exactly what we need, so ‘SynthMail’ was born.

Design

This is a diagram showing the overview of the proposed solution. It will be deployed as a set of microservices into our ‘worldwide’ Kubernetes cluster, which will run continuously and send/receive email from our SaaS clusters, essentially acting as a real customer’s mail server. It consists of three components:

• Test sender — creates emails based on config and sends them on a cron schedule to a list of SMTP endpoints. Informs test receiver every time an email is sent.
• Postfix relay — receives emails from the SaaS cluster and relays them to test receiver. Postfix was chosen for reliability, because it will queue mail if for some reason it can’t be sent, and it can also be tuned to induce extra SMTP faults to further test the system.
• Test receiver — waits to receive emails after being informed about them by test sender. When an email is received, computes flight time metrics and sends them to Datadog.

It’s just a huge loop of email, with some extra metadata being sent through via MIME headers. The metrics are sent to Datadog as a histogram, so we can easily query for percentiles and compute our SLOs.

Operation

The tests are defined in a YAML config file loaded as a ConfigMap by the test sender microservice. Here is an example that sends a plain email without an attachment, as well as an email with a PNG attached:

endpoints:
  ukprod: smtp.uk.ourendpoint.io
tests:
  - name: png_attached
    cron: "0 * * * * * *"
    spec: ./specs/attach_png.yml
    endpoints: [ukprod]
  - name: no_attachments
    cron: "0 * * * * * *"
    spec: ./specs/no_attachments.yml
    endpoints: [ukprod]

As you can see, SMTP endpoints are defined at the top, and then an array of tests is given. The tests are given a name (to appear in metrics and logs), a cron schedule of how often they should be sent (both here are every minute), a path to a spec file (which describes how to generate the email), and then which endpoints to send the test to.

A spec file is essentially a YAML syntax that can be converted directly into a MIME email:

headers:
  To: [ test@synthetic.mail ]
  Date: {{ now() }}
  From: [ test@synthetic.mail ]
  X-FileTrust-Tenant: <removed-for-security>
body: This contains a PNG
attach:
- png.png

It’s a really flexible way of generating emails, and can be hooked up to a cloud storage solution to access a shared library of attachments. This is very helpful for ensuring our system is correctly rejecting/sanitising bad files. We are hoping to open source this mail generation code soon.

When the test mails are received at the end of the system, the metrics are computed and submitted to Datadog as a histogram. This lets us directly see how many mails are meeting our SLOs and how many are not.

We can then use Datadog’s SLO monitor system to track our SLOs and error budget!

Conclusion

Ultimately, the synthetic mail system has been a huge success for us. We can now be 100% sure that we can detect any delays in mail flow before they begin affecting our customers, and we can mitigate any issues before they start burning large amounts of our error budget.

It’s also had a few fringe benefits:

• We can now accurately measure delays when correlated with external events such as cloud service failovers, and deployments.
• We can get a realtime view of the system performance according to our baseline of expected mail throughput. Previously this was difficult with the aforementioned ebb and flow of traffic.
• When we do have incidents, we can use the synthetic mail system to quickly assess the degree of mailflow impact. This greatly reduces operational complexity when it comes to call-outs.

Ultimately, if you are running a SaaS platform I would encourage using synthetic data to assist in monitoring wherever possible. It’s well worth the initial time investment of developing a system to generate the data.

At Glasswall SRE, we’ve recently been investigating using Pulumi as our Infrastructure as Code (IaC) solution, to speed up infrastructure deployments, reduce errors when creating new environments, and create a more stable platform.

This blog post is the first part in a guide to how we’re running Pulumi with a self-managed backend on Azure, complete with using Azure Key Vault as Pulumi’s encryption provider. It includes a short tutorial of bootstrapping a self-managed backend and automating Pulumi infrastructure deployments using Azure Pipelines.

Future parts will go into more detail about how to use Pulumi on a self-managed backend, and specifics about how we have implemented our SaaS infrastructure on Azure with Pulumi.

The story so far

For our FileTrust for Email SaaS product, most of our infrastructure is managed through IaC. Our infrastructure is hosted on Azure, and some of the tech we use on/with it is:

• Kubernetes
• Helm
• .NET Core
• Docker
• Python
• SQL Server
• Redis
• Prometheus

The Glasswall Site Reliability Engineering (SRE) Team is responsible for managing, maintaining, and supporting our infrastructure and the services that run on it in production. Our team is about five months old — comprised mainly of new hires — having inherited the existing CI/CD workflows from contractors before our time.

We use Azure DevOps as our CI/CD solution, utilising both build and release pipelines to package and deploy our microservices and infrastructure.

Our inherited IaC solution is written in Python, using the subprocess module to call Azure CLI commands. This has worked okay up until recently, where we’ve experienced a few growing pains with it as we’ve scaled:

• Using subprocess to call out to the Azure CLI can be brittle; if the CLI changes too much over different versions then your IaC is broken. This is amplified by the fact that some infrastructure deployments are run very infrequently. Everything is fine, until you need to redeploy a critical piece of infrastructure when it breaks, only to find your IaC is also broken due to API changes since it was last run.
• Logging errors during a CD run is troublesome when using subprocess. The error causing the run to fail will be a wordy Python exception, indicating that the subprocess exited unsuccessfully, with the actual CLI error hidden away in the logs above.
• The core infrastructure is fairly rigid, with the inability to pass outputs from one section of infrastructure as inputs into another. An example of this is our virtual networking. If we wanted to change a subnet prefix, we would have to modify many instances of it being hard coded (no doubt missing some and breaking deployments), as opposed to changing it in one place and having that propagate outwards. This won’t scale as we onboard new services and products.
• It’s a bespoke system, and difficult for new joiners to learn and adjust to. Documentation is sparse, and normally only says what something does as opposed to why it is doing it. There is no standardisation in terms of what is documented, where, and how.
• Best practices for writing software are generally not followed. Most of the code utilises Python’s globals() in order to read/modify global state. This leads to an almost complete lack of pure functions, numerous errors due to side effects, and code that is incredibly difficult to debug.
• Due to using the raw Azure CLI for a lot of functionality, there’s a tonne of boilerplate in place for ensuring idempotency (being able to run something multiple times without changing the end result beyond the initial run), and in some places operations are just not idempotent.
• It only creates infrastructure. It cannot destroy it.

So, for these reasons, we were interested in seeing if we could find a more scalable solution for infrastructure deployments. Enter: Pulumi.

Halloumi?

Pulumi. In its own words, it is “Modern Infrastructure as Code”. Basically, it lets you codify your infrastructure in a commonly-used programming language (Javascript, Typescript, Python, Go, or .NET), as opposed to learning a domain-specific language for it (like Terraform’s HCL).

Pulumi is hooked up to a backend (they offer a SaaS cloud solution, but we have opted for a self-managed backend using Azure Blob Storage) which stores the state of resources as you deploy/destroy things. This allows it be run idempotently, and ensures that when you make a change to your IaC, only the minimal set of changes to your cloud infrastructure are made — speeding up build time! It also automatically detects the optimal order of operations, and even runs some steps in parallel for maximum speed.

Below are some screenshots of code. The first is our current IaC solution for creating an Azure Container Registry (ACR) in Python using subprocess, and the second is the Pulumi version, also written in Python.

As can be seen, the Pulumi version is a lot simpler; it’s self-documenting, the infrastructure configuration exists directly within the declaration, and we can pass outputs from one stage as inputs to the next using their object model.

How do you then deploy the infrastructure? Using the old version, you’d have to ensure you ran the Python script on one of our build agents as part of an Azure Pipeline, making sure you were authenticating using the correct service principal in Azure AD.

With Pulumi, you just sign in with the Azure CLI, log in to the Pulumi backend, and run pulumi up. That’s it! Pulumi stores the state of the deployed resources in your storage backend, so you can run it on any machine and the results are entirely deterministic.

It’s the exact same process for CD pipelines too, except instead of logging in with the Azure CLI, you provide service principal credentials in environment variables.

So, without any further hesitation — let’s get stuck in!

Bootstrapping a Pulumi backend

You’ll need to set up two things on Azure in order for Pulumi to work:

• A storage account and blob container to store state files.
• A key vault and encryption key to encrypt/decrypt secrets.

This guide assumes you already have:

• An Azure AD account for yourself with permissions to create/destroy Azure resources.
• Azure DevOps set up for you to create Git repositories and pipelines.
• An Azure Pipelines service connection to your Azure subscription (via Azure Resource Manager), to allow build agents to create/destroy Azure resources.

Unfortunately, we can’t use Pulumi to create the Pulumi backend. It’s not a chicken/egg situation; the backend must be created naively first before we can harness the full power of Pulumi.

We’ll make a simple shell script to bootstrap Pulumi, and then put this in an Azure Pipeline with a CD trigger so we can deploy changes.

Create a bash script with the following contents:

#!/bin/bash
# make sure we exit on any errors
set -o errexit
set -o pipefile

readonly PULUMI_LOC=westeurope

This is literally just a bash script that’ll exit correctly if any errors happen. I also define a constant for the Azure location our Pulumi backend will be hosted in. If you want to change the location of the backend then change the constant.

As you follow this bootstrapping guide, add any code snippets given to the bash script. I’ll explain them along the way.

Resource group

In order to group our Pulumi backend cloud resources together, we need to create a resource group. Add this to your bash script:

az group create \
  --name rg-pulumi \
  --location "${PULUMI_LOC}"

It just creates an Azure resource group called rg-pulumi in our Azure subscription.

Storage

Now let’s set up the blob storage Pulumi will need to store its state files in. Add this to your bash script:

if ! az storage account show --name stpulumi \
      --resource-group rg-pulumi; then
  az storage account create \
    --name stpulumi \
    --resource-group rg-pulumi \
    --location "${PULUMI_LOC}" \
    --sku "Standard_LRS" \
    --encryption-services "blob" \
    --kind "StorageV2"
fi

This creates a storage account called stpulumi if it didn’t already exist. Enclosing it in the if statement like that ensures that it’s idempotent.

Now add this:

export AZURE_STORAGE_ACCOUNT="stpulumi"
export AZURE_STORAGE_KEY=$(az storage account keys list \
  --account-name stpulumi \
  --resource-group rg-pulumi \
  --query [0].value \
  --o tsv)

az storage container create \
  --name pulumi \
  --public-access off

We’re creating environment variables to store storage account credentials for the Azure CLI to be able to access it, then we’re creating a blob storage container for Pulumi to store its state files.

Key vault

This section is optional, as Pulumi has other methods of securely storing secret information, but this is what we used. If you’re gonna be using a different encryption provider, then feel free to skip this section and the one after it.

Sometimes you want to store secret information in your Pulumi projects — this is achieved through a Pulumi ‘encryption provider’. We’re using Azure Key Vault as ours. The basic principal is that you create an Azure Key Vault with an encryption key inside it. Pulumi will use this key for encrypting and decrypting secret values when they are stored in state files in the storage backend. This is an extra layer of security, and ensures that secrets don’t fall into the wrong hands, even if your state files happen to be compromised.

Add the following to your bash script:

az keyvault create \
  --name kvpulumi \
  --resource-group rg-pulumi \
  --location "${PULUMI_LOC}" \
  --enabled-for-template-deployment true \
  --sku "standard"

if ! az keyvault key show --vault-name kvpulumi \
      --name pulumi; then
  az keyvault key create \
    --name pulumi \
    --vault-name kvpulumi \
    --kty "RSA" \
    --protection "software" \
    --size 2048
fi

This creates an Azure Key Vault in rg-pulumi with name kvpulumi, then creates an encryption key inside it called pulumi that Pulumi will use to encrypt/decrypt secret values. Again, it has an if statement to ensure idempotency, otherwise the key would be recreated each time (which would be quite bad if we wanted to decrypt existing values in the state file — the key would no longer work!).

Authorisation

Finally, we need to make sure our user account and Azure Pipelines service connection principal have the correct access policies to encrypt/decrypt using the key we just created.

For this step you’ll need to know the object IDs of both your user account, and the Azure Pipelines service connection principal.

You can get the object ID of your user account by running this command:

$ az ad user show --id {your-email-address} --query objectId -o tsv

I haven’t been able to find an easy way of getting the object ID of the service connection principal, but you should be able to find it by searching your Azure DevOps project name and Azure subscription GUID in the ‘enterprise applications’ section of Azure AD in the Portal.

Now, add the following to your bash script:

az keyvault set-policy \
  --name kvpulumi \
  --object-id {your-user-account-object-id} \
  --key-permissions get list encrypt decrypt

az keyvault set-policy \
  --name kvpulumi \
  --object-id {service-connection-object-id} \
  --key-permissions get list encrypt decrypt

Here we’re simply adding the key permissions we need to the Azure AD objects that need them. As Pulumi ‘piggybacks’ your Azure credentials when you’re running it on your machine, you need to make sure your account is authorised to perform the required operations using keys in the vault. It’s the same story with the service principal.

Deployment

That’s the bash script done! If you run it (after running az login and ensuring you’re on the correct subscription, of course), it should create the backend for Pulumi!

Please note that this script is quite naive; if you change any of the values within it, it won’t know to tear down the existing infrastructure, as there is no state file. It also doesn’t let you destroy the infrastructure you have created. To do that you’ll have to manually delete the rg-pulumi resource group.

Deploying the Pulumi backend on Azure Pipelines

Now, let’s set up a CD pipeline to deploy our Pulumi backend.

Put your bash script in an Azure DevOps Git repository, and add a file called ‘azure-pipelines.yml’ somewhere in that repo. Inside this file, put:

name: $(BuildDefinitionName)-$(Date:yyyyMMdd)$(Rev:.r)
pool:
  vmImage: ubuntu-latest
trigger:
  - master
steps:
  - task: AzureCLI@2
    inputs:
      azureSubscription: '{your-service-connection}'
      scriptType: 'bash'
      scriptLocation: 'scriptPath'
      scriptPath: 'path/to/bootstrap.bash'
    displayName: "Deploy Pulumi backend"

Make sure you put the name of your Azure Pipelines service connection in the azureSubscription field, and the path to your bootstrap script in the scriptPath field.

Create a new pipeline in Azure DevOps from an existing YAML file in your repo, and point it at your ‘azure-pipelines.yml’ file (Pipelines / New Pipeline / Azure Repos Git / Your repo / Existing Azure Pipelines YAML file).

Now, every time you make a push to the master branch of your git repo, the Pulumi backend will be deployed. You shouldn’t have to do this very often, if at all, as the infrastructure should stay fairly static.

Wrap up

That’s it for part one! We’ve managed to codify the infrastructure required for Pulumi, as well as automate its deployment. Believe it or not, this is actually the hardest part — the rest is a breeze!

Look out for part two, where I go through how to start creating cloud infrastructure with Pulumi on our self-managed backend.

The Site Reliability Engineering (SRE) team at Glasswall is responsible for making sure our SaaS FileTrust platform is as reliable as possible for our end-users, whilst ensuring this focus on reliability doesn’t hamper innovation in terms of new product features. If you’d like to know more about this, then Alex (our glorious team leader) has previously written a great article all about it.

We manage our work via a Kanban board, similar to the one described by our cloud architect, Paul, in his article. We create work items on this board, and place them under ‘epics’ and ‘features’ depending on the overarching goal of the work. Based on their nature, pieces of work on this board get placed into one of four categories:

• Planned SRE — engineering/operations work that originates from within the team, with an end goal of improving reliability, visibility, and productivity. It’s ‘planned’ because we can backlog it and allocate time in our sprints for it.
• Planned business — engineering/operations work that originates from outside the team, with an end goal of improving reliability and product value. Again, we can backlog this and work on it in a sprint.
• Incident — unplanned work that is related to addressing defects in the platform, causing unreliability.
• Toil — unplanned work that could be automated, such as release requests, replaying dead letters, and performing common log analytics queries.

If you’re a fan of ‘The Phoenix Project’, you may notice that these categories of work align closely with its ‘four types of IT work’.

Every two weeks, we have a sprint planning meeting, where we pick work from our backlog based on priority and whether we think we can get it done in two weeks.

Going by Google’s SRE book (from which our entire profession has risen), we aim to put a hard cap on the amount of unplanned work that any given team member does in a sprint, as if you don’t put in time to do the planned work and make the system more reliable, the unplanned work will quickly pile up and become intractable without a larger team.

“Eventually, a traditional ops-focused group scales linearly with service size: if the products supported by the service succeed, the operational load will grow with traffic. That means hiring more people to do the same tasks over and over again.”

On unplanned work

It’s always there. You can’t backlog it, you can’t anticipate it, and it has the potential to completely defenestrate your sprint plan.

Google’s SRE book has an entire chapter dedicated to its systematic elimination. To them, unplanned work is ‘toil’, and is characterised by work that has any of the following descriptions:

• Manual
• Automatable
• No enduring value
• Repetitive
• O(n) with service growth (scales linearly)
• Tactical

Sidenote: There really needs to be a better acronym for these. ‘RATNOM’? ‘MAN ROT’? Those don’t sound great.

To us, toil tends to be things like fixing CD pipeline failures due to outdated scripts, updating software on our clusters, carrying out manual processes in the system backend that are currently unautomated, and manually running/checking CD pipelines for production deployments.

All of these things embody almost all of the values given above. They are also mainly driven by interruption.

As mentioned previously, it’s very difficult to backlog or anticipate toil. It mainly crops up in the middle of the day when something goes wrong, or someone wants something done, and it tends to be fairly urgent. This leads to members of our team being interrupted when carrying out our planned work, losing their focus on it, and making mistakes or oversights on what is usually critical production-related work.

This is a fundamental problem with the kind of work we do; how are we supposed to perform mission-critical work to reduce toil and make the system more reliable, when whenever this work is being carried out the team is interrupted and loses their focus?

The answer? Ducks.

‘Duck duty’

If you made it past the large image of the rubber duck and are still taking this article somewhat seriously, then I congratulate you! Bear with me.

Inspired by Chapter 29 of the SRE book, and Cal Newport’s ‘Deep Work’, we’ve come up with a solution to the interrupt problem: ‘Duck duty’.

We’re currently a three-member team, and our on-call rotations last for one week each. What if the person who is on-call handles all interrupts for the entire week that they’re on call? Interrupts are almost always related to changes in the system, and changes in the system are responsible for incidents more than half of the time (some sources place this figure at around 80% of the time!). It makes sense for the on-call person to have the entire week’s context of change requests if they are to also deal with any incidents caused by them.

If you have a dedicated ‘interrupt person’, this frees up the other members of the team to get their heads down, focus, and truly achieve the ‘flow state’ of deep work — increasing productivity, and decreasing mistakes.

So, where does the duck come in? Well, Ash (one of our SREs) brought it into work one day and we thought it’d be a good idea if the interrupt person had it on their desk for the week they were on duty. It’s an easily noticable sign for everyone else in the office that this is the person to grab for any queries, and ‘duck duty’ is a lot more fun to say than ‘interrupt person’.

We’ve been using this system for just over two months now, and it’s working. Routing all interrupts through one person allows us handle the unending flow of toil while still allowing for quality engineering work to be carried out. If you’re having problems managing toil, get a duck!

Sidenote: no one’s replying to my emails asking for Glasswall-branded ducks. If you’re a rubber duck manufacturer and are reading this, please route all requests through the team member with the duck.

Look out for my next post that will go into more detail about our approach to planned work that reduces toil.