Developers are spoiled.

Every single package management system these days will allow a developer to define their project’s dependencies in a simple format, whether that’s a cargo.toml in Rust, Gemfile for Ruby apps, a pom.xml for Maven-based projects, package.json for NodeJS, composer.json in PHP…

Declaring your dependencies and their desired version in a standard, easily-parseable language allows you to track outdated dependencies, make sure you keep up to date with security updates, ensures other developers working on the same project use the same version of dependencies, lets you set up reproducible builds… The benefits are immense, and it’s universally acknowledged as a best practice in modern software development.

But when it comes to the subfield of software engineering known as Infrastructure-as-Code (also variously referred to as DevOps, or SRE, but I’ll use infrastructure-as-code through this post) though, the alternatives become rare.

It is possible, as most sysadmins are used to, to follow individual projects through mailing lists, or periodically check projects’ websites for new releases and updates. But this doesn’t scale well as you add extra dependencies, and is at its core a very manual process.

Chef probably has the most full-featured dependency management system that I’ve encountered so far, the Chef Supermarket. But it only ensures Chef cookbooks are up-to-date, and only covers the subset of the infrastructure-as-code that Chef will support. What about your cloud orchestration software? What about Chef itself?

Part of the problem is that a modern infrastructure-as-code pipeline will rely on several separate, distinct, pieces of software, depending on which part of the infrastructure is being set up: Terraform, CloudFormation, Chef/Ansible/Puppet, Helm… These incompatible systems are written in different languages and follow different versioning schemes, so an overarching dependency management system isn’t easy to come by.

But dependency management is a worthy goal, so let’s take a crack at it!

Based on the findings above and lessons from the Kubernetes project’s handling of dependencies, I’d like to introduce my attempt at solving this issue: Zeitgeist.

Introducing Zeitgeist

Zeitgeist is a language-agnostic dependency checker.

Zeitgeist will let you define your dependencies in a YAML file, dependencies.yaml, and help you ensure:

• These dependencies versions are consistent within your project
• These dependencies are up-to-date

The key concept that allows Zeitgeist to be language-independent is that dependencies are declared with a reference to another plain-text file where that dependency is defined for the underlying tool, so any infrastructure-as-code tool that lets you define your dependencies in plain-text will be supported.

For example, let’s say you build your infrastructure via Terraform, and use a pinned version of Terraform to create a Docker container that applies your infrastructure in a reproducible fashion. Your Dockerfile may look something like:

FROM alpine:3.10

ENV TERRAFORM_VERSION="0.12.12"

RUN curl -LO https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \  
&& unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip \  
&& mv terraform /bin

# Install more tools, etc

You can define this version of Terraform as a dependency:

- name: terraform   
  version: 0.12.12
  upstream:
    flavour: github
    url: hashicorp/terraform   
  refPaths:   
  - path: Dockerfile
    match: TERRAFORM_VERSION

The important information for dependency checking here is the refPaths: path refers to a file, and match is a string that will be searched in this file.

Zeitgeist will read the file at path, searching for a line matching match, and when found, confirm that the version declared on this line is identical to the version declared in your dependencies.yaml.

The upstream block is used to check if your local version is up-to-date with the project it references. Here, Zeitgeist will check the local version against Terraform Releases on Github, and let you know when an update is available.

You may then define an AMI image for an AutoScaling group in a Terraform source file that we’ll call my-asg.tf as:

resource "aws_launch_template" "foobar" {   
  name_prefix   = "foobar"   
  image_id      = "ami-1a2b3c"   
  instance_type = "t2.micro" 
}

resource "aws_autoscaling_group" "bar" {
  availability_zones = ["us-east-1a"]
  desired_capacity   = 1   
  max_size           = 1   
  min_size           = 1

launch_template {     
    id      = "${aws_launch_template.foobar.id}"     
    version = "$Latest"   
  } 
}

Now, how do you ensure this AMI stays up to date? What if a new version gets released?

Zeitgeist can also take care of that for you! Define the dependency as:

- name: my-example-ami   
  version: ami-1a2b3c   
  scheme: random   
  upstream:     
    flavour: ami
    owner: somebody     
    name: "terraform-example-*"   
  refPaths:   
  - path: my-asg.tf     
    match: image_id

And Zeitgeist will let you know when a new version of the AMI is available, based on the owner and a regexp.

Another common issue is when a dependency is used in two different contexts, and these need to be kept in sync. This was the Kubernetes project’s most important requirement when implementing their solution, and I’ve followed the same yaml structure when designing Zeitgeist.

For example, Helm (up to version 3) needs two components:

• tiller, the server-side component of Helm, deployed on your Kubernetes cluster
• The helm CLI, used locally to interact with tiller

But using different versions of helm and tiller will often result in errors, so it's better to keep them in sync.

This can be expressed as a dependency with two refPaths:

- name: helm   
  version: 2.12.2   
  upstream:     
    flavour: github     
    url: helm/helm     
    constraints: <3.0.0   
  refPaths:   
  - path: bootstrap/tiller.yaml     
    match: gcr.io/kubernetes-helm/tiller   
  - path: helper-image/Dockerfile     
    match: HELM_LATEST_VERSION

And Zeitgeist will ensure these stay in sync with each other, as well as check the upstream for updates.

Working with Zeitgeist

We have introduced Zeitgeist in our infrastructure-as-code pipeline late last year to check dependencies on deployment and give us a nightly dependency report of available updates.

Since then, Zeitgeist has been fulfilling its stated aim of helping us keep our infrastructure up-to-date, but its major benefit in my opinion has been the peace of mind gained by knowing that pinned versions of dependencies are managed, and we don’t have to worry about them silently falling behind.

Zeitgeist supports Github, AMIs, and Helm as upstreams for version checking, with more in the pipeline!

How do I use it in my own projects?

To use Zeitgeist in your own project, create a dependencies.yaml and run the zeitgeist command-line tool. Pre-build binaries are available on Github.

Zeitgeist is written in Go and hosted on Github under the Apache 2.0 license. Contributions, bug reports, and feature suggestions are very welcome!

HTTP compression is ubiquitous across the modern web as a way to trade a small amount of computing power in exchange for vastly reduced bandwidth. It is usually achieved with the gzip algorithm, so I’ll refer to HTTP compression and gzip compression interchangeably in this post.

YOOX NET-A-PORTER GROUP uses compression across the board to load pages faster, which makes users happier, and reduce bandwidth costs, which also makes the finance department happier.

When we set up a new Kubernetes-based infrastructure for public-facing websites, we started using Ambassador as a reverse-proxy for all inbound HTTP calls, and Istio as a service mesh. Under the hood, both of these projects use the open-source Envoy reverse-proxy, originally developed at Lyft.

This setup has proved to be extremely flexible and performant, and after ironing out a few kinks we’ve been really happy with it.

However, after moving a few markets over, monitoring showed that our average page size and outbound traffic had grown a sizeable bit, which couldn’t be explained by the application payload: it had to be infrastructure-related. After investigating, we found out that gzip compression was much worse on the new stack than it was on the older stack.

… And that was despite the older stack not having gzip compression enabled at all!

Enter Akamai

We use Akamai as a Content Delivery Network. All calls to our infrastructure first go through Akamai, which ensures faster delivery thanks to having PoPs much closer to clients than our servers could be, and adds extra features like protection against DDOS attacks, caching, etc.

Akamai’s coverage is… Extensive:

And, crucially… When a client supports compression and the response that Akamai receives from our origin servers is not compressed, it will go ahead and compress it before returning it to the client. However, if the response from the origin server is already compressed, Akamai will send it to the client as-is: compression twice is useless, and decompressing + recompressing every request is wasteful.

So, counter-intuitively, enabling gzip compression on our origin backends made compression worse for the clients, as Akamai had better compression performance. We could have just deactivated gzip compression altogether on our origin servers and relied on Akamai to do it, but we also wanted to reduce bandwidth between origin servers and Akamai.

So we decided to investigate: what is wrong with our own gzip compression?

Gzip measurements & parameter tuning

We wrote a small benchmark to examine the compression ratio of various reverse-proxies to see:

• If the issue we were facing was specific to Ambassador/Envoy
• Which compression options had an impact
• How the competitors stacked up

The benchmark’s source and instructions to reproduce can be found on Github: https://github.com/Pluies/gzip-comparison

And here is a representative sample:

~/projects/gzip-comparison λ ./compare.sh assets/youtube.com.html
Comparing compression performance for file: assets/youtube.com.html (all sizes in bytes)
origin, uncompressed      247602
nginx, base               45826
nginx, optimised          38924
ambassador, base          55904
ambassador, optimised     55804
envoy, base               55907
envoy, optimised          43763
apache, base              38831
apache, optimised         38972
local gzip, base          39126
local gzip, optimised     38941

Or as a graph:

The two main takeaways from this test:

• Most servers compress gzip at very similar ratios, similar to the stand-alone gzip utility
• Ambassador produced files 40% larger than most servers, even “optimised”!

Using Envoy directly, we were able to reduce the size by setting the window_bits parameter to 15, which produced compressed files about 10% larger than other servers. Still not ideal, but a sizeable improvement.

We also started using this configuration in our Istio mesh to enable compression as widely as possible.

Ambassador did not expose this setting, but it is open-source! We opened pull request #1890 which got merged within a few days. 🎉

Envoy and gzip

Even after tuning all available parameters, we are still left with a ~10 to 12% performance degradation by using Envoy, compared to virtually all other options tested.

We’ve opened Envoy issue #8448 to report back our findings, with benchmark results. This is treated as a bug, and hopefully will be fixed in the future 🤞