Gathering events from applications in the dark, legacy, “serverful” world was a straightforward task — simply embed an agent process onto each server in your infrastructure to batch up system metrics and application events (“telemetry”) before asynchronously delivering them to your metrics/logs/event provider of choice.

In an AWS Lambda world, custom agents and unsanctioned background tasks are a pattern of the past. Function execution stats are collected exclusively by AWS CloudWatch, spans are sent via UDP to an X-Ray agent embedded on every Lambda host, and logs are collected asynchronously by some background process and relayed to CloudWatch Logs. In their serverless ecosystem, AWS has replaced all of the traditional agents with their own.

In many ways, these changes can be spun as a great convenience rather than as a vendor lock-in trap. After all, for those willing to buy in completely to the AWS observability ecosystem, no more configuration is required — simply emit traces to X-Ray or logs to CloudWatch, and make do with the AWS observability interface.

Yet many of us are already using non-AWS tooling or simply demand more than what CloudWatch or X-Ray can provide. What options exist to send events to observability tooling outside of the AWS walled garden?

In the current state of the world, the available strategies boil down to either:

1. Send telemetry directly to external observability tools during Lambda execution.
2. Scrape or trigger off the telemetry sent to CloudWatch and X-Ray to populate external providers.

Spoiler: neither option is ideal.

Send

Under the hood, Lambda functions are just containers that are unfrozen (or instantiated) when invoked by an event. Once a function has returned a response, it is frozen again where it waits for either another invocation or to be garbage collected and sent to Lambda heaven.

Since Lambda functions are frozen following their invocation, any telemetry must be sent during a function’s invocation. Otherwise, events batched up between invocations or sent in asynchronous background threads can be lost when functions are garbage collected between invocations.

This leads to the big problem with the “Send” approach: any runtime telemetry must either be sent across the network as events occur or batched into a report sent at the conclusion of the function’s execution (but before returning success). The sad state of affairs is that when using the “Send” approach, users must be comfortable either losing some of their events, or pay a latency penalty to send telemetry on every single Lambda invocation.

Major latency penalty aside, the “Send” approach has many benefits:

• No additional infrastructure is required within AWS, since the telemetry is sent directly to the provider.
• Telemetry is low latency. As soon as the Lambda function returns, we can feel secure that our events are being processed by our provider.
• No additional cost is incurred to process events after the fact (as we’ll see with the “Scrape” approach below)

Scrape

The most common approach to bypass the per-invocation performance penalty of the “Send” approach is to instead “Scrape” CloudWatch and X-Ray to gather metrics/logs/traces into your provider of choice.

A commonly utilized strategy is for functions to print events or metrics to standard output. These standard output events can then be wired up to trigger a separate Lambda function which can then forward events to external logging or metrics services. Frustratingly, AWS only allows a single Lambda function to attach to any single CloudWatch “Log Group”, so users must set up a fan-out Lambda function to a Kinesis Data Stream to send events or metrics to multiple providers.

Gathering trace or invocation stats for a given function is even less elegant — providers must actively scrape the AWS API every X minutes (which if set too frequently exasperates the hidden AWS API limit). Manually scraping traces leads to another vendor lock-in trap: how do you tie together the traces sent to X-Ray with the traces from calling services? Vendors have been forced to come up with clever techniques to tie these traces together.

The “Scrape” method solves the exact opposite problems of “Send” — users are able to save latency Lambda invocations, but in turn are forced to build (potentially expensive) Rube Goldberg style machines to relay and scrape logs and traces from AWS’s products.

An Ideal World

Observability for Lambda is in a particularly annoying place — either buy in to AWS’s offerings (CloudWatch/X-Ray), take a performance hit, or build a Rube Goldberg machine.

Yet there could be a better way — rather than force users to build elaborate systems to “scrape” or IPC messages from inside Lambda functions, AWS could provide some type of UDP listening agent on each Lambda host — these agents could perform a similar function to the existing X-Ray agent, but rather than send events to AWS’s X-Ray service, forward them to a customer owned Kinesis Stream. Maybe even call them Lambda Event Streams.

In this ideal new world, users could plug and play Lambda functions (custom or vendor provided) to consume whatever log/event/metric/trace telemetry that their Lambda functions are generating. No more latency penalty on event generation, and no more Rube Goldberg machines on the backend!

Since AWS isn’t going to fix this issue over night, we’re all left with this annoying trade off: use the AWS observability tooling wholesale, send metrics during execution time, or scrape into our own systems. Personally, I’m going to stick with the “send approach” — paying the additional latency is a small cost compared to the hassle of setting up a series of points of failure inside my own infrastructure. I’d love to use AWS’s tooling for everyone, and think AWS is getting a lot closer with products like CloudWatch Insights, but I think that AWS needs to take custom CloudWatch metrics just a little bit more seriously for me to abandon third party products entirely.

• How dramatic is the VPC cold start penalty really? This has a big impact on what database technology we choose (AWS DynamoDB and AWS Aurora Serverless have “public” APIs). We’d heard that ENI Cold Start could be up to 10 seconds, is that really true? How frequently?
• How does the size of the Lambda affect cold start times? If a smaller package can reduce cold start times, it may make sense to divide Lambdas into smaller packages. Otherwise, it might make more sense to leverage “monolith” Lambdas.
• We’d read that Python can actually exhibit faster cold start times than Golang in the context of Lambda execution. As a Ruby/Golang shop we’re curious to see how the performance of our runtimes stack up.

Terminology

If you read the above bullet points without skipping a beat, feel free to skip on to the next section. Otherwise, the below vocabulary section should help provide a refresher to some of the terms used throughout the post.

• AWS Lambda — Fully managed compute as a service. AWS Lambda stores and runs code (“functions”) on demand. Functions are run inside of sandboxed containers on host machines managed by AWS and created automatically to respond to changing demand.
• Warm Start — Between function executions, containers are “paused” on the host machine. A warm start is any execution of an AWS Lambda function where an idle container already exists on a host machine and is executed from this paused state.
• Cold Start — When a function is executed but an idle container does not exist, AWS Lambda will start a new container to execute your function. Since the host machine needs to load the function into memory (from presumably S3), cold start executions exhibit longer execution times than warm starts.
• ENI — Elastic Network Interfaces represent a virtual network card and are required for Lambdas to communicate to resources inside of an AWS VPC (Virtual Private Network) to access resources like internal load balancers, or databases like RDS or Elasticache.
• ENI Cold Start — In order to communicate inside a VPC, an ENI matching the security group of the Lambda must exist on the host machine when a function is initialized. If an ENI does not already exist on the host machine, it must be created before a function can be executed. ENIs can be reused between Lambdas that share the same security group, but cannot be shared across security groups even for the same VPC. AWS Plans to fix these issues sometime in 2019.
• Box Plot — A method to visually represent numeric data by quartile. In this post outliers are shown as points outside of the box.

The Setup

We began poking around on the internet to find the answers to our questions and found several great papers and blog posts. However, there were some questions that we didn’t feel had been directly answered, or at least not in the context of the specific technologies we were using. Besides, even if we trust the results of the tests, why not verify? So we decided to take an attempt at discovering our own answers to these questions.

We wrote a small testing harness and a series of simple Lambdas to perform these tests. Really we only needed our framework to perform a series of cold and warm invocations of a given Lambda package. We are able to detect whether a Lambda is invoked cold or warm by setting a global variable “cold” to false on first execution so that while the first invocation will return “cold = true”, every subsequent invocation will return “cold = false”. We are able to force cold starts by simply re-uploading a function’s payload.

We used three different sources to measure invocation time: billing duration, observed duration, and AWS X-Ray trace statistics. Since billing duration does not include cold start time, and X-Ray trace statistics do not include ENI creation time, we use observed time for most of our tests.

Database/VPC Performance

Our first test was designed to test the performance of our most common databases from inside of Lambda. Since most of the databases we leverage live inside an AWS VPC, this test would also inherently test the performance of Lambdas created inside a VPC (this mainly being the additional time necessary to initialize an ENI on the host where our Lambda lived).

We tested six databases, Aurora Serverless, Aurora MySQL, DynamoDB, Elasticache Redis, Elasticache Memcached, and MongoDB (Atlas). All except for Aurora Serverless and DynamoDB required us to create the Lambda inside a VPC.

The results from the cold start test surprised us. We had expected ENI creation to contribute more frequently to the cold start time of the Lambdas that were created inside a VPC. Instead, cold start times seemed to be consistent across the board other than significantly more outliers on VPC Lambdas.

It became clear from these results that not every VPC cold start requires ENI creation. Rather, AWS reuses existing ENIs during Lambda execution. So while technically Lambdas in a VPC were more liable to experience an ENI Cold Start, the number of cold starts experienced was dependent on the total number of existing ENIs in the invoking Lambda’s security group.

We wanted to understand more reliably the impact of an ENI Cold Start on Lambda invocation time. So we ran the test again and forced ENI creation by recreating VPC Lambdas in a temporary new security group before each invocation. These tests more clearly highlight the heavy penalty of an ENI Cold Start, requiring at minimum 7.5 seconds and frequently more than 20 seconds!

These tests remind us to be careful when placing VPC Lambdas in hot or customer facing paths. Some potential strategies we are looking at to mitigate the impact of ENI Cold Starts are to let related Lambdas share security groups (and therefore ENIs) or placing all VPC Lambdas on a 5–10 minute timer to ensure ENIs are created ahead of execution.

Package Sizes

Our second test was designed to understand the cold start performance of package sizes across the AWS Lambda memory sizes. We’d read that the amount of compute provided to a given AWS Lambda function is based on the provisioned memory of the function.

This test was no different than the previous test, except this time we simply included large randomly generated files in the zip we uploaded to Lambda.

The results for this test were clear: big package sizes equal big cold start penalties. It follows that Lambda pulls down a function’s package to the invoking host on cold starts, but what’s less clear is why there’s such a massive penalty on larger sized packages. The simple math is that a 10s cold start for a 249MB package is a download speed of 200mbps, quite a bit below the 25gbps that an r5.metal or similar could provide. This indicates that AWS is throttling cold start bandwidth on a per-lambda basis. The lack of a performance boost on larger memory Lambdas seems to imply that this is not dependent on Lambda memory size.

Runtime

Our final test was designed to understand the cold and warm start performance of the various AWS Lambda runtimes. We chose to compare Ruby and Golang (along with Python as a control) since they’re the primary languages we leverage internally. This test executes a very simple scripts that simply returns the “cold” global variable and the root X-Ray trace ID.

The results of the test indicate that while Golang comes out on top for both cold and warm start performance, there is not a dramatic difference in execution time between the three languages. The results of this test allow us to feel comfortable allowing engineers to write Lambda functions in whichever language they feel most comfortable.

Summary

In summary some of our major takeaways include:

• Any ENI Cold Start in a hot/customer facing path will result in what we consider to be an unacceptable spike in latency. ENI Cold Starts can be mitigated by allowing shared Lambda functions to share security groups (at least until AWS solves these issues sometime in 2019).
• Lambda package size does matter significantly for cold start executions. Users should be careful to avoid hot paths with packages in the 100MB+ range.
• Provisioned Lambda memory size matters less than anticipated — at the lower end of the scale (128MB) we were able to observe heightened response times, but the impact of size on lambdas larger than 512MB was negligible.
• The difference between compiled and interpreted languages (Golang vs Ruby) turned out to be far less dramatic than we had anticipated. As a result we can feel comfortable allowing developers to write functions in whichever language they feel most comfortable.

We’re excited to run these same tests in the future to see how AWS Lambda performance is changing over time!

If you’re interested in helping us build a modern, scalable platform for the future of crypto markets, we’re hiring in San Francisco!

This website may contain links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Unless otherwise noted, all images provided herein are by Coinbase.

Exploring serverless technology by benchmarking AWS Lambda was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

This workflow represents the best of many worlds. Log entries are easy to generate, and because they can contain fields with unlimited cardinality at microsecond granularity, it’s possible to slice, dice, and aggregate these entries to diagnose the root cause of even the most complex issues.

In 2016, we reaffirmed our decision to self-manage Elasticsearch. At the time, managed solutions like Amazon Elasticsearch Service were still in their infancy (for example, IAM policy changes would require a full redeploy of the cluster) and we were reluctant to trust not yet mature third party vendor technology at the time with the potentially sensitive nature of our applications’ standard output data. We built elaborate automation to manage and blue-green deploy our Elasticsearch data, master and coordinating nodes with our existing deployment pipeline and comply with our self-enforced 30 day project.

Managing this newly automated Elasticsearch ourselves actually worked out fine for over a year. As an organization of thirty engineers serving up two moderately trafficked web applications, both our indexing (~100GB per day) and search volume (a few queries per minute) were relatively tame.

However during 2017, interest in cryptocurrency skyrocketed, catapulting our moderately trafficked web applications into the spotlight. This massive surge in traffic to our customer facing applications (60x!) had the downstream effect of an equivalent surge in downstream log entries. Earlier technical challenges contributed to this surge of new entries as engineers worked overtime to add diagnostics to our platform (see our previous scaling blog post for more detail). During this period, a single Coinbase API call could generate up to 100 separate log entries, at times resulting in terabytes of log data per hour.

The cluster became unwieldy. As the size of the engineering team increased dramatically, the combined weight of terabytes per day of logs, a monolithic cluster, and unrestricted access patterns led to outages that halted engineering productivity and created operational hurdles.

It became clear that a monolithic Elasticsearch cluster would not be sufficient for our observability needs. Some of our problems included:

• A single bad Kibana query (hello, Timelion) could cause a hiccup severe enough to affect the entire engineering organization.
• Certain issues were difficult to diagnose or prevent — Elasticsearch, Kibana and X-Pack lack the ability to analyze slow/expensive queries by user nor the controls to prevent expensive queries like multi-week aggregations.
• Many of Elasticsearch’s load-related failure conditions (like a series of very expensive queries) would leave the cluster in a disabled state even after load had entirely subsided. Our only recourse to recover from these crippled states was to perform a full cluster restart.
• Elasticsearch does not have a separate management interface, meaning that during failures we would be unable to perform basic diagnostics on the cluster. Opening a support request with Elastic requires diagnostic dumps that we were unable to generate until the cluster had recovered.
• Alerting on data stored in Elasticsearch data was not intuitive — tools like Elastalert and Watcher don’t allow engineers to interactively build alerts and were difficult to integrate with PagerDuty and Slack.
• We were forced to reduce log retention to reduce the impact of massive queries and speed up cluster recovery following failures.
• Aside from application server standard output, we were also storing and querying security data like AWS VPC Flow Logs and Linux Auditd records which required separate security controls and exhibited different performance characteristics.
• Our two largest internal applications consumed 80–90% of our log pipeline capacity, reducing the performance and retention for other smaller applications in our environment.

We chose to solve these problems in two ways:

1. Introduce a new platform to provide engineers with features that are impractical to provide with Elasticsearch (namely inexpensive aggregations, alerting, and downsampling for long retention).
2. Split our self-managed, monolithic Elasticsearch cluster into several managed clusters segmented by business unit and use case.

Metrics

Based on our challenges self-managing Elasticsearch combined with our short experiments at developing a reliable blue-green deploy strategy for Prometheus backends influenced our decision to choose a managed service for our metrics provider. After evaluating several providers, we settled on Datadog.

Unlike log platforms which store data as discrete structured events, metric stores like Datadog store numerical values over time in a matrix of time by tag/metric value. In other words, in Datadog, if you send a single gauge or counter metric 100,000 times in a 10s period, it will only be stored once per tag value combination while on a logs platform that same value would result in 100,000 separate documents.

As a result of this tradeoff, metrics stores like Datadog allow for extremely fast aggregations and long retention at the cost of reduced granularity and cardinality. Specifically with cardinality, because every additional tag value added to a given metric requires a new column to be added to the time vs tag/metric matrix, Datadog makes it expensive to breach their 10k unique tag combination per host limit.

Despite these tradeoffs, we’ve found Datadog to be a near-perfect complement to our internal logs pipeline.

• Creating and iterating on alerts is fast and intuitive. Internally, we’ve built a tool to automatically monitor and codify our Datadog monitors and dashboards to prevent accidental modifications.
• Pulling up complex dashboards with metrics from the past several months is guilt-free and near-instantaneous.
• Features like distribution metrics allow engineers to easily understand the performance from a block of code with global percentiles.
• Limitations on cardinality can be confusing for engineers who want to keep track of high cardinality values like user_id or wallet address.

Metrics vs Logs

A common topic internally, we’ve found the distinction between Metrics and Logs to be important. At a high level the tools have similar feature sets — both allow applications to send arbitrary data which can be used to combine visualizations into fancy dashboards. However, beyond their basic feature sets, there are major differences in the features, performance, and retention of these tools. Neither tool can support 100% of use cases, so it’s likely that engineers will need to leverage both in order to provide full visibility into their applications.

In short, we think that Metrics should be used for dashboards and alerts; while Logs should be used as an investigative tool to find the root cause of issues.

Securing the Datadog Agent

From a security perspective, we are comfortable using a third-party service for metrics, but not for logs. This is because metrics tend to be numeric values associated with string keys and tags, compared to logs which contain entire lines of standard output.

Datadog offers operators a series of tight integrations — if you provide the Datadog agent access to a host’s /proc directory, Docker socket, and AWS EC2 metadata service, you’ll be provided rich metadata and system stats attached to every metric you generate. Running a 3rd party agent like Datadog on every host in your infrastructure, however, carries some security risk regardless of vendor or product, so we chose to take a more secure approach to employing this technology.

We took several actions in order to gain maximum exposure to these Datadog integrations while also reducing the risk associated with running a third party agent.

• Rather than use the pre-built Docker container, we built our own stripped down version with as few optional third party integrations as possible.
• By default, the agent is opt-in — projects need to explicitly choose to allow the Datadog container.
• On every host, put the container on a separate “untrusted” network bridge without access to other containers on the host, our VPC, or the EC2 AWS metadata service. We hard code the `DD_HOSTNAME` environment to the host’s instance ID to allow the AWS integration to continue working.
• Run a special Docker socket proxy service on hosts to enable the Datadog container integration without exposing the container’s potentially secret environment variable values.

Logs

Developing a strong metrics foundation at Coinbase helped to alleviate some of the problems we experienced with Elasticsearch as many workloads naturally migrated to Datadog. Now at least when an issue did occur on the monolithic cluster, engineers had data they could fall back on.

But the Elasticsearch issues and outages continued. As engineer headcount continued to grow, logging continued to be a source of frustration. There were about 7 new engineering productivity impeding incidents in Q4 2018. Each incident would require operational engineers to step through elaborate runbooks to shut down dependent services, fully restart the cluster, and backfill data once the cluster had stabilized.

The root cause of each incident was opaque — could it be a large aggregation query by an engineer? A security service gone rouge? However the source of our frustration was clear — we’d jammed so many use cases into this single Elasticsearch cluster that operating and diagnosing the cluster had become a nightmare. We needed to separate our workloads in order to speed incident diagnosis and reduce the impact of failures when they did occur.

Functionally sharding the cluster by use case seemed like a great next step. We just needed to make a decision between investing further in the elaborate automation we’d put in place to manage our existing cluster, or reapproach a managed solution to handle our log data.

So we chose to reevaluate managed solutions for handling our log data. While we’d previously decided against using Amazon Elasticsearch Service due to what we considered at the time to be a limited feature set and stories of questionable reliability, we found ourselves intrigued by its simplicity, approved vendor status, and AWS ecosystem integration.

We used our existing codification framework to launch several new clusters. Since we leverage AWS Kinesis consumers to write log entries to Elasticsearch, simply launching duplicate consumers pointed at the newly launched clusters allowed us to quickly evaluate the performance of Amazon Elasticsearch Service against our most heavy workloads.

Our evaluation of Amazon Elasticsearch Service went smoothly, indicating that the product had matured significantly over the past two years. Compared to our previous evaluation, we were happy to see the addition of instance storage, the support of modern versions of Elasticsearch (only a minor version or two behind at most), as well as various other small improvements like instant IAM policy modification.

While our monolithic cluster relied heavily on X-Pack to provide authentication and permissions for Kibana. Amazon Elasticsearch Service relies on IAM to handle permissions at a very coarse level (no document or index level permissions here!). We were able to work around this lack of granularity by dividing the monolith into seven new clusters, four for the vanilla logs use case, and three for our various security team use cases. Each cluster’s access is controlled by leveraging a cleverly configured nginx proxy and our existing internal SSO service.

Migrating a team of over 200 engineers from a single, simple to find Kibana instance (kibana.cb-internal.fakenet) to several separate Kibana instances (one for each of our workloads) presented a usability challenge. Our solution is to point a new wildcard domain (*.kibana.cb-internal.fakenet) at our nginx proxy, and use a project’s Github organization to direct engineers the appropriate Kibana instance. This way we can point several smaller organizations at the same Elasticsearch cluster with the option to split them out as their usage grows.

Functionally sharding Elasticsearch has not only had a massive impact on the reliability of our logging pipeline, but has dramatically reduced the cognitive overhead required by the team to manage the system. In the end, we’re thrilled to hand over the task of managing runbooks, tooling, and a fickle Elasticsearch cluster to AWS so that we can focus on building the next generation of observability tooling at Coinbase.

Today, we’re focusing on building the next generation of observability at Coinbase — if these types of challenges sound interesting to you, we’re hiring Reliability Engineers in San Francisco and Chicago (see our talk at re:invent about the types of problems the Reliability team is solving at Coinbase). We have many other positions available in San Francisco, Chicago, New York, and London — visit our careers page at http://coinbase.com/careers to see if any positions spark your interest!

This website may contain links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Unless otherwise noted, all images provided herein are by Coinbase.

Logs, metrics, and the evolution of observability at Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In 2017, the world saw an explosion in the popularity of cryptocurrency, and the ecosystem’s total market cap jumped from $20 billion to $600 billion. During this time, almost every component of Coinbase’s technology was battle-tested, and it showed our team that we need to focus on the reliability and scalability of our platform, just like we do with security. At MongoDB World 2018, myself, Michael de Hoog, and Jordan Sitkin, all engineers at Coinbase, gave a talk about the lessons we learned in 2017 and how we’re scaling our platform now. You can watch the talk here or read our recap below.

Lessons learned from 2017

Our traffic patterns in 2016, the year before the explosion in cryptocurrency popularity, had been remarkably consistent. Ahead of this boom, if we had drawn a red line where we expected our platform to experience issues, we would have put it somewhere around four or five times our typical daily maximum traffic of about 100,000 backend API requests per minute.

However, in May and June of 2017, the price of ether skyrocketed and traffic exploded past that red line. There were several days during this period where we experienced sustained traffic in this red zone, during which we experienced periods of downtime.

To solve these scalability issues fast, the Coinbase engineering team started by focusing on the low-hanging fruit in our environment. We worked around the clock to perform tasks like vertically scaling, upgrading database versions to take advantage of performance improvements, optimizing indexes, and splitting out hotspot collections into their own clusters. Each of these improvements bought us headroom, but these low-hanging fruit were beginning to dry up, and traffic was continuing to climb.

During each outage, the pattern was the same: our primary monitoring platform would show a 100x spike in latency, along with a strange 50/50 split between Ruby and MongoDB time. As our primary datastore, it made sense that MongoDB time would experience this high-latency during periods of heavy traffic, but the Ruby time wasn’t adding up.

We affectionately began to refer to this issue as “the Ghost,” as our existing monitoring tools were unable to provide clear answers to some of our most critical questions. Where were these queries coming from? What did they look like? Why was there a correlated spike in Ruby time? Could the issue be originating on the application side?

Simply put, our existing monitoring services were not able to utilize all of the context available to us inside our environment. We needed a framework for answering and visualizing the relationships between our environment’s components.

We began to further instrument database queries by modifying MongoDB’s database driver to log all queries above a certain response time threshold, along with important context like the request/response size, response time, source line of code, and query shape.

Our improved instrumentation provided us detailed data that allowed us to quickly narrow in on some strange characteristics that were present, even during non-outage situations. The first major outlier we saw was an extremely large response size object originating from a device find query. These massive queries would result in a massive network load when our users would sign in to make purchases or view the dashboard.

The reason for this extremely large response size was that we had modeled the relationship between the users and device classes as a many-to-many relationship. For example, some users might have multiple devices, while some devices may have multiple users. A poor device fingerprinting algorithm had bucketed a huge number of users into the same device, resulting in a single device object with a massive array of user_ids.

To solve the issue, we refactored this relationship to simply be a one-to-many relationship, where each device maps to just a single user. The performance impact was dramatic and gave Coinbase its single biggest performance boost in 2017.

This finding illustrated the power of good monitoring. Before granularly instrumenting our database queries, this was a near-impossible issue to debug. With the new tools, it was now obvious.

Another issue we set out to solve was large read throughput on certain collections. We decided to add a query-caching layer that would cache query results in Memcached. Any single document queries on certain collections would first query the cache before querying the database, and any database writes would also invalidate the cache.

We were able to roll out this change across a number of database clusters simultaneously. The query cache was written at the ORM and driver level, which allowed us to affect multiple problematic clusters at once.

As it turns out, the massive surge in traffic we experienced in May and June was nothing compared to the surge we experienced just a few months later in December and January. With the help of these fixes and others, we were able to withstand even larger surges in traffic.

Preparing for the future

Today we’re proactively working to make sure we’re prepared for the next surge in cryptocurrency interest. While it was easy to work on these improvements during the heat of a real firefight, we needed to find a way to improve our future performance, even during lower periods of traffic. The obvious answer is to load test our environment by emulating traffic patterns at several times the levels we experienced in the past to discover where our next weak point could originate.

Our chosen solution is to perform traffic capture and playback, specifically on our databases, to generate artificial “crypto mania” on demand. For us, this method is preferable to synthetic traffic generation, since it removes the requirement to keep synthetic scripts up to date. Every time we run the suite, we’ll be sure the queries map exactly the type of traffic our application is producing, based on our captured data.

To do this, we built a tool called Capture, which wraps an existing tool called mongoreplay. After choosing a specific cluster in our environment, Capture simultaneously kicks off a disk snapshot and begins to capture raw traffic on our application servers directed to that cluster. It then saves encrypted versions of these captures to S3 playback at a later time. When we’re ready to perform the playback, another tool called Cannon, based on mongoreplay, plays back the recorded traffic to a freshly launched cluster based on the previous cluster snapshot.

One challenge we faced is how to capture all of the MongoDB traffic for a single cluster across several application servers simultaneously. Cannon solves this by opening a 10MB buffer from each capture to simultaneously merge and filter the captures.

The final result is one merged capture file which can then be targeted by Cannon toward a freshly launched MongoDB cluster. Cannon allows you to choose exactly the speed to replay the capture in order to simulate loads thousands of times larger than what we may be experiencing on any given day.

We’re just getting started with Capture and Cannon and are excited to see the types of discoveries we find as we perform these types of load tests on all of our MongoDB clusters.

One major discovery as a result of our work with Capture and Cannon comes from one of Cannon’s debug features. Cannon has the ability to inspect a specific capture file and see the first 100 messages in it. Upon inspection, we noticed something interesting:

Notice the ping commands intermingled with the finds? Turns out that the MongoDB Ruby driver was not correctly following the MongoDB driver spec and was performing a ping command (to check replica set state) alongside each query to the database. While this behavior was unlikely to have been causing our downtime related issues, it was almost certainly the cause of the “ghost-like” behavior we observed in our monitoring.

After all the effort put into tackling these challenges, we’re proud of the current state of reliability at Coinbase. The events of 2017 reaffirm that a customer’s ability to access and view their funds is critical to our ability to fulfill our goal to be the most trusted place to buy, sell, and manage cryptocurrency. While security has always been our number one priority, we’re excited to focus on ensuring that the reliability of our platform is a top priority too!

We’ve formed three separate performance and reliability specific teams to prepare for future waves of cryptocurrency enthusiasm. If you’re interested in these types of challenges, please visit our careers page!

Except where noted herein, the images displayed are from Coinbase.

How we're scaling our platform for spikes in customer demand was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

On December 11th at 22:00 PST Coinbase underwent scheduled maintenance to improve database performance. One of the techniques we utilized during this maintenance window was to distribute database load by splitting existing datasets to new clusters.

On December 12th, starting at 04:00 PST, traffic began to climb as a result of large Litecoin price movements. At 04:15 PST, two of the new clusters from the maintenance window began to experience slowness as a result of the slower network attached disks used by these instances to complete the migration. This database slowness resulted in elevated response times and high error rates. We were able to resolve these issues by 05:42 PST by failing over to high performance nodes which we had begun provisioning the night before.

Following these improvements, site traffic continued to climb, reaching all time high levels by 06:14 PST. Starting at 05:58 PST, one of our primary clusters began to experience degraded performance as a result of overwhelming query volume. We worked throughout the day to reduce pressure on this cluster by deploying application level improvements to more efficiently cache database queries. By 14:00 PST services were fully restored.

Scaling is hard, but we’re working hard to support the millions of people who are discovering cryptocurrency for the first time. We are looking to hire senior backend engineers in San Francisco, London and New York. If working on this sort of challenge excites you please see our careers page.

December 12th Post Mortem was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

December 6th

On December 6th, Coinbase continued to experience high traffic as a result of large Bitcoin price movements. At 12:22 PST, one of our primary MongoDB clusters began to experience degraded performance leading to elevated response times across all API endpoints. We attributed this slowness to saturation of the storage engine’s concurrency mechanisms as a result of poor disk performance. Earlier in the day, we began the process of upgrading the hardware of several of our most important databases (including this saturated MongoDB cluster) as part of our ongoing efforts to scale Coinbase’s backend databases. At 13:13, we successfully completed the process of upgrading this database to an instance type with significantly faster instance storage.

By 13:25 PST, services were fully restored. We are continuing the process of upgrading all of our databases to prepare for further periods of high traffic.

December 7th

On December 7th, starting at 07:10 PST Coinbase began to experience extreme levels of traffic as a result of a rapid Bitcoin price movements. Starting at 07:20 PST this traffic began to bottleneck behind one of our primary database clusters, even though that cluster had been upgraded on December 6th. This resulted in elevated response times and periods of downtime. Over the course of the next several hours, we took several steps to reduce web requests and better cache queries to this cluster, yielding small improvements in usability.

At 13:55 PST, it became apparent that the cluster was suffering from a bottleneck in memory management, which lead us to alter lower level configuration to better utilize the upgraded resources. This yielded a significant improvement in query performance under the high concurrency we had been experiencing. By 14:00 PST services were fully restored.

Scaling is hard, but we’re eager to to help solve this surge of interest! We are looking to hire senior backend engineers in San Francisco, London and New York. If working on this sort of challenge excites you please see our careers page.

December 6th-7th Post Mortem was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The objective of this post is to provide a technical overview of the challenges we faced last week and actions we took to restore services to normal as quickly as possible.

November 29th

On November 29th, at 05:03 PST, Coinbase experienced a large increase in user traffic during due to large price movements in Bitcoin and Ethereum. As a result, one of our primary MongoDB clusters experienced CPU saturation leading to elevated response times across all API endpoints. These elevated response times resulted in saturated application queues which caused intermittent 502s for customers on both the Coinbase website and mobile apps.

To mitigate the database saturation, we worked to cache specific queries and reduce the load on the affected database. At 06:24 PST we began the process of vertically scaling the affected MongoDB cluster to new instance types with double the capacity. By 08:50 PST, site performance had been restored to normal.

Traffic levels continued to climb, and at 10:49 PST one of our primary Redis servers began to experience CPU saturation as a result of a series of expensive queries. To alleviate the issue, we developed fixes to help alleviate the pressure on this Redis cluster. By 13:25 PST the fixes, combined with slightly lower traffic, reduced CPU pressure to allow traffic to be served at normal response times. By 15:00 PST services were fully restored and operating normally.

At 20:10 PST we announced planned maintenance for November 30th at 23:00 PST. The purpose of this maintenance was to increase capacity by splitting both of the affected databases (MongoDB and Redis) into three new and separate clusters.

November 30th

On November 30th, starting at 05:30 PST Coinbase again began to experience elevated response times and intermittent 502s as a result of high traffic induced CPU saturation on our primary Redis cluster. Over the course of the downtime, we attempted to improve the performance of the cluster by analyzing analyzing Redis commands and pruning unnecessary query volume.

By 08:25 PST site performance was back to normal as a result of these Redis fixes and reduced overall site traffic.

At 23:00 PST, we began our scheduled maintenance. During this maintenance, we were able to evenly split out collections from our above mentioned MongoDB cluster across several new replica sets. Additionally, we took improved Redis performance by dividing up the misbehaving cluster’s keys across several new clusters.

December 1st

On December 1st, at 04:00 PST Coinbase began to experience elevated response times as a result of network saturation on one of the newly upgraded Redis clusters. During the November 30th planned maintenance, one of the newly split-out clusters had been inadvertently created with an undersized network interface (“Up to 10 Gbps” instead of “10 Gbps”).

Over the course of the morning (04:00–13:10 PST) Coinbase worked to identify the reason for the poorly performing database. During this period, Coinbase experienced intermittent periods of downtime — each period lasting between 3 and 7 minutes. At 13:07 PST the decision was made to accept 30 to 60 minutes of full downtime starting at 13:10 PST to upgrade the poorly performing database in order to resolve the elevated response times.

By 13:40 PST the new Redis cluster with higher network capacity became operational. At 14:10 PST Coinbase became fully operational in all countries. Later that day, we resolved the network saturation by identifying a query pattern which had been returning very large response bodies.

We are hiring senior backend engineers in San Francisco, London and New York. If working on this sort of challenge excites you please see our careers page.

November 29th-December 1st Post Mortem was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

On May 22, Coinbase started experiencing sustained high user traffic as a result of large Bitcoin and Ethereum price fluctuations. Over the next week, our low points in traffic far exceeded previous peaks in the preceding months. As a result, we bumped up against a series of capacity limits which led to long stretches of performance degradation and periods of full-site downtime.

MongoDB

On the first day of the incident (May 22), unoptimized database queries and a handful of missing indices on our primary MongoDB instances compounded, leading to a surge in database queues and degraded performance across coinbase.com. We were able to restore full service and solve our database queues by building key indices and optimizing expensive application queries.

On the second day of the incident (May 23), the MongoDB query planner for our clusters began to exhibit poor query selection under high load, resulting in query times several orders of magnitude slower than identical queries run on the correct index. This poor query selection led to short, intermittent downtime periods during which database queues would rise rapidly until the poorly-planned query completed.

To resolve these query planning issues, we spent the next 24 hours (May 23 — May 24) performing major MongoDB version upgrades, bringing all of our primary clusters up to version 3.0. We had been preparing to do these upgrades for weeks, and in consultation with MongoDB experts we decided to move forward under heavy load with the hope of mitigating these query planning issues. The upgrades to 3.0 proceeded smoothly despite the load and lead to drastic improvements in query planning and performance across the board, resolving our MongoDB issues for the duration of the incident.

In the weeks since the incident, we’ve worked hard to bring our MongoDB clusters up to date, and as of June 12 have successfully upgraded all of our clusters to version 3.2 with the new WiredTiger storage engine. In total over a period of three weeks we migrated 10 MongoDB clusters from a mix of 2.4 and 2.6 and have seen dramatically increased query performance and consistency.

Redis

As traffic continued to reach new heights on the third day of the incident (May 24), we hit the capacity limit of our primary Redis ElastiCache cluster. Due to the single-threaded design of Redis, we became CPU bound on one core which we fully saturated with over 20,000 requests per second.

Over the next day, we were able to restore service through a combination of improving expensive queries, sharding existing queries across multiple logically separated new instances, and reducing the total volume of queries within the application by 40%. By May 25 at 19:00 PST we were able to restore and maintain full service across our infrastructure.

Going Forward

We are committed to improving site performance so that we can serve our customers during even the most volatile periods in cryptocurrency.

We are rapidly expanding our team, if you are interested in solving challenges like these, we’re hiring!

May 22–24 Downtime Post Mortem was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our latest project to empower engineers was to codify all of our infrastructure with the goal of allowing any engineer in our organization to propose changes in a secure, consistent manner.

Choose the Right Tool

Prior to this project, the majority of our infrastructure was codified, although using different tools and different standards. As our clouds evolved in complexity, we ran into the shortcomings of our existing tools and experimented with a variety of ways to scale our effort:

• CloudFormation — Core infrastructure at Coinbase (e.g. VPCs, route tables, VPN configuration) as well as IAM policies were codified using CloudFormation. While this originally worked for our first application of the CloudFormation templates, CloudFormation’s weak ability to plan or review changes led us to fall back to manual changes and let these templates fall out of sync.
• Terraform — Next, we started to work with Terraform for newer services, however due to shortcomings with the tool, we never committed fully and were left with a mix codified and manual configured resources, (along with a lot of state files across our AWS accounts).
• GeoEngineer — To work around the aforementioned shortcomings in Terraform, we began developing GeoEngineer. As a result, many of our core apps deployed by Codeflow had already been codified using GeoEngineer.

After evaluating the above options as well as Ansible and SaltStack, we decided to double down on our GeoEngineer building upon Terraform’s power.

Picking a Metric

Leading into this project we validated GeoEngineer with well defined 3-tier-applications. This helped eliminate obvious bugs but we still had the large task of codifying existing and bespoke cloud resources. Our largest deployments often contained new, complex services like replicated Elasticache clusters, multiple encrypted S3 buckets for temporary storage and more.

Given that GeoEngineer has insight into all of the resources, codified or not, that exist in your infrastructure, we were able to use geo status to produce the exact percentage of codified versus uncodified resources in our infrastructure. Selecting and reinforcing this metric as our north star during the project made it simple to keep track of our progress towards 100%.

Along with returning the resource codification percentage, geo status returns a list of uncodified resources which we were then able to reference as a primary source during our effort. This output made collaboration easy, since it was never difficult to determine which resources to tackle next.

As a secondary metric, we utilized CloudTrail data to visualize the level of automation as Human (using dashboard) vs Computer (geo apply) actions. As a company with a deep focus on security, reducing the number of humans capable of taking actions, as well as reducing the total number of actions that those humans take, helps to reduce our exposure to both mistakes and malicious intent.

Tag Everything

At Coinbase, we use AWS tags to categorize, document, and define ownership of our infrastructure. Resource codification was the perfect opportunity to rethink our tag standards and ensure that all of our resources are properly classified.

Using GeoEngineer, we are able to implement resource level validations to require that any resources codified in our infrastructure upholds our tag standards. An example of standardized tags we use to classify our resources are Name, ProjectName, and Org which correlate respectively to the name of the specific resource, the project it is associated with, and the project’s organization within Coinbase.

Additionally, GeoEngineer can be expanded to automatically add default tags based on the context of resources.

Beyond classification, we use tags for determining the importance of a resource and directing notifications to the right channel. For example, our pd_channel tag ensures that if reliability with a resource can’t be auto remediated that we’re sending PagerDuty alerts to the right team.

Eliminate Cruft

As our codified effort illuminated not just some, but all of the resources within our infrastructure, we began to find dated configurations and even long-since deprecated resources. These ‘cruft’ resources are unused or unidentified resources which have been created by hand through the console or lost over time. Cruft resources are bad because not only because they’re an extra expense, but because unknown resources can lead to security holes or improper utilization.

Tackling cruft in your infrastructure can be a tedious process. One mislabeled SQS topic could actually be the SQS topic and inadvertently removing or modifying such a topic might take down an entire segment of the organization. Our technique to tackle this cruft began by using the output of geo status to list every single uncodified resource on a shared spreadsheet, and then carefully identifying and taking action on each resource.

We utilized three primary resources to ensure we were identifying and deleting the correct resources. First, CloudWatch’s metrics dashboard was invaluable to clarify utilization patterns for specific resources and show exactly when a resource stopped or started being used. Second, CloudTrail provided key insight into which specific user created a resource or interacted with it recently. Third, searching VPC flow logs allowed us to validate resources which hadn’t recieved any meaningful traffic.

In the end, we deleted over 270 resources across all of our environments, not including all of the resources we were able to rename and properly categorize through the effort. This was a long and tedious yet very rewarding process.

Keep it Codified

Our experience tells us that infrastructure codification follows the rule of entropy. No matter how hard you try, someone will want to log into the console and create or make changes to live resources. In order to protect our efforts, we want to set up the right tools and culture to keep our standards from slipping.

One of the forcing functions we utilize to prevent slippage is creating a bot called Mars to frequently run geo status and generate a percentage metric of uncodified resources that we can use to automatically generate notifications when we stray from our target.

Another strategy we use is to parse the output of geo plan to send notifications if there is a divergence between our code and the actual state of our AWS environments. This output alerts us to potential security issues like if someone were to modify an IAM policy without updating the policy file template.

What’s Next?

Ensuring that we’ve accounted for all of our resources is just the first step on the road to completely automating our infrastructure. One of the ideas we’re working towards next is building a bot to geo apply pull requests based on consensus to further reduce any individual’s access which following our mission to empower the breadth of engineering at Coinbase.

There’s a lot more to come - if you’re interested in being part of what we’re doing, we’re hiring!

Thanks to Rob Witoff, Graham Jenson, Brad Larson, Shane da Silva, Jack Kearney!

Codifying Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.