History

Envoy was released as OSS in the fall of 2016, and much to our amazement quickly gained traction throughout the industry. Users were drawn to many different aspects of the project including its inclusive community, extensibility, API-driven configuration model, powerful observability output, and increasingly extensive feature set.

Although in its early history Envoy became synonymous with “service mesh,” its first use at Lyft was actually as an API gateway / edge proxy, providing in-depth observability output that aided Lyft’s migration from a monolithic to a microservice architecture.

Over the last 5+ years, we have seen Envoy adopted by a tremendous number of end users, both as an API gateway and as a sidecar proxy in “service mesh” roles. At the same time we have seen a large vendor ecosystem spring up around Envoy, providing a multitude of solutions both in the OSS and proprietary domains. Envoy’s vendor ecosystem has been critical to the project’s success; without funding for all of the employees that work part or full-time on Envoy the project would certainly not be what it is today.

The flip side of Envoy’s success as a component of many different architecture types and vendor solutions is that it is inherently low level; Envoy is not an easy piece of software to learn. While the project has had massive success being adopted by large engineering organizations around the world, it is only lightly adopted for smaller and simpler use cases, where nginx and HAProxy are still dominant.

The Envoy Gateway project was born out of the belief that bringing Envoy “to the masses” in the API gateway role requires two primary things:

• A simplified deployment model and API layer aimed at lighter use cases.
• Merging the existing CNCF API gateway projects (Contour and Emissary) into a common core that can provide the best possible onboarding experience, while still allowing vendors to build value-added solutions based on Envoy Proxy and Envoy Gateway.

We strongly believe that if the community converges around a single Envoy branded API gateway core, it will:

• Reduce duplicative efforts around security, control plane technical details, and other shared concerns.
• Allow vendors to focus on layering value added functionality on top of Envoy Proxy and Envoy Gateway in the form of extensions, management plane UI, etc.
• Lead to a “rising tide lifts all boats” phenomenon in which more users around the world enjoy the benefits of Envoy, whether their organization is large or small. More users feed the virtuous cycle of more potential customers, more support for the core Envoy project, and a better overall experience for all.

Project outline

At a high level, Envoy Gateway can be thought of as a wrapper around the Envoy Proxy core. It will not change the core proxy, xDS, go-control-plane, etc. in any way (other than potentially driving features, bug fixes, and general improvements!). It will provide the following functionality:

• A simplified API for the gateway use case. The API will be the Kubernetes Gateway API with some Envoy-specific extensions. This API was chosen because deployment on Kubernetes as an ingress controller is the initial focus for the project and because the API has broad industry buy-in.
• A “batteries included” experience that will enable users to get up and running as fast as possible. This includes lifecycle management functionality that provisions controller resources, control plane resources, proxy instances, etc.
• An extensible API surface. While the project will aim to make common API gateway functionality available out of the box (e.g., rate limiting, authentication, Let’s Encrypt integration, etc.), vendors will be able to provide SaaS versions of all APIs, provide additional APIs and value added functionality such as WAF, enhanced observability, chaos engineering, etc.
• High quality documentation and getting started guides. Our primary goal with Envoy Gateway is to make the most common gateway use cases trivial to stand up for the average user.

On the topic of APIs, we believe that one area that has led to significant confusion is the effective reimplementation of Envoy’s xDS APIs in other projects when targeting advanced use cases. This pattern causes users to have to learn multiple sophisticated APIs (which ultimately translate back to xDS) in order to get their job done. As such, Envoy Gateway is committed to a “hard line in the sand” in which the Kubernetes Gateway API (and any permissible extensions within that API) is the only additional API that will be supported. More advanced use cases will be served by an “xDS mode” in which existing API resources will be automatically translated for the end user, who can then switch to utilizing the xDS APIs directly. This will lead to a crisper primary API while allowing an escape hatch for organizations that might outgrow the expressiveness of the primary API and wish to utilize the full power of Envoy via xDS.

On API standardization

Although a goal of Envoy Gateway is to provide a reference implementation for easily running Envoy in Kubernetes as an ingress controller, possibly the most important contribution of this effort will be standardizing the APIs that are used for this purpose. As the industry converges on specific Envoy Kubernetes Gateway API extensions, it will allow vendors to easily provide alternate SaaS implementations that may be preferable if a user outgrows the reference implementation, wants additional support and features, etc. Clearly, there is much work to do around defining the API extensions, determining which APIs are required versus optional for conformance, etc. This is the beginning of our standardization journey and we are eager to dive in with all interested parties.

Next steps

Today we are thankful for the initial sponsors of Envoy Gateway (Ambassador Labs, Fidelity, Tetrate, and VMware) and are excited to start on this new journey with all of you. The project is very early with the focus so far having been to agree on goals and high level design, so it’s a great time to get involved, either as an end user or as a system integrator.

We also want to make it extremely clear that existing users of Contour and Emissary will not be left behind. The project (and VMware and Ambassador Labs) is completely committed to ensuring a smooth eventual migration path for users of those projects to Envoy Gateway, either via translation and replacement, or via those projects becoming wrappers around the Envoy Gateway core.

We are extremely excited about bringing Envoy to a larger group of users via the Envoy Gateway project, and we hope you will join us on our journey!

Introducing Envoy Gateway was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Matt Klein on Twitter

Last week I sat for an internal interview about my career progression to high level IC engineer, with a focus on how I've never felt I needed to become a manager to gain influence. I thought I would share some of my career advice for aspiring IC "lifers." Thread!

I also spoke about this thread/topic in two podcast episodes which I recommend listening to for more color:

• Part 1: https://developertea.simplecast.fm/50464d4b
• Part 2: https://developertea.simplecast.fm/10bb00cd

Today we are thrilled to announce the initial OSS preview release of Envoy Mobile, an iOS and Android client network library that brings the power of Envoy Proxy to mobile platforms. This is the beginning of a journey that we hope mobile developers around the industry will join us on.

Background

When Lyft originally announced Envoy in 2016, the project goal was simply stated as:

The network should be transparent to applications. When network and application problems do occur, it should be easy to determine the source of the problem.

Envoy proxy was initially built at Lyft to solve the networking and observability issues inherent in large polyglot server-side microservice architectures. Over the last two and a half years, much to our surprise and satisfaction, Envoy has become incredibly popular throughout the industry. Today Envoy is used by all major public cloud providers, countless end user companies, and a plethora of infrastructure startups that have recognized Envoy’s extensibility as a useful base with which to build vertical applications and services. For those not familiar with Envoy’s architecture and feature set, please see the links in the further reading section at the bottom of this post.

As fundamental as Envoy has become in scaling both Lyft and other organizations’ distributed architectures, the reality is that three 9s at the server-side edge is meaningless if the user of a mobile application is only able to complete the desired product flows a fraction of the time. This may be due to a combination of network and application errors. Thus, in order to fully achieve the Envoy project’s goal of making the network transparent to applications, the service mesh and its inherent benefits (observability, consistency, etc.) must expand beyond the edge all the way to the mobile applications that are so critical to the end user’s experience. Envoy Mobile in conjunction with Envoy in the data center will provide the ability to reason about the entire distributed system network, not just the server-side portion.

Three 9s at the server-side edge is meaningless if the user of a mobile application is only able to complete the desired product flows a fraction of the time.

Envoy mobile goals

Whereas server-side Envoy proxy is a self-contained process that is meant to be deployed alongside a polyglot architecture, Envoy Mobile is distributed as a library meant to be compiled directly into client mobile applications. The library approach is required due to the practicalities of how applications are written and distributed on both the iOS and Android platforms. The high level goals of the library are discussed in the following subsections.

Consistent cross-platform advanced low level networking

Low-level networking improvements like HTTP/2, QUIC, QUIC to TCP failover, gRPC, DNS replacements, push/streaming, observability, analytics, state of the art TLS, retry and timeout policies, etc. take a tremendous amount of effort both to implement robustly as well as to make consistent across all of the Android and iOS versions in common use. Envoy Mobile will provide all of this functionality in a consistent cross-platform manner, especially when paired with Envoy running at the edge, similar to how server-side Envoy provides consistency to polyglot distributed application architectures.

Ubiquitous API and abstraction for networking

With the industry progressively moving towards specifying APIs via a strongly typed IDL such as protocol buffers, Envoy Mobile will standardize and abstract how mobile developers interact with IDL exposed endpoints. Via intelligent protobuf code generation and an abstract transport, both iOS and Android can provide similar interfaces and ergonomics for consuming APIs. Initially we are planning on focusing our efforts on Swift APIs for iOS and Kotlin APIs for Android, but depending on community interest we will consider adding support for additional languages in the future. Our ultimate goal is to make the low-level Envoy common C++ code an implementation detail that the average mobile developer does not need to be aware of. Instead, mobile developers will interact with high-level language specific APIs that encapsulate common concerns such as making API calls, analytics, tracing, etc.

Simple and explicit system for supporting advanced networking features

With protocol buffer’s powerful annotation/extension system, Envoy Mobile can add sophisticated cross-platform functionality in a simple and explicit way when using strongly typed IDL APIs. Examples of annotations that are planned on our roadmap include:

• Marking an API as offline/deferred capable
• Caching
• Priority
• Streaming
• Marking fields for exclusion both on the request and response in poor network conditions
• General Envoy policies such as retry and timeout specifications

Much like Envoy’s use in a server-side service mesh, the goal is to push as much functionality as possible into the common core so as to avoid reimplementing it in every mobile application language.

Our long-term plans include evolving the gRPC Server Reflection Protocol into a streaming reflection service API. This API will allow both Envoy and Envoy Mobile to fetch generic protobuf definitions from a central IDL service, which can then be used to implement annotation-driven networking via reflection. This model means that Envoy Mobile will not necessarily need to have prior knowledge of an organization’s APIs in order to provide enhanced cross-platform networking functionality.

xDS driven mobile client policy

One of the reasons that Envoy has become so popular as a platform is its rich configuration discovery APIs which are collectively known as xDS (more information can be found here and here). These APIs allow a distributed set of Envoys to be managed by an eventually consistent control plane. One of the long term goals of Envoy Mobile is to bring xDS configuration all the way to mobile clients, in the form of routing, authentication, failover, load balancing, and other policies driven by a global load balancing system. This will be an extremely powerful mechanism for bringing layer 7 / application networking concepts all the way to the mobile client. One day, it may even become possible to compile WASM code server-side, remotely distribute it to mobile clients, and remotely alter client behavior either to fix latent bugs or react more quickly to customer needs without needing to distribute a completely new client binary.

OSS strategy

This release is a proof of concept demo that shows Envoy compiled as a library and functional on both iOS and Android. Unlike when Lyft released Envoy in 2016, we are not providing a production-ready client networking solution. So why are we releasing it now?

Based on discussions with organizations around the industry, it seems clear that there is a strong desire for a comprehensive cross-platform client networking solution similar to what this project proposes. We have decided that it is in the industry’s best interest to release now, even while our team actively works on the base feature set required for shipping Envoy Mobile as the default networking transport layer in Lyft’s mobile applications. We look forward to collaborating with mobile engineering teams around the industry to ensure that what we build is generally useful, even while continuing to iterate at high velocity.

If we are successful in realizing our vision for the future of mobile client networking, we fully expect to grow the project’s maintainer ranks with engineers from around the industry, as well as ultimately migrate the project to a CNCF-owned vendor neutral location.

Join us and get started

In addition to the demo release which we encourage you to compile and play with, we are also open sourcing our roadmap and plan to develop Envoy Mobile entirely in the open. Please reach out to us via GitHub, email, or Slack if you would like to join us on our journey of defining the future of mobile client application networking. We look forward to hearing from you and collaborating with you!

Further reading

• Announcing Envoy: C++ L7 proxy and communication bus
• Envoy: 7 months later
• Envoy joins the CNCF
• Envoy graduates!
• Envoy documentation
• Envoy Mobile documentation

Este artículo también está en español: eng-espanol.lyft.com

Announcing Envoy Mobile was originally published in Lyft Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Although it’s true that, in some cases, crashes may be an indication of unreliable software and subpar development methods, crashing is also a valid error handling method that if used correctly can increase rather than decrease the overall quality, reliability, and velocity of a piece of software.

Code is a liability

Ultimately, every line of code in a program is a liability that may lead to software defects. One particularly pernicious source of code and defect volume is rarely (or never) used error handling code.

def doSomethingWithBar(bar):
  if bar is None:
    # Can this ever actually happen?
    return False

  bar.doSomething()
  return True

In the previous example I show some Python which checks to see if bar is a null value before doing something. To indicate failure (bar is null), the function returns a boolean value (before the committee for functional programming purity comes after me, instead of checking for null, assume bar is an optional and check whether it has a value — the same idea applies).

This example may seem contrived, but I see this pattern frequently in code review, and my review comment is always: can this ever actually happen? Sometimes it can happen, but very often it cannot, and the engineer has written the code in this style because they are under the mistaken belief that all code must have error checking.

The only error checking a program needs are for errors that can actually happen during normal control flow.

If in the previous example bar can never be null, doSomethingWithBar() can be removed from the code entirely and the doSomething() method call inlined. Additionally, the caller of the function no longer needs to deal with the cognitive load of understanding whether and how they need to handle an error from a call to doSomethingWithBar().

This last point is critical: even though this example is contrived, it or something like it is frequently found in real codebases, and through long call chains of additional needless error handling it can quickly become impossible to reason about the code logic. This extraneous call chain logic makes the larger codebase harder to reason about, harder to maintain and enhance, and more likely to contain bugs.

What if the author of the code was wrong and bar can in fact be null? Let the program crash! The resulting crash and stack trace will be extremely obvious, easy to debug, and fix. It may become clear that the invariant of bar always being null is not true. If may also become clear that some caller of doSomething() is broken and should have a proper bar. Either way, the code will stay as simple as possible and without extraneous logic that is hard to reason about.

Assert early and assert often

A corollary to “crash early and crash often” is “assert early and assert often.” Asserts are a tremendously powerful mechanism to verify code invariants (things that should always be true). I favor two types of asserts:

• Debug asserts: These asserts are compiled out in release builds and well and truly should never happen.
• Release asserts: These asserts are not compiled out of release builds. They are used to check situations that may theoretically happen, but if they do, it would be preferable to crash and restart instead of trying to handle the error.

Debug assertions should outnumber release assertions by a very large factor.

I view assertions as an incredibly powerful tool for reducing code complexity for the following reasons:

1. Assertions act as documentation. The reader clearly understands the state the program is expected to be in when the assertion runs.
2. Assertions by definition limit extraneous branches and error handling. Why handle a state which should never happen?

For languages that do not contain builtin assertions (e.g., Go) I recommend creating a simple assertion wrapper that crashes the program if not satisfied.

What happens if the programmer is incorrect and an assertion is not valid in all cases? Let the program crash! Like the null check in the previous section, an assertion failure is trivial to debug. The fix may either be to remove the assertion and add handling code, or to make the calling code adhere to the assertion.

Using code coverage and style guidelines to limit error handling complexity

Although code coverage is an imperfect tool (a high coverage percentage does not necessarily mean a program has quality test coverage), enforcing a very high level of coverage is very useful in yielding liberal use of assertions and limiting error handling to only what is necessary.

In the Envoy error handling guidelines we write:

Tip: If the thought of adding the extra test coverage, logging, and stats to handle an error and continue seems ridiculous because “this should never happen”, it’s a very good indication that the appropriate behavior is to terminate the process and not handle the error. When in doubt, please discuss.

In essence, we force all error branches and assertions to be covered by tests. If the programmer feels this time is wasteful it’s a useful forcing function for code logic simplification.

Ownership semantics used to prevent crashes can lead to complexity and bugs

One other area in which I often see extra complexity and bugs added in order to ostensibly prevent crashing is in object/data ownership semantics. Fundamentally, there are three different ways data can be allocated and tracked in a program:

1. Stack
2. Heap with a single owner (e.g., std::unique_ptr<> in C++, standard borrow checking in Rust, etc.). Note that in many popular languages this ownership type is not available in practice because all heap allocated objects are reference-counted and allow for possible unintentional sharing (e.g., Java, Python, JS, Go, etc.).
3. Heap with multiple owners (e.g., std::shared_ptr<> in C++, Java, Go, Python, Rust shared pointers, etc.).

Stack allocation is relatively simple and easy to understand so I’m going to primarily discuss how (2) and (3) relate to crashing early and code complexity.

At a high level, code which uses heap data with a single owner is substantially easier to reason about than code that uses reference-counted data. A single piece of code allocates data and a single piece of code frees it. Very simple. The alternative is shared ownership. The use of shared ownership can make code extremely difficult to reason about. How and when will an object be freed? Will there be any memory leaks due to circular references? (Somewhat ironically, I’ve seen many more production memory leaks in software written in Java and Python due to circular shared references vs. well written C++ that makes heavy use of single owner semantics.)

The downside of the single ownership approach is the ease of creating “use after free” situations in C/C++. Rust avoids “use after free” entirely with the borrow checker, while still allowing for single owner semantics. This is incredibly powerful from both a correctness perspective as well as a single data owner perspective and I look forward to the day in which most code is written in languages with Rust-like semantics. That said, given that the majority of code in the world is still written in C/C++, Java, Python, JS, and similar languages, I will continue this discussion from that viewpoint.

In C/C++, a “use after free” crash can sometimes be difficult to debug (again making Rust borrow semantics very enticing from a productivity standpoint), but it is very clearly a sign that the program crashed and an invariant has been violated. The alternative that I sometimes see tried in C/C++ is favoring Java/Python-like shared object ownership in an effort to avoid these types of crashes. The thinking goes that if an object is never freed while there is a reference to it, the program will never crash. Yet in my experience this inevitably leads to greater code complexity and more bugs due to circular references, hard to reason about logic, etc.

Only use shared memory ownership when the program logic actually calls for it.

For the same reasons that I advocate for limited error handling and extra assertions above, using single owner semantics is preferred. In Rust, the compiler will verify correctness. In C/C++ the compiler will not, but letting the program crash and fixing the uncovered invariant violation is far preferable to introducing needless ownership and code complexity in an attempt to avoid crashes of this type altogether.

For languages that do not allow for explicit single owner semantics, I recommend aggressively setting references to null when no longer in use. This reduces the chance of circular references and should make effective use after free issues more clear.

Conclusion

Limiting software complexity is one of the primary mechanisms available to us to limit defects. Very often, invariant violations that cause a fatal crash are substantially easier to debug and fix than complex code that attempted to prevent the crashes in the first place. Specifically, I recommend using the following three techniques for limiting error handling and code complexity:

1. Limit error handling to only errors that can actually happen during normal control flow. Crash otherwise.
2. Liberally use assertions to document invariant state and crash if violated.
3. Use single owner data semantics if at all possible to limit code complexity, and if doing this using C/C++, let the program crash if an ownership invariant is violated.

The three strategies above will limit code complexity and generally yield bugs that are more obvious and easier to fix.

It's new year's day and I'm arguing about how ridiculous monorepos are. 2019 is off to an inauspicious start. In spirit of this, I bring you a poll. Who are the bigger zealots? Supporters of:

Here we are at the beginning of 2019 and I’m engaged in yet another discussion on the merits (or lack thereof) of keeping all of an organization’s code in a “monorepo.” For those of you not familiar with this concept, the idea behind a monorepo is to store all code in a single version control system (VCS) repository. The alternative, of course, is to store code split into many different VCS repositories, usually on a service/application/library basis. For the purpose of this post I will call the multiple repository solution a “polyrepo.”

Some of tech’s biggest names use a monorepo, including Google, Facebook, Twitter, and others. Surely if these companies all use a monorepo, the benefits must be tremendous, and we should all do the same, right? Wrong! As the title says: please, do not use a monorepo! Why? Because, at scale, a monorepo must solve every problem that a polyrepo must solve, with the downside of encouraging tight coupling, and the additional herculean effort of tackling VCS scalability. Thus, in the medium to long term, a monorepo provides zero organizational benefits, while inevitably leaving some of an organization’s best engineers with a wicked case of PTSD (manifested via drooling and incoherent mumbling about git performance internals).

A quick note: what do I mean by “at scale?” There is no definitive answer to this question, but because I know I will be asked, let’s say for the sake of discussion that at scale means over 100 developers writing code full time.

Theoretical monorepo benefits and why they cannot be achieved without polyrepo-style tooling (or are a lie)

Theoretical benefit 1: Easier collaboration and code sharing

Monorepo proponents will argue that when all code is present within a single repository, the likelihood of code duplication is small, and it’s more likely that different teams will collaborate together on shared infrastructure.

Here is the ugly truth about even a medium size monorepo (and this will be a recurring theme throughout this section): it quickly becomes unreasonable for a single developer to have the entire repository on their machine, or to search through it using tools like grep. Thus, any monorepo that hopes to scale must provide two things:

1. Some type of virtual file system (VFS) that allows a portion of the code to be present locally. This might be accomplished via a proprietary VCS like Perforce which natively operates this way, via Google’s “G3” internal tooling, or Microsoft’s GVFS.
2. Sophisticated source code indexing/searching/discovery capabilities as a service. Since no individual developer is going to have all code in a searchable state, it’s critical that there exists some capability to perform a search across the entire codebase.

Given that a developer will only access small portions of the codebase at a time, is there any real difference between checking out a portion of the tree via a VFS or checking out multiple repositories? There is no difference.

In terms of source code indexing/searching/discovery capabilities, a tool can trivially iterate over many repositories and collate the results. In fact, this is how GitHub’s own searching capabilities work as well as newer and more sophisticated indexing and collaboration tools such as Sourcegraph.

Thus, in terms of collaboration and code sharing, at scale, developers are exposed to subsections of code through higher layer tooling. Whether the code is in a monorepo or polyrepo is irrelevant; the problem being solved is the same, and the efficacy of collaboration and code sharing has everything to do with engineering culture and nothing to do with code storage.

Theoretical benefit 2: Single build / no dependency management

The next thing that monorepo proponents will typically say is that by having all code in a single repository, there is no need for dependency management because all source code is built at the same time. This is a lie! At scale, there is simply no way to rebuild the entirety of a codebase and run all automated tests when each change is submitted (or, more importantly and more often, in CI when a change is proposed). To deal with this problem, all of the large monorepos have developed sophisticated build systems (see Bazel/Blaze from Google and Buck from Facebook as examples) that are designed in such a way as to internally track dependencies and build a directed acyclic graph (DAG) of the source code. This DAG allows for efficient build and test caching such that only code that changes, or code that depends on it, needs to be built and tested.

Furthermore, because code that is built must actually be deployed, and not all software is deployed at the same time, it is essential that build artifacts are carefully tracked so that previously deployed software can be redeployed to new hosts as needed. This reality means that even in a monorepo world, multiple versions of code exist at the same time in the wild, and must be carefully tracked and reconciled.

Monorepo proponents will argue that even with the large amount of build/dependency tracking required, there is still substantial benefit because a single commit/SHA describes the entire state of the world. I would argue this benefit is dubious; given the DAG that already exists, it’s a trivial leap to include individual repository SHAs as part of the DAG, and in fact, Bazel can seamlessly work across repositories or within a single repository, abstracting the underlying layout from the developer. Furthermore, automated refactor tooling can trivially be built that automatically bumps dependent library versions across many repositories, thus blurring the difference between a monorepo and polyrepo in this area (more on this below).

The end result is that the realities of build/deploy management at scale are largely identical whether using a monorepo or polyrepo. The tools don’t care, and neither should the developers writing code.

Theoretical benefit 3: Code refactors are easy / atomic commits

The final benefit that monorepo proponents typically tout is the fact that when all code is in a single repository, it makes code refactors much easier, due to ease of searching and the idea that a single atomic commit can span the entire codebase. This is a fallacy for multiple reasons:

1. As described above, at scale, a developer will not be able to easily edit or search the entirety of the codebase on their local machine. Thus, the idea that one can clone all of the code and simply do a grep/replace is not trivial in practice.
2. If we assume that via a sophisticated VFS a developer can clone and edit the entire codebase, the next question is how often does that actually happen? I’m not talking about fixing a bug in an implementation of a shared library, as this type of fix is identically carried out whether using a monorepo or polyrepo (assuming similar build/deploy tooling as described in the previous section). I’m talking about a library API change that has follow-on build breakage effects for other code. In very large code bases, it is likely impossible to make a change to a fundamental API and get it code reviewed by every affected team before merge conflicts force the process to start over again. Developers are faced with two realistic choices. First, they can give up, and work around the API issue (this happens more often than we would like to admit). Second, they can deprecate the existing API, implement a new API, and then go through the laborious process of individual deprecation changes throughout the codebase. Either way, this is exactly the same process undertaken in a polyrepo.
3. In a service oriented world, applications are now composed of many loosely coupled services that interact with each other using some type of well specified API. Larger organizations inevitably migrate to an IDL such as Thrift or Protobuf that allow for type-safe APIs and backwards compatible changes. As described in the previous section on build/deploy management, code is not deployed at the same time. It might be deployed over a period of hours, days, or months. Thus, modern developers must think about backwards compatibility in the wild. This is a simple reality of modern application development that many developers would like to ignore but cannot. Thus, when it comes to services, versus library APIs, developers must use one of the two options described above (give up on changing an API or go through a deprecation cycle), and this is no different whether using a monorepo or polyrepo.

In terms of actually making refactor changes across large codebases, many organizations end up developing automated refactor tooling such as fastmod, recently released by Facebook. As elsewhere, a tool such as this can trivially operate within a single repository or across multiple repositories. Lyft has a tool internally called “refactorator” which does just this. It works like fastmod but automates making changes across our polyrepo, including opening PRs, tracking review status, etc.

Unique monorepo downsides

In the previous section I laid out all of the theoretical benefits that a monorepo provides, and explained why in order to realize them, extraordinarily complex tooling must be developed that is no different to what is required for a polyrepo. In this section, I’m going to cover two unique downsides to monorepos.

Downside 1: Tight coupling and OSS

Organizationally, a monorepo encourages tight coupling and development of brittle software. It gives developers the feeling they can easily fix abstraction mistakes, when they actually cannot in the real world due to the realities of staggered build/deploy and the human/organizational/cultural factors inherent in asking developers to make changes across the entire codebase.

Polyrepo code layout offers clear team/project/abstraction/ownership boundaries and encourages developers to think carefully about contracts. This is a subtle yet hugely important benefit: it imbues an organization’s developers with a more scalable and long-term way of thinking. Furthermore, the use of a polyrepo does not mean that developers cannot reach across repository boundaries. Whether this happens or not is a function of the engineering culture in place versus whether the organization uses a monorepo or polyrepo.

Tight coupling also has substantial implications with regard to open source. If an organization wishes to create or easily consume OSS, using a polyrepo is required. The contortions that large monorepo organizations undertake (reverse import/export, private/public issue tracking, shim layers to abstract standard library differences, etc.) are not conducive to productive OSS collaboration and community building, and also create substantial overhead for engineers within the organization.

Downside 2: VCS scalability

monorepi on Twitter

here I updated the XKCD comic for you

Scaling a single VCS to hundreds of developers, hundreds of millions lines of code, and a rapid rate of submissions is a monumental task. Twitter’s monorepo roll-out about 5 years ago (based on git) was one of the biggest software engineering boondoggles I have ever witnessed in my career. Running simple commands such as git status would take minutes. If an individual clone got too far behind, it took hours to catch up (for a time there was even a practice of shipping hard drives to remote employees with a recent clone to start out with). I bring this up not specifically to make fun of Twitter engineering, but to illustrate how hard this problem is. I’m told that 5 years later, the performance of Twitter’s monorepo is still not what the developer tooling team there would like, and not for lack of trying.

Of course, the past 5 years has also seen development in this area. Microsoft’s git VFS which is used internally to develop Windows, has tackled creating a real VFS for git, as I described above, as a requirement for monorepo scalability (and with Microsoft’s acquisition of GitHub it seems likely this level of git scalability will find its way into GitHub’s enterprise offerings). And, of course, Google and Facebook continue to invest tremendous resources into their internal systems to keep them running, although none of this work is publicly available.

However, why bother solving the VCS scalability problem at all when, as described in the previous section, tooling will also need to be built that is identical to what is required for a polyrepo? There is no good reason.

Conclusion

As is often the case in software engineering, we tend to look at tech’s most successful companies for guidance on best practices, without understanding the monumental engineering that has gone into making those companies successful at scale. Monorepos, in my opinion, are an egregious example of this. Google, Facebook, and Twitter have invested extensively in their code storage systems, only to wind up with a solution that is no different from what is required when using a polyrepo, yet leads to tight coupling and requires a substantial investment in VCS scalability.

The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used. The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!

Last Monday we hosted our very first EnvoyCon, a conference devoted to Envoy, as a co-located event with KubeCon. The event sold out weeks in advance and around 350 folks attended.

I began the day with a very short welcome and brief history of the project. We then had a series of end-user and deep technology-focused talks. The day ended up with a brief road map discussion, followed by a Q&A session with all of the maintainers. I’m not going to highlight any particular talk because honestly, they were all fantastic! I would encourage anyone that wasn’t able to be at the event to check out the videos which have now been posted.

I want to personally say a huge thank you to everyone that made this event a huge success: the program committee; Megan Lehn from the Linux Foundation’s event planning group; the maintainers; and finally and most importantly our larger community that is passionate enough to both speak at the event and come to hear others.

It was such a special opportunity to get end-users, contributors, and maintainers together for talks and lively discussion. Feedback for the event has been extremely positive and we are all already looking forward to the second iteration which will be co-located with KubeCon NA 2019 in San Diego. Onward!

A huge thank you to the EnvoyCon sponsors!

Diamond:

• Datawire
• Turbine Labs

Gold:

• Lyft

Social:

• AWS

EnvoyCon recap was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today we are thrilled to announce that Envoy has become a CNCF graduated project, joining Kubernetes and Prometheus. The successful graduation demonstrates Envoy has reached a project maturity level that reflects a sustainable open source community, thus making it amenable to widespread enterprise usage.

Envoy began life at Lyft in May, 2015, a time in which the company was struggling to stabilize its rapidly growing microservice distributed architecture. Deployed in production at Lyft since September, 2015, Envoy quickly became an integral building block that allowed for more efficient human and technical scaling of Lyft’s architecture.

Envoy’s goal is to abstract the network from application developers so that they can focus on business logic. In order to provide rich and easy to use abstractions for modern cloud native applications, Envoy focuses on the L7 or application layer of the stack. In addition to support for HTTP and gRPC, Envoy also supports Redis, Thrift, MongoDB, global rate limiting, external authentication and authorization, and many other features geared towards building modern distributed applications.

In September, 2016, Lyft released Envoy 1.0 as OSS. The industry reaction to the project surpassed our wildest imagination of what was possible. We had clearly underestimated the demand for a modern network proxy built from the ground up for cloud native deployment. Envoy is not simple software; swapping or adding a network proxy in an existing production deployment is a non-trivial undertaking. This fact, the inherent implementation complexity of network proxy replacement, makes it all the more incredible the speed at which organizations across the industry have adopted the project.

In September, 2017, Envoy joined CNCF as an incubating project. The Foundation provided Envoy a neutral home to continue its rapid project growth. Now, nearly two and a half years after the initial OSS release, Envoy is in production at some of the world’s largest and most prominent technology companies. Additionally, Envoy has become a building block for higher level products and services provided by major cloud providers, startups, and other OSS projects. It’s been an incredible and wild ride.

Why has Envoy become so popular so quickly? We think it’s for a few different reasons:

• Performance: Envoy was written to perform well both with regard to throughput and latency. Stable latencies at the tail are critical for reducing cognitive load in large cloud native deployments.
• Reliability: The Envoy project takes quality and reliability extremely seriously. From test coverage to rigorous code reviews, our goal is to build a product reliable enough for the most critical workloads.
• Modern code base: Envoy is written in modern C++, which allows for a good balance of productivity and performance.
• Extensibility: Envoy has a multitude of extension points which allows it to be adapted to a large variety of end use cases, without compromising the design integrity and generality of the core software.
• Best-in-class observability: Envoy focuses on providing a staggering array of observability output. Although we aim to make the network transparent to application developers, when things go wrong it’s critical to be able to determine the source of the problem efficiently.
• Rich configuration API: Unlike previous generations of network proxies, Envoy was written to be centrally and dynamically configured using a well-specified, backwards compatible, and type safe API, and with a clean separation of data and control plane. The days of manually distributing configuration files to thousands of nodes and restarting the proxy are over. The configuration API has lead to a proliferation of higher level control planes that utilize Envoy in unique ways to provide end user value.
• Community first development: There is no commercial variant of the primary Envoy project. This means that the community collectively collaborates on making technology first decisions without having to worry about cannibalizing a commercial business model. This has turned out to be an incredibly powerful driver of Envoy’s success and one we hope will not be lost in the coming years.

What does the future hold for Envoy? From a feature perspective, 2019 will see investments in new protocols (e.g., UDP, QUIC, and Kafka), further scaling work to allow Envoy to be used by the world’s largest deployments, and foundational work to enable Envoy based solutions on mobile/IoT devices (more on this soon!). From a project and community perspective, our goal continues to be providing a cloud native building block that can be used to make the network transparent to application developers. We aim to do this with a community first development process that allows rich products and services to be built on top of Envoy and made available to end users. If we are successful, we believe that Envoy will be ubiquitously deployed over the next five to ten years, although the majority of end users will not know that Envoy is being used within the cloud native infrastructure they interact with on a daily basis.

If you have not yet learned about Envoy, there is no better time as we enter the project’s next stage of growth and adoption. Join us!

Envoy graduates! was originally published in Envoy Proxy on Medium, where people are continuing the conversation by highlighting and responding to this story.

In response I wrote this Twitter thread:

Matt Klein on Twitter

Won't defend Redis Labs, this is a dead end move, but there needs to be more recognition that the economics of OSS are fundamentally broken. (Thread) https://t.co/ROa8GDHqAR

Paul Dix, CTO of InfluxData, then followed up with an excellent blog post which I highly recommend reading. I responded on Twitter with another thread:

Matt Klein on Twitter

Great article by Paul as usual. Although I think we differ in opinion on the effectiveness of open core, I largely agree with his points. Fundamentally, FOSS is not actually free. The 💰 must come from somewhere. I would like to clarify my point from my previous thread though. https://t.co/o70BNllVkf

Given the above background, I want to provide a long form discussion of my two Twitter threads as this topic is nuanced and quite interesting.

Note: this post is heavy on opinion and light on facts/references backing up those opinions. Thus, preface everything that follows with “IMO.”

The history of OSS

In the early days of computing, primarily driven by academics, much of the intellectual property created was born out of the belief that all software should be free and open. During the late 70s and early 80s, proprietary software licensing took hold (driven by IBM, Microsoft, Apple, and others), and with it, the parallel free software movement that brought GNU, Linux, and eventually most of the software we use today in modern infrastructure computing (see Wikipedia for a more detailed history).

It’s important to point out that the early days of the free software movement were driven primarily by hobbyists and “true believers.” These developers donated their often considerable time to openly collaborate on various pieces of core functionality (compilers, linkers, operating systems, etc.) that would later enable an entire ecosystem of free software to be built on top.

Modern expectations: software is free and open

Since the early 90s, and primarily driven by Linux, there has gradually been a mental shift among those in the high-scale computing industry: proprietary software should be avoided at all costs. Why?

1. Open software quality has vastly increased over time. This is a key and subtle point. As mentioned briefly above, the origins of the modern free software movement were driven primarily by hobbyists and professional developers donating their time during nights and weekends. From a quality and functionality perspective, how could this group compete with Microsoft, Apple, Oracle, IBM, and others during the 80s and early 90s? They could not. However, the late 90s brought with it Yahoo, Amazon, and Google. The 00s brought Facebook, Twitter, and others. In short, starting in the late 90s, massively successful companies that were not directly generating revenue from software or hardware started driving forward many of the most popular free software projects by hiring professional developers (often from Microsoft, Apple, and other proprietary software companies) and putting them to work full-time on OSS projects. Thus, over time, the workforce contributing to the most popular free software has gradually shifted from hobbyists to well paid and highly skilled professional programmers subsidized by the profits of Internet companies.
2. Open software allows modification. What if a proprietary software company goes out of business? How will users continue to make changes to meet their business needs? What if users need to make a change that the company does not feel is valuable and refuses to prioritize?
3. Open software avoids cost lock-in. As computing has shifted progressively away from “shrink-wrapped” software to services, often run across thousands or even millions of computers, the largest Internet companies do not want to be locked-in to per-node, per-CPU, etc. pricing for critical systems that they depend upon such as hypervisors, operating systems, databases, load balancers, etc.

The previous points have led to a place in which software that is not free and open is treated with skepticism.

Why do developers create OSS vs. proprietary software in the modern era?

The computing industry has gradually shifted from generating revenue directly from software to generating revenue from value added services built on top (often supported by advertising). This is not to say that propriety software no longer exists. Clearly it does in the form of consumer applications (games, Windows, OSX, Android, apps, etc.). In the enterprise space it also still exists in the form of line of business applications (although I would argue that most large LOB applications are now custom developed in-house by the company that needs them). However, when focusing on the infrastructure (cloud) computing space, there is a dogmatic feeling throughout the industry that core tooling such as compilers, operating systems, databases, load balancers, etc. should be free and open.

Within the infrastructure space, given that the bias is now towards software being free and open versus proprietary, what are the reasons for creating new OSS in the modern era? There are a few:

1. Hobbyists. The simplest (and most noble) explanation for creating OSS remains that a developer (or group of developers) sees a problem domain they feel is undeserved by software (or served by a proprietary offering they don’t want to pay for) and decides to start a collaborative process to implement a solution. This is no different from the early days of the free software movement.
2. Internet mega-company byproduct. As described above, companies like Amazon, Google, and Facebook do not believe in using proprietary software for critical infrastructure computing functions. In addition to bolstering previously existing OSS, they have also started many critical projects and released them as OSS. Why? Isn’t the software they develop a competitive advantage even if not domain specific? Let’s be clear that Internet mega-companies do not do anything out of goodwill (Google for example is notorious for releasing whitepapers about internal infrastructure technologies but not the software itself). Releasing OSS is a calculation that incorporates recruiting (developers want to work on OSS as it gives them industry visibility), technical credibility (helps with recruiting and the perception of the company as a technical powerhouse), and platform building (the industry orienting around a set of OSS technologies driven by an Internet giant provides a substantial advantage when offering related cloud computing products).
3. Mind-share building for an infrastructure computing startup. There are still quite a few VC funded infrastructure startups forming every year. In the modern era, it’s essentially inconceivable that such a company can find success with a proprietary software offering. Instead, companies start with OSS, hope to gain mind and market-share, and then pivot to figuring out how to generate revenue on top (more on this below).

It’s important to point out that the above OSS creation options are not mutually exclusive. For example, software can be started by a hobbyist and then move under the umbrella of an Internet giant or a VC funded startup. Or software can be created by an Internet giant and then key maintainers leave and start a VC funded company.

Who pays for OSS?

Nothing is free. All created OSS is paid for somehow. For each of the OSS creation reasons described in the previous section, let’s look at how the software is paid for.

1. Hobbyists are paying with their time. This is time that they could be spending earning a living building proprietary software, spending with their family, engaging in recreation, etc.
2. Internet mega-companies are subsidizing OSS creation with their primary revenue generation stream. For Google, search and advertising. For Facebook, advertising. For Apple, iPhone. For Lyft, ride sharing. The list goes on.
3. Infrastructure computing startups are subsidizing OSS creation with VC dollars. VCs invest under the assumption that once an OSS project attains popularity, a revenue generating business model can be built on top.

It’s important to note that other than hobbyists and true altruists, the Internet companies and VC-backed startups all have ulterior motives for releasing OSS as described in the previous section. To believe otherwise is foolish.

OSS business models for startups

I’m not going to describe in depth the various ways in which startups are attempting to make money on OSS in the modern era. For more information see this blog post by Paul Dix as well as my own post on why I decided to not start a company around Envoy. However, I will briefly describe them here in enough detail that the rest of this post will make sense in isolation.

1. Open core. The definition of open core is not well established, which creates substantial miscommunication when discussing this business model. However, the general idea is that some subset of features in the OSS are placed behind a paywall. If users want to consume those features they need to pay.
2. SaaS. In this model, the OSS is free, but a company will run it for the customer (either in the cloud or on-prem) for a fee. The idea is that operating a sophisticated piece of software 24/7 is non-trivial and worth paying for. SaaS can also mean the layering of additional functionality on top of the core OSS product (e.g., UIs, management systems, analytics engines, policy tools, cost analysis, etc.). Thus, the line between open core and SaaS is blurry, a point which I will discuss in depth in the next section.
3. Consulting, services, and support. The OSS is free, but one or more companies are formed to provide consulting, feature creation for hire, and general support, all for a fee. This business model is what I would call a “lifestyle business.” It may generate enough revenue for a comfortable lifestyle, but it is extremely unlikely to generate enough revenue to excite the VCs.

If we take consulting, services, and support off the table as an option for high-growth revenue generation (the only thing VCs care about), we are left with open core, SaaS, or some blurring of the two.

What is open core really?

This is where things start to get interesting. There is no single definition of open core. After the initial Redis story broke, Salvatore Sanfilippo (creator of Redis) wrote a blog post laying out his case that Redis had not switched to an open core model. His argument can be reduced to the fact that Redis is a modular system, and that only modules developed by Redis Labs would be under a commercial license. The Redis core will remain fully open. He goes on to write:

In an open core business model around an open source system it is *fundamental* that you take something useful out of the free software part.

At a high level, I agree with Salvatore that this is the most basic definition of open core, however, I do believe that Redis Labs has fundamentally switched to an open core business model, and the implications of this switch are subtle and gradual.

The problem with “pure” open core

Whether an OSS project is modular or chooses to release commercial/enterprise closed source builds with extra features, there are some fundamental problems with the “pure” open core model:

1. Open core is at odds with the OSS community. What happens when a community member wants to contribute a feature/module that competes with the enterprise offering? How should the company handle that situation? They are in no-win scenario. If they refuse to take the contribution they alienate the community. If they take the contribution they cannibalize their own business model.
2. Cloud vendors and large Internet companies have the resources to take the core and build the rest. Large cloud vendors are well equipped to build managed SaaS offerings on top of popular infrastructure OSS. Even if certain pieces of critical functionality are missing from the OSS offering, if important enough, the features can be rebuilt with enough resources. Whether this rebuilding process is actually worth it or not is a complex equation and the subject of the next section, however at the scale that the “big three” clouds operate, it probably is worth it. As the inevitable march to the public cloud continues over the next 10–20 years, selling open core OSS independently to enterprises will become an increasingly tall order.

Our industry’s inability to accurately evaluate Total Cost of Ownership (TCO)

Although it’s been implied throughout this post, it should be explicitly stated that many (most?) software build versus buy decisions are subjective and made without any real data. All too often, potential customers feel that infrastructure software should be free, and grossly underestimate the true cost of running a piece of software over time, when taking into account both CAPEX and OPEX. Theoretically, the CAPEX for OSS is $0. But what is the OPEX? How many engineers will have to be employed to run the software and debug it? What features will have to be added to an unfamiliar code base? What additional ancillary management systems, UIs, etc. will need to built such that the software can be consumed by the rest of the organization?

If our industry was better at analyzing true TCO, I fully believe that making money off of OSS would be substantially easier. The experts in the software would be able to easily charge substantial per-node licensing fees in exchange for continued feature development and support to all but the largest enterprises and cloud providers. The fact that the software is free would really just be an insurance policy in case the company goes out of business.

Unfortunately, software buying decisions are made by humans, not machines, operating with imperfect information and biases on evaluating true cost. Thus, we have a logjam:

Everyone wants infrastructure software to be free and continuously developed by highly skilled professional developers (who in turn expect to make substantial salaries), but no one wants to pay for it. The economics of this situation are unsustainable and broken.

The advantages of “loose” open core and SaaS

Having laid out the problems with “pure” open core and the imperfect decision making process that our industry uses when deciding whether to buy or build software, we now come to what I have recently called “loose” open core and SaaS.

In the future, I believe the most successful OSS projects will be primarily monetized via this method. What is it?

The idea behind “loose” open core and SaaS is that a popular OSS project can be developed as a completely community driven project (this avoids the conflicts of interest inherent in “pure” open core), while value added proprietary services and software can be sold in an ecosystem that forms around the OSS. What are some examples of such services and software?

• Modern “cloud native” infrastructure software is robust, but typically configured via a giant pile of YAML. Custom UIs that allow organizations to manage configurations are extremely powerful. When coupled with RBAC, version control, etc. they become even more so.
• Analytics tooling.
• Security analysis and policy enforcement tooling.
• Observability and monitoring tooling.
• Hosted and on-prem managed SaaS offerings. Operating software at scale is extremely non-trivial.
• Domain specific features that can be implemented as add-ons without requiring any core modifications. For example, DDoS detection and mitigation tooling using Envoy’s OSS APIs and filters.

The other benefit of “loose” open core technologies are that they are more difficult for the large cloud vendors to subsume. Historically, the clouds are very good at creating basic managed SaaS offerings of popular OSS. However, they have been much less effective at building the “special sauce” on top. If executed well, a robust “loose” open core business can be built that is compatible, not competitive, with the cloud offerings.

Project popularity and business models

I had an interesting insight recently: I don’t think there is a one-size-fits-all business model for every OSS project. In fact, I think the best business model is dependent on the project’s popularity and might change over time. Why would that be?

New projects

For new projects being built with VC funding or by hobbyists, and without the ability to battle-test the software in the production environment of a large Internet company, the larger software community will remain skeptical of project quality for quite some time. These projects need to get a toehold in production wherever they can, and initially will rely on word of mouth, conference talks, Twitter, Hacker News, etc. As a project gains some traction, I think it’s perfectly reasonable to target a “pure” open core business model coupled with services and support in order to fund continued development (of course “loose” open core and SaaS are fine too but might require more development resources than the startup has). In fact, at this stage, I would argue it’s more likely that some small number of early adopters will pay for a “pure” open core product and/or a services and support contract, satisfying the relatively small amount of growth that VCs want to see during early funding rounds.

Popular projects

Unfortunately, there is an inflection point at which in some sense an OSS project becomes too popular for its own good, and outgrows its ability to generate enough revenue via either “pure” open core or services and support. How?

As a project reaches the upper echelon of popularity and number of production deployments, the community surrounding the project by definition will have commensurately increased in size (including most likely the very largest technology companies). Thus, the likelihood someone in the community is going to want to add features that directly compete with a “pure” open core business model, and refuse to purchase the enterprise version of the software that contains the feature, becomes much more likely. At this point, the project and surrounding community starts to become strained as a conflict of interest kicks in between project governance and business interests.

At this point, the community alienation makes it much more likely that an upstart OSS competitor will form (built by someone else who either doesn’t want to pay for the enterprise edition of a different project, or built for a different but related reason and happens to subsume the role of the competing enterprise product). This point is far from academic. In fact, it is the driving force behind Envoy’s rapid adoption in the industry.

A summary of the problems

As was discussed previously, as much as everyone would like to break the laws of economics, OSS is not free. Furthermore, modern OSS is big business, whether produced by the large Internet companies or VC-backed startups. The days of basement hacking producing the most popular projects are largely gone. What we are left with is a set of substantial problems that do not have a good answer:

• Everyone wants well-designed, bug-free, and feature-rich free software, but no one wants to pay for it.
• Very popular OSS projects started by the large Internet companies may appear to be community driven, but there are significant ulterior motives at play.
• OSS projects started by VC-backed companies are struggling to figure out how to monetize their creations. “Pure” open core may work initially, but if the project becomes popular enough, community alienation will become a substantial problem when deciding which features to accept in OSS, and conflict with the ability to reach the revenue targets that VCs are looking for in later funding rounds.
• Basic SaaS offerings by VC-backed companies are easily subsumed by the major cloud vendors.

From my perspective, since it seems unlikely that we are going to change the psychology behind how our industry evaluates TCO, and if we want to stay optimistic that the cloud vendors will not end up eating everyone’s lunch, building a vibrant community and then enabling an ecosystem of “loose” open core and SaaS businesses on top appears to me to be the only viable path forward for modern VC-backed OSS startups.

Maintainership, software foundations, and fellowships

If we assume that the path forward for revenue generating OSS is a vibrant community and an ecosystem of companies built on top, it becomes extremely important to talk about the maintainers. Who pays the key maintainers? How do they stay impartial such that they allow vibrant companies to form around the software in a way that does not favor any one company? How are they compensated such that they can earn a salary roughly commensurate with what they could earn if they worked for a major tech company either on OSS or on proprietary software and systems?

Today, we unfortunately do not have consistent answers to the above questions. Key maintainers currently operate under an almost feudal system of patronage, hopping from company to company, trying to earn a living, keep the community vibrant, and all the while stay impartial (which I would argue is almost impossible to do if earning a salary from a company that has a commercial interest in the OSS project).

One possible solution to this problem is for the various OSS foundations to step up and start providing fellowships to key maintainers. The foundations already raise substantial amounts of money from members to support their activities. It’s a natural progression for the foundation to raise money directly for fellowships to support the work of key maintainers on key projects. And I think it is a win-win for everyone involved. Here is why:

1. By allowing key maintainers to stay independent, it becomes much more likely that a project’s community remains robust and does not favor any individual corporate entity. This creates a larger community and a more vibrant ecosystem for “loose” open core and SaaS companies to be built on top.
2. By funding fellowships, companies that utilize the OSS for revenue generation ensure that implicit project governance does not inadvertently fall into the hands of a competitor.
3. Finally, because the fellowships are likely to be funded primarily by large Internet companies, they can pay well, with salaries commensurate with what the maintainers would earn elsewhere.

In short, everyone wins.

The future of OSS

It’s very difficult to foresee how the future of revenue generating OSS is going to play out over the next 10–20 years. Will most core infrastructure software be subsumed by the large clouds making all of this a moot point? Will VC-backed OSS startups continue to find success operating independently?

In any case, the march of OSS across the industry will not stop, that is a given. Because of this, as an industry, we are going to have to come to terms with the economic reality: nothing is free, including OSS. If we want vibrant OSS projects maintained by engineers that are well compensated and not conflicted, we are going to have to decide that this is something worth paying for. In my opinion, fellowships provided by OSS foundations and funded by companies generating revenue off of the OSS is a great way to start down this path.

What is DevOps

The term DevOps means different things to different people. Before I dive into my thinking on the subject, I think it’s important to be clear about what DevOps means to me.

Wikipedia defines DevOps as:

DevOps (a clipped compound of “development” and “operations”) is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, releasing to deployment and infrastructure management. DevOps aims at shorter development cycles, increased deployment frequency, and more dependable releases, in close alignment with business objectives.

I use a more narrow and slightly different definition of DevOps:

DevOps is the practice of developers being responsible for operating their services in production, 24/7. This includes development using shared infrastructure primitives, testing, on-call, reliability engineering, disaster recovery, defining SLOs, monitoring setup and alarming, debugging and performance analysis, incident root cause analysis, provisioning and deployment, etc.

The distinction between the Wikipedia definition and my definition (a development philosophy versus an operational strategy) is important and is framed by my personal industry experience. Part of the DevOps “movement” is to introduce slow moving “legacy” enterprises to the benefits of modern highly automated infrastructure and development practices. These include things like: loosely coupled services, APIs, and teams; continuous integration; small iterative deployments from master; agile communication and planning; cloud native elastic infrastructure; etc.

For the last 10 years of my career, I have worked at hyper-growth Internet companies including AWS EC2, pre-IPO Twitter, and Lyft. Additionally, primarily due to creating and talking about Envoy, I’ve spent the last two years meeting with and learning about the technical architectures and organizational structures of myriad primarily hyper-growth Internet companies. For all of these companies, embracing automation, agile development/planning, and other DevOps “best practices” is a given as the productivity improvements are well understood. Instead, the challenge for these engineering organizations is how to balance system reliability against the extreme pressures of business growth, personnel growth, and competition (both business and hiring). The rest of this post is based on my personal experience, which I recognize may not be applicable to all situations, especially slower moving traditional enterprises that may be used to deploying their monolithic software once per quarter and are being coaxed into more rapid and agile development practices.

A brief history of operating Internet applications

Over the past approximately thirty years of what I would call the modern Internet era, Internet application development and operation has gone through (in my opinion) three distinct phases.

1. During the first phase, Internet applications were built and deployed similarly to how “shrink-wrapped” software was shipped. Three distinct job roles (development, quality assurance, and operations) would collaborate to move applications from development to production over typically extremely long engineering cycles. During this phase, every application was deployed in a dedicated data center or colocation facility, further necessitating operations personnel who were familiar with site-specific network, hardware, and systems administration.
2. During the second phase, spearheaded primarily by Amazon and Google in the late 90s and early 00s, Internet applications at fast moving hyper-growth companies started to adopt practices similar to the modern DevOps movement (loosely coupled services, agile development and deployment, automation, etc.). These companies still ran their own (very large) data centers, but due to the scales involved, could also start developing centralized infrastructure teams to tackle common concerns required by all services (networking, monitoring, deployment, provisioning, data storage, caching, physical infrastructure, etc.). Amazon and Google, however, never fully unified development job roles (Amazon via the Systems Engineer and Google via the Site Reliability Engineer), recognizing the differing skills and interests involved in each.
3. During the third, or cloud native, phase, Internet applications are now built from the ground up to rely on hosted elastic architecture, typically provided by one of the “big three” public clouds. Getting product to market as fast as possible has always been the primary goal given the high likelihood of failure, however in the cloud native era the base technology available “out of the box” allows a rate of iteration that dwarfs what came before. The other defining feature of companies that have started in this era has been eschewing the practice of hiring non-software engineer roles. The available infrastructure base is so relatively robust they reason — I would argue correctly — that startup headcount dollars are better spent on software developers who can do both engineering and operations (DevOps).

The movement towards not hiring dedicated operations personnel in phase three companies is critically important. Although, clearly, such a company does not need full-time system administrators to manage machines in a colocation facility, the type of person who would have previously filled such a job also typically provided other 20% skills such as system debugging, performance profiling, operational intuition, etc. New companies are being built with a workforce that lacks critical, not easily replaceable, skillsets.

Why does DevOps work well for modern Internet startups?

DevOps, as I have defined it (engineers who develop and operate their services), works extremely well for modern Internet startups for a couple of different reasons:

1. In my experience, successful early stage startup engineers are a special breed of engineer. They are risk tolerant, extremely quick learners, comfortable getting things done as fast as possible regardless of the tech debt incurred, can often work in multiple systems and languages, and typically have prior experience with operations and systems administration, or are willing to learn as they go. In short, the typical startup engineer is uniquely suited to being a DevOps engineer, whether they want to call themselves one or not (I prefer not 😉).
2. As I described above, modern public clouds provide an incredible infrastructure base to build on. Most basic operational tasks of the past have been automated away, leaving a substrate that is good enough to ship a minimum viable product and see if there is product market fit.
3. I’m a firm believer that when developers are forced to be on-call and accountable for the code they write, the quality of the system improves. No one likes to get paged. This feedback loop builds a better product, and as I described in (1), the typical engineer attracted to working on an early stage startup product is perfectly willing to learn and do the operational work. This is especially true given that there is often little repercussion for an early startup product having poor reliability. Reliability needs to be just good enough for the product to find market fit and enter the hyper-growth phase.

What happens when a modern Internet startup undergoes hyper-growth?

Most startups fail. That’s the reality. As such, any early startup that is spending a lot of time creating infrastructure in the image of Google is just wasting time. I always tell people to stick with their monolithic architecture and not worry about anything else until human scalability issues (communication, planning, tight coupling, etc.) necessitate a move towards a more decoupled architecture.

So what happens when a modern (phase three) Internet startup finds success and enters hyper-growth? A couple of interesting things start happening at the same time:

1. The rate of personnel growth rapidly increases, causing severe strains on communication and engineering efficiency. I highly recommend reading The Mythical Man-Month (which is still largely relevant almost 50 years after its initial publication) for more information on this topic.
2. The above almost always results in a move from a monolithic to microservice architecture as a way to decouple development teams and yield greater communication and engineering efficiency.
3. The move from a monolithic to microservice architecture increases system infrastructure complexity by several orders of magnitude. Networking, observability, deployment, library management, security, and hundreds of other concerns that were not difficult previously are now major problems that need to be solved.
4. At the same time, hyper-growth means traffic growth and the resultant technical scaling issues, as well as greater repercussions for both complete failure and minor user experience issues.

Central infrastructure teams

Almost universally following the early startup phase, modern Internet hyper-growth companies end up structuring their engineering organizations similarly. This common structure consists of a central infrastructure team supporting a set of vertical product teams practicing DevOps (whether they call it that or not).

The reason the central infrastructure team is so common is that, as I discussed above, hyper-growth brings with it an associated set of changes, both with people and underlying technology, and the reality is that state of the art cloud native technology is still too hard to use if every product engineering team has to individually solve common problems around networking, observability, deployment, provisioning, caching, data storage, etc. As an industry we are tens of years away from “serverless” technologies being robust enough to fully support highly reliable, large-scale, and realtime Internet applications in which the entire engineering organization can largely focus on business logic.

Thus, the central infrastructure team was born to solve problems for the larger engineering organization above and beyond what the base cloud native infrastructure primitives provide. Clearly, Google’s infrastructure team is orders of magnitude larger than that of a company like Lyft because Google is solving foundational problems starting at the data center level, while Lyft relies on a substantial number of publicly available primitives. However, the underlying reasons for creating a central infrastructure organization are the same in both cases: abstract as much infrastructure as possible so that application/product developers can focus on business logic.

The fallacy of fungibility

Finally, we arrive at the idea of “fungibility,” which is the crux of the failure of the pure DevOps model when organizations scale beyond a certain number of engineers. Fungibility is the idea that all engineers are created equal and can do all things. Whether stated as an explicit hiring goal (as at least Amazon does and perhaps others), or made obvious by “bootcamp” like hiring practices in which engineers are hired without a team or role in mind, fungibility has become a popular component of modern engineering philosophy over the last 10–15 years at many companies. Why is this?

• As I already described, modern cloud native technology and abstractions allow extremely feature rich applications to be built with increasingly sophisticated infrastructure abstractions. Naturally, some specialist skills such as data center design and operations are no longer required for most companies.
• Over the last 15 years, the industry has focused on the idea that software engineering is the root of all disciplines. For example, Microsoft has phased out the traditional QA engineer and replaced it with the Software Test Engineer, the idea being that manual QA is not efficient and all testing should be automated. Similarly, traditional operations roles have been replaced with site reliability engineering (or similar), the idea being that manual operations is not efficient, and the only way to scale is through software automation. To be clear, I agree with these trends. Automation is a more efficient way to scale.

However, this idea taken to its extreme, as many newer Internet startups have done, has resulted in only generalist software engineers being hired, with the expectation that these (DevOps) engineers can handle development, QA, and operations.

Fungibility and generalist hiring typically works fine for early startups. However, beyond a certain size, the idea that all engineers are swappable becomes almost absurd for the following reasons:

• Generalists versus specialists. More complex applications and architectures require more specialist skills to succeed, whether that be frontend, infrastructure, client, operations, testing, data science, etc. This does not imply that generalists are no longer useful or that generalists cannot learn to become specialists, it just means that a larger engineering organization requires different types of engineers to succeed.
• All engineers do not like doing all things. Some engineers like being generalists. Some like specializing. Some like writing code. Some like debugging. Some like UI. Some like systems. A growing engineering organization that supports specialists has to grapple with the fact that employee happiness sometimes involves working on certain types of problems and not others.
• All engineers are not good at doing all things. Throughout my career, I have met many amazing people. Some of them can start with empty files in an IDE and create an incredible system from scratch. At the same time, these same people have little intuition for how to run reliable systems, how to debug them, how to monitor them, etc. Conversely, I have been on many infuriating interview loops in which a truly incredible operations engineer who could add tremendous benefit to the overall organization purely via expertise in debugging and innate intuition on how to run reliable systems has been rejected because they did not demonstrate “sufficient coding skills.”

Ironically and hypocritically, organizations such as Amazon and Facebook prioritize fungibility in software engineering roles, but clearly value the split (but still overlapping) skillset between development and operations by continuing to offer different career paths for each.

The breakdown

How and at what organization size does pure DevOps breakdown? What goes wrong?

• Move to microservices. By the time an engineering organization reaches ~75 people, there is almost certainly a central infrastructure team in place starting to build common substrate features required by product teams building microservices.
• Pure DevOps. At the same time, product teams are being told to do DevOps.
• Reliability consultants. At this organization size, the engineers who have gravitated towards working on infrastructure are very likely the same engineers who either have previous operational experience or good intuition in that area. Inevitably, these engineers become de facto SRE/production engineers and help the rest of the organization as consultants while continuing to build the infrastructure required to keep the business running.
• Lack of education. As an industry, we now expect to hire people who can step in and develop and operate Internet services. However, we almost universally do a terrible job of both the new hire and continuing education required to perform this task. How can we expect engineers to have operational intuition when we never teach the skills?
• Support breakdown. As the engineering organization hiring rate continues to ramp, there comes a point at which the central infrastructure team can no longer both continue to build and operate the infrastructure critical to business success, while also maintaining the support burden of helping product teams with operational tasks. The central infrastructure engineers are pulling double duty as organization-wide SRE consultants on top of their existing workload. Everyone understands that education and documentation is critical, but scheduling time to work on those two things is rarely prioritized.
• Burnout. Worse, the situation previously described creates a human toll and reduces morale across the entire organization. Product engineers feel they are being asked to do things they either don’t want to do or have not been taught to do. Infrastructure engineers begin to burnout under the weight of providing support, knowing that education and documentation are needed but unable to prioritize the creation of it, all the while keeping existing systems across the company running at high reliability.

At a certain engineering organization size the wheels start falling off the bus and the organization begins to have human scaling issues with a pure DevOps model supported by a central infrastructure team. I would argue this size is dependent on the current maturity of public cloud native technology and as of this writing is somewhere in the low hundreds of total engineers.

Is there a middle ground between the “old way” and the DevOps way?

For companies older than approximately 10 years, the site reliability or production engineering model has become common. Although implementation varies across companies, the idea is to employ engineers who can wholly focus on reliability engineering while not being beholden to product managers. Some of the implementation details are highly relevant however, and these include:

• Are SREs on-call by themselves or do software engineers share the on-call burden?
• Are SREs doing actual engineering and automation or are they being required to perform only manual and repetitive tasks such as deployments, recurring page resolution, etc.?
• Are SREs part of a central consulting organization or are they embedded within product teams?

The success of the program and its impact on the overall engineering organization is often dependent on the answers to the above questions. However, I firmly believe that at a certain size, the SRE model is the only effective way to scale an engineering organization to a number of engineers beyond the point at a which a pure DevOps model breaks down. In fact, I would argue that successfully bootstrapping an SRE program well in advance of the human scaling limits outlined herein is a critical responsibility of the engineering leadership of a modern hyper-growth Internet company.

What is the right SRE model?

Given the plethora of examples currently implemented in the industry, there is no right answer to this question and all models have their holes and resultant issues. I will outline what I think the sweet spot is based on my observations over the last 10 years:

• Recognize that operations and reliability engineering is a discrete and hugely valuable skillset. Our rush to automate everything and the idea that software engineers are fungible is marginalizing a subset of the engineering workforce that is equally (if not more!) valuable than software engineers. An operations engineer doesn’t have to be comfortable with empty source files just the same as a software engineer doesn’t have to be comfortable debugging and firefighting during a stressful outage. Operations engineers and software engineers are partners, not interchangeable cogs.
• SREs are not on-call, dashboard, and deploy monkeys. They are software engineers who focus on reliability tasks not product tasks. An ideal structure requires all engineers to perform basic operational tasks including on-call, deployments, monitoring, etc. I think this is critically important as it helps to avoid class/job stratification between reliability and software engineers and makes software engineers more directly accountable for product quality.
• SREs should be embedded into product teams, while not reporting to the product team engineering manager. This allows the SREs to scrum with their team, gain mutual trust, and still have appropriate checks and balances in place such that a real conversation can take place when attempting to weigh reliability versus features.
• The goal of embedded SREs is to increase the reliability of their products by implementing reliability oriented features and automation, mentoring and educating the rest of the team on operational best practices, and acting as a liaison between product teams and infrastructure teams (feedback on documentation, pain points, needed features, etc.).

A successful SRE program implemented early in the growth phase as outlined above, along with real investment in new hire and continuing education and documentation, can raise the bar of the entire engineering organization while mitigating many of the human scaling issues previously described.

Conclusion

Very few companies reach the hyper-growth stage at which point this post is directly applicable. For many companies, a pure DevOps model built on modern cloud native primitives may be entirely sufficient given the number of engineers involved, the system reliability required, and the product iteration rate the business requires.

For the relatively few companies for which this post does apply, the key takeaways are:

• DevOps style agile development and automation is required for any new technology company that hopes to compete.
• Publicly available cloud native primitives along with a small central infrastructure team can allow an engineering organization to scale to hundreds of engineers before the operational toll due to lack of education and role specificity starts to emerge.
• Getting ahead of the operational human scaling issues requires a real investment in new hire and continuing education, documentation, and the development of an embedded SRE team that can form a bridge between product teams and infrastructure teams.

Modern hyper-growth Internet companies have (in my opinion) an egregiously large amount of burnout, primarily due to the grueling product demands coupled with a lack of investment in operational infrastructure. I believe it is possible for engineering leadership to buck the trend by getting ahead of operations before it becomes a major impediment to organizational stability.

While newer companies might be under the illusion that advancements in cloud native automation are making the traditional operations engineer obsolete, this could not be further from the truth. For the foreseeable future, even while making use of the latest available technology, engineers who specialize in operations and reliability should be recognized and valued for offering critical skillsets that cannot be obtained any other way, and their vital roles should be formally structured into the engineering organization during the early growth phase.

Exception versus return code based error handling

I’m not going to spend a huge amount of time in this short post discussing the pros and cons of using exception versus return code based error handling. Much has been written, many arguments have been had, and many relationships have been destroyed.

My personal philosophy on this topic is nuanced. I think the least convoluted code uses both exception and return code based error handling. In my own code, I use roughly the following two rules to decide which method to use:

1. If an error is typically propagated at most one stack frame (i.e., to the immediate caller of a function), return code error handling should be used.
2. If an error is typically propagated more than one stack frame (i.e., up through several callers of a function), exception based error handling should be used.

The reasoning behind the above two rules is quite simple: When errors have to traverse a short distance, return codes are less verbose. When errors have to traverse a large distance, exceptions are far less verbose. Less code is generally easier to understand, easier to test, and easier to maintain.

As a language, Go makes a point of not supporting exceptions, claiming that the try/catch idiom leads to convoluted code. Although there are some things that I love about Go (and many things that I hate), dogmatic black and white statements such as these are not one of them. Engineering is a constant set of tradeoffs that are situationally dependent.

However, I’m going to let you in on a little secret. Go does support exceptions perfectly well and I’m going to show you how to use them in situations that will yield cleaner and more maintainable code.

Exception based error handling in Go

Let me start by doing some quick translations:

• defer = finally
• panic = throw
• recover = catch

Dogmatism that isn’t backed up by reality is the best kind, right? (It is for me given that I think exceptions are great 😉). In the following set of short code snippets I will demonstrate how to add exception based error handling to your Go code.

Defining the exception type

Figure 1 demonstrates defining a very simple “exception” type RateLimitConfigError. In this example, the type is simply an alias for a string, but it could just as easily be a struct if more complex error information needs to be carried. We also define an Error() function for use in logging.

Throwing an exception

Figure 2 demonstrates “throwing” an exception in Go code. Since panic() raises an exception, all we need to do is panic() with our exception type.

Catching an exception

Figure 3 demonstrates how to catch an exception in Go. This example is a little more interesting as we have to bend Go slightly in order to do what is needed.

1. reloadConfig() is a function that ultimately becomes highly nested, lending itself very well to exception based error handling based on the two error handling style rules I laid out above.
2. At line 2 we setup a defer statement with a finalizer.
3. At line 3 we attempt to recover (catch) any raised exceptions.
4. At line 4 we check the type of any caught exceptions. If the type is not what we expect we re-panic the exception on line 6. This prevents catching any errors that we did not intend.
5. Starting at line 9 we take action on the exception data.

This code is obviously more verbose than I would like, but it’s pretty clear that at the runtime level Go fully supports exception based error handling semantics. In fact, with a tiny amount of syntactic sugar on the catching side, the above code could be simplified to normal try/catch keywords that many programmers are familiar with.

(As an aside, figure 1 also demonstrates one of my favorite Go programming styles aimed squarely at annoying the dogmatic zealots: I refer to all object variables as this as seen on line 1. I know, I’m a bad person. 😃)

Conclusion

Depending on the situation, the use of exception based error handling can lead to vastly cleaner code. Conveniently, although Go promotes itself as a language that does not support exceptions, that is not really true. And in fact, the standard library itself is contradictory in this regard, presumably for the same reasons I have already laid out. The article on defer, panic, and recover goes on to say:

For a real-world example of panic and recover, see the json package from the Go standard library. It decodes JSON-encoded data with a set of recursive functions. When malformed JSON is encountered, the parser calls panic to unwind the stack to the top-level function call, which recovers from the panic and returns an appropriate error value (see the ‘error’ and ‘unmarshal’ methods of the decodeState type in decode.go).

When I was first learning Go, frustrated by the lack of exceptions, and found this text I actually laughed out loud.

For a more complete example of this style of programming, take a look at Lyft’s rate limit service, in which I break many of Go’s dogmatic rules in ways that, in my opinion, lead to cleaner and more readable code (and of course annoy the zealots).

With this newfound knowledge, go forth, spread the gospel, and write vastly more “exceptional Go!”