Early in my career, I thought about software systems as static, unchanging entities. If you’ve ever thought of service maps or architectural diagrams as a panacea (instead of yet another lens), or if you’ve ever talked about software being “done,” you’ve probably done the same.

When we think of software this way, though, we miss a fundamental property — people are constantly changing software, and those people almost certainly have different short and long-term goals, contexts, and pressures. Even software in “maintenance mode” has bug fixes, changes in its environment, and changes in the kind of usage patterns it supports.

One way to visualize a software system’s lifecycle is with a simple timeline and markers that represent deploys or changes to the software code and configuration. Let’s use red markers to represent deploys:

This visualization makes it clear that the software is changing over time. The new detail is good, as it prepares us for a mental model that embraces that software is a dynamic, ever-changing system. It still leaves a lot out, though.

The above visualization assumes that a software application lives in a perfect world without dependencies or changes in its environment. That’s not true. It’s impossible to separate operations from software development. So let’s add markers (in blue) to represent some of the changes we see in a software application’s environment:

Same system, but now we’ve added information about changes to the operational environment, like upgrading the version of MySQL that this fictional system uses, increasing the instance size of the AWS EC2 instances that this system runs on, or upgrading the OS kernel. These are all routine tasks when operating a web-based software system running in a public cloud environment, and they should impact the way we think about our software systems. The changes might have been made by the team responsible for deploying changes to code or a team responsible for purely operational work.

Our visualization represents a continuously changing system, both in its software code and the operational environment hosting it. These are both in the team’s control or teams responsible for developing and operating this system, but there is still something missing. What about customers or users of this application? They’re constantly interacting with it in ways that will influence its development and operation. Let’s add more markers (in green this time) that represent significant shifts in how users interact with our application:

Now that we’ve added humans to our timeline, we can see more of the story. Through this lens, it might become apparent that specific changes in user behavior may have motivated some of the operational changes. The ten-fold increase in average requests per second was probably a motivator in increasing the instance sizes. Or was it? The only people who can answer that are the humans who did the work, and we haven’t added them yet. Let’s add staffing changes to the team or teams responsible for this system. We’ll add these as purple markers:

This new dimension adds a fresh perspective to the system. We now know that we can ask Mary about the motivation for changing the instance size. Mary may not know anything about why the team upgraded MySQL when it did, though. In the world Mary lives in, the team has always been running MySQL 8. We know we can’t ask Aditya about the 10x spike in requests because he left the team before it happened. We also can’t ask the five engineers who joined later because that level of traffic has always been customary for them.

When we add human beings to our visualization, it becomes evident that the system is sociotechnical. The humans are doing the work and have the context behind the changes they are making. But where did they get that context? Often it’s from participating in specific incidents. I’m a big fan of the idea that incidents are unplanned investments. The return on those investments is learning. Therefore, the lessons learned from analyzing incidents can reveal important information about your technical systems, organization, and how various parts interact. Let’s add yellow markers for significant incidents into our timeline:

Incidents provide context for the work we do. If we have questions about the MySQL 8 migration, the answers are probably informed by whatever happened in the “MySQL collation” incident. Visualizing our system this way also makes it clear that incidents can be powerful storytelling mechanisms. Investing in good quality incident write-ups can extend the value of incidents for engineers who join after the fact.

Lenses into a System

Each set of markers we added to our system provided new context to form assumptions and frame our thinking. Everything in the visualization existed whether we were looking or not. It becomes clear when looked at this way that each of these dimensions is inextricably linked. It’s impossible to think holistically about software without thinking about the operational environment, or the users of the system, or the people involved in building and maintaining it. These things come together to create another lens through which we can view the world.

It’s important to point out that the final image here is still incomplete. We’ll never fit all of the contexts into a single model. We could keep going, adding more and more context. A fascinating one, for example, would be marking the beginning of the COVID pandemic, when a team that perhaps was colocated started working remotely, and when stress and risk of burnout increased considerably. We have to stop somewhere, otherwise, we’ll eventually include the whole world in our system, but it’s interesting to continually zoom out and see how a new lens helps frame our perspectives.

Embrace Sociotechnical Perspectives

So our systems are sociotechnical. What do you do about this? I think it’s clear that you start by investing in the social parts of the system. If you want to understand why your systems perform the way they do, it’s necessary to know how expertise is created and distributed amongst the people in your organization. The single best way to do this is to invest in incident analysis.

Many organizations have adopted the practice of doing “post-mortems” or “retrospectives” after incidents. Retrospectives are great! Unfortunately, I think a lot of learning is left on the table by the adoption of template-driven processes that produce shallow understandings of what transpired. I’ve spoken about how I think we can improve this. There are also experts in the field who provide training and consulting in incident analysis. There are also communities and companies dedicated to improving this practice.

Investing in incident analysis will empower teams to discover what they know, and almost as importantly, how they know what they know. It’s a necessary part of understanding the social aspects of your system, which hopefully this post demonstrates, are inextricable from any other part.

Sociotechnical Lenses into Software Systems was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Software is a big space and in the span of a career, you can move in a bunch of different directions, all interesting to different people. As I approach my 20th year as a professional software person, I thought it’d be interesting to reflect on where my career has taken me so far, and how I hope to spend the next 20 years. I found it useful to write down questions I keep coming back to and use them as a guide for my continued learning. I’ll be interested to come back to this post and see how it compares to whatever “future me” gets up to.

Some of these questions have answers and I just need to learn more about them. Others are active research areas with no “right” answer. Regardless, they’re things I keep thinking about, reading about, and occasionally applying as a practitioner.

Of course, these aren’t the only things I’m interested in, they’re what I think about the most. If anything, this exercise was a good way to make clear the areas I’m not interested in enough to dedicate serious time to. For example, I think that data visualization is an amazing topic. In a world with infinite time, I’d love to dive into it, but probably not in this career. So, with that said, here are my questions:

How can we ensure that data flows through a system in ways that are performant, scalable, and reliable? How do we verify the integrity of data, especially in increasingly large-scale and complex data systems?

This is a broad area, but it’s useful as a guide. My core technical interests gravitate towards backend engineering and this is why. I think that most of what we do when working on 24/7 internet services boils down to accepting requests to retrieve or mutate data. Retrieving data necessitates that we do it quickly and correctly for some definition of correct (I’m really interested in when that definition becomes flexible, see the fourth set of questions below). When mutating data, we want to ensure that a write is either accepted or rejected, i.e., the operation is reliable. It’s rarely acceptable to mysteriously drop writes, it's sometimes acceptable to repeat writes, and often it’s required that writes happen in a specific order. This is all trivial with one process working at a time but gets really hard when you have large numbers of concurrent operations operating on more data than can possibly fit on one computer, and in a world where you have to be able to tolerate the failure of one or more computers.

The fun thing about this question is how broad it is. You can start at a really high level, with systems design concepts, and deep dive in various directions. For example, load distribution and queuing theory are complete topics in their own right. You can also study the data structures used to store data in memory and on disk, to the protocols used to make decisions about how and where to store data, all the way down to lower-level concepts like filesystem and network performance. At every path, there are tradeoffs to consider and I doubt anyone could ever be considered an expert in all these areas, but it’s really fun stuff to think about and has real implications.

How can systems be built to maximize operability? What techniques can be followed to allow operators to observe the behavior of a system, make decisions about its operation, and act on those decisions?

I want to help make life better for those of us who keep things running, as well as the people who depend on the services we build and operate. What are the properties of a system that make it easier to operate compared to the properties of a system that operators fear? Luckily, I have a job that allows me to work directly on one aspect of this problem: Observability. Observability refers to the degree to which a system’s internal state can be inferred by looking at external outputs. Doing this well requires a lot of thought and context.

This question also touches on complexity in systems. It’s intuitive, but often wrong, to try and remove complexity — as system designers, we have this impulse to try and hide complexity, but this can often create dangerous situations for operators. Black boxes can have interesting, negative effects. This video, about why cars rarely crash into buildings in the Netherlands, explains it rather well, I think. In software systems, operators often need to be able to respond to operational failures never considered by the designer. Systems can fail in fundamentally surprising ways, and operators need to be able to act on whatever they are seeing.

I also think about the tunability of a system. Is it possible to affect the internal state of software without deploying a change? For example, feature flags are a useful way of toggling functionality and can be used effectively to handle unexpected failures or conditions that degrade functionality. Traffic control is another area where good complexity can be introduced. The ability to direct small amounts of traffic to alternate versions of running software can be tremendously helpful to operators responsible for rolling out changes safely. Being able to make quick decisions about configuration changes, for example, the version of a piece of software that a part of a system should run is crucial in my experience. Chaos engineering is another area where I think introducing complexity can be helpful.

How do operators learn about the systems they are responsible for? How does that knowledge help or hinder them in their duties? How do they teach others who are less familiar with the system? How do teams coordinate and make decisions during high-stress incidents?

I believe these are classic Cognitive Systems Engineering questions. Our software systems don’t run themselves, no matter what certain vendors might want you to believe. People keep systems running, and people are constantly doing things, constantly responding to things, and constantly changing things. Yes, change fail rates are a real thing, but mostly, these the actions that people take are what prevent outages or prevent outages from being worse than they are.

Have you ever worked with someone or a group of people who seemed to have an intuitive sense of what needed to happen when a certain alert fires or a graph doesn’t look right, or when a customer complains about a bad experience? Have you noticed how much you can learn by just being around that person or on their team? I’m fascinated by how operators obtain, continue to shape, and share that knowledge. I’ve heard this process referred to as sense-making and I think it’s an under-appreciated aspect of how we do our jobs.

I also know that we generally have different mental models of systems, and it’s usually true that none of them are correct, so the ways in which we use those different mental models to build shared understanding is fascinating to me.

People have written extensively on this subject, specifically in the context of software operations and outages and I want to dig in a lot more.

What are the tradeoffs when considering the needs of consistency, availability, and scalability? In order to guarantee coordination, is it necessary to introduce limits that interfere with a system's ability to scale? When is coordination unnecessary?

These are classic distributed systems questions and I find them really fun to think about. They come up in practical applications from time to time. If you think about data storage or any part of a distributed system that has to make decisions about how to handle a certain kind of operation, we run into some interesting constraints when considering needs of scale and availability. By scale, I mean handling volumes of data that can’t possibly be stored on a single disk, and by availability, I mean that the system will need to tolerate individual node failures, so data will have to be replicated. In such a system, how many nodes have to agree that a write has been accepted before a client can consider the operation successful? When adding nodes to a system, how is the performance impacted? Is it possible to create a system that requires no coordination, but can make safe guarantees about consistency? This is an active research area, and it’s fun to read papers from companies like Google, Facebook, and Amazon to see how practitioners are thinking about the various tradeoffs or fun ways to cheat. There seems to be a lot of back and forth between academia and industry here. CRDTs, for instance, is an attempt to get around the need for coordination while guaranteeing strong, eventual consistency. The Google Spanner team is no doubt doing some really interesting work here. It’s fascinating and I feel like you could spend a lifetime learning about all the work being done in this area and still not be caught up.

How do organizations learn? How do organizations internalize information necessary to guarantee their future success?

So you have a large scale data-driven system that incorporates a number of design decisions intended to prevent or isolate failures. You have groups of people who are constantly learning about and changing the system and coordinating their efforts in surprising and remarkable ways. How does an organization internalize this knowledge and avoid or make less likely certain organization failure modes? How do new members of a team learn about all the folk knowledge held within an organization? I’ve worked in organizations that were dangerously dependent on individuals, and not able to withstand commonplace things like vacation or attrition. These kinds of conditions are hell and lead to burnout for individuals, deteriorating quality for a product, and less reliability for users. I’m fascinated by the differences between teams who disseminate this knowledge well and teams that do not.

The next 20 years

Reading all of this, I could probably summarize my interests as: “building large scale distributed data systems that are safe to operate and discovering ways that teams internalize and share their operational knowledge.”. It’s fun to break it down further though, and useful as a guide for learning. As I described earlier, software engineering is a huge space with a seemingly infinite number of paths to explore. It was a useful exercise for me to write down the questions that seem to guide my learning and my career. I hope the next 20 years of my career afford me more opportunities to study the areas I’ve outlined, learn from my peers, mentors, and coworkers, and generally improve the software I work on. I feel incredibly lucky to work in a space with so many intellectually challenging problems, while also getting to help build products that people find useful.

Throughout my career, I’ve developed some opinions. Some have worn particularly deep ruts, reinforced by years of experience. I tried to figure out what these had in common, and it’s the idea that code in production is the only code that matters. Staging doesn’t matter, code on your laptop doesn’t matter, QA doesn’t matter, only production matters. Everything else is debt.

This perspective probably comes from years of sitting in between operations and product development. I strongly believe that teams should optimize for getting code to production as quickly as possible as well as responding to incidents in production.

This idea, and a lot of the practices it implies, can be counter-intuitive or controversial, so I want to dive in a bit. What follows is a set of practices and principles I believe are true, considering my underlying belief that code working in production is the only code that matters.

1. Engineers should operate their code.

Engineers are the subject matter experts for the code they write and should be responsible for operating it in production. In this context, “operating” means deploying, instrumenting, and monitoring code as well as helping to resolve incidents related to or impacting that code. The responsibility of operating code aligns incentives — it encourages engineers to write code that is observable and easy to debug, and connects them to what customers care about. It encourages them to be curious about how their code is performing in production. Importantly, engineers should be on-call for their code — being on-call creates a positive feedback loop and makes it easier to know if their efforts in writing production-ready code are paying off. I’ve heard people complain about the prospect of being on-call, so I’ll just ask this: if you’re not on-call for your code, who is?

If you’re not currently on-call for your code but want to be, and can help influence this decision, there are some things you can do. Set up PagerDuty (or similar) schedules for each group of engineers responsible for specific services or parts of your code. A good schedule has 6–8 engineers. There are plenty of variations, but a typical template is to have one-week rotations, where you’ll be on-call for secondary for a week and then primary for a week. Configuring alerts is a separate topic, which probably deserves it’s own blog post entirely, but focus on things that impact your customers (see: Symptom-based alerting) and remember that you’re ultimately responsible for how you respond to alerts, which means you can change them.

There are two talks I’d recommend watching that touch on the topic of configuring alerts: Liz Fong-Jones talks about SLOs in Cultivating Production Excellence and Aditya Mukerjee does a great job talking about techniques for managing alerts in Warning: This Talk Contains Content Known to the State of California to Reduce Alert Fatigue.

2. Buy Almost Always Beats Build

If you can avoid building something, you should. Code is the most expensive way to solve a problem that isn’t addressing a core area of your business. For most small to mid-sized companies, there are open source or better yet, hosted solutions that solve a wide range of common problems. I mean things like git repository hosting (Github, Gitlab, Bitbucket, etc), observability tooling (Honeycomb, Lightstep, etc), managed databases (Amazon RDS, Confluent Kafka, etc), alerting (PagerDuty, OpsGenie, etc) and a whole host of other commodity technologies. This even applies to your infrastructure — if you can help it, don’t roll your own Kubernetes clusters (side note: do you even need to use Kubernetes?), don’t roll your own load balancers if you can use Amazon ELB or ALBs.

Unfortunately, NIH syndrome is very real and some companies get burned badly by this. I’ve seen teams light time and money on fire reinventing components when better, more battle-tested alternatives exist in the market. Those same teams almost always end up spending years contending with the resulting technical debt. If you’re on such a team and have the will and ability to impact change, start rolling back these decisions one by one. Migrate your databases to a managed provider, migrate your feature flagging system to a SaaS tool (i.e. LaunchDarkly). Keep going until the only software you maintain yourselves is the software that delivers value to your customers. You’ll be much, much better off for it.

3. Make Deploys Easy

Deploying should be a frequent and unexciting activity. Engineers should be able to deploy with minimal manual steps and it should be easy to see if the deploy is successful (this requires instrumenting your code for observability, which — tada — is covered above), and it should be easy to roll back a deploy if something doesn’t go well. Deploying frequently implies that deploys are smaller, and smaller deploys are generally easier, faster and safer.

Many teams implement periods where deploys are forbidden — these can be referred to as code freezes, or deploy policies like “Don’t deploy on Fridays”. Having such blackout periods can lead to a pile-up of changes, which increases the overall risk of something going very wrong.

If you’re on a team that fears deploys, dedicate a large percentage of your engineering time to improvements in your deployment pipeline until the fear is gone. On a recent team I worked with, we were able to improve deploy times from 3 hours to 30 minutes, which drastically improved the teams’ confidence in the deploy process. A natural side effect of this was that engineers started deploying much more frequently instead of waiting for changes to pile up enough to warrant a “release” (which was synonymous with a deploy).

The book Accelerate has been getting a lot of attention. If you haven’t read it, I’d recommend it. The team behind it also publishes the State of DevOps reports, which are full of well-researched information about what various companies in the industry are doing. It’s not a coincidence that two of the four key metrics that the book focuses on are directly related to this (Deploy Frequency, Change Lead Time). Shipping is your company’s heartbeat.

4. Trust the People Closest to the Knives

The people who work with a system are the ones who understand it best. This applies to any part of the socio-technical systems within which we all work. In the case of software systems, the engineers who deploy every day and are on-call for critical services understand the level of risk they operate in. A sad trend is that managers tend to overestimate their teams’ progress on certain transitions — i.e. cloud-native, DevOps, etc. The higher up the management chain, the larger this overestimation tends to be. Engineers who deploy and get paged when things break know where the bodies are buried and they know what needs the most work. They should, therefore, be the primary stakeholders responsible for prioritizing technical work.

Another manifestation of this principle applies to platform or services teams. If you’re responsible for building some shared component that’s used within your organization (i.e. a messaging system, ci/cd infrastructure, shared libraries or services) there’s an uncomfortable truth lurking for you: the people who use your work know more about it than you do in many cases. They understand implicitly how it serves customers and they know what contortions or hoops they have to jump through to get it to work. Listen to them for clues on how to improve the UX of your services and tools.

5. QA Gates Make Quality Worse

Many teams have a manual QA step that gets performed before deploys. The idea, I guess, is to have someone run automated or manual tests to verify that a set of changes are ready to be released. This sounds like a comforting idea — having a human being (or team of human beings) “verify” a release before it goes out — but it falls victim to several false assumptions and creates some misalignments that do more harm than good.

First of all, if there’s manual work that needs to be done before a deploy can go out, that creates a bottleneck — if you’re making deploys easy, and deploying small changes frequently, no QA team is going to be able to keep up testing every deploy, and will inevitably block teams from deploying. That’s no good. If you have manual tests, automate them and build them into your CI pipeline (if they do deliver value).

Secondly, the teams doing QA often lack context and are under time pressure. They may end up testing “effects” instead of “intents”. For example, I’ve seen QA teams burn time testing that when something happens in a UI, something related happens in a database. What happens when an engineer refactors that UI component and changes the underlying data model? The functionality works, but the test breaks. Because two teams are involved, this takes coordination and time to fix. Similarly, I’ve seen QA teams block deploys because of failing tests when caching was introduced at the CDN layer — a TTL of 5 seconds on an activity feed may not ever be noticed by a user but it might break QA tests causing unnecessary conflicts between product and QA engineers.

Luckily, solving this one is easy. Instead of having a dedicated QA team work on creating manual and automated test cases that run in a fictitious QA environment, reassign that team to work on continuous testing in production. Instead of being a gate for deploys, a QA team could continuously verify that production is working as expected. QA teams are also well situated to lead Chaos Engineering initiatives, where faults are intentionally injected in production. QA engineers could also work on making the CI/CD pipeline more reliable, so that deploys are no longer a nightmare.

6. Boring Technology is Great.

With thanks to Dan McKinley, always strive for boring tech when possible. Systems are inherently unpredictable, and you want a wide area of expertise to fall back on when shit goes sideways. There are also routine operations that you’ll have to do (deploys, database migrations, etc) and it’s Very Nice to have widely used and tested tooling for this stuff. I think of databases most often when I think about this belief. MySQL is a database with many, many quirks, but it is so widely used, that you should still just use it most of the time.

Very few organizations have the bandwidth to debug unique problems. You don’t want unique problems, especially when performing routine operations — i.e. storing bytes on disk, choosing a new leader in a cluster, garbage collecting objects, querying time-series data, etc. Having unique problems will kill a small to medium size team. It will sap you of your creative energy, which is better used creating value for customers who want to pay you monies for your software. Use your innovation tokens wisely!

Using boring technology means you can lean on a large community of users. Shit on it all you want, but there are very few PHP issues that someone else hasn’t already encountered. Nowadays, the same is probably true for sufficiently widely used versions of Ruby on Rails. I often say that I like to be in the 3rd cohort of technology adoption. The 1st cohort is the bleeding edge organization. The 2nd cohort is the people who feel like they can take some risks. Let those two groups go before you, run into all the big problems, and then you can go, benefiting from all of their hard-won experience.

7. Simple Always Wins

I don’t have much to say about this, but we’re all writing YAML and JSON instead of XML and we’re all using HTTP instead of CORBA, RMI, DCOM, XPCOM, etc. Right? In that same spirit, I’d rather debug problems in a LAMP stack than a Microservices architecture any day.

Quick sidebar on Microservices: as with so many trends in tech, they are often sold as a panacea. Let me be clear: Microservices, designed well, solve some specific problems and as with most solutions to complex problems, involve several trade-offs. If you are going in this direction, I do have opinions on how you should do it, but I also think you should hold off for as long as you can.

8. Non-Production Environments Have Diminishing Returns

A more direct heading for this section would be “Non-Production Environments are Bullshit”. Environments like staging or pre-prod are a fucking lie. When you’re starting, they make a little sense, but as you grow, changes happen more frequently and you experience drift. Also, by definition, your non-prod environments aren’t getting traffic, which makes them fundamentally different. The amount of effort required to maintain non-prod environments grows very quickly. You’ll never prioritize work on non-prod like you will on prod, because customers don’t directly touch non-prod. Eventually, you’ll be scrambling to keep this popsicle sticks and duct tape environment up and running so you can test changes in it, lying to yourself, pretending it bears any resemblance to production.

9. Things Will Always Break

It’s impossible, even undesirable, to avoid failure. Lean into the fact that failure is inevitable, and focus on how you respond to it. This means investing in a continuously improving incident response process. There’s no one-size-fits-all for every company and team, but you should have a good idea of what to do when things go wrong, and you should have mechanisms in place to learn from those situations and improve your processes. Invest in Incident Analysis. It’s a huge field with lots of valuable tools and resources for maximizing the return on investment when incidents occur (or don’t!).

This is an area where Chaos Engineering can be helpful. Injecting failures into production can improve confidence in how to respond when a system starts behaving in unexpected ways. Game Days can be a particularly effective way to allow a team of engineers to practice various outage scenarios.

Conclusion

A lot of the beliefs outlined in this post are at least counter-intuitive, if not somewhat controversial, but I’m nevertheless convinced that they’re true. That doesn’t mean my mind cannot be changed, but it is unlikely. If you strongly agree or disagree, I’m on the internets. I’d be very curious to hear about your experiences.