Mental Models for Site Reliability

we’re recovering from too much dominoes and cheese

6 min readMay 4, 2025

Google’s SRE approach has become the unofficial standard for supporting digital services. If you want to get a job maintaining SaaS products or online services, you must read the Google SRE book: the interview questions will surely reference its jargon.

And, it’s great — as far as it goes. Concepts like the ‘4 Golden Signals’, SLOs and ‘Error Budgets’ gave us new ways to optimise application health, which are easy-to-learn and easy-to-track. Google-style Site Reliability Engineering supports the high rates of change that other practices like DevOps and Continuous Delivery demand.

But, not every company is a Google: your SRE career will throw up unique challenges that aren’t covered by any textbook. So, it’s good to have some awareness of the more general fields of Reliability, Risk and Safety Engineering, which inspire a lot of what we do nowadays in digital SRE. If you know the core principles, you have a head start whenever a novel solution is needed.

In this post, I’ll discuss some of the other ways people have modelled how failure happens and how we can understand causality in complex systems — and how these models help us fix things when they break.

Heinrich’s Dominoes

Herbert W. Heinrich published¹ his Domino model in 1931. It pictures industrial systems like a line of dominoes, standing on end. When any one domino is toppled, all the dominoes in line after it fall over too.

This model was popular for a while, then it went out of favour. Some of the criticisms were that:

It implies a perfectly ordered and safe system, which collapsed due to one single error (“everything was fine until Fred dropped a wrench in the gearbox”)
It suggests linear causality: a single failure starts a chain of events that leads inevitably to a single bad outcome. This is rarely realistic
It tends to blame individual human error, rather than lack of safety-by-design in the system. (Although this criticism might be unfair on Heinrich, who was an advocate of improved safety features in machinery)
It implicitly accepts Taylorism and Henry-Fordism, whose view of a fixed production chain — with humans as the weakest link — are hardly applicable to modern knowledge and service industries.

I’m sure any experienced SRE will recognise the Domino Model as too simplistic. No complex, distributed system is ever ‘perfectly’ working, and it is rare for a single error or component failure in a well-designed system to result in total failure.

This doesn’t mean the model has no value at all. For example, the useful technique of Root Cause Analysis (RCA) implies a domino-like model. RCA can be used to work through the sequence of events in an incident, and identify target points for improving safety or reliability. However, it’s dangerous to believe an incident is fully understood and preventable just because we identified a single ‘root cause’.

Reason’s Swiss Cheese

James T. Reason² imagined accidents and system errors by analogy to slices of Emmental cheese.

Each layer of protection against failures is like a slice of cheese. It will block some failure modes from having a negative consequence. The more layers, the more protection. When the system fails, it means that some specific problem managed to bypass all the cheese by wriggling through aligned holes in the stack.

The Swiss Cheese Model was prominent in the 1990s and early 2000s. It became part of more holistic fault analysis tools like Tripod Beta. It is still widely used.

Get Colm Campbell’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

When it comes to thinking about digital SRE, the Swiss Cheese Model has pros…

It aligns well with the Defence in Depth approach of InfoSec
It contains an implicit acceptance that no single safety feature will provide 100% assurance of reliability
It accepts that multiple people and multiple system components may influence the outcome of an unexpected condition

… and cons …

It’s easy to misunderstand it as recommending many, haphazardly constructed, safety features, to get the best probability of success. This undervalues carefully designing reliability into a system.
It conceals information about which measures are most important, most effective, and worth investing in.

At the very least, an SRE should be able to refer to this model, to justify having multiple monitoring, alerting and failover tools — to provide cover when errors evade detection in the primary tool.

Rasmussen’s Safety Envelope

Some recent models of industrial safety and reliability acknowledge that a range of social, commercial and technical factors are at play.

[image credit: https://risk-engineering.org/concept/Rasmussen-practical-drift]

Jens Rasmussen³ notes that any industrial system must be: commercially viable; operable by a limited number of humans; and have acceptable safety outcomes. The system can fail if it becomes too costly or unproductive, too time consuming or too complex to maintain, or too unsafe. It can operate well inside a (potentially) wide space of operating parameters: its Safety Envelope.

Rasmussen, and other researchers like Snook⁴ and Dekker⁵, note various ways that a well-planned system can drift into danger:

cost-cutting efforts may be unaware of their reliability or safety implications
incorrect practices may become normal, because they were done a few times without any visible ill effect.
Workers develop procedural shortcuts to save time or meet productivity targets.

In this kind of model, the linear causal chains of Domino theory can never really apply — since the ultimate cause of a failure or accident could be in a gradual shift in operational behaviours over several years.

Like the other models, this one is useful for SREs, but imperfect. The reality of digital systems is that most actions are logged, and most outcomes are traceable to specific causes. No SRE should be satisfied with submitting an incident report that attributes failure to “a complex range of socio-technical degradations of practice”, or similar. We have significantly more direct control over outcomes than our colleagues in some industry sectors, and significantly more visibility of causal factors.

Complexity Theories

When faced with a large, distributed technology system — a muddle of networks, microservices, and third parties — it can be tempting to accept that it is too complicated to understand. We can start to use analogies with biological systems, like rainforests or the human body. We often hear talk of ‘tech ecosystems’.

It is absolutely true that most modern software-plus-cloud domains are too big and too complex for any one human brain to fully map out and understand in detail. Nobody can see an APM alert and ‘just know’ for sure what has gone wrong. Even so, we don’t need to throw our hands up in despair.

Site Reliability Engineering is an extension of Software Engineering (with input from Safety, Risk and other Reliability Engineering fields). Like SWE, SRE depends on componentisation, with careful use of coupling between and cohesion within components. Like SWE, it is all about managing and reducing complexity, to make systems suitable to be maintained by people.

None of the models I’ve mentioned here is complete or sufficient for everything an SRE practitioner will do. But when used judiciously, they can all help us get to grips with an incident in progress, and plan for continually better reliability while our site is healthy.