DevOps Brain Exorcises

I first published this around 2017. Just came back and found errors, fixed and saved; the date has changed.

Before 2 disparate parties can have a conversation of any worth, a bridge of understanding must be built between them; this is an attempt at that bridge. This is mostly theory and craftwork. This is also an origin, not a destination.

Simplification

Within the context of engineering, the gods are fairly clear on this subject:

“Everything must be made as simple as possible, but not one bit simpler.” ~Einstein

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ~some French dude

In either case, I’m not qualified to disagree.

Technical Evolution

Were technology a biological entity, the rate at which it evolves would be, or in the case of AI, is considered alarming. The point of this is that, staying in one place too long is equivalent to falling behind. Processes/tools used only 5 years ago have been shed by the community in favor of other, more improved ones. It’s important to factor for this when:

1. introducing new technologies, and
2. reviewing existing policies surrounding tools/policies.

The Universe, our parents and especially technology all encourage us to expect and factor for change. Therefore, change must be a central theme for the design of any system. Our friends at HashiCorp put it quite nicely:

Workflows, not technologies

“…focus on the end goal and workflow, rather than the underlying technologies. Software and hardware will evolve and improve, and it is our goal to make adoption of new tooling simple…”

As Applied to Infrastructure

Infrastructure work has changed radically and it’s not always clear how. This diagram is ugly but outlines the contrast in the 2 types of infrastructure:

• Traditional
• Modern

Traditional vs. Modern

Starting at the beginning, Traditional work is typically begun by an individual learning to manage a single system; some day that will become multiple systems. For example, I was once responsible for <10K hosts, armed with nothing more than Vim, Bash and OpenSSH; these were effectively the “caveman days“.

Next, DevOps presented a way to meet all of the same requirements but, because of better tooling, more systems could be automated and therefore yield a higher quality solution; this marks a huge capability improvement.

Beyond that, Google published The SRE Handbook. This book distills many years of DevOps thinking and refactors it for newer, more late-breaking capabilities. This is not a value comparison between DevOps vs SRE, rather a linear evolution; one thing simply came before the other.

NOTE: Evolution is not a commentary on previous decisions. Rather, it’s an indicator that a properly functioning engineering team is working as intended. Good things come from this process:

• Evolution should increase capability.
• When capabilities increase, downstream processes have a chance to be optimized. The reverse is also true; if not, it becomes nearly impossible to optimize downstream processes.
• Teams that arrive later are able to “take ownership” of a discipline. This has the tendency to make newer team members more capable of supporting constructs they’ve helped to refactor/rebuild.

Identifiers

The hardest part of design work might be restraining ourselves from applying traditional thinking to modern infrastructure. There are many identifiers that serve to help us understand Modern vs Traditional:

Traditional infrastructure is generally more expensive and yields lower-quality systems due to the nature of how difficult it is to properly automate. Modern systems generally take less work to get better (higher-quality) results.

• Team Size: smaller teams performing higher-quality work typify modern infrastructure. The traditional method requires far more people.
• Number of Environments: modern environments are more structured and higher quality, unless there is some truly unique (generally data-related) condition there is no need for more than 2 environments. When looking at traditional methods, more environments are necessary as the quality of each is lower.
• Alerts: lower numbers typify modern, given higher quality systems. Higher alert volumes typify more traditional practices.
• Let’s do it the hard way first: this always means one thing: I know there are higher quality solutions but my biases are more important.

Of course, there are many more.

Environments

When discussing the number of environments, we must first identify why they exist at all:

“Variant environments exist to eliminate the assumptions of any environment.”

When environments are of a lower quality (reliability decreases) then we would need more of them to isolate and eliminate each environmental dependence the application might have. Of course, the reverse is also true.

When working within higher quality (increased reliability) less environments are necessary to isolate and eliminate each environmental dependence the application would have.

Quality

This has become a central theme; let’s assign a beginning definition. A brilliant engineer once told me,

“Quality is the perception of having expectation met or exceeded.”

This means quality is in the eye of the beholder and therefore subjective. Were that not true we would all be wearing the same clothing, driving the same car and using the same cell phone. From an engineering standpoint, we could assert that “no Mercedes-Benz has ever lasted longer than a Toyota”; yet, the Mercedes is minimally 3-times the cost. It’s subjective.

Beginning from the standpoint that both are a just machines, each of which has an MTF (mean time to failure), and an associated cost. One might be able to mathematically determine that the Toyota would cost less but last longer, yielding a higher quality. This lessens the subjective nature but would likely have little impact on the Mercedes owner.

Take-aways? We should all be self-aware of biases and tend towards objective truths, wherever possible. This is why peer review and following the data is critical to any team making proposals.

Deming would have us believe that no system should every be upgraded unless it increases quality, either of system components or over all of design; I would tend to agree. Higher quality solutions are generally preceded with solid requirements.

Requirements Gathering

This is a common gotcha. In the simplest terms:

• Goals are a generic (unordered) list of outcomes; boxes to be checked off that satisfy business objectives.
• Requirements are specific (ordered) list of agreed upon outcomes that serve business goals.
• Requirements are independent of tools and/or solutions.

Consider the process; in it’s simplest expression there are a few components:

1. Inputs
2. Processing
3. Outputs
4. Errors (deviations from Outputs)

This describes a computer system but any microservice or math function would follow the same path.

Think about what you would want to know before writing any program.

Get Todd Thomas’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

What are my inputs? What would we expect as outputs? Now I think I understand how to process data. But, I need to tell my human about any processing or errors (process deviation). It doesn’t have to be difficult.

We need to figure out where we are and begin at the right step:

1. not sure what requirements are; need to asses.
2. find tools to meet requirements (round peg, round hole)
3. make pre-selected tool fit requirements (sometimes: square peg, round hole)

Beware the BlogPost

It’s also important to understand a not-so-clear aspect of the research puzzle:

“For every good idea posted in a blog, there are 99 bad ones that you must get past first.”

This is anecdotal and distilled through the lens of (completely gut-reaction) empirical observation. The common thought is that, “great minds think alike“; I would posit that it’s more likely that:

great minds think independently.

Since engineering is ultimately a math problem, great minds do tend in a similar direction.

The young and old engineers both follow similar patters to arrive at a tool/proces, but they take separate paths.

Young engineers read a few blogs and pick a direction; get’s stuff done but there’s always the risk of that direction being wrong; i.e.: cargo culting.

Experienced engineers (great minds) look at data for patterns, find a tool or process that supports current patterns and is flexible enoug to migrate to future patterns, then generally end up agreeing on similar tooling; i.e.: engineering.

In one case the herd is following the herd. In the other, the herd will eventually end up following the engineers.

The cure: sound technical/engineering assertions, experimentation, demos and peer-reviews. Also, measure processing times before and after; these measurements will support the case.

If the concept still hasn’t resounded true then it could be lacking current relevance or, in some cases, preference has wheedled its way in. In the case of the former, a stronger argument is necessary, perhaps the proposed solution does not satisfy enough requirements? In the case of the latter, preference should always be identified and un-wheedled.

Design

Today, we all have access to the same open source tools. The only issue is which ones do we use?

Theoretically, two companies could be using the same tools but one is more efficient than the other. This is because every company has different:

1. Requirements
2. Constraints, and
3. Project drivers.

Given the same tools at both places, a similar tool set might satisfy 90% of requirements for company-x while it will only satisfy 70% of the requirements of company-y. I would argue that there are exceptions to this rule; for example: technologies like Kubernetes and Kafka would benefit nearly all projects.

The point is, this is 100% a design issue, starting with requirements but also factoring for:

Projects Drivers:

• Money
• Energy (human effort)
• Time

Example: sometimes time and money are unimportant but we’re short on energy. In this case it’s time to spend money on a managed service to make up for the lack of manpower the most elegant solution would require.

More importantly:

• one combination of these tools will represent expedience (Time-savings)
• one will represent a higher quality net result (lots of Energy)
• one might represent a large cost-reduction (Money)

Each project has its own driver; identify it early.

Most of the rules for tool assembly and workflow were discovered in the 1950s by an American mathematician who laid the conceptual foundation for Industrial Engineering as we know it today. This has a huge impact on DevOps work; example: DevOps teams heavily rely on the “feedback loop“. This is an old, industrial engineering idea conceptualized in the 1950s but so useful that we wouldn’t think of trying to live with out it.

For a fairly painless intro to these ideas, “The Goal“ is highly recommended. It’s somewhat dry though; snag the audiobook, otherwise it will likely put you to sleep.

Design: Good vs. Bad

How do you tell good design vs bad design? I’m not sure there’s a clear answer to this but I’ll share what a brilliant person gave to me:

Good design increases the number and quality of your options, bad design slowly takes your options away.

This person also added:

At some point, so many bad decisions have been made that there are only bad decisions left to make.

But by this time it’s usually too late and a plumber must be called.

Capability Assessments

Any new tool should save on one or more of the above categories. Should one tool be found to lessen all three, it should be included for review. Newer tools rarely decrease capabilities.

For example, The Military performs capability assessments on hardware (guns/tanks/destroyers) every 14 years; computer systems, every 2 years.

This should be done periodically to save an organization Money, Energy and Time, in pursuit of quality.

We’re all on a journey to the same place. And, everyone/team is at different points along the road. Buddhists would share:

To know the road ahead, ask those on their way back.

I hope this serves as an adequate starting point that we can all use as a bridge to understand one another.