By Tudor Girba and Simon Wardley

Software is a domain of mostly invisible knowledge work. It’s hard to see progress, quality or risks in the way that you can with building bridges or running a factory. In that vacuum, simple narratives often fill the gap. As Alan Kay said “computing has turned into a pop culture” and many of its most popular stories have become myths … “the 10x engineer”, “the genius founder”, “move fast and break things!”. In this chapter, we wish to explore specific myths focused on software engineering itself.

Myth 1: Software engineering is about building functionality

When we talk about building functionality, we mean writing code with the goal of transforming an input into an output that conforms to some specification. In chapter 4, we described how seven teams were tasked with producing a calendar application. Whilst their output was functionally equivalent, the internals were quite different. This is illustrated in figure 50, which visualizes the structure of each system in terms of code entities and their relationships.

The experiment shows that when we constrain functionality (through specification and tests) without constraining structure then this can lead to functionally equivalent but structurally diverse solutions. In other words, the structure can evolve independently of the functionality.

But why does structure matter? Isn’t it enough for a system to meet the required functionality? The short answer is that it is until it isn’t. The longer answer requires us to understand a little about evolution.

Our systems are built within a context, an economic and technological landscape. That landscape might appear to be static at a single point in time, but in practice it is constantly evolving. Take how our supermarkets have used bar codes to automate away the manual process of cashiers reading price labels that were stuck on products with label guns. The other technological systems (such as inventory or accounting) that the supermarkets used had to accommodate this new concept. The ability of a system to adapt to a change depends upon the structure of the existing system and not its existing functionality.

To demonstrate this point, let us return to our seven calendar applications. Let us suppose some unpredictable event (as bizarre as barcodes would have seemed in the 1950s) has happened to the environment and the weekly calendar now includes ten days (as happened in the French Revolution, 1793 to 1805). This would require a change to those systems. The difficulty in changing them would not depend upon the existing functionality (which they all fulfill) nor the new functionality (which they all would have to comply with). It will depend upon the existing structure that has been built into each. In some cases, the change might be trivial to implement. In other cases it would involve large structural changes.

Rather than changing the number of days in the week, let us pick something more commonplace such as splitting our monolith calendar application into two parts so that you can scale them differently. Such an activity depends almost exclusively on the structure of the system. Consider the case in which we are in team A which is working on the system depicted at the bottom-right of our visualization (figure 51). Cutting it in the middle crosses only a few dependencies. The situation for team B (top-left) looks radically different: cutting in the middle crosses many more dependencies of the system.

In summary, software engineering is more than just functionality, it also includes structure. Some structures facilitate the creation of new functionality better than others.

This is why concepts like refactoring (or restructuring, in general) are important. Avoidance of refactoring inevitably leads to systems that are out of date with their landscape and difficult to change. These we tend to call “legacy” which is just another way of saying that we have a structure that is stuck in the past which makes it difficult for us to change.

To solve the legacy problem we have to invest in refactoring. Unfortunately, such activity tends to be costly because we not only have to rewrite the structure, we first have to understand it. To make matters worse, this cost rises over time especially if new teams unfamiliar with the system are brought in and start their journey by reading code. To compound this further, there is often pressure to spend resources on adding new functionality rather than dealing with the structure. As a consequence of this, dealing with the structure is often left until the legacy problem is upon us. At which point, the high cost of changing a software system’s structure can be observed in the large number of failed modernizations.

To solve this problem, we need to somehow make refactoring inexpensive. In the digital world, the structure, the functionality, the tools and even the environment itself are constructed from code. We can therefore apply the concept of automation to refactoring.

This does occur but in most environments automatic refactorings only work on a small scale e.g., changing the name of a variable or moving a method. In 2010, Brant & Roberts et al showed that automatic transformation could work at a large scale with a case study of migrating a system of 1.5M lines of Delphi code into C# . Their work shows that large transformations can be achieved systematically through many micro automatic transformations that can be developed almost independently. We expand on these ideas later, but for now it is enough to note that software engineering is more than building and includes refactoring, and it is more than functionality and includes structure.

Myth 2: Refactoring is not a business problem

Most systems that support human processes and decision making have replaced what once happened manually. That human work used to be called knowledge work. It is, therefore, suitable to describe systems by the knowledge they encode and how they encode it.

The system’s functionality represents what the system accomplishes as a result of this encoded knowledge. The system’s structure represents that encoded knowledge in terms of the mechanics of how things are transformed. Some structures facilitate change better than others.

So what?

In the past, the core knowledge of an organization was contained in the heads of people and in the processes they followed. The knowledge management movement made the importance of processes more prominent. The idea was that new value can be created by exploiting the existing knowledge either by streamlining work or by combining it in new ways. Today, much of this core knowledge is inside of the structure of software systems. It is not in documents, it is not in people’s minds, it is inside the systems. That knowledge is still critical to organizations, and both streamlining and recombining are sources of value creation.

The importance of understanding the inside of systems can be observed in what engineers do. Most spend their energy not on writing code but in figuring out the knowledge encoded in the system to learn why, what and how to change things.

Let us demonstrate this with an example. At the height of the 2020 pandemic, a large national retailer (with annual revenues in excess of $10bn) decided it was strategically important to strengthen their online offering. At the core of their ecosystem, they had an old database containing customer information to which different applications were connected for read and write purposes. They wanted to have a more granular customer model and to migrate the database to a system that could be accessed via an API.

The project was defined and a significant team with resources were added. Their initial approach to deal with this had been to ask application teams about their needs. They then asked a couple of those teams to start isolating the calls to the database through a thin microservices layer (this is known as the strangler pattern). That seemed to work well at first, but six months into the project, they realized they still did not know which surrounding systems were using the database, let alone what changes should be performed.

In other words, half a year into a key strategic initiative they still did not have the first order information about the problem space. The knowledge was somewhere in their technology ecosystem, but they lacked the ability to decode it. They turned to outside expertise and brought in a team to help with that decoding. This is where things went from bad to worse.

To reason about the encoded knowledge within a system, you need access to the code. When asked for the code sources of the potential systems it turned out that they did not even know where the code for most of the systems was. The only known piece was the database itself and the few new microservices that were intended to wrap the database. You might find this surprising given the size of the company, alas, it’s all too common.

They could depict the situation in a diagram as seen on the left hand-side of figure 52: a central database (red) with systems (black) that were accessing it either directly or through services (blue). That was good enough to start from. The sysadmins then instrumented both the database and the microservices for a day (which was their typical business cycle) to gather fine grained logs. The logs contained all the queries over the database and the services, including the IP address of the machine performing the call. The logs were analyzed through multiple contextual tools that sliced and diced the information and later linked it with other sources of data. The automatic overview of the system is shown on the right.

This view (one of many created) shows in red the tables from the database, in blue the services and in black the systems. From here they could identify that not all systems have the same needs. For example, while some systems used tables in isolation, others relied on common tables. Thus the picture informed not only which systems have to change, but also provided input into how the work should be organized: which teams can continue working independently, and which should coordinate.

It’s worth taking a moment to look at the figure. The company had spent six months trying to understand what it had and despite all this effort, the left hand side was the best representation that they could provide. On the right hand side is what the actual system was and it took two people and one month to produce.

The challenge with this refactoring wasn’t in writing code, it was in understanding the encoded knowledge. That knowledge was in the system, not in people’s heads or in documentation. Tools were needed to unearth it but those tools had to be contextual to the system being examined.

Without this decoding, the organization could not execute at the business level. Hence whilst it might be commonplace to think of software refactoring as a technical activity about rewriting code, it often turns out to be a business activity about reshaping encoded knowledge.

Myth 3: Software engineering is an engineering practice

Software engineering should be an engineering discipline. But, most of it is still a craft. We should learn from the parts that are more evolved. Let us examine two major parts: development and testing. Let us start with testing.

There used to be a time when the decision to release a system undertook an elaborate process involving quality gates, manual attempts at regression testing and quality managers. People would define what quality means, other people would create checklists and then manually go through these. At the end, a report would be compiled with the results and someone would decide whether to release or not. The decision of whether the system fulfills the desired goal was long, laborious, and often wrong.

Granted, not all testing was manual. People aimed to automate the process with systems like SIMMON as early as the 1960s. In the same way how the price labeling in supermarkets has changed through automation, such a system removed the manual effort from the tedious part of testing. It was only much later that Kent Beck and Ward Cunningham realized we don’t have to wait until the end of the project before testing it, we could automate testing throughout the construction of the project. This eventually culminated in the ideas of test driven development (TDD) where we built the test for a system prior to writing the code that passed the test. This was a remarkable evolution.

Let’s explore it in more detail by looking at the topic through our map of decision making (our first wolf), and follow the path for testing. We begin with a hypothesis that the system lacks a feature and we build a test to check for this. That test is part of a systematic mechanism of exploring the space using a model created by the tests. If you’ve ever heard someone say “the best specification is the test suite” then this is what they mean, the tests model the system. The output (information) of those tests are traffic lights synthetized from specific coding. The overall experience is provided by the test suite which is composed of thousands of micro tests. When approached in this way, software testing is very firmly an engineering practice (see figure 53).

Let us summarize the changes in testing. In the early days of the industry, testing used to favor a gut feel driven approach. Traditional development handed code to testers who lacked system understanding and had to guess how things worked. Test-driven development (TDD) transformed testing from guesswork into an engineering discipline by embedding understanding directly into test suites which were made out of thousands of small, contextual tests that captured the system model and domain knowledge. TDD is more than just a testing technique, it’s a way of designing. But there is still more to this picture that’s worth understanding.

TDD sees the system in terms of functionality, inputs and outputs. As we’ve seen in myth 1, this point of view misses structure. To show this, let us examine the red-green-refactor cycle of TDD in more detail. The first step (red) is to write the test to show what needs to be done. This works great when we start with a known wish about the system that we can transform into a setup and an assertion. However, there are situations in which this is not possible because we do not know what a good assertion can be. For example, when we have to make sense of some data that we’ve never seen before, we cannot formulate a meaningful hypothesis upfront. We have to explore the data first. Similarly, green (implementation) is fine but how do we know what to implement? And what about the refactor step? How do we know what and how to refactor, and more importantly, how do we know when to stop? It’s not enough to guide the system through tests, you need to see the system as well. All of these questions lead us into development.

The limits of TDD don’t mean that development can’t learn from TDD. What might be less obvious to the reader is that those small tests used in TDD are in fact highly contextual tools i.e. they have inputs and outputs and they are applied to a specific problem (the test scenario). Every question about a system boils down to retrieving information from it. To decrease the time to answer, we need tools. But as software systems are highly contextual (an electronic healthcare record system is not the same as an online gambling site), surely we need our tools to be contextual, too. This leads to a question: What happens if we apply testing’s solution of using contextual tools to all development problems?

To answer that question, we need to look at how development is actually done today. Let us assume our organisation uses TDD (and sadly, that’s not everyone, for many testing is still a craft), once we run our tests and see it fails we switch to development in order to fix the problem. For most that means grabbing our monolithic coding environment (which we use for every problem), we inspect the code (often through reading) and we manually gather information to explain the problem (from architectural diagrams to logs). We like this because we want to be “data centric” but then most of those diagrams are actually statements of belief rather than fact. Our exploration is somewhat ad-hoc until we reach a point that our gut tells us that this is the right solution. We try it, we run the test again. For most, software development is very much in the craft phase (see figure 54).

If we want to change software development into an engineering practice then we have to follow that journey of testing. We have to start at the bottom of the map, with the foundations — the development experience. Tests are small tools built for our context. For software development, our experience should also mimic this with thousands of small tools molded to fit our problem. This allows us to climb the value chain all the way up to being hypothesis driven. This is the only path for development to become an engineering subject and it’s why it is our second wolf.

It’s worth reiterating once again what our two wolves are:

Wolf 1: Challenging the way we view software engineering and demonstrating that it is primarily a decision making process.

Wolf 2: Challenging the way in which we ask and answer questions and demonstrating that we need contextual tools.

Alas, tool vendors are trying to sell us the same tools everywhere with an added sheen of LLMs under the argument that AI is better at software creation than most people are. Well, in cases that might be true if we limit software development to a craft and never turn it into an engineering practice and optimize it. When we’ve seen groups take software development from a craft to an engineering practice (by abandoning those monolithic tools, even the ones with added LLMs) then we’ve seen performance increases of orders of magnitude and not the paltry benefits claimed by AI vendors.

Of course, this isn’t an exclusive choice though. Development becomes even faster when you start to use LLMs to build those contextual tools for you. By understanding both of our wolves this new path becomes possible.

Myth 4: LLMs will replace software engineers

We could talk about the endless prognostications of the great and good about how software engineers will disappear through vibe coding, but Simon has already covered the usual sort of myths in his medium posts from January 2023. Frankly, after two and a half years of this nonsense, both of us are bored rigid with it. However, for completeness, we will just repeat some of them here and move onto the more interesting topic.

1) You’ll need less engineers. Nope. See Jevon’s paradox. You’ll need to retrain to a new world but you’ll end up doing more stuff.

2) It’ll reduce IT budgets. Nope. See Jevon’s paradox again. You’ll end up doing more stuff, more cost efficiently.

3) You have a choice. Nope. See Red Queen Effect. This is only a question of “when” not “if”.

Now we are done with the dull, let us talk about a more interesting path which also appears in Simon’s article, the link between LLM and software engineering.

6) I can make a more efficient application by hand crafting the code. Nope. Well, technically you can but the time taken to hand craft it all will be vast compared to the speed at which competitors will move. I’d also suggest reading into Centaur Chess if you think even the most gifted engineer will outcompete an average engineer with an average AI.

Rather than rushing to get rid of software engineers, try instead getting rid of standard tools (even those with added LLMs) and focus on using LLMs to help compose your development experience out of thousands of micro tools just like testing. In the same way we have a test suite that is designed for a specific problem with small tests, we should have a tool suite for our application built out of micro tools.

By using LLMs in this way, we simultaneously turn development into an engineering subject and gain the benefits of LLMs in two areas where they are strong — creation of hypotheses and creation of specific coding for contextual tools (see figure 55).

Those are fancy words but a practical example of this way forward would be better. So, let us start by using LLM to analyze the Glamorous Toolkit. Trying to understand a system is a common problem faced by every developer when looking at a legacy environment.

We ask the LLM to provide us with a list of components and relationships. We give it the context of the code base, and we provide it with tools to search and retrieve code. We ensure that it knows about the specifics of how dependencies are defined. And we nudge the LLM to traverse the dependencies recursively and produce a systematic output in the form of the name of the component and its direct dependencies.

Find all components and their direct dependencies from Glamorous Toolkit.
For this you should:
- Traverse all dependencies recursively to identify all components starting from the root one.
- Provide the result concisely in the form of an array.
- An item in the resulting array is made out of an association between a key in the form of the name of a component, and the value made of an array with the names of only the component's direct dependencies.
- List all referred projects regardless of whether they are considered internal or external.
- Not list the recursive dependencies.
- Not list dependencies to packages as they are internal to projects.

The result should look as below:
```
{
'ComponentA' -> { 'ComponentB' . 'ComponentC' }.
'ComponentB' -> { 'ComponentE' }
}
```

To make the experiment as representative as possible, we picked a problem for which the LLM would have to explore the system whilst still relying upon prior knowledge for how dependencies are declared. We needed to test this because if the LLM didn’t know about how dependencies are declared then that would have introduced a confounding variable to the experiment.

We validated this setup by not telling it where or how to find the dependencies. The plan produced by the LLM confirmed that it did know those details:

Plan to build the dependency map:
- Find GT root baselines: search for classes named BaselineOf* that include 'GToolkit' or 'GlamorousToolkit'.
- For each root baseline, extract direct dependencies from its #baseline: method (look for spec baseline:/project: entries).
- Resolve each dependency to a local BaselineOf* class when present; keep the name string otherwise.
- Traverse recursively to discover all reachable components, but record only direct dependencies per component.
- Output a sorted array of associations: 'Component' -> { 'DirectDep1' . 'DirectDep2' }.
Proceed?

And then it produced a result with 154 components and 243 dependencies. It did so by querying and traversing the code as described in the plan. Quite impressive.

Of course, we do not need the LLM to answer such questions in the Glamorous Toolkit because we already have deterministic contextual tools that provide accurate answers. So, what’s the difference between the LLM answer and the one coming from our contextual tools? That’s a task for another tool and the result is figure 56 which shows a graph of components and their dependencies in which:

• the components and dependencies that both occur in the system and were correctly detected by the LLM appear in black;
• the components and dependencies that were missed by the LLM are colored in red.

The experiment shows us that the approach of using an LLM to understand a system, whilst convincing, fails. The result misses components, misses relationships and even shows relationships that do not exist. This is inevitable with any statistically based engine, particularly as the scale of the problem increases. What you need to remember is LLMs are not truth engines but coherence engines. They do not give you “what is” but “what is likely and sounds coherent”.

Armed with the results, we went ahead and tweaked the prompt in the hope that we could get the LLM to be more accurate. And in some cases it worked better. The emphasis here is on some.

Given a medium system with 1.6 million lines of code, 194 core components and 396 direct dependencies, the error rate of the LLM is roughly 5% to 30% of the components and dependencies missing along with components hallucinated. The problem is not so much the actual error rate (let us say 10% for example) but that you don’t know what 10% is wrong unless you have contextual tools (which most systems don’t) and if you had contextual tools then you’d simply run the tool. The only way to verify the LLM results in environments that lack contextual tools is through manual inspection which never happens because it is the very problem people are trying to circumvent. Even if it does happen, it is highly likely that significant errors will slip through any manual inspection.

It’s just not the error rate that is shocking, it’s also the cost associated with it. Once the contextual tool is created the cost to discover what the architecture is, what components are used and what relationships exist is barely measurable and constantly repeatable, capturing any and all modifications. The running of an LLM was orders of magnitude slower, more expensive (even at today’s highly subsidized prices) and humongously more error prone.

The LLM will give you an output that looks like an answer, it will do this more quickly than you can read the code but what it provides should not be used in an engineering context — it will lack those essential three characteristics of being representative, accurate or even explainable. Relying on this becomes a matter of faith.

The way to positively overcome the limitation of LLMs and not rely on blind faith, is to use the LLM to help generate the specific coding (the tool) that will provide the information we seek. This is our “way forward”. In this part of the example, we show you how to do this.

We start with our original prompt but change the first paragraph so that rather than giving us the answer directly, it gives us a snippet of code (a tool) to do this:

Produce a snippet that recovers all components and their direct dependencies 
from Glamorous Toolkit. The code should:
- Traverse all dependencies recursively to identify all components starting from the root one.
- Provide the result concisely in the form of an array.
- An item in the resulting array is made out of an association between a key in the form of the name of a component, and the value made of an array with the names of only the component's direct dependencies.
- List all referred projects regardless of whether they are considered internal or external.
- Not list the recursive dependencies.
- Not list dependencies to packages as they are internal to projects.
The result should look as below:
```
{
'ComponentA' -> { 'ComponentB' . 'ComponentC' }.
'ComponentB' -> { 'ComponentE' }
}
```

With this simple change, instead of getting data, we now get code that’s meant to retrieve it. In our case, the LLM produced the following snippet:

"Collect all GToolkit components and their direct dependencies"
| repo builder rootProject allProjects |
repo := 'github://feenkcom/gtoolkit:main/src'.
builder := GtRlDependenciesModelBuilder new.
rootProject := builder
buildProjectFromBaselineClass: BaselineOfGToolkit
withRepositoryDescription: repo.
allProjects := rootProject withAllChildProjects.
allProjects collect: [ :project |
project name -> ((project projectReferences collect: [ :ref | ref 
referencedProject name ]) asArray) ]

Executing the code produced the correct answer. It worked on the first and subsequent tries. You still need to review the code of the snippet itself, however, reviewing ten lines of coding in a tool is less daunting than manually finding and inspecting 194 component descriptions hidden inside a 1.6 million lines of code system. This works particularly well because we are facing a shallow problem.

The obvious counter from AI purists is to point out that we can use one agent to create a tool and another agent to understand. That’s fine but then you’re eliminating any human path to understanding and you are delegating the decision to the agent. This is a question of architecture.

Myth 5: Architects make decisions

In the previous myth, we showed a practical example of using micro tools with LLMs to expose the architecture of Glamorous Toolkit. We want you to take a good look at figure 52. Does that look like any normal architectural diagram that you’ve seen before?

In the world of moldable development, architectural diagrams that are generated out of the system itself are the norm. But they don’t look like diagrams you commonly see on whiteboards or powerpoint slides that aim to describe the existing systems. This is because those traditional diagrams are statements of belief i.e. what we would like the architecture of the current system to be and not what the architecture is.

Whatever you might prefer to understand by architecture, the only architecture that matters in software is the one that gets in the code. Thus the only architects that matter in the end are those that affect the code. Everyone else are wishful thinkers and potential influencers. It’s not architects that make decisions but the developer writing the code.

When the architecture is an emerging property, evolving it involves three activities:

1. Know where you are
2. Choose where to go
3. Ensure you go there

Of these, only the second involves design. The first and the third are assessment activities and they require visibility into the system (see figure 57).

The lack of vision into the system can lead to a difficult situation where a coder leaves the organization and a critical system is neither understood nor any documentation is accurate. Without the ability to generate the architectural diagrams from the system then the response by later developers is normally one of fear, not wanting to change or break anything in a system which is poorly understood but critical.

It is little wonder that there is a new lucrative market of vendors promising tools to understand your legacy through the use of LLMs. Their narrative exploits this fear but hopefully we have shown you in the practical example why their promises will rarely be met and that there exists a better alternative.

But can’t we just use agents? I write an architectural diagram, one agent codes it, another checks that it has been coded correctly?

Agents, being LLMs, have the same problems of hallucination and interpretation of your prompts. However, this is little different to human agents and at least in agentic swarms we are adding a process similar to building control that is found in physical architecture i.e. checking the code matches the diagram. In a sense by layering on multiple layers of agents you can clearly make an improvement in ensuring the code matches the architectural diagram. But, this is all wasted effort if you can simply get the system to generate its own architectural visualization to detect the delta between what is wished and what exists.

But I could use a specification and get swarms of agents to build that?

In the past, we fed programs into computers using things like punched cards, and on small micros you could even work directly in machine code or assembly. Take the once popular ZX80 which had Z80 chipset machine code and even higher level assemblers. These are languages close to the machine with hexadecimal commands such as 3E 2A (machine code) which means LD A, 42 (assembly) which means load register A with the number 42 (English enough for a technical person). Of course, if you were lucky enough to own a ZX80 then you used BASIC. You would type PRINT “HELLO” rather than “21 07 42 CD 0C 07 C9 2D 2A 31 31 34 01” (assuming a 4K RAM, rather than the 8K RAM with the ZX81, don’t ask).

We won’t continue with the metaphor, as it is giving both of us micro flashbacks to a time of fiddling with cassette tapes, listening for errors, wobbly RAM packs and typing in long strings of characters. It’s enough to say that the PRINT “HELLO” was more understandable.

The move to higher level languages, even a basic one like BASIC, made it possible to express ideas closer to how humans think and can better understand. These languages are executed on a machine, so the code eventually has to be translated to machine code. As that translation happens automatically with semantics guaranteed to be preserved, we no longer have to look at it, if we do not explicitly want to.

LLMs generate code, too. We can prompt “write a ZX80 program that prints hello” and the LLM readily produces PRINT “HELLO”. So, people concluded that we are facing a similar transition to a higher level of abstraction as we did when we moved from assembly to programming languages. Except that sometimes, our prompt produces PRINT AT 0,0;”HELLO” which looks similar but that clears the screen before printing. It’s not a large difference, but it is certainly not the same meaning.

What matters in this trivial example is that the semantics are not preserved. Unlike our translation from BASIC to machine language which always provides the same semantics (it is deterministic), the translation from prompt to machine language is not.

So, what does this mean in our current fad for specification driven development? It’s a bit like the calendar example above, yes you get an answer but the LLM could be acting like any one of those teams. You cannot guarantee that the semantics will be the same from one prompt to another. But isn’t that how real life is like? Yes, you don’t know what the team produces until you read the code in order to discover what decisions have actually been made. The specification alone is not enough. Unlike the translation from assembler to machine code, the translation from prompt to machine code moves the “developer” further away from the decision making in a material way.

Part of the problem, and why specification driven development sounds so seductive, comes from a misunderstanding of how we actually program things. The idea seems straightforward enough, in the physical world an architect creates a blueprint for a building and then the builder builds it. Following the construction metaphor in software engineering, the software architect creates a blueprint for the problem (the specification) and the engineer builds it! Almost, but not quite. Whilst we live in houses, we don’t live in the code that the engineer builds as there is a missing step. The computer translates the code and then runs the translation to provide the application. The code the developer produces is actually the blueprint. What we call software architecture (i.e. specification) is closer to a retail investors wishlist for new housing investment. Going from a retail investors wishlist to blueprints of the houses to be built takes a lot of decisions and those decisions are what code actually is.

This is not to say specification driven development isn’t useful, it certainly is as a learning tool. But we still have to decide the most important architectural question of where we value humans in the decision making process. If you’re happy handing over decisions to the machine then fine but you should do it being aware of the tradeoffs and the loss of comprehension.

But does losing control of system comprehension actually matter that much? It’s fine if you’re talking about prototypes, but if you’re going into production without that understanding, then you will increasingly be running the risk of becoming the next Knight Capital Group, a company that lost $440 million in 45 minutes because of behaviour within its systems that it did not understand. It’s only a matter of time before history repeats itself.

That’s not to say that human understanding is perfect; it’s not, as the Knight Capital example shows. It’s actually an argument for why good engineering practices matter. By choosing a path of devaluing human judgement in the decision making process, then you are accepting the structural choices and values the LLM uses as long as functionality seems correct. In doing so, you are ignoring that functionality is only one subset of what goes into building a system. Comprehension is your last line of defence in a world where we have not yet developed the engineering practices to manage this safely.

So let us now stretch the specification driven development to its next logical step. If we accept it as some higher level language (it is not) then why not get another agent to write the specification for you based upon some desired wish. Before some marketing person beats us to the punch, we will call this Wish Driven Development and quickly get an LLM to write us a book on this. We could even imagine prompting for “Build a successful product. Create a company around it. Reach an exit and deposit the amount in my crypto wallet”. Fame and fortune awaits!

Yes, you can even tell yourself that you’ll definitely read the specification this time before handing it to another swarm of agents to build it. You can snuggle under the comfort blanket that you have another swarm to check if the output matches the specification, another to build tests for it, to check if it is secure and so on. You can remove yourself far from the decision making processes and accept a world of convenience in return for no understanding. However, this is basically the storyline for E.M. Forster’s “The Machine Stops”, so he beat us to the punch by over one hundred years. Damn.

But hold on … why stop there, why not get the agents to create the desired wish? Do we really need humans to tell us what they want? The answers to all these questions is determined by where you think that humans should be involved in the decision making process and the nature of that involvement.

We’re not writing this to discourage people from using LLMs, quite the opposite. But we want you to think deeply about where you place human judgement in the decision making processes that are digital systems. Comprehension is not a nice to have. It is the scaffolding that makes safety, accountability, and learning possible. Without it, you are not engineering; you are relying on belief. By removing the conditions for comprehension under which mistakes can be understood, corrected, and learned from, you make the collapse that E.M.Forster discussed inevitable given enough time. In practice, we both suspect that it’ll take many Knight Capitals before we learn this lesson again, such is the seduction of LLMs.

To inform where human decision making matters, it is useful to know what humans can do through better engineering practices. Armed with that knowledge, we can then map the entire system and determine, for example, where do we wish to allow vibe coding for the creation of prototypes, where do we wish to use a combination of AI + Software Engineering (assuming we can make development an engineering subject) and where should we outsource to more utility providers. Remember, we are still learning the practices of how to do this safely but we should at least start by asking questions.

An example of this can be seen in figure 56. In this figure, we have taken a much earlier map used by James Findlay in the building of HS2 (UK high speed rail) in a virtual world. The original map was created in the early 2010s and was used to identify which system components should be built in a more agile way, which components should be bought off the shelf and which should be outsourced.

Today, on the same system map, we can ask a similar question on where human decision making matters. The answer is not a “one size fits all” but instead a combination. We might decide to vibe code those novel and new prototypes but simultaneously decide to use software engineering assisted by AI in building the core of the system. Where those boundaries exist will vary depending upon the capability of the technology, our willingness for risk and our requirements such as accuracy and reliability. The map is simply a communication tool for expressing this.

Architecture is supposed to be the expression of our values and it is unfortunate that most architects never ask, let alone answer, the most important architectural questions of the day. There are a number of key roles that architects can and should be playing:

1. Deciding where we value humans in the decision making process.
2. Determining the questions that need to be asked.
3. Understanding the existing system.
4. Being the arbiter of conflict between different decision makers.

Myth 6: Dealing with legacy is hard

To explain this myth, let us start with a real company. Lifeware offers software as a service systems that manage the core data and automate the core workflows for insurance companies. The first step when Lifeware works with a new insurance company is to migrate their legacy system to the Lifeware infrastructure. To do this, they use a streamlined process that has provided a 100% success rate for the migration of over 20 insurance companies over the last two decades. Furthermore, once migrated, Lifeware evolves these systems faster and with a fraction of effort compared to the teams that worked on the systems they displaced.

Contrast these results with the typical industry reality. For example, according to the Advanced 2020 Mainframe Modernization Business Barometer, 74% of organizations have failed to complete their migration projects. That gives an industry typical of 26% success rate. Lifeware is an extraordinary outlier with a 100% success rate over 20 examples. On probability alone, the likelihood of Lifeware’s success being a random effect is about 1 in 500 billion. So, either Lifeware is extraordinarily lucky or they are doing something right.

They achieve this exceptional success because they have invested in software engineering as a business capability for three decades. They spot and adopt new fundamental engineering practices early. So much so that they were a flagship case study in the book which first introduced the ideas of test driven development. Lifeware had the largest test suite in the world at the time (4K tests in 2002). Currently, their test suite holds 160K tests for a system built with 35M lines of code. While the amount of tests grew non linearly over the past decades, the amount of time to run them is still measured in minutes. Today, every developer can transparently and directly run them on AWS clusters containing thousands of processors.

They achieved this feat over time by investing considerably in their development environment. They have created a variety of specific tools, including for technical reporting of migrations status, for overviewing their integration pipeline or for contextual debugging. More recently, they took a further step and adopted Moldable Development which includes the practice of building small contextual tools. They have already accumulated more than 3k contextual micro tools to guide their workflows, with over 1k being constructed only over the past four months. These tools complement their functional tests and accelerate their work. As a result, Lifeware can already attack larger migrations than before which directly and positively affects their business.

What is more difficult to explain is the level of integration that they achieved throughout the company: whether they investigate a system change together with the business, whether they script a custom code transformation, or whether they investigate the performance of an AWS cluster run, they never leave their environment. Rather than splitting their work across tool boundaries, they make tools come to their environment. The result is a truly integrated development experience consisting of thousands of contextual tools.

So, it seems that dealing with legacy is only hard when you don’t invest in engineering practices. There is no obvious reason why other companies can’t be sharing the same rates of success.

This focus on engineering is somewhat counter to today’s AI mantra, which gives us an interesting experiment. If investment in engineering practices and system comprehension are the primary drivers of success, then you would expect that investment and use of AI in development without a focus on engineering practices would lead to anomalies such as people believing that performance has improved whilst instability in delivery has increased.

This is what the DORA 2025 State of AI-assisted Software Development Report seems to have stumbled upon (pages 38, 41): high degrees of adoption and perception that performance had improved combined with increasing instability in delivery. Of course, perception is hardly a good yardstick for measurement.

Fortunately, we have a randomized controlled trial from METR that looked at how AI tools affected the productivity of experienced open source developers. What they found was that while developers expected and perceived an increased productivity, the use of such tools actually slowed development by 19%.

So, we seem to be on a sticky wicket with high adoption, perceptions of improvement and associated marketing claims against quantitative slowdown and increasing instability. How this gets resolved in the future is unclear as the practices needed for development with AI are still slowly emerging and will take many more years to mature. All we can say for now is that a focus on engineering practice is the better bet since engineering isn’t going away anytime soon.

Myth 7: Non-technical people can’t understand code

In the early 1970s, computers started to offer interfaces made of screens and keyboards. Whilst this was a considerable improvement on what existed before (printouts, punchcards, pluggable cables), there was no graphical interface, only text based screens and command line instructions. Such limited interaction abilities meant that the audience became those educated on the corpus of these instructions i.e. “technical” people.

At the same time, a different paradigm appeared at Xerox Parc known as the Smalltalk system with a bitmapped display, radical graphical interface including icons & menus, application windows, keyboard and mouse. In this world, even children could understand and manipulate the system.

From technical experts in one world to children in another. Both were using similar underlying “computers”. Why such a huge difference?

It is the interface that determines the cognitive and practical barrier to entry, which in turn determines who can realistically use, manipulate and understand the system. In essence, an interface is a translation of a computation in a way that a human can understand and interact with. Interfaces are fundamental because we cannot perceive anything inside the computer that is not tied to an interface. In short, if you change the interface, you change the audience.

When we say “only technical people can truly understand how the code works”, what we are really admitting is that the interface limits the audience to technical people. It’s a design choice, not a law of nature. Whilst you can enable a wider audience to use and understand a system through interface design choices, a deeper understanding still requires that audience to understand its construction which is often equated with the ability to read the code.

However, in Cozy Corner from the previous chapter, we’ve seen an example of how presenting the inner workings of a system could empower non-technical people to understand and have a say in its evolution. Any part of a system can be presented through a tool in a way that an audience can comprehend without reading code. In fact, even technical people should not resort to reading code as the primary means to understand the system’s inner workings.

Understanding a system through a tool only requires the tool user to appreciate what it does and not necessarily its construction. A surveyor can use a theodolite to reason about a landscape without awareness of the intricacies of theodolite construction, such as the use of bronze bearings to create carefully controlled friction. That said, there are certainly advantages with understanding the construction, and that pathway should never be closed.

However, there is an assumption that a deeper understanding of the output of a tool requires a deeper understanding of the tool itself and this requires a technical person. Fortunately, this deeper understanding can be provided by another tool. Hence, the technical person might be the pathfinder but that doesn’t mean the non-technical person can’t follow. This is best shown with an example.

In a previous chapter, we used a treemap visualization showing the classes from Glamorous Toolkit grouped by packages. The visualization (repeated below) highlights in blue the classes that have at least one associated contextual tool.

This visualization is a tool inside Glamorous Toolkit (GT) whose subject is GT itself. This is a tool within a tool. This idea can be taken even further and can be applied at any granularity level. Whilst the tool was built by a technical user, a non-technical user is perfectly able to use the tool to gain a deeper understanding of GT than most technical users have of their own traditional development environments.

So, what if we want to know an even more technical aspect? For example, how does the layout work? That sounds like something only a specialized technical person could understand. So, let us give that a go with the aid of a tool.

To do that, let us consider a bit of history. The layout’s implementation is based on an algorithm called the Squarified Treemap. The article that introduced it contains the pseudocode and the math to describe what has to be implemented. More interestingly, though, it also includes a visual depiction of the steps the algorithm takes as seen below.

The picture shows multiple steps of laying out a set of 7 nodes with different areas. The sizes are denoted by numbers inside the boxes (6,6,4,3,2,2,1). The challenge the algorithm addresses is to arrange these so that we get a nice proportion between the overall width and height.

In the first step, it arranges a node with the largest area, which is 6. Then another, which is also 6. And then the third one with an area of 4. Only at this step, a condition about the proportions of the width and the height is not met, so the third step is discarded, the algorithm moves back one step and tries a different path. In this path, the node with weight 4 is placed to the right. And so the algorithm continues, hitting constraints, falling back to previous steps and trying again until all nodes are filled in. A non-technical person can easily understand what the algorithm is doing in this visual representation but may struggle to understand the code itself which is over 760 lines long. The visual depiction allows us to form an intuitive mental model of how the placement of elements work.

It was this same picture that guided the implementation of the algorithm into Glamorous Toolkit (GT). And once the implementation existed, it also included the tool to produce the same visualization for any given treemap. The picture below depicts this. On the left we see a concrete treemap of seven nodes like in the paper (just less dense than the one used to visualize the components in GT) and on the right we have the steps that the algorithm followed to produce the treemap provide by the same tool that was used in its implementation.

One minor detail to note is that by making the programmatic visualization available, we quickly discovered that for these specific 7 nodes and areas, the algorithm took 11 steps. However, the manual visualization in the original paper uses only 10 steps. This prompted a review of our interpretation of their algorithm (the right hand side of the image) even though the overall functionality (the left hand side) was equivalent. The person who used the visualization to challenge the interpretation of the algorithm was not the developer who had implemented it. Even non technical people can participate in highly technical conversations with the right interfaces. For those who are curious, what was discovered was the manual visualization contains a flaw.

Currently, GT contains over 5,000 such tools doing a similar job and making seemingly complicated technical details accessible to various audiences. Contrary to popular belief, any aspect of a software system can be explained in a way that’s understandable to an audience that is interested to learn about it.

One final thing to note, it is the artificial barrier created between technical and non-technical people that has led us to create misleading terms in an attempt to find common ground. One of our favourite, and probably the most misleading is technical debt.

Myth 8: Technical debt exists

Technical debt is an overloaded term today. It is used as a placeholder for all sorts of conversations, especially at the boundaries between technical and business areas. We should break free of it.

When Ward Cunningham came up with the term in 1992, it was a beautiful metaphor. At that time, the business people and technologists were often communicating through large documents. With just two words “technical” and “debt”, Ward brought together two worlds that were foreign to each other and gave them a common language to describe a specific problem in a financial system. He wanted to say that the structure of the system no longer fitted the new understanding that they gained by building and running it, and therefore they should stop and restructure it to mirror this. Like paying out a debt.

Due to the communication void caused by the chasm between the technical and non-technical world, this metaphor acted like a spark that ignited new kinds of conversations. And like that, the metaphor of technical debt took on a life of its own with many meanings being conflated into it. Ward later clarified that he never intended it to mean anything else than the difference between the current structure of the system, and the structure that we now wish the system to be. He never intended it to mean a reckless pile up of technology, for example.

A metaphor can be useful but this one has run its course. It’s time to outgrow it.

Ward Cunningham took a complex, multi-dimensional problem and reduced it to a single scalar that business could grasp. That scalar did not need to be accurate; it only needed to improve the business’s understanding of the system. The problem is that systems are influenced along multiple dimensions including code, context, skills, and tools.

The code represents what the system does (function) and how it does it (structure). But that code exists within an environment, and is shaped by the skills and tools used to build, operate, and understand it. What we are really dealing with is a tensor of function, structure, context, skills, and tools, compressed into a single scalar called “technical debt”, a value that people then attempt to measure by examining structure alone.

Whilst function, structure, context, skills and tools are real and tangible, the scalar is imaginary. Its role was simply to help provide the business with “insight” because “non-technical people can’t understand code” (see myth 7). This scalar is however as damaging to understanding as that other horrendously simplified scalar known as GDP is. In that case, Simon Kuznets (the inventor of GDP), warned from the very beginning that it should not be used as a measure of national success or wellbeing which is of course what it has been used for. People like simple scalars, like KPIs, regardless of whether they are useful or downright harmful. They have the one property that makes it appealing for busy people … they are simple.

However, the scalar being a compression of these multiple facets has one uncomfortable property. Its “value” can change by simply changing one or more of the facets i.e. if I change the tools or change the context then I could potentially make a huge debt disappear as if by magic. It’s not however magic, it’s just that the scalar is not real, it’s imaginary, a compression of things that are real.

Consider the following thought experiment: If we could change the tools and skills so that we could understand and change our systems at will and at zero cost, would we still talk about technical debt? We think not. A couple of decades ago, changing the name of a function was a tedious action potentially involving breaking code and requiring manual changes in multiple places; today, the same action can be handled through an automatic refactoring at zero cost. Without such refactorings, having a function with the wrong name could be qualified and even quantified as technical debt. But with refactorings, qualifying the same situation as technical debt would be misleading at best.

A simple change of tools and skills has made the technical debt disappear. It’s the sort of financial engineering that accountants could only dream of, alas in their world debt is something very real.

You might say that this only works in narrow cases. But the bottom line is that we will only find these other cases if we accept that the possibility exists, so that’s where we should start from. The good news is that we do not have to talk in hypotheticals. Throughout this book we show multiple examples of how something that appears expensive becomes many times less by changing the skills and tools.

Legacy IT estates are commonly considered to be part of an organization’s “technical debt”, but in myth 6 you saw a case in which a large legacy system can be moved much faster than the competition. This shows that investing in engineering practices makes the technical debt disappear and can even turn it into a competitive advantage.

Along with being an imaginary compression, it also suffers from being framed as a negative metaphor. The best case scenario is to not have debt. Having this monopolize the conversation drives a behaviour of always trying to reduce whatever is associated with that number, including the idea of software engineering as a cost structure, or of outsourcing it to other companies. That might be the right approach but as we saw with the legacy example, so might investing in engineering practices.

Another side effect is joy. When systems become understandable and changeable, engineering becomes joyful. So, not only do you get to solve your most expensive problem, you get to make it fun in the process, too. Isn’t that worth breaking free of a 30-year old metaphor that is imaginary?

What did we learn?

Before we move on, let us pause to look on the mighty works of our industry’s myths, and despair.

“Software engineering is about building functionality” crumbled first because software engineering is not only about functionality but also structure. This then exposed the myth of “Refactoring is not a business problem” because the organization’s knowledge is encoded in that structure. At this point, the tower of “Software engineering is an engineering practice” leans over and collapses because it was stretching the truth to breaking point. Testing has evolved into an engineering discipline by building contextual tools at scale. Development should follow the same path.

Recently, we have been deafened by a thousand AI purists chanting that “LLMs will replace software engineers”, but LLMs do not remove the need for engineers, they amplify the value of the engineering path that creates better questions, better tools, and better feedback. This matters because system decisions are made in code and not in the specifications, diagrams or other faith based artefacts that made people worship at the altar of “architects make decisions”. This doesn’t mean that architects don’t matter. They do because they should be deciding where we value humans in the decision making process.

By focusing on these engineering practices we can finally solve “the legacy is hard” trap which is even more pernicious than the idea that “non-technical people cannot understand code”. We finish our journey through the lands of Ozymandius with a search for mythical beasts. Yes, we too have heard (and even used the idea) that “technical debt exists”, alas, whilst well meant it’s mostly imaginary and misleading.

In overcoming all these myths, LLMs can be an enabler, for example in the building of micro-tools, but they also pose an ever present danger by removing the conditions for comprehension.

Before we leave, it’s worth noting that there are many more myths we could have explored. This is not an exhaustive list and we both had to make compromises. Simon, for example, would happily spend several chapters on how LLMs are non kinetic forms of warfare and why most of today’s digital sovereignty discussion (with its focus on territorial location) misses the mark by a mile. Tudor would have expanded on how the inability to read systems leads to a world of functional illiteracy that provides fertile ground for myths to flourish. However, for brevity, we made those compromises and chose only eight. We then traced their existence to our missing two wolves.

Homework

As a team, pick a legacy system that is often described as “technical debt”.

1. Determine what business value would be enabled if you could change it at will?
2. Pick three manual diagrams about that system. Do you believe these represent the system?
3. Write down the words: context, skills, tools. What change in each would diminish the apparent technical debt?
4. [For technical users] : Redo the experiment from myth 4. If you are happy sharing the information, use Claude Code (or an LLM/Agent of your choosing) to create a visualization of the module dependencies (such as npm). By changing only the beginning of the prompt you created, ask the LLM to create a tool to generate the visualization. Compare the two visualizations.
5. Write one sentence on where you want human judgement to remain non-negotiable in that system (and why).

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding Software Engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Tudor Girba and Simon Wardley

An apology

This book is about software engineering, but we start this chapter by talking to the business. We know that you’ve been wanting to fire those pedantic, inflexible, costly and annoying software engineers for ages. AI appears to be so damn helpful and so much more of a partner that you think that now is the right time to part ways. But give them a second chance, you will need them, software engineers can change and so can you.

We understand that in business you don’t care about the decisions made in code but you do care about how it works. Software engineers also care about how the systems work, but they care more about the decisions made in code. We can bridge this gap and bring you both together. We’ve done it before. We need to start by sitting down for a chat, somewhere comfortable, somewhere cozy.

The Cozy Corner

Until now we talked more about technical problems, like finding unnecessary data in a pipeline, debugging a stuck cursor, or explaining an algorithm. Those are problems which concern software engineers, and are far from your business language. So, let us talk about the restaurant business for a change.

During 2017, Tudor (as Simon was off elsewhere) had the privilege of observing how restaurants function and aim to produce value. Our observations were qualitative rather than quantitative and stem from personal experience mostly coming from eating at restaurants throughout the world. We have seen polite waiters, rushing bill boys, and bossy Oberkellners. We have seen food being served impeccably, and we have experienced the frustration of waiting for the bill.

From this journey, we came to believe that the core of the restaurant business was not yet digitized by anyone. There exists automation in different parts of the domain, but not at its core and not in general. But, what is that core?

From a consumer point of view, the value of a venue, being small or large, comes from two components: from the products it sells and from the experience it offers. From the business viewpoint, the value also depends on the costs with which the services are provided. To be complete, any optimization through automation must focus on all these dimensions in a uniform way.

A quick scan of the trade magazines shows there isn’t a shortage of systems trying to automate the restaurant business. Unfortunately, whilst they address some restaurant problems (the ones they were designed for), the restaurant business is highly variable. Hence, most of these systems either fail or try to force the restaurants to change their practice. There is no standard system that fits every restaurant. That was the challenge to address.

Our first clue was in the inconsistency of service found in busy restaurants when waiters become overloaded with demands. The problem to solve at the core of a restaurant seemed to be communication. It’s why a waiter sometimes comes over and asks whether you’ve got your order.

So how do you fix this? We need to start thinking about how messages are sent throughout the restaurant. Take Cozy Corner, a small restaurant with two tables served by a waiter and a kitchen with the cook. At its simplest we want the waiter to communicate with the chef, which they could do by talking to them. However, what happens if there is more than one waiter, maybe there’s two or three or six? Do we need a queue of waiters lining up patiently for the chef? And what happens when the chef needs to tell the waiter the food is finished? What if there is more than one chef? Maybe there are chefs that specialise in different dishes? Pretty quickly communication becomes messy.

We often try to solve this problem by introducing elements of queuing with paper notes. The famous carousel or ticket wheel. However, this is a carousel for creating food, but what about delivery and paying? The ticket wheel itself can often become a bottleneck in communication. So, the solution is to turn the ticket wheel into a digital solution and we’re done? Easy!

Think again, we’re back to the problem we started with — yes, you can automate some parts but the restaurant business is highly variable and contextual; so, the solution in one place doesn’t work in another. Since we’re talking about communication, the alternative is to build a language that enables an owner to describe their restaurant. That sounds a bit crazy but let us show you how in figure 34.

The guests have a wish — from I’d like to order drinks, can I buy a macaroni cheese or please can I settle my bill. The waiter transforms that wish into a message which is passed to the table. The table passes the message to the room. The room (there could be many) annotates this and passes it onto the business. The business interprets the message and potentially creates another one that is addressed to the kitchen. The chef gets an item on the to-do list.

The chef now cooks our sumptuous meal. When ready, they pass a message to the kitchen, the kitchen tells the business, the business tells the right room, the room tells the table and the table tells the waiter to deliver the food.

Through this mechanism the waiter and the cook can communicate without knowing of each other’s existence, and it can scale to arbitrary sizes and restaurant-specific processes i.e. we can describe any restaurant with this language. That’s the good news, the bad news is it’s just a theory. We would actually need to build a language on how to pass these messages. This is exactly what we did in 2017.

The idea is far from trivial. We sketched pictures like the one above to reason about our idea for an algorithm of propagating messages. These pictures offer a common ground for both technical and non-technical people to discuss the space. One thing which we should note here, because we return to it later, is the difference between where the work is done and where we learn, express and discuss ideas.

This sounds a bit counterintuitive but it’s one of those hidden in plain sight engineering practices. Walk into any engineering department and you’ll find the same problems being discussed in two different mediums. In software engineering those are the screen (where coding is done) and the whiteboard (where ideas are expressed). It’s the same with financial engineering groups — plenty of spreadsheets (where the work is done) and whiteboards (where ideas are expressed).

Domain Driven Design or DDD proposes the use of a ubiquitous language to bridge these two mediums. Here, the “domain” is the real-world problem that the software is trying to solve. If you’re building software for a bakery, the domain is baking and running a bakery. If you’re working on an app for doctors, the domain is medicine and healthcare. In our case, it’s the restaurant.

Instead of starting with tech stuff, we talk to the people who actually do the work — the bakers, the doctors, the waiters — using diagrams to understand how they think, how they talk, and what they actually do. We don’t try to solve the problem but instead we model the space like a baker would with orders, ingredients, ovens and timers and then use this to design our language. This is how DDD is practiced today — taking the hand drawn diagrams and transforming them into readable code. Perhaps like this one:

//Construct the restaurant and the staff
cozyCorner := Venue new name: 'Cozy Corner'.
kitchen := Area new name: 'Kitchen'; parentArea: cozyCorner.
room := Area new name: 'Room'; parentArea: cozyCorner.
table1 := Area new name: 'Table 1'; parentArea: room.
table2 := Area new name: 'Table 2'; parentArea: room.
johnWaiter := Contributor new 
                  name: 'John Waiter'; 
                  addArea: table1; addArea: table2.
jeffCook := Contributor new name: 'Jeff Cook'; addArea: kitchen.

//Setup logic for the parts (not shown here)
…

//Order some food
sirloinSteak := Consumable new 
                   name: 'Sirloin steak';
                   price: ((30 EUR asPrice discountedBy: 10 percent) 
                           discountedBy: 5 EUR).
sirloinSteakMenuItem := MenuItem new consumable: sirloinSteak.
order := Order new 
            addOrderItem: (OrderItem new menuItem: sirloinSteakMenuItem).
orderMessage := OrderMessage new order: order.
johnWaiter send: orderMessage to: table1.

Please take a moment to read the code even if you are not too familiar with programming. It’s in enough English that you should be able to get a gist of what is happening. Indeed, if you read the code above and squint hard enough, you can imagine the picture from the whiteboard.

In our case, this description is written in an internal domain specific language which is just a fancy way of saying an object specific model designed for that domain. It’s worth understanding that traditionally, variation in systems is often supported with configurations, each variant being treated as though it was an independent problem. This sort of solution becomes harder over time because the configuration combinations explode. The alternative is to find a language in which a set of variations can be expressed with simpler mechanics. In our case, this internal domain specific language not only describes the object model and variations but goes deeper into describing the logical setup of operators.

We have to go a little deeper here but bear with us. Let us look again at the code above. The code starts by creating and linking a few entities like the cozyCorner or jeffCook. These are of the same kind but with slight variations. We called them actors. Actors know how to send and react to messages. How they do those operations is defined in the logic part that we did not show above. So, let’s remedy that. The cozyCorner defines how it handles the OrderMessage something like this:

//Setup logic for the parts
cozyCorner
  when: [ :message | message is: OrderMessage ]
  act: [ :message :result |
    message handlingActor addOrder: message order.
    message orderItems do: [ :item | 
      result send: (ProduceMessage new orderItem: item) ] ].
...

Ok, we know that’s too technical. The gist of it is that actors can define logic based on a condition and associated action, and the result of an action can be one or more messages being sent. This then provides the basis for a language with which we can program complicated flows. It’s fine to not understand the code in all its details, but this should not mean that you are excluded from the discussion. What we need is a visualization that everyone can relate to, a true ubiquitous language. Hence, the importance of our whiteboard.

We now have a logical diagram and we have code that implements the logic. However, if these visualizations remain only on the whiteboard, drawing them every time we want to test new scenarios or updating them as we gain new insights into how the model should behave then this quickly becomes a highly tedious task. What’s worse is that a manually created picture represents a belief about an existing system and not the actual system. This is where we depart from DDD. We get our system to create our whiteboards for us.

Figure 35 shows a logical diagram produced automatically from the logic in the system. It captures the propagation of the order (red), the produce (blue) and the deliver (cyan) messages through the venue. How do we do this? We simply build the tool to automatically generate it out of the implemented logic. With a tool like this, the cost of recreating the whiteboard with different variations of venues now becomes close to zero.

Let’s take a moment just to think about this. Our normal way of operating is to update the whiteboard and then produce the code. However, because we now have a tool which automatically creates the whiteboard from the code, we can simply modify the code to see what it looks like and discuss the visualization with the group. This feels like it should be normal practice in any business process. This doesn’t seem to be a big ask. If one cog moves, the rest shifts too. Our visualizations should be part of the machinery — not just decorations taped to the wall or scribbled on a whiteboard.

The importance of visualization

As soon as we presented the generated diagram to the business, they immediately started to express ideas and asked for variations. Only this time, we did not start from an empty whiteboard but from a system generated reflection of itself. For example, an interesting variation is to allow for a guest (called Alice) that comes into the restaurant with a phone. Why shouldn’t Alice simply connect to the table and order directly through the digital interface without involving the waiter?

The business people sketched the idea manually on the generated diagram (see the left hand side of figure 36). The code was then changed and a new visualization automatically created (see the right hand side of figure 36) to ensure it matched. The entire development process was a few minutes and most of the development energy was spent on sketching and discussing ideas.

We can hear you saying “Ah, but you still have to turn that visualization into a running system”. No, the visualization is of the running system — that’s the point. This isn’t an exercise in how to more efficiently create whiteboard diagrams. The diagram is the running system, it’s a self representation. Once the change was in place, we could see that indeed, without changing anything in Cozy Corner, Alice could place the order and the cook was notified, when the cook said it’s ready, the waiter was notified.

As soon as the business group saw the result, that started the next conversation, the next set of ideas to be expressed: Should the waiter or the guest be notified? Well, it depends on the flow of the restaurant. And so we went on to discover an adjacent unexplored area. And so on.

This is a good time to talk briefly about the use of hand drawn diagrams. We used the one from figure 34 to communicate when there was no system. We used the left of figure 36 to depict the delta between what existed and what was wanted. Hand drawn diagrams depict either wishes (when they are about the future) or beliefs (when they are about the present). As soon as the system exists, the diagram should be generated i.e. there should be no place for beliefs, only wishes. Designing systems is hard enough. Not knowing what your current system is and relying on beliefs should not be an acceptable variable.

This seems like magic?

At first glance this can look like a BPMN (Business Process Model and Notation) or a workflow engine but this is just because we are modeling a concept that looks like that. The idea of creating contextual visualizations applies to anything in the domain we want to model.

For example, in our scenario code, the definition of the steak had a price associated with it. Like this:

sirloinSteak := Consumable new 
    name: 'Sirloin steak';
    price: ((30 EUR asPrice discountedBy: 10 percent) discountedBy: 5 EUR).

That price includes a couple of discounts like the discount by 10% or the one by 5 euro. Discovering the discounting mechanism also followed the same process as the one of discovering our order flow and it also came with its own tool and visualization as seen in figure 37. The price object was not specific to the restaurant industry but to any domain with finance within it.

Our restaurant business contained multiple languages — the language of order flow, the language of payment, the language of a specific object. When building our tool for the restaurant business, what we are in fact doing is creating a domain specific language by combining pre-existing languages with what is specific to the domain, Cozy Corner being a specific instantiation in that resulting language.

We are going to re-emphasise this part because it’s at the heart of why we can work so quickly. We don’t build solutions to a problem by writing code, we build the language of the problem. Then we use that language to solve the problem. That language is written in code and it always comes with a visualization (a tool) because building the tool is how we build the language.

The net effect of this is that we build domain specific tools for the problem and then we use those tools to solve the problem. Whilst most problems we face in technology are highly contextual — stuck cursor, cozy corner — there are a few highly recognizable and standard problems — HTTP load testing, code editing, DNS queries (see figure 38) — where ready made tools make sense and are available. Alas, we’ve been hoodwinked by tool vendors into believing that their tools are applicable everywhere which is unsurprising given their self interest. Kitchen blender salespeople will always sell you kitchen blenders for digging deep mine shafts if they could get away with it.

Despite tool vendor claims, for the majority of cases, to solve a problem we need to build a tool for that context. To communicate that problem, the tool needs to show a visualization of it but that requires a model. To build that model we benefit from creating the language of that highly contextual domain and in order to create that language we need to ask and answer questions. The process of asking and answering questions is captured in the development of the tool for a contextual problem. It’s also where we make our choices on what matters.

Hence, when dealing with a legacy migration project, we don’t start looking at or reading legacy code in a standard code editor, we start by building the tools to understand that legacy code and once we can effectively visualize the problem then we can migrate it. This is how you turn year-long migration efforts into a month: by spending the first three weeks building the tools you need. This isn’t magic, it’s just the result of rethinking what we mean by software engineering. Which is pretty convenient since the title of this book is “Rewilding Software Engineering”.

A note on reason

Language, medium and tools are core pillars for how we reason about the world around us. Any changes to these throughout history, such as the printing press (tool), the written word (language) and paper (medium), had profound effects on how we reason.

In the world of software engineering: the language, medium and tools have evolved over time leading to changes in how we reason both about and with the machine. For example, the language change from Fortran-like expression to message passing and protocols enabled us to create orders of magnitude larger systems. Today, languages are changing from exclusively deterministic to include non-deterministic forms such as prompts with LLMs.

The change of the medium through which we interact with the machine from text to GUI (graphical user interface) also changed how people adopted computation in their lives. Today, our medium is changing as well. In a traditional world of engineering we would often discuss the problem on a whiteboard and then translate this discussion to deterministic code (written text). The written text was the major medium of human to machine interaction. In a world of LLMs/LMMs then, the whiteboard (a mix of images and text) becomes a major medium of human to machine interaction, in which, the LLMs/LMMs often do the translation to deterministic code.

But, more than just inputs, the ability to visualize problem spaces is a critical part of the medium. This is not just in creating new business processes for the Cozy Corner but in how we understand problems. To give an example of why visualization matters and why it works, let us look at four constellations of ten small nodes like in figure 40. How many groups do you see in each constellation?

The answer to the question of “how many groups” depends upon what visualization is used. In the proximity configuration, you’d probably say three or possibly four — the only point of challenge being the node in the very center. In the enclosure, you’d probably say “two groups”. In the similarity or connectivity views then you’d probably say three. These pictures work because of our innate visual abilities. Visualization can be a powerful medium when leveraged appropriately.

Medium and language are changing. But what about tools? In our environment, the existing tool vendors are trying to persuade us that the tools of the future are the same tools they have sold us in the past plus added ChatGPT or equivalent. These future tools are commonly called copilots. Yes, they will add language elements (such as prompts) and will no doubt add medium elements (such as whiteboards). In 2025, it is clear that a copilot with a “build from a whiteboard” capability is just around the corner and an obvious (and somewhat tedious) change.

What this book argues is that we should use this opportunity to fundamentally change our tools as well and for those tools to become more contextual i.e. built to the problem. This might not be a popular message with tool vendors, but we have been sold a pig in a poke — we should have been using contextual tools all along. Now is the right time to change this.

These three concepts — of language, of medium, of tools — play off one another in Moldable Development. The practice of Moldable Development supports the creation of domain specific languages (as in the Cozy Corner example) for the problems you are looking at. Those languages may contain both deterministic and non deterministic forms but they are created through the process of building small contextual tools that can be combined in many ways. Those tools provide a highly visual medium for communication. By touching upon language, medium and tools then Moldable Development is in effect a way of reasoning about the world around us.

Shallow and deep (mining) problems

Ok, this has nothing to do with mining (or kitchen blenders). This has to do with how far we are from the problem we want to reason about.

To give an analogy, consider a stain you observe on the ceiling. You can just as well, say that it’s dirt and have it painted over. But what if the stain is a sign of a mold? Then you need to look beyond the thin paint layer and have the structure treated. But that stain can just as well be a leakage. In this case, the right solution is to go beyond the thin paint layer and the structure and look at the pipes. Limiting ourselves to observing only the outskirts of the system does not allow us to address or even understand anything but shallow problems. True understanding requires identifying root causes and that requires us to explore the system at various levels.

When the stain is just dirt, we are facing a shallow problem. When the stain is caused by the pipes, we have a deep problem on our hands. Yet, when we open the wall and get to the problematic pipes that should be replaced or patched, we are again facing a shallow problem.

Whether a problem is shallow or deep relates to the distance between the observation and the root cause. If we can peel off layers from our system and can explore them on demand, we can transform deep problems into shallow ones. A pipe leak is a relatively easy thing to fix. The hard part is knowing that the problem is a pipe leak and finding where the pipe leak is.

With the right tools even those harder problems can be broken down into more shallow problems. It’s why plumbers have tools like acoustic leak detectors, electronic leak detection, pipe inspection cameras and even use thermal imaging cameras. Notice how in the physical world, for every problem, we quickly break down into a plethora of tools, each designed for a specific kind of problem.

The same can happen with a software system. In this and previous chapters we offered examples at various levels of depth. In each case, when observed from outside the system, the problem looked deep and insurmountable. But when we dug deeper, it became shallow and tractable. You can observe this phenomenon using those questions and answers maps we introduced before (see figure 41).

An important point to recognize here is that questions at different levels are not necessarily provided by the same person, and that they require different levels of expertise. The houseowner asking about the stain often needs the plumber to ask about the source of the leak. Similarly, the business person wanting to know why the services are slow needs the technical person to navigate the technical details (see figure 42).

Those role variants — e.g., the manager, the architect or the developer — also point to who makes the decisions. Underneath the developer is the system itself and one of the interesting aspects of LLM/LMMs is the expansion of where the system makes choices for itself.

To vibe or not to vibe?

Given Eric Schmidt’s and others’ recent proclamations that software engineers will no longer be needed by the end of 2025, we need to return to the question of AI or more aptly the great seduction by it.

It’s probably worth clarifying a few popular terms here. These terms will change over time as people are still exploring and defining the field of generative AI. Software engineering is the act of making decisions in code. It can and often does use an AI as an assistant in this act but inspection of the code remains firmly in control of the human. Vibe coding (or as Simon prefers to call it vibe wrangling) is an exploratory style of programming where humans and AI work conversationally without the humans looking at the code.

Yes, you can use vibe coding to create marvellous prototypes. And both authors extensively use language models to do this. In the act of writing code, the coder is making decisions of what our system should look like. It should always be remembered that anything in the system, including what people call architecture or the ubiquitous domain language, is in the code. The one writing the code is the real architect and the real modeller.

When that coder is the AI then we are handing over control, decision making and understanding to the machine and that’s ok if it is something you trust the AI to do — for example, when working on a prototype. But when doing that, you are also deciding not to develop a language for that domain, not to model that system and not to build the tools to understand that context.

The troubles occur when you start to care about the result. At this point, you have no choice but to involve yourself in decision making. This often manifests itself through a review process i.e. we get the AI to code and then we review the code. In the case of technology, that means software engineering. Despite Mr. Schmidt’s claim, it is unlikely that software engineering is disappearing anytime soon. First, because building trust in AI systems is a social process. Second, because this assumes the AI systems are capable.

In addition, there is also a hidden assumption which is flawed. In early human history, we used to have scribes because literacy was not universal. The current ideas about AI have a hidden assumption that because the business has not in the past been able to be involved in software engineering that the the AI simply replaces those IT scribes rather than everyone becomes literate. It’s really important to grok this, from a business perspective using LLMs or using software engineers is the same thing because in both cases the business had no visibility into the system (see figure 43).

However, the Cozy Corner example shows that there is a mechanism for universal literacy through the views created from the system. These views democratize the reading of systems in a way that is decoupled from their writing.

So, why not get an AI to write the views and we use these views to discuss with the AI through prompts? That’s the future we are arguing for: a human will still be able to make decisions about the system at a slightly higher layer than code review. The key here is that the human doesn’t get detached from the decision making in the same way that the business became detached from the technology choices being made. This is the heart of rewilding software engineering, exposing the workings of the system in a language that is universal (in this case views) without the necessity to read code. See figure 44.

So, why not use AI agents to discuss with AI agents (e.g. some to code, some to read and review the code) in the pursuit of a goal set by the human? You can, and there are some very interesting practices being developed here which we will explore in a later chapter. However, you cannot avoid handing over decision making to the AIs if you cannot review the system. The use of such agents once again puts the business back into a situation of the system being a black box that it can’t understand.

The most important architectural question of today is: at what point do I value human decision making? Investors in AI and tool vendors of AI will tell you that you don’t need to care about human decision making and that AIs don’t need humans but then, they have a self interest in sizzling up that story. Hence claims that we won’t need software engineers by the end of 2025 are common amongst this group.

In general, using AI is perfectly fine if:

• the system is not something you care about. This can be particularly useful for exploring new ideas about a space, much like how you could use AI to get you beyond the writer’s block in front of an empty page.
• you are at the right level and see the right problem, you can specify what you want generated and you are also in the position to evaluate the AI’s output. Always remember that prompts are not a deterministic language and what you get back is the AI’s statistical interpretation of what you wrote, not necessarily what you were thinking.
• you are at the right level to answer your question, but do not have a tool for answering the question, then you might be able to use AI to generate that tool or view for you.
• you do not know where to look, you can also ask the AI for possible questions (even when they are presented as answers).

Don’t get us wrong, the dopamine hit of getting something done with vibe coding can feel delightful, and because of this it’s easy to get carried away in the excitement. But the economic impetus to adopt AI is based on the premise that we have reached the peak of human productivity and the only way to increase that productivity is to either add more people or adopt AI which is much cheaper and abundant. But that assumption is wrong. We are far from knowing what humans are capable of in this space simply because there is no universal literacy.

Bridging the gap

Shallow and deep mining, vibe or not to vibe, the different roles involved (both human and AI) and where decisions matter (both human and AI) — there is a lot of content here but at the root of all these is the skill of asking and answering questions about the context of our systems. This requires a common language between all participants hence the importance of universal literacy.

It’s worth noting there are also specific skillsets involved because these might not be obvious in the first place. Unsurprisingly, those skillsets cover the asking of questions (the stakeholder role) and the provision of answers (the facilitator role).

In our Cozy Corner example, we have multiple stakeholders, starting with the software engineer implementing the message flow algorithm that had the need to visualize it for debugging purposes, to the domain experts (or business managers) that had the need to see the flow to come up with variations at the domain level.

In the case of the data pipeline example described in previous chapters, there were multiple stakeholders as well. We had the CIO that wanted to know why the system was not improving, we had architects that wanted to know what data is and is not used throughout the pipeline, and we had the software engineers that wanted to understand the impact of individual data changes. In general, anyone with a stake in the system is a stakeholder.

The stakeholder’s role is one of asking questions, posing hypotheses and of interpreting answers. Questions can have different goals, like seeking exploration, refining the landscape, understanding the space, or helping define a domain specific language, but they should always be timely, specific and actionable. Once stakeholders get answers, they should know how to transform them into action.

The provision of those answers is determined by the skill of the facilitator. While the stakeholder can be anyone, the facilitator is necessarily technical. They construct contextual tools that provide explainable, representative and accurate answers from the systems. In figure 45, we provide an overview of these roles.

The facilitator is a support role and its function can be more clearly seen in the Q&A map. In order to explore a problem domain, the facilitator helps to industrialise answers through the production of tools to solve that problem. This enables not only new possible questions (the adjacent unexplored) but also the shift of questions from the concrete but uncodified (“how is this piece of data traversing the data pipeline”) to the more abstract but codified (a tool to show how many data is traversing the data pipeline) — see figure 46. The facilitator can be viewed as being similar to a data scientist but for providing answers about systems, and not only domain data. Yes, this is a possible role for AI.

The interplay between the roles can be observed at the highest conceptual levels all the way to the most granular levels of exploration. Consider, for example, the dynamic exploration of the tokenization algorithm described in the previous chapter, as seen in figure 47. The contextual tools materialize answers, but their meaning is given by the overarching questions prompting the exploration.

For example, our first question is “How does tokenization work?”. This leads us to a sub question of “How does tokenization work for a given input?”. This we answer through building the first pane which contains a code snippet that executes the tokenization algorithm which in turn creates raw data. Examination of this raw data leads us to a new question of “What are the resulting tokens?” which we answer by creating a new dedicated view (the 2nd pane). Exploration of an individual token from this view, creates a third pane with raw data. This leads us to a new question of “How was a token constructed?” which we answer with a new view (the 3rd pane).

The choice of a view must be a conscious step. This can be harder than it appears when you are used to working with standard tools with constrained views that you try to fit into your question. When following Moldable Development, we start with the question and then build or pick the tool with an appropriate view that helps drive the exploration of that question.

This can, and does, create multiple paths for exploration. For example, in the scenario above having created the output for the snippet in pane 1 then we could go down a different path. Instead of exploring “What are resulting tokens?”, we could explore “What steps did the algorithm take?” leading up to the question of “What happened at each step?” (see figure 48). Navigating these pathways is a skill that the stakeholder must learn.

Roleplaying with AI

How does AI fit into these roles? Is it a stakeholder, a facilitator or both? The answer to that question depends a lot upon context and time. For example, in today’s world, with our current systems, AI can act as a facilitator for prototypes and contextual tools. As its capabilities extend then that will change over time. Though these systems don’t reason (for now) despite sophisticated code generation, the combination of many agents can simulate such reasoning especially for shallow problems.

But, this is performative i.e. there is an appearance of competence without substantive problem solving. This can pose an issue when trying to solve deep problems. Hence, when a pattern is known or well defined or when we can quickly evaluate the output, then AI can act as a useful facilitator.

Perhaps unexpectedly, LLMs also have a role in asking questions (i.e. being a stakeholder) or at least coming up with alternative questions to ask. As time progresses, capabilities develop, practice co-evolve and trust is earned then that boundary of what we will use AI for will change.

We can summarize this all into one choice that needs to be continuously made, where does it matter to have humans involved in the decision making. The impact of LLMs is shown in figure 49 in both helping in the creation of hypotheses (a stakeholder role) and in generation of specific coding / tools (a facilitator role).

The reconcilliation

We started this chapter with an apology and a description of how software engineers can appear to be unhelpful, pedantic and inflexible. This is mostly caused by the exclusion of business from the decision making in the systems which in turn is mostly caused in the current practices of software engineering. This is why we are writing the book, it’s not just for software engineers, it’s also to bring the business closer.

Challenging the nature of software engineering (our first wolf in our rewilding metaphor) and the importance of this being a decision making process is as much for engineers as it is to highlight how the business should be involved. This requires us to challenge the way in which we ask and answer questions (our second wolf) and in particular the tools and visualizations we use.

That may not be comfortable for some, but both wolves are needed if we ever want to end up somewhere cozy.

What did we learn?

This chapter adds several important new concepts.

Stakeholder and Facilitator roles: there exists discrete but complementary roles with different skillsets that formalize the relationship between questions and answers. Each of those roles is focused on a different aspect and governed by their own metrics — ttA (time to Answer) and ttQ (time to Question).

Language, medium, and tools as pillars of reasoning: There exists three elements that are foundational to how we reason about software systems which is an essential part of decision making. Instead of building solutions directly, we should aim to build the language of the problem first, which includes contextual tools to visualize and manipulate that problem space.

Deep and shallow problems: visualizations are critical for transforming deep problems into shallow ones by revealing the system’s inner workings.

Vibe Coding vs. Software Engineering: AI has a role in code generation, facilitation and as an assistant to stakeholders. That role will expand over time but the boundary between human and AI decision-making is an architectural choice that should be made deliberately based on where human understanding and control are valued.

Homework

For this chapter’s homework, we need you to inspect your own organization and current practices. This is mostly an exercise in self reflection considering aspects of this chapter.

Step 1. Visualization audit:
Examine your current development process. How many of your visualizations (diagrams, charts) are manually created beliefs about the system versus automatically generated reflections of the actual system? Create a plan to move at least one key visualization from manual to automatic.

Step 2. Data sources audit:
Examine how you get information about your systems. How many surveys did you have in the past 3 years that were asking teams about how they use or deploy technical capabilities (such as for clouds)? Pick one of them and from one of the questions explore what it would take to get the answer through a tool rather than by asking for opinions.

Step 3. Role identification:
Identify the stakeholders and facilitators for a system in your organization. Document the questions typically asked by stakeholders and how facilitators currently provide answers. Where are the gaps in this communication? Does the organization think about how we ask and answer questions or these different types of roles? As a concrete way to identify facilitators, start from Step 1 and Step 2 and ask yourself who would implement those tools.

Step 4. Domain language exercise:
Choose a small but important process in your organization. Sketch a domain-specific language that would describe this process. What are the key actors, messages, and rules? How would you visualize the flow?

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding software engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Tudor Girba and Simon Wardley

Making systems explainable

In the previous chapters we introduced two important concepts. The first is that software engineering can be viewed as a decision making process. Our current processes are suboptimal, relying on beliefs and gut feel. When our mental model differs from the reality of the system, our actions will be misguided and we end up wasting a tremendous amount of energy. The way to solve this problem is to align the two. To do this, we need to look at how we inform ourselves about the system and for that, we have to look at the tools we use. In the physical world, you wouldn’t use a kitchen blender to create a deep shaft mine. In the digital world, this is precisely what we do, and this activity is highly aligned with how tool vendors tell us that building tools is hard. Well, it’s not. Challenging the nature of software engineering is our first wolf in our rewilding metaphor.

Metaphors are great, but shared experience is better. A large company (multi-billion dollar market cap) relied on robots to produce wafer technology. In turn, these robots relied on a real time distributed system. At some point, the company realized that they no longer knew how this system was working. The system was created a long time ago and became a black box for them. They gave our team at feenk one year to try to make sense of it. After a month, we had a meeting in which they asked what else we would need to get started. And we answered that we have finished and demonstrated it with a live running model of the system. That’s when they shared with us that there were bets in the company saying that we have no chance of succeeding in the allotted one year time. They assumed this because multiple past efforts over many years had been made and failed. What differed with us, is we started the project by building the tools we needed to solve the problem. This is how we succeeded and how we beat any and all expectations they had. It’s important to note here that this is not a story of 10x engineers. It is a story of a different method through which we made an opaque system explainable.

The second concept is that even if we accept that software engineering is a decision making process, we still have to think about, measure and optimize how we ask and answer questions. By decreasing the time it takes to ask and answer questions, we get to churn through more of them. The more we do that, the more likely it is to discover new value and interesting solutions. By not measuring or even asking questions about how we ask and answer questions, we deny ourselves this opportunity. Imagine a gold prospector panning for precious nuggets in a river. The traditional method of sifting through sediment is slow and laborious, allowing the prospector to process only a few pans of river mud each day. This is akin to our current approach of asking and answering questions without optimizing the process. Now, picture that same prospector inventing a more efficient sluice box system that can process ten times the amount of sediment in the same time. Not only does this increase the chances of finding gold, but it also exposes the prospector to a wider variety of minerals and geological formations they might not have encountered otherwise.

As we said, metaphors are good but shared experience is better. In finding our solution to the multi-billion dollar market cap company’s robotic problem, we spent three weeks of the total one month building dedicated inspection and debugging tools. These tools provided visibility which then allowed us to validate more hypotheses. And eventually, we ended up with a working solution.

Challenging the way in which we ask and answer questions is our second wolf in our rewilding metaphor. Both these wolves are visible in figure 20, in terms of the decision making process of what software is and what it should be, and in the separation of time to question and time to answer.

Both these wolves are connected by a common goal: the pursuit of explainable systems.

Systems’ explainability

For the longest time, ideas about quality regard the understandability of code as being a function of the code itself. If you write it in the right way, with descriptive variable names, with concise functions, with logical structure, with indentation, using the latest rituals of separating concerns and inline documentation then it would be explainable. Because explainable meant readable and that’s what we do, we read code to understand it.

This led to quality models that are made out of trees of measurements that are compared with industry accepted thresholds to identify problematic places in code. Metrics like McCabe complexity have a prominent place in this space as they are considered to denote difficult to grasp code. The same thinking also led to practices, like those behind Clean Code, which focus on the code structure and its readability as a differentiating factor.

While we agree that readability is relevant, readability is not the most effective path to understanding. What is required is to make a system explainable. This depends not only on the system itself, but also on how you extract the information from it, and how you use that information. If you attempt to make the system explainable only by focusing on the system itself, you miss the opportunities offered by the other two levers (see figure 21).

That explainability depends upon how we extract information and how we use the information can appear counter intuitive. But consider these two scenarios:

• You have a problem about a system that you do not know but wish to understand. If you are only equipped with a basic text editor such as a basic notepad then you are likely to fare much worse than if you have your favorite development environment. In other words, tools make a difference. How we extract information matters.
• Now consider two different engineers against the same problem: one with experience in the technology and the system, and an intern that just experienced their first assignment. The former will likely do better because they have a wider range of experiences and skills to bring to the problem and can find the root cause faster. How we use the information matters.

The structure of the system can certainly influence explainability, but skills and tools are at least as important. Meaningful tools and meaningful skills matter.

Example: The stuck cursor

For quite some time we had an issue in the text editor of a project known as Glamorous Toolkit. The problem was related to the cursor getting stuck when navigating with the keyboard and the cursor was at the end of a paragraph. It was annoying, it was visible but we did not know how to address it and it stayed open for half a year.

We knew that the issue was likely due to something at the lower levels of the editor implementation, but we did not know what it could be. We did not even have a meaningful hypothesis.

While Glamorous Toolkit is primarily written in Pharo Smalltalk, it also relies on Rust plugins. The text editor is based on such a plugin, and it is essentially made out of wrappers over Rust objects. This is nice because it allows us to reuse external libraries. At the same time, to find issues visible in the high level editor, we need to understand how the low level objects work. We can’t simply treat them as black boxes. Ideally we need at least the ability to inspect from the text editor to the wrappers to the Rust objects depicting the low level details of paragraphs. Unfortunately, back in 2022, we did not have this capability. For the Smalltalk objects we had extensible inspectors, but for the Rust objects we only had a separate and standard development environment.

As the problem with the cursor seemed to be related to its positioning, we naturally investigated the various positional indexes of the Rust objects using the standard inspection tools. We looked at paragraph objects and inspected their text ranges — at which index they start and at which index they stop. That did not lead to any insight even after several people looked at it over a longer period of time. For some inexplicable reason this was a seemingly impossible problem to solve.

As we do in cases in which we do not see the problem, we stop and work towards building a tool that shows it. In this case, this meant a longer detour. We had to extend our inspectors beyond Smalltalk and into Rust. Expanding a toolset into a new language is not a minor undertaking, this detour took a couple of months of work but now we have the capability to inspect any Rust object.

Once we had this capability we could return to the problem. And the problem became obvious within seconds. The screenshot below shows the actual exploration that led to the bug resolution (see figure 22).

From the figure, the left hand side is the Pharo Smalltalk text paragraph object. The middle pane shows an inspector of the corresponding Rust object. This one shows a view with details, such as the line ranges. To the right, we see the positional indexes. So, how did this help see the problem?

Take a closer look at the last pane. The first line has a range between 0..30. The second line has a range of 30..64. When we saw this, we realized our mistake: in our Smalltalk implementation we assumed that the ranges were inclusive i.e. 30 is the last position on the first line, and that the next line should start at 31. But when we saw that the second line starts with 30, we realised the ranges were not inclusive. 0..30 means 0 up to but not including 30. Our cursor problem was due to a mismatch between what we believed and what the framework actually did at a low level. The fix to the year long problem that had plagued us involved changing one character of code.

This real life example shows a couple of things. First, in contrast with the case study discussed in chapter 2, the current problem was tiny and low level. Yet, to address it, we relied on the same approach as we do for all problems. The view is not fancy at all but meaningful and seeing it made both the problem and the solution obvious. The explainability of the system depends more on how you seek the explanation than on the system itself.

To better understand the dynamics behind this statement, let us look at the questions and answers behind this example.

Questions and answers: the stuck cursor

Our first question was unsurprisingly “Why is the cursor stuck at the end of the paragraph?”. This question is actionable (if we know the answer we could do something about it), specific (it’s not some generic question but very precise to our context) and timely. We had no answer to this question but we suspect it has something to do with the Rust objects. This gave rise to our second question “What are the wrapped Rust objects looking like”. This question is a lot less specific and we had no tools for seeing the wrapped Rust object within the context of the Smalltalk object. We had to go outside of our environment and rely on generic tools for Rust in the hope of providing some sort of answer. This is shown in figure 23.

By building the Rust inspector we could answer the second question with a tool within our environment that would allow us to navigate across language boundaries. Seeing the interconnected objects led us to ask the question of “Are ranges inclusive on both ends?”, the answer of which also answered our original question of “Why is the cursor stuck at the end of the paragraph?” (see figure 24)

We didn’t answer our original question by creating answers to the things we suspected were happening. Instead, we invested in our ability to explore and our exploration led us to a new question we had not previously considered and the answer to that enabled us to solve our original question.

In our experience, this is a fairly common pattern. The answers to the questions we think we need often lead us to new questions we hadn’t thought about — the adjacent unexplored.

Exploring the stuck cursor with LLMs & copilots

But wait a second, should we not use an LLM instead to find the answer, especially for a problem of such reduced scope? Well, at the time there were no LLMs to use. Today, it would certainly be a legitimate path to consider. We will discuss the use of AI in a later chapter, but for now, let us take a brief detour and explore our stuck cursor problem with a couple of leading LLMs & copilots today.

The results of asking LLM-based chats were varied including a suggestion to search Stack Overflow for “Skia Paragraph cursor position”, “Rust text editor cursor stuck” and “Rust Skia keyboard navigation” — none of which resulted in useful articles.

We then used a copilot to digest the relevant code and when we asked “Why is the cursor stuck at the end of the paragraph?”, it provided us with several dozen possible suggestions. In our case, none of these described the range inclusion path.

But isn’t the problem just due to the wrong prompt or lack of context? Maybe a different prompt or tool could improve the situation. However, even if the range suggestion was covered, it would still be one of many that would require assessment against the system. The suggestions that come out of an LLM are in fact questions to explore disguised as answers. We still have to assess them, just as we would with suggestions or code created by a human.

But can’t we use agents and won’t they fix this assessment problem? Yep, that’s an answer. You can stop reading now. Unless we want a human to be able to make the choice in which case the human needs to be informed i.e. the answer should be explainable, representative and accurate. Choice leads us to a need to assess which leads us back to the problem of how we answer questions.

But we don’t make the choice in my self driving car, why should we care? It boils down to HITL, HOTL and HOOTL. If you can forever accept humans out of the loop (HOOTL) of decision making then by all means you can pursue the path of agents and stop reading. But be aware, you are consciously choosing to become the Eloi in HG Well’s Time Machine by letting your ability to reason about the problem atrophy and hence creating the potential for E.M. Forster’s The Machine Stops.

But aren’t machines more capable than humans? That argument assumes we have reached the pinnacle of human’s ability to make decisions about software systems. We argue that we’re nowhere near close. We are drawing conclusions about decision making without even considering what it is, what we do and whether it has been optimized. Because this has not been explicitly explored, we’ve ended up limiting our ability through constrained tools. And, the typical discussions are often the equivalent in the physical world of saying robots are faster at building deep mines with kitchen blenders than humans are.

If you are willing to accept that we’ve not optimized human decision making and you want humans to be able to participate i.e. Humans In The Loop (HITL) or On The Loop (HOTL), then keep reading. This is not to say that artificial intelligence is not useful — it is. However, we should not hastily discard our own ability to explore.

But isn’t there a way to use LLMs to enhance our ability to explore? Yes. The key is to use the LLM to help build the tools that you need. We do talk about this in a later chapter but figure 25 spells it out. LLMs can readily provide summaries, but the problem with these summaries is that the answer is not explainable. Instead you can use LLMs to build deterministic tools that you can use to produce explainable summaries. This approach allows you to inspect what the tool does and even adjust it before you use it.

But why do I need to use an environment like Glamorous Toolkit? Can’t I just ask the LLM to produce the tool when I need it? Eventually they will. LLMs will be able to take more context into account and produce more sophisticated tools. They will also be able to guide and support you in the use of those tools. The explainability of an answer depends on how well we understand what the tool does. And the way we understand tools is how we understand anything else in a system: through tools. Therefore, you need an environment in which understanding an external problem is the same as understanding the tool with which you understand the external problem. It’s recursive and of course, eventually LLMs will provide this but you’re back to the problem of handing over your decision making to an LLM.

But I don’t understand how electricity is generated. What’s the difference with handing over this creation and understanding of tools to an LLM? Eventually, there will be no difference as long as we have a path to understanding what the tool does. But that path would require the LLM to create tools to understand tools and then tools to understand tools to understand tools and … it’s tools all the way down. You will still need some sort of structure to manage this all.

But can’t an LLM create me such a structure? Yes, eventually. This will be greatly eased by having a systematic way to describe this structure. The environment itself can become a language made out of visual and interactive operators.

But can’t an LLM create that language? Eventually. This is what we show in Glamorous Toolkit, that a language exists for decomposing problems in the same way. In fact, that language was used in the building of Glamorous Toolkit. As mentioned we will talk more on this point later in the book.

Can’t you explain now? We will get there … eventually. Just like your LLMs but much sooner.

Unlike the stuck cursor problem, typical software engineering problems are vast by comparison requiring even more exploration. Yet, even for a small case, it proved useful to tackle the problem through the lens of contextual tools. This benefit is more visible when the problem grows in size. And all this is due to the highly contextual nature of software.

The contextual nature of software

Think of a simple calendar application. It has months, days, years and the ability to add appointments. It’s very simple. Many of the components needed to build such a simple thing are fairly standardised. So, what happens if you ask seven groups of engineers to build a highly well defined and constrained calendar application? Do they come up with the same structural design? Think again. In figure 26 we show the results of this very test.

In this experiment, seven groups of engineering students were given the task to build the same system. They were provided with a given set of libraries and a standard set of development tools to use. To constrain it even further, each system was evaluated externally using the same test suite.

In the figure above, we visualize the code structure of each system in terms of classes, methods and attributes and the relationships between them. Namely, the classes are depicted with black dots, the methods with red dots and the attributes with blue ones. A call between methods is shown with a red line, and an access to an attribute with blue lines. (Of course, we built a tool to produce this visualization)

What do we learn from this? Even in a highly constrained scenario, the produced structures are radically different.

To solve problems with any one of these systems, then you’re going to have to consider the context of the system itself. Yes, they are all calendar applications but they are not the same calendar application. A tool for one is not a tool for another unless we simplify the tool to ignore any context. This is basically what a standard tool does. It ignores context and treats every problem as the same. But, if the only tool you have is a hammer, then every problem becomes a nail regardless of whether you’re building a formula one racing car or a deep shaft mine.

The contextual nature of software means that we can predict classes of problems (e.g. the calendar speed) but we cannot predict the specific problems that appear in practice (e.g., when clicking on add meeting participant, the response is slow because the firewall throttles the access to the active directory of users).

Ready made rigid tools cannot be effective because they bake the problem in the tool upfront. They do solve a problem, but they solve someone else’s problem. A tool built before the context is known, will be solving generic issues, just not yours.

If we go back to the parallel with testing then tests can be created either after the code is written or before, as is the case with test driven development. However, tests are always created after the problem is known. We don’t go and download a suite of 100,000 ACME Certified Tests for Any Application because these tests would not know anything about our system. We want tests that are specific for the context of the system.

This principle is applicable to any tool. We should want tools that are specific for the context of the system. As we cannot predict the specific problem, and as we need a problem to build a tool, it follows that the only alternative is to create the tool during development, after each problem is known.

However, this leaves us with a question. If we need to explore to find the problem before we can build the tool, how do we explore?

Defined versus dynamic exploration

Let us suppose we decide to build tools to answer questions and each of those tools needs to be contextual. Doesn’t that lead to a situation where a large, core system could end up with thousands of tools? Yes, it does. It’s not that different to having thousands of tests. This leads to several further questions. Wouldn’t it be massively expensive to build and maintain thousands of tools? And, even if we can afford to have thousands of tools, wouldn’t that be overwhelming? Also, how do we hope to explore a system if we’re spending all our time trying to work out what tool we should be using? Let us take each question in turn.

1) Aren’t thousands of tools massively expensive?

In short: they do not have to be.

From a JetBrains study on Jupyter notebooks, we learn that people like creating notebooks copiously — they examined some 10 million of them. Each notebook represents someone’s analysis of a specific problem of interest to them. You can think of these as small tools that typically consist of only a few hundred lines of code. Jupyter Notebook is a toolkit that reduces the cost of creating and managing tools.

Another toolkit is Glamorous Toolkit (GT). This was used in the stuck cursor example. GT is itself built with and contains over 5,000 tools which were created during the development of the environment itself.

In figure 27, using a tool built specifically for this context, we provide a visualisation of the spread of contextual tools throughout Glamorous Toolkit. The visualization shows a treemap with all packages and their classes from the system. Each blue rectangle represents a class for which at least one contextual tool has been built.

The visualization demonstrates two things. First, that it is possible to create and maintain thousands of tools for a system even by a small team of eight people. Second, it shows that no part of the system is untouched by contextual tools once the possibility of creating them at low cost exists. The need is pervasive.

Furthermore, in this particular case, the average size of the extensions is 12 lines of code (of which 2 lines are the name of the function and a boilerplate annotation). Of course, some tools are more elaborate than others, but the start is most often a few lines that we build in minutes and see an increment and that we continuously expand as we need. The low threshold to start makes the approach feasible.

2) Aren’t thousands of tools overwhelming?

In short: they do not have to be.

There are billions of pages on the web. Yet, we do not qualify them as overwhelming. That’s because we are not exposed to them all at the same time. We do know the name of some (hello google.com), but most appear in context, during our exploration. Not only must our toolkit enable contextual tools to be built, it should also allow those tools to appear in the right context. In the case of systems, this is not a simple search for a tool but instead when inspecting an object or browsing a class, the right tools need to gravitate towards you.

Whilst obviously useful, given the number of notebooks, this is one of the limitations with Jupyter notebooks. The books are contextual tools as they are created for a specific problem, the toolkit itself does not mold itself to the context that you are looking at beyond simple searches. Where you do interact between notebooks (tools) you have to explicitly call that tool by name. This is a static or defined form of interaction. What is needed is for the right tools to appear based upon the context that you are exploring but that requires association of the tool to a context and some form of modelling of that context. However, whilst words are great this is best explained with an example.

Example: Andrej Karpathy’s tokenization explanation

Not long before the writing of this book, Andrej Karpathy put together a tutorial on tokenization — a prerequisite for the current functioning of large language models. The tutorial has multiple parts including the stepwise implementation of an algorithm and an explanation of what tokenization is. Andrej is a prominent engineer and a brilliant teacher, and he starts by saying that “before we dive into code, I’d like to give you a brief taste of some of the complexities that come from tokenization” and then he goes on to exemplify how tokenization works through an external application called Tiktokenizer.

The app is based on the tiktoken library which is provided by OpenAI to help people explore the amount of tokens for a given input string. In addition to the basic counting of tokens, the app provides an interactive way to graphically explore the tokens as well.

Andrej is relying on this tool because it helps newcomers build an intuition, a mental model of what the technical algorithm does. However, this explanation comes from an external tool (see figure 28), outside of the development tool he is using — in his case, a Google collab notebook which is based upon Jupyter notebooks.

It’s really important to grok this point, so we re-emphasise it. Whilst it might be technically feasible to rebuild the functionality of a tool like Tiktokenizer within a Jupyter notebook, the practice is not to do this but instead to constantly switch context to external tools, provided that those tools exist at all. So, what happens if we could easily (and cheaply) bring that tool into the context of the development experience?

To do this, we use a Moldable Development environment. In our case, that’s Glamorous Toolkit and we explain the algorithm’s output through contextual views. To illustrate the potential format of the explanation, one of our colleagues took the tiktoken library and wrapped it in a small project. While the tiktoken library works with strings and numbers, our mental model is about inputs and tokenization results. So, the wrapper adds entities like a TokenizerResult. When inspecting an instance of this entity, the inspector readily provides dedicated views, such as a graphical representation that’s equivalent to the one from the web application as shown in figure 29.

This took about an hour in total.

So, why is this interesting? We now have an explanation closer to the development process which is “nice”, but isn’t this just a waste of an hour on pointlessly trying to avoid context switching? By bringing the tool into our development environment, it enabled us to show something that was not previously possible to see — an explanation of the algorithm.

More importantly, if we can explain how the overall algorithm works in terms of input and output, we can also explain more detailed pieces. Let’s take a look. The tokenization logic is iterative. The input string gets transformed into numbers in steps, each step performing a compressing operation in which two tokens are identified and merged. We have a view which explains this logic, so let us now bring that view to each of these steps — see figure 30.

In the above we not only can see the visualization of the overall output, we see the visualization of each merge step. In this particular case, because we needed access to the code of the tokenization algorithm, we took the code that Andrej implemented during the tutorial instead of the blackbox tiktoken library. Once we had the previous infrastructure, the new increment was small, perhaps 15 minutes, to bring the total cost to 1 hour and 15 minutes.

By adding one more view to show the list of all merges, and concatenating it with the visualization of each merge, we basically get a postmortem debugger for the tokenization logic (see figure 31).

Now, Andrej is a fantastic teacher, but to provide his explanation he went to an app outside of his development environment. And when he got to explaining his algorithm, the explanations were delivered with plain numbers and strings. With just over an hour of effort, he could have had a visual debugger for the tokenization logic in the same environment and anyone else using the same class of Python objects would have access to the same.

Unfortunately, our expectation of how explainable systems are tends to be low. Interesting explanations are rare and when one does appear it is somewhere else, removed from where development happens. In contrast, our little demo shows that with little added effort we can make interesting explanations be an integral part of the development experience. These explanations should not be unusual events. They should be pervasive. Everywhere. All the time. And most importantly, in context.

Tools are overwhelming only if we see them all the time. If our tokenization debugger appeared every time we did an arbitrary exploration (e.g., inspecting network traffic) then we would quickly get overwhelmed especially if the 5,000 other tools in Glamorous Toolkit appeared at the same time. However, if they appear when they are meaningful, they become useful. Thousands of tools can co-exist by building and making them appear in context, e.g. only the ones that matter appear for the object under investigation.

3) How can we hope to explore a system if we’re spending all our time trying to work out what tool we should be using?

In short: you’re not. If the toolkit is designed for dynamic rather than defined exploration.

In order to understand this, we first need to clearly understand what a tool is. A tool is something that solves a problem. A generic tool is something that was designed for generic problems but we try to use this to solve our problem. This is known as exaptation. A contextual tool is something that is designed to solve our specific problem. In order to design it, we need to explore our problem. Hence, exploration should be part of the design process for the tool.

However, no problem is an island. In solving a problem, we often have parts of the answer but not the entire answer. These parts are reusable components that were created as part of other explorations. In our kitchen blender analogy, whilst a kitchen blender is not useful in deep shaft mining, it does contain nuts, bolts and copper wire which are part of mining equipment. In the same way, in solving our problem there will often be components from the solving of other problems. Hence our tool often contains components built for other explorations as well as components built specifically for ours. We assemble these into a contextual tool that answers our question.

For example, to answer the question “How does tokenization work” we need to answer the question “A) How does tokenization work for a given input?”, “B) What steps does the algorithm take?” and “C) What happens at each step?”. It is the combination of A) + B) + C) that gives us our overall answer, see figure 32.

Tools can be designed in at least two different ways: verb-noun or noun-verb. A verb-noun design starts with the action and then asks for the content. For example, when you invoke Open from a File menu, you get a dialog in which you must select the file to open. Open is the verb, the selected file is the noun. The tool is not contextual in nature as the context (e.g. the file) has to be provided to the tool.

In contrast, when you select the file in your file explorer tool and double click on it to open, you are facing the opposite design: the content comes first and the verb follows. This design also allows us to add options in a context menu to open the file with various other tools. Such a design is more contextual in nature because the content governs what can be done. This is the paradigm that a moldable development environment must follow — context first, action second. Hence in the figure above, the inspectors (the right hand panes) are reacting to the content under exploration by providing contextually appropriate tools. Furthermore, each tool allows us to interact with it and spawn dynamically connected contexts.

That last part is really important. Remember, each tool offers an answer to a question. Hence, in our exploration we are combining not only contextually relevant answers to other questions but also our environment is adjusting to the context under exploration, with tools popping out when relevant. This also lays the ground for surprising discoveries because as we explore, we wander into different contexts and are able to see what previous explorers have already found. We can combine previous explorations with our own and the overall experience emerges out of small increments without an overall dedicated tool. In other words, our tool literally molds out of our exploration.

This process is known as dynamic exploration which complements defined exploration (see figure 33).

A defined exploration can be seen in notebooks, such as Andrej Karpathy’s tokenization example or the notebook in the image above. These notebooks are composed of multiple small snippets of text, code and associated results. When the notebook is created, every new snippet is indeed added dynamically, mostly at the end, but as soon as the notebook has been created it can mostly only be consumed from top to bottom. This form is extremely useful for describing what someone has learnt in the past, as exhibited by the 10 million notebooks created in the data science space, but it offers little possibility of going off on a tangent starting from the middle of it. You can interject a snippet in the middle of the notebook, but this breaks the overall story. Moreover, we get to see the result of executing snippets, but we cannot explore their output beyond the scripted visualization.

But why do we need dynamic exploration? Because new questions require new answers that need exploration. Those are nice words, but what does this mean in practice? It means our tool should emerge from our exploration of the problem and should not be constrained to someone else’s problem. The critical change needed for this dynamic exploration is that the tools must not only be written for a context but also appear in a context. When exploring an object, only the few tools relevant to that entity should appear as discussed in the previous section. This is the equivalent of late binding answers to the question rather than setting off with tools with preconceived notions (i.e. early binding) of what the answer might be.

So, “How can we hope to explore a system if we’re spending all our time trying to work out what tool we should be using?” You’re not if your environment is moldable. However, more importantly, you should be asking of other tools “How can we hope to explore hard problems if our tool doesn’t mold to it?”

Dealing with hard problems

All interesting problems are inbetween existing tools. The hardest problems are those for which we do not know how to formulate an interesting question. An example of a hard problem is the stuck cursor one as described earlier in this chapter. In such cases, we start by describing what we know already. This then provides visibility into the adjacent unexplored and helps us discover interesting questions.

The difficulty of the problem is not in the solution i.e. providing the right answer but in finding the right question. In our case, once we saw the problem, the solution was easy and it involved changing exactly one character of the code (“=<” was changed into “<”). Finding this was possible through the serendipitous nature of the dynamic exploration

What did we learn?

The explainability of a system depends not only on the system itself, but also critically on how we extract information from it and how we use that information. Traditional approaches focus primarily on code quality metrics and readability, overlooking the crucial roles of tools and skills. The example of the stuck cursor problem demonstrates that once developers could see the actual problem through specialized inspection tools, the solution became easy. The system did not change. It was the creation and the use of the tools that changed the problem.

By building tools specifically designed for the context at hand, seemingly insurmountable challenges transform into manageable tasks. This can lead to thousands of contextual tools. These need to be inexpensive to create and they must only appear in the context for which they were defined to avoid overwhelming the engineer.

The need for contextual explanations is pervasive due to the contextual nature of software systems. Even highly constrained implementations will result in radically different structures, explaining why rigid, pre-built tools often fall short.

The pervasive use of notebooks shows that the need is not singular to software development. Notebooks offer defined explorations that are to be consumed mostly in the same combination they were created. But as any answer provides the context for new questions, we can still benefit greatly from an environment that allows for dynamic explorations as well. An environment that allows us to jump from context to context at will.

Homework

In this homework assignment, you will apply the concepts of system explainability by identifying, designing, and evaluating contextual tools for a software system of your choice. This will require some engineering skill, so if you’re not an engineer but instead run teams of them or interact with them, go find an engineer to help.

Step 1: Get a stopwatch. Set the timer for 60 minutes. Start working on the following steps and stop when the timer finishes.

Step 2: Select a large software system that you work with regularly. Open up a notebook. To the notebook add a title for the system.

Step 3: Determine the number of lines of code within your system. Add this to the notebook.

Step 4: Determine the distribution curve for function length within your system. Add this to the notebook.

Step 5: Determine the distribution curve for “if” statements within functions in the system. Add this to the notebook.

Step 6: Build a mechanism to select a particular distribution of “If” statements (such as 35 “If” statements in a function) and browse directly the functions that are responsible i.e. you should be able to click on the results in your notebook (from step 5) and examine the code.

Step 7: Find a function with a high number of “if” statements and go explore why this is.

Step 8: Generalise the tool for any type of statement, such as for loops.

Working in GT (Glamorous Toolkit), these tasks took about 18 minutes to complete. For example the distribution curve of IF statements is provided below (figure 34). How far down the list you get will help you understand how moldable your development environment is. Without a highly moldable environment or a pre-built tool designed for these specific problems then the list will be difficult to complete.

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding Software Engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Tudor Girba and Simon Wardley

In chapter two we discuss how we make decisions and introduce the idea of moldable development with an emphasis on legacy environments. We provide an example that we encountered in practice. The problem was large, costly and visible all the way to management. At the end of the chapter, we ask about the role of questions and answers. In this chapter we explore it in more detail.

To begin with, we’re going to make a bold statement. Moldable Development is applicable equally to any software engineering problem. This is not just about legacy environments but everything we do. To understand why, we’re going to discuss the nature, characteristics and finally the role of questions and answers with our example of a data pipeline.

On their nature

Let’s start from the Wardley Map we construct in the previous chapter, only this time we highlight the tight decision making loop in figure 8. We have a hypothesis on how something works, we explore the space through conversations with the system and people via the tools we have, we assess the situation and if we have confidence we make a decision or decide that we need to refine the hypothesis.

Unfolding that loop leads to the scientific method, as shown in Figure 9. Being a loop we can start from anywhere. Often, when applying the scientific method in an empirical situation, the start is an observation that results from exploration. In other cases, the start is an explicit hypothesis. Regardless of where we start, the loop typically follows the steps from Figure 9.

Our ability to make decisions is governed by the speed at which we move around the loop i.e. the flow of decision making. This has two major components, the time to ask a question (ttQ) and the time to answer (ttA) a question, as depictured in Figure 10.

One of the more surprising things about modern knowledge management and information theory is we never seem to record, evaluate or even ask the question of how long it takes us to ask and answer questions. At best, we examine our time to action i.e. to fix something or to make a decision. In the world of DORA, for example, we record, evaluate and ask about Time To Recover (TTR), Change Lead Time (CLT) and Deployment Frequency (DF) but at no point do we ever consider how long it takes us to formulate a hypothesis (as to what is going wrong with the system) and answer it. We examine the action and not the components that make the action.

The conceit behind moldable development is that systems are highly contextual. What that means is that we often do not have the most appropriate analysis tool at hand for a specific problem. In that situation, the default choice is to perform the analysis manually as discussed in the previous chapter.

The moldable development alternative is to stop and to mold or synthesize a tool with which we carry the analysis. The purpose is to speed up this loop and our time to answer questions. Of course, this implies that the synthetization cost plus the cost of using the tool and the speed gained must be more attractive than the manual alternative. The common practice today speaks against this idea as “tools are hard and expensive”. In the previous chapter we show that it is not the case. But our claim goes further than that example: creating micro tools works for every problem for which a specific tool does not yet exist, regardless of granularity.

As surprising as this may sound, it is not without precedent. For example, a few decades ago, regression testing was predominantly done manually, while today it is commonplace to automate much of the testing through small custom built scripts created specifically for different functional aspects of the system in an accumulating suite of test scripts. If a vendor tried to sell you a test suite that would work for your application and everyone else’s then you’d probably laugh at them. However, that’s exactly what we do with other tools. For example, static analyses, such as lint rules, are often downloaded from the web and used mostly “as is” in different systems. At the same time, it is not uncommon to have situations in which people have thousands of static analysis warnings, while all tests are green. It’s not because they do not care about the warnings, it’s that the warnings literally address someone else’s problems. The very act of creating a test within the context of a system encodes contextual value that is lacking when the check was created generically.

Another example of synthetization of tools can be seen in the space of observability, which advocates for the evaluation of the behavior of a system through custom signals that are system specific.

We’ve added this tool synthetization to our diagram (see figure 11).

While testing and observability are important, they focus primarily on the functional aspects of a system. The idea of creating micro tools applies to any other aspect of a system, including static checks, algorithm visualizations, architecture diagrams, performance evaluation, or configuration browsers.

The same Moldable Development cycle can also be visualised on our Wardley Map as seen in Figure 12, in which we linked hypotheses to Moldable Development and the synthesization of composable micro tools.

Given that our map and our flow diagram are equivalent, we can now import the concepts of ttA (time to answer) and ttQ (time to question) onto our map. To do so, we have to realise that the blue part of the map represents the technical details, how we extract information and how we answer questions. Whereas the red part denotes how we use information for value creation, i.e. the more business end of the system. In the red area, what matters most is paradoxically not the answer (something we use) but whether we’ve asked the right question. Finding the right question often involves asking many questions. Hence ttQ (time to question) governs.

So, the top of the map is about questions, the bottom of the map is about answers as shown in figure 13.

When we say that developers spend most of their time on manual inspection, it becomes obvious that the ttA is large and the energy of the organisation is going into finding answers. If all our energy is going into finding answers, then we probably can only answer a few questions and the process of decision making becomes one of striking it lucky. By taking a moldable approach, we reduce the energy cost involved in finding answers enabling us to invest more of our energy into finding questions to ask. It improves our odds of finding that right question. We essentially make it far more likely for us to strike it lucky i.e. rather than one or two hail Mary passes, with the same energy we get to try hundreds of times.

It may not be obvious, but the process described here involves dedicated skills. We say skills in plural because the skill needed to answer questions is not the same as the skill needed to ask questions. The process also requires appropriate technology to make the creation of tools inexpensive. We address both of these in detail in the later chapters. For now, let’s assume that it is possible to create those tools fast enough and that the skills are internalizable.

That still leaves us a problem which was demonstrated by a group of school children. When a small experiment was done in the form of a game to test the speed of asking and answering questions, one of the children quickly realised they could always win the game by shouting out “42” as the answer to every question because the game only cared about the time to answer and not whether the answer was right. This leads us to a new question, what are the characteristics of questions and answers?

On their characteristics

An answer has to be the right answer or, more appropriately, an “exact enough” answer for the context. For example, when wanting to know the amount of paint to use for marking the center circle of a football field, a fast and accurate enough answer to “What is Pi” could be 22/7. That answer is accurate enough for the purpose, given that the exact answer for Pi is an infinitely long sequence of numbers that will never be completed. In other contexts, such as rocket trajectory, you might need Pi to 15 decimal places.

Accuracy is one characteristic that an answer must have. There are others. In 1854, during a cholera epidemic in London Soho, Dr. John Snow convinced the authorities to close down a well as a means to stop the epidemic. At the time, the common consensus was that cholera was spread via airborne transmission, but Dr. Snow convinced them of a different hypothesis and that intervention did have the expected result. How did he do it? Through elegantly explained evidence including a map of where the deaths occurred in Soho and a detailed analysis of the water sources used by both those that died and those that survived. That explanation convinced people of a new possibility of understanding cholera and prompted them to take action upon this.

An answer has to therefore be accurate and explainable. It also has to be representative to what is being discussed. For example, Alice presents an architectural diagram that Bill has created for a system. The architectural diagram might be an accurate representation of what Bill believes, it is explainable in the sense that Bill created it but it might not actually represent the system being discussed.

So, if an answer has to be accurate, explainable and representative then what should we expect from a question? The first thing we need is hidden in that famous question “If a tree falls in a forest and no one is around to hear it, does it make a sound?” Even if we answer the question, given that the tree cannot be observed (no-one is around to hear it) then no action can be taken. It is an unobservable phenomenon and the answer is merely a curiosity. If we intend to make a decision then the first thing a question must be is actionable i.e. It has to be something that once answered, a decision or action can be taken on. If a question doesn’t result in some form of choice and is merely a curiosity then it has no role in decision making.

For the next characteristic we return to Dr Snow. In 1849, five years before the cholera epidemic, he wrote: “Having rejected effluvia and the poisoning of the blood in the first instance, and being led to the conclusion that the disease is communicated by something that acts directly on the alimentary canal, the excretions of the sick at once suggest themselves as containing some material which, being accidentally swallowed, might attach itself to the mucous membrane of the small intestines, and there multiply itself by the appropriation of surrounding matter, in virtue of molecular changes going on within it, or capable of going on, as soon as it is placed in congenial circumstances”. Dr. Snow had formulated a specific hypothesis after years of exploration into disease. The outbreak offered the testing ground for this hypothesis.

This teaches us that any question or hypothesis has to be meaningfully specific to the domain we are investigating. The number of leeches available to treat blood poisoning is not relevant to the management of cholera. The number of smarties in a tube is an interesting question but not relevant to nuclear power safety.

Along with being actionable and specific, a question also needs to be timely. The answer to the question “How do I travel from the Reform Club, London to Suez, Egypt” varies with time. In 1872, it involved a train to Dover, cross channel ferry to Calais, train to Paris, another train to Turin, a further train to Brindisi and finally a steamer across the Mediterranean sea. Today, it would be a direct flight from London Heathrow. There is little point to asking a question if the answer is going to take so long that the underlying environment will have changed.

For decision making, an answer has to be accurate, explainable and representative and a question has to be actionable, specific and timely. We’ve added this to figure 14.

Now that we understand the nature and characteristics of questions and answers, we need to move to an example in order to determine their roles. For this, we will return to our data pipeline case study covered in the previous chapter.

Example: Mapping out the problem space of the data pipeline

In true physics fashion, we teach you classical physics and then as you progress to a university degree we tell you “oops, my bad” because physics doesn’t quite work that way. In the previous chapter we said our first question was “What does the system look like?”

We lied.

In fact, it was “Why are our services slow?”. This was then refined to “Is there useless data generated?” and then further into “How is the data traversing the pipeline?”

Whilst the first question was what the client wanted to know, the latter question was akin to “What does the system look like?” Let’s depict these succession of questions in another Wardley Map (figure 15). The blue “?” represents answers we do not know or have any information for. The red texts are the questions.

The answer to that latter question was given by the client in figure 3 — the erroneous network diagram with the missing external component. As no model of the network existed, the diagram was created by hand and based on crude beliefs. However, a network diagram should be derivable from the traffic which flows across the network. For all its faults, the network diagram led to a question of “How is one attribute traversing the data pipeline?” as shown in figure 16.

In order to answer that the client was forced to use highly manual processes taking person-weeks for a single variable in a system which had tens of thousands. This was the first question for which a tool was built, to visualise the flow of an attribute across the network but to repeat this process tens of thousands of times. The output was a highly modelled and automated visualisation of flow, essentially transforming the answer into a commodity as depicted in figure 17.

The moving of an answer from left to right on the map through the creation of a small tool to turn a manual process into a modelled, automated and highly industrialised one, captures the key dynamic of Moldable Development. Once an answer is commoditized, it confers new opportunities to answer further questions along with industrialising higher order answers. In our case, we could use the answer to construct the answer to the higher level question of “how is data traversing the pipeline?” — as in figure 18.

This pattern can be repeated all the way up our chain of questions and answers until we reach an automated way of generating the answer to the top question: “Why are our services slow?”.

In practice, what happens is the industrialisation of answers enables new questions that we had not previously thought of. A similar effect can be seen in modern research lab automation. For example, once we had a complete overview of the pipeline, the team wanted to refine their question. It turned out that the pipeline supported multiple marketing campaigns, but some of those were less important or even not active anymore. So, instead of looking at the entire data space, they wanted to narrow it down to only the last major campaign. This led to a new question of “What data is used only in a specific campaign?” as in figure 19.

These new questions exist in an adjacent part of the map which was not previously explored. They are described as the adjacent unexplored. Many scientific discoveries have come from adjacent questions to that which was originally being explored. For example the discovery of penicillin stemmed from an experiment with an unexpected variation, and once Flemming saw the result he formulated a new observation that led to a new question and eventually to a new discovery. Of course, not all systems will lead to major scientific discoveries but the principle is the same and the faster we can explore the adjacent unexplored, the more likely it becomes that we will discover something useful.

It is worth noting that the new question is far more specific to the problem space. For example, “How is data traversing the pipeline?” can be asked of any system involving a pipeline, but “What data is used in a specific campaign?” only makes sense within a much smaller subset of systems. In general, the value of a question goes hand in hand with its specificity. This phenomenon is caused by increasing situational awareness of the problem space itself and is typically also seen in mapping where early maps will start with generic components and concepts but later on they become much more specific. James Joyce’s quote “In the particular is contained the universal” captures the idea that specificity in storytelling is paradoxically more relatable and universal than generic narratives. It suggests that while people may instinctively lean towards broad, generalised storytelling in an attempt to be widely understood, it is actually the precise, vivid details that resonate most deeply with audiences. Unfortunately, most people aren’t trained on specificity or even understand that it is a skill that must be trained.

On the role of questions and answers

At this point, it’s worth making a few observations.

Any problem can be broken down into a chain of questions and answers. Almost always, when starting with a new problem, we know some of the questions to ask but the problem space (the map) has not been determined. As a point of interest, the maps are normally far more branching than the examples given.

The faster we industrialize answers, the faster we get to industrialize higher order answers and create new questions. ttA is critical throughout and is a major constraint on our process of discovery. Though it’s not something that most organizations ever measure or attempt to.

In a chain of question and answers, many of the component questions we seek answers for are the same component questions that others need answers for. By focusing on small tools for specific questions, we can build up a library of tools to help us find answers. For example, our data pipeline investigation used existing tools for parsing and browsing JSON and XML files, and even an engine for defining dedicated parsers — such as for the internal DSL — and visualizing their output. If a toolkit is used, then once a tool is built to answer a question, it can be pre-built into the toolkit for further reuse. While reuse is great, the goal should be to amortize the cost of creating a custom tool on the first use.

The chain of questions and answers operates both ways: going top to bottom is seeking a more detailed answer; going bottom to top enables higher order questions. Often, when people first learn about Moldable Development, they tend to think that they need a special problem to start from. But Moldable Development can be used for all sorts of problems, if not every problem. Hence, you can start with it anywhere and you don’t need to use the approach across the entire system. It’s not unlike testing. It’s possible to do some tests for some part of the system, but that does not force you to do continuous and systematic testing for the entire system. Of course, testing across the entire system compounds the value of testing and hence we tend to do this. The same is true with Moldable Development.

In a fully adopted environment, we can address dozens of questions per developer per day. Perhaps surprisingly, we quickly learn that the technical part of building tools can be learnt rather easily and the bottleneck is asking questions i.e. ttA becomes so fast that ttQ becomes the constraint. In order to manage this constraint we need to introduce a new set of techniques that we discuss in a later chapter.

So, what is the role of questions and answers? Ultimately their role is to help us map out a problem space as per figure 18 until we have a good enough understanding (given any constraints such as time, money and resources) to make a decision on it. Moldable Development is an approach to speed up the flow between question and answers (i.e. ttA and ttQ) and hence enable us to explore a problem space more quickly. In the next chapter we get to flex those knowledge muscles more.

What did we learn?

Decision-making should follow a scientific method-like loop, involving hypothesis formation, exploration, assessment, and refinement. The speed of this loop is crucial for effective decision-making, yet traditional metrics often overlook the time taken to formulate and answer questions. Answers should be accurate, explainable, and representative, while questions need to be actionable, specific, and timely. These characteristics ensure that the decision-making process is both efficient and effective.

Moldable Development is a method for decision-making and is applicable to all software engineering problems, not just legacy environments. It focuses on improving the decision-making process by reducing the time to ask questions (ttQ) and time to answer (ttA). The core of this is the synthesis of micro tools tailored to specific problems. This approach can be more efficient than manual analysis, provided the tool creation cost is offset by the speed gained in analysis. This speeds up the process of finding answers, and more importantly, it allows for the exploration of new, previously unconsidered questions — the adjacent unexplored –- facilitating a more comprehensive mapping of the problem space.

Homework exercise: exploring a decision flow

Your task is to apply the process of questions and answers:

a) Choose a software system you’re familiar with e.g., a project you work on or use frequently.

b) Identify a question about the system. This question can be generic and applicable to many other systems e.g. “Why is this system slow?”

c) Create a map similar to Figure 15 in the chapter, showing:
i) The initial high-level question in red and the answer in blue. Use “?” symbols for unknown answers.
ii) Add at least three follow-up questions that you believe might help provide the answer you’re seeking in the first place. Try to make these questions ever more specific. Write the questions in red onto the map.
iii) Add blue “?” symbols for unknown answers to those questions unless the answer is known. If it is known then position it on the horizontal axis based on how the answer was obtained: if it was manually created by inspecting only some parts of the system, position it to the left; if it was generated out of the system, position it to the right.

d) For one of the lower-level answers that you positioned on the left, describe:
i) The current manual process of creating the answer.
ii) A potential tool or automation that could be created to answer it more efficiently.

e) Explain how the creation of this tool might enable answering higher-level questions or generate new questions you hadn’t considered before.

f) Reflect on the exercise by answering these questions:
i) How did breaking down the problem into smaller questions help you understand the system better?
ii) What challenges did you face in identifying specific, actionable, and timely questions?
iii) How might the creation of small, focused tools improve the time to Answer (ttA) for your chosen system?

Remember to consider the characteristics of good questions (actionable, specific, timely) and good answers (accurate, explainable, representative) as discussed in the chapter.

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding software engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Tudor Girba and Simon Wardley

Why talk about decision making?

Developers alone spend more than half of their time trying to figure out systems enough to determine what to do next. They might explore the impact of a change, the root cause of a bug, or how to migrate a component. The further we go away from the code editing and committing, the more it becomes about decisions such as architectural or product choices.

As a whole, software engineering can be viewed as primarily a decision making activity about a continuously changing system and surrounding environment. But what does that decision making process look like? This is rarely a question we ask ourselves.

Instead of questioning the process of decision making, we tend to focus on the outcomes of the decision made i.e. could we have made a “better” decision that is more aligned with the business (the outcome) rather than could the decision have been made in a “better” way (the process)? Ideally, we need to do both but we need to start with the process first.

In general, the rational way to make engineering decisions can be described through the following steps :

1. We assess the problem
2. To assess the problem, we need to explore the systems we are looking at.
3. To explore the system, we need to have a conversation with either a person or the system.
4. Having a conversation requires information that we can share or can be shared with us.
5. Information does not just appear but it must be synthesized in some way.
6. To synthesize information, we need some form of development experience to interact with the systems.

These steps are shown in figure 1.

The steps have been drawn in a Wardley Map. These maps provide a visualization of the problem space and are based upon a systematic notation (they also make Simon happy). We use a map because trying to explain the characteristics and consequences of decision making in software is an incredibly hard thing to do. We know this because we have tried to communicate these ideas both conceptually and through technology for 15 years and those explainations always fell short. The problem is the limitation of narrative. Maps allow us to present complicated and multi-dimensional information in a succint form.

The basics of a Wardley Map include a chain of connected components (known as a chain of needs as each component needs the next) with each component positioned according to how evolved it is on an x-axis. The x-axis always has four sections denoted with a label representing different stages of evolution of capital. That capital may include activities or practices or data or knowledge. For our maps, since we are using multiple forms of capital, we chose two labels for each section of the x-axis. These are concept/genesis, emerging/custom built, converging/product and accepted/commodity.

Written as red text, the labels of concept, emerging, converging and accepted describe how much agreement there exists on the process of information use. These correspond to the components and connections drawn in red on the upper part of the map. For example, exploration is mostly considered an emerging practice, for reasons we describe later.

The blue labels of genesis, custom built, product and commodity describe how industrialized the practices and tools for information extraction are (the connections drawn in blue). For example, there exist numerous competing products for development experience.

Many of the ideas in this book can feel as abstract in description as they are powerful in practice. Hence, to help the reader overcome this, we explain each of these steps using examples from our experience, in a question and answer format. Let us start with the question of “How do we go about making decisions in practice today?”. To answer this, we start at the bottom of the map (figure 1) and work up.

We can only experience a system through tools that expose that system through some form of development experience. Current practice is to use monolithic tools that are designed for general use without catering for specific contextual needs. Hence, the software tools we use to investigate a hospital system are usually the same tools that we use to investigate an online gambling site. In the physical world, this is the equivalent of trying to build a formula 1 racing car with the same set of tools used for digging a deep shaft mine.

These standard software tools are so pervasive that you can probably describe them without even looking. They will have a navigation pane on the left, a basic search function on top and some general window which displays information such as the code of the item you’ve selected in the navigation pane. You might even be able to right click on an item in the navigation pane to get some properties. Even so, how you view and interact with the “data” of the system has been pre-defined, i.e. the tool constrains how you view the digital world.

Using these tools, we manually inspect the system in order to synthesize the information we require. This is where we spend more than 50% of our entire development time including reading lots of code, investigating error traces or exploring log entries in order to find the information we think we need.

The information we find is then further consolidated manually into views of the system. For example, we might create an architecture diagram, a domain model or a network diagram. We often have to leave our monolithic tool that we used to synthesize the information in order to present it in a different format e.g. a powerpoint presentation of an architectural diagram that represents the code that we found with a code editor. Sometimes we don’t even use digital tools to create these views but instead rely on whiteboards, paper or even post-it notes.

Occasionally automated tools are used, such as a graph of events over time, but again they are typically generic and highly constraining with pre-defined views. You will have experienced this if you’ve ever used one of these tools and found that the graph isn’t quite what you needed but had no way of changing it within the tool, often resorting to trying to export the data into a new tool such as Excel in order to create a “better” graph.

The views we created are considered to represent the data about the system and our conversation is based upon those views such as the number of events over time in the graph and possibly connections to our network diagram. But ask yourself, how representative are those views? How many architecture diagrams have you seen that are actually wrong or have missing components that are discovered later or have changed since the time the architecture diagram was created? Hand drawn diagrams about a system are not unlike paintings that are used as a way to document history. They document the perspectives of the authors at that time more than the system itself. We call this “data centric”, but because we have no real idea how representative of the system these views are, we’ve added quotes.

This situation is exacerbated in a world of continuous deployment where it has almost become fashionable to give up on architectural diagrams by either keeping them high level with just enough information or relying on a discipline of regular manual updates. An InfoQ article states “One of the biggest mistakes is to create detailed architectural diagrams for parts of the system with high volatility”. This is basically an admission of defeat, an acceptance that we should rely on that which is unreliable. Even when vendor tools are used, it is typically considered to be insufficient and “need to be complemented by manually modeled diagrams” because of the inability of the tool to capture important contextual information.

These conversations are part of how we explore the system. In software exploration, the process is commonly driven by a series of ad-hoc conversations and interactions. Contrast this with geographical exploration which follows a more structured approach. It begins with a blank canvas and systematically models the landscape through movement, careful observation, and the use of specialized tools like theodolites to create reliable maps to help us visualize the space.

Without reliable system visualizations, any exploration that depends on manual diagrams might only be partial and can hold hidden beliefs. This leads to assessments that are predominately based upon whether we believe what we are being told by the views created and hence our gut feel.

In summary, the current practice can be described as gut feel assessment based upon ad-hoc exploration using what are perceived to be “data-centric” conversations built upon manual information that is synthesized through a process of manual inspection using a development experience that consists of monolithic tools. We’ve enhanced our Wardley map to reflect this current practice in figure 2.

When creating this new map, we’ve introduced the idea of pipelines and considered how evolved the components of today’s practice are. Pipelines represent a common meaning. Whilst there might be significant disagreement between software engineers over how exploration is achieved, there is general agreement (an accepted consensus) that their approaches are ad-hoc and there is little agreement over what a more structured method would look like. Hence, in the map the notion of ad-hoc exploration is considered more evolved than the notion of exploration itself.

Using this logic, each of the components from ad-hoc exploration to monolithic tools were added. Manual inspection was described as an emerging practice rather than an industrialised one because the rigidity of our monolithic tools often forces us into other highly manual processes such as reading code. There exists little consensus over what manual inspection means and topics such as reading code are rarely discussed. Whilst reading code does not scale with large systems containing millions of lines of code, it has the advantage that it can be adapted to any context and is hence used to circumvent the rigidity of today’s development experience. For example, when the tool doesn’t show connections between code objects, the software engineers are forced to read the code to find those connections for themselves.

While there are shortcomings to today’s approach, there are three takeaways worth noting:

1. Any decision about a system, regardless of whether technical or business focused, requires information from the system.
2. This information has to be synthesized somehow from the system, and the only way to do that is through a tool that provides a development experience.
3. Developers today write code for a fraction of their time. They spend most of their time reading because they want to understand what to do next. The largest single cost in software engineering is figuring the existing systems out which is something we do not optimize our work for.

How did we get here?

In a world of abundance of data, automation and systems we have somehow ended up with decision making processes that are more ad-hoc choices based upon manually created views. The cause of this situation appears to be the use of generic tools, which certainly suits the tool vendors to which we have handed over part of the process of understanding a system. We have accepted that building tools is hard — it takes money, it takes time — or at least, that is what tool vendors have always told us by highlighting their world class solutions and beloved tools that create lightning speed. “Software is a team sport” with the tool vendor as the referee, pitch and equipment provider.

If you want to pick a villain, a potential candidate would be Apple who took the concept of personal computing developed at Xerox Parc and encased it in a physical box and impenetrable apps with the Apple Macintosh. Another would be Microsoft which then sold its own concept of personal computing to the masses by removing the physical box — any x86 architecture would do. In both cases, the operating environment became more of a black box and what was sold was convenience of doing tasks rather than an understanding of what was happening. Despite the glossy marketing brochures, the vendors did not enable or empower understanding in people rather they infantilised them into tightly constrained spaces dependent upon the tools they sold.

Our natural inclination is to fight against this. Children start to build computational systems even within highly constrained environments like Minecraft or Roblox. If we wish to enhance these natural inclinations, we need to remove the constraints of tools, we need to provide more freedom. This is the central idea, expressed as four freedoms, behind the counter revolution to all of this control which is the open source movement.

Computers were supposed to augment human intellect, not diminish it in the pursuit of convenience. The path we have taken has had such a devastating effect that most professional software engineers don’t even conceive of building their own tools. Even within development forums, you commonly see examples of problem solving that reach the limit of the tool where the engineer gives up and suggests any further exploration is handed over to the tool vendor. Worse than this, many have almost replaced exploration and understanding with searching the web or glorified forums for answers. This is exemplified with endless infomercials on the “best answer to all your coding questions” — Stack Overflow.

Example: Trying to optimize a data pipeline in the traditional way

To explore these ideas further, we turn to a real world example. A large corporation wanted to optimize the performance of a central data pipeline by an order of magnitude. This was required from a business perspective. In their case it was the main marketing pipeline from which offers were sent to millions of customers, and they needed to be able to react much faster to changes in the market environment. The problem was visible all the way to the C-level and an initiative was started to reach the business goal of a quicker response to the market.

However, after a few years of effort, the data was still moving through the pipeline at the same speed as before. So, how can it be that all the effort made no difference in the end? People cared about the problem and they had spent many millions of dollars in pursuit of a solution.

To explain the environment, we will use their manually written high level architectural diagram (figure 3).

The pipeline consisted of an internal domain-specific language (DSL) based on Excel. Programmers would write elaborate queries in Excel spreadsheets and these were automatically converted to database transformations that were applied to the data which would end up in both a SQL database (Oracle) and a NoSQL database (Cassandra). Both of these databases were used as an input for a low code platform that also offered AI abilities and on which other programmers wrote scripts specific for various marketing campaigns.

The architecture was simple but no amount of engineering or reading stack overflow was helping to optimize it. Despite a lack of performance metrics, their investigation led them to believe that there might be too much data generated along the pipeline that was not used in the end.

Their best guess was that the problem was “dark data”, the equivalent of “dark matter” — lots of stuff we cannot see but has an impact. The teams working with the DSL did not have any visibility on what was actually being used. The teams working on the low code scripts could only see the data from the databases, but they did not know what transformations affect that data.

To make matters more complicated, the low code platform was indeed believed to be fulfilling its promise of helping people to create code faster, but it offered no support for either performance measurement or for tracing where a piece of data was used. Due to all these observations, they realized that because they were working in silos, they could not optimize the overall pipeline. They understood that they needed accurate data lineage to verify their beliefs but there were no tools for their specific combination of technologies or relevant stack overflow articles. In a system which contained tens of thousands of variables, they had been forced to spend person-weeks trying to manually see how just one variable was used through the pipeline.

In summary, the method of assessing the situation included a development experience consisting of monolithic tools which constrained what they were able to do for convenience. They synthesized information through manual inspection. For example, they attempted to create traces of how some variables were passed through the low code scripts by manually reading through the code. They consolidated information into manual views of the system. The high level architecture diagram in figure 3 was such an example. Their conversations were based upon those manual views and a belief they were representative of the system. These conversations led them to further beliefs that there “might be too much data generated along the pipeline that was not used in the end”. Their exploration of the system consisted of a series of ad-hoc conversations and interactions with the system. This method was unsystematic and mostly guided by the individual curiosity of the software developers and architects from different teams. They assessed and made their decisions based upon their beliefs in this exploration. The result was millions of dollars spent on investment with no discernible impact.

A new path: Moldable Development

In a world of rigid tools, manual inspection, ad-hoc exploration and gut feel, a group of researchers (that later found a home at feenk) asked whether another path for decision making was possible? This required challenging the way we make decisions, the way we think about tools and the way we interact with systems. Step by step they developed a new path known as Moldable Development. To understand this, let us once again run through the steps of decision making in reverse order.

We abandon the monolithic tools which normally define our development experience. Instead we choose to compose the experience out of micro tools, each of which was built to answer a specific question for a specific context. Using our micro tools we synthesize the information we require from specific coding i.e. a direct, unmediated feed of the system. This synthesization requires that the cost of creating the tool is cheap enough that it has no material impact e.g. built in minutes or hours rather than days, weeks or months. It is critical that the process of manufacturing micro tools should be as industrialised as possible, ideally through the use of a toolkit. Similar dynamics happened in the space of testing. When the cost of creating a single test became so small it did not matter, automatic and industrialized testing became adopted at scale. We can regard a test as a form of micro tool that takes the execution of a system and transforms it into a red/yellow/green signal. The same idea can be extended to any tool, including visualizations or queries.

The information is then presented in generated views. These views directly feed from the information which we synthesize from our system through micro tools. All the information, wherever it is presented, can be considered both live and directly extracted from the system. It has not been manually created. It is not unlike a photograph replacing a painting as a means of documentation.

The generated views in effect model the system. Our conversation changes from a “data centric” to a model centric approach that is generated live from the system and can be directly interrogated.

Our exploration changes from ad-hoc to a more formalized approach that we call Moldable Development. We use the model of the system to ask new questions as we explore more of the system. To answer these additional questions, we constantly create new micro tools synthesizing new information which we then codify into our model of the system. The more we explore, the more complete this model of the system becomes. This mimics geographical exploration, as we follow a more structured approach starting with a blank page and then systematically model the landscape through observation using micro tools as our own theodolites.

The assessment becomes hypothesis driven and we make our decisions based upon exploring the model which is created directly from the system and contains no necessity for belief. We do not need to trust what we are being told as we can directly interrogate the model which itself is generated live from the system.

The new practice has been added to our Wardley map in Figure 4.

Example: Optimizing the data pipeline following the new path

To explore these ideas further, we return to our real world example. A large corporation wanted to optimize the performance of a central data pipeline by an order of magnitude. After years of effort, the data was moving through the pipeline at the same speed as before. This led to a situation where the business and engineering were in conflict and the pressure was mounting on engineering to find a solution.

When you find yourself spending effort that seems to have no effect on reality, it is plausible that your model of the system differs significantly from the actual system. In such situations, you want to first improve your ability to see the actual system.

Our first hypothesis was that maybe their lack of results was because of an incomplete understanding of the system. Hence, our first question was “What does the system look like?”

In our case, we wanted a tool to give us an accurate perspective of the system. But given the combination of technologies and the specificity of the business case (such as, what data points are being used in a specific marketing campaign), there exists no such tool out of the box. We had to create it.

While building the model, we realized that the output from the first system did not match the data from the databases. This small observation made the team discover a whole external 3rd party system they were not aware of! This is shown in figure 5.

This confirmed our idea that their visibility into the system was partial and inaccurate, and past decisions had been based upon this perception. They needed an accurate overview. Since an overview is an aggregation of details, to create an accurate representation, the tools had to start from the smallest scope such as how a single property traverses the pipeline.

Figure 6 shows the generated view from one of the tools. A single input (the blue dots in the Internal system) impacted multiple other points of data (the red dots in the Internal system). These points of data then became inputs (blue dots) into the External system and impacted further data (the red dots in the External). Finally these points of data became inputs (blue dots) into their low code system.

Without an understanding of this flow, any attempt to optimize the entire system was likely to be ineffective and fraught with failure. This is exactly what they were experiencing.

Based on the property-level data lineage, we constructed tools to generate multiple pipeline overviews. One such overview quantified the amount of data produced along the pipeline and can be seen in Figure 7.

For each system, the visualization depicts groups of properties that are used or not used downstream in other systems. The red parts correspond to properties created but not used later on. In essence this was “dark data” i.e. data that had no interaction with the rest of the system but added “mass” to the system.

These visualizations showed the team that they did indeed generate data they were not using, but more importantly, it made them realize that the situation is not hopeless and that it is possible to get an overview over the system, even if it was made out of heterogeneous and old components. Based on the input, they decided to redo the pipeline by consolidating the information to eliminate the need for unnecessary transformations.

Taking a step back, we started with an initial question of “Why are our services slow?” which then led to a question of “Is there useless data generated?” which required a question of “How is data traversing the pipeline?”. To answer these questions, we constructed a development experience consisting of micro tools that synthesized live information across our data pipeline. The conversations were based upon the live information, such as the generated views from figures 6 & 7, rather than any erroneous belief as originally existed in figure 3.

This case shows that it is viable to have a systematic approach for exploration that consists of building a model of the system by asking questions and answering those questions through tools that use the system itself. In total, 54 distinct micro tools (i.e. at least 54 answers to questions) were needed to properly answer the top level question of “Is there useless data generated?”. This required only two person-months of effort. Compared with the previous attempts which were measured in hundred person years and many millions of dollars of direct costs, this is at least 600x factor of improvement.

How relevant is the example to you?

It’s tempting to look at the example and wonder “How could they miss that 3rd party system?” or state that “We wouldn’t do something like that”. The team of engineers at the client were extremely capable including a global leader in the service integrator industry. The engineers were all highly qualified with degrees and industry qualifications. However, this system was large with multiple smaller pieces that had been built over many years. People had retired, and component systems had been forgotten about. The team was also siloed into different groups — an internal system team, a database team, a low code team etc and no overall picture existed.

Whilst this might appear as an extreme, we would argue based upon experience that highly skilled teams, often operating in silos, trying to manage significant legacy environments with a less than adequate model of the system is commonplace, if not the norm. It does not surprise us that 83% of data and legacy migration projects are reported to either fail or exceed their budgets or schedules. What surprises us is that 17% succeed. However, even if that 17% is correct then we would expect that the results could be achieved at a much faster speed and lower cost.

If you have any form of legacy estate, then we suspect the example is relevant to you. To test, simply ask your engineering team to provide you with a model of the system. If the result is a manual hand written or powerpoint diagram (such as figure 3) then ask for a comparison to the live version.

Why not just use an AI to solve this?

When most CIOs in 2025 talk about AI, they usually mean a specific set of transformer architectures such as LLMs (large language models) and LMMs (large multi-modal models) that are commonly found in code co-pilots. Whilst there is nothing wrong with using such models to assist in the effort (and the authors both do), as of December 2024 marketing research campaigns claim that copilots can help developers write up to x1.55 (55%) faster. It should be remembered that these are marketing research and are often challenged by independent research which claims figures as low as x1.05 (5%).

Typically, using a model-centric approach as described here, we see improvements of orders of magnitude. For example, the case above showed 600x (60,000%) improvement. It is often difficult to quantify the difference because it is rare to have a baseline to compare against. Even where data exists, we are often not comparing like for like i.e. you have one group which has failed to find an answer or given up versus one where the answer is found. Furthermore, larger opportunity costs to the business due to delays are rarely factored into the equation. Despite the huge changes, it’s not possible to discount the influence of luck, the skill of the engineers or the bias of the authors due to the small sample sizes.

As a result, we are not in a position to quantify the impact of a more moldable approach and instead must rely on our experience of orders of magnitude. However, the two approaches — use of AI models and Moldable Development — are not in opposition and can complement each other. We discuss how, later in this book.

Throughout this chapter we have discussed the role of questions and answers. In the next part of our journey we explore this in more detail and why it matters.

What did we learn?

Software engineering can be seen as primarily a decision-making activity about continuously changing systems. The process involves assessing problems, exploring systems, having conversations, synthesizing information, and utilizing development experience.

Current practices often rely on gut feelings and ad-hoc exploration using manual, data-centric approaches. Furthermore, traditional software development tools are often monolithic and constraining, leading to manual inspection.

Moldable Development offers an alternative that relies on generating live views of the system. These are obtained through contextual micro tools that are created specifically for each problem.

We provided a real-world example of how Moldable Development was applied to optimize a data pipeline that had previously resisted improvement. In the case study, it took approximately 2 person-months to unlock a problem that had previously consumed hundreds of person-years without success. While the exact quantification of improvement is challenging, experience suggests that Moldable Development can lead to orders of magnitude improvement in problem-solving in software engineering.

Homework exercise: level setting

Gather a group of your software engineering colleagues and discuss the following questions:

1. Do we agree that we spend more than 50% of the engineering time on trying to learn about the system by reading through various artifacts?
2. When was the last time we talked about how we read code? Specifically, about how we do the reading, not about the code we read.
3. Do we have models of our system that allow us to investigate performance issues programmatically? What about security concerns? Or business domain events?
4. How do we obtain our architecture diagrams? Are they manually created, or do we generate them out of the system’s sources? How confident are we that the diagrams we use represent the system accurately? How would we verify this?

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding software engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

By Tudor Girba and Simon Wardley

What is this book about?

This book addresses a long-standing crisis in software engineering, first identified at a 1968 NATO Conference. The crisis, as described by Edsger Dijkstra in his Turing Award lecture, stems from the rapid increase in computer power:

“The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! […] as long as there were no machines, programming was no problem at all; […] and now we have gigantic computers, programming has become an equally gigantic problem.”

Since Dijkstra’s observation, computing power has continued to increase exponentially. As a result we’ve written ever more code, ever more abstractions and the total body of software has grown continuously year over year.

While growth itself isn’t inherently problematic, this ongoing crisis manifests when we fail to deal with the issue of legacy as we end up surrounded by old systems that we struggle to retire, to maintain or to take apart and refurbish for new purposes. From this perspective, the software industry behaves no differently from the plastic industry: we build faster and faster but are unable to recycle old systems. This is a systemic problem affecting every company.

Many developers initially shy away from legacy code, but those who learn to appreciate its intricacies and historical value can often find unexpected beauty in modernizing these past masterpieces. Unfortunately, this beauty is hidden in millions of lines of code making it difficult to understand. Hence, software legacy has become synonymous with being unpleasant and almost inhumane. Software engineering is the only domain in which “legacy” has a negative connotation even though it is where we spend the vast majority of our working time. Addressing the problem of legacy requires a humane systematic solution.

This book proposes that the best way to tackle this challenge is by reframing software engineering as a decision-making process about systems that are too large for us to fully grasp. Furthermore, these systems exist in ever-changing environments. To effectively refurbish such systems we must be able to take them apart, but before we can do this we first need to understand the parts. This process is highly manual, mostly consisting of reading. Trying to understand large systems in this way is an overwhelming task.

Consider the Apollo moon landing. The total number of lines of code were over 130,000. But, what does that mean to read? Fortunately, Margaret Hamilton, director of the software engineering effort and the one credited with coining the term “software engineering”, was photographed standing next to the printed source.

Now consider that the number of lines of code in a core banking system can be into tens of millions i.e. 100x what was used to land Apollo on the moon. Just reading all that code in minute detail is a daunting task requiring years of work. And that’s just reading the code — assuming the system remains static — and then we have to understand it in order to make a decision.

The perspective of software engineering as a decision making activity matches what people do implicitly. However, most people don’t think explicitly about the process of decision making and whether it can be improved. At the GOTO 2024 conference, a 500 developer audience was asked whether they agree with the statement that they spend most of their time reading through textual artifacts (e.g. code, logfiles) to figure systems out. Most did. But when asked when was the last time they talked about how they do the reading, it turned out that this conversation rarely occurs, and when it does, the reading part is never challenged. If we do not talk about the process, it’s not explicit. If it’s not explicit, it does not get optimized. Our lack of reasoning about decision making itself has led to, for example, a belief in architecture diagrams that don’t match reality, a failure to measure what really matters, and an atrophied ability to ask questions.

So what’s the solution? Are we just talking about becoming faster readers? No. You can’t speed up reading by a factor of 500x but you can improve decision making by that factor. This requires a significant change of perspective that starts with understanding why we read code and if we need to read it at all. This is the opportunity we put forward in this book through a technique known as Moldable Development that can be applied in a multitude of software engineering situations.

Who is this book intended for?

The lessons in this book are widely applicable to decision makers — whether software engineers, product managers, CTOs or CEOs — who need to ask and answer questions about systems. However, this book was mainly written for those poor folk dealing with legacy systems. We say “poor” because in today’s world of digital technology and AI, there is little love given to legacy environments or to those who have to manage the migration away from them. This book seeks to address that imbalance.

And this imbalance is all around us. According to Forbes, more than 60% of corporations rely on legacy systems to “power customer-facing applications”. Maintaining legacy systems is a $1.14 trillion market. For example, according to a Microfocus survey (2022) the world depends upon 800 billion lines of legacy COBOL code. A McKinsey survey revealed that the technical debt due to legacy systems accounts for 20% to 40% of the entire technology estate. In 2018, the US Federal Government spent 78% of its IT budget on operations and maintenance i.e. keeping the lights running. Acropolium (a specialist in legacy migration) estimates that the average migration project takes 4 years. At the same time, the Advanced 2020 Mainframe Modernization Barometer states that 74% of organizations have failed to complete their migration projects. Additionally, 28% claim that their legacy systems are between 20 to 30 years old.

Most of the statements above are created by surveys and hence are statements of belief. None of these reports should be considered rigorous by any measure as we do not have a means of interrogating the true legacy estate. They are a painting of the landscape, not the landscape itself. Nevertheless, as a whole they do indicate a global chorus singing a common refrain: legacy is expensive, hard and it needs to change.

In a world where nearly 90% of businesses are expecting Generative AI integration to free up 75% of their workload, is it any wonder that many CIOs are hoping that AI will solve the legacy problem for them?

In this book, we aim to change this picture and to also explain in the process why simply hoping for AI is not a solution. The major cause of all our headaches is the method by which we ask and answer questions about our systems.

How did the book get written?

The book is an overview of research and development that spans some 15 years involving colleagues at feenk and elsewhere. While Simon and Tudor are behind the writing and mapping, and even if Tudor was the one that coined the initial idea, the result of what we call Moldable Development today is very much the work of a collective. That research is still ongoing due to the size of the problem space, and so this book should not be seen as a simple recipe but the beginning of a conversation that we, as an industry, should have. In this spirit, the book is provided as Creative Commons Share Alike as we hope others will not only learn our lessons but build upon them.

Why rewilding?

Rewilding is a progressive approach to conservation that aims to restore ecosystems to a more natural state by allowing nature to take care of itself. An example of this is the reintroduction of gray wolves to Yellowstone in 1995 having previously been eradicated in the 1920s leading to significant ecological imbalances. The effect of rewilding was to change the grazing patterns of the elk population allowing for vegetation to recover and increasing biodiversity as the vegetation supported a wide range of diverse plant and animal species. This included a resurgence of beavers which caused river system changes through dam building resulting in changed hydrology which reduced runoff, recharged groundwater tables, improved water quality and led to a growth in fish populations. An apparently small change led to a new sustainable equilibrium that had far reaching consequences.

Of course, not all interventions in an environment have positive effects. For example, the introduction of the cane toad in Australia backfired dramatically due to the explosive growth of toads. It has reached the point at which it is now considered a threat to the environment. For these reasons, successful rewilding efforts require a lot of upfront analysis and continued observation. The overall goal is that changes to the environment should enable a new sustainable equilibrium.

In the world of software, we intend to reintroduce our wolf — the conversation on decision making — and our goal is to make this engineering field both humane and habitable. After 15 years of research and practice, we understand that far reaching changes are possible. We also understand that once a change is made it takes time for a new equilibrium to form. We would ask that if you decide to pursue our pathway that you allow time for changes to settle.

—

Rewilding Software Engineering

Chapter 1: Introduction
Chapter 2: How we make decisions
Chapter 3: Questions and answers
Chapter 4: Flexing those thinking muscles
Chapter 5: Different folks for different strokes
Chapter 6: Myths we tell ourselves

Rewilding software engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

A good enough pathway to reasonable decisions.

By @girba (Feenk & Glamorous Toolkit) and @swardley (Wardley Maps)
December 2024.

Chapter 1: Introduction

Whilst the lessons in this book are widely applicable to decision makers and software engineers who need to ask and answer questions, this book was mainly written for those poor folk dealing with legacy systems. We say “poor” because in today’s world of digital technology and AI, there is little love given to legacy environments nor those who have to manage the migration away from them. This book seeks to address that imbalance.

Why legacy? According to Forbes, more than 60% of corporations rely on legacy systems to “power customer-facing applications”. Mechanical Orchard claims that maintaining legacy systems is a $1.14 trillion market, and a Microfocus survey (2022) argues that the world depends upon 800 billion lines of legacy COBOL code. McKinsey highlights that technical debt due to legacy accounts for between 20% to 40% of the entire technology estate and in 2018, the US Federal Government spent 78% of its IT budget on operations and maintenance i.e. keeping the lights running. Acropolium (a self described specialist in legacy migration) estimates the average time for a typical migration project is 4 years and according to the 2020 Mainframe Modernization Business Barometer by Advanced then 74% of organizations have failed to complete their migration projects with over 28% of organisations claiming their legacy systems are between 20 to 30 years old.

In a world where nearly 90% of businesses are expecting GenAI integration to free up 75% of their workload, is it any wonder that many CIOs are hoping that AI will solve this legacy problem for them.

Most of the statements above are created by surveys as we do not have a means of interrogating the true legacy estate. Hence the quotations are statements of belief, none of these reports should be considered rigorous by any measure. They are a painting of the landscape, not the landscape itself. That said, as a whole they do indicate a global chorus singing a common refrain: legacy is expensive, hard and it needs to change.

In this book, we argue that simply hoping that AI will solve this problem is not the solution and that a major cause of all our headaches is the method by which we ask and answer questions itself. To examine this, we need to start our journey by first looking at how we make decisions.

How we make decisions

Why should we talk about decision making in software engineering? Developers spend more than half of their time trying to figure out systems enough to determine what to do next. They might explore the impact of a change, the root cause of a bug or they want to know how to migrate a component in order to make the decision to do so or to fix this or to make that change.

The further away we go from the code editing and committing, it’s only about decision making, such as the architectural choices that we make on a whiteboard. Software engineering can be viewed as primarily a decision making activity about a continuously changing system and surrounding environment. But what does that decision making process look like? This is rarely a question we ask ourselves.

Instead of questioning the process of decision making, we tend to focus on the outcomes of the decision made i.e. could we have made a “better” decision that is more aligned with the business (the outcome) rather than could the decision have been made in a “better” way (the process). Ideally, we need to do both but to begin with, we will concentrate on the process itself.

In general, the steps for how we go about making decisions can be described as:

1. We assess the problem
2. To assess the problem, we need to explore the systems we are looking at.
3. To explore the system, we need to have a conversation with either a person or the system.
4. Having a conversation requires information that we can share or can be shared with us.
5. Information does not just appear but it must be synthesized in some way.
6. To synthesize information, we need some form of development experience to interact with the systems.

These steps are shown in figure 1.

The steps have been drawn in a Wardley map using the axis of concept/genesis, emerging/custom built, converging/product and accepted/commodity. The terms concept, emerging, converging and accepted have been used to describe how much agreement there exists on the process of information use is (the red line). For example, exploration is mostly considered an emerging practice as there is little consistency between engineers over how they explore beyond it being “ad-hoc”. The terms genesis, custom built, product and commodity describe how industrialized the practices and tools for information generation are (the blue line). For example, there exist numerous competing products for development experience.

Many of the ideas in this book can feel as abstract in description as they are as powerful in practice. Hence, to help the reader overcome this, we will explain each of these steps using examples grounded in practice, in a question and answer format.

Let us start with the question of “How do we go about making decisions in practice today?”. To answer this, let us run through the steps in figure 1 in reverse order.

a) We can only experience a system through tools that expose that system through some form of experience — a development experience. Today, our development experience consists of monolithic tools that are not designed for our specific context but a general context. For example, the tools we use to investigate a hospital system are usually the same generic tools that we use to investigate an online gambling site. So pervasive are these standard tools that you can probably describe them for yourself without even looking. They will have a navigation pane on the left, a basic search function on top and some general window which displays information such as the code of the object you’ve selected in the navigation pane. You might even be able to right click on an object in the navigation pane to get some properties. How you view and interact with the “data” has been pre-defined, i.e. the tool constrains how you view the digital world. In the physical world, this is the equivalent of trying to build a formula 1 racing car with the same set of tools used for digging a deep shaft mine.

b) Using these tools, we manually inspect the system in order to synthesize the information we require. Examples include reading lots of code, investigating error traces or exploring log entries in order to find the information we think we need. On average, we spend more than 50% of our entire development time on reading code.

c) The information is then consolidated into views of the system that are manually pieced together. For example, we may use the information we have collected to create an architecture diagram, a domain model or a network diagram. We often have to leave our monolithic tool that we used to synthesize the information in order to present it in a different format. For example, using an editor to examine code and then a slideware tool to present an architectural diagram that represents the code that we found with the editor. Sometimes we don’t even use digital tools to create these views but instead rely on whiteboards, paper or even post-it notes. Occasionally automated tools are used, such as a graph of events over time, but again they are typically generic and highly constraining with pre-defined views. You will have experienced this if you’ve ever used one of these tools and found that the graph isn’t quite what you needed but had no way of changing it within the tool, often resorting to trying to export the data into a new tool such as Excel in order to create a “better” graph.

d) The views we create are considered to represent the data about the system and our conversation is based upon those views such as the number of events over time in the graph and possibly connections to our network diagram. We call this a “data centric” approach even when those views are little more than pen on paper or manually created and we have no real idea how representative of the system they are. Ask yourself, how many architecture diagrams have you seen that are actually wrong or have missing components that are discovered later or have changed since the time the architecture diagram was created? A hand drawn diagram about a system is not unlike a painting that is used as a way to document history. It documents the perspective of the author at that time more than the system itself.

This situation is exacerbated in a world of continuous deployment where it has almost become fashionable to give up on architectural diagrams by keeping them “high level” with “just enough” information or to force a process of regular manual updates and in some cases to rely on vendor tools that come with their own constraints and pre-defined views in order to help automate the generation. An example of this approach is summarised in an InfoQ article saying that “One of the biggest mistakes is to create detailed architectural diagrams for parts of the system with high volatility”. Even when vendor tools are used, it is often considered that “the automatic creation of diagrams is not a sufficient option. It needs to be complemented by manually modeled diagrams”. Architectural diagrams are seen as a craft.

e) These conversations are part of how we explore the system. In software exploration, the process is typically driven by a series of ad-hoc conversations and interactions with the system. This method is unsystematic, guided by individual curiosity, and often lacks a comprehensive strategy. In contrast, geographical exploration follows a more structured approach. It begins with a blank map and systematically models the landscape through movement, careful observation, and the use of specialized tools like theodolites. This results in a comprehensive, model-centric understanding of the area being explored e.g. a map.

f) We assess and make our decisions based upon this exploration. The result of the exploration might only partially describe the system and can hold hidden beliefs. Our decision is predominately based upon whether we believe what we are being told by the software engineers and the views created i.e. our gut feel whether this is right.

Our current practice can be summarised as gut feel assessment based upon ad-hoc exploration using what are perceived to be “data-centric” conversations but are built upon manual information that is synthesized through a process of manual inspection using a development experience that consists of monolithic tools. We quote the term “data-centric” because gut feel, ad-hoc, manual information and manual inspection seem to be orthogonal to the idea of being data centric.

We’ve added the practice today to our Wardley map shown in figure 1 to create the new map shown in figure 2.

When creating this new map, we’ve introduced the idea of pipelines and considered how evolved the components of today’s practice are. Pipelines represent a common meaning i.e. ad-hoc is a type of exploration but there are other ways of achieving exploration such as the more structured approach of geography. Whilst there might be significant disagreement between software engineers over how exploration is achieved, there is general agreement (an accepted consensus) that their approaches are ad-hoc and there is little agreement over what a more structured method would look like. Hence, in the map the notion of ad-hoc exploration is considered more evolved than the notion of exploration itself.

Using this logic, each of the components from ad-hoc exploration to monolithic tools were added. Manual inspection was described as an emerging practice rather than an industrialised one because the rigidity of our monolithic tools often forces us into other highly manual processes such as reading code. There exists little consensus over what manual inspection means and topics such as reading code are rarely discussed. Whilst reading code does not scale with large systems containing millions of lines of code, it has the advantage that it can be adapted to any context and is hence used to circumvent the rigidity of today’s development experience. For example, when the tool doesn’t show connections between code objects, the software engineers are forced to read the code to find out the connections for themself.

While there are shortcomings to today’s approach, there are three takeaways worth noting:

1. Any decision about a system, regardless of whether technical or business focused, requires information from the system.
2. This information has to be synthesized somehow from the system, and the only way to do that is through a tool that provides an experience. A development experience. Thus, the development experience is paramount for any meaningful system development.
3. Developers today write code for a fraction of their time. They spend most of their time reading because they want to understand what to do next. The largest single cost in software engineering is figuring the existing systems out which is something we do not optimize our work for.

How did we get here?

In a world of abundance of data, automation and systems we have somehow ended up with decision making processes that are more ad-hoc choices based upon manually created views. The cause of this situation appears to be the use of generic tools, which certainly suits the tool vendors to which we have handed over part of the process of understanding a system. We have accepted that building tools is hard — it takes money, it takes time — or at least, that is what tool vendors have always told us by highlighting their world class solutions and beloved tools that create lightning speed. “Software is a team sport” with the tool vendor as the referee, pitch and equipment provider.

If you want to pick a villain, a potential candidate would be Apple who took the concept of personal computing developed at Xerox Parc and encased it in a physical box and impenetrable apps with the Apple Macintosh. Another would be Microsoft which then sold its own concept of personal computing to the masses by removing the physical box — any x86 architecture would do. In both cases, the operating environment became more of a black box and what was sold was convenience of doing tasks rather than an understanding of what was happening. Despite the glossy marketing brochures, the vendors did not enable or empower understanding in people rather they infantilised them into tightly constrained spaces dependent upon the tools they sold.

Our natural inclination is to fight against this. Children start to build computational systems even within highly constrained environments like Minecraft or Roblox. If we wish to enhance these natural inclinations, we need to remove the constraints of tools, we need to provide more freedom. This is the central idea, expressed as four freedoms, behind the counter revolution to all of this control which is the open source movement.

Computers were supposed to augment human intellect, not diminish it in the pursuit of convenience. The path we have taken has had such a devastating effect that most professional software engineers don’t even conceive of building their own tools. Even within development forums, you commonly see examples of problem solving that reach the limit of the tool where the engineer gives up and suggests any further exploration is handed over to the tool vendor. Worse than this, many have almost replaced exploration and understanding with searching the web or glorified forums for answers. This is exemplified with endless infomercials on the “best answer to all your coding questions” — Stack Overflow.

Example: Trying to optimize a data pipeline in the traditional way

To explore these ideas further, we return to a real world example. A large corporation wanted to optimize the performance of a central data pipeline by an order of magnitude. This was wanted from a business perspective. In their case it was the main marketing pipeline from which offers were sent to millions of customers, and they wanted to be able to react much faster to changes in the market environment. The problem was visible all the way to the C-level and an initiative was started to reach the business goal of a quicker response to the market.

However, after a few years of effort, the data was still moving through the pipeline at the same speed as before. So, how can it be that all the effort made no difference in the end? People cared about the problem and they had spent many millions of dollars in pursuit of a solution.

To explain the environment, we will use their manually written high level architectural diagram (figure 3).

The pipeline consisted of an internal domain-specific language (DSL) based on Excel. Programmers would write elaborate queries in Excel spreadsheets and these were automatically converted to database transformations that were applied to the data which would end up in both a SQL database (Oracle) and a NoSQL database (Cassandra). Both of these databases were used as an input for a low code platform that also offered AI abilities and on which other programmers wrote scripts specific for various marketing campaigns.

The architecture was simple but no amount of engineering or reading stack overflow was helping to optimize it, or so they believed — they did not have any performance metrics. Despite this, their investigation led them to believe that there might be too much data generated along the pipeline that was not used in the end.

Their best guess was that the problem was “dark data”, the equivalent of “dark matter” — lots of stuff, we cannot see but has an impact. The teams working with the DSL did not have any visibility on what was actually being used. The teams working on the low code scripts could only see the data from the databases, but they did not know what transformations affect that data.

To make matters more complicated, the low code platform was indeed believed to be fulfilling its promise of helping people to create code faster, but it offered no support for either performance measurement or for tracing where a piece of data was used. Due to all these observations, they realized that because they were working in silos, they could not optimize the overall pipeline. They understood that they needed accurate data lineage to verify their beliefs but there was no tools for this or stack overflow article. In a system which contained tens of thousands of variables, they had been forced to spend person-weeks trying to manually see how just one variable was used through the pipeline.

In summary, the method of assessing the situation included:

a) A development experience consisting of monolithic tools which constrained what they were able to do for convenience.

b) They synthesized information through manual inspection. For example, they attempted to create traces of how some variables were passed through the low code scripts by manually reading through the code.

c) They consolidated information into manual views of the system. The high level architecture diagram in figure 3 was such an example.

d) Their conversations were based upon those manual views and a belief they were representative of the system. These conversations led them to further beliefs that there “might be too much data generated along the pipeline that was not used in the end”.

e) Their exploration of the system consisted of a series of ad-hoc conversations and interactions with the system. This method was unsystematic and mostly guided by the individual curiosity of the software developers and architects from different teams.

f) They assessed and made their decisions based upon their beliefs in this exploration. The result was millions of dollars spent on investment with no discernible impact.

This led to a situation where the business and engineering were in conflict and the pressure was mounting on engineering to find a solution. Engineering had responded by putting more resources into investigating the problem using the same path which had not yet yielded any results. The problem was considered by many to be insurmountable and they had practically given up.

A new path: Moldable Development

In a world of rigid tools, manual inspection, ad-hoc exploration and gut feel, a group of researchers (that later found a home at feenk) asked whether another path for decision making was possible? This required challenging the way we make decisions, the way we think about tools and the way we interact with systems. Step by step, over a period of 15 years, they developed a new path known as “Moldable Development”. To understand this new path, let us once again run through our steps to decision making in reverse order.

a) We abandon the monolithic tools which normally define our development experience. Instead we choose to compose the experience out of micro tools, each of which was built to answer a specific question for a specific context.

b) Using our micro tool we synthesize the information we require programmatically.

c) The information is then presented in generated views. These views directly feed from the information which we synthesise from our system through micro tools. For example, a graph (the generated view) of a collection of objects within the system and their relationships to other objects. All the information, wherever it is presented, can be considered both live and directly extracted from the system. It has not been manually created. It is not unlike a photograph replacing a painting as a means of documentation.

d) Our conversation about the system is now based directly on the system. A key part of the conversation is to codify that information into a model i.e. a representation of the live, running system itself.

e) These conversations are part of how we explore the system. We use the model of the system to ask new questions as we explore more of the system. To answer these additional questions, we constantly create new micro tools, synthesizing new information which we then codify into our model of the system. The more we explore, the more complete this model of the system becomes. This approach mimics geographical exploration. We follow a more structured approach starting with a blank model, and then systematically model the landscape through observation using our own theodolites (the micro tools).

f) We guide the assessment through explicit hypotheses and make our decisions based upon this exploration and the model of the system itself. The model is created directly from the system and contains no necessity for belief. We do not need to trust what we are being told as we can directly interrogate the model which itself is generated live from the system.

The new practice has been added to Figure 2 to create Figure 4.

Starting at the bottom of the map above and moving upwards:

Our development experience changes from the use of a monolithic product from a tool vendor that is designed to be generic to a composable environment that you customize for your problem. Why is this composable environment considered to be more industrialised? If you use a toolkit (such as the open sourced Glamorous Toolkit) then many of the micro tools you will need to solve your problem have already been solved by others and serve as examples. The process of manufacturing micro tools has also been industrialised within the toolkit. It is the digital equivalent of creating your own “customised” MINI Cooper by choosing from different but highly industrialised options such as roof color, chassis color, trim type and wheel options. The billions of different possible permutations gives an illusion that the MINI is custom built but in reality it is constructed from highly industrialised components that are chosen by you to fit your needs. This is the same design strategy behind composable micro tools.

Our synthetization changes from manual inspection of the system through monolithic tools to specific coding of tools that provide a direct, unmediated feed of the system. This synthesization requires that the cost of creating the tool is cheap enough that it has no material impact e.g. built in minutes or hours rather than days, weeks or months. Decreasing the cost of creating a tool contributes to the industrialization of micro tools. Similar dynamics happened in the space of testing. When the cost of creating a single test became so small it did not matter, automatic and industrialized testing became adopted at scale. We can regard a test as a form of micro tool that takes the execution of a system and transforms it into a red/yellow/green signal. The same idea can be extended to any tool, including visualizations or queries.

The use of small tests was behind an approach known as Test Driven Development (TDD) where we create small tests before we make code. Moldable development is somewhat akin to TDD but we now create small tools before we make decisions. Both approaches can be combined in something known as Example Driven Development, however we will discuss this later in the book.

Our information is no longer manual but generated live through micro tools. These micro tools in effect model the system.

Our conversation changes from a data centric approach focused on information provided in a manual view (e.g. a powerpoint composed graph or a CAD created network diagram or a handwritten architectural diagram) to a model centric approach that is generated live from the system and can be directly interrogated.

Our exploration changes from ad-hoc, highly dependent upon the understanding of the system by an engineer to a more formalized approach akin to geography. A skilled geographer with theodolites can map out any landscape and they do not require a deep understanding of the landscape prior to creating a map. In the same manner, a skilled explorer can model any system without requiring a priori in-depth knowledge of the system.

Our assessment changes from gut feel (and the level of belief we have in what is being presented in the manual views) to more hypothesis based where questions can be directly interrogated against the model.

Example: Optimizing the data pipeline following the new path

To explore these ideas further, we return to our real world example. A large corporation wanted to optimize the performance of a central data pipeline by an order of magnitude. After years of effort, the data was moving through the pipeline at the same speed as before. This led to a situation where the business and engineering were in conflict and the pressure was mounting on engineering to find a solution.

When you find yourself spending effort that seems to have no effect on reality, it is plausible that your model of the system differs significantly from the actual system. This happens frequently when models are mental models informed by manually created diagrams. In such situations, you want to first improve your ability to see the actual system.

Our first hypothesis was that maybe their lack of results was because of an incomplete understanding of the system. Hence, our first question was “What does the system look like?”

In our case, we wanted a tool to give us an accurate perspective of the system using the data of the system itself. Alas, given the combination of technologies and the specificity of the business case (such as, what data points are being used in a specific marketing campaign), there exists no such tool out of the box. We had to create it.

While building the model, we realized that the output from the first system did not match the data from the databases. This small observation made the team discover a whole external 3rd party system that they were not aware of!

This is shown in figure 5.

This confirmed the hypothesis that their visibility into the system was partial and inaccurate. Past decisions had been based upon this. To create a high level aggregation, the tools had to start from the smallest scope such as how a single property traverses the pipeline. These tools were built.

Figure 6 shows the generated view from one of the tools. A single input (the blue dot in the “Internal” system) impacted multiple other points of data (the red dots in the “Internal” system). These points of data then became inputs (blue dots) into the “External” system and impacted further data (the red dots in the “External”). Finally these points of data became inputs (blue dots) into their low code system.

As can be seen from figure 6, a single property (the blue dot in the “Internal” system) impacted multiple inputs (the blue dots in the “LowCode” system) due to the existence of the third party “External” system. Without an understanding of this flow, any attempt to optimize the entire system (from internal to lowcode) was likely to be ineffective and fraught with failure. This is exactly what they were experiencing.

Based on the property-level data lineage, we constructed tools to generate multiple pipeline overviews. One such overview quantified the amount of data produced along the pipeline and can be seen in Figure 7.

For each system, the visualization depicts groups of properties that are used or not used downstream in other systems. The red parts correspond to properties created but not used later on. In essence this was the “dark data” i.e. data that had no interaction with the rest of the system but added “mass” to the system.

These visualizations showed the team that they did indeed generate data they were not using, but more importantly, it made them realize that the situation is not hopeless and that it is possible to get an overview over the system, even if it was made out of heterogeneous and old components. Based on the input, they decided to redo the pipeline by consolidating the information to eliminate the need for unnecessary transformations.

This new path of assessing the situation included:

a) A development experience consisting of custom tools built specifically for the system to answer an initial question of “Why are our services slow?” which leads to a question of “Is there useless data generated” which requires a question of “How is data traversing the pipeline?” to be answered.

b) The tools synthesized information from the data pipeline that existed within the entire system between the various components. This was continuous live data.

c) The tools consolidated information into generated views of the system such as figures 6 & 7. It should be noted that figures 6 & 7 aren’t manual drawings but are continuously generated live from the systems themselves.

d) The conversations were based upon the live information within the system rather than any erroneous belief as originally existed in figure 3 with the missing 3rd party component.

e) The exploration of the system consisted of a systematic approach of building a model of the system by asking questions and answering those questions through tools that used the system itself. To achieve this, it is essential to use some form of toolkit which enables people to rapidly build tools whilst the toolkit models the system under examination. In the above case, the tools behind figure 6 & 7 were built using Glamorous Toolkit.

In total, 54 distinct micro tools (i.e. at least 54 answers to questions) were needed to properly understand the system in order to answer the top level question of “Is there useless data generated?”. This took approximately 2 person months of effort to solve the top level question and provide a model of the system compared to the previous attempts which were measured in hundred person years and many millions of dollars of direct costs (ignoring the opportunity cost of the time spent and the lack of solution for the business problem). In person months alone, this was a factor of 600x improvement with an actual working solution to the problem rather than failure.

f) They assessed and made their decisions based upon accurate information about the system. This led to a situation where the business and engineering could come to an agreement based on a common understanding of the problem.

How relevant is the example to you

It’s tempting to look at the example and ask “How did they miss that 3rd party system?” or state that “We wouldn’t do something like that”. The team of engineers at the client were extremely capable including a global leader in the service integrator industry. The engineers were all highly qualified with degrees and industry qualifications. However, this system was large with many pieces that had been built over many years. People had retired, and component systems had been forgotten about. The team was also siloed into different groups — an internal system team, a database team, a low code team etc and no overall picture existed.

Whilst this might appear as an extreme, we would argue based upon experience that highly skilled teams, often operating in silos, trying to manage complex and complicated legacy environments with a less than adequate model of the system is commonplace, if not the norm. It does not surprise us that 83% of data and legacy migration projects either fail, or exceed their budgets or schedules. What surprises us is that 17% succeed. However, even if that 17% is correct then we would expect that the results could be achieved at a much faster speed and lower cost.

If you have any form of legacy estate, then we suspect the example is relevant to you. To test, simply ask your engineering team to provide you with a model of the system. If the result is a manual hand written or powerpoint diagram (such as figure 3) then ask for a comparison to live. If you cannot get one, then you are likely to be navigating blind.

Why not just use an AI to solve this?

AI is a field and when most CIOs in 2024 talk about AI, they usually mean a specific set of transformer architectures such as LLMs (large language models) and LMMs (large multi-modal models) that are commonly found in code co-pilots. Whilst there is nothing wrong with using such models to assist in the effort (and the authors both do), as of December 2024 marketing research campaigns claim that copilots can help developers write up to x1.55 (55%) faster. It should be remembered that these are marketing research and are often challenged by independent research which claims figures as low as x1.05 (5%).

Typically, using a model-centric approach as described here, we see improvements of orders of magnitude. For example, the case above showed 600x (60,000%). In this case we had something to measure the impact against (the previous time and costs) but often it is difficult to quantify the difference because there is no baseline to compare against. Even in this case, we don’t know how long they would have had to continue before they found an answer and we can only compare against the time they had taken before they gave up with the traditional approach. The main issues with the comparison are:

1. It is rare to have a baseline to compare against.
2. Often, even where data exists, we are not comparing like for like i.e. you have one group which has failed to find an answer or given up versus one where the answer is found.
3. Small sample sizes. It’s not possible to discount the influence of luck, the skill of the engineers or the bias of the authors.
4. Larger opportunity costs to the business due to delays are rarely factored into the equation.

As a result, we are not in a position to quantify the impact of a more moldable approach and instead must rely on our experience of orders of magnitude. However, if we can’t quantify the impact then how do we know this path is right? The simple answer is we don’t but we do believe this is a better path. This is no different from my experience with Wardley Mapping back in 2005. There maybe a better approach but we’re not aware of it. All we can do is to help others become aware of this moldable approach and ensure the tools we use are open source as well as the knowledge.

That said, the two approaches — use of AI models and Moldable Development — are not in opposition and can compliment each other. We will discuss how, later in this book.

Throughout this introduction we have discussed the role of questions and answers. In the next part of our journey we will explore this in more detail and why it matters.

Rewilding Software Engineering was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

An example of molding custom source code analyses

Glamorous Toolkit, the moldable development environment, ships with extensive software analysis support for various languages. These analyses can be combined and integrated with the rest of the environment to assist developers in assessing their own system.

In this tutorial we explore how we can take architectural constrains for our systems that often reside in wiki pages and other documents, and express them in a testable form through custom analyses.

The software modeling part is based on FAMIX, a family of meta-models for representing various facts about software systems. FAMIX is developed within the context of the Moose analysis platform. Glamorous Toolkit integrates FAMIX and the core of Moose into the overall development environment.

On checking architecture

Software architecture can mean different things for different people. In a broad sense, we can look at architecture as the set of design decisions that guide the implementation of a software system. They are especially helpful when we need to make decisions about how to evolve our systems.

However, if our assumptions about a system’s architecture are out of sync with the reality from the code, we can easily make wrong decisions about how to evolve the system. This has a higher chance of happening as long as the architecture of our systems remains only described in wikis, word documents or presentations. We can reduce this chance by taking our architectural decisions and expressing them as concrete analyses that we can run automatically to check that our assumptions still match the reality found in the code.

Running example

As an initial case study for expressing architectural constraints we use ArgoUML, an open-source Java applications for creating UML diagrams. We use version 0.34 of this application, which has around 2400 classes and ~200k lines of Java code.

For this system we create an architectural report containing three constraints:

• No deprecated classes are used within the system;
• UI widgets are placed within dedicated UI packages;
• Modules implementing UML extensions for different programming languages have no inter-dependencies.

These constraints that are easy to write, and we use them as examples to explain the steps needed to create architectural constraints within Glamorous Toolkit.

Setup

To begin checking these constraints we need to load a model of ArgoUML into Glamorous Toolkit. We achieve this by means of an MSE file that must be exported by an external exporter. You can download the source code of ArgoUML, and an already created MSE model file using the snippet below. The MSE file was created using jdt2famix.

targetFolder := 'models' asFileReference ensureCreateDirectory.
archiveFileName := 'ArgoUML-0-34.zip'.
archiveUrl := 'https://dl.feenk.com/moose-tutorial/argouml/'. ZnClient new
  url: archiveUrl, archiveFileName;
  signalProgress: true;
  downloadTo: targetFolder.
(ZipArchive new 
  readFrom: targetFolder / archiveFileName) 
    extractAllTo: targetFolder.
targetFolder

The snippet creates a new models folder in the Glamorous Toolkit directory, and downloads and extracts an already prepared archive for this tutorial that contains the code and the MSE model file (the archive is about 33 MB). We can inspect the models folder to check if the download went ok.

We can load that model by executing another code snippet.

modelFile := 'models' asFileReference
  / 'ArgoUML-0-34'
  / 'ArgoUML-0-34.mse'.
model := MooseModel new
  importMSEFromFile: modelFile.
model

Now we have a loaded model of our system that we can inspect and explore.

Alternatively, if you want to also build the model file locally from source code, another article details the necessary steps.

The anatomy of an architectural constraint

What’s the best way to express an architectural constraint? There are many ways to approach it, but we suggest the following steps.

1. Describe the constraint

First, we need to describe the architectural constraint in plain words. The description can consist in at least a few sentences primarily focusing on the reason for which we want to verify the constraint. The code of the constraint describes what it does, and the plain text description should augment that with why it is important.

2. Ensure it is testable

Second, we need to ensure that we can actually test the constraint. For this we need to make sure that we can clearly identify the types of entities and relations involved in the constraint, and that we have all necessary data. For example, before testing that the database layer does not access code from the business layer we first need to identify which code belongs in the database layer and which code belongs in the business layer.

3. Attach a stakeholder

Every constraint should have a stakeholder. This is the person, group of persons, or team, for whom the constraint has value. If a constraint fails anyone can challenge the validity of a constraint. The stakeholder’s role is to defend it, or at least provide extra context for why it is relevant. If there’s nobody to defend it, the constraint should be removed as it does not add value anymore.

4. Formulate the analysis

Once we have a description of a constraint and a way to identify all relevant entities, we can proceed to specify an analysis that expresses that constraint. We should phrase the analysis so that it returns a list of entities (for example methods, classes, packages) that do not respect the constraint. This way, if the analysis returns an empty list the constraint passes. The constraint is considered as failing if the analysis returns one or more entities.

Implementing architectural constraints

Let’s look at defining a few concrete constraints.

Constraints in Glamorous Toolkit are subclasses of GtLeafConstraint; these kinds of constraints capture one single aspect about our system. To better structure all constraints relevant to ArgoUML we create a subclass named ArgoUMLConstraint. This class also stores the model on which we apply the constraint.

GtLeafConstraint subclass: #ArgoUMLConstraint
 instanceVariableNames: 'model'
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

ArgoUMLConstraint>>#model
 ^ model

ArgoUMLConstraint>>#model: aModel
 model := aModel

To create the actual constraint we need to override the following methods:

• name: Indicates the name of the constraint;
• description: Provides a description for the constraint;
• stakeholder: Indicates the stakeholder associated with the constraint;
• issues: Computes the list of entities that do not satisfy the constraint;
• status: Optional method indicating how to handle the case of a failing constraint when running constraints as tests. For example, we might not want to mark the build as broken due to a failing constraint, but it can still be valuable to know the list of entities that do not respect that constraint.

Checking for deprecated classes

Our first constraint checks that no deprecated classes are being used within the system. This is not the most useful constraint, however, it is a very simple one to implement and illustrates how to apply the steps:

1. Describe the constraint. To make deprecated classes easy to remove, they should not be used from classes that are not themselves deprecated.
2. Ensure it is testable. To test this we need to obtain the list of classes from our system and be able to detect if a class is annotated with @Deprecated. For each deprecated class we further need to access the list of classes using it. This data is readily available in the model.
3. Attach a stakeholder. We make the Development Team the stakeholder for this constraint.
4. Formulate an analysis. We first formulate the analysis as a sentence and then prototype an implementation in the playground.

From the model get the list of all classes that are deprecated. From this list select all classes that have at least a client that is not deprecated.

allDeprecatedClasses := model allModelClasses select: [ :each |
  each isAnnotatedWith: 'Deprecated' ].
allDeprecatedClasses select: [ :each |
  each clientTypes anySatisfy: [ :aClient |
    (aClient isAnnotatedWith: 'Deprecated') not ] ]

By prototyping the implementation in the Playground we can directly see the list of entities returned by the constraint and make sure it is ok.

To implement the constraint we first create a subclass of ArgoUMLConstraint named ArgoUMLDeprecatedClassesWithNoDeprecatedClients. The name and description methods are trivial to implement:

ArgoUMLConstraint 
 subclass: #ArgoUMLDeprecatedClassesWithNoDeprecatedClients
 instanceVariableNames: ''
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

ArgoUMLDeprecatedClassesWithNoDeprecatedClients>>#name
 ^ 'Deprecated classes with no deprecated clients'

ArgoUMLDeprecatedClassesWithNoDeprecatedClients>>#description
 ^ 'To make deprecated classes easy to remove, they should not be used from classes that are not themselves deprecated.'

To attach a stakeholder we create an instance of GtConstraintStakeholder and return it from the stakeholder method.

ArgoUMLDeprecatedClassesWithNoDeprecatedClients>>#stakeholder
 ^ GtConstraintStakeholder new 
     name: 'Development Team'

Now the most complicated method is issues that implements the analysis. In this case we can just reuse the code from the Playground where we prototyped the constraint.

ArgoUMLDeprecatedClassesWithNoDeprecatedClients>>#issues
  | allDeprecatedClasses |
  allDeprecatedClasses := self model allModelClasses select: [:each|
   each isAnnotatedWith: 'Deprecated' ].
  ^ allDeprecatedClasses select: [ :each |
      each clientTypes anySatisfy: [ :aClient |
        (aClient isAnnotatedWith: 'Deprecated') not ]

We stop here with the implementation and check the constraint. We can instantiate and inspect the constraint directly in the Playground. Constraint objects have an inspector view showing the list of entities that do not respect that constraint. In this case we see in that view the 10 classes that are not deprecated and use deprecated classes.

ArgoUMLDeprecatedClassesWithNoDeprecatedClients new
  model: model

Ensuring correct placement of classes

For the second constraint let us implement a less generic one that is more specific to our system.

ArgoUML is an application with a graphical user interface. If we inspect its list of packages we observe that UI related classes are placed in UI packages that contain the ui component in their name, for example org.argouml.ui, org.argouml.notation.ui, or org.argouml.ui.explorer .

To make it easier to identify UI related classes and better structure the code, we decide that all classes that represent graphical widgets should be placed in packages having the ui component in their name. This rule also allows those packages to contain sub-packages. Depending on our context we can also have a more strict rule. For now we go with the less strict version. We consider as graphical widgets those classes that inherit from java.awt.Component (here we can later decide to extend this definition).

We begin by creating a class for the constraint:

ArgoUMLConstraint subclass: #ArgoUMLWidgetsInTheWrongPackages
 instanceVariableNames: ''
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

Next we go through the four steps for creating a constraint.

1. Describe the constraint. Classes representing widgets (inherit from java.awt.Component) should be placed in packages that contain the ui component in their name to improve code modularity.

ArgoUMLWidgetsInTheWrongPackages>>#name
  ^ 'Widgets placed in the wrong package'

ArgoUMLWidgetsInTheWrongPackages>>#description
 ^ 'Classes representing widgets (inherit from ''java.awt.Component'') should be placed in packages that contain the ''ui'' component in their name to improve code modularity.'

2. Ensure it is testable. To test this constraint we need to access the list of classes from our system and for each class access all its superclasses including the ones from external dependencies like the graphical library AWT. For each superclass we need to access the package containing it. We can ensure we get all these information in the model, if during the generation of the model, we provide the importer with access to all relevant dependencies for the graphical libraries Swing and AWT.

3. Attach a stakeholder. As with the previous constraint the Development Team is the stakeholder for this constraint. Just in this case let us create a more specialized way of specifying the stakeholder. In the previous constraint, we used directly an instance of type GtConstraintStakeholder. We can create a subclass, named ArgoUMLConstraintStakeholder , and add a dedicated API for creating stakeholders.

GtConstraintStakeholder subclass: #ArgoUMLConstraintStakeholder
 instanceVariableNames: ''
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

ArgoUMLConstraintStakeholder class
 instanceVariableNames: 'developmentTeam'

ArgoUMLConstraintStakeholder class>>#developmentTeam
  ^ developmentTeam ifNil: [
      developmentTeam := self new name: 'Development Team' ]

Now we can specify the stakeholder using the method developmentTeam.

ArgoUMLWidgetsInTheWrongPackages>>#stakeholder
  ^ ArgoUMLConstraintStakeholder developmentTeam

We can also refactor the stakeholder method in the previous constraint.

ArgoUMLDeprecatedClassesWithNoDeprecatedClients>>#stakeholder
  ^ ArgoUMLConstraintStakeholder developmentTeam

4. Formulate an analysis. Last but not least we should create the analysis that checks the constraint. We start by formulating it in plain text:

From the model, get the list of all classes representing widgets. From this list, select all classes that are not placed in a package containing the ‘ui’ component.

We can now prototype the analysis in the Playground. As the first step we extract all widget classes from the model.

allWidgetClasses := model allModelClasses select: [ :aClass |
  aClass superclassHierarchyGroup anySatisfy: [ :aSuperclass |
    aSuperclass mooseName = 'java::awt::Component'] ].

As a second step, we reject all widget classes that contain the ui component in their package name. There could be multiple ways to implement this but for now let us simply check if the qualified name includes a component whose name begins with ui.

allWidgetClasses reject: [ :aClass |
  aClass namespaceScope mooseName includesSubstring: '::ui' ].

Now that we have the entire analysis we can place it in the issues method.

ArgoUMLWidgetsInTheWrongPackages>>#issues
 | allWidgetClasses |
 allWidgetClasses := self model allModelClasses select: [ :aClass |
   aClass superclassHierarchyGroup anySatisfy: [ :aSuperclass |
     aSuperclass mooseName = 'java::awt::Component' ] ].
 ^ allWidgetClasses reject: [ :aClass |
     aClass namespaceScope mooseName includesSubstring: '::ui' ].

Next we can also inspect this constraint in the Playground to ensure it works.

ArgoUMLWidgetsInTheWrongPackages new
 model: model

Verifying dependencies between modules

While the previous constraint is more contextual, it still can be applied to many systems that rely on a user interface and follow the same naming convention. As a last example, let us look at a constraint only relevant for ArgoUML.

ArgoUML can generate UML diagrams for systems written in various programming languages. Each language can have its own particularities. To support this ArgoUML has a core module and UML modules for different programming languages. We can decide that in our design these modules should not have any dependencies between them; they should just depend on a core module. We would like to create a constraint to ensure this.

Before going into the actual implementation, we can consider what should this constraint return. One option would be to return the list of modules that have those unwanted dependencies. The disadvantage of this is that when we get a failure we need to manually go and find the invalid dependencies.

Instead, we can structure the constraint to return the list of invalid dependencies between modules, and if we get invalid dependencies we know the exact places in the code where the invalid calls happen.

ArgoUMLConstraint subclass: #ArgoUMLInvalidUMLModulesDependencies
 instanceVariableNames: ''
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

Let’s go through the four steps for creating a constraint.

1. Describe the constraint. UML modules for different programming languages should not depend on one another.

ArgoUMLInvalidUMLModulesDependencies>>#name
  ^ 'Invalid dependencies between UML modules'

ArgoUMLInvalidUMLModulesDependencies>>#description
  ^ 'UML modules for different programming languages should not depend on one another.'

2. Ensure it is testable. To test this we need to obtain the list of UML modules, and the classes in each module. In this case we consider a module all classes from certain packages. For each class we further need to access all of its dependencies. We can get all these data from the model.

3. Attach a stakeholder. We make the Architecture Team the stakeholder for this constraint.

ArgoUMLConstraintStakeholder class>>#architectureTeam
  ^ architectureTeam ifNil: [
      architectureTeam := self new name: 'Architecture Team' ]

ArgoUMLInvalidUMLModulesDependencies>>#stakeholder
  ^ ArgoUMLConstraintStakeholder architectureTeam

4. Formulate an analysis. There are more ways in which we could create this analysis. One possible way is as below:

From the model get the list of all UML modules. For each module collect the list of outgoing dependencies. Select from each list dependencies where the target of the dependency is another UML module.

Let’s prototype the analysis in the Playground. We do not go into details here about the implementation of the analysis. We start by getting for each UML module the list of packages that we are interested in. These are the packages that begin with org.argouml.language followed by the name of a language (there are also other packages that begin with org.argouml.language that are not UML modules)

moduleNames := #('cpp' 'csharp' 'java' 'php' 'sql').
modulePackages := model allModelNamespaces select: [ :aNamespace |
  | fullName |
  fullName := aNamespace mooseName.
  moduleNames anySatisfy: [ :moduleName |
    fullName beginsWith: 'org::argouml::language::', moduleName ] ].
modulesByName := modulePackages groupedBy: [ :aNamespace |    
  aNamespace withAllParentScopes reversed fourth mooseName ].

The second step is to get for every module all its dependencies. We can obtain this using the method queryAllOutgoing, that returns all types of dependencies an entity has in a model. We then select as invalid those dependencies that are to other UML modules.

allInvalidDependencies := OrderedCollection new.
modulesByName keysAndValuesDo: [ :moduleName :packages |
 | moduleDependencies invalidDependencies otherModuleNames |
 moduleDependencies := packages flatCollect: #queryAllOutgoing. 
 otherModuleNames := modulesByName keys copyWithout: moduleName.
 
 invalidDependencies := moduleDependencies select: [ :aDependency |
  aDependency to asCollection anySatisfy: [ :aCandidate |
   otherModuleNames anySatisfy: [ :anotherModuleName |
    aCandidate mooseName beginsWith: anotherModuleName ] ] ].
 
 allInvalidDependencies addAll: invalidDependencies ].
allInvalidDependencies

We notice that there are no modules that do not respect the constraint. To double check that the constraint works we could also go and introduce some invalid dependencies in the code and see if the constraint detects them.

We can take now the previous snippets and put them into the constraint method:

ArgoUMLInvalidUMLModulesDependencies>>#issues
  |moduleNames modulePackages modulesByName allInvalidDependencies|

  moduleNames := #('cpp' 'csharp' 'java' 'php' 'sql').
  modulePackages := model allModelNamespaces select: [:aNamespace |
    | fullName |
    fullName := aNamespace mooseName.
    moduleNames anySatisfy: [ :moduleName |
      fullName beginsWith: 'org::argouml::language::', moduleName]].
  modulesByName := modulePackages groupedBy: [ :aNamespace |    
    aNamespace withAllParentScopes reversed fourth mooseName ].

  allInvalidDependencies := OrderedCollection new.
  modulesByName keysAndValuesDo: [ :moduleName :packages |
   | moduleDependencies invalidDependencies otherModuleNames |
   moduleDependencies := packages flatCollect: #queryAllOutgoing. 
   otherModuleNames := modulesByName keys copyWithout: moduleName.
 
   invalidDependencies := moduleDependencies select: [:aDependency |
     aDependency to asCollection anySatisfy: [ :aCandidate |
       otherModuleNames anySatisfy: [ :anotherModuleName |
         aCandidate mooseName beginsWith: anotherModuleName ] ] ].
 
    allInvalidDependencies addAll: invalidDependencies ].
  ^ allInvalidDependencies

This constraint requires more code than the previous two. Part of the reason is that this constraint is more complicated. Another is the way in which we chose to implement the constraint. However, the more important reason is that we just used “raw” operations over a model to perform all the necessary checks.

We will look in a future tutorial at how to extend a model with domain-specific queries that will make it possible to write analyses at a level of abstraction closer to the domain of our applications. Then we could, for example, express the logic for getting the packages in an UML module as below, instead of checking names of packages in the analysis.

modulePackages := model allModelNamespaces 
  select: #isArgoUMLUmlModulePackage.
modulesByName := modulePackages groupedBy: #argoUMlUmlModuleName.

Creating an architectural report

Up to this point we implemented three architectural constraints. To finish, we put all of them into an architectural report. An architectural report is nothing more than a composite constraint.

We created the individual constraints by subclassing GtLeafConstraint. To create an architectural report we subclass GtConstrainerReport, which is a subclass of GtCompositeConstraint with some helper methods. As we need to pass the model to the individual constraints we store it in the report.

GtConstrainerReport subclass: #ArgoUMLReport
 instanceVariableNames: 'model'
 classVariableNames: ''
 package: 'Argouml-ArchitectureRules'

ArgoUMLReport>>#model
  ^ model

ArgoUMLReport>>#model: aModel
  model := aModel

ArgoUMLReport class>>#onModel: aModel 
  ^ self basicNew  
      model: aModel;  
      initialize

Next we override the build: method and use addConstraint: to add individual constraints to the report.

build: aComposite
 aComposite name: 'ArgoUML Report'.
 aComposite
  addConstraint:(ArgoUMLDeprecatedClassesWithNoDeprecatedClients new 
   model: self model);
  addConstraint: (ArgoUMLWidgetsInTheWrongPackages new 
   model: self model);
  addConstraint: (ArgoUMLInvalidUMLModulesDependencies new 
   model: self model)

Now we can instantiate the report and inspect it.

report := ArgoUMLReport onModel: model.

We can decide that in our system using deprecated classes is ok for the moment. We could remove that constraint, however, then we cannot now just by looking at the report how many deprecated classes are currently in use. Instead we can mark the constraint about deprecated classes as neutral by overriding the status method in the constraint.

ArgoUMLInvalidUMLModulesDependencies>>#status
  ^ GtNeutralConstraintStatus new

Now when running the report, the first constraint is not marked as failed, and we can still get the list of deprecated classes.

Running the architectural report as part of continuous integration

Above we were looking at the report in the inspector. However, it can be useful to also run the report as part of our CI process. We can write a simple script that creates a model from an MSE file, instantiates our architectural report, and exports the results of running the report using the same format used by JUnit to return results.

$./glamoroustoolkit GlamorousToolkit.image eval "\
| modelFileName model|\
modelFileName := '.models/ArgoUML-0-34/ArgoUML-0-34.mse'\
model := MooseModel new \
importMSEFromFile: modelFileName asFileReference. \
ArgoUMLReport onModel: model.\
GtConstrainerHudsonReporter runOn: (ArgoUMLReport onModel: model)."

Below we see the results of running our architectural report for ArgoUML on a Jenkins server using the script above. Only the constraints about the placement of widgets is marked as failing.

Wrap-up

In this tutorial we explored the creation of an architectural report for a Java system. An architectural report consists of a set of constraints expressed through custom analyses. We can then automatically run those analyses and check them against the code. This is a first step in making sure that our assumptions about a system’s architecture do not become out of sync with the reality from the code.

Expressing architectural constraints for a Java system in Glamorous Toolkit was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

How we work at feenk

As I am writing this, billions of people are urged to socially distance themselves from others. Temporary hospitals are erected. Medical personnel face deadly situations without proper equipment. Factories change to produce ventilators. As physical places shut down, the economy takes a hard hit, too. Unemployment sky rockets. Production drops. People worry. And rightfully so.

We are all asked to do our part and stay home. But, some of us are more privileged than others: we can work and produce value while being at home. This article is addressed to them. In particular, to those creating software-based solutions.

Many people are experiencing working from home full time for the first time. And for many, it’s not a pleasant transition. Routines are shattered. Frustration grows. Productivity seems to go down.

It does not have to be like this. Well, let’s not talk about productivity right now. We are not going through a normal situation. It’s already a stressful time. At the very least, work should not make it more stressful. Still, we should be able to produce some value. And, perhaps, in the process, we can also bring some joy into our lives.

To help with this conversation, I’d like to offer our experience at feenk as a case study. Ever since we started in 2015, we work in a rather peculiar way: we work exclusively remotely, we work when it fits, we have no scheduled meetings, we do not rely on estimations, we have no centrally ordered backlog, and we guide our effort through the stories we tell each other.

According to the typical organizational and development process literature this sounds like we should not be able to produce anything of meaning. Whether we do or not, we leave it up to you to decide.

Perhaps a bit of background. We are a consulting company, but we work internally as a research lab. Our goal is to make the internals of systems explainable. We do this by rethinking the development experience:

• we invented moldable development, a new approach to programming that makes the system explainable through custom tools, and
• we develop Glamorous Toolkit, the environment that makes moldable development practical.

Let’s assume you consider what we produce as being not entirely meaningless. I am not suggesting you necessarily do what we do, but at the very least our experience could be considered as a case study to unearth assumptions that should not be cast in stone.

Humans first

People that work are not resources. They are humans. We see well-being having priority over work. We apply this as a matter of principle, not as a result of a crisis.

What does that mean? For one, well-being is personal. If we follow this to its natural conclusion, it means that it is the organization that should adapt to the individual, not the other way around. Every individual has contextual needs starting with availability. A person with family and kids will have different needs than a single person that wants to travel the world. Neither is better. They are just different.

We accept our friends to be different and those relationships are better and stronger for it. Why should this not apply for the people we want to work with?

Moreover, the individual needs likely vary in time. For example, for many of us, the individual needs during this pandemic are significantly different than they were a mere couple of months ago. Just consider the sudden transition to homeschooling that many families have to go through. This is dramatically disruptive. It’s likely that people no longer have long contiguous blocks of time to allocate to work.

We are humans first, workers second. Work should be embedded in the current reality of life, and only the individual knows how that should happen best. There are exceptions, of course. Right now, medical and support personnel throughout the world will have little choice but to work in terrible circumstances. But, those are exceptions. The majority of us can still choose.

This is particularly relevant when you set out to do work that has never been done before. When you want to create, it’s best to be in a position to be able to create. Yes, some external pressure can sometimes work, but not for an extended period of time, and not when it’s not voluntarily accepted. In general, creativity requires lateral thinking, and anxiety tends to inhibit it.

When we can work in our terms, we have more chances to enjoy our work. When we can decide what is important, we are more likely to explore a step further than usual, and by doing so, we increase the chance of uncovering value that otherwise would remain hidden.

As we like what we do, we sometimes tend to work more than we should. This is a tendency we actively try to counter balance. While we do not want to control how much people work, we do try to keep track of when people are explicitly on holidays. The contract stipulates a minimum amount of mandatory holidays. People are encouraged to take more than that, but at least the minimum must be taken. This seems to work reasonably well, but we likely still can improve in this regard.

Remote. No prescheduled synchronization

We regard the organization as a distributed system made of collaborative actors. There are multiple ways to design such a system. We chose the view that relies on asynchronous communication and on eventual consistency. This implies that the organization should not rely on any prescheduled synchronization mechanisms.

The most obvious is to not have to be in a certain place. We work remotely. We are currently a team of 11 spread over 5 countries in 3 continents. While some of us actually live in the same city, we do not meet in person to work. We do meet occasionally on company trips, conferences, or visits to our customers (during the COVID-19 crisis, we cancelled all of those). Still, our functioning does not depend on anyone being in a certain place.

Another synchronization mechanism is the work schedule. If everything is asynchronous by default, it does not quite matter when one works. This turns out to be rather convenient when someone is on a different timezone, and it goes hand in hand with having no scheduled meetings, too. A scheduled meeting requires someone to be available at a certain time, which is at odds with the asynchronous work principle. So, we have no such meetings for any of our internal work. That means we have no standup, no fix planning session. The only scheduled meetings we have is when we have to comply with an external factor, such as a customer, but we fight hard to not have many of those.

A meeting is a way to bring different parties into sync. However, when we rely on eventual consistency, we do not have to have everyone be on the same page all the time. We can be out of sync for a while and we are fine with that.

Yet another mechanism that induces synchronization is the ordered backlog. An ordered backlog implies that there is a single queue of work. So, we do not have one of those. We do have an issue tracker, but it is not ordered. In fact, everyone in the team can do what they want at any time.

That sounds like chaos. So, let’s look at what we put in place.

Organizing communication around conversations instead of topics

We do not schedule meetings, but we still meet. In fact, we like to pair often. Only we do it when it fits everyone involved. How do we do that? We simply ask. If I want to pair with someone, I’ll ask kindly if they are available. If they are available, we pair. If not, we do not.

Our communication setup is simple. Almost simplistic. We mostly use a chat tool, but more relevant is how we use it. The vast majority of communication (>90% of messages) are exchanged on public channels. The channels do not denote topics, but are named after pairs of people, such as #personA-personB. The goal is to connect people first and have work follow the connection, rather than the other way around. We find that this works reasonably well in our case, in particular given that the same persons often end up tackling rather distinct problems.

As all these channels are open, they can be seen by anyone and they can also be contributed to by anyone. It is not unusual for a conversation to start with two people and then have one or two join for a short while with various inputs.

A worry is that this leads to too much noise. That can happen. There are days of agitation when everything seems to be moving at once, but in most days, it is manageable. One interesting side-effect of having everything in the open is that there is less anxiety of missing out.

Beside chatting and pairing, we also communicate through narratives and through code. When we want to convey a larger wish, we might describe it in a longer internal blog post. But, when we want to convey what is in the system, we make it the job of the system to carry that narrative forward, in an executable manner.

Compacting communication

Remote communication does not have a good reputation. We hear people complain about the quality of connections, or about the lack of physical feedback during calls. This can happen when we assume that we need to replicate the same communication the happens when co-located. That’s a misguided comparison. Remote work is another new medium that offers other opportunities.

First, the lead time: How long does it last between the moment you want to do something and the moment you actually get to do it? If for talking with someone, I need to move to a room, there is a delay involved. If to move to that room, I first have to book it, there is another delay involved. These can take many minutes. In a remote environment, it can take seconds. This matters because shorter lead times can enable finer grained communication.

When remotely, we can pair for just a few minutes meaningfully, something that is not so convenient when co-located. This is just one example. Look for the shift in costs, and adjust accordingly.

There is at least one more thing to consider when it comes to communication costs. If we get the cost of communication down, say an order of magnitude, you get access to a whole new set of options.

Consider this conversation sample.

Orange has a question. Blue answers in brief plain text. Orange answers back with a picture showing the inheritance tree of the class that Blue pointed out, essentially saying that the indication does not seem to map to the system. This is not unusual. These back and forth are often necessary, but can be quite expensive. Blue then sends a script with a picture that exemplifies the message concretely. Orange commits the code a couple of minutes later without asking any further questions.

Being able to show what you mean in a way that the other can immediately relate to without much further explanation can bring the cost of communication down significantly. When the cost is large, we might want to have synchronization points with status/planning meetings and with associated ceremonies. But, if you can convey the same problem in depth through a picture that you can put together in a few minutes and have the person on the other side say “ahh, yes, I see it”, you have new options.

We explicitly and continuously optimize our communication cost. It is not always as easy, but when it does happen (and our chat is full of pictures we send back and forth) we literally rejoice. It’s a thing we aim for. In our case, constructing concise narratives about a system is at the core of moldable development and is enhanced by our development environment.

Distributed backlog

Theory says that we should be doing the most important work first. That might work well in a factory. Or when everything else is equal. But, in creative work we see two problems with it.

First, someone would have to define what important is. Second, someone would have to order the items linearly. That’s not trivial in a space that was not tackled before. We often hear that the ordering is the prerogative of the business. But, how often does it happen for the business people to order the backlog only to find the technical tasks be pushed at the bottom of the list?

Over the years, we have experienced many times how seemingly unimportant issues unravel opportunities that would have otherwise be missed. In our team, there is no one that tells others what to do. Anyone can choose what they want to work on.

If we look at the core of it, a backlog holds a set of wishes. Some of those wishes come from some existing pains. Others might come from dreams of new possibilities. And they might come at various levels of abstractions.

When you set out to create a new kind of value, one that a customer cannot yet quite define, all you are left with is an inner compass. Only when you work as a team on a large enough problem, there are potentially many compasses, each working in a certain area.

Software is multi-dimensional. Flattening all those dimensions in a single list is rather limiting, and we choose to not do that.

Planning around goals. No estimations. No handovers

Because nobody tells anyone what to do, nobody can expect that someome does something specific either. In practice, they do listen to one another and they do coordinate.

To coordinate, we do employ some informal planning. Our planning happens in pairs or small groups, and might take minutes. And we might do it multiple times a day, or just a few times a week.

We never plan too specific tasks. We mostly agree on the goal and on how to describe a problem. And we think of the demo we want to show at the end.

We might engage in some short term estimations mostly to evaluate whether we want to go one way or another, but nothing else depends on that estimation. There are no promises. There are no handovers. We do not have to wait for others to finish something. If it’s not finished we might continue it, or we might go a different route.

We should also mention the issue of trust. When people choose when and how to work, we must trust them. We might disagree with the path or outcome, but all debates should happen from a trust perspective. Trust is essential and it goes both ways: When we trust the source of feedback, we are less likely to perceive it as personal and focus on improving the work.

Throwing code away

When we describe what we do, we hear often that given that there is no upfront coordination, it might happen that there will be overlap in what different people work on. In practice, it happens very rarely. In most cases, people naturally work together.

Nevertheless, it can happen that two people work on the same thing at the same time, and we are more than fine with it. That’s because we throw away most of the code we write. If we do not like the story we can tell with the code, we throw it away. Even if it works. It’s rather liberating, really.

Just consider the alternative: if you never throw away code that works, it means that you got it right the first time. That’s unlikely. We find that we need some 3–5 variations of the same thing to reach a reasonably satisfying solution.

So, having two competing solutions is actually positive because it means we can learn faster. Throwing code away is only a waste when you think of programming as a process of construction. Instead, we regard programming as capturing our ideas in a larger, coherent, but yet-to-be-defined puzzle. In this context multiple perspectives are essential and throwing code away is almost mandatory.

Trunk-based development. Continuous delivery

We do synchronize eventually. In code, directly on trunk. We do not use development branches or pull requests, and we do not require code reviews. We simply commit, integrate and deploy intensely.

We might review code during pairing, or we might do it after a commit while working on the code. When we do find something we do not like, we transform it into automatic detections either in the form of tests, when the problem is decomposable functionally, or of structural checks, when the problem is better suited for some static analysis. In other words, we use reviewing for exploratory purposes, but the regression checking is always the job of the machine.

From a technical perspective, this way of working enables continuous delivery. The largest cost in software development is integration and we want to see problems as soon as possible. This constraints how we work and naturally pushes us to modularize our work both internally and from a user perspective.

From an organizational perspective, a pull request that requires a review is a synchronization point. It often builds a queue and one’s work waits for someone else. We want to avoid relying on synchronization mechanisms that are not necessary.

And, there is another perspective to it. We look at the systems we construct not as something we implement, but something that expresses our understanding of the problem. Each of us has an individual understanding from our own perspective. Given the multi-dimensional nature of software, it is not reasonable to expect a complete understanding to fit in a single head. We need to externalize it and the code is the ultimate place in which we get to synchronize our perspectives. When a mismatch occurs, we see it as an opportunity to learn from and adjust our understanding. This synchronization mechanism is mandatory, and we want to get to this point in the fastest possible way.

Storytelling

Storytelling glues our development together. When we want to convince a colleague about a particular direction, we tell a story. When we design, we think of the story the system will tell. When we document our system, we make the system show the story. And we release only when we deem that the story is beautiful.

Stories come in many shapes. A document can tell a story. A blog post can tell a story. A comment to an issue can tell a story. A tweet can tell a story, too. Each medium enables different kinds of stories.

We utilize all of these, but we spend most of our time inside software systems. That is a medium that lends itself to new opportunities. Most of our stories are executable and visual and are created and delivered right in the development environment. We want our system to tell a story. And we want these stories to be told at all level of abstractions.

Let’s look at some examples. We start from a single object. In this case, we see a price object constructed by the code on the left being inspected through a custom view on the right. The view tells a story of how the object is formed in a way that a non-technical person can relate to.

Once we can show individual objects in interesting ways, we can weave these representations into larger narratives. For example, take a look below at an example of how we went about documenting the implementation of an algorithm: the environment allows us to explore the detection of a neighbor in a circular treemap through views specific for the different steps in the algorithm.

And here is another similar example documenting another algorithm. This time, we assemble the object views into a larger narrative presented in a live document.

We went through thousands of scenarios like these, and we learnt that every piece of our systems hides at least one exciting story, regardless of how technical and dry appears at first sight. But, these stories are more than just excitement. They are engineering tools, too, as they enable us to reason with ease about otherwise impenetrable problems.

Just consider this story of a bug, a highly technical low-level bug, about how when pressing certain keys on a real keyboard (bottom left) we get a wrong signal internally (top right). The video depicts rather literally how we approached the debugging of the problem. We built a whole tool just to be able to reproduce this problem in a way that was pleasant for our brain. This then allowed us to tackle the problem effectively. And then, perhaps unexpectedly, it ended up as a tweet and became a marketing tool, too.

feenk on Twitter

When #gtoolkit is embedded in the intertwined logic of Morphic, pressing character [ works fine, but pressing Shift+[ is translated into Shift+LeftArrow. How do we know? Take a look. Having bugs is annoying, but working on them does not have to be. https://t.co/fhhqWbhf3T

A good story can make us both be productive and smile at the same time. It can be both an engineering tool and a marketing tool at the same time.

Storytelling is a powerful skill. Like any skill, it can be learnt. We make it a focus to train it, and we optimize all our work around it. By doing so, we find that most other things are secondary.

No “I am sorry”. Just “thank you”

We do have one rule though: we can only say “I am sorry” if we hurt someone, but we can say “thank you” as many times as we want.

We trust each other to do the best we can, and we assume mistakes are normal, be it in code or otherwise. As such, making a mistake should be associated with focusing on finding a solution, and not with being sorry for what could have been different.

At the same time, whenever we do get something we enjoy, we like to celebrate it. We noticed that a “thank you” does wonders, so we use that abundantly.

Parting remarks

This article does not put forward a recipe. It also does not describe an ideal solution. This is merely our story with its specific caveats.

As we develop the tools we use, we are our first customers, and this puts us in the strong position of understanding the domain rather deeply.

We are not a large team. Although we were told before that this approach will not work beyond a handful of people, 11 people still does not constitute a large team. Then again, many larger companies are made up of teams around this size.

We work for the long term. We never promise a specific outcome until it’s done. As such, we do not have to run after promises. Others might not be in this position, although we do believe that one should avoid depending on promises, especially in our field and especially when the promises are not made by the team.

In any case, this approach worked for us and for our values. We wanted to have a resilient organization that can mold around the individual. We did not follow a rule book. We borrowed some pieces from others, such as working remotely or focusing on the team members first, but we wrote our book the way it worked for us.

Resilience is hard to judge in normal conditions. In fact, resilience comes at an apparent cost whose utility is not obvious in normal times. Resilience pays off when you meet an unusual event. The current crisis is such an event. We noticed that while there were many changes for the individuals, there was little change for the organization. And we learnt that even in these times, there needs not be a tradeoff between individual well-being and group productive work.

#remote, #nomeetings, #noestimation, #nobacklog. #storytelling was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.

A software environmentalism story

A few days ago, the governor of New Jersey called explicitly for COBOL programmers to help with the development of the unemployment system during the COVID-19 crisis:

Jim Manico on Twitter

The governor of New Jersey just put out the call on live TV that he is desperate for Cobol programmers right now.

This is how the software environmentalism crisis looks like.

In the software development industry, we focus almost exclusively on building systems. The conversations are dominated by new languages, new frameworks, new deployment options. And the body of software grows exponentially. At the same time, we are unable to remove old systems.

From this perspective, our software industry behaves not unlike the plastic industry: we focus on building without any regards to recyclability. That is not sustainable. And, given the breadth of software’s reach, it is not responsible either.

The call for COBOL programmers by the governor of New Jersey takes people by surprise, including the governor himself who said that “There’ll be lots of postmortems and one of them on our list will be how did we get here where we literally needed COBOL programmers.”

And the apparent sudden growth of the #COBOL tag popularity is reason for surprise as well:

michael_nielsen on Twitter

Unexpected

After all, we associate a popular tag to be trendy, and COBOL is certainly not considered cool. But, COBOL is merely a symptom. The underlying problem is systemic. COBOL is not a problem per se. The problem is that we associate COBOL with extinction, while it is still one of the most used programming languages in the world.

We might think that changing a system is just a matter of will. Interestingly, in the case of the New Jersey unemployment system, there were many signs. For example, take a look at this one:

Joseph Steinberg on Twitter

COVID-19 Response: New Jersey Urgently Needs COBOL Programmers (Yes, You Read That Correctly) https://t.co/irjdR8eyCc #COBOL #NewJersey #Covid_19 #COVID #CoronavirusUSA #Coronavirustruth #News #NJ #USA #TechNews #CyberSecurity #InfoSec #Unemployment #CoronavirusPandemic #NYC

I would be surprised if the people that manage this essential system did not know that depending on a technology considered outdated poses a great risk. I’d also be surprised if they didn’t try to address it, and fail for one reason or another.

In fact, I am seeing this with every company we interact with. Everyone seems to have at least one large skeleton in a closet right at the core of the business that they simply cannot get rid of.

When I ask people at various conferences if their company relies on at least one mainframe system, the large majority does. This is not a local problem. It’s a systemic one that requires a systematic rethinking of our work.

We need to change the narrative. Recyclability must be embedded into how we build our systems.

The term “software” was born to contrast “hardware”. At that time, hardware was as large as a building and software was easy to replace. We now live in a different world: hardware fits in my pocket, while we are drowning in software we cannot move.

When developing systems, people often focus on current or immediate functionality. That is shortsighted and dangerous. Functionality is indeed a business asset, but the ability to adapt to changes in the surrounding environment depends on the non-functional parts. The New Jersey unemployment system did its job just fine until recently. Its functionality did not deteriorate. It was the environment that changed and this made the system is no longer adequate. It now needs to be recycled to match the new reality.

To recycle a system implies taking it apart and refurbishing it for other purposes. However, before we can take it apart, we first have to understand the parts.

Interestingly, the activity of understanding a system happens to be the single largest expense we have, with developers alone spending 50% or more of their time reading code. Yet, nobody talks about how this time is spent. That’s the first thing we should change.

We need to start talking about how we read code. Better yet, how to not read code.

Developers read code because they want to understand enough to know what to do next. This is called decision making. Given that we spend most of our energy doing that, it would be more interesting to define our work as being primarily a decision making one rather than construction one.

As the goal is decision making, reading is just the strategy through which we extract information out of our system. It also happens to be the most manual and error prone strategy.

When I ask technical people whether they have pictures about their systems, everyone says yes. But, when I ask whether they believe those pictures are 100% accurate, most people say no. That’s because the pictures were created manually.

Tudor Girba on Twitter

Do you have pictures about the inner workings or structure of your existing software system? If yes, how are they constructed?

A picture or report about a system exhibits either a wish (when the picture depicts a plan) or a belief (when the picture depicts what we think the system is). Both are essential for humans, but neither should be used as basis for decision making.

Decision making is hard enough as it is. Often uncertainties in the environment require probabilistic thinking. However, the reality of our own system should not be a probability. The reality of our system must become a certainty.

The sheer size of our systems far exceeds the ability of humans to grasp without summarization. A picture does tell a thousand words, but those words must be accurate.

Now, accurate pictures are still not enough. For a picture to be effective, it must be specific to the question we want to answer.

Software systems are likely the most complicated structures we have ever created as humans. To this end, it is impossible to predict specific questions. We can predict classes of questions but not specific ones.

That is why pictures have to be created after the question was made explicit. To make this happen the ability to craft a picture must reside inside the team. Given its impact, this ability is not merely a technical one. It has high level business strategic importance.

A good picture or narrative can make humans grasp a data problem orders of magnitude faster than without one. A decade ago this was less obvious, but by now we have ample evidence from all the efforts around data science and the impact it has on business decisions.

As everything about our systems is data, including code, the same principle applies to software development, too. We just have to think of our job through this new lens. And the good news is that this is less difficult than expected.

Just think of this: most systems take data, model it, manipulate it and present it to users that make decisions fast without ever seeing the raw data. So, we, software people, possess this magic skill of transforming data into information to speed up decision making. The skill is already present inside our teams. We just have to apply it to our own work!

Every time developers read, they imply that the only way to answer the question is manually going through raw data. However, through our work and that of others, we showed that most software assessment is rather decomposable as a search or visualization problem.

That might sound academic, but it’s really not. Consider this: when a developer address a data problem in the database, she first writes a query. When the same developer addresses a problem in code, she starts scrolling. Both are data problems. Only the mental models are different.

How do we change this?

We should listen to what Marshall McLuhan and John Culkin told us many decades ago about the nature of our tools. We should observe that there is a deep correlation between the interface of our tools, and the mental models they induce. A database tool has a big query box at the top. People query. A typical development environment has a giant text editor at the center. People scroll and read.

We shape our tools, and thereafter our tools shape us. We should carefully choose the characteristics of our tools because they dictate how we think. Our tools, and not only our languages, are essential. The stakes are too high to disregard this.

How would this look like in practice? Consider a case study of reasoning about evaluating @Deprecated classes in a Java system. In our experiments, we ask developers to estimate the time it would take to go over a given system to find which deprecated classes they can remove and what to do with the ones that cannot be removed right now. For that specific system, comprising of 2500 classes of which 25 are deprecated, the average estimation is 4h, the minimum being 2h.

We picked this problem because this is a rather basic evolutionary problem about which developers often learn in introductory courses. This should be a highly optimized problem that we could not optimize meaningfully any further.

We show how the same problem can be addressed better and more accurately in a matter of minutes. Minutes. Literally.

Here is a description of how that works:

Evaluating @Deprecated classes in Java systems with Glamorous Toolkit

This is an order of magnitude difference when dealing with a simple scenario. The difference is correlated with the nature of the tools. We built a new kind of an environment precisely so that we can think differently. And this difference only grows as the problems get larger.

But, the tool is not the interesting part. The specific tool used is relevant to show what is possible. The most interesting part is that people can understand what the tool conceptually does in minutes, even when they’ve never done any software analysis before! There is no conceptual learning curve because developers have already done the same thing many times, only on other data. This ability is essential and it is reusable.

Tools are essential in software development. The key characteristic is that they should be customizable inexpensively while working on the problem. They should be moldable.

Every time you replace manual work with automation, the real value is extracted when you change the business processes. The same applies here. We have seen historically that when tests became code, we shifted towards various forms of test-driven development. When infrastructure became code, we reshaped companies around DevOps.

Tools must become code, too. And this leads to moldable development, a way of guiding development through the lenses of custom tools.

This article is already way too long. Does it cover the whole space? No. Do we have all the answers? Certainly not. But, we do have the beginning of the conversation. And this is a conversation we must have.

Our kids will only know a world based on software. We already messed up the real world for them, and that is certainly something we must address. Still, while addressing the real world, we should not create a larger problem.

The New Jersey unemployment system is but a case that happened to gain media attention. But, the problem is systemic. We are drowning in systems we cannot move.

The software world is in a crisis of environmental proportions, too. But, that is addressable. And like with the real world problem, we must see it for what it is and make it an explicit subject of conversation.

This article is based on a Twitter thread I wrote not long ago:

Tudor Girba on Twitter

The governor of New Jersey called explicitly for COBOL programmers to help with the development of the unemployment system: https://t.co/7MVEhynS7k This is how the software environmentalism crisis looks like. An explanatory thread. 1/

Why the governor of New Jersey calls for COBOL programmers was originally published in feenk on Medium, where people are continuing the conversation by highlighting and responding to this story.