Previously I described a process for developing software while trying to provide as much value and create understanding as early as possible.

Building with Value

How would one iteration of this process look like?

I’ll walk you through one iteration using the Go programming language while deliberately showing the steps more explicitly. In practice, many of these steps could be replaced by some external tools or libraries.

I’ll continue with the example from the previous post, an Issue Tracker.

The Simplified Process

1. Structure Follows Value: find out what is the most valuable thing to implement right now.
2. Spiking: we create a draft of 2–4 changes.
3. Gradual Stiffening: build these changes incrementally,
4. Cleanup: observe the wholeness and try to notice any small mistakes.

Structure Follows Value

“What provides most Value in an Issue Tracker that we can implement right now?”

We may notice the essential pieces from the language used.

• I need to get an overview of the issues.
• I need to find information about this issue.
• I don’t have good visibility on issue progress.

We may also get this information from a domain expert. We also should try to understand why exactly it is important and how it fits into the rest of the system:

• It shows what we are doing.
• It shows what we have done.
• It shows what we still need to do.

Issue is definitely a valuable part. Hence we must capture it in code.

Spiking

To ensure that we can adequately represent the complexity of an Issue, we should create a package for it. That way, we make it significant and clear. At the same time, we create a locus of attention which allows to understand and examine the feature wholly and whether it is complete.

Of course, packages are not the only way to group things — there are also classes, functions, methods, constraints, etc. What you use to represent the idea will depend on how large, detailed, or important the contained thing is.

It is better to start with a notch larger grouping than is needed, and it is not difficult to make it smaller. However, the reverse, moving from smaller container to larger, is usually more complicated. e.g. Going from multiple packages to a single package vs. going from a single package to multiple.

We’ll add our first Spike to issue/info.go:

package issue

type ID int
type Status string

const (
    Created Status = "Created"
    Closed         = "Closed"
)

type Info struct {
    ID      ID
    Caption string
    Desc    string
    Status  Status
}

We must have an issue.ID to identify an issue uniquely. Each issue usually has an issue.Status associated with it. We need something to bring all the attributes together like issue.Info. Keep in mind that we are sketching the code and are not committed to this structure. We are not looking perfection, but rather a global view of how things will work together.

Notice that I don’t use long names such as IssueStatus, IssueInfo because the package for them already contains issue.

We also need some way to store and load those issues. The way we store and load them can change. Hence we should abstract this knowledge away. We create an interface issue.Manager for it. We put it into issue/manager.go:

type Manager interface {
    Create(info Info) (ID, error)
    Load(id ID) (Info, error)
    Close(id ID) (error)
    List() (issues []Info, error)
}

To get an overview of how we will use it, we write some usage code into main.go:

package main

import (
    "fmt"

"example.com/tracker/issue"
)

func check(err error) { // TODO: only for draft
    if err != nil {
        panic(err)
    }
}

func main() {
    var manager issue.Manager

id, err := manager.Create(issue.Info{
        Caption: "Hello",
        Desc: "World",
        Status: issue.Created,
    })
    check(err)

info, err := manager.Load(id)
    check(err)

fmt.Println(info)

infos, err := manager.List()
    check(err)
    fmt.Println(infos)
}

It doesn’t matter in which order you create these pieces, and sometimes it is easier to create the usage code first, other times it is easier to create the implementation first. The thing that does matter is that both exist to ensure that we have the implementation details right and that we can integrate it with the rest of the code.

The usage code can also be sketched as a test; this depends on how the sketched code will be used, how it needs to integrate with the rest of the system and other factors.

Gradual Stiffening

Notice that we don’t have any runnable code yet, it’s fine, because until now we were trying to grasp what we are implementing and that all the pieces work together as intended.

Now we will step-by-step start to flesh out the actual structure, until we have a reliable and stable runnable code. We are in the beginning stages of our project, so there isn’t much to worry about. We should skim over our code and notice anything that doesn’t feel nice.

The first thing we may notice is issue.Created. What would info.Status == issue.Created mean? This suggests that we haven’t captured the intent as well as we should have. Let’s refine our sketch, info.Status == issue.Open sounds much better. Hence we change issue/info.go:

const (
    Open Status = "Open"
    Done        = "Done"
)

In main.go the manager doesn’t feel clear; it feels like a fuzzy concept without specific meaning. There probably will be more things that need to “manage” things. Is there a better name for it?

What does the manager do? “It manages and tracks issues.” Here is a clue for a nicer name Tracker. We shall refine issue/manager.go into issue/tracker.go and change:

type Tracker interface {
    Create(info Info) (ID, error)
    Load(id ID) (Info, error)
    Close(id ID) (error)
    List() (issues []Info, error)
}

We also make all the necessary adjustments to main.go. At the end of this, we should have code that compiles however, it’s okay if it is not yet completely bug-free. We will do this in the next step, but gradual stiffening and cleanup are usually finished with a final cleanup pass.

Cleanup

Now we have a good idea about the feature and how to put it into code, we shall go over and fill in all the missing details and ensure that we have comments and a few tests and can use it in some form.

Here we could add an in-memory implementation of the tracker and then write some tests for it.

The other thing what we want to do here is to make it easier to understand and ensure whether the code behaves as it should. In most cases, you would wish to unit or behavior test, but they are not the only way. You could also write property tests. Or write output that could be verified by hand, if the correct behavior is difficult to describe in code.

Few interesting bits to pay attention to while solidifying code. When you come across questions, mark them as such. For example, while writing the tracker test case, I made a mistake while writing:

tracker.Close(id)
// ...
expect := Info{
    ID:      id,
    Caption: "Caption",
    Desc:    "Desc",
    Status:  Closed,     // compilation error, should be Done
}

I mixed up two things: the method is called Close , and the resulting status is Done. Because I made a mistake while writing this, it suggests to me that the code is not clear enough, but I’m not sure how to improve it. It probably isn’t that important, so I’ll mark it as a TODO and move on to other things:

const (
    // TODO: should tracker.Create renamed to "Open"
    Open = Status("Open") // Open means that issue is in progress
    // TODO: should "Done" be renamed to "Closed"
    Done = Status("Done") // Done means that issue is completed and delivered
)

I could try to figure out this immediately, but I really don’t think I have the necessary information right now and I will probably find out the details while implementing other things. In a similar vein, we might reconsider Tracker method names.

Now we have captured something of value in the code. There is plenty of things we can do to improve the ease of use that code, but we have something that someone would like to use already.

Using this process helps to keep focus on what matters and at the same time bring together a bigger picture.

Thanks to Taavi Kivisik for review feedback.

Beginners often overwhelm their brain with decisions and things they need to do. Which usually turns into a mess of trying to implement everything at once. Alternatively, programmers end up making very short-sighted decisions that end up becoming problematic later.

The most important part of programming is that you cannot do everything at once. The human mind is very limited in how much it can process at a time. To combat this limitation, we need to build things piece-by-piece to ensure that everything will fit together nicely.

Speaking as a builder, if you start something, you must have a vision of the thing which arises from your instinct about preserving and enhancing what is there… If you’re working correctly, the feeling doesn’t wander about.

 — Christopher Alexander

The question is — how do we pick the pieces we develop? How do we ensure that our mind and feeling doesn’t wander around?

Understanding Value

I covered some of the aspects of understanding value already in Value in Software.

Value in Software

When we are starting out writing some software, there is so much we could implement. There are also so many ways we could organize and implement things. This overwhelming amount of concerns can quickly push people into Analysis Paralysis.

One approach would be to start from what is easy to implement; however, if we do that, we may implement many things that are not valuable or even worse, not get to implementing what is valuable at all.

Let’s take an Issue Tracker as our example to investigate the problems.

Let’s say we start by implementing login to the system as the first thing. As discussed in another article, it’s not that valuable to people. As a programmer, you might think — I’ve done this many times, it’s easy, and I have a lot of necessary code available already. No doubt, you get something that looks nice and can interact with the page, but what are the downsides?

For one, we would need to login to the site before we can interact with the actual product; this slows down every step of our development. Similarly, our code base will end up more substantial. Hence, we have more distractions and more parts that could break. We also might get new requests for login, e.g., password recovery, user management, user groups. Triaging extra features sinks time from other vital parts of the code.

We could also start implementing the end goal. For issue tracking, this might be managing large projects with several teams. However, a big goal can take too much time before it can become practically useful.

Implementing larger things has an even more significant drawback: we make larger changes and hence we are likely to miss many mistakes.

Yet, to date, there is little recognition of the following commonsense point: If indeed the programs are so complex, then it is likely that they, too, will be potentially subject to hundreds of thousands, perhaps millions of egregious mistakes of adaption. Here I am not only talking about “bugs” — failures which stop a program from running altogether. I am talking about mistakes of adaptation, ways in which the program fails to do what it is supposed to do, fails to meet the needs of the people who use it, or is more awkward, more annoying, less useful, than it is supposed to be.

- Christopher Alexander, “The Process of Creating Life”

How do we avoid making mistakes in the Valuable parts of the system and at the same time, be as valuable as possible?

It somehow seems obvious and not-so-obvious at the same time to start with the most valuable thing we can complete right now.

For an issue tracker, the most valuable thing we can quickly complete is the Issue. We can detect the importance already from our conversation about the product. To do anything in the system, we need to be able to manage Issues. The rest of the system has no significant value without it.

After implementing the “Issue” as the first thing, we can already start using it in code and people may start getting value from it.

You might be asking, “What about Projects?” It’s quite easy to start multiple instances of the same server and track them on different URL-s.

Alternatively, “What about login?”. If it’s only for internal use, we could put it up on our intranet and actually trust your coworkers. If we really need outside access, we can add a proxy for authentication.

Just to clarify, I don’t think avoiding “login system” makes a huge difference in the project, however, it is a good example.

There are also other benefits we gain from putting Valuable things first.

We start to learn about them much sooner and understand how they fit together better, without being distracted by less important things. Maintaining Valuable parts gives us more insight into how it might provide even more value.

The code usually will reflect this as well. Since these are the things we implement first, the code will be cleaner and contain fewer bugs. Even more, the developers need to always think about why they are doing things, rather than crunching out features.

It’s important to figure out what is more valuable than something else; otherwise, we may end up back with Analysis Paralysis. Even, if we later discover we were wrong, it’s essential for figuring out what to do.

The Simplified Process

Christopher Alexander in “The Process of Creating Life” describes a fundamental process, how life and things we consider living, evolve. The process of creating complex structures, like software, does not differ.

However, it’s hard to describe “The Fundamental Differentiating Process” without also describing the foundations of Centers and Structure-Preserving Transformations, which in itself would take several articles.

The following is a simplification of the “Fundamental Process” combined with concepts from “The Timeless Way of Building”. It loses some of its generality and beauty in favor of pragmatic concerns. While the following is very useful, it’s not the whole story.

Structure Follows Value

We dedicated a lot of time explaining what Value is and why it is important. To ensure that we build something of value, we need to understand the value it provides and how it provides it. The better we can distinguish value, the better we can make decisions.

Decomposition

When we roughly know what we are building, then we can find the next step by trying to eliminating things. We can visualize the project as a graph of dependencies and associations:

We ignore things we cannot implement right now, can mock out, can achieve with other means or live without. Implementing data repositories don’t make sense without actually knowing what we need to store so that we can cross them out:

Since Projects without Issues are less valuable than Issues themselves, we can cross it out.

Since we can use Issues without Users and tracking Issues are more valuable than tracking Users, we can cross out users.

Since View requires the existence of an Issue, we can also cross it out.

Of course, different people may reach different conclusions on what is the most valuable thing, but it’s more consistent than inconsistent. If you are in doubt, ask another person what would they rather have “Issues” or “Issues View.”

Sometimes the value is so similar that it’s quite impossible to break the tie, but it is critical to do it nevertheless. You could pick the one that is the least understood, the one that is easier to implement or roll a die. If we do not create this ordering, we can end up overwhelming ourselves.

Enhancement

A slightly better approach is to do this in reverse and think of things in terms of unfolding. We ask ourselves which change would enhance the system wholeness and value the most.

This approach is more difficult to explain and to do, but it also works when we don’t exactly know what we are building. This approach requires feeling and a good understanding of the wholeness of the system.

You might think that when you start a new project, there is no system to enhance, but there always is. There’s always a reason you are implementing the project. In an Issue Tracker, there exist people, who want to use the system, and want to get an overview of what is happening. There are also many ideas on how things should work. World, People, and Ideas are a part of the system like any piece of code you write.

Spiking

Implementing ideas one-by-one would eventually cause problems since we might make mistakes from integrating with the rest of the system. We also shouldn’t try to implement too much, since our working memory cannot handle it.

If we try to break the previously discovered “Value” into 2–4 changes, we minimize the chance of the changes fitting into our working memory. We outline these changes as rough drafts called Spikes. For Spikes, you can use method declarations, type definitions, comments, or anything else that helps to get a feel for how things fit together. We could even do the spiking on a whiteboard or CRC cards.

Spiking in software development has its roots in Extreme Programming for getting a feel for and understanding whether we are on the right track and notice potential problems. The spiking idea comes from architecture, where architects would put spikes into the ground to get a better feeling of the building and the environment.

Gradual Stiffening & Cleanup

Once the spikes roughly fit and integrate with the rest of the system, we can start making the draft changes. We begin to implement methods, rough out testing, update documents, and so on.

While making these 2–4 changes, we also need to pay close attention to how things fit together and have a keen eye on how the process flows.

Recognize that you are not assembling a building from components like an erector set, but that you are instead weaving a structure which starts out globally complete, but flimsy; then gradually making it stiffer but still rather flimsy; and only finally making it completely stiff and strong. We believe that in our own time, the most natural version of this process is to put up a shell of sheet materials, and then make it fully strong by filling it with a compressive fill.

- “A Pattern Language” by Christopher Alexander

If we use a wrong type name and code doesn’t compile, it can be an important signal that the type could have a better name. We need to have an awareness of the whole.

This could even result in us moving our spikes or going back to evaluate what is the most valuable thing we are doing. If you are unsure, take notes they help remind where things can be further improved.

Gradual Stiffening and Cleanup happen together. One is about improving things, and the other is about observing the wholeness and fixing small mistakes.

Putting it all together

It might not be easy to comprehend this without practical examples, but the best analogy might be Drawing.

You don’t start drawing like a matrix printer; starting from the top-left corner and proceed to the right side and then the next line.

More often, the process looks like:

1. First, you figure out what you are drawing and what are the things you are going to draw (structure follows value) and the purpose of the drawing.
2. Then you do a few rough sketches until you see that the picture will work nicely (spiking).
3. Then you start to refine the sketches and draw more delicately (gradual stiffening).
4. Finally, you clean all the sketch lines and color it (cleanup).

The basic process of programming is the same. The process also may unfold recursively on different levels of abstraction. In a company, you will have multiple people doing similar processes in parallel.

This skill of being aware of value and wholeness takes time to develop, but the better you get at it, the more fluid and faster the process becomes.

In the next article, we’ll take a look at how one iteration of this process might look like:

Building with Value: An Example

Why do we write programs in the first place? It’s easy to get lost in the technological details, but this is not the primary goal. This might sound obvious but, we need to provide value to people.

There is a central quality which is the root criterion of life and spirit in a man, a town, a building, or a wilderness. This quality is objective and precise, but it cannot be named.

- Christopher Alexander, “The Timeless Way of Building”

Understanding Value helps us understand why we are doing things in the first place. If you are not creating something of value, then you are creating waste.

Valuable

To understand Value, we must first understand why do we call things Valuable.

In the simplest terms, when X is valuable to someone Y, it means, that X has importance to Y, it can bring benefit to Y in some way.

The definition is not rocket science:

• the internet is valuable because it can connect people to information with minimal cost,
• a keyboard is valuable because it allows interacting with the computer and
• a text editor is valuable because it allows to change and adjust the text with ease.

We know and feel when something is valuable to us, and we can empathize when something is valuable to someone else.

The Context

However, understanding Value is not that easy. We must always bear in mind that the value of something possesses is tied to the context.

• If there isn’t information on the internet, then it’s not as valuable.
• If there is no computer for the keyboard, then the keyboard is not that useful.
• If you have nothing to write, then a text editor is not valuable.

The previous is all common sense, but these examinations help us examine more interesting questions. Let’s take something more challenging.

Does a ‘login system’ have Value? The first reaction would be to say, “Yes, of course it does.?”

But, have you ever gone home and said: “I got so much value from logging into the system?” I doubt it. Should we conclude that “No, of course logins have no value, they are only an annoyance?”

As you may have already guessed, neither answer is correct.

- paraphrased example from James Coplien

The goal of “logging in” is to protect value, rather than to be of value. If there is no login screen for a system that you don’t use, then it is not of importance to you. When it’s an online bank login, the login is as valuable to you, as how much money you have on your accounts. For the banks, the login system is very important, although it may not be of any explicit value.

It’s never as simple as “Does something have value?” because we need to take the context into account.

Evaluator

You should also have noticed that Value will also depend on who is doing the evaluating. We have to keep in mind that there is always an evaluator when looking at value.

When creating things, we should always look through the eyes of the person benefiting from the Value. In the value stream, there can be many stakeholders who can benefit from it. Let’s say we are building a wooden box.

• In the simplest case, we are providing value for the end-user. You make a wooden box, and the end-user uses it.
• We could provide value to some other person in your team. You cut a few boards and give them to another person to assemble. The end-user doesn’t care about those boards, and they are not valuable to him.
• We could provide value to ourselves. I could make a jig that allows me to cut boards faster and with less effort. However, neither the person doing assembling the box or the end-user using the box, will care about the jig. But, they do care about the faster cutting.

We shouldn’t restrict our minds to creating boxes, but instead see the bigger picture.

How is the box valuable to the end-user, how will they use it? Does he hold very lightweight things in it — this means we can make the box sides thinner and reduce waste. Does he need to store different things, maybe we can add separators into the box?

Understanding what we create and why it is valuable to people is essential to making the right decisions.

Outside the box

We should have a bigger view and understand that we can get the same value from multiple solutions. Engineering is about understanding these different solutions and picking the most appropriate one.

When we look at the “login system” case, if the data can be public, then we won’t need a login. If we are only using it on our computer or intranet, then there is no login necessary.

Look for multiple solutions, and you can avoid a lot of work this way. Even better, you might end up with a product that is more valuable overall.

Understanding Value should be at the core of our decision making. It helps to give us clarity about why we are doing something and not doing something else. Always understand how value is provided for the end-user. Otherwise, we are bound to build something unnecessary.

In the next blog post, we’ll take a look at how we put this understanding into practice.

Building with Value

Let’s take a look at a simple example:

func BenchmarkReport(b *testing.B) {
    runtime.GC()
    for i := 0; i < b.N; i++ {
        r := fmt.Sprintf("hello, world %d", 123)
        runtime.KeepAlive(r)
    }
}

Running go test -bench . will give us the result:

BenchmarkReport-32      20000000               107 ns/op

This might be enough for getting a rough estimate where you stand in terms of performance, but a more detailed output is required for optimization. Condensing everything into a single number is bound to be simplistic.

github.com/loov/hrtime

Let me introduce you to the hrtime package which I wrote to get a more extensive output from benchmarks.

Histogram

The first approach to using it is to use hrtime.NewBenchmark. Rewriting the first example as a regular program.

func main() {
    bench := hrtime.NewBenchmark(20000000)
    for bench.Next() {
        r := fmt.Sprintf("hello, world %d", 123)
        runtime.KeepAlive(r)
    }
    fmt.Println(bench.Histogram(10))
}

Which will print:

avg 372ns;  min 300ns;  p50 400ns;  max 295µs;
  p90 400ns;  p99 500ns;  p999 1.8µs;  p9999 4.3µs;
      300ns [ 7332554] ███████████████████████
      400ns [12535735] ████████████████████████████████████████
      600ns [   18955]
      800ns [    2322]
        1µs [   20413]
      1.2µs [   34854]
      1.4µs [   25096]
      1.6µs [   10009]
      1.8µs [    4688]
        2µs+[   15374]

As we can see that P99 is 500ns, which means that 1% of all our measurements are above 500ns. We can try to optimize this by allocating fewer strings:

func main() {
    bench := hrtime.NewBenchmark(20000000)
    var back [1024]byte
    for bench.Next() {
        buffer := back[:0]
        buffer = append(buffer, []byte("hello, world ")...)
        buffer = strconv.AppendInt(buffer, 123, 10)
        runtime.KeepAlive(buffer)
    }
    fmt.Println(bench.Histogram(10))
}

And the result:

avg 267ns;  min 200ns;  p50 300ns;  max 216µs;
  p90 300ns;  p99 300ns;  p999 1.1µs;  p9999 3.6µs;
      200ns [ 7211285] ██████████████████████▌
      300ns [12658260] ████████████████████████████████████████
      400ns [   81076]
      500ns [    3226]
      600ns [     343]
      700ns [     136]
      800ns [     729]
      900ns [    8108]
        1µs [   15436]
      1.1µs+[   21401]

We can now see that the 99% has gone from 500ns to 300ns.

If you have a keen eye, you may have noticed that Go benchmark gave an average 107ns/op however hrtime gave us 372ns/op. This is the unfortunate side-effect in trying to measure more — it always has an overhead. The final results include this overhead.

Note: depending on the operating-system the overhead can be significantly less and hrtime does support repeated calls inside with https://godoc.org/github.com/loov/hrtime#Histogram.Divide.

Stopwatch

Sometimes you also want to measure concurrent operations. For this there is Stopwatch . https://godoc.org/github.com/loov/hrtime#Stopwatch

Let’s say you want to measure how long does a send take on a highly contended channel. This, of course, is a contrived example, but roughly shows the idea of how we can start the measurement from one goroutine, stop it in another and finally print it all out.

func main() {
    const numberOfExperiments = 1000
    bench := hrtime.NewStopwatch(numberOfExperiments)

ch := make(chan int32, 10)
    wait := make(chan struct{})

// start senders
    for i := 0; i < numberOfExperiments; i++ {
        go func() {
            <-wait
            ch <- bench.Start()
        }()
    }

// start one receiver
    go func() {
        for lap := range ch {
            bench.Stop(lap)
        }
    }()

// wait for all goroutines to be created
    time.Sleep(time.Second)
    // release all goroutines at the same time
    close(wait)

// wait for all measurements to be completed
    bench.Wait()
    fmt.Println(bench.Histogram(10))
}

hrtesting

Of course, writing a separate binary for all of the tests isn’t that convenient. For that there’s github.com/loov/hrtime/hrtesting, which provides wrappers for testing.B.

func BenchmarkReport(b *testing.B) {
    bench := hrtesting.NewBenchmark(b)
    defer bench.Report()

for bench.Next() {
        r := fmt.Sprintf("hello, world %d", 123)
        runtime.KeepAlive(r)
    }
}

It will print out 50%, 90%, and 99% percentiles.

BenchmarkReport-32               3000000               427 ns/op
--- BENCH: BenchmarkReport-32
    benchmark_old.go:11: 24.5µs₅₀ 24.5µs₉₀ 24.5µs₉₉ N=1
    benchmark_old.go:11:  400ns₅₀  500ns₉₀ 12.8µs₉₉ N=100
    benchmark_old.go:11:  400ns₅₀  500ns₉₀  500ns₉₉ N=10000
    benchmark_old.go:11:  400ns₅₀  500ns₉₀  600ns₉₉ N=1000000
    benchmark_old.go:11:  400ns₅₀  500ns₉₀  500ns₉₉ N=3000000

Unfortunately, with Go 1.12 it will print all of the runs of Benchmark instead of just the last. However, with Go 1.13, the output will be much nicer:

BenchmarkReport-32   3174566  379 ns/op  400 ns/p50  400 ns/p90 ...

And comparing results with benchstat will work as well.

hrplot

Leaving the best for last github.com/loov/hrtime/hrplot. Using my experimental plotting package, I decided to add a convenient way to plot the benchmark results.

func BenchmarkReport(b *testing.B) {
    bench := hrtesting.NewBenchmark(b)
    defer bench.Report()
    defer hrplot.All("all.svg", bench)
    
    runtime.GC()
    for bench.Next() {
        r := fmt.Sprintf("hello, world %d", 123)
        runtime.KeepAlive(r)
    }
}

Will create an SVG file called all.svg. It contains a line plot, which shows how much time each iteration took. Second is a density plot, which shows the distribution of timing measurements. And finally a percentiles plot.

Conclusion

Performance optimization is fun, but having tools to help you out makes it even more enjoyable.

Go try out github.com/loov/hrtime and let me know what you think.

I often get asked for recommendations. Here is a list of books and resources I vouch for because I’ve found them invaluable in my journey as a software engineer or have seen them greatly help others.

If you’ve never coded before, then my main recommendation is:

Learn Code The Hard Way course by Zed Shaw is one of those great starting places when you have never seen code before. It has plenty of good tips, tricks and exercises to get you started.

For little ones, playing around with Scratch and Sonic Pi is also great.

A few recommendations specifically for Go:

Go Training resources by Bill Kennedy is a nice introduction to programming in general and what is a good mental model for programming in Go. He covers a variety of topics and it is all free.

Web Development With Go course by Jon Calhoun is a comprehensive guide into writing a production ready web application.

The Go Programming Language book by Alan Donovan and Brian Kernighan.

After you have written your first if statements, loops and can manage a 1000 line program.

A Philosophy of Software Design by John Ousterhout is one of those books that helps you get started in moving the right direction for software development mastery. He guides you to find balance in the things you do using clear examples. His list of warning signs helps you notice when things have gone wrong.

The Practice of Programming by Brian Kernighan and Rob Pike is an elegant book which covers a variety of topics from comments to testing and performance.

The Pragmatic Programmer by Andrew Hunt and David Thomas is a step towards looking at the bigger picture of software craftsmanship, not just focusing on the coding.

Once you grasp basics you should continue learning by writing a bigger project — a few packages and around 10,000 lines of code. Otherwise, the following reading might be hard to relate to.

On the Criteria To Be Used in Decomposing Systems into Modules by David Parnas is a classic that will probably never age. It discusses main concerns when making software modular. It is a great introduction into software architecture.

Physics of Software by Carlo Pescio is a hidden gem that hones your intuition on how software evolves and how different pieces interact. Although these concepts may seem very abstract, I use them frequently.

Lean Architecture: for Agile Software Development by James Coplien and Gertrud Bjørnvig clearly reminds us to not start from technology, but rather from focusing on creating value and on people. In the long run it leads to software that adds more value and expresses ideas clearer.

Domain Driven Design by Eric Evans was interesting to me mainly with regards to how to design and figure out the domain part of the software and communicate it clearly.

Working Effectively with Legacy Code by Michael Feathers has sound advice on how to maintain software. I’ve learned so much from refactoring and making things clearer. Maybe even more than from writing things from scratch. Some day, all successful codebases become legacy, so learning how to manage them is essential.

Design Thinking by Dave West and Rebecca Rikner goes into fundamental perspectives, principles, concepts and practices that make design better and more insightful.

An Introduction to General Systems Thinking by Gerald Weinberg is an introduction on how to think about complex and chaotic problems where it is impossible to isolate the parts.

For performance and algorithms my favourites are:

The Algorithm Design Manual by Steven Skiena is my favorite algorithmics book. I especially recommend reading the War Stories where it makes clearer how people invent algorithms. It’s not a spark of insight, but rather a gradual improvement over days, months and years.

Latency Numbers Every Programmer Should Know is one of those tables that you should have roughly memorized for back-of-the-envelope calculations. In general, the skill of estimating is very useful in making smart decisions about performance.

Data Oriented Design by Richard Fabian offers a deep dive into using your computer clock cycles efficiently.

Algorithm Engineering goes in depth in all the different parts of designing algorithms (e.g. how to design, derive and test them properly).

As programmers we should read code more than we write it. Otherwise it’s easy to end up in our own bubble.

The Architecture of Open Source Application contains many different case-studies and interesting practical issues that people needed to solve.

Game Programming Gems series contains many very specific and interesting articles on different problems. It’s a great resource for building up your vocabulary about problems.

Two similar books which I recommend are Beautiful Code and Programming Pearls.

Of course you should read code, a lot of it. I can highly recommend reading SQLite, Plan9 and Lua. Libraries that you use daily are a good place to start.

While programming you eventually end up working with other people for one reason or another.

ScrumPLOP is like a chest of treasures where each pattern is thoroughly thought out and analyses different problems a software team might have. Every pattern has years of experience to back it up. I would recommend to skip the Scrum Guide and go straight to the patterns.

Five Dysfunctions of a Team goes over major things that can go wrong in a team so you can detect it early and counteract.

re:Work by Google shares research insights into how to make work environment and communication better.

Even as a programmer you should have a good grasp on how to design usable things and what a human brain is capable of.

The Humane Interface by Jef Raskin is the introductory book on usability and cognetics. I gained a lot of insight into how to measure and understand usability.

The Design of Everyday Things by Don Norman explains human psychology and makes it clear that human interaction is to be expected and failing to take it into account is an error on the designer side.

Nielsen/Norman Group if you want to stay up to date with the latest news on usability research or you have a very specific topic that interests you. When I have a usability question then that’s the first place I search.

The Best Interface is No Interface reminds that the best solution often is not an app or an interaction, but rather the lack of it.

As programmers we spend most of our time building and maintaining solutions, however reflecting on why and what we are building is just as important.

Zen and the Art of Motorcycle Maintenance by Robert Pirsig is an exploration of what is “quality” and what matters in life.

The Nature of Order by Christopher Alexander is a formulation of principles that create good environments and life in the world. It’s a description on how to make life better using design.

Patterns of Software by Richard Gabriel contains many insightful essays on the nature of building software.

Somewhat more specialist subject is networking, but nevertheless, you will end up using networks one way or another:

Falsehoods programmers believe about networks and Fallacies of distributed computing are very short articles, but will make it clear why networking code is difficult, and therefore make you approach the field with greater reverence.

Network Programming with Go by Jan Newmarch goes over multiple layers of network programming with really nice explanations.

Why Do Computers Stop and What Can Be Done About It? by Jim Gray is a classic and a great introduction to dealing with the inevitable failure of systems.

Making reliable distributed systems in the presence of software errors by Joe Armstrong talks about building reliable systems that can survive in the harshest of conditions.

And now the miscellaneous:

Structure and Interpretation of Computer Programs by Gerald J. Sussman and Hal Abelson is a classic that teaches how to bend programming to your own needs.

NAND2Tetris course is an introduction to building a virtual computer from the basic elements (almost). It covers machine language, computer architecture, assemblers, high-level languages etc. All in a course form. For an addendum you can watch Ben Eater’s Building an 8-bit breadboard computer.

Awesome Falsehoods contains a series of articles by different authors exploring various topics in depth, usually things that people get wrong. Just learning about names is enough to take precautions in every other concern.

One of my major insights was:

When the code-base is less than 30,000 lines of code, then code organization isn’t a big hindrance in getting work done.

This number also suggests that you cannot draw any significant conclusions from code-bases that have less than 30K LOC (lines of code).

Note: the actual size is actually more related to how much variance and how many ideas the implementation has. For example, on a website you probably would need 5 different domain ideas, where each is represented by 5 entities, all of them being manipulated and inter-connected. The 30K LOC cutoff serves more as a heuristic.

Alphabet

Why do people organize things in the first place? First, it helps with finding things. Take dictionary for example. I can find “serendipity” just by finding the letter “s” and then each following letter one at a time.

In a dictionary with words sorted in alphabetical order each word acts like a road sign pointing from where you are to where you want to go.

It would be wrong to conclude that organizing everything alphabetically is the right thing to do. Often the question is how to organize. For example when I would like to find all words ending with “-logy” then alphabetical order would be very difficult to use. There are specialized dictionaries that help with that exact question, but those in turn make finding a words with the same beginning more difficult.

When organizing we need to think:

Where is the value in a specific order?

One important implication of this is, that when we make something easier, something else becomes more difficult. The code structure should cater to our problems and needs.

No Organization

What does disorganized mean? It’s easy to state that something is disorganized, but how do we know whether something is organized?

Organization can be largely separated into three principles:

• grouping
• showing similarity
• showing relationships

Grouping creates a unifying idea that helps us find or situate something. In psychology terms, we create a new chunk out of multiple ideas. Showing relations or similarity helps us find the other idea and how it is situated in the bigger picture.

As stated before, a dictionary has alphabetical order. This creates implicit groupings. Each word might have a description or “see also” notes, which show relations to other words.

So, when we talk about no organization, it often means one of the following:

• there is no understandable grouping
• it is unclear how something is related to everything else

It’s important to notice that it’s about “understanding” and “clarity”. When you look at a reverse word dictionary, the usefulness probably won’t be visible immediately. It might be easy to say it’s unorganized or useless. However, when you understand the usefulness it “magically” becomes organized.

In most cases we also need to consider ease-of-learning the organization. Complicated approaches can become an overhead when trying to modify artifacts. A simplistic approach may not give useful benefits. More on that later.

Code

Code is mainly organized using the following language constructs:

• folders and files;
• modules, units or packages;
• types, classes, nested classes and objects;
• interfaces, traits, contexts, roles and aspects;
• methods, functions, nested functions and code blocks.

However, there are other ways to describe structures:

• spatial placement of code structures;
• similar naming (for example Hungarian notation or Design Patterns);
• regions (such as in C#);
• comments, documentation and diagrams.

It might be easy to forget that types serve as a way to organize code. Things derived from the same base-class (or object) are explicitly related. Using another type as a field in a structure is an implicit link to understanding that structure as well. The following descriptions applies to any or combination of these.

Canonical Layout

One of the most obvious ways of organizing things is the same way as everyone else does. This gives rise to structures that look very similar to each other.

You can recognize Java by its structure lib, src/main/java, src/main/resources, src/test etc. Or Linux by etc, mnt, dev. You also have model, view, controller. But there’s no real reason for naming these folders that way other than that everyone else does it the same way.

I’m not talking about the specific content that these folders have, but just the names. It could be library, instead of lib; it could be mdl or models instead of model. Keeping consistency can help a lot with understandability.

Familiarity is a useful feature as it helps us understand new things quicker by relying on what we know already.

My own rule is that when I cannot argue for more than 5% better results by naming it some particular way, then I will use the canonical approach; because the hit from non-familiarity (see principle of least astonishment) when other programmers read it, is bigger.

Often there are reasons why something ended up being named like the way it did. Usually they are human reasons. Someone thought that “this is clear to me” and other people picked it up. The benefit comes from familiarity rather than anything else.

Group by Dependency or Technology

Separating things by having a common dependency is another easy way of categorizing.

For example you might have a component that relies on Windows or Linux system calls, so it can be helpful to separate them with a folder. Similarly a database implementation built on top of Postgres and MongoDB might end up in different files or folders.

Separating things by dependency often makes it easier to do compilation where you don’t pull in a particular dependency. Similarly, separating by dependency allows you to focus more on the “abstraction” that the different dependencies need to implement.

This clear separation might end up with more lines of code compared to using things like C-macros IFDEFs. Similarly, the “abstraction” that describes both adds more code.

Group by Team

“organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.”
- M. Conway

Grouping by team makes it clearer when coordination between teams is needed. For example ui, backend and db need expertise from different teams. Hence, it becomes more obvious who is responsible for which part of the codebase.

However such separation can break locality of information of how something works. Locality of information itself can make harder to grasp the system as a whole. The isolation itself may result in ui, backend and db not fitting together nicely.

A change in the model often propagates through all of the groups. This increases communication or it may require more effort to define stable interfaces.

Group by Use

A grouping that focuses on usability is “group things that are used together”. Sometimes it is difficult to define a single clear grouping other than these things are used together.

Most languages have some sort of math or string manipulation package, with functions like power, log and trim, join.

Effectively these groups are toolboxes that you can use. The problem with such toolboxes is that they can easily degrade into kitchen sinks, where all sorts of arbitrary things end up together. You need to clearly figure out what belongs or doesn’t belong in there to avoid such problems.

Group by Connections

One similar approach to grouping by use, is grouping by connections. We can see this in abundance when looking at declarations of classes and it’s methods.

A method is tied to the class declaration because it operates on the fields that a class has declared. Of course it is declared there because we see it as a behavior of the type. We declare them together, because they are strongly bound together, so it helps to keep a locus of attention.

This is not limited to class oriented languages, all languages have a preference of keeping functions, procedures or predicates close to the type, datatype or struct.

The problem with this approach is similar to grouping by use. Commonly used structures can start accumulating features from all of the places it’s being used. This can make the structure itself less clear. Good examples are data-structures (Scala, C#). Of course, you do get a lot of convenience from this approach.

Group by Classification

Grouping by classification is about organizing things based on the function that things serve in a bigger picture.

One of the most recognizable ones would be Ruby on Rails folder structure.

Many frameworks force a particular folder structure. It’s common to see folders such as models, views, controllers and components. For games you would see folders such as textures, scripts, materials and meshes.

In some projects there is also a single separate folder such as plugins that serves as pluggable parts. This helps to organize and make it clear that the rest of the system doesn’t depend on them directly.

Separation by category can help and simplify framework implementation to automate and find things. For example web renderer can look views folder to find appropriate content to render. Similarly, micro-architecture problems are more visible. For example, it is much more visible when “models” requires “views” by accident.

With frameworks that require such layout it’s necessary to learn the proposed structures. The more complicated the layout, the more learning is needed. However, given a good layout this can significantly help getting small and medium sized projects up and running, since you don’t have to relearn how to do everything.

However the bigger the project the more likely it is that you need exceptions in your design. The fixed folder structure means you either end up creating new folders for capturing the exception or the folder becomes “dirty” because the artifacts don’t necessarily align with the category.

The classification problem can also happen with inheritance. “Evolve Your Hierarchy” describes this problem in game programming context. For example having a base class “Movable” and then a derived “Vehicle”. However, when we add a “Trebuchet”, which needs to be deployed, it makes the object “Unmovable”. We would then move all the different behaviors up to the base class or create a new base-class “MostlyMovable”. We can manage it, but the fundamental problem is about treating it as a classification problem rather than composition of properties.

Group by Feature

Instead of thinking in terms of classification we can also think of features that work together.

For example, you would end up with ideas blog, admin, user, comment, feed for a blog site. You still might have models and views, but in these folders (blog/model , blog/view etc).

This organization makes it much clearer what the software does and how it’s built. It also shows how different things communicate. It places less importance on technical considerations and more on the value proposal of the whole system.

It also makes it much easier to have variation in how things are implemented. For example some particular feature might much better be expressed as a model-view or page.

You can still have cross-cutting features, such as administration of pages. So you would need to figure out how to exactly implement it. One way would be to have a single folder admin with admin/blog, admin/users etc. An alternative would be blog/admin, user/admin etc. In the first case we make it easier to write the admin system itself. In the second case it’s easier to manage a particular feature in isolation.

When we have given less importance to technical considerations it becomes more difficult to enforce a particular micro-architecture and consistency between features. However, it is still possible to use code-review and specialized linters can also ensure such constraints.

With performance oriented projects it might become much more difficult to ensure that the system as a whole works tightly and with low-memory, because we may have separated the pieces that are commonly running together.

This approach also has problems when prototyping, when you are still figuring out what “features” you need.

Group by Value

In all of the previous groupings we have discussed pros and cons of different approaches. “Group by Value” can be seen as an mix of all them.

You start by figuring out what is “valuable” and important in your project. This can vary in different sections, classification or features. Then derive the organization.

This kind of organization can be seen in many projects that do not have an idiomatic way of writing things. You can look at Kubernetes as an example that uses many of these. Similarly many other command-line utilities, database implementations.

This kind of structure happens quite naturally when taking an incremental approach to building things.

One of the major downfalls of this approach is that it is difficult to explain and difficult to do well. Contrary to all the previous organization approaches, you need to be able to evaluate pros/cons of all of them instead of just one. Explaining these ideas to junior programmers is also difficult.

Grouping by value requires a good understanding of why and how your software is valuable. Getting a deep understanding can take a lot of time. And as I mentioned in the beginning, the extra effort of organizing may not pay-off when your project is small. Then again, with large projects forcing people to think about what is valuable can have big benefits, not just in code-organization, but also the product itself.

I try to think less about organizing code and more about organizing ideas about the software. Then again, it’s about finding a good balance between different approaches. There is no “right way”, but they do give more clarity to different aspects of a project.

“Just organizing things” doesn’t necessarily create value.

The question is, which organization gives the most value to you?

If you found this article useful you can buy me a coffee at https://www.buymeacoffee.com/egon

On reddit there was a question:

Before diving into answering it’s always useful to first dive into the question. It’s easy to lose track on what the actual problem is. Often by creating a better underlying model we can simplify or eliminate the problem altogether.

After a few questions I got that the input JSON looks like this (slightly simplified):

The goal of the program is to write code that computes with these values according to names. For example:

"Alpha" + "Gamma" = 25 float

We’ll ignore the multiplier in this post. Unfortunately he wasn’t able to share the business problem, which could provide even more insight.

Interface

The initial implementation was based on using interfaces. Interfaces are indeed one of the solutions for handling variability of the type. The most basic implementation could look something like this:

The content can be parsed from JSON using using a regular struct and unmarshal:

There are other packages that could be more performant, but it’s not a vital part and we can always switch it out later if needed.

Now the question is, how do we write the code. Just writing fields["Alpha"].Add(fields["Beta"]) , wouldn’t work since we only handle things that have specific type.

Type-Switch

I guess the brute-force implementation could use a type-switch to do this:

We use “Error” field type to propagate errors around. It also could be done with manual error propagation or panic, but the Error field seemed a nice solution. Finally we will use all of these things together as:

However, already with implementing two types a single operation becomes very verbose. This is also worsened with each new operation.

There’s a variation for this type-switch. Instead of using concrete type in Add we can use the interface Field. However this has the problem that it ties all the different Field implementations together.

func (a *Uint) Add(b Field) Field { ...

Type-Map

We can avoid some of the boilerplate by using a type-map, which creates a mapping from types to a function:

For each “name and type pair” we assign a corresponding function. We’ll implement the generic Add / Sub in by looking up the function with the right signature. We can see a variation of this in Ivy. It has some nice benefits, like making type-promotions easier.

However, due to interfaces we end up with a lot of different type-conversions to make this happen. We lose all of the type-safety that our language gives us. Similarly it is easy to make a mistake when specifying the functions. That can be somewhat fixed by using reflection to find the func arguments.

This can be a pretty good solution when we need to mix multiple types and the “math” comes from the end-user and not the programmer. In that scenario we cannot rely on the language type-safety anyways.

Reflection

Let’s consider whether using varying types is necessary at all. When we have the correct types from the beginning, we won’t have to deal with the conversions in the first place.

We can try to implement something similar to JSON unmarshaling on arbitrary types. In essence:

To make this implementation work, we first need to unmarshal JSON data into a structure. This is quite similar to the previous cases:

We then need to walk over all of the necessary fields with reflection. For each result field we also will find the corresponding field from JSON.

And finally assign the value to the field:

This solution is nice and neat to use, however extending it with new types would be somewhat annoying. Working with reflection is error-prone and easy to miss specific cases.

Composed serialization

We can avoid reflection by using a thing I call composed serialization. This is how we would write the Field interface:

Notice that the Value is a pointer not the value itself. This allows us to assign the value to an arbitrary place. We can assign the values like:

Finally we can use these all together as:

We can make this little shorter by hiding the JSON part, but it’s not that important for the example. The neat thing is that we can add new field types without having to change the scanning part. Of course this implementation comes at the annoyance of having to explicitly write out the target type twice.

Conclusion

This was another example in how a single problem can be solved in multiple ways. I definitely skipped many possible solutions. Remember that none of these is the “best solution”, each of them has different tradeoffs.

It’s easy to get stuck in a mindset where you want to solve a problem in a particular way. However, there are always many many solutions. Exercising this skill of infinite possibilities helps to find solutions that take less time to implement are more performant or easier to use.

First of all, there are quite a few ways of adding code to a Medium.com, however all of them are frustrating or lacking in some way.

Code Blocks

First of all the integrated code blocks. In essence they look like this:

package main

func main() {
    println("Hello World")
}

At least in theory. When you use tabs, this happens:

package main

func main() {
 println("Hello World")
}

When you copy-paste from Visual Studio Code, this happens:

package main

func main() {

println("Hello World")

}

When you edit, it will behave in all sorts of weird ways. I don’t really mind the no-syntax-highlighting, but the font-size could be smaller. Anytime I tried to use them then editing any indentation, deleting and undo made it break.

Gist

You can also link to gist.github.com, which looks much better:

However, the problem is when you try to add multiple code snippets. My most recent post had 12 different code paragraphs. When you put multiple files into a single gist you cannot link them to individually… so you end up with a mess like this:

Of course you can create separate gists, but manually making 12 of them is pretty annoying.

Medium API

Medium does support creating new posts via an API, where you can provide a markdown or html file. However the issue is that you cannot later update that post. There’s even an issue posted with no feedback from Medium.

I understand the complications. Since Medium tracks the highlights that people have made, updating the content is not as trivial as updating the html. However it definitely isn’t that complicated either since structural diffing does work.

Screenshots

The least frustrating approach I’ve found was using code screenshots. Obviously, this is very annoying for readers.

Conclusion

All in all, the general experience of sharing posts on Medium, is pretty good. The citing, highlighting and getting feedback part is really great. Interacting with comments is somewhat annoying and the stats aren’t that great. However the code part is a great deal frustrating.

None of the solutions I previously mentioned were usable, so I ended up writing my own “solution”.

loov/gistsnip

The idea was to take an existing working code, add some annotations and then the tool will automatically create gists based on the comments.

For example from the following code it will create automatically two gists:

func main() {
    //gistsnip:start:loop
    for i := 0; i < 100; i++ {
        fmt.Println(i)
    }
    //gistsnip:end:loop

    //gistsnip:start:example
    fmt.Println("hello world")
    //gistsnip:end:example
}

Of course, when you modify your code, it will update the existing gists rather than creating new ones. The folder where you run gistsnip contains a file called .gistsnip, which contains necessary metadata for it.

You can see a how your medium post looks like in:

Server and a Database

And the code repository for the medium post:

egonelbre/db-demo

This should allow me to continue working on Medium, until I find a better solution. The migration is going to be annoying, but probably worthwhile.

We are going to take a really small website, where you can submit comments and it will display them. We will cover interaction with the server and database, however we will mostly ignore the different ways of rendering the content and writing business logic. Also, you should probably have some familiarity with Go databases and servers.

You can imagine this as an example of “Infinite Possibilities” exercise.

YOLO

Code at https://github.com/egonelbre/db-demo/tree/master/00_yolo

This is the version, that takes the least effort. This approach can be described as “throw everything together”.

Everything in this file is mingled together and quite hard to follow. Any longer than this and it becomes unmaintainable.

However, this can be used as our first iteration until we figure out what needs to be done.

Functions

Code at https://github.com/egonelbre/db-demo/tree/master/01_funcs

The first step towards separating HTTP from data storage is to use functions. We create functions for different interactions and then provide the database connection as the first parameter.

We could also separate these functions into a separate folder. Regardless the database methods are harder to find, especially when there are many such functions.

Repository

Code at https://github.com/egonelbre/db-demo/tree/master/02_repo

Instead of keeping separate functions we can attach them to a type.

Where the comments repository looks like:

This keeps the comment related database interactions in a single place.

Server

Code at https://github.com/egonelbre/db-demo/tree/master/03_server

Before we refactor the database more, we need to make the server clearer. The endpoint wiring in main didn’t look that nice. Instead let’s try this:

The Server looks like this:

We have removed the global Comments repository and added it as a field to Server. Also, Server has become much easier to test.

Interface

Code at https://github.com/egonelbre/db-demo/tree/master/04_interface

It’s usually nice not to depend on the database implementation directly as it means we have to use specific database implementation for testing. By separating the Comments implementation and interface we make clearer our requirements and how we fulfill those requirements.

Since we wanted to separate them, the folder structure needs to change as well:

Scope

Code at https://github.com/egonelbre/db-demo/tree/master/05_scope

In the future we may need more repositories than Comments. There are few ways that we could handle it. One is to add all methods to a single type. However, that approach can get messy and unorganized after four different repositories.

The other approach is to create a separate implementation and pass them as arguments to Server. It has somewhat similar problem that we might end up with a lot of arguments.

We can also use hierarchical interfaces:

It might look that this is too complicated and unclear why this is beneficial. Let’s take a look at when we have more than one repository and have added basic access control:

We can implement this interface as:

Of course we can use different combinations of extending the DB.

These small things help us avoid forgetting different access checks without having to create a separate argument for each of the methods.

Conclusion

I think these five approaches should cover most of the needs, however specific scenarios might have more interesting solutions. Given a CRUD situation, you might need ORM. Sometimes, you may need a solution with multiple database engines (for example, one for content, one for text-search).

I think the most important thing is:

The right solution is the easiest that works for you.

The complicated version handle more cases, however this comes at the cost of more indirection and more code artifacts. Prefer to start with a simple solution and refactor to a more complicated one when the time is right.

On the surface, it seems like an easy question, and it looks like there are answers to that question:

In object-oriented design, a layer is a group of classes that have the same set of link-time module dependencies to other modules. In other words, a layer is a group of reusable components that are reusable in similar circumstances. In programming languages, the layer distinction is often expressed as “import” dependencies between software modules. — Wikipedia

and

In computing, an abstraction layer or abstraction level is a way of hiding the implementation details of a particular set of functionality, allowing the separation of concerns to facilitate interoperability and platform independence.[snip] An abstraction layer is a generalization of a conceptual model or algorithm, away from any specific implementation. — Wikipedia

However, this really doesn’t fully answer the question.

• How would you write a program to detect a layer?
• How would you write a program that detects a mixing of layers?
• Why is mixing layers bad for readability?

When does database layer transition over into domain layer? Could you write a program that draws the line between layers when they are mixed?

Before proceeding I recommend to think about it a little, because I’m interested in other opinions on it as well.

My first realization was that there really isn’t a fixed line between layers. It is something which is chosen by a programmer. There is always a gradient from one layer to another. For example Data Access Object is partly related to a domain and partly to a database.

Considering that there is a gradient from one layer to another, it’s not that surprising that we misclassify things or have trouble classifying. Sometimes these “misclassifications” end up as leaky abstractions, or in other words, things that leak leak hidden implementation details.

This might seem like we are getting further away from a measurable “layer”, but it gives one important insight:

Layers are a human concern not a technical one.

Let’s imagine having code which includes both database interaction and some inline-assembly. This is often called mixing of “abstraction layers”. However in an high-performance numeric library, using inline assembly (or intrinsics) might be completely fine.

One of the major reasons to use abstractions is to remove and hide the non-essential. That way we can focus more on the given task at hand.

In other words, it’s about keeping our locus of attention in a specific place in our code and mental model. The way we think about domain concepts is somewhat different from the way machine works. When we switch between those ways of thinking we are also switching between ideas.

With the high-performance numeric library using machine level details, such as SIMD, doesn’t add significant mental overhead, because for high-performance you already need to think how the machine operates. Effectively how machine operates is part of high-performance mental model.

For all intents and purposes, switching between layers has all the same downsides as multitasking.

We see a layer encouraging:

• mental model cohesion
• code reuse

Initially we asked how we would detect mixing of layers. Code reuse detection is conceptually not that complicated. When several projects end-up copying the same code with minor differences it can be classified as repeated code.

How do we detect mental model cohesion? I’m going to suggest using machine learning, which obviously isn’t as easy as I make it sound.

We could learn a word embedding using books on different topics. For example we could learn the domain of accounting from different laws or books. Then learn things from database books. Learn coding from coding books. In a sense, we could say that mental model cohesion is approximately the same as word distance in books.

Having learned the embedding, we can see whether the code at hand has terminology from different clustered areas. When a name which is not strongly aligned with the other words in a scope, then it may indicate a bad name, and possibly mixed layers. Of course, there are many tiny things that need to be considered to make this model useful. If you do end up implementing it, I’m definitely interested in the results.

After understanding these concepts, I think layer is a bad name and evokes bad imagery when used to describe the problems.

I think visualizing these things as bubbles of ideas that we want to keep as clear as possible is more useful than trying to stack things on top of each other. It’s important to note that in reality the classification has many more dimensions than would fit on a single flat image.

There are few important parts to layers / abstractions:

1. mental model cohesion
2. code reuse
3. shearing layers (which we ignored in this post)

We could also potentially add hiding information as well, but I think that it comes implicitly with mental model cohesion.

Thinking in terms of cohesive ideas is more vague, however, it makes us notice why we are organizing things in the first place. It gets us closer to visualizing the fundamental forces that created the layers in the first place.

If you found this article useful you can buy me a coffee at https://www.buymeacoffee.com/egon