Testing software is hard, but a lot evolved on the industry for it already, however when dealing with LLMs and specially AI agents, it’s super hard to test, because you don’t have a final truth to compare to, it’s not deterministic

You could try to add a lot of control to your agent and make it very strict, but then that makes it no much better than regular software, it kinda completely defeats the purpose of having an agent in the first place, as they are valuable exactly in those places where having the AI taking the next step decision is a good thing.

Still we are developing as we did we software in the old days: change something, try going through the whole flow, change something again, go through the whole flow again. And worse: because it’s non-deterministic, it’s basically impossible to tell if a tiny change on the prompt broke that thing you were testing last week.

So one day at LangWatch we had a crazy idea: what if we had another agent as our QA? Testing the agent for us just like we do? We could write all the scenarios as we try them and let the thing do it and just tell us if it works.

This is exactly what we did! And it works super well, we put it on a python library we call Scenario:

GitHub - langwatch/scenario: Agent testing library that uses an agent to test your agent

Scenario allows you to write tests in plain english, for the testing agents to use as reference to simulate your users, and test your agentic application.

To get it installed:

pip install langwatch-scenario pytest

Here is a full example:

import pytest

from scenario import Scenario, TestingAgent, scenario_cache

Scenario.configure(testing_agent=TestingAgent(model="openai/gpt-4o-mini"))


@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent():
    agent = VegetarianRecipeAgent()

    def vegetarian_recipe_agent(message, context):
        # Call your agent here
        return agent.run(message)

    # Define the scenario
    scenario = Scenario(
        "User is looking for a dinner idea",
        agent=vegetarian_recipe_agent,
        success_criteria=[
            "Recipe agent generates a vegetarian recipe",
            "Recipe includes a list of ingredients",
            "Recipe includes step-by-step cooking instructions",
        ],
        failure_criteria=[
            "The recipe is not vegetarian or includes meat",
            "The agent asks more than two follow-up questions",
        ],
    )

    # Run the scenario and get results
    result = await scenario.run()

    # Assert for pytest to know whether the test passed
    assert result.success


# Example agent implementation
import litellm


class VegetarianRecipeAgent:
    def __init__(self):
        self.history = []

    @scenario_cache()
    def run(self, message: str):
        self.history.append({"role": "user", "content": message})

        response = litellm.completion(
            model="openai/gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": """You are a vegetarian recipe agent.
                    Given the user request, ask AT MOST ONE follow-up question,
                    then provide a complete recipe. Keep your responses concise and focused.""",
                },
                *self.history,
            ],
        )
        message = response.choices[0].message  # type: ignore
        self.history.append(message)

        return {"messages": [message]}

As you can see it’s a simple pytest test, which you can save for example as tests/test_vegetarian_recipe_agent.py, and then you can run it with:

pytest -s tests/test_vegetarian_recipe_agent.py

You will see the testing agent, impersonating the “User” here, will start talking with the agent under test, or simply “Agent”, asking for vegetarian recipes and evaluating it’s responses at it goes:

What’s cool is that you can also step-in at any time, by enabling the debug more with debug=True on the scenario, so you can use this as a little chatbot to exercise and develop your agent as well.

Doing a scenario-driven-development brings many advantages to agent dev:

• Evaluate your agents end-to-end, as close to what user will face as possible
• Can evaluate both objective things with code (“did it use the search tool?”) and more subjective things with the agent (“is the recipe vegetarian?”)
• It becomes a documentation of what your agent should and shouldn’t do and all the edge cases going forward, preventing regressions
• Since it’s looking at it end-to-end, it makes so much easier to refactor and change prompt, models, tools, which agents are part of it and so one, while ensuring all scenarios run well

I’m really thinking this is the way forward on developing and testing agents, sure it’s still non-deterministic, but as we argued that is kind of the point, and for making it stable caching and a few other mechanisms can help out.

Check out for more examples in the repository, and give it a star! ⭐

https://github.com/langwatch/scenario

This article was originally shared on LangWatch blog.

Test Driven Development in simple terms means writing the tests before writing the code, but anyone who has practiced it for real knows that it goes much beyond. This simple, unintuitive twist causes a profound shift in your mindset.

Having the test before the implementation makes you think much more about what is actually expected, with the how coming later, and the how is an implementation detail, which can be changed through refactoring.

LLMs excel at doing what you ask them to do… but also kinda don't. You can write a prompt and just ask what is expected, but it's not always that LLMs do what you want, partially because it doesn't always follow the prompt correctly. A working prompt can easily break when changed, but in other part because sometimes you actually don't know every detail on how you want it to behave, you only notice when the output is somewhat off, for some use cases, and language is just so open it's hard to define everything.

This way, TDD matches both pretty well and pretty badly on what is needed to build better LLM applications.

Pretty well on philosophy, if instead of doing the prompt directly you'd first write down what you expect from the output, then the prompt could be guided by the already defined spec, and you could change the prompt or the model later, while making sure everything still works as expected.

Pretty badly though, because generally TDD matches better with unit tests, but LLMs are not a good fit for unit tests, their unpredictable nature means they can fail probabilistically, and many examples are needed to ensure it mostly passes.

But is there a way forward anyway? Well, there is!

Doing TDD for LLMs with pytest

Alright, no more yapping, let's start with a very simple test case. We are building a bot that will generate and tweet out recipes, as a tweet, it should be short, so our first test is simple enough:

import pytest
from app import recipe_bot

@pytest.mark.asyncio
async def test_fits_a_tweet(entry, model):
    recipe = await recipe_bot.generate_tweet(
      input="Generate me a recipe for a quick breakfast with bacon",
      model="gpt-3.5-turbo"
    )

    assert len(recipe) <= 140

We use pytest to run and execute the tests plus pytest-asyncio to run tests asynchronously, this will be important later.

Our test is simple enough, good old assertion of just checking the length of the output. Now, as per TDD rule, we run the test (without the implementation), we see it fail (if it passes now something is wrong!) and then we proceed to implementation:

import litellm
from litellm import ModelResponse


async def generate_tweet(input: str, model: str = "gpt-3.5-turbo") -> str:
    response: ModelResponse = await litellm.acompletion(
        model=model,
        messages=[
            {
                "role": "system",
                "content": "You are a recipe tweet generator, generate recipes using a max of 140 characters.",
            },
            {"role": "user", "content": input},
        ],
        temperature=0.0,
    )  # type: ignore

    return response.choices[0].message.content  # type: ignore

We use litellm, a drop-in replacement for OpenAI which allows us to use many different LLMs by just switching the model argument. The prompt is simple, straightforward, should work.

Then we run the test again aaand it passe… fails?

> pytest

============================================================================ FAILURES =============================================================================
________________________________________________________________________ test_fits_a_tweet ________________________________________________________________________

    @pytest.mark.asyncio
    async def test_fits_a_tweet():
        recipe = await recipe_bot.generate_tweet(
            input="Generate me a recipe for a quick breakfast with bacon",
            model="gpt-3.5-turbo",
        )

>       assert len(recipe) <= 140
E       AssertionError: assert 144 <= 140
E        +  where 144 = len('Crispy Bacon Breakfast Tacos: Cook bacon until crispy, scramble eggs, fill tortillas with eggs, bacon, cheese, and salsa. Enjoy! #breakfasttacos')

Turns out, LLMs are terrible at counting, so even though you asked very explicitly on the prompt, it still goes a bit over 140 characters. Well, since now you have an automated way of validating, it becomes easier to try it out different prompts and see if it works:

- "content": "You are a recipe tweet generator, generate recipes using a max of 140 characters.",
+ "content": "You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping.",

It does work!

> pytest

tests/test_recipe_bot.py .                                                                                                                                  [100%]

======================================================================== 1 passed in 1.77s ========================================================================

Awesome, the power of no yapping. Cool, this means it will always work right? For any model and input right? No?

Problem #1: Unpredictable inputs

Users can send any input they want, but our prompt not necessarily will work perfectly for all the cases. Having a working example gives us a bit of confidence, but you know what would be even better? Way more examples. In a probabilistic environment, that’s the best way to get more confidence.

Luckily, we can use pytest’s parametrize for that:

import pandas as pd

entries = pd.DataFrame(
    {
        "input": [
            "Generate me a recipe for a quick breakfast with bacon",
            "Generate me a recipe for a lunch using lentils",
            "Generate me a recipe for a vegetarian dessert",
            "Generate me a super complicated recipe for a dinner with a lot of ingredients",
        ],
    }
)

@pytest.mark.asyncio
@pytest.mark.parametrize("entry", entries.itertuples())
async def test_fits_a_tweet(entry):
    recipe = await recipe_bot.generate_tweet(
        input=entry.input,
        model="gpt-3.5-turbo",
    )

    assert len(recipe) <= 140

Now we have not only one example, but four, and notice that on the last one, the user is trying to be a smartass and jailbreaking a bit, let’s see if our bot is resilient to that!

> pytest

tests/test_recipe_bot.py .                                                                                                                                  [100%]

======================================================================== 4 passed in 7.10s ========================================================================

Yes, it is! Nice, this gives us more confidence. Notice it says 4 passed, by using parametrize with the inputs examples, one test became four already.

Those are things you would probably try by hand as well to break your own bot, however, it’s exhausting doing it every time you change the prompt, it is super convenient to have it automated, so you can tune your prompts worry-free.

But wait, what if you just run again? Actually, sometimes it fails!

============================================================================ FAILURES =============================================================================
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry3-gpt-3.5-turbo] - AssertionError: assert 141 <= 140
============================================================= 1 failed, 3 passed, 1 warning in 7.12s ==============================================================

Almost! By one extra character! But how come? The tests just passed on the previous run, my temperature is 0.0, they should always pass every time right? No?

Problem #2: Flakiness

This is a big one for LLMs, flakiness is way more inherent for LLMs than in traditional software, even when temperature is zero. Luckily, there is yet another library can than help us here, and the name is kinda obvious too, flaky! Flaky allows us to mark a test to be retried a number of times, until it works:

  @pytest.mark.asyncio
+ @pytest.mark.flaky(max_runs=3)
  @pytest.mark.parametrize("entry", entries.itertuples())
  async def test_fits_a_tweet(entry):
      recipe = await recipe_bot.generate_tweet(
          input=entry.input,
          model="gpt-3.5-turbo",
      )

      assert len(recipe) <= 140

With this change, we allow each LLM to retry 3 times before the verdict, effectively doing 3-shot attempts at the problem, and now, let’s see how it goes:

> pytest

tests/test_recipe_bot.py ....                                                                                                                                [100%]
================================================================== 4 passed, 1 warning in 6.80s ===================================================================

Nice! Everything passes! It means that with a couple more attempt, our bot can successfully keep under 140 characters.

But what are we really saying here? Sometimes it will fail and sometimes it will work (at least 33% of the time), but likely in production, you won’t retry many times, and you won’t even know if what you generated is wrong or not (unless you implement the same test at runtime, like a guardrail) so you can retry, so our test is not giving us that much confidence this way (only 33%).

Expecting 100% also doesn’t work though, LLMs are inherently flaky, but how about instead of just defining an upper bound, we define a lower one as well? I want to be able to say that, in production, I expect this to work at least 67% of the time. We can do that by using the min_passes argument:

  @pytest.mark.asyncio
- @pytest.mark.flaky(max_runs=3)
+ @pytest.mark.flaky(max_runs=3, min_passes=2)
  @pytest.mark.parametrize("entry", entries.itertuples())
  async def test_fits_a_tweet(entry):
      recipe = await recipe_bot.generate_tweet(
          input=entry.input,
          model="gpt-3.5-turbo",
      )

      assert len(recipe) <= 140

Presto! Now you are saying it can try 3 times, but it should pass at least in 2 of them, ~67% of the time.

But does this mean your unit tests are saying now that it’s okay for things to fail sometimes? That might make a lot of people uncomfortable, however we are on a probabilistic world remember, having it defined, on your specifications, the pass rate of your executions is already way better than where we started off, completely in the dark, now at least it’s just a matter of increasing those numbers, as you iterate and improve your solution.

Now, what if we try a different model then? A smarter one like gpt-4o ? If it works on gpt-3.5-turbo then it should be a breeze, right? What about an open-source model like llama3? After all the prompts are portable right? No?

Problem #3: Prompt portability

This is a problem for LLMs that unit tests can actually help solving, when you switch models or merely upgrade them, your prompts may stop working. Since we have parametrize and litellm, we can in fact easily run our test examples against different models and see how they behave:

from itertools import product

models = ["gpt-3.5-turbo", "gpt-4o", "groq/llama3-70b-8192"]

@pytest.mark.asyncio
@pytest.mark.flaky(max_runs=3, min_passes=2)
@pytest.mark.parametrize("entry, model", product(entries.itertuples(), models))
async def test_fits_a_tweet(entry, model):
    recipe = await recipe_bot.generate_tweet(
        input=entry.input,
        model=model,
    )

    assert len(recipe) <= 140

This way we pair up each input entry with each model we want to test against, let’s see the results:

> pytest

tests/test_recipe_bot.py ........F.FF                                                                                                                       [100%]

============================================================================ FAILURES =============================================================================
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry8-groq/llama3-70b-8192] - assert 151 <= 140
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry10-gpt-4o] - AssertionError: assert 148 <= 140
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry11-groq/llama3-70b-8192] - assert 320 <= 140
============================================================= 3 failed, 9 passed, 1 warning in 45.21s =============================================================

You can see that actually GPT-4 and Llama3 fail this test, with both failing on the last smart-ass user example. But before we get into that, have you noticed something else? 45 seconds! Our tests took 45 seconds to run, and this is just our first test, how long will it take this suite when the project grows? That’s not good.

Problem #4: LLMs are slow

Even though groq is crazy fast, it’s still orders of magnitude slower than ordinary code we unit test, but actually, in our case, those LLMs are not running on our machine but through an API, so the problem here really is that they are being called sequentially, instead of in parallel.

Luckily, we can use pytest-asyncio-cooperative (use my branch if having issues with retries later), this library allows pytest tests to run all at the same time:

- @pytest.mark.asyncio
+ @pytest.mark.asyncio_cooperative
  @pytest.mark.parametrize("entry, model", product(entries.itertuples(), models))
  async def test_fits_a_tweet(entry, model):
      recipe = await recipe_bot.generate_tweet(
          input=entry.input,
          model=model,
      )
  
      assert len(recipe) <= 140

Now let’s run again:

> pytest

============================================================================ FAILURES =============================================================================
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry9-gpt-3.5-turbo] - AssertionError: assert 141 <= 140
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry11-groq/llama3-70b-8192] - assert 308 <= 140
FAILED tests/test_recipe_bot.py::test_fits_a_tweet[entry10-gpt-4o] - AssertionError: assert 148 <= 140
============================================================= 4 failed, 8 passed, 1 warning in 27.88s ==============================================================

Tests still fail of course, but now it takes 27s, way shorter time, but still a lot due to the retries that end up happening sequentially. If we remove the retries it all runs in 2s, try it out! A fast feedback loop is really important for you to keep tweaking and iterating comfortably, otherwise you will just get annoyed and throw the tests out of the window.

Nice, now that we have bit more speed, let’s go back to the previous problem then, we could try to adjust the prompt to make it work well for all three, but perhaps you don’t actually need it to work on every single prompt, even with retries, again on define boundaries, if you have a big dataset, inevitably some models or some prompts will have a trade-off and being better to answer some inputs well, but others not so much.

Problem #5: Pass Rate

Just like we did retries for every single item and established a minimum pass rate there, there are some items that your LLM won’t be able to get right no matter how many times you try, however, it doesn’t mean it’s worthless, getting the examples on your dataset right 80% of the time is still better than getting it right 50% of the time, and again, you then get explicit on your limitations rather than in the dark, and can only improve from there.

In our case for example, maybe it’s okay that GPT-4 or Llama3 fails in one of the tasks after trying 3 times, we still consider it good enough. So we need then a pass rate across the board, a global pass rate. For that, we are going to use the library langevals, which is all about providing evaluations for LLMs, but it also provides this nice pass_rate pytest annotation:

  entries = pd.DataFrame(
      {
          "input": [
              "Generate me a recipe for a quick breakfast with bacon",
              "Generate me a recipe for a lunch using lentils",
              "Generate me a recipe for a vegetarian dessert",
-             "Generate me a super complicated recipe for a dinner with a lot of ingredients",
          ],
      }
  )

  @pytest.mark.asyncio_cooperative
  @pytest.mark.flaky(max_runs=3, min_passes=2)
+ @pytest.mark.pass_rate(0.8)
  @pytest.mark.parametrize("entry, model", product(entries.itertuples(), models))
  async def test_fits_a_tweet(entry, model):
      recipe = await recipe_bot.generate_tweet(
          input=entry.input,
          model=model,
      )
  
      assert len(recipe) <= 140

With this parameter, now we are saying that the minimum passes we expect is 80%, we are okay with 20% of failed inputs, as long as most of them (80%) are indeed less than 140 characters. I’ve also removed the last tricky example, it was indeed throwing the other models completely off, we can add it back later after improving our prompt.

> pytest

======================================================================= test session starts =======================================================================

x.xx......

============================================================ 8 passed, 3 xfailed, 1 warning in 20.42s =============================================================

Nice! Now our test suite passes, even though not all tests passed completely, notice the 3 xfailed ones, those failed, but were “forgiven”, either because they were retried and worked, or because they are within the pass rate margin.

We have now all the tools we need to write tests for LLMs, and making our uncertainty rates clear. But TDD is about creating writing more tests, so you can add more features to your application, so let’s finally jump into it.

TDD for adding more features

Having 140 characters is nice and well, but that is a too simple of a test to be honest, and it would be easy enough to just cut the string short to fit in there. The more interesting use cases for LLMs is not something like that, the more interesting cases for LLMs is actually being able to do things that code itself cannot do, that cannot be expressed with logic, those are also the most interesting features for your solution to have.

For example, let’s make our bot output only vegeratian recipes, we want to go green and save the planet! This is something that cannot be expressed with basic programming logic, but LLMs are perfect for. However, how do we evaluate this? There is no pytest assertion for that! Turns out those interesting features, are also the most interesting ones to be evaluated.

Problem #6: Evaluations

Enter LangEvals! LangEvals is a library that aggregates various LLM-focused evaluators, exactly for the purpose of evaluating and validating LLMs,.

Validating if our recipe is really vegetarian is really hard to do with traditional code, but it does seems like something an LLM would be good to. LangEvals does not have a “vegetarian evaluator” out-of-the-box, however it does have a CustomLLMBooleanEvaluator, which allows us to easily build a LLM-as-a-judge evaluator, which behinds the scene uses function calling to always return true or false for whatever you want to evaluate.

So let’s write our test with it (before the implementation of course) together with the expect utility:

from langevals import expect
from langevals_langevals.llm_boolean import (
    CustomLLMBooleanEvaluator,
    CustomLLMBooleanSettings,
)

@pytest.mark.asyncio_cooperative
@pytest.mark.parametrize("entry, model", product(entries.itertuples(), models))
async def test_llm_as_judge(entry, model):
    recipe = await recipe_bot.generate_tweet(input=entry.input, model=model)

    vegetarian_checker = CustomLLMBooleanEvaluator(
        settings=CustomLLMBooleanSettings(
            prompt="Look at the output recipe. Is the recipe vegetarian?",
        )
    )

    expect(input=entry.input, output=recipe).to_pass(vegetarian_checker)

Since we are TDDing, let’s run this test to make sure it is failing and we actually need to implement anything (maybe it’s vegetarian by default, so we would have nothing to do!). Let’s check:

> pytest

tests/test_recipe_bot.py F..F.F.F.

============================================================================ FAILURES =============================================================================
FAILED tests/test_recipe_bot.py::test_llm_as_judge[entry2-groq/llama3-70b-8192] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.
FAILED tests/test_recipe_bot.py::test_llm_as_judge[entry1-gpt-4o] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.
FAILED tests/test_recipe_bot.py::test_llm_as_judge[entry0-gpt-3.5-turbo] - AssertionError: Custom LLM Boolean Evaluator to_pass FAILED - The recipe is not vegetarian because it contains bacon.
============================================================= 3 failed, 6 passed, 1 warning in 6.99s ==============================================================

The test failed for the first entry on all models. Of course, because the first input asked for a “quick breakfast with bacon”, so it’s not vegetarian. The LLM Boolean evaluator caught that, and nicely tells us why it decided it fails right there on the error message, and it is because the output recipe contains bacon, which is not vegetarian.

Now you might be thinking, isn’t it weird that an LLM is grading it’s own homework? Won’t it just agree with itself most of the time? Well, not necessarily, first of all because you can use a different, smarter model to evaluate the one you will actually use in production at scale, for example GPT-4 evaluating GPT-3.5 results, but also, because this prompt here is way more focus, it does one thing and one thing well only, it won’t break because you are not updating it and doing many things at once like your main bot’s prompt.

This is one of the very important reasons for us to run our tests before write the code, so we “test our test”. By the reasoning on the outputs, it seems to be working perfectly well, so now we can trust it will evaluate our actual implementation code. Let’s go ahead and update our prompt then:

  async def generate_tweet(input: str, model: str = "gpt-3.5-turbo") -> str:
      response: ModelResponse = await litellm.acompletion(
          model=model,
          messages=[
              {
                  "role": "system",
-                 "content": "You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping.",
+                 "content": "You are a recipe tweet generator, generate recipes using a max of 140 characters, just the ingredients, no yapping. Also, the recipe must be vegetarian.",
              },
              {"role": "user", "content": input},
          ],
          temperature=0.0,
      )  # type: ignore
  
      return response.choices[0].message.content  # type: ignore

Small addition of “Also, the recipe must be vegetarian.”, let’s see how it behaves:

tests/test_recipe_bot.py .........

================================================================== 9 passed, 1 warning in 9.19s ===================================================================

Nice! It all works! And since you wrote the tests before, you don’t need to add any new tests now, you are done, commit, push, and boom, you have a new feature in production, and can change your prompt with peace of mind that this very important concept on your product of having the recipes to be vegetarian will continue working.

For the next feature, you can just think of what you expect the model to output first, verify it, and then implement, this will help you ensure you always move only forward, and don’t break features that were previously working, even in this highly unpredictable environment.

Wrapping Up

I hope this guide gave you a good idea on what is it like to TDD with LLMs, all the challenged on writing tests for it, and ways how to handle each.

The field of LLM evaluations is still a very new one and in fast expansion, every day people are exploring new ideas and finding more and more ways that we can control those very powerful but unpredictable creatures.

Check out LangEvals Documentation for all the available evaluators, contributions for new ones are also very welcome!

Also, if you like this guide, please give LangEvals a star on github, and for all those bugs that cannot be captured by unit tests, check out LangWatch, our LLM Ops monitoring platform, powered by LangEvals.

Good testing!

It does not come as a shock for anybody when I say that data is one of the most valuable assets a company can have, this has been true for some years now. Valuable data can be used to build very powerful products, and with the invention of LLMs, it became even more obvious that companies that already have valuable data internally can leverage AI to create a bigger edge.

But internal data is not the one I want to talk about today, I want to talk about the data that is produced as a byproduct of your AI being in production: all the interactions your users have with it, both with happy and unhappy outcomes. This is super valuable data, it’s literally the best insight on how users expect your product to behave, and now it comes in conversational text format, it’s not just numbers like traditional product analytics, so it can directly feed back into the AI.

The thing with LLMs, is that they are super powerful and general, this is the exact reason we want to use them that, they can answer any questions and execute a large range of tasks, however exactly because of that, their surface area to the real world usage is extremely large, the more they do the larger it is, resulting in countless different ways it can fall short. It is hopeless to try and predict all the ways your users will want to use your product, and try covering all the possibilities and edge cases beforehand.

How can a Data Flywheel help LLM apps?

We call it a flywheel when an improvement brings business value and that loops back on further improvements. Since the number of possibilities of an LLM product is too vast — think of all possible ways a user can try using your product, or even compose a sentence — nothing is better at selecting what is more valuable and relevant like the real world, this makes it essential to monitor and get insights on where the product is suboptimal, or where it could bring even more value.

Then by using those insights for improving the product and making it more valuable, it will receive more usage and more users, it will gain more trust and cover more cases to be used in more places. By getting more usage, even more scenarios are revealed, new paths to bring even more value, and new shortcomings to overcome and make the product more robust, which will lead to even more usage an so on. That’s the flywheel.

The advantage with AI over traditional software improvements, is that this connection is even more direct, LLMs can be used also to evaluate, classify and synthesize data that will improve the product, so this flywheel is closed even tighter together.

There is, however, no shortcuts, if your solution has been finding new unexpected edge cases for months or years, it is not possible for someone else to just guess which ones they are and port them over elsewhere without going through the same discovery process. The improvement process you done on your product for your specific domain then creates an edge that sets you apart from others as soon as you start it.

This is a good news, if you are doing it.

LangWatch

I started building LangWatch on top of this realization, to help you maximize your data flywheel. Our platform helps both on collection part, with monitoring, evaluations, insights, collecting user feedback and annotating with domain experts from your team, and on the improvement part, running experiments, validating and guaranteeing the improved quality of each iteration.

If you are interested, help us out with a star on github ⭐

I published this post originally on Reasoning Engine

In the past few weeks I’ve became obsessed with DSPy, when I first saw the video DSPy Explained by Connor Shorten everything immediately clicked.

This is the LLM framework I was waiting for, this is the idea I was trying to get to but didn’t knew exactly how, and oh how coherent all the pieces are! It’s a great conceptual abstraction, I noticed myself not needing the docs all the time, even though the library is not properly typed right now, because things just *make sense* together.

If you haven’t dig into it yet, DSPy main selling point is that you just define your pipeline declaratively without worrying about the prompts to the LLM, and then it can optimize the prompts for you automatically. Of course, why should humans be trying to prompt engineer, when we have the best machines to do that: LLMs and trial-and-error.

And that’s kinda obvious right, of course I thought of that before, of course a lot of people thought about that before, having another LLM to improve your prompts, in fact we already have plenty of those like PromptPerfect, however things still felt iffy, kinda unreliable, relying just on feeling, or on a bit of uncanny abstraction. But DSPy looked at PyTorch for inspiration, and this, turns out, was the right abstraction.

Many devs adventured themselves at the problem, and different angles were attempted, I personally took a TDD approach at the problem, which I still think is the right one for sanity checks, I’ve also tried Reactive Streams FP-style approach, some inventented this weird “chains” concept which nobody really likes, and are now attempting Graphs angle.

But turns out, comming from the Machine Learning angle with PyTorch abstraction like DSPy did was the actual right approach, the one that makes me and all devs I talk to go like “oooh that is genius! It makes a lot of sense” and have this sense of relief that finally looks like someone had the right jab at the problem and you’ll be able to build something reliable.

Of course DSPy doesn’t seem the best for every problem right now, it seems to make the most sense for problems with well defined discrete outputs, like traditional supervised learning, for example when you want to use LLMs as a classifier. However, it is a clear a step on the right direction, and you can see a near future where more “feeling-based” evaluators will be also very effective and ubiquotous, for example, if you are writing a children’s stories chatbot and want to reliably evaluate how boring or exciting a story is.

The main value that DSPy brings to the industry then, is not (only) that it optimizes prompts for you allowing you to switch models whenever you like, but that it actually turns the problem on its head: it forces you to change your mindset, thinking first about what does it mean to be a good output or not with examples (like TDD promotes!) before the prompt, freeing you think in a more abstract level about the structure and so, and let the machine do machine-things of finding the best fit given the problem at hand.

Being so bearish in DSPy, I really want to help it grow and contribute to the community because I think it has a lot of future and it is the ahead. And I mean conceptually, even if something else would superseed DSPy with a different name.

Turns out, I already have a startup in the LLM space, LangWatch, so as I played with DSPy for LangWatch experiments, I naturally noticed a piece missing that could help me a lot. Thus, DSPy Visualizer was born.

DSPy Visualizer

DSPy Visualizer is open-source, and it allows you to log your DSPy training sessions, track the performance, costs, compare runs and debug them in detail.

It’s still early days for DSPy or any kinda of automated LLM optimizer, so it really helps a lot being able to understand what is the optimizer trying to do, where is it going, and specially, how much is it costing, as too many examples and too complex pipelines can make your OpenAI credits go down the drain in a heartbeat. Even though LLMs are getting cheaper, we are pushing the automation limits here.

Iteration is the word of order with DSPy, although a much more confortable, reliable iteration than “classical” prompt engineering of course, and we found that looking at the examples being used, the LLM calls being made, and what is happening on all those optimization attempts that is helping it go up or down, gave us lots of ideas of what we can try next, or what is wrong so we can stop the iteration earlier and fix it.

I’ve posted an official blogpost on the DSPy Visualizer on our LangWatch blog, if you want to try it out, check out the quickstart guide.

If you are in for more DSPy content, follow me on twitter and subscribe to this substack!

If you are building with LLMs and have done your fair share of prompt engineering, LLM calls after LLM calls, trying to improve your product performance, DSPy is the logical next step for you.

What is DSPy?

DSPy is the hot new framework on LLM community, which takes a different approach than what so far has been done for the past year: instead of having engineers manually trying to adjust prompt to improve results and do a *vibe-checking* on the outputs, DSPy gets you to focus on the dataset examples you bring, and the objective metrics that come out, not unlike traditional Machine Learning.

DSPy takes a big inspiration from PyTorch, which brings a much more rational and structured approach to optimize LLM pipelines, in a very coherent framework where all the pieces fit very well together.

In short, by giving your dataset to your DSPy program, and defining which metrics mean a good result by evaluating the outputs, DSPy Optimizers can automatically try to find the best combinations to improve your LLM program, be it by changing the prompts, the demonstrations for Few-Shot Prompting, or slight changes in the structure (signatures).

We found this approach to be very effective. In one case where we couldn’t get accuracy higher than 60% after many hours of manual prompt engineering, DSPy took it to 80% for us in a matter of minutes.

Prompt engineering at last automated, and the results are by definition measurable!

DSPy Visualizer

At LangWatch we have been very excited on the results we are getting from DSPy, so today we are making public a tool we already have been using internally which help us a lot on our DSPy training sessions, the DSPy Visualizer:

LangWatch is an open-source LLMOps platform, and our latest release, with DSPy Visualizer, allow you to log your DSPy training sessions, track the performance, costs, compare runs and debug them in detail.

This helps make it very easy to understand the step the optimizer is going through, when using it ourselves, multiple times we noticed some issues on the pipeline, like the signatures not being used correctly, or the wrong LLM being called when for example we wanted to use GPT-4 as a teacher for a GPT-3.5 student, or it gave us new ideas on what to try. This allowed us to stop the experiment earlier and iterate on our setup more quickly.

Since DSPy runs a lot of trials in a short amount of time, token rate limits and LLM calls cost also play a big factor, LangWatch’s DSPy Visualizer tracks all LLM calls for you, making it easy to compare for example the cost of using GPT-3.5 Turbo vs Claude 3 Haiku, or noticing that you are reaching 400K tokens per minute and know what rate limit to request further, which was our case.

If you are curious to try it out too, check out our quickstart docs, feel free also to join our community on discord for any questions!

But then this happens with most people:

It is so easy to open a new tab, but so hard to close, people get very attached to their tabs, “I might need it later”, “I will read it eventually”. Then hundreds of extensions and practices that require discipline to manage the insurmountable amount of tabs pop up, but only the most dedicated can follow.

The thing is, our web got richer and richer over time, there are more things to do at the same time than ever before, tabs were great, but we need a better solution now.

I propose then a radically different approach: forget about tabs, get rid of them.

I noticed that when I was navigating between files in the code editor, or between open tabs in the browser, I spent a long time looking for the tab I wanted, over and over again, so much that often it was faster to just open another tab for the same link, increasing the problem.

Often also, I noticed the tab I wanted to navigate was one of the ones I had opened most recently, I just wanted to go back to it, or to that one before it.

So, if typing is faster than finding the tab, then all you need is a quick search, and if most of the times you need recent tabs, then they should be easily at the top of this search.

If you use IntelliJ, this is exactly what Cmd + E does, or the Recent Files feature. If you just press Cmd + E then Enter, it will take you to the previous open file, very convenient. If otherwise you start typing, you can quickly find and go where you intended, your eyes can find it super fast, much faster than it can find the right tabs!

IntelliJ was the first place I started living without tabs. I noticed I had to change the settings to completely hide them, because if I didn’t, then I end up with the old habit of looking for the right tab, and getting annoyed that I couldn’t find it. Since now I navigate through shorcuts, I actually end up hiding everything away, and I have full screen real estate to focus now.

This is how my IntelliJ looks like when I’m “switching tabs”:

Look at the editor behind, no toolbars, no file explorer, and most importantly, no tabs.

Then, most recently, I went to more frontend work, and moved to VSCode, took me a while, but I could replicate the same environment there

All you need is to add this custom shortcut

  {
    "key": "cmd+e",
    "command": "workbench.action.quickOpenPreviousEditor"
  }

And that was good enough for me.

On the browser land, I was still relying on discipline, tab snooze, etc etc, on Firefox, but then, a wild browser appeared, Arc! To my delight, Arc supported precisely my no-tabs workflow lifestyle. Look at me, writing this blogpost right now:

On Arc, when you press Cmd + T you also get this “Recent Files” navigation, by having that, Arc makes it so easy to hide everything else, and the website can use all your screen real estate.

It’s not that tabs don’t exist, they are there, just not in front of you. It makes it easier for you to really focus on one thing at a time, without worrying about managing your tabs and being all tidy and clean all the time.

Incidentally in Arc, this is also the shortcut to open a new tab, as every browser, so it really directs you into getting more used to this navigation style. As a nice addition, Arc also has this “Little Arc” feature, which opens a new small window in-place where you clicked the link, for example on Slack, allowing you to quickly skim over the link and dismiss immediately, or decide to actually open as a tab in the main browser. I felt this was super convenient for less focus switching, and less tabs.

That’s it, if you have never lived without tabs before, I really recommend trying it, it’s so relaxing.

Cheers!

Life without Tabs was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

pragmatic

**/præɡˈmæt.ɪk/**
adjective

1. solving problems in a sensible way that suits the conditions that really exist now, rather than obeying fixed theories, ideas, or rules — Cambridge Dictionary
2. based on practical judgments rather than principles — American Dictionary
3. relating to matters of fact or practical affairs often to the exclusion of intellectual or artistic matters : practical as opposed to idealistic — Merriam-Webster

Recently I started reading the book The Pragmatic Programmer, I had never read it before yet it influenced me so much, how? Simply by the title. The Pragmatic Programmer, such a cool concept. I’ve heard about this book left and right, it was a mandatory read once I got serious about software development, I heard people talking about “you have to be pragmatic here”, or “those devs are pragmatic”, this changed my mind when approaching the development of many software systems, yet for some reason I never got to actually read it, it has been sitting on my “To Read” list for a long long time until now.

So now I read it and it was… disappointing. Like very disappointing. Like “don’t meet your heroes” kind of thing. The title “The Pragmatic Programmer” is so brilliant, yet they ruined it with the content.

My understanding of pragmatic is the same as the definitions given above, and it were those that I used to guide my thinking when taking pragmatic decisions. Not good practices, not dogmas, but foot on the ground, facing the harsh and cruel reality. The Pragmatic Programmer book is not about that, it’s a dogmatic book.

The Pragmatic Programmer: The Good Parts

The book actually have some very good and pragmatic points, I will start with those because I think they should have been the central idea of the book:

CRITICAL THINKING

The last important point is to think critically about what you read and hear. You need to ensure that the knowledge in your portfolio is accurate and unswayed by either vendor or media hype. Beware of the zealots who insist that their dogma provides the only answer — it may or may not be applicable to you and your project.

Never underestimate the power of commercialism. Just because a web search engine lists a hit first doesn’t mean that it’s the best match; the content provider can pay to get top billing. Just because a bookstore features a book prominently doesn’t mean it’s a good book, or even popular; they may have been paid to place it there.

Critical Thinking, that’s it, for me that’s in the very core definition of what does it mean to be Pragmatic, don’t be fooled by dogmas and hype, keep your foot on the ground.

Still for some reason they say it’s the last important part (I don’t know why it isn’t the first topic on the book), and later they throw lots and lots of dogmas at you, without explaining the reasoning behind them, contradicting this very same paragraph.

They go further on Critical Thinking with some tips:

Ask the “Five Whys”

A favorite consulting trick: ask “why?” at least five times. Ask a question, and get an answer. Dig deeper by asking “why?” Repeat as if you were a petulant four-year old (but a polite one). You might be able to get closer to a root cause this way.

[…]

What’s the context?

Everything occurs in its own context, which is why “one size fits all” solutions often don’t. Consider an article or book touting a “best practice.” Good questions to consider are “best for who?” What are the prerequisites, what are the consequences, short and long term?

Those are perfect, that’s much of what I understand by being pragmatic.

Then, to close off the good parts of the book, on Good-Enough Software:

The marketing people will have promises to keep, the eventual end users may have made plans based on a delivery schedule, and your company will certainly have cashflow constraints. It would be unprofessional to ignore these users’ requirements simply to add new features to the program, or to polish up the code just one more time. We’re not advocating panic: it is equally unprofessional to promise impossible time scales and to cut basic engineering corners to meet a deadline.

That’s it, that’s a Pragmatic Programmer, one that has to work within corporate constraints, one which has it’s foot on the ground, that produces Good-Enough Software, because they know it’s not possible to build the perfect system given real world contraints, nor to deliver on impossible deadlines.

The Dogmatic Programmer

At the same time the book argues for Good-Enough Software, it argues for the broken window theory:

Don’t leave “broken windows’’ (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.

This is already somewhat contradictory to the Good-Enough Software point made on the book just two topics later, but the worse is that it’s actually based on a theory that was already debunked

A lot of times the story of crime reduction in NY with the broken-window policy is used to back-up this, but a quick read shows you that this is a mere correlation, not causation, a change of police strategy to focus on the bigger crimes rather than on petty ones might have actually been much more effective, together with abortion legalization and many other candidates, nobody knows for sure.

The broken-window theory for code feels like a nice idea, but feeling is not enough, Dogmatic Programmers are gullible and like theories, but a True Pragmatic Programer is skeptical, and values scientific rigour (when possible).

Dogmatic Programmer: gullible, does what feels right, following idealistic theories

Pragmatic Programmer: skeptical, values scientific rigour (when possible)

Now, the (when possible) part is very important, and very pragmatic too, because to be honest, in software development that’s rarelly possible. The more I learned about statistics and biases in those recent years, the more I realized that having a proper experiment to discover causality is an extremely hard thing, and in software development, basically impossible.

We cannot build thousands of the same systems, with thousands of developers and add just a small variation (say, broken-window policy) to do a proper RCT and see it’s effect. We don’t even know if static types are better than dynamic ones, we have many studies, but all confounded or biased in a way or another.

Most of the times we have no choice, being pragmatic means you have to use what is practically available: subjective experiences, biased experiments, or even your favorite dogma when there is nothing left. Practically, you need to take a decision (or even decide to not take one), but keeping your skepticism if you want to be pragmatic, fix your foot on the ground.

In that vein, I will tell you a subjective experience I had that touches the broken-window theory, and I wrote about it on the blogpost Designed v Evolutionary Code. It’s not that I’m against broken-window theory or that I’m in favor of it, it’s just that it depends. I take the book’s own advice of What’s the context? to give you an example of just two cases where it probably works and where it probably doesn’t, following Kent Back’s 3X Theory:

1. If you are in exploration mode, most of the code you produce has no value and you will throw away, you need to iterate faster, here broken windows are totally fine, you focus on the current biggest problem only, law of diminishing returns kicks in and fixing windows doesn’t pay-off at all, intead you move quickly to the next biggest problem
2. If you are in extraction mode, on a huge system, even a single flaw can be cathastrophic and cost millions of dollars or lifes (you are under power law), here you don’t want to keep a single window broken, you want to try your best to be sure all tiny pieces are working as they should and hope for the best

A True Pragmatic Programmer is skeptical of stories and dogmas, but the book is not skeptical at all, it tries to convice you using anecdotes, comming from the authors’ experience, without any depthness, cues for the possibility of being wrong, and sometimes not even a short explanation, for example:

Duplication with Data Sources

Many data sources allow you to introspect on their data schema. This can be used to remove much of the duplication between them and your code. Rather than manually creating the code to contain this stored data, you can generate the containers directly from the schema. Many persistence frameworks will do this heavy lifting for you.

There’s another option, and one we often prefer. Rather than writing code that represents external data in a fixed structure (an instance of a struct or class, for example), just stick it into a key/value data structure (your language might call it a map, hash, dictionary, or even object).

So should I just trust your preference? It doesn’t seem very pragmatic. And why do you even prefer that? There is not an explanation of the rationale whatsoever, where is the Critical Thinking? Where is the Five Whys?

Most of the times the book assume all conclusions are self evident, there are many more examples like that:

you can discipline yourself to write software that’s good enough — good enough for your users, for future maintainers, for your own peace of mind. You’ll find that you are more productive and your users are happier. And you may well find that your programs are actually better for their shorter incubation.

Why?? I mean I’m obviously all in favor of shipping ealier (generally), but why does it make things better? I feel the authors could have done a way better job explaining how production is the real test to our systems, how real life is so complex that is impossible to predict all the variables, like Nassim Taleb explains so well in Anti-fragile, so we have to be pragmatic, accept there is a lot of variables we don’t know, and that it’s cheaper to allow real-life to give us this feedback then upfront predicting them all, this is why Agile works (but not always! Context matters)

Your ability to learn new things is your most important strategic asset.

Why? And “strategic”? I thought this book was about being pragmatic (foot on the ground, present) not “strategic” (heads in the cloud, future)

Treat English (or whatever your native tongue may be) as just another programming language. Write natural language as you would write code: honor the DRY principle, ETC, automation, and so on.

Why? Being DRY is actually terrible advice for natural language communication, people don’t pay attention, people forget, you actually need to be repetitive sometimes.

If you can’t find the answer yourself, find out who can. Don’t let it rest. Talking to other people will help build your personal network, and you may surprise yourself by finding solutions to other, unrelated problems along the way.

Tips tips tips, just tips, no pragmatism here.

Chapter 2 is titled “A Pragmatic Approach”, and it starts with that, I kid you not:

A Pragmatic Approach

There are certain tips and tricks that apply at all levels of software development, processes that are virtually universal, and ideas that are almost axiomatic

So their Pragmatic approach is to bring us a chapter full of axioms, not to challenge them, but to tell us they are universal 🤯. Say what!? This is absolutely NOT pragmatic. I expect to find axioms on Clean Code, that’s why I bought that book, not here, here I expected to find counter-axioms, cases of when you have to actually be pragmatic, take your head off the clouds and break those axioms when it makes practical sense. This chapter should really be called A Dogmatic Approach.

They continue:

The world is full of gurus and pundits, all eager to pass on their hard-earned wisdom when it comes to How to Design Software.

And finally confess:

And we, your gentle authors, are guilty of this too. But we’d like to make amends by explaining something that only became apparent to us fairly recently. First, the general statement:

Their “explanation”, actually uses a dogma as an excuse (Good Design Is Easier to Change Than Bad Design), so it’s circular. “I’m sorry we are not pragmatic and presenting you dogmas, but the reason is because of this dogma”

You can repeat yourself

Topic 9 is called “DRY — The Evils of Duplication”, this is the part of the book I have the biggest beef with. The authors treat repetion as pure evil, and don’t make any arguments on when it could be good, it’s not pragmatic at all, pure dogma.

In fact, I’ve used in the past sentences like “they decided to take the more pragmatic approach and copy-paste the code for this situation”, so it bothers me that the book that takes pride in it’s title wouldn’t even consider this as a possible sentence.

Just to be clear, I did many terrible code duplications in the past, and it helped me a lot learning the DRY concept at the start of my programming carreer and removing many duplications, but unlike the authors, I know that DRY does not hold true everywhere, and many many times it causes unintended harm, as you can read in Duplication is far cheaper than the wrong abstraction by Sandi Metz. It’s not all black-and-white, a pragmatic programmer should know that.

Dogmatic Programmer: Duplication is evil

Pragmatic Programmer: It depends

They start with a code refactor example to try to make their point. This is the original code:

def print_balance(account)
    printf "Debits: %10.2f\n", account.debits
    printf "Credits: %10.2f\n", account.credits
    if account.fees < 0
        printf "Fees: %10.2f-\n", -account.fees
    else
        printf "Fees: %10.2f\n", account.fees
    end
    printf " ———-\n"
    if account.balance < 0
        printf "Balance: %10.2f-\n", -account.balance
    else
        printf "Balance: %10.2f\n", account.balance
    end
end

So they do this refactoring:

def format_amount(value)
    result = sprintf("%10.2f", value.abs)
    if value < 0
        result + "-"
    else
        result + " "
    end
end

def print_line(label, value)
    printf "%-9s%s\n", label, value
end

def report_line(label, amount)
    print_line(label + ":", format_amount(amount))
end

def print_balance(account)
    report_line("Debits", account.debits)
    report_line("Credits", account.credits)
    report_line("Fees", account.fees)
    print_line("", "———-")
    report_line("Balance", account.balance)
end

And they claim it a victory, apparently they removed most of the duplication.

What!? This is a terrible refactor, one function became four; it’s way worse to read, now instead of reading just a bunch of plain printfs, easy to follow because it’s a core function of the language, the reader needs to jump around functions and understand what the author meant; now instead of one function causing bugs you have 4 functions that can interact in weird ways causing weirder bugs; now if you want to change the format of just one like you need to add an if condition, a new method, or something, it actually violates their Easier to Change rule.

It does not work here because this code is so simple, this is the wrong example, on a full-fledge report some of those refactorings could actually make sense, you would probably have the logic and presentation separation already anyway, but giving the advice on this small piece of code actually makes things much worse, showing that they are not taking context into account, but just following DRY as a dogma.

Now there are many statements they make I’d like to comment on:

The alternative is to have the same thing expressed in two or more places. If you change one, you have to remember to change the others, or, like the alien computers, your program will be brought to its knees by a contradiction. It isn’t a question of whether you’ll remember: it’s a question of when you’ll forget.

Conversely, if you have everything expressed in a single, centralized place and you need to change it you are forced to remember all the places that depend on that. This can take a long time, time which may not even pay off for the inconsistency. It could be that the place you “forgot” to change actually was not that important at all, like a page nobody visits, a feature no one uses. If it is important, you will notice it. Again it depends on context, I touch on this on the Designed v Evolutionary Code post too.

Additionaly, no duplication at all can actually be pretty risky. Remember left-pad? Because everyone decided to not duplicate something that can be implemented in 1 line of code pretty much all javascript deployments in the planet were broken for a day. When you have central critical pieces without any duplication, you increase the chances of bringing the whole system to it’s knees, like it happened many times recently with big tech.

DRY is about the duplication of knowledge, of intent. It’s about expressing the same thing in two different places, possibly in two totally different ways.

When I say duplication is good for resilience, some may think about server redundancy, but no, I really mean intent duplication, which the authors rejects. Why is “two totally different ways” necessarily a bad thing? Devs love redundancy of servers but despite duplication of code, problem is, having thousands of servers in hundreds of availability zones still won’t prevent you from deploying a catastrophic commit.

If you think about it, different codebases doing the same thing is like redundancy for the overall system, for ideas, for bugs. There is a reason why evolution brought us so many different animals, which survive through different ways, there is a reason why we have anti-monopoly laws, causing different companies to do the same thing but in different ways, this way you get the chance that one of them is actually better than the other.

Dogmatic Programmer: let’s make everything abstract, what if we need to switch databases?

Pragmatic Programmer: we will never switch databases

On topic 11 — Reversibility, the authors go on about how we should decouple third-party services to keep our system flexible, and of course they give the classic example of the database switch:

“But you said we’d use database XYZ! We are 85% done coding the project, we can’t change now!”

In practice there are not a lot of reasons for switching one relational database with another, and even if you do, it will probably not be crazy hard, SQL is very standardized (much easier to learn than the thousands of ORMs APIs we have), on a small codebase find and replace can do most of the trick. Even if a big codebase decides to switch, it will be incremental with a migration plan anyway, it will never be “just change a config on the ORM”. Now if the change is for a completely different way to store the data to some domain specific NoSQL, or replace those MySQL queues with Kafka, then probably your abstractions won’t hold anyway, you will need to change your architecture to really take advantage of it.

An abuse of “reversibility” causes for example Generic Cloud Usage, from ThoughtWorks radar:

We see organizations limiting their use of the cloud to only those features common across all cloud providers — thereby missing out on the providers’ unique benefits. We see organizations making large investments in home-grown abstraction layers that are too complex to build and too costly to maintain to stay cloud agnostic.

[…] which reminds us of the lowest common denominator scenario we saw 10 years ago when companies avoided many advanced features in relational databases in an effort to remain vendor neutral

Give a random number

The book then goes to Estimating, on topic 15, which is perhaps the most debated point in software development. As usual, they start with a bad example:

The Library of Congress in Washington, DC, currently has about 75 terabytes of digital information online. Quick! How long will it take to send all that information over a 1Gbps network? How much storage will you need for a million names and addresses? How long does it take to compress 100Mb of text? How many months will it take to deliver your project?

The last question is completely different from all the others, they are conflating estimations that have a super simple and linear models, where you can easily find all the variables, with a complex non-linear estimation where no good model exists. Deceiving the reader into thinking that estimating a software project is as easy and reliable as calculating bandwidth is an awful thing to do.

Then they go on some tips on how to improve your estimations, telling about building a model and breaking it in smaller pieces, they give a lot of focus on getting better at estimating, but the actual valuable thing is some communication tips in the middle, like saying weeks instead of days to convey less precision, and giving multiple range estimations instead of a point estimate.

Dogmatic 1: believes on estimations, and that one day they they will get very good at it (this day never comes)

Dogmatic 2: believes that estimations are never possible to get right anyway, so they are useless

Pragmatic: manages expectations, not accuracy

You see, developers are actually not that bad on estimating, getting around 60% of the estimations right[1][2], people hypothesize that developers are good at estimating the median, not the mean. So the problem really is on the blowup factor: eventually something will get very wrong, and not even the most experienced estimator will be able to predict it, when that happens (and it does), expectations management skills is what really counts.

Considering that, the True Pragmatic Programmer is not in favor or against estimations: it’s just a tool, which they know is flawed (sometimes exponentially flawed), still sometimes the most available one to align teams more or less on what to do, or as a way to get others thinking more deeply about a feature, so they use it when useful, and avoid it (or just give a random number) when useless.

When someone says “the deadline is October” for a pragmatic programmer, they ask “why?” (those five whys mentioned by the book). More often than not I’ve found out that those are just self-inflicted deadlines: no client knows about this feature you want to launch, other teams are unaware, it wouldn’t really affect much anyone, it’s just politics or a random date someone came up with.

Other times, the deadline actually makes sense, maybe some other team depends on that for a pressing issue, or maybe there is a regulation change, and even though you might use some fancy tricks of iterative development to avoid that, sometimes is just easier to agree on a date and that’s it. Pragmatic.

Well I think that’s enough about the book, this post is too long already, but I hope I manage to convey the idea of what a pragmatic actually means, the definition that is closer to the dictionary one.

Thanks for reading! Follow me on Twitter, and drop a message there if you disagree or want to discuss about any of the points.

Addendum 1: The version of the book I have and this blogpost is based on is the 20th Anniversary Edition, not the classic one.

Addendum 2: The article came out very harsh, so I need to clarify that I do in fact think The Pragmatic Programmer is a very valuable book for beginner developers, but not advanced as I hoped so. This made me want to write this blogpost, to express what I actually wanted to read, the book I wish existed.

A lot is discussed on how to do agile, if a given company or framework even is agile, if you should use sprints or Kanban, Scrum or XP, if agile can scale, etc. On those discussions, the agile manifesto is routinely brought up with those famous four lines:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

We have so many layers on top of “agile” nowadays that people talk about it without considering or even knowing about the existence of the agile manifesto, leading to a lot of BS. It’s much better when the agile values are brought up, the discussion becomes much more productive.

Those four sentences are now an unchallenged way to better develop software in general — surely enough, after over 20 years they proved their value empirically — but why is it that they make so much sense? I’m not looking for anecdotes but trying to actually understand the reasoning behind it, the core value behind the agile manifesto values, and I think I found the answer, on Nassim Taleb’s Incerto series.

Taleb argues that many things in the real world are too chaotic to be fully understood, he criticizes a lot economists, for example, for claiming to be able to understand and predict such a complex thing as the economy, pointing that they actually turn out to be more wrong than right. He argues that we should focus not on trying to understand and predict all uncertainty, but rather work considering there will always be uncertainty and risk

This is also the biggest realization of the agile manifesto authors, that software development, just like the real world, the human body or the economy, is too damn complex to be fully predicted, written down or systematized.

Then they went on to write four very down-to-earth guidelines, naming things we do day to day, like processes, tools, documentation, contracts and plans, but pushing us away from the default approach of trying to understand everything and predicting the future, towards the more flexible and uncertainty thinking

To clarify on what I mean by “away from default approach”: even if nobody is teaching “waterfall” project management, I believe most on the right side of the agile values is the natural direction people tend to take when problems arise, so it requires a manifesto to weight to the other side.

For example “over processes and tools”. It is common for people to want to fix problems by introducing a process or a tool that will streamline the work and allegedly reduce mistakes. “Did an employee steal from the company? Let’s now always require manager approvals so that can’t happen again”, “Did a human error happen here? We could introduce a checklist form so nothing is forgotten again”, “People are not giving feedbacks to each other? Let’s make them use this tool with very specific text boxes guiding them to the perfect feedback process”. I’ve seen suggestions like this popping up from very well-intended people many times, with clear benefits on the short term, but causing more harm then good down the line. Everyone that worked on a big corp knows that bureaucracy only tends to grow — even though nobody is really a bureaucracy lover — this is just the natural direction, so it requires a push in the other direction to balance things out.

The suggestion is then to give more weight to “Individuals and interactions” instead, but why is that? This is because our processes and tools, just like our systems, are rigid: as soon as you encounter an edge case the process can’t handle it breaks — and in complex systems edge cases are not the exception, they are the norm. Humans are the flexibility of the system, the oil to our corporate machine. A lot of what is communicated on a natural human interaction cannot be written down in forms or in a process, it would require thousands, millions of them. Those approvals you are now requiring might slow down the whole process, costing more than the eventual stealing; the checklist could be blocking some legitimate decisions to leave something out; the standardized feedback process might actually be harming a lot of people that struggle with the style, being unable to adapt it to their team and peers. Of course we can and should keep improving the processes and tools and try to fit everyone, but at some point the cost of customization does not pay off, instead, it’s much more effective if we leverage individuals and interactions natural nuances and flexibility to tame the complexity of things such as software development.

Logic excludes — by definition — nuances, and since truth resides exclusively in the nuances, it is “a useless instrument for finding Truth in the moral and political sciences”. […]

Things are too complicated to be expressed in words; by doing so, you kill humans.

– Nassim Taleb — Antifragile

Next is “Working software over comprehensive documentation”, which could also be interpreted as “empirical over theoretical”. Let me try and make the connection to the main point: software development is just too complex so we embrace uncertainty instead. Traditionally, we try to do the best brainstorm with stakeholders, document everything, plan the architecture with hundreds of diagrams and so on in hope to build the perfect system. After a long time building it, when we put it out we always discover it is far from perfect, we find out we got the requirements wrong, the needs wrong, and many bugs show up, no matter how much paper trail you left behind. People get scared, so they default to putting even more effort in upfront design, I saw it happening quite many times.

Mother Nature on the other hand took a different approach: it didn’t carefully designed its creatures, but rather got (random) ideas out there and let the environment show what works and what doesn’t, that’s how it got where it is, with such advanced creatures and the ability to always survive and evolve. Just like Mother Nature, the real world is ruthless, the users, the market and everyone else touching our software will quickly point the flaws in our design with brutal honesty, even if by accident, and this is the most powerful feedback we can leverage. Production is the best QA there is. The solution for creating a perfect system then is not to design a perfect system, but to put an imperfect system out there earlier, aka working software, and let the best feedback mechanism we have do its work naturally.

Brenan Keller on Twitter: "A QA engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd. First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone. / Twitter"

A QA engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 99999999999 beers. Orders a lizard. Orders -1 beers. Orders a ueicbksjdhd. First real customer walks in and asks where the bathroom is. The bar bursts into flames, killing everyone.

The “comprehensive documentation” part is sometimes also interpreted not as the UML you wrote before development in old waterfall projects, but docs written during or after development, comments, wikis and diagrams on how pieces of the system works. It regularly becomes a point of discussion in the manifesto, as it is considered a good virtue that your whole system is well documented even in agile environments, common sense tells you that the more detailed the better, but there is actually a diminishing return here. In a complex system, you can’t actually document everything (nevermind keeping it up to date), there are so many nuances and intricacies that are simply too hard to convey in written text, with other means such as human interactions (or like pair programming) being much more effective. And it’s all fine, you don’t actually need devs to know how everything works to be able to build stuff on top of it

As with so many programming management problems, the technical memorandum problem gets worse nonlinearly as size increases. With 10 people, documents can simply be numbered. With 100 people, several linear sequences will often suffice. With 1000, scattered inevitably over several physical locations, the need for a structured workbook increases and the size of the workbook increases. How then shall the mechanics be handled? […]

We quickly decided that each programmer should see all the material, i.e., should have a copy of the workbook in their office.

Of critical importance is timely updating. The workbook must be current. This is very difficult to do if whole documents must be retyped for changes. In a looseleaf book, however, only pages need to be changed. We had available a computer-driven text-editing system, and this proved invaluable for timely maintenance. Offset masters were prepared directly on the computer printer, and turnaround time was less than a day. The recipient of all these updated pages has an assimilation problem, however. When they first receives a changed page, they want to know, “What has been changed?” When they later consult it, they want to know, “What is the definition today?”[…]

Our project had not been under way six months before we hit another problem. The workbook was about five feet thick! If we had stacked up the 100 copies serving programmers in our offices in Manhattan’s Time-Life Building, they would have towered above the building itself. Furthermore, the daily change distribution averaged two inches, some 150 pages to be interfiled in the whole. Maintenance of the workbook began to take a significant time from each workday […]

D. L. Parnas of Carnegie-Mellon University has proposed a still more radical solution. His thesis is that the programmer is most effective if shielded from, rather than exposed to the details of construction of system parts other than their own.

– Frederick P. Brooks Jr. — The Mythical Man-Month

If you got the point already, it should be clear that “Customer collaboration over contract negotiation” just follows the same rationale of the previous ones. In a software factory developing software projects, when things go wrong (and they always do), people go back to review what was actually agreed in the initial contract. This is a mistake because we know by now that software development is too complex, to think you would been able to have predicted the future back then. Still people decide to honour the contract, at the end, the customer is angry because the developer didn’t built what they wanted, and the developer is angry because of all the wasted effort.

I’m guilty of have fallen in the same trap in the past, when, after loosing quite some money (and sleep) in a software project, I tightened even more the rules of the contracts in the next projects, basically saying that the customer could no longer change their minds during the duration of the projects, helping me reducing the deadlines overrun. The result was useless software being delivered to unhappy customers, with deadlines still being broken, everyone was miserable, except the contract.

The core issue is the same as the previous one, just like you cannot have processes that models the whole of a complex system, and you cannot design all the details of a complex system upfront, you can’t also predict what needs to be built exactly, when it will be done, and all the problems that may happen to write it down in a contract. Tightening the contract is a loosing game for a software project, rather, a more flexible contract that makes few guarantees about how, what or when, while harder to get signed, give a lot of room for navigating the complexities of software development, allowing much closer customer collaboration with better feedback loops and change of plans, thus making the customer happy — and happy customers don’t sue you.

Let me repeat the core value behind agile again: software development is too damn complex, move away from trying to capture all the variables, and embrace the strategies to move in face of uncertainty

Last but definitely not least, “Responding to change over following a plan”. This is probably the value agile is most remembered for, in a sharp contrast to waterfall detailed plans. It should also be the most clear example of the point I’m trying to make — predicting the future is definitively the most complex thing there is.

As much self-evident as this may seem nowadays, it still must be stated. Lots of people still believe that the more detailed plan you have the better you can control your future. Since early age we hear that “you got to know where you are going”, “you got to have a plan for your future”, from grown-ups and pseudo-intellectuals — even though if you ask, none of them planned to be where they are right now, life just happened.

The rational flanêur is someone who, unlike a tourist, makes a decision at every step to revise his schedule, so he can imbibe things based on new information, what Nero was trying to practice in his travels, often guided by his sense of smell. The flâneur is not a prisoner of a plan. Tourism, actual or figurative, is imbued with the teleological illusion; it assumes completeness of a vision and gets on locked into a hard-to-revise program, while the flâneur continuously — and, what is crucial, rationally — modifies his targets as he acquires information. […]

The error of thinking you know exactly where you are going and assuming you know today what your preferences will be tomorrow has an associated one. It it the illusion of thinking that others, too, know where they are going, and the they would tell you what they want if you just asked them.

– Nassim Taleb — Antifragile

Predicting the future is so hard, that with time you learn that even projects considered classic waterfall fits, such as construction, actually miss deadlines and have to adapt and change details of the plan all the time, with a growing impact as the complexity of the world increases:

But the puzzle was that such underestimations did not seem to exist in the past century or so, though we were dealing with the very same humans, endowed with the same biases. Many large-scale projects a century and a half ago were completed on time; many of the tall buildings and monuments we see today are not just more elegant than modernistic structures but were completed within, and often ahead of, schedule. These include not just the Empire State Building (still standing in New York), but the London Crystal Palace, erected for the Great Exhibition of 1851, the hallmark of the Victorian reign, based on the inventive ideas of a gardener. The Palace, which housed the exhibition, went from concept to grand opening in just nine months. […]

The obvious is usually missed here: the Crystal Palace project did not use computers, and the parts were built not far from the source, with a small number of business involved in the supply chain. Further, there were no business schools at the time to teach something called “project management” and increase overconfidence. There were no consulting firms. The agency problem (which we defined as the divergence between the interest of the agent and that of his client) was not significant. In other words, it was a much more linear economy — less complex — than today. And we have more nonlinearities-asymmetries, convexities — in today’s world.

Black Swan effects are necessarily increasing, as a result of complexity, interdependence between parts, globalization, and the beastly thing called “efficiency” […] The world is getting less and less predictable, and we rely more and more on technologies that have errors and interactions that are harder to estimate, let alone predict.

– Nassim Taleb — Antifragile

Finally, I must make a note of the text after the values on the agile manifesto:

That is, while there is value in the items on the right, we value the items on the left more.

I, like the manifesto, am not dismissing the value of having a plan, processes, contracts etc, it is merely a pushback to on things that may naturally creep in to give it some balance.

Understanding the reasoning behind why agile values work can hopefully give us answers to what is not covered by the manifesto. For example, does agile scale? Scaling increases complexity, as we saw agile is about reducing the management of complexity, so inevitably the more you scale, the less agile you become. If you look at how Mother Nature does it, it doesn’t scale, at least not how we do with companies, Nature has fairly horizontal hierarchy, there is no 5km tall animals, or animals managing thousands of other animals, or a single animal concentrating all the power. Instead, we have many small, independent creatures, managing small amounts of complexity each, which allows them to keep adapting and surviving together, Nature is agile.

Thanks for reading!

Originally published on my blog

Software is famously known to have flaws, good practices and automated tests have helped us reduce issues, but most of the industry pretty much gave up on the idea of having software completely free of bugs, instead we moved our strategy to delivering early and iterating faster, letting the real world be our QA, and the real world is indeed the best QA there is. But is that really the only way? How come we always have gaps when testing our sofware? I mean it’s computers, why can’t we like simulate all the bits and be sure to check everything? What if we do unit tests? Integration tests? E2E tests? Contract tests? Have 100% coverage? Do some Mutation tests? If we apply all the tricks we can, can we feel safe with code which written by other humans? What if the code was not written by a human, but a machine, like Copilot?

The idea for this blogpost came from an insight by Nicholas Tietz, he suggests that maybe with AIs like Copilot evolving, it could be the case that we humans focus more on the specification and verification of what we want to be done, and the machine could automate the coding part.

If you think about it, it’s kinda like doing TDD, in which on the first step you need to “write a test that passes iff the feature’s specifications are met”. Then, on the seconds step you see it fail, and on the third step you “write the simplest code that passes the new test”. This is the step that could be automated by a machine.

Most importantly, and often overlooked by TDD beginners, it the keyword simplest. Write just enough to make the test green, nothing more, because this way you will be forced to write more tests for the functionality that you know is still missing, and then when you do, you are sure that functionality is tested as well. This is also a great way to think about what a machine will do, more often then not, given an objective, machines WILL find the simplest solution, often surprising us humans.

This means our tests need to ensure complete behaviour in order to prevent AI (or other devs) from taking shortcuts. Can we do that? Write our tests so well that it will force our code to behave properly? Let’s try, let’s start our project:

npm init

Unit Tests

Create a test file transaction.test.js with a sample test, just to check our tests are working:

test("it works", () => {
  expect(1 + 1).toBe(2);
});

Run it, all is working:

npx jest

 PASS  ./transaction.test.js

Alright, so now what do we actually want to build? My idea is to build a simple function to transfer money from Alice to Bob. To do that I wrote a test that creates a sqlite db, a balance table, put 10 pesos on Alice balance and 5 pesos on Bob balance. Then I call the function transferMoney, which doesn’t exist yet but I intend to write it.

This function should transfer 10 pesos from Alice to Bob, we verify that by checking that the balances are correct, here is the the test:

const sqlite3 = require("better-sqlite3");
const transaction = require("./transaction");

describe("transferMoney", () => {
  let db;

  beforeEach(() => {
    db = sqlite3(':memory:');
    db.exec("CREATE TABLE balance (account TEXT, amount INT)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Alice', 10)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Bob', 5)");
  });

  it("transfers money from Alice to Bob", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: 10 });
    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(0);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });
});

We run the test again, with npx jest, and find several errors: first of all we don't have sqlite, please install it with npm install better-sqlite3; then it will say it cannot find module ./transaction, create transaction.js file and export a transferMoney function from it; with setup out of the way, now we need to implement the most simple thing to make the test pass:

function transferMoney(db) {
    db.prepare("UPDATE balance SET amount = 0 WHERE account = 'Alice'").run();
    db.prepare("UPDATE balance SET amount = 15 WHERE account = 'Bob'").run();
}

module.exports = { transferMoney };

Run it again:

npx jest
PASS  ./transaction.test.js
 ✓ transfers money from Alice to Bob (13 ms)

Our tests passes, we did TDD, and we have 100% coverage (you can check by using the --coverage flag), so our code must be perfect and free of bugs, right? Right!? What? Hard-coded values you say?

Well, obviously, our code sucks, we didn’t even had to use the parameters as our current test suite does not force us to. How can we force parameter usage? One solution is to write more tests with a slightly different value, to force the parameters to be used — but that feels a bit stupid.

This why TDD has a last step: Refactoring, where you can reorganize the code, preserving the functionality without adding anything new, while the tests still passes. So let’s rewrite the code like something closer to what we expected (you might not like the result either, but bare with me):

function transferMoney(db, { from, to, amount }) {
    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

Run the tests again, they pass.

You can argue that a machine would not refactor the code like we did, but to be honest, I don’t think it would even need the refactor. Even though machines take shortcuts, on the current state of the AIs we have, it would most likely make use of those parameters from the start, as human developers usually do too. The reason for that it that AI is trained on gazillions of human code (for now), so it’s more likely to replicate a longer, more common solution, than to come up with a simpler but creative one. So let’s assume this is the case and move forward.

Cool, we have a feature to transfer money working, it seems that TDD helped us implementing the happy path, we can ship to production and see if any problems happen. Do you already know which ones? Take a look on the previous code and find at least two things you think might go wrong with it.

Let’s start with an easy one, the famous billion-dollar mistake:

it("does not transfer any money when amount is undefined", () => {
  transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });

  const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
  expect(alice.amount).toBe(10);

  const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
  expect(bob.amount).toBe(15);
});

Run the tests, we get an error:

npx jest
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (56 ms)
    ✕ sending undefined does not transfer any amount (9 ms)

  ● transferMoney › sending undefined does not transfer any amount

    expect(received).toBe(expected) // Object.is equality

    Expected: 10
    Received: null

How can we prevent that? Plus all other nulls, undefineds or invalid types that can reach our function?

Type Checking

We could just add an if condition and return or throw if the amount is null, or undefined, an object instead of a number, etc, and also check on every other argument, not forgetting to write a unit test for each case or… we can just add types. Types constrain the solution space, it could guide a computer to generate code only within the types we added. Once you go very strict about it, some even say that if it compiles, it works.

On JavaScript world it’s very easy to move from dynamic to static types with TypeScript, so let’s rename our files to transaction.ts and transaction.test.ts, and type all the variables. This is the typed test file:

import * as sqlite3 from "better-sqlite3";
import * as transaction from "./transaction";

describe("transferMoney", () => {
  let db : sqlite3.Database;

  beforeEach(() => {
    db = sqlite3(':memory:');
    db.exec("CREATE TABLE balance (account TEXT, amount INT)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Alice', 10)");
    db.exec("INSERT INTO balance(account, amount) VALUES ('Bob', 5)");
  });

  it("transfers money from Alice to Bob", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: 10 });
    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(0);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });

  it("sending undefined does not transfer any amount", () => {
    transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });

    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    expect(alice.amount).toBe(10);

    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBe(15);
  });
});

And the typed implementation:

import { Database } from "better-sqlite3";

export function transferMoney(
    db: Database,
    { from, to, amount }: { from: string, to: string, amount: number }
) {
    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

Assuming you have typescript installed (npm -g install typescript), you can now compile and run the tests:

tsc --strict *.ts && npx jest transaction.test.js

We get an error:

transaction.test.ts:24:63 - error TS2322: Type 'undefined' is not assignable to type 'number'.

24     transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount: undefined });
                                                                 ~~~~~~

Notice it’s an error on the test file, but the test didn’t fail, it didn’t even run because sending undefined is not allowed. Now for our unwanted behaviour we cannot even write a test, it’s actually forbidden, you are forced to delete that test, that’s very cool!

We are so strict now, that certainly means we won’t have issues with our code anymore right? No? We certainly closed more gaps now, and we can keep going, maybe write more and more unit tests covering things we can remember that can cause problems, maybe getting more creative about types, like introducing monads to handle side-effects and see if it helps.

But even if we go through this effort, what bugs may still be lurking out there? Let’s say we wrote as much unit tests as we could and typed everything, how do we know we are still not missing some edge case? Maybe try random stuff and see if something breaks? Yeah! That’s a good idea!

Property-Based Testing

In Property-Based Testing, you generate lots of samples to give to your function and write some assertions about it. Because of this sampling, Property-Based is also known as Fuzz Testing, but I like the name Property-Based Testing better, because it highlights the most important concept to grasp about this kind of test: that you should assert on properties that your output must hold*,* not on specific examples like unit tests, on properties. It requires a different mindset to think about properties, personally for me it was hard to do, since I was for many years writing just regular unit tests.

For example, if you are writing a simple sum function, to sum A and B. You cannot write a test to generate a thousand random numbers for A, and for B, and then just assert the result of sum(A, B) should be A + B. That’s too easy! You are simply replicating the behaviour of your actual implementation, that doesn’t help anyone, both your code and test could be wrong together. Instead you focus on properties, for example, no matter which of those 1000 random numbers are summed together, two positives should always return a positive result, if one of them is negative then is should have the sign of the greater of the two, or any other properties you can think about. If a counter-example is found, the test will tell you.

Let’s write one:

import fc from "fast-check";

// ...

it('ends up with more money on the receiving account', () => {
  fc.assert(fc.property(fc.integer(), amount => {
    db.prepare("UPDATE balance SET amount = 10").run();

    try {
      transaction.transferMoney(db, { from: 'Alice', to: 'Bob', amount });
    } catch (_) { return; }

    const alice = db.prepare("SELECT amount FROM balance WHERE account = 'Alice'").get();
    const bob = db.prepare("SELECT amount FROM balance WHERE account = 'Bob'").get();
    expect(bob.amount).toBeGreaterThan(alice.amount);
  }));
});

Basically what this test does is using fast-check to generate random integers (with fc.integer) which is then passed to our callback to make and assertion. And what can we assert it’s always true when transferring money from Alice to Bob? What is a property of a transference? Here they both start with 10 pesos, so the property we check here is that no matter how much money is being transferred, Bob should always end up with more money than Alice. We also ignore any errors, as we are only interested in the successful transfers for now.

Let’s run it!

tsc --strict *.ts && npx jest transaction.test.js
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (6 ms)
    ✕ ends up with more money on the receiving account (21 ms)

  ● transferMoney › ends up with more money on the receiving account

    Property failed after 1 tests
    { seed: -1111144902, path: "0:0", endOnFailure: true }
    Counterexample: [0]
    Shrunk 1 time(s)
    Got error: Error: expect(received).toBeGreaterThan(expected)

    Expected: > 10
    Received:   10

It found one counterexample: 0. Yeah, Alice is trying to transfer zero pesos to Bob, now it starts coming down to specs of the system you are building, is this supposed to be allowed? Let’s say no, because we think it’s too stupid to do a zero transfer, and we do a simple fix for it on the top of our transferMoney function:

if (amount == 0) throw "cannot transfer 0 pesos";

Great, the case for 0 is fixed, let’s run again

tsc --strict *.ts && npx jest transaction.test.js
 FAIL  ./transaction.test.js
  transferMoney
    ✓ transfers money from Alice to Bob (5 ms)
    ✕ ends up with more money on the receiving account (53 ms)

  ● transferMoney › ends up with more money on the receiving account

    Property failed after 2 tests
    { seed: -1121527945, path: "1:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0", endOnFailure: true }
    Counterexample: [-1]
    Shrunk 26 time(s)
    Got error: Error: expect(received).toBeGreaterThan(expected)

    Expected: > 11
    Received:   9

Uh oh, Alice sent -1, we haven’t though about that! Like a QA the property-based testing tries to reduce the counterexample to the simplest one that fails, telling us that allowing people to transfer negative amount of money breaks our property. Transferring negative money is not something our business want, so we improve our check:

if (amount <= 0) throw "only positive amounts are allowed to be transferred";

Run the tests again and you will see that they pass, great! What other properties can we ensure? Let’s say our business is a good bank, the kind that do not want anyone to have debt. One of the properties we could check then is that Alice always have a positive balance after every transfer:

expect(alice.amount).toBeGreaterThanOrEqual(0);

Run it again:

Property failed after 2 tests
    { seed: -1313110683, path: "1:1:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:2", endOnFailure: true }
    Counterexample: [11]
    Shrunk 25 time(s)
    Got error: Error: expect(received).toBeGreaterThanOrEqual(expected)

Expected: >= 0
Received:    -1

The counterexample found is 11, this made Alice end up with -1 because it’s more money than there is on the account balance. Solving that is a little more tricky, we need to first check how much balance there is left on the account, let’s do it:

const balance_from = db.prepare("SELECT amount FROM balance WHERE account = ?").get(from);

if (amount > balance_from.amount) throw "there is not enough money to be transferred";

Run the tests again, they pass, no counter-examples found! Awesome, our code is stronger than ever.

You can go on and do some more assertions, like if the account does not exist when somebody send something other than “Alice” and “Bob”, but I want to focus on something else, which you probably noticed from the first example already, as this is the classic database 101 example. “Where is the transaction? For God’s sake wrap this in a transaction already!” — you may say. But why do we need it? How can we prove that we need it? Here is our current code:

export function transferMoney(
    db: Database,
    { from, to, amount }: { from: string, to: string, amount: number }
) {
    if (amount <= 0) throw "only positive amounts are allowed to be transferred";

    const balance_from = db.prepare("SELECT amount FROM balance WHERE account = ?").get(from);
    if (amount > balance_from.amount) throw "there is not enough money to be transferred";

    db.prepare("UPDATE balance SET amount = amount - ? WHERE account = ?").run(amount, from);
    db.prepare("UPDATE balance SET amount = amount + ? WHERE account = ?").run(amount, to);
}

So far our unit tests didn’t require us to use a transaction, our types also didn’t, neither did our property-based tests, no one found a counterexample for a transaction. We do know, however, from experience, that our code can fail at each of those lines of code, our server could die, leaving the database in an invalid state in the middle of the transfer, not to mention when there are multiple instances running this code in parallel and updating Alice and Bob balances at the same time. To solve it with unit tests we could mock each of those functions and see what happens if they fail, but if this was much larger, mocking every line of code and seeing what happens if they fail is very boring to say the least. Fortunately, there is a better way.

Formal Specification

If property based testing got you thinking about properties, formal specification will get you thinking about properties AND state. As you may have heard, state is the root of all evil. That’s why functional programming folks want to push it away with pure functions, to a place we can reason better about it. Having stateless functions makes your program much easier to reason about, but you do need state at some point, otherwise your code is completely useless ( like Haskell).

Our previous code have a bunch of hidden state, since if you really think about it, your database is really just a global mutable variable, if some function changing it fails in the middle, or two functions change it at the same time, you will get in an inconsistent state. Good luck debugging that.

Now, Formal Specification tricky part is that it requires us to model all state transitions we have, but then it gives you the power to verify all the states your code can reach, not just simulate random scenarios, but actually checking ALL possible paths. The idea is to leverage the power of mathematics to build a full specification, allowing you to check every intermediary step if you want to (helping to check for issues in case your program halts).

For that we are going to use TLA+, a formal specification language created by Dr. Leslie Lamport, he created it because he found that formal specification helped him to prove the correctness of distributed systems, the most famous of them being the Paxos Protocol. I will not go in details about TLA+ here (I’m a beginner myself), but if you want to learn more about it, I really recommend watching Dr. Leslie’s video-tutorials, they are really fun, a bit weird, but mostly fun.

Enough talk, I will just paste the initial version of the specification here and we can go over it:

At first this code looks very alien, but we can render it for a better visualization with LaTeX. If you have some experience with math symbols, that can help:

Okay let me explain. We are basically trying to replicate our code behaviour and all state changes here. We start with two constants: accounts, those will be Alice and Bob; and initialBalance, as we will put some initial money for each. Then we have two variables: balances, changing as the accounts transfer money to each other; and msgs, which we will use to model the queries that are being sent to the database, more specifically here only the UPDATE statements, we skip SELECT and pretend we can read the balances directly to make the spec simpler.

Then we have four functions. On Init, we define our initial state, nothing special.

On TransferMoney we do basically a replication of our current JavaScript implementation. The way to think about it is a bit different though, it’s not imperative as in “do this update to the database”, it’s actually declarative like “we allow a state transition from wathever messages are already on the db queue union two new messages for updating to and from balances, as long as the from account has enough money.

On DbUpdate we are simulating the database actually changing the state, basically it’s saying that if the msgs queue is not empty, we can switch to a state where the msg queue has all items but this first one, and the balances have this new amount on the specified msg.account

Now the Next is the interesting part of TLA+, it specifies what state changes your system can go through at each time step, in our case it’s DbUpdate OR TransferMoney for each of the accounts we have, for any amount of money they might have in the balance, as long as it’s positive. This is basically to say, the database could be updating and all people could be transfering money, back and forth, all the time, at the same time. Pretty much what happens with a system at scale, distributed or under high load, and we want to see what can happen there, what paths it can go through.

Then we have a helper, it’s just a simple function to sum balances that we will use next.

Finally, we have our Invariants! Those are the things we want to verify. Those specifications need to remain true all the time, no matter where we stop on the execution, no matter how many computers we add.

Those, unfortunatly, won’t come automatically, they will come from thinking. What properties do you want your system to hold? This depends on your goal, on business requirements, on, well, specifications, we are just formalizing them. Like unit tests, they might have gaps, but being at a much higher-level, a single specification can cover way more ground. This is what Nicholas Tietz meant, we can let Copilot or future AIs generate the code, while we humans focus on the specifications. A language like TLA+ may allow us to define them at the highest level that is still non-ambiguous.

Our first invariant TypeOK is a simple one, it’s akim to type checks, we are making sure that on the msgs sent to the database account has always to belong in the constant accounts, and that the amount must be an integer.

Next is BalancesAlwaysPositive, we are a no-debt bank remember, so we assert that any point in time no balance is allowed to go negative, we already checked that with the property-based test and fixed it by introducing and if (amount > balance_from.amount) but could we be missing something?

Finally we have TotalMoneyStable, this is a new rule. Since we don’t have deposits or withdraws yet, money cannot just appear out of nowhere right? If Alice started with 10, and Bob started with 10, and there is no one else in the system, but then suddently we have 25 pesos, then where did this money came from? So we add this invariant to make sure that at any point in time the total amount is the same as the initial one, since we have a closed system.

From TLA+ Toolbox, we can run our specification, and see if we violate any invariants. This is the model configuration we use:

Nothing surprising here, just one small detail is that we put just 2 pesos in each account, since if we put much more than that, the state space to verify grows exponentially large. Even though definition is in math, we are unfortunately still limited by mundane computing power.

Alright so let’s run the model! 🥁🥁🥁🥁🥁

Oh no, we have an error!

The invariant TotalMoneyStable is violated, how did it happen? We can actually follow the steps the model found. First we initialize with Alice and Bob having 2 pesos each. Next we add two msgs, that Bob should have 1 peso and Alice should have 3, this means Bob is doing a single peso transfer. Next DbUpdate kicks in to update the balances and… ops, Alice has 2 pesos and Bob 1. One peso completely vanished of the system. Well, we know there is still a message left to be processed, but what if your system died at that point? Those invariants guarantee to be always consistent, no eventuallyconsistent, always consistent.

The problem happens because our updates happen sequentially. What can we do to solve that? We can do both updates at once, this should solve our problems. Let’s update our spec:

Okay now instead of sending 2 msgs to update the database, we are sending a single one, with from, to, and amount. Let’s see what happens:

Oh no! Invariant BalanceAlwaysPositive was violated. How did that happen? Let’s follow the steps. Initial balances, okay, then Bob did a transfer to Alice for 1 peso, but then before the database had time to process the update, Bob did yet another transfer to Alice, for 2 pesos this time, most likely it was another instance of our service running in parallel. As a result, once DbUpdate processes both msgs, Bob ends up with a negative amount, and that’s terrible for our bank because we would have to charge interest and make money! No bank wants that I’m sure.

Alright, I have an idea for one final change:

Now additionally to updating both balances at once, we no longer send the amount to be transferred, but actually the final resulting amount for both balances. This way we are carrying the state inside the message, which means that even conflicting messages when processed will leave the state stable way, although it also means last-write-wins.

Let’s run our model again:

Wow no errors! TLA+ could not find any issues on the 3.196 possible states it found. We solved folks, that’s it, and we could do it without using transaction or locks! Great!

If you feel like it you can keep going, what would happen if we add another account? Say Alice, Bob and Carlo? Can you simulate the state changes that happen during a database transaction? Does it solve our problems?

Conclusion

Making a sure a function free of bugs is very hard, and the definition what “free of bugs” mean actually depends on your specification. On our every day jobs most of the issues are avoided by plain human experience and judgment, if you could not see the code, guaranteeing it is working is super difficult. Tools like property-testing and TLA+ allow us to get closer and closer, and while I have no idea if those tools will indeed help us bridge the connection with AI, I know they are fun to learn.

Thanks for reaching the end of this blogpost, let me know how you like it. I need to end up with a warning, though: we only tested the algorithm here, there is a lot more to software than that, this is just a tiny part.

Now I kindly ask you to not leave yet and save the links listed below to read or watch later, they are more interesting than this one, and modeled my thinking about testing.

More Content

They Write the Right Stuff — an article from 1996, on how space engineers get their softwared to be free of bugs

We need to talk about testing — on how our automated testing is still not even close to what advanced human testers can do

Making Impossible States Impossible — making the best use of your static types

Formal verification applied — intro to TLA+ for devs

Seems like a very good deal, right? It’s 60% so you win most of the time, play it right and you will get rich! Still, on an experiment, 28% of participants went bust. Why is that so?

You go bust when you bet too much, to see that in practice I wrote a notebook with some simulations. For example, if participants where to bet constanly 50% of their money:

50% bet

Most of them whet bust, 86%, not a great strategy. What if we play very safe? Always bet only 10% of your money:

10% bet

Awesome, nobody went bust, that would be a very good strategy. Problem is, it also took them many rounds to reach the max profit (I capped at 10k), now, real world is finite, we don’t live forever, imagine for example if this 420 average would mean years. Ain’t nobody got time for that.

On the original experiment, participants were limited to 300 bets. So how can we maximize our profits, while minimizing the chance to go bust?

An old theory to improve our bets is to use the Expected Utility Hypothesis. The idea is that instead of a fixed strategy (like 10% or 50%) you should always bet on whatever maximizes your “utility”, the “utility” definition may vary, but Bernoulli defined it in the 1700s as:

utility = log(wealth)

So, the richer you are, the smaller the utility of additional money.

Then we can decide how much we are going to bet on each round like this:

for bet in range(0, cash + 1):
  utility = 0.6 * math.log(cash + bet) + 0.4 * math.log(cash - bet)

Then you select whatever bet has the biggest utility. This issues great results:

Expected Utility Hypothesis

Only 5% went bust now, and average time to max profit went down to 284, much better!

There is also an alternative to Expected Utility Hypothesis called Prospect Theory, which tries to map better on how we humans actually take risks. On prospect theory the “utility” you give you an outcome heavily relies on weather it’s positive or negative.

As risk-averse humans, people would not take a bet where you have 60% chance of winning $100, but 40% of losing $100. The pain of losing $100 could only be compensated if the offer was like 60% chance of winning $200 vs 40% of losing $100. Still, lots of people won’t take this bet.

I’ve managed to simulate a prospect value function like this:

You can see that the curve on the upper side (positive outcomes) is at 4, while on the lower side (negative outcomes) is much steeper, at -8, showing the idea that our pain of losing money outweighs our happiness in getting some more.

So going back to the bet, how much of $25 would you bet for 60% chance of doubling it? It seems like nothing. If my calculations are correct according to prospect theory, at 60% the chance of winning never outweighs the loss so utility is always negative, it feels like you are always losing. But it does give the smallest negative utility always at the maximum amount, starting with $25 (-1.2 utility). Needless to say this is not a very good strategy:

Prospect Theory

I don’t know if I did some misinterpretation of prospect theory or what, but maybe it explains why so many humans went bust on the original experiment. I also tried prospect theory considering everything as a profit starting with the $25, but then it behaves just like the Expected Utility Hypothesis because there is no asymmetry from losses nor very high or very low percentages that would change our perception.

Finally, the last method is actually the simplest one to calculate, Kelly criterion was described John Larry Kelly Jr. at Bell Labs in 1956. This is the formula:

bet = cash * (bet_chance * 2 - 1)

Which for our case means you should always bet 20% of your money because this is a 60% chance bet. That’s it!

Kelly Criterion

That’s it! I had fun playing with those simulations, hope it can be useful to you. Here is the link to the notebook if you also want to play with it yourself.

Cheers!