Evals Before Features: How to Test a Strands Agent You Cannot Unit-Test

Write the eval cases before the prompt, run them on every commit with a fake model and on every model change with a judge, and let a threshold gate decide the release.

We upgraded the model behind our RAG support chatbot from GPT-4.1 to GPT-5. The change looked harmless: the same prompts and tools over the same retrieval. We had unit tests for the Python implementation logic and manual checks of the answers, and both passed before the release.

But we were wrong: we found out from users and they were not happy. The answers had become less correct, less relevant and less helpful, because the way the new model reasoned had changed under the hood. It cost us reputation and another release cycle to get the old behaviour back. Not to mention the knock-on effect on the planned upcoming releases.

Nothing in our code had changed, so nothing in our tests failed. The thing that had changed was the model’s choices, and we had nothing that measured or tested them.

Lesson and my position: write the evals before the agent logic and the prompts. Only once the business behaviour is written down as cases can you write a correct implementation of it in the agents. The same cases tell you within minutes, before the release and not from user complaints, whether a model…