A problem well-defined is half solved: Measuring Hypothesis Quality at Booking.com
Even after an experiment (or A/B test) ends, it may not be clear what action to take or what was learned.
The test may be properly powered, run for the right amount of time, and produce a statistically clear result. But if the hypothesis never made the original problem, reasoning for the change, or decision criteria explicit, the result is difficult to interpret.
This is why hypothesis quality matters. A well-written hypothesis does more than improve an experiment at launch; it improves the quality of the decision that follows. By recording the evidence behind an idea, the user response we expect, and the metrics that would support or challenge it, teams create a clearer standard for interpreting the outcome.
The benefits compound over time. A hypothesis that records only an implementation is difficult for another team (or an AI system), to find, compare, or reuse. A hypothesis that captures the underlying theory and expected behavioural response becomes a searchable explanation of why an experiment was run. Over time, this gives Booking.com a richer map of what we have learned: which user problems recur, which mechanisms tend to work, and where our assumptions repeatedly fail.
In our earlier posts on experimentation quality and scaling that quality across Booking.com, we described how shared standards, measurement, education, and tooling can make better experimentation easier to practice at scale. This post focuses on the next layer of that work: how we are helping teams make the reasoning inside their hypotheses clearer, more consistent, and more useful long after an experiment has ended.
The Hypothesis Template
A hypothesis is not just a description of the feature we are about to build. It is the bridge between an observation, a theory about user behaviour, and the evidence that could support or challenge that theory. When that bridge is weak, the experiment may still run correctly, even score highly on other pillars of experimentation quality, but the result becomes much harder to interpret.
Consider a mock hypothesis such as:
“By making the button rounded, we will increase conversion.”
It is short, but it leaves critical questions unanswered. What problem are we solving? What evidence do we have to support this hypothesis? Why should shape change behaviour? Which users are affected? What should they do differently? If conversion does move, do we understand why well enough to apply the learning elsewhere?
To address these questions, we developed internal guidance on writing a solid hypothesis around a simple structure:
Based on [previous evidence] we think [theory].
We believe we can [outcome] by changing [implementation] for [target group].
We know this hypothesis is supported if they will [new behaviour] and we see [effect on metrics].
We do this because of [motivation].
The template is useful because it captures more than the implementation. It asks us to record the prior evidence, the theory being tested, the audience, the expected user response, the observable effects, and the reason the experiment matters. It also encourages teams to connect immediate behavioural metrics with more distant outcomes, rather than relying on a single end metric to explain the whole causal chain
Hypothesis Quality at Scale
The hypothesis template initially gave us a shared vocabulary, but coaching and internal reviews couldn’t keep pace with our experimentation volume. Scaling required a lightweight mechanism to assess adoption of our guidelines and deliver real-time feedback directly to teams as they register hypotheses.
The Hypothesis Quality Score was designed to address that gap. As a lightweight, advisory score from 0 to 3, it evaluates how clearly and completely an experiment hypothesis is written. Crucially, the score attaches to the hypothesis, not the author. It is not meant to judge the quality of an idea, prioritize experiments, or replace product intuition. Instead, it simply creates a deliberate pause between “let’s test this” and “what exactly do we believe will happen?”
What does it evaluate?
The checker looks for five components of a useful, testable hypothesis:
- Prior evidence or insight: Why is this idea worth testing?
- Change being made: What specific intervention or treatment is different from the baseline?
- Target audience or sample: Which users or traffic segment will experience the change?
- Expected user response: How should users think, feel, or behave differently?
- Observable effects or metrics: What measurable outcomes would support or challenge the hypothesis?
How is the score calculated?
To evaluate hypotheses consistently at scale, an LLM assesses the text against the five pillars. Each element is marked as present or missing, giving us an overall score mapped as follows:
Improving Quality through the Platform & Community
Since Booking.com uses an in-house experimentation platform, we have the unique advantage of being able to enhance the tool itself to drive better hypothesis quality. Our objective was to ensure that writing high-quality hypotheses becomes an effortless, actionable, and standard practice.
Get Luka Beverin’s stories in your inbox
Join Medium for free to get updates from this writer.
Proactive Intervention — Experiment Creation: To assist users as they draft their hypothesis, we embedded the checker directly into the experiment creation form. This lightweight large language model automatically evaluates the text and responds within 10 seconds on average, clearly highlighting failed pillars and pinpointing areas for improvement.
Learn and improve — The Quality Tab: As mentioned in our previous post on experimentation quality, we leverage the “Quality Tab” to highlight violations, explain the issue, and provide instructions on how to fix it. This feature empowers users to immediately act on the feedback and refine their hypotheses.
Make it actionable — Community engagement: Instead of the core experimentation platform team acting as a centralized bottleneck, we rely on embedded experimentation ambassadors within each product track. A key part of their role is explaining the actual value of clear hypotheses to their teams, ensuring this is viewed as a critical product practice rather than a compliance exercise. To support them, we built a dashboard that automatically flags weak hypotheses, enabling these reviewers to quickly spot where teams are struggling, step in with guidance, and scale best practices locally.
Results
After releasing the checker, it was important for us to understand the broader impact of this feature on our teams and their experimentation habits. To achieve this, we took advantage of the fact that our experimentation platform logs all hypothesis text, allowing us to backfill the quality score for historical experiments. This retrospective analysis was essential for validating the scoring framework and measuring the tool’s impact after the official rollout.
Since launching the feature, the share of product improvement experiments with an “Excellent” hypothesis score grew steadily from 62% to almost 90%. To put it another way, 9 out of 10 hypotheses currently being written satisfy at least four of the five guideline pillars.
However, top-line improvements only tell part of the story; drilling into individual pillars reveals where the real progress happened. In product improvement experiments, inclusion of Expected User Responses and Observable Metrics grew by 14 and 13 percentage points, respectively. Clearer expected responses help us understand whether user behavior actually aligns with our theory. Meanwhile, pre-defining metrics guards against goalpost-shifting and enables downstream tooling to recommend the right metrics for future experiments.
Discussion
It’s important to acknowledge what the checker is and isn’t. It assesses whether a hypothesis clearly captures the evidence, change, audience, expected response, and metrics. It does not determine whether the idea is strategically valuable, or the experiment is well designed. Even an excellent hypothesis can fail to improve decision-making metrics when its implementation is buggy or if it’s poorly designed
This is also how we should interpret the increase from 62% to 88%. It shows that teams are documenting their reasoning more completely, but it does not prove that the checker alone led to better product decisions. The score is an operational signal: useful for making progress visible and identifying where teams may need support, but not a complete measure of design or decision quality.
There is always a risk that teams optimise for the score rather than the underlying practice. That is why we keep it advisory and pair it with education and human review. We also chose not to generate automatic rewrites. The goal is not to produce better-looking text, but to encourage teams to pause and clarify what they believe before running the experiment. The LLM supports that reflection; it does not replace human judgment.
Ultimately, the shift from manual reviews to LLM-powered checks has been transformative. By making hypothesis quality feedback available at the point of experiment creation, LLMs have helped integrate what was once a manual bottleneck into the experimentation workflow. More importantly, this structure creates a rich, searchable knowledge base of experiments. By feeding high-quality hypotheses into our systems, we are building an experiment improvement loop where LLMs can synthesize historical learnings, suggest relevant past insights for new ideas, and continuously refine our understanding of what drives better customer experience and product growth.
Acknowledgements
This work builds on contributions from teams across Booking.com. Special thanks to Luis Matute (Engineer) for leading the build, Feza Roheel (Engineer) for managing database architecture, and Melanie Muller (Product Manager) and Ilya Eroshkin (Data Science Manager) for their strategic guidance across product and data science.
