The relationship between hypothesis and prediction is many-to-many. A single hypothesis (i.e. proposed explanation for a phenomenon) might result in many different predictions (i.e. expected observable outcomes if the hypothesis is true); and a single prediction might be explained by many different hypotheses.

• Hypothesis 1: light helps plants grow.
• Hypothesis 2: heat helps plants grow.
• Prediction 1: if I put my plants in the warm sun, they will grow more.
• Prediction 2: if I put my plants in the cold cellar, they will grow less.

Both these hypotheses are consistent with both these predictions. If I put my plants in in the sun and they thrive, I will not know if this was because of the light or because of the heat. To disentangle these two, I need a prediction which is consistent with one of the hypotheses but not the other.

• Prediction 3: if I put my plants in a cold place under the sun, they will grow more.
• Prediction 4: if I put my plants in a warm dark place, they will grow more.

These two predictions are only consistent with one of the hypotheses. If we can test and reject one of them, we can pinpoint which hypothesis has been falsified.[1]

By teaching people to include a prediction in the their hypothesis, I may have been stunting their ability to effectively reason through these sorts of situations. By simplifying one thing too much, I have made other things more complicated.

If you are one of the many, many people potentially affected by my mistake, then I apologise for the confusion. I will try to do better next time.

Teachers are also forever learning. Please give us feedback so that we may improve. Thank you Erin and Craig for helping me grow. ❤️

[1] Plants probably need both heat and light, because life is never as simple as the contrived examples we use to explain things.

(This post was originally published on my blog.)

This might seem too obvious to even mention. Clearly we need to release features (e.g. speed) in order for those features to add value (e.g. velocity). But the connection between speed and velocity is more complicated than it might at first seem. Mantras such as “fail fast” and “rapid iteration” hint at a deeper relationship which I will try to make explicit here.

Every time we release a feature through an experiment, we get feedback from our customers about that feature. This feedback allows us to measure whether this feature will help our velocity and in turn decide whether to ship this particular release.

The same feedback also allows us to learn and improve future features we develop. Every time we release a feature through an experiment, we might learn something which will improve our chances of developing features that our users will use and love.

If we release more features per sprint, we will not just have more opportunities to ship; we will also have more opportunities to gather feedback and learn. It is this second order learning effect that “fail fast” and “rapid iteration” are referring to. Increasing speed of execution will — through reliable feedback — improve our aim, which will in turn increase our future velocity.

This second order learning has a compounding effect: any individual release might result in insights that will influence all other releases that will come after it. If we want to optimise for the long-term, then we should be willing to take (a little or a lot) more risk with each individual feature release—for example by building an MVP rather than a full-fledged product—if we believe it will increases our rate of learning.

Speed is not velocity, but speed is still critically important for velocity. The key phrase missing from both “failing fast” and “rapid iteration” is “… so that we can learn faster”.

We want to be running with our eyes wide open.

(This post was originally published on my blog.)

Replacing a meeting with an email is problematic for several reasons.

1. Emails allow for communication with a bounded and clearly defined audience only. It is the responsibility of the sender to determine who can access the information in it. There is no way for people to “subscribe” to emails related to a certain topic (although they could sign up to a distribution list; it is still up to the sender to decide whether or not to include that list in the recipients). This setup is not great for the sender (who has to make a decision before sending), the recipients (who have no control over the decision to receive this information), and everybody else (who have no control or visibility over the decision as well as the information; e.g. they are completely unaware the information exists).
2. There is no (easy) way to add emails to a shared archive which can be searched and accessed later by anyone outside the recipient list (distribution lists are again the exception to the rule here). This sort of fungible communication can be useful at times — for example when the content of the meeting is sensitive in nature — but it is also a (often unnecessary) restriction which limits impact and opportunities for feedback and improvement.
3. Email doesn’t support commenting, copy editing, updating, collaborative editing, or versioning very well. While “reply all” does allow for discussion, long email threads with inline comments are difficult to follow (Outlook makes this doubly so, because it does not natively support interleaved posting, which forces people to resort to using font colours — ”responses below in red” — like it’s 1999) and the recipients cannot (easily) opt-out of the conversation.

If the purpose of the original meeting was only to send information to a limited audience, then these limitations of email might not be a problem. But if any sort of interaction was expected — or if the audience has a fuzzy boundary (e.g. in the future someone else might be interested in the content discussed) — then I would argue a wiki page would be a better fit.

When speaking at conferences, I get this question a lot. And my answer is always the same: yes and no. Let me explain.

There is a difference between “speed” and “velocity”. Speed is how fast something is going, while velocity is how fast something is going in a certain direction. For example, if you’re running on a track, your speed is how fast you are running, while your velocity is how fast you are running in a certain direction. Imagine running around a track in a circle. You could be running really fast, but your velocity would be zero because you’re not going in any particular direction.

When talking about the speed of product development, we must take care not to confuse speed with velocity*. We want to release features quickly (i.e. speed), but we also want to release features that add value (i.e. velocity). Experimentation helps distinguish between the two.

The answer to the question above is “yes and no” because experimentation will reduce product development speed while at the same time increasing product development velocity. We will ship fewer features when it turns out they don’t add the value that we expected. But we will also learn which features add value and which do not, and this will help us better decide which features to build next.

Speed matters, but we also need to ensure that we are speeding in the right direction. Running into a wall faster is not a recipe for product success.

*) What in Agile/Scrum is often referred to as “velocity” (e.g. the amount of work a team can tackle during a single Sprint) is from a product development perspective actually a measure of raw speed. Marking more tickets as completed or shipping more features to production might contribute to more product development speed, but customers don’t care how many tickets are done or how many features are released. Customers only care about things that bring value to them; they only care about your product development velocity.

One month ago, I started a new job. I left a place I love to go exploring. After eight years of following a clear path, I was ready to be left to my own devices and seek out a new course.

What came after was to choose the next step. I did not want to leave one place without going to another destination. Deciding where to go next was almost as difficult as deciding to leave. But I knew that—more than anything specific—I wanted something different, something new, and some new things.

A new challenge. A new perspective. A new setting.

I understand and am grateful for the fact that I am speaking from a place of immense privilege. I had options to choose from. This is not the case for many others. Still, I hope this post may help others who have the luxurious challenge of having to choose their own adventure.

I chose Vistaprint, and here’s why.

I wanted to be part of something new

My decision to leave came from an immense sense of curiosity and a desire to learn. The impulse was motivated by the idea that testing my abilities in different ways will help me grow in different ways. The path we take today often shapes which roads are open to us in the future. I was looking for a chance to start from something small and build the skills needed to shape it into something bigger.

In my new role, I will be helping a company chart new territory, journeying into the data and analytics space. It is a mission that will require me to depend on everything I have learned about online experimentation and data science. And the challenge is to do it all without the benefit of the support and legacy of a long-established leader in those fields that I had grown accustomed to.

All I am equipped with now is what is in my backpack, the ideas in my head, and the lessons I have picked up along the way.

I wanted to experience new customs and traditions

Traveling well means exploring everything a place has to offer — its tastes, sounds, and unique celebrations. To not do what we always do; truly appreciating what makes a place special. Ordering the local lunch special stir fried beef with tree ants, instead of the usual sandwich.

The differences we discover, no matter how minute and deeply ingrained, are refreshing reminders that there are often many answers to the same question. It shows us that our way is not the only way. Challenging our assumptions about what it means to live and do right.

Starting a new job and joining a new group of people likewise means adopting new actions, behaviours, and updating our opinions. In the first few weeks in my new role, I have discovered alternative tools, techniques and ways of thinking at every step. In one of the very first meetings I attended, I experienced first-hand the six-page narrative technique that I’d previously only read about in the book The Everything Store. It was a small but striking reminder that there are still many more new things for me to see and do.

It is one thing to read about something in a book. It is something entirely different to experience that thing first-hand. Even if we end up never eating tree ants again, the experience itself will change our minds forever and for the better.

I wanted to feel at home

Like many of us, I realised the enormous benefits of working from home last year. I am grateful for the privilege to have access to a nice home office with a door that I can close. I also live in a house that is big enough for my kids to play, while I work. Again, I understand this is not true for everyone, but for me, it makes all the difference.

Working from home has helped me improve my work-life balance. I stopped spending three hours of my working day commuting. I started taking my kids to school in the morning. And I go for a run between meetings at least twice a week.

So even as I set out to explore new worlds — I wanted the freedom to work from my spare bedroom office. To adventure far and wide, from the safety of my own home.

This new opportunity has given me these things, and more. I am eager to see where this journey takes me next, and what I will learn along the way.

I am choosing my own adventure.

Written by Molly Stevens and Lukas Vermeer.

User research and experimentation are both powerful tools that allow us to narrow down and select product options. Each tool has a range of core capabilities that ensure our customers have great experiences and the business reaches their goals.

At Booking.com, our teams collaborate together within an insights organization, focusing on how to measure and understand the customer experience across all aspects of our business. When we (Molly and Lukas) started working together a year ago, people expected that we would not get along. This expectation stemmed from the mistaken idea that somehow our respective fields are in conflict. However, we both know that our disciplines can work well together, although it can be challenging to help our stakeholders and partners in the business understand how our areas are incredibly complimentary.

In this post we will examine in some detail why we consider research and experimentation to be the “peanut butter and jelly” of the customer insights world — two unique things that are even better together. We will give general explanations and specific Booking.com examples. We will also walk you through a recent case study where we’ll examine how focusing on each of our core strengths allows for highly effective evidence-based decision making. We hope this will give you the language for stakeholder discussions to talk about experimentation and research, and how they belong together.

Quant times qual

We have noticed some prevailing myths around experimentation and user research. We believe that these cause confusion and block the ability to collaborate and move forward on key topics. These myths center around a “numbers vs. people” (i.e. experimentation vs. user research) type mentality, which is an unhelpful and unnecessary division.

These myths include:

• Bigger numbers are always more believable, so experimentation is “better”
• User research always takes a long time AND experiments are always faster.
• There is no subjectivity in experimentation, so it’s the only way to be “data driven”.
• User research is expensive and experimentation is free.
• These fields, and the people in them, don’t work well together.
• Choosing just one to focus on is a good business, as well as personal, strategy.

Boiling down the inherent complexity to these black and white views highly limits our ability to recognise how our fields can (and should) work together. When we base our work practices and processes on collaboration, experimentation and research multiply the positive effects of their respective areas. This multiplicative effect on the kinds of positive experiences that can delight our customers and build business. This is especially important to support a company culture that rewards not just success, but learning and evolution.

Instead of thinking in terms of research VERSUS experimentation, we should be thinking how we can use research TIMES experimentation.

Without more knowledge about our areas it could be confusing to understand how this makes sense. In the following section we will define our areas more precisely, illuminating the elements that make us perfect partners.

Research at Booking.com

At Booking.com, research is a multi-method approach to answering questions in two main areas of inquiry around the what and why behind human behaviors. We turn our observations into insight and knowledge which can help us work with our cross-functional partners.

1. What do we see currently in the world and on our products?
2. What are the hypotheses about future behavior, where we can make changes and improvements?

Gathering the insights to answer these questions, gives us an opportunity to apply the knowledge to three major product development areas that help us understand major customer problems or opportunities.

1. Roadmaps. What kinds of products or services should Booking.com be building? What are the key characteristics of these that would meet customer needs?
2. Products/Features. When a product/feature is created, or added, how will people react? What are their needs and expectations? Who is the target audience and does this work for them?
3. Experiments. Are we taking into account the right potential solutions? Is the experiment targeting the top use cases in the right context?

Understanding the customer, the scope of the problem — and the space in which we are working, helps us to partner with cross functional partners and identify the best solutions. Defining the hypothesis, and understanding what kinds of data will help them to prove or disprove that hypothesis are essential for strong decision making.

Experimentation at Booking.com

At Booking.com, experiments are an important part of our product development cycle. Experiments allow us to measure, in a controlled manner, what impact our changes have on customer behaviour. Every product change we make has a clearly-defined purpose. Measuring the impact of our changes through experimentation allows us to validate that we have achieved the desired outcome. Experiments also allow us to abort changes quickly when they do not have the intended effect.

The term experiment itself seems to be often misunderstood, yet it is actually very clearly defined. Two key concepts are important to understand when talking about experimentation:

1. An experiment is a procedure carried out to support, refute, or validate a hypothesis.
2. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is changed.

This definition is probably a bit more broad than most people tend to call “experimentation”. For example, this concept of experimentation is not limited to only certain kinds of changes. It is also not limited to randomised experiments (often called A/B tests). Put differently, any changes made to a product, with or without the intent to have an impact, could be considered an experiment.

The more prior evidence and understanding is available, the more clearly the hypothesis and change — also called treatment — can usually be formulated. More often than not, much of this prior evidence and understanding will come from prior research. Without clearly defining the hypothesis in advance, there really can be no good experimentation. Data quality and statistical rigor are obviously important factors when running experiments, but it is important to acknowledge that these cannot fix a weak hypothesis or an inferior treatment.

We use experimentation to help us validate hypotheses with the goal of addressing well defined user problems. It’s all about learning as fast as possible in the most rigorous way possible.

— (Finn Hansen, Senior Product Manager, Booking.com)

At the product strategy level, there are always (too) many ideas to work on. Here, experiments help us learn which changes have impact. The quicker we manage to validate new ideas, the less time is wasted on things that don’t work, and the more time is left to work on things that make a difference to our customers. Experiments help us decide what we should ask, test and build next.

Evidence-based decision making

It is often said that Booking.com is a “data driven” company. However, data itself is not driving the decisions we make. Instead, we use data as evidence to support our decisions. Humans are still the driving force behind making choices about that data — and how to bring it to each of the decisions they make. Both research and experimentation provide evidence for this decision-making process.

In this context, research can be used to gather data for a broad range of topics, which may or may not have a clearly-defined hypothesis yet. We can use the data from research as evidence in a decision-making process — for very broad and very precise topics. Experimentation, on the other hand, is always centered around clearly-defined hypotheses. Every experiment is designed specifically to collect evidence for one of these hypotheses.

To illustrate how these two fields interact, we will discuss a recent case study.

Traveller needs in coronavirus times

Initially, about a year ago, when COVID-19 closed down the world and travel became both less possible, and in some ways less comfortable, we paused our research for a bit.

Finally as summer came, and things began to open up again, we set out to understand changes in traveller sentiment that had happened during the lockdowns. We knew that we could make a lot of changes to our product offerings, but which ones would be the most meaningful and useful to our population?

Conducting remote interviews and online surveys, we dug into the following topic.

Identify how Booking.com can help customers with the “new” needs:

• What are customers concerned about? What are they looking forward to?
• What do customers expect from Booking.com specifically
• What are the must > nice-to-have product features?

We had multiple findings from this work, however, one significant finding was the following:

Travelers are more likely to stay at apartments/homes when compared to the previous year; hotels might have higher cleaning standards, but homes offer isolation and more control of the environment.

We combined this with knowledge of our customer segments and previous experiments, and then worked with our product teams to develop potential experiments.

From insight to experiments

Whenever we have a key insight like that, there are always many ways in which we can try to address the customer’s needs. In this case, we wanted to help customers find a particular kind of property that we think they will enjoy. The question is how we should change the product to help them do just that.

We could for example visually highlight these properties on our search page. Or we could allow customers to explicitly filter for them. Both of these ideas could potentially help people find what they are looking for. At Booking.com, we use experimentation to identify which of these implementations actually do have the expected effect.

In the case of this example, our product teams explored both of the implementation ideas — highlighting and filtering — in a series of experiments. Both lines of experimentation were based on the same core research. Both were trying to address the same customer needs. Where the two differed was in our approach to implementing the solutions. This shows that the relationship between an insight and an experiment is often many-to-many instead of one-to-one.

Both approaches required different product implementation trade-offs to be made. For example, visually highlighting these properties could be added to the existing interface without the need to remove something else. Conversely, the way filtering was implemented meant that another filter was made less prominent in order to make room for the new one.

These differences in implementation turned out to be important. Even though both lines of experimentation were based on the same research and customer needs, the outcomes were very different. Visually highlighting properties proved to help people find what they wanted, while filtering did not have a positive effect. In fact, the filtering approach seemed to have negative side effects for certain customer groups that were not part of the original research. Ultimately, our product teams decided to ship the visual highlights and drop the filtering idea. Experimentation helped them gather the evidence required to decide which change helped our customers reach their goals best.

Our customers got added value from the successful experiment — and they showed this in their behavior. Research can use this knowledge about large scale behavior to investigate and understand what is motivating and exciting for our customers. As we move into this summer, we can try to understand how this is performing — from a qualitative and customer journey POV, which will lead to more hypotheses and experiments.

Conclusion

Experimentation and research are important tools in your toolbox for gathering data — and identifying different hypotheses for customer focused decisions. Our areas are not at odds — each of these disciplines is useful for different things and under different circumstances. We’re a multiplying effect — increasing the efficiency of each other in getting to the BEST solutions thoughtfully and quickly. Partnering together is even more important with the amount of data that our company gathers to understand the customer experience. Turning this data into supporting or disproving evidence supports efforts to de-risk our decisions at each step.

We hope this article has helped you understand as well as express how experimentation and research can partner up to provide deeper and more meaningful insights and better customer experiences. We look forward to hearing your stories about how experimentation and research evolve in your businesses and contexts

Acknowledgements

Thanks to the teams at ADAPT and UXRConf Anywhere 2021 who helped us craft the talks on which this article was based. Thanks to Kristofer Barber for helpfully suggesting improvements and additions to an earlier version of this article. Lastly, we would also like to thank everyone who was involved in the research and experiments outlined in this article.

Multiplication instead of division was originally published in Booking Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

At the end of this month, I will be leaving the company where I have spent the past eight amazing years. I am not sure I can adequately explain my motivation (decisions of this nature rarely have a single cause) but I will try.

Not all decisions in life can be data driven. I am leaving because it simply felt like it was time for me to leave. I will use three metaphors to try to convey how I feel. I hope this may help others who are in a similar position.

I am leaving for the same reason that I love travel.

Booking.com’s mission is to make it easier for everyone to experience the world. Our jobs are part of our own experience of that world. We could spend our entire lives sitting in one place, but most of us don’t. Many of us crave travel. We want to see new places. We want to experience new things.

We leave our homes not because we don’t like it there; we leave because we want to see what else is out there. I am leaving because I want to see what is outside my current company, and I chose my new job because it is different.

I am leaving now for the same reason that I get out of bed on Sunday morning.

Most of us get up before Sunday noon (on most weeks, anyway). We could stay in bed longer and into the afternoon. We could lie in and enjoy the warmth just a little bit more. But if we stay too long, the day will be gone, and with it many opportunities to experience the world.

I am not leaving now because of something that is happening now; I am leaving now because I fear I will regret not leaving now later. FOMO made me do it.

I will miss the past years like I miss my university days.

I was a student once. I stayed up late. Meeting new people and constantly reinventing myself trying to figure out who I was. Making plenty of mistakes, but constantly learning. I collected memories which I will carry with me for the rest of my life. I think back to those times often and fondly. I also know that I would not go back to that life now even if I could.

In some sense, I have been missing my first few years at this company already during my last few years at the very same company. Things are different now. I am different now. This is as it should be. Time is a funny thing, and change is the only constant. My staying or leaving makes no difference to the relentless passage of time.

Reminiscing about something is not the same thing as longing for something. I will continue to miss parts of the past, and that is okay. My long tenure at this company is an indication of how much I loved, and still love it there. It is an amazing place to work and I will dearly miss many of the people there. But there are more places to see; and so little time. It is time for me to move on.

I am leaving a place I love.

Thanks to Kristofer Barber for helpfully suggesting improvements and additions to an earlier version of this article.

Kickstarting and keeping the A/B testing momentum

Written by Aleksander Fabijan (Mircosoft), Benjamin Arai (Microsoft), Pavel Dmitriev (Outreach.io, and Lukas Vermeer (Booking.com). Originally published on the Microsoft Experimentation Platform blog.

At the time when we published the Experimentation Evolution Model in 2017 [1], we never expected such an interest in it. Practitioners from hundreds of companies and several different industries reached out with thoughts, concerns and tips on how to kickstart and scale an A/B testing program from “Crawl” or “Walk” milestones, to the “Run” or “Fly” stages [1]. Without doubt, scaling A/B testing in an organization is a challenge with which all of us can empathize. The reason for this is that there is no easy path to achieve A/B testing at scale (i.e., “Fly” stage). The Experimentation Evolution Model has the right ingredients, yet what you cook with them and in what way determines how successful the evolution will be. To experience the full benefit of A/B testing, organizations need to evolve both technically and culturally, and this needs to happen in an iterative way. One of the best ways to describe this iterative path to “Fly” stage is by using a Flywheel.

What is a Flywheel?

The concept of a flywheel was presented in the book “Good to Great”, illustrating that good-to-great transformations never happen at once but, you guessed it, in turns [2]. Every flywheel consists of a few carefully derived steps that logically follow each other in a loop. The goal is to kickstart the flywheel to help it start moving. The intention is to reach enough momentum such that it continues to spin. Note that value can be derived from various outcomes; for example, confirmation that a change will or won’t improve the customer experience is not the only valuable learning to be derived from A/B testing on its own. In this article, we present the A/B Testing Flywheel — a tool for implementing and growing a successful A/B testing program. Our flywheel focuses on navigating the value-investment cycle which supports and motivates the continued funding required to scale up an A/B testing program.

Introducing the A/B testing Flywheel

We developed the A/B Testing Flywheel based on our experience and learnings from introducing and embedding A/B testing in over two dozen product teams at Microsoft (where Aleksander & Benjamin work), Outreach (Pavel), and Booking.com (Lukas).

We refined it with insights extracted through surveys, tutorials, and other types of collaboration with companies that tried, failed, and succeeded in scaling their A/B testing culture. As you see from the figure, it consists of five steps that require continuous and iterative investment.

We introduce the steps next, starting with the goal-step of being able to support more decisions through A/B testing.

1. Running more A/B tests to support more decisions. At the top of the flywheel is the goal — using A/B tests to support decision making. With every turn of the flywheel, we aim to run more A/B tests to support more decisions.
2. Measure value to decision making. While not every A/B test has substantial impact on decisions, some A/B tests do, and the impact and value added by A/B tests need to be measured and captured. The more A/B tests we run with each turn of the flywheel, the more aggregate value and impact the A/B testing program will provide.
3. Increasing interest in A/B testing. When A/B testing is ran by one team and those results are shared broadly, this can lead to interest in A/B testing by new teams and new A/B testing scenarios. To support this spread of interest, the value of A/B tests needs to be broadly communicated. Additionally, educational and support efforts are needed to support new teams interested in using A/B testing to ensure their initial experience with A/B testing is a success.
4. Investing in A/B testing infrastructure and data quality. The more interest and willingness there is to try A/B testing within an organization, the more resources can be justifiably allocated to make an A/B testing program successful. These resources should be directed towards two key areas: improving A/B testing capabilities for managing and analyzing A/B tests, and increasing data quality. The goal of these investments is to make A/B tests are more trustworthy and easier to execute.
5. Lowering human cost of A/B testing. Improvements in A/B testing infrastructure and data quality create conditions for streamlining the A/B testing process and lowering its human cost i.e., the amount of investment required to start doing A/B testing in a new area. Lowering the cost of A/B testing is a critical step in the flywheel that is often missed. If running A/B tests remains costly, A/B testing will remain limited to highly interested early adopters with a lot of resources. This step increases the return on investment (ROI) of A/B testing by reducing the “I” and making it worthwhile for more teams who may be less convinced about the “R” or are more resource constrained, to try running A/B tests.

Each of the steps above merits a thorough explanation that is beyond the scope of this article. Instead, we provide a few tips on how to make that first turn, and what keeps the Flywheel spinning once it’s on a roll (pun intended).

The first turn

An initial investment will be needed in each of the five steps of the A/B Testing Flywheel.

1. Running the first A/B test. The analysis of our data shows that the growth of A/B testing program often starts with a single team trying out A/B testing in one specific scenario, such as the signup flow on a website, or ranking optimization in a shopping app. In this initial stage, A/B tests need to be chosen wisely. Why? Well, initial success may quickly propel the A/B testing program forward, while failure may stop the program in its tracks before it had a chance to get going. Note that it is important to differentiate between the success of an A/B test and the success of an idea being tested. For example, an A/B test that correctly prevents rolling out a feature that regressed important metrics can be as valuable as an A/B test in which improvements were detected. We found the following factors to be important when selecting initial A/B tests:

• High value potential. Focus on A/B testing features that have potential for high impact and relate to team goals. A/B tests on a small feature in an insignificant area of the website or product is unlikely to deliver significant value.
• Simple to execute. Given the lack of A/B testing infrastructure at this stage, initial A/B tests should be as simple to configure and execute as possible. Extra complexities may result in issues with design, execution, and analysis of the A/B test, delaying results and undermining trust in A/B testing. For example, A/B tests requiring coordination across website and mobile app or coordination of several backend and frontend components should be avoided. As a rule of thumb, we recommend a simple A/B test with 1 treatment and 1 control, in a single component/area of the product, utilizing 100% of users split 50/50 to maximize sensitivity.
• Easy to measure. The success metrics for the A/B test should be easily computable. Ideally, the product should already be instrumented, and data pipelines for metric computation should already exist. It is best to avoid creating new instrumentation and data pipelines at this stage and rather validate and reuse existing solutions. New infrastructure requires more time and resources and may introduce data quality issues.

2. Measuring value. Every successful A/B test that demonstrates clear value will push the flywheel forward, while every failure will slow or push it backward. Note that value can be derived from various outcomes; for example, confirmation that a change will or won’t improve the customer experience is not the only valuable learning to be derived from A/B testing. There is one specific type of value we have seen the most success with in terms of showcasing the power of A/B testing and generating more interest. We call this type of value a counterintuitive result. A counterintuitive result is a result of an A/B test that contradicts stakeholders’ expectations. To obtain a counterintuitive result, one needs to pick a test where the outcome is uncertain — there is a disagreement among the stakeholders about what they think the result will be. By choosing something over which there is disagreement, we are likely to choose something where insight has value. If stakeholders disagree about something, it’s probably something worth disagreeing about. For example, if there are several possible A/B testing ideas on the table, one can do a quick survey of decision makers asking them to predict the result and explain their prediction for each test. One should document the answers and then pick the test with the most disagreement. Once the results are in, someone is bound to be surprised!

3. Increasing Interest. If the A/B test was planned and executed correctly it will certainly result in learnings that will be of interest to many in the company. Review the results broadly (e.g., in a shiproom) and share them across the organization (e.g., in a newsletter). Make a game out of it before you share the outcome and measure how many predict the outcome of the A/B test correctly. Some teams like Azure Identity share broadly the variants that were tested and questions on which options received better outcomes [4]. Furthermore, at Microsoft we frequently organize Best Experiments Talks where A/B tests that resulted in valuable learnings are presented company-wide to inspire and spark interests in the scientific method.

4. Investing in Infrastructure. From an infrastructure perspective, we should focus on trustworthiness, especially in the early steps. One of the most fundamental components for running a trustworthy A/B test is treatment assignment — make sure that it is working correctly by running test A/As (A/B tests where both versions are identical) to validate your system end-to-end [5]. There is a small set of statistical tests necessary to interpret the results and ensure their trustworthiness. Before you run those, measure the observed split of users between variants and compare it to the configured one to detect a Sample Ratio Mismatch which commonly invalidates the results [6].

5. Lowering the cost. To lower the cost for the next A/B test, start automating the fundamental data quality checks such as the Sample Ratio Mismatch [7] and make it easy to start an A/A test to validate new set-ups and new metrics [5]. Since A/B testing is based on trustworthy data, it is instrumental that teams do not have to do common data checks manually. And because we intend to iteratively improve over time, we need to put checks in place to ensure that improvements do not accidentally introduce problems. Another aspect of lowering the cost are shared libraries. Teams starting A/B testing will hugely benefit from a shared library for assigning units into variants and a common script for computing metrics and performing statistical tests. What cannot be automated in this stage (e.g. interpreting analysis data) should be supported through office hours or some other easily accessible channel.

Making the A/B testing Flywheel Spin

Now that we know how to make the first turn, we need to invest in keeping the momentum.

1. Running more A/B tests. Eventually, A/B testing should expand scenario wise (e.g., from testing changes on the web to mobile to backend) and from one team to the next one. It is valuable for new teams to have a role model, e.g., another team which runs “similar A/B tests” and is successful. Another great way to facilitate the growth of A/B testing in a new area is to transition people with expertise in A/B testing to work with a new org. This is particularly powerful at the executive level, as it can greatly speed up A/B testing growth by making available resources that normally would only become available at the later stages of A/B testing growth.

2. Measuring more value. While the practice of collecting and communicating counterintuitive results should be continued, the focus of value creation needs to shift to connecting test results to business and customer value. Such connection is established by developing good Overall Evaluation Criteria (OEC) — a set of metrics and decision criteria used to judge the outcomes of A/B tests — which is predictive of long-term business and customer value and sensitive to the changes being tested. At this stage it is also important to focus on obtaining a more comprehensive understanding of test results by creating a richer set of metrics. In addition to a small set of OEC and Data Quality metrics that help determine whether the test was executed correctly and which variant is better, a comprehensive set of Local and Diagnostic metrics should be created [8], [9]. These metrics capture fine grained aspects of user behavior and often provide insight into the “why” behind the top level positive/negative result of an A/B test. Such insights help better understand user behavior, generate ideas for new features, and determine most promising focus areas within the product.

3. Increasing Interest even more. With more teams running A/B tests, a systematic program for communicating results and value needs to be created. For example, you can send monthly newsletters about interesting A/B tests that were completed, include A/B testing as part of new employee onboarding, share valuable learnings with executives, etc. The continuous supply and sharing of valuable A/B tests will reinforce the interest. Make sure to celebrate the learnings.

4. Investing in Infrastructure. A platform that has the capability to manage A/B testing operations will be instrumental for growth. We’ve extensively published about Microsoft’s Experimentation Platform (ExP) and features that are valuable for an A/B testing platform in our papers (see e.g. [10]–[12]) and hence will not go into the details here. Instead, what is important is that any steps that can be taken to reduce the need for manual intervention allows for decisions to be democratized and reduces the overall support cost.

5. Lowering the cost of A/B testing even more. Making the management of A/B tests and analysis consumption easy is critical for lowering the cost of A/B testing at scale. A/B tests should be easy to configure and iterate on (e.g. some might not even require code changes), and results intuitive to practitioners that might not be statisticians. Keep in mind that the tools need to be trustworthy as well as the decisions made based of them. Color coding and summarization results are two such examples that help with the interpretation [13]. Investing in ways to guide analysis through summarization of experiment result will drive down support costs and increase the trustworthiness of decisions making.

Steps in this flywheel allowed Microsoft, Outreach and Booking.com to scale A/B testing for many products in various stages of user growth, maturity in data pipelines and infrastructure, and the ability to adopt data-driven decisions. We hope that this iterative nature of the A/B testing Flywheel will be helpful to everyone else that strives to kickstart or grow their A/B testing program.

References

[1] A. Fabijan, P. Dmitriev, H. H. Olsson, and J. Bosch, “The Evolution of Continuous Experimentation in Software Product Development,” 2017, doi: 10.1109/ICSE.2017.76.

[2] J. Collins, “Good to great: Why some companies make the leap … and others don’t,” Meas. Bus. Excell., vol. 7, no. 3, pp. 4–10, 2003, doi: 10.1108/mbe.2003.26707cae.010.

[3] L. Peppler, “The Amazing Flywheel Effect,” 2019. .

[4] “Microsoft Identity sign-up flow A/B test.” https://twitter.com/azuread/status/1255957652976422914.

[5] “p-Values for Your p-Values: Validating Metric Trustworthiness by Simulated A/A Tests.” https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/p-values-for-your-p-values-validating-metric-trustworthiness-by-simulated-a-a-tests/.

[6] A. Fabijan, “Diagnosing Sample Ratio Mismatch in A/B Testing.” https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/.

[7] A. Fabijan et al., “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining — KDD ’19, 2019, pp. 2156–2164, doi: 10.1145/3292500.3330722.

[8] P. Dmitriev and X. Wu, “Measuring Metrics,” in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management — CIKM ’16, 2016, pp. 429–437, doi: 10.1145/2983323.2983356.

[9] “Understanding Mechanisms of Change in Online Experiments at Booking.com.” https://booking.ai/understanding-mechanisms-of-change-in-online-experiments-at-booking-com-629201ec74ee.

[10] S. Gupta, L. Ulanova, S. Bhardwaj, P. Dmitriev, P. Raff, and A. Fabijan, “The Anatomy of a Large-Scale Experimentation Platform,” in 2018 IEEE International Conference on Software Architecture (ICSA), Apr. 2018, no. May, pp. 1–109, doi: 10.1109/ICSA.2018.00009.

[11] A. Deng, U. Knoblich, and J. Lu, “Applying the Delta method in metric analytics: A practical guide with novel ideas,” no. March, Mar. 2018, [Online]. Available: http://arxiv.org/abs/1803.06336.

[12] W. Machmouchi, S. Gupta, R. Zhang, and A. Fabijan, “Patterns of Trustworthy Experimentation: Pre-Experiment Stage,” 2020. https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/patterns-of-trustworthy-experimentation-pre-experiment-stage/.

[13] A. Fabijan, P. Dmitriev, H. H. Olsson, and J. Bosch, “Effective Online Experiment Analysis at Large Scale,” 2018.

It takes a Flywheel to Fly was originally published in Booking Product on Medium, where people are continuing the conversation by highlighting and responding to this story.

I know. I know. I know! Using my inbox as my to-do-list is considered bad practice, and I should feel bad for even trying.

Several well known time management books explicitly advise against this approach. Based on this advice, I have tried different tools and techniques to create a to-do-list separate from my inbox. Yet the honest truth is that all of these attempts didn’t work nearly as well for me as my current Gmail setup.

I learnt to stop worrying and made the inbox-as-to-do-list approach work for me. I will try to explain why and how. Perhaps it can work for you too.

Realisation #1: Most of my tasks are emails already.

My job revolves around communicating with people. Emails inevitably form a large part of that communication. Most of the things I get “done” get done through email. Most of the things I need to get done next will also get done through email.

When I try to keep a to-do-list separate from my inbox, most of the tasks that end up on that to-do-list started life as an email, or will at least result in an email eventually. Moving all of those emails to a separate to-do-list is a chore; moving them back to an email doubly so. Using my inbox as my to-do-list saves me a lot of work simply moving tasks back and forth.

Obviously, not all tasks start with an email. However, effective to-do-list management becomes a lot easier when all tasks are collected in one place (as opposed to being distributed across notebooks, sticky notes on my desk, apps on my phone, and my email). So, since I have decided to use my inbox as my to-do-list, I need to ensure that somehow all my tasks arrive as emails there, even those tasks that were not sent to me by other people.

The solution to that problem is rather straightforward.

Trick #1: I email tasks to myself.

Emailing myself about the odd task that does not already involve an email in my inbox feels like less work than moving everything else back and forth between two lists.

The great thing about this approach is that it is extremely flexible. Sometimes I email myself just a subject. Sometimes I write whole paragraphs of text in the body to remind myself of the details of the task. Sometimes I add links to other things. Sometimes I reply to myself and post updates to keep track of my progress. Sometimes I carbon copy people who depend on the completion of the task, letting them know that I have documented this task and will get back to them later.

Inbox-as-to-do-list supports rich notes and collaborative tasks out of the box. Everyone I know already has an account and knows how to use this open platform.

Realisation #2: Time management is attention management.

Visual clutter clutters my mind. Anything I can see can distract me. The less there is to see on the screen, the less I will be distracted. To help me focus on the most important tasks, I try to hide everything else.

Trick #2: Simplify Gmail.

Simplify is an awesome Chrome extension that simplifies the Gmail interface to the bare minimum. Most of the interface elements are useless anyway, because I use the keyboard to navigate (more on that later).

(I used to maintain a personal fork of Simplify that would hide even more of the interface — such as the giant“plus” in the bottom right of the screen — but then I realised I could just use my ad-blocker to block those specific components. Much easier, and just as effective.)

Trick #3: Separate Gmail from the browser.

I use Gmail through the web interface. Often I have several other tabs open at the same time. When I’m browsing, the Gmail tab distracts me. When I’m emailing, the other tabs distract me. Browsing and email are separate tasks that should live in separate apps.

Chrome can create a shortcut to Gmail that opens as a new window. That way, Gmail behaves like a separate app, rather than just another tab.

Trick #4: Snooze.

Tasks I do not need to focus on right now need to be out of sight and out of mind. If they are not, these currently unimportant tasks continue to take up valuable brain space, even when I am not actively working on them.

In my experience, there are two main ways in which a postponed task can impinge on my attention:

1. If I see the task, I will think about the task. Simply looking at a list of tasks is a great way to destroy my concentration. So many things to do! So little time!
2. If I postpone a task, I need to know for sure that I will get back to it later. This one is a bit more subtle, but nevertheless important. A postponed task tends to linger in my mind if I am not absolutely confident that I will be reminded about it later. Fear of forgetting is the mind-killer.

I need a way to completely hide a task, but I also need to know that it will be back. Luckily, Gmail has a “snooze” feature:

Postpone emails and temporarily remove them from your inbox until you need them. Your email will come back to the top of your inbox when you want it to, whether that’s tomorrow, next week, or this evening.

I snooze frequently. I snooze ruthlessly. I snooze without regret.

Snoozing helps me avoid both of the attention traps above: the email disappears from view, and yet I can be absolutely confident that I will be reminded of it in the future, because it will come back directly to my inbox.

Deliberately deciding to postpone a task by actively snoozing it feels much better than pretending to ignore it for a while. Managing my own attention makes it more likely I will actually get to the task eventually. It’s not “goodbye”, it’s “see you later”.

Can it wait? Great!

Trick #5: Disable notifications.

Email is an asynchronous communication channel. It can usually wait a few hours. I will be checking my to-do-list several times a day in between tasks anyway. Notifications are a distraction and a waste of time.

Trick #6: Close Gmail while working on a task.

The purpose of a to-do-list is to help me decide what task I should be working on next. I do not need my to-do-list if I already know what I should be working on, or while I am working on a task. To avoid being distracted by other tasks on the list, or by new emails arriving in my inbox, I close Gmail while working on a task.

Realisation #3: Time spent managing the tasks is not spent doing the tasks.

The best to-do-list is obviously an empty list with no tasks on it remaining to be done. Empty is what we should be striving for.

I get more things done when I spend more time doing the things to be done. I want to spend as little time as possible navigating, organising, and managing my tasks.

Trick #7: Use the keyboard for navigation.

Using a mouse requires squinting at the screen, searching for the pointer, and directing it with precision. After clicking on the compose or reply button, writing an email then requires switching context and placing both hands on the keyboard for textual input. Over the course of a day, quite a bit of time is spent just navigating around and switching context.

Conversely, using the keyboard for navigation relies more on muscle memory and hammering out well worn patterns (open email, delete email, archive email, snooze email, reply-all, send email, etc). In addition, there is no switching cost between the navigation and the writing of an email.

I avoid using the mouse as much as possible. I use keyboard shortcuts instead. After a few days of practice (and hitting ? for the cheatsheet a lot), I never looked back. I now spend far less time navigating, and thus more time doing.

Trick #8: Don’t use filters or folders.

I used to have filters and folders. Lots and lots of filters and folders. They were only hiding the real problem: I was getting too many emails. More emails than I could ever hope to read. Worse still, I wasn’t even sure why I was even trying just to read them.

If an email requires an action from me, it should go to my inbox so I can prioritise it. If an email does not require an action from me, I should not be getting it.

Do or do not, there is no FYI.

(I realise this trick is probably the most controversial of all the tricks listed in this post. If you very strongly disagree — and you want to explain to my why I am wrong — feel free to send me an email. You can be assured it will go straight to my inbox.)

Trick #9: Use labels as visual cues.

Some tasks are more important than others. I use coloured labels to draw attention to more urgent and important tasks in my inbox.

My labelling scheme is intentionally simplistic. I use labels as temporary indicators, not for permanent organisation or structure. Organisation and structure are only distractions from the ultimate goal of ending up with an empty list anyway. An empty list does not need either.

Realisation #4: I cannot do everything today.

Every morning there are more tasks in my inbox than I can reasonably hope to complete in one day. After denial, anger, bargaining, and depression comes acceptance of the simple fact that effective time management involves deliberately deciding what not to spend time on.

Trick #10: Start the day snoozing.

Every morning I start my day by snoozing everything that I don’t expect to get done that day. This usually takes a few minutes scanning through my inbox. If something new and important comes up later, I immediately reevaluate if another task should now be pushed out to tomorrow. If so, I snooze without delay.

The reality is that I cannot do everything, so I must eventually always choose. Choosing as early as possible ensures that I will have as much time available as possible for the most important tasks. Snoozing as early as possible ensures that less important tasks take up as little of my attention as possible during the day.

When in doubt, I snooze rather than wait or deliberate. Snoozing is a two-way door decision: it is easily reversed. If I happen to finish all my tasks earlier than expected, I can always check my snoozed folder and un-snooze a task.

Trick #11: End the day snoozing.

At the end of every working day, I carefully review any tasks left over and thoughtfully snooze them one by one. It is the KonMari Method for Inbox Zero.

This should not be necessary, because I should have either snoozed a task earlier or completed it that day. Any tasks left over are a testament to my personal failure to ruthlessly prioritize or effectively execute.

Ending the day snoozing these tasks one by one serves as a reminder of my own limitations as well as a learning opportunity for the future. Tomorrow I will to-do better.

Ending the day snoozing also provides closure for the working day. After this daily ritual, there is (literally) nothing left to-do. Tomorrow is another day.

I hope these realisations and tricks can be of some use to you. If you have any questions or suggestions, feel free to email me a task.

Written by: Tolga Oztan, Zoe van Havre, Chad A Davis and Lukas Vermeer.

At Booking.com, one of the key ingredients to customer-centric product development isn’t just bright minds having great ideas, but collecting the evidence to support these ideas. We test each idea addressing a customer pain point via an A/B test using our in-house experiment platform. This platform is able to test thousands of changes simultaneously, with real customers, collecting data on the outcome within minutes of being implemented. However, as Booking.com grows to more countries, more languages, and our products grow in scope from hotels to other types of accommodations, cruises, car rentals, and more, our product development procedures must also become more flexible and more sensitive to the interactions of these many goals.

An A/B test (also known as a “randomised controlled trial”) helps us assess the causal effect of the implementation of an idea (from now on referred to as treatment) on some desired outcome. Even though we can never observe the outcome for a given person under both exposed and not exposed conditions — a problem known as the Fundamental Problem of Causal Inference — we can still get an unbiased estimate of the effect of the treatment on the intended population, referred to as the Average Treatment Effect (ATE). If the ATE was all we cared about, we would be done. However, at Booking.com, we care about two things:

1. The ATE, the total effect of our treatment on the outcome variable
2. The mechanism of change, how the treatment affected our visitors’ behaviour and, as a result, how the outcome variable changed.

While A/B tests are a powerful tool for estimating the overall effects of product changes on business metrics, they often fall short in explaining the mechanism of change. This becomes problematic when decision-making depends on being able to distinguish between the direct effect of a change or treatment on some outcome variable and its indirect effect via a so-called mediator variable.

In this post, we use a working example to explain the need for mediation analyses in online experimentation. We use simulated data to show how these methods help identify and estimate direct causal effects. We show that failing to take into account all confounders can lead to biased estimates, and we include sensitivity analyses to help gauge the robustness of estimates to missing causal factors.

Reducing ‘cancellations per visitor’

Cancellations due to unclarity around accommodation policies, facilities or prices lead to a bad customer experience, which we want to avoid as a general principle. In addition, cancellations make availability calculations more difficult for our accommodation partners and lead to a bad experience on their side as well. A confused, dissatisfied customer or partner is more likely to call our customer service, which also increases our customer service agents’ load.

We have teams working on solutions to address the pain points of customers and partners regarding cancellations. In one experiment, one such team might change the design of a page to bring more clarity to potential guests before they make a reservation, adding a text box containing an explanation of the property policies. Their goal is to make guests more aware of the prices around their trips, so that they have a lower chance of cancelling later, resulting in a reduction of the metric ‘cancellations per visitor’. We can represent such a scenario visually using a causal graph:

If the only goal was to reduce cancellations, the experimenter could go ahead and use the ATE to see if this was achieved, testing the difference between the average cancellations per visitor between the control and treatment group. However, this would never tell us how this result was achieved. The learning comes with understanding the mechanism of change, and teams need this understanding to explore other ideas or abandon those that don’t work.

Modelling the flow of effects

We encourage teams to monitor complementary metrics that can provide additional support for their hypothesised mechanism. For the example above, as the new information is displayed in a text box, a supporting metric might be if visitors hovered on the text box or not. We can extend the causal graph to include these supplementary metrics:

Here the team is explicitly checking one specific mechanism which is the reduction of cancellations complemented by a hover on the text box. If hovers increase and cancellations decrease, we understand the mechanism better. However, if cancellations change without the number of hovers changing, then either an unforeseen mechanism is at work or we’re dealing with a false positive. Either way, this additional insight can help the team better understand how their treatment is affecting visitor behaviour.

Direct & Indirect Treatment Effects

This approach gets more complicated when the metric that will help explain the mechanism is directly related to the outcome variable, while treatment is also expected to directly affect the outcome variable. For instance, if we observed a decrease in cancellations per visitor, but the number of bookings was also reduced. Did the reduction in cancellations originate from the new feature saving customers from making bookings, most of which would have been cancelled anyway, or did it inadvertently scare off previously satisfied customers from making bookings at all? We can express this mediation scenario graphically:

In this case, Bookings is a direct parent of the child metric Cancellations, meaning any change in bookings will automatically cause a change in cancellations. If you have more bookings, you have more opportunities for one of the bookings to get cancelled. If you have fewer bookings, you have fewer opportunities for a cancellation to happen.

The Average Treatment Effect in this example can be broken down into two effects:

The Indirect Effect is the effect of treatment on cancellations via bookings and the Direct Effect is the effect of treatment on cancellations directly. A standard A/B test will only tell us the sum of these two effects (the ATE) which is a very crucial quantity. If cancellations increase beyond an amount that we can tolerate, they may not care if the increase is due to additional bookings or the treatment’s direct effect. However, in most cases, to be able to understand the mechanism and to be able to make a decision, we need to disentangle these two effects.

The trouble with confounders

Another lurking problem is that of known and unknown confounder variables. In the example above, imagine some visitors are business travellers. We can extend the causal graph with one confounder (‘is visitor travelling for business’):

Confounders can add a layer of non-identifiability to the problem of interpreting the causal effect. While methods do exist which enable us to adjust for post-treatment covariates, the moment we do this, we lose the unbiasedness of the causal effect, unless we can control for all potential confounders. The reason for this is that the confounders and treatment are not independent conditional on the mediator variable, bookings.

By design, we randomise the visitors so that treatment and whether a visitor is a business traveller or not are independent. However, they are not conditionally independent. Imagine a scenario in which business visitors book more and also visitors in the treatment group book more. If we know how many bookings a visitor has, then knowing if they are in treatment or not would give us information about their likelihood of being a business traveller or not, and vice versa.

A method for dealing with confounders

The method we employ follows Imai et al.. Given a set of pre-treatment confounders X, we assume the following, known as the sequential ignorability assumption

The first equation says that given a set of pre-treatment covariates X, treatment assignment T is independent of the potential outcomes and potential mediator values. The second equation says that given the same set of pre-treatment covariates X and the actual treatment value T the mediator potential outcome is independent of the potential outcome.

The first equation is guaranteed to hold by design, as the context we work in is randomised online experiments. The second equation says the mediator variable is ignorable conditioned on the pre-treatment confounders X. Looking at it from the perspective of Directed Acyclic Graphs notation, we assume that all the back-door paths we may be opening by conditioning on the mediator variable will be blocked by the set of pre-treatment variables X. If this assumption holds, then the direct and indirect effects of treatment on the outcome variable are identified and we can estimate them non-parametrically.

For the estimation of causal effects, we use two-stage modeling as proposed in Imai et al. The form of these models are not important as long as sequential ignorability holds; so we use generalized linear models that are suitable for the outcome variables.

Testing the method with simulated data

We simulated 100,000 data points with one pre-treatment confounder, whether a visitor is a business traveller or not, where half the visitors are randomly assigned to treatment and half to control groups and within each group, the probability of a visitor being a business traveller is 0.4. On average a business traveller makes one booking while a non-business traveller on average makes also one booking, draws coming from Poisson distributions. The treatment adds two more bookings for business travellers but doesn’t affect bookings of non-business travellers. In addition, treatment has no direct effect on cancellations; therefore, cancellations per booking stays the same for each group; 14% for business travellers and 7% for non-business travellers, drawn from binomial distributions with said probabilities.

We can see that the treatment has no direct effect on cancellations, as the cancellation rates per booking stay the same for the subpopulations. All the treatment does is make business travellers book more, from 1 booking per visitor on average to 3 on average. In this scenario, adding the number of bookings as a covariate in a regression model yields a positive regression coefficient for the effect of treatment on cancellations even though there is no direct effect of treatment on cancellations. Since there is a pre-treatment confounder and treatment and bookings are not conditionally independent, this approach gives a biased result. However, using the two-stage method and including the pre-treatment confounder in the models, we get direct effects close to zero (as expected).

We can also look at the point estimates for no adjustment, adjustment in a linear regression, and two-stage model results. (Here, we assume the experiment was run for 30 days, in order to bring the point estimates down to familiar per day units.)

Next, we do a sensitivity analysis to see how robust the point estimates are to missing covariates. Data can never tell us whether we have successfully taken into account all pretreatment confounders. However, a sensitivity analysis can tell us how robust the estimates are. In our simulation data, we know there is only one pre-treatment confounder. So we expect the estimate to be quite robust when we include this variable in our models, and not robust when we omit it.

This shows how the direct effect of the treatment on cancellations changes with the correlation of the error terms of stage 1 and stage 2 models when we include the pre-treatment covariate. Observe that even when the correlation term ρ on the x axis is at the extremes, -0.9 and 0.9, the 95 percent intervals still include 0, meaning we can never conclusively conclude that the effect is non-zero. On the other hand, when we omit the pre-treatment covariate, business booker, the conclusions change drastically depending on the value of ρ; as shown here:

In summary, A/B testing allows for rapid customer-centric product development, but is likely to produce biased direct effect estimates, as the method cannot account for confounders, nor identify missing confounders. This leads to decisions being governed by hidden factors, which will at best inject randomness into the decision making process (costing time and effort), and at worst erode the quality of a product. For example, in the hypothetical scenario above, naive use of standard A/B testing might lead to a product better tailored for business travellers, and by consequence shrink our customer pool significantly.

If identifying the mechanism of change is important for decision making, then we suggest:

1. using multiple experiments to replicate findings and protect against false positives,
2. measuring important health metrics, as well as metrics known to be related (causally) to the outcome (and to include these in the model as demonstrated), and
3. performing a sensitivity analysis to check for missing confounders.

We encourage experimenters to always interpret results in context; making decisions which keep in mind the big picture, the mechanism of change, and the long term impact on customers.

Acknowledgements

This post was based on a paper presented at the 2018 Conference on Digital Experimentation CODE@MIT. These ideas were greatly influenced by our work on the in-house experimentation platform at Booking.com, as well as conversations with colleagues and other online experimentation practitioners. In particular, this work would not have been possible if not for the relentless challenging by raphael lopez kaufman. Causal graph figures were crafted by our amazing designer Alimsky Sergey. Thanks also to Lucas Bernardi and Steven Baguley for their inputs.

Understanding Mechanisms of Change in Online Experiments at Booking.com was originally published in Booking.com ML & DS Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.