A/B Testing
Even Tiny Improvements can generate tons of revenue!
Imagine you’ve built an app for students where they can track their studies and create to-do lists for different topics. Now, you’re thinking of changing the colour of the “Add” button, which students use to add topics to their study list. But here’s the question: Will changing the button colour actually make students notice it more and encourage them to use it to add their study topics?
But what if their was a way to test idea before rolling it out to everyone?
Now lets talk about A/B testing: Small Name, Big Impact.
In Simple Words: A/B testing helps us compare two versions and use actual user behaviour to understand which version performs better.
What exactly is A/B Testing?
A/B testing is a simple way of comparing two versions of something to see which one performs better with real users. You create Version A (the existing version) and Version B (the changed version), and show them to different groups of users. Then, you track a specific success metric, such as clicks, sign-ups, purchases, or engagement. Instead of guessing what users will prefer, A/B testing helps companies make decisions based on actual user behaviour and data.
Why We Use It:
We use A/B testing to understand what actually works for users instead of simply assuming what they will like. A small change in a product might look better to us, but that doesn’t necessarily mean users will respond to it.
A/B testing lets companies try out these changes with real users and compare the results. This helps them make safer, data-backed decisions, reduce the risk of making the wrong change, and improve the product over time.
What exactly is being compared?
In A/B testing, we compare two versions of the same thing to see which one performs better with users.
For example, imagine a student study app with an “Add” button. Version A has a blue button, while Version B has a green button. Different groups of students see different versions, and we then compare how many students click the button.
So, we’re not just comparing the design, we’re comparing how users actually behave with each version.
How does an A/B test work?
A/B testing may sound technical, but the process is actually quite simple:
- Find something to improve — The company identifies a problem, such as fewer clicks, sign-ups, or purchases than expected.
- Make a hypothesis — They come up with an idea about what change might improve the result.
- Create two versions — Version A is the current version, while Version B includes the proposed change.
- Show them to different users — Users are randomly divided into two groups, and each group sees one version.
- Choose what to measure — The company decides on a success metric, such as clicks, sign-ups, purchases, or engagement.
- Run the test and collect data — Both versions are shown to real users for a certain period while their behaviour is measured.
- Compare the results — The company looks at which version performed better based on the chosen metric.
- Take action — If one version clearly performs better, the company can roll it out to more users. If not, they can keep the existing version or test another idea.
What should we measure?
Get Bhavana’s stories in your inbox
Join Medium for free to get updates from this writer.
Changing the “Add” button colour is not enough. We need to measure whether the change actually makes students more likely to use the button.
For example, if our goal is to increase the number of study topics students add, we could measure:
- Add button click-through rate — How many students click the “Add” button?
- Topics added per user — How many study topics does each student add?
- Topic addition rate — What percentage of students actually add a topic after seeing the button?
- Repeat usage — Do students come back and use the “Add” button again?
Here, we can have a primary metric and guardrail metrics.
Primary metric → Number of study topics added
Guardrail metrics → App crashes, loading time, or students abandoning the page
The primary metric tells us whether we achieved our main goal, while guardrail metrics make sure the change isn’t causing other problems.
So, we’re not just asking “Do students like the green button?” We’re asking “Does the green button actually change their behaviour?”
Comman Mistakes in A/B Testing
A/B testing can give misleading results if it isn’t done carefully. Some common mistakes are:
- Ending the test too early: Early results may change as more users participate.
- Testing with too few users: Small sample sizes can make results unreliable.
- Changing too many things at once: You may not know which change caused the result.
- Choosing metrics after seeing results: Decide your success metric before starting the test.
- Ignoring negative side effects: One metric may improve while another gets worse.
- Assuming higher conversion means success: A higher conversion rate is useful only if it also supports the overall product goal.
Good A/B testing is not just about getting a higher number. It’s about making reliable decisions that actually improve the product.
A/B Testing Isn’t Only About Buttons, A/B testing is a decision-making method, not just a UI technique.
E-commerce Free delivery message vs no message
Netflix Different artwork for the same movie
Food delivery Different ways of displaying delivery time
ChatGPT Different wording or placement for a feature
Conclusion:
A/B testing is more than simply comparing two versions of a product. It helps product teams make better decisions using real user behaviour instead of assumptions.
The key is to start with a clear hypothesis, choose the right success metric, collect enough data, and look at the overall impact before making a decision. When done correctly, even small changes can help create a better user experience and improve product outcomes.
A/B testing is ultimately about one thing: making smarter product decisions with evidence, not guesswork.
