How to Interpret A/b Testing Results

🛑 Spoiler alert, plot A and plot B are actually from the same data set. 🛑 You just finished an A/B test using eye tracking and you ran a t test. You look at the p value and bar plot (Plot A)… nothing. No significant difference. You sigh, assume the design change did not matter, and start writing your report. Then, just to double check, you plot the histogram (Plot B) for both groups. And your jaw drops. One group is heavily skewed to the right. The other is skewed to the left. Their medians are far apart. The only thing that happened to be the same was the mean. In that moment you realize something very painful. You picked the wrong analysis and the wrong plot. The t test completely missed the effect because it only compares means, and your means were so close. But the actual distributions were dramatically different. The effect was very real and actually very large. This happens more often than people admit. It has happened in many UX studies and the results have been misleading. Wrong conclusions, wrong design decisions, and sometimes wasted weeks of research. Not because the data were bad, but because the wrong statistical test was used. This is why you must always check your distributions before you choose your test (Please do it!). The shape of your data matters. When your data are clean and roughly normal, parametric tests like the t test work beautifully. But UX data rarely behave that nicely. Eye tracking metrics, time on task, fixation counts, click delays, stress ratings, Likert responses… they are often skewed, heavy tailed, or full of outliers. When that happens, parametric tests lose accuracy. They can completely miss real effects. Non parametric tests do not have this problem because they compare the overall distribution and the median, not just the mean. Parametric tests, like the independent samples t test, assume normal data and equal variances. They only check whether the means differ. Non parametric tests, like the Mann Whitney U test, do not need normality. They pick up differences in medians and differences in overall distribution shape. They are often the correct choice in real UX research. Here is what happened in our example. We created two datasets with the exact same mean. One was right skewed. The other was left skewed. The medians were very different. 🔴 Results from the parametric t test t statistic: 0.00000000000000531 p value: 0.9999999999999958 Conclusion: no difference 🟢 Results from the non parametric Mann Whitney U test U statistic: 102112 p value: 0.000000538 Conclusion: very strong difference The t test completely failed. The non parametric test detected a very real and important effect. This is the entire point. If you only look at means, you are flying blind. If you check distributions first and choose the correct test, you avoid costly mistakes that have already ruined many UX studies.

Your A/B test launched 4 weeks ago. And the p-value is 0.0501. In this situation you usually find three types of CRO practitioner or data scientist: The bad: "It's fine, we won, the uplift looks great, ship the variant!" The ugly: "Let's just wait one more week and see what happens." The good: "Let's repeat it and meta-analyze." The closer your p-value sits to alpha (0.05), the more likely your recommendation lands on a false positive (you call the variant a winner when it isn't) or a false negative (you call it a loser when it isn't). Think about it: a p-value of 0.051 is almost identical to 0.049, yet your recommendation flips completely. In most cases a handful of extra conversions is all that separates them. That points to an uncomfortable truth: interpreting a p-value is gradual, not binary. So when a p-value lands close to alpha, don't rush to declare a winner or a loser. Most of the time you are sitting in an FP/FN gray zone. What can you do instead? Repeat it. If you run the experiment again under strictly the same conditions (same design, same metrics, same audience definition), you get a second p-value that you can combine into a meta p-value: a single combined probability that aggregates the results of multiple independent studies testing the same hypothesis. Sometimes you can't rerun the test, because you can't preserve independence between tests or because of security/legal constraints. But in digital products, especially in product and UX, tests can often be repeated safely. How do you combine two p-values? There are plenty of resources online (link in comments) with easy calculators using different methods: Stouffer, Pearson, inverse-variance weighting... What's the payoff? If two experiments give you 0.061 and 0.0501, Stouffer's method returns a meta p-value of around 0.0067. When both tests point the same way, the combined p-value comes out smaller than either of them. One important caveat: when you replicate and combine, hold yourself to a stricter bar. A common best practice (Ron Kohavi) is to demand p-value < 0.01 rather than 0.05, especially when the change is costly to ship or the prior that it works is low. The combined number looks reassuringly small, so raise the threshold (alpha) to match the higher standard you are now claiming. Next time you see a p-value hugging 0.05, don't cry and don't celebrate. Just say "let's repeat" and let the statistics do the work. The cost of shipping a decision that doesn't move the needle (false positive) is just as real as missing a genuine opportunity (false negative). #CRO #Growth #stats #abtesting #data #analytics

A company runs an A/B test. Version B wins—12% lift, statistically significant. Champagne. 🎉 Six months later, revenue is flat. What happened? They averaged over their customers. Rookie move. (I've done it too.) Version B: +20% for new users. But -8% for returning customers. New users outnumbered returners in the test, so B "won." Then the customer mix shifted. More returners. The "winning" variant was slowly bleeding its best users. This is Simpson's Paradox—when aggregate trends reverse at the subgroup level. It's not exotic. It's everywhere. Data-driven teams walk into this constantly when the first rule of being data-driven is "run the test and trust the average." The fix isn't more data. It's asking: for whom did each version win? Averages describe populations. They don't describe people. The most dangerous phrase in analytics isn't "we don't have data." It's "the data is clear." For those wrestling with weird A/B results—I see you! Ask who's in your sample before you pop the champagne.

How can an A/B test be “statistically significant” but not be totally trustworthy? I’ve been wrestling with this question for over a decade. Through extensive research, I've now come to understand that a big part of the answer lies in confidence intervals. Here’s the simplest way to explain it: Imagine you run an A/B test with very small numbers: 🚥 Version A: 82 visitors, 3 conversions 🚦 Version B: 75 visitors, 12 conversions The math shows the result is statistically significant. ⚡ The p-value is 0.0088, well below the common p < 0.05 threshold ⚡ Observed power is reported as 95.09%, well above the standard 80% rate The results look convincing. Statistical significance, long treated as the gold standard, has been achieved! But, here's the problem. Statistical significance can be "gamed" with low traffic tests because it only answers one narrow question: 🔦 If there were truly no difference between versions, how likely is it this result would happen by chance? That’s it. That's all statistical significance answers. It doesn't tell you whether the result is stable or repeatable. And, as you can imagine, with tiny samples, like 3 vs. 12 conversions, you get exaggerated effects. Every single conversion has an outsized influence. One or two people behaving differently can completely flip the outcome. ➡️ This is where confidence intervals come in. A confidence interval is the range of outcomes that could reasonably be true given the data. In small tests, that range is really wide. So the actual conversion effect might be smaller or larger than what you achieved in the test. You can't know with precision. So you don't have a dependable estimate of how big the improvement really is, or whether the result would hold if you ran the test again. It's important to realize, a confidence interval is not the same as a confidence level. Remember: a confidence interval is the range of values that could reasonably be true given the data. A label of “95% confidence” describes how that range was constructed, not how certain or correct the result is. Which means, a 95% confidence interval can still be very wide, creating substantial uncertainty around the estimate. When there's such uncertainty, the numbers may appear exaggerated. That's where Twymann’s Law comes in. It states, anything that looks interesting or unusual is usually wrong. In small samples, results are extreme because the noise does most of the work. So while a statistically significant difference can be measured in a small-sample study, you can't reliably measure how big that difference actually is. That's why 3 vs. 12 conversions often fail to replicate once more data is collected. 📣 Call to action for 2026: Run tests that are not only statistically significant, but also have a large enough sample size to produce narrow confidence intervals, so you can not only detect effects, but also estimate them precisely enough to make accurate, trustworthy test decisions.

🚨 Your A/B test results are not the real impact. A happy PM runs an A/B test → sees a +15% lift in revenue → scales the feature to all users → shares the big win in Slack 🎉 But… once the feature is fully rolled out, the KPI impact isn’t there. Why? Because test results often don’t reflect the true long-term effect. Here are a few reasons why this happens: 1️⃣ Confidence intervals matter → That “+15%” is actually a range. The lower bound might be close to zero. 2️⃣ Novelty effect → Users are excited at first, but the effect fades as they get used to the change. 3️⃣ Experiments aren’t additive → Three +15% lifts don’t stack to +45%. There’s a ceiling, and improvements often cannibalize each other. 4️⃣ Sample ≠ population → The test group might not represent your entire user base. For example, you have more high-intent users in the variant. 5️⃣ Time-to-KPI effects → We see that a lot, especially in conversion experiments. The experiment could improve the time to conversion, so when you close the experiment, it seems like you’re winning, but actually if you monitor the users a few days/weeks after the experiment ends, there are no differences in total conversions between the variant and the control. 6️⃣ Type I error → With P-value=0.05 (or worse, 0.1), there’s still a decent chance the “win” is a false positive. 👉 That’s why tracking post-launch impact is just as important as running the experiment itself. Methods like holdout groups, simple correlation tracking, or causal inference models (building synthetic control) help reveal the real sustained effect.

Senior Data Scientist Question Series - (Q3/25) Your A/B test reaches significance in just 3 days. 100,000 users. P-value = 0.01. The PM wants to launch immediately. Should you? Most people say: "Yes. We reached the required sample size. The math is valid." The senior answer: "Not necessarily. You may be measuring a temporary spike rather than a real effect." The misconception: Many data scientists treat sample size as the finish line. Senior data scientists know statistics are blind to time. A calculator doesn't know what a weekend is. It doesn't know traffic patterns change. It doesn't know a viral campaign just flooded your experiment with a completely different audience. If you stop the test during a temporary spike, you didn't validate a feature. You validated a moment. The Election Poll Analogy Imagine conducting a national election poll immediately after India wins the World Cup final. You collect 100,000 responses. The sample size is huge. The result is statistically significant. Would you trust it to represent the entire country? Of course not. Emotions are running high. The population isn't behaving normally. The sample is heavily biased by the moment. Data science works the same way. Junior data scientists optimize for sample size. Senior data scientists optimize for representative behavior. The Two-Lock Framework Before declaring an A/B test winner, unlock both doors: Lock 1: Volume Did we reach the required sample size? Lock 2: Time Did we observe at least one full business cycle (7-14 days)? Most teams stop after Lock 1. The best teams refuse to launch until both locks are open. Key Takeaway Statistical significance is not a race to the finish line. Before asking: "Did we collect enough data?" First ask: "Did we observe enough reality?" ♻️ Repost this if you believe A/B testing requires full business cycles, not just lucky p-values. #DataScience #Analytics #Experimentation #Statistics #ProductManagement #AI

This A/B test result with 99% confidence is 87% likely to be wrong: An AI redesigned landing page beat the human-designed version by 44%. A/B tests are the best scientific way to test causality, but TRUSTWORTHY A/B tests require a proper setup and analysis. I was asked today for my thoughts about a result published by Crazy Egg at https://lnkd.in/eWhQ7B2M. This experiment result has several red flags worth adding to your experimentation checklist (calculations use reasonable default industry assumptions): 🚩 1. No power calculations.  A huge red flag: the word “power” is not mentioned in the article and it appears that no power calculations were done. My power calculations suggest that this experiment had only 6.9% power for the right tail, dramatically below 80% recommended: https://lnkd.in/eXss97-N. In such low-power situations, the EXPECTED exaggeration (type M error) is 5x, the probability of a sign-error (you thought AI was better, but the human is better) is 9.6%, vastly higher than the 1% implied by “99% confidence.”   - See the recently published article on Why Power Calculations are Essential at https://lnkd.in/gVskVh_j . - Gelman has a great visualization of 6% power in his book. See https://lnkd.in/eJ_Zt3tT 🚩 2. Misinterpreting p-values.   As Pierre Paqueton mentioned in response to the LinkedIn post about this, the 99% confidence (p-value 0.01) is mis-interpreted:   https://lnkd.in/eVMEdcey . A more meaningful statistic to share is the False Positive Risk: the probability that a statistically significant result is a false positive. With such low power and for the median success rate in the industry, the estimated false positive risk is 87% (see https://lnkd.in/gaEbvCXM, FPR tab, change E1 to 93.1% (6.9% power) and read the number in E25 for one-tailed test of 0.05 that was used). - See https://lnkd.in/gM67hFTe and https://lnkd.in/gVjj4Wpv 🚩 3. Borderline Sample Ratio Mismatch (SRM) (explained in comment) 🚩 4. Twyman’s law (explained in comment) I’ll conclude with a shameless plug: these sanity checks are all taught in my flagship interactive online course on A/B Testing at https://bit.ly/ABClassRKLI. #ABTesting #PowerAnalysisIsEssential #ExperimentGuide Despina G. Lukas Vermeer 🃏 Jakub Linowski

📊 𝗕𝗿𝗲𝗮𝗸𝗶𝗻𝗴 𝗗𝗼𝘄𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝘁𝗼 𝗘𝗻𝗵𝗮𝗻𝗰𝗲 𝗗𝗲𝘁𝗲𝗰𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗶𝗻 𝗔/𝗕 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝘀 In experimentation, we often measure broad success metrics that can be too noisy or unclear, making it difficult to detect meaningful changes. The solution? Break them down. 𝗧𝗵𝗲 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲: High-level metrics like "User Activation Rate" are valuable, but they aggregate many underlying behaviors. When you run an A/B test on a specific feature (like an improved onboarding flow), the impact might only affect one small step in the user journey. If you only measure the overall metric, that signal can get lost in the noise. 𝗧𝗵𝗲 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻: 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹 𝗠𝗲𝘁𝗿𝗶𝗰 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 Implement a MECE (Mutually Exclusive, Collectively Exhaustive) framework that decomposes your output metric into granular input components. Instead of just tracking activation, measure each step of the journey: - % of users who view the onboarding screen - % who interact with the first core feature - % who complete their first session - % who return the next day 𝗪𝗵𝘆 𝗧𝗵𝗶𝘀 𝗪𝗼𝗿𝗸𝘀: By measuring closer to the intervention, you increase precision. If your new onboarding flow primarily improves the transition from viewing to interaction, you'll catch that lift at the granular level even if the overall activation rate barely moves. You're essentially reducing variance and increasing statistical power by focusing on where the action actually happens. 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁: Enhanced detectability means faster learning, more confident decisions, and better product outcomes. This approach transforms vague signals into clear, actionable insights. What metrics have you found most valuable to decompose in your experimentation work? (Note: Used NotebookLM to generate the infographic. A bit dramatic, but I like it)

If an A/B test is 'inconclusive', it does not necessarily mean that the change does not work. It rather just means that you have not been able to prove whether it works or not. It is entirely possible that the change does have an impact (positive or negative), but that it is just too subtle for you to detect with the volumes of traffic you have. Mostly though, subtle (if you could detect it) would still be meaningful in terms of revenue. If you discard everything which is inconclusive, how do you know you are not throwing away things which would be worth implementing? So what to do? Well, experimentation is really about degrees of risk management. If you cannot prove the positive benefit of a change, then the first thing is to accept that the risk surrounding that decision is greater. BUT, you can understand the parameters of that risk. The image is from the awesome sequential testing calculator in Analytics Toolkit, created by Georgi Georgiev. This is the analysis of an inconclusive test, which is nevertheless able to show, based on what was determined by the observation, that there is a 70% likelihood of the effect falling between around -8.5% and +5%. This particular case is vague, but at least you know the boundaries of the risk you're playing with. In some cases the picture is more heavily skewed in one direction. An A/B test is a way of making a decision, and the outcome of that test is always simply an expression of the degrees of confidence you can have in making that decision. How you make the decision is always still up to you. #cro #experimentation #ecommerce #digitalmarketing #ux #userexperience

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.