But that can be misleading.

A variant may have the highest CTR and still fail to provide enough statistical evidence for a confident decision. At the same time, a Bayesian analysis may suggest that the same variant is probably better than the control, even if the frequentist test does not call it statistically significant.

This project explores that tension using the Upworthy Research Archive, a public dataset of headline experiments from the media company Upworthy. The goal was to analyze one five-variant A/B/n experiment using two different statistical frameworks:

• a frequentist approach
• a Bayesian approach

The main question was:

Can we trust the variant with the highest click-through rate?

The answer turned out to be more interesting than a simple yes or no.

Project Motivation

A/B testing is one of the most common ways companies make data-driven decisions. It is used in digital products, marketing, media, e-commerce, and user engagement.

However, one common mistake is to rank variants by click-through rate and choose the highest one without asking whether the result is reliable.

This project was motivated by a practical question:

When an experiment appears to have a winner, can we actually trust that winner?

To answer this, I analyzed one experiment from the Upworthy dataset using a complete statistical workflow. I first used a frequentist framework to check sample-ratio mismatch, effect size, power, multiple comparisons, and confidence intervals. Then I repeated the analysis using a Bayesian framework to estimate the probability that each variant was better than the control.

The goal was not just to find a winner. The goal was to understand uncertainty.

Dataset

The Upworthy dataset contains A/B tests where different headlines and visual elements were tested to improve engagement.

For each variant, the dataset includes:

• impressions
• clicks
• headline text
• experiment ID
• eyecatcher ID

The main metric used in this analysis was click-through rate:

CTR = clicks / impressions

For this article, I focused on one experiment with five variants. Each variant had a different eyecatcher_id.

Because the dataset did not explicitly identify a true control group, I treated the variant with the lowest CTR as an exploratory baseline.

This is an important limitation. In a production A/B test, the control group should be defined before seeing the results. In this project, the baseline was selected only for analysis and learning purposes.

Step 1: Sample-Ratio Mismatch Check

Before testing which variant performed best, I first checked whether traffic was distributed roughly equally across the five variants.

This is called a Sample-Ratio Mismatch check.

Since there were five variants, the expected traffic share was approximately 20% per variant.

To complement the visual check, I used a chi-square goodness-of-fit test.

The chi-square test was not statistically significant. This means there was no strong evidence that traffic allocation deviated from the expected distribution.

This gave more confidence that the experiment traffic was reasonably balanced.

Step 2: CTR and Relative Lift

After checking traffic balance, I calculated CTR for each variant.

One variant had the highest observed CTR and a relative lift of approximately 45.6% compared with the exploratory baseline.

At first glance, that looks impressive.

But relative lift alone does not answer the most important question:

Is the difference reliable enough to act on?

This is where effect size and power analysis become important.

Step 3: Effect Size with Cohen’s h

Because CTR is a proportion, I used Cohen’s h as the effect size measure.

Cohen’s h is used to compare two proportions, such as the CTR of a control group and the CTR of a treatment group.

For the best-performing variant, the effect size was small:

Cohen’s h: 0.0346

This is an important result.

A 45.6% relative lift sounds large, but the absolute CTR values were very small. The control had 26 clicks, while the best-performing variant had 37 clicks.

So even though the relative lift looked strong, the amount of information in the data was still limited.

Step 4: Power Analysis

Next, I asked:

Did this experiment have enough sample size to reliably detect the observed effect?

Using the observed Cohen’s h value, the required sample size for 80% power was estimated.

The experiment had far fewer impressions than the required sample size. This means the experiment was likely underpowered.

At the observed effect size, the test would detect a true effect only about 30% of the time. That implies a high Type II error risk.

In simple terms:

The experiment may not have had enough data to confidently detect the improvement, even if the improvement was real.

Step 5: Multiple Comparisons and Dunnett’s Test

This experiment had one exploratory baseline and four treatment variants.

If I ran four separate tests at α = 0.05, the chance of at least one false positive would increase.

The family-wise error rate would be:

1 - (1 - 0.05)^4 ≈ 18.5%

That means there would be about an 18.5% chance of finding at least one false positive across the four comparisons.

To address this, I used Dunnett’s test.

Dunnett’s test is designed for comparing multiple treatments against one control while controlling the family-wise error rate.

The frequentist result was:

The best-performing variant was variant 4.

It had:

• the highest observed CTR
• the largest relative lift
• 37 clicks compared with 26 clicks for the control

However, after Dunnett’s correction, the adjusted p-value was 0.209.

Since this is greater than 0.05, the frequentist analysis failed to reject the null hypothesis.

The conclusion was:

Variant 4 looked better, but the frequentist test did not provide enough evidence to declare it statistically significant mainly because the test was under powered.

Step 6: Confidence Intervals

Confidence intervals were used to visualize uncertainty around the CTR estimates.

The control is in black and the treatments are in blue. This plot helped show why the highest CTR alone was not enough. The treatment estimates had uncertainty, and the observed differences could still be explained by random variation. There is a clear overlap between the confidence interval of all the tests.

Please see a detailed analysis including the code at my github page.

Part 4: Bayesian A/B/n Testing

After completing the frequentist analysis, I repeated the same experiment using a Bayesian framework.

The Bayesian approach asks a different question.

Instead of asking:

Is the result statistically significant?

It asks:

What is the probability that a treatment variant is better than the control?

This is often easier to interpret from a business perspective because it translates uncertainty into probability.

The frequentist approach is useful when we want a formal hypothesis test and strict error control.

The Bayesian approach is useful when we want a probability-based decision framework.

The Bayesian Model

In this A/B test, each impression can result in one of two outcomes:

• click
• no click

That makes the click outcome a binary event.

For each variant, I modeled the true CTR as a probability parameter:

θ = true click-through rate

Because CTR is a probability, θ must be between 0 and 1.

I used a Beta distribution as the prior because the Beta distribution is naturally defined between 0 and 1 and works well with binomial click data. The fantastic properties of the Beta distribution can be explored more at its Wikipedia page. The usefulness of the Beta distribution for A/B testing is as follows:

• Perfect Mathematical Fit to Probability Boundary: The Beta distribution naturally has a domain (support) of exactly ([0, 1]). It is physically impossible for a Beta random variable to fall outside these boundaries making it perfect for probability distribution.
• Conjugate Prior to the Binomial Likelihood: This is the most powerful algebraic reason. When running A/B test tracking conversions, the incoming data follows a Binomial distribution(Likelihood), because of the conjugate pairing rule, the outgoing data (posterior) will also following a beta distribution. Because posterior and prior distribution are conjugate pairs, there is no need to run heavy simulations to find the posteriror distribution. One calculate the exact answer instantly using simple addition.
• Incredible Shape Flexibility: The Beta distribution is controlled by two shape parameters, usually called α (alpha) and β (beta). By changing these two numbers, a Beta distribution can twist into almost any shape to represent the prior state of knowledge.

For this analysis, I used a uniform prior:

Beta(1, 1)

This prior treats all CTR values between 0 and 1 as initially possible before observing the data.

The Beta(1, 1) prior distribution used before observing the experiment data.

Updating the Prior with Data

Because the Beta distribution is conjugate to the Binomial likelihood, the posterior can be calculated directly.

For each variant:

α_new = α_prior + clicks
β_new = β_prior + impressions - clicks

This gives each variant its own posterior distribution for the true CTR.

The posterior distribution combines:

• the prior belief
• the observed clicks
• the observed non-clicks

The black line which is barely visible is the prior distribution of beta(1,1). The blue line is the posterior distribution for the control, while the red line is the posterior distribution for the best performing variant. Instead of only having one CTR number per variant, we now have a full probability distribution for each variant’s true CTR. The best-performing variant shifts to the right, but overlap still remains. The posterior distributions make the uncertainty visible.

Probability That Each Variant Beats Control

Using simulation, I sampled from each variant’s posterior distribution and estimated how often each treatment variant had a higher CTR than the control.

This gives the Bayesian probability of out performance:

P(θ_treatment > θ_control)

The results were:

The meaning of the columns in the results

The most important columns are P(Variant > Control) and the 95% Credible Interval. A high probability of beating the control suggests that a variant is promising, but a wide credible interval means there is still uncertainty. If the credible interval includes negative lift, the variant may still be worse than the control even if it has the highest probability of being better.

Lastly in the Bayesian analysis, the variant with the highest posterior probability of beating the control is not automatically declared the winner. A decision rule is needed to define how much evidence is required before choosing a variant. The decision rule used was:

1. The probability of the P(variant>Control) is greater than 90%
2. The observed lift has to be greater the mean of all the lifts.

Using the rule above it looks as if variant 4 can be a winner. The Bayesian model estimated that it had a 92.99% probability of beating the control.

That is much more intuitive than saying “p-value = 0.209.”

However, the result was still not fully conclusive.

The 95% credible interval for relative lift was:

[-11.40%, +138.05%]

This interval is wide and includes negative values.

That means the variant could still be worse than the control, even though it is more likely to be better.

Why the Frequentist and Bayesian Results Differ

The frequentist and Bayesian analyses do not truly contradict each other.

They answer different questions.

The frequentist analysis asks:

If there were no real difference, how surprising would this result be?

The Bayesian analysis asks:

Given the data and prior, how likely is it that this variant is better?

For variant 4:

The frequentist method was conservative because it controlled the risk of false positives across multiple comparisons.

The Bayesian method was more decision-oriented because it estimated the probability that the variant was better.

Final Business Decision

From a business perspective, I would not immediately declare variant 4 the final winner.I would treat it as the best candidate for a follow-up test.

Why?

Because the evidence is promising but still uncertain.

The frequentist analysis did not find statistical significance after correcting for multiple comparisons.

The Bayesian analysis estimated a 92.99% probability that variant 56437 was better than the control, but the credible interval was wide and still included possible downside.

Also, the absolute difference was small. Variant 4 generated 37 clicks, while the control generated 26 clicks. That is only about 11 additional clicks in this experiment. Depending on the revenue model, if these 11 additional clicks generate a lot more revenue, then variant 4 can be a winner.

So practical recommendation is:

Run a larger follow-up experiment comparing variant 4 directly against the control.

This would reduce uncertainty and provide stronger evidence before making a production decision.

What I Learned

This project helped me understand that A/B testing is not just about picking the highest CTR.

A good A/B testing workflow should ask:

• Was traffic assigned correctly?
• Is the observed lift practically meaningful?
• Clicks impact on revenue?
• Was the experiment powered enough?
• Did we account for multiple comparisons?
• How much uncertainty remains?
• What decision risk are we willing to accept?

The frequentist framework helped me control false positives and understand statistical significance.

The Bayesian framework helped me think in terms of probabilities, uncertainty, and business decision-making.

The biggest lesson was:

The best A/B testing decision is not always “launch the highest CTR.” Sometimes the best decision is “this variant is promising, but we need more data.”

Conclusion

The Upworthy dataset was a useful case study for comparing frequentist and Bayesian A/B testing.

In the selected experiment, variant 4 had the highest observed CTR and the largest relative lift. The Bayesian analysis estimated that it had a 92.99% probability of beating the control.

However, the frequentist analysis did not find the result statistically significant after Dunnett’s multiple-comparison correction. The experiment was also underpowered, and the Bayesian credible interval remained wide.

A/B testing is not just about finding winners. It is about making decisions under uncertainty.

Please see a detailed analysis including the code at my github page.