AB Testing

AB Test is used when mathematical averages are not sufficient and a comparison between two groups is desired.

This refers to the control and test groups A and B. Generally, innovations in mobile or web applications are used to test features that are desired to be tested. Or it is used to conclude whether the result of the developed machine learning algorithms is useful or not. Did the new model make a better, significant difference compared to the old model? We must look for the answer.

Hypotheses (365 Data Science)

We will establish two types of hypotheses regarding this. H0 and H1. H0 is the null hypothesis. That is, there is no difference between the two group averages. According to the result of this hypothesis, we aim to determine whether there is a difference between the two groups.

H0 => µ1 = µ2
H1 => µ1 != µ2

We will verify the hypotheses in the following order :

# AB Testing (Independent T Test)

# 1. Setup Hypotheses
# 2. Assumptions check
#    1. Assumptions of normality
#    2. Homogeneity of variance
# 3. Application of hypotheses
#   1. If these assumptions are met, Unpaired( Independent ) T Test(parametric test)
#   2. If these assumptions are not met, Mann-Whitney U test. (non-parametric test)
# 4. Interpret the results based on P-values.
#

Let’s try to understand this test by examining the “tips” dataset with Python. In this case, is there a statistically significant difference between the account averages of smokers and non-smokers ?

import seaborn as sns
df = sns.load_dataset("tips")

To examine the data set, let’s see the averages of smoker and non-smoker users.

Mathematically there seems to be a difference. Is it really like that statistically ?

1. Setup Hypotheses :

H0 => µ1 = µ2
H1 => µ1 != µ2

2. Assumptions check :

1. Assumptions of normality
2. Homogeneity of variance

H0 : Normal distribution assumption is met
H1 : Normal distribution assumption is not met

We use Shapiro method in python. Shapiro test helps us test whether the distribution of a variable is normal. The H0 hypothesis here is the assumption of normal distribution. It says that there is no significant difference between the standard normal distribution and the sample distribution.

Get Murat Arın’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If the p-value is less than 0.05, HO is rejected. If not, it cannot be rejected. So, according to this result, the normal assumption distribution is not met for Smoker = Yes group.

If the p-value is less than 0.05, HO is rejected. If not, it cannot be rejected. So, according to this result, the normal assumption distribution is not met for Smoker = No group.

# Smoker = Yes
from scipy.stats import shapiro, levene, mannwhitneyu
test_stat, pvalue = shapiro(df.loc[df["smoker"] == "Yes", "total_bill"])
print('Test Stat = %.4f, p-value = %.4f' % (test_stat, pvalue))

#Result
#
# Test Stat = 0.9367, p-value = 0.0002
#

#Smoker = No
from scipy.stats import shapiro, levene, mannwhitneyu
test_stat, pvalue = shapiro(df.loc[df["smoker"] == "No", "total_bill"])
print('Test Stat = %.4f, p-value = %.4f' % (test_stat, pvalue))

#Result
#
# Test Stat = 0.9045, p-value = 0.0000
#

Since the H0 hypothesis is not met in both groups(‘Yes’ , ’No’ ), we will have to use a non-parametric thesis. Therefore, we will use the assumption hypothesis. H0: Assumptions are homogeneous. H1: Assumptions are not homogeneous. Levene test is used for the assumption of variance homogeneity.

test_stat, pvalue = levene(df.loc[df["smoker"] == "Yes", "total_bill"],
                           df.loc[df["smoker"] == "No", "total_bill"])
print('Test Stat = %.4f, p-value = %.4f' % (test_stat, pvalue))

# Result
#
# Test Stat = 4.0537, p-value = 0.0452
#

As a result p-value < 0.05. The variances are not homogeneous. In this case, both assumptions are not met. We will implement this article(3b)

3. Application of hypotheses :

a — If these assumptions are met, Unpaired( Independent ) T Test(parametric test)

b — If these assumptions are not met, Mann-Whitney U test. (non-parametric test)

Mann-Whitney U test (mannwhitneyu) It is a non parametric mean comparison test.

test_stat, pvalue = mannwhitneyu(df.loc[df["smoker"] == "Yes", "total_bill"],
                                 df.loc[df["smoker"] == "No", "total_bill"])
print('Test Stat = %.4f, p-value = %.4f' % (test_stat, pvalue))

# Result :
#
# Test Stat = 7531.5000, p-value = 0.3413
#

4. Interpret the results based on P-values :
Hypothesis H0 cannot be rejected because the p-value is greater than 0.05.Therefore, there is no statistically significant difference between the two means.

Thanks.