A/B Testing Project
Statistics for Business Project: A Data-Driven Approach through A/B Testing
In today’s dynamic marketing landscape, the pursuit of successful campaigns is a constant challenge. With a multitude of strategies and options at their disposal, marketing companies find themselves navigating a complex and ever-evolving market. This complexity necessitates a methodical approach to campaign optimization, which is where A/B testing comes into play.
A/B testing, a fundamental tool in the marketer’s arsenal, is a randomized experimentation process that involves the simultaneous presentation of two or more versions of a variable (such as a web page, page element, banner, etc.) to different segments of the audience. Its primary objective is to unveil which version wields the greatest influence and significantly impacts the desired business metrics.
For marketing companies, two pivotal questions loom large:
1. Will the campaign be a resounding success?
2. If the campaign proves successful, to what extent can that success be attributed to the advertisements?
It is in the quest to answer the second question that A/B testing assumes a crucial role. In this process, the vast majority of the target audience is exposed to the advertisements, constituting the experimental group. Conversely, a smaller portion of the audience forms the control group, and they are presented with a Public Service Announcement (PSA) or nothing, occupying the precise location and size where the ad would typically appear.
The essence of the dataset lies in its ability to dissect these two groups, discern the effectiveness of the advertisements, calculate the revenue generated from these ads, and determine whether the disparities between the groups are statistically significant. This empirical analysis equips marketing companies with the insights they need to make informed decisions, ensuring their campaigns not only thrive but also deliver tangible, data-backed results.
Data dictionaries given below :
- Index: Row index
- user id: User ID (unique)
- test group: If “ad” the person saw the advertisement, if “psa” they only saw the public service announcement
- converted: If a person bought the product then True, else is False
- total ads: Amount of ads seen by person
- most ads day: Day that the person saw the biggest amount of ads
- most ads hour: Hour of day that the person saw the biggest amount of ads
Data Preparation
- Data Eksploration
import pandas as pd
# Create a sample DataFrame for demonstration purposes
data = {
'Data Feature': df.columns,
'Data Type': df.dtypes,
'Null Values': df.isnull().sum(),
'Count Unique': df.nunique(),
'Unique Sample': [df[col].sample(5).to_list() for col in df.columns]
}
info_df = pd.DataFrame(data)
info_df
2. Data Cleaning
Get Saprilian Hidayat’s stories in your inbox
Join Medium for free to get updates from this writer.
First, we will remove the “Unnamed: 0” column and then rename each column by adding underscores in place of spaces.
import pandas as pd
class DataFrameOperations:
def __init__(self, df):
self.df = df
def remove_unnamed_column(self):
if "Unnamed: 0" in self.df.columns:
self.df = self.df.drop(["Unnamed: 0"], axis=1)
def rename_columns(self):
self.df.rename(columns=lambda x: x.strip().replace(" ", "_"), inplace=True)
df_ops = DataFrameOperations(df)
df_ops.remove_unnamed_column()
df_ops.rename_columns()
The data we have is already clean and we can proceed with further exploration.
Bostrap Resample
Bootstrap method is a resampling technique used in statistical analysis.Basically,it involves repeatedly sampling observations from a dataset with replacement to create multiple resamples. This resampling process allows for estimating the sampling distribution of a statistic or making inferences about population parameters. In this project, I’m going to resample from the original datarfame to create resampled dataframes of 1000 sample means for both experimental group and control group.
Before doing bootstrapping, we can first subset the original dataframe into experimental group dataframe and control group dataframe.
#subset the original dataframe
ad_experimental=df[df['test_group']=='ad']
psa_control=df[df['test_group']=='psa']
#find the average converted rate of each group
ad_converted=ad_experimental['converted'].mean()
psa_converted=psa_control['converted'].mean()
print(ad_converted,psa_converted)Based on the sample data we’ve got, the average converted rate of ad (2.55%) is 0.76% higher than the converted rate of psa(1.79%). It seems running ad is successful at improving the converted rate. However, is it real to a larger population? Are these numbers got influenced by the greater sample size of ad? To answer these questions, we need to do statistical significance tests (z-test or t-test) based on data with balanced sample sizes (bootstrap).
#create an empty list to store bootstrapped means
boot_ad=[]
#Set up a loop that will iterate 1000 times. In each iteration, a new bootstrapped sample will be generated
for i in range (1000):
boot_mean=ad_experimental.sample(frac=1,replace=True)['converted'].mean()
boot_ad.append(boot_mean)
boot_ad=pd.DataFrame(boot_ad)
#Create a density plot of the bootstrapped means
boot_ad.plot(kind='density')
From the density plot,we can clearly see the sampling distribution of the mean (obtained through the bootstrap resampling process) is approximately normal. It allows for the application of statistical tests and confidence intervals that rely on the assumption of normality. For example, we can perform hypothesis testing or construct confidence intervals through using methods like the t-test or z-test based on the assumption of normality.
We will repeat the bootstrap resampling process to the psa group.
boot_psa=[]
for i in range(1000):
boot_mean=psa_control.sample(frac=1,replace=True)['converted'].mean()
boot_psa.append(boot_mean)
boot_psa=pd.DataFrame(boot_psa)
boot_psa.plot(kind='density')The bootstrap means of psa group also follows the normal distribution. Now, we are confident to do hypothesis test by running z-test or t-test based on the bootstrap data we’ve just got.
A/B Testing
1. Z-Test
Null Hypothesis (H0): The conversion rate of the ad group is equal to or less than the conversion rate of the psa group.
Alternative Hypothesis (H1): The conversion rate of the ad group is significantly greater than the conversion rate of the psa group.
#conduct a z-test
from scipy import stats
boot_ad_converted=boot_strap['ad_converted']
boot_psa_converted=boot_strap['psa_converted']
boot_ad_mean=boot_ad_converted.mean()
boot_psa_mean=boot_psa_converted.mean()
boot_ad_std=boot_ad_converted.std()
boot_psa_std=boot_psa_converted.std()
pooled_std=np.sqrt((boot_ad_std**2+boot_psa_std**2)/2)
z_score=(boot_ad_mean-boot_psa_mean)/pooled_std
p_value=1-stats.norm.cdf(z_score)
print('z score',z_score)
print('P Value',p_value)Based on the z score is 12.5287 and p value is 0.0, we can conlude that the conversion rate of ad group is significantly greater than the conversion rate of psa group.
The high z-score indicates a large difference between the sample means, suggesting a substantial effect. The p-value of 0.0 indicates that the possibility of observing extreme results under null hypothesis is very low, providing strong evidence against the null hypothesis.
Theefroe, we reject the null hypothesis and support the alternative hypothesis that the conversion rate of ad group is significantly greater than the conversion rate of psa group.
2. T-Test
# conduct a t-test
t_stat, p_value = stats.ttest_ind(boot_ad_converted, boot_psa_converted, alternative='greater', equal_var=False)
print("T-statistic:", t_stat)
print("P-value:", p_value)Similarly, the large t-statistic of 280.150 and a p-value of 0.0 suggest strong evidence against the null hypothesis and provides support for the alternative hypothesis.
Overall, both the z-test and t-test provide strong evidence of a significant difference in the conversion rates, supporting the conclusion that the ad group has a higher conversion rate than the psa group.
