Hypothesis Testing with Python

1. Intro

Hypothesis testing is a statistical method used to determine whether there is enough evidence in a sample of data to support or reject a hypothesis about a population.

2. Definition

Hypothesis testing is a statistical method used to infer population parameters based on sample data i.e the process of using sample data to make educated guesses or conclusions about characteristics (parameters) of a larger population.

It involves two competing hypotheses:

• The null hypothesis (H₀) which claims there is no effect or difference.
• The alternative hypothesis (Hₐ), which claims there is an effect or difference.

The process includes:

1. Formulating Hypotheses: Define H₀ and Hₐ.
2. Selecting a Significance Level (α): Define a cutoff point in order to reject the null hypothesis i.e claim that there is a change, typically 0.05.
3. Computing a Test Statistic: Form a statistic such as z, t of the sample.
4. Determining the p-value: It calculates what is the probability of seeing the test statistic under H₀. In simpler terms, it helps us measure the chance that the patterns or differences we see in our data happened just by random chance. A small p-value means it’s unlikely the results are due to chance, suggesting the null hypothesis might be wrong.
5. Making a Decision: Reject H₀(means: there might be a significant changes) if the p-value ≤ α and fail to reject H₀ if it isn’t.

2.1. Type I and Type II Errors

Type I error occurs if you reject a true null hypothesis, that is, if you believe an effect exists when there isn’t. A Type II error occurs when you do not reject a false null hypothesis, that is, when you do not detect a real effect. So to see the clear difference with these two check the illustration below.

Type I & Type II Errors Hypothesis Testing

3. Hypothesis Testing with Python

Now, let’s start to conduct hypothesis testing with Python, with emphasis on hypothesis testing for population mean & one tailed and two tailed tests.

Before, diving into the testing we have to install (if yet not installed) and import required libraries.

# 'pip install scipy' if you install it from terminal
!pip install scipy # installing directly from notebook

# import the required packages
import scipy
import pandas as pd  
import numpy as np  
import matplotlib.pyplot as plt  
import seaborn as sns

%matplotlib inline
import scipy.stats as stats  # this library has a significant amount of probability distributions and a (growing) amount of statistical functions

3.1 Hypothesis Test for Population Mean 𝜇

Problem:
The mean delivery time for an E commerce company is 5 days with a standard deviation of 1.3 days. The new customer service manager samples 45 ordersand worries about delays, finding a mean delivery time of 5.25 days. Can we have enough statistical evidence to believe that the delivery time exceeds 5 days with a significance level of α = 0.05?

Let’s define null (H₀) and alternative hypothesis (Hₐ).

1.1 Let μ be the mean delivery time of the products.

The null hypothesis:

H₀:𝜇=5

The alternate hypothesis:

Hₐ:𝜇>5

1.2 Are the Assumptions of the Z test met?

The Central Limit Theorem states that if the sample size is greater than 30 then the distribution of sample means will be normal. However, if the sample size was less than 30, the Z test could be applied if we knew that the population distribution was normal. In addition, the manager’s collection method indicates that the observations are from a simple random sample, and the standard deviation is known.

Now we will calculate the Z test statistic.

mu, sigma = 5, 1.3 # mu is 5(population mean) with 1.3 standard deviation

x_bar = 5.25 # ~!!sample mean!!~# calculating the test statistic
test_stat = (x_bar - mu) / (sigma/np.sqrt(45))
test_stat

1.2900392177883402

Now, let’s plot the distribution of the Z test statistic and see where the computed test statistic lies on the plot.

# NOTE:to make content more concise, simpler plote code provided instead of full code as the below graph
# thus, expect your graph to be VISUALLY different than below
from scipy.stats import norm

# We are plotting the distributions here to better visualize the calculations
x = np.linspace(-4, 4, 100)
plt.plot(x, norm.pdf(x, 0, 1))
plt.axvline(x = test_stat, c = 'r')
plt.show()

Distribution of Z Test statistics Hypothesis Testing

The above distribution is a distribution of test statistic. What is the probability of making this test statistic or a bigger value in the mentioned distribution?

1 - norm.cdf(test_stat)

0.09851852092578695

If you wonder the logic of CDF, you can check this link:

Continuous Uniform Distribution with Python

The probability is very small (which might indicate the rejection of H₀), but is it significant enough to reject the null hypothesis in favour of alternate hypothesis?

1.3 Rejection & Acceptance Region | p-value

It is a very small probability, though we cannot actually conclude that the probability is sufficient to reject the null hypothesis in favour of the alternative hypothesis. We can assess this using one of the following approaches:

Get Ja X’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

1.3.1 Rejection Region Approach

• Select the Level of Significance (⍺): That means the probability of rejecting the null hypothesis when it is true.
• Identify the Rejection Region: Find out where on the graph the null hypothesis can be rejected.
• Make a Decision: If the test statistic is found in this region, reject the null hypothesis.
  In our example, Z test statistic is a standard normal. If the Z values at the right end are strong evidence against the null hypothesis, then we will find the critical Z value for an area of ⍺ to the right.

from scipy.stats import norm

# find the value
critical_val = norm.ppf(1-.05)
critical_val

1.6448536269514722

The critical value separates the region — will reject the null hypothesis from the region won’t reject the null hypothesis. See the below plot.

# NOTE:to make content more concise, simpler plote code provided instead of full code as the below graph
# thus, expect your graph to be VISUALLY different than below

x = np.linspace(-4, 4, 100)
plt.plot(x, norm.pdf(x, 0, 1))
plt.axvline(x = critical_val, c = 'r')
x1 = np.linspace(critical_val, 4, 50)
plt.fill_between(x1, norm.pdf(x1, 0, 1), color='r')
plt.annotate('Reject Null', (2, 0.20))
plt.annotate('  Do Not Reject\n        Null', (-1, 0.20))
plt.show()

Acceptance and Rejection region Hypothesis Testing

Our test statistic (~ 1.29) does not lie in the rejection region( which is the left of ~1.65), so we can not reject the null hypothesis.

1.3.2 p-value Approach

The rejection region approach allows determining opportunity of rejecting null hypothesis, but doesn’t quantify the strength of the evidence. This is why statisticians are often partial to the p-value approach, which measures that strength.

• Choose the Level of Significance (⍺): Set your significance value.
• Calculate the p-value: It is the probability of observing the test statistic (or more extreme) if the null hypothesis is true.
• Whenever we have decided p-value ≤ α, we reject the null hypothesis and otherwise, we fail to reject.

In the example that was given, the p value was the area to the right of the test statistic to the standard normal curve.

# calculate the p-value
p_value = 1 - norm.cdf(test_stat)
p_value

0.09851852092578695

Since the p-value ~0.098 is greater than significance level we can not reject the null hypothesis, considering the formula p-value ≤ α. Therefore, we cannot say that the mean delivery time of a product has exceeded 5 days.

3.2 One-tailed and Two-tailed Tests

Example: A standard deviation of 25 ml is associated with a best selling Coke soft drink of 330 ml. A customer suspects that a bottle contains less than 330 ml, leading to the following hypotheses:

- Null Hypothesis (H₀): μ = 330
- Alternative Hypothesis (Hₐ): μ < 330

Meanwhile, the quality control team aims to maintain exactly 330 ml per bottle, forming these hypotheses:

- Null Hypothesis (H₀): μ = 330
- Alternative Hypothesis (Hₐ): μ ≠ 330

Context determines whether a one sided or a two sided alternative hypothesis is chosen.

2.1 Two-tailed Test Example
Being part of the quality control team, you know that the mean amount of Coke is 330 ml and standard deviation is 25 ml. You collect 36 random samples and find the mean amount is 320 ml. You want to test if the amount above is different from 330 ml at 0.05 significance level.

Let’s calculate the Z-Test Statistics

mu, sigma = 330, 25 # mu population mean
x_bar = 320 # sample mean

# calculate the test statistic
test_stat = (x_bar - mu) / (sigma/np.sqrt(36))
test_stat

-2.4

In this given example, the Z Test statistic is normally distributed. Strong evidence against the null hypothesis are the Z Test at the left and right end of the distribution. We will find the values of Z (critical values), which give an area of 𝛼/2 to the left and right ends as rejection region.

Visualisation:

# NOTE:to make content more concise, simpler plote code provided instead of full code as the below graph
# thus, expect your graph to be VISUALLY different than below

from scipy.stats import norm# find the critical values
critical_val1 = norm.ppf(1-(0.05/2))
critical_val2 = norm.ppf(0.05/2)# plot the rejection and acceptance region
x = np.linspace(-4, 4, 100)
plt.plot(x, norm.pdf(x, 0, 1))
plt.axvline(x = critical_val1, c = 'r')
x1 = np.linspace(critical_val1, 4, 50)
plt.fill_between(x1, norm.pdf(x1, 0, 1), color='r')
plt.axvline(x = critical_val2, c = 'r')
x1 = np.linspace(-4, critical_val2, 50)
plt.fill_between(x1, norm.pdf(x1, 0, 1), color='r')
plt.annotate('Reject Null', (2.2, 0.20))
plt.annotate('Reject Null', (-3.5, 0.20))
plt.annotate('  Do Not Reject\n        Null', (-1, 0.20))
plt.show()

Two Tailed Hypothesis Testing

Our test statistic (~ -2.4) is in the rejection region so we can reject the null hypothesis. Therefore, we can say that the amount of coke in the bottle is not exactly 330ml.

2.2 One-tailed Example

Suppose that you are a customer who wants to test if the amount of coke in a bottle is less than 330 ml and you know that the mean amount of coke is 330 ml and the standard deviation is 25 ml .
You have 36 random bottle that you have sampled. In the 36 samples the mean amount of coke is 320 ml.
Do you have enough Statistical evidence for it?

Z-Test Statistics

mu, sigma = 330, 25
x_bar = 320

# calculate the test statistic
test_stat = (x_bar - mu) / (sigma/np.sqrt(36))
test_stat

Let’s plot the one-tailed graph

# NOTE:to make content more concise, simpler plote code provided instead of full code as the below graph
# thus, expect your graph to be VISUALLY different than below

from scipy.stats import norm# find the critical value
critical_val = norm.ppf(0.05)# plot the rejection and acceptance region
x = np.linspace(-4, 4, 100)
plt.plot(x, norm.pdf(x, 0, 1))
plt.axvline(x = critical_val, c = 'r')
x1 = np.linspace(-4, critical_val, 50)
plt.fill_between(x1, norm.pdf(x1, 0, 1), color='r')
plt.annotate('Reject Null', (-3.5, 0.20))
plt.annotate('  Do Not Reject\n        Null', (-1, 0.20))
plt.show()

One Tailed Hypothesis Testing

Since our test statistic (~-2.4) falls in rejection region, we reject the null hypothesis. This is enough statistical evidence with which we can say that the amount of coke in the bottle is less than 330 ml.

The key takeaway for hypothesis testing regarding the population mean μ is that both the Rejection Region and p-value approaches yield the same conclusion: there is not sufficient statistical evidence to assert that the mean delivery time of a product is greater than 5 days.

Furthermore, in one tailed tests, differences are examined in only one direction, whereas in two tailed tests differences are examined in both directions; therefore, such a choice depends on the nature of the problem.