Hypothesis Testing
We often encounter the concept of hypothesis testing while studying Data Science, yet it can still be confusing. Let’s clarify what hypothesis testing is and how it works.
What is hypothesis testing?
Hypothesis testing is a statistical approach used to make inferences or decisions about a population based on sample data. Conducting analysis directly on the entire population data can be challenging — if not impossible — so we use sample data instead to draw conclusions. This reverse approach allows us to infer information about the larger population without having to collect all of its data.
Before diving deeper into hypothesis testing, it’s essential to understand the Central Limit Theorem (CLT). According to the CLT, the distribution of sample means (calculated from multiple random samples drawn from a population) tends to follow a Gaussian (normal) distribution, regardless of the original population’s distribution. In other words, if we repeatedly take random samples from a population, calculate the mean of each sample, and then plot these sample means, we will observe a bell-shaped curve — a normal distribution.
Get Santosh Kumar Jami’s stories in your inbox
Join Medium for free to get updates from this writer.
If the population itself follows a Gaussian distribution, the distribution of sample means will also be a proper Gaussian distribution. However, even if the population follows a different distribution, the distribution of sample means will still approximate a Gaussian distribution, given a large enough sample size.
If the sample statistic falls at the extreme ends of this Gaussian distribution, we can assume that the sample may contain some rare values from the population. Our goal is to determine whether our sample statistic falls at the edges or in the center of the distribution. To make this decision, we use the following key factors:
- Confidence Interval: This is the range of values calculated from the sample data that is likely to contain the true population parameter with a certain level of confidence. Commonly used confidence levels are 95% and 99%.
- Significance Level: This is the probability of rejecting the null hypothesis when it is actually true, also known as a Type I error. Typical significance levels are 0.05 (5%) or 0.01 (1%).
- P-Value: This is the probability of observing a sample statistic as extreme as, or more extreme than, the one observed, assuming the null hypothesis is true. The p-value supports the null hypothesis. If the p-value is less than the significance level, there is not enough evidence to support the null hypothesis, so we reject it. Conversely, if the p-value is greater than the significance level, we fail to reject the null hypothesis.
We determine the p-value using statistical tests like the z-test and t-test, depending on sample size and assumptions about the population distribution.
Let’s understand how this works in detail. Suppose our significance level is 0.05, and we obtain a p-value of 0.03, which is less than the significance level. This lower p-value indicates that the probability of our statistic falling in the extreme region is low, meaning it likely falls within the normal range. Since the p-value supports the null hypothesis, it should ideally be high if we want to avoid rejecting the null hypothesis. In this case, however, the p-value is low, so we reject the null hypothesis.
On the other hand, if the p-value is greater than the significance level, then the probability of the statistic falling in the extreme region is high, which supports the null hypothesis, and we fail to reject it.
Note: The null hypothesis (H0) assumes there is no effect or no relationship between the sample data and the population.
In summary, hypothesis testing follows a reverse approach where we start with a “negative” assumption (the null hypothesis) and use the sample data decide whether to reject or accept it.
