Hypothesis Testing & Rejection Region Approach Explained

Hypothesis Testing, P-value, Rejection Region Approach and Types I & II Error Explained

Photo by Doncoombez on Unsplash

⬅️ Previous: [Point Estimate and Confidence Intervals]

Hypothesis Testing

A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. Hypothesis testing allows us to make probabilistic statements about population parameters.

Null hypothesis (H0):

In simple terms, the null hypothesis is a statement that assumes there is no significant effect or relationship between the variables being studied. It serves as the starting point for hypothesis testing and represents the status quo or the assumption of no effect until proven otherwise. The purpose of hypothesis testing is to gather evidence (data) to either reject or fail to reject the null hypothesis in favour of the alternative hypothesis, which claims there is a significant effect or relationship.

Alternative hypothesis (H1 or Ha):

The alternative hypothesis, is a statement that contradicts the null hypothesis and claims there is a significant effect or relationship between the variables being studied. It represents the research hypothesis or the claim that the researcher wants to support through statistical analysis.

Example: Hypothesis tests are similar to jury trials, in a sense. In a jury trial, H0 is similar to the not-guilty verdict, and Ha is the guilty verdict. You assume in a jury trial that the defendant isn’t guilty unless the prosecution can show beyond a reasonable doubt that he or she is guilty. If the jury says the evidence is beyond a reasonable doubt, they reject H0 (not guilty) or accept Ha (guilty).

Null and Alternative Hypothesis

Important Points

• How to decide what will be Null hypothesis and what will be Alternate Hypothesis — Typically the Null hypothesis says nothing new is happening.
• We try to gather evidence to reject the null hypothesis.
• It’s important to note that failing to reject the null hypothesis doesn’t necessarily mean that the null hypothesis is true; it just means that there isn’t enough evidence to support the alternative hypothesis.

Steps involved in Hypothesis Testing:

1. Formulate a Null and Alternate hypothesis.
2. Select a significance level (This is the probability of rejecting the null hypothesis when it is actually true, usually set at 0.05 or 0.01).
3. Check assumptions (example distribution like normal or non-normal).
4. Decide which test is appropriate(Z-test, T-test, Chi-square test, ANOVA).
5. State the relevant test statistic (example z-score or z-value for Z-test).
6. Conduct the test.
7. Reject or not reject the Null Hypothesis.
8. Interpret the result.

Significance level: denoted as α (alpha), is a predetermined threshold used in hypothesis testing to determine whether the null hypothesis should be rejected or not. It represents the probability of rejecting the null hypothesis when it is actually true, also known as Type 1 error.

Rejection Region Approach:

The rejection region is the region of values that corresponds to the rejection of the null hypothesis at some chosen probability level. Here, we compare Compare Calculated Test Statistic to Critical Value to decide to reject or not reject the Null Hypothesis.

The Problem with the Rejection Region Approach:

• It lacks magnitude: The fundamental flaw of the Rejection Region approach is that it is a rigid, binary “Yes/No” decision framework that hides the actual strength of your evidence. It does not tell you how strongly you should reject the null hypothesis. For example, if your critical Z-value is 1.96, a test statistic of 1.97 and a test statistic of 5.50 both result in the exact same conclusion: “Reject Ho.” However, a Z-score of 5.50 is astronomically stronger evidence than 1.97. The rejection region method treats them as identical.
• It is highly inflexible: The critical value is strictly tied to the alpha you choose at the very beginning. If you calculate your test statistic using alpha = 0.05 and later decide you want to test at alpha = 0.01, you have to go back to the tables and recalculate your critical values all over again.
• It relies on archaic tables: Historically, this method was popular because it was easier to look up one critical value in a physical textbook table than to calculate the exact area under a curve by hand.

Type-I vs Type-II Error

Type-I (False Positive): The error occurs when the sample results,
lead to the rejection of the null hypothesis when it is in fact
true. In other words, it’s the mistake of finding a significant effect or
relationship when there is none. The probability of committing
a Type I error is denoted by α (alpha), which is also known as
the significance level. By choosing a significance level,
researchers can control the risk of making a Type I error.

Type-II (False Negative): The error occurs when based on the
sample results, the null hypothesis is not rejected when it is in
fact false. This means that the researcher fails to detect a significant
effect or relationship when one actually exists. The probability
of committing a Type II error is denoted by β (beta).

Trade-off between Type-I and Type-II errors:

The Inevitable Seesaw Effect: The most crucial concept to understand about these errors is that they are inversely related. You cannot simultaneously minimize both without increasing your sample size.

• If you want to be absolutely certain you don’t make a Type-I Error (False Positive), you might lower your significance level from α = 0.05 to α = 0.01. You are making your test much stricter. However, by making the test so strict, you make it much harder to reject the Null Hypothesis. This directly increases the probability that you will miss a real effect, thereby increasing your chance of a Type-II Error (False Negative), β.

One Sided vs Two Sided Test

One-Sided (One-Tailed) Test

A one-sided test is used when the researcher is interested in testing the effect in a specific direction (either greater than or less than the value specified in the null hypothesis). The alternative hypothesis in a one-sided test contains an inequality (either “>” or “<”). Example: A researcher wants to test whether a new medication increases the average recovery rate compared to the existing medication.

Example: Suppose a company is evaluating the impact of a new training program on the productivity of its employees. The company has data on the average productivity of its employees before implementing the training program. The average productivity was 50 units per day with a known population standard deviation of 5 units. After implementing the training program, the company measures the productivity of a random sample of 30 employees. The sample has an average productivity of 53 units per day. The company wants to know if the new training program has significantly increased productivity.

NOTE: Check normality of sample before proceedings. Here, i am not doing.

Z-Test, One Sided

• Advantages:

1. More powerful: One-tailed tests are generally more powerful than two-tailed tests, as the entire significance level (α) is allocated to one tail of the distribution. This means that the test is more likely to detect an effect in the specified direction, assuming the effect exists.
2. Directional hypothesis: One-tailed tests are appropriate when there is a strong theoretical or practical reason to test for an effect in a specific direction.

• Disadvantages:

1. Missed effects: One-tailed tests can miss effects in the opposite direction of the specified alternative hypothesis. If an effect exists in the opposite direction, the test will not be able to detect it, which could lead to incorrect conclusions.
2. Increased risk of Type I error: One-tailed tests can be more prone to Type I errors if the effect is actually in the opposite direction than the one specified in the alternative hypothesis.

Two-Sided (Two-Tailed) Test

A two-sided test is used when the researcher is interested in testing the effect in both directions (i.e., whether the value specified in the null hypothesis is different, either greater or lesser). The alternative hypothesis in a two-sided test contains a “not equal to” sign (≠). Example: A researcher wants to test whether a new medication has a different average recovery rate compared to the existing medication. The main difference between them lies in the directionality of the alternative hypothesis and how the significance level is distributed in the critical regions.

Get Mr. Vivek Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example: Suppose a snack food company claims that their Lays wafer packets contain an average weight of 50 grams per packet. To verify this claim, a consumer watchdog organization decides to test a random sample of Lays wafer packets. The organization wants to determine whether the actual average weight differs significantly from the claimed 50 grams. The organization collects a random sample of 40 Lays wafer packets and measures their weights. They find that the sample has an average weight of 49 grams, with a known population standard deviation of 4 grams.

Z-Test, Two Sided

• Advantages:

1. Detects effects in both directions: Two-tailed tests can detect effects in both directions, which makes them suitable for situations where the direction of the effect is uncertain or when researchers want to test for any difference between the groups or variables.
2. More conservative: Two-tailed tests are more conservative because the significance level (α) is split between both tails of the distribution. This reduces the risk of Type I errors in cases where the direction of the effect is uncertain.

• Disadvantages:

1. Less powerful: Two-tailed tests are generally less powerful than one-tailed tests because the significance level (α) is divided between both tails of the distribution. This means the test requires a larger effect size to reject the null hypothesis, which could lead to a higher risk of Type-II errors (failing to reject the null hypothesis when it is false).
2. Not appropriate for directional hypotheses: Two-tailed tests are not ideal for cases where the research question or hypothesis is directional, as they test for differences in both directions, which may not be of interest or relevance.

One Sided vs Two Sided Test

Where can be Hypothesis Testing Applied?

1. Testing the effectiveness of interventions or treatments: Hypothesis testing can be used to determine whether a new drug, therapy, or educational intervention has a significant effect compared to a control group or an existing treatment.
2. Comparing means or proportions: Hypothesis testing can be used to compare means or proportions between two or more groups to determine if there’s a significant difference. This can be applied to compare average customer satisfaction scores, conversion rates, or
   employee performance across different groups.
3. Analysing relationships between variables: Hypothesis testing can be used to evaluate the association between variables, such as the correlation between age and income or the relationship between advertising spend and sales.
4. Evaluating the goodness of fit: Hypothesis testing can help assess if a particular theoretical distribution (e.g., normal, binomial or Poisson) is a good fit for the observed data.
5. Testing the independence of categorical variables: Hypothesis testing can be used to determine if two categorical variables are independent or if there’s a significant association between them. For example, it can be used to test if there’s a relationship between the type of product and the likelihood of it being returned by a customer.
6. A/B testing: In marketing, product development, and website design, hypothesis testing is often used to compare the performance of two different versions (A and B) to determine which one is more effective in terms of conversion rates, user engagement, or other metrics.

Hypothesis Testing ML Applications:

1. Model comparison: Hypothesis testing can be used to compare the performance of different machine learning models or algorithms on a given dataset. For example, you can use a paired t-test to compare the accuracy or error rate of two models on multiple cross-validation folds to determine if one model performs significantly better than the other.
2. Feature selection: Hypothesis testing can help in identify which features are significantly related to the target variable or contribute meaningfully to the model’s performance. For example, you can use a t-test, chi-square test, or ANOVA to test the relationship between individual features and the target variable. Features with significant relationships can be selected for building the model, while non-significant features may be excluded.
3. Hyperparameter tuning: Hypothesis testing can be used to evaluate the performance of a model trained with different hyperparameter settings. By comparing the performance of models with different hyperparameters, you can determine if one set of hyperparameters leads to significantly better performance.
4. Assessing model assumptions: In some cases, machine learning models rely on certain statistical assumptions, such as linearity or normality of residuals in linear regression. Hypothesis testing can help assess whether these assumptions are met, allowing you to determine if the model is appropriate for the data.

P-value Approach

P-value is the probability of getting a sample as or more extreme(having more evidence against H0) than our own sample given the Null Hypothesis(H0) is true. In simple words, p-value is a measure of the strength of the evidence against the Null Hypothesis that is provided by our sample data.

Here, We compare the p-value with the significance level.

Example: Acoin is tossed 100 times and the below graph is showing probability of getting a discrete number of heads. alpha, α = 0.05. Ho: p(H) = p(T) and Ha: p(H) > p(T).

PMF of Binomail Distribution of Coin Tossing

We tossed 100 times the coin and got 53 Heads. So, p-value = p(H > 53) = 0.3086 and p(53) = 0.0666. Failed to reject the null hypothesis: There is not enough evidence to conclude that the coin is not fair.

PMF of Binomail Distribution of Coin Tossing | p-value = p(H > 53) = 0.3086

We tossed 100 times the coin and got 60 Heads. So, p-value = p(H > 60) = 0.0284 and p(60) = 0.0108. Reject the null hypothesis: The coin is not fair.

p-value = p(H > 60) = 0.0284

We tossed 100 times the coin and got 83 Heads. So, p-value = p(H > 83) = 0.0000 and p(83) = 0.0000. Reject the null hypothesis: The coin is not fair.

p-value = p(H > 83) = 0.0000

With significance value, alpha (α):

If p-value ≤ alpha (α) then reject the null hypothesis. Else p-value > α, Failed to reject the null hypothesis.

Without significance value, alpha (α):

1. Very small p-values (e.g., p < 0.01) indicate strong evidence against the null hypothesis, suggesting that the observed effect or difference is unlikely to have occurred by chance alone. Rejecct the Null Hypothesis.
2. Small p-values (e.g., 0.01 ≤ p < 0.05) indicate moderate evidence against the null hypothesis, suggesting that the observed effect or difference is less likely to have occurred by chance alone. Rejecct the Null Hypothesis.
3. Large p-values (e.g., 0.05 ≤ p < 0.1) indicate weak evidence against the null hypothesis, suggesting that the observed effect or difference might have occurred by chance alone, but there is still some level of uncertainty. need to talk to the stack holders and do some more investigation. Failed to reject the Null Hypothesis.
4. Very large p-values (e.g., p ≥ 0.1) indicate weak or no evidence against the null hypothesis, suggesting that the observed effect or difference is likely to have occurred by chance alone. Failed to reject the Null Hypothesis.

P-value in context of Z-test:

Example 1: Suppose a company is evaluating the impact of a new training program on the productivity of its employees. The company has data on the average productivity of its employees before implementing the training program. The average productivity was 50 units per day. After implementing the training program, the company measures the productivity of a random sample of 30 employees. The sample has an average productivity of 53 units per day and the population standard deviation is 4. The company wants to know if the new training program has significantly increased productivity.

One Tailed Test | p-value in z-test

NOTE: p-value = P(Z ≥ 4.11) = 1 — P(Z < 4.11) = 1–0.99998 = 0.00002

Example 2: Suppose a snack food company claims that their Lays wafer packets contain an average weight of 50 grams per packet. To verify this claim, a consumer watchdog organization decides to test a random sample of Lays wafer packets. The organization wants to determine whether the actual average weight differs significantly from the claimed 50 grams. The organization collects a random sample of 40 Lays wafer packets and measures their weights. They find that the sample has an average weight of 49 grams, with a pop standard deviation of 5 grams.

Two Tailed Test | p-value in z-test

NOTE: p-value = 2 * P(z<-1.26) = 2*0.103 = 0.206 or 20.6% and on camparing with alpha 0.05 or 5%, p-value is greater than alpha.

T-tests

A t-test is a statistical test used in hypothesis testing to compare the means of two samples or to compare a sample mean to a known population mean. The t-test is based on the t-distribution, which is used when the population standard deviation is unknown and the sample size is small. There are three main types of t-tests:

• One-sample t-test: The one-sample t-test is used to compare the mean of a single sample to a known population mean. The null hypothesis states that there is no significant difference between the sample mean and the population mean, while the alternative hypothesis states that there is a significant difference.
• Independent two-sample t-test: The independent two-sample t-test is used to compare the means of two independent samples. The null hypothesis states that there is no significant difference between the means of the two samples, while the alternative hypothesis states that there is a significant difference.
• Paired t-test (dependent two-sample t-test): The paired t-test is used to compare the means of two samples that are dependent or paired, such as pre-test and post-test scores for the same group of subjects or measurements taken on the same subjects under two different conditions. The null hypothesis states that there is no significant difference between the means of the paired differences, while the alternative hypothesis states that there is a significant difference.

Why a Paired 𝑍-Test Doesn’t Really Exist?

A paired test looks at the difference scores (Before minus After) for a single group of people.

• To run a paired 𝑍-test, you would need to know the true population standard deviation of the differences (𝜎𝑑).
• In the real world, while we might occasionally know the standard deviation of an overall population (like IQ scores or height), we almost never know the exact population variance for how much people change after a specific, localized treatment.
• Because 𝜎𝑑 is unknown, researchers always use the sample standard deviation of the differences (𝑠𝑑), which forces the use of a paired 𝑡-test.

Single Sample t-test

A one-sample t-test checks whether a sample mean differs from the population mean.

Degree of Freedom = n — 1

Assumptions for a single sample t-test
1. Normality — Population from which the sample is drawn is normally distributed
2. Independence — The observations in the sample must be independent, which means that the value of one observation should not influence the value of another observation.
3. Random Sampling — The sample must be a random and representative subset of the population.
4. Unknown population std — The population std is not known.

Example: Suppose a manufacturer claims that the average weight of their new chocolate bars is 50 grams, we highly doubt that and want to check this so we drew out a sample of 25 chocolate bars and measured their weight, the sample mean came out to be 49.7 grams and the sample std deviation was 1.2 grams. Consider the significance level to be 0.05.

p-value in Single Sample t-test

Critical t-value at df=24 and 5% Significance Level = +- 2.064 and on comparing with Calculated t-value =-1.25.

And p-value = 2*P(t-value>-1.25, df=24) = 2*0.11 = 0.22 and on comparing this with alpha 0.05.

Independent 2 sample t-test

An independent two-sample t-test, also known as an unpaired t-test, is a statistical method used to compare the means of two independent groups to determine if there is a significant difference between them.
Assumptions for the test:

1. Independence of observations: The two samples must be independent, meaning there is no relationship between the observations in one group and the observations in the other group. The subjects in the two groups should be selected randomly and independently.
2. Normality: The data in each of the two groups should be approximately normally distributed. The t-test is considered robust to mild violations of normality, especially when the sample sizes are large (typically n ≥ 30) and the sample sizes of the two groups are similar. If the data is highly skewed or has substantial outliers, consider using a non-parametric test, such as the Mann-Whitney U test.
3. Equal variances (Homoscedasticity): The variances of the two populations should be approximately equal. This assumption can be checked using F-test for equality of variances. If this assumption is not met, you can use Welch’s t-test, which does not require equal variances.
4. Random sampling: The data should be collected using a random sampling method from the respective populations. This ensures that the sample is representative of the population and reduces the risk of selection bias.

Independent Two sample T-Test formula

Degree of Freedom

Example: Suppose a website owner claims that there is no difference in the average time spent on their website between desktop and mobile users. To test this claim, we collect data from 30 desktop users and 30 mobile users regarding the time spent on the website in minutes. Assuming one use only use one device. The sample statistics are as follows:

• desktop users = [12, 15, 18, 16, 20, 17, 14, 22, 19, 21, 23, 18, 25, 17, 16, 24, 20, 19, 22, 18, 15, 14, 23, 16, 12, 21, 19, 17, 20, 14]
• mobile_users = [10, 12, 14, 13, 16, 15, 11, 17, 14, 16, 18, 14, 20, 15, 14, 19, 16, 15, 17, 14, 12, 11, 18, 15, 10, 16, 15, 13, 16, 11]

Independent 2 sample t-test

p_value = 2*P(t < 5.25) = 0 and on comparing with alpha, 0.05

Paired 2 sample t-test

A paired two-sample t-test, also known as a dependent or paired-samples t-test, is a statistical test used to compare the means of two related or dependent groups.
Common scenarios where a paired two-sample t-test is used include:
1. Before-and-after studies: Comparing the performance of a group before and after an intervention or treatment.
2. Matched or correlated groups: Comparing the performance of two groups that are matched or correlated in some way, such as siblings or pairs of individuals with similar characteristics.

Assumptions
1. Paired observations: The two sets of observations must be related or paired in some way, such as before-and-after measurements on the same subjects or observations from matched or correlated groups.
2. Normality: The differences between the paired observations should be approximately normally distributed. This assumption can be checked using graphical methods (e.g., histograms, Q-Q plots) or statistical tests for normality (e.g., Shapiro-Wilk test). Note that the t-test is generally robust to moderate violations of this assumption when the sample size is large.
3. Independence of pairs: Each pair of observations should be independent of other pairs. In other words, the outcome of one pair should not affect the outcome of another pair. This assumption is generally satisfied by appropriate study design and random sampling.

Paired 2 sample t-test

Example 3: Let’s assume that a fitness center is evaluating the effectiveness of a new 8-week weight loss program. They enroll 15 participants in the program and measure their weights before and after the program. The goal is to test whether the new weight loss program leads to a significant reduction in the participants’ weight. Significance level (α) = 0.05
Before the program: [80, 92, 75, 68, 85, 78, 73, 90, 70, 88, 76, 84, 82, 77, 91]
After the program: [78, 93, 81, 67, 88, 76, 74, 91, 69, 88, 77, 81, 80, 79, 88]

Paired 2 sample t-test

NOTE: before — after = diff and mu diff = 0 as assumed

➡️ Next Up: [Chi Square Distribution, Goodness of Fit Test & Test for Independence]

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!