Reviewing P-value
Yes, I am going back to the basics, reviewing the concept of P-value, after soullessly reading and accepting that “P-value < 0.05 as statistical significance” for a thousand times.
Code to reproduce the figures: https://github.com/ambercheny/hypothesis-testing
To understand P-value, we need to first stop thinking about the term P-value (stop the echo in your brain). Let’s zoom out and look at the first important concept, hypothesis testing.
Hypothesis testing
Every day we look at hundreds or thousands of different numbers, and each interpretation has a hidden agenda, is this value higher/lower than I would expect?
Hypothesis testing is a statistical framework that aims to compare two values and the null hypothesis is that two values are no different whereas the alternative hypothesis is that they are different (or one is higher than another; this is two-tailed/one-tailed problem given what you want to know). Essentially, we want to know how far away the two distributions are.
However, we need to have criteria to establish how much different the two values have to be for them to be truly different, in which case we can “reject” the null hypothesis.
Get YaLinChen (Amber)’s stories in your inbox
Join Medium for free to get updates from this writer.
So we need to introduce another concept, significance level.
Significance level
The significance level ($\alpha$) is actually the probability of falsely rejecting the null hypothesis when it is actually true. The commonly set threshold is 0.05, which is why we read a lot in scientific papers that “P-value < 0.05 as statistical significance.” For example, if we want to know if a group of students are taller than the average height of their age, then if their height mean falls into the shaded area (covering 5% of the sampling height curve, then it means that it is true that these students on average are taller than average. This method that checks whether a value falls in the shaded area (critical area as most statistical textbooks refer to) is the critical value method.
Critical value method checks whether the observation falls into the shaded area (critical area) defined by the significance level. Falling into the critical area means the observed value is so different from the null and that there is a less probability than our significance level to falsely reject the null hypothesis, giving us the justification to reject the null.
Z-test
Notice that we have converted the null hypothesis curve as a standard normal distribution, with mean = 0 and standard deviation (std)= 1. So when we compare the group mean to the null hypothesis, we need to compare on the same scale. This conversion is called Z-test. Nowadays statistical software like Python, R, etc. handle this automatically for you when you run hypothesis testing such as t-test. But conceptually this is important to understand how the comparison works.
By converting our data to a standard normal distribution, we can use universal reference tools — like a Z-table or t-table — to instantly find the exact mathematical boundaries for our test. For example, a significance level of 0.05 corresponds to 1.64 std with a one-sided comparison (e.g., is students’ average height taller than average) and 1.96 std for a two-sided comparison (e.g., is students’ average height different from the average).
The P-value approach
The p-value is then the tail area of the observed value Z-score, meaning the probability of seeing the observed result on the null hypothesis distribution. When the P-value is lower than significance level, meaning the observed value is more unlikely than what expect, we “reject” the null hypothesis and conclude that there is statistical difference.
If you still cannot fully understand, what you should take away
Next time, when you read a p-value, convert the p-value to z-score (see a quick script for conversion), draw a bell curve, and put the z score on the curve. Then draw the significance level on the curve. If the p-value falls OUTSIDE of the critical area, then we reject null hypothesis and conclude that there is statistical significance. Otherwise, the comparison is not satistically different.
If you notice something…
All the math above uses/assumes that the null hypothesis comes from a normal distribution. This needs to be true for Z-test to be valid and such assumption is usually met when you are testing with large samples. For questions such as when the group of students is less than 30 people, we use non-parametric tests like Mann-Whitney U test to accommodate for non-normal distribution.
