Sampling bias is a Systematic error in statistics that occurs when some members of a population are more likely to be included in a sample than others. This results in a non-representative sample, which can skew results and lead to incorrect conclusions.
Understanding sampling bias is essential for students, as it directly affects the validity and reliability of statistical analyses.
For example, in Figure 1:
The image illustrates a population from which a sample is drawn. Notice that only individuals with similar characteristics (e.g., same clothing) are selected into the sample. This highlights how sampling bias can occur when the sample does not adequately represent the diversity of the population.
This article will cover the concept of sampling bias and its types, and provide examples and practice problems to help students grasp this important topic.
Types of Sampling Bias
Below are the most common types of Sampling Bias are as follows:
Type of Sampling Bias | About it |
|---|---|
Selection Bias | Some members of the population are systematically more likely to be included in the sample. |
Survivorship Bias | Only surviving subjects are considered, leading to an overestimation of the success rate. |
Undercoverage Bias | Some members of the population are inadequately represented in the sample. |
Voluntary Response Bias | The sample consists of volunteers who choose to participate, often leading to a non-representative sample. |
Non-response Bias | Individuals chosen for the sample are unwilling or unable to participate. |
Time interval Bias | Results are influenced by the specific time period during which the sample is collected |
Minimizing Sampling Bias
Some methods can help mitigate or identify bias in sampling methods. These are crucial for ensuring that statistical analyses are accurate and reliable. Here are some important related concepts:
Random Sampling
Random sampling is a technique where each member of a population has an equal chance of being selected. This method helps ensure that the sample is representative of the population, thereby reducing sampling bias.
Example: Drawing names from a hat where each name has an equal chance of being picked.
Stratified Sampling
Stratified sampling involves dividing the population into subgroups (strata) based on a specific characteristic (e.g., age, gender, income level) and then taking a random sample from each subgroup.
Example: In a survey on education, dividing the population into strata based on educational level (e.g., high school, undergraduate, graduate) and then randomly sampling from each stratum.
Systematic Sampling
Systematic sampling involves selecting every nth member of the population after a random starting point.
Example: Choosing every 10th person on a list after randomly selecting a starting point between 1 and 10.
Cluster Sampling
Cluster sampling involves dividing the population into clusters, usually based on geography or other natural groupings, and then randomly selecting entire clusters for the sample.
Example: Dividing a city into districts and randomly selecting some districts, then surveying all individuals within those districts.
Weighting Adjustments: Oversampling and Undersampling
These are deliberate techniques used after initial sampling or during dataset construction, primarily for analytical purposes:
- Oversampling: Increasing the proportion of a particular subgroup within the sample to ensure adequate representation. (e.g., surveying extra people from a small ethnic minority to ensure you have enough data for reliable subgroup analysis).
- Undersampling: Reducing the proportion of a dominant subgroup to balance the sample. (Common in machine learning to balance datasets where one class is vastly overrepresented, like fraud detection).
Example: In a health study, oversampling a minority group to ensure their health outcomes are adequately represented.
Results from oversampled/undersampled data CANNOT be directly generalized to the overall population without careful re-weighting. The primary goal is often robust analysis of the subgroup or improved model training, not estimating overall population parameters.
Applications of Sampling Bias on CS
Machine Learning (ML) & Artificial Intelligence (AI)
- Training Data Bias: The most pervasive impact. If the data used to train an ML model suffers from sampling bias, the model learns and amplifies those biases.
Examples: Resume screening tools disadvantaging women if trained on historical hiring data biased towards men. - Active Learning: The selection strategy for choosing which data points to label next can introduce bias if not carefully designed.
- Reinforcement Learning: The environment simulator or the distribution of initial states/actions can bias the learned policy.
Data Science & Analytics:
- Feature Engineering: Proxies like "zip code" or "browser type" encode historical biases (e.g., redlining).
- A/B Testing: Non-random cohort assignment (e.g., testing features only on engaged users) → Misleading success metrics.
- Big Data Fallacy: Large datasets (e.g., social media scrapes) inherit platform-specific biases (e.g., age, political leaning).
Human-Computer Interaction (HCI) & User Experience (UX):
- Biased participant recruitment → Non-inclusive designs.
- User Studies: Recruiting via tech forums → Excludes elderly/digitally inexperienced users.
- Log Data Analysis: Analyzing only data from active users ignores the needs and pain points of users who churned (Survivorship Bias).
Practice Problems on Sampling Bias: Solved
Problem 1: A university wants to survey students about campus facilities. They decide to survey students only in the library. What type of sampling bias might this introduce?
Solution:
This might introduce selection bias because students in the library may not represent the views of all students on campus.
Problem 2: A company wants to understand the job satisfaction of its employees, but only surveys employees who have been with the company for more than 5 years. What type of bias is this?
Solution:
This introduces survivorship bias, as it ignores the opinions of newer employees who may have different perspectives.
Problem 3: An online retailer sends out a customer satisfaction survey via email, but only 10% of recipients respond. What type of bias could this lead to?
Solution:
This could lead to non-response bias, as the opinions of the 90% who did not respond are not considered.
Problem 4: In a survey about a new product, only the first 100 customers who bought the product are surveyed. What type of sampling bias might this cause?
Solution:
This might cause selection bias, as the first 100 customers might have different views than those who purchase the product later.
Problem 5: A political poll is conducted by calling landline phones. What type of bias might this introduce?
Solution:
This might introduce undercoverage bias, as many younger people or those in urban areas might only have cell phones.
Practice Problems on Sampling Bias: Unsolved
1. A school surveys students about their favorite subjects by asking those in advanced placement classes.
2. A health study only includes participants who regularly visit a gym.
3. An online poll about internet usage is conducted on a tech news website.
4. A car manufacturer surveys customers who have purchased their most expensive model.
5. A retail store surveys customers who purchased on Black Friday.
6. A study on dietary habits surveys only those who visit a health food store.
7. A survey on employee satisfaction is conducted by interviewing only those who received a promotion in the last year.
8. A survey on work-from-home experiences is conducted by asking employees who volunteered to work remotely.
9. A survey on commuting times is conducted by asking employees who arrive early at the office.
10. A survey on public transportation is conducted by asking passengers at a train station during peak hours.