Sampling Techniques for Machine Learning

Hello everyone 👋

In our previous blogs, we explored the fundamentals of Inferential Statistics and the core concepts of Probability and Probability Distributions — understanding how uncertainty is measured and how data behaves.

Now, before diving deeper into data-driven modeling, let’s explore something equally important — Sampling Techniques.

Sampling is the bridge between the massive population of data and the manageable subset we actually analyze.

What is Sampling?

Sampling is the process of selecting a subset (sample) from a larger group (population) to draw conclusions about the entire group.

In simple terms — instead of studying everyone, we study some, but in a representative way.

Example:
If a company wants to know customer satisfaction levels, it’s not practical to ask every customer.
Instead, they can select a sample of 1,000 customers — ensuring that this sample accurately represents the whole customer base.

Why Sampling is Important in Machine Learning

Sampling is not just a statistical concept — it’s an everyday reality in Machine Learning.

Here’s why:-

Efficiency: Working with smaller, manageable datasets saves time and computation.

Cost-effectiveness: Collecting and processing entire populations is often expensive.

Model performance: Good sampling helps ML models generalize better to unseen data.

Fairness and balance: Sampling ensures class balance in imbalanced datasets (e.g., fraud detection).

In short — sampling helps us create a mini version of reality that our models can learn from.

Key Terms to Remember

Population: The complete set of all possible observations.

Sample: A subset of the population selected for analysis.

Sampling Frame: The list from which the sample is drawn.

Sampling Bias: When the sample does not represent the population correctly.

Types of Sampling Techniques

Sampling methods can be broadly divided into two categories:

1. Probability Sampling
2. Non-Probability Sampling

Let’s understand both :

1. Probability Sampling

In probability sampling, every member of the population has a known (non-zero) chance of being selected.

This type of sampling ensures unbiased representation and is widely used in statistical research and ML model validation.

a. Simple Random Sampling

Every individual has an equal chance of being selected.

Example:
Selecting 100 customers randomly from a list of 10,000 using random number generation.

import numpy as np

population = np.arange(1, 10001)
sample = np.random.choice(population, 100, replace=False)
print("Random Sample:", sample[:10])

Advantages: Unbiased and easy to implement.
Disadvantage: May not represent subgroups well.

b. Systematic Sampling

Selecting every kth element from a list after a random start.

Example:
If you have 1,000 employees and you select every 10th employee, your sample size = 100.

import numpy as np

population = np.arange(1, 1001)
k = 10
start = np.random.randint(0, k)
sample = population[start::k]
print("Systematic Sample:", sample[:10])

Advantages: Simple and ensures uniform coverage.
Disadvantage: Can introduce bias if there’s a hidden pattern in the population.

c. Stratified Sampling

Population is divided into groups (strata) based on shared characteristics, and samples are drawn from each stratum proportionally.

Example:
If your population has 60% males and 40% females, your sample should follow the same ratio.

import numpy as np

males = np.random.choice(range(1000), 60, replace=False)
females = np.random.choice(range(1000, 2000), 40, replace=False)
sample = np.concatenate([males, females])
print("Stratified Sample Size:", len(sample))

Advantages: Ensures representation of all subgroups.
Disadvantage: Requires prior knowledge of population structure.

d. Cluster Sampling

The population is divided into clusters (usually based on location or natural grouping), and a few clusters are selected randomly for study.

Get Learner’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example:
To survey students in a country, you might randomly select 10 schools (clusters) and then survey all students within those schools.

Advantages: Cost-effective and practical.
Disadvantage: Can be less accurate if clusters are not homogeneous.

e. Multi-Stage Sampling

A combination of several sampling techniques applied in stages.

Example:
Select 5 cities → from each city, choose 3 schools → from each school, choose 30 students.

Advantages: Very flexible and efficient for large populations.
Disadvantage: Slightly complex to design.

2. Non-Probability Sampling

In non-probability sampling, not every member of the population has a known or equal chance of being selected.

These methods are quicker and cheaper but may introduce bias.

a. Convenience Sampling

Choosing samples that are easiest to access.

Example:
Surveying friends or coworkers because they’re readily available.

Advantages: Easy and inexpensive.
Disadvantage: High risk of bias and low generalizability.

b. Judgmental (Purposive) Sampling

Researcher selects individuals based on their knowledge or judgment.

Example:
Choosing only “expert” users to test a new AI tool.

Advantages: Useful when expertise matters.
Disadvantage: Depends heavily on researcher’s subjectivity.

c. Quota Sampling

Similar to stratified sampling, but selection within each group is non-random.

Example:
Choosing 50 males and 50 females but selecting whoever is available first.

Advantages: Maintains proportionality.
Disadvantage: Still introduces bias due to non-random selection.

d. Snowball Sampling

Used when subjects are hard to find; existing participants recruit others.

Example:
Used in research involving rare diseases or niche online communities.

Advantages: Helps reach hidden populations.
Disadvantage: May not represent the broader population.

Comparison Table

Type Selection Basis Bias Risk Use Case

Simple Random Random Low Balanced datasets
Systematic Interval-based Medium Ordered data
Stratified Based on subgroups Low Demographic data
Cluster Based on natural groups Medium Large geographic areas
Convenience Based on availability High Pilot studies
Snowball Based on referrals High Hidden or rare groups

---

Sampling in Machine Learning Practice

In ML, sampling is used at multiple stages:

Data Splitting: Dividing dataset into training, validation, and test sets.

Cross-Validation: Repeatedly sampling subsets for model evaluation.

Handling Imbalance: Oversampling minority class or undersampling majority class.

Bootstrapping: Sampling with replacement to estimate model accuracy (used in Random Forests).

Example: Using Stratified Sampling to split training and testing data proportionally in Python:

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)

print("Training data shape:", X_train.shape)
print("Testing data shape:", X_test.shape)

In Short

Sampling helps us study a representative part of a population efficiently.

Probability Sampling = unbiased, reliable results.

Non-Probability Sampling = quick but potentially biased.

In Machine Learning, correct sampling ensures fair, balanced, and generalizable models.

💭 Final Thoughts

Sampling might look like a simple data collection step — but it’s much more than that.

> A poor sample leads to poor insights, and poor insights lead to poor decisions.

In Machine Learning, where data is everything, choosing the right sampling technique determines whether your model learns reality — or just noise.

In our next blog, we’ll dive into Hypothesis Testing and Statistical Inference, where we’ll learn how to validate assumptions and measure the significance of our results.

Until then, remember — a small but well-chosen sample can reveal the truth of the whole population.