Data Leakage in Machine Learning For Beginners

Imagine a student who gets the exam paper one night before the test. He memorizes every answer, walks in, and scores 98%. Everyone is impressed. But did he actually learn anything? Not really. Put him in front of a new question and he is completely stuck.

Machine learning models can fall into the exact same trap without anyone noticing. This is called data leakage, and it is one of the most common (and sneakiest) mistakes beginners make.

But what is it, why is it so dangerous, and how does it happen? Let’s break it down in the simplest way possible.

What is Data Leakage?

Data leakage happens when your model gets to see information during training that it wouldn’t have access to in the real world. Because of this, your test results look amazing, but the model falls apart the moment it faces real, unseen data.

Think of it as the exam paper being “leaked” to the model before test day.

The dangerous part is that nothing crashes and there is no error message. Your code runs smoothly, your loss drops nicely, and your accuracy looks incredible. You feel great until the model is actually used in production and fails completely.

The 4 Common Types of Data Leakage

Data leakage usually sneaks in through one of these four doors:

1. Train-Test Contamination (Preprocessing Leakage)

This happens when information from your test set quietly sneaks into your training pipeline.

The classic example is preprocessing. Suppose you have missing values or want to scale your numbers using StandardScaler. If you run the scaler or calculate the average across your entire dataset before splitting it, the scaler has already peeked at the test data to calculate the overall mean and spread.

Because of that peek, your test set is no longer truly “unseen.”

2. Target Leakage (The “Cheating” Feature)

This happens when one of your input columns secretly contains the answer or includes data generated after the event happened.

Suppose you want to build a model to predict whether a customer will cancel their subscription. One of your dataset columns is refund_processed_date.

• Active customers have an empty value.
• Canceled customers have a timestamp.

Your model quickly discovers: “If there is a date here, the customer canceled.” It achieves 99.8% accuracy. But in the real world, you want to identify at-risk customers before they leave. At prediction time, that refund column doesn’t exist yet!

3. Time-Travel Leakage (Temporal Leakage)

If your data involves dates or time (like stock prices, daily retail sales, or website traffic), a standard random split will ruin your model.

With a random split, your model might train on data from Friday, test on data from Wednesday, and then try to predict Thursday. You are essentially giving the model a time machine to look into the future to guess the past.

For time-dependent data, you must always split chronologically: train on the past, test on the future.

4. Group Overlap Leakage

Imagine building an app to detect skin conditions from photos. Patient A has 6 photos taken from slightly different angles. If 4 of those photos end up in your training set and 2 in your test set, the model doesn’t learn the skin disease it just recognizes Patient A’s skin tone, background, or lighting.

Get Ashwin Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Whenever multiple rows belong to the same person, device, or store, all data from that specific group must stay strictly together in either the train set or the test set, never split between both.

Example: The Wrong Way vs. The Right Way

Here is how leakage looks in code using scikit-learn.

The Wrong Way:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Scaling BEFORE splitting (Leakage!)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)   # Scaler has now seen test distribution

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

The Right Way:

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# 1. Split FIRST
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 2. Learn ONLY from training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Calculate mean/std on train only
X_test_scaled = scaler.transform(X_test)        # Apply those exact train values to test

The golden rule to remember: fit on training data only, transform on test data. The exact same logic applies to SimpleImputer, CountVectorizer, and TfidfVectorizer.

How to Avoid It the Easy Way: Pipelines

Remembering to manually separate fit and transform across every preprocessing step is tedious and easy to mess up especially when doing cross-validation.

Luckily, scikit-learn has Pipeline, which handles this isolation under the hood:

Python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Bundle preprocessing and model together
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression())
])
# The pipeline guarantees the scaler only learns from X_train
pipe.fit(X_train, y_train)
# During testing, it only transforms X_test without fitting
print("Test accuracy:", pipe.score(X_test, y_test))

By wrapping your steps inside a Pipeline, you create a firewall that stops accidental leakage from happening.

How Do You Spot Data Leakage?

Because data leakage doesn’t throw red error codes, watch out for these telltale symptoms:

• Accuracy that feels too good to be true: If a simple baseline model hits 99% accuracy on a messy real-world problem, be suspicious, not celebratory.
• One feature dominates feature importance: If one column accounts for 85% of your model’s predictive power, check whether that column was created during or after the target event.
• The “Lab vs. Reality” drop: Your validation score is near-perfect, but performance crashes when you feed it live, raw inputs.
• Differences in train vs. test feature distributions: If the model struggles unusually hard on the test set right after minor preprocessing changes.

Quick Checklist Before You Train

Run through these four questions before every experiment:

• Did I split my raw data before applying scalers, encoders, or imputers?
• Am I only calling .fit() on my training split?
• Does any feature rely on information that wouldn’t exist at the exact moment of prediction?
• If working with time-series or repeated patient/user IDs, did I use a temporal split (TimeSeriesSplit) or grouped split (GroupKFold)?

Almost every machine learning practitioner gets burned by data leakage early in their journey it is practically a rite of passage. Don’t worry if you’ve already made this mistake; catching it before production is what matters.

Keep practicing, stay curious, and lean on Pipelines to do the heavy lifting for you!

Happy Coding ❤️