Data Leakage: When the ML Model Knows Too Much

Why impressive machine-learning scores can measure hidden access to answers and how to rebuild an evaluation you can trust.

At 9:05 on Monday morning, a package receives its first carrier-acceptance scan. In an offline evaluation meant to recreate that moment, a model predicts that the package will miss its promised arrival time.

If that prediction could be reproduced live, the timing would be useful. An operations coordinator might review the route, escalate the shipment, or prepare an early customer notification. The model also looks extraordinary. Its test results are almost perfect. The team then examines the variables behind the prediction.

Near the top are actual transit time and delay compensation issued. Neither value exists at the acceptance scan. In our synthetic generator, both are created after the cutoff and partly from the final delivery outcome. The model has not developed remarkable foresight. It has received tracking updates from the future. Nothing is necessarily wrong with its mathematics. The code runs. The synthetic labels are internally consistent. The score is reproducible. The model learned the relationship it was given.

The problem is that the development table answered one question:

Given everything eventually recorded about a package, can we identify whether it was late?

The operations team needs an answer to a different question:

Using only information available at the acceptance scan, which packages are likely to arrive late while there is still time to respond?

Leakage occurs when that mismatch lets unavailable information cross the evaluation boundary.

Freeze the clock before choosing features

Before selecting an algorithm, write a prediction contract:

Using information available at [prediction time], predict [outcome over a stated horizon] for [unit], so that [user] can consider [action].

For our hypothetical example:

Using information recorded by the carrier-acceptance scan, estimate whether a shipment will miss its promised delivery time, so an operations team can prioritize shipments for review. This is not a claim that the score reveals why a package may be delayed, or that intervention will prevent the delay. It simply defines the predictive task.

Now freeze the tracking record at the acceptance scan.

Figure 1. A field’s validity depends on whether it exists at the prediction cutoff. Prior completed-shipment history and acceptance-time conditions may be available; actual transit time and final delivery records are not. Author-generated diagram.

A feature is not automatically leaky because it is strongly predictive. It becomes leaky when the model cannot legitimately know it for the stated prediction. Actual transit time is forbidden at the acceptance scan. It may be perfectly valid for a later model that estimates compensation after delivery. Timing, not the column name, determines whether the information belongs.

What data leakage actually means

Scikit-learn defines leakage as using information during model building that would not be available at prediction time, producing overly optimistic estimates that can deteriorate on genuinely new data (common pitfalls). That definition is broader than accidentally placing the target column inside X. Leakage can happen when:

• A feature contains information created after the outcome.
• Future observations influence past predictions.
• Records from the same person, machine, or document family appear on both sides of a split when the evaluation claims generalization to unseen entities.
• A scaler, imputer, feature selector, or sampler learns from the evaluation data.
• The team repeatedly changes the model after examining the final test score.

Kaufman, Rosset, and Perlich described leakage as introducing information about the target that should not legitimately be available and argued for a clear separation between learning and prediction (KDD paper).

These are diagnostic categories, not a universal formal taxonomy. A project can violate more than one boundary at once.

Information leakage: the answer wearing a disguise

Target leakage occurs when an input directly reveals the target or when its value or availability is determined by the outcome. Future information can also invalidate a forecast because it was unavailable at prediction time, even when the outcome did not create it; that is temporal leakage. Train–test contamination and repeated adaptation to a test set can invalidate an evaluation even when no feature directly contains the target.

The obvious mistake is using late_delivery as an input while predicting late_delivery.

Real leakage is often less direct:

• Actual transit time recorded after delivery
• A final delivery-status code
• The total number of delivery attempts
• A refund or compensation amount caused by the delay
• A customer complaint filed after the promised arrival time
• An exception-resolution code entered after the journey ends
• A missing-value pattern caused by a post-outcome administrative workflow

The model may never see the target itself. It sees the target’s footprints.

The same pattern appears across domains. A hospital model may use a discharge code to predict deterioration at admission. A fraud model may use a chargeback filed later to classify a transaction at authorization time. A factory model may use a repair order to predict a failure before inspection. In each hypothetical example, the field may be accurate — it simply belongs to a later moment.

This is why deleting a column named outcome is not an adequate leakage audit. You must understand how every feature was created, when it became available, and whether the outcome influenced its existence or value.

Temporal leakage: tomorrow quietly enters yesterday

Time leakage can also arise without an obviously post-outcome column. Suppose the model uses a route’s seven-day average delay rate. A centered window for Tuesday might include deliveries completed on Wednesday, Thursday, and Friday. The name sounds historical, but tomorrow has traveled backward across the prediction boundary. Other examples include:

• Calculating normalization statistics using future delivery periods
• Joining the newest address record instead of the version available at acceptance
• Training on later months and testing on earlier months
• In a historical backtest, training on labels whose outcome windows had not yet matured by the simulated model-training date
• Allowing overlapping windows to share many of the same underlying events

For time-dependent data, shuffled cross-validation may create unrealistic similarity between training and testing. Scikit-learn recommends time-aware evaluation when observations are generated by a temporal process and warns against training on future observations while evaluating the past (cross-validation guidance, TimeSeriesSplit).

A chronological split is not automatically sufficient. If input windows overlap, outcomes arrive late, or records are corrected after the fact, the evaluation may need a gap between training and testing. An immature outcome is often a right-censoring or label-quality problem rather than feature leakage; it becomes a temporal boundary failure when future outcome information is used inconsistently across training and evaluation.

Independence leakage: a new shipment may be an old merchant

Even if every feature obeys the clock, the evaluation can still recognize the sender. Our synthetic case contains four shipments for each fictional merchant. If we split rows randomly, one merchant can appear in training through three shipments and in testing through the fourth. The shipment is technically held out. The merchant is not.

That may be acceptable if the deployed system will predict later shipments for the same known merchants and the split preserves time correctly. But it does not support a claim that the model generalizes to merchants it has never encountered.

The split must imitate the intended future:

• New shipments from known merchants: preserve time and test future records appropriately.
• Entirely unseen merchants: keep every shipment for one merchant in a single partition.
• Later delivery periods containing known and new merchants: use a design that respects both time and identity.

Scikit-learn’s grouped cross-validation guidance explains that when several dependent samples come from one subject or entity, that entity should not appear in both training and validation if the goal is generalization to unseen groups (grouped evaluation).

The same issue appears elsewhere:

• Several visits from one patient
• Multiple sensor windows from one machine
• Crops or augmentations from one source image
• Reviews written by the same customer
• Near-duplicate documents
• Records copied across source systems

Rows can be different while their information is almost the same.

Adaptation leakage: fit means more than training the model

Most beginners understand that the test set should not be passed to model.fit().

But many other operations learn from data:

• Imputation learns replacement values.
• Scaling learns means, variances, minima, or maxima.
• PCA learns directions of variation.
• Feature selection learns which columns look useful.
• Target encoding learns relationships between categories and labels.
• Probability calibration learns a mapping from raw scores to probabilities.
• Threshold selection learns which cutoff produces a desired trade-off.

If these operations inspect the test set, the test has participated in training — even if the final estimator never directly receives those rows.

Scikit-learn demonstrates this with random features and random labels: performing supervised feature selection on the full dataset produces an inflated score, while splitting first and fitting feature selection only on training returns performance toward chance. Its recommendation is to place learned transformations inside a Pipeline.

Here are condensed excerpts, assuming X and y are already loaded. The unsafe pattern is:

# Unsafe: feature selection sees every label, including test labels
selected_X = SelectKBest(k=25).fit_transform(X, y)
X_train, X_test, y_train, y_test = train_test_split(
    selected_X, y, test_size=0.25, random_state=42
)
This version is safer for preprocessing:
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

# Split before anything learns from the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

model = make_pipeline(
    SelectKBest(score_func=f_classif, k=25),
    LogisticRegression(max_iter=2_000)
)
model.fit(X_train, y_train)

Inside cross-validation, the complete pipeline must be fitted separately inside every training fold. This still requires a group- or time-aware split when rows are dependent.

Resampling must also remain inside training

Oversampling and undersampling are not inherently leaky. Their placement determines whether the evaluation remains honest.

Get Rayan Yassminh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If SMOTE or another sampler is applied before splitting, synthetic or selected observations can carry information across the boundary. The evaluation distribution may also stop resembling the natural deployment population.

The imbalanced-learn documentation demonstrates both problems and recommends resampling only within training partitions (common pitfalls).

The correct pattern is:

1. Create the training and validation fold.
2. Fit the sampler using only the training fold.
3. Train the model on that resampled training fold.
4. Evaluate on the untouched validation fold.

Case study: watch an impossible score appear

We created a reproducible dataset containing 8,000 completely synthetic shipments: four shipments from each of 2,000 fictional merchants. The target is late_delivery, an authored binary label indicating whether a shipment arrives after its promised time. It is not a real carrier’s service definition or an estimate of any delivery network. The stated prediction moment is the carrier-acceptance scan. Allowed a cceptance-time inputs include:

• A synthetic merchant prior on-time rate, treated as available before the current acceptance scan
• Origin-hub load at acceptance
• Route distance
• Package weight
• Promised transit days
• Service level
• Weekend-pickup indicator

The deliberately forbidden inputs are actual transit hours and delay compensation. Both are generated after — and partly from — the synthetic outcome.

For the main comparison, GroupShuffleSplit places 1,500 merchants in training and 500 entirely different merchants in testing. The test set contains 2,000 shipments and has a 47.4% late-delivery rate.

We compare a prior-only baseline, a leakage-safe logistic-regression pipeline, and the same pipeline with the two post-outcome fields added.

Accuracy counts correct decisions at the chosen threshold. ROC AUC measures how well a model ranks positive cases above negative ones across thresholds; 0.5 is chance-like ranking. Log loss evaluates predicted probabilities and strongly penalizes confident mistakes; lower is better.

Table 1. Measured results on 2,000 synthetic test shipments representing 500 merchants absent from training. These values demonstrate evaluation mechanics and provide no evidence about real packages, merchants, carriers, or delivery networks.

Figure 2. The same unseen-merchant test set tells three stories. The acceptance-time model improves on the prior baseline. The deliberately leaky model appears to have impossible foresight because its inputs are generated after delivery. Synthetic data; n = 2,000 test shipments.

The perfect score was deliberately constructed. It is not an estimate of how much leakage usually helps. It makes one point visible: an offline model can look flawless precisely because it was given information that the real prediction system could never possess. Removing those fields lowers the score. That is not the corrected model becoming worse. It is the evaluation finally answering the declared question.

Two quieter leaks

The obvious future feature is valuable for teaching, but real leakage is often less theatrical. The row split that nearly repeats every merchant

In a random row split, the test set contained 1,355 unique merchants. Of those, 1,348 (99.48%) also appeared in the training set through another shipment.

We held the model settings constant and ran a two-by-two comparison. Under the random-row split, adding merchant ID increased ROC AUC from 0.632 to 0.691, while log loss changed only slightly from 0.663 to 0.666. Under the group split, where all 500 test merchants were unseen, adding ID did not meaningfully help: ROC AUC remained at 0.628, while log loss changed from 0.667 to 0.671.

This result does not prove that merchant ID is always forbidden or that group splitting is always superior. It shows that the random-row experiment cannot support the stated claim about unseen merchants.

That detail matters. A model can improve its ranking metric while producing worse probability estimates. Leakage and split mismatch do not have to improve every metric, and a cross-split difference should not be interpreted as the isolated causal effect of one feature.

The feature selector that examined the test labels

In another experiment, we added 600 pure-noise variables so the effect of supervised selection would be visible.

When SelectKBest was fitted only on training data inside the pipeline, test ROC AUC was 0.609. When it selected features using all 8,000 labels before the split, ROC AUC increased to 0.635.

The increase is modest compared with the perfect post-outcome model, which is exactly why this error can be harder to notice. The code looks organized. The test rows never enter LogisticRegression.fit(). Yet their labels already influenced which features the model received.

Figure 3. Leakage can hide in evaluation design. The controlled merchant-ID comparison shows an apparent ranking benefit only when almost every test merchant also appears in training. Whole-data supervised feature selection creates a second, smaller source of optimism. Synthetic data; n = 2,000 test shipments per comparison.

These are authored synthetic relationships, not estimates of real leakage effects. They demonstrate mechanisms separately so one mistake is not used to explain another.

A pipeline helps but it cannot understand time

A pipeline is one of the best defenses against procedural leakage. It keeps imputation, scaling, feature selection, and model fitting together so cross-validation can refit every learned step using only the appropriate training fold.

But a pipeline cannot know that actual_transit_hours arrives after the acceptance scan. It cannot decide whether two shipments from the same merchant should cross folds. It cannot identify near-duplicate images. It cannot know that the final test score has quietly guided twelve rounds of feature engineering.

Pipelines protect statistical boundaries. They do not understand organizational time or deployment intent.

That judgment still belongs to the people defining the problem.

The test set can leak without entering the training table

The records can also cross the boundary without moving at all — the test set only needs to influence what the pipeline learns. Imagine evaluating a model on the final test set, disliking the result, adding features, and evaluating again. Then changing the model family. Then the seed. Then the threshold. Then removing a subgroup that looks difficult.

The test data never appears in the training table. But its results repeatedly influence development. The team is adapting to the test set.

Dwork and colleagues formalized the broader problem of adaptive holdout reuse: when later analyses depend on earlier holdout results, standard assumptions about statistical validity can fail (The Reusable Holdout). This does not mean a test set becomes worthless after one human glance. The issue is influence. If test performance changes what you build, the test is functioning as development data.

A cleaner workflow is:

• Use training data for fitting.
• Use validation data or inner cross-validation for model and threshold choices.
• Freeze the complete pipeline.
• Evaluate the final test set for the release or publication claim.
• Obtain new evaluation data when the old test has materially shaped development.

A practical leakage audit

Before trusting a model score, reconstruct the room in which the deployed prediction will be made.

1. Write the prediction contract: State the unit, cutoff, target, horizon, user, action, and permitted information.

2. Build a feature-availability table: For every feature, record:

• When the underlying event occurs
• When the value is calculated
• When it becomes available to the production system
• Whether the outcome can change the value or its missingness

The availability timestamp matters more than the event timestamp. A lab measurement may occur today but reach the model tomorrow.

3. Define what “new” means: Will deployment encounter:

• New independent rows?
• Later periods?
• Familiar entities with new events?
• Entirely unseen entities?

Choose random, temporal, grouped, or combined evaluation accordingly.

4. Trace everything that learns: Search the code for fit, fit_transform, aggregation, vocabulary building, feature selection, resampling, calibration, and threshold optimization. Each learned operation must remain inside the appropriate development boundary. Preprocessing and feature selection must be refitted within training folds; calibration, tuning, and threshold selection may use properly separated validation data. The final test must influence none of them.

5. Search for relatives, not only duplicates: Check entity IDs, source files, event windows, hashes, timestamps, and upstream joins. An augmented image or adjacent sensor window may contain nearly the same information without being byte-for-byte identical.

6. Protect the final test: Record which results the team has seen and which choices they influenced. A test set is not a permanent public scoreboard.

7. Compare the leaky and corrected systems: The difference can reveal where optimism entered. But do not treat the gap as a universal correction factor; it belongs to that dataset and design.

Not every suspicious feature is forbidden

• A closed repair order is leakage for a model predicting failure before inspection. It may be valid for a later model predicting repair duration.
• The same merchant appearing in training and testing is inappropriate for an unseen-merchant claim. It may be legitimate when the system is designed to predict a later shipment for a known merchant, provided the split preserves time and no future fields cross backward.
• A very high score is a reason to investigate, not proof of leakage. A modest score is not proof that the evaluation is clean.
• Leakage is not determined by whether a feature looks suspicious. It is determined by whether the complete experiment matches the decision it claims to support.

The honest score may be lower and more valuable

Return to the package at its first carrier scan. The rebuilt model no longer sees actual transit time, compensation issued, or final delivery records. Its transformations learn only from training data. If the claim concerns unseen merchants, every merchant remains on one side of the evaluation boundary. The final test stops acting as a development dashboard.

The corrected evaluation metrics may be less exciting. But for the first time, it could have been produced while the package was still at the beginning of its journey. That is not weaker machine learning. It is stronger evidence.

Data leakage makes models appear more capable by granting them access to answers, consequences, related cases, or evaluation feedback. Removing that access does not damage the problem. It restores it.

Before trusting the next remarkable score, ask:

Could the deployed system have known every fact, learned every transformation without touching held-out evidence, and faced cases that were new in the way deployment requires?

If the answer is unclear, the project may not need another algorithm yet. It needs a clock, a boundary, and an honest test.