<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mercy Mwangi on Medium]]></title>
        <description><![CDATA[Stories by Mercy Mwangi on Medium]]></description>
        <link>https://medium.com/@mwangi25.mercy?source=rss-ed5479ff5fa4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mEsAh4swfBAIfzk9ZKh8YA.jpeg</url>
            <title>Stories by Mercy Mwangi on Medium</title>
            <link>https://medium.com/@mwangi25.mercy?source=rss-ed5479ff5fa4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:03:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mwangi25.mercy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data leakage: The many ways you accidentally cheat in Machine Learning]]></title>
            <link>https://medium.com/shift-enter/data-leakage-the-many-ways-you-accidentally-cheat-in-machine-learning-b803e4375e5d?source=rss-ed5479ff5fa4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b803e4375e5d</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai-data-leakage]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning-python]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mercy Mwangi]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 08:34:29 GMT</pubDate>
            <atom:updated>2026-08-17T08:36:24.832Z</atom:updated>
            <content:encoded><![CDATA[<p>A machine learning script that throws an exception is easy to fix, but a script that runs perfectly while subtly corrupting your validation metrics is a quiet disaster. In my time as a technical mentor, I have seen time and again students proudly present scripts with 99% model accuracy only for the model to completely collapse when it touches real-world data. The code ran without throwing a single error, the syntax was clean and yet the pipeline was fundamentally broken. The culprit is almost always data leakage- in this blog I go into the common ways it creeps into your machine learning workflow and accidentally inflates model performance during training.</p><p>To understand why data leakage is so destructive, we have to look at why we use holdout sets and cross-validation in the first place: to get an unbiased estimate of real-world performance. A model only has value if it generalizes well to data it hasn’t seen yet. We want a model that learns patterns, not one that simply memorizes the training set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYSrcT8JxHYfa9_-cR8Ffw.png" /><figcaption>AI-generated image by author</figcaption></figure><p>The cruel part about data leakage is how quietly it sneaks in. It’s subtle enough to go completely unnoticed, right up until you tell your team you’ve built a model with 99% accuracy. What makes this worse is that many of the current tutorials only talk about leakage in theory, and very few resources explain the actual mechanics of why (and how) everyday code patterns introduce it.</p><p>So this article is my solid attempt to guide data nerds on how to diagnose data leakage, explaining the reason it happens and in part 2, how to quickly avoid it all together.</p><h4>1. Feature Scaling: Why `fit_transform()` on your entire dataset leaks data</h4><p>If you’ve been scaling your input features like shown below, sorry, but this is the clearest sign of data leakage.</p><pre>from sklearn.preprocessing import StandardScaler<br>from sklearn.model_selection import train_test_split<br><br>#DON&#39;T DO THIS: Scaler computes mean and std from ALL data<br>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X)<br><br>X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2, random_state=42)</pre><p><strong>Why this breaks your pipeline:</strong></p><p>Remember, the rule is simple- the model and transformers must ‘learn’ <strong>ONLY</strong> from the training dataset.</p><p>When you call scaler.fit_transform(X) before splitting, StandardScaler calculates the mean (<strong>𝜇</strong>) and standard deviation (𝛔) across your entire dataset, including the rows that will end up in your test set. That test set distribution subtly shifts the scale of your training inputs. Even though your model never sees the test labels, it trains on features that were computed using statistics from unseen test rows.</p><p>This is what you should do instead:</p><pre>from sklearn.preprocessing import StandardScaler<br>from sklearn.model_selection import train_test_split<br><br># DO THIS: Split raw features first!<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br>scaler = StandardScaler()<br><br># Learn parameters (mean, std) ONLY from the training set, then transform it<br>X_train_scaled = scaler.fit_transform(X_train)  <br><br># Apply the ALREADY LEARNED parameters to the test set (do NOT call fit here)<br>X_test_scaled = scaler.transform(X_test)</pre><blockquote><strong>The Golden Rule of Transformers:</strong></blockquote><blockquote>.fit_transform() learns parameters and applies them to the training set. .transform() <em>only</em> applies previously learned parameters to new data. If you ever call .fit() or .fit_transform() on test or validation data, information has leaked.</blockquote><h4>2. Feature Encoding: The problem with target encoding</h4><p>Here, the risk of data leakage is a little nuanced, because the answer is not one size fits all.</p><p>For standard techniques like <strong>One-Hot Encoding</strong> or <strong>Label Encoding</strong>, the risk of data leakage is generally low. These transformers don’t calculate statistical properties across rows, they simply map existing category labels to numbers or expanded binary columns.</p><p>The real danger arises with <strong>Target Encoding</strong> (Mean Encoding), a popular technique where categorical values are replaced by the mean of the target variable for each category.</p><pre># DON&#39;T DO THIS: Target encoding computed on full dataset target (y)<br>from category_encoders import TargetEncoder<br><br>encoder = TargetEncoder()<br>X_encoded = encoder.fit_transform(X,y) # Computes mean target using ALL rows!</pre><p>Target encoding replaces a category label (e.g., city = &quot;Nairobi&quot;) with the average target value across all rows where that category appears. If you fit a target encoder on your entire dataset before splitting, the encoded feature for a training sample will directly incorporate target values from the test set. Your model is getting a direct peek at the actual target outcomes of unseen test data.</p><p>The simple fix is to always perform target encoding after splitting your train and test set.</p><pre># DO THIS: Split your raw data first, then encode<br>from category_encoders import TargetEncoder<br>from sklearn.model_selection import train_test_split<br><br># Split train and test sets<br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.2, random_state=42<br>)<br><br># Fit target encoder ONLY on training features and training labels<br>encoder = TargetEncoder()<br>X_train_encoded = encoder.fit_transform(X_train, y_train)<br><br># Apply learned category target means to test set (never pass y_test!)<br>X_test_encoded = encoder.transform(X_test)</pre><blockquote><strong>Key Rule for Target Encoding:</strong> Notice that on the test set, you only call .transform(X_test). You never pass y_test to the encoder during evaluation—the encoder must strictly use the category-to-target mappings it learned from X_train and y_train.</blockquote><h4>3. Resampling: Applying SMOTE before splitting</h4><p>Class imbalance is extremely common in real-world machine learning (e.g., fraud detection, churn prediction). A frequent instinct is to apply <strong>SMOTE</strong> (Synthetic Minority Over-sampling Technique) to balance the target distribution right away.</p><p>If you run SMOTE before train_test_split, you are creating a massive data leak:</p><pre>from imblearn.over_sampling import SMOTE<br>from sklearn.model_selection import train_test_split<br><br># DON&#39;T DO THIS: SMOTE applied across the entire dataset<br>sm = SMOTE(random_state=42)<br>X_resampled, y_resampled = sm.fit_resample(X, y)<br><br>X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2)</pre><p><strong>Why this breaks your pipeline:</strong></p><p>SMOTE creates new synthetic data points by calculating the <em>k-nearest neighbors</em> of real minority samples and interpolating between them.</p><p>When you apply SMOTE to the full dataset, a minority sample that belongs in your test set will be used to generate synthetic points for the training set. When you subsequently perform your train_test_split, two things happen:</p><ol><li>Synthetic points in your training set contain direct structural information from original samples sitting in your test set.</li><li>Near-identical synthetic duplicates of test points end up in your training set, allowing the model to simply memorize them.</li></ol><pre>from imblearn.over_sampling import SMOTE<br>from sklearn.model_selection import train_test_split<br><br># DO THIS: Split first! Keep your test set raw and imbalanced.<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)<br><br>sm = SMOTE(random_state=42)<br># Resample ONLY the training set<br>X_train_resampled, y_train_resampled = sm.fit_resample(X_train, y_train)</pre><h4>4. Cross- validation: The most common hidden mistake</h4><p>This is one of the most frequent mistakes I see in student notebooks. Because Cross-Validation feels like an automated evaluation step, it’s easy to forget that every single validation fold must be treated as completely unseen data.</p><p>Suppose you apply a preprocessing step , like feature scaling, across your entire training dataset <em>before</em> passing it into cross_val_score:</p><pre>from sklearn.preprocessing import StandardScaler<br>from sklearn.model_selection import cross_val_score, KFold<br><br># DON&#39;T DO THIS: Preprocessing applied to the entire dataset before CV<br>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X_train)  # Scaled across ALL 5 folds at once!<br><br>cross_val_score(model, X_scaled, y_train, cv=5)</pre><p><strong>Why this breaks your pipeline:</strong></p><p>During 5-fold cross-validation, your data is split into 5 chunks. In the first iteration, Folds 1–4 serve as the training set, and Fold 5 acts as the validation set.</p><p>However, because fit_transform() was called on X_train beforehand, the mean (<strong>𝜇</strong>) and standard deviation (𝛔) used to scale Folds 1–4 were calculated using values from Fold 5 as well. Fold 5 is no longer &quot;unseen&quot; validation data, its distribution has already infected the features used for model training in every single fold iteration.</p><p>Your cross-validation score is now artificially high, giving you false confidence in a model that will drop in accuracy the moment it hits production.<strong> The fix </strong>is simple: any preprocessing transformer that learns parameters from your data (scalers, encoders, imputers, or resamplers) must be re-fitted <strong>inside each fold</strong> strictly using that fold’s specific training split.</p><p><strong>So How Do We Fix Data Leakage altogether?</strong></p><p>Manually writing custom loops to isolate transformers, resamplers, and target encoders across cross-validation folds is messy and prone to human error. One misplaced .fit() call and the leak returns.</p><p>But there is a neat and easy fix, provided in scikit-learn, in <strong>Part 2 of this series</strong>, I’ll cover the production fix: using imblearn pipelines to automate preprocessing isolation across every fold with zero data leakage.</p><p>If this hit home, and want to continue learning about why things work the way they work in Data , AI and Machine Learning. Welcome to Shift + Enter. Follow along and enjoy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b803e4375e5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shift-enter/data-leakage-the-many-ways-you-accidentally-cheat-in-machine-learning-b803e4375e5d">Data leakage: The many ways you accidentally cheat in Machine Learning</a> was originally published in <a href="https://medium.com/shift-enter">Shift+Enter</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>