<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mihai Boldeanu on Medium]]></title>
        <description><![CDATA[Stories by Mihai Boldeanu on Medium]]></description>
        <link>https://medium.com/@mishuboldeanu?source=rss-2ba2d51c8d77------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zTbx2tKr10NMB1_tec4cDA.jpeg</url>
            <title>Stories by Mihai Boldeanu on Medium</title>
            <link>https://medium.com/@mishuboldeanu?source=rss-2ba2d51c8d77------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:44:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mishuboldeanu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Severely Imbalanced Datasets, Part II: Synthetic Data Boogaloo]]></title>
            <link>https://medium.com/ing-tech-romania/severely-imbalanced-datasets-part-ii-synthetic-data-boogaloo-96214c53274e?source=rss-2ba2d51c8d77------2</link>
            <guid isPermaLink="false">https://medium.com/p/96214c53274e</guid>
            <category><![CDATA[imbalanced-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[oversampling]]></category>
            <dc:creator><![CDATA[Mihai Boldeanu]]></dc:creator>
            <pubDate>Sun, 23 Mar 2025 13:30:25 GMT</pubDate>
            <atom:updated>2025-03-24T07:43:26.739Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*mHT8sozYOUpnDRLk-nL4nQ.png" /></figure><p>If you’re back for more fraud detection fun, welcome! If this is your first time here, you might want to check out my previous <a href="https://medium.com/ing-tech-romania/severely-imbalanced-datasets-or-how-i-learned-to-stop-worrying-and-love-the-fraud-ce2aeee615ac">article</a>, where we explored the challenges of severely imbalanced datasets and the complexities of fraud detection.</p><h3>Quick recap</h3><p>Fraud is exceptionally rare. Yet, paradoxically, it feels like it’s happening all the time. This contradiction exists because the sheer number of transactions creates countless opportunities for fraud. Every time you tap your card, shop online, or click a link in an email, there’s a small risk someone is trying to exploit the system. Fraud is as rare as flipping a coin and having it land on its edge — but when you’re flipping millions of coins per hour, every hour, every day, fraud attempts are bound to slip past safeguards and impact people’s lives.</p><p>In the previous article, we tackled this problem by experimenting with random oversampling and undersampling to train ML models capable of detecting fraud. But those were just the simplest ways of handling class imbalance. As promised, this time we’re diving into more advanced techniques to overcome data scarcity and improve fraud detection.</p><p>In this article we will be investigating synthetic data generation — a smarter way to boost model performance without simply duplicating existing fraud cases. By using methods like SMOTE, Borderline-SMOTE, ADASYN, and SMOTE-Tomek, we’ll see if artificially generated fraud examples can help our models generalize better.</p><p>So, whether you’re a fraud analyst, data scientist, ML enthusiast, or just someone who enjoys watching models struggle with severely imbalanced data, stick around! This is where things get interesting.</p><h3>Advanced Synthetic Oversampling</h3><p>In the previous article, we tackled data imbalance using random oversampling and undersampling — simple techniques that can help but ultimately come with major drawbacks.</p><ul><li>Random oversampling just duplicates existing samples, which is like copy-pasting the same question and answer on an exam — it doesn’t actually teach the model anything new.</li><li>Random undersampling is like throwing out half the test questions just to make studying easier. Sure, it simplifies the problem, but at the cost of losing valuable information.</li></ul><p>Now, it’s time to move beyond basic resampling and explore synthetic data generation — where we don’t just duplicate samples but aim to create entirely new, realistic ones to improve model training.</p><p>Enter <a href="https://www.jair.org/index.php/jair/article/view/10302">SMOTE</a> (Synthetic Minority Over-sampling Technique), one of the most widely discussed methods for handling imbalanced datasets. Unlike simple duplication, SMOTE generates new synthetic samples by interpolating existing samples, filling in gaps in the data distribution.</p><p>This allows us to create new fraud examples that resemble real cases, rather than simply increasing their count. But while SMOTE is an improvement over random oversampling, it comes with its own set of challenges and limitations.</p><p>SMOTE can introduce new problems that hurt model performance rather than help it:</p><ul><li>Class Overlap — SMOTE generates new points by drawing straight lines between existing samples, even if they are far apart. This can create synthetic fraud cases in regions where real fraud never occurs, making fraud and legitimate transactions harder to separate.</li><li>Outlier Sensitivity — If a rare fraud case sits in isolation, SMOTE will generate more synthetic versions of it, reinforcing potentially noisy or unrepresentative patterns rather than improving generalization.</li><li>Boundary Distortion — Since SMOTE does not consider class boundaries, it can inject synthetic fraud cases into majority-class regions, confusing the model rather than improving fraud detection.</li></ul><p>To avoid these pitfalls, we can also try smarter synthetic oversampling methods — techniques that adapt to the data distribution rather than blindly generating new points.</p><p>A major issue with standard SMOTE is that it generates synthetic samples across the entire minority class, even in regions where fraud cases are clearly distinct. <a href="https://link.springer.com/chapter/10.1007/11538059_91">Borderline-SMOTE</a> improves this by focusing oversampling on cases that lie near the decision boundary — where fraud detection is most difficult.</p><p><strong>How it works:</strong></p><ul><li>It identifies fraud cases that are near legitimate transactions (i.e., those surrounded by majority class samples).</li><li>Synthetic samples are only generated near these borderline cases, with the goal of reinforcing the model’s ability to distinguish between fraud and legitimate transactions.</li><li>This ensures that new fraud cases are strategically placed where they are most needed, rather than randomly scattered across the dataset.</li></ul><p>Use Case: Useful when fraud cases are often borderline suspicious rather than clearly fraudulent, improving the model’s sensitivity to subtle fraud patterns.</p><p>Borderline-SMOTE improves sensitivity around the boundary but may ignore underrepresented clusters away from the boundary and risk overfitting.</p><p>Not all fraud cases are equally difficult to detect. Some are well-separated from legitimate transactions, while others closely resemble normal behaviour, making them harder for the model to classify.</p><p><a href="https://ieeexplore.ieee.org/document/4633969">ADASYN</a> (Adaptive Synthetic Sampling) builds on SMOTE by focusing oversampling efforts on the hardest-to-classify fraud cases, rather than treating all fraud cases equally.</p><p><strong>How it works:</strong></p><ul><li>It measures how difficult each fraud case is to classify based on its proximity to majority class samples.</li><li>Fraud cases that are surrounded by legitimate transactions receive more synthetic samples, while easier-to-classify cases receive fewer.</li><li>This helps the model focus on fraud cases that are most likely to be misclassified, improving precision without overwhelming the dataset with unnecessary synthetic samples.</li></ul><p>Use Case: Ideal when fraud cases closely resemble legitimate transactions, ensuring that the model improves where it struggles most.</p><p>ADASYN adapts oversampling based on local difficulty, potentially leading to a more balanced model, but it might over-sample in regions where noise is present, if the difficulty measure is affected by outliers.</p><p>One downside of oversampling is that it increases the overlap between fraud and legitimate transactions, sometimes making fraud appear more common than it actually is. To counteract this, <a href="https://www.researchgate.net/publication/221322870_Balancing_Training_Data_for_Automated_Annotation_of_Keywords_a_Case_Study">SMOTE-Tomek</a> combines oversampling with data cleaning to refine the dataset and improve model generalization.</p><p><strong>How it works:</strong></p><ul><li>First, SMOTE generates synthetic fraud cases, increasing the minority class.</li><li>Then, pairs of nearest fraud-legitimate samples, called <a href="https://ieeexplore.ieee.org/document/4309452">Tomek links</a>, are identified.</li><li>Borderline legitimate transactions are removed, ensuring a clearer separation between fraud and non-fraud cases.</li></ul><p>This prevents fraud cases from being placed too close to legitimate transactions, reducing false positives and improving the decision boundary.</p><p>Use Case: Useful when the dataset has significant class overlap, helping models differentiate fraud from normal transactions more effectively. It is also useful to make a trade-off when we are interested in reducing the False Negative rate at the cost of increasing the False Positive rate.</p><p>SMOTE-Tomek improves class separation by combining synthetic oversampling with data cleaning — removing ambiguous majority samples via Tomek links clarifies the decision boundary and reduces false positives — however, it may inadvertently discard informative borderline instances and introduces additional complexity in tuning the parameters.</p><h3>Visualizing generated data</h3><p>When generating synthetic data, one of the biggest questions is whether these new samples truly resemble real cases. If our synthetic samples don’t reflect real-world patterns, our model will learn patterns that don’t exist, leading to poor generalization and ineffective detection.</p><p>In practice, we can use dimensionality reduction to visualize both real and synthetic data in two or three dimensions, helping us see how well synthetic samples align with real ones. While methods like PCA or t-SNE are popular, this article focuses on <a href="https://joss.theoj.org/papers/10.21105/joss.00861">UMAP</a> (Uniform Manifold Approximation and Projection) — a powerful technique that often provides clearer global structure and is computationally efficient.</p><p>Our dataset can contain tens or hundreds of features, making direct visualization of data in high-dimensional space nearly impossible. By projecting these high-dimensional points down to 2D or 3D, we can visually inspect how real and synthetic samples cluster.</p><p>If our synthetic fraud data truly mimics real fraud, we expect to see these new samples overlapping or closely clustering with real fraud points. If they appear in completely separate regions, that’s a sign the synthetic generation process might be creating unrealistic patterns.</p><p>UMAP is a nonlinear dimensionality reduction algorithm that builds a graph of your high-dimensional data and learns a low-dimensional representation that aim to preserves both local and global structure.</p><p>When applying UMAP to our dataset, we train the embedding on the real data only (both legitimate and real fraud). Then we transform the synthetic fraud samples into the same 2D space. This ensures that synthetic data must “fit in” to the structure UMAP learned from actual transactions, rather than artificially shaping the embedding to suit itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WVc-fUUcN736hLK_3j0X8w.png" /><figcaption>Figure 1: Legitimate Transactions (Majority Class) typically form one or more large clusters, reflecting their prevalence. Real Fraud Transactions (Minority Class) often cluster in smaller pockets, showing their rarity and potentially distinct characteristics. Synthetic Fraud Transactions should ideally appear within or very near the real fraud clusters if they are truly realistic. If they scatter across legitimate clusters or form bizarre “islands,” it may indicate the synthetic method isn’t generating credible fraud. All synthetic generation algorithms were used to double the number of fraud samples compared to baseline.</figcaption></figure><p>From the UMAP plots ( Figure 1), we can see that SMOTE (top-left) scatters synthetic fraud samples broadly across the main fraud region, Borderline-SMOTE (top-right) places them closer to the boundary where legitimate and fraud transactions meet, ADASYN (bottom-left) adapts to local difficulty and thus clusters synthetic points where real fraud is most intermixed with legitimate samples but it also adds synthetic samples near outliers, and SMOTE-Tomek (bottom-right) appears to behave similar to SMOTE but it also remove borderline legitimate transactions while adding synthetic fraud cases. These patterns highlight how each method reshapes the minority class in a unique way, emphasizing different regions and potentially influencing model performance.</p><h3>Experimental setup</h3><p>We used the same <a href="https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud">credit card fraud dataset</a> and followed the same preprocessing steps as in Part 1, including stratified sampling, train–test split, and using Precision–Recall AUC as our primary metric. This consistency ensures that our results remain directly comparable to the previous study.</p><p>To assess the impact of synthetic oversampling on model performance, we create multiple versions of the training dataset by applying different oversampling strategies — namely, Original Imbalance, SMOTE, Borderline-SMOTE, ADASYN, and SMOTE-Tomek. For each method, we generate different resampling ratios (e.g., M:m, M:2m, M:10m) to evaluate how varying levels of synthetic oversampling affect the results. Because adding synthetic samples changes the original data distribution this must be applied only on the training data and never of the set used to test the models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*3eiCwSFnxd8ezqHndN7R1w.png" /><figcaption>Figure 2: Diagram of the data splitting and augmentation process. The initial dataset is split into train and test sets. The train set is split into 3 folds and the oversampling algorithm are applied such that we always keep one fold unbiased. Each model type and hyperparameter combination is trained on the 3 Runs presented. For the final evaluation of the models an oversampled Train set is used and evaluated on the Test set.</figcaption></figure><p>We then perform 3-fold cross-validation on these datasets and conduct hyperparameter tuning for the same four models (Gradient Boosting, Random Forest, MLP Classifier, and Logistic Regression). This repeated validation yields robust performance estimates and insights into model stability under different imbalance corrections, allowing us to isolate the impact of the new synthetic techniques from inherent differences among model architectures. To further clarify the cross-validation approach for each oversampling approach the folds are augmented only when used for training and not when used for evaluation of the model.</p><p>Our evaluation primarily relies on the Precision–Recall Curve (PRC) and Average Precision (AP) scores, which are particularly well-suited for imbalanced datasets like those used in fraud detection. By comparing these metrics across the various resampling strategies — from the original imbalance to multiple levels of synthetic oversampling — we can assess how each method influences the classifier’s ability to accurately distinguish fraud from legitimate transactions. Additionally, analyzing model stability across these variants helps us identify which combination of oversampling technique and classifier provides the most consistent and reliable performance, ultimately guiding us toward the most effective fraud detection strategy.</p><h3>Results and discussion</h3><p>We performed a hyperparameter search for each model type — Gradient Boosting, Random Forest, Logistic Regression, and MLP Classifier — across various oversampling methods (ADASYN, Borderline-SMOTE, SMOTE, SMOTE-Tomek) and different imbalance ratios (e.g., 2×, 10×). Each combination was evaluated under 3-fold cross-validation, and the best hyperparameter set for each model was then tested on a holdout set to estimate real-world performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*URzsfBZAl0vYs2kwLdrrUQ.png" /><figcaption>Figure 3: Box plots showing the distribution of average precision scores for four models (Gradient Boosting, Random Forest, Logistic Regression, and MLP Classifier) under different oversampling methods (ADASYN, Borderline-SMOTE, SMOTE, SMOTE-Tomek) and imbalance levels (2×, 10×). Each box summarizes the cross-validation performance of the best hyperparameters found for each model type, illustrating how both the choice of oversampling technique and the magnitude of minority class augmentation impact model effectiveness in detecting fraud.</figcaption></figure><p>Random Forest and Gradient Boosting both benefited from larger minority-class multipliers (e.g., 10×) and cleaning-based methods. In particular, ADASYN ×10 consistently enhanced Random Forest’s detection capabilities, while SMOTE-Tomek ×10 emerged as the strongest option for Gradient Boosting. These results suggest that ensemble methods can leverage additional synthetic samples effectively without succumbing to overfitting.</p><p>Logistic Regression showed relatively modest gains compared to the ensemble models. Although ADASYN ×10 offered a slight performance edge — likely by focusing on the hardest-to-classify fraud cases — the overall improvement was limited, indicating that a linear model may not fully capitalize on the extra minority-class information.</p><p>The MLP Classifier was more sensitive to large-scale oversampling, with performance often declining under aggressive synthetic augmentation. In contrast, SMOTE ×2 provided a balanced approach, offering enough new minority samples to boost recall without introducing excessive noise or redundancy.</p><p><strong>Best oversampling choices per model type</strong>:</p><ul><li>Gradient Boosting: SMOTE-Tomek ×10, leveraging both synthetic generation and removal of borderline majority samples.</li><li>Random Forest: ADASYN ×10, adaptively targeting the hardest fraud instances.</li><li>Logistic Regression: ADASYN ×10 for modest but consistent gains.</li><li>MLP Classifier: SMOTE ×2 to avoid overfitting while still enriching minority data.</li></ul><p>The results indicate that ensemble methods generally benefit from more aggressive synthetic oversampling strategies, while simpler or more noise-sensitive models (e.g., Logistic Regression, MLP) require more conservative approaches. This highlights the importance of matching the oversampling technique and ratio to the model’s capacity for handling synthetic data, as well as the potential utility of additional data-cleaning steps when significant class overlap or noise is present.</p><p>To further illustrate the impact of our synthetic oversampling approach relative to our baseline methods, we examined the Precision–Recall curves and computed the Average Precision (AP) for each model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lXJg8DZ0WjJdBfg80DeauQ.png" /><figcaption>Figure 4: Plot of the Precision–Recall (PR) curves for each model. On the left we have the results from the first article when using only random over or undersampling. On the right we have the best performing models using the synthetic data generation approach.</figcaption></figure><p>In our baseline, Random Forest — trained using random oversampling of the minority class by a factor of 10 — achieved an Average Precision (AP) of 82%. With the introduction of synthetic oversampling via ADASYN at a 10× ratio, Random Forest’s AP increased to 85%. This improvement suggests that adaptive synthetic sampling effectively enriches the minority class in a way that enables the ensemble to capture subtle fraud patterns without overfitting. In conclusion, Random Forest benefits significantly from synthetic oversampling, demonstrating enhanced detection performance with a more balanced and informative training set.</p><p>For Gradient Boosting, the baseline method of oversampling the minority class by a factor of 10 resulted in an AP of 81%. By employing SMOTE-Tomek at a 10× ratio in the current experiment, the AP rose modestly to 82%. This slight improvement indicates that while Gradient Boosting can leverage the refined synthetic samples — created by both generating new minority instances and removing borderline majority samples — the gains are less dramatic compared to Random Forest. In summary, the enhancement in Gradient Boosting’s performance, though modest, confirms the potential of integrating data cleaning with synthetic oversampling to slightly improve model sensitivity</p><p>Logistic Regression, which originally achieved an AP of 84% using random undersampling of the majority class by a factor of 2, experienced a decline to 81% AP when synthetic oversampling via ADASYN at a 10× ratio was applied. This drop implies that the linear model may not fully benefit from the enriched minority class, likely due to the introduction of noise and the model’s limited capacity to capture non-linear patterns. In conclusion, Logistic Regression appears less suited to aggressive synthetic augmentation, highlighting the need for alternative strategies such as cost-sensitive learning for improving performance in such settings.</p><p>The MLP Classifier initially achieved an AP of 86% under a baseline approach that used majority undersampling by a factor of 2. With the current approach using SMOTE at a 2× ratio, the AP slightly decreased to 85%. Although the drop is modest, it suggests that neural networks like the MLP are sensitive to the quality and extent of synthetic data, potentially leading to overfitting if oversampling is too aggressive. In summary, while the MLP may benefit from synthetic oversampling to a degree, careful tuning is essential to balance the benefits of additional minority samples against the risk of introducing noise.</p><p>Overall, these PR curves are consistent with the observations derived from cross-validation and holdout evaluations: aggressive oversampling can substantially enhance recall for ensemble models, whereas simpler or noise-prone classifiers require careful tuning of oversampling parameters to avoid overfitting and undue noise amplification.</p><h3>Conclusions</h3><p>In this study, we investigated the <strong>effectiveness of advanced synthetic oversampling techniques</strong> to mitigate class imbalance in fraud detection. The experiments revealed that ensemble methods — specifically Random Forest and Gradient Boosting — benefit substantially from aggressive oversampling, achieving notable improvements in average precision when enriched with synthetic minority samples. In contrast, simpler models like Logistic Regression and neural networks (MLP) are more sensitive to oversampling, often requiring more conservative augmentation to avoid noise amplification and overfitting. These findings emphasize the importance of tailoring oversampling strategies to the model architecture.</p><p>Despite these promising results, <strong>synthetic oversampling is not a panacea</strong>. Its effectiveness is contingent upon the model’s capacity to assimilate new, artificially generated data, and it may not perform well in scenarios characterized by highly non-linear relationships or substantial noise. Furthermore, the risk of overfitting and the potential loss of informative samples during data cleaning underscore the need for careful tuning and, in some cases, alternative strategies. Another important aspect to keep in mind is that if a method or technique works well on toy datasets, such as the one we used in this article, it does not mean this will have the same effect universally.</p><p>Moreover, the <strong>No Free Lunch theorem reminds us that no single method is universally optimal for every problem</strong>. This principle underscores the inherent limitations of any algorithm or technique, including synthetic oversampling. We cannot create new information out of nothing; rather, we can only manipulate the existing data to better suit our needs. In the context of fraud detection, this means we can adjust our models to prioritize either false positive rates or false negative rates, depending on the specific requirements and constraints of the application. For instance, in scenarios where the cost of missing a fraudulent transaction is high, we might bias the model towards minimizing false negatives. Conversely, if the cost of investigating false alarms is prohibitive, we might aim to reduce false positives. This trade-off highlights the importance of carefully tuning our models and strategies to achieve the desired balance between precision and recall.</p><p>In Part 3, we will extend our investigation by exploring anomaly detection and unsupervised learning techniques to complement synthetic oversampling. These approaches aim to further enhance fraud detection in environments where conventional oversampling may fall short, ultimately striving for a more robust and comprehensive solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96214c53274e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ing-tech-romania/severely-imbalanced-datasets-part-ii-synthetic-data-boogaloo-96214c53274e">Severely Imbalanced Datasets, Part II: Synthetic Data Boogaloo</a> was originally published in <a href="https://medium.com/ing-tech-romania">ING Hubs Romania</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Severely Imbalanced Datasets or: How I Learned to Stop Worrying and Love the Fraud]]></title>
            <link>https://medium.com/ing-tech-romania/severely-imbalanced-datasets-or-how-i-learned-to-stop-worrying-and-love-the-fraud-ce2aeee615ac?source=rss-2ba2d51c8d77------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce2aeee615ac</guid>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[imbalanced-dataset]]></category>
            <category><![CDATA[classification-metrics]]></category>
            <category><![CDATA[oversampling]]></category>
            <category><![CDATA[undersampling]]></category>
            <dc:creator><![CDATA[Mihai Boldeanu]]></dc:creator>
            <pubDate>Mon, 04 Nov 2024 11:24:28 GMT</pubDate>
            <atom:updated>2024-11-04T11:24:28.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bdd2XM3memqzcHEOPxs7cQ.jpeg" /></figure><p>As bad and unlucky as you might feel when you are the victim of fraud, you have to keep one thing in mind: you were just witness to a very rare event. Even though the news would have you believe that a fraudster is behind every corner of the internet, waiting to steal your data, or more directly your hard-earned cash, the truth is that fraud is an exceptionally rare event when compared to the huge number of transactions that happen daily.</p><p>In this material, I will discuss fraud from a data scientist’s perspective. Together, we’ll explore real-world fraud prevalence, tackle severely imbalanced datasets, and hear from both statisticians and those dashing, results-driven data scientists. Finally, we’ll try our luck at spotting fraud in a publicly available dataset.</p><h3>Why do imbalanced datasets exist</h3><p>Imbalanced datasets are aggregations of data where one class is vastly more present than the other. Usually, we call them the Majority class(M) and the minority class(m) and we describe the ratio between them as the imbalance.</p><p>There are two types of imbalanced datasets: the ones that correctly capture the distribution of the underlying phenomenon and the ones that are just bad.</p><p>To get the bad ones out of the way, if you are trying to do the most basic ML tasks, identifying dogs from cats, your dataset should be balanced, and if it’s not, the way to solve it is to just spend 2 hours on the internet collecting extra images.</p><p>On the other hand, datasets that are imbalanced because the real-world distribution of events is naturally imbalanced present a trickier problem to solve. These sorts of data sources are usually tied to rarely occurring events that have significant negative impact.</p><p>Going from less to more imbalance we can have:</p><p>1. Slightly imbalanced data such as in the case of churn prediction or predicting when a client is likely to cancel a subscription (telecom, SaaS, Netflix etc.) can have an imbalance 70:30 to 89:11 depending on industry<a href="https://customergauge.com/blog/average-churn-rate-by-industry">¹</a>.</p><p>2. In loan default prediction you can have an expected default rate of about 15% with an imbalance of 85:15 in some public datasets<a href="https://www.kaggle.com/datasets/subhamjain/loan-prediction-based-on-customer-behavior/data">²</a>, but in practice the imbalance is even greater<a href="https://www.fitchratings.com/research/corporate-finance/us-leveraged-loan-default-rate-for-2024-revised-up-amid-deterioration-24-07-2024">³</a>.</p><p>3. In cybersecurity, in intrusion detection systems for networks, the number of malicious attacks or intrusions is vastly outnumbered by normal traffic. Normal traffic typically constitutes over 90% of the data, while attack traffic is less than 10% and imbalance of 90:10 depending on the exact attack type<a href="https://www.unb.ca/cic/datasets/ids-2018.html">⁴</a>.</p><p>4. A more severe case of imbalance may come from the medical sector where medical datasets for rare diseases, like cancer detection or rare genetic conditions, are naturally imbalanced because these diseases affect a small proportion of the population. In breast cancer detection datasets, the positive class may only make up 2–5% of the data<a href="https://pmc.ncbi.nlm.nih.gov/articles/PMC9778593/">⁵</a> and imbalance of 95:5 to 92:2.</p><p>5. Moving to an even worse imbalance there is insurance fraud where insurance companies face fraud in claims processing (health, auto, or home insurance). A very small fraction of claims is fraudulent, but they can be costly. Often less than 1% of the claims are fraudulent<a href="https://www.forbes.com/advisor/insurance/fraud-statistics/">⁶</a>, leading to an imbalance ratio as high as 99:1.</p><p>6. Some of the most imbalanced datasets available are those with credit card fraud which typically have an extreme imbalance, where fraudulent transactions may only make up 0.1% to 1% of the entire dataset<a href="https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud">⁷</a>, imbalance ratio 99:1 to 999:1. Even worse is the case the online transaction/banking where prevalence can be as low as 0.01%<a href="https://www.ecb.europa.eu/press/pr/date/2024/html/ecb.pr240801~f21cc4a009.en.html">⁸</a> due to the high number of events happening and equivalent imbalance ratio of 9999:1.</p><p>While the fact that transactions are rarely fraudulent is good for the average Joe, it makes our life much more difficult when trying to teach models to detect fraud. Next, we will look over what the experts have to say about handling the problem of imbalanced datasets.</p><h3>The Statistician vs the Data Scientist view on imbalance</h3><p>Statisticians’ opinions are summarized by a great response on a specialty statistics forum: <br> “Class imbalance almost certainly is not a problem, and there is no need to use under sampling, oversampling, or artificial balancing to solve a non-problem. It might be helpful if you said why you find the imbalance problematic. Statisticians do not see such a problem” by <a href="https://stats.stackexchange.com/questions/555584/classification-with-imbalanced-data#comment1021699_555584">Dave circa 2021</a>.</p><p><strong>What the statistician means is </strong>that trying to apply classification methods to a severely imbalanced dataset and complaining when that does not work as expected is usually the wrong approach. They propose using the right statistical methods for the problem (estimating a log likelihood or other such tendency estimations).</p><p>In statistics, classification is seen as a forced choice where, based on a threshold (e.g. 0.5), we decide something is fraud or non-fraud. This type of approach is seen as inferior to that of modeling probabilities and providing a risk score for each event. Having a risk score allows the end user to select their own threshold based on their risk appetite and alert checking budget.</p><p><strong>The data scientist, on the other hand</strong>, values his job more and will know when to give in to the dark side and build classifiers, if that’s what the business requests. But this does not mean that we will ignore the knowledge of the statistician.</p><p>While probabilistic thinking is crucial for capturing uncertainty, practical scenarios exist where oversampling and undersampling can shine, particularly in classification tasks like fraud detection in imbalanced datasets. These resampling methods address the bias inherent in skewed datasets, where traditional classifiers tend to favor the majority class.</p><p>Balancing the class distribution through oversampling or undersampling lets the model focus on the minority class, avoiding trivial majority class predictions. While probabilistic approaches might offer nuanced decisions in some cases, resampling methods help models manage severe imbalance, ensuring minority class instances are properly accounted for. This is crucial in fraud detection, where missing fraudulent transactions carries significant costs.</p><p><strong>Another important aspect to consider</strong> is that most production ready or off-the-shelf models have a large time and memory complexity linked to the training set size. This sometimes forces us to select certain model types not based on merit or performance but rather on whether they can be trained on datasets size we are interested in. By under sampling, we can reduce the size of the training dataset and have more flexibility in the types of models we can use for classification. As an example, Gaussian Processes have O(n³) time and O(n²) memory complexity. This makes training on more than a few thousand samples unfeasible in most cases.</p><h3>Public credit card fraud dataset</h3><p>But enough with the story telling and let’s look at some data. In this section we will investigate a fraud related <a href="https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud">Kaggle dataset</a>:</p><p>Credit Card Fraud Detection: This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions, 581:1 imbalance ratio.</p><p>Using this dataset, we will first look at the data and get an idea of the available features and how they behave for both fraud and non-fraud events.</p><p>The dataset was created from real transactions, and this means it had to be anonymized before publication. The way this was achieved was by applying PCA to the features and saving them as a reduced set of 28 columns (labeled V1-V28). This means we can skip the step of feature selection or doing checks for strong correlation between features.</p><p>Another column in the dataset is Time, and while this would be an important field to use if we had months or years of data, we would ignore it in our case and drop it. The final two columns are Amount and Class, where we see the value of a transaction and the label whether it was fraud or not.</p><p>To be able to evaluate any classification model we will need to split the data: first we take 10% and set aside on the shelf (holdout set), for the rest we will split the training data in such a way that we can do 3-fold validation. The data splitting was done using stratified sampling to guarantee the same data imbalance between train and holdout. This means that the remaining samples will be split into 3 equal folds, and they will each have the role of the validation fold once.</p><h3>Solutions</h3><p>Let’s move to the fun part of the data science journey, trying to find the best Metrics, Methods and Models (Triple M) to solve our problem. As we said at the beginning of the article, the problem is simple, we want to create a model capable of identifying fraudulent credit card transactions, but that doesn’t create too many false alerts.</p><h3>Metrics</h3><p>The best way to choose a metric is to think like a bank for a second. Let’s figure out what the cost of fraud is. There is an obvious cost of fraudulent transactions, which is the money taken from the customer account, but there is also the cost of evaluating each alerted transaction and having an agent reach out to the customer to check if the alert was fraud.</p><p>These two costs can be mapped to well-known metrics Recall, or the amount of fraud that the model identifies from all the fraud that happens, and Precision, which is the percentage of actual fraud out of all the events the model will flag as suspect. Recall will serve the end user, and Precision will benefit the bank agent that has to investigate the alerts. With these two metrics in mind and remembering what our friend the statistician said, we will use the Precision-Recall curve and the area under this curve as a way of evaluating model performance at all the threshold levels simultaneously. Average Precision (AP) summarizes a precision-recall curve as the weighted mean of precision achieved at each threshold, with the increase in recall from the previous threshold.</p><h3>Over sampling minority class</h3><p>Oversampling is a method that tries to balance out the minority class by adding more samples with that label. How we get “new” samples is the actual complex part, but for this initial article we will stick to RandomOversampling. This method will just duplicate samples with the fraud label.</p><p>The main concern, around fixing imbalance using Oversampling, is about when to apply the method. If we apply this method before we separate the holdout from the train set, we might be causing some information leakage that will artificially inflate the performance of the model. This would happen as positive samples from the holdout could be duplicated and end up in the train set. Another important reason on why we should avoid oversampling the holdout part of the data would be that we would be changing the distribution of fraud we are expecting on real data. To avoid this, we will apply the method only after the data is split.</p><p>When oversampling is done on slightly imbalanced data the minority class is usually augmented until the ratio of M:m (Majority to minority) reaches 50%, but in our case that would be a very bad idea as we start with so few samples of fraud.</p><p>What we can do in our case is to raise the number of positive labels in the dataset by some multiplier and see the effect it has on trained models. We will set up 2 arbitrary levels so that if we start with an initial M:m ratio of samples we can have a M:2*m and M:10*m version. This will allow us to see if this method enhances model performance compared to the normal training.</p><h3>Under sampling majority class</h3><p>Under sampling is another favorite in these sort of imbalance situations and this is sometimes for practical reasons such as fitting the dataset into memory, being able to train the model using finite memory and time and of course improving the performance of the model. What under sampling does is remove samples from the Majority class until the imbalance is less severe. Again, we will not be able to hit a 50% ratio between the classes so we will choose to arbitrary targets to under sample too. If the initial ratio is M:m we will aim for M/2:m and M/10:m.</p><p>After the two operations we can have the following combinations of imbalance M/10:m, M/2:m, M:m, M:2*m, M:10*m. Of course, we could also try both under and over sampling, but this is a little too much for this article and will be discussed in part II.</p><h3>Hyperparameter tuning of classifiers</h3><p>The final step is finding the last M, as in models. Because we are not sure which models are best for our specific problem, we will use a shotgun approach where we try many models and select the best performer. To give ourselves the best chance of finding a great solution we will do hyperparameter tuning with a fixed budget of 20 tries for each model type.</p><p>The models we will try are Logistic regression, RandomForest, Gradient boosting, and a Multi-layer perceptron. We are leaving out SVMs, Gaussian Processes and Naïve Bayes type models as they are difficult to train when we have more than a few thousand samples.</p><h3>Results</h3><p>The hyperparameter search was done for all models on the 3 folds and we use the mean across the folds to aggregate the results and select the best performing variant of each model type. This search was repeated for all 5 variants of the folds to also capture the effect of different imbalances and dataset size on the performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*wEZ1b57CeTjWl9R-_fIt6Q.png" /><figcaption>Figure 1 Results of hyperparameter search on the multiple dataset variants. This is aggregated for all hyperparameter combinations for each model type</figcaption></figure><p>In Figure 1. we can see that the effect of the imbalance is different for each model type. Logistic regression appears to not be impacted at all by the different imbalance ratios and different sizes of the datasets. Gradient boosting seems to be more impacted by dataset size than imbalance ratio, with the dataset that had the imbalance reduced by oversampling the minority class times 10 is the best performing and the one where the majority class was under-sampled by 10 times is the worst performer. Random forest appears to have a small benefit from correcting the imbalance, but the effect is small. For Multi-Layer Perceptron’s the effect seems to slightly lower the performance of the models when we are changing the imbalance.</p><p>A total of 5 imbalance sizes * 4 model types * 3 folds * 20 hp combinations =1200 experiments were run to find the best performing models. The final selected models will be evaluated on the holdout set to see if our selection of hyperparameters and dataset imbalance is optimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*RC7TOE_wQr82W4gF95pETQ.png" /><figcaption>Figure 2 Best performing hyperparameters for each of the 4 model types at various imbalance levels</figcaption></figure><p>When looking at the best performing model variants across all dataset sizes and imbalances we have Figure 2. This plot confirms our findings from the previous figure. Gradient boosting prefers a larger dataset size. Random forest is working best when either we over sample minority by 10 or under sample majority by 10. Logistic regression is stable across all the dataset variants and MLP is best either on the full dataset or when under sampling by a factor of 2 and the performance degrades when we oversample or under sample to aggressively.</p><p>Taking the best performing models and imbalance ratios we replicate the setup on the initial train holdout split. The train set is under and over sampled until we have the 5 variants with original imbalance, minority class *2, minority class * 10, majority class /2, majority class /10.<br> We then train each model on the preferred dataset size and evaluate on the holdout set. To keep the statisticians happy, we evaluate the Precision-Recall Curve to check the model behaviors at all possible threshold values and we use the Average Precision as a final score for each model. This comparison can be seen in Figure 3. The PR curve is a visualization of the trade-off in Precision and Recall that happens when changing the threshold used as the classification boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*d4iCQCxKYe94PQOMadxY8A.png" /><figcaption>Figure 3 PR-Curves for best performing models. An ideal or perfect model will have the Precision of 1.0 when Recall is 1.0 and the AP will be 100% and the figure will look like a full square</figcaption></figure><p>The models perform comparable to what we observed on the folds, the only surprise being the logistic regression model which performs significantly better on holdout. For this toy dataset the best model we have is the MLP Classifier, but this was a very resource constrained search and there are still many more options to explore to get a better model.</p><h3>Conclusion</h3><p>In this article, we explored various scenarios of imbalanced datasets and strategies to address them, enhancing our chances of success in classification tasks. We examined both oversampling and undersampling techniques, along with hyperparameter and imbalance tuning.</p><p>This is the first in a series of articles focused on imbalanced datasets. Stay tuned for the next installment, coming in early 2025, where we’ll delve into advanced over-sampling methods like SMOTE and ADASYN. We’ll also explore anomaly detection algorithms as unsupervised learners and other innovative approaches to combating fraud.</p><p>In the realm of fraud detection, “loving the fraud” means valuing the few precious fraud samples amidst a sea of normal transactions. These rare instances guide us in training and refining our models to uncover hidden patterns. It’s akin to treasure hunting — leveraging those scarce fraud samples to prevent future incidents and safeguard against the unknown. Embracing this challenge, we learn to appreciate every rare fraud case as a key to more effective prevention.</p><p>I hope you enjoyed this brief insight into the world of fraud detection through the lens of data science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce2aeee615ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ing-tech-romania/severely-imbalanced-datasets-or-how-i-learned-to-stop-worrying-and-love-the-fraud-ce2aeee615ac">Severely Imbalanced Datasets or: How I Learned to Stop Worrying and Love the Fraud</a> was originally published in <a href="https://medium.com/ing-tech-romania">ING Hubs Romania</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>