<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Valeria Verzi on Medium]]></title>
        <description><![CDATA[Stories by Valeria Verzi on Medium]]></description>
        <link>https://medium.com/@valeria.verzi1?source=rss-fa6d10644812------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hO3R3NCZJs-rWclwRR3vqQ.jpeg</url>
            <title>Stories by Valeria Verzi on Medium</title>
            <link>https://medium.com/@valeria.verzi1?source=rss-fa6d10644812------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:46:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@valeria.verzi1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Monitoring Fraud Models in Production: Challenges and Solutions]]></title>
            <link>https://medium.com/@valeria.verzi1/monitoring-fraud-models-in-production-challenges-and-solutions-477ac99760f8?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/477ac99760f8</guid>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Wed, 31 Dec 2025 10:02:44 GMT</pubDate>
            <atom:updated>2025-12-31T10:02:44.260Z</atom:updated>
            <content:encoded><![CDATA[<p>From drift to missing labels — practical ways to keep your models effective over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*R7j5Nku19tm5st9DVNl4sw.jpeg" /></figure><blockquote>Hey there! If you’re working on fraud detection, you know that keeping your models sharp in production is like trying to hit a moving target while blindfolded. Let’s dive into why monitoring fraud detection models is tricky and how to do it right.</blockquote><h3>Introduction</h3><p>In our previous article of the <a href="https://medium.com/@valeria.verzi1/%EF%B8%8F-%EF%B8%8F-the-data-science-playbook-for-fraud-detection-53153bb9aef9?postPublishedType=repub">series</a>, <a href="https://medium.com/@valeria.verzi1/fighting-decay-the-subtle-art-of-keeping-fraud-models-relevant-cb416d19cab6?postPublishedType=initial"><em>Keeping Your Model Sharp: Stability and Drift in Fraud Detection</em></a>, we explored how models can degrade over time and why stability matters. But even when you understand drift, you still need a practical framework to <strong>monitor your models in production</strong>.</p><p>Fraud detection monitoring is full of unique challenges: delayed labels, missing outcomes from rejected transactions, and the interaction of multiple models and rules all create a complicated feedback loop. Traditional performance metrics, like ROC-AUC or precision, can be misleading — relying on them alone is risky.</p><p>In this article, we break down the <strong>core challenges</strong> of fraud model monitoring and explore <strong>practical solutions</strong>. You’ll learn how to detect different types of drift, leverage statistical methods like PSI, and integrate business metrics to make informed decisions. By the end, you’ll have a clear picture of <strong>what to watch, how to watch it, and when to act</strong> — ensuring your models remain effective and your business stays protected.</p><h3>The Core Challenges</h3><p>When it comes to monitoring fraud detection models, we’re dealing with three major headaches that make traditional ML monitoring look like a walk in the park.</p><h3>1. Label Maturity</h3><p>Here’s the thing about fraud — it’s sneaky! There is an inherent delay between transaction processing and fraud confirmation. A transaction that looks perfectly legit today might turn out to be fraudulent weeks or even months later. This temporal gap creates a significant challenge in evaluating model performance, as we’re constantly working with incomplete information about recent transactions.</p><p>This delay in getting our labels (fraud confirmation) makes it super tricky to evaluate our model’s performance in real-time. Think about it: if we look at transactions from the last 30 days, we might be missing fraud labels that haven’t surfaced yet. This leads to to artificially inflated performance metrics.</p><h3>2. Missing Labels for Rejected Transactions</h3><p>This is where it gets really interesting — the “reject inference” problem. When our system blocks a transaction because it smells like fraud, we create a paradox: we’ll never know if we were right or wrong! This creates an incomplete feedback loop that fundamentally affects our ability to measure model performance accurately. Unlike other machine learning applications where we can eventually obtain ground truth for all predictions, in fraud detection, we permanently lose information about rejected transactions.</p><h4>The A/B Testing Solution</h4><p>One way to tackle this is A/B testing, where we split transactions into two groups:</p><ul><li>Group A (Control): Business as usual</li><li>Group B (Treatment): Let everything through (yes, even the suspicious ones!)</li></ul><p>Sounds crazy, but here’s why it’s useful:</p><ol><li>We get complete information about fraud patterns</li><li>We can see what we’re missing in our rejected transactions</li><li>We can validate our model’s decisions</li></ol><p>But hold up — there’s a catch. A/B testing in fraud detection is like playing with fire:</p><ul><li><strong>Business Risk</strong>: Letting all transactions through from group B exposes the business to potential fraud losses</li><li><strong>Complexity</strong>: When you have multiple rules and models, it gets messy fast</li><li><strong>Scale</strong>: Testing multiple things at once isn’t feasible due to combinatorial complexity</li><li><strong>Cost</strong>: Your boss might not be thrilled about the potential fraud losses</li></ul><p>While A/B testing is great for initial validation or checking specific issues, it’s not something you want to run continuously. Most companies save it for special occasions, like when they’re updating their model or investigating weird performance issues.</p><h3>3. Complex Decision Systems</h3><p>Modern fraud detection is like a symphony orchestra — it’s not just one instrument playing. We’ve got multiple ML models and rules working together, making it hard to tell which part is doing what. When a transaction gets blocked, was it the ML model? A rule? Both?</p><h3>Understanding the Impact of Missing Labels</h3><h4>The Confusion Matrix Problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*ttSsz6kP_WID1IMff2AO1g.png" /></figure><p>The foundation of most model performance metrics is the confusion matrix, which traditionally looks like this:</p><p>Actual FraudActual GenuinePredicted FraudTrue Positives (TP)False Positives (FP)Predicted GenuineFalse Negatives (FN)True Negatives (TN)</p><p>The Reject Inference problem tells us that in a real-world fraud detection system, we face a fundamental limitation: we can only observe outcomes for transactions that we allow to proceed. When our model predicts a transaction is genuine and allows it through, we eventually learn whether this decision was correct. But when we block one? We never know if we were right or wrong.This creates an asymmetric view of our model’s performance, where we can only populate the second row of our confusion matrix.</p><h4>Mathematical Implications</h4><p>This missing information messes with our standard metrics:</p><ul><li><strong>Precision</strong> (Positive Predictive Value): Can’t calculate this properly because we don’t know our true TP count.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/1*Zd0_MLIjRINHebWxok0LNw.png" /></figure><ul><li><strong>Recall</strong> (True Positive Rate): [We can estimate this, but it’s incomplete because we’re missing fraud labels.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*46pNKev0tXLPDMTQGPTOqA.png" /></figure><ul><li><strong>False Positive Rate</strong>: Forget about calculating this — we don’t have the FP values!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*m2svEHf20IJryNMzJd90Uw.png" /></figure><h3>Traditional Performance Metrics and Their Limitations</h3><h4>The Challenge with Standard Metrics</h4><p>In credit scoring, they largely use the Gini index — it’s like their go-to metric. But for fraud detection? Not so much. The Gini index works great when you have a nice, clean dataset with clear labels, but in fraud detection, we’re dealing with missing labels and delayed feedback.</p><p>PR-AUC (Precision-Recall Area Under Curve) is our jam in fraud detection because it focuses on the minority class (fraud). But even this isn’t perfect because of those missing true outcome of rejected transactions.</p><p>ROC-AUC? Yeah, that’s not really our thing either. Here’s why:</p><ol><li>Class Imbalance: ROC curves get all confused when one class is super rare</li><li>Business Reality: FPR (used in ROC) doesn’t mean much in fraud contexts</li><li>Operational Focus: We care about precision and recall, not TPR/FPR</li><li>Missing Data Impact: The reject inference problem makes FPR calculation impossible</li></ol><p>More on this topic in another article of mine: <strong><em>[[TODO]-add link to model evaluation metrics]</em></strong></p><h3>Effective Solutions for Production Monitoring</h3><h4>Model Drift: Understanding the Types</h4><p>In fraud detection, we encounter three main types of drift that require monitoring. Understanding these different types of drift is crucial for maintaining model effectiveness over time.</p><h4>1. Data Drift</h4><p>Data drift occurs when the statistical properties of our input features change over time. In the context of fraud detection, this can manifest in several ways. Customer behavior patterns naturally evolve — for example, the rise of mobile payments might shift the distribution of transaction devices. Seasonal variations in spending can cause temporary but significant changes in transaction amounts and frequencies. New transaction patterns might emerge with the introduction of new payment methods or changes in merchant behavior.</p><p>For instance, during holiday seasons, we typically observe higher transaction amounts and increased shopping frequency, which can temporarily alter the distribution of these features. Similarly, the COVID-19 pandemic caused dramatic shifts in online versus in-store transaction patterns, creating significant data drift that many fraud detection systems had to adapt to.</p><h4>2. Model Score Drift</h4><p>Model score drift focuses on changes in how our model assigns risk scores to transactions. This type of drift is particularly important because it directly impacts business decisions. When we observe changes in the distribution of risk scores, it might indicate that our model’s understanding of fraud patterns is shifting.</p><p>A classic example is when a model gradually begins assigning higher risk scores to a larger proportion of transactions. This could indicate several things: either fraud patterns are evolving and becoming more sophisticated, or our model’s calibration is drifting from its original state. The stability of decision thresholds becomes crucial here — if we notice that our previously established thresholds are leading to significantly different rejection rates, this might indicate score drift that requires attention.</p><h4>3. Concept Drift</h4><p>Concept drift represents perhaps the most challenging type of drift in fraud detection. It occurs when the fundamental relationship between our features and fraud patterns changes. This is particularly relevant in fraud detection because fraudsters actively adapt their strategies to bypass detection systems.</p><p>For example, if fraudsters discover that large transaction amounts trigger more scrutiny, they might switch to making multiple smaller transactions. This changes the relationship between transaction amount and fraud probability — a concept drift that our model needs to adapt to. Similarly, the emergence of new fraud techniques, such as synthetic identity fraud, can create entirely new patterns that existing models might not be equipped to detect.</p><h3>Monitoring Approaches for Different Types of Drift</h3><p>Each type of drift requires specific monitoring approaches and statistical tests to detect and measure changes effectively.</p><h4>Data Drift Detection</h4><p>For monitoring feature distributions, we can employ several statistical methods depending on the nature of the data:</p><p>For continuous variables, the Kolmogorov-Smirnov test helps us identify significant changes in distribution shapes. This is particularly useful for features like transaction amounts or frequencies. The test compares the cumulative distribution functions of our reference and current datasets, helping us identify when feature distributions have significantly shifted.</p><p>For categorical features, such as merchant categories or payment types, we use chi-squared tests to detect changes in the distribution of categories. This helps us identify when certain categories become more or less prevalent than expected.</p><p>When dealing with complex probability distributions, Jensen-Shannon divergence provides a sophisticated measure of similarity between distributions, helping us detect subtle changes that might not be apparent with simpler tests.</p><h4>Model Score Drift Analysis</h4><p>Model score drift requires a comprehensive monitoring approach that looks at:</p><p>Score Distribution Stability: We track how the overall distribution of risk scores changes over time. This includes monitoring both the shape and central tendencies of the score distribution.</p><p>Rejection Rate Analysis: We carefully monitor how rejection rates evolve across different score bands. This helps us understand if our model’s risk assessment is becoming more or less conservative over time.</p><p>Threshold Impact: We analyze how our decision thresholds perform over time, including their effectiveness in separating fraudulent from legitimate transactions based on the labels we do receive.</p><h4>Concept Drift Monitoring</h4><p>Monitoring concept drift requires a combination of quantitative and qualitative approaches:</p><p>Performance Metrics: We track metrics on transactions where we do receive feedback, understanding that this represents a biased sample but can still provide valuable signals.</p><p>Business Metric Correlation: We analyze how changes in model predictions correlate with business outcomes, helping us identify when our model’s understanding of fraud patterns might be degrading.</p><p>Expert Review: Regular review of false positives and false negatives by fraud analysts helps identify new patterns that the model might be missing or misinterpreting.</p><p>Pattern Analysis: Systematic analysis of emerging fraud patterns through case studies and fraud reports helps us stay ahead of evolving fraud strategies.</p><h3>Population Stability Index: A Robust Monitoring Tool</h3><p>The Population Stability Index (PSI) has emerged as a particularly valuable tool for model monitoring in production. PSI measures the stability of distributions over time, making it especially useful for tracking both feature and score distributions. The metric works by comparing the distribution of a variable between two time periods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*LrYzbSiGgQmhefHslBl8OQ.png" /></figure><p>where:</p><ul><li><em>Actual%_i</em> is the percentage of observations in bin i in the current period</li><li><em>Expected%_i</em> is the percentage of observations in bin i in the baseline period</li></ul><p>PSI Interpretation:</p><ul><li>PSI &lt; 0.1: No significant distribution change</li><li>0.1 ≤ PSI &lt; 0.2: Moderate distribution change</li><li>PSI ≥ 0.2: Significant distribution change</li></ul><p>However, it’s important to note that PSI’s effectiveness depends heavily on how you set up your analysis. The choice of bin sizes for continuous variables can significantly impact the results, and thresholds for acceptable PSI values should be carefully calibrated based on your specific use case and risk tolerance.</p><h3>Integrating Business Context</h3><p>Technical metrics alone aren’t sufficient for comprehensive model monitoring. A robust monitoring system must incorporate business metrics that provide context and help interpret the technical signals.</p><h4>Fraud Rate (FR) Monitoring</h4><p>FR monitoring is crucial for understanding the real-world impact of your model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*_Q-yy9DdhnyM7GD0CrZ6BQ.png" /></figure><p>An unexpected increase in fraud rate might indicate new fraud patterns that your model isn’t catching. However, it’s important to distinguish between genuine performance degradation and temporary spikes due to coordinated fraud attacks.</p><h4>Conversion Rate (CR) Analysis</h4><p>CR monitoring helps ensure that your fraud prevention measures aren’t creating excessive friction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*97TWE3vrsd5xtU10CvDT_g.png" /></figure><p>A declining conversion rate might indicate that your model has become too conservative, potentially harming the business while trying to prevent fraud.</p><h3>Best Practices for Implementation</h3><h4>Establishing Your Monitoring Framework</h4><p>A successful monitoring system starts with proper baseline establishment. When deploying a new model, thoroughly document its initial performance characteristics across all relevant metrics. This baseline serves as a reference point for detecting future degradation.</p><p>Create a comprehensive monitoring schedule that defines when and how often different metrics should be evaluated. Some metrics might need real-time monitoring, while others can be evaluated on a daily or weekly basis.</p><h4>Multi-layered Monitoring Approach</h4><p>Implement monitoring at multiple levels to create a comprehensive view of your model’s health:</p><ol><li>Technical Monitoring:<br>- Input feature distributions<br>- Model score distributions<br>- Data quality metrics<br>- Performance metrics where available<br>- Resource utilization</li><li>Business Monitoring:<br>- Fraud losses<br>- Customer friction metrics<br>- Acceptance rates<br>- Manual review rates<br>- Chargeback rates</li><li>Operational Monitoring:<br>- Model latency<br>- System throughput<br>- Error rates<br>- API response times<br>- Resource utilization</li></ol><h3>Action Planning</h3><p>Develop clear, documented procedures for responding to different types of alerts. These procedures should specify:</p><ul><li>The threshold levels that trigger different types of responses</li><li>Who needs to be notified when issues are detected</li><li>What immediate actions should be taken to mitigate risks</li><li>Criteria for when model retraining or redeployment is necessary</li></ul><h4>Response Framework Example:</h4><p><strong>1️⃣ Alert Levels:</strong></p><ul><li>Level 1 (Warning): Single metric outside normal range</li><li>Level 2 (Concern): Multiple metrics showing deviation</li><li>Level 3 (Critical): Significant performance degradation</li></ul><p><strong>2️⃣ Response Actions:</strong></p><ul><li>Immediate investigation of root cause</li><li>Temporary adjustment of decision thresholds</li><li>Emergency model retraining if necessary</li><li>Stakeholder communication</li></ul><h3>Conclusion</h3><p>While traditional model performance metrics may be limited in production fraud detection systems, a comprehensive monitoring approach focusing on drift detection, business metrics, and population stability can effectively track model health and trigger timely interventions when needed.</p><p>The key to successful model monitoring in fraud detection lies in combining multiple approaches and maintaining a balance between technical and business metrics. By implementing a robust monitoring framework that accounts for the unique challenges of fraud detection, organizations can ensure their systems remain effective and adaptable to changing patterns.</p><h3><strong>Further Reading:</strong></h3><ul><li><a href="https://medium.com/ai-enthusiast/keeping-your-models-relevant-methods-and-metrics-for-drift-detection-in-production-f6df9fe0e35b">Methods and Metrics for Drift Detection in Production</a></li><li><a href="https://towardsdatascience.com/statistical-methods-for-machine-learning-model-monitoring-3c4434cfc454">Statistical Methods for Machine Learning Model Monitoring</a></li><li><a href="https://www.listendata.com/2015/05/population-stability-index.html">Understanding Population Stability Index (PSI) in Model Monitoring</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=477ac99760f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fighting Decay: The Subtle Art of Keeping Fraud Models Relevant]]></title>
            <link>https://medium.com/@valeria.verzi1/fighting-decay-the-subtle-art-of-keeping-fraud-models-relevant-cb416d19cab6?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb416d19cab6</guid>
            <category><![CDATA[stability]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Tue, 30 Dec 2025 13:30:23 GMT</pubDate>
            <atom:updated>2025-12-30T13:30:23.034Z</atom:updated>
            <content:encoded><![CDATA[<p>How to detect drift, monitor stability, and know when your model is losing its edge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPnyqilv6tangdFMRm3WGw.png" /></figure><h3>Introduction</h3><p>So far in this series, we’ve focused on building fraud detection models that make sense <strong>offline</strong>: choosing the right metrics, handling class imbalance, respecting time, and avoiding the many ways models can overfit or leak information from the future.</p><p>But even if you get all of that right, there’s still a hard truth waiting in production: <strong>models age</strong>.</p><p>A fraud model can launch with solid validation results, reasonable generalization, and no obvious signs of overfitting — and still degrade quietly over time. Fraud patterns evolve, user behavior shifts, and operational constraints change. What once worked well can slowly lose its edge.</p><p>This doesn’t automatically mean the model was wrong or poorly designed. In many cases, it reflects a deeper question: <em>is the model still stable under changing conditions?</em></p><p>In this article, we focus on <strong>model stability in fraud detection systems</strong>. We’ll look at how performance changes over time, how different types of drift manifest in practice, and which signals help you distinguish between normal variation and genuine degradation. The goal isn’t just to detect failure, but to understand <strong>when a model is losing relevance — and what to do about it</strong>.</p><h3>ROC and PR curves after production</h3><p>Remember how we discussed PR curves being your friend in model evaluation? Well, they’re even more important in production. While ROC curves might give you a false sense of security, PR curves will tell you the real story about how your model is performing.</p><h3>Why ROC Curves Can Be Deceptive in Production</h3><p>ROC curves can be misleading in production — they remain stable even when the model’s real-world performance degrades significantly. They are like that friend who always says everything is fine. Here are two common scenarios where they can trick you:</p><ol><li>Changing Fraud Rates</li></ol><ul><li>ROC curves might stay stable even as fraud rates change</li><li>Translation: Your model might look stable, but it’s actually struggling</li><li>PR curves will show you the real impact on your business</li></ul><ol><li>Population Drift</li></ol><ul><li>When fraud patterns change but overall fraud rates stay similar</li><li>ROC curves might maintain their shape, suggesting stable performance</li><li>Translation: Fraudsters are getting smarter, but your metrics aren’t showing it</li><li>PR curves reveal if your model is still catching the new fraud patterns</li></ul><h4>Model Performance Across Different Fraud Rates</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQF1sMYVrkEFx9fBNc-6pA.png" /></figure><p>I put our model through its paces with different fraud rates:</p><ul><li>Original fraud rate (baseline)</li><li>2x fraud rate</li><li>4x fraud rate</li><li>8x fraud rate</li></ul><p>What I found:</p><ol><li>ROC curves stayed pretty stable across different fraud rates</li><li>PR curves showed the real story — they’re much more sensitive to changes</li><li>Higher fraud rates gave better PR-AUC scores, but don’t get too excited — this might not reflect real-world performance</li></ol><h4>Impact of Drifting Fraud Patterns</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3tyu9sqOSXu-8jiCR_5Kw.png" /></figure><p>Fraudsters are like chess players — they’re always thinking several moves ahead. That’s why monitoring how your model handles changing fraud patterns is crucial.</p><p>I simulated real-world drift scenarios by creating different versions of our fraud population:</p><ol><li>Original patterns (baseline)</li><li>Positive shift (increasing feature values)</li><li>Negative shift (decreasing feature values)</li><li>Mixed shift (random direction per feature)</li></ol><p>The results were eye-opening:</p><ul><li>Model performance varied significantly with shifting fraud patterns</li><li>Some shifts were sneakier than others to detect</li><li>Regular model retraining and monitoring became crucial</li></ul><h3>The Role of PR Curves in Production Monitoring</h3><p>PR curves are particularly valuable for production monitoring because they:</p><ul><li>Provide early warning signs of model degradation</li><li>Help identify when retraining is necessary</li><li>Give clear insights into the model’s ability to detect new fraud patterns</li><li>Better reflect the business impact of model changes</li></ul><h3>True Positive Rate vs False Positive Rate Trade-off</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHyVWiXVkrOokjnVZyXRSQ.png" /></figure><p>In fraud detection, there’s always a trade-off between:</p><ul><li>Catching more fraud (higher TPR)</li><li>Minimizing false alarms (lower FPR)</li></ul><p>The optimal operating point depends on business factors:</p><ul><li>Cost of investigating false positives</li><li>Cost of missing fraud</li><li>Customer experience impact</li><li>Regulatory requirements</li></ul><h3>Key Takeaways</h3><ol><li><strong>Don’t Trust Accuracy Alone</strong>: In unbalanced datasets, accuracy can be misleading. Use precision, recall, and F1 score.</li><li><strong>PR Curves &gt; ROC Curves</strong>: For fraud detection, PR curves and PR-AUC provide more meaningful insights than ROC curves.</li><li><strong>Monitor Performance Changes</strong>: Both fraud rates and patterns change over time. Regular monitoring using appropriate metrics is crucial.</li><li><strong>Consider Business Context</strong>: Choose operating thresholds based on business constraints and costs, not just mathematical optimization.</li></ol><h3>Conclusion</h3><p>Building a stable fraud detection system is like maintaining a high-performance car — you need to keep an eye on multiple indicators and make adjustments before things go wrong. By understanding how different metrics behave in production and choosing the right ones to monitor, you can keep your fraud detection system running smoothly.</p><p><strong>Remember</strong>: The goal isn’t just to optimize metrics, but to build a robust system that effectively fights fraud while keeping your customers happy. Stay vigilant, keep monitoring, and don’t be afraid to retrain when needed!</p><p>Do you want to know more on fradu modeling topics? Check out our other articles on model evaluation, sampling strategies, and time series analysis!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb416d19cab6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting the Past: The Subtle Art of Not Cheating Time]]></title>
            <link>https://medium.com/@valeria.verzi1/predicting-the-past-the-subtle-art-of-not-cheating-time-225d82f7b001?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/225d82f7b001</guid>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[fraud-prevention]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-leak]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 19:33:27 GMT</pubDate>
            <atom:updated>2025-10-31T19:33:27.563Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to design fraud-detection features that stay true to real-world data and avoid time leakage.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TNSTp1V5Y33eaU3d2RmLA.png" /><figcaption>Image AI Generated with <a href="https://deepai.org/">https://deepai.org/</a></figcaption></figure><blockquote>“Your model looks amazing in testing! But why isn’t it catching any fraud in production?”</blockquote><p>If you’ve ever heard this (or said it yourself), you’re not alone. After a decade of building fraud detection systems, I can tell you that this is one of the most common problems I see. And surprisingly, it’s not because of complex algorithms or fancy models — most of the time it’s because of how we handle time.</p><p>In our last article, we explored how time affects data splitting. Now, let’s dig into something even trickier: how to build features that actually work in the real world, where you can’t peek into the future.</p><p>This article is part of a series where we explore different aspects of fraud detection. If you haven’t already, check out my articles on<strong> </strong><a href="https://medium.com/@valeria.verzi1/understanding-model-evaluation-metrics-in-fraud-detection-beyond-accuracy-52b224ac0418"><strong>model evaluation</strong></a><strong> </strong>and<strong> </strong><a href="https://medium.com/@valeria.verzi1/understanding-overfitting-in-fraud-detection-the-basics-79eb2795b70c"><strong>imbalance handling</strong></a><strong> — </strong><em>[model monitoring soon to come]</em><strong> </strong>— they’ll give you a solid foundation for what we’re about to discuss.</p><h3>The Time Factor in Fraud Detection</h3><p>Here’s a story that might sound familiar: A team builds a fraud detection model that looks incredible during testing — we’re talking near-perfect scores. Everyone’s excited. Then they deploy it… and suddenly their amazing model can barely catch any fraud. What happened?</p><p>It might be overfitting or the answer can lie in how we built our features. In development, it’s surprisingly easy to accidentally use information that wouldn’t be available in the real world. Think about it — when a transaction happens in production, you only have information up to that exact moment. No peeking at fraud labels that get confirmed weeks later, no using patterns that haven’t emerged yet.</p><p>But here’s the tricky part — this mistake is so common that I’ve seen it happen to everyone from junior developers to seasoned ML engineers. It’s subtle, it’s sneaky, and it can make your model look amazing in testing while setting it up for failure in the real world.</p><p>Ready to learn how to spot and fix this? Let’s dive in.</p><h3>The Label Maturity Challenge</h3><p>Let me tell you about one of the biggest gotchas in fraud detection — label maturity. Here’s what you need to know:</p><ul><li>Fraud labels can take 30, 60, or even more days to mature after a transaction</li><li>Your dataset should have a column showing when each fraud label was updated</li><li>When building past behavioral features, always check the label’s timestamp</li></ul><p>Let’s look at a real example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*qOG-yUWxmjQDzD_G-7Tehg.png" /></figure><p>See how those fraud labels take their sweet time to show up? That’s why we need to be super careful with our feature engineering.</p><h3>The Data Leakage Trap</h3><p>Here’s where it gets interesting. Let’s say you want to know how many fraudulent transactions a customer had in the past. Sounds simple, right? But watch out — there’s a trap!</p><p>Let’s look at our example data. If we want to predict for transaction c1511 (20/04/2024), here&#39;s what NOT to do:</p><pre>select sum(is_fraud) <br>from transactions<br>where transaction_date &lt; &#39;20/04/2024&#39;</pre><p>This would give you 2 fraudulent transactions. But wait! The fraud label for transaction b14891 wasn&#39;t even confirmed until 01/05/2024 - that&#39;s after our prediction date! We&#39;re using information from the future! 🚫</p><p>Here’s the right way to do it:</p><pre>select sum(is_fraud) <br>from transactions<br>where is_fraud_date &lt; &#39;20/04/2024&#39;</pre><p>Now we correctly get 1 fraudulent transaction. The image below shows this concept in action — we only use information from the green area (the past) when making our prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*mpwUGkCqCIHrgnKon24sPQ.png" /></figure><h3>The Golden Rule of Feature Engineering</h3><p>Here’s a simple trick to make sure you’re getting this right: always ask yourself, “Would this data be available at prediction time?” This small check will help you avoid data leakage every time.</p><p>Think about it like this: if you’re building a real-time fraud detection system, you can only use information that would have been available at the moment the transaction happened. It’s like trying to predict the weather — you can use yesterday’s data, but you can’t use tomorrow’s!</p><h3>Key Takeaways</h3><ol><li>Always check your label timestamps when building features</li><li>Never use information that wouldn’t be available at prediction time</li><li>Think about what information would actually be available in production</li><li>Test your features with real-world timing scenarios</li></ol><p><strong>Remember</strong>: In fraud detection, timing isn’t just important — it’s everything. Get it wrong, and you might as well be using a crystal ball for predictions!</p><p>Want to learn more about how to handle time in your fraud detection system? Check out our other articles on model evaluation metrics and data splitting strategies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=225d82f7b001" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fraud Doesn’t Flow — It Loops: Rethinking Time in Fraud Detection]]></title>
            <link>https://medium.com/@valeria.verzi1/fraud-doesnt-flow-it-loops-rethinking-time-in-fraud-detection-28ab59f733a6?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/28ab59f733a6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[carousel]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Wed, 22 Oct 2025 18:35:20 GMT</pubDate>
            <atom:updated>2025-10-22T18:35:20.106Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fraud Doesn’t Flow — It Loops: Rethinking Time in Fraud Detection</h3><p>Let’s be real — if you’re working on fraud detection, you’ve probably tried to treat it as a time series problem at some point. We’ve all been there. But here’s the thing: fraud patterns don’t follow a nice, neat timeline. They’re more like a carousel — patterns keep coming back around, but each time they’re a bit different. Let’s dive into why traditional time series approaches fall short and how to build better fraud detection systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*WUbaDjLuwH6QmkLxZJ4W_w.jpeg" /><figcaption>Time AI Generated with <a href="https://deepai.org/">https://deepai.org/</a></figcaption></figure><p>In the realm of fraud detection, a common misconception persists: treating fraud as a pure time series phenomenon. While temporal aspects play a crucial role, the reality is far more nuanced. Fraud patterns exhibit what we might call a “circular” nature, where patterns resurface with variations, rather than following strict temporal progression.</p><h3>The Circular Nature of Fraud</h3><p>Imagine a carousel rather than a straight timeline. Fraudsters often revisit successful techniques, adapting and refining them. For instance, holiday season fraud patterns may repeat annually, but with sophisticated variations that reflect learned experiences from both sides — fraudsters and detection systems.</p><h3>Why Not Pure Time Series?</h3><p>A time series is a sequence of data points collected or recorded at regular time intervals, where each observation is dependent on its temporal position. In traditional time series analysis, we assume that:</p><ol><li>Future events are directly dependent on past events</li><li>Patterns follow a consistent temporal progression</li><li>Seasonality and trends are predictable</li></ol><p>However, fraud doesn’t strictly adhere to these principles. While a customer’s past behavior might indicate fraud risk, we cannot definitively say that all previous events directly cause subsequent ones. Fraudsters are adaptive, creating an ever-evolving landscape of patterns.</p><h3>Building Robust Fraud Detection Models</h3><h4>Data Preparation: The Foundation</h4><p>When it comes to fraud detection, time isn’t just a dimension — it’s the key to understanding how fraud patterns evolve. Let’s tackle this in two interconnected steps:</p><p>First, we need to build features that capture the temporal nature of fraud. This means creating features that can spot when behavior patterns start to shift over time. But here’s where it gets tricky — you need to be super careful about not accidentally using future information to predict the past (we call this data leakage). I talk about this in other article of mine: <strong><em>feature_engineering </em></strong>[coming soon — stay tuned!].</p><p>Second — but not in order of importance or timing — we need to focus on data splitting. Remember that carousel we talked about? We need to split our data in a way that captures both the circular nature of fraud patterns and their evolution over time. Following, we focus on this key step.</p><p>Consider these critical aspects:</p><p>1. Training Window Selection</p><ul><li><strong>Historical Depth</strong>: Capture enough history to understand pattern variations</li><li><strong>Recency Balance</strong>: Include recent data to reflect current tactics</li><li><strong>Labels Maturity</strong>: Remember to consider a buffer for your labels to be mature (e.g. do not take the most recent month, go back at least 60 days)</li><li><strong>Optimal Window</strong>: Usually 6–12 months, depending on fraud velocity</li></ul><p>2. Validation Split in Fraud Detection: The Two-Pronged Approach</p><p>In fraud detection, the traditional split of data into training, validation, and test sets requires special consideration due to the temporal nature of fraud patterns and the need to prevent data leakage. More on this in the following sections.</p><h4>Validation Split</h4><p><strong>what does it serve for</strong></p><blockquote>Training: Model development<br>Validation: Model selection &amp; hyperparameter tuning<br>Test: Final performance evaluation</blockquote><h3><strong>Standard validation split</strong></h3><p>The most common structure used for timeseries events maintains temporal ordering and has no random splitting across time periods:</p><blockquote>Timeline:<br>[Training Set (60%)] -&gt; [Validation Set (20%)] -&gt; [Test Set (20%)]</blockquote><ul><li>Example: For 1 year of data</li></ul><blockquote>Training: Jan-Jul<br>Validation: Aug-Sep<br>Test: Oct-Dec</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIzkRWivKEUm256OHE7GLA.jpeg" /></figure><p>This way of splitting the dataset makes sense only if you see or assume strong causality and time dependency in your dataset. The model gets always tested on the future data, answering to the question: how good is my model as predicting next event?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CmrRjn92BqNS64i2rd5nAQ.jpeg" /></figure><h3>Validation Split in Fraud Detection: The Two-Pronged Approach</h3><p>In fraud, these assumptions are really stretched. As we saw in the introduction, fraud patterns are cyclical and adaptive rather than following traditional time series progression, making temporal-only approaches insufficient for effective detection.</p><p><strong>How can we split the dataset to be effective, then?</strong></p><p>The fundamental concept to remember is that you want to test two aspects when looking at model performance:</p><ol><li>How good the model can explain the patterns he already seen.</li><li>How stable the model is over time.</li></ol><p>With this two in mind, the image below should become clear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFbkhADy5TFRK1iENxCjjg.jpeg" /></figure><p><strong>Out-of-Time (OOT) Validation:</strong> An out of time set (oot) that the model has never seen (in the future wrt training set)</p><blockquote>Training Data (Jan-Jun) → Future Data (Jul-Sep)<br>Purpose: Test model robustness against new patterns</blockquote><p><strong><em>Example code</em></strong></p><pre>_expression = &quot;trans_date_trans_time &lt; &#39;2020-07-01 00:00:00&#39;&quot;<br>train = data.query(_expression)<br><br>oot = data.query(f&quot;~({_expression})&quot;)<br>oot_y = oot.is_fraud<br>oot_X = oot.drop(columns=[&#39;is_fraud&#39;])</pre><p><strong>In-Time Holdout:</strong> A validation set sampled from the training population. This serves to test that the model is able to generalize between what it has already seen. For simplicity, I prefer calling this set Holdout to avoid confusion with out-of-time validation set and cross-validation test set.</p><blockquote>Training Data (e.g., Jan-Jun) → Random Validation Sample<br>Purpose: Test pattern generalization within known timeframe</blockquote><p><strong><em>Example code</em></strong></p><pre>X = train.drop(columns=[&#39;is_fraud&#39;])<br>X.drop(metadata_columns, axis=1, inplace=True)<br>y = train[&#39;is_fraud&#39;]<br><br>train_X, holdout_X, train_y, holdout_y = train_test_split(X, y, test_size=0.2, random_state=42)</pre><h3>Validation Checks</h3><h4><strong>Volume Check</strong></h4><ul><li>Ensure sufficient fraud cases in each split</li><li>Maintain class distribution across splits</li></ul><h4><strong>Pattern Verification</strong></h4><ul><li>Check for pattern stability across splits</li><li>Verify business cycle representation</li></ul><pre>def validate_split(train, val, test):<br>    &quot;&quot;&quot;<br>    Verify split quality<br>    &quot;&quot;&quot;<br>    checks = {<br>        &quot;Size Ratios&quot;: check_size_distribution(train, val, test),<br>        &quot;Class Distribution&quot;: check_class_distribution(train, val, test),<br>        &quot;Temporal Order&quot;: verify_temporal_order(train, val, test),<br>        &quot;No Overlap&quot;: verify_no_overlap(train, val, test)<br>    }<br>    return checks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xslx8p9yXpbXTRXQP7euMw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28ab59f733a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting in Fraud Detection, Part 2: Advanced Techniques]]></title>
            <link>https://medium.com/@valeria.verzi1/overfitting-in-fraud-detection-part-2-advanced-techniques-72ddb44ffd92?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ddb44ffd92</guid>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[k-fold-cross-validation]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 16:32:15 GMT</pubDate>
            <atom:updated>2025-10-13T16:32:15.949Z</atom:updated>
            <content:encoded><![CDATA[<h4>So you’ve tried the basic approaches to handling overfitting in fraud detection, but you’re still not getting the results you need. Let’s dive into a more sophisticated solution that combines strategic sampling with robust cross-validation — the Ensemble Resampling technique.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Cyg14AraB33gHUQUNQirsw.jpeg" /><figcaption>Image AI generated with <a href="https://deepai.org/">https://deepai.org/</a></figcaption></figure><p><strong>If you missed the first article</strong> in this chapter on overfitting, I invite you to take a look <a href="https://medium.com/@valeria.verzi1/understanding-overfitting-in-fraud-detection-the-basics-79eb2795b70c">here</a> — it’s our starting point for this more advanced discussion. That piece covers the fundamentals and sets the context for what we’re diving into today.</p><p>If you’ve already read it, then let’s jump right in — I hope you enjoy this next step in the journey!</p><h3>The Advanced Approach: Cross-Validation with Dynamic Downsampling</h3><p>This is where it gets interesting — the state-of-the-art approach for fraud modeling. The problem with doing downsampling once for all CV is that we lose variability and information on the negative class. The solution? Sample in each iteration of cross-validation. This technique was discovered in the paper <a href="https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/tsmcb09.pdf">“Exploratory Undersampling for Class-Imbalance Learning”</a>.</p><p>This method can be implemented in many ways. For big parameter spaces Optuna works best (please, have a look at my <a href="https://github.com/valever/Fraud-Modeling/blob/main/notebooks/model_training/03_optuna_model_training.ipynb">repo</a> for this). For the sake of simplicity, I will show the way using <em>imblearn</em> and <em>scikit learn</em>, which requires less code and it is more self contained then the <em>optuna</em> way.</p><p>Here’s how we implement it:</p><pre>from imblearn.pipeline import Pipeline<br>from lightgbm import LGBMClassifier<br>from sklearn.model_selection import GridSearchCV<br><br>hyperparameters = {<br>        &#39;class__n_estimators&#39;: [20, 50, 200, 500],<br>        &quot;class__objective&quot;: [&quot;binary&quot;],<br>        &quot;class__early_stopping_round&quot;: [10],<br>        &quot;class__num_leaves&quot;: [5, 10, 80, 100],<br>        &quot;class__min_data_in_leaf&quot;: [10, 50, 100, 200],<br>    }<br>undersample_pipe = Pipeline([(&#39;sampling&#39;, RandomUnderSampler(sampling_strategy=0.1, random_state=42)) <br>                             , (&#39;class&#39;, LGBMClassifier(objective=&#39;binary&#39;))])<br>score_balanced_parameter_model = GridSearchCV(undersample_pipe, param_grid=hyperparameters, cv=3, scoring=&#39;average_precision&#39;)<br>score_balanced_parameter_model.fit(train_X, train_y, class__eval_set=(holdout_X, holdout_y))</pre><h3>Model Performance</h3><p>Let’s see how this advanced approach performs:</p><p><strong>Metrics Comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*v54R1PbrIlKhx89RMyeRpg.png" /></figure><p>This is where we start seeing real improvements — let’s compare the 3 experiments of the chapter:</p><ol><li>Balanced Precision-Recall Trade-off:<br>- Precision starts at 0.55, more balanced than our previous attempts<br>- Recall maintains high performance (1.00 → 0.92 → 0.85)<br>- Better balance between false positives and missed frauds</li><li>Rock-Solid Stability:<br>- Accuracy remains remarkably stable (only 0.04% drop)<br>- Precision decay is more controlled: 0.55 → 0.52 → 0.42<br>- Recall maintains strong performance even in OOT (0.85)</li><li>Robust Learning:<br>- F1 score shows gradual decline (0.71 → 0.66 → 0.56)<br>- Performance drops are more predictable<br>- Better generalization to unseen patterns</li></ol><p><strong>Precision Recall Curves</strong></p><figure><img alt="pr curves" src="https://cdn-images-1.medium.com/max/1024/1*CJUicAc18YzDN7osVUhwYg.png" /></figure><p>The PR curve shows these improvements:</p><ul><li>Smoother transitions between training and holdout curves</li><li>More consistent area under the curve across datasets</li><li>Better maintained precision-recall relationship in OOT</li></ul><h3>Considerations</h3><p>This approach is the real deal. Compared to our previous attempts: vs. Baseline: More stable metrics, better fraud detection, less severe overfitting vs. Simple Downsampling: Better precision, more balanced metrics, improved generalization</p><p>The dynamic CV approach shows superior handling of overfitting through:</p><ul><li>More realistic training performance</li><li>Better retention of model capabilities in OOT</li><li>More balanced precision-recall trade-off</li><li>Consistent performance across different time periods</li></ul><h3>Advanced Best Practices</h3><p>Here’s what you need to remember when implementing this advanced approach:</p><ol><li>Use Stratified Cross-Validation</li></ol><pre>skf = StratifiedKFold(n_splits=self.n_splits, shuffle=True, random_state=42)</pre><ol><li>Implement Sampling in Stratified KFold This ensures proper sampling within each fold while maintaining the class distribution.</li><li>Monitor Multiple Metrics</li></ol><pre>eval_metric=[&quot;average_precision&quot;, &quot;auc&quot;]</pre><ol><li>Consider Alternative Methods While this approach works well, there are other options worth exploring:</li></ol><ul><li>BalancedBaggingClassifier</li><li>SMOTE and other oversampling techniques</li><li>Hybrid approaches combining multiple sampling methods</li></ul><h3>Conclusion</h3><p>Remember:</p><ul><li>Never downsample validation or test sets</li><li>Implement sampling within cross-validation folds</li><li>Use appropriate metrics for imbalanced classification</li><li>Consider the business impact of false positives vs. false negatives</li></ul><p>This advanced approach might require more computational resources and careful tuning, but the improved stability and performance make it worth the effort for production fraud detection systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ddb44ffd92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Overfitting in Fraud Detection: The Basics]]></title>
            <link>https://medium.com/@valeria.verzi1/understanding-overfitting-in-fraud-detection-the-basics-79eb2795b70c?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/79eb2795b70c</guid>
            <category><![CDATA[imbalanced-data]]></category>
            <category><![CDATA[fraud]]></category>
            <category><![CDATA[fraud-prevention]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Tue, 07 Oct 2025 16:02:07 GMT</pubDate>
            <atom:updated>2025-10-13T11:22:00.465Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overfitting in Fraud Detection, Part 1: The Basics</h3><h4>Picture this: You’ve spent weeks building your fraud detection model. The training metrics look fantastic — 99% accuracy! Your stakeholders are impressed, and you’re ready to deploy. But then reality hits: your model starts flagging legitimate transactions left and right, missing actual fraud cases, and generally making a mess of things. Sound familiar? You’re not alone. This is the classic tale of overfitting in fraud detection, where our models become too “clever” for their own good.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kR_kSuYoXYERfonWVJs9tg.png" /><figcaption>Image AI generated with <a href="https://deepai.org/">https://deepai.org/</a></figcaption></figure><h3>The Challenge: Imbalanced Data in Fraud Detection</h3><p>The problem isn’t just about model complexity — it’s about the unique nature of fraud data. We’re dealing with extremely imbalanced datasets where fraud cases are like needles in a haystack (and the haystack is on fire). In our dataset spanning 2019–2020, we observed months where only 258 fraudulent transactions occurred among 139,538 total transactions (0.18%). And get this — the European Bank Authority reported in their 2024 report that in Q1 2023, fraud was only 0.015% of total card payments. That’s not just imbalanced — that’s severely imbalanced!</p><p>But wait, it gets worse. Our “non-fraud” class isn’t even clean — some fraudulent activities might be hiding in there, undetected due to sophisticated schemes or delayed discovery. This means we’re training our models on potentially noisy data, which is like trying to learn chess while some pieces keep changing their moves.</p><p>Technically, this creates three major headaches:</p><ul><li>Models get lazy and just predict “not fraud” for everything</li><li>Traditional accuracy metrics become completely useless</li><li>The model might miss important fraud patterns entirely</li></ul><p>Quick heads up before we dive in: While we’re focusing on the most widely-used sampling technique here, there are plenty of other ways to tackle class imbalance. Don’t feel boxed in — you might find that a different approach (or even a combination of methods) works better for your specific fraud detection challenge. The key is to experiment and find what fits your use case best!</p><h3>Three Approaches to Model Building</h3><p>Class imbalance is like having a biased referee in a game — it makes the whole system unfair. When we don’t address it, our models become overly optimistic and miss critical fraud patterns. But here’s the catch: fixing it with basic resampling can create its own problems, like overfitting and unstable performance.</p><p>To demonstrate the evolution of fraud detection modeling and its best practices, let’s look at three different approaches, each building on what we learned from the previous one:</p><ol><li>The Baseline Approach: A standard classification model without any sampling techniques (spoiler: it’s not great)</li><li>The Basic Debiasing Solution: Simple downsampling to address class imbalance (better, but still not perfect)</li><li>The Advanced Cross-Validation Framework: A sophisticated solution that combines strategic sampling with robust cross-validation (this is where it gets interesting)</li></ol><p>Following, we will see the first two approaches, while a separate article deeps dive into the more advanced third methodology — feel free to jump to the article if the others are too basic for you [COMING SOON].</p><p>As we walk through each approach, we’ll not only learn how to build better fraud detection models, but also get a feel for why some techniques work better than others. Think of it as a journey from “meh” to “better” — we’ll see how each step helps us build more stable and reliable models.</p><h3>1. Standard Model (No Debiasing)</h3><p>First up, let’s look at what happens when we don’t do anything about the imbalance or the parameter tuning. This will serve as our baseline and show us exactly why we need better approaches.</p><pre># Data preparation<br>X = train.drop(columns=[&#39;is_fraud&#39;])<br>X.drop(metadata_columns, axis=1, inplace=True)<br>y = train[&#39;is_fraud&#39;]<br><br># Basic train-test split<br>train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.2, random_state=42)<br># Model configuration<br>hyperparameters = {<br>    &quot;objective&quot;: &quot;binary&quot;,<br>    &quot;n_estimators&quot;: 500,<br>    &quot;early_stopping_round&quot;: 10,<br>    &quot;first_metric_only&quot;: True,<br>    &quot;num_leaves&quot;: 10,<br>    &quot;min_data_in_leaf&quot;: 20<br>}<br># Train the model<br>lgbm_model = LGBMClassifier(**hyperparameters)<br>model = lgbm_model.fit(X=train_X,<br>                      y=train_y,<br>                      eval_set=[(test_X, test_y)],<br>                      eval_metric=&#39;average_precision&#39;)</pre><h4>Model Performance:</h4><p>Let’s see how this basic approach performs across different datasets:</p><p>Metrics Comparison</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnSSVg4G7u2R7zNOUiDmdg.png" /></figure><p>This is a classic case of overfitting in fraud detection:</p><ol><li>That Deceptive Accuracy: Looks great, right? But remember our earlier warning about accuracy in imbalanced datasets…The model is essentially “mastering” the prediction of non-fraudulent cases,</li><li>The Precision Plunge: Watch that precision drop from 83% to 45% — that’s our model’s ability to correctly identify fraud going down the drain as he sees unseen data,</li><li>Recall Reality Check: The declining recall means we’re missing more and more actual fraud cases as we move from training to out-of-time data.</li></ol><p>Precision Recall Curves</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1lWLyIF3Yxf_Ft3e77AgVQ.png" /></figure><p>The Precision-Recall Area Under Curve (PR-AUC) plot tells a clear story: our model’s performance is falling apart when it hits real-world data. The area under the curve drops from 0.8 to 0.4 from training to out-of-time data — that’s like going from an A to an F in model performance. The steep decline in the curve for the OOT dataset indicates that the model cannot maintain high precision without severely sacrificing recall, making it impractical for real-world fraud detection where both metrics are crucial.</p><h4>Considerations:</h4><p>This model is about as reliable as a chocolate teapot. The huge gap between training and OOT metrics means it’s learning patterns that don’t generalize well to new fraud patterns, a critical flaw in fraud detection where patterns evolve rapidly. And that 63.87% recall in OOT? That means we’re missing about 36% of actual fraud cases — exposing the business to significant financial risk.</p><p>This analysis clearly demonstrates why a standard modeling approach is insufficient for fraud detection. The combination of class imbalance and the temporal nature of fraud patterns requires more sophisticated techniques, which we’ll explore in our next approaches.</p><p><strong>Note (to self / caveat):</strong> I’m not a huge fan of using the scale_pos_weight parameter in LightGBM, because combining those with other hyperparameters can distort the model’s output and lead to <strong>poor calibration</strong> of individual class probabilities. According to the LightGBM docs, if you mix in class weights or scale_pos_weight, the resulting probability estimates may be unreliable. <a href="https://lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMClassifier.html">lightgbm.readthedocs.io</a></p><p>If you go this route, you’ll probably want to <strong>recalibrate</strong> your model (e.g. via Platt scaling or isotonic regression) afterward to get meaningful probabilities.</p><h3>2. Simple Downsampling</h3><p>Now let’s try something smarter — downsampling the majority class. This is like leveling the playing field by reducing the number of non-fraud cases we train on:</p><pre># Import necessary libraries<br>from imblearn.under_sampling import RandomUnderSampler<br>from sklearn.model_selection import train_test_split<br><br># Initialize the undersampler with a sampling ratio of 0.2 (1:5 fraud to non-fraud ratio)<br>undersample_func = RandomUnderSampler(sampling_strategy=0.2, random_state=42)<br># Apply undersampling to the training data only<br>X_train_resampled, y_train_resampled = undersample_func.fit_resample(X_train, y_train)<br># Train the model on resampled data, with the same hyperparameters as baseline<br>lgbm_model = LGBMClassifier(**hyperparameters)<br>model = lgbm_model.fit(X=X_train_resampled,<br>                      y=y_train_resampled,<br>                      eval_set=[(X_test, y_test)],<br>                      eval_metric=&#39;average_precision&#39;)</pre><p>This approach:</p><ul><li>Improves fraud detection capability</li><li>Reduces bias toward the majority class</li><li>But may lose important information from the majority class</li></ul><p>The sampling ratio of 0.2 means we keep all fraud cases and randomly sample non-fraud cases to achieve a 1:5 ratio, which significantly reduces class imbalance while still maintaining some of the natural class distribution’s characteristics.</p><h4>Model Performance:</h4><p>Let’s see how this improved approach performs:</p><p><strong>Metrics Comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7jGec_TdS8jquvx2POsR7Q.png" /></figure><p>This is much better! Let’s break down what we’re seeing:</p><ol><li>More Stable Precision:<br>- Starts at a more realistic 0.36 (vs. baseline’s 0.83)<br>- Performance degradation is more gradual: only 6% drop (vs. baseline’s 38% drop)</li><li>Better Fraud Detection:<br>- Maintains high recall across all sets (only -6% drop)<br>- Much better than baseline’s OOT recall of 0.64<br>- Model keeps detecting frauds even as patterns evolve</li><li>Smoother Performance Decay:<br>- <em>Precision</em>: 0.35 → 0.34 → 0.30 (gradual decline)<br>- <em>Recall</em>: 0.97 → 0.50 → 0.45 (controlled degradation) Compared to baseline’s sharp drops, these smoother transitions indicate better generalization</li></ol><p><strong>Precision Recall Curves</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZP8GzoAoP1fzbdO5b7DDQ.png" /><figcaption>PR Recall curve behavior on a balanced models</figcaption></figure><p>The PR curve shows these improvements:</p><ul><li>Smaller gap between training and holdout curves</li><li>More gradual performance decline in OOT</li><li>Better maintained relationship between precision and recall across datasets</li></ul><h4>Considerations</h4><p>While the lower precision might seem concerning, this model is actually more reliable than our baseline. It’s learning more generalizable patterns and maintaining better stability across different datasets. Sure, it’s not perfect (that perfect training recall is a bit suspicious), but it’s a huge improvement over our first attempt.</p><p>This analysis demonstrates that while simple downsampling isn’t perfect (note the perfect training recall), it significantly reduces overfitting compared to the baseline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79eb2795b70c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Model Evaluation Metrics in Fraud Detection: Beyond Accuracy]]></title>
            <link>https://medium.com/@valeria.verzi1/understanding-model-evaluation-metrics-in-fraud-detection-beyond-accuracy-52b224ac0418?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/52b224ac0418</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[performance-metrics]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[model-performance]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 10:31:51 GMT</pubDate>
            <atom:updated>2025-10-01T10:31:51.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWk7AOhkpcJ8EuUF-6oiFQ.png" /><figcaption>AI Generated with <a href="https://deepai.org/">https://deepai.org/</a></figcaption></figure><p>Let’s be real — if you’re reading this, you’ve probably had that moment where you had to explain to stakeholders — or to yourself — why your model’s 99% accuracy is actually terrible.</p><p>We’ve all been there. Let’s dive into why traditional metrics fail us in fraud detection and what actually works in production.</p><h3>The Challenge: When 99% Accuracy is Actually Bad</h3><p>In fraud detection, achieving 99% accuracy is not just insufficient — it’s a red flag. Let’s examine why:</p><h3>The Numbers Game</h3><p>In our dataset spanning 2019–2020, we observed months with only 258 fraudulent transactions among 139,538 total transactions (0.18%). The European Bank Authority’s 2024 report shows even more extreme imbalance: fraud represents only 0.015% of total card payments in Q1 2023. This severe class imbalance fundamentally changes how we must evaluate model performance.</p><h3>Why Traditional Metrics Lie</h3><p>Here’s a fun thought experiment: A model that simply predicts “not fraud” for all transactions would achieve:</p><ul><li>Accuracy: 99.985%</li><li>Precision: Undefined (0/0)</li><li>Recall: 0%</li><li>F1 Score: 0%</li></ul><p>While technically achieving high accuracy, such a model would be completely ineffective for fraud detection. It’s like having a security guard who just waves everyone through — technically very efficient, but not exactly doing their job.</p><h3>The Real Metrics That Matter</h3><p>Lets have a look at the confusion matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8G3MJHj5fK5Cq2fw.png" /></figure><p>The confusion matrix above illustrates the fundamental components of model evaluation:</p><ul><li><strong>True Positives (TP)</strong>: Correctly identified fraud cases</li><li><strong>False Positives (FP)</strong>: Legitimate transactions incorrectly flagged as fraud</li><li><strong>True Negatives (TN)</strong>: Correctly identified legitimate transactions</li><li><strong>False Negatives (FN)</strong>: Missed fraud cases</li></ul><h3>Precision, Recall, and F1 Score: Core Evaluation Metrics</h3><p>Let’s examine these metrics in the context of fraud detection:</p><p><strong>Precision</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/1*Zd0_MLIjRINHebWxok0LNw.png" /></figure><ul><li>High precision indicates few false positives</li><li>Critical for operational efficiency</li><li>Impacts investigation resource allocation</li><li>Translation: How many of your fraud alerts are actually fraud?</li></ul><p><strong>Recall</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*46pNKev0tXLPDMTQGPTOqA.png" /></figure><ul><li>High recall indicates comprehensive fraud detection</li><li>Directly impacts financial risk</li><li>Essential for regulatory compliance</li><li>Translation: How many actual fraud cases did you catch?</li></ul><p><strong>F1 Score:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*uBZosC5QMcI3FYZsdvz9qw.png" /></figure><ul><li>Harmonic mean of precision and recall</li><li>Balances detection capability with operational efficiency</li><li>Provides a single metric for model comparison</li><li>Translation: The sweet spot between catching fraud and not driving your team crazy with false alarms</li></ul><h3>Real-World Example: Model Performance Analysis</h3><p>Our model’s performance across different datasets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*kcKPD_7UjuBMRBZ1PWSw1Q.png" /></figure><p>This data perfectly illustrates why accuracy alone is misleading:</p><ol><li><strong>High Accuracy (98–99%)</strong>: Looks impressive but doesn’t tell the whole story</li><li><strong>Moderate Precision (29–36%)</strong>: Shows that about 1/3 of our fraud alerts are actual fraud</li><li><strong>High Recall (90–97%)</strong>: Indicates we’re catching most of the actual fraud cases</li><li><strong>Moderate F1 Score (44–52%)</strong>: Reflects the trade-off between precision and recall</li></ol><h3>ROC-AUC vs PR-AUC: Technical Analysis</h3><h4>Why ROC-AUC Can Be Deceptive</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LFhPfYyzd32likGh.png" /><figcaption>Roc Curves for Training, Testing and OOT samples</figcaption></figure><p>ROC curves plot True Positive Rate (TPR) against False Positive Rate (FPR) across different threshold values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*m2svEHf20IJryNMzJd90Uw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*igvnpj2R2UO61uqfBwUIPA.png" /></figure><p>While widely used, ROC curves can be misleading in highly imbalanced datasets.:</p><ul><li>ROC-AUC can remain high even with poor minority class performance, since the focus is on majority class</li><li>alse Positive Rate (FPR) becomes less meaningful due to the large number of true negatives</li><li>The curve’s shape may not reflect practical model utility</li><li>Translation: ROC-AUC is like that friend who’s always too optimistic</li></ul><h3>Why PR-AUC is Your Friend</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Dr5E_6DTnrSs_IVu.png" /><figcaption>PR-AUC curves for Training, Testing and OTT samples</figcaption></figure><p>PR curves plot Precision against Recall. The PR-AUC plots provide a more realistic picture of model performance in fraud detection scenarios, as they better capture the challenges of detecting rare fraudulent transactions while maintaining reasonable precision. This is particularly important in fraud detection where both false positives (blocking legitimate transactions) and false negatives (missing fraud) have significant business implications.</p><p>Advantages in fraud detection:</p><ul><li>Directly focuses on the minority class</li><li>More sensitive to changes in false positive rate</li><li>Better reflects operational impact</li><li>Provides clearer insight into model’s practical utility</li><li>Translation: PR-AUC is like that brutally honest friend who tells you exactly what you need to hear</li></ul><h3>Performance Analysis Across Datasets</h3><p>Our model’s performance reveals:</p><ul><li>PR-AUC: Shows significant overfitting (0.4 drop from Training to OOT)</li><li>ROC-AUC: Remains stable (around 0.99) but masks underlying issues</li><li>Key indicators of overfitting:</li><li>Poor generalization to new data</li><li>Precision degradation</li><li>Recall instability</li></ul><h3>Best Practices: Technical Implementation</h3><ol><li><strong>Metric Selection</strong><br>- Primary: PR-AUC for overall performance<br>- Secondary: Precision and Recall for specific aspects<br>- Business metrics for operational impact</li><li><strong>Threshold Optimization</strong><br>- Consider cost matrix for false positives vs false negatives<br>- Implement multiple thresholds for different risk levels<br>- Regular recalibration based on performance monitoring</li><li><strong>Validation Strategy</strong><br>- Stratified cross-validation to maintain class distribution<br>- Out-of-time validation for temporal stability<br>- Multiple evaluation periods for robustness</li></ol><h3>The Bottom Line</h3><p><strong>Remember</strong>: In fraud detection, the perfect model doesn’t exist. The goal is to build something that:</p><ul><li>Catches enough fraud to protect your business</li><li>Doesn’t drive your customers crazy with false alarms</li><li>Keeps working as fraud patterns change</li><li>Makes business sense</li></ul><p><em>Ready to dive deeper into any of these topics? Check out our other articles on model stability, sampling strategies, and time series analysis!</em></p><p><strong>Reminders on this series</strong>:</p><ul><li>All the plots, code, and results I will discuss in this series are taken from my<strong> </strong><a href="https://github.com/valever/Fraud-Modeling/tree/main"><strong>repository</strong></a> and produced using the credit card fraud detection dataset from<strong> </strong><a href="https://www.kaggle.com/datasets/kartik2112/fraud-detection"><strong>Kaggle open data</strong></a><strong>.</strong></li><li><a href="https://medium.com/%40valeria.verzi1/%EF%B8%8F-%EF%B8%8F-the-data-science-playbook-for-fraud-detection-53153bb9aef9">🕵️‍♂️ The Data Science Playbook for Fraud Detection</a> is master article, telling what this series is talking about and containing all the relevant links to already published article.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52b224ac0418" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Inside the Fraud Ecosystem]]></title>
            <link>https://medium.com/@valeria.verzi1/inside-the-fraud-ecosystem-6471482d56b0?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/6471482d56b0</guid>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Tue, 23 Sep 2025 06:13:49 GMT</pubDate>
            <atom:updated>2025-09-23T06:26:43.705Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why analysts and data scientists need each other to beat both everyday fraud and large-scale attacks.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QE9bCXqjoVzMrsCWx-sUkw.jpeg" /></figure><p>Fraud is messy, tricky, and always changing. At first glance it can look totally random, but if you dig in, you’ll find recurring patterns — each one with its own quirks. The very first step in building any successful fraud system is <strong>mapping those behaviors</strong>: figuring out what type of fraud you’re looking at, spotting the signs, and connecting the dots.</p><p>Think of stolen identity fraud, phishing, or account takeovers. They might all fall under “fraud,” but each plays by different rules and might require its own set of defenses.</p><p>Now, if you zoom out and look at your fraud rate as a whole, a bigger picture emerges. You’ll usually see two layers: a steady “baseline” of background fraud, and sharp peaks whenever organized attacks hit.</p><p>In this article, we’ll stick with that zoomed-out view and talk about the bigger ecosystem behind fraud detection: the rules and models, the analysts and data scientists, and — most importantly — how they need to work together to stay ahead of the game.</p><h3>🎯 Two Types of Fraud</h3><h4>1. Organized Attacks 🎯</h4><p>These are typically well-organized operations focusing on specific areas, merchants, or items (or combinations thereof). They represent organized crime, often involving groups from the same region who design and execute coordinated attacks.</p><p>The hallmark of organized attacks is their sudden appearance — you’ll see spikes in similar fraud patterns within specific perimeters. While difficult to predict, they do present some recurrent patterns that can help raise alerts. The real challenge lies in distinguishing these from legitimate events like flash sales or seasonal promotions.</p><h4>2. Baseline Fraud 🎲</h4><p>This is what I call the “background noise” of fraud — individual cases rather than organized attacks. These cases stem from various individual circumstances like shipping issues, impulse purchases, or poor financial planning. Unlike organized attacks, baseline fraud is always present in the system, highly diverse in nature, with no clear unified patterns.</p><h3>🎯 The Business Impact</h3><p>Fighting fraud is always a balancing act. On one hand, you need to push fraud rates down to cut losses. On the other, you can’t frustrate good customers with too much friction. That means decisions need to be sharp and precise, without putting conversion rates at risk.</p><p>It’s all about finding that sweet spot: blocking the bad guys while letting genuine customers flow through smoothly.</p><h3>📉 Lowering Fraud Rates</h3><p>The fraud rate curve presents itself as a combination of a baseline curve (representing individual fraud cases) and multiple peaks (representing organized attacks). To effectively combat this, we employ a <strong>two-pronged strategy</strong> involving both Analytics and Data Science teams.</p><h4>Analytics Team 🕵️‍♂️</h4><p>Analysts are real-time detectives. They play a crucial role in identifying and addressing specific peaks in fraud activity. They’re great at spotting sudden spikes, connecting the dots, and deploying quick countermeasures. Their work is what holds the line during live attacks</p><h4>Data Science Team 🤖</h4><p>Data scientists, on the other hand, focus on the long game. They build predictive models and automated detection systems that keep baseline fraud under control. Their tools:</p><ul><li><strong>Long-term models</strong> that reduce overall fraud trends by learning from analyst discoveries.</li><li><strong>Anomaly detection systems</strong> that help surface new attack patterns before they blow up.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKIu7bx8aBx3TZSXN3ooFA.png" /></figure><h3>🤝 The Power of Collaboration</h3><p>Here’s the key: <strong>neither team</strong> can win this fight alone. Analytics and Data Science need to feed into each other constantly.</p><ul><li>Analysts spot the signals first and provide the raw evidence.</li><li>Data scientists turn those signals into scalable models and smarter detection systems.</li></ul><p>And here’s the kicker: <strong>data scientists can’t be effective without the deep knowledge that comes from working side by side with analysts</strong>. At the same time, analysts are far less effective without the tools, automation, and numerical evidence provided by data science.</p><p>When the loop works, the whole ecosystem gets stronger. A well-oiled setup means not just blocking fraud, but also innovating, scaling, and keeping business goals front and center.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v3ZkVYCVE1_J0q0eQ5QknA.png" /></figure><h3>💡 Key Takeaways</h3><p>Fraud will never disappear, but it can be managed. The trick is staying flexible, combining short-term responses with long-term solutions, and making sure Analytics and Data Science work as one team.</p><p>At the end of the day, it’s all about balance:</p><ul><li>stopping fraud without hurting user experience,</li><li>reacting fast without losing the long view,</li><li>and blending human judgment with automated systems.</li></ul><p>The goal isn’t perfection — it’s building a system that keeps risk under control while letting your business grow and your customers stay happy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6471482d56b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[️‍♂️ The Data Science Playbook for Fraud Detection]]></title>
            <link>https://medium.com/@valeria.verzi1/%EF%B8%8F-%EF%B8%8F-the-data-science-playbook-for-fraud-detection-53153bb9aef9?source=rss-fa6d10644812------2</link>
            <guid isPermaLink="false">https://medium.com/p/53153bb9aef9</guid>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fraud]]></category>
            <dc:creator><![CDATA[Valeria Verzi]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 19:19:23 GMT</pubDate>
            <atom:updated>2025-12-31T13:22:10.804Z</atom:updated>
            <content:encoded><![CDATA[<p>Diaries &amp; Lessons from Machine Learning in the Real World</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IC7ixe_ONr8U3O1hcrTSqw.png" /></figure><p>Hey there! 👋 Welcome to my first series of articles on fraud detection. After spending over a decade in the trenches of fraud detection for insurance and fintech companies, I’ve learned a thing or two about what works and what doesn’t. And you know what? I want to share this knowledge with you!</p><h3>🎯 Why I’m Writing This Series</h3><p>When I chat with my team, new joiners, or anyone interested in fraud detection, one thing becomes crystal clear: this is a niche area where we’re seriously lacking well-established best practices and literature. Everyone’s doing it differently, and there’s no clear guide to follow.</p><p>Fraud brings some unique challenges that no other application has. They are numerous and not always straightforward. As a result, without guidance from an expert developer in the area, it’s easy to get things wrong!</p><p>That’s exactly why I’m writing this series. I want to share what I’ve learned through years of trial and error and establishing some solid best practices for our modeling approach. But here’s the thing — this isn’t the only way to do fraud detection. It’s just the approach that I’ve seen work consistently across different use cases.</p><p>There are also many use cases that requires different type of models, like clustering or anomaly detection models. Or again, you might have the chance to exploit the potential of graphs. Here we will talk about classification models.</p><p>And — reality check — with technology evolving at lightning speed, some of these methods will change. But the core business concepts and technical challenges? They’re here to stay. So don’t get too attached to the specific methods — use this as your starting point. If you’ve found something that works better for your use case, please share it with the community! We’re all in this together.</p><h3>🔍 What Makes Fraud Detection Special?</h3><p>Fraud detection is tricky. Here’s why:</p><ol><li><strong>The Numbers Game 🎲 </strong><br>- We’re talking about fraud rates as low as 0.015% (yes, that’s from the European Bank Authority!)<br>- Your typical ML metrics will lie to you in this context<br>- It’s like trying to spot a black cat in a dark room… that’s also moving</li><li><strong>The Label Problem 🏷️</strong><br>- Sometimes fraud takes weeks to surface<br>- You might never know if a rejected transaction was actually fraud<br>- Non-fraud data are more like “genuine-ish” transactions</li><li><strong>The Moving Target 🎯</strong><br>- Fraudsters are always one step ahead<br>- Patterns repeat but with twists<br>- Your fraud system needs to be as adaptable as the fraudsters</li></ol><h3>📚 What We’ll Cover</h3><p><em>[I will update this list with links as I write articles — stay tuned! 😎]</em></p><h4>1. 🤝 Collaboration makes you powerful</h4><p>Fraud isn’t just random chaos — there are patterns, baselines, and spikes that tell a bigger story. To make sense of it, you need both:<br> 🔍 Analysts who spot behaviors in the wild<br> 🤖 Data scientists who build scalable defenses</p><p>Read the full article:<a href="https://medium.com/@valeria.verzi1/inside-the-fraud-ecosystem-6471482d56b0"> Inside the Fraud Ecosystem</a>!</p><h4>2. 📊 Metrics That Matter</h4><ul><li>Why accuracy is basically useless</li><li>PR-AUC vs ROC-AUC: The real debate</li><li>How to interpret your model’s performance</li><li>Real-world examples and gotchas</li></ul><p>Read the full article: <a href="https://medium.com/@valeria.verzi1/understanding-model-evaluation-metrics-in-fraud-detection-beyond-accuracy-52b224ac0418">Understanding Model Evaluation Metrics in Fraud Detection: Beyond Accuracy</a></p><h4>3. ⚖️ Fighting the Imbalance</h4><ul><li>Sampling strategies that work</li><li>Cross-validation that makes sense</li><li>How to avoid the overfitting trap</li><li>Real performance comparisons</li></ul><p>Read the full articles here: <a href="https://medium.com/@valeria.verzi1/understanding-overfitting-in-fraud-detection-the-basics-79eb2795b70c">Overfitting in Fraud Detection, Part 1: The Basics</a>; <a href="https://medium.com/@valeria.verzi1/overfitting-in-fraud-detection-part-2-advanced-techniques-72ddb44ffd92">Overfitting in Fraud Detection, Part 2: Advanced Techniques</a></p><h4>4. ⏰ Time is Everything</h4><ul><li>Why traditional time series fails</li><li>How to split your data properly</li><li>Robust Data preparation</li><li>Validation approaches that make sense</li></ul><p>Read the full article: <a href="https://medium.com/@valeria.verzi1/fraud-doesnt-flow-it-loops-rethinking-time-in-fraud-detection-28ab59f733a6">Fraud Doesn’t Flow — It Loops: Rethinking Time in Fraud Detection</a></p><h4>5. 🛠️ Feature Engineering: The Time Factor</h4><ul><li>Why great models fail in production</li><li>The label maturity challenge</li><li>Avoiding the data leakage trap</li><li>Building features that work in the real world</li></ul><p>Read the full article: <a href="https://medium.com/@valeria.verzi1/predicting-the-past-the-subtle-art-of-not-cheating-time-225d82f7b001">Predicting the Past: The Subtle Art of Not Cheating Time</a></p><h4>6. 🔄 Keeping Your Model Sharp</h4><ul><li>Different types of drift (and how to spot them)</li><li>PSI: Your new best friend for monitoring</li><li>Business metrics that actually matter</li><li>When to hit the retrain button</li></ul><p>Read the full article: <a href="https://medium.com/@valeria.verzi1/fighting-decay-the-subtle-art-of-keeping-fraud-models-relevant-cb416d19cab6?postPublishedType=initial">Fighting Decay: The Subtle Art of Keeping Fraud Models Relevant</a></p><h4>7. 📈 Model Monitoring in Production</h4><ul><li>Setting up effective monitoring systems</li><li>Detecting and responding to drift</li><li>Balancing technical and business metrics</li><li>Creating actionable alerts and responses</li></ul><p>Read the full article: <a href="https://medium.com/@valeria.verzi1/monitoring-fraud-models-in-production-challenges-and-solutions-477ac99760f8">Monitoring Fraud Models in Production: Challenges and Solutions</a></p><h3>🚨 Some of the Real-World Challenges</h3><h4>Data Leakage: The Silent Killer</h4><pre># The wrong way (don&#39;t do this!)<br>df[&#39;future_fraud&#39;] = df[&#39;is_fraud&#39;].shift(-1)  # Data leakage!<br><br># The right way<br>df[&#39;past_fraud_rate&#39;] = df[&#39;is_fraud&#39;].rolling(&#39;60d&#39;).mean()</pre><h4>Label Maturity: The Waiting Game</h4><pre>Transaction Timeline:<br>Day 0: Transaction happens<br>Day 1-60: We wait... and wait...<br>Day 61+: Finally, we know if it was fraud</pre><h4>Monitoring: Beyond the Basics</h4><pre># Basic monitoring (not enough)<br>model.predict_proba(X)<br><br># Better monitoring<br>from sklearn.metrics import average_precision_score<br>from fraud_detection.monitoring import PopulationStabilityIndex<br># Track both technical and business metrics<br>psi = PopulationStabilityIndex()<br>drift_score = psi.calculate(feature_distributions)</pre><h3>🚀 Getting Started</h3><p>Not sure where to begin? Here’s your cheat sheet:</p><ul><li>New to fraud? Start with the metrics article</li><li>Building a new system? Jump to the debiasing article, followed by the time series and feature engineering articles</li><li>Maintaining models? Check out the stability metrics</li><li>Monitoring models? Begin with the time monitoring article</li></ul><h3>💡 The Bottom Line</h3><p>Fraud detection isn’t just about building a model — it’s about creating a system that:</p><ol><li>Actually catches fraud</li><li>Doesn’t drive your customers crazy</li><li>Keeps working as fraud patterns change</li><li>Makes business sense</li></ol><p><strong>Remember</strong>: The goal isn’t to build the perfect model (spoiler: it doesn’t exist), but to create a system that effectively protects your business while keeping your customers happy.</p><p><strong>Ready to dive in? Let’s get started!</strong> 🚀</p><p><strong>Additional note</strong>: All the plots, code, and results I will discuss in this series are taken from my<strong> </strong><a href="https://github.com/valever/Fraud-Modeling/tree/main"><strong>repository</strong></a> and produced using the credit card fraud detection dataset from<strong> </strong><a href="https://www.kaggle.com/datasets/kartik2112/fraud-detection"><strong>Kaggle open data</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53153bb9aef9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>