<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Eding on Medium]]></title>
        <description><![CDATA[Stories by Matt Eding on Medium]]></description>
        <link>https://medium.com/@matteding?source=rss-5181614e8007------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*L__hMe_vaXOf5Xkt.jpg</url>
            <title>Stories by Matt Eding on Medium</title>
            <link>https://medium.com/@matteding?source=rss-5181614e8007------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:46:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@matteding/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Imbalanced Data —Fraud Detection]]></title>
            <link>https://medium.com/@matteding/imbalanced-data-fraud-detection-3185c1cdaa77?source=rss-5181614e8007------2</link>
            <guid isPermaLink="false">https://medium.com/p/3185c1cdaa77</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Matt Eding]]></dc:creator>
            <pubDate>Wed, 18 Sep 2019 16:46:01 GMT</pubDate>
            <atom:updated>2019-09-18T16:46:01.060Z</atom:updated>
            <content:encoded><![CDATA[<p>A Journey in Credit Card Fraud Detection</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1deAOUP9xLXmAf2mo9g2A.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@nandhukumar">Nandhu Kumar</a></figcaption></figure><p><em>For brevity, some code has been elided. Check out </em><a href="https://github.com/MattEding/CreditCardFraud"><em>my GitHub</em></a><em> for details.</em></p><p>For many real world data sets where we would like to perform classification, using the data as is simply won’t cut it. The reason is that most classification algorithms such as Logistic Regression assume that the number of observations from each class are roughly the same. When the data is heavily imbalanced, the algorithm will start to make predictions in favor the majority class. Unfortunately the minority class tends to be the one that we are interested in identifying, as is the case with finding people with a rare disease. In this article we will investigate the process of dealing with an imbalanced data set to maximize the predictive power of our models.</p><p>The data I chose to analyze was the <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud">Credit Card Fraud Detection</a> data set from Kaggle. As a quick overview of the data, the following features are provided:</p><ul><li><em>Class</em>: Non-Fraud (0) and Fraud (1)</li><li><em>Time</em>: seconds elapsed from when data collection began</li><li><em>Amount</em>: the transaction amount</li><li><em>V1-V28</em>: features created via <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">PCA</a> which also preserves anonymity of users</li></ul><p>To mimic the notion of trying to make predictions of unseen data, the first step I undertook was splitting my data 80% train and 20% test sets and put aside my holdout test set until the final evaluation at the end. With training data to work with, it was now time to delve deep into the structure of data I was working with to see if I could squeeze out as much information as possible to improve model performance.</p><pre>import pandas as pd<br>from sklearn.model_selection import train_test_split</pre><pre>credit = pd.read_csv(&#39;creditcard.csv&#39;)<br>credit_train, credit_test = train_test_split(credit, test_size=0.2)</pre><h4>Data Exploration</h4><p>First I chose to look into the <em>Time</em> feature to see what the temporal span was.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>bins = len(np.histogram_bin_edges(credit_train[&#39;Time&#39;], &#39;auto&#39;))<br>plt.hist(credit_train[&#39;Time&#39;], bins=bins)</pre><figure><img alt="Transactions over time histogram" src="https://cdn-images-1.medium.com/max/732/1*ePtQE7pOIMw7xCMF9Hve3g.png" /><figcaption>Two day cycle of transaction data</figcaption></figure><p>From the histogram we can see two days worth of data, where I assume the peaks are daytime and the valleys are night. Since there seems to be a regular pattern, I decided to convert the <em>Time</em> to hour of day. To do this, I made the assumption that midnight occurs at the minimums.</p><pre>from sklearn.base import BaseEstimator, TransformerMixin</pre><pre>class TimeToHour(BaseEstimator, TransformerMixin)<br>    def __init__(self, copy=True):<br>        self.copy = bool(copy)</pre><pre>    def fit(self, X, y=None):<br>        bins = len(np.histogram_bin_edges(X[&#39;Time&#39;], bins=&#39;auto&#39;))<br>        cuts = pd.cut(X[&#39;Time&#39;], bins)<br>        counts = cuts.value_counts()<br>        midnight = int(counts.index[-1].mid)<br>        self.midnight_ = midnight<br>        return self</pre><pre>    def transform(self, X):<br>        X = X.copy() if self.copy else X<br>        day_sec = 24 * 60 * 60<br>        hr_sec = 60 * 60<br>        X[&#39;Hour&#39;] = ((X[Time] — self.midnight_) % day_sec) / hr_sec<br>        return X</pre><figure><img alt="Hourly transactions histogram" src="https://cdn-images-1.medium.com/max/732/1*QmsN9Z5DtPYjFdrlgwxUFA.png" /><figcaption>Hourly representation of the credit card data</figcaption></figure><p>Much better! Having normalized our temporal feature to an hourly format should prove to be more robust than elapsed time as it embeds trends in the data better. As an added bonus, having created a transformer, I could easily inject this into pipelines later down the line when doing cross-validation.</p><p>Next I decided to investigate further into the other feature that I could derive meaning from —<em> Amount</em>. I’ll split the data by class so we can look for differences between the two that may prove useful for improving our classification model.</p><pre>import seaborn as sns</pre><pre>is_fraud = (credit_train[&#39;Class&#39;] == 1)<br>nonfraud = credit_train[~is_fraud]<br>fraud = credit_train[is_fraud]</pre><pre>x, y = &#39;Hour&#39;, &#39;Amount&#39;<br>fig, (ax0, ax1) = plt.subplots(1, 2, sharey=True, sharex=True)<br>sns.scatterplot(nonfraud[x], nonfraud[y], ax=ax0)<br>sns.scatterplot(fraud[x], fraud[y], ax=ax1)</pre><figure><img alt="Hourly transactions scatterplots for non-fraud and fraud classes" src="https://cdn-images-1.medium.com/max/961/1*tw634FBRI2NrDBW59eRBsw.png" /><figcaption>A poor choice for visual representation of the data</figcaption></figure><p>Looking at the scatterplot, it is difficult to see any patterns due to the huge range of y-values and the densely clustered points at the bottom. Luckily we can apply logarithms to remedy the vertical issue, and we can use hexbins to group nearby values into bins to evaluate which regions have the most density. Note that since some transaction amounts are $0.00, I used log(<em>Amount</em> + 1) to circumvent undefined values.</p><pre>x, y = &#39;Hour&#39;, &#39;Log1pAmount&#39;<br>fig, (ax0, ax1) = plt.subplots(1, 2, sharey=True, sharex=True)<br>ax0.hexbin(nonfraud[x], nonfraud[y], gridsize=10)<br>ax1.hexbin(fraud[x], fraud[y], gridsize=10)</pre><figure><img alt="Hourly transactions hexbins for non-fraud and fraud classes at log scale" src="https://cdn-images-1.medium.com/max/942/1*Bv0ubn3_enNLoYidEn5ejg.png" /><figcaption>Good representations of data expose patterns much more clearly</figcaption></figure><p>Wow! What a difference a simple transformation and an appropriate visualization choice can make. We can observe that non-fraud activity has quite a different structure with most activity during daytime hours and is most dense in the center that gradually fades towards the extremities. On the other hand, fraud activity is much more uniform both in time of day and the transaction amount.</p><p>Another area I wanted to delve into was the distribution of outliers within each class. It might be possible that outliers could confuse our model we want to train.</p><pre>from sklearn.neighbors import LocalOutlierFactor</pre><pre>lof = LocalOutlierFactor(contamination=&#39;auto&#39;)</pre><pre>nf_preds = lof.fit_predict(nonfraud)<br>nf_inliers = nonfraud[nf_preds == 1]<br>nf_outliers = nonfraud[nf_preds == -1]</pre><pre>fr_preds = lof.fit_predict(fraud)<br>fr_inliers = fraud[fr_preds == 1]<br>fr_outliers = fraud[fr_preds == -1]</pre><p>When using hexbins to compare inliers/outliers to the entire class distribution, I found that the graphs of inliers vs all looked basically the same regardless of class. Likewise the fraud outliers vs fraud all yielded similar plots. However, the non-fraud outlier vs non-fraud all were drastically different:</p><figure><img alt="Hourly transactions hexbins for non-fraud outliers vs all" src="https://cdn-images-1.medium.com/max/942/1*8AcwGy6uEB3bjw2tDOS4iw.png" /></figure><p>I hypothesized that non-fraud outliers may be misclassified as fraudulent activity. So when it comes to training I will run my pipelines on the full train set and also on the train set minus non-fraud outliers. <em>(Spoiler alert — my hypothesis proved to be wrong.)</em></p><h4>Balancing Act</h4><p>When it comes to imbalanced data, we should consider balancing out the minority and majority classes to avoid misleading pitfalls otherwise. There are several options at our disposal: undersampling the majority class, oversampling the minority class, or a combination of the two. The issue with undersampling is that a lot of data gets discarded which is usually undesirable. So instead I used two oversampling techniques that make new, synthetic data points from existing minority sample data: <strong>S</strong>ynthetic <strong>M</strong>inority <strong>O</strong>versamping <strong>TE</strong>chnique (SMOTE) and<strong> ADA</strong>ptive <strong>SYN</strong>thetic (ADASYN).</p><p>For each minority sample, both SMOTE and ADASYN find nearby samples from that class, compute their difference, and scale by a random factor between 0 and 1. The main difference between the two algorithms is that SMOTE oversamples each point the same amount whereas ADASYN places greater emphasis on points that are more difficult to distinguish. Since these algorithms use spacial distance, we need to standardize the features using StandardScaler which subtracts the mean and then divides by the standard deviation.</p><pre>from imblearn.over_sampling import ADASYN, SMOTE<br>from imblearn.pipeline import Pipeline<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import recall_score<br>from sklearn.linear_model import LogisticRegression</pre><pre>hour = TimeToHour()<br>scaler = StandardScaler()<br>smote = SMOTE()<br>adasyn = ADASYN()<br>logreg = LogisticRegression()</pre><pre>no_sampling_pipe = Pipeline([<br>    (&#39;hour&#39;, hour),<br>    (&#39;scaler&#39;, scaler),<br>    (&#39;logreg&#39;, logreg)<br>])<br>smote_pipe = Pipeline([<br>    (&#39;hour&#39;, hour),<br>    (&#39;scaler&#39;, scaler), <br>    (&#39;smote&#39;, smote), <br>    (&#39;logreg&#39;, logreg)<br>])<br>adasyn_pipe = Pipeline([<br>    (&#39;hour&#39;, hour),<br>    (&#39;scaler&#39;, scaler), <br>    (&#39;adasyn&#39;, adasyn), <br>    (&#39;logreg&#39;, logreg)<br>])</pre><pre>X_train = credit_train.drop(columns=[&#39;Class&#39;])<br>y_train = credit_train[&#39;Class&#39;]</pre><p>Note that instead of the regular Scikit-Learn Pipeline, I use a specialized one from Imbalanced-Learn which allows for resampling within the pipe. Unfortunately, we have to make three separate pipelines since there is no FeatureUnion to do each case in parallel that handles resampling algorithms.</p><p>To evaluate how well the models work, I settled on using recall as my metric of choice. <a href="https://en.wikipedia.org/wiki/Precision_and_recall">Recall</a> is defined as TP / (TP + FN) which essentially measures the efficacy of a model capturing as much fraudulent activity as possible. Using cross-validation, the average recall scores from running these pipelines are as follows:</p><p><strong>All Train Data</strong></p><ul><li>No Resampling: 59.34%</li><li>SMOTE: 90.29%</li><li>ADASYN: 92.33%</li></ul><p><strong>Train Data with Non-Fraud Outliers Removed</strong></p><ul><li>No Resampling: 66.24%</li><li>SMOTE: 90.29%</li><li>ADASYN: 90.80%</li></ul><p>As one would expect, the pipeline where class imbalance was left unaddressed performed worse considerably, and it was the only one were removing outliers had an improvement. On the contrary, the oversampling results suggest that the pipeline that uses ADASYN on all train data should be the one to use for our final model. However there is more than meets the eye when the confusion matrices of SMOTE vs ADASYN are compared. To illustrate, I chose one of the resultant <a href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrices</a> that arose during a cross-validation fold for each (the other folds had similar results).</p><figure><img alt="SMOTE confusion matrix during cross-validation" src="https://cdn-images-1.medium.com/max/340/1*kg4P1VxMkh03bwb_38E7HA.png" /></figure><figure><img alt="ADASYN confusion matrix during cross-validation" src="https://cdn-images-1.medium.com/max/340/1*7zY6FFsP3rS-wJ94QWU1Xg.png" /><figcaption>Confusion matrices for SMOTE and ADASYN pipelines</figcaption></figure><p>From the diagrams we can see that ADASYN identifies more frauds (2 more in this instance), but at a cost — it incurs almost 4 times as many false positives than SMOTE. Therefore the best model really depends on the cost for a business auditing accounts flagged as fraud vs the average cost of leaving a fraud uncaught. Consequently I will be using both models with my holdout test set and work out the math for when one model would be preferable over the other.</p><p>As an additional measure to improve performance, I used GridSearchCV on both the SMOTE and ADASYN pipelines that use all the training data and saved them for later use. In the parameter dictionary, the word(s) in front of the double underscore corresponds to a pipeline step and the word(s) after is the hyper-parameter to fine-tune.</p><pre>params = dict(<br>    smote__k_neighbors=range(4, 6),<br>    logreg__C=np.power(10.0, range(-2, 3)),<br>)<br>smote_gridcv = GridSearchCV(smote_pipe, params, scoring=&#39;recall&#39;)<br>smote_gridcv.fit(X_train, y_train)<br>smote_best = smote_gridcv.best_estimator_</pre><pre># ...<br>adasyn_best = adasyn_gridcv.best_estimator_</pre><h4>The Results</h4><p>Time to use the data we kept aside at the beginning and see how well the models generalize to new data.</p><pre>from sklearn.metrics import confusion_matrix</pre><pre>X_test = credit_test.drop(columns=[&#39;Class&#39;])<br>y_test = credit_test[&#39;Class&#39;]</pre><pre>smote_pred = smote_best.predict(X_test)<br>smote_conf_mtx = confusion_matrix(y_test, smote_pred)</pre><pre># ...<br>adasyn_conf_mtx = confusion_matrix(y_test, adasyn_pred)</pre><figure><img alt="SMOTE confusion matrix on test set" src="https://cdn-images-1.medium.com/max/340/1*TuAtfbowRN70Bia8zRg0gA.png" /></figure><figure><img alt="ADASYN confusion matrix on test set" src="https://cdn-images-1.medium.com/max/340/1*Qh4rtW6slrJTN-QeiWiv4w.png" /><figcaption>Final model results</figcaption></figure><p>SMOTE Recall: 94.06%<br>Audits: 1520<br>Uncaught: 6</p><p>ADASYN Recall: 97.03%<br>Audits: 5555<br>Uncaught: 3</p><p>Finally let’s do some math for which model to use.</p><ul><li>Let <em>F</em> be the cost of leaving a fraudulent account being uncaught.</li><li>Let <em>A</em> be the cost of auditing an account to see if it is fraudulent.</li><li>Let <em>f</em> be the number of uncaught fraudulent accounts (TN) for a model.</li><li>Let <em>a</em> be the number of flagged accounts (TP + FP) for a model.</li></ul><figure><img alt="SMOTE and ADASYN breakeven equation" src="https://cdn-images-1.medium.com/max/1024/1*X67qYC8-B65ltZkrMXi9CA.png" /></figure><p>From the above equation we deduce that if the average cost of leaving a fraudulent account uncaught is 1345 times the cost of an audit, the ADASYN model should be the one used. Otherwise SMOTE would be better fiscally.</p><h4>Conclusion</h4><p>As is typical with data science projects, spending time to carefully explore the data plays an important role. We were able to construct new features that provide more information than what we had to begin with, and also gain insights about data structure which we used to adjust class imbalance. Finally some analysis was required to determine when a model would be appropriate depending on business costs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3185c1cdaa77" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>