<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suneil on Medium]]></title>
        <description><![CDATA[Stories by Suneil on Medium]]></description>
        <link>https://medium.com/@suneil.shrivastav?source=rss-317ee1a9dedd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jFHLheABB-X3NHuo.</url>
            <title>Stories by Suneil on Medium</title>
            <link>https://medium.com/@suneil.shrivastav?source=rss-317ee1a9dedd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:22:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suneil.shrivastav/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The case of the churning customer]]></title>
            <link>https://medium.com/loopio-tech/the-case-of-the-churning-customer-fc4e0fd442a0?source=rss-317ee1a9dedd------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc4e0fd442a0</guid>
            <dc:creator><![CDATA[Suneil]]></dc:creator>
            <pubDate>Fri, 15 Jul 2022 13:00:07 GMT</pubDate>
            <atom:updated>2022-07-15T13:51:44.662Z</atom:updated>
            <content:encoded><![CDATA[<h4>Choosing appropriate metrics to assess model performance in imbalanced datasets</h4><p>When I started building binary classification models in an academic setting, I mostly focused on getting the training right, such as comparing different algorithms and hyperparameter tuning, and ended up devoting little to no time to model evaluation. At the time, “accuracy” used to be the single evaluation metric I was required to report most often. Since we are talking in the context of classification tasks, accuracy does seem to be a fair metric. Its intuitive meaning, “the number of correct predictions”, just works. And in the typical classroom setting, it does often work.</p><p>The reason accuracy worked as a metric is that the toy datasets most often used in such settings are balanced: for example, one of the most well-known datasets used in classrooms is the iris dataset, in which the three types of iris flowers are each represented by 50 rows in the dataset. Since all the classes are represented equally, the dataset is balanced. Another equally famous dataset, the Titanic survivor dataset, is imbalanced, since its target variable, the column representing whether or not someone survived, has 38% survivors. There are more samples in the survivor category compared to non-survivors.</p><p>In industry use cases, we are most often interested in building classification models on significantly imbalanced datasets. Classification models allow us to construct algorithms that can predict which category a new observation belongs to, based on the patterns it learns from past data.</p><h3>Commonly used terms</h3><p>It is helpful to define a few terms at this point. The variable of interest on which we want to make the prediction is commonly called the <em>target variable</em>. When this target variable is of a categorical nature, the prediction problem is referred to as a <em>classification</em> problem. When the target variable can take one of only two potential values, the prediction problem is reduced to <em>binary classification</em>.</p><p>A classification model is an algorithm that can learn the relation from feature data, or inputs, to the “class” of the target variable. An example that is quite common across many different types of companies is predicting whether a current customer will continue to remain a customer in the next period or within the next <em>n</em> periods (this period could be a day, week, month, quarter, or year) We can reframe the target variable as an answer to this question: “Will this customer remain a customer in the next period?” Since this can be answered with a yes or a no, this is a binary classification problem.</p><h3>Accuracy is an unreliable performance metric for imbalanced datasets</h3><p>Imagine you work at a telecom company. You will likely encounter the very common practice of your competitor consistently luring your subscribers away from you with ingenious marketing ploys. This leads to a steady stream of customers leaving your network (“churning”) to become your competitor’s customers.</p><p>In many mature markets with stable, established competitors, churn could be around 5% a year. This creates an imbalanced dataset of 95% non-churning customers and 5% churning customers. Additionally, let us assume that the new acquisition rate is roughly the same as the base churn rate so that the net customer base remains approximately the same year to year.</p><p>Now, if you said no customer will ever churn, you will still be right 95% of the time!</p><p>Is that a problem?</p><p>Yes.</p><p>Our main motive is to make sure that the loyalty campaign budget, earmarked for marketing campaigns to retain customers, gets redirected to those customers that are going to churn and redirected away from those customers that will not churn. More concretely, this means that each campaign outreach activity gets directed to the true churners and away from the true non-churners.</p><p>Since the fact of churn is still unknown at the time of budget allocation, executives depend on their modeler to tell them who the churners are and who the non-churners are.</p><h4>Breaking down customer actions to define true and false outcomes</h4><p>Imagine that the modeler has created a list of 600 customers (out of 10,000) that the model predicts to be churners in the following month. The campaigners acted on the list, reaching out to these 600 customers, and to nobody else. The following month, the loyalty team ran the numbers on who actually churned and found the following scenarios:</p><ol><li>Suppose 450 customers out of the 600 in the churn list actually churned. These 450 are <strong>true positives</strong> — people whom the model predicted to be churners and that actually churned.</li><li>The remaining 150 were <strong>false positives</strong> — people deemed at risk of churning but who did not actually churn.</li><li>Suppose the true number of churners was 500, and there were 50 that the model failed to catch at all. These 50 represent <strong>false negatives</strong>: the model failed to identify these people correctly as being at risk of churning. (Note: it is a common enough occurrence in this particular case. It could mean something as unavoidable as circumstances you cannot possibly know about, including all the different competitor marketing campaigns that your customers receive, their financial health, their family emergencies, and so on)</li><li><strong>True negatives</strong> are the largest hence uninteresting (but comforting) segment. These are those wonderful customers whom our model did not predict to be at risk of churn for the following period, and the prediction turned out to be right: they did not churn!</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*0hY53cxpVGABWZwY" /><figcaption><em>True vs False Outcomes</em></figcaption></figure><h4>Adding to the confusion: enter the matrix</h4><p>The definitions above become much clearer when presented as a 2x2 matrix called the ‘Confusion Matrix’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*KNjwJ5aqBeyz2Rx2KNxfvg.png" /><figcaption>Confusion matrix</figcaption></figure><h4>How to recall with precision</h4><p>There are three different but related metrics you can try to use to assess model performance:</p><ol><li><strong>Precision: </strong>Out of all your <em>predicted </em>churns, how many were <em>actual</em> churns? In this example, this metric is 450/600 or 75%.</li><li><strong>Recall: </strong>Out of all <em>actual</em> churns, how many did you manage to label correctly? In this example, this number is 450/500 or 90%.</li><li><strong>F1 score: </strong>The harmonic mean of precision and recall is called the F1 score. That is, F1 score = 2 * precision * recall / (precision + recall). This is a specific form of the general metric called the <em>f-beta score</em>, where the “beta” represents the relative weight of recall in the harmonic mean calculation. In the F1 score calculation, the value of beta is 1, which implies that precision and recall are equally weighted.</li></ol><h3>Measuring the true cost of false predictions</h3><p>In broad strokes, we can say that the values of <em>precision</em> and <em>recall</em> allow the modeler to diagnose the balance between false positives and false negatives. From there on out, the modeler is more or less the messenger and not the decision maker. Strictly speaking, the <em>business consequences</em> of favoring precision or recall, and the extent to which this favoring should tilt the scales, is not a decision for the modeler to make.</p><p>It is the business stakeholders who need to make this decision. In the churn example, we saw above, what would be the consequences of false positives and false negatives?</p><ul><li>False positives — customers who the model <em>predicted </em>would churn but did not <em>actually</em> churn — represent a waste of the campaign budget.</li><li>False negatives, on the other hand — customers who the model <em>predicted</em> would not churn but did <em>actually</em> churn — represent a lost revenue stream for the company.</li></ul><p>Consider the following assumptions:</p><ol><li>There are some costs associated with running campaigns.</li><li>There is a limited budget for campaigns.</li><li>The total number of customers is much higher than what retention campaigns can address in any given campaign.</li></ol><p>Then we can understand better why campaign outreaches need to be targeted: reaching out to customers that are not going to churn is a waste of money, whereas not reaching out to customers that intend to churn represents a lost outreach opportunity. The first is a case of false positives: sending directing prevention campaign activities to non-churners. With ever more sophisticated campaign management tools and deepening penetration of digital channels, the cost per outreach activity is trending downwards.</p><p>The cost of a false negative could be considerably higher, given that a telecom customer represents hundreds of dollars in annual revenue. The exact balance of how many outreach campaigns the company may be willing to waste for each additional churner saved is a business decision that modelers can typically help inform but are not accountable for.</p><h4>In conclusion, accuracy is inconclusive.</h4><p>To reiterate, accuracy is a potentially misleading metric to assess a model’s performance when the target variable is imbalanced since it takes into account true negatives, which often constitute a majority of the dataset. Instead of accuracy, the model’s performance is better assessed with metrics such as precision, recall, and f-beta score, which drop true negatives from the calculation.</p><h3>Up next in <em>The Case Of The Churning Customer</em>: moving from model metrics to probabilities and thresholds</h3><p>There is a gap in the narrative presented above: the confusion matrix used to evaluate the model’s performance assumes we know who churned and who did not. When we use the model to make decisions, however, we will not have this knowledge available at the time of prediction. Instead, we need to rely on the confidence scores that the model produces, and it will be a business decision to fix a cutoff threshold — any customer that scores above the cutoff will be labeled a potential churner, and those that score lower will be labeled a non-churner.</p><p>In future blog posts, I will write more about the following topics to bridge the gaps stated above:</p><ul><li>How to move from class-based model metrics to confidence score-based metrics</li><li>How to move from model metrics to predictions</li><li>How to move from measuring model performance to measuring business impact</li></ul><p>While you wait with bated breath for the next installment to drop, check out <a href="https://bit.ly/3vA3IXg">career opportunities</a> across Loopio’s fantastic teams — we are hiring in Engineering, Product, and Design groups!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc4e0fd442a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/loopio-tech/the-case-of-the-churning-customer-fc4e0fd442a0">The case of the churning customer</a> was originally published in <a href="https://medium.com/loopio-tech">Loopio Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>