<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carl gold on Medium]]></title>
        <description><![CDATA[Stories by Carl gold on Medium]]></description>
        <link>https://medium.com/@data-science-churn?source=rss-fbfc6b1ae37a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*rqE87TpLhcrZBLKZ2h8VzA.jpeg</url>
            <title>Stories by Carl gold on Medium</title>
            <link>https://medium.com/@data-science-churn?source=rss-fbfc6b1ae37a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:59:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@data-science-churn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Did It Actually Work? Why Martech Performance Is So Hard to Measure]]></title>
            <link>https://data-science-churn.medium.com/did-it-actually-work-why-martech-performance-is-so-hard-to-measure-3bfa64bc09e1?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bfa64bc09e1</guid>
            <category><![CDATA[martech]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[metrics]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 03:00:37 GMT</pubDate>
            <atom:updated>2026-11-19T05:07:09.185Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mkpOAzperqr-twjhz5VKvw.jpeg" /></figure><p>Every lifecycle marketer eventually gets the same question from leadership: is this working? You’ve added a tool that decides which customers get which message, and when. Revenue is up. Engagement looks healthy. But how much of that change came from the technology, and how much would have happened anyway?</p><p>It’s a harder question than it looks. Most AI decisioning tools in use today, including propensity models and contextual bandits, learn from correlations in historical data. That makes it difficult for a model to separate the effect of its own decisions from everything else going on with a customer. The same challenge shows up one level higher, when a business tries to evaluate the vendor. Separating what a tool caused from what was already happening is genuinely difficult.</p><p>This post looks at why martech performance is so hard to measure, and what buyers can do to get a clearer picture.</p><h3>Measurement is hard even inside one company</h3><p>Attributing a change in customer behavior to a specific lifecycle campaign is difficult in its own right. Customers encounter many touchpoints, purchase cycles vary, and seasonality moves everything at once. Companies don’t agree with each other on a standard way to measure campaign impact, and it’s common for marketing, finance, and product teams within the same company to use different definitions of success.</p><p>When the internal yardstick isn’t settled, comparing results across vendors becomes harder still.</p><h3>Case studies tell part of the story</h3><p>Most martech vendors share results through case studies. That makes sense: a well-chosen example is the easiest way to show what a product can do. But case studies are built to highlight success, and the methodology behind the numbers usually isn’t published. With no objective industry benchmark, an impressive uplift figure can be hard to interpret without more context.</p><p>This isn’t unique to marketing technology. The ways statistics can mislead have been well documented for decades, and they tend to appear whenever results are summarized for a sales audience. Here are four patterns worth knowing, with a question to ask for each.</p><p><strong>Selective examples.</strong> A result drawn from the single best campaign, or a favorable time window, can look very different from typical performance.</p><p><strong>The baseline.</strong> Uplift is always measured against something. Comparing against a weak prior campaign, or against sending nothing at all, produces a bigger number than comparing against the best strategy already in place.</p><p><strong>What drove the change.</strong> Some gains come from factors other than the technology. A common example is message volume: sending more messages often produces a short-term revenue bump, but it can also accelerate unsubscribes and churn over time.</p><p><strong>The success metric.</strong> Opens, clicks, and downloads are easy to move and easy to report. Profitability, retention, and customer lifetime value are what the business actually cares about, and they’re slower and harder to measure.</p><p>None of these problems requires bad intent to manifest. Every team has an incentive to lead with its best results, and without independent review, nothing in the industry says otherwise.</p><h3>The information gap buyers face</h3><p>The bigger challenge is what isn’t shared. Few vendors publish validated average performance across their full customer base. That leaves buyers making significant investments based on examples they can’t independently verify.</p><p>We believe the industry would benefit from moving toward objective, verifiable measurement: clear methodology, controlled comparisons, and results reported against business outcomes. That’s a large topic, and no single company will solve it.</p><p>In the meantime, buyers can close much of the gap by asking the questions above, insisting on holdout groups, and evaluating results over enough time to see long-term effects. Vendors can help by building measurement into the product itself, so results can be checked rather than taken on faith.</p><p><em>Originally published at </em><a href="https://www.treatmachine.com/treat-machine-blog/did-it-actually-work-why-martech-performance-is-so-hard-to-measure"><em>https://www.treatmachine.com</em></a><em> on September 11, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bfa64bc09e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Limits of A/B Testing, and What Comes Next]]></title>
            <link>https://data-science-churn.medium.com/the-limits-of-a-b-testing-and-what-comes-next-8f3203fc863e?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f3203fc863e</guid>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Thu, 27 Aug 2026 04:00:38 GMT</pubDate>
            <atom:updated>2026-10-15T03:07:57.006Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FjvvjR0tQti2W2rt71xTng.jpeg" /></figure><p>“Right message, right person, right time” has been the working principle of marketing for the last decade. In practice, most organizations approximate it through a mix of experience, intuition, and workflows that were set up some years ago and rarely revisited since. A/B testing has been the discipline that keeps this process honest — and it remains valuable — but it has real structural limits worth understanding.</p><h3>What A/B testing does well, and where it runs out of room</h3><p>The A/B test is the rigorous foundation of marketing measurement, and its logic is sound: form a hypothesis, split traffic, wait for statistical significance, encode the winner as a rule, and repeat. Each cycle produces a defensible answer.</p><p>The constraint is throughput. By the time a test reaches significance and its result is implemented, customer behavior may already have shifted, and the “winner” is applied uniformly to the whole audience going forward, whether or not it’s still the best choice for every individual within that audience. A team can only run so many tests in parallel, and each one yields a single answer rather than an answer tailored to the person receiving it.</p><h3>A different way to frame the problem</h3><p>Much of the current conversation in marketing technology is focused on generative AI: producing new text and images. A separate, less visible shift is happening in a different part of the stack — not in generating content, but in deciding which content to send, to whom, and when. A useful analogy is a recommendation algorithm: the mechanism that decides what to surface next isn’t concerned with producing new content, only with selecting among what already exists.</p><p>This is the role Causal AI Decisioning (CAID) plays. Rather than guessing which message will resonate, it uses a brand’s own outcome data to estimate the effect of each possible action, and selects the one most likely to produce the result the brand actually wants — for that specific customer, not for the audience as a whole.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*twWDpsX_ZzMSx_vD.jpg" /></figure><h3>Moving past general-purpose assumptions</h3><p>Generative AI models are trained on the general text of the web, which makes them very good at producing fluent copy and not well suited to a narrower, more commercially specific question: would offering this particular customer a 10% discount actually change their behavior, or would they have converted regardless? Causal decisioning is built to answer exactly that question, using a brand’s own first-party data and learning continuously from the outcomes of its own decisions.</p><h3>Extending the test-and-learn cycle rather than abandoning it</h3><p>None of this makes A/B testing obsolete — its underlying discipline is exactly what makes causal decisioning possible in the first place. What changes is the object of optimization. Instead of identifying the single best-performing subject line across an entire audience, the system estimates the best subject line for each individual, and updates that estimate continuously as behavior changes, rather than waiting for the next full test cycle. In practice, this also reduces the amount of time teams spend reconciling dashboards after the fact, freeing more of that time for strategy and customer-facing work.</p><h3>Where this applies</h3><p>If a brand already runs a marketing automation platform — Klaviyo, Iterable, Marketo, or similar — and can observe whether a given customer converted, the two prerequisites for this approach are already in place.</p><ul><li><strong>Nurture and lifecycle: </strong>deliver the message most likely to move a specific lead or trial user forward, rather than the one that performs best on average.</li><li><strong>Retail and DTC: </strong>recommend the product most likely to convert a given shopper, not simply the one that’s broadly popular.</li><li><strong>Subscription and SaaS: </strong>distinguish the customer who’s ready for an upgrade from the one who needs a retention offer, and reach each at the right moment rather than defaulting to the same prompt for everyone.</li><li><strong>Churn prevention: </strong>separate a persuadable customer from one who has already decided to leave, and direct retention budget toward the former.</li></ul><h3>The distinction that matters</h3><p>Most personalization tools available today are, underneath the marketing language, rule-based systems built on correlation: useful, but fundamentally observational. Causal AI decisioning is interventional — it estimates the effect of the action itself, rather than simply predicting an outcome, which is what allows it to select the action that actually changes where a customer ends up. That’s the shift that matters, and teams already running disciplined A/B tests are, in a meaningful sense, closer to it than they might expect.</p><p><em>Originally published at </em><a href="https://www.treatmachine.com/treat-machine-blog/the-limits-of-a/b-testing-and-what-comes-next"><em>https://www.treatmachine.com</em></a><em> on August 27, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f3203fc863e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Causal AI Decisioning?]]></title>
            <link>https://data-science-churn.medium.com/treat-machine-causal-marketing-automation-5151392b731a?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5151392b731a</guid>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 17:12:27 GMT</pubDate>
            <atom:updated>2026-09-25T02:30:11.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoMmudoWOZdL3pcwDBOWSg.jpeg" /></figure><p>For the last decade, “right message, right person, right time” has been the marketer’s mantra. Most teams get closer to it every year through experience, judgment, and workflows built up over time. AI Decisioning is a way to keep that learning going automatically, and to do it at the level of the individual customer.</p><h3>The quieter revolution alongside Generative AI</h3><p>Generative AI creates new text and images, and it is now genuinely good at writing the ten versions of an offer email that a skilled copywriter would produce. Deciding which of those versions goes to which customer is a different job. It calls for different evidence: not what reads well, but what actually moved the outcome last time you tried it. Causal AI Decisioning mathematically selects the action most likely to drive a specific result, then learns from what came back.</p><p>This is similar to the division of labor in a marketing team: Designers and copywriters produce the concepts, and they are the reason you have good options at all. Then the Lifecycle Marketing/CRM manager decides which one actually goes out, to whom, and watches the results to decide what goes out next. Both jobs are skilled and neither substitutes for the other.</p><h3>Your data, not the internet’s</h3><p>Generative AI is built on the general wisdom of the web, and it is genuinely good at that. It can write a polite email in seconds. What it cannot tell you is whether offering Customer A a 10% discount will keep them from churning, or whether that customer was going to stay anyway.</p><p>Causal AI Decisioning answers that question using your proprietary data, learning from the results of its own actions. Every interaction adds information, so the system improves as it runs.</p><p><strong>AI Decisioning</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BhbeNno5ejH-WZVj.jpg" /></figure><p><em>Unlike static workflows, AI Decisioning gets smarter with every interaction.</em></p><h3>Building on the A/B test</h3><p>A/B testing is the rigorous foundation of modern marketing measurement, and the discipline behind it is exactly what makes AI Decisioning possible. The classic loop looks like this:</p><ol><li>Form a hypothesis</li><li>Split traffic</li><li>Wait for statistical significance</li><li>Encode the winner as a rule</li><li>Repeat</li></ol><p>Every cycle produces a trustworthy answer. The constraint is throughput. A team can only run so many tests, and each one yields a single winner for the whole audience.</p><p>Causal AI Decisioning applies the same logic continuously and at scale. Rather than finding the best subject line on average, it learns the best subject line for each individual, and it keeps learning as behavior shifts. Your experiments still matter, and their results compound without waiting for the next test cycle to finish. It also gives time back: less time spent reconciling spreadsheets and dashboards, more time on customers and strategy.</p><h3>Where it works well</h3><p>If you have a marketing automation platform (Klaviyo, Iterable, Marketo) and a feedback loop that tells you whether someone bought, you already have what you need to start.</p><ul><li><strong>Nurture: </strong>deliver the message that resonates on a 1:1 basis, moving lead to deal, or trial to paid.</li><li><strong>Retail: </strong>recommend the item that triggers a purchase for that specific user, not only the popular one.</li><li><strong>SaaS: </strong>identify who is ready for an upgrade and who would respond better to a retention offer, then reach them when it matters.</li><li><strong>Churn prevention: </strong>tell the difference between a persuadable customer and a lost cause, and put your budget where it changes the outcome.</li></ul><h3>What makes it different</h3><p>Many personalization engines today work from rules and correlations, which is a sensible place to start and a real improvement over sending everyone the same thing. Causal AI Decisioning takes the next step by being interventional. It measures the effect of the action itself, so it can choose the action that changes where the customer ends up, not just the one that predicts it.</p><p>That is the shift worth understanding, and if you are already running experiments and tracking outcomes, you are closer to it than you might think.</p><p><em>Originally published at </em><a href="https://www.treatmachine.com/treat-machine-blog/what-is-ai-decisioning"><em>https://www.treatmachine.com</em></a><em> on July 30, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5151392b731a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Churn Simulation Advances — Fighting Churn WIth Data]]></title>
            <link>https://data-science-churn.medium.com/churn-simulation-advances-fighting-churn-with-data-2296c915208c?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2296c915208c</guid>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[churn-rate]]></category>
            <category><![CDATA[simulation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Sun, 22 Oct 2023 22:19:07 GMT</pubDate>
            <atom:updated>2023-10-22T23:41:23.453Z</atom:updated>
            <content:encoded><![CDATA[<h3>Churn Simulation Advances — Fighting Churn With Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/0*kmK6Npvo1QQSpyu6" /><figcaption>Figure 1: Overview the churn simulation.</figcaption></figure><p>This post introduces the new advanced customer churn simulation for learning the techniques of Fighting Churn with data (FCWD). The book was originally released with a simple simulation. The simple simulation served a purpose, but was lacking: Some of the code from Fighting Churn with Data either would not run or produced meaningless results on the simple simulation. But the recently advanced churn simulation solves this. Now, all of the code in Fighting Churn With Data can be studied, tested and practiced using the new simulation. This tablelists the new features of the simulation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBcWbbzGrRhwsY7EKEZ5fg.png" /><figcaption>Table 1: New Features of the Advanced Churn Simulation</figcaption></figure><p>This means two things for users:</p><ol><li>You can practice all the techniques in Fighting Churn With Data.</li><li>The data is challenging enough to test machine learning models</li></ol><p>All of the simulation code is available <a href="https://github.com/carl24k/fight-churn">open source on github</a>. And you can customize the simulation to generate unique data sets. Customizing the simulation code will be the subject for a future post. For more information on the advanced churn simulation check out the <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4540160">ChurnSim White Paper hosted on SSRN</a> (the Social Science Research Network.)</p><h3>Churn Simulation Overview</h3><p>Figure 1 (above) summarizes the ChurnSim Mode. Customers choose product plans and take actions. Customers take actions at a rate determined by a Log-Normal behavioral model. Customer actions provide positive or negative utility. This is utility in the economic sense, meaning usefulness or enjoyment. Next, the customer’s utility determines if they churn, or upgrade or downgrade. Finally, the observable data is written to a simple but realistic churn database.</p><h3>Realistic Churn Rate Simulation</h3><p>A realistic set of churn rates is one result of the new simulation, as shown in the figure below. Figure 2 shows a realistic simulation of churn rates for an enterprise SaaS company: Firstly, the MRR churn rate is higher than the logo (count based) churn rate. Secondly, the Net retention rate is greater than 100%. Net retention greater than 100% is sometimes called “negative churn”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*VON-RDJZnM5N0fkq" /><figcaption>Figure 2: Comparison of realistic churn rates generated by the simulation.</figcaption></figure><p>The realistic relationship between churn rates result from careful construction of the simulation. The simulation balances the utility that customers receive with the price they pay for accounts with different numbers of users. To learn more about churn rate calculation, check out this <a href="https://www.fightchurnwithdata.com/2019/01/01/calculate-churn-rate-sql/">post about churn rate calculation with SQL</a>.</p><h3>Realistic Behavior Model in a Churn Simulation</h3><p>The Log-Normal behavior rate model makes the simulated behavior frequency realistic. Figure 3 shows that the frequency has a long tail: A few simulated customers have rates far above the mean. Further, the log-log plot (right) shows that the frequency of high rates follows a power law.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*CI0RI7dkiw32CjEp" /><figcaption>Figure 3: Long tail / power law distribution of customer behaviors</figcaption></figure><p>More and more actions by the customer has no impact on the churn rate. Figure 4 shows the diminishing returns to customer actions: A few actions per month leads to a big churn rate reduction. However, after a certain point, additional actions have no impact. This diminishing returns is a core part of the simulation model for customer utility. <a href="https://www.fightchurnwithdata.com/2019/07/27/understand-churn-with-behavioral-cohorts/">For more information, check out this post on customer behavioral analysis.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/0*9gY3KGgt_eTE1ysn" /><figcaption>Figure 4: Diminishing Churn returns in churn reduction</figcaption></figure><h3>Realistic Machine Learning Results</h3><p>One interesting use of the simulation is analyzing variability in the results of machine learning models. Figure 5 shows the results of SHAP analysis for an XGBoost model trained on three different instantiations of the data set. The top three most important features are the same for every model. However, after the top three features there is a lot of variability in the ranking. To learn more, read this post on <a href="https://www.fightchurnwithdata.com/2021/03/31/explainable-machine-learning-churn-prediction/">explainable machine learning with SHAP analysis</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*u80lkhEMFCxfSbZ8" /><figcaption>Figure 5: Results of SHAP analysis for an XGBoost model trained on three different instantiations of the data set.</figcaption></figure><h3>Learn More!</h3><p>In conclusion, the new simulation has a lot of great feature realistic details. By using the new simulation students and machine learning researchers can learn a lot about customer churn. For more information, check out the <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4540160">ChurnSim White Paper hosted on SSRN</a> (the Social Science Research Network.) You can also <a href="https://github.com/carl24k/fight-churn">download the simulation code from github.</a> Lastly, if you want to setup and run the churn simulation yourself watch the demonstration video:</p><ul><li><a href="https://www.twitch.tv/videos/1933481374">Twitch VOD</a></li><li><a href="https://youtu.be/RNXeB8qV-_g?si=JU62tQmrAFIQsT0u">You Tube</a></li></ul><p><em>Originally published at </em><a href="https://www.fightchurnwithdata.com/2023/10/22/churn-simulation/"><em>https://www.fightchurnwithdata.com</em></a><em> on October 22, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2296c915208c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Churn Data Science Strategy — Fighting Churn WIth Data]]></title>
            <link>https://data-science-churn.medium.com/churn-data-science-strategy-fighting-churn-with-data-a69529ebaae0?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a69529ebaae0</guid>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[churn]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Sat, 03 Sep 2022 01:36:44 GMT</pubDate>
            <atom:updated>2023-02-04T23:12:31.339Z</atom:updated>
            <content:encoded><![CDATA[<h3>Churn Data Science Strategy — Fighting Churn WIth Data</h3><p>Churn means that your customers cancel your service, or stop spending money on your site or app. Churn is the most common Data Science problem in the world. That’s because nowadays every company has a churn problem! All companies want to keep their customers coming back.</p><h3>Top 3 Challenges for Data Science with Churn</h3><p>Most data scientists have fit a churn model in a class or bootcamp. But really fighting churn is much harder than it sounds.</p><ol><li>Churn is Hard to predict. It may be easy to know a customer is at risk. But it is very hard to predict the timing. There is too much randomness and stuff you don’t know.</li><li>Churn is Harder to prevent. Because customers know the product. You can’t fool them with marketing.</li><li>Churn is Hard to communicate about and organize around.</li></ol><p>Your churn strategy can use data science, but it’s not as simple as it sounds.</p><h3>Churn Data Science Strategy Pyramid</h3><p>You can think of your churn data science strategy like a pyramid.</p><ul><li>The foundation is event data. You store it in a data warehouse or data lake house. Its like a foundation in a house, because you need to have it but a foundation alone does not do that much for you.</li><li>The first level is calculating customer metrics. You will use customer metrics for feature engineering in a model, but they have many more uses in your company. This level also includes calculating your churn rate.</li><li>The next level is to do basic segments and targeting using your metrics. A/B testing is also done at this level.</li><li>The pinnacle of the pyramid is AI. Or machine learning, or automation — whatever you want to call it. This is the most advanced level. AI for automated churn reduction has a lot of pitfalls — your actions will create bias in the model. You need to constantly retrain, and include information about your actions in the model. Reinforcement learning is a natural fit for optimizing the impact of interventions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*FORpynM6eSIS0i2h" /></figure><h3>The Good News</h3><p>You can get a lot of good results with not that much effort. You may have heard of the Pareto principal: 80% of the effect from 20% of the causes. People take it to mean you can get a lot from a little effort. It’s not quite that good with churn and data science. But consider the pyramid above.</p><ul><li>You can get 50% of possible churn reduction with 25% of the effort. That corresponds to Calculate churn and customer metrics from data and look at them</li><li>You can get 75% of possible churn reduction with 50% of the effort. That means interventions targeted by simple metrics and some low complexity A/B testing.</li><li>To get the maximum possible churn reduction using AI/ML and automation is 50% more of the total effort.</li></ul><h3>Get More Info</h3><p>To learn more check the <a href="https://fightchurnwithdata.com/churn-blog/">Churn Blog</a> or the book, <a href="https://www.manning.com/books/fighting-churn-with-data?a_aid=fight-churn&amp;a_bid=61baa361">Fighting Churn With Data</a>.</p><p><em>Originally published at </em><a href="https://www.fightchurnwithdata.com/2022/09/02/churn-data-science-strategy/"><em>https://www.fightchurnwithdata.com</em></a><em> on September 3, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a69529ebaae0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explainable Machine Learning Churn Prediction — Fighting Churn With Data]]></title>
            <link>https://data-science-churn.medium.com/explainable-machine-learning-churn-prediction-fighting-churn-with-data-895dcb0b01c?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/895dcb0b01c</guid>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Wed, 31 Mar 2021 15:14:26 GMT</pubDate>
            <atom:updated>2021-04-05T00:49:08.558Z</atom:updated>
            <content:encoded><![CDATA[<h3>Explainable Machine Learning Churn Prediction — Fighting Churn With Data</h3><p>In my previous story on <a href="https://medium.com/@carl.steven.gold/forecasting-churn-risk-with-machine-learning-part-2-fighting-churn-with-data-f45ea5646f49">Machine Learning for Churn prediction</a>, I showed that Machine Learning models (ML also known as AI) are the most accurate. But in the past, machine learning models suffered from being hard to explain. In my book, <a href="https://www.amazon.com/Fighting-Churn-Data-Carl-Gold/dp/161729652X">Fighting Churn With Data</a>, I wrote that you should use Logistic Regression to explain churn. But I also showed that the machine learning model XGBoost gave higher accuracy. Now I recently learned about a great new technique known as SHAP! SHAP makes machine learning and AI models more explainable than they have ever been! SHAP stands for SHapely Additive exPlanations. So in this post I will demonstrate using SHAP to achieve explainable machine learning churn prediction.</p><p>SHAP analyzes how much each metric (or “feature” or “independent variable”) contributes to a machine learning prediction as if the variable were a player on a team. The analysis depends on looking at the predictions with each subset of features. It relies on clever algorithms that solve the problem exactly for tree models, and approximates it for other models. There are many good articles giving you an explanation of SHAP, so I will focus on the application to churn. I recommend:</p><ul><li><a href="https://towardsdatascience.com/explain-your-model-with-the-shap-values-bc36aac4de3d">This blog post by Dr. Dataman on Medium which explains the general theory of SHAP values</a>.</li><li><a href="https://shap.readthedocs.io/en/latest/index.html">The documentation of the SHAP Python package</a>.</li></ul><h3>The SHAP Explanation for XGBoost Machine Learning Churn Prediction</h3><p>Below is an example of the output of SHAP for the churn case study from the book, <a href="https://www.manning.com/books/fighting-churn-with-data">Fighting Churn With Data</a>. This gives a visual illustration of the contribution of each metric in the model.</p><ol><li>The features are listed on the left: Each row are the results for one feature</li><li>The colored bars are stacked dots representing the contribution of each feature to each example in a set of predictions.</li><li>The vertical line in the center shows the baseline model prediction. If a point is to the left of the line, for that example the feature contributes to reducing the model output prediction. If a point is to the right of the line, for that example the variable contributes to increasing the model output prediction.</li><li>The color of each point represents whether the point is from an example where that variable (feature) is high (red) or low (blue.)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*pTeg1IPIH_ASYK-b.png" /></figure><p>This is a model for churn probability, which is the model output. These are a summary of the key findings:</p><ul><li>Message per month are New Friend per month are the two most influential variables. High values of these measurements are associated with lower values of the churn probability.</li><li>Adview per post is associated with increasing the churn risk forecast for those examples with high values on this metric. But this metric is less influential than Message per Month and New Friend per month — you can see this by noting that the stacked dots are never as far from the 0.0 influence line as they are for Messages and New Friends. Dislike percent is similar, and somewhat less influential.</li></ul><h3>Why this is an improvement over previous methods</h3><p>If you have ever used other methods to look at the importance and influence of features in XGBoost then you will recognize what a big advance this represents! (The methods were equally bad for ensembles of decision trees like Random Forest.) You can try try this yourself by calling the function from the xgboost module: xgboost.plot_importance(xgb_model)). That older approach will tell you what was the most important according to a few different definitions of importance. But those methods do not give you a sense of the direction or magnitude of the influence each variable in the model. Understanding the direction and magnitude of influence is key to explainability. So this is a big improvement!</p><h3>Code for SHAP Explainable XGBoost Machine Learning Churn Prediction</h3><p>Now lets look at the code. You can find this example in the Fighting Churn With Data Github repo here: &lt;TODO add the link. The code assumes there is already a saved XGBoost model and a data set. The data is necessary because the explanation runs on actual examples. But the analysis is done in 3 lines of code! It’s that easy.</p><pre>import pickle import shap import pandas as pd def shap_explain_xgb(model_pickle_path, data_set_path): # loading saved model and pickle with open(model_pickle_path, &#39;rb&#39;) as fid: xgb_model = pickle.load(fid) current_df = pd.read_csv(data_set_path) # create the SHAP explanation summary explainer = shap.Explainer(xgb_model) shap_values = explainer(current_df) shap.summary_plot(shap_values, current_df, show=False)</pre><h3>SHAP for Logistic Regression Churn Prediction</h3><p>For comparison, here is the result from using SHAP on the Logistic Regression model. For this model, the result was already explainable using the model coefficients (as explained previously in my story on <a href="https://medium.com/@carl.steven.gold/customer-churn-probability-fighting-churn-with-data-36d91b08fa00">Customer churn probability</a> forecasting). You can see that both XGBoost and Logistic Regression find many of the same variables to be influential:</p><ul><li>New friend per month is the most influential, and high values reduce the churn probability.</li><li>Metric group 2, a result of dimension reduction, consists of the combination of messages and replies. (Explained in the post on <a href="https://fightchurnwithdata.com/customer-behavior-correlation-and-churn/">Customer Behavior Correlation and Churn</a>.)</li><li>Adview per post is (like in the XGBoost case) the most influential metric that is negative (in the sense of high values increasing the churn probability.)</li></ul><p>There are other differences of course, most notably the very different shape of the influence! For XGBoost (above) the influence of the variable is exactly the same for many points, because XGBoost is based on decision trees: For one cut point in a tree, all values above the cut point may reach the same terminal node. But Logistic Regression is a linear model, and increasing one measure always results in a monotonic increase in the influence. This results in the more continuous type of influence shown in the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*wDlqrhnmgmetbNaJ.png" /></figure><h3>Adding Explainable Machine Learning To Fighting Churn With Data</h3><p>Unfortunately, when I wrote Fighting Churn With Data I did not know about using SHAP to explain XGBoost. So in the book, I said to use Logistic Regression as the sole method for interpreting the influence of the metrics on churn. So this new method will be a great improvement for the second edition! For now, I have included the code to produce SHAP plots for XGBoost and Logistic Regression along with rest of the code for the book in the Github repository. You can find them as the last two listings under chapter 9:</p><ol><li><a href="https://github.com/carl24k/fight-churn/blob/master/listings/chap9/listing_9_8_shap_explain_xgb.py">listings/chap9/listing_9_8_shap_explain_xgb.py</a></li><li><a href="https://github.com/carl24k/fight-churn/blob/master/listings/chap9/listing_9_9_shap_explain_logreg.py">listings/chap9/listing_9_9_shap_explain_logreg.py</a></li></ol><p>If you have the rest of the code from the book running, it should be no problem for you to run those as well. If you want more information and details about using XGBoost to predict churn, check out <a href="https://www.manning.com/books/fighting-churn-with-data?a_aid=fight-churn&amp;a_bid=61baa361">Chapter 9 of my book, Fighting Churn With Data</a>. Its packed with more information about analyzing churn, coming up with good metrics (features) for predicting churn, and using all that information to actually reduce churn in practice.</p><p><em>Originally published at </em><a href="https://fightchurnwithdata.com/explainable-machine-learning-churn-prediction/"><em>https://fightchurnwithdata.com</em></a><em> on March 31, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=895dcb0b01c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Churn Risk with Machine Learning, Part 2 — Fighting Churn With Data]]></title>
            <link>https://data-science-churn.medium.com/forecasting-churn-risk-with-machine-learning-part-2-fighting-churn-with-data-f45ea5646f49?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f45ea5646f49</guid>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[forecasting]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Sat, 11 Jul 2020 21:14:13 GMT</pubDate>
            <atom:updated>2020-07-11T21:14:13.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>Forecasting Churn Risk with Machine Learning, Part 2 — Fighting Churn With Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*vtpb2jyNhjWknilh2KApKQ.jpeg" /><figcaption>Fighting Churn With Data: The Science and strategy of customer retention</figcaption></figure><p><em>From </em><a href="https://www.manning.com/books/fighting-churn-with-data?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07"><em>Fighting Churn with Data</em></a><em> by Carl Gold</em></p><p><em>This article discusses using machine learning algorithms to forecast churn risks.</em></p><p><a href="https://medium.com/@carl.steven.gold/forecasting-churn-risk-with-machine-learning-part-1-manning-217bc190d0e1?source=friends_link&amp;sk=f1f0c94507b080a46af18dabaa31ec5c">Continued from Part 1</a>, in which the XGBoost machine learning model and how it can be used to predict churn is presented. This post demonstrates the code and some results for real data. All the code from Fighting Churn With Data can be found at <a href="https://github.com/carl24k/fight-churn">https://github.com/carl24k/fight-churn</a>. Below, chapter 9 listing 6 shows XGBoost cross-validation for regression in python.</p><pre>import pandas as pd <br>import pickle from sklearn.model_selection <br>import GridSearchCV, TimeSeriesSplit from sklearn.metrics <br>import make_scorer <br>import xgboost as xgb #A </pre><pre>from listing_8_2_logistic_regression import prepare_data <br>from listing_9_2_top_decile_lift import calc_lift</pre><pre><br>def crossvalidate_xgb(data_set_path,n_test_split): <br>    <br>    X,y = prepare_data(data_set_path,ext=&#39;&#39;,as_retention=False) #B <br>    tscv = TimeSeriesSplit(n_splits=n_test_split)<br>    xgb_model = xgb.XGBClassifier(objective=&#39;binary:logistic&#39;) #C</pre><pre>    score_models = {<br>        &#39;lift&#39;: make_scorer(calc_lift, needs_proba=True), <br>        &#39;AUC&#39;: &#39;roc_auc&#39;<br>    } </pre><pre>    test_params = { <br>        &#39;max_depth&#39;: [1,2,4,6], #D <br>        &#39;learning_rate&#39;: [0.1,0.2,0.3,0.4], #E <br>        &#39;n_estimators&#39;: [20,40,80,120], #F <br>        &#39;min_child_weight&#39; : [3,6,9,12]<br>    } #G <br><br>    gsearch = GridSearchCV(estimator=xgb_model,n_jobs=-1,     <br>                           scoring=score_models, #H <br>                           cv=tscv, <br>                           verbose=1, <br>                           return_train_score=False, <br>                           param_grid=test_params,<br>                           refit=&#39;AUC&#39;) #I <br>    gsearch.fit(X.values,y) #J </pre><pre>    result_df = pd.DataFrame(gsearch.cv_results_) #K    <br>    result_df.sort_values(&#39;mean_test_AUC&#39;, <br>                          ascending=False,inplace=True) #L <br>    save_path = data_set_path.replace(&#39;.csv&#39;, &#39;_crossval_xgb.csv&#39;)    <br>    result_df.to_csv(save_path, index=False) <br>    pickle_path = data_set_path.replace(&#39;.csv&#39;, &#39;_xgb_model.pkl&#39;) #M    <br>    with open(pickle_path, &#39;wb&#39;) as fid:       <br>        pickle.dump(gsearch.best_estimator, fid) #N </pre><ul><li><strong>A Imports XGBoost, which is in a separate package</strong></li><li><strong>B Most of this function is the same as listing 9.5, the regression cross-validation.</strong></li><li><strong>C Creates an XGBClassifier object for a binary outcome</strong></li><li><strong>D Tests tree depths from 1 to 6</strong></li><li><strong>E Tests learning rates from 0.1 to 0.4</strong></li><li><strong>F Tests number of estimators from 20 to 120</strong></li><li><strong>G Tests minimum weight from 3 to 12</strong></li><li><strong>H Creates the GridSearchCV object with the XGBoost model object and tests parameters</strong></li><li><strong>I Refits the best model according to AUC after cross-validation</strong></li><li><strong>J Passes as values, not a DataFrame, to avoid a known package issue at the time of this writing</strong></li><li><strong>K Transfers the results to a DataFrame</strong></li><li><strong>L Sorts the result to get the best AUC first</strong></li><li><strong>M Creates a pickle of the best result</strong></li><li><strong>N The best result is in the best_estimator field of the GridSearchCV object.</strong></li></ul><p>The main steps are</p><ol><li>Prepares the data</li><li>Creates a model instance (in this case, an XGBoost model)</li><li>Defines the accuracy measurement functions to use (lift and AUC)</li><li>Defines the sequences of parameters to test</li><li>Passes these in the GridSearchCV object and calls the fit function</li><li>Saves the results (with no additional analysis like in the regression cross validation).</li></ol><p>One important and slightly subtle difference between listing 1 a regression cross-validation is that the dataset would be created from the original unscaled metrics, and it doesn’t use scores or groups like you would for the regression. Re-scaling the metrics for XGBoost (or decision trees, generally) is pointless because the cut points in the rules operate as well on the metrics regardless of scale or skew. And grouping correlated metrics doesn’t provide any benefit. In fact, it can hurt the performance of this type of machine learning model. Grouping correlated metrics is beneficial for interpretation and helps avoid the problems that correlated metrics can cause in regression.</p><p>On the other hand, for XGBoost, a diversity of metrics is beneficial and correlation does no harm. If two metrics are correlated, either one can make a suitable rule node in a tree. For these reasons, the prepare_data function is called with an empty extension argument, so that it loads the original dataset rather than loading the grouped scores (the default behavior).</p><p><strong>Comparison of XGBoost accuracy to regression</strong></p><p>Because XGBoost takes much longer to fit the larger number of parameters, you should expect that it provides some improvement in the forecasting accuracy. This expectation is confirmed in figure 1, which compares the AUC and lift achieved by regression and XGBoost models for the simulation data from Fighting Churn With Data, as well as three real company case study datasets for the companies. The AUC improvement ranges from 0.02 to 0.06, and XGBoost always produces more accurate forecast than the regression. In terms of lift, the improvement is from 0.1 to 0.5.</p><p>Are those improvements significant? Remember that the full range of AUCs you’re likely to see in churn forecasting is from around 0.6 to 0.8. The maximum AUC, therefore, is 0.2 more than the minimum, and in relative terms, an improvement of 0.02 in AUC represents around a ten percent improvement in terms of the overall possible range. By the same token, a 0.05 improvement in AUC represents twenty-five percent of the difference between worst and best in class, and these really are significant improvements. Still, the forecasting isn’t perfect, even with machine learning. This is why I advise that predicting churn with machine learning isn’t likely to live up to some of the hype in the machine learning field.</p><p><strong>TAKEAWAY Though machine learning algorithms can produce forecasts that are significantly more accurate than regression, churn is always hard to predict due to factors like subjectivity, imperfect information, rarity, and extraneous factors influencing the timing of churn.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cTK4FIUlq4bPbD3u.png" /></figure><p><strong>Comparison of advanced and basic metrics</strong></p><p>Another important question that you might have been wondering is how much improvement in accuracy can be attributed to advanced metrics? The relationship to churn in cohort analysis must have improved the model, but like you want to validate your data and modeling by showing your model can predict out of sample, it also makes sense to confirm that the work you did creating more metrics contributed something empirically.</p><p>In order to make the comparison on the simulated social network datasets, there are additional versions of the cross-validation testing command which run on the original dataset. That is, you run the dataset without advanced metrics or the basic metrics. To run the regression cross-validation on the basic metric dataset from Fighting Churn With Data, use the following:</p><pre>fight-churn/listings/run_churn_listing.py --chapter 9 --listing 5 --version 1</pre><p>That produces a cross-validation table. You’ll probably find that the maximum accuracy of any model is somewhat less for the data with only basic metrics compared to the data with advanced metrics. As illustrated in the bar chart in figure 2, the maximum accuracy that I got on my regression simulation with only basic metrics was 0.63; for the regression on the simulated data with advanced metrics, the maximum AUC was 0.75: a 0.12 difference. The time spent creating advanced metrics was well spent. In fact, the regression accuracy with advanced metrics is significantly better than XGBoost with basic metrics, and the additional improvements from the machine learning algorithm are relatively small.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gcbs8CLPG5KzohHf.png" /></figure><p>You can also perform the same check on the XGBoost model by running the second version of the XGBoost cross-validation command with these arguments:</p><pre>fight-churn/listings/run_churn_listing.py --chapter 9 --listing 6 --version 1</pre><p>In this case, you’ll probably find that the XGBoost forecasts did a bit better with the advanced metrics. For my simulation, I got an AUC of 0.774 using the XGBoost with the basic metrics alone as compared with 0.797 for XGBoost with advanced metrics: the improvement attributable to advanced metrics is 0.023.</p><p>Figure 4 also contains similar comparisons for forecasts made on three real company case studies introduced. These each show different relationships between accuracy with and without the advanced metrics. These three cases illustrate the range of scenarios you can encounter on your own case studies:</p><ol><li>For the first case study, the regression accuracy is significantly improved by the advanced metrics, but the XGBoost doesn’t get any improvement, and XGBoost is best overall. This shows you can’t always expect advanced metrics to improve machine learning.</li><li>For the second case study, both the regression and the XGBoost are significantly improved by the addition of advanced metrics. The regression accuracy with the advanced metrics is about the same as the XGBoost accuracy with only basic metrics. The XGBoost accuracy with advanced metrics is the highest of all by a significant amount: around 0.1 more than regression with basic metrics.</li><li>For the third case study, the regression using advanced metrics has higher accuracy than XGBoost without advanced metrics. But the highest accuracy is achieved by XGBoost using advanced metrics: more than 0.1 improvement over the basic metrics and regression. This case study is most similar to the social network simulation.</li></ol><p>These cases demonstrate that if high accuracy on churn forecasts is a high priority for you, then both machine learning and advanced metrics are important. In my experience, advanced metrics usually improve the accuracy of churn forecasts for both regression and machine learning models like XGBoost.</p><p>That’s all for this article. If you want to learn more about the book, you can preview its contents on our browser-based liveBook platform <a href="https://livebook.manning.com/book/fighting-churn-with-data?origin=product-look-inside&amp;utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07">here</a>. Take 37% off <a href="https://www.manning.com/books/fighting-churn-with-data?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07"><em>Fighting Churn with Data</em></a> by entering <strong>fccgold</strong> into the discount code box at checkout at <a href="https://www.manning.com/?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07">manning.com</a>.</p><p><em>Originally published at </em><a href="https://freecontent.manning.com/forecasting-churn-risk-with-machine-learning-part-2/"><em>https://freecontent.manning.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f45ea5646f49" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting Churn Risk with Machine Learning, Part 1 | Manning]]></title>
            <link>https://data-science-churn.medium.com/forecasting-churn-risk-with-machine-learning-part-1-manning-217bc190d0e1?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/217bc190d0e1</guid>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[customer-success]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 22:48:59 GMT</pubDate>
            <atom:updated>2020-07-18T19:01:58.791Z</atom:updated>
            <content:encoded><![CDATA[<h3>Forecasting Churn Risk with Machine Learning, Part 1 — Fighting Churn With Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*vtpb2jyNhjWknilh2KApKQ.jpeg" /><figcaption>Fighting Churn With Data — The Science and Strategy of Customer Retention</figcaption></figure><p><em>From </em><a href="https://www.manning.com/books/fighting-churn-with-data?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07"><em>Fighting Churn with Data</em></a><em> by Carl Gold</em></p><p><em>This article discusses using machine learning algorithms to forecast churn risks. In this article, I’m going to introduce the basic ideas of machine learning (ML) and a particular algorithm called XGBoost. If you are already familiar with these topics then skip to </em><a href="https://medium.com/@carl.steven.gold/forecasting-churn-risk-with-machine-learning-part-2-fighting-churn-with-data-f45ea5646f49"><em>my next post which dives into the details of using these methods to forecast churn</em></a><em>.</em></p><p><strong>Forecasting churn risk with machine learning</strong></p><p>You can forecast churn with <a href="https://medium.com/@carl.steven.gold/customer-churn-probability-fighting-churn-with-data-36d91b08fa00">a regression (as described in my previous post)</a> in which predictions are made by multiplying metrics by a set of weights. You can also predict churn with other kinds of forecasting models which are collectively known as <em>machine learning</em>. |No official definition of what constitutes a machine learning model exists, but for the purpose of this article, I use the following:</p><p><strong>DEFINITION A <em>machine learning model</em> is any predictive algorithm which has the following two characteristics: 1) the algorithm learns to make the prediction by processing sample data (as compared with making predictions with rules set by a human programmer), and 2) the algorithm isn’t the regression algorithm.</strong></p><p>The second condition can seem strange because the regression algorithm certainly meets the first condition. This is really a historical distinction because the regression approach predates machine learning methods by decades.</p><p><strong>The XGBoost learning model</strong></p><p>This book covers only one machine learning algorithm named XGBoost, but the same techniques for fitting the model and forecasting apply to most other algorithms you may consider. The XGBoost algorithm is based on the concept of a <em>decision tree</em>, illustrated in figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xn1xmOB_FlukT2D3.png" /></figure><p><strong>DEFINITION A <em>decision tree</em> is an algorithm for predicting an outcome (such as a customer churning or not churning) that consists of a binary tree made up of rules or tests.</strong></p><p>Each test in a decision tree takes a single metric and checks whether it is greater than or less than a predetermined cut point. The prediction for an observation (of a customer) is determined by starting at the root of the tree and performing the first test. The result of the test determines which of the two branches to follow from the node leading to one of the second level tests. The result of all the tests determines a path through the tree, and the leaves of the tree each has a designated prediction.</p><p>Small decision trees seem simple, and they used to be seen as an easy-to-interpret machine learning model. But in practice, large decision trees for datasets with many metrics become hard to interpret. Fortunately, no one has to read the rules in the tree to make a prediction.</p><p>An algorithm is used to test metrics and decide on the cut points to optimize performance when making predictions using the sample data. If a backtest shows the results are accurate, you can make predictions using a decision tree without being too concerned about the substance of the rules, but there are methods to interpret a decision tree. If you have more than a few metrics, then understanding the influence of metrics on the likelihood of churn is more easily done through grouping and regression methods, but I won’t spend time on interpreting decision trees.</p><p>Apart from being difficult to interpret, decision trees are no longer state-of-the-art in terms of prediction accuracy. But decision trees are the building blocks for more accurate machine learning models. One example of a more accurate machine learning model is a random forest<em>,</em> illustrated in figure 2.</p><p><strong>DEFINITION A <em>random forest</em> is an algorithm for predicting an outcome such as a customer churning by randomly generating a large set of decision trees (a “forest”). All the trees try to predict the same outcome, but each does this according to a different set of learned rules. The final prediction is made by averaging the predictions of the forest.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q8djTK_K5TqQ8Obz.png" /></figure><p>The random forest is an example of what’s called an <em>ensemble</em> prediction algorithm because the final prediction is made from the combination of a group of other machine-predicting algorithms. <em>Ensemble</em> means a group evaluated as a whole rather than individually. A random forest is a simple type of ensemble in that each tree gets an equal vote in the outcome, and additional trees are added at random. <em>Boosting</em> is a name for machine learning algorithms that make some important improvements over ensembles like random forest.</p><p><strong>DEFINITION <em>Boosting</em> is a machine learning ensemble in which the ensemble members are added to correct the errors of the existing ensemble.</strong></p><p>Rather than randomly adding decision trees like in a random forest, each new tree in a boosting ensemble is created to correct wrong answers made by the existing ensemble, rather than re-predict on the already correct examples. Internal to the boosting algorithm, successive trees are generated on subsamples of the data that over-emphasize the observations that were incorrectly classified by earlier trees. Also, the weight assigned to successive trees in the vote is related to how much they contribute to reducing the mistakes. These improvements make boosted forests of decisions trees more accurate than a truly random forest of decision trees.</p><p>XGBoost is short for <em>extreme gradient</em> boosting, and it’s the name for a machine learning model which (at the time of this writing) is the most popular and successful model for general-purpose prediction. XGBoost is popular because it delivers state-of-the-art performance, and the algorithm to fit the model is relatively fast (compared to other boosting algorithms, but as you’ll see, it isn’t fast compared to regression). Details about the XGBoost algorithm are beyond the scope of this article, but there are many excellent free resources online. For example, see <a href="https://en.wikipedia.org/wiki/XGBoost">https://en.wikipedia.org/wiki/XGBoost</a> and <a href="https://en.wikipedia.org/wiki/Gradient_boosting">https://en.wikipedia.org/wiki/Gradient_boosting</a>.</p><p><strong>XGBoost cross-validation</strong></p><p>Machine learning algorithms like XGBoost can be accurate at predictions, but this accuracy comes with some additional complexity. One area of complexity is that the algorithms have multiple optional parameters that you must choose correctly in order to get the best results. The optional parameters for XGBoost include ones that control how the individual decision trees are generated, as well as parameters for how the votes of the different decision trees are combined. Here are a few of the most important parameters for XGBoost:</p><ul><li>max_depth-The maximum depth of rules in each decision tree</li><li>n_estimator-The number of decision trees to generate</li><li>learning_rate-How heavily to emphasize the weight of votes from the best trees</li><li>min_child_weight-The minimum weight of each tree in the vote, regardless to how well it did</li></ul><p>Because there’s no straightforward way to select the values for many parameters, they’re set by out-of-sample cross-validation, like you used for the control parameter on the regression.</p><p><strong>TAKEAWAY State-of-the-art machine learning models have numerous parameters and the only way to make sure you pick the best values is to cross-validate a large number of them.</strong></p><p>You test a sequence of plausible values for each parameter and choose the ones with the best values on a cross-validation test. Figure 3 shows an example of such a cross-validation result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*arCEFkJlGGYSsZzp.png" /></figure><p>Figure 3 was created by running listing 1 on a simulated social network dataset.</p><ul><li>Four columns of parameters are there because four parameters were part of the test: max_depth, n_estimator, learning_rate, and minimum_child_date.</li><li>It has many more rows in the output table: 256 parameter combinations to be precise. The reason there are 256 parameter combinations is clear when inspecting listing 9.6: the test is made over four parameters, and the sequence of values for each parameter has four entries. The total number of combinations is the product of the number of values for each parameter; in this case, 4 x 4 x 4 x 4 = 256.</li></ul><p>You should run listing 1 on your own simulated data using the following usual Python wrapper program command with these arguments:</p><pre>fight-churn/listings/run_churn_listing.py --chapter 9 --listing 6</pre><p>Don’t be surprised if the cross-validation for the XGBoost model takes a lot longer than it did for the regression. Not only are there a lot more parameter combinations to test, but each time a model is fit, it takes significantly longer. The precise time can vary (depending on your hardware), but for me, the XGBoost model takes about forty times longer to fit in comparison with the regression model. The regression takes only a few hundredths of a second to fit on average; figure 3 shows that the XGBoost fits take from around one to four seconds.</p><p><strong>NOTE XGBoost is in its own Python package, and if you haven’t used it before now, you need to install it before running listing 1.</strong></p><p>Check out what this looks like in code in the continuation in part 2 of this series: <a href="https://medium.com/@carl.steven.gold/forecasting-churn-risk-with-machine-learning-part-2-fighting-churn-with-data-f45ea5646f49">the next post shows the code, and illustrates real results of forecasting churn</a>.</p><p>That’s all for this article. If you want to learn more about the book, you can preview its contents on our browser-based liveBook platform <a href="https://livebook.manning.com/book/fighting-churn-with-data?origin=product-look-inside&amp;utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07">here</a>. Take 37% off <a href="https://www.manning.com/books/fighting-churn-with-data?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07"><em>Fighting Churn with Data</em></a> by entering <strong>fccgold</strong> into the discount code box at checkout at <a href="https://www.manning.com/?utm_source=freecontentcenter&amp;utm_medium=blog&amp;utm_campaign=book_fightingchurnwithdata&amp;utm_content=article_07">manning.com</a>.</p><p><em>Originally published at </em><a href="https://freecontent.manning.com/forecasting-churn-risk-with-machine-learning-part-1/"><em>https://freecontent.manning.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=217bc190d0e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer Lifetime Value and Churn — Fighting Churn With Data]]></title>
            <link>https://data-science-churn.medium.com/customer-lifetime-value-and-churn-fighting-churn-with-data-9c7fd1db64c4?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c7fd1db64c4</guid>
            <category><![CDATA[customer-success]]></category>
            <category><![CDATA[churn-rate]]></category>
            <category><![CDATA[customer-lifetime-value]]></category>
            <category><![CDATA[churn]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Sun, 31 May 2020 00:52:18 GMT</pubDate>
            <atom:updated>2020-07-29T03:39:48.276Z</atom:updated>
            <content:encoded><![CDATA[<h3>Customer Lifetime Value and Churn — Fighting Churn With Data</h3><p>Customer lifetime value (CLV) is the foundation of recurring revenue business models: you recoup the costs and profit from a customer over their lifetime using your product, not from a single transaction. CLV is important because you need to make sure that the costs you spend to acquire and maintain your customers are worth it! If you calculate CLV correctly, you can use it strategically to evaluate the ROI of different acquisition and retention tactics. But if you calculate CLV wrong those decisions will be off the mark!</p><p>In this post I will explain the concept behind CLV and dispel some common misconceptions. I’ll also show you the <em>right</em> way to calculate CLV, but it requires you know your customer churn probability. So this post assumes you have calculated your average churn rate as I described in <a href="https://medium.com/@carl.steven.gold/how-to-calculate-churn-with-sql-fighting-churn-with-data-f4e5fb13a4df">my post on churn rate calculation with SQL</a>. Or, better yet, you have calculated <a href="https://medium.com/@carl.steven.gold/customer-churn-probability-fighting-churn-with-data-36d91b08fa00">individual customer churn probabilities with a regression as I explained in my last post</a>.</p><p>But what does churn have to do with customer lifetime value? I’ll show you the details in a moment. But in a nutshell, the lower the churn rate the longer customers stay with you. And the longer customers stay with you, more lifetime value. So the lower your product churn rate the higher your customer lifetime value.</p><h3>What Goes Into Customer Lifetime Value?</h3><p>The picture below illustrates the components of CLV :</p><ul><li>Customer Acquisition Cost : <strong>CAC</strong>. The total cost you pay to acquire each customer, Including both sales and marketing expenses.</li><li>Recurring Revenue : <strong>RR</strong>. The amount the customer pays you each month.</li><li>Cost of goods sold: <strong>COGS</strong>. The amount it cost you to maintain the customer each month. COGS includes things like the cost of your cloud computing or data center, and customer support. (But Not the R&amp;D cost of creating your product.)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*6v5EWliE03KOlEpG.png" /></figure><p>As you can see in the picture, CLV Is the combination of all the costs and revenue over the customers journey From sign up to churn. If you like equations, here’s how that all goes together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*jqYTiTiIoh1GehiV.png" /></figure><h3>Misconceptions about Customer Lifetime Value</h3><p>The biggest misconception about CLV is that it is based on the total of the payments your customer made to you in the past. In fact, it takes into account all payments including those you expect to make in the future!</p><p>But how can you know how much your customer will pay you in the future? You don’t have a crystal ball to see the future. That’s where the churn rate comes into play.</p><h3>How Customer Lifetime Value depends on Churn</h3><p>The relationship between churn and customer lifetime value is actually very simple. That’s because customers <em>expected</em> lifetime As your customer is just the reciprocal of the churn rate. Meaning, one divided by the churn rate. If you like equations, heres what that looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/0*KSRP7381SOvuQnqX.png" /></figure><p>Importantly, If you measure your churn with a monthly rate this gives you the customers lifetime in months. But if you measure your churn with an annual rate then this gives you the customer lifetime in years. Here are some examples:</p><ul><li>If the churn probability is 5% per month, the expected customer lifetime is 1.0 / 0.05 = 20 months</li><li>If the churn probability is 30% per year, the expected customer lifetime is 1.0 / 0.30 =3.33 years</li></ul><p>If you have calculated individual customer churn probabilities, then you can estimate lifetime individually for every customer. If you don’t have individual customer churn forecasts, then you can just use your churn rate. In that case, you are estimating the average lifetime for all your customers.</p><h3>CLV From Churn</h3><p>Now that you know you can predict customer lifetime from the churn rate, you should see where this is going. You can multiply the periodic recurring revenue and cost by the lifetime! Then just subtract your acquisition costs and the result is customer lifetime value. Here’s how that all comes together in an equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*PqsCoC5lYAJK52Ai.png" /></figure><p>If you are familiar with the concept of the margin, then you can write CLV in a simpler equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*WtVPP9PDWegvSAbi.png" /></figure><p>where <strong><em>m</em></strong> is the margin, defined as the proportion of recurring revenue that you keep after costs of goods sold, or: m= (RR — COGS)/RR.</p><h3>Advanced CLV: Cashflow Discounting</h3><p>Calculating CLV isn’t that hard, but there is one more misconception for pitfall. The truth is, the simple method I showed you above only works when your customers expected lifetimes are going to be shorter than a couple of years. If your customer lifetimes are longer than a couple of years, then you should also use a technique known as cash flow discounting to reduce the value of payments far in the future. So that means you only have to worry about this if your return rate is around 20% per year or less.</p><p>How do you do the discounting? That’s an advance subject I’m not going to tackle in a blog post. I recommend that you read this article called <a href="https://www0.gsb.columbia.edu/mygsb/faculty/research/pubfiles/721/gupta_customers.pdf">Customers as Assets by a Columbia professors Sunil Gupta and Donald Lehman</a>. They can do a better job explaining the concepts than I could. I just want to make sure you’re aware, and do the right thing if you’re at one of those lucky companies with a low churn rate.</p><h3>Learning more</h3><p><a href="https://www.manning.com/books/fighting-churn-with-data?a_aid=fight-churn&amp;a_bid=61baa361">Want to learn more? Check out my book, Fighting Churn With Data.</a></p><p><em>Originally published at </em><a href="https://fightchurnwithdata.com/customer-lifetime-value-and-churn/"><em>https://fightchurnwithdata.com</em></a><em> on May 31, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c7fd1db64c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer churn probability — Fighting Churn With Data]]></title>
            <link>https://data-science-churn.medium.com/customer-churn-probability-fighting-churn-with-data-36d91b08fa00?source=rss-fbfc6b1ae37a------2</link>
            <guid isPermaLink="false">https://medium.com/p/36d91b08fa00</guid>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[Carl gold]]></dc:creator>
            <pubDate>Mon, 27 Apr 2020 17:50:13 GMT</pubDate>
            <atom:updated>2020-07-18T19:00:07.099Z</atom:updated>
            <content:encoded><![CDATA[<h3>Customer churn probability — Fighting Churn With Data</h3><p>Calculating churn probability is an important part of fighting churn because of three key use cases:</p><ol><li>Evaluating which behaviors are most important for engagement</li><li>Calculating customer lifetime value</li><li>Segmenting customers for high cost interventions</li></ol><p>In this post I give an introduction to logistic regression: Logistic Regression is the most common and versatile way to calculate the churn probabilities. I assume you have <a href="https://medium.com/@carl.steven.gold/churn-feature-engineering-d9e997eb1752">calculated metrics</a> and <a href="https://medium.com/@carl.steven.gold/how-to-create-a-churn-dataset-a2fde8044379">created a data set</a>, following the instructions in my previous posts (or in the book, <a href="https://www.manning.com/books/fighting-churn-with-data">Fighting Churn With Data</a>.)</p><h3>Model of Churn Probability</h3><p>The logistic regression model for churn consists of two key concepts.</p><ol><li>More engagement causes a higher chance of retention. But there are diminishing returns for both low and high levels of engagement.</li><li>Engagement follows a multiplicative model. That means you multiply behavioral metrics by a “weight” (or coefficient) and add the results to get engagement.</li></ol><p>First, the concept of diminishing returns to engagement (and disengagement) is shown in the picture below. The relationship is “S” shaped because of the diminishing returns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*kRTrE7D5Kk8znpnj.png" /></figure><p>Next, the multiplicative model for calculating engagement from behaviors and weights is shown in the next diagram. Don’t worry if it looks complicated — you don’t have to keep track of all that multiply yourself (That’s what computers are for!)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*ipeftfXijeeamVj1.png" /></figure><p>Of course, you are probably wondering how you determine the engagement weights. Finding the weights is what the Logistic Regression algorithm is for. The code in python the code is shown below.</p><pre>from sklearn.linear_model import LogisticRegression </pre><pre>retain_reg = LogisticRegression(fit_intercept=True, solver=&#39;liblinear&#39;, penalty=&#39;l1&#39;) retain_reg.fit(churn_data, churn_outcomes) </pre><pre>engagement_weights = retain_reg.coef[0]</pre><p>For details, check out the <a href="https://github.com/carl24k/fight-churn/blob/master/listings/chap8/listing_8_2_logistic_regression.py">source code (listing 8.2 from Fighting Churn With Data) in my Github repository</a>.</p><h3>Churn Prediction Case studies</h3><p>With this in mind, let’s look at some real examples. The table below shows a result from a case study of <a href="https://broadly.com/">Broadly</a>. Broadly is a SaaS product for businesses. Companies use Broadly to manage their customers and reviews. As I explain in my <a href="https://fightchurnwithdata.com/customer-behavior-correlation-and-churn/">post on behavioral correlation</a>, metrics are in groups into related areas. The top two groups for Broadly are metrics for 1) customers &amp; messages; and 2) reviews. The results show that the engagement weight for reviews is higher: 0.55 compared to 0.16. That illustrates using logistic regression to find the most engaging behavior areas on your own product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/0*Ib3X4O3kRm01aZKm.png" /></figure><p>The next trick is to use histograms to visualize the churn probabilities of all your customers. Histograms count how many customers have a churn probability in a sequence of ranges. The count for each range is shown as the height of a bar in the plot. To illustrate this, three examples of churn probability histograms are shown below. Note that the exact churn rates are hidden for confidentiality reasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HsY9p6hbr6IJyTAY.png" /></figure><p>As you can see, there are at least three patterns to customer churn probability. Unfortunately, explaining all the details would be too much information for a blog post! In future posts I will say more about using churn probability forecasts to segment customers, and calculate customer lifetime value.</p><p>If you want more information and details about logistic regression and calculating churn probabilities, check out <a href="https://www.manning.com/books/fighting-churn-with-data?a_aid=fight-churn&amp;a_bid=61baa361">Chapter 8 of my book, Fighting Churn With Data</a>. You can now get the entire book in e-book edition through my publisher Manning’s Early Access Program (MEAP)</p><p><em>Originally published at </em><a href="https://fightchurnwithdata.com/customer-churn-probability/"><em>https://fightchurnwithdata.com</em></a><em> on April 27, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36d91b08fa00" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>