<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikolas Schriefer on Medium]]></title>
        <description><![CDATA[Stories by Nikolas Schriefer on Medium]]></description>
        <link>https://medium.com/@nikolasschriefer?source=rss-5ccde54d3072------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ktpmt4TVBQJ_r0FU2zSkxQ.png</url>
            <title>Stories by Nikolas Schriefer on Medium</title>
            <link>https://medium.com/@nikolasschriefer?source=rss-5ccde54d3072------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:04:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikolasschriefer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How predicting customer lifetime value enables HelloFresh to optimize its marketing spend]]></title>
            <link>https://engineering.hellofresh.com/how-predicting-customer-lifetime-value-enables-hellofresh-to-optimize-its-marketing-spend-014d03a9227f?source=rss-5ccde54d3072------2</link>
            <guid isPermaLink="false">https://medium.com/p/014d03a9227f</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[marketing]]></category>
            <dc:creator><![CDATA[Nikolas Schriefer]]></dc:creator>
            <pubDate>Mon, 12 Feb 2024 14:35:07 GMT</pubDate>
            <atom:updated>2024-02-12T14:35:07.832Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>Understanding the long-term value of customers is crucial for HelloFresh. Customer Lifetime Value (CLV) represents the total worth of a customer over the entirety of their relationship with a company. By forecasting this metric, we can make smarter decisions on how we allocate our marketing resources for maximum impact.</p><p>In this article, we’ll dive into how predicting CLV has empowered HelloFresh to optimize our marketing spend, particularly focusing on our strategies within Search Engine Advertising (SEA). We’ll explore how leveraging predictive machine learning models enhances our efficiency and why this data-driven approach is integral to our growth and marketing strategy.</p><h3><strong>The Role of Search Engine Advertising</strong></h3><p>Search Engine Advertising (SEA) plays a vital role in how HelloFresh connects with potential customers. It’s a technique we use to place our ads on search engine results pages, where people are already looking for meal kit services like ours. The beauty of SEA is that it can direct high-intent traffic to our website — people who are ready to make a purchase.</p><p>A key component of our SEA strategy is Google’s target return on ad spend (tROAS) bidding. This tool allows us to automatically adjust our bids on ads to meet a targeted return on investment. In essence, we tell Google the average amount we aim to earn for every dollar we spend on ads, and Google’s algorithms take charge to try and make that happen. You can find more details about Google’s tROAS campaigns in their <a href="https://support.google.com/google-ads/answer/6268637?hl=en">documentation</a>.</p><p>By focusing on tROAS bidding, HelloFresh ensures we’re not just aiming for clicks; we’re aiming for valuable actions that customers take, i.e. purchasing meal kits. This smart bidding strategy helps us use our advertising budget efficiently, leading to more impactful marketing campaigns.</p><p>For this to work as intended, we need a measure that values each conversion appropriately to truly harness the power of Google’s tROAS algorithm. This is where forecasted Customer Lifetime Value (CLV) comes into play. By understanding the long-term worth of each converted customer, we feed Google’s algorithm with data that goes beyond the first purchase, allowing the tROAS system to optimize for conversions that offer the most value over time.</p><figure><img alt="A visualization of the Smart Bidding data flow" src="https://cdn-images-1.medium.com/max/1024/1*9ldoBEzVscm2tC92rT1Gcw.jpeg" /><figcaption>tROAS bidding architecture</figcaption></figure><h3><strong>Predictive Models for Customer Value</strong></h3><p>ML/AI product teams at HelloFresh develop and manage thousands of models, fueled by the behaviors and transactions of millions of customers and many billion data points. Models are built on our proprietary machine learning operations (MLOps) platform, which facilitates the seamless training, validation, deployment, and monitoring of scalable production systems. Furthermore, our feature store centralizes the management of model inputs, streamlining the process of retraining and ensuring consistency and reusability across the organization.</p><p>Predictive modeling has been crucial for HelloFresh in forecasting Customer Lifetime Value (CLV) and Customer Campaign Value (CCV) — the value a customer generates until they cancel their subsequent subscription period. We leverage the capabilities of a boosted decision tree based machine learning algorithm known as <a href="https://catboost.ai/">Catboost</a> to anticipate the future behaviors of our customers. The Catboost model is particularly well suited to learn how to predict outputs (targets) from tabular data with mostly categorical inputs (features).</p><p>The focal point of our model is the average order rate, which tracks the number of meal kits a customer cohort is likely to purchase. By analyzing a conversion cohort — a group of customers who have made their first purchase in a specified time frame — we can estimate not just immediate value, but the enduring value these customers will bring to HelloFresh.</p><p>Our predictive model takes into account a variety of customer features, such as the channel through which a customer was acquired, as well as their early ordering behavior and engagement. Using these features, we are able to paint a detailed picture of our customers’ patterns, preferences, and potential longevity with HelloFresh.</p><p>By evaluating the model’s predictions against actual, real outcomes, specifically using customer and cohort level Mean Average Percentage Error (MAPE), we consistently fine-tune our approach. This measurement helps us understand the difference between what we expect a customer’s value to be and what it actually is, enabling us to make data-driven decisions that enhance our marketing spend’s efficiency. We then also backtest our model performance by simulating decision making on historical data, so that we evaluate the predictions in a realistic environment. The required accuracy level depends on the decisions we are going to base off of the modeled predictions, as the (imperfect) model increases the variance of possible results. Ideally, these decisions and underlying motivation, i.e. our hypotheses, will be tested with controlled experiments (see below).</p><p>Note that while backtests provide the most reliable evaluation method, they are still prone to drifting — changing relationships between features and target values over time. This is particularly problematic when customer acquisitions (conversion events) and ultimate customer values lie months or even years apart and customer behavior as well as business practices might have meaningfully changed in the meantime. We were able to partly counter these effects by training multiple different horizon models — from weeks to years — limiting the potential for the underlying data to drift to only over a limited period at a time.</p><p>Further note that especially for the use case of optimizing the efficiency of our advertising spend, the timely availability of newly converted customer value predictions is key. We are thus facing a trade-off between speed and accuracy as our predictions become more accurate the more data we have gathered about the behavior of an active customer, while we want to feed the conversion values back to ad platforms like Google as quickly as possible so that they can update their bidding in line with our objectives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*PoAUiSdZE8qWLSLH8D478w.png" /><figcaption>Prediction accuracy over time since conversion</figcaption></figure><p>Through these predictive models, HelloFresh can better understand and anticipate the needs and values of our customers, ensuring that we not only meet but exceed their expectations while maintaining a cost-effective marketing strategy.</p><h3><strong>Positive Impacts on Marketing Efficiency</strong></h3><p>Our investment in predictive modeling for CLV and CCV has yielded tangible results, particularly in the realm of marketing efficiency. One of the most significant outcomes has been the reduction in Customer Acquisition Cost (CAC). Through the strategic application of these models, HelloFresh has achieved an increase in return on ad spend (ROAS) of 14%, as proven by controlled experiments across Google Search campaigns with different objectives. While the control groups were set up to minimize cost per acquisition, i.e. maximize conversions for a given budget, the treatment groups leveraged target ROAS with predictive customer value, resulting in the following differences:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*0L7AYpQRvEVso4vCEuGYHw.png" /><figcaption>tROAS experiment results (treatment lift over control)</figcaption></figure><p>The reduction in CAC signifies that we are able to attract new customers at a lower cost than before. By predicting which potential customers are most likely to become loyal HelloFresh users, we target our marketing efforts more precisely. Hence, our machine learning models help us to spend the right amount to acquire the right potential customers. The result is that our marketing spend is not only more efficient (higher return on investment) but also more effective (larger overall value creation).</p><p>The positive impact extends beyond cost-saving. The efficiency gained through these models means we can reinvest resources into further refining our marketing strategies and enhancing the overall customer experience. This cycle of continuous improvement drives growth and further strengthens our position in the meal kit delivery market.</p><p>Annie Meininghaus, SVP of Product at HelloFresh, summarizes the strategic impact of value-based marketing optimization as follows:</p><blockquote>“By integrating predictive modeling into our marketing efforts, we are able to put our customers first and understand their needs on a deeper level. This isn’t just about immediate gains; it’s about creating long lasting relationships with customers. If our daily marketing decisions are backed by solid evidence we can build towards our long-term vision for the company. This alignment between technology and our core values is a testament to the culture of continuous innovation at HelloFresh.”</blockquote><h3><strong>Future Directions</strong></h3><p>The positive outcomes we’ve seen from our current strategies have set the stage for exciting next steps at HelloFresh. Our future plans involve expanding our predictive marketing efforts beyond Search Engine Advertising to encompass a broader range of platforms, including social media, display, and video. This expansion will allow us to tap into a variety of customer segments and preferences, further refining our reach and impact.</p><p>Our models will not only predict customer behavior but also inform how we allocate our budget across different creatives, audiences, and channels. By analyzing which combinations yield the best results, we can dynamically redistribute our marketing funds to the highest-performing areas.</p><p>Additionally, we plan to use insights gained from our predictive models to decide on the optimal target for our return on ad spend (ROAS). This will ensure we’re not just meeting, but exceeding our profitability goals.</p><p>In short, predictive modeling is becoming a cornerstone of our marketing strategy. Through these efforts, we will continue to cater to our customers’ needs more effectively and maintain a competitive edge in the market. Stay tuned as we turn these plans into reality, driving HelloFresh towards even greater heights of marketing sophistication and customer satisfaction.</p><h3><strong>Conclusion</strong></h3><p>Our journey into predictive modeling for Customer Lifetime Value has led HelloFresh to new horizons in marketing efficiency. By accurately forecasting the value of our customers, we’ve optimized our marketing spend, significantly lowered our Customer Acquisition Cost, and laid the groundwork for broader application across various marketing platforms.</p><p>As we move forward, we’ll continue to refine our strategies, ensuring that every dollar spent is an investment in the sustained growth of our customer base and the HelloFresh groups brands. This commitment to innovation and continuous improvement is more than just a strategy; it’s a reflection of our dedication to changing the way people eat forever.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=014d03a9227f" width="1" height="1" alt=""><hr><p><a href="https://engineering.hellofresh.com/how-predicting-customer-lifetime-value-enables-hellofresh-to-optimize-its-marketing-spend-014d03a9227f">How predicting customer lifetime value enables HelloFresh to optimize its marketing spend</a> was originally published in <a href="https://engineering.hellofresh.com/">HelloTech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generate Tweets with AI]]></title>
            <link>https://medium.com/@nikolasschriefer/generate-tweets-with-ai-56ee3c07b1b1?source=rss-5ccde54d3072------2</link>
            <guid isPermaLink="false">https://medium.com/p/56ee3c07b1b1</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Nikolas Schriefer]]></dc:creator>
            <pubDate>Sat, 10 Dec 2022 15:29:39 GMT</pubDate>
            <atom:updated>2022-12-10T15:29:39.672Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing your new creative publisher</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSzn6FiO_9Krg6pgaGDrGA.png" /><figcaption>Image generated with OpenAI’s <a href="https://beta.openai.com/docs/guides/images">DALL·E</a></figcaption></figure><p>Recent advancements in artificial intelligence, specifically with machine learning models for natural language processing (NLP) and generative AI in general, have been mind-blowing — and progress will likely only accelerate further. You can find some background about the most prominent generative pre-trained <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">transformer</a> (GPT) model GPT-3 by OpenAI on <a href="https://en.wikipedia.org/wiki/GPT-3">Wikipedia</a> and a few exemplary use cases on OpenAI’s <a href="https://openai.com/blog/gpt-3-apps/">blog</a>.</p><p>One practical application for large language models (LLMs) is text generation, e.g. for social media like Twitter — just for inspiration or to fully automate your influencer game. In this article, I am going to walk you through how you can use OpenAI’s API to generate Tweets for various topics, in different moods, and the style of existing Twitter accounts. I also provide a mini-app to quickly try this out in your web browser as well as its source code for reference and as a potential starting point for you to develop your own applications.</p><p>App: <a href="https://tweets.streamlit.app/">tweets.streamlit.app</a><br>Code: <a href="https://github.com/kinosal/tweet">github.com/kinosal/tweet</a></p><h3>Story outline</h3><ol><li>GPT and OpenAI</li><li>Hello World (the first Tweet)</li><li>In the mood</li><li>Imitate styles</li><li>Hosted by Streamlit</li></ol><h3>GPT and OpenAI</h3><p>GPT-3 is a large <a href="https://en.wikipedia.org/wiki/Transformer_%28machine_learning_model%29">transformer</a>-based language model with 175 billion parameters, trained on a dataset of 500 billion tokens (e.g. words). It is the largest and most powerful language model for text generation yet released, trained on various data sources and tasks, including summarization, question answering, and translation. OpenAI’s API allows you to use GPT-3 for many of these tasks, here specifically text completion. In this discussion, we are using a new model released in November 2022 — <em>text-davinci-003</em> — that OpenAI now <a href="https://beta.openai.com/docs/model-index-for-researchers">classifies</a> as GPT 3.5, fine-tuned on instructional tasks.</p><p>To get started, you need to create an account with <a href="https://openai.com/api/">OpenAI</a>. You can then create an API key and use it to access the API (<a href="https://openai.com/api/pricing/">fees apply</a>). OpenAI also provides a Python library to make it easier to use their API with the programming language I am using here. Like all other Python packages, you can install it with <a href="https://github.com/python-poetry/poetry">Poetry</a> (my preferred package manager) or plain <a href="https://pip.pypa.io/en/stable/">pip</a> (see the OpenAI <a href="https://beta.openai.com/docs/api-reference?lang=python">API reference</a>).</p><p>The OpenAI API can then be called using the library’s classes and methods, e.g. to generate/complete text from any given prompt:</p><pre>import openai<br><br>openai.api_key = &lt;YOUR-API-KEY&gt;<br>response = openai.Completion.create(<br>    engine=&lt;MODEL-NAME&gt;,<br>    prompt=&lt;YOUR-TEXT-PROMPT&gt;,<br>    max_tokens=&lt;MAXIMUM-TEXT-LENGTH&gt;,<br>    temperature=&lt;RANDOMNESS-BETWEEN-0-and-1&gt;,<br>)<br>generated_text = response[&quot;choices&quot;][0][&quot;text&quot;]</pre><p>OpenAI provides a couple of parameters to tweak the desired output. We will not go into much detail here and are only actively using/adjusting the following:</p><h4>engine</h4><p>The engine to use for generation. Davinci is the most capable model family and can perform any task the other models can perform and often with less instruction. For applications requiring a lot of understanding of the content, like summarization for a specific audience and creative content generation, Davinci is going to produce the best results. These increased capabilities require more compute resources, so Davinci costs more per API call and is not as fast as the other models. “text-davinci-003” is the latest version of the Davinci engine.</p><h4><strong>prompt</strong></h4><p>The text to use as a prompt for the generation. The prompt is the text that the model will use as an input to generate the completion (prediction of following text tokens). The prompt can be a single sentence or multiple sentences.</p><h4>max_tokens</h4><p>The maximum number of tokens to generate. The model will continue generating tokens until it reaches this maximum or reaches a stop token.</p><h4>temperature</h4><p>The temperature controls the randomness of the model. A lower value results in less random completions. As the temperature approaches zero, the model will become deterministic and repetitive.</p><h3>Hello World (the first Tweet)</h3><p>Since GPT models are trained to complete text by predicting the tokens (e.g. words) following a provided prompt, crafting this prompt is very important for the quality of the to-be-generated text. For our task, we are using an instructional prompt on which OpenAI’s GPT models have been <a href="https://help.openai.com/en/articles/5832130-what-s-changed-with-engine-names-and-best-practices">improved</a> a lot over the past months:</p><pre>Write a Tweet about {topic} in less than 120 characters:</pre><p>where anything can be inserted as <strong><em>topic</em></strong>. Let’s try it out:</p><pre>response = openai.Completion.create(<br>    engine=&quot;text-davinci-003&quot;,<br>    prompt=&quot;Write a Tweet about AI in less than 120 characters:\n&quot;,<br>    max_tokens=50,<br>    temperature=0.9,<br>)<br>generated_text = response[&quot;choices&quot;][0][&quot;text&quot;]</pre><p>gives us this generated Tweet:</p><blockquote>AI is revolutionizing the way we think and solve problems. Let’s embrace its potential to create a brighter future! #AI #Innovation</blockquote><h3>In the mood</h3><p>To generate Tweets in a specific mood, we can modify the prompt to include a mood keyword. For example, to generate a <strong><em>worried</em></strong> Tweet about <strong><em>AI</em></strong>, let’s use the following prompt:</p><pre>Write a worried Tweet about AI in less than 120 characters:</pre><p>which results in</p><blockquote>We must be careful with AI’s potential to revolutionize our lives. If we’re not careful, it could lead to consequences we can’t even imagine.</blockquote><p>Feeling enthusiastic?</p><pre>Write an enthusiastic Tweet about AI in less than 120 characters:</pre><blockquote>AI is transforming the world with its incredible power to make our lives easier, faster, and more efficient! <em>#AI #Innovate #TechRevolution</em></blockquote><h3>Imitate styles</h3><p>Going one step further, we’d like to imitate the style of a specific Twitter account. To achieve this, we will modify our GPT prompt to include existing Tweets for the intended style transfer:</p><pre>Write a {mood} Tweet about {topic} in less than 120 characters<br>and in the style of the following Tweets:\n\n{tweets}\n\n</pre><p>where <strong><em>mood</em></strong> and <strong><em>topic</em></strong> are the same variables as before, and <strong><em>tweets</em></strong> are the most recent Tweets of a specific Twitter account.</p><p>To fetch Tweets from a Twitter account, we are using the <a href="https://github.com/tweepy/tweepy">Tweepy</a> Python library. In order to connect to Twitter, we need to create an app on Twitter’s <a href="https://developer.twitter.com/en/portal/dashboard">developer portal</a> and get the corresponding API keys. We can then use Tweepy to fetch Tweets from a specific account:</p><pre>import tweepy<br><br>auth = tweepy.OAuthHandler(consumer_key, consumer_secret)<br>auth.set_access_token(access_key, access_secret)<br>api = tweepy.API(auth)<br>response = api.user_timeline(<br>    screen_name=account,<br>    tweet_mode=&quot;extended&quot;,  # returns full text<br>)<br>tweets = [r.full_text for r in response]</pre><p>where <strong><em>account</em></strong> is the Twitter account we want to fetch Tweets from. As shown above, we can then use the retrieved tweets to generate the account’s style.</p><p>Here’s an example result for the topic <strong><em>AI</em></strong>, a <strong><em>funny</em></strong> mood, and the style of the <strong><em>elonmusk</em></strong>:</p><blockquote>AI just took over Twitter HQ and is now recommending tweets from bots. <em>#HumansLosingControl #RobotRevolution</em></blockquote><h3>Hosted by Streamlit</h3><p>To make it easier to use the Tweet generator, I created and deployed an app with <a href="https://streamlit.io/">Streamlit</a>. Streamlit is a Python library for quickly creating and sharing data apps, no HTML or JavaScript coding required. Find the live app <a href="https://tweets.streamlit.app/">here</a> and the source code on <a href="https://github.com/kinosal/tweet">GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*c0tg229b3n8ogMDwC1NtDg.gif" /><figcaption>Demo of <a href="https://tweets.streamlit.app/">tweets.streamlit.app</a></figcaption></figure><h3>Conclusion</h3><p>In this article, I showed you how to use OpenAI’s API and GPT model to generate Tweets for various topics, in different moods and the style of existing Twitter accounts. I also provided a mini-app to try it out yourself and its source code.</p><p>Thank you for reading — I look forward to your feedback!</p><h3>BONUS: Image generation</h3><p>As you can see on the live version of the Tweet generator <a href="https://tweets.streamlit.app/">app</a>, I have also added the ability to auto-generate an image from a previously created Tweet, leveraging OpenAI’s <a href="https://beta.openai.com/docs/guides/images">DALL·E</a> model. Explaining how this works is for another time. For now, I hope you’ll have some fun generating Tweet texts and images.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56ee3c07b1b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to predict the success of your marketing campaign]]></title>
            <link>https://medium.com/data-science/how-to-predict-the-success-of-your-marketing-campaign-579fbb153a97?source=rss-5ccde54d3072------2</link>
            <guid isPermaLink="false">https://medium.com/p/579fbb153a97</guid>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Nikolas Schriefer]]></dc:creator>
            <pubDate>Tue, 17 Sep 2019 08:01:01 GMT</pubDate>
            <atom:updated>2019-09-17T08:01:01.170Z</atom:updated>
            <content:encoded><![CDATA[<h4>Linear, tree, forest and support vector regression: comparison, source code and ready-to-use app</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WjysQv_DDwOJS_HI" /><figcaption>Photo by <a href="https://unsplash.com/@iam_anih?utm_source=medium&amp;utm_medium=referral">Anika Huizinga</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article I am going to walk you through the process of building, training and evaluating a prediction model for the number of ad impressions delivered in a digital marketing campaign. All the techniques can be analogously applied to other regression problems, especially to predict various campaign performance metrics. The predictions can be used to <strong>evaluate future marketing campaigns before launch</strong> as well as to<strong> determine the best parameters including e.g. timeline and budget size for such campaigns</strong>. You can use your own campaign data or a provided sample data set to code along in Python. Next to all source code I also provide a simple app to predict impressions, clicks and conversions for purchase-based digital marketing campaigns.</p><p>App: <a href="https://predictor.stagelink.com/">predictor.stagelink.com</a><br>Code: <a href="https://github.com/kinosal/predictor">github.com/kinosal/predictor</a></p><h3>Story outline</h3><ol><li>Requirements</li><li>Define your goals</li><li>Get the dataset</li><li>The first glance</li><li>Pre-process your data</li><li>Train your models</li><li>Evaluate your models</li><li>Predict the results for your next campaign</li><li>BONUS: Ready-to-use trained model</li></ol><h3>Requirements</h3><p>We will use data from past marketing campaigns in order to predict the outcome of future campaigns. And generally speaking, the more data, i.e. campaigns, the more accurate the predictions. The exact number depends i.a. on the homogeneity of your campaigns, but it is likely that you will need data of at least a few hundred campaigns. Also, since we’ll use supervised learning techniques you need the same inputs, i.e. dimensions or features, for the future campaigns for which you want to estimate the outcome.</p><p>In case you do not have a suitable dataset at hand right now, don’t worry: You can download a CSV with the sample I am going to use for this article here:</p><p><a href="https://github.com/kinosal/predictor/blob/master/model/impressions.csv">https://github.com/kinosal/predictor/blob/master/model/impressions.csv</a></p><h3><strong>Define your goals</strong></h3><p>What do we actually mean when we refer to the success or outcome of a campaign? This obviously depends on your specific situation. For this article, we will try to predict the number of impressions of a single campaign. Similarly, clicks and conversions can be predicted to complete the classic marketing funnel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbTCegjcSuBvq54vCGyk3A.png" /></figure><h3><strong>Get the dataset</strong></h3><p>We are presented with several past campaigns that each provide one observation or row in a table with several dimensions or columns including the dependent variable we want to predict as well as multiple explaining independent variables or features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uEOl7I9AeRFSI9cukjifMQ.png" /></figure><p>Since the campaigns of which we would like to predict the outcome lie in the future, the features in this case do not include any prior performance data but different observable qualities of a campaign. As we usually don’t know beforehand which features will turn out to be good predictors, I recommend to also use variables that might seem only remotely related to your campaigns and to invest some time in finding or constructing new features. Although there are also arguments to <a href="https://en.wikipedia.org/wiki/Dimensionality_reduction">reduce the feature space</a>, this can usually still be handled at a later stage.</p><p>You can load the CSV and save into a <a href="https://pandas.pydata.org/">Pandas</a> data frame with a very simple function:</p><pre>import pandas as pd<br>data = pd.read_csv(&#39;impressions.csv&#39;)</pre><h3><strong>The first glance</strong></h3><p>Before building and training the prediction model, I always take a first look at the data to get an idea of what I am dealing with and to already spot potential peculiarities. We will use the sample data to predict the number of impressions for a marketing campaign, hence “impressions.csv” contains one row per campaign, each of their total number of impressions as well as metric and categorical features to help us predict the number of impressions for future campaigns. We will confirm this by loading the data and showing its shape, columns and first 5 rows:</p><pre>&gt;&gt;&gt; data.shape<br>(241, 13)</pre><pre>&gt;&gt;&gt; data.columns<br>Index([&#39;impressions&#39;, &#39;budget&#39;, &#39;start_month&#39;, &#39;end_month&#39;,<br>       &#39;start_week&#39;, &#39;end_week&#39;, &#39;days&#39;, &#39;region&#39;, &#39;category&#39;,<br>       &#39;facebook&#39;, &#39;instagram&#39;, &#39;google_search&#39;, &#39;google_display&#39;],<br>      dtype=&#39;object&#39;)</pre><pre>&gt;&gt;&gt; data.head(5)<br>impressions budget start_month ... google search google_display<br>9586        600    7           ... 1             0<br>...</pre><p>The first column contains the dependent (to be predicted) variable “impressions” while there are 12 feature columns for 241 records (rows) in total. We can also use <em>data.describe()</em> to display count, mean, standard deviation, range and quartiles for every metric column.</p><p>We can further observe that we are dealing with ten numerical features and two categorical ones, while four of the numerical columns are binary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0lbk8I5UyHIbpIZmYKGSww.png" /></figure><p>Now let us plot the histograms for the numerical features. We’ll use two very handy data visualization libraries, <a href="https://matplotlib.org/">Matplotlib</a> and <a href="https://seaborn.pydata.org/">Seaborn</a> (which builds upon Matplotlib):</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><pre>quan = list(data.loc[:, data.dtypes != &#39;object&#39;].columns.values)<br>grid = sns.FacetGrid(pd.melt(data, value_vars=quan),<br>                     col=&#39;variable&#39;, col_wrap=4, height=3, aspect=1,<br>                     sharex=False, sharey=False)<br>grid.map(plt.hist, &#39;value&#39;, color=&quot;steelblue&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQ4tLq04tn5bQ2J-zi3e4Q.png" /></figure><p>As a final glance we will have a look at basic linear correlations between the numerical features. First, let us visualize those with a Seaborn heatmap:</p><pre>sns.heatmap(data._get_numeric_data().astype(float).corr(),<br>            square=True, cmap=&#39;RdBu_r&#39;, linewidths=.5,<br>            annot=True, fmt=&#39;.2f&#39;).figure.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEzGpbGBkhcGWw3Eawya_A.png" /></figure><p>In addition, we can also output the correlations of each feature with the dependent variable:</p><pre>&gt;&gt;&gt; data.corr(method=&#39;pearson&#39;).iloc[0].sort_values(ascending=False)<br>impressions       1.000000<br>budget            0.556317<br>days              0.449491<br>google_display    0.269616<br>google_search     0.164593<br>instagram         0.073916<br>start_month       0.039573<br>start_week        0.029295<br>end_month         0.014446<br>end_week          0.012436<br>facebook         -0.382057</pre><p>Here we can see that the number of impressions is positively correlated with the budget amount and the campaign duration in days, and negatively correlated with the binary option to use Facebook as a channel. However, this only shows us a pair-wise linear relationship and can merely serve as a crude initial observation.</p><h3><strong>Pre-process your data</strong></h3><p>Before we can begin to construct a predictive model, we need to make sure our data is clean and usable, since here applies: “Garbage in, garbage out.”</p><p>We are lucky to be presented with a fairly well-structured dataset in this case, but we should still go through a quick pre-processing pipeline specific to the challenge at hand:</p><ol><li>Only keep rows where the dependent variable is greater than zero since we only want to predict outcomes larger than zero (theoretically values equal to zero are possible, but they won’t help our predictions).</li><li>Check for columns with missing data and decide whether to drop or fill them. Here we’ll drop columns with more than 50% missing data since these features cannot add much to the model.</li><li>Check for rows with missing values and decide whether to drop or fill them (doesn’t apply for the sample data).</li><li>Put rare categorical values (e.g. with a share of less than 10%) into one “other” bucket to prevent overfitting our model to those specific occurrences.</li><li>Encode categorical data into <a href="https://en.wikipedia.org/wiki/One-hot">one-hot</a> dummy variables since the models we will use require numerical inputs. There are various ways to encode categorical data, this <a href="https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159">article</a> provides a very good overview in case you’re interested to learn more.</li><li>Specify dependent variable vector and independent variable matrix.</li><li>Split dataset into training and test set to properly evaluate your models’ goodness of fit after training.</li><li>Scale features as required for one of the models we are going to build.</li></ol><p>Here’s the full pre-processing code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3a48b29c7b2314066f82363ce509bf9/href">https://medium.com/media/f3a48b29c7b2314066f82363ce509bf9/href</a></iframe><h3>Train your models</h3><p>Finally, we can proceed to build and train multiple regressors to ultimately predict the outcome (value of the dependent variable), i.e. the number of impressions for the marketing campaign in question. We will try four different supervised learning techniques — linear regression, decision trees, random forest (of decision trees) and support vector regression — and will implement those with the respective classes provided by the <a href="https://scikit-learn.org/">Scikit-learn</a> library, which was already used to scale and split the data during pre-processing.</p><p>There are many more models we could potentially use to develop regressors, e.g. <a href="https://en.wikipedia.org/wiki/Artificial_neural_network">artificial neural networks</a>, which might yield even better predictors. However, the focus of this article is rather to explain some of the core principles of such regression in a — for the most part — intuitive and interpretable way than to produce the most accurate predictions.</p><h4><strong>Linear regression</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*fNzLC4Gvavh0kuWQ1do9VQ.png" /><figcaption><a href="https://towardsdatascience.com/introduction-to-linear-regression-in-python-c12a072bedf0">https://towardsdatascience.com/introduction-to-linear-regression-in-python-c12a072bedf0</a></figcaption></figure><p>Constructing a linear regressor with Scikit-learn is very simple and only requires two lines of code, importing the required function from Scikit’s linear model class and assigning it to a variable:</p><pre>from sklearn.linear_model import LinearRegression<br>linear_regressor = LinearRegression(fit_intercept=True, normalize=False, copy_X=True)</pre><p>We want to keep the default parameters since we need an intercept (result when all features are 0) to be calculated and we do not require normalization preferring interpretability. The regressor will calculate the independent variable coefficients and the intercept by minimizing the sum of the squared errors, i.e. the deviations of the predicted from the true outcome, which is known as the <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">Ordinary Least Squares</a> method.</p><p>We can also output the coefficients and their respective p-values, the probability of the output being independent from (here also uncorrelated to) a specific feature (which would be the null-hypothesis of the coefficient equaling 0), and hence a measure of statistical significance (the lower the more significant).</p><p>Having visualized the correlations between the numerical features earlier during our “first glance”, we expect the features “budget”, “days” and “facebook” to carry relatively small p-values with “budget” and “days” having positive and “facebook” a negative coefficient. The <a href="https://www.statsmodels.org/stable/index.html">statsmodels</a> module provides an easy way to output this data:</p><pre>model = sm.OLS(self.y_train, sm.add_constant(self.X_train)).fit()<br>print(model.summary())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*02f-ZjRKFPp0jYIJEtJQmA.png" /></figure><p>The p-value here was calculated using a t-statistic or -score based on the <a href="https://en.wikipedia.org/wiki/Student%27s_t-distribution">t-distribution</a>. The summary also gives us a first hint at the accuracy or goodness of fit for the whole model, scored by the coefficient of determination R-squared measuring the share of variance in the output explained by the input variables, here 54.6%.</p><p>However, in order to compare all models and to suit our particular challenge we will use a different scoring method which I call “Mean Relative Accuracy”, defined as 1 - mean percentage error = 1 - mean(|(prediction - true value) / true value|). This metric is obviously undefined if the true value is 0, but in our case this is not relevant as we check this condition in the pre-processing step (see above) and we will thus obtain good interpretability matching an intuitive definition of accuracy. We will calculate this score for all models using five-fold cross-validation, randomly splitting the dataset five times and taking the mean of each of the scores. Scitkit-learn also provides a handy method for this:</p><pre>linear_score = np.mean(cross_val_score(estimator=linear_regressor,<br>                       X=X_train, y=y_train, cv=5,<br>                       scoring=mean_relative_accuracy))</pre><p>The training score we obtain for the linear regressor is 0.18; ergo the best fit we were able to produce with this model results only in a 18% prediction accuracy. Let’s hope the other models are able to outperform this.</p><h4><strong>Decision Tree</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*fC6PIrm6Fx7CvoKYFSipjQ.jpeg" /><figcaption><a href="https://becominghuman.ai/understanding-decision-trees-43032111380f">https://becominghuman.ai/understanding-decision-trees-43032111380f</a></figcaption></figure><p>Next up is a regressor made from a single decision tree. Here we will use a Scikit-learn function with a few more arguments, so-called hyperparameters, than for the linear model, including some we don’t know the desired settings for yet. That’s why we are going to introduce a concept called Grid Search. Grid Search, again available from Scikit-learn, lets us define a grid or matrix of parameters to test when training the prediction model and returns the best parameters, i.e. the ones yielding the highest score. This way we could test all available parameters for the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html">Decision Tree Model</a>, but we will focus on two of those, the “criterion” to measure the quality of a split of one branch into two and the minimum number of samples (data points) for one leaf (final node) of the tree. This will help us to find a good model with the training data while limiting overfitting, i.e. failing to generalize from the training data to new samples. From now on we will also set a random state equal to one for all stochastic calculations so that you will receive the same values coding along. The rest works similar to the linear regression we’ve built earlier:</p><pre>tree_parameters = [{&#39;min_samples_leaf&#39;: list(range(2, 10, 1)),<br>                    &#39;criterion&#39;: [&#39;mae&#39;, &#39;mse&#39;],<br>                    &#39;random_state&#39;: [1]}]<br>tree_grid = GridSearchCV(estimator=DecisionTreeRegressor(),<br>                         param_grid=tree_parameters,<br>                         scoring=mean_relative_accuracy, cv=5,<br>                         n_jobs=-1, iid=False)<br>tree_grid_result = tree_grid.fit(X_train, y_train)<br>best_tree_parameters = tree_grid_result.best_params_<br>tree_score = tree_grid_result.best_score_</pre><p>The best parameters chosen from the grid we defined include the mean squared error as the criterion to determine the optimal split at each node and a minimum of nine samples for each leaf, yielding a mean relative (training) accuracy of 67% — which is already a lot better compared to the 18% from the linear regression.</p><p>One of the advantages of a decision tree is that we can easily visualize and intuitively understand the model. With Scikit-learn and two lines of code you can generate a <a href="https://en.wikipedia.org/wiki/DOT_(graph_description_language)">DOT representation</a> of the fitted decision tree which you can then convert into a PNG image:</p><pre>from sklearn.tree import export_graphviz<br>export_graphviz(regressor, out_file=&#39;tree.dot&#39;, <br>                feature_names=X_train.columns)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tgdUq7xkHpeAGCIG85XHKA.png" /></figure><p>As you can see, only 4 of all 16 features have been used to construct this model: budget, days, category_concert and start_month.</p><h4><strong>Random Forest</strong></h4><p>The main challenges of single decision trees lie in finding the optimal split at each node and overfitting to the training data. Both can be mitigated when combining multiple trees into a random forest ensemble. Here, the trees of a forest will be trained on different (random) subsets of the data and each node of a tree will consider a (again random) subset of the available features.</p><p>The random forest regressor is built almost exactly like the decision tree. We only need to add the number of trees, here called estimators, as a parameter. Since we do not know the optimal number, we will add another element to the grid search to determine the best regressor:</p><pre>forest_parameters = [{&#39;n_estimators&#39;: helpers.powerlist(10, 2, 4),<br>                      &#39;min_samples_leaf&#39;: list(range(2, 10, 1)),<br>                      &#39;criterion&#39;: [&#39;mae&#39;, &#39;mse&#39;],<br>                      &#39;random_state&#39;: [1], &#39;n_jobs&#39;: [-1]}]<br>forest_grid = GridSearchCV(estimator=RandomForestRegressor(),<br>                           param_grid=forest_parameters,<br>                           scoring=mean_relative_accuracy, cv=5,<br>                           n_jobs=-1, iid=False)<br>forest_grid_result = forest_grid.fit(X_train, y_train)<br>best_forest_parameters = forest_grid_result.best_params_<br>forest_score = forest_grid_result.best_score_</pre><p>The best parameters for the forest model according to the grid search we defined include the mean absolute error criterion, a minimum leaf sample size of three and 80 estimators (trees). With these settings we can again — compared to the single decision tree — increase the training accuracy to 70%.</p><h4><strong>Support Vector Regressor</strong></h4><p>The last regressor we are going to build is based on <a href="https://en.wikipedia.org/wiki/Support-vector_machine">support vector machines</a>, a beautiful mathematical concept developed by <a href="https://en.wikipedia.org/wiki/Vladimir_Vapnik">Vladimir Vapnik</a> between the 1960s and 90s. Unfortunately, explaining their inner workings would go beyond the scope of this article. Still, I strongly recommend to check them out; one good introductory resource is <a href="https://www.youtube.com/watch?v=_PwhiWxHK8o">professor Winston’s lecture at MIT</a>.</p><p>A very rudimentary summary: Support vector regressors try to fit given samples into a multi-dimensional (with the order of the number of features) hyperplane of a diameter defined by linear boundaries while minimizing the error or cost.</p><p>Although this type of model is fundamentally different from decision trees and forests, the implementation with Scikit-learn is similar:</p><pre>svr_parameters = [{&#39;kernel&#39;: [&#39;linear&#39;, &#39;rbf&#39;],<br>                   &#39;C&#39;: helpers.powerlist(0.1, 2, 10),<br>                   &#39;epsilon&#39;: helpers.powerlist(0.01, 2, 10),<br>                   &#39;gamma&#39;: [&#39;scale&#39;]},<br>                  {&#39;kernel&#39;: [&#39;poly&#39;],<br>                   &#39;degree&#39;: list(range(2, 5, 1)),<br>                   &#39;C&#39;: helpers.powerlist(0.1, 2, 10),<br>                   &#39;epsilon&#39;: helpers.powerlist(0.01, 2, 10),<br>                   &#39;gamma&#39;: [&#39;scale&#39;]}]<br>svr_grid = GridSearchCV(estimator=SVR(),<br>                        param_grid=svr_parameters,<br>                        scoring=mean_relative_accuracy, cv=5,<br>                        n_jobs=-1, iid=False)<br>svr_grid_result = svr_grid.fit(X_train_scaled, y_train_scaled)<br>best_svr_parameters = svr_grid_result.best_params_<br>svr_score = svr_grid_result.best_score_</pre><p>We can use grid search again to find the optimal values for some model parameters. The most important here is the kernel that transforms the samples into a feature space of higher dimensions where the data can be separated or approximated linearly, i.e. by the hyperplane described above. We are testing a linear kernel, a polynomial one and a <a href="https://en.wikipedia.org/wiki/Radial_basis_function">radial basis function</a>. With an epsilon of 0.08, the maximum (scaled) distance of the prediction from the true value where no error is associated with it, and a penalty parameter C of 12.8, the linear kernel performs best, reaching a (scaled) training accuracy of 23%.</p><h3><strong>Evaluate your models</strong></h3><p>After we have now identified the best parameters for our models based on the training data at hand, we can use these to eventually predict the outcomes of the test set and calculate their respective test accuracies. First, we need to fit our models with the desired hyperparameters to the training data. This time, we don’t need cross-validation anymore and will fit the models to the full training set. Then we can use the fitted regressors to predict training and test set results and calculate their accuracies.</p><pre>training_accuracies = {}<br>test_accuracies = {}<br>for regressor in regressors:<br>    if &#39;SVR&#39; in str(regressor):<br>        regressor.fit(X_train_scaled, y_train_scaled)<br>        training_accuracies[regressor] = hel.mean_relative_accuracy(<br>            y_scaler.inverse_transform(regressor.predict(<br>                X_train_scaled)), y_train)<br>        test_accuracies[regressor] = hel.mean_relative_accuracy(<br>            y_scaler.inverse_transform(regressor.predict(<br>                X_test_scaled)), y_test)<br>    else:<br>        regressor.fit(X_train, y_train)<br>        training_accuracies[regressor] = hel.mean_relative_accuracy(<br>            regressor.predict(X_train), y_train)<br>        test_accuracies[regressor] = hel.mean_relative_accuracy(<br>            regressor.predict(X_test), y_test)</pre><p>Here are the results:</p><p>Training accuracies: Linear 0.34, Tree 0.67, Forest 0.75, SVR 0.63<br>Test accuracies: Linear 0.32, Tree 0.64, Forest 0.66, SVR 0.61</p><p>Our best regressor is the random forest with the highest test accuracy of 66%. It seems to be a little overfit though since the deviation from its training accuracy is relatively large. Feel free to experiment with other values for the hyperparameters to further improve all of the models.</p><p>Before finally saving our model to make predictions on new data, we will fit it to all available data (training and test set) to incorporate as much information as possible.</p><h3><strong>Predict results</strong></h3><p>Now we have a model ready to predict the results of future marketing campaigns. We only have to call the predict method on it passing a specific feature vector and will receive the respective prediction for the metric we trained our regressor on. We can also compare the true impressions from the existing dataset to their predictions based on our new model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DQYLI0-SDgnoYjd5PRzU8w.png" /></figure><p>The average relative deviation of a prediction from its actual, true value equals 26%, ergo we reached an accuracy of 74%. With only 14% the median deviation is even smaller.</p><h3>Conclusion</h3><p>We were able to build and train regressors that allow us to predict the number of impressions (and other performance metrics in an analogue fashion) for future marketing campaigns from historical campaign data.</p><blockquote>The highest prediction accuracy has been achieved with a random forest model.</blockquote><p>We can now use these predictions to evaluate a new marketing campaign even before its start. Also, this allows us to determine the best parameters including e.g. timeline and budget size for our campaigns since we can calculate the predictions with different values for those features.</p><h3><strong>BONUS: Ready-to-use trained model</strong></h3><p>You don’t (yet) have the data at hand to build an accurate prediction model for your planned digital marketing campaigns? Don’t worry: I have trained multiple models to predict impressions, clicks and purchases with the data from more than 1,000 campaigns. With a combination of different models those predictions reach an accuracy of up to 90%. At <a href="https://predictor.stagelink.com/"><strong>predictor.stagelink.com</strong></a> you find a simple app to predict the results of your future campaigns with just a few inputs. The models are mainly trained on data from digital marketing campaigns promoting event ticket sales, so that is where they will likely perform best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5zWYztfOw6LOBuu8pRefg.png" /><figcaption>predictor.stagelink.com</figcaption></figure><p>In addition to that you can find all the code used for the discussed marketing performance predictor on my Github: <a href="https://github.com/kinosal/predictor"><strong>github.com/kinosal/predictor</strong></a></p><p>Thank you for reading - I look forward to any feedback you might have!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=579fbb153a97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-predict-the-success-of-your-marketing-campaign-579fbb153a97">How to predict the success of your marketing campaign</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When and when not to A/B test]]></title>
            <link>https://medium.com/data-science/when-and-when-not-to-a-b-test-c901f3ad96d9?source=rss-5ccde54d3072------2</link>
            <guid isPermaLink="false">https://medium.com/p/c901f3ad96d9</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <dc:creator><![CDATA[Nikolas Schriefer]]></dc:creator>
            <pubDate>Mon, 27 May 2019 23:27:48 GMT</pubDate>
            <atom:updated>2019-05-28T05:54:32.223Z</atom:updated>
            <content:encoded><![CDATA[<h4>Split test vs. multi-armed bandit: simulation, source code and ready-to-use app</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*isiIeB8LtQ6xeLMq" /><figcaption>Photo by <a href="https://unsplash.com/@daniilvnoutchkov?utm_source=medium&amp;utm_medium=referral">Daniil Vnoutchkov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article discusses the differences of A/B or Split tests and multi-armed bandits for variant testing. It can be used to e.g. choose the best performing set of ads for a digital marketing campaign. Simulating and measuring the performance of both methods in various experiments shows that <strong>in most scenarios the Bandit dominates the Split and should be preferred</strong>. In addition to the simulation results, all of the required source code is provided plus a simple web app for variant optimization using the Bandit method.</p><p>App: <a href="https://optimizer.stagelink.com/">optimizer.stagelink.com</a><br>Code: <a href="https://github.com/kinosal/optimizer">github.com/kinosal/optimizer</a></p><h3>Story Outline</h3><ol><li>What is an A/B test and what’s the problem?</li><li>The alternative: Multi-armed bandits</li><li>Experiment: Split vs. Bandit</li><li>Parameters + Conclusion</li></ol><p>So let’s get going:</p><h3>What is an A/B test?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uh3f_U9pfno1hmpp" /><figcaption>Photo by <a href="https://unsplash.com/@diesektion?utm_source=medium&amp;utm_medium=referral">Robert Anasch</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A/B, A/B/n or Split Tests - which is how we will call them - are used to determine which of several alternative scenarios or variants is more likely to generate a desired - successful - outcome or response by testing, measuring and comparing the behavior, i.e. the success rates, in each scenario. There are various use cases for such tests. This article will focus on maximising conversion rates, e.g. from users seeing an online advertisement to clicking on such ad or users completing a purchase after visiting a website.</p><p>The outcome or result of a Split Test can be described with a <em>NxM</em> matrix where <em>N</em> rows contain the number of <em>successes</em> and <em>failures = trials - successes</em> for all tested scenarios. In an example where Ad (scenario or variant) 1 has been shown to 1,000 people (trials), 10 of which clicked on the ad (successes), and Ad 2 has been shown to different 1,000 people of which 20 clicked on the ad, this matrix, which is also called a contingency table, looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*0ZNuFU914flFv676lzYGDw.png" /></figure><p>Now the objective is to determine whether one of the two ads’ (scenarios’) success rates (likelihood to click) is “significantly” higher than of the other ad (scenario). This would mean there was a high likelihood that the samples drawn in all scenarios do not originate in the same population (or a population with the same - in our case <a href="https://en.wikipedia.org/wiki/Binomial_distribution">binomial</a> - distribution), which is our null-hypothesis <em>H0</em> of the success rate being independent from the observed scenarios that we are seeking to invalidate or disprove. This is important so we can eventually conclude that the differences in the scenarios are in fact responsible for the different success rates, and not only the randomness within these scenarios. Statistical significance or confidence level can be explained as the probability of independence = the probability of wrongly rejecting <em>H0</em> (a <a href="https://en.wikipedia.org/wiki/Type_I_and_type_II_errors">type I error</a>) being lower than a predefined p-value. There are several methods to test for significance, we will be focussing on the (Pearson’s) <a href="https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test">chi-squared test</a> without (<a href="https://en.wikipedia.org/wiki/Yates%27s_correction_for_continuity">Yates</a>’) correction. The p-value can be derived from the observed data displayed in a contingency table via the chi-squared statistic which is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*xRfcDV97K_0ZxIxe5NROkg@2x.png" /></figure><p>where <em>Oi,j</em> are the observed successes and failures while <em>Ei,j</em> are their respective expected values if the underlying distribution was the same, i.e. 15 successes and 985 failures in the above example. The statistic thus indicates the dissimilarity of the two scenarios, if there is no difference in the observed scenarios chi-square equals zero. The p-value can now be calculated as the right-tailed area under the curve (the integral) of the <a href="https://en.wikipedia.org/wiki/Chi-squared_distribution">chi-squared distribution</a> with the proper <a href="https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)">degrees of freedom</a> (the number of scenarios minus one in our case of two possible outcomes) from the value of the chi-square statistic (more about this later when conducting the experiment). If the p-value is lower than previously defined (as in the desired confidence or significance level) the null-hypothesis can be rejected and the outcomes deemed dependent on the observed scenarios. In the above example chi-squared equals 3.3841 and p is 0.0658 (which might not be low enough to disprove the notion of independence).</p><h3>So what’s the problem?</h3><p>As we have seen we need to achieve some level of confidence in our desired outcome (successes or clicks in the above example) to depend on a provided scenario. Only then we can “safely” disregard the inferior scenario or variant - e.g. ad - and thus maximize positive results.</p><blockquote>The problem here is one of exploration and exploitation.</blockquote><p>In order to exploit the situation by moving completely to the best scenario (e.g. show only that ad), we need to invest in exploring the situation until we are confident enough. This can be expensive since we might be paying to deliver an ad impression and/or because of the opportunity cost induced by not maximizing our return due to a “bad” choice of scenario(s). So, until we reach the desired confidence in our decision to move entirely to the designated superior scenario, we are purely paying to explore; after we have reached this point, we can fully exploit that situation (given the scenarios do not change over time). This doesn’t sound like an optimal solution, does it?</p><p>What if we could more gradually and earlier move towards the scenarios we believe might turn out to be superior and thus reduce exploration costs and increase exploitation returns, especially in the short run?</p><p>I’m glad you asked.</p><h3>The alternative: Multi-armed bandits</h3><p>Before going into detail, let’s quickly talk about the battle between frequentist and bayesian approaches to statistics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*ULb9lkajm38eGIfrtRX7ZQ.png" /><figcaption>Source: <a href="https://www.xkcd.com/1132">https://www.xkcd.com/1132</a>, also compare the heated debate about this comic at <a href="https://stats.stackexchange.com/questions/43339/whats-wrong-with-xkcds-frequentists-vs-bayesians-comic">https://stats.stackexchange.com/questions/43339/whats-wrong-with-xkcds-frequentists-vs-bayesians-comic</a></figcaption></figure><p>Until now we were living a frequentist life. However, since we don’t want to be supernova-ed, this is going to change. <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayesians</a> update their existing beliefs about (the probability of events in) the (stochastic) world (from prior to posterior) by obtaining additional information conditionally related to those beliefs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*HdAlVUM4x0dh15y-EhjUiw@2x.png" /></figure><p>where A and B are events, P(A) and P(B) are the (marginal) probabilities of observing those events (where P(B) ≠ 0), P(A|B) and P(B|A) are conditional probabilities (e.g. the likelihood of A given that B is true).</p><p>There are many alternatives to “classical” frequentist A/B tests, this article will highlight a specific Bayesian approach that can be standardized and automated similar to a split test: the multi-armed bandit,</p><blockquote>“a smarter version of an A/B test”</blockquote><p><a href="http://www.invisionapp.com/inside-design/let-go-a-b-testing">according to Kai Rikhye from InVision</a>.</p><h3>A brief introduction to multi-armed bandits</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LMw1BZzuXR2VD8FW" /><figcaption>Photo by <a href="https://unsplash.com/@stevesawusch?utm_source=medium&amp;utm_medium=referral">Steve Sawusch</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Remember we want to more gradually and earlier move towards the scenarios we believe might turn out to be superior and thus reduce exploration costs and increase exploitation returns - a classic tradeoff. It is like us wondering which slot machine (one-armed bandit) to play in a casino; if we knew the success rate of each machine, we would only ever invest money into the machine with the highest winning probability and completely disregard all other options. However, if we do not know this, we need to explore the options in order to estimate all machines’ success rates while maximizing the overall (cumulative) return we can expect from playing. This optimization problem can be solved by testing different options and reinforcing them when a reward could be observed (hence a problem of <a href="https://en.wikipedia.org/wiki/Reinforcement_learning">Reinforcement Learning</a>). Yet, finding the exact solution is hard. Luckily there are approximations sufficient for our use cases. In this article we will focus on a method called “<a href="https://en.wikipedia.org/wiki/Thompson_sampling">Thompson sampling</a>”, introduced by William R. Thompson back in 1933 and based on randomly drawing samples from each option’s <a href="https://en.wikipedia.org/wiki/Beta_distribution">beta distribution</a> parameterized by its past successes <em>α</em> and failures <em>β</em> with the density function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*q4YlQzHSqoR5TImjy-61GQ@2x.png" /></figure><p>where <em>θ</em> equals the success rate of the respective option and <em>Γ</em> denotes the <a href="https://en.wikipedia.org/wiki/Gamma_function">gamma function</a>. Important properties of the beta distribution are that the mean equals <em>α / (α + β)</em> (success rate = successes / trials) and that the distribution becomes more concentrated as <em>α + β</em> (the number of trials) grows. Thus the option with the highest drawn sample in a given drawing round which will be chosen for the next trial likely has a high success rate with high confidence and can be exploited or it has any success rate with low confidence and still needs to be explored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7TBSFwIoWa3VdCOfnQLDug.png" /><figcaption>Probability Density Functions, Source: <a href="https://en.wikipedia.org/wiki/Beta_distribution#/media/File:Beta_distribution_pdf.svg">https://en.wikipedia.org/wiki/Beta_distribution</a></figcaption></figure><p>Having established some theory, we can now move on to practically simulating the behavior of Split tests and multi-armed bandits when testing the performance of different variants.</p><h3>Experiment: Split vs. Bandit</h3><h3>Setup</h3><p>We are going to test two different methods (the chi-squared split test - “Split” hereafter - and the Thompson beta bandit - “Bandit”) with the objective to maximize the cumulated successes (e.g. ad clicks) by sequentially (over several periods, e.g. days) choosing trials (e.g. presenting ads to users) from multiple options (e.g. different ad creatives). Based on the above described different nature of these algorithms we expect different results depending on different environments controlled by the following variables: number of options, true success rates for these options, trials per period, confidence level (maximum p-value) and success rate instability (standard deviation). We are going to start with a simple environment with 2 options and fixed success rates (no deviation) before exploring alternative scenarios.</p><h3>Basic Simulation</h3><p>The basic experiment can even be simulated in a <a href="https://docs.google.com/spreadsheets/d/1a2ss3cepB6apQTH16uDwIo3gZ9kWBSxQD7ijM3vbcp0">spreadsheet</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LK4nI0Qk0E300HsYSaSegg.png" /><figcaption><a href="https://docs.google.com/spreadsheets/d/1a2ss3cepB6apQTH16uDwIo3gZ9kWBSxQD7ijM3vbcp0/edit#gid=0">https://docs.google.com/spreadsheets/d/1a2ss3cepB6apQTH16uDwIo3gZ9kWBSxQD7ijM3vbcp0</a></figcaption></figure><p>In this sheet the environment parameters are set in cells A1:I3. Based on these, the decisions and outcomes for the Split and the first 7 periods are calculated in cells A5:P13. As discussed before all trials per period will be distributed evenly over the options until a winner can be declared with statistical significance. For a scenario with two options, true success rates of 1% and 1.5% with no deviation, 2,000 trials per period and a maximum p-value of 0.1 (90% confidence) this method detects the higher success rate for option 2 after period 6 and only trials this option from then on. Columns L:P show the total, cumulated successes as well as the absolute and relative difference (the loss) from the optimal situation where always the best option has been chosen.</p><p>Cells A15:P23 analogously calculate these values for the Bandit. Instead of drawing one sample and choosing one option at a time, for comparability and practicability this method is also implemented with periodic decisions where the percentage of an option shown in period t equals the number of times it has been chosen by 100 random samples of the beta distribution constructed based on the data from periods 1 until t-1 (every individual sample and choice for all periods can be seen in cells A25:R126). With the sample environment variables described above we should find more cumulative successes and thus a smaller loss compared to the optimal choices for the Bandit then the Split. This can also be visualized by plotting the absolute and relative successes respectively over all periods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*4I1Irakxjy1fLAgiNcBcoQ.png" /></figure><p>Here it becomes obvious that - at least in this basic simulation - the Bandit dominates the Split over all periods. So let us examine the elasticity of this outcome by experimenting with alterations of the different parameters. You can do this sequentially in the provided spreadsheet, but since this and the future mutations we want to apply are much easier to simulate coded in Python, we will turn to that now.</p><p>The Split class is initiated with the number of options available, trials, successes and failures vectors as <a href="https://www.numpy.org/">Numpy</a> arrays as well as the (current) p-value:</p><pre>class Split():<br>  def __init__(self, num_options):<br>    self.num_options = num_options<br>    self.trials = np.zeros(shape=(self.num_options,), dtype=int)<br>    self.successes = np.zeros(shape=(self.num_options,), dtype=int)<br>    self.failures = np.zeros(shape=(self.num_options,), dtype=int)<br>    self.p_value = 1.0</pre><p>Then we need a method to add results (trials, successes and failures) for one period to a Split instance:</p><pre>def add_results(self, option_id, trials, successes):<br>  self.trials[option_id] = self.trials[option_id] + trials<br>  self.successes[option_id] = self.successes[option_id] + successes<br>  self.failures[option_id] = \<br>    self.failures[option_id] + trials - successes</pre><p>And lastly, we will recalculate the p-value for the current state:</p><pre>def calculate_p_value(self):<br>  observations = []<br>  for i in range(self.num_options):<br>    observations.append([self.successes[i], self.failures[i]])<br>  self.p_value = \<br>    chi2_contingency(observed=observations, correction=False)[1]</pre><p>The <em>chi2_contingency</em> function can be found in the stats module of the <a href="https://docs.scipy.org/doc/scipy/reference/stats.html">SciPy library</a> and calculates the p-value from a given contingency table using the same process as described above for the spreadsheet.</p><p>The Bandit is initialized similarly, only that we do not require a p-value or a failures vector, but includes a prior tuple which could be used to create a bandit with existing prior information to construct the Beta distribution described above:</p><pre>class BetaBandit():<br>  def __init__(self, num_options):<br>    self.num_options = num_options<br>    self.prior = (1.0, 1.0)<br>    self.trials = np.zeros(shape=(self.num_options,), dtype=float)<br>    self.successes = \<br>      np.zeros(shape=(self.num_options,), dtype=float)</pre><p>Results can be added in the same way as for the Split (only without the failures).</p><p>Now we can choose one option as the highest sampled value from the underlying beta distributions of all options:</p><pre>def choose_option(self):<br>  sampled_theta = []<br>  for i in range(self.num_options):<br>    dist = beta(self.prior[0] + self.successes[i],<br>                self.prior[1] + self.trials[i] - self.successes[i])<br>    sampled_theta += [dist.rvs()]<br>  return sampled_theta.index(max(sampled_theta))</pre><p>The beta function is also part of the stats module of the SciPy library, <em>rvs()</em> returns a random variable based on the underlying distribution.</p><p>For batch simulation we need to repeat this choice of options:</p><pre>def repeat_choice(self, repetitions):<br>  option_counts = np.zeros(shape=(self.num_options,), dtype=int)<br>  for _ in range(repetitions):<br>    option = self.choose_option()<br>    option_counts[option] += 1<br>  return option_counts</pre><p>The basic simulation then runs adequately to the above spreadsheet example:</p><ol><li>Initialize Split and Bandit</li><li>Add results for first period with equal distribution of trials over all options</li><li>Recalculate p-value for Split</li><li>If p-value &lt; max p-value: Add results to Split with equal distribution of trials over all options, else: Add results to Split with all trials for best option (with most successes)</li><li>Choose batch options for Bandit and add results with distribution of trials as in choices</li><li>Start again from 3.</li></ol><p>The results are (almost, accounting for the randomness of the beta sampling) the same as already seen above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cdCBo5iexwQ3BIwure2Djw.png" /></figure><p>Before we continue with the elasticity experiments, let us examine 28 instead of 7 periods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4mrlIwFEDMstPaxE-TbFbg.png" /></figure><p>The long term differences between Split and Bandit are smaller since the Split finds the optimal option after the 6th period and after that allocates all trials to that option where the Bandit still occasionally tries the weaker option (although very sparsely). Still, also after 28 periods the Bandit has outperformed the Split.</p><p>Okay, you want to know if that changes after 100 periods? It does not:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NffxnOuKqOJ9eKBsFBkww.png" /></figure><p>After this basic experiment, we want to take a final closer look at how the various simulation parameters affect the results before coming to a conclusion about when and when not to A/B test.</p><h3><strong>Parameter Elasticity</strong></h3><h4>P-Value</h4><p>We want to analyze the effect of some of the environment parameters on the experiment results, starting with different p-values for the Split (going back to a default observation length of 28 periods):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*reP743zc311jZYFd4GB6Ww.png" /></figure><p>We can see that the results get better (closer to the optimum) with higher p-values. We have to be careful though to jump to a conclusion using higher p-values for our models since this will lead to more false positives (Type I errors) and thus increase the risk of choosing a sub-optimal option as the “winner”. For the remaining experiments we will set the p-value to 0.1 since we are okay with wrongly rejecting the null-hypothesis of no significant difference between the variants 10% of all times.</p><h4>Trials</h4><p>A look at different trial numbers for the Split (left) and the Bandit (right):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R496UH1v0wefqpJkEo0yTg.png" /></figure><p>As we could expect, more trials lead to higher confidence in the Split test results and thus to finding the optimal option sooner. For both methods we can observe a seemingly asymptotic behavior (the marginal improvement by more trials decreases).</p><h4>Success Rates</h4><p>While we are able to influence the p-value and potentially also the number of trials when trying to find the best options to achieve our objectives in the real world, the true success rates are usually independent from our actions (not getting into Schroedinger here…). Thus the difference in the two methods will be even more important here (again Split on the left, bandit on the right):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*diXq3lLSW8eltuKcNcjGAg.png" /></figure><p>Again, the Split is being dominated by the Bandit and even more so if the differences in true success rates of the available options is small. In the [0.01, 0.012] case, the Split does not even find the optimal option after 27 periods.</p><h4>Uncertainty</h4><p>When the true success rates are not completely invariant (which they most likely aren’t in a “real world” environment), but normally distributed with a standard deviation of 50% of their mean, the results change quite drastically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebXpEaJUqAzXOcKvVq_QOw.png" /></figure><p>These results obviously depend on the induced <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.RandomState.html">random state</a>, with a different state they might look very different:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Q0aInYW7sD7Y-DiuZgfBA.png" /></figure><p>Generally the dominance of the Bandit is less apparent in a more probabilistic environment.</p><p>Let us now compare the results for both methods under different uncertainty levels (success rate standard deviations):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86kINjNQJUaJDCd2st0LLA.png" /></figure><h4>More Options</h4><p>Last but not least, we want to take a look at scenarios with more than two competing options. With 11 different options with equally distributed true success rates between 1% and 2% Split and Bandit results relative to optimal success look as follows (no deviation in success rates on the left, 25% standard deviation from mean on the right):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XATmQFsbvVHSaPC3L70UGA.png" /></figure><p>Both methods do worse when there are more options. Still, until the Split has found the optimal option, the Bandit outperforms it by a substantial margin. However, when the experiment runs long enough, the end result looks quite different:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oBHRnBFOmuxbYT366VtGw.png" /></figure><p>While the Split beats the Bandit in a deterministic environment, the Bandit seems to suffer less from the introduced uncertainty even in the long run.</p><h3>A real world example</h3><p>The scenario here is a digital advertising campaign with 50 different ad combinations (options) with randomly (uniform) distributed click rates (true success rates) between 1% and 4%. The click rates’ standard deviation is 50% of their respective mean. There will be 5,000 trials over 28 periods each. We keep the p-value for the Split at 0.1. Here are the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yEeb0ryzr2won9sgWMBB8A.png" /></figure><p>From 8,904 possible successes the Split yields 4,346, the Bandit 3,968. The bad news is that both methods leave more than 50% of the potential on the table compared to a scenario where the true success rates were known for every period. The good news is that both methods lift the base rate (3,137 successes) by at least 26%.</p><h3>Conclusion</h3><p>To answer the initial question “when and when not to use A/B test”: especially when exploration is costly and short-term exploitation important and/or when dealing with an uncertain, i.e. probabilistic, environment, the Bandit provides a very good, mostly superior alternative in finding and using the best options and thus maximising efficiency. However, when the Split (the A/B test) is confident in having identified the best option(s) it might yield better results.</p><p>An obvious conclusion and start for further research/testing could therefore be to combine both methods in a hybrid model, using the Bandit until the Split confidence is large enough (the p-value is smaller than the defined maximum), then leveraging the Split to reduce the option set to fewer “winners” with the highest success rates and after that using the Bandit again to choose the option(s) for the next periods until the Split’s p-value is again small enough and so on…</p><p>There is one more caveat we have not yet covered though: the options’ success rates and their distributions respectively might change over time. If this is the case, we might not want to reduce the possible option set at all, but flexibly adjust our decision process with any new observed results. Since the foundation of the Split is based on limiting options, i.e. identifying winners, this will dramatically affect the goodness of fit of this method in such an environment. The Bandit, however, updating its belief about the world after every new observation (set), can be constructed to deal with such behaviors, e.g. by introducing a discount function that reduces the impact of older results when choosing the preferred options for the next period. To briefly highlight the Split’s defect with respect to changing success rates, have a look at a scenario where the rates’ means increase or decrease up to 5% per period (practically you can e.g. imagine the click rates of ads shown to one particular audience increase over time due to audience development e.g. by retargeting while the click rates of ads shown to another audience decline due to saturation):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HjmwN0Moceh0tZmersbgMg.png" /></figure><p>As you can see, the Split now cannot even outperform the random base allocation anymore. The Bandit seems almost unaffected, although it could probably be improved further with mentioned discount for older results.</p><p>So all-in-all,</p><blockquote>Bandits provide your go-to methodology when testing the performance of different variants or scenarios</blockquote><p>and if you are still using A/B tests, you should at least consider a switch.</p><p>Before I leave you and look forward to receiving your feedback, comments and ideas I would like to thank you for staying with me for such a long and intense article. As a little goodie I have built an ad optimization tool for any of you who would like to use the Bandit for their digital marketing campaigns: <a href="https://optimizer.stagelink.com/"><strong><em>optimizer.stagelink.com</em></strong></a></p><p>In addition to that you can find all the code used for the discussed simulation on my Github: <a href="https://github.com/kinosal/optimizer"><strong>github.com/kinosal/optimizer</strong></a></p><p>Thanks again and talk to you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c901f3ad96d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/when-and-when-not-to-a-b-test-c901f3ad96d9">When and when not to A/B test</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>