With Google pulling the plug on third party cookies in 2023 and data privacy regulations in place, most brands have been looking at other methods to measure their marketing effectiveness.

One such method which can aid marketers in creating effective strategies is Marketing Mix Modeling (MMM).

Market Mix Modeling (MMM) is a technique which helps in quantifying the impact of several marketing inputs on the KPI (e.g. Sales, Market Share, CTR etc.). The purpose of using MMM is to understand how much each marketing input contributes to the KPI, and how much to spend on each marketing input.

We have been requested by many to give our take on open-source libraries like Robyn.

We have been using our proprietary techniques + Robyn for some of our MMM projects.

We are starting a series “Robyn under the hood” to educate and inform MMM users across the globe on Robyn.

Through this series, we dig deep into the methods/ techniques used and try to answer the ‘How’ & ‘Why’ behind these techniques like:

· Why Robyn uses Ridge Regularization?

· Does Ridge reduce overfitting?

· What loss function is used?

· Why is Weibull distribution used for Adstock? Why not any other distribution?

· How Multi-objective optimization works for tuning hyper-parameters?

· What is Pareto optimality?

To kick-off, we will first cover Ridge Regularization in Robyn.

Robyn Under the Hood

“Robyn is an experimental, semi-automated and open-sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. It uses various machine learning techniques (Ridge regression, multi-objective evolutionary algorithm for hyperparameter optimization, time-series decomposition for trend & season, gradient-based optimization for budget allocation etc.) to define media channel efficiency and effectivity, explore adstock rates and saturation curves.”¹

Under the hood, Robyn uses Ridge regression to regularize multi-collinearity and automate hyper-parameter optimization using evolutionary algorithms from Facebook AI’s library Nevergrad. It also makes use of Facebook’s prophet library to decompose the time series into trend, seasonality and holidays.

So, first let’s do a drill down on Ridge Regression.

1. Ridge Regression:

Ridge Regression (depicted on the right hand size in the above image) is a regularization technique which helps combat overfitting in the model. For more detailed explanation on why Ridge Regression does not reduce the coefficients to zero, check out the resources here and here.

There are some models which perform well on the training set but fail to perform on the test set. In other words, the model has poor predictive power when it comes to unseen data and is overfitted on the train set. To help combat this, regularization technique is applied which reduces the variance at the cost of adding a bias.

One such method is Ridge Regression.

2. Why is Ridge regression used in MMM?

It is very common to observe multicollinearity among different independent variables in MMM. This leads to model being overfitted. Through Ridge regression, a penalty term is introduced to the model cost function. The penalty term is the sum of squares of all the model coefficients multiplied by Lambda (λ). Refer to the equation below:

The penalty term penalizes the regression model and shrinks the coefficients towards zero. Here, lambda controls the severity of the penalty. Higher the value of λ, higher is the impact of shrinkage.

So, if there are too many variables in the model, the impact of some variables will be closer to zero. Ridge Regression does not remove predictors from the model, it just shrinks them towards zero (but does not make it zero). The coefficients are not exactly 0 and we can still see the impact of all the predictors in the model.

3. Ridge Regression in Robyn

In the next post, we will try to cover the functions responsible for implementing Ridge regression in Robyn.

Resources:

1. https://facebookexperimental.github.io/Robyn/docs/about

2. https://facebookexperimental.github.io/Robyn/docs/analysts-guide-to-MMM

Want to implement Marketing Mix Models (MMM) and other Marketing Analytics Solutions?

Get in touch with us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

So, Google decided to postpone the sunsetting of cookies to 2024. Sigh.. I know, I can hear the anguish of ‘Not Again’ .

In fact just a few days ago, I had created a meme on how the cookieless winter is coming in 2023. Thanks to Google’s announcement I had to edit it. :)

As a fallout of this announcement from Google, Some of my clients and friends from AdTech/Martech messaged and asked me:

• What will be the effect of this postponement ?
• How does it affect companies that has already started to implement MMM based solutions?
• What happens to those who still have not figured out a non cookie based solution?

So here is my take:

The Head Start

• Some companies started experimenting and implementing MMM based solutions as early as 2021 (just few months after Google had announced then that they would deprecate cookies by 2022). Not to mention, there are companies/agencies who have been implementing MMM for past 20–30 yrs. In my opinion, these companies have a huge head start vis a vis companies that are still figuring out solutions.

The Taste of Privacy

• Some companies and their clients have had a sweet taste of the cookie free solutions like MMM. A taste of freedom in a sense.
  For them going back to the cookie based solutions would be a downgrade of sorts.

2024 is not that far off

• Some companies might be tempted to take their foot of the peddle thinking that 2024 is far away. From my experience, getting MMM based solutions ready, requires a lot of R&D and experimentations. The more you experiment, the more quickly you will arrive at a good solution. So as the saying goes, “the best time to plant a tree was 20 yrs ago and the next best time is now”. Now would be the right time to experiment and adopt MMM solutions.

Want to implement Marketing Mix Models (MMM) and other Marketing Analytics Solutions?

Get in touch with us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

I have been having interesting conversations with my clients and prospective clients of late.

One question that is on everybody’s mind is “Will the adoption of MMM and MTA increase or decrease given the looming macro economic conditions ?”

Here is my take :

1) Being prudent with marketing spends.

Organizations want to become more prudent with their marketing spends. They want to maintain same level of marketing spends and yet retain same (or higher) levels of sales (or other KPIs).

MMM is tailor made for scenarios like this. Budget optimization post MMM helps answer exactly this.

2) MMM’s time in the sun just arrived sooner

With Google about to pull the plug on third party cookies in 2023 and data privacy regulations in place, most brands have been looking at other methods to measure their marketing effectiveness.

Many marketers had opined that the adoption of MMM will hence skyrocket. I believe the impending economic conditions will make the adoption of MMM even quicker.

3) MMM will be the barometer every organization needs.

During these tumultuous times, every organization would like to constantly measure their Marketing ROI. They would want to know how best to deploy their marketing budget into various marketing media. MMM is the best MROI or ROAS barometer.

As they say, every cloud has a silver lining. The upcoming financial tough times will perhaps make organizations adopt more prudent marketing strategies aided by good domain knowledge and data driven methods.

I believe we all will learn to make more with less.

For innovative marketing analytics solutions, you can reach out to us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

With browsers pulling the plug on third party cookies in 2023 and data privacy regulations in place, most brands have been looking at other methods to measure their marketing effectiveness.

One such method which can aid marketers in creating effective strategies is Marketing Mix Modeling (MMM).

MMM is being used in various industries for quite some time now, but many marketers have started to realize its importance due to third party cookies getting deprecated.

There are many advantages which MMM offer. Most data required for MMM are easily available and MMM doesn’t need customer personally identifiable information (PII).

However, there are several myths associated with MMM:

1. MMMs take lot of time to build

2. MMMs are expensive

3. MMMs are data intensive

4. MMMs don’t factor in digital data

5. MMMs can only be used by CPG industry

Well, neither of the above myth is correct.

Here is a take on how MMM has stood the test of time and will be useful for a long time to come.

The revamped MMM is more agile

With digital disruption taking place, MMM has seen a big revamp. Gone are the days when Marketing Mix Models used to take months to build. MMMs used these days are more agile and can be built faster and for shorter time periods. The quick and ready to use MMMs can provide actionable insights at a quicker cadence to marketers who are looking to get more real time insights.

The new MMM can use more granular data

Most digital campaigns have a short-term impact on Brand Awareness. Marketers need to feel the pulse of the customers/markets faster, and course correct their campaigns to get higher returns. To cater to such demands, MMMs can be easily adjusted to factor in more granular data (weekly/daily)

MMM goes hand-in-hand with the new digitally savvy world

Due to Covid-19, e-commerce sector has seen an unprecedented growth in the last year. With the rise in e-commerce revenue, many brands are interested in knowing which campaigns move the needle of online sales. Many brands have started to realize how MMM can help them understand that.

• Brands have started using MMM to understand which social media platforms work better and which campaigns are generating higher ROAS.
• MMM is not just backward looking anymore, the new agile MMM can be used for understanding which campaigns could work better.

MMM is not just backward looking anymore, the new agile MMM can be used for understanding which campaigns could work better.

MMM adoption going up

The purpose of using MMM is to quantify the impact of various marketing investments on KPIs such as Sales, Market Share, Brand Awareness, Leads, Orders etc.

Any marketers goal is to get the bang for their buck. This goal is same across all brands irrespective of whichever industry or category they are into.

Gone are the days when MMM used to be applied only in CPG. MMM is now readily adopted in various industries such as like e-commerce, pharma, manufacturing, banking, travel/hospitality etc.

Gone are the days when MMM used to be applied only in CPG. MMM is now readily adopted in various industries such as like e-commerce, pharma, manufacturing, banking, travel/hospitality etc.

The Takeaway

With the data privacy concerns increasing and the world becoming less, and less cookie driven, MMM will be a crucial tool in the armor of marketing teams.

For innovative marketing analytics solutions, you can reach out to us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

We have built 20 Markov Attribution models for various companies across geographies in the last 2 yrs.

Here are the 7 lessons we learnt along the way.

Reality is different from toy examples:

Things don’t work as easily as illustrated on toy examples. Markov Attributions implementation on a Real data set is a different beast altogether.

It is all about the Paths:

Roughly, 60% of the Markov attribution projects is about Data pre-processing and data transformation (forming the paths).

Pay attention to the timestamp:

While aggregating the data, one needs to pay attention to the order of timestamp (both in terms of format and chronology). The chronology of path is extremely important.
Any mix up will result in wrong attributions.

It is good to break the problem:

Customers are more interested in which paths lead to conversion rather than which has not. Hence it makes sense to break your data set into two
Paths leading to conversion
Paths leading to No conversion

Apply the algorithm selectively:

Building up on the last point, apply the algorithm only on the ‘path s leading to conversion’. Applying the algorithm to the whole data set (sometimes having millions of records) can lead to intractable solution.

Don’t forget the removal effects:

The removal effect tells you how much is the contribution of a channel by removing that channel from the path and seeing how many conversions are happening without that channel. Applying Removal effects is crucial to get the right attributions.

Interpret Markov results through lens of Domain Experience:

The Markov results only show which paths lead to higher conversions. It does not quantify the value of each conversion per say.

For example path A may have resulted in maximum number of conversion relative to path B but path B though having less number of conversion could have resulted in the conversions with higher revenue.

For innovative marketing science solutions, you can reach out to us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

“Logistic Regression is not Regression but a Classification Algorithm”.

You might have seen this in latest popular machine learning books, blogs or you might have heard *Data Science Gurus* utter the same in their highly subscribed YouTube channels.

Machine learning has usurped and renamed many statistical techniques. Often to the extent that they now disbelieve and reject its statistical origins. Case in point is “Logistic regression is not Regression”

However, nothing can be further from the truth than this assertion. We live in a meme culture, memes even become cryptocurrency.

So why not use a meme to drive home the point that Logistic Regression is indeed Regression.

Logistic Regression is indeed Regression.

This meme would have made things clearer for most of you. However, let me unpack it for more clarity.

Logistic Regression models a continuous outcome (the probabilities). The probabilities range between 0 and 1.

Binary Logistic Regression is used as a Classification algorithm when we want the response variable to be dichotomous (Churn/Not Churned, Pass/Fail, Spam/No spam etc.)

Usually, we make Logistic Regression into a classification algorithm by setting an appropriate probability cut-off or threshold (0.4, 0.5, 0.6 etc.).

The problem of classifying using a threshold value

Fixing the probability threshold is purely a business call and not a statistical one. The threshold could vary from one domain/industry to another.

Frank Harrell in his blog¹ aptly makes the point “classification is a forced choice”.

Classification is a forced choice.

Now consider this example, you choose a threshold value of 0.5. Now the ML algorithm outputs the probability of default or no default (1- default, 0 — no default) for 4 customers as 0.51, 0.49, 0.23 and 0.92. Based on the threshold, 2 are classified as ‘default’ and 2 are classified as ‘no default’. However, ask yourself this, isn’t it too close to call for the customers with probability 0.51 and 0.49? 0.51 is surely closer to 0.49 (which is classified as no default) than it is to 0.92 (which is classified as default).

Some popular machine learning packages and low code tools do not depict the predicted probabilities to the user explicitly. The user is hence oblivious to what predicted probabilities are gotten. He /she simply gets the decision — default or no default (1 or 0). In the case of 0.49 and 0.51, the user happily made the decision that the person will not default and will default respectively. But a peek inside the predicted probabilities reveals that it was too close to call!

The other problem with thresholds is that, when we use an improper scoring rule like classification accuracy, it can be easily gamed. For e.g., if out of 100 people, 95 of them default on loan, and 5 don’t. If the classifier classifies everyone to default on loan, then it would have an accuracy of 95% !!

So, is there a better way to use Logistic Regression?

The answer is Yes.

Industries such as Finance and Marketing use Logistic Regression in a more appropriate way for Credit risk modelling and marketing campaign targeting respectively.

A real use case

Let us say, you are a CMO in charge for sales and marketing of a product in your organization. You plan to launch a marketing campaign to increase sales of that product. You have been given a fixed budget for this. Now you would like to get the best ROI possible i.e., spend the exact fixed budget or even lesser and get the maximum sales possible. Here is what you have.

You have the data for 10,000 customers who had either purchased or not purchased a similar product in the past.

You would like to understand which customers should be targeted to increase the likelihood of purchase this time.

You would obviously like to target those people who are more likely to buy the product since you have a fixed budget for the campaign. How to go about it?

Decile Analysis and Gain Chart come to rescue here.

What is a Decile Analysis?

Decile analysis was once a popularly used technique, however the convention of teaching and bucketing machine learning problems into either ‘classification’ or ‘Regression’ types, lead people to forget Decile analysis type analyses. I am pretty sure, most freshly minted data scientists would not have even heard of Decile analysis. So, coming back to what is Decile Analysis.

Decile Analysis is used to categorize dataset from highest to lowest values or vice versa. (Based on predicted probabilities)

As obvious from the name, the analysis involves dividing the dataset into ten equal groups. Each group should have the same no. of observations/customers.

It ranks customers in the order from most likely to respond to least likely to respond.

How is Decile Analysis done?

Step 1: Build a Logistic Regression model. In this case the dependent variable was ‘probability of buying the product’. With 1 denoting purchased, 0 denoting not purchased. Also, the relevant independent variables were selected.

Step 2: Get the predicted probabilities from Logistic Regression algorithm. Arrange the probabilities in descending order.

Step 3: Divide the whole dataset into 10 groups, each group should contain equal no. of observations. So, if there 10,000 records, each group would have 1000 records/customers.

Step 4: Compute % of responders for each decile

Step 5: Compute response rate for each decile

Step 6: Compute lift for each decile

The top decile would have customers who are most likely to respond followed by the decile 2 which would have customers who are next most likely to respond and so on.

One major advantage of the decile analysis is that the probabilities and probability range are their own error measures. That is, if the top decile has the probability range 0.75–0.81, then then the probability of this person not buying the product i.e., the error would be (1- [0.75–0.81], here ‘[]’ denotes the interval of values between 0.75–0.81 including 0.75 and 0.81)

What does a Decile Analysis output look like?

The below table depicts a typical decile analysis output.

Let’s unpack the output table.

As stated earlier, each decile has equal number of customers (1000 in each decile).

% of responders for each decile = No. of responders in that decile/ No. of total responders for all 10 deciles

Refer to Table 1:

Here, % of responders for Decile 1 = 224/984 = 22.8%

984 is the total no. of responders in all 10 deciles

Similarly, % of responders for Decile 2 = 16.5% (162/984)

And cumulative % of responders for top 2 deciles = 39.2%

Gains and Gain Chart:

From table 1, Decile 1 contains top 10% of the customers who are most likely to buy. Decile 1 has the highest no. of responders (224 responders).

So out of the total responders in all deciles, 22.8% of the responders are in Decile 1. So, for 10% of the customer base, there are 22.8% responses in Decile 1.

Similarly for 20% of the customer base in decile 1 and decile 2, there are 39.2% responses.

Gain Chart below explains this better.

Takeaway from Gain Chart: Gain Chart can be used to assess what % of customers respond in each decile. So, rather than targeting customers from lower deciles, customers can be picked up from the top deciles only.

The dashed line is the Baseline. Baseline tells how much % of customers would respond if we target customers randomly without a model.

Response Rate

Response Rate tells what is the percentage of customers who responded in each decile. The response % is highest in Decile 1 followed by Decile 2 and so on.

Response Rate for each decile = No. of respondent in that decile/ No. of customers in that decile

Refer to Table 1:

Here, Response Rate for Decile 1 = 224/ 1000 = 22.4%

Takeaway from Response Rates comparison: Response Rate of each decile is depicted in the chart below. The average response rate of all deciles is 9.8%. So, customers from Decile 1 to Decile 4 lie above the average response rate and should be targeted for the campaign.

Lift and Lift Curve

Lift = cumulative % of responders / customers % at each decile

Refer to Table 1:

Lift for Decile 1 = 22.8%/10% = 2.28

Lift for Decile 2 = 39.2%/20% = 1.96

How to interpret: If we target top two deciles, then we would target 20% of the customers. In the same deciles, the cumulative % of responders are 39.2%. Hence, there is a lift of 1.96.

A lift of 1 means there is no gain compared with the no. of customers targeted at random. Lift greater than 1 means, model approach is better than selecting the customers at random.

Takeaway from Lift Chart: Can be used to identify the deciles which have a higher Lift.

How to use Decile Analysis in business decision making:

Now that we have done building Decile Analysis, the next pertinent question is how we use it to make effective business decisions:

Let us revisit our Decile Analysis Table

Going by the above results, we make a decision that we will target customers in the Top 4 deciles as they have higher probability of buying the product.

From the business point of view, the ROI of targeting the Top 4 deciles is more. As we move down the decile, the ROI rapidly decreases and is not worthwhile pursuing these customers.

So, that wraps up our decile analysis article. Your comments are welcome.

“This article was contributed by Aryma Labs. Co-authored by Venkat Raman and Ridhima Kumar.

Aryma Labs is a Data Science Firm specializing in Marketing Mix Modeling, Marketing/Digital analytics, Machine learning and NLP.

We cater to clients across FMCG, E-Commerce, Banking, Manufacturing, Media and Advertising industries.”

References:

1. https://www.fharrell.com/post/classification/

The Lost Art of Decile Analysis was originally published in Analytics Vidhya on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Logistic Regression is not Regression but a Classification Algorithm”.

You might have seen this in latest popular machine learning books, blogs or you might have heard *Data Science Gurus* utter the same in their highly subscribed YouTube channels.

Machine learning has usurped and renamed many statistical techniques. Often to the extent that they now disbelieve and reject its statistical origins. Case in point is “Logistic regression is not Regression”

However, nothing can be further from the truth than this assertion. We live in a meme culture, memes even become cryptocurrency.

So why not use a meme to drive home the point that Logistic Regression is indeed Regression.

Logistic Regression is indeed Regression.

This meme would have made things clearer for most of you. However, let me unpack it for more clarity.

Logistic Regression models a continuous outcome (the probabilities). The probabilities range between 0 and 1.

Binary Logistic Regression is used as a Classification algorithm when we want the response variable to be dichotomous (Churn/Not Churned, Pass/Fail, Spam/No spam etc.)

Usually, we make Logistic Regression into a classification algorithm by setting an appropriate probability cut-off or threshold (0.4, 0.5, 0.6 etc.).

The problem of classifying using a threshold value

Fixing the probability threshold is purely a business call and not a statistical one. The threshold could vary from one domain/industry to another.

Frank Harrell in his blog¹ aptly makes the point “classification is a forced choice”.

Classification is a forced choice.

Now consider this example, you choose a threshold value of 0.5. Now the ML algorithm outputs the probability of default or no default (1- default, 0 — no default) for 4 customers as 0.51, 0.49, 0.23 and 0.92. Based on the threshold, 2 are classified as ‘default’ and 2 are classified as ‘no default’. However, ask yourself this, isn’t it too close to call for the customers with probability 0.51 and 0.49? 0.51 is surely closer to 0.49 (which is classified as no default) than it is to 0.92 (which is classified as default).

Some popular machine learning packages and low code tools do not depict the predicted probabilities to the user explicitly. The user is hence oblivious to what predicted probabilities are gotten. He /she simply gets the decision — default or no default (1 or 0). In the case of 0.49 and 0.51, the user happily made the decision that the person will not default and will default respectively. But a peek inside the predicted probabilities reveals that it was too close to call!

The other problem with thresholds is that, when we use an improper scoring rule like classification accuracy, it can be easily gamed. For e.g., if out of 100 people, 95 of them default on loan, and 5 don’t. If the classifier classifies everyone to default on loan, then it would have an accuracy of 95% !!

So, is there a better way to use Logistic Regression?

The answer is Yes.

Industries such as Finance and Marketing use Logistic Regression in a more appropriate way for Credit risk modelling and marketing campaign targeting respectively.

A real use case

Let us say, you are a CMO in charge for sales and marketing of a product in your organization. You plan to launch a marketing campaign to increase sales of that product. You have been given a fixed budget for this. Now you would like to get the best ROI possible i.e., spend the exact fixed budget or even lesser and get the maximum sales possible. Here is what you have.

You have the data for 10,000 customers who had either purchased or not purchased a similar product in the past.

You would like to understand which customers should be targeted to increase the likelihood of purchase this time.

You would obviously like to target those people who are more likely to buy the product since you have a fixed budget for the campaign. How to go about it?

Decile Analysis and Gain Chart come to rescue here.

What is a Decile Analysis?

Decile analysis was once a popularly used technique, however the convention of teaching and bucketing machine learning problems into either ‘classification’ or ‘Regression’ types, lead people to forget Decile analysis type analyses. I am pretty sure, most freshly minted data scientists would not have even heard of Decile analysis. So, coming back to what is Decile Analysis.

Decile Analysis is used to categorize dataset from highest to lowest values or vice versa. (Based on predicted probabilities)

As obvious from the name, the analysis involves dividing the dataset into ten equal groups. Each group should have the same no. of observations/customers.

It ranks customers in the order from most likely to respond to least likely to respond.

How is Decile Analysis done?

Step 1: Build a Logistic Regression model. In this case the dependent variable was ‘probability of buying the product’. With 1 denoting purchased, 0 denoting not purchased. Also, the relevant independent variables were selected.

Step 2: Get the predicted probabilities from Logistic Regression algorithm. Arrange the probabilities in descending order.

Step 3: Divide the whole dataset into 10 groups, each group should contain equal no. of observations. So, if there 10,000 records, each group would have 1000 records/customers.

Step 4: Compute % of responders for each decile

Step 5: Compute response rate for each decile

Step 6: Compute lift for each decile

The top decile would have customers who are most likely to respond followed by the decile 2 which would have customers who are next most likely to respond and so on.

One major advantage of the decile analysis is that the probabilities and probability range are their own error measures. That is, if the top decile has the probability range 0.75–0.81, then then the probability of this person not buying the product i.e., the error would be (1- [0.75–0.81], here ‘[]’ denotes the interval of values between 0.75–0.81 including 0.75 and 0.81)

What does a Decile Analysis output look like?

The below table depicts a typical decile analysis output.

Let’s unpack the output table.

As stated earlier, each decile has equal number of customers (1000 in each decile).

% of responders for each decile = No. of responders in that decile/ No. of total responders for all 10 deciles

Refer to Table 1:

Here, % of responders for Decile 1 = 224/984 = 22.8%

984 is the total no. of responders in all 10 deciles

Similarly, % of responders for Decile 2 = 16.5% (162/984)

And cumulative % of responders for top 2 deciles = 39.2%

Gains and Gain Chart:

From table 1, Decile 1 contains top 10% of the customers who are most likely to buy. Decile 1 has the highest no. of responders (224 responders).

So out of the total responders in all deciles, 22.8% of the responders are in Decile 1. So, for 10% of the customer base, there are 22.8% responses in Decile 1.

Similarly for 20% of the customer base in decile 1 and decile 2, there are 39.2% responses.

Gain Chart below explains this better.

Takeaway from Gain Chart: Gain Chart can be used to assess what % of customers respond in each decile. So, rather than targeting customers from lower deciles, customers can be picked up from the top deciles only.

The dashed line is the Baseline. Baseline tells how much % of customers would respond if we target customers randomly without a model.

Response Rate

Response Rate tells what is the percentage of customers who responded in each decile. The response % is highest in Decile 1 followed by Decile 2 and so on.

Response Rate for each decile = No. of respondent in that decile/ No. of customers in that decile

Refer to Table 1:

Here, Response Rate for Decile 1 = 224/ 1000 = 22.4%

Takeaway from Response Rates comparison: Response Rate of each decile is depicted in the chart below. The average response rate of all deciles is 9.8%. So, customers from Decile 1 to Decile 4 lie above the average response rate and should be targeted for the campaign.

Lift and Lift Curve

Lift = cumulative % of responders / customers % at each decile

Refer to Table 1:

Lift for Decile 1 = 22.8%/10% = 2.28

Lift for Decile 2 = 39.2%/20% = 1.96

How to interpret: If we target top two deciles, then we would target 20% of the customers. In the same deciles, the cumulative % of responders are 39.2%. Hence, there is a lift of 1.96.

A lift of 1 means there is no gain compared with the no. of customers targeted at random. Lift greater than 1 means, model approach is better than selecting the customers at random.

Takeaway from Lift Chart: Can be used to identify the deciles which have a higher Lift.

How to use Decile Analysis in business decision making:

Now that we have done building Decile Analysis, the next pertinent question is how we use it to make effective business decisions:

Let us revisit our Decile Analysis Table

Going by the above results, we make a decision that we will target customers in the Top 4 deciles as they have higher probability of buying the product.

From the business point of view, the ROI of targeting the Top 4 deciles is more. As we move down the decile, the ROI rapidly decreases and is not worthwhile pursuing these customers.

So, that wraps up our decile analysis article. Your comments are welcome.

For innovative Marketing solutions through Data Science, get in touch with Aryma Labs — Your one stop solution for Marketing Science

Data Science Consulting

References:

1. https://www.fharrell.com/post/classification/

© Copyright 2021 www.arymalabs.com All Rights Reserved.

The Lost Art of Decile Analysis was originally published in Aryma Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

If the words ‘Dynamic Time Warping’ evokes a sense of time travel science fiction, you would not be at fault.

So let us first unpack this science fiction terminology.

Dynamic Time Warping

Dynamic Time Warping (DTW) is a time series analysis technique used for measuring similarity between two temporal sequence. These sequences or time series could be of different length as well.

It calculates the optimal match between two series using a set of rules. Suppose there are two series — A and B. The DTW algorithm minimizes the distance between them by creating a warping path W (now you know where the word ‘warping’ comes from in DTW).

How is it different from other distance metrics like Euclidean Distance?

Euclidean distance is useful when both the time series are in sync and will compare the value of series 1 at time t with the value of series 2 at time t. Basically, it does one-to-one match and does not take into account time shifts.

DTW on the other hand applies one-to-many or many-to-one matches in such a way that the overall distance between the two series is minimized. It allows two similar shaped series of different lengths to match.

DTW has been applied in diverse fields such as speech pattern recognition and signal processing. However, the application of DTW in the field of Marketing has been rare.

In this post we will showcase a novel application of DTW in the field of Marketing.

Business Problem:

• The client ran marketing campaign in one market for a period of one year (August 2019 — July 2020) and noticed that the campaign was highly successful in generating incremental leads. This market was termed as ‘Ideal Market’.

• The client planned on implementing the campaign in other new markets (identified as candidate markets) and wanted to identify which of these candidate markets were like the ‘ideal market’.

• The issue was, they did not want to run the campaigns in all the ‘candidate markets’ for a period of 1 year owing to cost concerns.

• The ask was to identify early signs whether a ‘candidate market’ is like the ‘ideal market’ in terms of campaign response instead of having to wait till the year end.

Approach:

Dynamic Time Warping algorithm was used to understand which candidate markets were similar in behavior to the ‘Ideal Market’.

• The client had identified one market as an ideal market where the campaign was successful (Campaign was run from Aug’19 to Jul’20 and leads were recorded for this period).
• 7 candidate markets were identified by the client to run the similar campaign.
• For these candidate markets, the leads and campaign data were available for a period Aug’20 — Dec’20 (5 months).
• The leads from these candidate markets (Aug’20 — Dec’20) were compared with the leads from the ideal market (Aug’19 — Dec’19) using DTW algorithm.
• The markets with lowest distance w.r.t ideal market were selected to continue the campaign for further period.

Our Recommendation:

• It was observed that out of the ‘7 Candidate Markets’, there were 4 markets which were most similar to the ‘Ideal Market’ (highlighted in the image below).
• We suggested the client to continue the campaign in these 4 candidate markets.

Result Evaluation & Impact

The client was recommended to continue campaigns in these 4 most similar markets instead of running campaign in all ‘Candidate Markets’.

In the post implementation evaluation conducted in April 2021, it was found that out of the 4 markets suggested by us:

• Candidate market A, E and F got 5%, 9% and 6% incremental leads, respectively.
• Candidate market D got 2% incremental leads.

Overall, this solution helped the client in:

• Correctly identifying similar markets out of a cohort of markets thereby cutting costs.
• The identified markets netted higher number of leads.

References

1. http://www.mathcs.emory.edu/~lxiong/cs730_s13/share/slides/searching_sigkdd2012_DTW.pdf
2. https://en.wikipedia.org/wiki/Dynamic_time_warping

For innovative marketing science solutions, you can reach out to us at:

Website: https://www.arymalabs.com/

LinkedIn: Ridhima Kumar

If the words ‘Dynamic Time Warping’ evokes a sense of time travel science fiction, you would not be at fault.

So let us first unpack this science fiction terminology.

Dynamic Time Warping

Dynamic Time Warping (DTW) is a time series analysis technique used for measuring similarity between two temporal sequence. These sequences or time series could be of different length as well.

It calculates the optimal match between two series using a set of rules. Suppose there are two series — A and B. The DTW algorithm minimizes the distance between them by creating a warping path W (now you know where the word ‘warping’ come from in DTW).

How is it different from other distance metrics like Euclidean Distance?

Euclidean distance is useful when both the time series are in sync and will compare the value of series 1 at time t with the value of series 2 at time t. Basically, it does one-to-one match and does not take into account time shifts.

DTW on the other hand applies one-to-many or many-to-one matches in such a way that the overall distance between the two series is minimized. It allows two similar shaped series of different lengths to match.

DTW has been applied in diverse fields such as speech pattern recognition and signal processing. However, the application of DTW in the field of Marketing has been rare.

In this post we will showcase a novel application of DTW in the field of Marketing.

Business Problem:

• The client ran marketing campaign in one market for a period of one year (August 2019 — July 2020) and noticed that the campaign was highly successful in generating incremental leads. This market was termed as ‘Ideal Market’.

• The client planned on implementing the campaign in other new markets (identified as candidate markets) and wanted to identify which of these candidate markets were like the ‘ideal market’.

• The issue was, they did not want to run the campaigns in all the ‘candidate markets’ for a period of 1 year owing to cost concerns.

• The ask was to identify early signs whether a ‘candidate market’ is like the ‘ideal market’ in terms of campaign response instead of having to wait till the year end.

Approach:

Dynamic Time Warping algorithm was used to understand which candidate markets were similar in behaviour to the ‘Ideal Market’.

• The client had identified one market as an ideal market where the campaign was successful (Campaign was run from Aug’19 to Jul’20 and leads were recorded for this period).
• 7 candidate markets were identified by the client to run the similar campaign.
• For these candidate markets, the leads and campaign data were available for a period Aug’20 — Dec’20 (5 months).
• The leads from these candidate markets (Aug’20 — Dec’20) were compared with the leads from the ideal market (Aug’19 — Dec’19) using DTW algorithm.
• The markets with lowest distance w.r.t ideal market were selected to continue the campaign for further period.

Our Recommendation:

• It was observed that out of the ‘7 Candidate Markets’, there were 4 markets which were most similar to the ‘Ideal Market’ (highlighted in the image below).
• We suggested the client to continue the campaign in these 4 candidate markets.

Result Evaluation & Impact

The client was recommended to continue campaigns in these 4 most similar markets instead of running campaign in all ‘Candidate Markets’.

In the post implementation evaluation conducted in April 2021, it was found that out of the 4 markets suggested by us:

• Candidate market A, E and F got 5%, 9% and 6% incremental leads, respectively.
• Candidate market D got 2% incremental leads.

Overall, this solution helped the client in:

• Correctly identifying similar markets out of a cohort of markets thereby cutting costs.
• The identified markets netted higher number of leads.

For innovative Marketing solutions through Data Science, get in touch with Aryma Labs — Your one stop solution for Marketing Science

https://www.arymalabs.com/

References

1. http://www.mathcs.emory.edu/~lxiong/cs730_s13/share/slides/searching_sigkdd2012_DTW.pdf
2. https://en.wikipedia.org/wiki/Dynamic_time_warping

© Copyright 2021 www.arymalabs.com All Rights Reserved.

Devising better Marketing Campaign Strategy through Dynamic Time Warping was originally published in Aryma Labs on Medium, where people are continuing the conversation by highlighting and responding to this story.

Or which forecast accuracy metrics to use?

Many CPG brands across the world would be focusing on keeping a tab on their sales and demand numbers during the Covid-19 pandemic. In my previous article, I had covered points on doing Marketing Mix modeling during these testing times.

The brands might have already forecasted sales or demands for the first 3–4 months of 2020. But what could have been missed from their forecasts is the effect of Covid-19.

With the increase in sales being reported by many brands, the forecasted sales would be way off the mark and there would be a huge divergence in the forecasted sales from actuals.

The situation calls for checking the forecast accuracy of the models.

In this article, I am going to discuss different forecast accuracy metrics and how to assess which of these forecast accuracy metrics would be the ideal one to choose.

Let’s start with a brief overview of these metrics:

1. MAD/MAE: This is calculated by subtracting the actual sales from forecasted sales. The error term is then converted to absolute error and the average of absolute errors is calculated for the whole time series.

This forecast measure depends on the scale and the error term also has the same unit as the data. For e.g. if the sales volume is in Kg, then the error term would also be in Kg.

This method cannot be used to make comparison between time series which have different units e.g. sales volume (Kg) error cannot be compared with sales value (USD).

The drawback of this accuracy measure is that it averages out the error across the whole series which doesn’t give a very accurate result.

The error of 500 units might look very large when the sales are 2000 but the same error would be minuscule if the actual sales were 20,000.

2. RMSE: Root mean square error is the square root of the average squared error.

Like MAE, this measure is scale dependent and cannot be used to make comparisons between series on different scales. And the error term also has the same unit as the data.

3. MAPE: Mean Absolute Percentage Error is the most widely used measure for checking forecast accuracy. It comes under percentage errors which are scale independent and can be used for comparing series on different scales.

where eᵢ is the error term and yᵢ is the actual data at time i.

The disadvantage of MAPE is that it becomes undefined if the value for actual figure is 0 for any observation in the data.

4. MASE: Mean Absolute Scaled Error is a scale free error.

Basically, MASE is nothing but a ratio of MAE on test data divided by MAE using one-step naïve forecasting method on the training set.

where,

Here, Q is the naïve forecast computed on the training data. MAE is calculated on the test data and Q is calculated on the training data.

If MASE is less than 1, it means that the forecast is better than the one step naïve method on training data. If it is more than 1, it means that the forecast method is worse than the forecast using one step naïve forecasting approach on the training data.

To better understand this, below are the few links explaining the formula correctly:

Interpretation of mean absolute scaled error (MASE)

https://robjhyndman.com/papers/foresight.pdf

Now that we have the introduction to error metrics, let us see where each of these forecasting metrics work better or worse.

Scenario 1: Forecast varying drastically from actuals

Before Covid-19 is considered as training data and post covid-19 (March-20 to May-2020) is considered as test data.

In this scenario, MAPE can be used as a good accuracy measure since it is scale independent and can be used to compare different series or forecast scenarios.

Scenario 2: Actuals plummeting to zero

Doesn’t the below graph remind of you something recent in the news ?…. If you guessed ‘Crude oil’ you would be correct !!

In this scenario, unfortunately the actuals plummet to zero (highly unexpected scenario) This scenario has been true for industries like Crude oil where the West Texas Intermediate (WTI) oil price benchmark plummeted to negative, and also for automobile industry which has recently recorded zero sales for these months.

MAPE is not a good measure for this scenario as it becomes undefined.

MAE or RMSE could be used for comparing forecast accuracy here. MASE also does not seem like a good KPI here as it is greater than 1. When MASE is greater than 1, it is implied that the method used for forecasting is worse than the naïve method used.

Scenario 3: The ideal scenario (Back to normal scenario)

In this scenario, there is no covid-19 interference (oh how we wish for those normal days). The models are behaving properly. In this case, any measure can be used.

References:

1. https://robjhyndman.com/papers/forecast-accuracy.pdf
2. https://robjhyndman.com/papers/foresight.pdf
3. https://qz.com/1841668/oil-prices-are-negative-for-first-time-ever/

You can get in touch with me:

LinkedIn: Ridhima Kumar

Twitter: @kumar_ridhima

MAD Over MAPE? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.