In one of my earlier articles, we talked about the basics of the statistical significance and I got some really promising feedback which brought me back here to try and cover a few more concepts.

Let’s get started with t-tests first. A t-test is done when we do not know the population mean and standard deviation to find out how the mean of the two groups in question differ from one another. The basic assumption behind this test is that it assumes that the samples are coming from a population which has a normal distribution. I will try to explain this better with an example:

There are two different restaurants that you like. One of them has an average rating of 3.5 and the other one has an average rating of 3.7. You need to take your date to one of these restaurants. Should you just pick the one with the higher rating? I suppose you should since its a date but what if the first one has a per-person cost of $30 while the second one costs $60. Is the difference in the rating statistically significant enough for you to double the burden on your wallet? The only way to find out is to run a t-test on the rating set of restaurant A v/s B. You pick the dataset for all the ratings of both the places and run them through a t-test. This way you answer the question whether the difference in ratings is significant enough (or simply a coincidence)? Of course, I don’t recommend sharing with your date that you ran a t-test to choose the place but I hope I managed to show you the use case for this test.

The next part is how to run a t-test. I will post a simple code snippet here on python indicating the same and interpreting the results for you. For the sake of this same exercise, I will pick The Rocks Cafe & Dare Cafe as the two cafes in question. The first thing to do is to test the hypothesis of normality. Here I refer to the statement that the samples are coming from a population which has a normal distribution. Crazy hypothesis, right! How do I check for the population of the rating to find if the ratings are normally distributed! Common sense tells me that it should be normally distributed but is there a way to check this? Probably there is… but it’s not something straightforward. So while I am going to assume here that they are normal and proceed forward, I am also going to show what to do in cases where you are not going to simply say ‘common-sense’ and how the hypothesis results differ in that case.

#Getting the Dataset ready!
import itertools
import numpy as np
ratings = [1,2,3,4,5]
ratings_rocks=[27,35,111,402,526] #number of 1,2,3,4 and 5 rating for The Rocks
ratings_dare=[9,4,6,51,156] #number of 1,2,3,4 and 5 rating for Dare
ratings_rocks_list=list(itertools.chain.from_iterable(itertools.repeat(x, y) for x,y in zip(ratings,ratings_rocks)))
ratings_dare_list=list(itertools.chain.from_iterable(itertools.repeat(x, y) for x,y in zip(ratings,ratings_dare)))
print ("Rating for Rocks is: "+str(np.mean(ratings_rocks_list)))
# Rating for Rocks is: 4.239782016348774
print ("Rating for Dare is: "+str(np.mean(ratings_dare_list)))
# Rating for Dare is: 4.508849557522124

Now that we have the dataset ready, let’s create the mean and the std dev for both followed by the standard errors and the degree of freedom to get… Naah… none of that… all u need to do is run the ttest_ind from the scipy.stats library to get the following

from scipy.stats import ttest_ind
stat, p = ttest_ind(ratings_rocks_list, ratings_dare_list)
print('t=%.3f, p=%.4f' % (stat, p))
#t=-3.943, p=0.0001

If you happen to interchange the two lists, all that will change is the sign of the t-value. Don’t be too bothered about the t-value as it is linked to the p-value and if you can read the p-value corrected then it's not of much import. So what the results here indicate is that even though the two cafes have a rating difference of 0.25, this is a pretty substantial difference and merits a change of venue for your date.

However, I think it's only to fair to highlight that student’s t-test (the one we just examined above) was the best practise quite when we were very restricted with the processing power available at our disposal. Now that we have access to much better computing power, its a much better choice to go with something like Wilcoxon test which does not form assumptions like the one that we had to make above like normal distribution for the population. Let’s try the same thing with the Wilcoxon t-test (2 sample Wilcoxon test is also called the Mann Whitney U test) to find what kind of results we get.

from scipy.stats import mannwhitneyu
stat, p = mannwhitneyu(ratings_dare_list, ratings_rocks_list)
print('t=%.3f, p=%.4f' % (stat, p))
#t=98019.500, p=0.0000

From this, we see that the results indicate the same thing though the p-value is different in both the scenarios due to the underlying methodology and assumptions behind both the tests. If you are ever comparing two different samples, I will recommend using this method rather than the student’s t-test due to the assumption of normality in the student t-test method.

Let's move on to a different scenario. Instead of 2 different samples/cafes, what if I had 3 different cafes that I want to compare. One way is to do a Wilcoxon t-test for A vs B, A vs C and B vs C. Simple enough! But what if we had 4 cafes or even 10? The combinations will just keep on increasing. To deal with such a scenario, we do something known as 1 Way Anova test. The one-way analysis of variance (ANOVA) is used to determine whether there are any statistically significant differences between the means of three or more independent (unrelated) groups/samples.

It is important to note here that one-way ANOVA CAN be used as a substitute for t-test for multiple groups but mathematically speaking, it is more of a regression model which is used to establish causality.

For instance, if we go to my article about segmentation, once I had a cluster from the hierarchical clustering method, I ran ANOVA on these clusters to determine whether these clusters are actually different given these difference in all the features.

The best way to execute ANOVA is by using the below set of code

from statsmodels.formula.api import ols
formula='y~C(X1)+C(X2)'
results = ols(formula, data=df).fit()
results.summary()

The things to note in the formula above is that it borrows from R and that why you see the format to be a bit non-pythonic. As a summary from the code above, you will get a table like this

Let's try to interpret the table above. The first thing to note, as you might have guessed, is the overall Prob (F-statistic) which indicates the p-value of the result. This indicates if, on an overall level, the groups are different. If it is, then you move on to check the P>|t| for each of the independent variables which indicate which of these variables are actually different in the model. In the example above, we can see that only x2 is significantly different due to this value being less than 0.05. Of course, there are other important parameters that we should be looking at here- most important of them being adjusted R2 which explains how much of variabilities in y are explained by variabilities in x- indicating the robustness or completeness of the model- but that’s a discussion for another day. For the time being, let’s just say that just looking at these parameters should be good enough for our purpose.

It should be noted that often times multicollinearity can throw off the results of individual t-tests. This is an especially commonplace scenario that can often be interpreted by reading this table correctly. A symptom of multicollinearity is when none of the individual coefficients is significant but the overall p-value is significant.

Let's suppose the Anova gives us that the groups are significantly different from each other. The next question then comes- so what? To answer this question, we use the post-hoc test.

Post-hoc literally means ‘after-this’ and it should only be applied once you have established that the Anova is significant at the required level. The post-hoc test establishes where the difference in the two groups actually lies. Let’s go back to the movie example I had. Once I established that the cluster difference was significant on an overall level, I went ahead with applying the Tukey test, which is one type of post-hoc test. Using the code below:

import statsmodels.stats.multicomp as multi
for col in city_final.columns:
    mc1=multi.MultiComparison(city_final[col],city_final['Cluster'])
    res1=mc1.tukeyhsd(alpha=0.05)
    print col
    res1.plot_simultaneous()
    plt.show()
    print res1.summary()

With the cole snippet above, I went about creating a family-wise check for all categorical columns to find out how the clusters were different from each other. An example of one of the plots that were created as a result of the code above is

The above plots indicate that for this particular feature, where 0 and 2 clusters are quite similar to each other, cluster 1 has a significantly higher value(p<0.05 level since that is what we are using in the code above). We took note of all the features which were dominating for each of clusters and voila, we have the definitions of each of these 3 clusters.

I hope that with these examples and code snippets, I have made things easier to understand and that you are not (anymore!) confused by the xkcd post below :)

Statistically Significant Chronicle was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

The why what where and how of various statistical tests explained in simple terms

I would like to start with this news article on Huffington Post almost 4 years back which might have been a piece of news that almost all chocolate lovers like me would have been jubilant over!

Every time I hear someone say ‘statistically significant’, my interest is suddenly piqued and I feel “Well! Here is a well-documented experiment. Well Done”. But then, I resign my fate to accept the results at face value since more likely than not, the results usually end up missing most of the parameters that are vital to establishing the validity of the test. Keeping this in mind, I thought it might be a good idea to start by highlighting what are the things you should pay attention to in a test and why are they important.

Let’s start with a simple question: “Does having LeBron James in a basketball team make any difference in the possibility of the team winning?” Now, I don’t consider myself a basketball fan so I am going to come up with some hypothetical numbers here so please don’t hold it against me.

How we are likely to go about proving this point is, we first assume that having LeBron James in a team makes no difference in the winning record of a team (I know it’s a ridiculous idea but bear with me here). Now we see that when LeBron James (let’s call him LJ, shall we) was in the team, they won 6 games out of 10 and when he was not in the team they won 5 games out of 10. To keep things simple, we will assume that the 10 games in both the scenarios were against the same team and the rest of the players were the same, with the same substitute for LJ in all of the second set of 10 games. Now, what you will be asking here is: we have just one more game that we won with LJ on the team, is that simply a fluke or does that mean he is a game-changer? Will having 7 wins in 10 convince you that it’s not a fluke, or perhaps it’s 8 out of 10 wins?

The takeaway here is simple: when we mention statistical significance, what is also really important to mention is something we call the “effect size”! How that changes the conversation is something like this: With LJ in the team, the likelihood to win the game is 10% more than without him (with a p-value of less than 0.05- but let’s worry about this last part later).

I will reiterate the importance of the effect size with another commonplace example. We must have come across statements like- “Studies of the benefits of massage demonstrate that it increases the lifespan”. Now the question we should be asking here is how much increase are we talking about? Is it just 5 mins added to your life or are we talking about days? In both cases, the study has a right to say that the lifespan is increased but in the former case, I would probably not pay 100 dollars just to get 5 minutes added to my life :)

Though it’s a very simplistic example, the idea here was to showcase the importance of asking the right questions and how much the effect size can make or break the importance of a study.

Now, let’s move to the second important parameter that we should be paying attention to. This one is the sample size of the experiment. Let me highlight the reason for this one. Let’s toss two coins 5 times just to get 4 heads with the first coin v/s 2 heads with the second one. It’s not an unreasonable scenario. But what really takes the cake here is that the statistical significance test on these results is likely to establish that the first coin is biased towards head!

There is no right answer on what should be the minimum sample size required for a test and it largely depends on what effect size you are trying to measure and the confidence interval you are trying to determine. The small the effect size you are trying to highlight, the more the sample size you will need for establishing the results with the same level of confidence. The higher the confidence interval you are looking for, the more the sample size needs to be.

The last and I believe the most obvious one is the p-value of the test. Most often than not, we go for a p-value significance and simply establish it to be the gospel truth. It’s like the xkcd webcomic strip below:

Now a p-value of less than 0.05 simply means that there is less than a 5% chance that you are likely to see these results IF we assume that the Null hypothesis is correct. In other words, if the p-value is 0.05 then the probability of no false positive is 95%. A false positive means we assume that the test is positive but in reality, that is not the case. A 5% chance of a false positive is indeed low, but it is exactly what is being portrayed in the green jelly case above. If you measure a large number of things for small sample size, you are almost guaranteed to get a “statistically significant” result by chance.

The study with the chocolate was done along similar lines. That study included 18 different measurements — weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc. — from 15 people. That study design is a recipe for false positives. Let's look at it mathematically. If the probability of getting no false positive is 0.95 for one measure when you extrapolate it to 18 measures the probability of having no false positive in all 18 measures becomes 0.95¹⁸=39.7%. So, obviously you will end up with one result that will give you false positive and voila, weight reduction by eating chocolate is born!

That’s why when someone mentions a statistical significance test, you should immediately start looking for these other parameters which are:

1. Effect size
2. Sample size
3. p-value

When it comes down to customer segmentation, one of the most interesting questions I have come across that ties right to the heart of the problem is

“Why do I need customer segmentation. I already have all the data for each individual. Why do I need to tag them into bundles?”

If you are NOT one of those individuals who wrestles with the question above, then probably you can skip the next section. But I think it well worth the effort to make a believer out of everyone.

The group-of-one (individualisation) is definitely better as compared to a group-of-many WHEN we have unlimited resources and time on our hands. I mean, sure, if we know past transactions records for each individual, we can very well customise the experience for each individual (which is what google news does by showing us news based on our individual interests). But that is only feasible when the service or product has a way to reach each of the individuals differently. Let’s consider a billboard business in Sydney CBD area. What sort of audience should we target our advertisements towards for the Above-The-Line (ATL) campaigns? Do we make the tone of the message more serious to target elderly or lighthearted for young? Do we make the message a bit complicated to engage high IQ individuals? A lot of these are my imaginations which need to be substantiated by proper research but I hope you get the direction in which I am going. Even for the companies that do have a channel to reach each individual, it’s difficult to directly jump from one-size-fits-all approach to individualisation. For such companies, it makes more sense to take baby-steps and the first step towards the road to individual personal customisation is group-of-many or as we call it fondly- Segmentation.

Now that everyone is on the same page, let’s move on to describe what a good segment looks like and towards the end we will talk about how it can ideally be used. In my experience, there are 2 very basic characteristics of a good segment.

The first one is that the segments in which the customers (or entities- in case you are trying to segment something else) belong to should not change too much with time. I am sure the first question that comes to your mind here will be- how much is too much? I am afraid I can’t answer that just yet. It depends entirely on a case to case basis and depends a lot on the industry we are talking about. Personally, as a rule of thumb, I try to make sure that in the past one year, not more than 10–15% of the customers should have changed the segment they belong to.

The second important characteristic of the segment is to ensure that the segment is designed based on the use case. Now this might seem like a very general rule (common sense even!) but let me try and explain my rationale behind this. At a few places, I have come across RFM segmentation which segments the customers based on their Recency (how recently they interacted with the platform), Frequency (how frequently they come to the platform) and Monetary (how much money they have spent during a fixed period). This kind of segment works really well if you want to come up with a method to reward the customers - something which does not involve any sort of prediction but rather action based on what they have done in a given time interval. But when this same system is used to predict their behaviour in the next quarter, that’s when it becomes a tricky scenario. For starters, this kind of segmentation does not capture what motivations or preferences these customers have. To give you a flavour of what I am talking about, let’s take 2 customers both of whom have spent $1000 in the last one month on a mobile phone e-commerce platform. One of them is the young technologist who purchased Apple i-Phone and is likely to purchase a phone whenever a new Apple product is launched and then there is the other person who made a purchase because his/her last phone was outdated after years of use. Both of them fall in the same category as far as RFM is considered but while we can expect the first person to make another purchase in an year, the second one is probably not going to return to the platform in the next couple of years. What’s worse is that if we keep on sending regular messages to the second guy by lumping him with the first one we are likely to lose him/her for good. Do you see where I am going with this? In such a scenario, RFM is not the best way forward and instead what we really need is a forward-looking-segmentation approach that defines who those customers are and what their tastes are like based on their purchase decisions.

Now that the theory is out of the way, let’s try and discuss the approach which incorporates all the points illustrated above with an example. In this case, I have 1 year transactions (980K rows) of hashed customer movie ticket purchase data consisting of the following features (after a lot of cleaning):

1. Transaction ID: Unique number for each line item.
2. Customer ID: Unique hashed number for each customer.
3. Number of tickets: How many tickets were purchased by that person.
4. Ticket Type: What type of tickets were purchased for each purchased ticket
5. Cinema Site: Suburb for each of the cinema locations.
6. Date and Time of Show
7. Name of the film

Based on all of these features, I used Pandas DataFrame structure to put data in the following structure

1. Transaction ID was kept as the index
2. Number of tickets (continuous variable) was kept as it is.
3. Ticket type was a very unique column as it was in the form of “Adult;Adult” for 2 Adult tickets. So I had to dummy-fy it in a unique fashion. Based on the unique values, I created column headers like in dummy but the values assigned for each transaction were based on the number of times that value occurred the raw data. For example if the input value was “Adult; Child; Adult” then under Adult column I get 2 and under Child column I get 1. I don’t know if there is a name for this approach or not but I like to call it quantised dummy.
4. Cinema Site, Name of the film and date of the film were columns I used for the join so they were pretty much kept as it is (except cleaning the column which goes without saying).

With this table pretty much ready (took a couple of days to get the cleaning and processing done), I moved on to movies. For movies, I used the OMDB api which provided me a pretty fast way to get the details of each of the movies. The problem I faced here was how to establish the primary key based on the movie name. OMDB data is pretty reliable but as it is rightly said, “Garbage in, garbage out!”. For authenticity of the data, I had to make sure that the movie I am getting the details for is the right one. For this, I put my trust in the IMDB search engine along with a cross check of dates.

To elaborate my process with an example, let’s take the movie The man from Uncle. First I searched for ‘http://www.imdb.com/find?q=The+Man+From+U.N.C.L.E.&s=all’ which gives me a couple of search results. From there, I picked the first result link which gives me ‘https://www.imdb.com/title/tt1638355/?ref_=fn_al_tt_1’. This brings me to the number tt1638355 which is what I was after in the first place. When I search for OMDB with this movie key, I get a json with all the details from where I can take my pick of all the columns that I want to include in my database. To make sure that I am getting the result as required, I am using the validation method of checking the release year with the first date on which that movie was screened in the data. The idea is that the first show for most of the movies is in the same year of their launch. There were a few movies for which I had to manually find the link but more than 99% of the results cleared this sense check.

Once I had these results, I was able to come up with the join between each of the user and the movie database to get the pivot (or groupby) for each of the suburb location based on how many customers went for each of the genres of the movie, how many customers bought each of the different ticket types (Discount, Adult, Child, Group etc), how many users went for movies starring top actors, how many users went for movies by top production houses and so on. With this, I had 17 suburbs (location of the cinemas) with their movie profiles.

Now based on these 17 suburbs (I was only able to get the dataset for these suburbs), let’s try to come up with some kind of recommendations for these locations based on what kind of people watch these movies. A basic assumption behind the next set of analysis is that most of the people who go for movies in these suburbs are those from nearby areas. In other words, I am assuming that most of the people watch movies closer to their homes. With this assumption in mind, I tried multiple algorithms for segmentation. The idea here is to first do the clustering on the suburbs to see if we can come up with different kind of suburbs. Once we have those, we use the same cluster number to do a post-hoc test to test the causality of whether the fact that the suburbs are different causes any difference in the movie behaviour.

Post-hoc tests are tests that believe that because one event follows another, the first must be a cause of the second.

Following this iterative process, I got the best result with a hierarchical clustering approach with 3 clusters. It followed a lot of trial and error for me to come up with these results and since I am already planning to hog at least 15 minutes of your time with this article, I would rather not push my luck by pushing it to 20 mins by taking you through all the iterations!

The way to find decent clusters and measure their performance is to find the silhouette score for each of the different number of clusters and see if we can find a good number for any of the clusters. In very simple terms, silhouette score shows how much the items in the same cluster are similar to each other while being distinct from items in the other cluster and ranges from +1 to -1 with 0.6+ being a decent number, in my experience. We need to find a cluster which is separated enough with respect to customer preference and also can be distinguished well enough with respect to its demographic profile. The idea is to first define boundaries for cluster demographics (profiles) followed by defining testing of whether that difference in profile leads to a significant difference in any of the behaviours.

The most tricky part with respect to customer segmentation is naming the segments.

Another thing that needs to be kept in mind is that the values in each of the suburbs (with respect to movie goers) need to be normalised across rows (except for total number of people who go to movies) to ensure that the values are compared for each of the suburb in a proper context. To illustrate how this can be an issue, lets assume that suburb 1 has population of 500 with 10 people going for Comedy and 490 going to Sci-Fi movies, while suburb 2 has population of 5000 with 100 people going for comedy and 4900 for Sci-Fi. Now if we compare the absolute numbers then obviously people in Suburb 2 are more likely to watch Sci-Fi than Suburb 1 but if we normalise both the suburbs, we realise that the actual value is 0.1 and 0.9 for both the suburbs and so both these suburbs do not have any difference in terms of taste.

From the demographics data, I was able to get the suburb profiles for each of these 17 suburbs with respect to gender, marriage, professions, country of birth, income, professions and qualifications and made clusters on them based on these values. Now with respect to these clusters, we need to do a Post-hoc (Tukey’s range) test on the preferences to find out if the profiles lead to a difference of behaviours. Tukey’s test is similar to pairwise t-test between each pair. What it does in simple terms, is tell us that with respect to the clusters we have defined, is there a significant difference between each of the key behaviour of each of the clusters. The iteration that I mentioned earlier, is between defining the cluster number and naming it based on the post-hoc test (Tukey test).

Based on the results of all these multiple iterations, let me tell you the results of this iterative exercise.

With a very high average income (with a very high representation of those from China, India and New Zealand descent), these suburbs are happy to buy full priced tickets. The average number of post graduate degree levels are much higher in these suburb which is again reflected by higher number of managers and professionals that exist in these suburbs. Most number of Gold/Luxury Class are from these suburbs. The people living here are not big fans of comedy and romance movies. Action and sci-fi movies are quite likely to do well in these areas.

With one of the lowest education level (certificate level) and a lot more kids in the age group of 0–10 years old, these suburbs have the least movie watching rate. These are probably not the suburbs, I would recommend business to open new cinema centres at. People in these suburbs only go for movies which are really famous (either by top production houses or by top actors). Mostly comprised of personal and community service workers and labourers, these suburbs are very partial towards comedy, animation (makes sense due to all those kids) and adult movies.

The last cluster has a very high number of people in sales, administrative and clerical roles. The average salary in these clusters in quite low with a fairly high number of unmarried people. The average qualification in this area is quite moderate with most of the people having a Bachelors background. These clusters are very high discount seekers and they prefer romance, drama and musical genres. Probably, the fact that this cluster has the highest number of people from Australian descent leads to very high number of people being interested in Biographic and history movies. Sci-Fi and other intellectual movies are a no-go. There are a large number of policemen in these areas as well which is reflected in the high number of police discount tickets sold in these areas. The people in these areas are very high movie watchers which can be explained by the high discount driven culture.

To avoid any undue feeling of profiling, I would rather not talk about the name of actual suburbs under each of these clusters. But now that we have these clusters, how do we use these results? Let me come up with some business decisions centred around these clusters:

1. The number of days for which a new movie should be screened and the number of shows per day for each of these suburbs could be planned more efficiently based on the upcoming movie calendar.
2. Location for new cinema sites based on the profile of the suburbs and what ROI could be expected from these suburbs.
3. What kind of movies should be marketed in each of the clusters (suburbs)?

These are just a few of the questions which can be answered based on the research carried out by these data points. If there were specific data points associated with each of the customers, like their home location postcodes, gender of each movie goer, work location etc then it could be used to get much more precise and accurate clusters.

With this I hope you enjoyed reading my approach to segmentation and it will be great to read if anyone has any thoughts or comments on this approach.

Edit: Now, I have been asked to explain the Tukey’s test and how to run and read it in details by those who reviewed this blog. But in the interest of time (and to make sure I have some follow-up article for the next blog), I am going to take that as a topic for the next time. Special thanks to Niketa, Surendra and Josh for their fantastic input and making sure I don’t get carried away in my own flow.