Measurement Problems
’Social Proof’ notion Is very important marketing ideas in recent years. When customers decision buy any product on web sites , select movie or listen music playlist etc. they check any other customers points/stars and comments about that product. In addition ‘The Wisdom of Crowds’ notion , if majority of society bought some think tv,pc,shampoo etc. other people thinking its a good think product .
When purchasing a product on today’s websites, there is no one who does not examine the comments and average scores about the product. These reviews help us learn better about the product and are the best way to ensure a better shopping experience. Therefore, we respect these results on the basis of concepts that affect purchasing processes, such as “Social Proof” and “The Wisdom of Crowds”, the belief in the wisdom of the community.
Let’s say we need to buy a mouse. As seen in the image above, the mouse on the left has 5 stars, while the other mouse has slightly less than 5 stars. When we look only at the number of stars, we prefer the mouse on the left. However, when we look at the majority of the audience who rate the product, the mouse on the right is preferred even though it has a lower star rating due to the large number of raters. This is a good example to understand the wisdom of society and the concept of social proof.
Major Measurement Topics :
-Rating Products
-Sorting Products
-Sorting Reviews
-AB Testing(https://medium.com/@arinmurat/ab-testing-40615f287e42)
Rating Products
Let’s try to determine the correct order of product ratings. Unfortunately i didn’t share datasets because its special, but maybe you try this following links.
Ps: Some results may appear numerically different from my examples. Don’t worry :)
import pandas as pd
import math
import scipy.stats as st
from sklearn.preprocessing import MinMaxScaler
# some visible display settings
pd.set_option('display.max_columns', None)
pd.set_option('display.max_row', None)
pd.set_option('display.width', 500)
pd.set_option('display.expand_frame_repr', False)
pd.set_option('display.float_format', lambda x: '%.5f' % x)
# read dataset
df = pd.read_csv("....../course_reviews.csv")
Analyzed Rating and Questions Asked Distributions
Let’s examine the ‘Rating’ distributions arithmetically. The score given based on the breakdown of questions asked. For example, how many points did users who asked 2 questions give? Here we consult the pandas library.
## Rating Distributions
df["Rating"].value_counts()
## Questions Asked Distributions
df["Questions Asked"].value_counts()
##
df.groupby("Questions Asked").agg({"Questions Asked": "count",
"Rating": "mean"})
df["Rating"].mean()
4.764284061993986
According to analysis , 3267 student vote 5 star this course , 3867 student has no questions and there are 15 people asking 4 questions and their average rating score is 4.83333, the average score given by users who asked 2 questions is 4.80625. All rating average calculate(4.764) but there will not always be an up-to-date rating because trends is fastly changed.
This course may have a good rating over a certain period of time, but as trends change perhaps the ratings will go up or down. If we can catch this somehow, we will have taken a good precaution. In addition hand when you bought a new tv ,three months later thats your new tv is old tv :) So we’ll use “Time-Based Weighted Average” & “User-Based Weighted Average”. Timestamp column mean last comment time — analysis time. How result days is minimum this comments is new for us.
Time-Based Weighted Average
# Analysis Time : (The date we want to analyze)df.info()# Timestamp variable is object. We convert to type of time# 1 Timestamp 4323 non-null objectdf["Timestamp"] =pd.to_datetime(df["Timestamp"])
current_date = pd.to_datetime('2021-02-10 0:0:0')# We need to express all comments made in "days".
df["days"] = (current_date - df["Timestamp"]).dt.days# Result ex:
Rating days
0 5.00000 4
# How can I access the comments made in the last 30 days in this data set?## checked last 30 days rating
df.loc[df["days"] <= 30, "Rating"]
# how much comment in 30 daysdf.loc[df["days"] <= 30, "Rating"].count()
# or 0> df[df["days"] <=30].count()# Average Rating, last 30 days
df.loc[df["days"] <= 30, "Rating"].mean()
# Result : 4.775773195876289#Averge Rating 30 & <= 90 days
df.loc[(df["days"] > 30) & (df["days"] <= 90), "Rating"].mean()
# Result : 4.763833992094861# let's calculate average rating other days
df.loc[(df["days"] > 90) & (df["days"] <= 180), "Rating"].mean()
# Result :4.752503576537912
df.loc[df["days"] > 180 , "Rating"].mean()
# Result :4.76641586867305According to the above results , it seems that this course has increased its scores in recent days.
By giving different weights to these calculations according to time, we can reflect the effect of time in the weight calculation.
## separated some ratio values ,new to old
## time_base_av
df.loc[df["days"] <= 30, "Rating"].mean() * 28/100 + \
df.loc[(df["days"] > 30) & (df["days"] <= 90), "Rating"].mean() * 26/100 + \
df.loc[(df["days"] > 90) & (df["days"] <= 180), "Rating"].mean() * 24/100 + \
df.loc[df["days"] > 180, "Rating"].mean() * 22/100
# 4.765025682267194User-Based Weighted Average
Should everyone’s scores have the same weight ? So in this example, should the score given by the user who watched the entire course be the same as the score given by the user who watched it only once? In another example, should the ratings or comments given by the user who bought the same product from a shopping site once and the user who bought it more than once be the same ?
This is actually an assumption. However, it should be taken into consideration to prevent some deceptions, that is, for social proof.
Let’s take a look at the averages of those who watched the course according to their “Progress” rate.
df.groupby("Progress").agg({"Rating": "mean"})
# Result
# Progress Rating
# 0.00000 4.67391
# 1.00000 4.64269
# ...
# 97.00000 5.00000
# 98.00000 5.00000
# 100.00000 4.86632
In this case, there is a significant difference in point weights between those who almost exclusively previewed and rated the course and those who completed the course.
# user_base_weig_av
df.loc[df["Progress"] <= 10, "Rating"].mean() * 22/100 + \
df.loc[(df["Progress"] > 10) & (df["Progress"] <= 45), "Rating"].mean() * 24/100 + \
df.loc[(df["Progress"] > 45) & (df["Progress"] <= 75), "Rating"].mean() * 26/100 + \
df.loc[df["Progress"] > 75, "Rating"].mean() * 28/100
# Result
# 4.800257704672543Are those who watch the entire course the same as those who do not ? Here, we give a weight according to the rate of users watching the course and find a more reasonable result.
Get Murat Arın’s stories in your inbox
Join Medium for free to get updates from this writer.
Weighted Average
Normally, user and time weights may change with the initiative of the business unit, but here, let’s make a realistic rating calculation by giving equal weights to both. Let’s refine it further by considering viewership as a quality metric.
time_base_av(df) * 50 / 100 + user_base_weig_av(df) * 50 / 100
# 4.782641693469868Sorting Products
When we search for something such as a product, service, or personnel on the website, the products appear in a sorted order. The results are sometimes multi-star, sometimes sponsored, etc. It can happen. In this regard, I will try to explain how to sort the products we are looking for on a site in a meaningful way.
We will focus on how to make a classification based on the data that may occur specifically for the products.
Sorting by Rating
In the example database we will see the ratings on a training course. We can consider these trainings as devices on an electronic shopping site.
Now, let’ s make a simple ranking according to the points given.
https://www.kaggle.com/datasets/muratarin/productsorting
If you want to use this dataset, I can share it.
df.sort_values("rating", ascending=False).head(20)
There were 3 problems with simple search values.
- We saw that there were course topics that were unrelated to each other. Ps: There is nothing we can do analytically about this, we will filter it at the beginning of the run anyway :)
- Although the purchase numbers are very low, it seems to have more points than the more watched course. The number of comments is very low.
- We have ignored the concepts of ‘Comment_Count’ and ‘Purchase Count’.
As a result, filtering only based on ‘Rating’ will not guide us correctly.
Sorting by Rating, Comment Count and Purchase Count
Even if the purchased amount is good in terms of customer satisfaction, I may not want to recommend products with bad reviews. Some courses may have been preferred because they were cheap, but they may not have been liked.
Since the ‘Purchase_Count’ and ‘Comment_Count’ values are high, we will use MinMaxScaler to ensure that they are within certain ranges to simplify it a little more.
from sklearn.preprocessing import MinMaxScaler
# pruchase_count
df["purchase_count_scaled"] = MinMaxScaler(feature_range=(1, 5)).fit(df[["purchase_count"]]).transform(df[["purchase_count"]])
#commment_count
df["comment_count_scaled"] = MinMaxScaler(feature_range=(1, 5)).fit(df[["commment_count"]]).transform(df[["commment_count"]])
Under normal circumstances, companies can give weight according to the levels they consider important. We will make a random distribution according to ourselves.
# Random Distirb.
(df["comment_count_scaled"] * 32 / 100 +
df["purchase_count_scaled"] * 26 / 100 +
df["rating"] * 42 / 100)
# Functional code
def weighted_sorting_score(dataframe, w1=32, w2=26, w3=42, scl = 100):
return (dataframe["comment_count_scaled"] * w1 / scl +
dataframe["purchase_count_scaled"] * w2 / scl +
dataframe["rating"] * w3 / scl)
df["weighted_sorting_score"] = weighted_sorting_score(df)
df.sort_values("weighted_sorting_score", ascending=False).head(20)
You can filter for a specific course with the query below.
df[df["course_name"].str.contains("Veri")].sort_values("weighted_sorting_score", ascending=False).head(20)Bayesian Average Rating Score (BAR Score)
Also called , “Sorting Products with 5 Star Rated” , “Sorting Products According to Distribution of 5 Start Rating” . Here we will make an evaluation based on “Points”. In summary; Using the Bayesian Method, we will calculate a weighted average over the score distributions. If our only focus will be on the scores to be awarded, we will evaluate with BAR Score.
# n points
def bayesian_average_rating(n, confidence=0.95):
if sum(n) == 0:
return 0
K = len(n)
z = st.norm.ppf(1 - (1 - confidence) / 2)
N = sum(n)
first_part = 0.0
second_part = 0.0
for k, n_k in enumerate(n):
first_part += (k + 1) * (n[k] + 1) / (N + K)
second_part += (k + 1) * (k + 1) * (n[k] + 1) / (N + K)
score = first_part - z * math.sqrt((second_part- first_part * first_part) / (N + K + 1))
return score
df["bar_score"] = df.apply(lambda x: bayesian_average_rating(x[["1_point",
"2_point",
"3_point",
"4_point",
"5_point"]]), axis=1)
df.sort_values("bar_score", ascending=False).head(20)
Hybrid Sorting (BAR Score + Other Custom Factors)
Although BAR Score is a good method on its own, it is necessary to include the data of different companies in the calculations and the calculations of the companies themselves. We calculate “Bayesian Average Rating” and “Weighted Sorting Score” by giving certain weights.
def hybrid_sorting_score(dataframe, bar_w = 60, wss_w = 40):
bar_score = dataframe.apply(lambda x: bayesian_average_rating(x[["1_point",
"2_point",
"3_point",
"4_point",
"5_point"]]), axis=1)
# Special Company Score etc.
wss_score = weighted_sorting_score(dataframe)
return bar_score * bar_w / 100 + wss_score * wss_w / 100
hybrid_sorting_score(df)
df["hybrid_sorting_score"] = hybrid_sorting_score(df)
df.sort_values("hybrid_sorting_score", ascending=False).head(20)
Sorting Reviews
The importance of ranking comments is to see the comments of the products purchased by the customer and the scores they received. Afterwards, the focus is on whether other customers who have not yet purchased these reviews will find them useful. We should move the comments that are useful and have many stars to the top so that they do not end up in an infinite loop as these are frequently asked questions :). In addition, the concept of time can also be added to this calculation.
Wilson Lower Bound Score
It offers the opportunity to score any item product or review through binary interactions. Let’s consider YouTube videos as an example. We can give examples of scoring the like, dislike button of the videos and the like, dislike button for the comments and answers given under this video.
This method technically; “For p in the Bernoulli Equation, it calculates a confidence interval. And it accepts the lower limit of this confidence interval as another score.
def wilson_lower_bound(up, down, confidence=0.95):
"""
P score is a WLB score
if Scores 1-5
1,3 Negative
4,5 Positive
:param up: int
:param down: int
:param confidence: float
:return: wilson score : float
"""
n = up + down
if n == 0:
return 0
z = st.norm.ppf(1 - (1 - confidence) / 2)
phat = 1.0 * up / n
return (phat + z * z / (2 * n) - z * math.sqrt((phat * (1 - phat) + z * z / (4 * n)) / n)) / (1 + z * z / n)
wilson_lower_bound(600, 400)
#0.5693094295142663
wilson_lower_bound(5500, 4500)
#0.5402319557715324
Dear readers, my article “Measurement with Python” has been completed :)
Thanks
