Labeling Opportunity Cohorts in Ad and Content Ranking Algorithms

With examples in Python

This article demonstrates a model-agnostic methodology for identifying and sizing groups of users with poor algorithmic personalization experiences. An example is provided in Python.

Executive Summary

1. Ranking algorithms are a common solution in ad targeting and content recommendations. These algorithms do not treat all users equally.
2. When users are treated differently by algorithms, some have better experiences than others. Some users with negative experiences group into cohorts based on common treatment by the algorithm. These cohorts represent product opportunities.
3. Analysis of algorithm error can highlight which cohorts have poor experiences. Explainable AI can further paint a picture of who these users are and how to improve their experiences.
4. This article, and associated Python notebook, explain how to combine Pytorch, SHAP, and HDBSCAN to identify opportunities with underperforming cohorts; including an example using a book recommender whose recommendations underperform for readers of romance fiction.

Introduction

Algorithms provide personalized experiences to users, whether these be personalized ads, music, news, etc. These algorithms are trained to have little error; however, it is common to find examples of users with relatively lower quality experiences. Cohorts of users with shared lesser experiences are good targets for product teams to plan improvements.

This article shares a methodology for identifying underperforming cohorts that product teams can use to improve user experiences, retention, and revenue. It includes a toy example, demonstrating that a simple book recommender systematically misunderstands, for example, readers of romance and women’s fiction.

Example Scenario

Suppose that you are a data scientist working on an e-book app whose central feature provides recommendations of what to read next. We know that good recommendations are important and have learned that a subset of users report poor recommendations. Worse, we believe that these users are more likely to churn. We want to understand who the users receiving poor recommendations are. More importantly, we want to understand the correlates of poor experience and devise a correction plan.

Our recommendation engine does not treat everyone the same. Maybe it fails to find a reading pattern for users with more eclectic tastes. Maybe certain languages of books underperform in our model. These are examples of model underperformance that are cohort specific, not general. General underperformance is often tied to architecture or hyperparameter optimization; cohort-specific underperformance is typically due to bias in the data or assumptions in the model. Our goal is to identify the characteristics of users with worse experiences such that we can design a solution for future improvements.

Our Recommendation Engine

We might use explainable AI (XAI) to understand how our recommender treats users. However, it is hard to apply XAI techniques to production models because the algorithms are often combinations of multiple models. Consider this (simplistic) depiction of a content recommender:

Multi-Stage Recommendation Engine

At Stage 1, a crude but very fast, algorithm retrieves only the content that is most relevant to the user. At Stage 2, a slower model introduces more data and optimizes the output from the prior stage. This multi-stage approach is necessary in many architectures to ensure timely responses. Understanding this recommendation engine requires considering two stages of algorithms, which is typically incompatible with established XAI methods.

To address this, one can build a single model of algorithm error, to which we apply XAI techniques. Pulling from production data, we generate a vector ŷ, corresponding to predicted book ratings. We then retrieve the actual rating, y. Using an error function (f) of our choosing, we define error e such that e = f(ŷ, y). We will identify underperforming cohorts by explaining a model m of e, highlighting groups of users with higher predicted error.

Analyzing Algorithm Error

Analysis of algorithm error, like any data analysis, can be thought of in terms of exploratory or confirmatory analysis. Exploratory analysis emphasizes generating new hypotheses; confirmatory analysis emphasizes evaluating specific hypotheses.

Exploratory modeling of algorithm error merges explanations of why a model behaves as it does with correlates of algorithm performance. In this stage, we will build a model to predict algorithm error and cluster its Shapley values. We will then use the clusters to identify cohorts with similar personalization experiences.

In contrast, confirmatory modeling of algorithm error seeks to validate our hypothesis that we have correctly identified a feature that captures underperformance in the model. This is important for opportunity sizing because while model lift is the gold standard of evaluation, it can be prohibitively expensive to validate opportunities this way. Instead, we will evaluate opportunities through a variant of calibration analysis.

Confidence in Opportunity through Calibration

We expect algorithm error to be proportionate at all values of all features. Anywhere a feature (or a subset of its values) has outsized algorithm error, it is considered miscalibrated and an opportunity to improve the algorithm.

For example, suppose that we hypothesize that user grade-level is correlated to the quality of book recommendations because more sophisticated readers have more diverse interests, which are harder for a recommender to identify. In this scenario, we expect the algorithm to disproportionately deliver poor recommendations to users with greater grade level:

Grade-Level Calibration Curves

The above chart shows two scenarios: randomly assigned grade level (top) and grade level inferred from review text (bottom). The blue line represents the difference between our recommender’s predicted rating and real-world rating.

Get Dan Mallinger’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The random variable (top) has no relationship to algorithm error, as captured by the horizontal trend of the line. However, review-based grade level (bottom) has a clear relationship: the book recommender is unjustifiably optimistic about users with higher grade-level. The area under the curve is not directly interpretable; but features with greater area under the curve are expected to drive larger opportunities than those with smaller area. Calibration curves can be used to rank opportunities by size.

Tying It Together: Identifying Opportunity Cohorts

Thus far, we have described the motivations for finding opportunity cohorts and high-level methodologies for measuring them. In this section, we will walk through an example using the following recipe:

1. Create a dataset of algorithm error and potential correlates.
2. Build a model of algorithm error, m.
3. Calculate the Shapley values of m and cluster them into cohorts.
4. Use calibration analysis to assess the opportunity size of cohorts.

In this example, we investigate if users who read different types of books, e.g. History versus Comic Books, have different quality of recommendations. Call these “user interests.” We will ask “Do users with different interest profiles have different recommendation experiences?” This is a toy example with open source data. Professional applications will want to include additional variables that represent product features, experiences, and expectations.

All code, with explanations, can be found in this notebook.

Step 1: Create Algorithm Error Dataset

We start by acquiring the Amazon 2023 review dataset and building an SVD recommendation engine over book reviews. We then make recommendations over the test dataset to calculate algorithm error.

# load data
infos = datasets.get_dataset_infos("McAuley-Lab/Amazon-Reviews-2023")

# generate training/test splits for our recommendation engine
review_data = datasets.load_dataset("McAuley-Lab/Amazon-Reviews-2023", "raw_review_Books", split="full", trust_remote_code=True)
book_metadata = datasets.load_dataset("McAuley-Lab/Amazon-Reviews-2023", "raw_meta_Books", split="full", trust_remote_code=True)
recsys_train_df = datasets.load_dataset("McAuley-Lab/Amazon-Reviews-2023", "5core_last_out_Books", split="train", trust_remote_code=True).to_pandas()
recsys_test_df = datasets.load_dataset("McAuley-Lab/Amazon-Reviews-2023", "5core_last_out_Books", split="test", trust_remote_code=True).to_pandas()

# lightweight singular value decomposition-based recommendation engine
recommender = surprise.SVD()
reader = surprise.Reader(rating_scale=(1, 5))
recsys_train_data = surprise.Dataset.load_from_df(recsys_train_df[["user_id", "parent_asin", "rating"]], reader).build_full_trainset()
recommender.fit(recsys_train_data)

# the prediction dataframe has a row per prediction and its outcome
# because of the nature of the train/test split, there is one row per user in test
prediction_df = pd.DataFrame(recommender.test([(row["user_id"], row["parent_asin"], row["rating"]) for _, row in recsys_test_df.iterrows()])).rename(columns={"uid": "user_id", "iid": "parent_asin", "r_ui": "rating", "est": "prediction"})
prediction_df["rating"] = prediction_df["rating"].astype(float)
prediction_df["delta"] = prediction_df["prediction"] - prediction_df["rating"]
prediction_df["delta_zscore"] = scipy.stats.zscore(prediction_df["delta"])

Step 2: Model Algorithm Error

The recommendation test set is then further divided into train, validation, and test for algorithm error modeling. Any prediction model compatible with Shapley values is viable; however, we use a neural network as it is potentially more suited to the sparse nature of user interests data.

The creation of these datasets and model are handled by this code:

sample_mask = pd.qcut(np.random.rand(len(interest_model_df)), 7, labels=False)
interest_model_train_df = interest_model_df.iloc[sample_mask >= 2]
interest_model_validation_df = interest_model_df.iloc[sample_mask == 0]
interest_model_test_df = interest_model_df.iloc[sample_mask == 1]

# CUDA test and train datasets
X_train = torch.tensor(interest_model_train_df[list(valid_book_categories)].to_numpy(), device=device, dtype=torch.float32)
y_train = torch.tensor(interest_model_train_df["delta_zscore"].to_numpy(), device=device, dtype=torch.float32)
X_validation = torch.tensor(interest_model_validation_df[list(valid_book_categories)].to_numpy(), device=device, dtype=torch.float32)
y_validation = torch.tensor(interest_model_validation_df["delta_zscore"].to_numpy(), device=device, dtype=torch.float32)
X_test = torch.tensor(interest_model_test_df[list(valid_book_categories)].to_numpy(), device=device, dtype=torch.float32)
y_test = torch.tensor(interest_model_test_df["delta_zscore"].to_numpy(), device=device, dtype=torch.float32)


# building a neural network to predict algorithm error.
hidden_layer_width = 12
interest_model = nn.Sequential(
                     nn.Linear(in_features=X_train.shape[1], out_features=hidden_layer_width),
                     nn.BatchNorm1d(hidden_layer_width),
                     nn.ReLU(),
                     nn.Dropout(p=0.3),
                     nn.Linear(in_features=hidden_layer_width, out_features=hidden_layer_width),
                     nn.BatchNorm1d(hidden_layer_width),
                     nn.ReLU(),
                     nn.Dropout(p=0.1),
                     nn.Linear(hidden_layer_width, 1)
                 ).to(device=device)

loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(interest_model.parameters(), lr=0.00001, weight_decay=0.0075)

Step 3: Calculate the Shapley Values and Cluster Them

After creating a model of recommendation error, we can calculate its Shapley values. In our case, Shapley values correspond to the impact of user interests on recommendation algorithm error:

# Shapley background data needs to be smaller for our code to run quickly, so we use the kmeans
# method to identify representative background data.
X_train_summary = shap.kmeans(X_train.cpu().detach().numpy(), int(X_train.shape[1] ** 0.5))

# get shapley values
interest_model.eval() # disable dropout layers
explainer = shap.DeepExplainer(interest_model.to(device=device), torch.tensor(X_train_summary.data, device=device, dtype=torch.float32, requires_grad=False))
shap_validation = explainer.shap_values(X_validation).squeeze()
shap_test = explainer.shap_values(X_test).squeeze()

By clustering Shapley values, we are grouping users whose recommendations were erroneous for similar reasons. It is critical to understand that we are not clustering user interests broadly but the impact that these interests have on recommendation quality. Intuitively, Shapley value clusters pull together users whom the book recommender misunderstands in similar ways.

We opt to explore opportunity cohorts whose size is at least 1% of the user base by clustering the SHAP values with HDBSCAN including hyperparameters for cohort size:

# cluster shapley values such that each cluster must be at least 1% of users
clusterer_validation = hdbscan.HDBSCAN(min_cluster_size=int(shap_validation.shape[0]*0.01), prediction_data=True)
clusters_validation = clusterer_validation.fit(shap_validation).labels_
clusters_test = hdbscan.approximate_predict(clusterer_validation, shap_test)

HDBSCAN is particularly useful because it allows users to not belong to a cluster (labeling them as noise). In the case of modeling recommendation error, we expect the majority of our dataset to be noise (especially for mature products).

Step 4: Calibration Analysis and Opportunity Sizing

Our analysis produced five user cohorts. In practice, the number of cohorts will vary by application and quality of covariate data.

As an example, we focus our attention on Cohort A, which accounts for 1.5% of our user sample and has an average miscalibration of -6% standard deviations of rating discrepancy. A negative miscalibration implies that users report interest in books that the book recommender labeled as less relevant. This suggests that latent demand may exist and our book recommender is worse at identifying books for Cohort A.

With TF-IDF we can explore these cohorts further, seeing that Cohort A consists of users who read Romance and Women’s Fiction, as exemplified by publishers like MIRA and Harlequin. However, the cluster has a secondary focus on mystery and detective novels. With further investigation we could evaluate if this cluster represents a bimodal distribution of romance and mystery; and we may further hypothesize that this bimodality confuses our book recommender.

User Interest Word Cloud

We now have confidence that some readers of romance and women’s fiction receive subpar book recommendations compared to the average user, which is a surprising gender bias of our algorithm! We also now have techniques for prioritizing this opportunity relative to others (calibration size). Next steps follow differently in different fields. A book recommender may use this insight to augment training data. An ad network may decide to change attribution windows. A social media app might try to map sessions to specific interests. Once opportunity cohorts are identified, the real work can begin!

Closing Thoughts

This article demonstrates a methodology for identifying users with lower quality personalization experiences (opportunity cohorts) in arbitrary settings. In practice, one can use this methodology to improve any algorithm, whether it be identifying why certain consumers have worse ad click-thru rates or labeling the users most likely to churn from a music recommender.

Be warned that the article is deceptively simple: real-world use cases require effort to optimize the error model, tune the clustering hyperparameters, and iterate between exploration and confirmation. But the juice can be worth the squeeze.

Enjoy!