Recommender Systems and Ranking Systems

How Platforms Know Exactly What You Want?

Have you ever wondered how Spotify always seems to know the perfect song to play next, or how Netflix consistently recommends the exact movie you’re in the mood for? The secret lies in Recommender Systems (or Ranking Systems) — a powerful subclass of information filtering systems designed to predict a user’s preference or rating for a given item.

At their core, recommendation systems are algorithms that monitor user behavior to predict what they will like, adapting dynamically to changes in user patterns. Whether it’s media streaming (Netflix, Spotify), e-commerce (Amazon, Flipkart), or social media (YouTube, Instagram), these systems are the invisible engines driving digital engagement.

Recommender Systems are a subclass of information filtering systems that aim to predict the “rating” or “preference” a user would give to an item. They are algorithms that monitor user preferences and behavior to predict what they will like. They can also detect changes in user patterns and adapt to it.

There is no “one size fits all” algorithm. Recommender systems generally fall into four distinct categories, each with its own strengths and weaknesses.

i. Popularity-Based Systems

This is the simplest form of recommendation. It suggests items based purely on their overall popularity or trending status.

• The Pros: Highly scalable, incredibly easy to implement, and requires absolutely no historical user data to function.
• The Cons: Zero personalization. It suffers from a heavy bias toward already-popular items, completely ignoring niche or local content, which leads to a severe lack of diversity in recommendations.

ii. Collaborative Filtering (CF):

Instead of looking at the attributes of an item, Collaborative Filtering looks at the behavior of the crowd. It recommends items based on the preferences of similar users, functioning entirely on user-item interactions rather than metadata. They are highly diverse and scalable but are computationally expensive for large datasets and still suffer from the cold start problem for new users and items.

There are two main flavors of CF:

1. User-Based Collaborative Filtering (UBCF): Recommends items that similar users have liked. It calculates the similarity between users based on their preferences, selects the most similar users to you, and recommends what they liked. This is ideal when the number of items is much larger than the number of users (e.g., Instagram posts).
2. Item-Based Collaborative Filtering (IBCF): Recommends items that are similar to items you have liked. It calculates similarities between items based on how users interact with them. This is the go-to approach when the number of users is much larger than the number of items (e.g., Netflix movies).

Note: Advanced CF often uses Model-Based Recommendation Systems, leveraging techniques like SVD Factorization to uncover latent features in user-item interactions.

iii. Content-Based Filtering:

This system acts on the premise of: “Because you liked this, you might like that.” It recommends items by comparing the content or metadata of the items with a user’s profile. For example, if you listen to an upbeat, acoustic indie-folk song or interacted positively with a song with certain content attributes in the past, the system looks at those specific attributes (genre, tempo, acousticness) and recommends songs with similar attributes.

Get Mr. Vivek Kumar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Take a Spotify Hybrid Recommendation System as an example. You could process a massive dataset (like the Million Song Dataset) by generating 10 recommendations based on song metadata (Content-Based) and generating another set of recommendations via an Item-User Interaction Matrix (Collaborative Filtering).

• The Pros: Highly personalized and capable of recommending niche items that popularity-based systems would miss.
• The Cons: It suffers from over-specialization (creating an “echo chamber” where you are only recommended things virtually identical to what you already like). It also struggles with the Cold Start Problem, it is very difficult to recommend items to a brand-new user with no history, or to recommend a newly released item with no interaction data.

iv. Hybrid Recommender Systems:

To mitigate the weaknesses of individual models (like the cold start problem or over-specialization), industry giants use Hybrid Recommender Systems. By combining multiple techniques, you get highly personalized, dynamic, and diverse recommendations.

Take a Spotify Hybrid Recommendation System as an example. You could process a massive dataset (like the Million Song Dataset) by generating 10 recommendations based on song metadata (Content-Based) and generating another set of recommendations via an Item-User Interaction Matrix (Collaborative Filtering).

How do you combine them?

• Weighted Hybrid: Assigns calculated weights to the similarity scores of both systems and mathematically combines them to generate a final, master recommendation list.
• Switching Hybrid: Dynamically chooses which system to use based on the context (e.g., using popularity for a new user, then switching to collaborative filtering once they build a history).
• Feature Combination Hybrid: Feeds collaborative and content-based features into a single, unified machine-learning model.

The Math Behind the Magic: Measuring Similarity

How exactly does a computer know that User A is “similar” to User B, or Song X is “similar” to Song Y? It relies on mathematical distance and similarity measures:

Cosine Similarity

1. Cosine Similarity (The Industry Favorite): Measures the cosine of the angle between two vectors in a multi-dimensional space. It is incredibly popular because it is immune to the “curse of dimensionality.” Scores range from -1 (total opposites) to 1 (identical). Note: In datasets where all interactions are positive (like play counts), the score elegantly ranges from 0 to 1.
2. Pearson Correlation: Measures the linear correlation between two variables, adjusting for users who might generally rate everything higher or lower than average.
3. Jaccard Similarity: Perfect for binary data (like clicked vs. didn’t click). It measures similarity by dividing the size of the intersection of two sets by the size of their union.
4. Euclidean Distance: The standard “straight-line” distance between two points in multi-dimensional space.
5. Manhattan Distance: Measures distance in a grid-based system by summing the absolute differences of their coordinates.

How Do We Know It’s Working? (Evaluation Metrics)

You’ve built the system, but how do you prove to stakeholders that it’s actually driving that desired Click-Through Rate? Recommender systems require labeled data and specific metrics to evaluate performance:

• Precision@K: Out of the top K items the system recommended, what proportion were actually relevant (clicked/liked) by the user? (Focuses on recommendation quality).
• Recall@K: Out of all the relevant items that exist for this user, what proportion did the system successfully recommend in the top K? (Focuses on recommendation coverage).

Below is the working example of Hybrid Recommendation System on top of Collaborative filtering and Content based filtering with Weighted Hybrid approach to combine the outputs (similarity scores) to generate final recommendations of songs on Million Song Dataset Spotify Last.fm.

If you found value in this content, I invite you to stay connected for more insightful posts. Your time and interest are greatly appreciated. Thank you for reading!