Building Similar and Personalized Product Ads with Two-Towers

1. Problem Overview

Modern e-commerce platforms need to show relevant product advertisements and recommendations to users across multiple surfaces such as product detail page, home page, search results, shopping feeds, etc.

Showing irrelevant ads reduces engagement and revenue, while relevant recommendations can significantly improve:

• Click-through rate (CTR)
• Conversion rate
• Average order value

The goal of this system is to build a semantic recommendation engine that can retrieve and rank relevant products based on user behavior and product similarity.

1.1 Available Data

To build this system we rely on two main data sources.

Item Metadata: Title, features, color, manufactures

User Behavior Data: Product views, Purchases

1.2 System Output

The system exposes an online inference endpoint that returns relevant product recommendations.

The endpoint accepts:

anchor_item_id (optional)
user_id (optional)

and returns:

Top N recommended products

This endpoint can be directly integrated with the front-end UI to render ads or recommendations.

1.3 Recommendation Scenarios

1.4 Approach

To solve this problem we use a Two-Tower Embedding Architecture.

The system includes:

• Sentence-BERT model for semantic embeddings
• PyTorch for model training
• Approximate Nearest Neighbor (ANN) index for fast retrieval
• Cloud inference deployment (e.g. SageMaker / Vertex AI)

This architecture allows us to efficiently search through millions of products in milliseconds.

In the next section, we explore why Two-Tower architectures work well for recommendation systems.

2. System Architecture

A recommendation system at scale requires both offline training pipelines and online inference services.

3. Why Two-Tower Models Work

Two-Tower (or dual-encoder) architectures are a popular choice for retrieval-based recommendation systems because they separate item and user (or anchor) representations.

Get Ankur Porwal’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Key Intuition

• Each tower encodes its input into a dense vector embedding.
• Similarity between vectors (usually cosine similarity) indicates relevance.
• By precomputing embeddings for items, we can perform fast retrieval using Approximate Nearest Neighbor (ANN) search.

Training Signals

• Positive pairs: Products frequently viewed together by the same user in the same session.
• Negative pairs: Randomly sampled unrelated items.
• Loss function: Contrastive or cosine-similarity-based loss.

4. Training the embeddings

Once the model is trained:

1. Use model.encode() to generate embeddings for all items.
2. Store embeddings in a database or vector store.
3. These embeddings are used for fast retrieval with ANN in production.

This offline pipeline ensures that item embeddings are precomputed, enabling millisecond-level retrieval during online inference.

5. Retrieval with ANN

After training the Two-Tower model and generating item embeddings, the next step is retrieving relevant items efficiently. Searching millions of embeddings linearly is too slow, so we use Approximate Nearest Neighbor (ANN) search.

Similarity Metric:

• Most commonly cosine similarity is used:

Similarity(x, y) = (x · y) / (||x|| * ||y||)

• Embeddings that are semantically or behaviorally similar have a higher cosine score.

Why ANN?

• Linear search scales O(n), too slow for millions of items.
• ANN (e.g., FAISS, ScaNN, or Annoy) reduces retrieval to milliseconds.

6. Personalization: Ranking Based on User

After retrieving candidate items using item-to-item similarity, we further personalize recommendations based on the user’s recent activity. This ensures the top items are aligned with the user’s preferences.

6.1. Input for Personalization

For each user, we consider:

• Last 5 viewed items in the past T days
• Last 5 purchased items in the past T days

These items capture recent interests and strong signals of preference.

This is a lightweight way to include user context without maintaining dynamic user embeddings.

6.2. Multi-Anchor ANN

1. Fetch embeddings for the recent 10 items (5 viewed + 5 purchased).
2. Retrieve the top 30 recommendations for each item
3. Aggregate them by summing cosine similarities
4. Select final recommendations with the highest cosine scores

7. Full Inference Pipeline

The full inference pipeline combines item-to-item retrieval and user personalization to serve top recommendations in real-time.

This modular, scenario-based flow ensures low latency, scalability, and relevance, making it production-ready for large e-commerce platforms.

9. Trade-offs and Next Steps

Building a recommendation system at scale involves practical trade-offs between accuracy, latency, and cost. Here are the key considerations for the next set of improvements:

9.1. Dynamic User Embeddings

• Full user embeddings incorporating browsing history, demographic info, and real-time interactions can improve personalization.
• Challenge: Updating these embeddings continuously is computationally expensive and can increase latency.
• Our approach simplifies this by using last 5 viewed + 5 purchased items as anchors, allowing fast, online multi-anchor ANN without storing dynamic user embeddings.

9.2. Latency vs Accuracy Trade-off

• Multi-anchor aggregation (10 × 30 candidates → sum cosine scores → top 30) balances personalization and speed.
• More anchors or larger candidate sets improve relevance but increase computation.
• The current approach is a practical middle ground for low-latency, high-quality recommendations.