<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by pradeep karuturi on Medium]]></title>
        <description><![CDATA[Stories by pradeep karuturi on Medium]]></description>
        <link>https://medium.com/@pradeepkaruturi?source=rss-4f6cfe60cb8a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*VMa1maTfGLzlMiQK.jpg</url>
            <title>Stories by pradeep karuturi on Medium</title>
            <link>https://medium.com/@pradeepkaruturi?source=rss-4f6cfe60cb8a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:42:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pradeepkaruturi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sequence models for Contextual Recommendations at Instacart]]></title>
            <link>https://tech.instacart.com/sequence-models-for-contextual-recommendations-at-instacart-93414a28e70c?source=rss-4f6cfe60cb8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/93414a28e70c</guid>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[contextual]]></category>
            <dc:creator><![CDATA[pradeep karuturi]]></dc:creator>
            <pubDate>Wed, 23 Oct 2024 17:20:27 GMT</pubDate>
            <atom:updated>2024-10-23T18:56:04.583Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONO9MhfCYVSOMg5zWHlvtg.png" /></figure><p><strong>Authors</strong>: Pradeep Karuturi, Young Rao, Sharath Rao, Shishir Kumar Prasad<br><strong>Key contributors</strong>: Brian Lin, Cheng Jia, Karuna Ahuja, Shrikar Archak, Jichao Zhang, Taesik Na, Haixun Wang</p><h3>Introduction</h3><p>At Instacart, we help our users find their favorite products organically or discover new ones through Ads across various shopping surfaces such as search, browse and recommendations. Behind the scenes, we use machine learning algorithms to power these surfaces to provide a delightful user experience. Each of these surfaces have diverse optimization goals and multi-step ranking pipelines (retrieval, ranking, re-ranking). For example, while organic content is often optimized for user engagement and transaction revenue, sponsored content additionally takes into consideration advertiser value and ad revenue. In this blogpost, we describe how we built a centralized <strong>contextual retrieval system</strong> that powers diverse recommendation surfaces, even though their end goals and ranking layers are different. Having a common retrieval system across both ads and organic surfaces has lowered our maintenance costs and allowed us to deprecate many legacy ad hoc retrieval systems. Using in-session contextual signals, we built a <a href="https://arxiv.org/abs/1810.04805">BERT</a>-like language model to power sequence recommendations for this system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*-TU_cH6cEyM_R0dX" /></figure><p>Our customers use Instacart for the convenience and time-savings we provide and often choose to fulfill the weekly shopping needs of their entire family by placing large basket orders. We take pride in making their shopping experience as efficient and effortless as possible. Our contextual retrieval system reacts in real time to a user’s actions within a shopping session and retrieves products relevant to that session. For example, when a user adds pancake mix to their cart, views bacon and later adds eggs to their cart, there is probably a breakfast intent and our contextual recommendation system will retrieve breakfast products as shown in the below diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GRWvjJ3dVOwlDvna" /><figcaption>Contextual Recommendations</figcaption></figure><p>We use this contextual recommendation system across many surfaces at Instacart such as search, item details page, cart, pre-checkout and post-checkout recommendations. Below, we show a couple of visual examples of where this contextual recommendation system is used.</p><p><strong>Item Details Page: </strong>Here we show detailed information about a product. We introduced a carousel titled, “Items to add next” that is based on user actions within a shopping session (cart adds, product page views etc.). In this example, the user already had milk and bread in their cart. In the context of these cart products and current olive oil product, we make relevant recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ajXej8EGJX-ADKKa" /></figure><p><strong>Cart Page: </strong>Before a user checks out, we show cart recommendations in the cart details page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ih1ci4sbzTn3cvSfc1cC9g.png" /></figure><h3>Problem Formulation</h3><p>We formulate the retrieval problem in terms of the next product prediction task. Given a sequence of products that the user interacted with, we predict the probabilities for products that the user may interact with next.</p><p>Predict p(Pᵢ| Pₜ₁,Pₜ₂…) for i in [1, N], where N is the size of the catalog i.e., we predict the probability of the next product given a sequence of user product interactions at times t1,t2,…</p><p>Once we predict the probabilities across all product ids, we pick the top K products based on predicted probabilities for further downstream ranking. We evaluate the models offline using Recall@K metric. This metric roughly gives us the percentage of times when the last token in the test sequence dataset is present in the top K predictions from the model. This metric indicates how effectively the model predicts which product a user might be interested in next.</p><h3>Model</h3><p>In order to train a model for this problem, we used the Masked Language Model (MLM) approach that is very common in the NLP domain. Prior work in this domain (<a href="https://arxiv.org/abs/1904.06690">BERT4Rec</a>, <a href="https://research.facebook.com/publications/transformers4rec-bridging-the-gap-between-nlp-and-sequential-session-based-recommendation/">Transformers4Rec</a>) showed good results using this approach to e-commerce datasets. But unlike BERT4Rec and Transformers4Rec, which show results on tens of thousands of products, our production models have an order of 10x more products. Moreover, our catalog has millions of products spread across multiple retailers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rBZwUAj6HaElfXkh" /></figure><p>In the case of BERT, the model is trained on token sequences of text data. We train our model on historical sequences of product ids that a user interacts with in a session. At training time, we use a Masked Language Modeling(MLM) approach. At inference time, we use the encoded session representation from the transformer block to predict probabilities over all product ids. We tried different Language model architectures like XLNet and BERT, and based on offline evaluation we converged on a simple BERT-like model for our problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4SN5x3Qb7TpBHWdLwne7A.png" /><figcaption>Typical BERT model vs our use case</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PepszTHnqaJEic6a" /></figure><p>Just using product ID sequences without any additional user or product features has shown significant impact and we limit ourselves to describing a preliminary version of this system that does not use any other features. We will follow up in a later post on how we leverage more contextual features in this system.</p><h3>Impact</h3><p>Prior to building this contextual retrieval system, we had a disparate set of retrieval systems for different recommendation surfaces across ads and organic surfaces. By leveraging a unified retrieval system, we were able to deprecate old ad hoc systems.</p><p>Prior retrieval systems did not properly use contextual sequence information and were mostly based on an ad-hoc combination of product co-occurrence, similarity and popularity signals; all of these are important but the next product purchase prediction turns out to be a principled path forward as well as empirically more successful. Our initial offline evaluation showed a significant uplift over prior systems and led to outsized impact across multiple metrics across transaction volume and ad marketplace. We also saw a 30% lift in user cart additions when we launched this system on cart recommendations.</p><h3>Discussion: Does sequence information matter?</h3><p>We have seen big wins from sequence based models at Instacart. But how significant is the precise sequence information in predicting user behavior? In other words, what happens if we mask the real sequence information and just input a group of products to the model? We evaluated this using two approaches:</p><h4>Different Training procedure</h4><p>When training a transformer-based sequence model, we randomized the tokens in the training sequences and saw how offline metrics are affected compared to that of our control model, which has access to precise sequence information. In the control model, we did not randomize token sequences in training data. We then compared the relevance metrics on non-randomized test dataset on the last product prediction task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E1XC6lphE22CoV4_" /><figcaption>Evaluating the importance of sequence information</figcaption></figure><p>Even though we may have gotten meaningful recommendations with completely randomized sequence training data, the relevance metrics (Recall@K) for the model trained on randomized training data were 10–40% worse depending on K. This indicates that sequence information at training has a meaningful impact on the quality of next product predictions.</p><h4>Different evaluation</h4><p>For a different evaluation, we kept the control model as-is at training time and changed the evaluation methodology. We got relevance metrics for the test dataset once by randomizing the token sequence in the test set and once without randomization. We kept the last product of the product sequence in the test data set and randomly shuffled all the products preceding the last product in the sequence. Evaluation metrics degraded by 20–45% depending on the value of K. This indicates that during model inference, if we do not have proper sequence information, prediction quality(and recommendations) will suffer.</p><p>We also evaluated the impact of more recent products in the user interaction sequence on recommendations. The last 3–5 products in the sequence have an outsized influence on the recommendations with prior products having a diminishing influence as the sequence length increases. So in practice, we resorted to using sequence length of 20 for training and inference.</p><h3>Ongoing and Future work</h3><p>We have described a preliminary contextual retrieval system, on which we have since iterated to tackle the challenges of applying language models to our use cases, including:</p><ol><li><strong>Catalog Size: </strong>Our catalog has millions of products spread across thousands of retailers. For the first iteration of our system, we restricted our models to have a product-id vocabulary of under one million. We choose these products based on a combination of product popularity and other business rules. Since we restrict the model vocabulary to be a fixed number, all the product_ids not in the datasets will be mapped to an Out of Vocabulary Token (OOV). We have explored ANN approaches to scale our model to millions of products.</li><li><strong>Canonical Product Identity: </strong>Millions of products are sold at Instacart across thousands of different retailers. Some of the products share a common identity across retailers, but some products, such as non-branded strawberries, may not share a common product identity. Since our preliminary model is just based on a sequence of product-IDs, it is possible that non-popular product-IDs (e.g., non-branded produce) may not surface in the recommendations even though such products are popular at a particular retailer. Text-content inclusive approaches, such as <a href="https://arxiv.org/abs/2305.05065">TiGER</a>, can help solve these problems.</li><li><strong>Popularity bias: </strong>As is commonly the case with retail — both online and offline — the majority of our user interaction data is with a small subset of our catalog and our training data reflects this. The preliminary model has a bias towards popular products when recommending products. So we get top-K products from the model for a particular retailer.</li><li><strong>More Context: </strong>So far we have focused our model training and inference using cart add and product view sequences. We are working on incorporating more context information such as user searches into our model.</li></ol><h3>Conclusion</h3><p>Implementing transformer-based sequence models in our contextual retrieval layer has significantly enhanced the performance of our recommendation surfaces. Leveraging user interaction sequences has enabled us to develop more effective models than previous ones. Furthermore, by adopting a unified retrieval layer that serves both ads and organic surfaces, we have been able to phase out legacy systems that didn’t utilize sequential information. We have discovered that a simple BERT-like architecture is optimal for our datasets when focusing solely on product-ID sequences. Upcoming blog posts will delve into incorporating additional context and product and user features into our sequence models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93414a28e70c" width="1" height="1" alt=""><hr><p><a href="https://tech.instacart.com/sequence-models-for-contextual-recommendations-at-instacart-93414a28e70c">Sequence models for Contextual Recommendations at Instacart</a> was originally published in <a href="https://tech.instacart.com/">tech-at-instacart</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>