Scroll into the Future: How eBay’s Multi-Arm Bandits Elevate Product Recommendations Through Dynamic Pagination
By Peijun Zou, Vincent He, AB, Xin Yin, Xiaoqin Chen and Dan Schonfeld
By leveraging pagination and multi-arm bandit techniques to present customers with more relevant item listings, we greatly improved the user experience on eBay’s Feed Recommendations pages.
Background
Currently when eBay customers shop on eBay, we frequently present them with a diverse combination of item recommendations depending on their stage in the shopper journey. The real estate on the site where we render these recommendations are referred to as “Placements”, with each eBay page featuring a different number of placements. For example, the product listing page, also known as the View Item page, shown in Figure 1.1 contains six placements while the See More page (Figure 1.2) accommodates three. The purchase behavior may be hindered if the provided recommendations are not relevant enough to meet users’ needs. In this context, surfacing more placements increases the probability that eBay users will be presented with their desired items. However, increasing the number of placements on the page will incur higher cost in terms of latency and computation power to load the page. Additionally, most users don’t have the patience to check an extensive amount of placements.
For example, we currently have three placements on the Desktop View Item (VI) page with the percentage of users who view each placement as follows:
Thus, we would like to utilize pagination (Figure 2) to seamlessly and automatically load new content onto the page as the user scrolls to the end of the current content section to decrease the drop-off in user engagement. We are also using a multi-arm bandit technique to rank the new paginated placements so that the more relevant recommendations will be shown first.
Pagination Engineering
The recommendation feed service, known as “merchfeed”, delivers continuous, high-quality product recommendations to customers. This service employs pagination to predict and identify the content that will appear in the subsequent sections, as illustrated below:
Each page section displays recommendations from a specific algorithm, curated by a “placement”. Common algorithms include “similar” (“more like this”) and “complementary” (“customers who bought this also bought”). In addition to these, we offer numerous advanced, personalized algorithms to enrich the shopping experience, with hundreds of options available. A primary challenge is integrating a new, “never seen” algorithm into subsequent pages, ensuring the recommendations are relevant and align with user preferences.
To address this, a caching system for algorithm deduplication is essential, while the Page Optimization Service, a machine learning platform, predicts the optimal algorithm candidates using a multi-arm approach.
The workflow involves four main parties:
- Client: This includes mobile native and web platforms, which render the feed UI with pagination and display it to customers.
- Merchfeed Service: A module provider processing feed requests for different campaigns and managing pagination requests.
- Recommendation Service: Executes algorithms and provides item recommendation recalls.
- Page Optimization Service: A machine learning service that selects the most effective algorithms, key in implementing “multi-armed bandit” strategies.
For each new algorithm presented to the client, it’s cached in eBay’s key-value cache service. This ensures that the Page Optimization Service, when generating new algorithms, excludes those already used in the same session.
Get Dan Schonfeld’s stories in your inbox
Join Medium for free to get updates from this writer.
For initial page requests, Merchfeed Service generates a new session ID, linking all algorithms run within that session. Page Optimization uses this ID to avoid previously used algorithms, selecting new ones for subsequent pages. The chosen algorithms are sent back to the Recommendation Service to fetch corresponding item recalls, which are then assembled in Merchfeed and delivered to the client. The workflow is illustrated below:
This system efficiently handles millions of sessions per day and is used on high-traffic pages like the view item page. Future plans include optimizing the workflow further, such as directly calling Page Optimization from Merchfeed to reduce transaction times and enhance stability.
Bandit Model
Pagination techniques enable us to present more content to the user, but the user will lose interest while scrolling if they are not presented with relevant content. Thus, we utilize a multi-arm bandit model to rank the placements so that the user is shown the most relevant items to their shopper journey first. Each arm is an algorithm candidate, or in simpler terms a placement theme. The theme could be Similar Items, Complementary Items, Sellers’ Other Items, and many more. The multi-arm bandit ranks placements using a reinforcement learning technique to balance exploration and exploitation. It contains different bandit models such as Thompson sampling, cascading bandit, linear bandit, and neural bandit. We tested different bandit methods and decided to use linear bandit as the production method due to its easy implementation and relatively powerful ranking by incorporating various contextual and personalization features. The diagram below characterizes the main differences between each bandit approach:
Linear bandit was first introduced by Shipra Agrawal and Navin Goyal in 2013 and since then it has been well studied. As the name implies, the linear bandit is a linear model which assumes that likelihood at time t of a reward (click or purchase) r(t) given context b(t) , constant v and parameter 𝜇 following the probability density function of a Gaussian distribution:
During the training process, our goal is to estimate the parameter 𝜇 based on user’s feedback and context features. The detailed algorithm steps are listed below. Note that I is the identity matrix and ƒ, B are the intermediate variables here. The variance of the parameter will become smaller as the bandit model collects more data.
In our production implementation, we follow the Bayesian Linear Regression method from Riquelme et al. 2018 where the noise parameter v is replaced with a prior belief of an inverse-gamma distribution and will be updated over time.
As a reinforcement learning technique, the linear bandit model will train the model based on user feedback (clicks or purchases). We train the parameter for each bandit arm and store the parameter into OpenSearch and during inference time the parameter is retrieved from OpenSearch to calculate the ranking scores of the placements. The user will see the ranked page and react based on the relevance of the placements. Lastly, the feedback will be tracked back into our ETL pipeline and the arm parameters will be updated daily.
Online Experiments Results
We first A/B tested the pagination and multi-arm bandit combination on pages with less traffic such as the Native See More page to compare the effectiveness of different bandit models:
The results show that by incorporating the contextual and personalization information, linear bandit has much better ranking results.
We then A/B tested the pagination and linear bandit combination on See More page across all device platforms. The results are shown below:
To rule out the cannibalization effect where the new placements ends up taking sales away from its existing old placements rather than generating additional revenue, we then tested our techniques on the Desktop View Item (VI) page, yielding similar results with significant positive site level impact:
In conclusion, the pagination and multi-arm bandit techniques exhibit great improvement in terms of purchases, clicks, and ad revenue. It enables more users to fulfill their shopping needs.
Future Development
When users scroll down and request more placements, it’s becoming more and more important to generate diverse results. Otherwise, shoppers may become fatigued by the same placement theme and the user experience will be compromised. We plan to utilize diversity techniques such as Determinantal Point Processes (DPP) or Maximal Marginal Relevance (MMR) to promote page diversity and provide users with more customized and personalized shopping experiences.
