<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pavel Kostin on Medium]]></title>
        <description><![CDATA[Stories by Pavel Kostin on Medium]]></description>
        <link>https://medium.com/@PavelKostin?source=rss-48f370875dce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3i8712TmRlEqs1qeAcNpvw.png</url>
            <title>Stories by Pavel Kostin on Medium</title>
            <link>https://medium.com/@PavelKostin?source=rss-48f370875dce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:04:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@PavelKostin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Measuring e-commerce product recommendations: pro tips and tricks]]></title>
            <link>https://medium.com/@PavelKostin/measuring-e-commerce-product-recommendations-pro-tips-and-tricks-b0c559d5ec27?source=rss-48f370875dce------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0c559d5ec27</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[product-recommendations]]></category>
            <category><![CDATA[e-commerce-business]]></category>
            <category><![CDATA[personalization]]></category>
            <category><![CDATA[marketing-automation]]></category>
            <dc:creator><![CDATA[Pavel Kostin]]></dc:creator>
            <pubDate>Tue, 05 Jul 2022 09:06:51 GMT</pubDate>
            <atom:updated>2022-07-05T09:06:51.333Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*IiEV5yYwQ_z58wCo.png" /></figure><p>I’ve been working with the e-commerce market for the last 10 years and, before going deeper into the personalization/recommendation engines, found myself that I did not pay enough attention to the quality of the product recommendations.</p><p>When discussing this with some colleagues from the market, I found that I’m not alone in my approach to them. During one of last discussions on how to measure the quality of the recommendations, I came to conclusion that there’s not much info about this topic on the web and decided to share my experience on it, so that anyone can use my approach and enlarge their knowledge and experience on this matter.</p><h3>Introduction into the product recommendations</h3><p>For sure, if you’ve worked at any e-commerce project or browsed any website with at least 50 SKUs on board, you have faced product recommendations, presented on the homepage, catalogue, product detailed page or checkout.</p><p>The product recommendations are an essential part of any e-commerce website and are known to increase the average order value, number of items in cart, conversion rate and total revenue. And they really do, in the wide majority of the cases, because they help you, as a customer, find most relevant items on the web or app.</p><p>But do you actually know, with the product recommendations implemented on your website, how these product recommendations are performing?</p><p>Are you sure, that the recommendations you have — are maximizing your value, do show correct products for cross-sell and similars and have a positive impact on the overall website / mobile app performance rather than drag attention from the right products to some other, irrelevant to the products a user sees?</p><p>From what I saw, a huge number of e-commerce businesses just do the implementation of the product recommendation blocks, be it their own widgets or any external SAAS product. And pay little attention to the final quality of the implemented recommendations or what percentage of revenue is delivered from them.</p><p>This is even more true for the small-to-medium businesses where you do not have a whole in-house team or web-analysts and product managers and do not have the option to get a detailed and deep understanding of how to measure the product recommendations.</p><p>Yes, many of the companies do the A/B testing, try to measure the performance of different designs of the blocks and different strategies, with some attribution applied and results in uplifts or downgrades in revenue.</p><p>But generally the majority of these A/B tests is crap — and here is why:</p><p>— To get a better understanding of the problem, try to answer the following question: <em>if you have your recommendation blocks implemented on your homepage</em> <em>— how do you measure the quality of the recommended products in these blocks?</em></p><p>If you try to measure it by revenue from the clients, who saw or clicked into some products in this block, this will allow a lot of irrelevant data into the testing — since users very rarely tend to buy from the homepage and their customer journey is far more complicated, they will most of the time end purchasing a totally different product from the one, they clicked into on the homepage.</p><figure><img alt="User’s customer journey can be quite complicated" src="https://cdn-images-1.medium.com/max/1024/0*O1ydgJFj0fS2rWsn.png" /><figcaption>User’s customer journey can be quite complicated</figcaption></figure><p>On the other hand, if you measure it by CTR, do your clients actually buy the item, they clicked into? And did it increase the actual conversion rate, compared to not having any recommendation blocks on the homepage and using standard catalog browsing instead?</p><p>These questions on the way of measurement the effectiveness of product recommendations brought me to a number of thoughts I’d like to discuss in this article.</p><h3>Metrics I find most useful for measuring the effectiveness of product recommendations</h3><ol><li><strong>Direct revenue</strong></li></ol><p>During my work on a various number of personalization projects, using products, like Exponea and Dynamic Yield, with a team of personalization managers and solution consultants, we have come to a conclusion that the best way to measure the effectiveness of product recommendations is a metric called <strong>“Direct revenue”</strong> — which is revenue from the product, which the user clicked and bought within an attribution window (which could vary from 1 to 30 days, depending on your business model).</p><p>To visualize this example, let’s pretend you are working in an online food retailer, with a frequency of 2–4 purchases per month. In this case, the attribution window for the purchased product from recommendation widget should be the most narrow, 1–2 days usually is enough.</p><p>However, if you work in an online DIY retailer, especially selling complicated and expensive products, you might use attribution up to 30 days, because the customer journey of this user would be much longer and might take days or even weeks for the purchase to happen.</p><p>Why “Direct Revenue”? Because it is the key metric to understand that the recommendation seen on the page — is the one the user bought, thus the recommendation engine is showing the most relevant items.</p><p><strong>2. Assisted revenue</strong></p><p>The next metric, used for measuring your product recommendations, will be “<strong>Assisted revenue”</strong>, which is generally all the revenue from products clicked and bought using product recommendation block.</p><p>For instance, a user clicked on some product in recommendations, but initially finished his purchased buying some other product. In this case, this purchase will be associated into the assisted revenue (actually, the revenue which was generated with the usage of a recommendation block). This is not an effective enough metric to measure the quality of your recommendations, because the user might end with buying a totally different item, irrelevant to the one recommended, and thus, the assisted revenue will end with a huge figure in your dashboard. But still I find this metric useful, and we will discuss how to use it below.</p><p><strong>3. CTR</strong></p><p>I will not actually pay a lot of attention to the typical metrics, you should be already aware of — like CTR, number of impressions and clicks, assuming you already understand all these metrics and how they affect on the overall performance.</p><p>However, there are a few important things here for discussion:</p><ol><li>CTR is a good metric of measuring the quality of recommendations. However, in the end, high CTR does not mean high direct revenue from the recommendation blocks. On the opposite, sometimes strategies with lower CTR will show much better results in final revenue. Because the recommendation engine might have narrow filtering of products and show less of them; but the most relevant ones — which will decrease the CTR, but increase the final revenue, showing only the most relevant items.</li><li>You should be aware of how you calculate your impressions — for instance, when a recommendation block is not showing on the first screen, you should be aware of when your attribution windows starts. In the case when your recommendation block is at the bottom of the page, the «load page» attribution will make your data irrelevant, because some users will not see the recommendation block at all. In these cases, the start of attribution should be, when the recommendation block is in your viewport (when they were actually seen by the user).</li></ol><p><strong>4. Incremental revenue</strong></p><p>Incremental revenue means additional revenue of one variation (recommendation block or recommendation strategy in our case) against the other one or against control group (CG).</p><p>Do you need to measure it, or use the metrics above? Despite trying to find the easiest answer «Yes, you should be paying a lot of attention to increment», since it is one the main KPIs for any conversion rate optimization / personalization manager — let’s look deeper into this question:</p><ul><li>First of all, when you measure the presence of product recommendation block against no recommendation block at all (CG) — imagine cases when the CG (no recommendation blocks at all) could win. Struggling? For sure, there will be these cases, for example I’ve seen cases in product cart when recommendation blocks disrupt user’s attention on his customer journey. But actually, there are not many of them, when the absence of recommendations increases revenue. And in the majority of these cases — the question is mainly to the quality of recommendations, not to the UI or the necessity of the block itself on the page.</li><li>So, mainly you should be comparing <strong>different strategies</strong> within the recommendation blocks using <strong>one and the same method of measurement</strong> (this is the most important). <br>To say a few more words about this — if you are doing a bake-off of two different SAAS services, or a third party SAAS recommendation service against your own recommendation engine, the increment should be measured by one of the parties, and the measurement approach is to be agreed from all sides. Otherwise, you might be ending that data from one SAAS gives them +N% increment whereas the other SAAS service has uplifts +M% on their side. And you can’t understand who wins at the end, because both sides measure the results differently. Here, I suggest focusing on direct revenue as the key metric at a bake-off stage.</li><li>Should you always have a control group within all your recommendation block strategies? That’s more tricky — but I’d generally say no — you should be comparing different strategies and always striving to find the ones which work the best in any particular moment. Rather than comparing a recommendation block to emptiness. However, there’s nothing bad, if you want 5–10% of users to be always in your CG.</li></ul><p><strong>5. Introducing the “RQI index”</strong></p><p>Now, to understand the effectiveness of your recommendation strategies, in addition to “direct revenue” metric, I will introduce a new metric called RQI: Recommendation quality index.</p><p>The principle is easy — we measure the percentage of direct revenue from the recommendation block to the assisted revenue using a simple formula: <em>Direct revenue / Assisted Revenue * 100%</em>.</p><p>Why we need RQI? This formula gives us an overview of the total quality of the recommendations, when we compare the revenue from directly bought items to the total revenue, that came through the recommendation block.</p><p>From my experience on tens of various projects, <strong>10% RQI</strong> is a very good result, while everything above is great!</p><p>To sum up, let’s once again recap the most important points:</p><ol><li>Measure and pay attention to the attribution window — it should be different depending on the type of products you sell and the length of purchasing path of your clients.</li><li>Use direct revenue as a main point of measurement of the quality of your recommendation blocks.</li><li>When measuring increment — the best way is to compare recommendation strategies with each other rather than compare a recommendation block vs emptiness.</li><li>Try using the RQI index as one of your new KPIs to measure the effectiveness of your product recommendations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*DKQTIipdFwZ6ao1tiYfDcA.jpeg" /><figcaption>Pavel Kostin, E-commerce and personalization expert, ex-General Manager, Exponea.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0c559d5ec27" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>