<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Eyal Michael Cohen on Medium]]></title>
        <description><![CDATA[Stories by Eyal Michael Cohen on Medium]]></description>
        <link>https://medium.com/@eyalmc?source=rss-3f9ba2cdb7b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xb4IiVjD4Wg7TM1bTN507A.png</url>
            <title>Stories by Eyal Michael Cohen on Medium</title>
            <link>https://medium.com/@eyalmc?source=rss-3f9ba2cdb7b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:27:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eyalmc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How update messages push data-driven companies to wrong conclusions]]></title>
            <link>https://medium.com/@eyalmc/how-update-messages-push-data-driven-companies-to-wrong-conclusions-d586e61399cc?source=rss-3f9ba2cdb7b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d586e61399cc</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[noteworthy-stories]]></category>
            <dc:creator><![CDATA[Eyal Michael Cohen]]></dc:creator>
            <pubDate>Wed, 11 Oct 2017 12:40:16 GMT</pubDate>
            <atom:updated>2017-11-08T14:12:41.520Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*jDLrLOF-Pbxbghj67KcZ9g.png" /></figure><h4>And how a Product trick can fix a Data problem</h4><p>This is the first in a series of posts on how data-driven companies end up drawing wrong conclusions about their products. Yes, even companies with highly skilled, well-intentioned Product and Data teams. It is the area of overlap where danger lurks.</p><h4>Featuring:</h4><ul><li>One sneaky data bias that takes different guises.</li><li>A specific case where it is particularly likely to go unnoticed - when a company asks users for permission to update to the latest version.</li><li>The basic solution that works for most cases.</li><li>TOWEL: an advanced solution that offers a Product fix to a Data problem.</li></ul><blockquote><em>👋🏾 Get to know the people and ideas shaping the products we use every day. </em><a href="https://usejournal.typeform.com/to/YDVRgX?utm_source=usejournal.com&amp;utm_medium=blog&amp;utm_content=eyal_michael_cohen&amp;utm_campaign=guest_post"><strong><em>Subscribe to Noteworthy</em></strong></a><em> — the product &amp; design newsletter written by the </em><a href="https://usejournal.com/?/utm_source=usejournal.com&amp;utm_medium=blog&amp;utm_content=eyal_michael_cohen&amp;utm_campaign=guest_post"><em>Journal</em></a><em> team.</em></blockquote><p>The age of data-driven product development is upon us. Managers actively seek data on which to base their product decisions and business strategies, as opposed to relying chiefly on their experience and intuition.</p><p>Top goals are expressed as measurable KPIs<em>*</em> and intensive A/B testing is carried out to meet these, not to mention the data-science teams producing complex prediction models based on state-of-the-art machine learning techniques.</p><p>Yet in this data-fueled utopia it is not uncommon for companies to get the very basics behind scientific experimentation wrong.</p><p><em>* If you are unfamiliar with these concepts, or wish to refresh your memory, the box below is for you.</em></p><pre><strong>Refresh your memory: core concepts in data-driven decision-making</strong></pre><pre><a href="https://en.wikipedia.org/wiki/Performance_indicator">KPI</a> (Key Performance Indicator): A data-driven company tracks the progress of a product over time using core metrics that reflect on its business success. Popular choices revolve around growth in new users, retention of existing users, and revenues.</pre><pre><a href="https://en.wikipedia.org/wiki/A/B_testing">A/B test</a>: To compare alternatives, the company conducts experiments according to the scientific method. In particular, every new version of the product will be tested against the previous version in a controlled experiment, where users assigned to the experiment group are served with the new version and users assigned to the control group the previous version (<em>side note: independent features released in the same version may be tested in independent experiments, but since this makes no difference in our case let us assume one experiment per version</em>). To conclude which version is more successful, the company compares their performance according to (at the very least) the KPIs. The size of the experiment group is often limited to a small fraction of the user base, to minimise the damage in case it proves a flop. The purpose of an experiment is therefore dual-fold: (1) provide protection against unsuccessful versions and (2) draw valuable insights about the product and its user base, for planning of future versions.</pre><p>Update messages provide a useful case study.</p><p>Many companies have a policy to require the user’s consent to update to a newer version.</p><p>This is a considerate approach towards users, who may not wish to deal with the woes of major changes at this time, or any time. In fact, when users are paying a handsome sum of money (a common case with enterprise users) it may even be part of the <a href="https://en.wikipedia.org/wiki/Service-level_agreement">SLA</a>.</p><p>As innocuous as these update messages are in themselves, they can lead data-driven companies astray, fooling them into drawing wrong conclusions and making possibly harmful decisions.</p><p>To see how this tragic state of affairs comes about, let us look at the naive and straightforward method a data-driven company might use to deduce whether a new version is performing better than its predecessor:</p><ol><li>A new version is ready for release, including major changes to the user experience. For such a version, the policy is to ask existing users of the product for permission to update. The company will probably not risk offering the update to all users, but rather to a randomly selected subset of them. Only these users will receive the update message.</li><li>To determine whether the new version is an improvement compared with its predecessor, the company measures the effects of each version by computing the KPIs (core metrics) separately for users using each of them. Once enough data is gathered to reach conclusions, the winner is declared.</li><li>If the new version is deemed superior, it is offered appropriately to all users, and the Product Manager buys the team a round of celebratory beers. Otherwise, the new version is reverted (if the policy allows it), and the PM buys everybody a round of commiseratory gin and tonic. Being a PM can be quite expensive in companies on a short release cycle.</li></ol><p>Can you see what is wrong here?</p><h3>Self-selection bias</h3><blockquote>Not everything that resembles an A/B test is an A/B test. Treat it as such and you are in for trouble.</blockquote><p>Perhaps the reason it feels natural to compare the groups of users as described above has to do with the similarity to <a href="https://en.wikipedia.org/wiki/A/B_testing">A/B testing</a>.</p><p>In a typical A/B test users are randomly selected to one of two (or more) groups and served with the appropriate version accordingly.</p><p>However, the scenario above differs from an A/B test in one crucial parameter.</p><p>What the company missed is the importance of comparing the different experiences of <strong>similar</strong> users.</p><p>When two or more user groups are selected at random, user traits (and combinations of traits) are evenly distributed between the groups.</p><p>This means that given enough data, there will be roughly the same number of French-speaking users in each group (or Internet Explorer users, early adopters, risk takers..).</p><p>This in turn implies that any differences in measurements almost surely result from the different experience users had encountered, and not from traits inherent in the users of any of the groups.</p><p>When comparing a group of users who chose to update to the latest version to those who declined the offer (or were not even presented with the update message), it could well be that the differences in measurements result from inherent differences in user traits.</p><p>Perhaps users who accept are more tech-savvy? Perhaps they are more critical?</p><p>And if so, to what extent can we expect this to bias the results? We simply cannot know how much of the effect reflected by the results should be attributed to the new <strong>experience</strong> and how much to <strong>user traits</strong>.</p><p>If we inadvertently attribute the results to the experience, we will draw conclusions that may lead the company to harmful courses of action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*3suPV9R6vQPWsvMd1UzI7Q.jpeg" /></figure><p>This scenario is affected by what is known as <a href="https://en.wikipedia.org/wiki/Self-selection_bias">self-selection bias</a>. Whenever users are divided into groups according to their conscious decision whether to participate in an activity or not, there exists a considerable risk of bias.</p><p>This is also the case when analysing data gathered from beta testers, or from users who choose to answer an email survey.</p><p>Most PMs see right away that whereas beta testing is a fantastic tool for surfacing critical bugs and usability issues, it is not a reliable indicator: the success of a version with beta testers may not be representative of the mainstream market.</p><p>However, self-selection bias which takes a sneakier guise such as in our scenario flourishes throughout the industry.</p><p>Another “classic” that proudly presents this same notorious bias occurs when companies try to conclude whether a feature that has not been A/B tested upon release<em>**</em> benefits users.</p><p>Users are divided into two groups according to a single parameter: whether or not they have used the feature (what accounts for “usage” is up to the company to decide - it can be recurring, one-time, or even just enabling the feature in a settings menu).</p><p>The KPIs are computed separately for each group and then compared. If users of the feature perform better, the company will conclude that this is thanks to the different experience these users have been through and triumphantly declare that the feature benefits users. The other explanation, that of users of the feature having a completely different set of traits to begin with, is often overlooked.</p><p><em>** One common reason for not A/B testing a feature is having to release it to all users as part of a media event.</em></p><p>Alrighty then. So what should the company do?</p><h3><strong>When in the Science Lab, do as the Scientists do</strong></h3><p>Conduct an A/B test, of course. A proper one, though, with randomisation and everything. Not some biased lookalike.</p><p>This is probably what the company would have done in the first place had it not been bound by its policy to require users’ consent to update.</p><p>But how can randomisation and explicit consent live together side by side? They cannot, but they can live happily ever after one following the other.</p><p>The simpler method is to place the A/B test before the update message. First, randomly assign users to either the control or the experiment group. Then, users in the experiment group will be presented with the update message. Some will update to the new version, others won’t.</p><p>Meanwhile, users in the control group will go on using the old version, oblivious to the whole experiment.</p><p>The secret to avoiding the self-selection bias is to compare the control group to the experiment group en masse, users of both the new and the old versions bundled together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*aJXwC4VZDfqfNSPI_Ycrsw.png" /><figcaption>Compare apples to apples, not to bits of apples.</figcaption></figure><p><em>“Now wait a minute”</em> I hear you cry, <em>“it can’t be that simple!”</em>. And indeed, it isn’t. But it’s almost that simple. The only problem with this method is the “noise” added by the users in the experiment group who reject the update. This means more data is required to reach statistically significant conclusions compared with the alternative method I shall present next.</p><p>To save you (and me) the agony of college statistics, let’s skip a mathematical proof of this. Instead, below is an example calculation to give some sense of it. If you are not into figures right now feel free to skip to the next paragraph.</p><pre><strong>Calculating the number of users required for the experiment</strong></pre><pre>In this example, assume the following:</pre><pre>1. If presented with the update message, 40% of the users would accept it, 60% would reject it.</pre><pre>2. The KPI in question has a 5% conversation rate (e.g. 5% of users submit a purchase on a weekly basis in the old version). This breaks down to an 8% conversion rate among users who would hypothetically accept (those above the dashed line in the apples illustration), and 3% among hypothetical rejectors: 40%*8% + 60%*3% = 5%.</pre><pre>3. When measuring the impact of the new version, we are interested in being able to detect a 5% effect (increase or decrease) on the KPI at a level of certainty of 99%. Less than that is of no interest to us. But the experiment group contains a mix of 40% users of the new version and 60% users of the old version. Therefore, the observable effect on the entire experiment group is diluted down: (40%*8%*105% + 60%*3%) / 5% = 103.2%. This means that if the new version has a 5% effect on the KPI only a 3.2% effect will be observed in the experiment group.</pre><pre>We can now use a <a href="https://www.optimizely.com/sample-size-calculator">calculator</a> to figure out how many users are required for the experiment. The inputs are:<br> * <em>Baseline Conversion Rate</em> = 5%<br> * <em>Minimum Detectable Effect</em> = 3.2%<br> * <em>Statistical Significance</em> = 99%<br>It comes out we need <strong>390,000 users</strong> in each of the A/B test groups.</pre><pre>For comparison, the inputs for the alternative method are:<br> * <em>Baseline Conversion Rate</em> = 8%<br> * <em>Minimum Detectable Effect</em> = 5%<br> * <em>Statistical Significance</em> = 99%<br>which yields that only <strong>85,000 users</strong> are required for each group.</pre><p>As imperfect as this method may be, it suffices in many situations. You can use an <a href="https://www.optimizely.com/sample-size-calculator">A/B test sample size calculator</a> to check whether your user base is large enough to apply it. When feeding the <em>Baseline Conversion Rate</em> take into account both the conversion rate of the update message and that of the target KPI. If this method turns out impractical in your case, don’t panic. Just don’t forget to bring a TOWEL.</p><h3>TOWEL: a Product fix to a Data problem</h3><p>Normally, data issues are handled by the Data team. Biased data is cleansed to the extent possible and analyses results are cautiously interpreted with the assistance of error estimation models. But this rocky road is no force majeure. With the right product tweaks, certain data issues can be avoided altogether.</p><p>Such is the case with the alternative method, where the update message is placed before the A/B test. First, present all users with the update message. Only those who accept shall take part in the experiment. Then, randomly assign these users to either the control or the experiment group.</p><p>Ideally, we would like to give users in the control group the equivalent of a placebo, that is: serve them the old version and pretend it is the new one. But alas, there are no black-box pills in the realm of software, at least not when it comes to versions with changes to the user experience. Existing users will not be fooled by such pretence (they will, however, get rightfully annoyed).</p><p>This occasion calls for a trick I have sentimentally named TOWEL, short for</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*tUGyFT-pPWen4SReF9Y-FA.jpeg" /><figcaption><em>Stop telling me the ‘E’ is missing. It’s an E-List, alright? Geez..</em></figcaption></figure><p>It works like this: users who accept the update message are prompted with</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*1ONF65SzWKF4fpUZB3TwSg.png" /></figure><p>Those in the experiment group stay on the waiting list for a short while (say, a minute, or until the start of the next session) and then presented with</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*jSxhy3hT4ldcKI_Ixbya-w.png" /></figure><p>and are given the new version.</p><p>Now comes the twist: users in the control group, however, encounter a completely different fate. They are put on the “<em>other”</em> waiting list, where they will blissfully stay until enough data has been gathered to conclude the experiment. In the “meantime”, they go on using the old version.</p><h4><strong>Interpreting the results</strong></h4><p>Let us look at how the experiment results are to be interpreted. First of all, note that only users who accepted the update message participate in the experiment, thus eliminating the original self-selection bias. We compare the performance of the experiment and the control group. Since users have been split randomly into groups, we assume they possess roughly the same traits. Therefore, all differences in measurements can now be attributed to the different experience they received. Voilà!</p><p>But wait, the different experience users received is almost, but not entirely, limited to the differences between the new and the old versions. The other difference is that users in the control group were promised a new version and then evasively denied of it. If they end up feeling deluded (or on the contrary, excited about the prospect of an upcoming version) then this may affect their experience as a whole. The key is to minimise the mental impact TOWEL has on users by making your messages as agreeable to them as possible.</p><pre><strong>Naturally cautious? Verify TOWEL is well accepted by your users:</strong></pre><pre>Personally, I believe (for what it’s worth) that for most user bases the bias introduced by TOWEL is markedly smaller than the original self-selection bias. If you are worried, you can verify it on your user base using the following experiment:</pre><pre>1. Take a feature that has failed to make any impact on your KPIs in a recent experiment (hereinafter experiment A). Create a new version that is simply the current version plus the benign feature. We expect this new version to perform the same as the current version.</pre><pre>2. Conduct an experiment B, on a set of users <strong>different from that of experiment A</strong>, in which the <strong>control</strong> group gets the update message, then put on a short waiting list, then given the new version (same as the experiment group in TOWEL), and the <strong>experiment</strong> group is put on the indefinite waiting list (like the control group in TOWEL).</pre><pre>3. Compare the KPIs computed for the control and experiment groups of experiment B. Significant differences suggest that TOWEL is not well accepted by your current user base.</pre><h4><strong>Tying up loose ends</strong></h4><p>When a new version turns out a success you want to offer it to all users so they can enjoy its benefits. This also has the added value of allowing you to stop supporting older versions and clean up your code-base.</p><p>Unfortunately, you cannot simply apply the conclusions to all users. Your experiment was conducted on <strong>existing</strong> users who <strong>accepted</strong> the update offer. New users, or those who declined the offer, may hold entirely different traits and may thus react completely differently to the new version.</p><p>So what are you to do?</p><p>As for new users, this is easy. There is no policy to respect in their case, so simply run an experiment with them.</p><p>As for the users who declined, not being able to make the analogy is your lesser problem.</p><p>Even if you could conclude that they would benefit from the new version, you would not be able to forcefully foist the update on them without breaking your policy anyway.</p><p>When such a policy is in place (and as mentioned earlier, if you wish to make business in certain markets you may not have much choice about it) then the immediate outcome is having to support older versions and maintain legacy code.</p><p>If, however, the new version turns out unsuccessful, you will not eventually be releasing it to users in the control group, despite your promise. What should you do then?</p><p>The first option is to do nothing unless a user contacts your support team. In such case, I suggest you simply tell him the truth: that the version did not prove itself and will not eventually be rolled out. Alternatively, when a future version proves successful in an experiment (on a different set of users) you can roll it out to these users and deliver on your original promise.</p><h4><strong>OK, self-selection bias is a problem. But how bad is it in <em>my</em> case?</strong></h4><p>The magnitude of the effect of the bias depends on your user base at a certain point in time. To get an estimate, do the big no-no and compare your control group to the group of users who declined the update offer.</p><p>If you accept the postulate that TOWEL has hardly any effect on users in the control group, then you can relate any differences between the control group and the users who declined to the inherent difference in traits between these user groups.</p><p>If the effect of the bias comes out negligible you may choose not bother yourself and your users with TOWEL again, but beware: trends change quickly in the world of software and with them so does your user base. Even if your users demonstrate practically no self-selection bias, it does not guarantee this will still be the case months from now.</p><p><strong>Special thanks</strong> to Roy Degani for his meticulous and excellent feedback.</p><blockquote><em>👋🏾 Get to know the people and ideas shaping the products we use every day. </em><a href="https://usejournal.typeform.com/to/YDVRgX?utm_source=usejournal.com&amp;utm_medium=blog&amp;utm_content=eyal_michael_cohen&amp;utm_campaign=guest_post"><strong><em>Subscribe to Noteworthy</em></strong></a><em> — the product &amp; design newsletter written by the </em><a href="https://usejournal.com/?/utm_source=usejournal.com&amp;utm_medium=blog&amp;utm_content=eyal_michael_cohen&amp;utm_campaign=guest_post"><em>Journal</em></a><em> team.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d586e61399cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>