<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by David Mannheim on Medium]]></title>
        <description><![CDATA[Stories by David Mannheim on Medium]]></description>
        <link>https://medium.com/@davidleemannheim?source=rss-c2550b94b3c9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-ZfWL50fQtY-1ZKmrV2RIg@2x.jpeg</url>
            <title>Stories by David Mannheim on Medium</title>
            <link>https://medium.com/@davidleemannheim?source=rss-c2550b94b3c9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:03:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@davidleemannheim/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Experimentation platform selection is really, really hard]]></title>
            <link>https://davidleemannheim.medium.com/experimentation-platform-selection-is-really-really-hard-7991792467ad?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7991792467ad</guid>
            <category><![CDATA[ab-testing-software]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 30 Jul 2021 06:02:12 GMT</pubDate>
            <atom:updated>2021-07-30T06:02:12.432Z</atom:updated>
            <content:encoded><![CDATA[<p>A message to all businesses out there looking for experimentation vendors: I feel for you.</p><p>The selection available is unreal. It’s actually the same issue with most SaaS products. Not just in the sense of “I want a, say, behavioural analytics tool — which one do I go for?” but even in the sense of “what martech platforms should I have that make up my tech stack?”</p><p>It’s insane. It’s <a href="https://chiefmartec.com/2020/04/marketing-technology-landscape-2020-martech-5000/">7,040 levels of insane</a>. That’s how many martech platforms there are to pick from in 2020. Over 200 of which are classified as optimisation, personalisation and testing.</p><p>I’ve been involved in a lot of these discussions because of the breadth of experience I’ve personally had with each one, it tends to be the number one question I get asked. “Which one is best”. Scratch that. It’s definitely the number one question. So much so I even wrote a <a href="https://www.userconversion.com/wp-content/uploads/2020/08/How-to-select-an-experimentation-platform.pdf">white paper</a> on this very approach and subject with Damian at User Conversion.</p><p>That’s not me tooting my own horn, but I feel I’ve learnt one significant truth with these platforms.</p><blockquote>They’re all very samey with samey features packaged up in a less-so-samey way. Some have more features than others, some better features than others. But the understanding of your business challenges is the most important thing to communicate when in a vendor selection process. It allows those vendors to respond with contextually-specific, persuasive answers on how their platform can support your journey.</blockquote><p>Otherwise, what I often see happening is that the organisation will ask for a list of features. Too many. Non-prioritised because the list is a democratic request. Vendors will have 90% of them. Vendors will then seduce the organisation with sexy features that weren’t asked for. Ahem, personalisation. We then end up in feature-tennis. This over-stimulation of feature wars means that — really — the only differentiation, then, either becomes price, how many sales people the vendor has to pester you.</p><p>It’s harsh but true.</p><p>This isn’t dissing those platforms. Actually, I want to help both sides of the party. If anything, this is a plea to businesses selecting an experimentation platform. It’s a difficult market to navigate and I really feel for you. Wade through that sea of sameness. You can create platform differentiation by thinking about selecting a vendor in the same way you’d select a consultancy. Provide your challenges with context and allow the vendor to respond on how their platform specifically can help solve those challenges.</p><h3>What are my primary vendor considerations</h3><p>Here are the list of questions I ask a business when trying to understand what platform will be best for them.</p><p><strong>1 — Definition: Why do you experiment?</strong></p><p>I think this is one of the best questions to ask. Not only does it make people think about the why this vendor selection process is happening, but it’s a broad enough question to set the scene.</p><p>Truly understanding the purpose of experimentation.</p><p>Whilst the evangelists will state the true purpose of AB testing is validation, the <em>real</em> purpose might be something different. Increase conversion rates. Personalise behaviours. Marketing promos. Bypassing development (yes, it’s a thing). Creating a test and learn culture to…tick a box (yes that’s also a thing)</p><p>This can influence your decision in which platform. Some focus on their easy to use UI for the non-coders out there. Some have a commercial arrangement based on volume of usage meaning that you should understand how many tests you’re producing now and what that looks like in 12 months, too.</p><p><strong>2 — People: Who experiments now and who do you want to experiment in the future?</strong></p><p>Is your experimentation process centralised or decentralised? Does it sit with marketing, product or engineering? Knowing the resources available, their skill sets can help you understand the maturity of platform that you’re after. Decentralised model where engineers are heavily involved and running hundreds of tests a month? You might want a platform that’s slightly more enterprise, has great permission allowances and a versioning system within the code editor. That’s more important than, say, a really nice UI to a WYSIWYG editor.</p><p><strong>3 — Process: What does your experimentation process currently look like?</strong></p><p>Do you have an experimentation platform currently — what are its limitations since using it and why was it selected? Is it centralised by one team, or decentralised in product teams? How mature is your experimentation program currently? ← I think the <a href="https://www.optimizely.com/maturity-model/">Optimizely maturity quiz</a> is the best to answer this.</p><p><strong>4 — Technical: What does your tech stack look like and how do you want to experiment</strong></p><p>Some companies might be embedded in the Adobe stack. Or completely wedded to a behavioural analytics tool like Content Square or Segment. Or run everything through JIRA. What are the integrations <em>really</em> like with the platforms? And yes, everyone offers these integrations more or less, but give me examples and detail of how to make our life easier.</p><p>Not to mention how the site is built. Clean code bases. Site structure. Underlying technologies — for example React is a beast to experiment within and some platforms from experience handle it much better than others.</p><p>Equally, where experimentation sits (product vs marketing vs engineering, centralised vs decentralised) and how tests are created are vital. Is server side required? Who builds tests — do we need marketing to add banners to bypass development (I know, I know, sad face). But if so, we need a simple WYSIWYG editor — some being far better than others because UI is an easy differentiator.</p><p>If you’re not on the technical side and don’t know most of what I’m talking about, sit down with your engineers. You’ll waste a lot of time in your selection process if your chosen tool ultimately won’t work with your existing stack — and in many cases, it might not. But, a word of warning, I’ve seen engineering teams “take over” the process. They should be heavily involved, especially if that’s where experimentation sits. But democratising experimentation where most if not all evangelise it, is the only route to success.</p><p><strong>5 — Personalisation: How important is user or feature personalisation?</strong></p><p>Everyone will say it’s super important, we know this. But everyone will also have different definitions of personalisations, too. My first recommendation is to align in <em>your</em> definition of personalisation (and please don’t say 121).</p><p>As a pre-requisite it’s vital to understand the maturity of the business currently, and the purpose of experimentation before determine how <em>really</em> important it actually is. So more detailed questions like of the tests that you run currently, how many are ran at a segment level? What are the OKRs and vision for the business in the next 3 years and is personalisation part of that strategy? Trying to test the company to see <em>how</em> important personalisation really is, apart from just lip-service.</p><p>As a bonus, I’d also consider the level of customer support or service you think you’ll need. There’s technical support, sure, but what about ‘strategic consultancy’. I’d be wary of any vendor offering consultancy support — that’s a whole topic altogether. And I won’t even claim to be able to articulate it better than Steven Pavlovich’s “<a href="https://conversion.com/blog/managed-service-sucks/">Managed Services Suck</a>” musings. How much and what type of support you need as a business depends on your maturity, resources and skill set. Inform the vendor of this, contextually, to allow them to respond appropriately note: not upsell.</p><h3>What type of vendors are there?</h3><p>We categorise vendors into 4 segments. Some platforms naturally straddle categories. Some platforms want to categorise themselves as all 4 (!) (I’ve had those conversations, they’re not pretty) There will be some cross-over. And therefore, this isn’t hard and fast. But I think what these categories do is try to link your <em>maturity</em>, <em>ambition</em> and your <em>tech architecture</em> to a category.</p><ul><li>Enterprise Tech Stack. These are your Adobes (Target) and your Oracles (Maxymiser) and perhaps your Googles (Optimize), where being embedded within a tech stack is super important and efficient</li><li>Pure and Mature Experimentation. The platforms that are pure players within the market</li><li>Wider experimentation suites. There’s a bit more than just experimentation here, it might be heatmaps, or session recording functionality.</li><li>Personalisation suites. These platforms pride themselves on a personalisation proposition and are more pure-play in this area.</li></ul><p>I think it’s important to highlight the personalisation suite as independent because, as experimentation becomes a broader need in the market, so too does personalisation. The two are different, albeit inextricably interlinked. I’ve had conversations with companies where personalisation suite X is awesome at personalisation, but not as good at experimentation as experimentation platform Y. So what’s more important to you? There are companies I know that have a personalisation suite and an experimentation platform. Again, we come back to the ‘purpose of experimentation’ for your business.</p><p>I’ll also just mention the trend of experimentation platforms becoming more CDPs (customer data platforms) that give them this ability to hold data, and personalise on it’s behalf. We have the acquisition streak of KIBO with Monetate, or EpiServer with Optimizely, or Oracle’s Unity platform. We even see the other way around with Exponea, a CDP, introducing their experimentation platform. I think this is an interesting set of moves, making AB testing more involved</p><p>If you’re after more information you can read the whitepaper we wrote <a href="https://www.userconversion.com/wp-content/uploads/2020/08/How-to-select-an-experimentation-platform.pdf">here</a>.</p><h3>If I were you…</h3><p>If I were you, I’d whittle down my choices to &lt;5 vendors. Ideally staying within the categorisation structure above based on your maturity, ambition and tech architecture.</p><p>Base your decision from similar companies that you’re aware of, or go to <a href="https://www.brainlabsdigital.com/">agencies</a> or consultants that can offer, mostly, impartial advice. Note: it will likely be different than the format above, this is just my experience. Also note; some agencies do receive a kick back from said vendors so do ask prior to remove potential bias.</p><p>Then I would write the RFP as a series of problem statements. Give the vendors context and challenges to solve. Think about questions framed like the below, encouraging long for, descriptive responses.</p><ul><li>Our code base is react and we’re looking at optimisation experiences for our users across web and app. We have an agile, disciplined development process and need to ensure any tech is fully aligned with the engineering department goals, without hindering site speed. How will your platform impact our site and how might we prevent daily conflicts with our releases?</li><li>We’re just starting our experimentation journey. We have dabbled with a few tests in Google Optimize and have ambitions to run &gt;100 tests in the next 12 months. We have 1 x CRO manager, a data analyst and an engineering team of 8. How can you help us grow this velocity and evangelise how AB testing can help us to our senior management team?</li></ul><h3>Summary</h3><p>I feel like I’ve been a bit unfair on experimentation platforms. All I’m trying to do is help based on my experience. And that might involve me revealing some ugly truths. Even more so if you read that bonus section below. And it will almost definitely mean I get in trouble. But what the heck. I care more about supporting our industry through justice, more so than being trolled.</p><p>I’m trying to help create a clear method of differentiation, because. frankly, currently, there is very little in a saturated market. We need to allow businesses to make a more informed, less-bias decision on which platform is best for them; to do that we must explain how your tech can help solve their individual challenges. For that to happen, businesses must provide context and explain what those challenges are. The same goes for any vendor, not just experimentation. Indeed, the same goes for agencies. The more information you can provide, the better. A 30 minute phone call wont suffice.</p><p>I don’t think it’s rocket science, nor difficult. I just feel we get mesmerised by the overwhelming amount of features available and the price conversation. There should be some form of regulatory or, at least, standardised method of selecting a platform.</p><p>In my opinion, to do that, we need to help these vendors by telling them what the actual challenges are with the businesses. That way they can truly differentiate by explaining — not just that they have a WYSIWYG — but how that can help assist the marketing team because of it’s ease of use, and easy to select divs. Or not just that they have permissions and version controls, but how businesses who have a centre of excellence can easily review the code or test set up of their product teams and revert back if necessary.</p><p>Challenge vendors by giving them enough context to help you.</p><h3>Bonus: The problem with price</h3><p>I didn’t know whether to write this. It’s a tangent from the above and I think it’ll get me in trouble. But screw it. It’s important.</p><p>Price is often a core differentiator between all vendors (not just experimentation, but particularly so). But I often feel as though pricing models are inconsistent. And it would appear I’m not alone in this statistically insignificant and probably bias’d poll…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B9EgnHE2gu_VUbFKBWS4sg.png" /></figure><p>Business development teams are commissioned per sale; it’s in their personal interest to sell you their platform. And I’m not saying this happens — but I can see it being at the expense of what’s right for you as a business. That’s a whole new conversation on greed I won’t entertain publicly. Yet.</p><p>I do find it incongruent that for the same service price differs — a lot. I’ve seen companies been quoted £300k p.a. and then suddenly, because it’s the end of the quarter, they’re quoted £200k p.a. Or for sexy brand names and case studies, the price suddenly drops by £40k. Or as soon as the competition heats up for a brand, vendors will willingly match others. There seems to be a lot of give in the price. I was even privy to one discussion where just 30 minutes before a big meeting with a prospect, the vendor made up the figures with the rationale “they can afford it”.</p><p>Some pricing models are based on performance. Some on usage. Both of which I find bizarre.</p><p>I also find it hard to accept that all platforms want to move more “enterprise” because at the end of the day that’s where the money is. I won’t name names, but there are at least 12 platforms I can name that have moved away from their entry price proposition, toward a more expensive proposition. This usually happens when they have had external investment in their business and forced to make more profit — but how can they seemingly justify doubling prices from one year to the next?</p><p>It happens. Unethically so. This isn’t knocking platforms. Much. Yes, I’d like to see some of their methods disrupted, but they do have a job of promoting and selling their platform. I just feel that sometimes those methods can swerve into defamation given the experimentation market is more about substitution, rather than adoption — especially when it’s as price orientated as it is.</p><h3>Don’t forget</h3><ol><li>Please feedback on this article — was it helpful? Awesome? Crap? <a href="https://forms.gle/eX9ztvxDXSdFqJ9CA">https://forms.gle/eX9ztvxDXSdFqJ9CA</a></li><li>Ask questions and vote <a href="https://app.sli.do/event/orj9eqab/live/questions">https://app.sli.do/event/orj9eqab/live/questions</a>. I answer these on a weekly basis by taking the top voted topic and writing about it with my stories and experience.</li><li>Subscribe to my newsletter at <a href="https://optimisation.substack.com/">https://optimisation.substack.com/</a></li><li>Leave a comment. Create debate. Have a drink.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7991792467ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimenting for versioning, visioning and venturing forms of innovation]]></title>
            <link>https://davidleemannheim.medium.com/experimenting-for-versioning-visioning-and-venturing-forms-of-innovation-331d1cbb149b?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/331d1cbb149b</guid>
            <category><![CDATA[innnovation]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 16 Jul 2021 06:24:41 GMT</pubDate>
            <atom:updated>2021-07-16T06:24:41.586Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve really been torn recently. And found it hard to articulate my paradox.</p><p>On one hand, I discussed in my last article, <a href="https://www.linkedin.com/pulse/does-experimentation-encourage-innovation-through-david-mannheim/">I believe that AB testing encourages innovation</a>. It can create a fountain of knowledge and lead us on a path of discovery. Perhaps that path is not the perceived intention, but that’s what creates ‘the original’.</p><p>In my musings before that though, I mentioned how <a href="https://www.linkedin.com/pulse/does-experimentation-inhibit-encourage-innovation-david-mannheim/">experimentation held limited value for innovation</a>. I just don’t understand how we can test and learn about user behaviours, when the users themselves are not ready for such an innovation? 🤷🏼</p><p>Sit on the fence, David.</p><p>I’ve been thinking about where this clashing of two minds is coming from.</p><p>I think it’s two fold.</p><ol><li>Innovation holds many different categories and is more a spectrum, rather than a binary definition</li><li>Our societal perception of what experimentation is designed to do is to tweak, iterate, move stuff round on a page — rather than truly innovate</li></ol><p>I think innovation needs a level of categorisation to understand whether experimentation is better for some categories over others. So we’ll explore that.</p><p>But I also think, that within those categories, especially in ecommerce, experimentation is pigeon-holed. It’s poster-boy’d and seen as incremental ‘improvements’, rather than innovative leaps forward. Stuck in a world of moving images from the left to the right, or making fonts bigger or small. In this world, cyclical case studies create a perception of experimenting for the tweaking and versioning, rather than the bold ‘go where no person has gone before’ venturing.</p><h3>WTF is innovation</h3><p>Innovation is many things; not one thing.</p><p>I’ve previously explained how the economist David Galenson came up with two different types of innovators; the conceptual and the experimental. The conceptual are the strokes of genius you come across once in a century. Where the experimental, the more common of the two, is the type of innovation that occurs over time from learning through trial-and-error. Sounds familiar doesn’t it, experimenters?</p><blockquote>There are two very different types of artists: those I call Old Masters, who work by trial and error and tend to improve with age, and conceptual people, or Young Geniuses, who generally do their best work early in their careers.</blockquote><p>I began to wonder if we can expand on Galenson’s categorisation of innovation. And whether experimentation plays a bigger role in one type of innovation more than others.</p><p>Have you ever seen Kevin Dame’s, Head of Product at Youtube, talks about <a href="https://www.youtube.com/watch?v=fMCc89kO5BI">versioning, visioning and venturing</a>? It’s fab, watch it.</p><p>I wonder if, and how, experimentation has a place within each of these categories? Does it hold more value in versioning than venturing?</p><p>I think we think it does because of how the industry, and society, treat experimentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*MYTejVa8VLhY_3GJn1rIlw.png" /></figure><h3>Versioning</h3><p>Experimentation is complex. So to make it more accessible (and to sell their services!), those companies that advocate it — the agencies, the SaaS platforms, the communities — create case studies that resonate by making it seem….simple.</p><p>That’s needed, I get it. But in doing so, over-simplifying this methodology has led us to believe that the purpose of experimentation is one of moving shit round on a page to increase a metric. Usually a conversion rate. Hence the term “conversion rate optimisation”.</p><p>Enter versioning. The notion of incremental improvements to a behaviour or aesthetic. The Kindle is now on its seventh generation. Samsung Galaxy on its 21st. iPhone on its 13th.</p><p>There’s nothing wrong with this, but I think that we think experimentation holds the greatest place here; within versioning. I reiterate, we think it thrives here. Because of how case studies, communities, and success stories talk about it.</p><p>This type of trial-and-error approach of validation is exactly what David Galenson was talking about when he was advocating the experimental innovator. That’s fine, but there are other types of innovation of which experimentation are suited to; I just feel this is the most popular because it is, by virtue, the most accessible.</p><p>And, for reference, versioning is a necessity. And experimentation to support that is also a necessity.</p><p>Take Netflix. They have been working on the <a href="https://askgib.substack.com/p/a-brief-history-of-netflix-personalization">hypothesis of personalisation since 2000</a>. They introduced Cinematch (a recommendation system where member ratings were used to predict how much a member would like a movie). But their evolution of the same hypothesis has seen multiple iterations, in fact, from (to name a few)</p><ul><li>2006 — Ratings Wizard with a “Movies You’ll Heart” tab</li><li>2009 — collaborative filtering algorithms focussed on popularity</li><li>2009 — category match personalisation (because you watched X, you’ll like Y)</li><li>2011 — AB tested personalisation vs no personalisation (the former won)</li><li>2012 — Multiple profiles for individual personalised recommendations</li><li>2016 — floating rows for a personalised UI</li><li>2018 — personalised movie art</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kScprUWBGZeADU5puZzyVg.png" /></figure><p>Netflix innovated by experimenting by versioning their recommendations system. Some failed, some became anachronistic. But they did it.</p><p>Ironically, I find when it comes to ecommerce, rarely do they version with their experiments. Experimentation is seen as versioning, but, in my experience, it stops at version 1.1, and rarely moves beyond that.</p><p>Despite community companies distorting our lens towards “this is how experimentation should be”, I often find that when we, as optimisers, experiment and reach a conclusion, we give it a binary “win” or a “lose” label.</p><p>This isn’t innovation.</p><p>Taking that concept, we’ve worked with Flannels before and gave a talk on the power of iteration with AB Tasty previously. We discussed how some of our experiments were only really classified as ‘successful’ once they were iterated on because we learned from the behaviours of the previous incarnation. What we really should have spoken about is how they pushed our creative thinking. Look at the below — I’ve personally never seen filters within a product card designed to encourage filter usage before. That’s versioning. That’s iteration. That’s innovation. And all thanks to validating behaviours, and learning from them, within a series of experiments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUxjJxh7_iJZXmT-nDH1pw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mopYBU-kxroOpwjinlHgA.png" /></figure><h3>Visioning</h3><p>Visioning is moving beyond the 1% improvements for how we see our product in 2–3 years. What’s a leap in value we can create for our users?</p><p>I think there’s a different process required for visioning, then there is for versioning. And I wonder whether experimentation holds more of a common place in the former than the latter. Or, at least, again our perception is that experimentation works better in the former rather than the latter.</p><p>In versioning, we ask “what’s the quickest way we can get this out of the door? What’s an MVP to prove our hypothesis? What’s feasible? How can we seek out a 1% improvement?” These are experimentation mindsets.</p><p>Especially when we prioritise based on “time” (or effort) as an indicator, often the quickest thing to engineer is prioritised first and, adversely, that which takes the longest is prioritised last. So will we ever develop visioning ideas when our experimentation prioritisation frameworks are so biased against time?</p><p>Visioning requires someone to slow down. Take a leap for 10% improvements. And often, think more about the problem, rather than the solution. Sometimes at the cost of speed of production within versioning, comes a negligence in “am I <em>really</em> tackling the problem here, or am I just putting a plaster over it?”</p><p>Take the below example. YouTube creators wanted to promote donations to social causes they cared about. YouTube took a versioning mindset and quickly added information within the “i” icon that came with every video. Convergent thinking. This satisfied the hypothesis and creators saw an uplift in donations to their causes.</p><p>It wasn’t until a visioning mindset was taken where that feature really became its own and the user experience excelled. This was as a result of divergent thinking where they got a tonne of ideas together, honed in on the one they thought was best, validated that, and went with it.</p><p>The result was a 30x uplift in donations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p8eZLFJMYxzG9yxa-1XMqw.png" /></figure><p>Experimentation is absolutely perfect for such a situation, because it validates an understanding. That understanding might be assumed or seen as reaching for the stars, but the methodology helps you understand it’s plausibility.</p><h3>Venturing</h3><p>Venturing are the invention breakthroughs. The moonshot projects. Those that take 5–10 years to complete.</p><p>When I wrote “<a href="https://www.linkedin.com/pulse/does-experimentation-inhibit-encourage-innovation-david-mannheim/">does experimentation inhibit or encourage innovation</a>”, this is the type of innovation I was hinting at. Can we test ideas on our users when the users don’t know any different? Or when they aren’t ready? Where the innovation is so new the reticence to change outweighs or dilutes the potential of the idea. Think a “if I asked people what they wanted they would have said faster horses” moment.</p><p>I could argue experimentation doesn’t work as well here. But I would be wrong.</p><p>The benefits of validation (are we heading in the right direction) and discovery (learning from our experiments) is too great that, in my opinion, it outweighs the “I don’t know what I don’t know” argument. Again, similar to the visioning debate, experimentation is extremely powerful here for the purposes of discovery and validation; arguably more the former than the latter.</p><blockquote>Rowing harder doesn’t help if the board is headed in the wrong direction. Kenichi Ohmae</blockquote><p>All experimentation does is show us which direction to row in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcS3CgqbKwY7NAesDyLkVw.png" /></figure><h3>How experimentation is perceived amongst the many</h3><p>Experimentation works across all these categories of innovation.</p><p><em>I want</em> to say that experimentation thrives more in versioning. But whilst I want to say it, I don’t think that’s true. I think I’m forced to say it because of the societal perception of experimentation.</p><p>The real power of experimentation is one of validation and discovery.</p><p>But, from personal experience, in 90% + cases of how I see experimentation being used, it primary use is to change web design layouts, to tweak the user experience onsite or worse still, for marketing campaigns to bypass engineering teams.</p><p>Perhaps that’s because I live more in an ecommerce world. In this world, AB testing largely sits within a marketing function, rather than an engineering function.</p><p>Yet because of this bias towards small improvements experimentation is seen as better for versioning because it’s stuck there; for most companies. Only until we really understand what AB testing can give us can we use it for the purpose it was intended; for innovative leaps forward.</p><blockquote>“I’d spend 55 minutes defining the problem and 5 minutes solving it” Albert Einstein</blockquote><p>When the majority of experiment value is stuck within a world of versioning, the purpose of experimentation is adversely believed to be one of a solution. Really, the main value of experimentation comes from helping to define the problem itself.</p><p>In this respect, regardless of the innovation categorisation, experimenting is vital because it helps define the problem through validation and discovery.</p><h3>What’s next?</h3><p>In my next series of articles, I am going to dive a bit deeper into questions like:</p><ul><li><a href="https://www.linkedin.com/pulse/does-experimentation-inhibit-encourage-innovation-david-mannheim/">Does experimentation inhibit or encourage innovation?</a></li><li><a href="https://www.linkedin.com/pulse/does-experimentation-encourage-innovation-through-david-mannheim/">Experimentation validates creativity</a></li><li>Can a hypotheses limit creativity?</li><li>Experimenting for iteration and experimenting for innovation ← that’s this one, spoilers!</li><li>How can we be more creative within our process of experimentation?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=331d1cbb149b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does experimentation encourage innovation through validation?]]></title>
            <link>https://davidleemannheim.medium.com/does-experimentation-encourage-innovation-through-validation-4f1df2e7d82c?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f1df2e7d82c</guid>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 25 Jun 2021 05:56:38 GMT</pubDate>
            <atom:updated>2021-06-25T05:56:38.205Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>“Experimentation-driven innovation will kill intuition and judgement”​. Stefan Thomke</blockquote><p>This is the first of seven myths Stefan Thomke writes about in his book, Experimentation Works.</p><p>In my last <a href="https://www.linkedin.com/pulse/does-experimentation-inhibit-encourage-innovation-david-mannheim/">article</a>, I argued that experimenting could do the exact opposite. That it restricts creativity and innovation just as much as it fosters it. Why? Well, I think there can be inherent bias within validation itself, given that users might not know any better or be ready for said innovation. It could be a timing thing or a “I don’t know what I don’t know”​ thing. Give it a read.</p><p>Most disagreed. Although, I do wonder what would happen if my Linkedin connections were within fields of creativity rather than fields of experimentation. And yes I know experimentation is arguably a creative field but I not here to satisfy trolls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*S1qu00aNFWfcctiPaVM8GA.jpeg" /></figure><p>These guys aren’t wrong.</p><p>Experimentation fully supports innovation as a method for validation and exploration. It does not create it. I do truly believe that. And when I was writing this, I thought to myself “is this even worth writing about? People already know this. That much is clear from the comments of my last post alone”​.</p><p>But I believe in it. As much as I believe that it can, on occasion, inhibit innovation. There’s a place for intuition and a place for experimentation. What is that place?</p><p>To make it interesting, I’ve tried to back it up with a tonne of sexy examples from the commercial, economic and scientific world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d0A2ReIAfQQb_kCel6Mobw.png" /></figure><h3>Intro.</h3><p>“Experimentation supports innovation, it doesn’t create it”​. I just said that. But is it that simple to just say?</p><p>I think the complexities of this topic are within the semantics themselves; creativity and innovation. The economist, David Galenson, spent years studying this topic and came up with two distinct personas and definitions of <em>innovators</em>.</p><p>The first being <em>conceptual</em> — think Mozart or Galilei — where bold new ideas are pursued born out of genius or prodigy. The second being <em>experimental</em> — think Jobs or Wright Brothers — using a trial-and-error approach to innovate and create.</p><p>Interestingly, Galenson is known for saying that the latter approach of ‘experimental’​ is far more “interesting”​ and “common”​.</p><p>And there are hundreds of advocates and examples of where experimentation leads to innovation. <a href="https://www.vbivaccines.com/evlp-platform/louis-pasteur-attenuated-vaccine/#:~:text=Louis%20Pasteur%20in%201857.&amp;text=By%201878%2C%20Pasteur%20had%20succeeded,looking%20for%20safer%20inoculation%20methods.">Louis Pasteur’s discovery of artificial vaccines</a> came by experimenting with injecting hens with mild doses of cholera. <a href="https://www.ideatovalue.com/insp/nickskillicorn/2017/04/true-story-post-notes-almost-failed/">3Ms discovery of Post-It notes</a> came from a series of experiments developing a polymer-based adhesive.</p><blockquote>“I had sat down and factored it out beforehand, I wouldn’t have done the experiment. If I had limited my thinking only to what the literature said, I would have stopped”​ Spencer Silver [ref the Post-It note]</blockquote><p>But why and how does experimentation foster innovation?</p><h3>Failing.</h3><p>Failure breeds learnings. And learnings breed knowledge. And knowledge breeds innovation.</p><p>It’s important. In an experimentation context everyone states “there’s no such thing as a losing test”​. Yes yes, I know. You can’t get to adulthood without going through awkward teenage puberty.</p><p>And yes, there have been so many examples of note where innovation derived from failure. The lightbulb had 9,000 experiments before it worked. Slack was born out of a <a href="https://techcrunch.com/2019/05/30/the-slack-origin-story/?guccounter=1&amp;guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&amp;guce_referrer_sig=AQAAAIqzIB8MA9BFgeDDBVFE0iOc25qkmvAJtUdzIwm2EGM1rA34MgI6dL0KnXkdtdtT1ixyK57A_itpVZqehRI4K28xwtdDTO-QNMFJ5lD3QL5cj0Pm5uRQsp87WT4f1Ww__AoP3mQmrLbDzGBAYdNUOhBMX9H7vk8XaEWkWTpzyY0j">failed online multiplayer game called Tiny Speck</a>. Amazon Marketplace (which now accounts for 40% of revenue by the way) was <a href="https://www.businessinsider.com/amazons-jeff-bezos-the-business-insider-interview-2014-12?r=US&amp;IR=T">born from the Z Shops product</a> which was born from the failed Auctions product. WD40 was born from a series of experiments — <a href="https://www.wd40.com/history/">40 of them to be exact</a> — fancy that. Cordless vacuum cleaners from <a href="http://www.ideafinder.com/history/inventions/dysonvac.htm">Dyson took 5,271 prototypes.</a> Sounds like a competition, doesn’t it? Who can fail the most?</p><p>The concept of failure is a complex one. And I could cite something like the below from Andrew Stanton — which is beautiful both because it’s Pixar and authentic — but it’s not rooted in commercial environments where, I believe, stakeholders naturally fear failure. For reasons that are ego-driven. For reasons that they are human and therefore naturally avoid catastrophe. But also for reasons that commercial environments pulsate the economy and need for money is perpetual (*cough or greed).</p><p>So whereas ‘failing’​ is often cited as an aspirational state of mind, in a commercial environment, from experience, I often find it rarely is true.</p><blockquote>“My strategy has always been: be wrong as fast as we can. We’re gonna screw up, let’s just admit that. But let’s do it as fast as we can so we can get to the answer.”​ Andrew Stanton</blockquote><p>But failure is needed because to understand whether you’ve failed, you must validate and measure; and to do that you must experiment. So surely, innovation ‘from failure’​ only ever stems from an experiment in understanding, not necessarily failure itself. You must be willing to be wrong on your way to being right (Godin)</p><blockquote>“If I find ten thousand ways something wont work, I haven’t failed. Every wrong attempt discarded is just one more step forward”​ Thomas Edison</blockquote><h3>Practicality.</h3><p>Experiments are quicker. They get shit done. Galenson siad it himself, thinking the <em>experimental innovator</em> is much more common because of this level of practicality.</p><p>I got given a great article the other day (h/t to Ben Labay, which came from Simon Elsworth, which I think came from Matt Lacey), where Michael Schrage argues that <a href="https://hbr-org.cdn.ampproject.org/c/s/hbr.org/amp/2014/12/a-testable-idea-is-better-than-a-good-idea">a testable idea is better than a good idea</a>. He mostly talks about the practicality of testable solutions; they’re measurable, action-orientated, deliverable, quick.</p><p>Where non-testable ideas, those moments of “I had a good idea”​, are discussed, debated, passive and placid. We beat around the bush and over-polish. Seth Godin talks about good work never seeing the light of day because finance professionals are paralysed by their fear of getting something “wrong”, herein we come back to failure.</p><blockquote>“Testable hypotheses seemed a faster and, frankly, better gateway to innovative action and active innovation”​ Michael Schrage</blockquote><p>Testable ideas are better both operationally and emotionally.</p><p>Operationally because of the aforementioned. Emotionally they’re better because they remove that subjectivity, pride and ego that comes with the <em>next big idea</em>. “Business ideaholics, not unlike meth addicts and other junkies, are always looking for the next fix.”​ say Schrage. Here we are talking about ego again. And you’re going for the jugular there, Michael. But he’s basically saying that there is a serious temptation to fall in love with the process of generating the idea itself. We’re human beings. Naturally creative; it’s what sets us apart from the primates. To proliferate these ideas endlessly and forget that something needs to be done with the ideas is all too common.</p><p>Those big ideas, too, are impractical and are often a result of what Jeff Bezos calls, a “hail-mary”​ event; a desperate attempt to save a companies corporate existence. Blockbuster. Blackberry. Nokia. The contrary, and more popularised approach of what we’re discussing today, is smaller ideas or, what Peter Sims calls in his book, <a href="https://www.amazon.co.uk/Little-Bets-breakthrough-emerge-discoveries/dp/1847940498"><em>Small Bets</em></a><em>.</em></p><h3>Validation.</h3><p>And here’s the big one. Experimentation validates ideas. The illusion of rationality is a known phenomenon when ideas or assumptions seem logical, but they haven’t been validated in the real world.</p><p>Chris Rock didn’t just win 3 x Grammys for “best comedy album”​ by being ‘naturally’​ funny. The albums are tested hundreds, if not thousands, of times at smaller gigs. In a 45 minute set, Chris will ramble, lose his train of thought, and only sometimes stumble on something where he’s audibly heard as saying “this needs to be fleshed out more if it’s going to make it”​</p><p>Amazon, similarly, are known for their innovative approaches to delivery or video with Prime subscriptions or hosting with AWS. Jeff Bezos often talks about how they go down “blind alleys”​ But, he goes on to state “every once in a while, you go down an alley and it opens up this huge, broad avenue”​</p><p>Both Chris and Jeff accept that you can’t rely on intuition what works and what doesn’t work. You must experiment.</p><blockquote>“If you follow your intuition, you will more often than not err by misclassifying a random event as systematic. We are too willing to reject the belief that much of what we see in life is random”​ Daniel Kahneman</blockquote><p>Intuition can mislead or misrepresent.</p><p>Mislead. Intuition holds no evidence behind a claim. “I feel it in my gut”​ is as risky as putting it all on 17 Red. In fact, research suggests that our predictions of how customers behave are wrong 9 out of 10 times (Thomke and Beyersdorfer, <a href="http://booking.com/">Booking.com</a>: Harvard Business School Case).</p><p>Going back to our mate Jeff, Amazon once launched a feature where in one click, Amazon would show you what other customers purchased allowing you to find customers with similar order histories. No one used it. And of course, there are thousands of examples of this too, aren’t there? Where people have created stuff based on pure intuition. I think it’s what the majority of the popularised entrepreneurs still do when pitching to the scary Dragons on BBC One. Pride can get in the way.</p><blockquote>“Our history is full of things like that. Where we came up with an innovation that we thought was really cool and the customers didn’t care.”​ Jeff Bezos</blockquote><p>Mis-represent. Intuition can mis-represent because of the relationship between A and B. If intuition is telling you A because of B, what happens if the relationship between A and B is purely coincidental. Like how <a href="https://paularowinska.wordpress.com/2018/08/18/deadly-ice-cream/">eating more ice cream increases your chances of drowning</a>. Or <a href="https://www.youtube.com/watch?v=VMUQSMFGBDo">forrest fires</a>. Or <a href="https://www.youtube.com/watch?v=lbODqslc4Tg">polio</a>. In other words, there’s a difference between correlation and causation, and pure intuition poses a risk of the former. In simple terms, when your intuition is wrong.</p><p>Data, especially through experimentation, can verify correlation. Yet, and this should be in the section above, causal questions can never be answered from past data alone (Pearl and McKenzie, The Book of Why: The New Science of Cause and Effect, 2018). So does experimentation provide the ability to truly innovate?</p><h3>Continually learning.</h3><p>Maybe this is the big one actually.</p><blockquote>“Experimentation matters because it fuels the discovery and creation of knowledge”​ Stefan Thomke</blockquote><p>Stefan wrote that as the very first sentence in his book, Experimentation Matters in 2003.</p><p>Stefan was talking both about the exploration of experiments, as well as the exploitation. It’s not just the validation but the why. Analysed right, iterated on with further hypotheses, experimentation can be the most effective and efficient knowledge source in your organisation.</p><p>Over 75% of innovations come from your active users, a study by the economist Eric von Hippel found. So ask them and find out. Demonstrate behavioural and attitudinal shifts with them, practically, through variable changes. It’s how films tinker with their story line. Or how P&amp;G prototype quick and dirty ideas for new products. And now we’re just talking about how good user research is, so I’m going to stop.</p><p>Whilst Steve Jobs stated that “no one wants to rent their music”​ (cue Spotify), he also stated that “people don’t know what they want until they’ve seen it”​. Thus, in a paradoxical conversation with himself proving, that practical feedback — be that through an experiment or not — does indeed support innovation, but only through learning itself.</p><p>We’ve (FKW User Conversion) drastically altered how staff speak to customers and package products for one client before — by testing something online. Not because the the experiment ‘won’​, but because of what we learned from the attitudes of those within the variant group.</p><p>That won vs loss thing, by the way, is such a red herring. It places experimentation as this binary method of being. When it’s a method for further knowledge as much as validation.</p><p>For my final semi-tenuous example, did you know Beethoven had hundreds of manuscripts with annotations and notes that clearly demonstrate his experimental approach? They highlighted a clear evolution of his musical style from, arguably, the same as everyone else — especially Mozart — to a highly distinctive and innovative style; so much so it ushered in a new era of classical romantic music. Just like Chris Rock (which I never thought I’d be comparing the two) both took a process of testing their art, understanding what did and didn’t resonate along the way, leading them to new heights.</p><p>See. Experimentation gives us knowledge we didn’t have before which allows us to create and innovate.</p><h3>Summary</h3><p>Here’s something which I bet you didn’t know (or perhaps care). I consider creativity to be one of my core values*.</p><p>Experimentation is also process I really genuinely believe in.</p><p>These two things are beautiful in their own right. And when you put them together? Love and music, baby. I do believe the two work together and I even advocate this on my <a href="http://www.davidmannheim.com/">website</a>. “I believe creating a culture of experimentation is an enabler to learn and iterate”​. I wrote that months ago. So, nur nur nur nur.</p><p>But, I do feel there is still a place for intuition. It might mislead and misrepresent; ut it creates. If we’re going for moonshot innovation, theres going to be a point where you’ve just got to take a leap. And I’m going to talk about that in “Experimenting for iteration and experimenting for innovation”​</p><p>Having a process of experimentation, at least in some form, will support that leap though validation and extra learnings.; it just might not create the leap itself.</p><h3>What’s next?</h3><p>In my next series of articles, I am going to dive a bit deeper into questions like:</p><ul><li><a href="https://www.linkedin.com/pulse/does-experimentation-inhibit-encourage-innovation-david-mannheim/">Does experimentation inhibit or encourage innovation?</a></li><li>Experimentation validates creativity ← that’s this one, spoilers!</li><li>Can a hypotheses limit creativity?</li><li>Experimenting for iteration and experimenting for innovation</li><li>How can we be more creative within our process of experimentation?</li></ul><p>*if you’re interested / bored / nosey in the other 4, at least for me, it’s autonomy (self-determined and independent), industry (to work hard and well in life), hope (maintain a positive outlook) and courtesy (to be considerate and polite). I did a great exercise with my wife not too long ago on our values because we are just that cool. DM me if you fancy the exercise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f1df2e7d82c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does experimentation inhibit or encourage innovation?]]></title>
            <link>https://davidleemannheim.medium.com/does-experimentation-inhibit-or-encourage-innovation-9ca21c571e91?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ca21c571e91</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[creativity]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 11 Jun 2021 06:36:22 GMT</pubDate>
            <atom:updated>2021-06-11T06:36:22.945Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR; Experimentation has its place. Intuition has its place. There is inherent bias with validating the new be that a pre-conditioned way of thinking, fear or timing which is what I want to explore…</p><blockquote><em>“Don’t experiment when you should think; don’t think when you should experiment” Jon Bentley</em></blockquote><p>This is going to sound counter-intuitive coming from someone who has built his career up evangelising experimentation. And as someone who considers creativity one of his core values. But I’m not sure the two work that well hand in hand.</p><p>I genuinely feel that experimenting can restrict creativity and innovation just as much as it fosters it.</p><p>What?!</p><p>I’ve set out exploring the concepts of intuition, innovation, disruption and how the methodology of split testing works alongside them, if at all, and found that the majority of people believe that <strong>yes</strong> “experimentation encourages innovation”.</p><p>How <a href="https://www.headspacegroup.co.uk/creative-entrepreneurs-the-story-behind-innocent-drinks/">Innocent drinks experimented with their smoothies</a>. Or how <a href="https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_eef64075-6b15-40cf-b045-2eae5726d357">Dropbox experimented with their product and pricing offering</a>. Or how <a href="https://hbr.org/2018/11/using-experiments-to-launch-new-products">Uber launched UberPool to influece Uber usage</a>. The moon landing was a series of experiments was it not?</p><p>Heck, I even put a biased, small sample sized poll on Linkedin just last week and the results speak for themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4JNPBIF27RPigW1DrsOh8w.png" /></figure><p>But what about the adverse?</p><p>What about Henry J Ford asking his customers what they would have wanted? Faster horses of course. Is that considered too radical an innovation that experimentation can’t support such a concept because at some point you’ve just got to take the leap? Users don’t know what they don’t know right?</p><p>Does experimentation corrode the purpose of intuition? Tim Berners-Lee didn’t wake up one day and say “I’m going to test the concept of this interconnected web thing called the Internet” or iterate towards it. His boss even called it “<a href="https://webfoundation.org/about/vision/history-of-the-web/">vague</a>” and took 4 years before it becoming non-proprietary.</p><h3>Not even users know what they want.</h3><blockquote><em>“People have told us over and over again ,they don’t want to rent their music…they don’t want subscriptions” Steve Jobs in 2003.</em></blockquote><p>When we listen to our customers, they can be often wrong. Just as we are; despited being professed ‘experts’ within our field.</p><p>More precisely, when we interpret or analyse what customers want, in theory they know no better. I’m not devaluing the need for research — heck, even Jakob Nielsen says <a href="https://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/">don’t listen to your users</a>.</p><p>I’m curious as to the role that research, which includes experimentation, plays in the innovation of new things. How can users know what they want when it’s not been created or adopted yet? How do companies like Spotify innovate when the apparent <em>research</em> suggests customers ‘don’t want to rent their music’? Or how does Ford innovate with the car when the apparent <em>research</em> suggests that customers ‘want faster horses’? Or the adverse happens when responding to a survey Walmart posed, customers asked for less clutter, <a href="https://www.notion.so/Does-experimentation-inhibit-or-encourage-innovation-0c341e216f424e3c9630744ea20ab7fd">Walmart did so and in the process lost $1.85bn in sales</a></p><p>Within innovation or new product development, I question, can feedback dilute the innovation of what’s intended to be? The <a href="https://www.pressgazette.co.uk/new-look-mobile-friendly-bbc-news-website-attracts-mostly-negative-comments-from-readers/">BBC didn’t concede to negative feedback when they radically redesigned their homepage in 2015</a>. It soon calmed down as they became more conditioned to the change itself (h/t to Tim Axon)</p><h3>That’s because we don’t know what we don’t know.</h3><blockquote><em>“What’s really valuable in life is when you allow people to do different things. Things they otherwise wouldn’t have contemplated or things that didn’t simply occur to them” Rory Sutherland</em></blockquote><p>When we come up with solutions, be they for experiments or not, we tinker the experience based on what our experience with the experience is. This is because of our pre-conditioning as well as <a href="https://www.verywellmind.com/what-is-functional-fixedness-2795484">functional fixedness</a>, a cognitive bias that involves a tendency to see objects as only working in a particular way.</p><p>In other words, we are conditioned to variables or stimulus. i.e we only know what we know. In our world, a PDP has to look a certain way for a specific purpose, or filters must be located here and have these functions.</p><p>We use our past experiences to help solve the issues we face. This can make it difficult to see novel or creative ways of fixing current problems (<em>Valee-Tourangeau; Einstellung defused Interactivity and mental set</em>).</p><p>My hypothesis goes further beyond ‘creating the same solutions to the same problems’ because of our conditioned stimulus. And I propose, the more we homogenise solutions, the more users become desensitised to the effect of the ‘new’. Meaning the homogenisation of solutions will impact user conversion (<a href="https://www.userconversion.com/">h/t</a>) less and less over time. Making it <em>even harder</em> to innovate.</p><h3>The new requires time and effort that we don’t have</h3><blockquote><em>“The best, maybe the only real measure of innovation, is a change in human behaviour” Stewart Butterfield</em></blockquote><p>From a user perspective, I question whether learning a <em>new</em> product or feature, because it carries a change in behaviour, carries both effort and risk; and therefore brings with it avoidance.</p><p>In that, as humans, we fear catastrophe. There are many more ways to die than there are to survive. We know we want something that’s pretty good, but we definitely know that we don’t want it to be terrible. Therefore, we are conditioned to safety in what we know, and in what we don’t know. Well. We avoid because we fear.</p><p>I think we also avoid because we associate effort and time in learning the new. There’s a bit to be said about McDonalds and Amazon here. In that, we want things fast. Like, super fast. There’s even the coined term “<a href="https://en.wikipedia.org/wiki/McDonaldization#:~:text=McDonaldization%20is%20a%20McWord%20developed,of%20a%20fast-food%20restaurant.">McDonalidsation</a>” of society, of which efficiency is the first principle, and, to my point earlier, predictability is the third. But learning something new comes at the cost of speed.</p><p>I remember testing Klarna for a famous beauty brand in 2016, less than 6 months after it “came to market” in the UK. We tried 9 different iterations of promoting it to clients and every single time it ‘failed’. There was next to no uptake on a service that, in theory, should have been perfect for this brand. Our assumption as to why was as much about the communication of the innovative buy-now pay-later service, less so the product itself.</p><p>It’s one thing to come up with a product or feature. It’s a whole new thing — remarkably more difficult and often lesser discussed — to get people to adopt a new behaviour. Therefore, when it comes to experimentation, the hypothesis process should have as much, if not greater emphasis, on the communication of the product, as much as the product itself.</p><h3>The more we understand about something, the more restricted our river of thinking.</h3><p>Indeed, does greater understanding of something lead to an inability to innovate in that area? We are so conditioned to how a pricing within SaaS products should be, for example, does that inhibit our ability to innovate or disrupt that model?</p><p>There’s certainly the argument to suggest that experiments validate and provide knowledge. I truly believe that, also. (note: it’s something I’m going to unpack more in “can hypothesis limit creativity”)</p><blockquote><em>“Experiments are fact-finding missions that, over time, inch towards greater understanding” Andrew Stanton</em></blockquote><p>But there’s the counter argument to this. Having a, what <a href="https://www.youtube.com/watch?v=A-mO8H-mutA">Duncan Wardle</a> says, “naive expert” in the room can create more than those with greater understanding. Someone who isn’t an expert within the field and therefore does not have any preconceptions, nor bound by the limits of what is and isn’t possible. It’s how <a href="https://www.forbes.com/sites/jeffkauflin/2019/05/02/lemonade-fintech-insurance-unicorn/?sh=4888250b6cde">Lemonade insurance</a> was born, now valued at over $2bn in just two years. It’s how, in Duncan’s case, when asked to draw a house you always draw a door, two windows and a triangle for a roof because you understand how houses should look.</p><p>I question the outcome of Andrews statement; can greater understanding limit the art of what’s possible or remove preconceptions and therefore innovation?</p><h3>Summary</h3><p>I think there’s a place for the methodology of experimentation when it comes to innovation.</p><p>In my opinion, experimentation supports innovation as its a method for validation and exploration; but it doesn’t create it. There are inherent bias within validation, too, given users might not know any better or be ready for that innovation itself.</p><p>Which means there’s a place for intuition.</p><p>That’s not to say that you read the results of an AB test, feel in in your gut that it’s wrong, and ignore the data. In fact the data can lead you to other areas of curiosity — think the creation of the Post It. But stepping outside our comfort zone, our preconceptions, our river of thinking, testing in different ways that are <em>more than data-driven,</em> understanding that a split tests and iterations of that are not just the only way to solicit feedback and understanding; **are methods of creative thought that can really innovate.</p><p>It’s a curious topic that needs more unpacking. In my next series of articles, I am going to dive a bit deeper into questions like:</p><ul><li>Experimentation validates creativity</li><li>Can hypotheses limit creativity?</li><li>Experimenting for iteration and experimenting for innovation</li><li>How can we be more creative within our process of experimentation?</li></ul><p>What do you think? Does experimentation inhibit innovation as the solutions and the users that interact with it are too preconditioned in their river of thinking? Is experimentation vital in the creativity process because it allows us to learn and understand relationships, therefore iterate to places we’ve never been before?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ca21c571e91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yes. We need to attribute revenue to experimentation. Here’s how: Part 2]]></title>
            <link>https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/72e9d39d2a20</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 28 May 2021 05:35:51 GMT</pubDate>
            <atom:updated>2021-06-01T08:12:41.877Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR; Start to re-frame the conversation about experimentation. Talk about expected loss as much as expected gain, use OKRs to focus and align, talk in ranges, use degradation, highlight your program quality metrics.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><p>In the first part, we spoke mostly about communication (Part 1 is <a href="https://www.linkedin.com/pulse/yes-we-need-attribute-revenue-experimentation-heres-how-mannheim/">here</a>, read that first). Being honest, I split this article up because it was just too damn long. This is also, really, about communication 😝 It’s all about communication. But, more specifically, re-framing the conversation.</p><p>Whether you’re talking about an individual experiment or a series of experiments (a program, if you will) how that is communicated will impact how it is received. In my experience, as someone who has dealt with a breathe of stakeholders; junior to senior, practitioner to C-suite, mature to immature, I feel that there are certain strategies you can utilise to <em>re-frame the discussion of experimentation away from revenue</em>.</p><p>Inherently, that educates the client by prioritising on metrics and attributes that aren’t directly (perhaps indirectly) associated with the program itself. A cause and effect, if you will. So previously I spoke about defining your purpose of experimentation, setting standards, accuracy thresholds and telling stories. Today, I’m going to talk about:</p><ol><li>Expected loss</li><li>OKRs for focus and alignment</li><li>Talk in ranges</li><li>Using Degradation</li><li>Program quality metrics</li></ol><h3>Expected loss</h3><p>What about open discussions on the alternatives to experimentation? What is the cost of not experimenting?</p><p>I admit, I think this is a hard sell. In experience, to companies, it feels like a hard sales pitch by associating negative connotations to not doing a very valuable practice, instead of displaying the positives of doing experimentation. And I know, there are the concepts of loss aversion and that <a href="https://usabilla.com/blog/how-loss-aversion-and-risk-influence-decision-making/#:~:text=Research%20studies%20into%20the%20psychological,times%20more%20than%20the%20gain.">golden ratio of 1.5 to 2.5</a>, but just from my experience, I’ve never seen it work very well.</p><p>But we can explore these alternatives. What about a website redesign? What about placing budget into acquisition channels like PPC? What about dumping loads of money into organic? Interestingly, what about their associated levels of revenue attribution?</p><p>I sometimes like to talk in measurement terms and gear the conversation towards efficiency, rather than ‘loss’. As an experiment validates a variable through measurement, when you don’t experiment, you clearly don’t measure. Asking teams how well that feature performed or how that deployment faired or how well utilised that feature is sparks the conversation. Or comparing against a discussion of measurement elsewhere in the business. If we measure and guide our decisions of acquisition based on data, why don’t we do the same with onsite behaviour? (etc)</p><blockquote>Experimentation is as much for protecting revenue, as it is incrementing it. Failed experiments are effectively saving a business from productionising shitty experiences. Firstly to stop poor UX being deployed to production that would have cost the business money. But also, in the time saved of your R&amp;D team — where, quite often, the effort to deploy an experiment is a fraction of the effort to build a new experience” <a href="https://www.linkedin.com/in/luke-frake/">Luke Frake</a></blockquote><p>When it comes to individual experiments, we could re-frame discussions to talk about how a, quote unquote, experiment ‘saved us’ losing money by not releasing it. <a href="https://abtestguide.com/bayesian/">ABtestguide.com</a> calculates the business impact of a Bayesian result in 6 months, where you could balance revenue yield against chance. This is a nice little tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vjbrah1XjNskhSRw" /></figure><p>This conversation is geared towards, as much protecting revenue, as it is gaining it.</p><p>It can also be a conversation geared towards efficiency of development. In one example that comes to mind, when working with Travis Perkins, “instead of embarking on a lengthy card-sorting process that would have taken us months to re-categorise and implement, we chose to redesign the navigation; how users interacted with it. We saved our team from an estimated £95k on a solution that took just 3 days to build” That “£95k” was worked out by estimating the resource needed to plan, restructure and analyse the build requirements as a cost per day and no. of days. Rather arbitrary, I admit, but it gave a scale rather than an accuracy, and helped reframe the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fvHXWyDAMSG3bw6s" /></figure><h3>OKRs for focus and alignment</h3><p>Want to know one of the best decisions we ever made at FKA User Conversion (in my humble opinion)?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*It4ImHvb997n5_tc" /></figure><p>To say reading <a href="https://www.amazon.co.uk/Measure-What-Matters-Simple-Drives/dp/024134848X">Measure what Matters</a> was life-changing is an understatement (and I read a lot around that subject of objectives and key results to be sure). Implementing OKRs for our conversion programmes allowed us to tell a story or narrative around where we’re heading that wasn’t necessarily routed in revenue gain. Again, it helped <em>reframe the conversation</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pyglm8rCfNwT3Ui1" /></figure><p>Don’t get me wrong, it was difficult (and still is) to implement OKRs; well. Some businesses don’t have this methodology. Some don’t even understand their objectives. We were dealing with, largely, trade business meaning that everything is an objective or priority and usually in a fast-paced environment. Individual consultants have different styles and OKRs are fairly process driven. That being said, by calling out “focus and alignment” as the key to unlocking a re-prioritisation of outcomes, the method of doing that (i.e an O and KRs) wasn’t as important as what the conversation geared towards.</p><p>By framing success as the confidence value of achieving specific objectives and specific key metrics, the focus is taken away from revenue attribution. Equally, it helps the wider team inc. stakeholders understand the direction of the optimisation program, as well as appending focus and providing alignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S8VxgTg5mQN2hnl_" /></figure><h3>Talk in ranges</h3><p>This seemed like a fairly obvious solution to the ‘problem’ when it originally reared its head. And it’s a tactic, after speaking with a lot of practitioners, that’s well used.</p><blockquote>“An experiment doesn’t mean you’ll make 3.9% revenue uplift, it means with 90% certainty you’ll make an uplift between a range — point A and point B. This is what confidence intervals are and dictate” <a href="https://www.linkedin.com/in/hpourkhalkhali/">Hazjier Pourkhalkhali</a></blockquote><p>Hazjier recommends to “always err towards the lower side of that range as a matter of best practice”</p><p>Because we work with confidence values, presenting ranges back to your stakeholders makes perfect sense. It alleviates the issue to a degree, in that you’re showing it’s not an 11.96% uplift on the experiment but somewhere between 9.5% and 14.2%. It still needs the same degree of qualification, it’s just a more visible way of making the point. As a result, we generally provide forecasts between two values, rather than hanging our hat on a single value; again, it sets expectations.</p><p>We’re ultimately talking about presenting a range of values that the data hasn’t ruled out with a degree of certainty. Visualising this, too, will help stakeholders understand the risk associated with implementing the experiment.</p><blockquote><em>We don’t work in a world of exact numbers, we work in a world of ranges. So at best we can say to a degree of certainty that an experiment will have increased a metric by roughly our confidence intervals” </em><a href="https://www.linkedin.com/in/luke-frake/"><em>Luke Frake</em></a></blockquote><h3>Using Degradation</h3><p>We spoke in the previous article about not collating experiment attribution together because, “you’re adding fuzz on top of fuzz”.</p><p>And in the previous section we talking about ranges, designed to showcase the conservative estimate. Why? Because it’s better to over-deliver than over-promise. For this reason, we <em>always</em> add degradation into experiments.</p><p>That rate should be contextualised to the business, however.</p><blockquote><em>“I have not seen a better take yet than just having diminishing returns from 100% to 0% in 1 year. For one reason or another, a gold standard has emerged of including an impact of 6 months in business case calculations (we contributed to this ourselves — we often presented: 1 year of effect that expires linearly, which is equal to 6 months of impact at implementation immediately after the test)” </em><a href="https://www.linkedin.com/in/tonwesseling/"><em>Ton Wesseling</em></a></blockquote><p>Why does the value of a test degrades over a number of months? <em>The novelty made fade, or a competitor may copy; we regress towards the mean.</em></p><p>In particular, that point about novelty fading is an interesting one, and something that I tend to discuss within my talks around creativity.</p><ol><li>We are conditioned to variables. Like, how a product page should be structured (thank you Amazon)</li><li>…when, we are conditioned to these variables, we become desensitised to them</li><li>…meaning the impact of that stimulus is reduced.</li></ol><p>I have no proof on this whatsoever, might I add, apart from a small sample size of individual experiments achieving the same thing.</p><p>How much does it degrade? I don’t know. The below is from a talk Ton did in Manchester (great city, one great football team, let’s not talk about the Europa League Final please) about creating new baselines and standards based on cumulative experiment attribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i-p6OC_Tf8aZz88C" /></figure><p>Cumulative collective experiments could be contextualised in some form by degrading based on sales lifecycle and seasonality.</p><p>Taking the below example from a client at FKA User Conversion, we noted that the cumulative value of the experiment degraded over time, but <em>loosely</em> contextualised it to the business lifecycle. We knew there were peaks in sales as different seasons, new streams of traffic based on TV advertisements etc.</p><p>Still, I agree, I don’t think there’s a ‘better’ way (at least that I have seen) than just diminishing the return of an experiment to null after a period of time — I’ve actually done 6–8 months in some cases; all in the interest of being conservative in the <em>hope of over-delivering rather than over-promising.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7aETsuuWNlrELKGY" /></figure><h3>Program quality metrics</h3><p>To remove the focus of revenue attribution, we must re-frame the focus towards something more attributable. Something that defines performance. Something that showcases we’re moving in the right direction, at least.</p><blockquote><em>“The most important KPI was the number of tests you are running because, naturally, the more tests you have, more winners you’ll have. The second KPI is around the time to implement — if you have a lot of successful experiments they’ll take time to implement both in terms of development and also in terms of time to go live. Finally, the third KPI is success rate. If your success rate is too low then the chance that you’ll input a false positive gets bigger and bigger.” </em><a href="https://www.linkedin.com/in/annemarie-klaassen-4a55a310/"><em>Annemarie Klaassen</em></a></blockquote><p>In the case above, Annemarie Klassen noted that she wanted to focus on the quality of the experiment program itself, by focussing on 3 x main KPIs:</p><ul><li>Number of tests</li><li>Time to implement</li><li>Success rate</li></ul><p>On that last point, FYI, plugging once more the great<a href="https://abtestguide.com/fdr/">ABtestguide.com</a>; they have a false discovery rate calculator for your experimentation program which I would recommend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/0*NsFKuPLPpZAsSklm" /></figure><p>All the way back in 2015, Paul Rouke wrote an article for eConsultancy on <a href="https://econsultancy.com/vanity-versus-sanity-metrics-in-conversion-optimisation/">Vanity vs Sanity metrics</a> (it’s a good read, give it a shot, even I remembered it to this day nearly 6 years later). Here, he spoke about the ‘sanity metrics’ for an effective CRO program being the percentage of tests that deliver an uplift, the average % uplift per test, the percentage of successful tests that deliver over 5% CR increase, the percentage reduction in cost-per-acquisition (CPA)</p><p>….and even… The percentage ROI per test. Let’s not dig into that last one 😆</p><p>Indeed, Booking.com take a similar approach in a recently released article about their KPIs around <a href="https://medium.com/booking-product/why-we-use-experimentation-quality-as-the-main-kpi-for-our-experimentation-platform-f4c1ce381b81">‘experiment quality’</a>. And OK, we’re talking about Booking.com here — I would question (and I don’t know the answer) how important revenue attribution is to them on a per experiment or collective basis. Their maturity is as such that they require KPIs that measure good product decisions. They therefore focus on:</p><ul><li>Good design (things which happen before the start of an experiment)</li><li>Good execution (mainly about the planned experiment duration and the adherence to that plan)</li><li>Good shipping (validates that the decision is in line with the shipping criteria.)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_TI6GjzZi9V418HZ" /></figure><h3>Summary</h3><p>How should you attribute revenue to experimentation? In part 1 we spoke about ‘<em>communication</em>’. How to best communicate</p><ol><li>Align and define on your purpose of experimentation</li><li>Set Standards and therefore expectations</li><li>Define what accuracy is acceptable</li><li>Tell Stories</li></ol><p>In part 2, I would argue this is more about ‘<em>re-framing the conversation</em>’ (but really #3 and #4 should be in the first series). I told you, I just split up my list of 1–9 because it was super long and no one would read this. Hell, I’m impressed you’ve read this far 👀</p><ol><li>Expected loss</li><li>OKRs for focus and alignment</li><li>Talk in ranges</li><li>Using Degradation</li><li>Program quality metrics</li></ol><h3>My final final summary about this “Experimentation Revenue Attribution Series”</h3><p>I really hope this series has helped, at least, spark some debate and encourage thought. I too, struggle with this and so do many others. You’re not alone. The “theory” often doesn’t match the “practice” and I’m trying to balance the two; often slipping into practitioner and evangelical mode. Sorry about that.</p><p>I strongly believe that there is an over-indexed view of revenue attribution towards AB testing. And that has over-simplified the art of it. It’s commercialised this methodology, and sometimes for personal gain or greed. Falsified case studies, intentional or not. Breeding bad practice after bad practice. Creating mis-aligned thought and education.</p><p>Remember SEO in c. 2015? Stakeholders mostly cared about “getting to the top of Google”. Which bred the same as above; falsified case studies, bad practice and poor education for years to come. But those who really cared about the craft spoke of presence and authority.</p><p>I feel we are at a similar inflection point within the CRO (specifically AB testing) industry in reference to our craft.</p><p>In most cases, it has diminished the importance and investment of, what I would consider to be, one of the most valuable practices within business.</p><p>Forgive the cynical old man view but it is the single reason why I embarked on this discovery nearly 3 years ago; because I care about the craft and really want others to see it’s true potential. By over-indexing revenue attribution, I feel that we over-simplify and therefore won’t reach that point.</p><blockquote>Experiment revenue attribution, in an ideal world, should not be done. It’s not the purpose of the methodology. But we don’t live in an ideal world. Our world is contextual and complex. So, when doing so, we must have guidelines that educate our stakeholders over time and ensure acceptable levels of accuracy, to help re-frame the conversation as to the true purpose of AB testing</blockquote><p>My favourite quote in the whole series is all about humility and empathy with others. Just because we believe something to be true does not mean we should stomp our feet until we get it. Thank you Matt.</p><blockquote>Ultimately, it comes down to approaching conversations with empathy. It’s unreasonable to expect the FD or commercial team to be as knowledgeable about experiment driven product development as us. We’ll get the most buy-in when our story is simple, compelling and delivered with humility. <a href="https://www.linkedin.com/in/matthewpaullacey/">Matt Lacey</a></blockquote><p>Here is a list of the history, the why, and the how (part 1). A huge thank you to all the contributors both in quote form, those who reviewed the articles, and who sparked debate.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72e9d39d2a20" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yes. We need to attribute revenue to experimentation. Here’s how: Part 1]]></title>
            <link>https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccf90a4416e6</guid>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[cro]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 21 May 2021 05:43:53 GMT</pubDate>
            <atom:updated>2021-06-01T08:12:10.431Z</atom:updated>
            <content:encoded><![CDATA[<p><em>TLDR: Communication is key. Align and define on your purpose of experimentation for the business. Set your standards and therefore expectations. Define what accuracy is acceptable. And tell stories.</em></p><p>Woah, this is a long one. Sorry. So much so it’s a two-parter. I actually think the second part is better than the first part, too; this is Joss Whedon — all about communication — and next Friday is Zach Synder — the Batman.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><h3>Intro</h3><p>There’s a stigma that experimentation is inextricably linked to revenue. I know, I’m banging my drum again.</p><p>It’s because I have to. And it has to be. In a world where cash is king and competition is high, methods that are advertised as pushing us forward, or in some cases silver bullets, will win. Unfortunately, sometimes that advertisement is falsified. Poor statistics. Case studies that promise the world. Copy-cat behaviour. I unpack that more in the previous two articles of this attribution series.</p><ul><li><a href="https://www.linkedin.com/pulse/why-do-we-assume-experimentation-financial-gain-david-mannheim/?trackingId=e9%2BkDWecS6en5wsXOyoMBA%3D%3D">Why do we assume experimentation is about revenue gain?</a></li><li><a href="https://www.linkedin.com/pulse/you-cant-accurately-attribute-nor-forecast-revenue-heres-mannheim/">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li></ul><p>I continually refer back to the true purpose of experimentation; validation.</p><p>The one thing we haven’t discussed is <em>how</em> to attribute revenue to experimentation.</p><p>Because my world comes from stakeholder expectations and operations within a conversion rate optimisation environment, I cannot sit here and discuss statistics as articulate as some of my data scientist colleagues. Attributing and forecasting revenue through data modelling, be it predictive or not, is complex and contextual; it’s not something I’m going to be able to dig into.</p><p>But I do understand humans; despite being an introvert.</p><blockquote><em>“CRO is not a pure science. We’re dealing with things that are just messy. We’re testing with humans and humans are complex.” </em><a href="https://www.matthewedgar.net/"><em>Matt Edgar</em></a></blockquote><p>The purpose of experimentation is not designed for revenue. And to do so is to over-simplify the performance of a method that, inherently, is complex. Yet we have to discuss the elephant in the room with our stakeholders. So we use the power of communication to educate them and create a relationship of authenticity and trust that builds a balance of validating decisions practically; rather than creating features commercially.</p><p>As a result, when we communicate experimentation to others within our business, my recommendations for how to educate and reframe mindsets are (at least for this article) — part 2 next Friday.</p><ol><li>Align and define on your purpose of experimentation</li><li>Set Standards and therefore expectations</li><li>Define what accuracy is acceptable</li><li>Tell Stories</li></ol><h3>Align and define on your purpose of experimentation</h3><p>Different businesses have a different purpose for experimentation.</p><p>Understandably, AB testing is designed for validation, but from experience I’ve had the following quotes, from stakeholders of various businesses, when asked the question “why do you want to test?”. It’s actually a really powerful question to ask your stakeholders — and one we ask within our on-boarding quiz at FKA User Conversion. Some responses have included:</p><ul><li>“I want to know what works above and beyond what’s already in our roadmap”</li><li>“We can add more £££ to our service and know what works and what doesn’t work”</li><li>“Are we doing the right things? What should we be focussing on?”</li></ul><p>André Morys and the team at Konversions Kraft set those expectations clearly by asking each client, before starting a project, “what their most favourite outcome is and what ‘price tag’ they would put on each of the outcomes to reach their goals”. This helps prioritise and frame the conversation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ABu7BnIDjIJif9a-" /></figure><p>Similarly, as part of our on-boarding process at FKA User Conversion, we send a <a href="https://docs.google.com/forms/d/e/1FAIpQLSdwE3pKOm8F6XHt2apiXn5HJ0suGLQGhTVwE6M5IaU8YDVXig/viewform?usp=sf_link">quiz</a> out to all stakeholders involved or have a touchpoint in the project and it’s been one of the most influential and enlightening things that we do as an agency. There are a series of questions within the quiz that identify:</p><ol><li>How important is experimentation to their roles and responsibilities</li><li>How involved they want to be within experimentation*</li></ol><p>*I really like understanding the difference between importance and involvement. There’s a venn diagram there, somewhere.</p><p><a href="https://docs.google.com/forms/d/e/1FAIpQLSdwE3pKOm8F6XHt2apiXn5HJ0suGLQGhTVwE6M5IaU8YDVXig/viewform?usp=sf_link">That quiz is here for reference.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6svdKa-r_E6D5fBr" /></figure><h3>Set Standards and therefore expectations</h3><p>At the end of the day, we’re just talking about good levels of communication, aren’t we? Therefore, we should follow the unwritten rules of decent conversations with human beings who are complex and emotional. Setting those expectations first is vital to success so you don’t get the shock factor later on in your career of “woah, hold on, you said this was worth £5m but we’re not seeing it on our bottom line — why?”.</p><blockquote>“The most important tip is that you need to sit down with your stakeholders and talk about this early on. About how you want to report it, what the downsides are of reporting it like that — they should be aware what the limitations are.” <a href="https://www.linkedin.com/in/annemarie-klaassen-4a55a310/">Annemarie Klaasen</a></blockquote><p>The agreed approaches can be individual to you, the business or both. They enable you to set standards with how you are working within the business itself. Whether it comes down to the nuance of individual experiment reporting, collective experiment reporting or the purpose of experimentation as a concept, those standards should be both set and maintained.</p><p>We are dealing with human behaviour and different individuals react and learn differently; find a combination of methods that your stakeholders react to (ooo, look at that, a test and learn approach) and set those expectations early on.</p><h3>Define what accuracy is acceptable</h3><p>Defining levels of accuracy is vital to the success of your program and communication.</p><p>When I originally put out the poll out of ‘do you believe you can accurately attribute and forecast revenue to AB testing’ there was a clear split. But the word “accurately” was what most were hung up on and caught people off guard. Rightly so. Accurately is a range of confidence — so how accurately do we need to be?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6H8_cGJ_SLv8M6hf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3LXSG9ulAAQxs5Oy" /></figure><blockquote>“If we were trying to get a 121 perfect match attribution, then 1. That’s a lot of time and time is money you’ll spend to check how accurate it was 2. you’re negatively impacting the ROI because the more money and time you spend on the accuracy of the figures, the less return you’ll naturally receive. It’s an exponential curve to be that much more sure” <a href="https://www.linkedin.com/in/timstewart/">Tim Stewart</a></blockquote><p>Tim relates the time spent to discovering this ‘value’ to lost opportunity cost. Perhaps, one of naval-gazing (Not navel gazing as Tim pointed out to me. This isn’t to do with belly buttons. I’m sure he’s coined that term to me on the phone in the past; most likely in one of his many descriptive essays).</p><p>Given moving averages and changing external variables, I wonder if the only true method of collective experiment attribution reporting is having a hold back group? Test and release all the changes you need to and have a hold back group of, say, 5% of users.</p><p>I don’t know of anyone that does this, personally, but, I’m curious; is this lost opportunity (of those 5%) for the purpose of being more precise with our attribution? We ask the question, once more, of the ‘alternatives to experimentation’…</p><blockquote>“Find a common sense approach of what’s the area we want to be in. Do we have enough grit and courage, to commit ourselves to 80% significance level for our experiments or do we make 5x more experiments and, thus, calculate for false negatives and the costs of not implementing a winner?” <a href="https://www.linkedin.com/in/andremorys/">André Morys</a></blockquote><h3>Tell Stories</h3><p>Of course, you can’t just set the expectations, you need to continually manage them. Continually communicate. And in a way that’s engaging.</p><blockquote>“At the end of the day, we’re all talking the same language — dollars, numbers and we’re all data driven. A lot of CFOs are willing to have those conversations. Their job is to figure out how to make the company more successful from a financial standpoint. So almost all of them are willing to work with you to help figure that out. With CFOs, I try to get them involved and excited in the project to see what we can prove from a financial standpoint” <a href="https://www.matthewedgar.net/">Matt Edgar</a></blockquote><p>One of my favourite methods of communication to subvert the expectation of pure revenue, is to provide test results in a story-arc.</p><p>I’ve seen a lot of different test results — I admit, some of the best ones are within a presentation deck (despite how much presentation decks, suck, give this a read <a href="https://www.inf.ed.ac.uk/teaching/courses/pi/2016_2017/phil/tufte-powerpoint.pdf">“the cognitive style of powerpoint pitching out corrupts within”</a> ). At FKA User Conversion, we showcase results in a three-pronged attack of</p><ol><li>What did we learn</li><li>What does it mean</li><li>What next</li></ol><p>… and then we provide the results. Giving this narrative gives you the opportunity to tell that story of what was observed and why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5W05BNpB3oBpXhuw" /></figure><blockquote>“Define those parameters upfront. Remind people of every opportunity of what you’re working to. If they want to change those parameters — give them the cost; time or money.” <a href="https://www.linkedin.com/in/timstewart/">Tim Stewart</a></blockquote><h3>Part 1 — Summary</h3><blockquote>“Let’s say you won 20 experiments on metrics other than revenue. I am confident that if you bundled all 20 of your winners together in a single test, you would then also see a meaningful impact on revenue.” <a href="https://www.linkedin.com/in/hpourkhalkhali/">Hazjier Pourkhalkhali</a></blockquote><p>I think we need to question, once again, the purpose of experimentation. That being validation, not revenue generation. In other words, teams only release those which have already been deemed as sufficiently ‘successful’ and not quantifiably detrimental to the experience. As I’ve already mentioned, attributing revenue — at least in some form — is a necessity to some, if not most, if not all businesses.</p><p>How is that done? Well, I would recommend that the business determine this for themselves.</p><p>If they set budgets based on a business case, they will have to support this value themselves. Set those expectations around the accuracy of attribution, manage those expectations and reframe KPIs accordingly. To judge <em>every</em> business decision on revenue is nonsensical; consider the alternatives to experimentation and discuss loss as much as gain would be my core recommendations for reframing the communication stream.</p><p>Why are you doing what you’re doing…</p><blockquote>“ It’s more important to me that we’re making the right product decisions at pace, rather than getting hung up on the specific pounds and pence calculations.” <a href="https://www.linkedin.com/in/matthewpaullacey/">Matt Lacey</a></blockquote><pre>Part 2: Next Friday 21st. </pre><pre>&quot;Yes. We need to attribute revenue to experimentation. Here’s how (Part 2 - Zach Synder)&quot;</pre><pre>--- Sign up to my Substack to not miss a post <a href="https://optimisation.substack.com/">https://optimisation.substack.com/</a> ---</pre><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><h3>A quick note on methodology…</h3><p>I wanted to quickly mention why I’m quoting people left right and centre for this particular article. Or why I’m even writing this.</p><p>Because this topic is contentious (it’s widely used, often wrongly), complex and contextual, in my mind, the only way to add objectivity is to crowd-source opinion. I did this back in 2019 as part of a piece of research to understand attribution. Those that have been kind enough to provide quotes have approved the use of that quote, in the past few weeks, and in some cases changed it.</p><p>I strongly believe in sharing news and practices about this topic and in doing so creating debate. Because I do feel that a prioritised view of revenue attribution has over-simplified the art of experimentation. It’s commercialised that methodology, and sometimes for personal gain or greed. In most cases, it has diminished the importance and investment of, what I would consider to be, one of the most valuable practices within business.</p><p>Hence, I’ve created the narrative from experience and undertook the research for it, but without the opinions of those who helped provide quotes, or review this, it wouldn’t have that level of objectivity. A huge thank you should go out to those people.</p><p>Right or wrong in my approach, I just want to help.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccf90a4416e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.]]></title>
            <link>https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a22b1ac2a5c7</guid>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[ab-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[cro]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 14 May 2021 05:37:03 GMT</pubDate>
            <atom:updated>2021-06-01T08:11:17.993Z</atom:updated>
            <content:encoded><![CDATA[<p><em>TLDR; I don’t believe you should attribute revenue to individual experiments, but I get the fact that we have to; it’s fuzzy. I don’t believe you can accurately forecast that revenue attribution; it gets even fuzzier. And, when you collate experiments together to determine overall revenue impact? I certainly don’t believe you can attribute that type of revenue; that’s super fuzzy. </em>Experiment results tell you which way you’re headed, not how far you’re going.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><p>I’m expecting debate. Please comment. Aggressively. This is something, I admit, has been a burning hole in my pocket for many years. And trying to convince many stakeholders that revenue attribution is not the main purpose of experimentation; especially to ecommerce clients where the main focus is, indeed, trade, has proven difficult for me personally.</p><p>In 2019, I took it upon myself to interview 30 or so of the best minds in the business. Why can we not attribute revenue to experimentation accurately? What does accurately mean? Can we forecast it? What do you do when stakeholders ask? How do we survive as individuals and collectives?</p><p>The response was clear; you cannot attribute revenue to AB testing accurately, but we need to.</p><p>Fast-forward two years later and it would appear it’s less clear-cut. (Poll is <a href="https://www.linkedin.com/posts/davidleemannheim_a-burning-hole-in-my-pocket-can-it-be-done-activity-6798374948483166208-XAn6">here</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zoH4MDstSV4sSvC2" /></figure><p>I truly believe that it is difficult to attribute revenue to individual experiments. And that it shouldn’t be done.</p><p>Adding to that, forecasting that revenue is less possible, with less accuracy.</p><p>Adding to that, collating experiments to understand overall revenue impact is nigh impossible, with even less accuracy than the less accuracy that came before it.</p><p>Perhaps it’s the word accuracy and the determining degrees of acceptable confidence levels? From what I’ve seen in the industry, most experiment attribution falls on the wildly inaccurate rather than the moderately inaccurate. I’m talking about taking a winning experiment, doubling its impact (given it’s a 50:50 split) times it by 12 for each month, and cumulatively adding it to other winning experiments’ level of inaccuracy.</p><p>For me; experimentation done poorly is a very expensive way to slow down and over complicate bad decisions. Add in misattribution into that dangerous mixing pot, and you have an unethical delusion of “we’re doing things right”. Maybe that’s why my paranoia subverts the need to attribute?</p><p>In my previous article I wrote about <a href="https://www.linkedin.com/pulse/why-do-we-assume-experimentation-financial-gain-david-mannheim/">why we assume experimentation is all about financial gain</a>. The history of where this stigma comes from.</p><p>In this article, I’m going to talk about why experimentation isn’t about financial gain; literally. As in:</p><ul><li>why you shouldn’t attribute revenue to individual experiments</li><li>why you cannot accurately forecast that revenue attribution</li><li>and why you cannot collectively attribute from multiple experiments</li></ul><h3>You shouldn’t, really, attribute revenue to individual experiments</h3><p>…but we have to. So we have to deal with it. Businesses are run on performance metrics and experimentation is a method of validating performance. Here are some reasons why we shouldn’t attribute revenue to experiments, and why the accuracy of said attribution can throw you under a bus.</p><h3># It takes longer, so ask what is the lost opportunity cost</h3><p>The purpose of experimentation is to prove or disprove a hypothesis. It’s designed to validate. It is not designed to measure the difference of said validation.</p><blockquote>“Most of the time, we run an experiment to see if there’s a difference in performance between the variations. Simply put, “Is B better than A?” But experiments aren’t typically designed to show “By how much is B better than A?” <a href="https://www.linkedin.com/in/stephenpavlovich/"><em>Stephen Pavlovich</em></a></blockquote><p>We’re not looking at attributing revenue uplift because, typically, most of the time we’re testing on a binomial metric. Did X improve Y: yes or no. But when we look at non-binomial metrics that are a continuous scale, or a range, we introduce another complication altogether.</p><p>Revenue, the performance difference, is a non-binomial figure. And thus there are complications.</p><p>A single conversion could be £1,000 or £500 or £50,000 (and herein we could debate that we should be removing outliers, reducing our testable sample). Hubert Wassner, Chief Data Scientist at AB Tasty, calls this concept “a statistical nightmare”. You can never be really sure on how much you’re gaining (non-binomial), the only thing you can assess is that you are gaining something (binomial). That something, Wassner continues, is associated with probability.</p><blockquote><em>“When we’re talking about a hypothesis, this is simply a “yes or a no” answer. The user either successfully converted, or they didn’t. When we look at revenue, we can assess that you are gaining or losing something, but the size is near impossible to assess. From a statistical solution, we can assess that there is a difference in the size, but the size of the difference is too difficult to assess because it will be very variable.” </em><a href="https://www.linkedin.com/in/hubert-wassner-ab4aa84/"><em>Hubert Wassner</em></a></blockquote><p>That being said, could you report against a non-binomial metric to varying degrees of accuracy? Yes. Taking pharmaceutical samples of, say, a drug that looks to increase lifespan. That lifespan is a range, and whilst we can state whether the drug impacted the lifespan, the question of “by how much” is required as part of the hypothesis in of itself. In other words, whilst the hypothesis is binomial (yes or no), the ability to measure the hypothesis can be non-binomial (a continuous scale) and is achievable to degrees of accuracy.</p><p>The problem? It takes much longer. Your primary metric to determine the output of the hypothesis is a continuous scale, meaning you have more variables to be certain on the exact outcome.</p><blockquote><em>“To have more confidence in the size of the uplift (not just that fact that it exists), you’d have to run the experiment for far longer. For example, you might continue to serve a losing variation (A) to 50% of your traffic so you can have greater confidence in the value of the winning variation (B). That comes at the expense of your customers and business.’ </em><a href="https://www.linkedin.com/in/stephenpavlovich/"><em>Stephen Pavlovich</em></a></blockquote><p>Given our experiments are required to reach a level of statistical significance to be confident in proving a validated outcome, if we were looking at revenue as a statistically significant metric, this would take significantly longer than the duration of the experiment to prove and accurately attribute.</p><p>The largest contributor to the “delay”, when using revenue as a significant measure of test outcome, is that it is much more difficult to gather sufficient testable data simply because the vast majority of outcomes (at a session level) will be £0 revenue. Added to that, we obviously get “null” outcomes when looking at conversion rate, in other words outcomes generated by users who don’t convert.</p><p>The summary of this is that there is more variability in non-binomial metrics (eg revenue) than bi-nominal metrics, therefore experiments require a larger sample size, take longer and could be a lost opportunity cost — or should certainly be a consideration. At which point accuracy becomes skewed due to the complexities.</p><h3># The purpose of experimentation is not about revenue gain</h3><p>At least, not all experiments, certainly. Surely not every decision or improvement we make is designed to make a direct positive impact on revenue? We worked with retailers who needed to appease brands for merchandising or aesthetic looks. We’ve created experiments to impact post-purchase customer satisfaction. We’ve ran tests to prove the CEO’s change in navigation right (or wrong); not a tactic I endorse or agree with by the way.</p><blockquote>“The second you start measuring every experiment on just revenue, you lose the ability to think strategically. You have to balance two things: solving your customers’ most urgent pain points and proving your impact to the business. if you let one of these two overwhelm the other, you’re headed for a dead end.”<em> </em><a href="https://www.linkedin.com/in/hpourkhalkhali/"><em>Hazjier Pourkhalkhali</em></a></blockquote><p>When Bravissimo approached us in 2018, the purpose was to reduce returns. A behemoth metric that took years to impact. It was impacted by making customers feel better and more confident in their decisions to ensure the fit was right (who knew so much went into choosing a bra?!). Qualitative metrics that did have quantitative values placed on them, but when cumulatively put together we knew we were positively impacting the customer experience. The attribution of which was less important, than the knowledge of “we knew we were moving in a positive direction”.</p><h3>Forecasting experiment attribution is even harder</h3><h3># Experiment results are based on a series of averages.</h3><blockquote><em>“Bottom line? What you’re doing is shift an average. On average, an average customer, exposed to an average AB test will perform averagely. There’s nothing earth shattering about that.” </em><a href="https://www.linkedin.com/in/craigsullivan/"><em>Craig Sullivan</em></a></blockquote><p>It’s very common to hear ‘conversion is down YOY’ or ‘AOV is down YOY’ but those top line metrics are a series of averages. Hell, the acronym AOV literally stands for average order value. It’s a metric compiled of hundreds of different metrics.</p><p>A common scenario, for example, might be that within the variant of an experiment, the variation has, luckily, received a few £1000 orders that hugely impact the revenue uplift and thus AOV. The same is the case for conversion rate; where a cohort of users eg. a broad-stroke of PPC, or users landing on a product page from PPC, might hugely affect the average conversion rate.</p><p>Both metrics are non-normal distributions. They change over time based on the composition of users, behaviour and metrics. For example, running the same experiment in, something as arbitrary as December and June, will result in two very different average results.</p><blockquote>“You can’t add the uplifts of winning experiments because you don’t know how they will interact or how your site and users will change. If you want to prove your impact, be conservative. You’re better off reporting a lower number everyone will defend than a higher number only a few believe.” <a href="https://www.linkedin.com/in/hpourkhalkhali/"><em>Hazjier Pourkhalkhali</em></a></blockquote><p>We need to ask questions like who’s inside the sample, who was exposed to the test, what kind of people, how much have they spent with you, how long have they been with you, what cohort of segment are they in? <a href="https://www.linkedin.com/in/ben-labay/">Ben Labay</a> talks about “lowering the denominator” i.e if you isolate the analysis to the audience and experience, the analysis and measurement is simplified. Of course, this is true of all experiments, not just non-standard distributions (like AOV). And a reason why I’m a big fan of testing cohorts, rather than averages.</p><p>Ultimately, there are percentages of the segment within the experiment that will respond strongly, and some that will respond poorly, if you don’t know the composition and response of those, then you’re reporting on averages; which lacks a level of accuracy.</p><blockquote><em>“Given an experiment we’ve run, how can we make a prediction for future behaviour? You can never do this based on average response on the site. As soon as you start plotting it out over a month, three months, twelve months, either you’ll hopelessly underestimate or hopelessly overestimate. Why? Because averages fundamentally lie.” </em><a href="https://www.linkedin.com/in/craigsullivan/"><em>Craig Sullivan</em></a></blockquote><h3># Experiment results are a snapshot of time and behaviour</h3><p>An experiment is a snapshot of time; the results of which are of a specific sample, who reacted in a specific way, providing a specific result.</p><blockquote><em>“We are using retrospective to forecast the future. There is a limit to how much this discipline can tell you” </em><a href="https://www.linkedin.com/in/timstewart/"><em>Tim Stewart</em></a></blockquote><p>Forecasting doesn’t take into account all the unforeseen external variables that could, and often do, happen. Regardless of degree of accuracy, forecasting is still, at best, an educated guess — because it largely involves understanding the past to understanding the future. And we’re not Marty McFly here betting on sports games.</p><blockquote><em>“We can never value the forecasted and expected return to 100% because of the conditional factors that are unexpected — and ultimately we’re talking about attribution here. “When we are forecasting business revenue — there are a f*ck-tonne more variables to consider.” </em><a href="https://www.linkedin.com/in/timstewart/"><em>Tim Stewart</em></a></blockquote><p>A new feature could have the “shiny” effect meaning usage is novelty and novelty wears off. A change in layout could have the “shock” effect meaning that users respond negatively to it because of uncertainty. Adversely, as users become conditioned to a way of behaving — a burger menu as your navigation or filters being at the top of a page — they respond positively over time because it’s what they know. Amazon being a prime (pun intended) candidate of this and how they’ve dictated the way a PDP looks for ecommerce despite not making any significant changes in years.</p><p>One consideration is that you can estimate the impact of an experiment within the testing period, but you assume there are no external factors that would influence it post the experiment concluding — of which there always are. For example, if the company decides to double the price of their products, the experiment will, naturally, not have the same uplift. Think about the stuff that you have control over changing (usually internally like brand) vs the stuff that you don’t have control over (usually external like new competitors)</p><h3>Collective experiment attribution is nigh impossible</h3><p>Then there’s the issue of, not just forecasting revenue uplift from a single experiment, but forecasting it from multiple experiments.</p><p>What happens if we create an experiment A designed to increase add to carts, and experiment B designed to increase product views. Where both experiments might be successful in their own right, the approach to increase product views might negatively impact the number of add to carts because of the intent of the users now coming through to the PDP. You can increase users getting through to the product, but it might not be the <em>right</em> product for example?</p><blockquote>“I’d be hesitant on grouping together multiple experiments and grouping the impact. You are grouping fuzzy results, which adds a lot of… fuzz. It tends to be how a lot of people look it and it’s misleading” <a href="https://www.linkedin.com/in/stephenpavlovich/">Stephen Pavlovich</a></blockquote><p>The more factors you change within your product roadmap — tested or not — or the more noise in the metrics you are using, the greater that degree of fuzziness, too. I can experiment on the checkout all I like, but if we’re adding in a new payment provider because our marketing department gets a kickback from Mastercard, surely this will impact the impact of the experiment; retrospective or predictive.</p><p>One option to combat this is to have a hold-back set of, say, 10%. In other words, 10% of users always see the ‘original’ or control. I question the purpose behind this — is it just naval-gazing? Proving a further degree of accuracy that ‘we were right to the degree of x%’ rather than just ‘we’re moving in the right direction.</p><h3>Bonus: Which leads me to the discussion of the term ‘accurate’…</h3><p>When I posted the poll up for others to see, the majority of comments were along the lines of this.</p><p>Questioning the term ‘accuracy’. It’s my view that if you can’t attribute accurately, then you should attribute at all. If we attribute a single experiment wrong, and forecast that, we’re magnifying the issue.</p><blockquote><em>“As experiments are typically designed to show that a difference exists (not in the size of the difference), it can be challenging to forecast from a single experiment. When you scale that across an entire experimentation programme, it magnifies that and can be misleading.” </em><a href="https://www.linkedin.com/in/stephenpavlovich/"><em>Steven Pavlovich</em></a></blockquote><p>There needs to be some give and take. We’re managing risks of making the wrong decision, so we should consider the impact of expected loss vs expected gain. Perhaps even more so.</p><p>We’re managing levels of effort, too. How much effort is required to increase your degree of accuracy. Is this just naval-gazing to try to get as near to 100% as possible? What about the lost opportunity time? Would you rather run more experiments or be more accurate in any form of forecasting that you are able to achieve? I’ll address this in my next article “Yes. We need to attribute revenue to experimentation. Here’s how”</p><blockquote><em>“Most of the time being accurate about the direction of the decision is more important than being precise about the exact nature of the outcome” </em><a href="https://www.linkedin.com/in/matthewpaullacey/"><em>Matt Lacey</em></a></blockquote><p>Am I being a bit over-precise with my language? There’s definitely some semantics in here, right? But I think there’s your answer, if revenue is an indicator of performance, <em>how accurate does your business need you to be with your experimentation attribution and does it require forecasting?</em></p><h3>Summary</h3><p>Revenue attribution from individual experiments is difficult and fuzzy and, in an ideal world, shouldn’t be done.</p><p>Revenue attribution from collective experiments is even fuzzier and nigh impossible to understand the impact of.</p><p>Forecasting attribution mostly lacks a reasonable degree of accuracy and, like any attribution, relies on the past to predict the future.</p><p>Why can you not accurately attribute nor forecast revenue to experimentation?</p><ul><li>Metrics and stimulus change over time</li><li>Experiments are designed as a snapshot in time</li></ul><p>We need to:</p><ul><li>Appreciate and agree on what levels of accuracy are needed for revenue attribution within your business</li><li>To that end, appreciate that if we need a high degree of confidence, as revenue is a distribution, experiments will take longer to run, perhaps reducing your cadence</li><li>And thus, what is the lost opportunity cost of doing so</li><li>Not focus on revenue attribution as much as “what the purpose of experimentation is for your business”. Different maturities will yield different approaches to revenue attribution</li></ul><p>I’m going to save “what to do” for next Friday’s article. How I, and others, attribute revenue to experiments. Perhaps, more importantly, how to communicate that to the rest of the business.</p><p>Be sure to subscribe to my newsletter at<a href="https://optimisation.substack.com/"> https://optimisation.substack.com/</a> where, every Friday at 7am, I’ll release a new series of thoughts and advice based on my time within CRO.</p><p>Leave a comment. Create debate. Have a drink. Disagree.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a22b1ac2a5c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why do we assume experimentation is about financial gain?]]></title>
            <link>https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8834715b83e6</guid>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 07 May 2021 06:20:31 GMT</pubDate>
            <atom:updated>2021-06-01T08:10:43.868Z</atom:updated>
            <content:encoded><![CDATA[<p>The over-simplification and virality of click-bait headlines for case studies designed for commercial gain and a wider audience has led to associating financial gain to an output. I blame Optimizely, the $300m button and WhichTestWon.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Em1SEd2JlsB2IltL" /></figure><p>A beautiful juxtaposition. A real email from a real client and one that I’ll never forget. Not because it wasn’t unusual asking to attribute individual and cumulative solutions. And then annualise them.</p><p>But because the sender went on to state that they were off to a 10-day TV advertising shoot. (and clearly understood my reticence in providing financial returns against investment)</p><p>It left me curious. If our contract renewal was so dependant on the attribution of 48x solutions, how must the bigger advertising agencies feel? You know, with the cost and subsequent lack of attribution of a national TV advertising campaign.</p><blockquote>“…people hire brand agencies from New York with double digit rebranding and they don’t care at all about ROI. They hire for someone like us for AB tests and they start discussing if the uplift is 3.1% or 3.2%.” <a href="https://www.linkedin.com/in/andremorys/">André Morys</a></blockquote><p>OK, we’re being slightly flippant. I won’t comment on how much Beyonce cost for starring in her Pepsi commercial all those years ago (<a href="https://www.billboard.com/articles/news/1481503/beyonce-partners-with-pepsi-for-50-million-deal">it was $50 bloody million</a>). But I am curious how people assume that we can easily attribute and annualise revenue to an experiment. Or that they expect to get a financial gain from experimentation, period.</p><p>Perhaps it’s the assumption of CRO managers wearing white lab coats and the belief that were smart?</p><blockquote>“We are supposed to be scientists with white lab coats and nerdy people — we should not only know what the result is, but exactly how much it’s going to pay back over the next 3 years.” <a href="https://www.linkedin.com/in/craigsullivan/">Craig Sullivan</a></blockquote><p>Or perhaps it’s the fact we’re number people. And therefore surely we must know what that return will be, because we deal with numbers all the time?</p><blockquote>“Optimisers in the industry have created a culture and influence that’s born out of numbers. It’s all numbers numbers numbers. We put our blinkers on and tear off in the direction of those numbers” <a href="https://www.linkedin.com/in/chriscallaghan/">Chris Callaghan</a></blockquote><p>I sound bitter, I know. It’s just; I firmly believe experimentation was not designed for the purpose of revenue attribution. It never was. You might say that the notion of an experiment was designed to <a href="https://www.bbvaopenmind.com/en/science/leading-figures/james-lind-and-scurvy-the-first-clinical-trial-in-history/">prevent scurvy</a>. The beauty of AB testing comes from validation; a level of confidence that your hypothesis is true or not. Why should that come with a financial gain?</p><p>To assign a financial figure against such an art is a gross over-simplification and commercialised view of what experimentation is designed to do. Much like the most recent, European Super League proposal commercialising the history and art of football in the Premier League. That’s probably the subconscious inspiration for this writing.</p><p>Hell, I ask, where did that notion of experimentation and financial benefit even come from?</p><blockquote>Should experimentation be about revenue? It’s nonsensical to judge every aspect of your business on revenue just as it’s nonsensical to judge experimentation on just revenue, <a href="https://www.linkedin.com/in/hpourkhalkhali/">Hazjier Pourkhalkhali</a></blockquote><h3>Why do we think like this?</h3><p>Why? Why is it so common for people to think that we can accurately attribute a revenue gain to AB tests or changes we make on site. Before the why, comes the how. How did this phenomena come about?</p><p>We’ll go into <em>why</em> that is the case in other musings next week, and the week after. And, like my writings on prioritisation, we must first understand why we hold a preconception that experimentation can easily be attributed, before we understand what to do about it.</p><h3>AB testing is over-simplified</h3><p>It should be to be more widely accessible.</p><p>Experimentation is a complex methodology where a lot of things can go wrong and the fate of a poor feature or bad experience rests on the shoulders of a decision. To accelerate adoption, we have had to over-simplify the methodology in of itself, in order to evangelise it. Therefore are we our own worst enemy?</p><p>Perhaps this is, indeed, our fault. We have set a rod for our own backs.</p><blockquote>As the experts, we set the expectation for the rest of the industry. There are so many stupid case studies out there that directly attribute the experiment impact — people that do stupid tests that have a 70% confidence level, 70% uplift and create £70m euros for a client”. <a href="https://www.linkedin.com/in/andremorys/">André Morys</a></blockquote><p>André isn’t wrong. Does showing ‘related products’ bar above the reviews section really increase the revenue of my entire site by 28%? Or does moving the words “men” and “women” to either side of the logo in the navigation really increase site revenue by 9.4%?</p><pre>* those are real examples - I was going to link to them, but thought I&#39;d be thoroughly told off by outing such examples</pre><p>Going back to the days of WhichTestWon, which was an online game that reported on individual experiments, users were asked to choose which experiment ‘won’ and subsequently provided the results of that experiment in £ and conversion % uplift.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*fytaaaeZEJF7nKKN" /></figure><h3>When did we start attributing financial gain?</h3><p>Optimizely might be one of the first.</p><p>Their ‘<em>first’</em> piece of marketing was an absolute barn-stormer that they used for years to come. And so they should have done. I genuinely think it was a brilliant, engaging experiment. I understand why they don’t use it anymore given that a certain republican may have divided judgement, somewhat.</p><p>“<a href="https://blog.optimizely.com/2010/11/29/how-obama-raised-60-million-by-running-a-simple-experiment/">How Obama raised $60m by running a simple experiment</a>”</p><p>The experiment was really nice from a marketing perspective — it was gamified, well-known (I think used in almost every Optimizely presentation I saw between 2010 and 2017), demonstrated a surprising series of results, engaged, topical. But most importantly, it held the belief that a series of 4 x variations unlocked the key to a disproportionate revenue gain of $60m. It’s in the title. It grabs your attention and simplifies a complex concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/0*6XU_jyDG7n4z_h6X" /></figure><p>Attributing financial gain to experimentation has long since been the staple of AB testing tools and agencies alike (inc. ourselves. More on that later)</p><p>Oh, wait, there’s the <a href="https://articles.uie.com/three_hund_million_button/">$300m button</a>, too (published a year earlier than the Obama example). Much like a Manchester United transfer rumour, no one knows who this was for; but some say Best Buy. In essence, the button copy was changed — whether within an experiment or not, we do not know — replacing the “Register” button with “Continue as Guest”, and included some reassuring copy on what would happen next</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xZPi89pM2T6GUiWp" /></figure><h3>And where are we now?</h3><p>The popularity of and exposure of AB testing has led to a viral effect. If one company saw a $300m uplift from the changing of the wording of a button, then can these effects be replicated on a different website or platform?</p><p>The answer is assumed to be yes. And one (folly or not) case study, spawns another, with spawns another, which gets the attention of other professionals who try the same on their website; creating a domino effect of, what we now know today as , ‘best practice’.</p><p>That best practice is both in the solution, and the approach. The focus on the output, the numbers, and therefore the perceived measurability of what we do as a practice, has created a perfect storm of “a practice that generates unprecedented revenue and returns” and one that is “accurately measurable”.</p><blockquote>“Because we say everything is measurable online — now people also believe that you can put an exact number on it. But that’s not possible. That’s the downside of believing that you can measure everything online.” <a href="https://www.linkedin.com/in/annemarie-klaassen-4a55a310/">Annemarie Klaassen</a></blockquote><p>The sheer fact that the industry has framed experimentation on, what Morys calls “hard ROI” (which he goes on to say is a “a big pain in the ass”) means that those who purchase such services, tools or consume that content are conditioned to thinking in the same way.</p><p>People are becoming more and more educated — I believe but the damage has been done. We need to make the case why these case studies aren’t reliable and inform our stakeholders.</p><h3>Am I being overly precious?</h3><p>Sometimes I admit, I easily slip into practitioner, evangelical mode. It’s easy to do so. To fly the flag for those experimenting, rather than truly understanding how a single methodology fits into a wider business purpose, or accepting the maturity of a practice into a commercial field. That being said, I will never accept a ESL (European Super League) bid. We must be emotionally mature enough to appreciate how the piece fits into the jigsaw.</p><blockquote>“It’s an area of grace; I don’t expect the FD to be as knowledge about experiment driven product development as I am; that’s an unrealistic expectation” <a href="https://www.linkedin.com/in/matthewpaullacey/">Matt Lacey</a></blockquote><p>At the end of the day, we’re just talking about attribution. And it’s been a difficult, almost subjective, nut to crack. Actually we’re talking about humans.</p><blockquote>“CRO is not a pure science. You can’t truly separately out all your variables. We’re dealing with things that are just messy. We’re testing with humans and humans are complex.” <a href="https://www.linkedin.com/in/matthewedgar/">Matthew Edgar</a></blockquote><p>I’ll go into this in more detail in the next two or three posts.</p><h3>Summary</h3><p>It is difficult, if not impossible, to provide an accurate ROI for experimentation. The over-simplification and virality of click-bait headlines for case studies designed for commercial gain and a wider audience has led to associating financial gain to an output.</p><p>Should we do this? No.</p><p>Do we have to? Conversely, I believe so.</p><p>To add to Craig Sullivan’s quote above, we need to appreciate the business value of experiments. That doesn’t have to be a financial gain, but I believe it does help stakeholders understand impact. How that is communicated without being over-evangelical is something I’m delving into next week and the week after.</p><blockquote>There’s no point being scientific about testing if you’re also not scientific about measuring the business value of experiments. <a href="https://www.linkedin.com/in/craigsullivan/">Craig Sullivan</a></blockquote><p>We need to help our stakeholders make informed decisions. Any attempts to quantify that are challenging, yes, regardless, they will still want to look at that overall impact.</p><ol><li>because financial gain is, ultimately, what they associate performance with</li><li>because of the preconception that an experiment is directly linked to revenue gain</li></ol><p>In the next article, I’ll be looking at why we can’t accurately attribute revenue to experimentation.</p><p>This is part of a 4-article series on whether, and how, you can attribute revenue to experimentation:</p><ol><li><a href="https://davidleemannheim.medium.com/why-do-we-assume-experimentation-is-about-financial-gain-8834715b83e6">Why do we assume experimentation is about financial gain?</a></li><li><a href="https://davidleemannheim.medium.com/no-you-cant-accurately-attribute-nor-forecast-revenue-to-experimentation-here-s-why-a22b1ac2a5c7">No, you can’t accurately attribute, nor forecast, revenue to experimentation. Here’s why.</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-1-ccf90a4416e6">Yes. We need to attribute revenue to experimentation. Here’s how: Part 1</a></li><li><a href="https://davidleemannheim.medium.com/yes-we-need-to-attribute-revenue-to-experimentation-heres-how-part-2-72e9d39d2a20">Yes. We need to attribute revenue to experimentation. Here’s how: Part 2</a></li></ol><h3>Don’t miss my spurious thoughts on optimisation</h3><p>I currently work just 2 days a week (and thoroughly enjoying it). But for the other 3 days, in a bid to keep me off Twitter and reading about the bull markets of Cryptocurrency, I learn, write and share. For nothing more than creating debate and enjoyment 😎</p><p>Subscribe to my newsletter at <a href="https://optimisation.substack.com/">https://optimisation.substack.com/</a>. New articles come out every Friday morning at 7am for you early risers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8834715b83e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do you decide whether an idea should be tested vs those things that should just go live without…]]></title>
            <link>https://davidleemannheim.medium.com/how-do-you-decide-whether-an-idea-should-be-tested-vs-those-things-that-should-just-go-live-without-9fd016278c7e?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fd016278c7e</guid>
            <category><![CDATA[experiment]]></category>
            <category><![CDATA[experimentation]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 30 Apr 2021 05:34:45 GMT</pubDate>
            <atom:updated>2021-04-30T05:34:45.540Z</atom:updated>
            <content:encoded><![CDATA[<h3>How do you decide whether an idea should be tested vs those things that should just go live without testing?</h3><p>TLDR; I am not a believer that you should test everything and, instead, it should have a swim-lane of prioritisation. Testing is a (brilliant) methodology for decision making. Factors that affect this inc. purpose, traffic, type of test and time.</p><p>Reminder; There’s no such thing as right or wrong, just contextualised experiences. I’ll be sure to give my opinion, with others, on questions people have asked at my <a href="https://app.sli.do/event/orj9eqab/live/questions">Slido</a>. Please comment, like, share, debate, drink when you’re done. I’d love us learn more and share our experiences together to help our CRO community. Particularly, moving away from revenue-only, sticky-filter-changing methodologies who call our beautiful discipline “CROW”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*P6lbBuEHyDcCmyCc" /></figure><p>Twice, I was invited to talk to the product team at ASOS. The first question I got from both sessions was “how do you decide what to test vs what do just do?”</p><p>Next to “which AB testing platform is best”, I’d say this is the number one question I get asked in general, and probably why it was upvoted so highly on my <a href="https://app.sli.do/event/orj9eqab/live/questions">Slido</a>.</p><p>There was a subscription business who approached us last year. 100,000 sessions a week. Very few conversions, mind you (less than 1%). Booming in a pandemic environment. They were sold on an AB testing platform that they used… twice in 12 months. Each test, we worked out, cost them nearly £10,000 to execute.</p><p>Clearly this is an extreme example, but should they have been testing? My argument is no. They lacked the maturity, resource and infrastructure to do so (hence why they needed an agency, I suppose). That AB testing platform, too, I felt, mis-sold them. I’m not sure if you’ve seen this, too, but it really makes my blood boil. Commissioned sales people, lacking an understanding of the true challenge of the business, over-selling a ‘solution’, over-promised, over-sold as a plug n’ play.</p><p>Testing is just a methodology to get more data on something and, therefore, validate. But it needs certain things to determine whether you should be testing full-stop, or whether you should be testing individual ideas. Those, are:</p><ol><li>What is the purpose of testing in your organisation</li><li>Do you have the traffic?</li><li>What is the type of change you’re looking at making?</li><li>Do you have time?</li></ol><p>It’s sexy; sure. So the theoretical evangelists amongst us will say “test everything”. How idealistic. But in the majority of cases that’s not fair or true because they lack the process, culture and in some cases the traffic and the time to execute.</p><blockquote>“In an ideal world you’d test pretty much everything, because every test you run gives you an opportunity to learn. We don’t live in an ideal world though and are tasked to deliver a commercial impact to a business” <a href="https://www.linkedin.com/in/-ryanjordan/">Ryan Jordan, Brainlabs</a></blockquote><p>So lets remove that stigma of testing everything.</p><p>We’re not Booking.com who tests <a href="https://blog.usejournal.com/how-booking-com-a-b-tests-ten-novenonagintillion-versions-of-its-site-25fc3a9e875b">Ten Novenonagintillion versions of its site</a>. I don’t even know what that number is. We live in a practical world where, as much as we’d like a testing culture of “test everything, fail fast”, in 99% of cases, it’s not going to happen.</p><h3>Purpose</h3><p>Testing is designed to validate and learn. All experimentation is, is another method of data collection. But I think we can categorise what the perceived purpose is of testing, or an individual experiment, for different people.</p><h3>1.Validation</h3><p>You should test when have a hypothesis that needs to be validated. Proven or disproven. The question therefore becomes; how much validation do you require? It’s a matter of confidence in an idea, and experimentation might be needed if you have little understanding of that idea in the first place.</p><blockquote>“Consider the strength of your hypothesis, how much supporting data and insight do you have to support this idea? If it’s an idea your CEO dreamed up over the weekend, there’s more risk involved than an idea backed up by Analytics data and user feedback” <a href="https://www.linkedin.com/in/emma-travis-032ab24a/">Emma Travis, Speero</a></blockquote><h3>2. Reduce risk</h3><p>Experimentation is (also) the mitigation of Type 1 errors ie. preventing changes or behaviours that could lead to harm.</p><p>Below we ask “is B better than A”. Flipping that on it’s head, a method of testing could be about demonstrating that “B is not worse than A”. This is occasionally known as non-inferiority testing.</p><blockquote>“I’m not sure experimentation is really about uplift; a lot of it is about mitigating risk. We may not experience a really harmful event, we still would have had the protection against it. For example, you may have collision car insurance, but not have had any accident over the year. What was the value of the insurance? Certainly not zero” <a href="https://www.linkedin.com/in/mattgershoff/">Matt Gershoff, Conductrics</a></blockquote><p>Take the example of <a href="https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7">AirBnB’s redesigned search page</a>. They all thought it was ‘clearly’ better; and their users agreed in qualitative user studies. To keep in the spirit of their testing culture, they did test the new design to ensure they wouldn’t create a negative effect. Actually, also, more importantly, to gather knowledge about which aspects did and didn’t work.</p><p>They ended up with a neutral result. (in the end they found a bug that affected IE users and noted it increased click-through action of more than 2%)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R5BGBA1qQetSUofj" /></figure><p>I’d advise you to read <a href="https://www.linkedin.com/in/geoprofi/">Georgi Georgiev</a> <a href="https://blog.analytics-toolkit.com/2017/risk-vs-reward-ab-tests-ab-testing-risk-management/">risk vs reward article here</a> and review how to get an optimal ratio between risk and reward and, using calculations, achieve a better ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Kei9VS8KmSKRyBy" /></figure><h3>3. Revenue Attribution</h3><p>A super quick one here, if you’re testing for revenue attribution, forecastable or not * this purpose will often dictate what you should and shouldn’t test.</p><blockquote>“If you are looking for long-term effects, you need to find metrics that predict these (the OEC problem), or run experiments for long and continue measuring. The latter is hard in practice due to survivorship bias in online cohorts, when cookies are used to track users” <a href="https://www.linkedin.com/in/ronnyk/">Ronny Kohavi, AirBnB</a></blockquote><p>*not like you should do, but let’s be honest, most / everyone does or, at the very least, appends a level of ROI to their results.</p><p>It will dictate what you should and shouldn’t test by testing on higher priority pages or segments, looking for attributable revenue to demonstrate return and value. I’m not the biggest fan of this ‘purpose’ as it a) inherently skews your prioritisation modelling towards value away from risk-management b) is not the designed nor foundational purpose for AB testing and c) assumes revenue is attributable from a test; when in reality it is a non-binomial metric.</p><h3>4. Learning</h3><p>Experimentation is an explanation. Did we prove that B is better than A?</p><p>…but it is limited in its exploration. Why is B better than A.</p><p>It’s because you’re proving or disproving a hypothesis, only. You can learn from it, sure, but it is limited in what you learn because your parameters are set. There’s only so much data can infer, rather than inform.</p><p>I encourage you to think about the question “what am I wanting to learn from this test?” That will help you decide whether you answer your questions through experimentation, or something else. It will also help you understand whether your change is solves a genuine problem. Do you really want to learn something from changing button copy…?</p><p>In which case other exploratory methods might be more suited to your answers of “what am I wanting to learn from this?”</p><blockquote>“Testing doesn’t have to mean AB testing, it isn’t the only method to validate an idea. User testing, design testing and copy testing can all provide you with data to support decision making and identify optimisation opportunities.” <a href="https://www.linkedin.com/in/emma-travis-032ab24a/">Emma Travis, Speero</a></blockquote><p>In summary, always linking it back to the “why are you testing” should help you keep you on track whether you should be testing a solution.</p><h3>Traffic</h3><p>Some claim a minimum number of users (e.g. 10,000) is necessary. I only partially agree, as it’s a question of confidence. Presidential polls are done using samples of 1,000–2,000 people. Many studies in sociology are done on hundreds of users.</p><blockquote>“…being comfortable with the fact that you’re not going to get [a] 100% solution, and understanding that you’re dealing with probabilities, so that you don’t get paralyzed trying to think that you’re going to actually solve this perfectly” Barack Obama</blockquote><p>Barack goes on to state that being just 51% confident is enough and that, along the way, your approach can be moulded. I’m not suggesting at 51% statistical significance we should stop our tests; I’m advocating that I’d rather be 51% confident than 49% confident. And experimenting gives me confidence that a treatment will have a positive impact on our users.</p><p>The alternative to experimentation is opinion.</p><p>That all being said, the approach of adequate sample sizes and needing to run tests to a degree of confidence is still preferred in my opinion.</p><p>If sample sizes are low, alongside of a less immature process that I would potentially consider other methods of testing — user testing perhaps — rather than practical AB testing.</p><h3>Type</h3><p>What is the change that you’re looking at making? I tend to categorise changes into one of three pots:</p><ol><li>Usability</li><li>Anxiety</li><li>Motivation</li></ol><p>There’s bugs and load time which are foundational to the above, too</p><p>The first, usability, is inherently linked to just making things easier. These are both a) the changes that industry ‘test’ the most and b) the changes that, on average, have the least impact. Sticky filters. Sticky call to actions. Sticky everything. I wrote about how this is causing us to <a href="https://www.linkedin.com/pulse/losing-our-way-experimentation-david-mannheim/">lose our way with experimentation here</a>.</p><blockquote>“I’d prioritise any experiments that look to change the way our users think or feel about how they behave. Changes to address usability issues (which, are usually less impactful) should be de-prioritised from testing, because it’s either an already validated behaviour or problem and / or it’s unlikely to carry significant risk” <a href="https://www.linkedin.com/in/-ryanjordan/">Ryan Jordan, Brainlabs</a></blockquote><p>The second and third and intrinsically linked to attitude that then affect behaviour. They are 34x and 100x more impactful than usability respectively.</p><p>Qubit <a href="https://drive.google.com/file/d/1FnSCrvLezC8scDmGZHdyLcNVVNZyVvxt/view">published a paper</a> in which they found that “Of 6700 experiments, 90% had an effect of less than 1.2% on revenue. Most simple UI changes are ineffective. Of the 29 common categories of treatment included in this paper only 8 have a greater than 50% probability of having a positive impact on revenue per visitor.</p><p>As User Conversion, we also wrote about <a href="https://www.userconversion.com/wp-content/uploads/2021/01/What-type-of-experiments-have-the-biggest-impact.pdf">what type of experiments have the biggest impact here</a>.</p><blockquote>“With the small changes, the effort and time to test is often unnecessary. Resources and time are limited, we do not want to spend X man hours on an experiment that is going to run for 60 days only to give us an inconclusive result and little insight. Particularly when running this experiment may have come at the expense of another potential test.” <a href="https://www.linkedin.com/in/maxbradleymarketing/">Max Bradley, Zendesk</a></blockquote><p>Time</p><p>Resource is often cited as a reason why teams don’t experiment; particularly time.</p><blockquote>“At <a href="http://www.teamsnap.com/">TeamSnap</a> we have 15 million users so we can run AB tests fairly quickly. However, it is still expensive to run lots and lots of AB tests from a labor standpoint”, Ken McDonald, ex-TeamSnap</blockquote><p>I genuinely think the biggest barrier to adoption or acceleration of experimentation is the perceived notion that: it adds time.</p><p>In the short term, it probably does. But it validates the good decisions from the bad. The high performing development releases, from the non-impactful releases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*8dmFumEd87VXhqdT" /></figure><p>I’m an advocate of MVP (minimum viable product or feature) and through experimentation, surely you’d want to understand whether something works or not before fully building it and releasing it into the wild? That’s what <a href="https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_ec4a6228-96cc-4437-86a4-a196e5582ee1">Dropbox</a> did with their product before releasing it. And <a href="https://buffer.com/resources/idea-to-paying-customers-in-7-weeks-how-we-did-it/">Buffer</a>. And Hubble. And eSalon. And <a href="https://www.theguardian.com/business/2004/aug/07/1">Innocent</a>. I wrote a whole article on <a href="https://davidleemannheim.medium.com/experimenting-everywhere-de4030d7a83a">experimenting everywhere</a> that I presented to the AO team, discussing MVPs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sD8vdhrUG_adwX40" /></figure><blockquote>“Instead of embarking on a lengthy card-sorting process that would have taken us months to re-categorise and implement, we chose to redesign the navigation; how users interacted with it. We saved our team from an estimated £95k on a solution that took just 3 days to build” Adrian Hobson, Travis Perkins</blockquote><p>I like to think that experimentation gives us agility.</p><p>Whilst I appreciate the notion of exploration vs exploitation in this <a href="https://towardsdatascience.com/when-and-when-not-to-a-b-test-c901f3ad96d9">debate</a>, I think the notion of a positive result of the time and resource that could have been spent over a poor decision. trumps the sequential discussion here; although still worth pointing out.</p><p>“In order to exploit the situation by moving completely to the best scenario (e.g. show only that ad), we need to invest in exploring the situation until we are confident enough. This can be expensive since we might be paying to deliver an ad impression and/or because of the opportunity cost induced by not maximizing our return due to a “bad” choice of scenario(s). So, until we reach the desired confidence in our decision to move entirely to the designated superior scenario, we are purely paying to explore; after we have reached this point, we can fully exploit that situation (given the scenarios do not change over time). This doesn’t sound like an optimal solution, does it?”</p><p>Summary</p><p>I’d recommend asking yourself a variety of questions.</p><ol><li>What is the purpose of testing in your organisation</li><li>Do you have the traffic?</li><li>What is the type of change you’re looking at making?</li><li>Do you have time?</li></ol><p>It doesn’t have to be in a specific process, it’s just about feeling it and understanding your prioritisation methodology; the sexiest of topics.</p><blockquote>“If the primary goal for the CRO programme is to increase the average revenue per user (in which most cases it is) and we are observing data which shows hesitation to complete the purchase on the checkout page then we should test a treatment on the checkout page as it aligns with the CRO goal.” <a href="https://www.linkedin.com/in/amit-chhatralia83/">Amit Chhatralia, Cisco</a></blockquote><p>Consider risk and resource as much as possible; even if the potential return may be unknown.</p><blockquote>“When the cost of running A/B tests outweighs the potential return [is when you don’t test]” Alhan Keser, ex-American Express</blockquote><p>As a recommendation, I’m a big fan of MVP, low-fidelity options. If you’re looking at creating a product finder, consider a quick survey instead. If you’re looking at restructuring your information architecture, consider the front-end look to the user first. If you’re looking at creating a new feature, create a false-door test and gauge user interest first.</p><h3>One last thing …</h3><ol><li>Please feedback on this article — was it helpful? Awesome? Crap? <a href="https://forms.gle/eX9ztvxDXSdFqJ9CA">https://forms.gle/eX9ztvxDXSdFqJ9CA</a></li><li>Ask questions and vote <a href="https://app.sli.do/event/orj9eqab/live/questions">https://app.sli.do/event/orj9eqab/live/questions</a>. I answer these on a weekly basis by taking the top voted topic and writing about it with my stories and experience.</li><li>Subscribe to my newsletter at <a href="https://optimisation.substack.com/">https://optimisation.substack.com/</a></li><li>Leave a comment. Create debate. Have a drink.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fd016278c7e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How are you best evangelising CRO into the wider company so that everyone understands the value…]]></title>
            <link>https://davidleemannheim.medium.com/how-are-you-best-evangelising-cro-into-the-wider-company-so-that-everyone-understands-the-value-1e38ac87ed31?source=rss-c2550b94b3c9------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e38ac87ed31</guid>
            <category><![CDATA[optimisation]]></category>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[cro]]></category>
            <dc:creator><![CDATA[David Mannheim]]></dc:creator>
            <pubDate>Fri, 23 Apr 2021 05:55:14 GMT</pubDate>
            <atom:updated>2021-04-23T05:55:14.154Z</atom:updated>
            <content:encoded><![CDATA[<h3>How are you best evangelising CRO into the wider company so that everyone understands the value and wants to meaningfully contribute to the success?</h3><p>TLDR; You can’t do it alone. It must be democratised in the form of structure and mindset. If you educate, encourage failure, gamify and demonstrate humility, I think you’ll take a good swing at it.</p><p>—</p><p><em>Every week, I answer the top voted question in my </em><a href="https://app.sli.do/event/orj9eqab"><em>Slido</em></a><em>. There are currently 37 questions on there. Id love for you to upvote or contribute to ask a question about my experience or if I can crowd-source other opinions.</em></p><p><em>Please also leave a comment on this article and give me your opinion. The debate genuinely helps others and me in learning — especially if you disagree.</em></p><p>—</p><p>In 2011, Lea Hickman led product for Adobe’s Creative Suite.</p><p>She knew that she had to optimise the product for a more sustainable, demanding customer base. And research demonstrated that they needed to move away from a desktop-centric, annual-upgrade model to a subscription-based model supporting all devices. Mobile, eh?</p><p>What a task to convince engineers to moving away from annual upgrades to continuous development. Or to convince finance to moving from a license model to a subscription one. Or convince sales to moving away from a reseller model to a direct relationship with the customer. Who, by the way, are now needing to rent their software rather than own it. They were furious. There was even a <a href="https://en.wikipedia.org/wiki/Change.org">Change.org</a> petition which reached over 30,000 signatures within a few weeks of the announcement.</p><p>Lea articulated a vision, rather than a methodology. Instead of compartmentalising finance, and engineering and marketing; she communicated a mission where they would all reach, and, if anything, over-communicated it. She created continuous prototypes getting all others involved in a more democratised process designed to rally product teams and executives and keeping them excited.</p><p>Now, Adobe has more than tripled its’ market cap to $60bn, thanks in large part to this transition.</p><blockquote>“This is one of the most impressive, nearly superhuman examples I know of a product leader driving massive and meaningful change in a large enterprise company.” Marty Cagan</blockquote><p>Lea is, indeed, superhuman and, being honest, I think you can evangelise CRO all you want, but without the right structure and mindset, you’re a lonely island.</p><p>The things I take from this story are that:</p><ul><li>There’s no such thing as over-communication</li><li>The clarity, story, proposition (the why) and the vision of the communication, is almost more important than the communication itself. Where do you want the company to be? Why?</li><li>She had permission</li><li>Democratising, empowerment and giving autonomy to others is vital for the vitality and therefore, buy-in</li></ul><p>I’ve been in the position of looking outside in. Watching those CRO programs that fail and those that succeed. And I can tell you now, those that succeed are those that:</p><ol><li>Decentralise the optimisation process</li><li>Have an optimisation mindset</li><li>The ABCs of EFGH (Educate, failure, gamify and humility)</li></ol><p>And that’s how I’m going to answer this question with experiences of my own.</p><p>—</p><h3>Decentralise the optimisation process</h3><p>So I don’t think you can do it alone. Sorry. Not even if you’re Lea. She had permission, for one, but she was also in a position of higher power.</p><p>Decentralisation is the best, arguably only, way you can reach the heats and minds above. Period. I’m all for centre of excellence centres by the way. That’s like an optimisation union. But a decentralised optimisation process is a structure where everyone optimises; or at least everyone can optimise.</p><p>In working with — must be over 100 digital companies by now — I’ve never seen a successful optimisation practice where it is centralised. By success I mean, listening to data and customers for every decision or release.</p><p>Amongst many reasons, I think that a centralised approach is discipline-led (of which optimisation is not a discipline, more on that later), and a decentralised approach is mindset-led. It’s an embedded belief; like a virus.</p><p>Think, too, about domain expertise. I’ll let Bhav explain.</p><blockquote>“In my opinion, the limitations of a centralised team only become apparent when one moves beyond low hanging fruit and enters a space where deep domain expertise is required. We kid ourselves into thinking that a centralised team has the necessary subject matter expertise to solve real problems just because they have a designer, an engineer and a CRO person who has a couple of test ideas. Decentralised teams obsess over the area they look after and all they need help in is how to take all their domain expertise and knowledge and turn them into hypotheses and tests.”</blockquote><blockquote><a href="https://www.linkedin.com/in/dodonerd/">Bhavik Patel,</a> Product Analytics Lead, Gousto</blockquote><p>But my overlying reason why decentralisation is the only route forward is that of autonomy and trust. A de-centralised approach decouples stakeholders from product changes.</p><p>When I was working with a large retailer, I remember the concept of a new checkout being discussed between the product and ecommerce teams. The approach was clearly centralised as the statements were along the lines of:</p><blockquote>“We need these 5 features releasing within the checkout to hit the target of a 65% progression rate”</blockquote><p>A decentralised approach would have gone along the lines of:</p><blockquote>“Hey product team can you get us a 65% progression rate in the checkout in the next 4 months?”.</blockquote><p>Therefore, letting them work out how to achieve this, thus creating autonomy. Thus creating empowerment.</p><p>A final note on structure, there’s a cracking <a href="https://medium.com/swlh/how-do-you-choose-the-best-growth-team-model-632ad5a85be9#.qid1mvx7s">series of interviews undertaken</a> by Andrew McInnes. He interviewed over 20 growth leads at companies like Uber and Pinterest to identify which was the best growth model — an independent or a functional one. The answer?</p><blockquote>“Choosing a growth team model based upon cultural, organisational, and strategic fit may be the better path forward. Don’t select for growth. Select for fit.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*h5aACi_osrBopMXI" /></figure><h3>Have an optimisation mindset</h3><p>What is an optimisation mindset?</p><p>Let’s not forget that experimentation is not optimisation. All experimentation is, is a method (a rather sexy one at that) to validate a decision by adding data. It’s sexy because a) it is (or can be) accurate and b) gamifies the decision making process. Who will win? A or B? By virtue, experimentation is a game. We’ll come on to this later.</p><p>Experimentation and optimisation aren’t terribly dissimilar as they champion data-driven user-first cultures. So I’m going to throw <a href="http://%20https//www.nesta.org.uk/blog/towards-an-experimental-culture-in-government-reflections-on-and-from-practice/">this</a> diagram out there which is excellent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MFNFbGOfDsNUyMRS" /></figure><p>Structure is a pre-requisite, perhaps, but mindset will determine it’s success.</p><blockquote>“Even [with the right structure], leadership would need to buy into a data driven and iterative mindset to drive growth. If leadership does not, you will fail no matter how you structure your growth team.” <a href="https://medium.com/@ahmcinnes?source=post_page-----632ad5a85be9--------------------------------">Andrew McInnes</a></blockquote><p>It sounds platitudinal, but conversion rate optimisation is a mindset, or culture, not a discipline. Structure is needed, but without mindset (or belief) you have nothing.</p><p>When I started User Conversion in 2015, I had the single mantra of: conversion rate optimisation is not a specialism, it’s a series of specialisms that work together. Here’s a screenshot from something hidden in our About Us page in 2015.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SgrrfvKCKy3gac-_" /></figure><p>Fast forward to today, and I still thoroughly believe this. Perhaps even more so. I believe that a single person should not be responsible for “optimisation”. Sorry “Optimisation mangers” but if you’re on your own in a business … you can try and do what Lea did but its very rare (and she was already in a high product position). It will take years, you do need to bring others along the ride with you, you will need that permission.</p><p>Look at the biggies and, for the sake of argument, look at experimentation as a methodology, instead of optimisation. There’s a theme to them all and it is one of mindset.</p><blockquote>“Netflix runs on an A/B testing culture: nearly every decision we make about our product and business is guided by member behavior observed in test”</blockquote><blockquote>“Microsoft’s Analysis &amp; Experimentation team consists of more than 80 people who on any given day help run hundreds of online controlled experiments on various products”</blockquote><h3>The ABCs of EFGH (Educate, failure, gamify and humility)</h3><p>OK, OK, we’ve spoken about structure and mindset as being foundational prerequisite. But let’s assume you do need to evangelise CRO. How do you best go about doing that?</p><p>Think &gt; Feel &gt; Act.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*nkONXPgRBSphZBav" /></figure><p>You want your business to behave in a different way, they must think in a different way. And if you’re not a lonely island, and you do have — at least concepts — of a decentralised structure in place, then we’re in prime position to change behaviour by affecting the think and then the feel.</p><p>Well. It’ll be hard.</p><blockquote>“Experimentation culture is something you earn, you don’t buy. Why? Because culture is completely changing the way you build your products or services.” Craig Sullivan.</blockquote><h3>1. Educate</h3><p>Educate is the think, to precursor the feel and act.</p><p>What is the purpose of optimisation within your business and how does it link to the wider business goals? Think about Lea continually reiterating that vision. And over-communicating it.</p><p>Break it down further (and I’m doing this because AB testing is often seen as the definition of optimisation). What is the purpose of experimentation within your optimisation process? The answer is is is designed to validate a change by adding more evidence.</p><p>We should know by now the purpose of experimentation is not to ‘win’ or ‘lose’ but to learn. It’s hypothesis testing at its very core. “Was I correct in this statement of thinking?”</p><p>Some of the ways at User Conversion that we’ve encouraged this mentality is to remove ROI from the equation of individual test analysis — and explain why (I won’t go into this RN). In our test analysis, demonstrating “primary metric analysis” “secondary metric analysis ”followed by three simple questions:</p><ul><li>What did we learn</li><li>What does this mean</li><li>What next</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KviCgSODGP3wJmZ2" /></figure><p>I remember an old conversation with the fabulous <a href="https://www.linkedin.com/in/annemarie-klaassen-4a55a310/">Annemarie Klaassen</a> stating before she even took a job at X company, she dictated how she would report on experiment success for this very purpose. The company wasn’t called X, I just forgot the name of it.</p><h3>2. Failure</h3><p>Failure is the feel, the precursor to act.</p><p>I could write a book on failure. <a href="https://www.amazon.co.uk/Adapt-Success-Always-Starts-Failure/dp/0349121516">Some have</a>. Failure is what makes us truly feel something. We often say “I feel it’s a failure” rather than “I think it’s a failure” I won’t delve too much into the concepts of failures because I think it’s widely versed. But I would recommend encouraging it, and celebrating it.</p><p>I remember a story from <a href="https://www.linkedin.com/in/simon-elsworth/">Simon Elsworth</a> and the team at Sky that stated when they first ‘started’, all the heads of got round a table. Of the 7, 8, 9 or so teams that were there, the experimentation team was the only was that talked about their failures, in comparison to all those who talked about their successes.</p><p>Another recommendation is to turn your tales of failure into stories. Stories help us engage. I LOVE stories. For those of you that don’t know me, I love Disney. Like. Obsessed. And so I’m fascinated with story-telling and will always point you in the direction of <a href="https://www.amazon.co.uk/creativity-inc/s?k=creativity+inc">Creativity Inc</a> or <a href="https://www.ted.com/talks/andrew_stanton_the_clues_to_a_great_story?language=en">Andrew Stantons TED talk</a>. But telling a good story can help us feel more and therefore be more engaged, especially when it resonates.</p><p>My most read article I’ve written is about <a href="https://www.linkedin.com/pulse/why-i-sold-user-conversion-david-mannheim-1f/">why I sold User Conversion</a> — at the heart of the story, it’s a success story about hustle and coming to terms with my own shortcomings. But people do empathise making the story that much stronger. The book, <a href="https://www.amazon.co.uk/Pixar-Storytelling-Effective-Pixars-Greatest/dp/1517699452">Pixar: The Rules for effective Story Telling </a>, actually goes into great depths of how their films have moments of physical or emotional life-or-death moments which creates conflict, and therefore emotional attachment to character and story. Apply the same to your mission.</p><h3>3. Gamify</h3><p>Gamify is the act. Well. It’s actually the best way to change mindset by acting.</p><p>Jamil Qureshi said it’s the best way to change behaviour, and I really like him.</p><blockquote>“We must take away the existing parameters that we believe hold us in place, and play the game of ‘what if’ more often. What if we had an unlimited budget? What if we had no budget? What if we gave this away for free? What if we charged for this? All great journeys start with a question. Move outside of what currently holds us in place to create new possibilities.”</blockquote><p><a href="https://www.linkedin.com/in/duncanwardle/">Duncan Wardle</a> says the same too in, I’m going to throw it out there, the <a href="https://www.ted.com/talks/duncan_wardle_the_theory_of_creativity">best talk I’ve ever seen</a>. I’m slightly bias as it’s a talk about Disney, but listen to how he describes the What If scenario mapping in relation to Blockbuster, Fantasia and Car washes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0_nFk5ouI6-xHVbZ" /></figure><p><em>[image courtesy of qoorio.app/o/agne.nainyte/i/qg2n55-s5]</em></p><p>But really, experimentation is the ultimate game. It’s A vs B. It’s win or lose (despite what I said before). It’s aha moments. And although experimentation is just a form of optimisation, it is the sexiest form. Let’s be honest. It is. It’s sexy because it’s competitive and a game by it’s very nature.</p><h3>4. Humility</h3><p>A lot of optimisers I have seen have pride; which teeters over into ego. Myself included. We all have ego. They focus on others’ failures. With a microscope. Demanding that they change through a critical lens. “You’re doing that wrong” “What, why didn’t you test that first” “Hold on, the button is … RED”</p><p>Such criticism creates isolation. Can you imagine continually forcing the need or drumming the message of “failure” to a product team who, inherently, want to succeed?</p><p>Really, optimisers need humility. The emotional intelligence to coach others through the why and how to reach the outcome. I should change this section from educating to coaching really. <a href="https://www.linkedin.com/in/mrmarkleach/">Mark Leach</a> always taught us that there is a distinct difference between being “passionate” about something and “caring” about something. I think he was on to something.</p><p>Less about ego, but consider who you’re speaking to. Like Craig said, culture is earned. My good friend Manuel says that (in a wicked article <a href="https://effectiveexperiments.com/blog/questions-to-ask-when-democratizing-experimentation-in-an-organization/">here</a>)</p><blockquote>“democratising [testing] is about people first and understanding them” <a href="https://www.linkedin.com/in/manueldacosta/">Manuel Da Costa</a></blockquote><h3>Summary</h3><p>Let’s answer your question.</p><p>In order for everyone to “understand [conversion rate optimisation’s] value and meaningfully contribute to its success”, one must democratise. Because to behave you must first think and feel. To think and feel, you must empower.</p><p>You democratise through structure and mindset as foundational to your efforts.</p><p>I’d recommend following the model of EFGH.</p><ol><li>Educate; continually relate back to the vision and over-communicate</li><li>Failure; celebrate and encourage through stories</li><li>Gamify; what ifs remove the art of the possible</li><li>Humility; demonstrate care, not passion.</li></ol><h3>Finally</h3><p>One last thing … I’d love it if you could complete one of the below. It really helps me to get feedback, speak to people, create relationships and learn from others.</p><ol><li>Ask questions and vote <a href="https://app.sli.do/event/orj9eqab/live/questions">https://app.sli.do/event/orj9eqab/live/questions</a>. I answer these on a weekly basis by taking the top voted topic and writing about it with my stories and experience.</li><li>Subscribe to my newsletter at <a href="https://optimisation.substack.com/">https://optimisation.substack.com/</a></li><li>Leave a comment. Create debate. Have a drink.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e38ac87ed31" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>