<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Joseph Kay on Medium]]></title>
        <description><![CDATA[Stories by Joseph Kay on Medium]]></description>
        <link>https://medium.com/@joseph.kay?source=rss-ce6ae7115614------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BLEXQKoL_cEYQUpbMUBIKg.jpeg</url>
            <title>Stories by Joseph Kay on Medium</title>
            <link>https://medium.com/@joseph.kay?source=rss-ce6ae7115614------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:12:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@joseph.kay/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Users don’t come first]]></title>
            <link>https://uxdesign.cc/users-dont-come-first-47e457043e66?source=rss-ce6ae7115614------2</link>
            <guid isPermaLink="false">https://medium.com/p/47e457043e66</guid>
            <category><![CDATA[strategy]]></category>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[ux-design]]></category>
            <category><![CDATA[product]]></category>
            <dc:creator><![CDATA[Joseph Kay]]></dc:creator>
            <pubDate>Wed, 18 Jul 2018 11:43:27 GMT</pubDate>
            <atom:updated>2018-07-24T13:17:39.907Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9TANbw3pvehrBih-JLN8dw.png" /><figcaption><a href="http://govdesign.tumblr.com/image/147037699728">Image credit</a></figcaption></figure><p>It’s pretty common for people working in product or UX teams to talk about “user advocacy” and “putting users first”. But what does this actually mean? And is it a good idea?</p><p>Understanding your users and your potential users is central to building a successful product, but this isn’t the same thing as advocating for users or putting them first.</p><p>In fact, this line of thinking can damage the way your team and your company works.</p><h3>What does user advocacy mean?</h3><p>Being an advocate for the user suggests that you ought to be on the user’s side, making sure that their interests are heard, and then fighting for them.</p><p>“Users first” is more often used to describe a whole business that puts the interest of its users ahead of everything else. It’s user advocacy at the company level.</p><p>I understand what the potential benefits of user advocacy are. There are plenty of companies that don’t pay enough attention to their users. They could build better products if there was someone fighting for the user.</p><p><strong>But this is a problem that could be solved by educating people about the value of user research and working with them to make good decisions.</strong></p><p>Advocacy, on the other hand, is not a good way to approach this.</p><h3>You don’t want what the users want</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FZKZiW6GSx8eSA%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FZKZiW6GSx8eSA%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FZKZiW6GSx8eSA%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="193" frameborder="0" scrolling="no"><a href="https://medium.com/media/17d62d079cacc41c17a9de66c083f9f7/href">https://medium.com/media/17d62d079cacc41c17a9de66c083f9f7/href</a></iframe><p>There are plenty of ways in which a user’s interests might not align with the company’s. I’ll give an example from Lyst (where I work as a user researcher).</p><p><a href="http://www.lyst.com/">Lyst</a> is a website and app that helps you find clothes, shoes and accessories from thousands of brands and retailers in one place.</p><p>Instead of searching for clothes to buy, some people come to our website to get the original professional photos of things that they want to sell on eBay.</p><p>At one point, we made a change to the website that made it harder to get hold of these photos (for a completely unrelated reason) and some users complained. We ignored their complaints because they aren’t using Lyst in the way we intended and they aren’t making us any money.</p><p>There’s nothing wrong with people using Lyst in this way, but should we be advocating for them? Fighting for them? Designing for them? No.</p><p>User advocacy puts you on the user’s side, but that’s only <em>sometimes</em> the right place to be.</p><h3>Companies that put users first go bankrupt</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FOPRRK6P5Erbt6%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FOPRRK6P5Erbt6%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FOPRRK6P5Erbt6%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="228" frameborder="0" scrolling="no"><a href="https://medium.com/media/44038f14db4a19cf234d7ed182165bdc/href">https://medium.com/media/44038f14db4a19cf234d7ed182165bdc/href</a></iframe><p>The idea of having a whole company that puts users first is ludicrous.</p><p>What would it look like in practice? Imagine a bank that put its users first, or a sweet shop. I’d ask for free sweets and free money!</p><p>Users, by and large, don’t care about the success of your business. If I saw that a sweet shop was offering free sweets, I probably wouldn’t stop to think about their bottom line. I’d assume that they had that covered.</p><p>Any company that put its user’s interests ahead of everything else would quickly go bankrupt.</p><p>Now, you could reasonably argue at this point that this isn’t what people mean when they say “users first”. <em>Obviously we aren’t going to deliberately go bankrupt in the pursuit of pleasing our users.</em></p><p>Fair enough. But what does it mean then? If the users are first, what comes second? It has to mean something to be worth saying.</p><p>I don’t think there’s much of a risk of companies bankrupting themselves like this, but there is a risk of specific people or teams taking this language to heart and then making worse decisions as a result.</p><p>For example, someone might focus on the wrong feature, or carry out research into the wrong area.</p><h3>Advocacy erodes credibility</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FhTRtnHe0fWfYs%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FhTRtnHe0fWfYs%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FhTRtnHe0fWfYs%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="245" frameborder="0" scrolling="no"><a href="https://medium.com/media/54b275793946ab514c57e3389fed6aeb/href">https://medium.com/media/54b275793946ab514c57e3389fed6aeb/href</a></iframe><p>What if every team was advocating for something different? Could that work?</p><p>The UX team advocates for the users, another team advocates for the employees and another one advocates for the balance sheet. Then we have a big fight and end up somewhere sensible in the middle.</p><p>The problem is that there’s no way for anyone to have any credibility if they’re always fighting for a predetermined side. What about when there’s a fight that a user researcher really needs and deserves to win? How will they convince anyone?</p><p>It would be very easy to dismiss someone’s opinion on the basis that they <em>always say that</em>, regardless of whether it’s a good idea.</p><h3>Not doing research is OK</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fo5oLImoQgGsKY%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fo5oLImoQgGsKY%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fo5oLImoQgGsKY%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="239" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0dc846763f8d8442114a78befc76aa9/href">https://medium.com/media/d0dc846763f8d8442114a78befc76aa9/href</a></iframe><p>This is an important part of having credibility.</p><p>As well as “advocating for users”, sometimes user researchers are expected to “advocate for user research”.</p><p>Again, it’s very important to educate people about the value of user research, but user researchers need to be experts in <em>not doing research</em> as much as they are experts in <em>doing research</em>.</p><p>The idea of advocating for user research suggests that we should always want to be doing more research. But doing research often isn’t the best thing for the business. There is always a trade-off between the value that the research will provide and the time and money that would have to go into the research.</p><h3>Conclusion: put the business first</h3><p>Every business has to put its own survival ahead of the interests of its users.</p><p>This doesn’t mean ignoring the users. Every business will benefit from understanding its users better and making the product better for them. Being <a href="https://www.usability.gov/what-and-why/user-centered-design.html">user-centred</a> is a good thing.</p><p>But it’s a good thing <em>because</em> it’s good for the business.</p><p>User advocacy fails as a strategy because it implies that you should be on the side of the users, which isn’t always in the interest of the business. And adopting advocacy as a general decision-making strategy would just lead to nobody having any credibility.</p><p>Instead, everyone should do their best to understand the important factors in how a business makes decisions, and do their best to educate their colleagues about the factors that they know the most about.</p><p>User researchers should educate people about the value of user research and understanding users. They should be experts on how to do research and whether to do research.</p><p>That way they will be credible if they need to persuade someone that more research is needed.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2F50d69a%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2F50d69a%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/05d5fd32eda31cbd1b83287606744532/href">https://medium.com/media/05d5fd32eda31cbd1b83287606744532/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47e457043e66" width="1" height="1" alt=""><hr><p><a href="https://uxdesign.cc/users-dont-come-first-47e457043e66">Users don’t come first</a> was originally published in <a href="https://uxdesign.cc/">UX Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The problem with NPS benchmarking]]></title>
            <link>https://uxdesign.cc/the-problem-with-nps-benchmarking-ae6dd37df2fb?source=rss-ce6ae7115614------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae6dd37df2fb</guid>
            <category><![CDATA[net-promoter-score]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[user-experience]]></category>
            <dc:creator><![CDATA[Joseph Kay]]></dc:creator>
            <pubDate>Wed, 11 Jul 2018 10:29:01 GMT</pubDate>
            <atom:updated>2018-07-13T03:39:15.612Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QE9Im_uFYeZGMytqxGN5IQ.png" /></figure><p>Net Promoter Score is a type of survey used to measure customer loyalty. A lot of companies use NPS, so it’s seen as a good way to benchmark your performance against other companies.</p><p>This article is about how the inconsistencies in how NPS is implemented makes accurate benchmarking impossible.</p><p>I had been meaning for some time to write about the problems with the NPS. But then in December 2017, <a href="https://medium.com/u/b90ef6212176">Jared M. Spool</a> wrote an article criticising NPS pretty thoroughly. <a href="https://articles.uie.com/net-promoter-score-considered-harmful-and-what-ux-professionals-can-do-about-it/">You can read it on his blog here</a>.</p><p>I’m going to talk about an area that Jared Spool didn’t completely cover. But if you aren’t familiar with NPS, I’d recommend that you read Jared Spool’s article first.</p><h3>How is it meant to be done?</h3><p>Fred Reichheld, the inventor of NPS, wrote the question like this:</p><blockquote>“How likely is it that you would recommend [company X] to a friend or colleague?”</blockquote><p>Then you’re supposed to get the user to pick a number between 0 and 10, where 0 is labeled “not at all likely”, 5 is labeled “neutral” and 10 is labeled “extremely likely”.</p><h3>What does it look like in practice?</h3><p>Here are some examples of NPS questions I’ve been asked on websites and in emails:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*P6UHgByG898lWvDYuhYRJA.png" /></figure><p>This one is from 123-reg. The main thing to note here is that the question is slightly different. Instead of asking if you “would” recommend, this is more about whether you will recommend. The question is no longer hypothetical.</p><p>They also don’t have the “neutral” label for the number 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*Z8lQP68glyixV9NZSjUO3w.png" /></figure><p>Typeform have chosen to use different end-labels. “Not likely” and “very likely” are less extreme than “not at all likely” and “extremely likely”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*-sTdEI0sEottMs2keJESPA.png" /></figure><p>This one from Coverwise has uses one of the original end-labels, but not the other one. Also, they’ve chosen to have the options run vertically, rather than horizontally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*MhavU07I-CLdJURzrt5j1Q.png" /></figure><p>Twitter has “extremely unlikely” as their 0 label, instead of “not at all likely”. Most NPS implementations have the numbers increasing from left to right, but this one has it the other way around.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*56EbzpSepn1lxHfjhj7n7A.png" /></figure><p>7Geese have split the number scale in two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*HM0dHeRPtD9iW36jAqkhjw.png" /></figure><p>This text from my GP surgery uses a fully labeled 5-point scale. It also adds a caveat about the scenario in which you would make the recommendation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*V0f-m2LjIcNFPC4x0NOkGg.png" /></figure><p>I found a screenshot of this one from Orbitz. Again, it specifies what kind of recommendation you might make. It also combines a colour scale with the number scale.</p><h3>Inconsistencies make fair comparisons impossible</h3><p>Here are some of the inconsistencies that are out there:</p><ul><li>How the question is phrased</li><li>The number of options that are labeled</li><li>The labels that are used</li><li>The size of the scale</li><li>The direction and layout of the scale</li><li>Other indicators of how to interpret the scale (colour)</li></ul><p>These factors can influence the responses that you receive. For example, there’s a well known <a href="https://measuringu.com/left-side-bias/">left-side bias</a>, so whatever is on the left side of your scale is more likely to be selected.</p><p>All of this doesn’t make much difference when you consider all the other flaws in NPS. A purity test for a broken system isn’t much use.</p><p>However, one of the arguments I’ve heard in favour of NPS is that, in spite of it’s flaws, it’s still useful for comparing the quality of your user experience with other companies, since so many companies use NPS.</p><p>These inconsistencies in how NPS is implemented show that this isn’t true. You can’t make a fair comparison when everyone is asking slightly different questions and presenting the options in different ways.</p><h3>Also, there’s the issue of where you ask it</h3><p>Jared Spool did cover this in his article, but it’s worth repeating briefly.</p><p>The other major inconsistency in how NPS is implemented is the circumstances in which the question is presented to users.</p><p>If you present it earlier or later on in someone’s journey, you’ll get different results. If you present it on a particular kind of page, you’ll get different results from other pages. The same goes for emails.</p><p>It comes down to who you’re asking and when you’re asking them. Where you ask people the question is a way of selecting which people will see it. For example, if you only show it after 20 page views, most of the people who are frustrated may have already left. And people will give different responses at different stages in their journey too.</p><p>This can come down to the random variations in how companies choose to implement NPS, but it can also be deliberate. Some companies reward their employees for better NPS scores, which can lead to them manipulating their NPS implementation to improve the score.</p><h3>Conclusion: don’t bother with NPS</h3><p>There are lots of reasons why NPS is rubbish. If you’re clinging on to it because you hope that it will help you to benchmark your performance against other companies, you should stop.</p><p><em>You might also like to read:</em></p><p><a href="https://uxdesign.cc/need-to-use-a-survey-stick-with-the-original-8ae723bdb78d">Need to use a survey? Stick with the original</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Ff%2F50d69a%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2F50d69a%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/05d5fd32eda31cbd1b83287606744532/href">https://medium.com/media/05d5fd32eda31cbd1b83287606744532/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae6dd37df2fb" width="1" height="1" alt=""><hr><p><a href="https://uxdesign.cc/the-problem-with-nps-benchmarking-ae6dd37df2fb">The problem with NPS benchmarking</a> was originally published in <a href="https://uxdesign.cc/">UX Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why user researchers should care about sample size]]></title>
            <link>https://medium.com/designing-lyst/why-user-researchers-should-care-about-sample-size-fbb0b4215b3a?source=rss-ce6ae7115614------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbb0b4215b3a</guid>
            <category><![CDATA[user-testing]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[qualitative-research]]></category>
            <dc:creator><![CDATA[Joseph Kay]]></dc:creator>
            <pubDate>Tue, 20 Feb 2018 10:17:32 GMT</pubDate>
            <atom:updated>2018-02-20T10:17:32.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*PYKZKt99vQVIX83LQAuohw.png" /><figcaption>Most of the circles are blue</figcaption></figure><p>User researchers often find themselves having to defend small sample sizes in their research.</p><p>Some kinds of research really do require large sample sizes and it’s common for researchers to have to explain why that doesn’t apply to user testing or research interviews.</p><p>Here’s the problem: constantly defending small sample sizes can make us complacent. We feel that user research is a sample-size-worry-free zone, but it isn’t.</p><h3>What’s the deal with sample size?</h3><p>I was always told that you need around 6 to 10 participants for user testing. <a href="https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/">Nielsen has a good article about why you only need 5.</a></p><p>You only need a small number of participants because the objective is to uncover potential issues with the design being tested. Nielsen has shown that 5 users will uncover about 85% of those issues.</p><p>You can apply the same logic to research interviews as well. If you use them to explore the various experiences and opinions that are out there about a specific subject, you will uncover most of them with a small sample. However, in my experience, interviews are usually broader in scope than user tests, so you would probably need more participants to get to 85%.</p><h4><strong>So why do we sometimes need large sample sizes?</strong></h4><p>Perhaps you don’t want to just uncover problems or insights. You might want to know how common something is.</p><p>Let’s say I want to know how many of my product’s users are women. If I speak to just 5 of them and 4 of them are women, I couldn’t confidently say that 80% of our users are women.</p><p>So instead, it would be better to survey a few hundred of them, which would give me a more accurate percentage. I can then use statistics to work out what the error margins are, which will tell me how confident I can be that the true percentage falls within a certain range.</p><h4><strong>So what’s the problem?</strong></h4><p>There are two problems.</p><p>Firstly, some people think that you always need a large sample size for any research to be valid. This can make life difficult for user researchers who have to convince them otherwise.</p><p>Secondly, by digging in on the side of low sample sizes, we (the researchers) can end up missing the examples in our own work where we needed a large sample size but didn’t have one.</p><h3>When are we getting our sample sizes wrong?</h3><p>If you think that sample size and statistics isn’t an issue in user testing or research interviews, there’s a chance that you might fall into one of these traps.</p><h4><strong>1. Comparing two versions of something</strong></h4><p>Let’s say you have two versions of a particular webpage and you want to know which one is better. They’re prototypes, so you can’t just do an A/B test.</p><p>If you do user testing with 5 participants, you won’t be able to have much confidence that the version that performs better would still be better if you tested it with everybody.</p><p>Instead of treating user testing like it’s an A/B test, researchers should stick to the process of identifying problems, making changes and then retesting. It’s better to focus on creating the best possible option, rather than selecting what appears to be the best performing option.</p><h4><strong>2. Percentages and proportions</strong></h4><p>If you have a large enough sample size you get to say things like “63.1% (±4.6%) of our users are women.”</p><p>If you do some user testing or interviews with 6 participants, you can calculate a percentage like that, but it wouldn’t be very precise or helpful.</p><p>Even if you don’t use a percentage, you might be tempted to say that “most” users do something. That sounds safer, but it’s just another way of saying “more than 50%”. You’d still need a large sample to be confident about that.</p><p>The same is true of negative claims like “our users have no difficulty doing X”. User testing can tell you that some people have a particular experience, because you’ve seen it happen. But it can’t tell you that nobody has a particular experience. That’s the same as saying “0%”.</p><h4><strong>3. Suggestive language</strong></h4><p>Even if you don’t make claims about users in general, you can still cause trouble.</p><p>If you report that “4 out of 6 participants were able to complete the task”, you’re making a factual claim.</p><p>But unfortunately there’s still a problem. Maybe <em>you </em>aren’t generalising about all of your users, but you’re handing a loaded gun to whoever reads the report.</p><p>That kind of statement encourages the reader to think in terms of numbers and proportions, which can very easily lead to generalisations. It also encourages them to rank the findings in terms of how many participants had each experience. Again, this is the sort of thing that requires a large sample size.</p><h3>We all need to care about sample sizes</h3><p>I decided to write this post after listening to <a href="https://www.mixed-methods.org/episodes/2017/11/29/the-true-value-of-user-experience-research-matt-gallivan-airbnb">an episode of the Mixed Methods podcast</a>, where <a href="https://medium.com/u/18e3a8e14a14">Aryel Cianflone</a> talked to <a href="https://medium.com/u/ffa12acc7070">Matt Gallivan</a> about how researchers feel under pressure to be scientific and more like data scientists.</p><p>Matt talks about facing skepticism about small-sample-size user research at Facebook and how he has dealt with that issue. They both make a case for embracing uncertainty in research, which is something I definitely agree with. It’s an interesting episode and a great podcast in general.</p><p>But I think that they create a bit of a false dichotomy, with user researchers and uncertainty on one side and data scientists and error margins on the other. I don’t think we can do our jobs well unless we understand some of that science too, if only to know what we can and can’t do with our data.</p><p>An understanding of sample sizes affects research design, analysis and reporting:</p><ul><li>Choosing the right method with the right sample size to match the questions we’re trying to answer.</li><li>Understanding what we can and cannot conclude from our data.</li><li>Communicating our findings in a way that makes it clear what is and is not supported by the sample size.</li></ul><p>We don’t have to become data scientists, but we do need to understand statistics well enough to avoid making mistakes. Or at least, we need to make sure that we are working very closely with people who understand statistics, so that we can get their input when we need it.</p><p><a href="https://lyst.workable.com/j/24613D9F79">Lyst is looking for another user researcher to join the team</a>. So if you like statistically sound user research, click on that link and send us an application.</p><p>We’re also hiring for a number of other roles too, including <a href="https://lyst.workable.com/">designers, product managers and developers</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbb0b4215b3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/designing-lyst/why-user-researchers-should-care-about-sample-size-fbb0b4215b3a">Why user researchers should care about sample size</a> was originally published in <a href="https://medium.com/designing-lyst">Designing Lyst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 presentation lessons from Jam London 2017]]></title>
            <link>https://medium.com/designing-lyst/3-presentation-lessons-from-jam-london-2017-900049d93abb?source=rss-ce6ae7115614------2</link>
            <guid isPermaLink="false">https://medium.com/p/900049d93abb</guid>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[public-speaking]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[presentations]]></category>
            <category><![CDATA[conference]]></category>
            <dc:creator><![CDATA[Joseph Kay]]></dc:creator>
            <pubDate>Fri, 17 Nov 2017 14:57:44 GMT</pubDate>
            <atom:updated>2017-11-25T16:52:07.459Z</atom:updated>
            <content:encoded><![CDATA[<p>I think presentations skills are under-appreciated. They’re seen as a nice-to-have, rather than a must-have.</p><p>In research, design and product management there are few things that are more important that communication. Not all communication involves standing up in front of an audience, but when somebody has to do a presentation, it makes a huge difference if they are a good presenter.</p><p>Two weeks ago, eight of us from Lyst went to Jam London 2017, a conference about building products and the lessons we learn along the way.</p><p>I want share a few presentation highlights from the conference, and talk about the art of presenting and how important it is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*JEZe2iH9RxrwLK0J." /><figcaption>Eva Liparova speaking at Jam London 2017 — <a href="http://www.jamlondon.io/">www.jamlondon.io</a></figcaption></figure><h3>1. Confidence</h3><p>Confidence is like the oil in the gears of a presentation. When it’s there, everything runs smoothly. When it’s missing, you really notice it.</p><p><a href="https://medium.com/u/abfbfeebe731">Eva Liparova</a> gave a presentation about her work at the V&amp;A. She was definitely one of the best speakers.</p><p>The way she stood, the tone of her voice and the pattern of her speech made it seem like she was talking to a handful of people, rather than hundreds.</p><h4>Why is it important?</h4><p>Confidence puts people at ease.</p><p>Have you sat through a presentation given by someone who was visibly nervous? It can be uncomfortable. The chances are you’ve been in their shoes, so you empathise with them and you feel nervous for them.</p><p>A nervous presenter won’t get their point across as clearly and the audience can be distracted by their nervousness too.</p><p>So what’s the solution?</p><p>I think we need to view presenting as an important skill and do more to help people develop it. It’s not enough to just throw people in front of an audience every now and then — they need practice, but they also need more guidance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*UhPf6ihhOrbChsv5." /><figcaption>Ed Cooke speaking at Jam London 2017 — <a href="http://www.jamlondon.io/">www.jamlondon.io</a></figcaption></figure><h3>2. Energy</h3><p>If you’re confident and your audience is at ease, the next important thing to have is energy.</p><p><a href="https://medium.com/u/39e75eb8a103">Ed Cooke</a>’s talk about Memrise was many things: it was funny, it was interesting, but most of all it was energetic.</p><p>As with confidence, energy is about how you move, how you speak and what you say. Ed moved around the stage, he gesticulated and he spoke quickly and colourfully.</p><h4>Why is it important?</h4><p>Energy keeps people engaged.</p><p>If you’re presenting without any energy it can give the impression that you’re not very interested in what you’re saying. In contrast, Ed seemed excited by what he was telling us.</p><p>Excitement is contagious. Energy is contagious. I was hanging on Ed’s every word.</p><p>It’s very common for a presentation to lack energy, and that isn’t always a problem. Maybe the content requires a more measured approach, or maybe you’re just a very laid back presenter.</p><p>But there are so many presentations that would be improved by an injection of energy. The content isn’t always the most exciting stuff. Maybe the audience has sat through several presentations already. If you want them to pay attention and remember what you’re saying, you have to make them feel it.</p><p>My colleague <a href="https://medium.com/u/4119c955d1fe">Tom Petty</a> recently gave an internal presentation about design at Lyst. He’s a great speaker who really gets into it, and it made a huge difference.</p><p>It’s common for senior people to give lots of presentations and, while they usually have lots of confidence, they don’t always create the kind of atmosphere that Ed and Tom do.</p><p>We should view presenting as a serious skill and require senior people to develop that skill if they are going to be entrusted with that job. Or we should nurture people who have a talent for presenting and give the job to them more often.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*pf5-7ersHkYCoKVdJe0ykw.png" /><figcaption>A slide from Pantelis Korovilas’s presentation at Jam London 2017 — <a href="http://www.jamlondon.io/">www.jamlondon.io</a></figcaption></figure><h3>3. Humour</h3><p>Humour is like the icing on the cake. If your audience is at ease and they’re engaged, you’re doing a great job. If you can make them laugh, even better.</p><p><a href="https://medium.com/u/6a0b0c1e0c3d">Pantelis Korovilas</a> spoke about a project that he had worked on at Hopper where it didn’t all go to plan.</p><p>It was a great presentation in general, but it also had a brilliant and funny moment that stood out.</p><p>He was talking about the folly of highlighting a feature to users with a call-out, instead of questioning the way the feature was implemented. I can’t recreate the joke here, but his basketball slide (above) got a big laugh.</p><h4>Why is it important?</h4><p>Humour lifts the mood and reinforces important messages.</p><p>A presentation is like a contract. People have decided to sit and listen to you, so you’d better make it worth their while.</p><p>There’s lots of ways to do that, but smart use of humour just makes the whole experience more enjoyable. It makes people more engaged in what you’re saying. It can turn a good talk into a great talk.</p><p>You have to make sure that the humour is helping your presentation though. <a href="https://medium.com/u/cafd3fe907fa">Gabriel Valdivia</a> gave a great talk about VR, and just like in Pantelis’s talk there was a big laugh at one point, when he showed this gif:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*GOaJ0HbXm9yrVMS9.gif" /><figcaption><a href="https://i.imgur.com/YNOZ0pg.gif">https://i.imgur.com/YNOZ0pg.gif</a></figcaption></figure><p>I can’t remember what point he was making with it, and that’s the problem really. The gif was <em>too good</em>. I couldn’t take my eyes off it. I still can’t.</p><p>The best thing about Pantelis’s basketball joke was that it helped to reinforce the point he was making. That’s the gold standard.</p><p>But there doesn’t always have to be a point to reinforce. There are a few people at Lyst who regularly make us all laugh when they get up and talk about something. It just makes it a better place to work, and that’s really valuable too.</p><h3>Elevating the art of presenting</h3><p>My main feeling post-Jam is that presenting is really important: more important that we usually think.</p><p>It’s important for the audience, because they want to enjoy your talk and learn something.</p><p>It’s important for you, because being a great speaker gets you the opportunity to present at things like Jam.</p><p>And it’s important for your company, because better communication leads to better products.</p><p>So we need to value presentation skills and teach people to present with confidence, energy and humour.</p><p>Do you agree?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*CnNoY4oO0emJ-M19XndyXw.gif" /></figure><p>Communication in general is such an important part of working in a product team. If you share these values, <a href="https://www.lyst.com/careers/">Lyst is looking for product managers, designers and a user researcher</a>. Send us an application, or get in touch with me for a chat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=900049d93abb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/designing-lyst/3-presentation-lessons-from-jam-london-2017-900049d93abb">3 presentation lessons from Jam London 2017</a> was originally published in <a href="https://medium.com/designing-lyst">Designing Lyst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 mistakes in 1 survey question]]></title>
            <link>https://medium.com/designing-lyst/7-mistakes-in-1-survey-question-ab2726308000?source=rss-ce6ae7115614------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab2726308000</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[surveys]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Joseph Kay]]></dc:creator>
            <pubDate>Thu, 09 Nov 2017 14:10:05 GMT</pubDate>
            <atom:updated>2017-11-09T14:34:31.120Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve always found it useful to see examples of how not to do something. You learn a lot from your own mistakes, so it’s even better if you can learn from everyone else’s mistakes too.</p><p>A company sent me an email recently containing a one-question survey that they had run.</p><p>I won’t name the company that sent it. I’m pretty sure that they weren’t conducting serious research — it looks like this was just a bit of fun for them. Still, my apologies to them for picking on them like this, even if they are anonymous.</p><p>I’ve spotted seven things in their research design and their analysis that I think would be important mistakes in a real piece of research. I’m going to go through them one by one.</p><p>Here’s how they presented the question and the survey results in the email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*rg-DfVt709_PM61dlxYdjg.png" /></figure><p>Take a moment and see how many mistakes you can spot. I’m not talking about typos or anything like that.</p><p>Ready? Here’s what I found.</p><h3><strong>1. Wrong question format</strong></h3><p>Every survey question should have a purpose. They should be designed to help meet a research aim.</p><p>This is the question they asked:</p><blockquote>“Would you rather buy from a brand you know or buy the cheapest product?”</blockquote><p>I don’t know for sure what the research aim was, but I think it’s fair to assume that it’s about uncovering the relative importance of brand familiarity and price.</p><p>So this is really two questions masquerading as one. It’s asking how important brand familiarity is, and it’s also asking how important price is.</p><p>Why is this a problem? By trying to do too much, the question can’t accurately record information about either subject, let alone both of them.</p><p>If someone indicates that they would rather buy from a brand they know, does that mean that brand familiarity is really important? Maybe. Or maybe it means that they prefer not to buy the cheapest brand. There’s no way to know.</p><p>Shoehorning it into this “would you rather” format obscures other possible data too. What if someone doesn’t care about brand familiarity or price? They would have to just pick one option, and their choice would be indistinguishable from someone who cared deeply.</p><p>It would make much more sense to have two separate questions that measure how much people care about each one. However, for reasons that I’ll come to, I don’t think a survey is the right approach at all.</p><h3><strong>2. Uneven comparison</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6z8FtGHqgYkakTykMwUKsA.png" /></figure><p>This one is more subtle.</p><p>If they want to know relative importance of brand familiarity and price, the wording of the question should present the two options fairly and evenly. Otherwise, the results of the survey won’t represent the underlying issue well.</p><p>Why then are we comparing <em>any</em> known brand with <em>the single</em> cheapest product? That doesn’t seem even to me.</p><p>It would make more sense to compare the product from the most familiar band with the cheapest product, or perhaps any known-brand product with any cheap product.</p><h3><strong>3. High likelihood of bias</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7vcLYqAqWi4_IGX11q5jrA.png" /></figure><p>There’s a much bigger problem with the wording.</p><p>We are emotional creatures. We like to think of ourselves in a certain light.</p><p>When asked whether we prefer known brands or the cheapest product, there’s a good chance that we would feel more comfortable selecting the former, whether it’s true or not.</p><p>There’s a stigma associated with buying the cheapest option. Even in an anonymous survey, we can’t always escape the effect of that stigma.</p><p>Is there a way of wording the question to avoid this problem? I’m not sure. I suspect that this subject really isn’t something that you can get accurate information about from a survey.</p><p>What would work better than a survey? If you’re worried about people misrepresenting themselves, it may be better to observe what they do rather than ask them about it.</p><h3><strong>4. High likelihood of misreporting in general</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PuStQ4R_88BrVHjuK0Aoiw.png" /></figure><p>There’s another reason why a survey isn’t the right research method for this subject: people won’t know the answer to the question.</p><p>Do you know how you make decisions about brand and price? I personally would find it incredibly hard to tell someone which one has more influence on my decision-making.</p><p>People in general are very bad at explaining why they did something, or even what they did. How often do you go to the supermarket? How many times per week do you eat tomatoes?</p><p>Our memories are faulty and our powers of introspection are even worse. So when it comes to conducting research into these sorts of issues, it’s far better to observe what people do than it is to ask them what they do.</p><p>In this specific case, you might be able to run an experiment to determine the effects of brand familiarity and price. For example, you could expose shoppers to different brands at different price levels and record their purchase behaviour.</p><h3><strong>5. Overly broad scope</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQB13Xj1jEFMpe--bVq1ew.png" /></figure><blockquote>“Would you rather buy from a brand you know or buy the cheapest product?”</blockquote><p>What were you imagining buying when you first read that question?</p><p>A brand you know <em>of what</em>? The cheapest product <em>in what category</em>?</p><p>They asked the question in general terms, but people don’t make decisions in general: they make specific decisions in specific contexts.</p><p>For example, the roles of brand and price might be very different for the purchase of tinned beans compared with leather sofas or travel insurance.</p><p>Furthermore, people’s feelings about tinned beans might change when they’re hungry. Maybe they would be willing to spend more. Maybe they would be more easily enticed by a premium brand.</p><p>We don’t know how the results of this survey would apply to any specific case. Without that, the results are pretty much meaningless. Maybe everyone was imagining beans, maybe there was a wide variety. We can’t know.</p><p>Let’s go back to the experiment where we expose shoppers to different brands at different price levels. It would be important to study a specific product category. We would also want to keep a close eye on other variables, like income or hunger.</p><h3><strong>6. Misleading analysis</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIQyEng9MBUJ5iiTEIi6zw.png" /></figure><p>This one might sound picky, but it’s arguably the biggest problem of the lot.</p><p>The title on the chart is:</p><blockquote>“Brand always wins”</blockquote><p>The data they have collected says that 69% would rather buy from a known brand. How is that evidence that “brand always wins”? Maybe brand <em>usually </em>wins, but not always. It isn’t even close.</p><p>When you’re communicating research findings, it’s important that people come away with the key things that they should know. It’s good to sum things up in a concise way, and they managed to make it very concise, but they got the message wrong!</p><p>This undermines the whole point of doing research. Communication is key.</p><h3><strong>7. Incomplete data visualisation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVLrntR2-t5vuY58m_efRA.png" /></figure><p>There’s a lot of people who abhor all pie charts, but this one isn’t so bad. It’s just comparing two percentages and pie charts do that quite well.</p><p>However, this particular visualisation has a problem. They only have a sample of 100 people, and while this isn’t a terrible sample size, it isn’t so big that they can get away with having no error bars to represent the confidence interval.</p><p>If you aren’t familiar with confidence intervals, they indicate the range within which the true number would most likely fall if the research were conducted with all the relevant people in the world, rather than just 100 of them.</p><p>In this case, a 95% confidence interval would go from 59.9% to 78.1%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*Gn4PvO0mrT_7audnFdt7iA.png" /><figcaption>p is the proportion who chose “buy from a brand I know” and n is the total number of respondents (<a href="https://onlinecourses.science.psu.edu/stat100/node/56">source</a>)</figcaption></figure><p>That’s quite a big range and it’s worth showing in the visualisation. I’ve never seen error bars on a pie chart before, so perhaps it would be better to use a different kind of visualisation, like the stacked bar chart above (except with labels and things).</p><h3><strong>Bonus: Poor legibility</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9gL3fMLSPBh4Nv9qn4wVNw.png" /></figure><p>A small extra one for free!</p><p>Why did they write the labels in the same colours as the pie chart segments? That text written in peach is very difficult to read. This is an accessibility issue and also just an example of poor communication and design.</p><p>In the spirit of learning from our mistakes: what mistakes have I made? Have I missed anything? Is there anything I’ve got wrong? Please let me know in the comments.</p><p>Also, <a href="https://lyst.workable.com/j/24613D9F79">Lyst is looking for another user researcher to join the team</a>. At Lyst we’re always learning, and we love good research design and analysis. If that sounds good to you, click on that link and send us an application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab2726308000" width="1" height="1" alt=""><hr><p><a href="https://medium.com/designing-lyst/7-mistakes-in-1-survey-question-ab2726308000">7 mistakes in 1 survey question</a> was originally published in <a href="https://medium.com/designing-lyst">Designing Lyst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>