<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mitchel Seaman on Medium]]></title>
        <description><![CDATA[Stories by Mitchel Seaman on Medium]]></description>
        <link>https://medium.com/@mitchelseaman?source=rss-973c3794cf14------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yRNmJMyVG1KNu_HLOGcYvg.jpeg</url>
            <title>Stories by Mitchel Seaman on Medium</title>
            <link>https://medium.com/@mitchelseaman?source=rss-973c3794cf14------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:47:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mitchelseaman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My First Week at Grafana Labs]]></title>
            <link>https://medium.com/@mitchelseaman/my-first-week-at-grafana-labs-a62f1dde5fde?source=rss-973c3794cf14------2</link>
            <guid isPermaLink="false">https://medium.com/p/a62f1dde5fde</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[growth]]></category>
            <category><![CDATA[remote-working]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Mitchel Seaman]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 18:41:53 GMT</pubDate>
            <atom:updated>2020-07-04T00:50:08.318Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sK7swfRRdoVV0fevxU-iAA.jpeg" /></figure><p>My first day as the product manager for <a href="https://grafana.com/products/enterprise/">Grafana Enterprise</a> felt weird. I woke up at 5:30am, got ready, sat down with my personal computer at the same desk I’ve used since coming home for COVID, had one 7am meeting with my new manager Torkel (based in Stockholm), and just… started doing the job. No protracted HR onboarding, no desk assignment, no midday welcome lunch. Just a Google Doc with some ideas for what to do in my first few weeks. It was extremely calm and surprisingly productive. In my first two days I gained access to all the tools, signed up for all the benefits, learned about all my products, met my team, and began contributing to product discussions. I’m sort of mystified by the experience.</p><p>Grafana emphasizes its remote-first and dramatically-open culture, so I thought I’d share some observations from my first week.</p><h3>Highlights</h3><p><strong>I onboarded myself. </strong>There were warm virtual welcomes and good documentation, but I spent big swaths of my first few days quietly working on my own, setting up accounts, reading about the products, and watching recordings of recent company meetings. I also bought a regular MacBook at the regular Apple Store and set it up myself. The regular way. No hard drive images, no visit from IT.</p><p><strong>Some implementation details really make the global, remote-first thing run smoothly.</strong></p><ul><li>People are diligent about making their availability and working hours clear. There are Google Calendar Working Hours and out-of-office events, a shared company calendar with upcoming vacation linked to the HR system where holidays are requested, liberally-emoji’d Slack statuses, do not disturb and notification preferences, and out-of-office email messages. If someone doesn’t respond right away, I usually know why.</li><li>Meetings aren’t the primary way information gets passed around. Grafana uses a lot of documentation and invests in links between information, like a tool called <a href="https://www.getguru.com/">Guru</a> that holds all of the company’s HR and go-to-market knowledge. It seems to actually do what it’s supposed to do.</li><li>When meetings do happen (like monthly all-hands and weekly product updates), they’re always recorded and they always go into the same GDrive folder. I watched them at 1.75x. <em>I do wonder if there’s a more efficient way to manage meeting recordings — I want to see them linked from completed calendar events, annotated with notes, transcribed and connected to the decks and docs shared within. Product opp? </em>🤔</li><li>Benefits are consistent worldwide. Thirty days of vacation (no non-accrued financial skullduggery). A cash contribution to health insurance. Four percent 401(k) and pension matches when countries provide them, and a 4% pay increase when they don’t. Salaries that remain constant wherever I move. This feels fair and makes moving easy.</li><li>The travel policy is broad (present circumstances notwithstanding 😷). Teams travel to spend time together all the time. At 100 people, Grafana spent $100k per month on travel.</li></ul><p><strong>The leadership team is <em>open</em>.</strong></p><ul><li>I watched recordings of the last two company town halls, which included new customers, deal sizes, company revenue and cashflow, and <em>a screenshot of the company’s bank balance. </em>This all helps me trust the company’s financial position during COVID, and gives me confidence in the company equity I’m working for.</li><li>Everyone’s calendar is public. Like, I can see the CEO’s event details. The <em>Head of HR’s</em> event details, save for a couple of events made private. This default-to-total-openness seems to solve a lot more problems than it might create. And it’s easier to schedule meetings.</li><li>Everyone actually creates <a href="https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/">OKRs</a> each quarter, talks about them, updates progress weekly and posts them publicly using a tool called <a href="https://www.15five.com/">15Five</a>. Teams do the same.</li></ul><p><strong>The open source is strong with this one.</strong> I’ve worked at other companies with open-source products, but Grafana’s founding team are real advocates of OSS. You can tell by the number of pull requests they merge… personally. The business model predicates on an involved open-source community of users and contributors. Every interview and meet-and-greet I’ve had here has involved a discussion of how Grafana can grow as a business <em>and</em> stay true to its open-source software and community.</p><h3>Challenges</h3><p><strong>Boy, do I hate videoconferencing.</strong> I hate talking over people because of latency, seeing myself all the time as in a rearview mirror, missing out on pre-meeting chats, sitting in the same chair all day long, and having to pee reeeally badly by the end of every meeting. I suspect that everyone working from home right now feels similarly. But video chat is a reality of remote work, and I hope to adapt. Limited time zone overlap is nice, since there will always be a few meeting-less hours each day.</p><p><strong>Grafana is in growth mode.</strong></p><ul><li>The company is fast approaching <a href="https://en.wikipedia.org/wiki/Dunbar%27s_number#:~:text=Dunbar&#39;s%20number%20is%20a%20suggested,relates%20to%20every%20other%20person.">Dunbar’s Number</a> (~150 people). Half of the team has been around for less than a year. This year will be a test of the <em>systems</em> the leadership team are putting in place to communicate strategy, structure teams, and disseminate knowledge, because tribal knowledge is not enough. This seems to be front-of-mind for everyone, and certain people (including, I suppose, product managers) are arriving to help scale. Maybe the documentation-rich knowledge base, built because the team is all remote, will help too.</li><li>Speaking of product managers — <a href="https://medium.com/u/7573ef234b99">Richard Lam</a> and I are the first two PMs at a company with 60 engineers, a half-dozen designers, and established go-to-market and operations teams. This is intimidating, and we’ll need to reverse-engineer and retool in order to really pull the products’ priorities together and fit into teams.</li></ul><h3>Addendum</h3><p>It’s great here so far, and I hope to post more about Grafana’s products in the future. If you want to <a href="https://grafana.com/">check out Grafana</a>, it’s free and open-source. With all the growth of course the company’s hiring, and with all the remote-first emphasis people can work from pretty much anywhere. <a href="https://grafana.com/about/careers/#jobs">Job postings are here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a62f1dde5fde" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get More Personal Feedback]]></title>
            <link>https://medium.com/product-labs/get-more-personal-feedback-3fa7d7416388?source=rss-973c3794cf14------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fa7d7416388</guid>
            <category><![CDATA[feedback]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[balanced-team]]></category>
            <dc:creator><![CDATA[Mitchel Seaman]]></dc:creator>
            <pubDate>Thu, 16 Feb 2017 18:20:28 GMT</pubDate>
            <atom:updated>2017-05-01T19:30:58.217Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Speed-Dating Retro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_fSY8VdFe9QTIeb1nMQyg.jpeg" /></figure><p>One of my favorite teams at Pivotal is building an all-digital German energy company called <a href="https://www.getfresh.energy/">Fresh</a>. They’re masters of feedback. I walk over to their desks to find half the team standing up, talking over user stories in progress or sketching out ideas for their next feature. They’re constantly talking and they respect each other’s opinions. I think a lot of their openness comes down to the quality of their relationships — every member of the team knows every other member individually. One of the ways they created those connections so quickly was by giving one-on-one feedback periodically during retrospectives.</p><p>Once every couple of months, each person on the team sits down for a couple of minutes with <em>each other person</em> for a couple of minutes to share their personal observations, things they appreciate or have learned from the other person, and areas where they think the other could improve. This makes for a nice contrast to normal retros, where the entire team discusses common topics.</p><p>Great software and teams depend on quality feedback. Customer feedback drives product changes through research and analytics, and internal feedback drives team changes through standup and retrospectives. I’ve noticed, though, that repetitive activities like retrospectives can become dogmatic and lull teams into repetitive, single-track conversations. Maybe the team always talks about technical blockers or a tough stakeholder, and that becomes the theme of a series of retros. Maybe some people on the team overpower the conversation, and others hang back. Breaking from those well-worn topics takes a shake-up.</p><h3>How to Do It</h3><p>We’ve tried this with a couple of other teams at Pivotal and it’s worked out great. However, good feedback delivered well depends on courage from team members, as well as some forward planning and facilitation. Here are some pointers:</p><h4>Planning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bMWS4m3jeusesCNiyujKGg.jpeg" /></figure><ol><li>Talk to your team about the session. Check to see if everyone is interested in giving and receiving feedback, and if some people aren’t, talk to them about why.</li><li>Schedule a one-hour to 90-minute time slot 2 weeks or so in advance, to give everyone the chance to think about feedback for other people on the team. It’s a lot of work to think of both positive and constructive things to say about each person, so give some notice. Reserve somewhere with enough space for everyone to have a private conversation. I like using our kitchen area, which has long tables and chairs that face each other.</li><li>You might also book some time before the meeting for everyone to write down their feedback prior to the session. People often forget to think through or write down feedback unless it’s an explicit activity.</li><li>Find a third-party facilitator who can manage the session. This person can keep time and make sure things run smoothly.</li><li>Give the team a primer on Actionable, Specific, and Kind feedback. You could send an article for the team to read, or better yet hold a little white-boarding session and go over some examples of good and bad feedback by acting it out.</li><li>One week before and one day before the retro, send reminders to everyone that they should be thinking over feedback. I really liked how each person on the Fresh team wrote feedback on note cards, and gave each note card to their partner as we rotated through. It gave everyone a souvenir that they could refer back to.</li><li>Get some drinks and snacks for the session. People are not as kind when they’re hungry.</li></ol><h4>Day-of</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BC1BnMfjVoFFXvPFfCZ50w.jpeg" /></figure><ol><li>Figure out timing. Leave 5–10 minutes at the beginning to set up and 5 minutes at the end to debrief, and I’d recommend about 5 minutes per round for partners to give each other feedback and rotate. A team of up to 10 people can fit into a one-hour session. Remember that partners don’t have to try to fill all the time if they don’t have anything else to say. It’s often a hint that you need to collaborate more with this individual if you don’t have any feedback for them or vice versa.</li><li>Set up around a long table with each pair facing each other. Your rotation pattern varies depending on whether you have an even or odd number of team members.</li></ol><p>If you’re odd, then everyone can rotate around the table to the left or right, with one person taking a break each round:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liYhd6DxVnrAgl_go7ouZA.png" /></figure><p>If you’re even, then one person should stay stationary while everyone else rotates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frKkHW-d4_MDuXcSQ5jeQg.png" /></figure><ol><li>The facilitator should keep track of time (2 minutes per person per pair) and call when partners should switch or pairs should rotate. We used a little gong for a signal, which worked out better than shouting.</li><li>After giving feedback, pairs can hand each other their feedback cards.</li><li>Chat after the last round of feedback to see what people learned, what they thought of the session, and ways to improve it for next time. Plan the next one; I think it’s useful to run speed-dating retros every 6 weeks or so, but your cadence might be different.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDCcrnW03MZvYJVJEZAKTA.jpeg" /></figure><p>Giving good feedback is hard, but retros like this make space for people to open up to each other, become closer, and ultimately become much more collaborative and productive as a result. If you work it in periodically, say every 6 or 8 weeks, partners can discuss progress and help each other improve.</p><p>I’d love to hear about your favorite resources for giving and receiving feedback, or other things you’ve tried to make teams closer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsciWvH2J0D-b4NHOAYvuw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fa7d7416388" width="1" height="1" alt=""><hr><p><a href="https://medium.com/product-labs/get-more-personal-feedback-3fa7d7416388">Get More Personal Feedback</a> was originally published in <a href="https://medium.com/product-labs">Product Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Right Number of User Interviews]]></title>
            <link>https://medium.com/@mitchelseaman/the-right-number-of-user-interviews-de11c7815d9?source=rss-973c3794cf14------2</link>
            <guid isPermaLink="false">https://medium.com/p/de11c7815d9</guid>
            <category><![CDATA[user-research]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[ux]]></category>
            <dc:creator><![CDATA[Mitchel Seaman]]></dc:creator>
            <pubDate>Mon, 28 Sep 2015 10:43:59 GMT</pubDate>
            <atom:updated>2015-10-05T09:17:04.839Z</atom:updated>
            <content:encoded><![CDATA[<p>When planning out product research, I usually have to estimate the number of generative user interviews to conduct. Teams need to know how many users it will take to confidently make product decisions based on their feedback. Budgets, schedules, and the success of the product itself depend on a reasonable estimate.</p><p>I estimate a round of generative user interviews like this:</p><ul><li>Start with 5</li><li>Increase by up to 5 more depending on domain complexity</li><li>Multiply by the number of personas covered</li><li>Add one to three interviews to work out process kinks if this is new territory or a new team</li></ul><p>Justification below, but first some background:</p><h3>You’re after saturation.</h3><p>The key goal qualitative researchers shoot for is <strong>saturation</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*52q07RiDJX-P_5WZJh8ZFQ.jpeg" /></figure><blockquote>Theoretical saturation occurs when all of the main variations of the phenomenon have been identified and incorporated into the emerging theory. -<a href="http://api.ning.com/files/bzSRIccncYIHoa9GPd2I1ad0jUj6NIyGREx1RMTzJqjgYoBcPtmAyKny8kWTJW7MhnnIQkEyyo5j9FuEaO8Lr-6LlGYXEoZZ/FM.HowManyInt.pdf">Guest et al</a></blockquote><p>A qualitative researcher’s job is to learn all of the ideas, behaviors, stories, or experiences from a set of subjects within a domain. The researcher starts with an open-ended question, breaks it down into a series of smaller questions or topics, recruits, and begins interviews. Research hits a saturation point when incremental interviews don’t introduce new information. Since every new insight is important, you want to get reasonably close to a complete understanding of your domain.</p><p>So, interview until you don’t learn much from new interviews.</p><p>To understand whether you’ve arrived at saturation, compare successive synthesis sessions after interviews. How many new flashes of insight came up? How many new categories did you create to account for user sentiments? When the learnings taper, move on. This should be accompanied by a collective sense among the team that you’re learned enough.</p><h3><strong>There’s no single correct number</strong></h3><p>Expert recommendations are all over the map regarding the “right” number of interviews to conduct, across scientific and user research communities.</p><ul><li>Gaskin, Griffin, Hauser &amp; Co recommend <strong>10–30</strong> interviews that produce 75–150 statements from customers in <a href="http://www.mit.edu/~hauser/Papers/Gaskin_Griffin_Hauser_et_al%20VOC%20Encyclopedia%202011.pdf">Voice of the Customer</a>.</li><li>Daniel Bertaux said “<strong>15</strong> is the smallest acceptable sample” in a 1981 paper on sociological research.</li><li>Greg Guest and co concluded from an ethnographic study that they created 97% of their research codes within <strong>12</strong> interviews and 94% in the first <strong>6</strong>.</li><li><a href="http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/">Nielsen Norman</a> says that <strong>5</strong> <strong>usability</strong> test subjects are all you should target per round.*</li><li>The 560 qualitative research papers that Mark Mason surveyed in 2010 conducted anywhere from <strong>1–95 </strong>interviews, averaging <strong>31 </strong>with a large standard deviation.</li></ul><p>There’s no clear consensus on the topic because the researchers are conducting different kinds of interviews, in different domains, with different populations.</p><p>Studies claiming that very few user interviews are sufficient work within a tight, agreeable set of conditions, like usability testing an update to one mobile app scenario. This is like saying you can feel confident driving 120mph on the freeway when it’s sunny with good visibility, the road is straight, and your Porsche is in good repair. There’s a good reason why the speed limit is a lot lower than 120.</p><p>Usability testing a product among one homogenous group of users will start to feel repetitive after just a few interviews, and you should rightfully plan for a small number of them (5 or so). On the other hand, performing ethnographic research on a large domain and a diverse set of individuals requires a larger sample.</p><h3>How to (Roughly) Estimate Your Sample Size</h3><p>To roughly estimate the number of interviews you’ll need to conduct, consider a few factors:</p><h4>1. Diminishing Returns</h4><p>You’ll learn less from each successive interview, in a diminishing-returns shape that looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*QUUd7TtQovmNqSI1E_CV-Q.png" /></figure><p>When interviews start to feel repetitive, you’re not likely to gain much by doing more. Pull the rip cord and don’t conduct unnecessary interviews.</p><h4>2. The size of your domain</h4><p>The size of the topic you’re researching changes the <em>amount</em> of information you’ll need to gain before you’ve hit saturation. Usability testing for a single scenario or screen is a relatively tight domain, so 5 interviews might be just fine. Bigger, more nuanced, and less-understood domains bump up your expected sample size. Exploring a big issue like young peoples’ opinions about healthcare coverage, a broad emotional issue like postmarital sexuality, or a poorly-understood domain for your team like mobile device use in another country can drastically increase the number of interviews you’ll want to conduct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*9xz3GZVXtvJOlmvIigA06g.png" /></figure><h4>3. Your population’s heterogeneity</h4><p>The diversity of your population is a key indicator of how many interviews you should conduct. I like to think about this in terms of personas. Lauren Gilchrist’s article <a href="http://blog.pivotal.io/labs/labs/how-to-define-your-persona">here</a> is a good explanation of how we do that at Pivotal. Your personas are basically a set of self-consistent people who represent your research population or market. If you’re researching around one persona (say, “Bill,” a 45-year-old married tax accountant from Milwaukee who hates his morning commute), you don’t need to add interviews to your baseline. But each additional persona is a <em>multiplier</em> — if you want to understand two groups of people, interview the baseline number of <em>each</em> of them.</p><p>Take this example for a product that connects politicians to their constituents:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_v9HirhX71YObISU6fsrxA.png" /></figure><p>I have one persona for my politicians and one for my constituents — they have different characteristics, wants, needs, and problems, so my learning curve for each of them will be relatively independent of the other. I double the number of interviews in my estimate and recruit from both groups.</p><p>For personas who are similar or who interact similarly with your product, you might not quite need to multiply your sample. For example, if I’m making a shopping app that men and women use similarly, I don’t need to double my sample size just because I have male and female personas. However, I’ll want to make sure to represent those groups correctly in interviews to control for <a href="https://en.wikipedia.org/wiki/Selection_bias">selection bias</a>.</p><h4>4. The skill and experience of the interviewer and note-takers</h4><p>It takes time to warm into a round of user interviews. The script isn’t quite right, the interviewers are new or rusty, the stupid Google Hangout keeps glitching — whatever. Unless interview conditions are completely solid, I need an interview or two to warm up and make adjustments. Likewise, the team usually needs some time to sync on note-taking and synthesis methods. I treat this like a learning curve in front of the diminishing-returns curve for saturation. I tack on an extra interview or three when things are still new.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*oYXgqXV81pdyDDpK7mf6YQ.png" /></figure><h3>Summing it up</h3><p>Repeating the above, I estimate a <strong>round</strong> of user interviews like this:</p><ul><li>Start with 5</li><li>Increase by up to 5 more depending on domain complexity</li><li>Multiply by the number of personas covered</li><li>Add one to three interviews to work out process kinks if this is new territory or a new team</li></ul><p>This method has held up well on dozens of rounds of user interviews across several of my client projects at Pivotal Labs, but I always use it as a planning tool — <em>not</em> the determination of the absolute number of user interviews we conduct. This is to budget out time and money for interviews.</p><p>It goes without saying that there are a ton of other factors to consider when planning your actual interviews. Coming up with the right questions, controlling your researcher bias, pulling a reasonably representative sample, and learning to explore the extremes of your research topics are the difference between gathering the right information and the wrong information, no matter how completely. The primary goal of your research should always be to gain empathy for your customers — the better you can understand things from their perspective, the better you’ll be able to make something they find valuable.</p><p><em>* Ethnographic research, exploratory interviews, contextual inquiry, and usability testing are different types of qualitative research, but they all involve performing user interviews to a saturation point. I believe the type of research you’re conducting influences the size of your domain, so I account for research type using that lever.</em></p><h4>References</h4><p>Bertaux, Daniel (1981). From the life-history approach to the transformation of sociological practice. In Daniel Bertaux (Ed.),<em>Biography and society: The life history approach in the social sciences</em> (pp.29–45). London: Sage.</p><p>Gaskin, S. P., Griffin, A., Hauser, J. R., Katz, G. M. and Klein, R. L. 2010. Voice of the Customer. Wiley International Encyclopedia of Marketing. 5.</p><p>Guest, Greg; Bunce, Arwen &amp; Johnson, Laura (2006). “How many interviews are enough? An experiment with data saturation and variability”. <em>Field Methods</em>, <em>18</em>(1), 59–82.</p><p>Mason, Mark (2010). Sample Size and Saturation in PhD Studies Using Qualitative Interviews [63 paragraphs]. <em>Forum Qualitative Sozialforschung / Forum: Qualitative Social Research</em>, <em>11</em>(3), Art. 8, <a href="http://nbn-resolving.de/urn:nbn:de:0114-fqs100387.">http://nbn-resolving.de/urn:nbn:de:0114-fqs100387.</a></p><p>Morse, Janice, M. (2000). Determining sample size<em>. Qualitative Health Research</em>, <em>10</em>(1), 3–5.</p><p>Nielsen, Jakob. (2000). Why You Only Need to Test with 5 Users. <em>Nielsen Norman Group. </em><a href="http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/">http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/</a></p><p>Ritchie, Jane; Lewis, Jane &amp; Elam, Gillian (2003). Designing and selecting samples. In Jane Ritchie &amp; Jane Lewis (Eds.),<em>Qualitative research practice. A guide for social science students and researchers</em> (pp.77–108) Thousand Oaks, CA: Sage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de11c7815d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Mechanical Turk Accelerated our Product R&D]]></title>
            <link>https://medium.com/product-labs/how-mechanical-turk-accelerated-our-product-r-d-587feb67953?source=rss-973c3794cf14------2</link>
            <guid isPermaLink="false">https://medium.com/p/587feb67953</guid>
            <category><![CDATA[mechanical-turk]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[user-research]]></category>
            <dc:creator><![CDATA[Mitchel Seaman]]></dc:creator>
            <pubDate>Wed, 02 Sep 2015 13:05:21 GMT</pubDate>
            <atom:updated>2017-05-01T20:19:33.127Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*L46SsbuWTjPgW_JpH0-7tw.gif" /><figcaption>Photo credit <a href="http://gizmodo.com/my-brief-and-curious-life-as-a-mechanical-turk-1587864671">Gizmodo</a></figcaption></figure><p><a href="https://www.mturk.com/mturk/welcome">Mechanical Turk</a> combines human judgment with processing tasks. This is pretty useful, because it means that with a little effort and about fifty bucks, you can get thousands of humans to answer basic questions for you. My team and I used it to help with a research question for <a href="http://www.songmonkey.io/">songmonkey.io</a> that interviews and quantitative research couldn’t quite answer.</p><p><em>There are some Mechanical Turk tips at the bottom of this article if you’re looking for more of a how-to. But first, the story!</em></p><h4>Background</h4><p>We wanted to publish videos to teach people how to play popular songs on the piano. Research and interviews told us that a beginner musician’s go-to method for learning to play a song was to head to Google and type something like “how to play A<em>ll of Me</em> on piano.” Their number-one result? A ton of YouTube videos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*kfDLOUYUXAaS5RihKOckzQ.png" /></figure><p>Some of the videos attracted a lot more views and likes than others, so we set out to discover what separated the million-view videos from the 10-view videos.</p><h4>Setup</h4><p>First I performed a bunch of YouTube searches for instructional videos, sorted them by popularity and checked out things they had in common.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJrdVlpNIqewpWtE-Ihj8A.jpeg" /><figcaption>Synthesia in action</figcaption></figure><p>Some included video of an instructor playing piano, some used a tool called <a href="http://www.synthesiagame.com/">Synthesia</a><strong> </strong>to create a Guitar Hero effect, some displayed sheet music, some broke the songs up into different sections, and so on. After a few dozen videos, their attributes started to congeal.</p><p>Great — but what made a video popular? The manual-research results weren’t overwhelming — we needed a bigger sample of videos to figure out the relationship between a video’s content and its popularity.</p><p>With the help of the engineer on our team, we wrote a script to pull the top ~5,000 video results and some stats (date published, total views and likes, URL, etc.) for our search terms from the <a href="https://developers.google.com/youtube/v3/?hl=en">YouTube Data API</a> . Some interesting patterns emerged; for example, certain naming conventions and video length correlated with video popularity.</p><p>The API gave us some useful information, but in order to classify the actual content of the video we needed a human eye.</p><h4>Enter Mechanical Turk.</h4><p>MTurk was developed for just this type of situation — getting a lot of people to perform a bunch of small, straightforward, digital tasks. We asked them to categorize our YouTube videos.</p><p>I made a Mechanical Turk project that asked workers to categorize each video and identify some attributes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*KGsUA3hBLW8MWgNMKS-pxA.jpeg" /></figure><p>The workers categorized each video by its purpose, which allowed us to filter out unrelated videos. Workers also identified the elements of the video that we found in our initial qualitative research, so we could see which of those elements correlated most strongly with video popularity.</p><p>We were able to filter out a lot of videos that weren’t really about teaching songs on piano, which gave us a cleaner data set than just the YouTube search results on their own.</p><h4>Results</h4><p>Synthesia “falling bars” and a clear view of hands on the keyboard were found more often in popular, well-liked videos than were verbal lessons, song segmentation, or notes displayed onscreen. That was useful to know, because it encouraged us to focus on producing a certain type of video.</p><p>The team decided to make videos using the Synthesia tool and a teacher’s hands on the keyboard. They were relatively cheap to make and turned out to be really popular in usability tests.</p><p>MTurk and some data analysis were a great place to start making decisions on how to move forward. The larger video sample size gave us greater confidence that the trends we noticed were real, not just anecdotal. It’s a solid tool to consider next time you want to add a little en-masse human judgment to your research.</p><p><strong>Thanks to Pivotal Labs designers </strong><a href="https://medium.com/@aa_lawrence"><strong>Aaron Lawrence</strong></a><strong> and </strong><a href="https://twitter.com/technicka"><strong>Becky Gessler</strong></a><strong>, and the team at </strong><a href="http://www.songmonkey.io/"><strong>songmonkey.io</strong></a><strong>. Amy, Tanner and the team have crafted a pretty fantastic piano teaching tool. Grab a keyboard and check it out; you’ll be playing Taylor Swift in 5 minutes.</strong></p><figure><a href="http://www.songmonkey.io/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sjJXFoEzWOjExQzgN6O-9A.png" /></a></figure><h3>Mechanical Turk Tips:</h3><ul><li><strong>Make questions as straightforward as possible</strong>, breaking them down into multiple easier questions if necessary.</li><li>In order to verify responses, <strong>ask multiple MTurk workers to answer each of your questions</strong>. For example, I asked 3 Turks to categorize each of my videos and made sure their responses matched before adding that video to my results.</li><li>Mechanical Turk has some tools you can use to <strong>vet workers </strong>— you can create qualifications that include tests to make sure you’re asking the right people your questions, and you can enlist Mechanical Turk Masters who have demonstrated a certain type of proficiency in a given area. Use this if it makes sense. It didn’t for me, because my task was simple and I used redundancy instead (see previous tip).</li><li><strong>Work in small, increasing-size batches of tasks</strong>. Try starting with a one-item batch, then 10, then 100, then 200 and so on. I made a lot of mistakes setting up my tasks and could have saved time and money by being a little more patient and ramping up.</li><li><strong>Do your truly qualitative research separately</strong> in user interviews — MTurk workers make more money by performing tasks quickly, rather than answering questions thoughtfully, so don’t expect essays or advanced insight. That’s not really MTurk’s sweet spot.</li><li>Like surveys, Mechanical Turk can give you a lot of data that <em>looks</em> authoritative because it’s in a spreadsheet. <a href="https://explorable.com/research-bias"><strong>Be on the lookout for research biases</strong></a> like leading questions, selection bias, and confusion between correlation and causation in your research. MTurk is best used alongside other experiments. My experiment, for example, included some sampling bias because I performed the YouTube searches to feed the video data set, rather than including every YouTube video.</li><li><strong>Control for external variables </strong>— the most obvious external variable that gave us bogus results at first was time — we judged video by their absolute number of views, rather than their <em>rate</em> of views (per month, year, etc.). Older videos have more views than newer videos, even though their quality isn’t necessarily as good.</li></ul><p><em>I’d love to hear tips and questions if you have them. We also kept the YouTube scripts if you’re interested in specifics. Hit me up in the comments or on Twitter: </em><a href="https://twitter.com/mitchelseaman"><em>@mitchelseaman</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=587feb67953" width="1" height="1" alt=""><hr><p><a href="https://medium.com/product-labs/how-mechanical-turk-accelerated-our-product-r-d-587feb67953">How Mechanical Turk Accelerated our Product R&amp;D</a> was originally published in <a href="https://medium.com/product-labs">Product Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trello for Retrospectives]]></title>
            <link>https://medium.com/built-to-adapt/trello-for-retrospectives-bd9dcf56d942?source=rss-973c3794cf14------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd9dcf56d942</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[retrospectives]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[trello]]></category>
            <dc:creator><![CDATA[Mitchel Seaman]]></dc:creator>
            <pubDate>Tue, 25 Aug 2015 15:10:40 GMT</pubDate>
            <atom:updated>2017-11-26T04:14:31.064Z</atom:updated>
            <content:encoded><![CDATA[<p>David Ghent recently published a great post on <a href="https://medium.com/product-labs/using-trello-for-user-research-synthesis-fb1abdfc7c4b">using Trello for research synthesis</a> and I thought I’d follow up with another Trello application on Agile projects — retrospective.</p><p>I just finished a remote project with 8 team members who needed a good alternative to a whiteboard for retro. It’s pretty hard to beat a whiteboard for simplicity and ease of use…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rLE5bAkck8MR6ImtZeWYHg.jpeg" /><figcaption>retro on a whiteboard</figcaption></figure><p>…but with people coming in from 3 cities and 2 time zones, we had to find another way to make everything visible to everyone all the time. So we went to Trello, and it actually worked out pretty great:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-9LTHpY1sY84Oa2waWm6w.png" /><figcaption>super retro board</figcaption></figure><h3>Here’s how to do it:</h3><h4>One-time setup:</h4><ol><li>Set up a <a href="https://trello.com/">Trello</a> board called Retro (or String Cheese, or whatever). Put it within your team’s Trello organization if you have one.</li><li>Set up your lists: Action Items, Done, [DATE] :), [DATE] :/, [DATE] :(</li><li>Share the board with everyone on your team and give them edit permissions <em>before</em> retro (it’ll eat into your time if you do it during)</li><li>Get your snacks and drinks ready, and have everyone bring a laptop or tablet or phone — anything that can run Trello.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBEJKgNz5-o_ov8l2GAxDg.png" /><figcaption>Starter board</figcaption></figure><h4>At retro, here’s the process:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*-kaCwjvJNagZuq14Fwg1kA.png" /></figure><ol><li>Everyone adds their cards to the appropriate list just like it was a whiteboard</li><li>Folks vote for items using one of two methods. They can use the Voting Power-up (enable it using Menu &gt; Power Ups &gt; Voting, and then people vote by selecting a card and clicking the Vote button on the right-hand side). I’m not a huge fan of Voting because you have to click into each card and you can only vote once. What’s better than voting? <em>stickers! </em>Everyone gets a few votes and puts stickers on the cards they’d like to discuss. Trust me, it’s way more fun.</li><li>The facilitator drags the top-voted items to the top of the list</li><li>The group discusses them one by one</li><li>The facilitator triages them. Give a color label to a card that has been discussed by clicking on it and clicking the Label button, or hovering over the card and pressing a number key between 1 and 6. Move cards that are actionable to Action Items.</li><li>The facilitator adds action items.</li><li>The group celebrates! Another week over!</li></ol><p>Once the retro has finished, the facilitator archives the lists and creates new ones for next week. The dates keep all the archived lists in order and it’s easy to retrieve old lists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*po3UNYY_Otp1cpChfpWWLQ.png" /><figcaption>archive old lists</figcaption></figure><p>The next week, before starting items the group goes over action items from the week before. Items move from the Action Items column to the Done column. Everyone cheers.</p><h3>What’s great about it:</h3><ol><li>Retro items are saved for future reference</li><li>Action items stay up, can be assigned to people and given due dates (Trello will even send reminders by email). Retro items can be dragged straight into the Action Items column.</li></ol><h3>What’s not so great:</h3><ol><li>It’s still not in-person. For collocated teams, I still recommend the whiteboard.</li><li>Action items aren’t physically radiated in the team’s space, so they can be harder to remember. Using due dates and assigning items to people within Trello helped keep up reminders and accountability.</li><li>People have a tendency to drift off when their computers are in front of them. Treat this by having everyone get up once in a while during discussion, and keep laptops closed when not in use.</li></ol><h3>Pro Tips</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*SpvzMIpaAZR8ptoRqZ-9QQ.png" /><figcaption>custom stickers!</figcaption></figure><ul><li>Trello Gold gives you access to a lot more stickers and you can make custom ones — it’s more fun.</li><li>If you’re the one to invite your whole team to Trello, you get free Trello Gold, you mastermind.</li><li>Make it fun! Change the background, make crazy stickers, make jokes, use weird emoticons… One of the biggest things lacking from remote teams is joking around — Trello’s a pretty playful tool and we had a really good time messing with it.</li></ul><figure><a href="http://builttoadapt.io/"><img alt="" src="https://cdn-images-1.medium.com/max/71/1*9fg8oHTkroGjylRUKyKQAw.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd9dcf56d942" width="1" height="1" alt=""><hr><p><a href="https://medium.com/built-to-adapt/trello-for-retrospectives-bd9dcf56d942">Trello for Retrospectives</a> was originally published in <a href="https://medium.com/built-to-adapt">Built to Adapt</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>