<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Kraemer on Medium]]></title>
        <description><![CDATA[Stories by Jeff Kraemer on Medium]]></description>
        <link>https://medium.com/@jeffkraemer?source=rss-c2eb03213a21------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*NZPxqTt3aQb__y-Y.jpg</url>
            <title>Stories by Jeff Kraemer on Medium</title>
            <link>https://medium.com/@jeffkraemer?source=rss-c2eb03213a21------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:44:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeffkraemer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How do you know when a design fails a usability test?]]></title>
            <link>https://medium.com/building-freshbooks/how-do-you-know-when-a-design-fails-a-usability-test-3afbf0b8a4c4?source=rss-c2eb03213a21------2</link>
            <guid isPermaLink="false">https://medium.com/p/3afbf0b8a4c4</guid>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[interaction-design]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[web-design]]></category>
            <category><![CDATA[usability]]></category>
            <dc:creator><![CDATA[Jeff Kraemer]]></dc:creator>
            <pubDate>Tue, 14 Nov 2017 20:49:59 GMT</pubDate>
            <atom:updated>2017-11-14T20:49:59.300Z</atom:updated>
            <content:encoded><![CDATA[<h4>A handy technique I learned from the wrong job…</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYHq2dqUCYCyKrYd2KfGHA.png" /><figcaption>Task failure.</figcaption></figure><p><em>Originally published at </em><a href="https://www.webdesignerdepot.com/2017/03/how-do-you-know-when-a-design-fails-a-usability-test/"><em>WebdesignerDepot.com</em></a><em>.</em></p><p>Years ago, I spent an awkward patch of my career as an instructional designer, creating courses for online learning. It was a bad fit and I moved on happily, but one part of that job has made me a better UX designer: <strong>learning objectives.</strong></p><p>Learning objectives are simply what you want the student to learn by the end of the training. If there’s a test, the test questions should be based on those objectives — otherwise, what’s the point of the test?</p><p>The same approach comes in handy for figuring out whether a design has passed or failed a usability test. Just remember: it’s the design that’s being tested, not the participants.</p><p>What does the test participant need to do or say for you to feel confident that the design has succeeded? Do they need to track three hours of time for a particular project? Generate an invoice to a client based on that tracked time? Send the invoice? That’s your test criteria (which my <a href="https://www.freshbooks.com/invoice">FreshBooks</a> colleague Melina Stathopoulos named “success criteria”).</p><p>Of course usability testing is about observing how users complete tasks, but what will you get them to do, exactly? The beauty of these criteria is that they steer you away from vague testing goals like, “understand how time tracking works.” How will you know they’ve understood it? You get them to describe it. And once they’ve described it accurately, you can say that aspect of the design was successful.</p><p><strong>Success criteria help you twice over</strong>: they clarify whether your design is really successful, and they make it easier to share those results.</p><h3>Verbs are magical</h3><p>The book that taught me about learning objectives, George Piskurich’s <a href="https://books.google.ca/books/about/Rapid_Instructional_Design.html?id=DcmuNAEACAAJ&amp;redir_esc=y&amp;hl=en">Rapid Instructional Design</a>, offers a handy list of behaviours to start your success criteria.</p><p>For example, the objectives for comprehension might be “describe” or “demonstrate”. Again, “understand” is no good — you need them to <em>say</em> (that is, describe) or <em>do</em> (that is, demonstrate) something that proves to you that they’ve understood.</p><p>And then, at a higher degree of difficulty, a participant might “explain” or “organize”; at a higher level still, they might “create” or “evaluate”.</p><p>Whatever verb you choose to start your success criteria, <strong>the point is that you can observe whether or not a user has actually said or done whatever constitutes task success.</strong></p><h3>“By the end of this session…”</h3><p>So, when you’re planning your next usability test, and you’re working on tasks, start by asking, “What should a user be able to do with (or say about) this design?”</p><p>Then, you might write something like this:</p><blockquote>By the end of the session, the participant should be able to:</blockquote><blockquote>- Track three hours of time for a particular project</blockquote><blockquote>- Generate an invoice to a client based on that tracked time</blockquote><blockquote>- Describe the difference between tracking time and logging time</blockquote><p>Now you have three success criteria and, based on those, you’ve also got a pretty clear sense of what tasks you’ll need to give the participants.</p><p>One caveat: success criteria aren’t quite the same as tasks. Tasks have more context; they’re written to be read to the participant, and might include some context about the task, particularly if you’re steering them to find something in your prototype. For example:</p><blockquote><strong>Success criteria:</strong> Generate an invoice to a client based on that tracked time<br><strong>Task:</strong> “Now that you’ve tracked three hours on the Atlas project, show me how you would invoice Acme Products for your time.”</blockquote><p>Pretty similar, obviously, but <strong>success criteria are for you and your team; the task is for the participant</strong> in the context of the usability session.</p><p>And you’ll notice that one of the success criteria above is about describing something, rather than completing a task. It might be a follow-up question to a task. These are handy for validating whether your design’s mental model is clear to users. I’ve seen users find their way through a task, but then describe to me a mental model of the app which is at odds with how it was designed. That’s task success for one participant, but more importantly there’s an underlying problem with matching that participant’s mental model.</p><p>So, start with your success criteria, then write your tasks and follow-up questions based on your criteria.</p><h3>Stakeholders love success criteria</h3><p>Stakeholders don’t necessarily care about your process, but they really care about the results. And if your presentation of the results is vague, they will be rightfully irritated.</p><p>“The user managed to track a few hours, but we weren’t sure whether she understood that tracking time isn’t the same as logging it against a client…” Well, <strong>why aren’t you sure? Isn’t it your job to figure this out?</strong> You’re wasting their time, and not giving them clear direction on how to fix the UX problems — which is also your job, right?</p><blockquote>Success criteria help you twice over: they clarify whether your design is really successful, and they make it easier to share those results.</blockquote><p>We’ve had some success tracking success criteria in a simple table, and colour-coding the results. Like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ke5mAAk0s6zA2FOSZ5vFNQ.png" /><figcaption>We whip up a colour-coded table of results (green = success, red = failure) on our wiki. In the top row, we list participants; in the left column, we list our success criteria. It’s ugly, but quick and useful.</figcaption></figure><p>This is easy to scan, shows pretty clearly where the problems are, and grounds the results in the experiences of actual participants. We also list a bullet-point summary of results and a list of usability problems and recommendations just beneath it. At FreshBooks, we’ll zero in on those problems and iterate until we believe they’re solved. Your process might be a little different — maybe you’re a consultant handing over a report to a client, for example — but the benefits are the same.</p><p>We love to hear how our peers are practicing design. Do you use something like success criteria? If you try it, is it useful to you? Are you using some other method to track the usability of your designs? Let us know in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3afbf0b8a4c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/building-freshbooks/how-do-you-know-when-a-design-fails-a-usability-test-3afbf0b8a4c4">How do you know when a design fails a usability test?</a> was originally published in <a href="https://medium.com/building-freshbooks">Building FreshBooks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bite-Sized UX: The Handiness of Assumptions and Hypotheses]]></title>
            <link>https://medium.com/building-freshbooks/bite-sized-ux-the-handiness-of-assumptions-and-hypotheses-b1880d7cad7c?source=rss-c2eb03213a21------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1880d7cad7c</guid>
            <category><![CDATA[design]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Jeff Kraemer]]></dc:creator>
            <pubDate>Thu, 10 Aug 2017 18:38:59 GMT</pubDate>
            <atom:updated>2017-08-10T18:38:59.808Z</atom:updated>
            <content:encoded><![CDATA[<p>At FreshBooks, our product team talks a lot about assumptions and hypotheses. What’s the difference, and why are they useful?</p><h4>Assumptions: “We assume…”</h4><p>An assumption is basically an informed guess… which might be biased, and which is definitively unproven. Assumptions lurk everywhere, and they’re often wrong. What’s really useful, then, is calling out assumptions so that we can turn them into falsifiable hypotheses.</p><p>In our design process, assumptions often come up when we’re brainstorming or critiquing a design. “A user would probably click on that button if it was a different colour,” one of us will say, and another will call out, “That’s a testable assumption.” Excellent! We have a test to run, which means we need a hypothesis.</p><h4>Hypotheses: “We believe…”</h4><p>A hypothesis is a bit more formal. It’s a way of stating an assumption in a testable form. We also call a hypothesis a “we believe” statement since we start our hypotheses with those words. For example, “We believe that making all our buttons purple will double conversion.” We can test that and learn whether our belief was right. By testing, we can spin our assumption straw into insight gold.</p><p>A key part of a sound hypothesis is that it’s falsifiable. For example, “We believe that users will love our purple buttons” isn’t falsifiable; it’s subjective. And (this is going to sound grouchy, but:) why do we care if they love them? Aren’t we trying to achieve a result? Our belief that a purple button will double conversion has a condition and result we can test — if the button’s purple, the conversion rate will double, and that’s quantifiably either true or false.</p><p>A hypothesis forms the backbone of our design sprints; the purpose of the sprint is to validate it. That’s why we usually end our brainstorming session at the start of a design sprint by agreeing on a hypothesis. The designer and product owner then test it within the week.</p><p>In short, when you identify an assumption, rewriting it as a falsifiable hypothesis will help you get data you can use. And what if your hypothesis fails? That’s great, because you’ll have learned something!</p><p><em>Thanks to Danielle Klein, Kiley Meehan, and Aaron Wright for feedback on this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1880d7cad7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/building-freshbooks/bite-sized-ux-the-handiness-of-assumptions-and-hypotheses-b1880d7cad7c">Bite-Sized UX: The Handiness of Assumptions and Hypotheses</a> was originally published in <a href="https://medium.com/building-freshbooks">Building FreshBooks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How FreshBooks Prepares Usability Tests in Only a Day]]></title>
            <link>https://medium.com/building-freshbooks/how-freshbooks-prepares-usability-tests-in-only-a-day-3f4a8367f356?source=rss-c2eb03213a21------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f4a8367f356</guid>
            <category><![CDATA[design]]></category>
            <category><![CDATA[prototyping]]></category>
            <category><![CDATA[ux]]></category>
            <dc:creator><![CDATA[Jeff Kraemer]]></dc:creator>
            <pubDate>Wed, 13 Apr 2016 19:55:42 GMT</pubDate>
            <atom:updated>2016-04-14T20:30:01.104Z</atom:updated>
            <content:encoded><![CDATA[<h4>Prototyping and preparing for usability tests doesn’t have to be a drawn-out, painstaking process.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mIuEeM3GNy2vxTZZJ5VPag.jpeg" /><figcaption>We don’t actually use calendars like this. Haha, so analog! (Photo by Eric Rothermel)</figcaption></figure><p>If you follow a Lean UX process you already know that you need regular user feedback. At FreshBooks, we put our work in front of our users every Thursday and use prototypes as the basis for getting feedback. Once you’ve got your rhythm, you can do it in a day.</p><p>We’ve learned that prototyping and preparing for usability tests doesn’t have to be a drawn-out, painstaking process. Unfortunately, I’ve seen plenty of organizations where it’s just that.</p><h3>How we structure our Lean UX sprint</h3><p>How do we know what to test? Our week-long design sprint is designed to test a hypothesis:</p><ul><li><a href="https://medium.com/freshbooks-product-development/the-power-of-the-charrette-9f57d410f2e5#.39ofd9jgu">Monday: Charrette</a> to ideate a hypothesis to a design problem</li><li><a href="https://medium.com/freshbooks-product-development/why-design-circles-are-so-frickin-productive-5f5cc5441d0d#.37f7968b8">Tuesday: Gather in Design Circle</a> to evaluate our hypothesis</li><li>Wednesday: Build prototypes, to test our hypothesis with real users</li><li><a href="https://medium.com/freshbooks-product-development/don-t-light-a-fire-in-your-tent-user-testing-without-burning-the-place-down-7688d62812d6#.8hl7opii6">Thursday: Test our prototypes</a> with users to validate our hypothesis</li><li>Friday: Gather again in Design Circle to debrief on test results</li></ul><p>So our Thursday tests aren’t just a “let’s show stuff to users and get feedback” session (although those are good too). Each usability session tests our hypothesis; the idea that our sprint is pinned to.</p><p>Since we test on Thursdays, our prototype and test plan need to be prepared each Wednesday. We know we’ll return to our circle at the end of the week to say whether we disproved the hypothesis. So we can’t half-ass it; every week, we’re held accountable by our team.</p><p>That means a good prototype and test plan are crucial.</p><h3>How we spend those 8 hours on Wednesdays…</h3><p>… varies a little from team to team, and week to week. We don’t follow this process to the letter every week, but it should give you an idea of how FreshBooks UX designers keep our prototyping and testing lean.</p><h4>10 a.m.: Get final circle feedback into mockups</h4><p>We’ve already sorted out our hypothesis for the week — for example, “we believe that introducing friction into the signup process will increase conversion” (just go with me on this). This is usually squared away on Mondays.</p><p>On Tuesday the designer is exploring design options to test that hypothesis. At the end of the day, we show those options in the Design Circle. We end Tuesday afternoons with a design circle, where the team finalizes what we’re testing.</p><p>So Wednesday mornings are spent incorporating feedback from that circle into our mockups.</p><h4>11 a.m.: Write your test plan for Thursday</h4><p>Once we’ve polished off the tweaks to the mockups (we hope they’re tweaks!) we plan our test. Here’s what goes into a good test plan:</p><ul><li><strong>Goals: </strong>How do we know we’re testing our hypothesis? What trouble spots do we anticipate? You might phrase your test goals as questions you need answers to, like “Can the user sign up for FreshBooks?”</li><li><strong>Tasks:</strong> How will we formulate a task that will meet our goals, but not lead test participants? For example, “Sign up for FreshBooks”. We also draft the follow-up questions to each task based on the expected outcomes, like “Why did you drop out after step 4?”</li></ul><p>Because we use the same template every week, including the same script to start tests and a few key questions we ask everyone, like “tell us about your business,” we don’t need to over-document or reinvent the wheel. (Here’s a <a href="https://www.sensible.com/downloads/test-script.pdf">good template from Steve Krug</a> if you’re starting from scratch.)</p><p>It’s a good idea to validate your test plan with your PM. Hey, it’s a good idea to write it together, if you can.</p><h4>12 p.m.: Have lunch</h4><p>We like lunch. Try <a href="http://tuckshopkitchen.ca/">Tuck Shop</a>!</p><h4>1 p.m.: Plan and building the prototype</h4><p>We almost always use <a href="https://medium.com/@sketchapp">Sketch</a> or Illustrator and <a href="https://medium.com/@InVisionApp">Invision</a>. They’re terrific, quick tools; they allow us to test both desktop and mobile apps, and they fit with our practice of iterating at a fairly high fidelity. (Each designer can use their preferred tool, as long as it gets the design across). Once our screens are designed, dropping them in Invision and building a prototype by linking hotspots is quick and easy.</p><p>It’s important to note that we jump from sketch to Sketch. We skip lo-fi prototyping and use the sketches from charrette to design directly in high-fidelity tools. That’s not always a great practice, but it works for us because we have a strong pattern library. So, we’re never really starting from scratch. In fact, we’re usually iterating on an existing pattern.</p><p>On a typical Wednesday, we’re pulling those sweet mockups into Invision and building a prototype that maps to the test we planned in the morning.</p><h4>4:30 p.m.: Check in with participants</h4><p>Since we test on Thursdays, we remind each participant with a quick phone call on Wednesday afternoon. We book our participants well in advance, by emailing a segment of FreshBooks users and asking them to sign up to our <a href="https://youcanbook.me/">You Can Book Me</a> form. YCBM sends automated reminder emails, but frankly, the call is more personal (and therefore more FreshBooksy) and more effective.</p><h4>5:00 p.m.: Test the test</h4><p>What could go wrong? So much!</p><p>We like to test our tests by conducting dry runs. It’s never a bad idea to do these with a PM or fellow designer if possible. Ideally, we test our test with whomever is going to observe our session the next day.</p><p>Often FreshBooks designers will conduct tests solo, but we encourage designers to pull in observers. A PM is great, but getting any scrum team member to participate is a win, plus it’s handy to have them walk through the test with us. Together, we’ll find blind spots and technical issues we might’ve missed on our own.</p><p>We try to cut any questions from our test script that aren’t <em>directly</em> related to our test goals, i.e. based on the hypothesis we want to validate. We often spot pointless questions when we’re testing our test. Cutting them feels great.</p><h4>6:00 p.m.: Give High Fives</h4><p>And home we go.</p><h3>What have we learned?</h3><p>Every designer should be able to build their own prototypes, and to test their own work. Building and testing a prototype can easily be a part of your weekly process.</p><p>We’ve made it part of our weekly process for almost two years now, and it’s enabled us to create a <a href="https://www.freshbooks.com/new-freshbooks">completely new platform for FreshBooks</a> (we’ll have much more to say about that later). It also helped us generate more insights in a shorter amount of time, with much greater confidence.</p><p>Have you tried lean prototyping and testing? What’s worked for you?</p><h3>And another thing:</h3><p>FreshBooks is looking for participants for<a href="http://labs.freshbooks.com/"> Collaborative Design Labs</a>. And if you like this article, please pass it on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*ZtfGhIS10PcWIgP5w_6sLQ.png" /><figcaption>Jeff Kraemer is a Principal UX Designer at FreshBooks.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f4a8367f356" width="1" height="1" alt=""><hr><p><a href="https://medium.com/building-freshbooks/how-freshbooks-prepares-usability-tests-in-only-a-day-3f4a8367f356">How FreshBooks Prepares Usability Tests in Only a Day</a> was originally published in <a href="https://medium.com/building-freshbooks">Building FreshBooks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>