The Challenge: The Marketer-Designed IA

Years ago, a marketing team at a well-known enterprise financial company was working with their IT group to create a content microsite. The site would house informational and brand-focused content, which had previously been scattered across various websites, newsletters, and blogs.

The team had 125 content pieces ready to go and plans to create thousands more. The site would have its own navigation, and the team needed a way to organize the content that made sense to users.

They contacted us to lead information architecture research with users.

The problem: the team already had strong opinions about how the IA should be structured and labeled, none of which were informed by user research. On our initial call, they told us:

We just want some research to help us pick a winner among our 3 existing ideas, maybe with some tweaks. Ideally we want this done within a week or two.

We responded with a simple question:

What if the best IA for customers looks nothing like the 3 versions that you’ve come up with on your own?

After a long silence, we continued:

If we limit ourselves to a quick round of IA testing, we won’t have an opportunity to come up with something truly human-centered if the existing versions perform poorly.

Our hunch, we told them, was that the existing IA concepts would cause many findability problems for users. We nicely said:

You are marketers that live in this content every day. A structure that makes sense to you will likely trip up the average consumer who only needs this type of content occassionally.

There was a more obvious problem with the existing drafts.

Your category names sound cute and clever; for example, one version uses “Live | Laugh | Learn”. While this might work well in a marketing campaign, when it comes to web navigation it’s likely to cause confusion. Your users are trying to quickly find information or complete a task. To support them, your best bet is to use plain language that matches the words they use.

How do you discover the language of your customers? How do you understand the way they approach this content? That brings us to …

The Solution: Card Sort Study

After some back and forth, the project team agreed to a 1-month research and design project that started with a card sort study and later tested the IAs through multiple rounds of tree testing, with design iterations between each round.

In this case study, we cover the most challenging phase of the project: designing and running the card sort study.

In a card sort, we present users with a collection of cards that represent content or functionality; we ask users to group the cards in ways that make sense to them and then label the groups. After watching enough users go through the process, thinking aloud as they go, we’re in a great position to generate intuitive IAs.

Running a card sort is difficult. But it’s the best starting point for creating a usable IA. Here are the steps we went through for this project, along with some of the challenges that we overcame.

Our 12-Step Card Sort Process

We started this project with a quick discovery phase that included a kickoff meeting and a review of relevant resources. As with most of our UX research studies, our primary goals in this phase were to:

• Align with project stakeholders on project goals, research questions, and measures of success
• Develop an understanding of the target audiences and of the core user goals, scenarios, and tasks.

With this critical foundation in place, we were ready to move onto the unique steps of this research method.

1. Review content inventory.

The marketing team sent us a content inventory spreadsheet with existing content and some future content. The spreadsheet included article titles, sub-titles, and links to the actual articles that existed. We spent a good chunk of time reading the articles.

2. Select representative content.

Out of 125 potential content pieces, we picked 100 for the card sort. Often we aim for 30 to 50 cards; in this case, the content was relatively easy to understand and group, so we went higher. The 25 we removed were already well represented in the 100 articles we had.

We made sure that all of the cards were at the same bottom “level”; i.e. we didn’t mix article pages with category pages.

3. Get feedback on content and revise.

We shared the proposed list of articles with the project team and asked if it was a representative sample of future content. We made some revisions based on their feedback.

4. Write card labels.

While it would have been nice to reuse the article titles as the card labels, we ended up rewriting nearly all 100 of them to 1) be easy for participants to understand, 2) accurately represent content, and 3) spell out any acronyms or jargon. For most articles, we used the sub-title as the starting point since those were more descriptive than the titles.

5. Fix biasing labels (don’t skip this!).

It’s very easy to bias participants in a card sort study and end up with categories that are less about how users approach the concepts and more about their keyword- and pattern-matching abilities. Since unchecked card labels can result in poor IAs, removing label bias is one of the most important jobs in designing a card sort study.

It’s very easy to bias participants in a card sort study with common keywords or patterns in card labels.

We reviewed the list of cards and spotted common words used across labels. We then found synonyms or alternate phrasings for those words. For example, “quiz” appeared on 5 of the card labels. If we’d left them as is, many users would have grouped “quiz” cards together even if the underlying content didn’t go together in users’ minds. So we replaced one “quiz” with “exam”, turned another label into a “what do you know about …?” question format, and so on.

We also looked for common structures in the cards. For example, cards containing “ideas for homeowners” and “tips for parents” followed this pattern:

{Resource} for {Audience}

Left as is, many users would have grouped those cards together purely based on the structural similarities, regardless of the content similarities. So we mixed up the word order and structures of those cards, e.g. “how parents can …”.

These types of changes may feel counterintuitive to you as a researcher, designer, or writer. After all, your job is typically to make tasks easier for users. But as Jakob Nielsen points out:

Card sorting isn’t a user interface design; it’s a knowledge elicitation exercise to discover users’ mental models. [Your study] has to make users work harder and really think about how they’d approach the concepts written on the cards.”

6. Get feedback on card labels and revise.

Just as we’d done with the content list, we shared the card labels with the project team — along with explanations and instructions — and captured their feedback. We also shared the list of cards with a couple of people unfamiliar with the project and asked them to tell us what each card meant. This helped us to identify and fix card labels that were vague or misleading.

7. Run a pilot study and revise (don’t skip this!).

We ran a pilot card sort with 2 participants. Pilots are always a good idea in user research, and they’re especially important when your study includes a quantitative component that becomes expensive to redo if you spot study design problems later on.

Based on what we saw in our pilot sessions, we reworded more labels to further reduce common words and needless words. We also removed 25 more cards that now seemed unnecessary and made the sort easier to do in our target time of 15 to 20 minutes.

After numerous iterations, we now had 75 cards that we felt good about in terms of representativeness, descriptiveness, clarity, and lack of bias. We were ready for the real study.

8. Recruit participants.

Recruitment was relatively easy for this study because the microsite’s target market was consumers of a widely-used line of products. We lined up 55 participants that we sourced from a research panel and screened with a few questions.

9. Run qualitative sessions.

We ran 5 users through a remote think-aloud card sort, one at a time. Hearing their thoughts allowed us to understand why people group certain topics together and, more generally, it helped us to see the content from a user’s point of view.

10. Run quantitative sessions.

We ran 50 users through a remote unmoderated card sort, one at a time. We couldn’t hear what these participants were thinking, but we now had a more robust set of data to spot patterns and validate our moderated session insights. We now had the right mix of qualitative and quantitative.

11. Analyze results.

For the qualitative sessions, we captured observations on how participants grouped the content, and then we looked for patterns in our observations.

We ran the quantitative card sort through OptimalSort software, which provides great starting tools for analysis. As good as these tools are, they don’t hand us the best IA designs. Alongside qualitative data, they give us insights that inform our designs.

Card sort analysis doesn’t give us the IA design. It gives us insights that inform our designs.

We focused much of our quant analysis on standardizing categories — a tedious process that produces the most accurate picture of user-generated categories and category names. We then added some insights from two data visualizations: the dendrogram and the similarity matrix.

12. Generate new IAs.

Armed with our analysis, we were ready to propose changes to the project team’s proposed IAs, and generate new IAs. After some back-and-forth with the project team, we ended up with 3 IAs.

Two of these were completely new versions. The third IA was adapted from the best option from the marketing team’s original drafts. We improved the labels but left the organization mostly untouched. Since it was the closest thing we had to a control version, we referred to this as “the original IA”.

Now we were ready to test our 3 IAs with a new set of users.

The Result: 75% Increase in Findability Plus Design Durability

To evaluate an IA, tree testing is our method of choice. We ran an A/B/C tree test with our 3 IAs. We recruited 540 participants and randomly assigned each person to one of the three IAs — giving us about 180 participants per IA. We randomly gave each user a subset of tasks from a total set of 18 tasks. As with the card sort, we ran both qualitative and quantitative sessions, giving us the right mix of insights and confidence.

Based on what we observed in tree testing, we picked the 2 top-performing IAs and made improvements to each of them. We then ran a final test with our 2 updated finalists — recruiting 360 new participants and giving them the same tasks. We compared the performance of the finalists against each other and against the 3 IAs we tested initially.

The top-performing IA outscored the original IA by 75% in terms of content findability.

By the end of the tree testing process, we had tested 5 user-centered IA variations with 900 users. The top-performing IA outscored the original IA by 75% in terms of content findability.

A couple of months later, the IT team launched the microsite with the top-performing IA.

This project and product launch took place 8 years ago.

Today, the microsite is still there. Its content has grown exponentially. The web ecosystem it sits within has completely changed. No one remains from the original product team.

Follow-up user research has led to redesigns. And yet, the IA’s organization and category names have barely changed from the version we landed on years ago.

With the help of one card sort study, the team was able to transform its company-centered IA into a user-centered IA that has stood the test of time.

How a card sort helped a top financial firm create an intuitive IA was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

• In 2021, a leader in the moving and storage container space was eager to transform its customer experience.
• After observing in-depth customer research, their team used journey mapping workshops to brainstorm ideal experiences.
• The team walked away with the foundation for a multi-year vision and concrete ideas for large and small innovations.

How Do You Achieve a Best-in-Class User Experience?

As the Covid-19 pandemic ignited the U.S. housing market, it added fuel to the already-hot moving and storage container space. 1–800-PACK-RAT is one of the largest players in this industry, landing on the Inc. 5000’s list of fastest-growing companies 7 times in the last 8 years. In late 2020, the North Carolina-based company secured $160 million in financing from The Carlyle Group and PNC Bank to help it increase its market share. With strong competition from PODS®, U-Haul, and others, 1–800-PACK-RAT was eager to find ways to differentiate its service.

In early 2021, 1–800-PACK-RAT’s marketing and technology teams began work on a transformation of the customer experience. This initiative included an overhaul of pricing, order management, and all other backend systems as well as a complete website redesign. To guide this transformation, the teams wanted a concrete, multi-year vision and plan to achieve a best-in-class user experience. Since technology work was underway, this plan would need to include both long-term innovations as well as short-term changes.

1–800-PACK-RAT partnered with Marketade to build the foundation for a customer experience vision based on qualitative discovery research. To get there, our teams collaborated throughout a series of steps over 2 main phases: discovery research followed by journey mapping workshops

Phase 1: Qualitative Discovery Research

Kickoff & Planning

In this initial step, we learned what we needed to plan the project, recruit the right participants, and design the research and outputs to be as impactful as possible. Our steps included:

• A Zoom kickoff meeting with 4 members of the 1–800-PACK-RAT team
• 1:1 interviews with 3 executive stakeholders to understand their short-term and long-term goals for the company and its customer experience
• 1:1 interviews with 3 sales reps to better understand key personas, customer flows, and known/assumed pain points

Based on what we learned from these discussions, we wrote a plan for our qualitative user research, including detailed recruitment strategies, recruitment screener surveys, and interview discussion guides.

Recruiting

We recruited 10 participants for research interviews. We used a few sources to recruit candidates, including an email to recent 1–800-PACK-RAT customers. To narrow down our pool of candidates and select 10 high-quality, representative participants, we relied on a written questionnaire followed by a phone screen interview.

While our phone screens primarily focused on selecting the best candidates, they also served as mini-research sessions; by the time we finished these interviews, we knew much more about the audience and product and were better prepared to conduct in-depth interviews.

Based on audience criteria defined during the discovery phase, we recruited:

• A mix of recent 1–800-PACK-RAT customers and those of similar services
• A mix of long-distance move customers and storage customers
• A mix of desktop and mobile users

After scheduling the participants, we were ready to dive into the research phase.

Customer Interviews

We conducted all customer interviews via Zoom. Sessions lasted about 1 hour and followed this general structure:

1. Introduction to the study and session
2. Interview about their recent moving/storage needs and experience, including the digital experience
3. Observation of participants demonstrating steps in their flow, summarizing relevant information, and completing key tasks

The interview portion of the session included questions to help understand:

• What prompted their need for a container
• What types of competitive research they did
• How they completed and managed their booking
• What touchpoints they interacted with (website, phone call, email, etc.)
• The continuity of their overall experience

Here are examples of interview questions we asked participants:

• ​​Could you tell me the story of your move/storage needs, and how [1–800-PACK-RAT] fit into that?
• How did moving/storing with [1–800-PACK-RAT] compare to other moving or storage solutions you have used previously?

During the observation section, we asked the participants to screen share while they took us through a re-creation of their research and booking process. In many cases, customers were able to pull up their actual confirmation emails and recall both delights and pain points that arose during the process.

Once the research was wrapped up, we were ready for the most important step: workshops for collaborative analysis, journey mapping, and solution ideation.

Phase 2: Journey Mapping Workshops

Over 3 days, 10 stakeholders from 1–800-PACK-RAT collaborated via Zoom and Miro to analyze the research, map customer journeys, and prioritize opportunities. The workshop participants represented a diverse set of teams, roles, and seniority levels, including:

• Chief Digital & Technology Officer
• SVP of Pricing & Revenue Management
• SVP of Sales & Marketing
• Senior Sales Supervisor
• Director of Ecommerce/Customer Experience
• Ecommerce Product Owner
• User Experience/Brand Designer.

Pre-Work & Research Recaps

Prior to the workshops, each team member watched 4–6 research session recordings that we had assigned to them. While observing, they took structured notes based on guidelines that we provided. This homework produced a set of direct research-based observations that set the foundation for our workshop activities.

Each team member watched 4–6 research session recordings prior to the workshops.

To kick off the first day, each team member presented a summary of 1 of their assigned interviews while their colleagues jotted down additional observations.

Customer Needs Analysis

Team members posted their top research observations on a Miro whiteboard. They worked together to sort the notes into groups based on common themes.

To encourage a user-centered approach, the team labeled each group with an “I need” or “I want” statement, such as “I want to know where my container is.” The team then voted for the groupings they felt were most important to address during journey mapping.

Next, the team built out the groupings into 9 full customer need statements using the format of “I want/need _____, so that I can _____.” For example: “I want … to know where my container is so that I can … plan my move-in date at my destination.” For each need statement, the team used their research observations to generate 4 to 8 “so that I can” benefits, reaching over 50 benefits in total.

These statements summarized customers’ goals and motivations and helped the 1–800-PACK-RAT team start to see what it needs to achieve a best-in-class user experience.

They were now ready to start mapping the existing customer journeys.

Current Customer Journeys

To start the journey mapping process, we worked together as a full team to:

• Define the 5 main stages in the containerized moving/storage process
• List 1–800-PACK-RAT’s internal goals for each stage

We then broke into small groups to build the journey maps. Each group took a customer segment (e.g. long-distance mover) and completed the following steps for each of the 5 journey stages:

• List the specific steps the user took.
• Define the customer’s goals and touchpoints.
• Identify the customer’s thoughts and emotions.
• Note friction points and pain points.

Ideal Customer Journeys

Once they’d built their existing customer journey, each group created an ideal customer journey for their segment by following these steps that we’ve adapted from Jared Spool’s approach:

• Brainstormed ways to remove instances of frustration
• Explored ways to add delight
• Individually sketched detailed solutions
• Discussed and combined their solution ideas to build a new journey

Innovation Ideation & Solution Prioritization

Groups presented their existing and ideal journey maps to the rest of the team. During each presentation, team members identified the biggest opportunities to transform the customer experience.

The last 2 workshop exercises helped the team turn their journey maps into actionable solutions. First, we facilitated a “bang/buck” activity where team members . One by one, we discussed the innovation opportunities that rose to the top in the journey mapping presentations. The team plotted each opportunity on an impact/effort matrix. By the end of the activity, the team had organized the opportunities by relative impact and effort, making the prioritization of efforts easier and more objective; e.g. “let’s prioritize high impact/low effort opportunities over low impact/high effort ones.”

Finally, we led a session to brainstorm new content ideas for short-term wins. The team generated over 40 specific ideas for new content on the website or to be pushed through communication channels; for example, a content piece on “how to pack your container.”

These final exercises helped the team convert their journey maps into actionable solutions.

The team sorted the ideas into the 5 journey stages to show where customers would most benefit from that content. They then voted on the content ideas based on estimated impact to help prioritize the next steps.

Summary Report

After the workshops, we wrote a report that summarized the project’s process and outputs. The report captured digital versions of the journey maps, the opportunity matrix, and content brainstorming — and explained how the team arrived at these outputs.

By itself, the report would have done little to drive change at 1–800-PACK-RAT. But it provided a useful reference point for a team that participated actively in the workshops and throughout the project.

Outcome: Alignment & Actionable Insights

In a little over a month, Marketade and 1–800-PACK-RAT had conducted in-depth customer research and used that research to generate a series of actionable outputs:

• 9 customer need statements that summarized customers’ top goals and motivations when interacting with the service.
• 3 current journey maps of the containerized moving and storage experience for key customer segments.
• 2 ideal journey maps that turned current frustrations into future instances of delight.
• 15 innovation opportunities generated from the ideal journeys, prioritized by relative effort and impact.
• 40+ content ideas to improve the short-term customer experience, all mapped to journey stages and informed by user interviews.

How did the 1–800-PACK-RAT team describe the outcomes and benefits of the project after it was complete?

• A brand/UX designer remarked how the project “produced curated ideas and helped the team align on a vision.”
• A C-suite officer appreciated how the project “gave us a breakdown of actionable experiences for each journey phase.”
• And an e-commerce manager called the project “hands down amazing [at producing] clear feedback, unbiased discussion and action items.”

After the project, the 1–800-PACK-RAT team would use the need statements and journey maps as the foundation for a multi-year customer experience vision. And they would begin their efforts to transform the experience by implementing the product innovations and content ideas that they had agreed on and prioritized in the workshops.

Moving Forward: How 1–800-PACK-RAT Used Journey Maps to Start a Customer Experience Transformation was originally published in Marketade on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a case study from Marketade’s work with an EU-based firm that provides services and solutions across various markets, including security, transportation, and aerospace. With over 75,000 employees in over 50 countries, they sell to governments and enterprise organizations around the world. Their annual revenue is over $20 billion USD.

Marketade has partnered with this firm on numerous projects since 2019, all of which included qualitative research and stakeholder workshops. Through in-depth customer interviews, our work gives the firm’s UX, product, and innovation teams a deep understanding of the journeys and needs of their buyers and users. And through collaborative workshops, we help these teams gain alignment and generate evidence-based solutions and strategies.

Market Validation & Value Mapping in Enterprise Security

A recent discovery research project focused on enterprise cybersecurity leaders who are the key decision-maker for security strategies and purchases at their organization. The team we worked with was exploring a new type of security solution that would challenge the model used by most organizations. The team had 2 critical questions, the generic versions of which are:

Question #1: What types of organizations are most ready to adopt a [certain type of] security solution?

Related questions included:

• What is the value that such a solution would bring to the organization?
• Who are our target buyer personas at the organization?
• How ready are they in terms of budget and strategy?

Question #2: What are the barriers preventing such a tool from being viable, desirable, and feasible?

To help the team answer these questions, Marketade:

• Recruited and interviewed 10 senior-level cybersecurity leaders from large and mid-size organizations
• Facilitated a collaborative workshop to observe, analyze, and act on the research
• Delivered a report that summarized the process, research, and workshop outcomes

Step 1: Discovery

We assigned 1 lead researcher and 2 supporting researchers to this project. After a virtual kickoff meeting and follow-up discussions with the client team, our team designed the research and delivered a 6-page research plan.

The research plan started with clearly-stated project goals and research questions. The bulk of the plan was a detailed discussion guide for our moderators to follow during research sessions. This guide included over 50 potential interview questions — organized into various buckets and levels — that all tied back to the research questions.

To get the most value from each interview, our researchers always depart from the script and adapt based on the participant’s responses. Still, we find it critical to fully think through the likely paths an interview might take and have a plan for them. We collaborated closely with the project team to develop the questions and interview structure.

Here’s a small excerpt from our plan:

As you can see, we rely heavily on open-ended questions, which is especially important for generating insights from buyer journey studies like this one. We want to put the team’s assumptions to the test and be open to surprises.

In addition to the discussion guide, we wrote a 3-page recruitment plan that shared details about the target audience and selection criteria for research participants.

Step 2: Recruitment

The recruitment of participants for this project was particularly important and challenging. The target audience is relatively small, hard to reach, and — due to the topic — reluctant to be interviewed. This makes it tempting to find easier-to-recruit proxies: people who are less senior in their organization but understand the security space. We agreed with the client team that, given the research questions, we could not cut corners with recruiting; we needed to speak with the primary decision-makers.

For this project’s recruitment, we had to be more creative, rigorous, and persistent than usual. We sourced candidates from a variety of places, including LinkedIn ad campaigns that targeted professionals by job title and company size (e.g. CIOs at companies with 10,000+ employees). All of our recruitment sources pointed candidates to a 6-question web-based screener; to get more reliable responses, we used 4 open-field questions, such as: If a data security breach occurred at your company, what would your role be?

We then conducted phone screening interviews with candidates that passed the web-based screener. Again, we included some open-ended questions, such as: Related to your job, what keeps you up at night? What is your biggest worry or pain point? Since our skilled researchers conducted the phone screens, we captured the data necessary to select the right candidates — while increasing our grasp of the project’s research questions.

Based on our rigorous process, we were able to find 10 qualified participants from a mix of industries that met a desired split between large and mid-size organizations.

Step 3: Research

We conducted interviews with each of the 10 participants. These sessions were:

• One-on-one moderated interviews
• Conducted remotely via Zoom
• Recorded so the team could watch them in full

While our researchers are not cybersecurity experts, through the discovery and recruitment process we learned enough about the subject to be able to conduct effective interviews with experts in this area. 1 or 2 client team members joined many of the live sessions as observers, allowing us to leverage their expertise during a Q&A period at the end of each interview.

Step 4: Workshop

Following these interviews, we held a 1-day collaborative workshop to observe, analyze, and make decisions based on the research. 10 stakeholders participated in the workshop, which was facilitated by 2 Marketade researchers. The workshop team represented a diverse cross-section within the relevant business unit, including stakeholders responsible for innovation, competitive analysis, product, UX, and engineering.

We started by sharing focus questions for the workshop; these were based on the project’s research questions. We then gave instructions on what to look for in the research sessions and how to take notes on “stickies” based on the focus questions. After each session, the team put the important notes on 2 Value Proposition Canvases: one for mid-sized organizations and one for large organizations.

The Value Proposition Canvas is a tool developed by Alex Osterwalder, creator of the Business Model Canvas and other tools to help teams visualize, simplify, and collaborate on complex business problems.

After all of the sessions were complete and the notes were posted, we led an exercise where we grouped similar observations to uncover trends and consolidate and elevate findings.

Next, we used a series of value proposition mapping exercises that helped the team understand and reach alignment on:

• What their customers are currently experiencing
• How those desires and challenges might be addressed by a potential solution
• What the product should provide in order to address the customers’ needs

These exercises helped the team gain alignment on a minimum viable product or MVP.

We then facilitated discussions on 4 questions:

1. To whom should an MVP be tailored?
2. How ready (if at all) are potential customers to consume a product like this?
3. What are the barriers to it being viable?
4. What’s their biggest “burning fire” or security concern?

We split this discussion between 2 groups: one for mid-sized organizations (1K-10K employees) and one for large organizations (10K+ employees). Each group identified barriers to the team’s solution concept for the 2 types of organizations.

Step 5: Report

After the workshop, we wrote and delivered a 13-page report to the team that:

• Began with an executive summary
• Covered the research goals and process
• Shared basic information about interviewees and their session recordings
• Documented the workshop process and outcomes
• Summarized key takeaways and next steps

“Identified the Main Gaps”

While we can’t share the specific outcomes, we can say that the team made a number of key strategic decisions — and implemented critical changes — based on the alignment resulting from this project.

A chief data officer and innovation VP was the project’s key stakeholder. After the project, he praised the “excellent research and workshop facilitation” and said the effort “identified the main gaps to focus our efforts on for the next steps.”

Understanding Enterprise Buyers Through User Research & Value Proposition Mapping was originally published in Marketade on Medium, where people are continuing the conversation by highlighting and responding to this story.

A fitness tech company visits its customers’ homes to gain a deep understanding of their needs — paving the way for actionable personas and journey maps.

Home Visits Before a Home Fitness Explosion

The Covid-19 pandemic sparked a dramatic increase in demand for home fitness equipment. Powered by AI and focused on strength-building, Tonal is the “smartest home gym you’ve ever seen,” according to Men’s Health. Valued at over $500 million, the Bay Area company is backed by Amazon and athlete-investors like Stephen Curry and Serena Williams.

A couple of months before the pandemic, Marketade led in-home qualitative research and a collaborative journey mapping workshop for Tonal. The project’s main goal was to uncover design opportunities to improve the Tonal customer experience and lead the market on connected strength.

Related goals included:

• Learn the habits, needs, and patterns of a representative customer base
• Understand Tonal user fitness journeys and customer expectations
• Create actionable insights to improve and iterate the product

9-Step Process

Here’s an overview of the step-by-step process we used for this project. Our process is adapted from Jared Spool’s “LiRPPS” approach.

1. Kickoff & Planning

We facilitated a kickoff meeting to align on project goals, roles, and measures of success. From the start, Tonal rejected the traditional user research approach — one where researchers disappear to conduct sessions in isolation and report back to the team what they learned.

Instead, Tonal embraced the idea that research works best as a team sport — and committed to actively participating throughout the process. Based on the kickoff and some follow-up conversations, we designed the research and planned the project in detail.

2. In-Home Research

In December 2019 and January 2020, we conducted in-home interview/observation sessions with 12 Tonal customers in 4 cities. We interviewed customers about their fitness history and current fitness routines, and their experiences using Tonal. We also observed them as they completed a Tonal workout of their choice.

A diverse mix of Tonal designers, product managers, marketers, and engineers joined our research sessions live — 1 or 2 at each stop — and took structured notes as they watched.

3. Workshop Pre-Work

In addition to observing 1–2 in-home sessions live, each Tonal stakeholder watched full recordings of at least 4–5 other sessions — for a total of at least 6 sessions per stakeholder.

Stakeholders tracked all of the sessions they watched in a shared Google Doc, which increased accountability, participation, and distribution across the 12 sessions.

In addition, for each session that they observed in person, stakeholders prepared a 3–5 minute verbal summary.

4. Research Review

In mid-January, the Tonal project team gathered at their Bay Area office for a 2-day workshop.

To kick off the workshop, the team:

• Watched a short highlight reel of all 12 research sessions;
• Presented their 3–5 minute session summaries, with different stakeholders covering the most interesting and important takeaways from a session they observed in person;
• Took structured notes as they watched highlights and listened to summaries;
• Collaboratively created an action list that encompassed the major themes that emerged from the research — and grouped those themes into easy-to-remember categories.

5. Persona Generation

The team broke into small groups and followed a step-by-step process to create personas based on the research. As part of the process, they identified critical information about each persona, such as goals, challenges, motivations, and other elements they deemed relevant to the problems encountered by that persona. They then picked the most critical personas.

6. Mapping the Current Journey

Drawing directly from the research they observed, small groups mapped the current journey for each persona. The maps illustrated process steps, moments of frustration and delight, goals, and pain points.

7. Sketching the Ideal Journey

Groups crafted an ideal journey that removes moments of frustration and adds elements of delight. As part of this process, each team member sketched product changes and solutions to turn that ideal journey into a reality — and received feedback on their ideas.

8. Defining Next Actions

We facilitated an exercise to define next steps required for each solution or feature that was approved by the workshop team. We assigned owners and deadlines to each action.

9. Digitized Reporting

After the workshop, we worked with Tonal and a professional designer to transform the sketches of the personas and journey maps into a more distributable format. Tonal stakeholders spent time adjusting and wordsmithing the copy, and Marketade provided visual design. We also created a final report that captured all research, workshop steps, team decisions, and next steps.

10x Growth

As Tonal’s workshop wrapped up in mid-January 2020, we could feel the team’s excitement and optimism about the future. Yet no one could have predicted how fast the company would grow in months ahead.

CNN reported in September that the “company’s business is performing over ‘10x’ compared to one year ago,” based on an interview with Tonal’s CEO Aly Orady. “We’re well over $100 million run rate in terms of our sales momentum,” Orady told CNN. “We’re expecting a really, really sizable holiday season.”

The company’s business is performing over ‘10x’ compared to one year ago.

The pandemic gets much of the credit for Tonal’s phenomenal growth in 2020. Yet plenty of companies in “fortunate 2020 industries” failed to grow, especially as opportunistic competitors jumped into their space.

Tonal capitalized on its opportunity in part by being a deeply customer-centered company. Plenty of companies claim to know their customers because they send surveys or run focus groups or create journey maps based on big data. Usually, their understanding is shallow and misleading.

With this project, Tonal took a different approach. As a home fitness company, they knew they needed to go into customers’ homes — one at a time — to truly understand their journeys and pain points. They also knew that paying a consultant to do the research and map the journeys wouldn’t do the trick. So they sent team members out to 4 U.S. cities to gain direct exposure to customers. Then they carved out 2 days to come together as a group, align on the research, and build actionable journey maps.

With this shared customer understanding and action plan, the team was well prepared for the challenges and opportunities that 2020 would bring.

How a Fitness Company Turned Customer Journey Mapping into a Team Sport was originally published in Marketade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Challenge: Low Online Adoption for a Key Service

A Fortune 500 company with over 10 million customers was making steady progress on a key business goal: getting more customers to use its digital self-service channels rather than phone into its call centers. Across numerous product/service lines and transaction types (sales, support, etc.), for years the company had consistently reduced costs while improving the customer experience by making it easier to transact online.

And yet, for one critical service line, the online transaction share was stuck. Only 1 out of 10 customers completed the transaction online. Despite plenty of focus and investment — and data showing that many customers preferred not to call a support center — this service team struggled to move the needle.

The online transaction share was stuck at 10%.

Their lack of progress drew the attention of executives. As the company’s customer base grew, having 90% of these transactions handled via phone threatened to overwhelm its support center staffing and costs. Management also feared that customer retention and profitability were suffering as the company disappointed customers who wanted to transact digitally.

Action: Collaborative Research & Iterative Testing

The company partnered with Marketade to conduct a mixed-methods, “team sport” research project. The project aimed to:

• Give a diverse set of teams a deep understanding of this service’s customer experience and customer mindset.
• Identify barriers to customers using the self-service channel, including: gaps between product positioning and customer perceptions; confusion around the product and how it works; and reasons that customers call rather than transact online.
• Generate viable initiatives to combat these barriers and increase self-service adoption for this product line.
• Test and launch the most promising solution ideas, and measure the impact.

Here are the key steps we followed:

1. Current State Research

In order to gain a wide and deep understanding of the service line’s challenges, we drew upon both qualitative and quantitative methods to lead a multifaceted research initiative focused on the current state of the customer experience. In cooperation with multiple company stakeholders across various departments, we conducted the following research:

• Interviews with 10 stakeholders representing a diverse set of teams across the organization (IT, business, marketing, etc.)
• Review of existing user research
• Analysis of 4 competitor products and their customer experience
• Review of analytics data for digital transaction flows (mobile app and web) and phone vs. self-service results
• 1-day field visit to a call center, observing/interviewing 15 phone reps
• One-on-one interviews with 20 customers, many of whom had recently used the product
• Usability testing with 12 customers, to identify friction points in digital flows
• Card sorting study with 30 participants, to understand what broader product areas people associated with this service
• Survey of 200 current customers, including a subset who had recently transacted by phone for this product

Based on our research, we identified 3 broad problem areas that were hurting self-service adoption.

Within each problem area, our researchers documented a number of specific barriers to online adoption. Yet we knew from experience that if we just presented these barriers in a report, our research would fail to drive action and results. So we kept the specific findings in our back pocket and turned to what we know works: team sport analysis and ideation.

2. Stakeholder Workshop

We facilitated a virtual workshop with key stakeholders across 5 different teams. We structured the workshop as follows:

1. We outlined the 3 broad problem areas we’d identified in our current state research.
2. Stakeholders listened to in-depth excerpts of the research interviews and took structured notes that captured specific self-service barriers.
3. We guided the team through an affinity mapping exercise to reach a consensus on the biggest barriers and opportunities. Stakeholders added their findings as notes to the virtual whiteboard. They grouped similar findings together and gave each group a name. They then voted to identify the top 3 biggest barriers.
4. The team brainstormed solutions to the top 3 barriers, then voted on those solutions.

Prior to this project, these 5 teams had worked in silos as they tried to improve KPIs for this service. Yes, they shared their tactics and results by email and in meeting presentations. But they lacked a coordinated effort built on a shared understanding of problems and opportunities.

This workshop was the critical step in shifting from silos to alignment. By gathering in the same (virtual) room, observing the same user research, and analyzing and brainstorming together, the workshop magically got the team on the same page.

The workshop magically got the team on the same page.

It also motivated the team to act. At one point during the workshop, the VP of the service line leaned over to the UX team manager and said: “This is probably one of the best WebEx sessions I’ve ever sat through — very interactive and it’s keeping me engaged!”

Based on the alignment and excitement generated from the workshop, it was tempting to move straight into solution design and implementation. But we weren’t quite ready; the ideas generated in the workshop were broad, rough, and full of untested assumptions. To be able to run with solution ideas that had the best chance to move the “stuck” KPI, we needed a few additional research outputs.

3. Personas, Journey Maps & Content Audit

Based on the current state research and the workshop, we created a set of customer personas for this service. Each persona focused on a specific problem area identified in the collaborative analysis.

Next, we took the most important persona and mapped their existing customer journey, chronologically plotting the barriers the team had previously prioritized. This mapping exercise helped the team:

• Visualize the most important obstacles in the context of a real customer’s experience with the service;
• Appreciate how barriers that are small in isolation can combine to create a very poor experience over the entire journey;
• Focus on a previously ignore gap that existed early in the journey: the lack of customer awareness that this service transaction could be completed online.

Again drawing from the research and the collaborative ideation process, we then created an ideal customer journey. In this exercise, we turned the barriers and frustration points into solutions and areas of delight. The visual demonstration of where we could gain the most by turning frustration into delight helped us to focus and prioritize our solution exploration.

Finally, using the lens of personas and journey maps, we conducted a content audit across a wide range of channels and digital properties. We knew from our previous steps that, whether in the form of a marketing campaign or a call center script or mobile app microcopy, content was a major contributor to existing barriers.

Tie a content audit’s findings to personas and journey maps — which in turn are grounded in research and team collaboration — and you create a formula for action.

The journey maps helped the team to see how many channels a typical customer touches for this service — and how turning content inconsistencies into a cohesive experience could transform user uncertainty into confidence. Our audit also spotted many instances of confusing or missing content that were driving customers to call.

Too often, personas, journey maps, and content audits sit on stakeholders’ shelves and fail to drive change. By combining the tools and tying the content audit’s findings to personas and journey maps — which in turn are grounded in research and team collaboration — we created a formula for action.

We were now ready to regroup with the teams, design specific solutions, and run experiments.

4. Rapid Experimentation

At this point in the project, we had:

1. Conducted in-depth current state research, using a wide range of methods to interview/observe a diverse group of audiences;
2. Gathered stakeholders to observe and analyze the research — and prioritize the top problems and opportunities as a team;
3. Used the top problems/opportunities to generate personas and journey maps — which in turn provided visual, real-world context for a content audit.

By the end of stage 3, the team had generated plenty of solution ideas that they were excited to explore, design, and develop. And critically, they had a robust framework to pick the best of the best ideas — the ones that research indicated would give the most bang for the buck.

Over a couple of months, we coordinated 8 experiments across the following channels and products: Email, Website, Customer Service, Sales, and Mobile App.

Experiments included content and design changes to the following:

• Confirmation emails
• Customer login pages
• Mobile app flows
• Website product marketing pages

For many of these changes, we first tested the new copy and/or design through prototype UX testing with a small number of participants. We iterated based on what we learned before launching the new design. Where possible, we worked with the company’s metrics team to run an A/B test.

Results: Measurable Impact, Team Alignment & Executive Excitement

The experiments produced exciting wins for the organization. Of the 8 tests, 5 showed meaningful improvement to online adoption.

One of the results was particularly exciting. Based directly on what we’d observed in our current state research and what we documented in the personas/journeys/content audit, we spotted an opportunity to add one line of copy to one page of the mobile app. It was a point where analytics data showed lots of users were switching from online to phone, and where qualitative research told us why. The new copy we wrote directly addressed the uncertainty we’d uncovered in the research, and produced a notable shift in the percentage of users completing the transaction online.

One line of copy was all it took to cause a big lift.

In combination, the content, design, and service changes helped ignite a significant increase in the percentage of customers completing this service transaction online.

The project also laid the foundation for long-term improvements to the customer experience with this service. In particular, the analysis workshop and follow-up sessions brought a diverse team together and, through the user research, gave them common ground for discussions.

To empower these teams to continue innovation, iteration, and experimentation, we built content guides, led interactive copywriting training, and created call center talking points.

A few months later, online adoption for this service was back in the spotlight with the company’s executives. This time, however, the attention was welcome. Senior management joined a meeting to hear the positive results and to learn about the research-based process behind them. As the meeting was ending, one of the execs thanked the core project team, and said “this is exactly what I was hoping for.”

How Holistic User Research Helped Remove Self-Service Barriers was originally published in Marketade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Years ago, an enterprise company wanted to improve the user experience of the internal web application used by its call center support reps. With thousands of reps helping millions of customers, support task efficiency was a key driver of both business KPIs and customer satisfaction.

Many companies invest heavily in the UX of their customer-facing software yet ignore the usability of employee-facing systems.

This company was ahead of the curve. They recognized the potential ROI of a redesigned support app built on a deep understanding of the reps’ tasks, needs, and pain points.

The company’s UX team partnered with Marketade to launch and operationalize a new user research program for its call center software. Together we built from scratch a repeatable research program that has:

• Transformed the user experience of call center staff
• Shown a strong ROI and become a point of pride for the company
• Continually expanded and inspired similar research programs

Here’s the story of how we did it, focusing on the steps that we’ve found to be critical for any new research initiative.

Phase A: Create

1. Discovery & Pilots

We started with a discovery research process that included stakeholder interviews, industry research, and most importantly, call center field visits. At the call centers, we observed and interviewed 20+ service reps, helpdesk staff, supervisors, trainers, and more — providing critical data for the following steps.

Based on what we learned, we quickly launched a few pilot usability studies with reps. We captured key learnings about recruiting, moderating, and working with the design and development teams.

2. Research Cycle Design

By now, we’d learned enough to design the first version of a repeatable, 2-week testing process — one that mapped to the scrum team’s agile development cycle. The 10-day cycle looked like this:

• 3 days to prototype, design the research, and recruit participants
• 3 days to test (primarily qualitative testing of prototypes supplemented by quantitative testing)
• 3 days to analyze the research, identify problems, generate solution ideas, and collaboratively iterate on those solutions
• 1 day to present the findings and changes to stakeholders

3. Program Design

We also designed and planned an initial version of the larger research program, mapping out:

• Research topics and methods for the first 2 months
• A repeatable process for topic and method selection going forward
• A target number of program participants
• Key stakeholders and management check-ins
• Program KPIs and measurement methods
• A detailed rollout schedule

Phase B: Launch

4. Buy-In Building

With an MVP and early feedback in hand, we shifted our focus to building support from critical stakeholders — from senior management to call center supervisors to scrum team managers.

5. Phased Rollout

We identified a couple of call centers for a full-blown pilot — testing the end-to-end process multiple times, gaining recruiting practice, and building ground-level support. We then prepared for a nationwide launch.

6. Marketing Program

After getting buy-in from management on a program name and brand identity, we built an internal marketing website that explained the program and benefits to reps and featured an application form. We also designed and wrote recruiting email templates and built out an email communication process. Then we launched the program across all call centers.

Phase C: Grow

7. Panel Building

We coordinated with call center management to send out program invitations to curated lists of reps. We also built a process for capturing key attributes of program applicants (job role, tenure, etc.) — and used these as screening criteria to select a representative panel that would support different types of studies. To date, we’ve recruited well over 1,000 participants into the panel.

8. Story Sharing

A few months into the program, we had settled into a steady cadence and had completed over 10 studies. To accelerate program recruitment, we needed stronger support from management across all call centers.

We decided that the best way to do this was to show the impact this research was having in concrete ways. We designed a case study template for the marketing site. For each study completed to date, we produced a story that showed what we tested, learned, and changed. We then promoted the case studies to call center managers.

9. Constant Iteration

As the program matured and surpassed KPI targets, we maintained a focus on continuous experimentation and evolution. Improvements include:

• Shifting from prototype testing to live environment testing
• Creating more realistic user tasks by modeling them on real call flows
• Expanding the research cycle to match changes in the development cycle
• Transitioning panel recruitment and maintenance to new team members

Results: Transformation, Expansion & ROI

UX Transformation

To date, we’ve completed over 60 research studies through this program — across a wide range of software features/flows and supported by a diverse set of research methods. Even high-quality, recurring UX research often gets forgotten in a report and fails to drive the change. By creating a highly collaborative research process that syncs with the company’s agile development cycle, we’ve helped the UX team achieve a high implementation rate for research study recommendations.

Corporate Visibility

The program has received praise throughout the company, from executives to HR. It was a finalist for a company-wide quality award and was promoted in public-facing employee recruitment materials.

New Units, New Programs

As word has spread, new business units have approached the UX team about rolling this program out to their applications and teams. We’ve helped to launch spinoff programs for other call center applications. The program has also served as a poster child and a starting framework for new research programs in other product areas.

Business ROI

Best of all, the UX team has shown a return from the program that resonates with metrics-focused executives: significant cost savings from faster and more productive call center reps, tied back to research-driven usability improvements.

How to Build an Enterprise UX Research Program: 9-Step Case Study was originally published in Marketade on Medium, where people are continuing the conversation by highlighting and responding to this story.

Within the UX research space, there are numerous platforms, panels, and agencies for participant recruitment. There are also many researchers, designers, and product owners who conduct recruiting on their own.

Despite these resources and experience, participant recruitment is the #1 pain point for many teams conducting user research.

Through our work with a wide range of teams, we have learned that frustration with recruitment is strongest when:

1. The audience is hard to recruit; and
2. The team lacks a shared baseline understanding of the audience.

No matter how good the recruiter or the platform, this combination will produce disappointing participants — and disappointing research.

In response to this common struggle among our clients, our team at Marketade developed a strategic alternative to the traditional recruitment approach. We also developed a process to help teams determine when this approach makes sense to use.

What Is Strategic Recruitment?

In short, strategic recruitment treats recruiting as a valuable research asset and alignment process — and invests in it accordingly. It offers a holistic, collaborative alternative to traditional recruitment.

With traditional recruitment…

• Recruitment is solely a means to an end (the research)
• The research team outsources this step if possible — to a recruiter or a platform
• Speed and cost are prioritized over quality, further creating incentives to use panels or platforms
• Team discussion of the target audience is rushed and focused on demographics
• Screening is limited to a web-based, multiple-choice questionnaire
• Rejected candidates are treated as waste, as is the screening process
• Knowledge gained throughout the process is not captured; the process starts from scratch with the next research project

By contrast, with strategic recruitment…

• Recruitment is a means to an end and an end in itself
• The research team owns recruitment — rarely outsourcing it — and allocates the time and resources it deserves
• In particular, the team invests time at the outset to gain a baseline alignment on the target audience
• Screening includes both a written survey and a phone-based interview
• The screening process is part of the research process; the team learns a lot about the audience — and about their own assumptions — as they select and reject candidates
• Knowledge gained throughout the process is captured; over time it becomes a critical, strategic research asset that informs other projects

Is Strategic Recruitment Right for You?

Strategic recruitment is a larger investment than traditional recruiting approaches, and it is not the right fit for all teams or situations. To help determine whether this approach is a good fit for your project or program, ask a series of questions in 2 areas.

1. Audience Factors

Start by estimating the difficulty of recruiting your audience. Take existing information and evaluate your audience across these criteria:

• Size — How large is your target audience? How easy is it to locate them?
• Reachability — How easy is it to reach your audience through digital communications or other sources?
• Busyness — How busy is your audience? How valuable is their time? How easy is it to get them to want to participate in any kind of research?
• Willingness — How sensitive are your research topics for this audience? How willing are they to share the information or show the tools you’ll be asking about?
• Screenability — How easy is it to screen for key factors among a larger pool of candidates? How easy is it to score and select the top candidates?

If you need to recruit multiple distinct personas or user groups for a given research project, estimate the difficulty for each one based on these factors.

An example of a hard-to-recruit audience: Fortune 500 C-suite cybersecurity executives for a research project that requires them to talk about specific parts of their security process. In particular, they are hard to find, their time is scarce, and the topic is highly sensitive.

2. Organizational Factors

Just as important as the audience is your organization’s experience with and understanding of that audience. In particular, ask about:

• Past Recruitment Experience — How many times have you recruited a similar audience for research in the past — either directly or through an agency?
• Past Recruitment Satisfaction — How satisfied have you been with participant quality when you have recruited this type of audience recently?
• Team Clarity and Alignment — Regardless of how difficult or easy the audience is to recruit, how clear is your team’s understanding of the audience? How aligned is your team on the target audience?

These 3 factors have a high degree of overlap and correlation. Generally speaking, the more experience you have recruiting this audience, the happier you are with recent participants and the stronger your team’s alignment.

If you’ve only tried recruiting them once or twice — and participants are hard to recruit — your team is less likely to have a solid, shared picture of that audience; and you’re more likely to have been disappointed by the participant quality.

How Strategic Recruitment Works: Step by Step

If your audience is hard to recruit and your team lacks a clear, shared picture of the audience, you are a strong fit for strategic recruitment. Here’s a step-by-step overview of our approach:

1. Recruitment Team Selection

• Assign a “Recruiter” who has solid experience in both user research and participant recruitment. This person will own most of the action items throughout this process.
• The Recruiter should be able to quickly understand your research goals and methods. Having run research sessions themselves, they should appreciate the research impact of a high-quality vs. low-quality participant.
• Support your lead with a “Writer”. This is typically a marketer or copywriter. Their job is to ensure that your recruitment campaigns, ads, and messages are as effective as possible at attracting and converting candidates.
• Assign a “Decider” (name borrowed from GV Design Sprint). While you may have multiple people reviewing research candidates, the Decider will have the ultimate say on who gets approved or rejected. Often this role is filled by a research lead, a product owner, or a design lead.

2. Discovery Review

• Hold a kickoff call with the Recruitment Team and other relevant stakeholders; discuss your research goals, audience, timeline, and process.
• Review relevant resources (e.g. software demo, persona documents) to get up to speed on your product and audience. also review any past recruitment efforts for similar audiences (screeners, participant lists, etc.). Build an understanding of what has worked and failed in past recruiting efforts.

3. Audience Alignment Workshop

Soon after the kickoff and before you start recruiting, hold a half-day remote workshop with your recruitment team and key stakeholders. This workshop lays a critical foundation for your recruitment efforts and the larger project. It helps your team to:

• Ensure basic alignment on project/program research goals, and understand how effective audience recruitment supports the larger project.
• Gather, document, and organize the audience-related assumptions currently scattered across your stakeholders’ minds — and flag the biggest and riskiest areas of disagreement.
• Share the knowledge of the people within your organization who have the most contact with this audience (e.g. sales reps, customer support).
• Use collaborative exercises to reach high-level alignment on your target audience and their needs and behaviors. Your goal is just enough understanding to move forward with recruiting, not detailed personas.
• Generate and prioritize high-level sourcing ideas, screening questions, and selection criteria.
• Draft a candidate rating scale, scoring system, and selection process.
• Identify the “Review Team” for candidate selection and input: this might just be your Decider, or it might be the Decider plus one or two other stakeholders.

4. Reusable Recruitment Plan

• Draft a recruitment plan that includes screening criteria/questions, sourcing plans, and proposed incentives; adapt the plan based on stakeholder feedback.
• As part of your recruitment plan, deliver a summary of the alignment workshop’s outputs. Beyond the current project, this summary serves as a useful tool for future research and recruitment.

5. Custom Sourcing

Conduct full-service, customized sourcing of participants, such as:

• Google, Facebook, and LinkedIn campaigns with advanced audience targeting, ads written from scratch, and ongoing optimization
• Live intercept ads on websites or within products
• Online/offline community outreach (e.g. industry groups, Slack channels, hobbyist forums)
• 1:1 outreach via LinkedIn, Slack, and other channels

Monitor campaign metrics and adjust your investment based on performance.

6. Advanced Screening

• Direct all recruitment campaigns to a web-based screener that includes both multiple-choice and open-ended questions.
• Review web screener responses against your recruitment criteria, and select the top respondents.
• Conduct 5–10 minute phone screening calls that you record.

7. Collaborative Review & Selection

• Use a group recordkeeping tool like Airtable to send the Review Team batches of candidates that meet a minimum scoring threshold.
• Ask the Review Team to review key candidate information, including the phone screen recording, and then quickly score the candidate and share qualitative feedback. Depending on the team’s comfort level and bandwidth, they can choose to review and score as many candidates as they like — all, some or none.
• Based on scoring and feedback from the team, the Recruiter selects the top candidates for research sessions — enough to give you backups in the event of no-shows. For judgment calls, the Recruiter asks the Decider.

8. Scheduling & Payments

• Work with your researcher(s) to set up a session scheduling process that works best for them. We typically use an online scheduling tool like YouCanBookMe that syncs with the researcher’s calendar and that allows participants to schedule their sessions.
• Create customized, automated email and/or text messages for participants (e.g. confirmation, reminder). Also, call participants to remind them of their sessions.
• Create an easy way for your researcher(s) to notify the Recruiter when sessions are complete so he/she can quickly send incentive payments to participants.

9. Final Report

Write and share a final report that includes:

• All raw data collected during the recruitment process (e.g. respondent information, screener recordings, scoring of candidates).
• Analysis of top-tier candidates.
• Takeaways and recommendations for future recruiting of similar audiences.

It took our team a lot of trial and error to discover this more strategic approach to research recruitment. We’ll continue to evolve our steps based on what we learn from our projects and from others in the research community.

What ways have you found to make your recruitment more strategic and effective? Please share them in the comments!

Strategic recruitment: a new approach to finding participants for UX research was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Faced with management requests for bigger numbers, many research teams dig in on qualitative data. Here’s a more effective approach.

Kimberly kept surprising us.

She was using an expert marketplace website, trying to find the right specialist based on her needs. The bare-bones homepage had one call to action: a large search box followed by a button, with some supporting text.

Kimberly ignored it and clicked the navigation at the top. After a few minutes, she scrolled down another page, found an advanced search option, and used the last of 4 possible filters — “expert location” — to start her search.

“Let’s see who they have near me,” she thought aloud.

“Huh,” muttered one of the stakeholders on the virtual meeting.

We were gathered remotely with a team of designers, marketers, IT managers, and product owners, watching usability testing sessions. One by one, research participants were using an expert finder tool. Similar to Zocdoc, FindLaw, or Realtor.com, it is the largest online marketplace of its kind in North America.

Kimberly was User #1.

User #2 behaved more as we expected. She went straight to the homepage search box, did a search, and spent the rest of her session exploring search results and expert bios. And so, we figured Kimberly was an anomaly.

That is until User #3 started. He behaved just like Kimberly, quickly finding the advanced search and filtering by location.

User #6 followed the same pattern.

In the end, half of our 6 participants used the marketplace in the same, surprising way: mostly ignoring the default path and using what seemed like a workaround.

“Only 3 People?”

A year earlier, product owners had designed the site based on assumptions about what matters to target users, including:

• When searching for this type of expert, users care most about the expert’s specialization.
• Since they are unlikely to visit the expert in person, most users do not care about the expert’s location.

Now, as we analyzed and discussed the research, team members were saying things like “we were wrong about our users” and “we need to change how this works.”

Not so fast, said the marketing VP who was popping in and out of the meeting:

“This was only 3 people from a sample of 6 — in a research study. Are we really going to change the site based on this? I’d like to see some bigger numbers …”

The Common Response: Dig In on Qual

Stop, and think for a moment about how this story usually goes.

For research experts who understand the power of in-depth testing with small numbers, it’s easy to be annoyed at stakeholder skepticism around qualitative data — and at requests for “proof” or “bigger data.”

Often we respond by doubling down on qualitative research. Here are 4 types of responses that we’ve used:

“The Nielsen”

We point to Jakob Nielsen’s article or graph showing that you can often uncover 85% of UX problems by watching just 5 users.

“The More-of-the-Same”

We offer to increase the qualitative sample size to 10 or 20 or whatever it takes to convince the stakeholder.

“The Buck Pass”

We respond by saying “big numbers are the [Analytics / Metrics / Data Science / Marketing / Other] Team’s job.”

“The Dismissal”

We conclude that “they don’t get it” and ignore the question or request, moving onto a project with stakeholders who think more like us.

Sometimes, I think these responses are the right ones.

Often, a sample of 5 is plenty to spot the top problems … and big data does live in another department … and there are lots of other teams knocking on your door for qualitative research.

But more and more, I pause before using one of these responses. Since we are user researchers who preach empathy and understanding, I try to ask:

• How can we be more empathetic toward our metrics-loving stakeholders and better understand their context, language, and goals?
• Why does the stakeholder want validation with larger numbers?
• What are some ways that we could quantify a critical qualitative finding?
• And, most importantly, how can we turn this request into an opportunity? How can we use it to win over new teams — and get them excited about doing more user-centered work in the future?

A Better Approach: Marry Qual & Quant

In the case of the marketplace site, we started by asking questions to better understand what was driving the marketing VP’s skepticism.

We learned about his bosses, their concerns, and how he shares information with them. We learned about past failures and how much was at stake for him. We learned about what it takes for him to secure additional funding for site redesigns.

After we listened and explored some options, we said:

“We get it. We’ll look at analytics data and see how it compares to what we just watched — and get back to you.”

We grabbed a member of the project team with strong Google Analytics experience and paired her with the research lead on this project. They spent an afternoon digging into the data and creating some custom reports.

The next day, we had our answer: 3 times as many users were searching by expert location as by specialty.

Far from an anomaly, Kimberly behaved just like most people on the site. We now had data from tens of thousands of sessions to prove it.

We shared this quantification of the usability finding with the workshop team. The skeptical VP was excited, partly by the specific finding but even more by what the exercise showed: that we could use real-world analytics data to validate messy “lab” research findings.

Short-Term Lift, Long-Term Transformation

A few weeks later, the team sketched and implemented a simple change to the site’s homepage.

Based on the qual and quant research, they hypothesized that adding a location search option to the existing specialization search option would increase leads.

They were right: in an A/B split test, the new homepage converted visitors to leads 50% better than the original. It seemed like a small change — 2 CTAs instead of 1 — but it had a huge impact on the conversion rate. The team quickly made this the new default design, and monthly KPIs jumped.

The VP was thrilled. He now could see a complete story of how qualitative and quantitative research can work together to drive UX improvements and business results.

1. Qualitative research uncovers an insight;
2. Analytics data quantifies the insight;
3. The quantified insight inspires a design change;
4. A/B testing validates the design change.

What if we had instead dismissed the executive’s initial skepticism around qualitative research? What if we had dug in on qualitative research? The story would have ended with Step 1.

Without quant backing, the VP would have pushed back on the design change. Frustrated and demotivated, the team would have moved onto an easier battle. And the company would have left millions in revenue on the table from this missed opportunity alone.

The long-term loss would have been much greater.

What nobody knew at the time was that this one project and story would set in motion a series of events that led to an organizational transformation. It went like this:

• The marketing VP shared the story and early results with his bosses and executive peers, paving the way for more UX investment in the marketplace.
• The team turned the 4-step qual/quant process into a habit, developing a recurring research program.
• The VP rallied the team around the mantra “evolution not revolution.” Their model: place small bets based on research, expand the winners, toss the losers, and repeat.
• Within a year, marketplace leads tripled. A year later, they doubled. Two years later, they doubled again. Over the course of 7 years, leads increased by nearly 2000%.

The product and process became a huge success story across the 4,000-person company with over 20 brands and business units. The marketing VP became the company’s most vocal cheerleader for qualitative user research, helping to spread user-centered design to new business units.

In one meeting, executives from a different business unit invited the VP and asked for his advice as they prepared to launch a new product. As they peppered him with questions, he returned to the same response, eventually saying:

“You just have to watch your customers use the product. I don’t know what else to tell you. You will be blown away.”

A New Way to Look at Requests for Bigger Data

Every week or so, we facilitate some kind of research workshop with a new team. The details are always different, but the starting point is the same: the team spends 2–3 hours watching in-depth qualitative user research.

Often there are qual newbies in the workshop — developers or product managers or marketers who have never watched a moderated usability test or a user interview. Over and over, we see this immersive process erase skepticism around small-scale research. After the workshop, we’ll hear things like “I didn’t think that 6 users would be enough to see clear patterns, but yeah, it was — I get it now.”

When it comes to convincing stakeholders that 5 or 10 participants are enough, there’s no substitute for getting them to sit down and watch a few full sessions.

And yet, executives remain the toughest sell. It’s much harder to get them in the workshop in the first place. When we do, it’s hard to get them to stay put for a morning. And even then, there’s often a remaining hurdle.

As with the marketing VP in our story above, many executives are surrounded by big-data metrics and dashboards every day; their bosses give them KPIs or OKRs, and challenge their resource requests by asking for data.

Even when they watch mind-blowing qual research, executives often need quantitative backing before they sign off on design changes or UX initiatives.

For user researchers, this can be frustrating. But if we view these requests as an opportunity — to partner with new colleagues, to marry lab findings and real-world data, to win over critical stakeholders — the payoff can be huge. The once-skeptical executive might become the company’s most influential champion for user research, allowing it to scale in ways we never imagined.

And the biggest winner, along with users and business results, will be the messy, qualitative research that we love.

Can mess-loving UX researchers win over metrics-loving executives? was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

User researchers love to hate website surveys, and justifiably so. Poorly-designed surveys are everywhere on the web, in part because tools make it easy to throw one up with lots of questions. Most website surveys annoy users and fail to capture data that results in UX improvements. At best, some of the data ends up in a presentation or dashboard.

But done right, web surveys can have real UX impact. On a recent project, we ran 2 short surveys that achieved actionable insights at a minimal cost to the user experience.

The setting was a usability study for a major regional health system that includes one of the top-ranked hospitals and medical schools in the U.S. Our primary research was moderated usability testing with 7 participants. To supplement our qualitative research, we gathered data from 2 large-scale website surveys.

Here’s what we did and what we learned.

Step #1: Pilot, Iterate & Launch

Every website is different, as is every website audience. We started by piloting a few surveys that we’ve found to be effective at both measuring UX and collecting qualitative responses on similar websites. We then tweaked the question wording and display criteria until we saw a strong response rate on this site. We settled on these 2 surveys.

Ease-of-Use Survey

This survey starts with a simple, multiple-choice ease-of-use question. For those who select “Difficult”, we ask an open-ended “why” follow-up. We served this to users as a popup after they’d visited 4 pages on desktop or 3 pages on mobile.

Task Survey

Here we flip things and start with an open-ended task question, and follow it with a multiple-choice question. We served this to users as they were about to leave the site — an exit-intent survey. (We used Qualaroo to run our surveys because it’s one of the few tools we’ve found with good exit-intent technology.)

We like these surveys because:

• They’re short — a max of 2 questions each — helping to maximize the response rate and minimize user annoyance. In both surveys, we still get useful data even if some users don’t answer the second question.
• They give us both quantitative and qualitative data.
• In combination, we get a good picture of both common tasks and top frustration reasons, and the relationship between the two.

Step #2: Analyze

Once we received 600 responses, we turned the surveys off and started our analysis. To analyze the results, we:

• Gathered all survey responses in a single spreadsheet
• Calculated some ease-of-use metrics
• Filtered the list to responses with likely task failure or frustration
• Manually categorized the task failure and frustration reasons
• Counted the responses within each of those categories

Here were our top findings:

• The #1 task area causing failure/frustration was Medical Records, in particular the task of trying to access or log into that part of the site. Example comments: “Hard getting the activation code”, “Can’t get on [records section]”, “Total waste of time”, and “almost impossible to get into.”
• The #2 task area causing failure/frustration was Search, in particular the task of searching for a doctor. Example comments: “circle around to the same information or dead end links” and “not able to search for a doctor and see their patients feedback.”
• The #3 task area causing failure/frustration was Contact Info & Appointments. Example comments: “cant find email addresses for physicians or their admins”, “where is email for [name]?”, “Patients should be able to communicate with Doctor through email”, and “Can I make an appointment online?”

The surveys also showed that 46% of the respondents found the site easy to use; 22% found it difficult to use. By itself, this was the least useful finding. But it gave us a baseline to compare the site against peer/competitor sites and against future versions. And it allowed the team to set a measurable target that motivates action. Can we make usability improvements that increase the “Easy” share from 46% to over 60% by next year?

Step #3: Share

We shared these survey findings at a stakeholder workshop, just after the health system’s marketing team had watched 5 of the moderated usability sessions. After sharing the top frustration/failure areas, we showed quotes from each area similar to the ones above.

While the biggest insights on this project came from qualitative testing, the survey allowed the team to:

• Establish and understand their benchmark usability score
• Validate/invalidate/quantify their qualitative findings
• Identify new UX issues that did not come up in moderated testing

The Surprising Insight

While surveys often help us find small UX issues that qualitative research failed to surface, it’s rare that we uncover major new issues. On this project, we did.

At the start of the project, the marketing team said that the patient records section of the site was out of scope. A different team owns that section of the site and it’s a hard section to change, we were told.

So we excluded records-related tasks from our moderated testing, and ensured that users in the records section of the site did not get the survey popup.

Despite its exclusion, survey data showed Patient Records as the #1 website frustration area, by far. How was this possible? Because the specific problem was access: many users could not even get into that part of the site.

After the workshop, the attendees shared this survey finding with the team that owns patient records. They included powerful user quotes that supported the finding. It worked: motivated by this surprising insight, a project is underway to improve this critical website flow.

Keys to Survey Success

Here are some lessons learned from this project and similar projects we’ve led:

1. Keep surveys super-short — 2 or 3 questions at most.

Long surveys lead to low response rates and less representative data. As Jakob Nielsen says, “you can’t trust [survey] data if it doesn’t represent your users.”

2. Include an open-ended question.

It’s tempting to think you know the top possible answers to your “why” question or task question — and to give people a list of responses to choose from. It certainly makes the analysis cleaner and easier. Avoid the temptation, at least for the first round. You’ll often be surprised at the responses, and there is huge value in hearing about problems in your users’ own words.

3. Pilot and iterate before you launch.

Because it’s so important to get a good response rate, and because every site is different, always test and iterate before launching the final survey.

4. Focus on user pain during analysis.

User frustration, problems, task failures. That’s where the biggest insights and opportunities come from.

5. Quantify qualitative survey data by categorizing and counting.

Categorizing open-ended responses is tedious, but it allows you to count, compare, and quantify insights.

6. Share survey data and insights after showing qualitative research.

Surveys should almost always be a supporting act. Once your stakeholders have qualitative observations in mind, survey data can help to validate or supplement those findings.

Follow these tips for your next website survey, and you’ll increase your chances of generating surprising insights and a better UX.

How to launch a website survey that drives UX insights and action was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

For our agency’s first 5 years, we used the traditional approach to user research: teams hired us, then we disappeared to conduct research and analysis, and returned with a big report.

Over and over, our research failed to drive change. No matter how clear the problems and opportunities were to us, teams nodded politely — and then largely ignored our findings and recommendations.

Frustrated by this lack of impact, about 6 years ago we began to experiment with more collaborative approaches to research. We started by adopting Jared Spool’s variation of the KJ-Method, where we brought diverse teams into the observation and analysis process. Immediately, we saw a difference: teams that previously ignored research began to internalize it, act on it, and ask for more.

Since then, we’ve borrowed, adapted, and mashed up many other collaborative techniques, including those from design thinking, the Google Ventures Design Sprint, the Lean Startup, and Gamestorming.

While we are not done experimenting, we can confidently say this: if your goal is to drive human-centered change, team sport research blows away traditional expert-led research.

And yet, based on what we hear and see from clients, we estimate that 90–95% of consultants and in-house teams use the traditional approach to user research. Their impact lives or dies with their report or presentation. And most often, it dies — no matter how great the research and analysis — because other teams were not part of the process.

The result: most research is failing to drive the positive changes that it could — whether it’s an organizational transformation, a new product strategy, or a small usability improvement.

Ready to try switching from traditional research to a more collaborative approach? Based on what we’ve learned during hundreds of projects, here are 6 key differences between the approaches, along with specific tactics you can adopt to make the switch.

How to Switch from Traditional Research to Team Research

1. Depth of Observation

Traditional Approach: Researchers conduct the research and share highlights and clips in a report and debrief. Result: Stakeholders gain a superficial understanding of users and their needs.

Team Approach: The stakeholder team observes significant amounts of raw user research directly and in depth. Result: Stakeholders develop a deep understanding of users and their needs.

2. Stakeholder Diversity

Traditional Approach: The team that watches extended research is limited, often to just researchers. Result: Development, product and design teams fail to align on many critical decisions.

Team Approach: Non-researchers, including designers, developers, product owners, and management, are part of the team gaining in-depth exposure to users. Result: Teams develop a shared understanding about users and reach consensus on key decisions more often.

3. Frequency of Observation

Traditional Approach: Most stakeholders’ research exposure is limited to large, ad hoc, and infrequent studies — twice a year at best. Result: It is easy to ignore persistent problems and forget about user needs.

Team Approach: Over time the diverse stakeholder teams observe in-depth user research at least once every 6 weeks. Result: It is hard to ignore persistent problems — and easy to keep an accurate picture of users in mind during design decisions.

4. Problem Analysis

Traditional Approach: Researchers analyze and synthesize the research and share their problem findings in a report and debrief. Result: Stakeholders often hear what they want to hear, and dismiss findings that contradict their assumptions. Disagreement about top problems persists.

Team Approach: Researchers teach stakeholders design thinking techniques and facilitate a structured process to analyze and synthesize problem findings as a team. Result: False assumptions die quickly, and the team gets on the same page about the biggest problems worth solving.

5. Solution Generation

Traditional Approach: Researchers brainstorm ideas and recommend solutions in a report or debrief — sometimes supported by sketches or wireframes. Result: Stakeholders dismiss solutions they don’t like — often rightly so. But because they lack the research context and/or design skills, they struggle to generate alternative solutions that move the needle.

Team Approach: Researchers lead stakeholders through a step-by-step process to generate, sketch, combine, and critique ideas — both individually and in small teams. Result: Because they generated the solutions themselves, teams are more likely to implement them. And because those solutions embraced design thinking and the diverse knowledge of the team, they succeed more often.

6. Researcher Role & Value

Traditional Approach: Researchers focus on 1) analyzing research, 2) designing solutions, 3) and presenting reports. Their value lies in UX expertise plus their ability to persuade others. Result: Relationships are often tense. Partner teams “just don’t get” researchers’ advice; researchers “just don’t get” the team’s business/IT/design constraints.

Team Approach: Researchers focus on 1) exposing teams to users, 2) teaching research methods and design thinking, and 3) creating a habit of user research. Their value lies in facilitation, empowerment, and program-building. Result: Because the partner team is an active player throughout the process, both sides work through disagreements in real time, with data. Relationships are more harmonious and productive.

From Telling to Teaching

When we bring these 6 keys together, we get this single, guiding principle statement:

For a user research program to be effective and impactful, it must empower a diverse stakeholder group to frequently observe users directly and in depth, and to analyze and solve problems collaboratively — supported by expert facilitation.

The team approach to research isn’t a silver bullet. It has plenty of challenges, such as getting partner teams to invest the necessary time to observe research and teaching engineers to ask open-ended interview questions.

But in our experience, these are solvable challenges in many organizations, especially when you start small and practice patience. As 1 or 2 teams experience the power of collaborative user research, they spread the word to other teams, and momentum builds.

Over time, instead of telling teams about their users, you’ll be teaching teams how to conduct their own user research. And eventually, your impact as a researcher will grow in ways you never thought was possible when your main deliverable was a report.

Are teams ignoring your user research? 6 keys to increasing impact was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.