A few weeks back I was invited to guest-author a post on Chris Chapman’s Quant UX Blog. Chris is a highly respected member of the quant UXR community so I was thrilled to be asked. Below is a summary of that content, but please visit the blog here for the full story.

In the data-obsessed world we live in, companies are constantly reaching for numbers as the foundation for improving the performance of their products and services. A commitment to gathering and analyzing user data is a critical component for building an understanding of how, and whether, we’re meeting customer needs. Harnessing this information allows teams to move beyond assumptions, identify pain points, and strategically design solutions that enhance usability, satisfaction, and ultimately, business outcomes. We know in UX research that numbers are often only part of the story.

• Starting with qualitative research before quantitative studies ensures you’re measuring the right things and that metrics accurately reflect real-world behaviors
• A loan servicing company discovered their call center metrics were misleading: agents marked 80%+ of calls as “make a payment” when many were actually customers checking on payment status due to system delays
• This inaccurate data led to wasted investment in improving an already-functional online payment system instead of addressing the actual issues: payment processing delays and poor messaging
• Qualitative research revealed agents had no appropriate category to describe calls where reassuring customers was the primary focus, forcing them to select “make a payment” as the closest option
• Leading with qual research would have let the team identify blind spots, refine data collection instruments, and ensure the quantitative categories matched reality before large-scale implementation
• The decision of which method to use first depends on team confidence in understanding the problem, timing constraints, and the risk of collecting inaccurate data
• Secondary datasets can sometimes provide useful context, but direct qualitative observation often reveals critical insights that metrics alone cannot capture
• Rotating between qual and quant methods creates a powerful research approach, with each method strengthening the other when properly sequenced

Do you remember copy machines?

Of course you do. They’re not like rotary phones…yet. But anyways, copy machines take an original paper document and, you know, make a copy. If you’ve ever seen what happens when you make a copy of a copy, and then make a copy of that copy of a copy, etc, you end up with something that has no meaning. In the case of a copy machine it’s because you can’t read it anymore due to the amplification of all the tiny errors and the blurry edges introduced by the process.

This is similar to, although less complicated than, the problem of using synthetic data in research.

Synthetic Data is Never New

What you are doing when you use synthetic data, whether in quantitative or qualitative work, is you take an original data source (and I need a big asterisk on the word “original” due to the way AI models are sourcing their content) and you duplicate it. A lot. And then you take all that duplicated data and treat it as an original datasource. So you amplify whatever was said in that first dataset. And if there was anything wrong with that original dataset, such as poor sampling or various forms of bias, then you’ve amplified those problems. And you won’t see any conflicting data that might notify you of that because you are not collecting any new data, you’re just regurgitating the old data. I’ll highlight that for you:

“You are not collecting any new data, you’re just regurgitating the old data.”

You’re Locking in Errors

Thinking out several years I begin to wonder what would happen with continual heavy usage of synthetic data.

As with the copy machine, with used and reused and recycled and reprocessed synthetic user data you will essentially be stripping out any original meaning over time. Proponents of synthetic data will now be saying to me “but our synthetic data is not made from a single original.” No, but generating a document from a collection of similar documents is still locking in the errors and assumptions of those multiple originals. (Also, the copy machine is a metaphor, not an accurate technical point of comparison.)

What I’m saying is, we will always need fresh data. We can duplicate the data we have but it will never tell us anything new. And in fact it might blur and consolidate the image so much that it loses all useful meaning.

Synthetic Data Only Blurs the Picture was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

The voice was saccharin. Oozing obsequiousness.

And then, my friends, she giggled.

Or, “it” giggled, I suppose.

I’m referring to the AI moderator being demo’d in a new qualitative research tool. One of a slew of “AI-powered offerings” emerging in the product and customer experience space, this tool (pun intended) was intended to free up trained researchers from the time-consuming task of actually talking to people by following a pre-prepared script and leading multiple participant interviews simultaneously. Or, if not trained researchers then whoever it was within a company that wanted to collect descriptive feedback about their product without bothering (hiring/paying for) trained researchers.

The idea itself can be understandably threatening to people who typically do research as part of their job, but I do agree that things change and sometimes that means jobs change. So I was curious to see how good this AI moderator would be. Honestly, getting qualitative data quickly and at scale is pretty exciting. Finally we could halt those “but how big was your sample size” questions in their tracks. I decided I would watch the demo with curiosity and an open mind.

I was not prepared to feel myself filled with quiet rage.

An AI-engineer’s perspective of qualitative research

The moderator started off with what most researchers would consider a very boiler-plate intro. Standard stuff you inform your participant about. The tone was definitely tuned to what was likely described in the product requirements doc as “friendly and approachable.” It felt a bit phooney, but I assumed my ear would acclimate. Then she (the voice was female) asked for the participant’s name, explaining that she likes to know who she’s talking to, and then, inexplicably, gave a soft little giggle. As in “I like to know who I’m talking to, tee-hee.”

I’m sorry, what?

What is this, flirting? Was this a juvenile attempt at building rapport? Was it deflecting possible aggression?

And why does that make me so angry?

Well, it makes me angry because it was designed, and the intent behind this design was an interpretation of what the creator’s thought about what I, and many other trained professional researchers, do for a living.

It makes be angry because it was designed

Do they think that’s how we operate? That we seek to simulate a bond through spritely intonation and soft platitudes, our voices gooey as we coo at our conversational partner?

No, not “conversational partner” because THIS IS NOT A DATING APP. It is an established data collection approach, complete with theory and methodological rigor.

The giggles (there were more to come) and the unctuous utterances take a serious endeavor capable of generating deep understanding and strategic foresight and transform it into a performative toy. I watched the participant play along, but to my view that’s all they were doing. They were gamely tolerating the newest thing.

That the product team responsible for this thought they’d solved the need for a human being by making something performatively agreeable says more about them then it does about the job it was intended for. The sycophancy present in most generative models today might be more feature than bug. And that should feel insulting to everyone.

There is another way

A trained moderator is not artificially sweet. They adjust to the participant. Some participants need more encouragement, and some react poorly to repeatedly being told things like “great job, thanks for hanging in there with me!” A good researcher also knows when a participant is getting fatigued and can slow down the pace or skip topics that are more challenging — it’s about getting good data, not finishing the script.

It’s about getting good data, not finishing the script

On that note, sometimes a topic becomes less relevant as a series of interviews progresses. Possibly something that was expected to be an issue really isn’t, and so the researcher scraps it from the guide and adds in something that didn’t make the first cut. If all your interviews are happening simultaneously, or nearly that, you may get lots of not-so-useful input on that non-issue. The responsive nature of qualitative interviews are one of their biggest strengths. An emergent technique is highly valuable for, actually, saving time.

In projects requiring deep dives, spontaneous exploration, or the ability to adapt to unexpected responses, human moderation remains critical. But there are other interesting applications for AI moderation. For example, where an exploratory approach is unnecessary, a robust set of questions is well-defined, and participants are unlikely to deviate significantly from the intended topics, an interview bot can efficiently bring in a high volume of descriptive (qualitative) data. A product feedback session focused on specific features is a likely application. But a simpering voice and adolescent tittering suits none of that.

The Giggle was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

Younger workers are less engaged in the post-pandemic workplace than their more experienced peers. They need more support with workplace norms, especially in hybrid environments. Structured programs for intergenerational collaboration are essential, as younger workers expect companies to facilitate these connections. Physical space and technology are also vital pieces of this puzzle. Spaces should encourage interaction, and tech tools should enable seamless collaboration, especially in hybrid work. A comprehensive strategy encompassing support programs, space design, and technology is key to engaging and retaining younger talent.

_______

The Emerging Workforce Needs a New Approach

The modern workplace is facing new challenges in engaging and supporting its younger workforce. Generational differences in communication styles, expectations, and work preferences have become increasingly apparent, particularly in the wake of the pandemic. Let’s cover some of the specific needs of younger employees and examine the ways in which organizations can better support their development and integration into the professional environment.

First, younger, less experienced workers need more support in learning workplace norms and standards. This has always been true but is especially important in today’s work environment, where remote or hybrid work schedules can make it difficult for new employees to connect with their colleagues and learn the unwritten rules of the office. Furthermore, a recent Gallup poll found that Millennials and GenZers reported the steepest declines in various elements of engagement. They may not be getting the support they need from their managers and colleagues, and they may be feeling isolated and disconnected as a result. This can lead to a number of problems, including lower productivity, higher turnover, and slower career growth.

The Need for Structured Connection

One way to help younger workers is to provide them with more opportunities to connect with their colleagues. This can be done through regular team meetings, social events, and mentoring programs. Younger workers also need more opportunities to learn about workplace norms and standards. This can be done through training programs, workshops, and shadowing experiences.

By providing younger workers with the support they need, organizations can help them to succeed in the workplace. This will lead to a more engaged and productive workforce, and it will help organizations to retain their top talent.

As Gallup stated, “Post-pandemic engagement trends vary widely across age generations, with younger employees feeling the most detached from their work and employers but baby boomers remaining engaged.” This is a clear indication that younger workers need more support in the workplace. Organizations that want to attract and retain top talent need to invest in their younger employees.

Additionally, research from Fortune highlights a key disconnect in expectations. “Although most Gen Zers surveyed reported that communicating with their wider team would improve their productivity and learning, 64% are waiting for their company to do more to encourage intergenerational collaboration.” This indicates that while younger workers recognize the value of interaction and knowledge sharing, they are not proactively seeking it out on their own. They expect their organizations to take the lead in facilitating these connections.

Highlighting the importance of this, Forbes notes that, “The connection gap experienced by employees in the post-pandemic workplace is no longer a personal issue; it is an organizational health challenge.” This underscores the importance of fostering networking and connections, particularly since employees may feel like strangers in the current work environment, leading to trust issues. Encouraging connections between colleagues is not just about individual comfort but about the overall health and functioning of the organization.

There is a clear need for companies to develop and implement structured programs that foster intergenerational collaboration. Simply hoping that these connections will form organically is insufficient. Organizations must actively create opportunities for younger employees to interact with and learn from their more experienced colleagues. This may involve formal mentorship programs, cross-functional project teams, or even informal social gatherings designed to bridge generational gaps.

Work Spaces Play a Role

However, the success of these initiatives is not solely dependent on programs and policies. The physical environment plays an equally critical role in enabling these interactions and achieving the broader organizational goals of fostering collaboration, learning, and a sense of belonging. As David Dewane CXO, Space at the global experience consultancy geniant notes:

“Workplace design isn’t just about aesthetics; it’s foundational in nurturing meaningful connections, enhancing collaboration, and cultivating a genuine sense of belonging — especially for emerging talent.” — David Dewane, CXO Space, geniant

Without a thoughtful approach to workplace design, even the most well-intentioned programs can fall short of creating a truly connected and supportive environment for younger employees.

Physical spaces can either facilitate or hinder the connections that are so crucial for engaging younger employees and fostering a healthy organizational culture. For example, open-plan layouts can encourage interaction and spontaneous conversations, with designated collaboration areas providing spaces for team projects and mentoring sessions, all while allowing for dedicated focus time in thoughtfully designed quiet areas. Conversely, isolating workspaces or a lack of common areas can exacerbate feelings of detachment and hinder the development of relationships. Designing spaces that accommodate flexibility, community, and interaction is essential for supporting the needs of a diverse workforce and addressing the ‘connection gap’ highlighted by Forbes. The physical environment should actively support the organizational goals of fostering collaboration, learning, and a sense of belonging.

The Importance of Supportive Technology

There is one final element to consider. Technology plays a crucial role in facilitating communication and collaboration between younger workers and their colleagues, particularly in a hybrid work environment. Younger employees, who are more accustomed to using technology in all aspects of their lives, expect their workplaces to provide them with the latest tools and resources to enable efficient and seamless remote work. This includes access to video conferencing platforms, instant messaging apps, cloud-based collaboration tools, and other technologies that facilitate real-time communication and knowledge sharing.

CBRE, a leading commercial real estate services and investment firm, has highlighted this need, reporting that “A notable 70% [of hybrid programs] prioritize investment in technology tools, ensuring that employees have the necessary resources to facilitate seamless collaboration.” These tools also include room booking and scheduling technology that integrates with existing platforms. By investing in workplace technology improvements, organizations can empower younger workers to thrive in a hybrid work environment, fostering a more connected, engaged, and productive workforce.

A Cohesive Approach

To effectively support and engage younger employees in the modern workplace a cohesive strategy that encompasses policies and support programs, space design, and technology is essential. Policies and support programs should focus on providing opportunities for learning, connecting with colleagues, and navigating the unique challenges of the modern work environment. Space design should prioritize flexibility, community, and interaction, creating physical environments that facilitate connections and foster a sense of belonging, while still allowing for “get away” spaces that facilitate deep work. And finally, technology plays a pivotal role by providing younger workers with the tools and resources they need for efficient hybrid or remote work and seamless collaboration. By integrating these elements into a comprehensive strategy organizations can create a more inclusive and supportive work environment that meets the growing needs of younger and newer employees, leading to increased engagement, productivity, and retention of this cohort.

Researchers are at a distinct disadvantage in the UX job market. Why? Because companies hire people to get things done. And researchers spend most of their time talking about the things other people do.

In a job interview researchers might talk a lot about how they did the research that led them to learning about the things other people do, but this is still off the mark. What did the researcher do for the company?

Yes, yes, we are advocates for the users and customers and members and all the other types of people who we research. But we can’t ignore that we’re doing this research for a business. And at the end of the day a hiring company needs to know why our research matters.

Last year I spent several months volunteering alongside other UX Researchers doing mock interviews and providing resume guidance for UXRs who’d been laid off. I’ve also served on numerous interview panels when I worked at Google. I have interviewed a lot of UXRs and the ones who stand out are the ones who know how to talk about their impact.

How to talk about Impact

When I ask most researchers about the impact of a project they often say something like, “I concluded the research with a presentation that was later shared with senior-level leadership.” Or, “The research resulted in several new journey maps.” Or even, “The team took my advice.”

• Your impact is not your deliverables.
• Your impact is not your recommendations.
• Your impact is not that a team did the thing they were probably going to do anyway and that your research generally agreed with.

Those aren’t impact. The last one gets close (sometimes) but it’s still not really talking about how the work you did impacted the company. Because it doesn’t talk about change. Change in a company largely shows up in the form of increasing or decreasing.

I am aware that researchers, especially qualitative researchers, get a little nervous when asked to add numbers to their resumes and talking points. You don’t necessarily need to have numbers (although doing so can catch the eye of people who scan resumes quickly). You do however have to think a little more deeply about the chain of actions coming before and after your research.

Let’s fix those example statement so that the impact they represent is clear.

What does it mean to the company that your presentation was shared with senior-level leadership?

It might mean that you helped boost the visibility of the research function, leading to an increased frequency of research requests from higher-level stakeholders, and a new standing invitation for your manager to join the products team’s monthly update meeting. You changed the way the company interacted with the research team.

What does it mean to the company that your research produced journey maps?

Perhaps it meant that the company now had a more nuanced understanding of the many steps involved with learning how to use their product, and this new understanding led them to create a new onboarding program. You changed the way the company brings users into the product. Bonus points if you know how much the abandonment rates went down after this program began, or the resulting increase in revenue.

What does it mean that the team followed your recommendation?

This one sounds kind of good already. The team took your advice. That’s a start, but the team believing in your results does not help you stand out as a job candidate. Pull back a little further in time. How long had the team been debating about this feature change? What prompted the discussion? Without your research, how would the decision have been made? It’s possible that your impact here is around increasing the confidence of a decision that was holding back other work. You changed the velocity of the team by unblocking their decision making.

Seeing your impact

Clearly, if your project did not lead to these changes, and you lack the supporting evidence I outline above, don’t make these claims. But I hope as examples they help you think more profoundly about what changes might have come from your work, and the impact that made on the company.

To see your impact requires lifting your head up out of the nitty-gritty detail we tend to love as researchers. You need to think about the signals you get from how your research is requested, how it’s received, how it’s shared, what decisions come from it, and what happens next.

Not every project will result in a strong impact story. It’s not realistic to expect it to. That’s why you curate which projects to highlight in your resume or talk about in an interview. Review your project history in your mind and think about the work that makes you feel good. Why do those projects stand out? You probably have some hidden intuition about the impact of this work that’s making you reflect on it so positively. As an astute researcher you’ve picked up on a signal. Follow it.

Addendum

Are you thinking: Thanks for all this Kelly, but I’ve been bounced from job to job for a couple years now and I have zero insight into any of the changes you described. As for numbers? Ha!

I am so sorry.

This is a factor in the research job hunt today. Don’t despair. (Or, despair a little less.) What you do instead is turn the discussion from organizational impact to your proficiency as a researcher. Your flexibility in research approach, your ability to adjust to new project circumstances, your creativity in recruiting a nearly-impossible sample, something about how you turn a bad situation into a better one. You say something like,

“Because I’ve had the opportunity to work with a lot of organizations it’s difficult to track impact, but I’ve really had the chance to flex my adaptability muscles. For example…”

In short: be straight forward about your reality, and shift the conversation toward one of your strengths. Talk about why the team brought you into that project and how you handled each challenge that came up. Similar to having an impact story, an effectiveness story can set you apart from candidates who talk about how they watched people do things and then created the assigned deliverables that reflect that. Because the company needs to know what you’ll do for them.

I hope this helps. Keep at it!

Researchers: Let’s talk about Impact was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quite a few years ago I worked with a team that was helping a client better understand the emerging “green” environmental movement. Their product line, unrelated to an emerging earth-friendly wave of concern over waste and unnecessary consumption, was stale and they were seeking ways to differentiate. These new greenies, as they called them, were a group they wanted to understand better. The research team was tasked with probing into question such as:

• What motivated their commitment to being green?
• How did being green manifest in their day-to-day lives?
• What criteria did they have when choosing green products or services?

We tackled this project with a combination of photo diaries and in-context interviews. We dropped off disposable cameras (back then a much more common product) with instructions to spend the week photographing the green things in their lives. At the end of the week we collected the cameras and spent time as a team reviewing the photos. We looked for similarities, proposed some themes, and developed our interview guide.

One theme that came up in the photos was the inclusion of the 7th Generation brand of cleaning products. The brand promises its users to help “transform the world into a healthy, sustainable, and equitable place for the next seven generations.” We were intrigued by how often we saw these products pictured in the diaries and were interested in hearing more from users.

Context Matters

Back in our participant’s homes we went over the photos and asked them to tell us more about them. Then we’d ask for a tour to be shown where these items were and how they were used. One participant provided an eye-opening demonstration of why context matters.

When talking to a participant I’ll call Sam, we asked why she used 7th Generation brand cleaning products. Sam very seriously explained to us that her responsibility as a mother was to raise her children in a home clean and free from dangerous chemicals and questionable ingredients. Her children were young, and touched or put their mouths on any and every surface. Standard cleaning products could leave a residue they might ingest. To top it all off, these products are detrimental to the larger environment. We asked if she used any other cleaning products, and she maintained that she did not.

Sam’s commitment to her children’s health and safety, as well as in doing whatever she could to ensure they had a lasting world to live in, was clear. She followed many other green practices in compliance with this ethos.

After the photo share it was time for the tour. When Sam got to the kitchen she opened the cabinet below the sink to point out a collection of 7th Generation products. What was sitting right next to them was interesting.

A large gallon jug of bleach.

Not typically considered a “green” product, we asked about it.

“Oh, that’s for the really big cleaning jobs.”

Aspirational Selves

At this point you might decide that Sam lied to us. We outright asked about other cleaning products, and she outright told us there were none.

Stories like this are common when product teams talk about why they’re not doing research, or why they only do certain very limited types of information gathering. “Users lie.” It’s been the topic of articles, talks, and podcasts.

I prefer to reframe this. It’s not that users lie when you ask them questions about what they do or what they want. They are telling you about their aspirational selves.

Everyone has values they want to uphold and impressions we hope to make. Many of us struggle to meet these. But if you ask people about their wants and needs, and especially if you ask them about their future behaviors, they will almost always tie these to those desired values and impressions. These are their aspirational selves talking to you.

Aspirations can be helpful to know about, particularly when you need to understand the value proposition of your product. But for understanding usage you need to get to true behaviors and real needs.

Meet Them Where They Are

By meeting users where they are you will learn vastly more about how they do things, where their pain points are, and what really delights them.

By “meet them where they are” I mean this both literally and figuratively.

In a literal sense, go meet them in their environment whenever this is feasible. Observe them in context and ask them to “show me” rather than just “tell me.” Make efforts to do this with your user base at least annually.

In the figurative sense I mean to meet users where they are by letting them lead the conversation a bit. You do this by listening. You should have learning objectives and you can absolutely set out the topics you want to cover in any interview or intercept, but your users or customers have interesting stories to tell if you allow them some space.

I sometimes refer to this as letting them show you other rooms in the house. Your product sits in one room, or at least you think it does, but there are hallways and rooms related to that room that are likely very influential to how your product fits in. Allow, and encourage, your users to draw you through these other areas.

You can make space for these occasionally wider conversations, and still collect the data you need for specific projects, by scheduling regular touch points with your user base. Monthly is a fair starting point, but I know teams who get a weekly cadence talking to just a couple users each time. Some interview questions are evergreen, allowing for deeper data collection, and some are new, broadening the understanding over time.

When users tell you things, and you can’t ask “show me,” instead ask them “can you tell me about the last time.”

Remember:

Users aren’t lying.

They’re telling you about their aspirational selves.

Asking them about a recent time helps to anchor them in true behavior.

Applying the Insights

Luckily our team was meeting users where they were by visiting them in home and asking about recent usage. We soon understood there was an aspiration among our target users that they would utilize eco-friendly products exclusively, but in reality they had concerns about efficacy. This meant we needed to make sure any resulting products stood up to their promises and did so in easily proven ways. We’d need to combat the suspicion that green products don’t quite cut it.

We learned much more by visiting in home — including using the photo diaries to ramp-up fast — than we would have if we’d just done phone interviews. Or worse, just sent out a survey. We learned about their aspirational selves and *knew* that’s what we were learning about, and we learned about how things really got done. There’s two things I hope you’ve learned:

• Get out there. Watch.
• Ask questions. Listen.

Now keep up the curiosity!

Why you can’t just ask was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stuck in a rut of evaluative research? Lift yourself out by elevating your insights

This article is for UX Researchers who are frustrated with an inability to participate in more strategic research opportunities.

*****

It happens. Your research roadmap has been stuffed full of round after round of evaluative testing. It’s valuable, and the team devours it with a satisfying enthusiasm, but you have the foresight to know that larger problems with the product loom ahead.

You fell in love with research for the chance it brings to dig deep into the things people do and how they structure their world, and you know this insight can make a positive impact on users or customers. You’ve likely noticed that while your tactical work is being used by the associated team it’s not making a larger or more strategic impact on the wider company.

You may have pitched some strategic work (you may call it foundational or generative). You may even have built out an alternate roadmap showing how several studies could build off one another and even feed into other product workstreams. But the immediate need is to see if the UI is understandable, or if the design resonates, or if the workflow…works. UX Researchers are typically a scarce resource and your time is essential to making measurable, tactical, impact in the product today. How do you get out of this rut?

Lift yourself out by looking up

One immediately implementable way is to expand your tactical research to include more exploratory questions. Budget in 10–15 additional minutes to your study plan, or remove a section that has had decreasing impact over time, to sit and talk more openly with your participants.

Lift your focus up and out of the product and let your participants tell you about the goals they have and how they meet them. You probably know how to do this already, you’re just more familiar with doing it in longer sessions. It can still be insightful in bite sized chunks, it just won’t (likely) lead to an entire lengthy presentation’s worth of content. But it will lead to a larger picture of the house your product sits in and to interesting previews of hallways your standard tactical research is too narrow to detect.

There are two places in your study where you can add in time for this higher-level exploration.

Peeking down the hallway

Tactical research is most often highly focused on one area, or let’s say “room,” of a much larger house. You structure your research sessions to delve into whether or not the room’s furniture is arranged in a way that makes sense. Or you’re assessing if the size of the room is adequate for the activities intended there. Etcetera. Quite frequently though you’ll get a sense from your users of a hallway off to one side. In the interest of time your general procedure would be to acknowledge this hallway with the participant and then redirect them to the room. This often looks like: “Oh, that’s an interesting point, thank you for sharing; do you think you’ve found the information you were looking for yet?”

Stop doing that.

Allow time in your protocol to document these hallways and then revisit at the end of the session. Gather just enough in the moment, it can take just a minute, by asking “tell me a little more about that,” and make a note to yourself for returning to it at the end of the session. Come back in the last 5 minutes and follow-up on the most promising hallways. You can say, “Thank you for your help understanding how X works for you. I want to go back to something you said earlier about Y.”

By keeping your eye out for these moments you may very well discover that a certain hallway comes up quite often among your participants.

Make it work for you:

In your study report give your stakeholders a peek down that hallway. Ask them what it would mean to the business if you went down that path in a larger, more purposeful study.

Note that this approach works best if you’re comfortable going off script and good at managing time during those initial hallway offshoots.

Talk about the house first

Another, more time efficient, approach is to add in an intentional section for exploring the larger concept or process your product sits within — the house, if you will. I personally like to add this to the beginning of the session, in the warm-up portion that is often unfortunately a bunch of fluff. Instead of asking about what users like to do in their down time, or other common warm ups, talk about the larger journey or the things that lead up to using the product. This has the added benefit of letting you situate the study tasks into a more relevant context for each user.

I hear you asking me, “But what about building rapport first? I ask about their hobbies so they know I’m interested in them.”

I don’t think that’s necessary. Well, the rapport part yes, but the artificial interest in hobbies, no. Reframe your idea of rapport building as more of an ongoing activity than as something you do in the first 2–5 minutes. Let’s look at an imaginary example.

You (after completing all necessary informed consent steps): Thanks again for coming in today to help us with the online movie ticket purchasing experience. To get started, can you tell me a bit about the last time you saw a movie in the theatre?

Participant: Sure. I guess it was last weekend. I saw Barbie.

You: And how did you like it?

Participant: I liked it, but my friend said it was silly.

You: So your friend was with you? Was anyone else there with the two of you?

Participant: Well, it ended up being just us.

You: Sounds like there were more people who were going to go before? Tell me more about that…

Here we have the chance to learn more about the movie selection process, how other people are involved, and how the experience is greater than just getting the tickets and watching the movie. We’re learning about the house.

Make it work for you:

In your final deliverable you can either report out on this section all at once as a way to give context to the product experience, or you can spread insights from it throughout the report as a way to add color and nuance to individual findings. If interest in this content is high, pitch a new strategic project that builds off of this piloted section with in-depth detail or by targeting a special segment of your target population.

Houses and hallways

Now that we’ve identified two moments in your study where you can add in more strategic research, how do you choose between them?

• Peeking down hallways will help you identify topics for new generative research. Report out on this as emerging areas of interest and get your team’s take on what knowing more about these will do for their decision making in today’s product.
• Surveying the house as a whole lets you identify the larger context of your product. This can help the team identify a strategic direction for marketing or for building out a fuller offering.

These are both useful. Sometimes choosing between them is a matter of where the larger knowledge gap is amongst the team. Do they already know the larger context pretty well? Do they have a detailed journeymap of where users leave and reenter the product? Have these been recently updated or re-validated? Choose the approach that will open up the largest possible window in their view of the user experience.

With just an extra or repurposed 10–15 minutes you can surface strategic questions with your tactical research and dramatically change the way the team views both the user experience, and you as a contributor.

Homer got a bad rap in UX. There’s a well-worn example using the episode where the inept Nuclear Safety Inspector and father of three is given the opportunity to innovate the car of the future.

I am speaking of course of Homer Simpson from the 35-year running animated sitcom, The Simpsons.

Homer ends up building a monstrosity of an automobile. Features are globbed on haphazardly, the passenger section is an entirely separate cabin, and the horn is an ear-splitting “awooga” noise. This endeavor ends with bankrupting the company and, spoiler alert, costs Homer his relationship with his long-lost brother.

The moral of this story is that users (aka regular people) aren’t great at thinking up product ideas. The resulting implication then is that we as creators of products should not bother asking users for their input.

This is only half true.

Users, in general, are not super great at coming up with usable, sustainable, innovative, or ultimately successful product ideas. But why should they be? That’s our job. What users are good at though, is knowing what it’s like to use the products we come up with.

Users don’t have your experience

Homer is not an engineer. What happens when you ask users what to build, and then take them at their word, is that you unfairly place the burden of developing your product onto their inexperienced shoulders.

Homer doesn’t have automotive product innovation experience. What he does have, are problems. His children are noisy and distracting, so he gave them a separate cabin. The team responsible for building The Homer needed to ask Homer about his need for fewer distractions. Or, on the flip side, his difficulty focusing on the task of driving. Those are the problems Homer was inelegantly trying to solve.

Users have their own experience

There are two moments in time where the engineers in this story could have done better. Both center on understanding the user’s experience before getting to the build stage. They could have done either, or both, of the following.

• Understand Homer’s current day experience. What is it like driving around in the car he owns right now?
• Dig deeper into Homer’s feature requests to discover what problem they solve. This is more than just asking “what problem does this solve?” although that’s not a bad first step.

Each of these moments can lead to detailed and nuanced information about user needs, goals, aspirations, and pain points; and yield eye-opening product insight. This data then feeds into your own industry-specific experience of creating a product.

The Homer story, while fictional, demonstrates an important lesson:

Use your experience to translate the user’s experience into better product design.

*****

Oddly, the car Homer designed is available in toy form from Hot Wheels, and one guy even built a working model. Like many things, it may just have been ahead of its time.

The Homer Simpson Effect: Asking users can lead to unusable products. Here’s why to do it anyway. was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is for any researcher who is ever worried about getting pushback based on their choice of method, or for managers or mentors who want to help those researchers reframe that experience.

*****

“Once they start asking methodology questions you know you’ve lost them!”

This was the lament I overheard at a recent gathering of UX researchers. The situation is common. A researcher prepares a beautiful deck of findings (hopefully complete with well-considered recommendations) only to be stopped at the first signal of user dissatisfaction because a stakeholder says they must have used the wrong method to get these answers. The sample size was too small, or the forced-choice responses didn’t allow for enough nuance, or the population wasn’t representative of the target market. Too often this is effectively the end of the presentation. But it doesn’t have to be.

Skepticism is a starting point for conversation

I wanted to shout over to this researcher, “Congratulations, you’ve got them right where you want them!” Skepticism is a great starting point for a conversation. And it shows a healthy curiosity among your partners and stakeholders.

First, every method has tradeoffs, so it’s important to not only recognize those yourself, but be able to explain them to non-researchers. Don’t hide behind the fact that your small usability study can not determine a fool-proof winner, or that your large scale survey can’t tell you why people quickly click out of a new feature. You selected the method after careful consideration of these tradeoffs. Lay them out for the team. What did this choice bring that you found valuable? It was worth your time. Tell them why.

Your stakeholder’s concern is your chance to engage them

Second, no research study should ever be done in isolation. You may not have the follow-up planned yet (well done if you do though!) but hearing your stakeholder’s concerns is your chance to engage them now in that next round. Be mindful here that you’re not offering a re-do — instead you want to discuss what a successful follow-up would add to the team’s emerging understanding.

Anyone who asks a tough question can become an evangelist for your research

Each question from a stakeholder is fodder for building your shared perspective of the user, the product, and how the two combine to make a worthwhile experience. Once you’ve had that discussion, you’re closer to securing your place as a go-to resource and winning that stakeholder’s trust. It’s like securing territory. Think of the goal as turning each person who asks you a tough question into an evangelist for your research.

The real way you know you’ve lost them is when they simply thank you for the presentation and leave the meeting.

Stakeholders questioning your methods? That’s a good thing! was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sunday, August 8, 2021

A little over a year ago I wrote about my first year working at Google as a senior-level UX Researcher. Today I reach the milestone of a thousand days.

A few key events:

• Teaming up on foundational work that led directly to the adoption of high level, multi-year strategic goals
• Incorporating research insights into design frameworks and product vision documentation
• Achieving proof of concept as I helped a proposed new product demonstrate its usefulness
• Successfully establishing the case to stop pursuing a product which did not meet existing user needs
• Seeing what I’ve worked on show up on the Search page (this is the best)
• Putting myself up for promotion (this is the worst)

Directly comparing where I am now, about a quarter shy of the 3-year mark, to where I was at a year it seems accurate to say that most elements of my experience have intensified.

The Atmosphere

Last time I talked about the vibrant and collaborative atmosphere at Google. In the interim we had a global pandemic. I think we did pretty well with how we handled things, going to voluntary work-from-home very early for example, but obviously my work atmosphere was massively impacted. This might be myopic but I do feel like UX research suffered the worst of it in many ways (in the tech industry I mean, not of all jobs possible). Talking to a participant face-to-face, handing them a testing device, experiencing their context — were all out of the question. This was tough to adjust to, but adjust we did. Since everyone else has had a pandemic too, I’ll leave rest of the impact of that out of the rest of this post.

The Work Itself

Keeping a manageable schedule is even harder than before. I support more teams across a broader space and lead an area covered by a few other researchers as well. I’m not a manager, more I take the strategic perspective that keeps our work cohesive. It means a lot of meetings, but I get to see a lot of progress.

Responsibilities

My prediction that I would be doing less actual research and more strategic planning as time went on was correct. Collaborating with other researchers across teams in service of this is wonderful, and I love that I am free to take part in these engagements. It’s just so. many. meetings.

That stuff in the news

This still happens. As long as tech is a gigantic industry there will be publicized missteps. One might ask for fewer missteps.

The promotion thing

I did not get it.

Promotion at Google is a tedious process, and they prefer that you kickstart it by nominating yourself during a regularly scheduled performance review. Then you build your case. A solid amount of arduous work is on the shoulders of your manager, but for the level I was aiming for I still had to document several impactful projects and write a short paper explaining why I was ready for promotion, complete with links to exhaustive quantities of evidence for each of my claims.

It was gut-wrenching having my application denied, but that’s a pretty common result. I’m up again soon — wish me luck!

Update 10/22/21: My second attempt at promotion was a success! I am thrilled to move into the role of Staff UX Researcher. (This probably means more meetings.) There are many perspectives on both the hiring and the promotion processes at Google, but from my perspective promotion was much more difficult than the initial hiring.

Overall

Last time I said, “Working at Google has been exciting, overwhelming, uplifting, defeating, and so, so, so much fun.” Same.

***

Kelly Moran utilizes an innate curiosity and unceasing desire to ask “why” to understand how people use products and services to accomplish their goals — whether those goals be work or play. Kelly was formerly Principal Experience Researcher at the Dallas-based consultancy projekt202, and now works as a Senior UXR at Google. Find her other writing at: https://medium.com/@Kel_Moran or follow her on twitter: https://twitter.com/Kel_Moran

1,000 Days at Google was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.