Researchers. What are they? What do they do? How many sides do they have?

You’ve come to the right place to get some answers. As a UX researcher, I’m going to show you why you absolutely need a someone like me in your design team, and how to best work with me to help you reach your goals.

What, exactly, is a ‘researcher’?

In this case, I’m talking about a role that is variously termed UX (User Experience) Research, Product Research, or Design Research.

At Atlassian, we tend to call ourselves UX Researchers. We work with cross-functional product teams, but we can — and do — conduct research with strategy, data science, and marketing teams. Speaking from my own experience and those of other researcher-types I talk to, this is typically the case in other organisations.

Researchers help us understand…

• Our customers (the people who use our products and pay for them)
• Our users (people who use our products but don’t pay for them)
• The people who don’t use our products…yet

…as well as how we can improve those product experiences.

Put simply: we conduct research with our customers, users, and people who aren’t our customers (yet), to understand their behaviours, needs, and problems.

As a cross-functional team, the insights gained by research help us understand what to build and how to build it, so we’re not only building the right things, but building them right. Research also helps us prioritise what problems to solve and when we should solve them, and separate what’s an essential need from what’s an added bonus.

Now, let’s talk about why you should embed a researcher in your design team.

Service model vs embedded researchers

The Research & Service Experience team at Atlassian has moved from a ‘Research as a Service’ model to an embedded model. Let’s look at this Service model first.

‘Research as a Service’ model: in and out of the swimming pool

In a ‘Research as a Service’ model, researchers all sit together in a pool, of sorts, for a larger product space. They’re allocated to projects within product teams based on demand.

It kind of looks like this:

This model works well when there aren’t enough researchers to go around: researchers can be allocated to projects based on business priority and the relative ambiguity of the problem space to be explored. You might also see this referred to as a ‘centralized’ model.

However, Research as a Service has its drawbacks:

• Because the researcher is jumping between projects and products, they may not have enough product knowledge, or context on the problem space, to conduct research effectively and deliver actionable insights that the product team can use.
• Because the embedded designers and product managers typically know the product better than the researcher, the researcher often ends up coaching them on how to conduct research for their product.
• Coaching designers and product managers works well up to a point. Speaking from my experiences at past organisations, most product designers struggle to conduct enough research on top of their day job, as do product managers.
• Another issue that crops up is that the research that gets done tends to be the more tactical stuff (such as validating that a particular concept solves a problem) rather than the more strategic stuff (uncovering new problems, solutions or opportunities to), simply due to the time constraints (and sometimes skill level) of the design team.

The embedded model attempts to resolve some of these issues. (As you may have guessed, I’m a big fan of the embedded model!)

Embedded Researchers: towelling off, home & dry

Wait, what does embedded mean?

In the embedded model, researchers are allocated to a product or problem space, and are treated as members of the design team, much like a product designer would be.

As with product designers and product managers, embedded researchers also retain their ‘home’ for the discipline, which focuses on things like craft sharing and the overall strategy.

When embedded, our researchers are now completely out of the swimming pool — towelled off, home and dry with their individual product teams.

Because the researchers are now embedded, we’re teammates, not consultants. This addresses the issues we looked at earlier in the ‘Research as a Service’ model, and changes the expectation of how we work with design teams:

• Because we remain in the one domain, we can develop specific domain or product expertise, and build enough context on the problem space in order to conduct research that’s really actionable.
• We can proactively contribute to business decisions, and provide understanding of users’ needs, problems, and opportunities at the right time to better inform those decisions.
• We’re able to build strong, trusted, relationships with our cross-functional partners. We partner with designers & product manager to deliver more research, more effectively (more on this later).
• We can work closer with our cross-functional partners to embed research insights into the product strategy or roadmap — the research outcomes becoming better-integrated. By being in the product team, the researcher understands deeply what insights are relevant to the roadmap and what’s feasible for the team to implement now versus later. This means product teams can actually deliver on outcomes from research!
• There’s reduced friction for the cross functional product team to participate in research, because we’re right there with you — regular involvement in sessions means we’re all building a greater understanding of our users over time, and able to make better decisions that impact those users.
• Similarly, research becomes more valuable because everyone in the team has a say in its planning, who the participants will be, and what we want to learn from them. The ownership of the research is shared, so everyone feels the need to act upon it.
• We get more strategic and tactical research done, together! This means we build better products and roadmaps that aren’t just validating what we think will work, but really getting under the hood of everything that we could solve — and all the possible solutions.

Embedded researchers and design teams: how we win together

Now that we’ve looked at the difference between centralized and embedded researchers, it’s clear that having researchers embedded provides benefits to both the researcher and the design team.

So, how can we best work together?

Make decisions together

Think of your embedded researcher as another member of your decision-making cohort. For your embedded researcher to be most effective, they need to be at the table when it comes to prioritising research, actioning that research, and deciding on the best way forward based on research insights.

You might be thinking ‘when do I bring an embedded researcher into the conversation?’ The beauty of embedded research is that it’s no longer about ‘bringing them in’ — they’re already there! Ideally, they should be present in all the meetings or discussions a product designer is in.

Big shout out to my teammates who demonstrated this understanding by inviting me to their Slack triad channels, weekly triad meetings, and ideation sessions the week I started! Having your researchers contribute in these forums means we incorporate customer insights at every decision point. In addition, because I know what’s going on, any research I conduct will be better contextualised and actionable.

Collaboration & partnership

As an embedded researcher, my philosophy is that research is a team sport. Embedded researchers work closely with their design teams in order to:

• Shape our quarterly and yearly plans
• Add a layer of insight to progress reports
• Prioritise what research needs to happen (and when)
• Shape research questions and hypotheses
• Create research approaches that will answer those questions in a valid, time-efficient way
• Get you involved in any research conducted (e.g. note-taking, facilitating, synthesis)
• Embed those insights into our product strategy/roadmap to inform where we’re going
• Develop a shared understanding of the people whose problems we’re solving.

Better solutions, stronger products, and happier teams

As I mentioned earlier, having an embedded researcher in your design team means you’ll get more research done — together, we can tackle both the longer term strategic stuff to identify new opportunities, and the shorter term tactical stuff to identify ideal solutions.

If you don’t have an embedded researcher, you’re missing out on those longer-term opportunities and problems to solve. You run the risk of having too narrow a focus into your product and organisation rather than getting the full picture of ‘what’s out there’.

Not having an embedded researcher is a bit like trying to get somewhere without a map — you might have a vague idea of how to do it, and you might be right — but its better to be sure!

But who does what research?

In the embedded model, the researcher usually takes the lead on conducting all forms of research (exploratory, strategic or evaluative). Time constraints, however, often mean that product design, product management and product marketing can, and do, conduct more tactical or smaller pieces of research with the guidance of the embedded researcher.

NB: These aren’t hard and fast rules, but generally how I’ve seen this model play out in organisations at which I’ve previously worked.

Going back to the partnership theme from earlier, I’d expect my design team to be super-involved in any research I conduct, and I’d expect any designer or PM conducting research to involve me. That involvement might include a peer review of the approach, data collection mechanism (e.g. moderation guide, survey), note-taking, or synthesis.

Partnerships = more brains = better research

In summary…

• Embedded researchers don’t swoop in, do some research, and throw it back over the fence.
• We’re with you every step of the way along the product development cycle.
• We’re with you as decisions are made, ensuring the insights we collect together play a part in shaping those decisions.
• We’re partners. And it’s going to be beautiful.

Yes, you should embed UX researchers into your design teams. Here’s why… was originally published in Designing Atlassian on Medium, where people are continuing the conversation by highlighting and responding to this story.

(Lost? Here is Part 1 and Part 2).

How we as a design research community talk about what we do — and how we may exclude others through language we use

Have you ever sat in a presentation and thought to yourself…what is this person talking about? They could be talking very quickly, so that it’s hard to understand what’s going on. Or they could be talking too slowly and you’re struggling to pay attention. Maybe they’re using a bunch of words you’re not sure of the meaning of, or words that you’ve not come across before.

Below is a (fictitious) paragraph of someone talking about design research:

“Last month I conducted an exploratory research study. The study comprised 20 contextual inquiry sessions, which are a form of ethnography typically used in anthropological studies at the tertiary education research level. Participant ephemera was collected in-situ to ensure validity and robustness”

Do you think this is reasonable in the context of…

• Two design researchers chatting over a coffee?
• A design researcher explaining their methodology to a General Manager of Product?
• Me telling my partner what I got up to at work yesterday?

You’re probably thinking that paragraph sounds a bit full-on. And you’d be right. It *might* be appropriate when talking to a bunch of seasoned design researchers, but it’s certainly not appropriate to use anywhere else.

Here’s another example. My ‘languate’ friend from earlier often talks to their partner about their work. I asked their partner how they respond to this.

“They’ll start rattling off a bunch of terms specific to their work that I don’t know, and it doesn’t click for them that I don’t know them. I find it really hard to know what they mean.

I don’t remember what they say because I have no concept of these things”

What do you think is happening in this example? My ‘languate’ friend clearly knows what they are talking about, and they’re passionate about it. But the way it’s explained is confusing for their partner — so they don’t remember.

These examples demonstrate our final sociolinguistic concept. Let’s look at in-groups and out-groups.

According to social psychology, an in-group is a social group to which people identify as being a member. The in-group classifies those who are not in that group negatively — the out-group.

This is a kind of ‘us vs them’ mentality, when it’s deliberate. In much of what we do, it’s typically subconscious.

In sociolinguistics, in-group members impose their dominant ideology (and language use) on the out-group to advantage themselves and legitimise the status quo. The out-group are then made to believe that they (and by extension, their culture and language), are inferior to the dominant group.

Have you ever been in a situation where a lot of complex words were used that you didn’t understand? How did that make you feel?

What you felt was exclusion. The in-group may have been using a number of complex words, or jargon, specific to the subject matter that they were talking about. You, as the outsider, weren’t given a way in through them explaining things to you. Maybe you were made to feel inferior because you didn’t understand something. (Again, this isn’t necessarily deliberate).

In some cases, the use of jargon is a result of our privilege. Perhaps you’ve been guilty of this. I’m certain I have.

Ask yourself, what are you trying to prove by using complex words?

In our research sessions with participants, we can sometimes make them feel excluded if we use words that are too complex, or don’t explain concepts or ideas in a way that’s easily understood. We can even do this through our accents.

We use language to exclude, consciously or subconsciously, all of the time. We use it to make ourselves feel good, and to make others feel bad. If there’s one thing I want you to take away from this article, it’s this.

When discussing your research plans, methods, and results with your stakeholders, you need to use juuuust enough technical language to show that:

• Design research is a legitimate skill that requires training
• You know what you’re talking about

But, you need to also explain what your technical language means so that your stakeholders come away feeling as though they’ve learned something, and that they’re on an equal footing with you. Plus they’ll remember what you were talking about!

The best design researchers are not the ones who use all the jargon and sound super smart in meetings because it is going over everyone’s head. The best design researchers are able to make technical concepts easy for anyone to understand, no matter their experience.

You’re striking a balance between affording your research the respect it deserves, and explaining what you do/why it’s important in a way that’s easy to understand, which avoids alienating your stakeholders.

If you can’t explain a concept in a simple way, do you really understand it?

Let’s summarise Parts 1, 2, and 3

You made it! Thanks for coming on that sociolinguistic journey with me. We covered:

• Prescriptivism, and why it’s not OK
• Registers and jargon, and how they sneak into our research approach
• How we express identity through language, and how we can use this
• In-groups and out-groups, and how to avoid creating these.

Here’s how you mitigate these things:

• Capturing exactly what the participant says, and mirroring what they say.
• Avoiding jargon and complex words
• Employing ‘word swaps’ to avoid overly technical language creeping into your facilitation
• Code-switching to reflect the way of speaking your participant is using
• Using phatic communication to make your participant feel at ease
• Literally just explaining what you mean

You won’t get this stuff right 100% of the time. We’re humans, we make mistakes. As a researcher, be aware of these things, so you can try to do something about them.

Finally, think about how you can use language to do good.

As before, please leave your comments below — I’m curious to understand any thoughts you had while reading this.

Special thanks to…

• Dr. Cameron Hunt, who languates good
• Dr. Nathan Eva
• Danya Azzopardi

If you’re keen to learn more about the weird world of language, David Crystal’s Encyclopaedia of Language is a great place to start.

Sociolinguistics, Design Research, and You-How to Languate Gooder: Part 3 was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How our experiences, the way we speak, and those of our research participants influence the collection of research data

Do you think your experiences always mirror those of your participants? Probably not.

Let’s explore how we use language to express our identity, subconsciously or consciously, and what that means for research.

What assumptions might you make about someone based on the way they speak? What assumptions might they make about you? I’m talking about accents. A good example of this is the TV show Family Guy.

Lois, the mum, comes from a well-to-do family from New England. How do you as the viewer know this? She doesn’t pronounce her r’s. This ‘non-rhotic’ r sound is a feature in some north-eastern American accents, and was in the past, a feature of how well-to-do people in the northeastern U.S. would typically speak (that’s called a Mid-Atlantic accent if you fancy going down a Wikipedia rabbit hole).

Peter, her husband, does pronounce his r’s, and he’s a bit more working class. The creators of Family Guy are subtly giving us clues about Lois and Peter, because of how we interpret their accents. This pops up in Australian media too…

These perceptions are exactly that — perceptions. Our society chooses to place these values on the different ways people speak.

How do you work with this as a researcher? Let’s look at some concepts from sociolinguistics to find out.

A major function of language is expressing personal identity. Through subconsciously or consciously choosing to speak a certain way, you can convey to the person listening who you are, or who you want to be.

When I delivered this as a talk at Design Research Australia, I probably sounded a little more “Melburnian” compared to when I speak with my regional Victorian parents or cousins — and this is subconscious. I want to be able to blend in with my family more. So, my accent changes when I speak with them — it sounds more broad.

When I speak with my researcher colleague at SEEK, who is in Kuala Lumpur, I try to avoid using too much Australian slang, and slow down a bit to ensure they understand me. Similarly, my KL counterpart avoided using the Malaysian English slang term ‘lah’ — until she found out my partner’s family is Malaysian Chinese and use this all the time! We’re both adapting to each other’s way of speaking.

So we know language helps us express identity, and we also know that we may change how we use language, in order to be perceived in a certain way. What else do we do? How do we apply this to design research?

Changing our language use according to the social context is known as situational code switching. The different codes you might use depending on the situation are called styles.

You can use this code-switching to meet your research participant where they are, without mockery or mimicry. You must adapt to them. Like this:

1. Become aware of how you sound, and the words you use. Maybe you *are* starting to think about this now you’re reading this article.
2. Depending on who your participant is, maybe you start to broaden your vowels ever so slightly. Maybe you start to use more of the words they use to describe things. Mirror the way they pronounce things, just a little, to break down those assumptions and establish some trust.
3. Once you get a ‘feel’ for who the person is, you can adapt your language to suit them. Tap into your own experiences to find some common ground.

This is a tricky balance to strike — you don’t want to become another person entirely! All you’re doing is meeting the participant where they are.

We’ve covered how we use language to express identity, and how we can use that to endear research participants to us. But wait, there’s more!

Aside from communicating ideas, we also use language to establish a comfortable relationship with other people. It can be as simple as saying ‘Morning’ to someone you pass on the street, or, in Japanese, saying ‘Itadakimasu!’ before you eat.

Neither of these convey ideas, but they serve to unite us over a common experience — building rapport. This is called ‘phatic communication’.

Think about starting a research session. Perhaps you ask the participant how their day was, or joke about needing another coffee (I do this a lot). You may resort to commenting on the weather. You’re using these things to establish a bit of rapport with your participant before you’ve commenced the session.

You may be thinking to yourself that all of this sounds a bit inauthentic. And you’re not wrong — sometimes, we run the risk of taking these things too far. What you want to avoid doing is stereotyping.

Stereotyping in a linguistic context involves contrasting two cultures or groups on the basis of a single dimension, and focusing on that as a problem for communication. Stereotyping can limit our understanding of human behaviour and can lead to miscommunication and misguided assumptions.

In your research sessions, focus very much on the individual who is your research participant. You as the researcher should be responding to them. This nicely segues us into the last sociolinguistic concept for Part 2 — status and contextual identity.

Status is the position a person holds in the social structure of a community or space. Roles are the conventional behaviours that society expects someone to adopt when they hold a particular status.

Contextual identity refers to how language use correlates with the characteristics of the context in which it is used.

As the researcher, you are likely expected to behave and speak in a certain way by the participant. There’s a power dynamic at play here. The setting of the research session is also artificial, and you need to break this down by placing yourself at the same or a lower status than the research participant.

Who are you in the research ‘room’? Who has the power here? If you need to, give your participant the power by adapting your status. Of course, that depends on how elite you think you are in comparison!

Be deliberate about the words you use, and the way you sound. Again, try to fit in with your participant’s world — don’t make them feel as if they have to fit in with your world.

So, what did you think?

That it’s for Part 2! Leave your comments below — I’m curious to understand any thoughts you had while reading this.

Part 3 is about how we as a design research community talk about what we do — and how we may exclude others through language we use.

Sociolinguistics, Design Research, and You-How to Languate Gooder: Part 2 was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hello. My name’s Caylie, and I’m a Design Researcher. In a past life, I studied Japanese, Mandarin, and Linguistics.

The use of language — spoken, signed, gestured, or written — unites us as humans. Beyond serving as a means of communication, language also has the power to influence the way we think, talk, act, and behave — the power to do good, and the power to cause harm.

Given my background, I’ve always wanted to explore the intersection of sociolinguistics — how and why we talk the way we do — and design research. Applying core concepts in sociolinguistics can improve the way we conduct design research, resulting in less biased research outcomes. That’s what these series of articles is all about.

Have you ever thought about…

• How the way you sound, and the words you use, might influence a research participant’s responses?
• How our experiences and the way we speak, and those of our research participants, influence the collection of research data?
• How we as a design research community talk about what we do — and how we might exclude others through language we use?

I’ll cover each of these 3 topics, in a 3-part particle. Below is Part 1.

But before I get into that, I wanted to give some context.

Why UX Research and Sociolinguistics?

I never really knew what I wanted to do. In secondary school, I suggested to my careers counsellor that I’d study anthropology, because I’d always had an interest in humans, history and biology. He replied — ‘what will you do for a real job’? (We got along well, so this was fine!)

I chose to study Japanese, Mandarin, and Linguistics, thinking I wanted to be an interpreter. I then studied a Master of Banking and Finance, thinking the combination of finance and Asian languages might be useful, given the increasingly global nature of money markets and financial services. This landed me a graduate position at National Australia Bank (NAB). I rotated through what was then the Human-Centred Design team.

From that point on, I was hooked. It made so much sense to me to conduct research with the people you’re building things for. I remained at NAB as a UX Researcher for a few more years, then landed at SEEK.

Having worked at two large corporates, I’m surprised by three things:

1. People who design products and services for ‘all Australians’ don’t always learn how to communicate to all Australians
2. There’s an assumption that the people who use the things we make speak and think in the same way as us
3. There’s a lack of focus on the language used in conducting research, as well as product & service designs.

My background, and these 3 things I’d noticed, made me think about how to join it all together. I first presented this as a talk at Design Research Australia — check it out here. But lucky you! You get to read this article!

Socio-what-now? Some definitions

Linguistics is the study of language. Not studying languages (although that does help to describe the things you observe), but studying how language is acquired, how it’s used, and its structure, sounds, and evolution.

Linguistics is a big field, and there’s lots of different branches. Here’s a few:

• Language acquisition (how humans learn first and additional languages)
• Historical linguistics (language change over time)
• Phonetics & phonology (sounds, and how we make them)
• Morphology (how words are made)
• Syntax (sentence structures) — mention the phrase ‘syntax trees’ to a linguistics student and they’ll cower in fear.

Our focus for this article, sociolinguistics, is the study of how people use language, and society’s effects on language — why we talk (and write) the way we do.

Cool. Let’s jump into Part 1.

Part 1: How the way you sound, and the words you use, might influence a research participant’s responses

We need to be careful with the words we use in design research & testing, as they can influence the participant’s responses.

Have you really listened to the way you speak?

• Does the way you speak change depending on who you’re speaking to? What about the words you use?
• Do you do this deliberately, or subconsciously?
• What about the words your participants use, and the way they speak? Have you noticed them before? Perhaps the words they use are different to yours. They might sound different to you.

If you haven’t noticed these things yet — you will now!

I was once conducting usability testing with a participant on the SEEK website. The participant referred to the job category section (it’s the bit where you choose what industry you think your job title fits in, before clicking ‘SEEK’) as the ‘clarifications’. It’s reasonable to assume due to the context that they meant classifications, but perhaps they misplaced the word due to some nerves, or just couldn’t remember the right word.

I didn’t say anything, and the session continued on.

What was I doing by not saying anything? Let’s look at some concepts from sociolinguistics to find out.

Prescriptivism is the view that one variety of language has an inherently higher value than others, and this ought to be imposed on the whole of the speech community [Crystal, Encyclopaedia of Language].

Think about that example I gave of a research participant using the SEEK website. What does that have to do with prescriptivism?

• I didn’t correct the participant, or ask what they meant. I didn’t repeat or ‘parrot’ the word back at them (a favoured trick of many researchers). Instead, I inferred from the context and their actions what they were talking about.
• Asking what they meant, or running the ol’ parrot trick, would have meant they may have realised their ‘mistake’ in calling the job categories clarifications (which, isn’t far off the ‘correct’ term).
• I would have risked making them feel embarrassed, therefore jeopardising the rest of the session — they may have tried to ‘prove’ they were clever or ‘tried’ harder to make up for it, which could have biased the data.
• If I had asked what they meant, or parroted — this would be prescriptivist. In my head, I’m thinking — “Oh, they clearly mean classifications, better ask just in case”. I’m telling them in a roundabout way that what they said, the way they talk, is wrong. And we don’t want that.

It’s a linguist’s role to describe what’s being said or written, avoiding judgement. The same is true of design researchers — we describe exactly what’s being said and done, with no editing, even though sometimes it may be ‘wrong’. There is no right or wrong way to speak. (That doesn’t mean it’s OK to use language that marginalises people from under-represented groups). It can be interpreted as elitist to define and expect adherence to a certain way of speaking.

I try to note the language used by participants to describe what they’re doing, the tasks they want to complete, their behaviours, and their desires. It’s our responsibility as researchers to not only capture our users’ behaviours, but also how they talk about these things — so we can mirror this in how our products are built, increasing their ease of use.

I have a funny anecdote to illustrate this, from a friend of mine. They were once getting tongue tied and struggling with their words in a conversation. Naturally, the rest of us were teasing them about this — to which they sarcastically replied “I languate good!” (They were also the inspiration for this article’s title).

What they did was take a noun (language) and turn it into a verb by sticking an ‘-ate’ ending on it. It’s not “correct”, but we all understood what they meant.

I want to introduce you now to another concept from sociolinguistics — registers and jargon. And yes, I realise the irony in this — I’m using a heck of a lot of jargon in this article!

A register is a variety of language that is determined by the things you are talking about. For example, the phrases ‘contextual inquiry’ and ‘usability’ may tell us we are dealing with the register that relates to design research.

A funny example of a register is ‘Legalese’ — the stuff in terms and conditions that no one ever reads.

Really technical words that we find in different registers are called jargon. In our field, we might casually refer to things like…

Platform

Concept

Design

Prototype

Feature

Product

We might call things “platforms”, “prototypes”, and “features”, but our participants who don’t work in our industry probably won’t use these terms. We want our participants to understand what we mean, so we get honest, clear responses, and the participant doesn’t feel like they have to ‘prove themselves’ by matching our language.

Here’s some ‘word swaps’ you can use instead:

Platform = website

Design = concept/idea

Prototype = fake website/website that doesn’t fully work

Feature = thing/thing you can use/thing you can do/part of the website/this bit here etc

Product = service, or, ‘things you use and do’

To figure out if my research approach is easily understood, I like to rely on what I call “the Parents test”. That is, if I ran a pilot research session with one of my parents, would they understand the things I was asking them about? You can use this too.

If your research participants belong to a group of people who use a particular register, get familiar with that — and work to understand from your participant what all those special terms mean to them.

So, what did you think?

That it’s for Part 1! Leave your comments below — I’m curious to understand any thoughts you had while reading this article, or your own experiences with language & design research.

Part 2 talks about how our experiences, the way we speak, and those of our research participants influence the collection of research data.

Sociolinguistics, Design Research, and You-How to Languate Gooder: Part 1 was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What I’m writing about below is not by any means the one-stop-shop guide to usability testing. It’s simply my process — and what I’ve found to work well through…doing this a lot.

Many of the steps below are also applicable to concept tests.

All right. On with the show.

What’s all this then?

Earlier this year, I was asked by my friend Kristen Hardy to present to the team at Telstra Health on conducting usability testing. I then spoke with Design Lead David Bacon to determine what would most benefit the team.

The steps below are drawn from content I’d created for previous in-house workshops both at SEEK and my previous workplace, and some stuff I along with my co-organiser Emily Murray facilitated externally at the meetup we run, Design Research Melbourne.

The 7 Steps of Usability Testing

1. Setting research and testing objectives (a very important but usually neglected step)
2. Working out who you need to test with (how to determine who you need, and recruit for that)
3. Getting consent (SUPER important)
4. Setting the scene (couching the testing in as much reality as possible, letting participants know if there are prototypes, and so on)
5. Creating the interview guide (includes testing live sites vs. prototypes vs. concepts, plus facilitating and note-taking tips)
6. Running the sessions (plus what to do when things go wrong)
7. Actioning those findings (duh)

Dotted throughout each topic was getting stakeholders on board and involved.

Step 1: Setting research and testing objectives

This is the most important step — getting clear on what you want to find out.

Objectives are also known as questions, goals, outputs, or y’know, the reasons why you’re doing the research.

It’s helpful to think of objectives in two “buckets” — the things that you, the designer/researcher, need to find out, and the things that your stakeholders want to know. They will sometimes be different!

Prioritise what’s in your head, and what your stakeholders are telling you. Start with what’s most important for you to know, so you can change what’s not working in your product. Normally, I think to myself — “at the end of this project, what’s gonna make me sad if I don’t know it?”

There’s a few different ways you can start to define objectives. What I like to do is make a draft list of what’s in my head, and then:

• Have a kick-off meeting with my stakeholders (this ensures they are more involved, and they have more ‘skin-in-the-game’ — therefore taking the findings more seriously)
• Map any assumptions and hypotheses — what they think they know already, and why

Remember…

There are many pain points you could discover in any usability testing session. A clear set of research objectives will help you stay on track, so you don’t end up with a disparate set of findings, or little snippets that don’t really connect with one another. If you’re finding that there are a lot of research objectives or questions needing to be answered, perhaps you need two or three discrete usability testing ‘rounds’, each with a different set of research objectives.

Think about…

• How realistic are the scenarios for your usability test? How ‘legit’ can you make them so the user doesn’t have to suspend their disbelief so much? This will ensure your research findings are more accurate — the more real the scenario, the more real the reaction!
• How many scenarios are you testing? Can you fit them all in with enough time to probe deeper if necessary?
• Do the scenarios reflect your research objectives?
• How the objectives will help you to write the scenarios in your test — if you have a clear set of questions to answer, the scenarios should write themselves.

Lastly…

Document, document, document! Documenting your research objectives is vital for the following reasons:

• You and your stakeholders have an agreed upon scope to fall back on (useful in case of scope creep)
• You and your stakeholders are on the same page about what this research is going to do, and why
• When other folks in your organisation check out your research (and they will), they’ll know exactly what you were trying to find out. Recording previous research allows for further understanding of your users and prevents duplication of research effort — saving us all time and money.

Step 2: Working out who you need to test with

Work out the variables

Map out the different variables (or elements that differ) which apply to your participants. Back in ‘ye old days’, I did this by scribbling on a whiteboard or piece of paper. Now, I’ll do it in Miro. Here’s an example:

In the study above, I was conducting research with our Corporate users. That meant I had EIGHT variables to consider to ensure I was getting a decent representation of users in my sample. I mixed and matched the different types of variable I had under these post-its in their columns and then wrote an idealised recruitment spec based on who I was looking for.

Here are some more common variables you might use to recruit for usability testing:

• The ages/age groups of your participants
• How frequently or not they use your product or service
• Where they live (important for accessing services)
• What other products or services they may use
• In some cases, their socio-economic status.

How do I even start to think about these?!

Map out what your participants must have and must not have:

• Must have/be (e.g. must use the SEEK mobile app, must be actively looking for a job…)
• Must not have/not be (e.g. must not have participated in any previous studies with SEEK, must not solely use SEEK to look for jobs…)

Then, think about general demographics, like age groups, languages spoken (i.e. English as an Additional Language), accessibility needs, and where they live.

Keep in mind, the more variables you have, the bigger your participant sample needs to be. A good rule of thumb is a minimum of 5 participants per variable, for example, product usage. Here’s an article for that.

Like the example above, start with your ideal recruitment specification and see what you can get. Then, you can loosen your variables a little. This is especially true of larger studies — pragmatism is important here, especially for larger projects in large organisations with some semblance of a deadline. Starting with the ideal shows you’ve done your homework and enables you to justify how, when, and why you made trade-offs in your recruit.

Creating a recruitment screener

This is where you write down exactly who you need to test with. Provide a summary at the start of the “must be’s and must not be’s” and list out the mix of demographics you are after.

Then, create a set of questions that will “screen out” who you DON’T want. For example, let’s say I want to test with people who are located in regional Australia (because its normally harder to find jobs in those areas), aged between 18–65, who don’t use SEEK frequently:

1. Which of these best describes where you live?

• Melbourne metro/inner suburbs
• Melbourne outer suburbs
• Regional Victoria…etc

2. Which age group do you belong to?

• 18–25
• 26–35
• 36–45…etc

Depending on the complexity of the study, you can then map out in a table an example of what your recruits might look like.

You’ll need to get quite specific in some cases to remove any chances for error in recruitment. It’s best to spell it out as much as you can.

Of course, this assumes you’re recruiting through a recruitment provider. You can also conduct recruitment through pop-ups on your own website, or via your Sales/Customer Service teams, for example. Point is, always use a screener. That way, you know for sure your participants fit the bill and you won’t get any surprises in your session!

Step 3: Getting consent

Wait, but why?

Ponder this for a moment. WHY do we need to get consent from our research participants? What if I’m just having a chat with them? Surely getting consent makes the session all formal and scary and therefore they won’t be as honest with me if I put a fancy piece of paper in front of them?

Nope. Nope. NOPE.

By asking for consent, we create a space in which the participant feels safe to share.

Think about this scenario — you tag along with a Sales representative and have a bit of a chit chat with the client about how they find your product. That’s all very well and good, but…

Will that client be honest with you when you’re there, with a representative of your company, and there’s finances involved?

Let’s replay that scenario again. You turn up with a notetaker and say you’re a researcher. You explain…

• What will happen in the research session
• How the session will be recorded (if at all) — video, photo, note-taking, the whole shebang
• What information will be collected, how it will be shared within your organisation, and stored
• What needs to be protected internally (i.e. market sensitive information).

Luckily, you’ve got all this written down so the participant can take their time to read through it, ask questions, and feel comfortable. This document is also in plain English (no Legal-ese!) and you’ve got some translated versions in your back pocket, to conduct research in different markets.

Sound familiar?

Yep, I’m describing a consent form.

Think about the consent form(s) you may have in your own place of work/study. Where did it come from? Who made it? What’s in it? Why?

Does it meet the needs of your research participants?

Are you able to answer questions about it?

Using a consent form means that not only does the participant understand what’s going to happen, and what’s going to happen with their data — it also creates a safe space for them to say what they really think. You’ll get much more honest and rich data this way, and this will in turn help you improve your products by solving the real problems for all of your users (not just the users who already love your products, for example!).

Step 4: Setting the scene

Now you need to do some more expectation setting. This typically takes about 5 minutes of your allocated 60 minutes with the participant (assuming a one hour session).

Things to think about

• Building some rapport with the participant. Be polite, but not too effusive— no one likes a robot, but you’re not there to make friends either!
• Explaining in more detail what the session will be about, and what kinds of things you’ll be doing (i.e. interviewing, doing some things on the screen, on paper, etc.) This helps the participant mentally prepare to context switch if necessary.
• Describing what a prototype might be like to interact with. We like to refer to these as “a website that doesn’t work yet”. If you’re testing concepts, be very clear they are in fact concepts (and don’t say you designed them!) — this means the participant will feel more comfortable saying what’s wrong with it. If you’re testing a live website, you can skip this.
• Ensuring the participant feels comfortable to be themselves — and to be honest (building on the consent form idea)
• Letting them know there’s no ‘right’ way to do things — we’re testing the usability, not them!

It’s important here to introduce yourself as a ‘researcher’ even if that’s not your actual job title! In this environment, everyone is a researcher. This reveals that you’re not attached to the products or services in any way, so the participant again feels comfortable to give negative or constructive responses. They also don’t feel like they ‘have to get it right’ to make you happy (we’re all people pleasers, after all).

Lastly…

Avoid using technical or jargon-y language. We might call things “platforms”, “prototypes”, and “features”, but our participants who don’t work in our industry likely won’t. We want to ensure our participants understand what we mean, so we get honest, clear responses, and the participant doesn’t feel like they have to ‘prove themselves’ by matching our language. Here’s some swaps:

• Platform = website
• Product = service
• Prototype = fake website/website that doesn’t fully work
• Feature = thing/thing you can use/thing you can do/part of the website

Avoid using language of ownership too:

• Our website = the/this website
• My design = the/this concept
• Product owner/manager/designer = researcher

Again, this helps the participant be more honest as they don’t feel like they have to protect your feelings!

If you’re a product manager/someone who works on the product or thing you’re testing…don’t say so. Using the term ‘researcher’ neutralises your role within the product — you are literally there to just find stuff out, meaning the responses you get will be more honest, and the participant feels more comfortable sharing with you.

Step 5: Creating the interview guide

Right. Now that’s all out of the way, you can write your interview guide. I use ‘guide’ as you’ll inevitably go off-book at some point, but you will need a standardised set of tasks for the participant to attempt in usability testing. This means you can easily compare results across participants to understand what things need improvement.

Start simple

Always start your session by asking some simple questions about your participant. This builds rapport, gets them in the right headspace for the session, and gives you some info you can re-use about them later:

• Tell me a bit about yourself?
• Tell me what a typical day looks like for you?
• What sorts of things do you like to do outside of work?
• Talk me through the last time you used [product/service]? (and is that a typical experience?)

These questions will also help you to validate the answers they gave in the screener. Think about how you can build rapport with these questions, and relate them to the product or service you are testing. Allow 5–10 minutes for this part.

Getting into the detail — what you want to test

Now, you need to think about those testing scenarios in more detail. What are the particular features or user flows you want to test? How common are they?

I normally chat with my Analytics friends to find out what the data is telling us about the most commonly performed tasks on site, then build some tasks around those.

You will need to create scenarios around these that are as realistic as possible. You can use the answers the participant gave you in the first part of the session to add a bit more “flavour”. For example, let’s say we’re testing the MYKI website and the participant told you they normally top up at the station:

“OK, imagine you are waiting at a tram stop, but there is no MYKI machine. You know you need to add some money so you can ride the tram, so you pull up the MYKI website on your mobile. Your MYKI is already registered.

Could you please have a go at logging into the MYKI website?”

Be sure not to alter the actual task itself! The more stringent/complex the tasks, the less you should add flavour so as not to bias the research in any way.

Tips on writing test scenarios…

• Make sure the tasks/scenarios start simple and get more complex as they go on. Ensure that the order of the tasks makes sense! (e.g. you wouldn’t make a payment and then login to a website).
• Always make the scenarios as realistic as possible. If it’s a live website, ask your participant to come prepared with any logins/information they might need to input. Otherwise, give them a set of “dummy” details to use so they’re not having to think up fake information to input on the spot.
• For a 1-hour session, allow approximately 40–50 minutes. This allows you to test 4–5 tasks/scenarios.
• Be sure to run a pilot session beforehand (ideally with someone who is not familiar with your product) to make sure everything goes as planned. You can make tweaks to the wording and timing after this.
• Avoid asking about future behaviour (e.g. would you do…). People are generally bad at predicting what they will do! Relying on past behaviours is a much better indicator of how people will respond to things.
• Ask open-ended questions as much as possible. An open-ended question allows for the user to give a more unstructured answer, so you can find out the surprising things you didn’t anticipate. Here’s an article on this.

• For the tasks, give the participant their own task sheet with the scenarios written on it. This means they have something to refer to if they forget what is being asked of them, and helps them feel more in control. If you’re using the Single Ease of Use Question or other metrics, they can fill in their answers themselves on the sheet.

Step 6: Running the sessions

Remember those stakeholders?

• Before you run the sessions, send around the interview guide for approval. This keeps them engaged.
• Keep them informed — let them know when you start recruiting, and the available time slots for note-taking.

Get your notetaker ducks in a row

• Create a spreadsheet or physical sheet for your stakeholders to sign up to be notetakers. I used to stick a laminated sheet on my desk that people could come and write their names on.
• Then, send a calendar invite with a buffer of 15 minutes before the session — this means they arrive on time, and you can use that time before the session to brief them on what they should be doing.
• Think about how you want them to take notes. I implemented Dovetail at SEEK because it’s digital, easy to share/pull verbatims out later (for those ‘Do we have any research on…’ requests), and you can tag the notes up afterwards with your team. Electronic is always easier! Figure out a system that works well for you and your team.

Note-taking tips

• I get my stakeholders to type straight into a blank Dovetail note or Word doc. This means all they have to do is listen and type — they are not worrying about typing answers under the right question (useful if you like to go ‘off-book’ and dig deeper at certain points).
• Get them to look out for mistakes, whether the task is completed successfully, body language, verbal cues (“hrm, um”), misunderstandings of words/functions, and typical behaviours (e.g. she said “I always pay my bills via BPAY”)
• If they do have questions for the participant — get them to note those down and ask them at the end. Having your session interrupted by someone asking questions is risky because you may lose control of the session. It also means the participant is having to focus on an extra person , which is distracting— they look like they’re watching a game of ping pong as their head keeps swivelling between you and the notetaker!
• Capture verbatim as MUCH as possible. This is crucial for quick synthesis. The notes should look like a conversation.

Facilitation tips

Be quiet and don’t interrupt

You’re there to learn from the user, you’re not there to sell an idea. Be quiet. Let them talk. When a user is about to answer a question, or starts talking, you stop. Listen.

Neutral responses

Say things like “mm”, “cool” and “thank you” in response to what the user is saying. This shows you’re listening, but you’re not being positive (which could lead them to say things that please you), or negative (which could lead them to lie to avoid the negativity). In Japanese, this is known as “aizuchi” — all the little noises you make to indicate you are listening.

Play dumb.

Participant: “If I click Submit, will the employer get my application?”

You: “Um, you’re thinking…?”

You might be tempted to reassure them that yes, the employer will get their application, or whatever. But you’re not there to say what’s going to happen, you’re there to find out what they think. Playing dumb allows us to draw this out. And if the user thinks something that’s not our intention, it’s our job to fix it.

Be a parrot.

Participant: “I normally apply for jobs, um…at night”

You: “At night?”

Play back exactly the words that the participant has said. By using the exact same words and an upward inflection, you’re asking for more info without putting ideas in their heads. You can use this technique also to go back and dig deeper on certain topics — this is also an indicator to your note-taker that you want them to capture it, as it’s significant:

“You mentioned earlier that you normally apply for jobs at night. Tell me more about that?”

Pause.

After you’ve asked the participant something, pause. Wait a good 5–10 seconds for their response. Don’t ask another question, don’t try to expand on it or use different words. Often, people will need a few seconds to digest what is being said, and formulate an answer.
This is also true of looking at designs or completing tasks — give the participant some breathing room!

Bounce it back.

People generally like to please other people. This means a participant will often ask for reassurance e.g. “Is that the Apply button there”, “Is that right?” Just put it back on them by asking “What do you think?” It can be tempting to explain or answer — but understanding what the participant thinks is a learning opportunity for us.

Testing a live site vs. a prototype vs. a paper/static concept

• If you are testing paper concepts, do more pointing and talking through individual elements on static or paper concepts (or get them to draw on it!)
• Live sites are easiest to test with as they are the most “real”. This is where it can be helpful to ask the participant to bring their own information to input.
• For prototypes, it is VERY important to make the data in the prototype as realistic as possible. Use real names, numbers, and information. Make sure there are no duplicates and everything is spelled correctly. If not, participants WILL comment on it, and it takes them out of what you are asking them to do.

What to do when things go wrong

Participants go off track

I once had a participant complain about an experience with Telstra for 20 minutes. Note I have never worked at Telstra.

There’s a few ways to tackle this:

• Acknowledge what they’re saying, but gently remind them why they’re with you — e.g. “That sounds incredibly frustrating/interesting/annoying, and if we have time at the end of our session today, we can talk more about that. But for now, I want us to focus on X”
• Interrupt. I don’t normally condone this, but if you’ve got a talker, you will need to butt in and steer them back on course: “Thanks for sharing that. Could you go back to the screen for me, and talk me through what’s happening for you?”
• If all else fails…just let ’em go.

The participant doesn’t fit the recruitment criteria

Sometimes, you discover that what the participant said in the screener doesn’t quite match what they’re telling you in the session. That’s OK — depending on how far you are in, you can politely let them know that they’re not quite the right fit for the study after all, and offer them a portion of the incentive (or the whole lot if you have cash to splash).

The participant gets frustrated or upset

Stop the task, let them have a break/get some water, and just move onto the next task. If they are really distressed, bring it back to a general chat about their typical use of your product, and let them let it out if they need to. Or, just stop the session and pay their incentive. It’s not worth pushing on at the cost of their mental health (or yours).

Your prototype breaks/the WiFi carks it

As a backup, have printed copies of the prototype that you can talk through. You could also have saved digital versions of the prototype (e.g. PNGs) and get the participant to interact with it as they would a working website — and ask them what they’d expect to happen.

The participant isn’t super responsive or chatty

You need to get creative. Ask them every possible question you can think of, always ask ‘why’ in response to their response e.g. “It’s good” Why? “It’s fine? Why? They are being paid, you can work to get a little more out of them!

Your participant wants to prove how smart they are

Sometimes participants will answer questions as if to say “Duh, don’t you know this?”. That’s fine. You’re not there to prove how smart you are or how much you know about the product. Play dumb and let them tell you how they think everything works. Use it as an opportunity to dig real deep.

Step 7: Actioning those findings!

Phew, you made it! You’ve created your objectives, recruited your participants, got consent, set the scene, conducted the sessions…so now what?

Remember those stakeholders?

These stakeholders are usually are made up of…

• Other designers
• Product Manager types
• Developers
• A BA or two
• Occasionally, a very highly paid person.

Hopefully, you got them all to take notes for you at least once. Now, you’re going to get these people together and run a synthesis session.

Why would I do that?

Getting your stakeholders to help you synthesise means that they will feel more attached to the findings. You can do this in a few ways:

• If you have notes in Word or Excel, get your stakeholders to transpose these to Post It Notes, and affinity map these as a group. This works well if you have a couple of hours on your hands
• If your notes are in Dovetail or some other online tool, assign a note to a stakeholder and get them to tag it up using some pre-determined tags.
• If your notes are in Excel, simply print them off and have a “cutting session” where you cut the notes out and then affinity map them.

Be sure to start and end the session with a discussion — get people to share what they found interesting/surprising from the testing. This means they come away with a broader view of what happened, rather than just their 1 session.

Now what?

To action the big stuff quickly, I print off the concepts/designs that I’m testing, and stick Post-It notes next to the bits I’d tested, summarising the findings. I’d put these up on a wall near my workspace and then the designers I was working with could get quickly stuck into making changes. This works well for rapid test and learning cycles, and you can also walk stakeholders through it. In our remote, digital world, you can do the same thing in Miro or another digital whiteboard tool — simply upload the concepts and put any commentary next to these in Post-Its.

Document, document, document

You’ll need to write up your findings — not just for presenting them, but also for longevity. Documenting our research ensures that we can justify certain design or product decisions, and also re-use findings instead of running research all over again, building our knowledge of our users over time.

This is the structure I follow:

• Executive summary (why you did the research, and the high-level findings)
• Key results — this is what you found out. Order these according to severity — if 5 out of 5 participants had trouble with something, that goes first! Throw in some verbatim and video to really drive home your points.
• The insights, and your recommendations. This is the ‘so what’ part of your report. Not only should you include what you saw/heard (the finding or observation) — but also what it means for your design, product, or business. What changes need to be made? What should your team be mindful of going forward when making prioritisation trade-offs? What has broader implications? What are recurring issues/broad problems that haven’t been solved yet?
• Then, in your appendices, list out your methodology (what tasks you tested, why, and how), the participant demographics, and the in-depth findings.

Bring your stakeholders back together to present these. Because you’ve kept them engaged, they will hopefully recognise:

• Any assumptions or objectives they helped to shape
• Verbatim from the notes they took
• Findings from the data they helped to synthesise.

This makes it way easier to get buy-in for the big changes. You’ll need to remain close to your stakeholders to ensure any changes are implemented, and the broader insights considered as part of decision-making as time goes on.

The big finish

Practice. Get someone who knows what they’re doing to watch you. Accept feedback and iterate…yourself!

7 Steps to Great Usability Testing was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At SEEK, we’ve been experimenting with different ways to conduct research remotely (moderated and unmoderated) for a little while.

Given the current situation with COVID-19, Mimi Turner and I had to quickly whip up some guidelines for our UXers and product teams to continue conducting research remotely — while being sensitive to, and respectful of, people’s experiences.

1. Recruitment and incentives

Our processes for recruiting participants remains much the same:

• We use an external recruitment provider to recruit our job seeker research participants for us, and in some cases, our small business employers. We create a recruitment screener that details exactly who we’re looking to talk to. Mimi and I review this for the UXer or team member before sending it to our recruitment provider.
• Recruiting other types of employers (for example, our corporate and recruiter customers) is typically done via Intercom, through a pop-up. Potential participants then complete a screener survey via SurveyMonkey to see if they qualify. They then progress to Acuity Scheduling to book a time with the UXer or UX Researcher (i.e. me) in question. We try to keep tabs on who in the UX team is doing what research so our website isn’t flooded with pop-ups! The other option to recruit for these employers is via our Sales & Service teams.

Given the current situation with COVID-19, we’re currently working through how we communicate and conduct research with our job seekers, and how we can approach this with our employers.

Regarding incentives, we use an electronic gift card provider called GiftPay. This service enables us to simply email our research participants a gift voucher that can be spent at most major retailers. We have an account for both AUD and NZD.

In some cases, our employers can’t accept gift cards. So, we offer a hamper instead through Halcyon Homewares that they can share with their teams.

2. Consent forms and recording

We have a remote consent form and Code of Conduct stored in SurveyMonkey. Our UXers can copy the link and email it out to research participants a day before their session starts, which also serves as a handy reminder!

Conveniently, this consent form link also lives in the calendar invites that are sent by Acuity Scheduling.

For recording, we have a few options (we’re currently reviewing which tool to use going forward):

• Our preferred tool is Lookback. You can set up a live moderated session, and send the link to the participant 5–10 mins before the session start time. All the participant needs to do is install the Lookback extension on Chrome, and we’ve got instructions for this just in case. Recordings are handily saved in Lookback, meaning that past research projects are easily found.
• In cases where the WiFi is misbehaving — we use Zoom. If the UXer in question does use Zoom, they save their recording in our centralised OneDrive folder, under ‘Candidate User Research’ or ‘Hirer User Research’. This ensures we can find our old research again (I am a big fan of re-using previous research where applicable).

We’ve even got a sexy filing structure template for the UX team to copy:

3. Notetaking

• On Lookback, notetakers can log in to watch the live session and take notes in real time in Dovetail. It’s really important to let the participant know that you have a notetaker online watching the session, and that some others may be watching the session after the fact. Some Product Managers and UXers have done this for me a few times now and it works quite well!
• On Zoom, the downside to this (so I’m told) is that the notetaker has to join the same meeting as the participant to watch the session — so, the participant can see who and how many people are observing them! Keeping our Ethics Principles in mind, our recommendation for our UXers is that they introduce the remote notetaker at the start of the session, and tell the participant they are only there to take notes. If others in the team wish to watch the sessions, they need to do so after the fact. We don’t want participants to feel put off or uncomfortable by being able to see a number of people watching what they are doing.

4. Other remote tools

• For conducting remote unmoderated research, we have set up an account with Validately.
• We have Optimal Workshop for card sorting, IA, and first-click tests. For both tools, we ‘turn on’ the licence as needed. Mimi and I then help the UXer set up the test.

Lastly, a few tips…and the ethics of it all

Wanting to run some more ‘hands on’ activities? Think about how you can run them digitally — it could be something as simple as a Word doc you send ahead of time. Then they share their screen while completing the activity — or even getting your participants to draw stuff out on paper. Have them email you a photo afterwards. Or like, literally get them to hold it up to the screen and take a screenshot.

Synthesising? You can do it as a team in Dovetail, all logged in at the same time! Divide and conquer by assigning each person a different note to tag up. I did this recently for a research project with our job seekers — I had my stakeholders jump into the Dovetail project, pick a note each, and tag it up. We were all doing it together on Zoom, and even had some groovy tunes playing to spur us on.

This halves your synthesis time and gets your stakeholders really stuck into the data. You will need to go in and do a bit of a tidy-up afterwards, but the stakeholder buy-in is invaluable.

And last but not least — let’s touch on Ethics. We reminded our Product teams of our SEEK UX Research Ethics Principles — namely that they still apply in a remote context!

These Principles enable us to understand how to store participant data correctly, and how to keep ourselves and our participants safe. In light of COVID-19, we included the following paragraph for job seekers:

We appreciate that this self-isolation period is a difficult time for everyone, and may impact your ability to work, or to find and gain employment. If you have any concerns about this, do let the researchers know.

We are also asking participants at the start of each session a few questions to acknowledge the situation and give them space to talk about what’s been going on for them:

Given the current situation with the COVID-19 pandemic — what has changed for you?

How are you feeling at the moment?

We’ll also use the responses to contribute to a broader monitoring program tracking how our job seekers are going.

So, what does this have to with Ethics? By using a consent form, and actively checking in on our participants’ mental states, we give them a safe space to talk about how they are affected. We are acknowledging the situation without taking on the responsibility of solving it.

We don’t conduct research in a vacuum, we conduct it in the real world — which is messy, unpredictable, and full of feelings. We need to be able to deal with that.

4 Guidelines For Doing Remote User Research — Ethically was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Often, we’re so focused on pumping out the next bit of research/design/Keynote presentation that we forget to stop and smell the lightbulb-moment-roses, so to speak.

The role I undertook at SEEK was a step up from my previous one. This was great, because I was after a challenge — but I’ve also learned some lessons along the way about what it means to build and lead a research practice.

With that in mind, I wanted to take the time to document where I started, where we’re at now, and what I’ll do going forward. I hope this helps those of you out there stepping into UX roles with a leadership focus, or building a UX Research Practice.

In the beginning…

When I began at SEEK in June 2018, my super-powered predecessor Mimi Turner had gone to great lengths to establish the UX Research Practice. I was delighted to find many processes and how-to guides had been documented and a number of tools were already in use. (For those wondering, she’s back now. Praise be.)

At that time, the UX and product management team each were around 25 people big. A number of the UXers had undergone UX Research training with Mimi before I started, and were comfortably conducting their own research for their particular product, expecting guidance from the UX Research Practice (i.e. me) when required.

The organisation was shifting to an environment where product managers also contributed to research activities focused on their product — as part of a new way of working called Continuous Discovery.

I encouraged the expectation that “PM/UX pairs” would conduct research together in a conversation with our Chief Product Officer (yes, I’m cheeky).

All of this left me with two questions:

1. In this new world, what purpose did the UX Research Practice at SEEK serve beyond operational and coaching support?
2. How would I ensure research quality in a world of decentralised research?

Read on for the answers I developed — but before I got stuck into tackling those questions, I first needed to prove myself.

Lesson 1: Establish your turf and get some runs on the board

Who is this newbie trying to tell me what to do? They’re like, 12

When I started the role at SEEK, one of the thoughts bouncin’ around my noggin was something like “What if people think I’m too inexperienced for this?”

I also suffer from the unfortunate condition of BabyFace(TM), which didn’t help. (But hey, I’ll look great when I’m 60…I hope).

Related to these thoughts was the desire as well as the need to prove myself. I remember speaking about this with my manager at the time and he agreed that I needed to get some runs on the board before the teams would start to trust that a) I ~usually~ knew what I was doing and b) they would, y’know, remember who I was.

My first research project at SEEK was a fairly straightforward contextual research piece with underserved job seeker segments across Victoria. This was great because I got to understand the very real struggles some of our users face when looking for work — and display my findings in a 2-day expo, building empathy for our job seekers. That research now sits in a mental model of the job seeker experience, representing this group of users alongside the ‘typical’ job seeker.

After that, the requests for research advice (and projects!) came in. I could start poking my nose into other research happening around the business.

I think this victory was really evident for me when, in October last year, one of the Heads of Product came to ask my advice and help on a small research project his team needed to conduct. That wouldn’t have happened 18 months ago. I nearly fell out of my chair from shock, but the fact he came to me, wanting to get the research done right (to ultimately make better decisions), is a big step in the right direction. (Cheers Jay!)

Lesson 2: Get a seat at the table, and own the space you are in

Why is this newbie at this meeting? What are they, like, 12?

So, my first research project at SEEK was complete. Thus began the nose-poking (nose-pokery?). I was now getting up to speed with what research was going on, and start to lend a hand or lead projects where I could — again, getting more runs on the board.

Compared to my previous role, SEEK is a very collaborative environment and decisions are usually made via consensus. Looking back, one thing I wish I’d done more of was build relationships with the various Heads of, and Product Managers, and gotten to know their spaces a little better. I also focused a lot more in recent months on strengthening relationships outside the Product space — with Marketing, Sales, and Strategy to name a few. This enables me to better find supplementary research data or complementary initiatives — increasing the sharing of user insight. Everyone wins!

It’s taken me a while to adjust my approach from ‘research it and people will come’ to working out how to get research insights into the right heads — and determining what research needs to happen in the first place.

Related to this was the first question I mentioned earlier. Let’s look at it again:

In this new world, what purpose did the UX Research Practice at SEEK serve beyond operational and coaching support?

Firstly, I felt we needed a common language/vocabulary to talk about all of our research. Creating a shared understanding of research process and practice ultimately gets things done faster — work is universally understandable and thus repeatable. (Repeatable, recordable, and rigourous — the three ‘r’s of good UX research).

So, I set about developing a framework to address this. It would be something around which to anchor our conversations and help us identify what kinds of research we were and weren’t doing. It needed to incorporate some product management thinking (i.e. align with Continuous Discovery and Marty Cagan’s Four Risks, to name some examples), as well as UX terminology. It also needed to reflect a space for UX Researchers to bounce around in. This is what I came up with (thanks to Steph Moss for tightening the visuals):

So — the UX Research Practice would lend coaching and operational support to Product teams (i.e. those PM/UX pairs I mentioned earlier) conducting exploratory, generative, or evaluative research for their product. While collaboration between product teams did exist, research conducted at SEEK previously typically focused on individual products, and still appeared that way when I showed up.

This left me with some gaps to play with:

• Exploratory research looking into certain customer segments
• Evaluative research assessing the site-wide experience
• Exploratory/generative research into new product spaces (well OK, I didn’t come up with this point myself but we’ve started this recently… that’s another article for another time).

This meant the UX Research Practice would be responsible for informing multiple product teams at a time, enabling them to make better decisions and eventually helping to set the direction of products. Space = owned.

Real-world example: I recently conducted two research projects — one looking into the experience of our corporate clients, and another looking into the experience of white-collar professional type folks. Both of these projects have informed decisions made in the business. However, this style of research is very new to SEEK, and we’re still hammering out the best way to implement it.

Following on from this, I’ve been running quarterly prioritisation sessions with the Heads of Product and UX Leads in order to determine what research is most needed. I’ve divided these projects into “Tracks” of research — all of these projects sit above individual product team work, and are led by the UX Research Practice with Product team support. I don’t have a pretty diagram for this yet, but it’s something like this:

Track 1: Streams & Segments

This research focuses on the 3 or so user streams we have at SEEK, and user segments within those streams. For example, a “stream” would be our job seekers who use our site, and a user segment within that might be parents returning to full time work. It can also focus on how our users engage with competitors.

Track 2: UXR Experiments

This research focuses on as-yet unsolved problems for our users, and the business. The problems are sourced from the business, or from various leadership forums.

Track 3: Site Hygiene

As the name suggests, this track focuses on those site-wide evaluative projects I mentioned earlier — assessing the whole experience against a number of metrics (as opposed to the experience for one product).

It’s taken some time to figure out just how we prioritise these forms of research — but hey, that’s what iteration is for! (And now I have an idea for another article…)

Lesson 3: Take 3 months to observe what’s happening and what’s not happening

We’re doing a lot of this, but not so much of that…so let’s do less this and more that

Jumping into a new, more senior role is not easy — and changing everything the moment you get in is a bit silly. You don’t necessarily know the reasons why things are done a certain way, and you need to earn people’s trust. Take time to listen, watch, and think. Understand what people are frustrated with and understand what’s working well.

As I mentioned earlier, there was a lot of documentation on research process and tools, but fewer documents describing how research is conducted at SEEK, and what kinds of methods you can use.

I worked with our Principal Designer Cheryl Paulsen on documenting some common methods used at SEEK (she’s also made a very cool tool for this that you can find here) and created a few more guidelines here and there to meet this need. Everyone loves a good how-to doc!

It became apparent that finding old research was a problem. This is definitely not unique to SEEK, but something I wanted to mitigate so that all our research was in one spot and re-usable.

I ran a workshop in which we assessed a number of different UX research tools. We came up with a shortlist of tools that we would continue to use, and one that I implemented to help us better synthesise research data and keep it all in one place. While we’re still getting all the teams familiar and comfortable with the tool, we now have a budding repository of raw research data, tags, and insights — meaning a juicy verbatim is more findable than ever.

Another initiative I began early on was the creation of SEEK’s UX Research Ethics Principles, which enable product teams to conduct user research that’s safe, respectful of our participants, ethical (duh), and looks after participant data. This clears up a lot of the ins and outs around what tools to use, getting consent (and storing it), what to do when things go pear-shaped, and how to store participant data. Massive thanks to Danya Azzopardi, Leah Connolly, and Mimi Turner for continuing to drive and deliver this piece with me (another lesson — you can’t work in a silo!).

Lastly, with direction from our Chief Product Officer, I created a series of training workshops for 5 “guinea pig” Product Managers. These covered the product lifecycle and research, facilitation, ethics, synthesis, and a quick look at different methods. We (Mimi and I) are now working out how we can turn this into a repeatable training module for product teams.

You may have worked out by now that these initiatives happily address my second question from earlier — How would I ensure research quality in a world of decentralised research practices?

It’s taken me time to get these things up and running, but I was grateful I allowed myself the time to absorb before jumping in.

Lesson 4: Impostor syndrome is real, and that’s OK

*arms flail, screams into void*

It’s amazing the different the skills people have. I’ve worked with PhD candidates, people *with* PhDs, data scientists and a whole bunch of other people at SEEK who are probably much smarter than me.

¯\_(ツ)_/¯

There are times where I have felt like a literal potato (specifically, a chat) in comparison (and that’s the impostor syndrome talking). I frequently had to remind myself — and still have to remind myself — that I am not a potato. I may have skills and experiences other people don’t have, just as other people have skills and experiences I don’t have. At the end of the day, we’re all here to make the world a better place (you’d hope) — so it makes sense to share skills and learn from each other.

When you’re starting a new role, focus on what you’re good at, and take the time to learn new things from the people who have the skills you don’t (books and articles also come in handy here). Once you’ve got those runs on the board we talked about, you can then start pushing yourself out of your comfort zone.

I felt very keenly that quantitative research skills, namely surveys, was a gap for me (not having had the means to conduct large surveys online previously). So I leapt on a survey writing course, and set about practicing my survey skills. Writing my first big fat survey wasn’t a piece of cake, but I’m super glad I pushed myself to do it, and I’m grateful for the support I had in doing so.

Another thing to remember is to keep checking back on, and documenting, the work you have done. When I reached my one-year anniversary at SEEK, people said to me “You’ve done a lot in a short amount of time”. At first I was like “Have I really?” but when I looked back at what I’d documented it wasn’t too shabby. Always continue learning, but don’t forget to look back on what you have done.

The reflection for me also helped justify Question Number 1 — the navel gazing, what is my like, purpose maaaan? question.

Lesson 5: There are so many things to do! Stop trying to do all of them

What do you mean, that can wait?! Everyone needs all the things NOW!

In my last role, I sat within a product or platform team, conducting the nittier and grittier research pieces like usability testing, concept testing, and some exploratory work for the product/platform.

As mentioned above, at SEEK we’ve moved to a world in which UXers and PMs conduct the evaluative or generative research for their product, which freed me up to tackle exploratory pieces of research into the holistic experience for a segment/user type, and future-focused pieces looking into unsolved problems.

I was very excited by all of this, but it was hard for me not being in a product team, doing the nitty gritty research that has a direct impact on the product. I needed to let go of what I was familiar with and start tackling the strategic problems.

Another facet of this was the operational side — not a day went by where I wasn’t dealing with a question related to ops, process, general how-do-I-do-this’s, or…complaints!

Don’t take it personally if people aren’t happy about things — you as an individual can try to improve as much as you can, but you’ll never please everyone. Aim for better, not perfect! It can be helpful here to document process and practice as much as you can so the team can help themselves (and make it clear what things they can and should be doing themselves), as well as limiting yourself to working on the projects (research or research ops) that will have the most bang for their buck. For me, this meant taking a good chunk of my downtime in December 2018 to write a bunch of how-to guides in our company intranet.

When making trade-offs on pieces of work, I’d always prioritise actual research projects and larger-scale operational pieces for the benefit of the team over the smaller things. It meant that some things were left by the wayside (sorry, OneDrive folder structure…) but it meant that I could focus on the macro — like where I wanted the UX Research Practice to go.

Lesson 6: Communicate that you are available

I may need to do some calendar gymnastics, but yes, I have 30mins for that interview guide review

Your team will come at you with last minute requests to check things, workshop things, be in meetings, or just to bounce ideas around. I learned to work at 80% capacity so I had time to deal with these curveballs. Being available for your team means they feel supported — that you care. (And to my UX and Product colleagues — I care about you all very much. Mimi and I will always make time to help you if you need it.)

I’d operated under the assumption that people would come to me if they needed coaching — but I found that sometimes it was better if I reached out and asked, as a sort of reminder that I was there to help.

I also learned to ask lots of questions to get a better understanding of what people wanted to do, and how. For example, I remember chatting with one of our UX Designers in the Mobile Apps team about research they intended to conduct with our mobile app users. After I’d asked some pertinent questions on what they were hoping to learn, we ended up reframing the research as a semi observational, semi usability testing exercise.

Chewing over the product-y problems keeps your brain fresh too. Sometimes it’s a lot for me to take in. But I usually have a rough idea of what each of the product teams are doing — which helps me connect various initiatives for the teams and gather information easily.

Lesson 7: Stop talking. Stop it

Well perhaps you should…nah wait. What do you think?

In UXR coaching conversations with my team and with Product folks, I tried to ask questions, not only to understand the problem, but also to enable the team member in question to come up with the answers themselves.

This was incredibly important for my Associate UX Researcher to grow her confidence in preparing for and conducting user interviews, among other methods. Now, she’s been able to start coaching team members herself.

Another benefit of this is that you’re giving space and time for the individual in question to practice how they explain things, and really allowing them to own their space and their work. I like to think of this as giving someone a leg up so they can climb over a fence — you’re not visible, but you’re making them more visible (rather than say, clambering over them).

Funnily enough, this reflects the behaviours of a good UX researcher — ask the ‘participant’ what they think.

Understand before you can be understood, and try to let folks speak for themselves.

Lesson 8: Be flexible — don’t be afraid to bend the rules to get the job done

You *can* take that metric…just don’t report it like that

Not gonna lie, when Continuous Discovery came along I was hesitant. However, I’m glad that I was able to work the process into UX Research methodologies and ultimately get the two to play together nicely in the sandpit, so to speak. It ultimately benefited me because I was free to explore chewier chunks of research (many UX Research folk get a bit over usability testing for months on end…).

Anyone working in a corporate environment needs to be adaptable and flexible, while (in a UX Research context at least) maintaining rigour of process and practice. You can achieve this by ensuring processes are well documented, and being very clear on what is and isn’t UX Research.

For example, the Product Managers I work with at the moment know very clearly the difference between what I do when I conduct research with our clients, and the conversations they have with the same clients. The first is research. The second is anecdotal or useful context. And they’re cool with that.

A colleague of mine was curious to learn about a variety of UX metrics, and wanted to use them in some usability testing he was conducting.

I’d previously referred to a study in which I ‘baselined’ our user experience using these metrics — and typically you’d need a high-ish number of participants to ensure your metrics are solid.

My colleague didn’t have enough participants, but, for the purposes of his learning, and to gain an indicative understanding, we took the metrics anyway. We then reported the resulting numbers as a range (luckily there were no huge outliers), and stressed that this was an indication of usability we could keep an eye on over time.

In short, pull an Elsa from Frozen — and let it go.

Lesson 9: Things will never be perfect, things will never be right

Look, the 34 participants I recruited don’t EXACTLY reflect our user base but it’s darn near pretty close

The second example above is a good reflection of this. My colleague’s use of metrics wasn’t necessarily by the book, but it was good enough for them to learn what to do, and gave an indication of usability. We were careful in our terminology of reporting it so our stakeholders didn’t get the wrong idea.

The research I conducted with our clients recently (as the italics allude to) was another example. Our clients are notoriously hard to recruit, and I’d spent a good deal of time working out the perfect recruitment spec that reflected our user base according to a number of variables — like spend, location, role type, industry, software used…and so on. (I’m ashamed to say I had fun doing it).

BUT — with these sort of recruits, sometimes you get what you’re given. I think our resulting recruit was pretty good — I’d have liked a few more of this and that, but ultimately I had to accept who I could get.

Starting with perfection gave me something to work towards, but I quickly learned that to get the work done, tweaks needed to be made.

Similarly, getting the team to use a new synthesis tool, updating our documents, and encouraging re-use of research are all slow processes — however, better is still better! Striving for perfection every time will mean you get burnt out pretty quickly.

Lesson 10: Do a thing. Get feedback on the thing. Iterate the thing. Present the thing. Working group the thing. Keep banging on about the thing

Remember this? Yeah? Good. Let’s talk about it again

I think this lesson wraps up all the other lessons quite nicely. Maybe its a super-lesson.

Let’s revisit those two questions I posed at the start of the article…am I meeting my objectives?

1. In this new world, what purpose did the UX Research Practice at SEEK serve beyond operational and coaching support?

• I led the charge to shift the UX Research Practice to exploratory or broad research foci, through developing and communicating a framework
• I and my Associate UX Researcher actually *did* research in this space, and we’re working through different ways to implement it
• I developed a number of ‘tracks’ of research to better communicate what kinds of research the practice can do, and how that benefits the business

2. How would I ensure research quality in a world of decentralised research practices?

• We developed the SEEK UX Research Ethics Principles
• We implemented a new synthesis tool
• I updated our documented processes (this is ongoing really)
• I made myself available for UXR coaching
• I developed a training program, which we’re now iterating

I know I’ve used the word “I” here, but all of these things I did not do alone. While each one was I guess spearheaded by me, they all had constant feedback from my peers and stakeholders, and they were iterated on accordingly. The Framework? Folks from the UX team had input into that. The tracks? I ran that past a Head of Product and the Head of UX before circulating it more widely. The training program? That had input from both product managers and UXers.

I continue to reference these initiatives whenever the opportunity arises (to the point that I’m one of ‘those people’ that’s usually got something to say in meetings).

It is much, much easier to get stakeholder buy-in when you’ve taken your stakeholders on the journey with you, heard their thoughts, and incorporated them.

That doesn’t mean you need 50 people involved in each project — just a mix of folks who you know will provide thoughtful, nuanced feedback. Take your time to find those people, and cherish them.

Most importantly — don’t give up. Change takes time (a lot of time!), and it takes people power. UX Research Practices are not built in a day!

With thanks to Timo Hilhorst, Kayla Heffernan, Mimi Turner, and the SEEK Product team for putting up with me banging on about UX Research. You’re stuck with me now! Mwahaha!

10 Lessons in Building a UX Research Practice was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

So. I’m going to write about spiders.

If you don’t like spiders, or if you’re an arachnophobe — beware! Pictures of spiders ahead.

Wait, you helped design WHAT?

In late 2017, two dear friends of mine let me know that they were attempting to build an app that would help Australians (and tourists) identify spiders.

I didn’t like spiders much. I didn’t like looking at them, them looking at me (with all 8 of their eyes!), or even being in the same room as one.

Despite that, my UX brain kicked into overdrive. The first question I asked was “How do you know this is a real need?”

Caitlin Henderson (Friend #1) has been working with spiders and insects her whole life. As part of this, she’d spent a lot of time responding to terrified individuals via her father’s Minibeast Wildlife Facebook page telling them that no, this spider was harmless and it wasn’t going to kill them. At that point, she was responding to multiple messages a day.

OK. That’s a little bit of a need. Minibeast Wildlife thus developed the app, bringing Caitlin and Cam onto the team for the project. It was to be called Spidentify.

I said to Caitlin — “Look. I’m a UX Researcher. Let me help you out”. Caitlin and Cameron Hunt (Friend #2) agreed to send me what they’d developed so far for review.

Now, UX folk. I can practically HEAR you thinking— “Wait a minute? They started developing without designing first?! Where are their task flow diagrams? The wireframes? The STYLE GUIDE?” That’s fairly common in start-up land though.

Cam and Caitlin had never developed an app before. The reason Cam had jumped into coding the app straight away was to see if he actually could (spoiler alert — he could).

Caitlin meanwhile was developing the content to describe the 250 or so spiders that would feature in Version 1 of the app.

Right. With that background, let’s take a look at what Cam sent me first.

Development Without Design

Here’s where Cam had got to so far:

Putting Design Back In

I had Caitlin and Cam pop over, and we got down to business.

First, I quizzed them both on their user base. We knew a lot of people on Facebook asked Caitlin about spiders. I asked Caitlin who exactly these people were.

“Worried mums and dads, sometimes kids, people who are just curious, and occasionally arachnologists” was the answer.

OK. Our user base was pretty broad — and the majority of them weren’t spider experts. This meant the app needed to:

• Use simple language
• Have biggish buttons for those little kiddy fingers
• Pander to the spider aficionados out there. (Arachna-nados?)

However, the main focus of the app was to help the average Australian-slash-tourist identify whichever arachnid happened to be making itself comfortable in their bed. (Eep.)

I mapped out a simple user flow, and then asked Caitlin to have a go.

The benefit of mapping out these flows before you start designing is that you can figure out all of the things that need to go in the website or application, and all of the choices that the user needs to make. Then, you can carve off one bit at a time to design.

Here’s my flow:

And here is Caitlin’s:

We then spent some time on design-y stuff.

Visual hierarchy of information

When you look at any kind of content, it’s important to know which bits are the most important — or, which bits you want your users to look at first.

Let’s take a look at a spider description page. Remember, the aim for the majority of the population is to understand “What is this, and will it kill me?”

Check out the below image. Can you tell at a glance what this spider is — and if it’ll hurt?

The largest text on the page above is the scientific name of the spider — so that’s where most people’s eyes are going to go first. Then, we’re greeted with a large image of the spider, and we don’t come across the spider’s common name — or what most people would call it — until halfway down the page. (A wolf spider — they have lovely reflective eyes!) The scientific name is also repeated, taking up valuable real estate on the screen.

In re-jigging this, I got very used to looking at images of spiders. Almost to the point of feeling…they’re kind of cute?

Anyway, here’s a screenshot of the page as how it looks today. What differences do you notice?

The common name is now at the top of the page, followed by a key so you can understand how dangerous the spider’s bite is. Then if you’re a keen bean, you can swipe over to read more about the spider and it’s behaviours. So, we’ve catered to the multiple user groups here, while addressing the most important pieces of info first.

Accessibility

We then talked about making the app more accessible, in terms of colours used, language used, and the types of selectors and buttons.

Firstly, identifying and describing spiders sometimes uses language that’s not so common. We needed to think carefully about what words we were using to describe certain items in the flow to identify spiders.

For example, “habitat” might not be a super clear word depending on the age and educational background of the user. We ended up employing light-boxes to explain words like these:

Secondly, I wanted to ensure that the colours we used had enough contrast so that they were legible, and that the buttons and selectors we were using were large enough to be used by all kinds of fingers, and clearly differentiated in their purpose. So — that meant it was time for a style guide!

Creating a visual language

I wanted to ensure the designs had some level of consistency. The designs DO look different now, but here’s what I started off with as a way to explain consistent visual design. (Not the full shebang, but you get the idea):

These guidelines incorporated a few tips and tricks I’d picked up:

• Putting words and icons together in the menu (not one or the other — the current version of the app does this when you click on the menu item)
• Differentiating between checkboxes (to select multiple items) and radio buttons (to select single items)
• Changing up the error messaging from red (which can be anxiety-inducing) to orange (less so). Turns out that’s not so easy to read on a white background.So, the error messaging is now white text on a black background. A nice little prompt instead of shouty red, which jumps out of the page.
• Increasing the font sizes, and using colour, font weights, and underlines as a means to differentiate items.
• A couple of the pages use sliders. The sliders are both tappable and slidable.

It’s important to note that these were very early stage designs — the designs have evolved since, as is normal with most projects.

The tools of the trade

After all that, I took these VERY simple designs that I’d whacked together in Sketch (praise be unto it), and showed Caitlin and Cam how to string them together using InVision. This meant they could then test simple flows without having to go to the effort of building the app.

Getting Annoyed with Each Other

Fast forward a month or two, and Caitlin and Cam were well on their way to becoming fully fledged designers/content strategists/developers. At this stage, they were consulting with me on some design choices, but for the most part were hurriedly building things and creating the content and designs themselves.

I downloaded a beta version of the app for a design review. Now, given that we were all friends, the conversation got a little…casual. I was using shouty capitals in our Google Doc communications out of frustration, because I felt as though my advice wasn’t being followed. Cam, I found out later, had been working fairly long hours to code the app and wasn’t taking kindly to being Google Doc’d at.

It’s important to note we were all working full time at this point — the app was never going to be pixel-picture perfect, and it likely still isn’t. That’s what iterations are for!

We all learned to be a little kinder to one another, and ourselves.

What Now?

We’re currently on version 2 of the app (now with added Jovial Jumping Spiders!) and I’m about to sit down and conduct a review of the designs and flows ahead of version 3 — as with anything, there are always improvements to be made, and it’s been a while since I’ve had a proper look.

I said at the start of this article I didn’t like spiders much.

Do I like them now? Definitely.

You can download Spidentify through Google Play, or the App Store. With thanks to Chloe Bloem, Aaron Orellana, and David Powlett for the extra design advice.

Designing a Good Spider Experience — Or, How I Learned to Stop Worrying and Love the Orb Weavers was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The ask

About 3 months into my role at SEEK, the Candidate product stream (that is, the side of our site that you’re probably most familiar with — looking for jobs) wanted to better understand job seeker sentiment toward SEEK.

As a researcher, this piqued my interest so I poked my nose in.

The goal in measuring sentiment is to provide a leading indicator of how our users are feeling about SEEK, regardless of their experience in finding a job. This would be used to adjust the things we were working on, and ensure we were solving the right problems. It was intended to be a rolling metric.

Previously, the SUPR-Q and the NPS had been used in an ad-hoc fashion in individual product spaces, but not to measure sentiment specifically.

There had been some discussion around using SUPR-Q to measure sentiment — because SUPR-Q covers loyalty, appearance, trust, value and comfort, and usability.

However, any researcher worth their salt knows it’s important to get stakeholder buy-in from the get-go. (For a more in-depth discussion on that old chestnut, read this article). So, we surveyed the product stream first to understand what people thought sentiment meant, and how they expected it to be measured. This also meant we got ourselves aligned on the idea.

Remember, the title of this article is bending the rules. At this juncture, I was already bending the rules by getting stakeholders involved in defining what we were measuring, how, and why. A research purist will tell you that’s a bad idea, because stakeholders might not understand what constitutes “good” research.

A pragmatist will tell you that getting your stakeholders involved sooner will mean they’re more likely to listen to and use your research results (in some way).

The problem

The survey asked our product stream what sentiment meant to them, their product area, and what they’d like to learn. We also asked how they thought this information (i.e. sentiment) would be used, and how sentiment could be measured.

It turned out that what the product stream wanted to learn from measuring sentiment, differed to how it had been measured previously. Metrics like the SUPR-Q and NPS only tell the “what” part of the story. What the stream were after appeared to reflect a need for some qualitative data.

The product stream told us that candidate sentiment means:

• feelings toward SEEK or about SEEK
• candidate experience and emotion
• why a candidate uses SEEK
• whether they would come back
• how they go using SEEK

It was not how a candidate feels about not having a job (more on that later).

There’s a lot to unpack here — the above responses incorporated a little bit of usability, a bit of loyalty, a bit of trust, value and comfort, and a bit of qualitative stuff — the why behind the what. So, a single metric probably wasn’t going to tell us what we needed to know.

Our survey respondents also told us that they planned to use the learnings in the following ways:

• as a leading indicator of what needs to be fixed
• to help us focus on the right priorities
• to shift us towards looking at the end-to-end experience
• to understand the most important problems and prioritise work
• to understand the effectiveness of work we do.

After the survey analysis was complete, I ran a workshop with the Product Managers, UX designers, and Delivery Managers to work out how we were going to tackle this.

The workshop

Based on the above survey results, I decided to put together a simple approach for measuring sentiment, and get feedback on it during a workshop.

First, I jumped on Google to see what other researchers had to tell me about measuring sentiment.

“Traditional” sentiment analysis seeks to understand what words people are using to describe your product, and what that means. Usually, you’d be looking at things like customer feedback data, or social media, and some kind of AI wizardry would aggregate that for you. At that time, we didn’t have anything that could do that neatly. Secondly, customer feedback data and social media can be overly skewed positive or negative.

I put that thought aside for later, and tweaked my search to see if there was anything “UX-y” I could do — such as any UX research metrics, or other ways to solicit users’ thoughts that addressed the research objectives as shaped by the team.

I happened across this article, which summarised nicely how one could capture user sentiment as part of a usability test, using a combination of emojis and Microsoft Product Reaction Cards (MPRCs).

I also looked at Nielsen-Norman Group’s article on the cards, and reflected upon my own experience of using emojis and language as a way to gather sentiment in a diary study. It had worked pretty well then — we asked users to choose an adjective and an emoji to describe their experience.

So, based on what the team wanted, and what others had done before, this is what I came up with:

Conduct face-to-face usability testing, incorporating the SEQ, SUS, and SUPR-Q, and throwing in some to-be-defined sentiment exercises using emojis and the MPRCs.

I figured that doing face-to-face research would enable us to understand the “why” behind the what. Secondly, if the sentiment portion was a bust, we’d at least get a usability baseline out of it.

An on-site survey to collect SUS and SUPR-Q, collected at key interaction points on the site

The idea here was to keep it simple with the collection of metrics, building on previously collected SUPR-Q data. Collecting it at key points across the site meant we’d get a nice cut-through of what people were doing on-site.

Looking at aggregated customer feedback data

Given we already had this data, this idea was popular, but we didn’t have a neat way of pulling this all together and analysing it.

During the workshop, I ran through the survey results and the above approach, asking people to vote on and talk through what approach they felt would work best.

It was decided that we do the face-to-face research first. Then, we’d use that as a basis to survey a wider audience, collecting the SUS and SUPR-Q, and asking the same sentiment questions.

The research

At this stage, I wasn’t feeling 100% sure that this would work. This approach *did* address the research objectives, but I’d sort of smooshed together some methods as a result of my own experience, along with some magical Internet resources. It wasn’t “tried and true”.

So, I ran this past my boss at the time. He also jumped on Google and came to much the same conclusion I did.

In an earlier conversation, he’d told me “You know all the user research methods out there — perhaps it’s time you started to break some of the rules”. I figured that was as much of a blessing as I was ever going to get.

I began to flesh out the interview guide for the face-to-face research. For the sentiment portion, we’d ask participants to complete two sentences about how they felt about SEEK (e.g. “SEEK makes me feel…, I would prefer to use SEEK if…”). Then, they’d choose 5 words from the MPRCs, and choose an emoji that best described their feelings toward SEEK.

After that, they’d attempt the usability testing portion, looking at 5 key tasks across the site. The metrics collected here were the SEQ, SUS, and SUPR-Q. I recruited a mix of people who used SEEK as their primary job seeking site, and people who hadn’t used SEEK, or used a competitor as their primary job seeking site.

We’d repeat the sentiment exercises after they had completed the usability portion of the session, performing the defined tasks on the SEEK website. I had a hypothesis here that usability would impact on sentiment.

After the face-to-face rounds were completed, I put the SUPR-Q and SUS surveys up on the site as intercept polls. I also put up the MPRCs and the complete the sentence exercises as a way to collect some qualitative data. 2000 responses later, the results were much the same as what people told us in the face-to-face testing (and I never wanted to look at Excel again!)

So…did it work?

Kind of.

We found that because SEEK is…well, pretty usable, it didn’t have a huge impact on job seeker sentiment toward SEEK.

With regard to the MPRCs, the words participants chose tended to be fairly practical and functional. A lot of the other data we collected reflected things we already knew about what job seekers found valuable on the site, and what they wanted in addition to that. These things played into sentiment.

Secondly, despite the team wanting a view of sentiment toward SEEK, and not how job seekers feel about the job-seeking experience — we found the two were inextricably linked. No matter what, a person’s sentiment towards a means of getting a job is going to be related to their experience of getting a job!

What next?

I’d like to run this research again, to see if the things people raised as issues have changed, or are still true. Our product stream seemed to get the most value out of seeing verbatim from users, and seeing their responses to what exists at the moment, and what’s not yet on the site.

However, I don’t think I’d run it as a usability test sandwiched by sentiment again — it’d likely be a contextual research piece to make the experience more “real” for the user. I would still use the sentiment exercises in some way, and follow it up with on-site surveys again. I think it’d be useful to do this sort of thing every 6 months — tracking some of the negative responses, or “warning signs” and see if they get any bigger.

Remember how this metric was intended to be rolling? We can’t really run the surveys on-site constantly — that would slow down the site experience and be way too much data to sift through. We’re still working on a way to aggregate all of our customer data into one spot. When and if that happens, expect to see another article!

With thanks to Mark O’Shea, Leah Connolly, Kayla Heffernan and the Candidate product stream at SEEK. Y’all are awesome.

Bending the Research Rules: Measuring Sentiment was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In August 2018, I presented a 10 minute talk at UX Australia. My talk focused on a pattern I’d observed emerging among UX researchers and teams — that UX research sometimes felt a little unloved, and difficult to execute. I wanted to share some tips and tricks I’d learned to help UX research flourish.

This is the written and slightly expanded version of my talk. It comes with a great SketchNote summary by Judy Shaul.

The audio can be found here: http://www.uxaustralia.com.au/conferences/uxaustralia-2018/presentation/from-seedlings-to-sequoias-nurturing-ux-research/

So let’s talk about UX research. Do you work in an organisation where…

• UX is relatively new?
• UX research is unheard of?
• You have to justify the need for UX research?
• Or, the size of your UX practice has exploded?

I imagine some of you are nodding along at this point — but our tale of woe doesn’t end there. Say you’ve cleared the above hurdles and you get some research done. Lucky you! But at the other end, maybe the research findings get ignored, or forgotten about. Then, your organisation hires even more UX folk with varying levels of research experience…

I know what you want. You just wanna do good UX research.

So how do you do that? By following these steps three:

1. Plant the seeds
2. Fertilise (…feed) your saplings
3. Propagate the trees

1. Plant the seeds

Get your research sprouting with tools and templates. Planting the seeds has three sub-steps:

• Create the basics
• Talk to people about it
• Build a crew

Create the basics

If you care about getting research done, and doing it well, this makes you a research champion. It’s up to you to provide your team with a toolkit to help make research happen. Don’t make your team think any more than they have to.

What kinds of things might such a toolkit include?

Perhaps a template interview guide, some consent forms, a research brief template (so your team can document why they are doing the research, and who it’s for), and a set of ethics (protect your team and your participants). Dream big.

While there is much more you can include in such a toolkit, the above things form a foundation. They give people in your team (particularly new folk) something to bounce off. It’s a lot easier starting with something as opposed to staring at a blank Word document — how scary is that?

Secondly, creating a toolkit means the team now has a standardised way of communicating — your stakeholders will come to know what to expect from UX research, and UX research develops a brand of its own.

It also means any research programs of work are:

• Understood 1, 5, 10 years from now
• Repeatable — particularly important for usability baselining or building on a set of themes (building a picture of your users)
• Reportable — there’s not much point in conducting research unless the approach and findings are documented somewhere, somehow, so that they can be referred back to

Lastly, you may want to outline some common forms of UX research as a guide — this gives your team yet another springboard and helps your stakeholders understand why guerrilla testing isn’t always an appropriate method.

Talk to people

Great. So you’ve got the beginnings of a toolkit, and some guidelines on what kind of research to do. Now check that it works.

Sit with a few people in your team and run them through the templates you’ve created. Do they have research coming up? Ask them to trial the toolkit for you.

Once you have people in your team trialing your toolkit, you learn what works and what doesn’t work. It could be that your research brief is too complex, or that your team wants to understand how to use usability metrics.

That’s OK. You’re in UX. You know what UXers are really good at? Iterating things! You can learn how people are using (or not using) your toolkit and adapt it so that people will use it. Then test it again.

Build a crew

Your toolkit guinea pigs now become your research crew. Once they’ve used and contributed to the creation of the toolkit, they’ll tell people about it — then those people will start using it too.

Use your crew to help you build and refine the toolkit over time, and then work together to help define other processes like remote testing tools, ethics frameworks, and equipment use. This is especially useful if your team doesn’t have a dedicated UX researcher — your team can work together to tackle the big research process problems in short bursts.

Seeds planted.

2. Feed the saplings

Make sure that the research conducted is of a high quality, and appropriate for the problem or gap its trying to address. Feeding the saplings has three sub-steps:

• Coach your team
• Coach each other
• Use your resources

Coach your team

Run some practical coaching sessions for your team. These could be things like questioning and responding techniques, choosing an appropriate research method, usability metrics or how to synthesise data — whatever you’ve heard your team asking for.

You could run these yourself, get someone external in to do it, or even look at some articles/videos and discuss them as a team — a sort of book club.

Coach each other

Now that your team have the basics down, continue to make yourself available for them to ask questions, and bounce ideas off you.

Keep coaching each other by observing each other’s research sessions, checking your scripts, and providing feedback. Remember that crew you built? Use it!

We often engage in design critique. It is equally important to engage in UX research critique. Check your approaches, scripts, technique, and reporting with each other. By doing this, you foster a culture of peer review in your team. Secondly, you ensure any research outputs are of a high quality — two brains are better than one! Peer reviews also help us to recognise our own biases and thought patterns, and ensure we learn something new, or look at things from a new perspective.

Use your resources

By now, you may be thinking “This is all very well and good, but I don’t have a dedicated researcher in my team!” You may not have a tame UX researcher* on hand, and that’s OK. You can still build a strong culture of research…with The Internet™️.

Use your networks to find great people to chat to about their research practices. Use The Internet™️ to look up research methods, recommendations, articles, usability testing tips and tricks (find some at the end of this blog post!). There are plenty of digital and paper resources out there — use them. If you’re struggling with the best way to tackle a problem, it’s likely that someone else out there has tackled the same problem, and done the thinking for you.

Now, your research saplings are loving the attention you’re giving them, and are being fed with knowledge. They’re thriving.

3. Propagate the trees

Spreading the Word of Research.

• Co-create from the get-go
• Put your stakeholders to work
• Workshop it

Co-create from the get-go

Get your stakeholders involved in your research project right from the start. I’m not talking about note-taking — think waaaay further back than that.

Co-create the research brief with them. That is, the research objectives (what you want to find out), potential questions to ask, the audience, and the method. This is useful for not-well-defined exploratory pieces (you can really dig out assumptions and deep-seated beliefs here) as well as faster-paced concept validation.

This part can sometimes worry dyed-in-the-wool researchers (what?! Letting stakeholders give you ideas?! But the leading!) — however, your stakeholders are much more likely to buy into the research findings if they’ve had some hand in shaping the objectives. And hey — hypotheses are there to be (dis)proven.

Secondly, it’s important to have people with different perspectives in the room. You’ll build a better product if you have people of different disciplines working with you and sharing their ideas. Research does not exist in a vacuum — it has to play nicely with technical feasibility and business viability, as does UX design.

You can pull out your stakeholders’ hypotheses and assumptions by using:

• Assumption Mapping
• a Lean UX Canvas
• a group brainstorming exercise
• a Business Model Canvas

Or literally just paper and pens, you don’t need to be fancy.

Put your stakeholders to work

Continue to keep your stakeholders engaged by asking them to take notes for you. Stress the importance of note-taking as reducing synthesis time, thus getting your team to the results more quickly. You can email your team with a sign up list, or stick a physical sign up sheet somewhere near your desk (I once laminated mine for an easy wipe n’ reuse).

You’ll need about 15 minutes before each session to brief each stakeholder on the content of your interview guide, and how to take good notes.

After the session, ask for their thoughts and what they found interesting/surprising. Go through their notes and make suggestions on how they could improve (provided they have asked you for this feedback!)

It’s important to remind your note-takers that observing user research is a privilege — we’re not there to make friends with participants, or sell an idea — we’re there to learn.

Workshop it

Once the research sessions are complete, run a debrief and synthesis session with your note-taking stakeholders.

Use an hour — spend about 20 minutes discussing what sorts of things they observed in the session. Finding commonalities or surprises helps cement in their minds the things they saw, and also broadens their viewpoint from just one or two sessions, to a big picture of common themes.

In the next 20 minutes, give each stakeholder a set of notes (ones that they did not write), a Sharpie (every designer’s fave tool) and some sticky notes. Ask your stakeholders to write down one observation per sticky note, and stick them up as they go, reading each one aloud.

As more and more notes are put up, get a few helpers to start affinity mapping these — this gives your stakeholders some ownership over the eventual synthesised findings, and ensures what they observed from the session really sinks in.

Lastly, when your synthesis and reporting is complete, share the insights via a workshop with your broader team — quickly run through the findings, and print these off. Then, ask your workshop attendees to sketch up concepts or ideas based on the findings.

All you need is one person walking away saying how interesting the session was — the rest will follow. Some never will — but that’s their loss.

In summary…

Keep in mind that nurturing UX Research takes time. It is a SLOW process — don’t expect an attitude change overnight. Each step takes a few months to embed itself. Rely on your research crew to keep using and talking about your tools and techniques as outlined above, and it’ll soon become the norm.

Secondly, there will be instances when you simply don’t have time to complete all the steps above — we’re all human, we all get busy! Keep in mind that doing some of the above is better than nothing, though. You’re still making a difference.

Here’s those resources I mentioned…from The Internet™️…

Note-taking and Observer Guidelines: https://www.nngroup.com/articles/observer-guidelines/

Some research templates: https://uxmastery.com/resources/templates/

Making a UX Research plan: https://www.smashingmagazine.com/2012/01/ux-research-plan-stakeholders-love/

Sharing research: https://medium.com/facebook-research/beyond-bullet-points-four-creative-ways-to-share-research-c10fa047f025

Even more sharing research: https://blog.prototypr.io/ux-research-is-boring-and-nobody-reads-it-668edbfc804a

Affinity Mapping: https://www.nngroup.com/articles/affinity-diagram/

*When I say tame UX Researcher, I am making fun of myself.

With thanks to Danya Azzopardi, Kayla Heffernan, and Judy Shaul.

From Seedlings to Sequoias — Nurturing UX Research was originally published in SEEK blog on Medium, where people are continuing the conversation by highlighting and responding to this story.