Chapters
Shownotes Transcript
This is the Nielsen Norman Group UX podcast. If you know anything about us, then you know that we love research. User research, to be precise. And our research has shown that understanding customers isn't optional, but essential if you want to succeed as a business. That said, research is hard to do.
harder to refine, and even harder to manage at scale when you have many people contributing and using that research. But there is hope. Today, you'll hear a conversation I had with Cara Pernice, Senior Vice President at Nielsen Norman Group, about how to manage research processes and insights so that they actually impact the designs that we are working so hard to create.
With that, here's Cara. Hi, Cara. Welcome to our podcast. How are you doing today? How was your weekend? It was fabulous. I actually did something amazing.
really strange for these days, which is we, my family came over, we had like 11 people and we celebrated my birthday early. And I actually started crying when they had a birthday cake. Cause I was like, it's been so long since we've gotten together like that. And I, I just felt really like, I can't even believe this is happening. And I really appreciated it. So. Yeah, that's right. Your birthday. I just remembered. Oh,
Happy birthday. Your birthday is tomorrow. That's very exciting. Well, happy early birthday. Thank you. And happy belated birthday to you because you just had a birthday. Yes. Thank you. I appreciate it.
Well, awesome. Well, yeah, I'm glad your weekend was great. And thank you for joining us today. I'm curious because I feel like I know bits and pieces, but I would love to know the full story, the full scoop. How did you get into UX and what brought you to NNG specifically? Oh, man. How much time do we have? Yeah.
I got into UX in a very strange way. The year was 1991. And when I graduated from college in Boston, the cover of the Boston Globe said, 91 graduates, no hope. And I know I majored in communications and public relations. And I was about to take this job at a communications agency on Newberry Street in Boston. And my sister, Susan, was working for a really awesome company, Lotus Development in Cambridge, and
And she said, Carrie, you can't take that job. You have to come work with me. This company is amazing. And she made it her mission to find a job that would maybe fit me, someone who had taken a few psych and a few programming classes, you know, and found this job that
asked that it required someone who had a master's in cognitive psychology and some, some other things I had never heard of before. I had no business going for this job and I can't even believe I got it after I think eight interviews, they basically hired me because I could write and I knew how to interview people from my journalism background background. And, and
And then my boss at the time, Mary Kate Foley popped up, like just put a whole bunch of papers on my desk and said, read these. And the top one was by a guy named Ben Schneiderman. And the one below it was a guy by the name of Jakob Nielsen in Denmark. And I was like, wow, what is this? So he's sort of interviewing people, but not really. He's kind of watching them. And I kind of got sucked in and just started doing research. And I've been doing it ever since. That was 1991.
and that was that. I ended up at Nielsen Norman Group by way of probably Sig Kai and some other groups. I used to see Jacob at different conferences, and then I actually, he was my idol. He was my screensaver for a while on my computer. I don't know if I've ever told him that. And yeah, and this person, Mary Beth Redker, he was in the Boston area, and he was
asked for a tour of our usability labs and Mary Beth wanted to do it, but she had a really big meeting and she said, I know you really, you know, are really interested in meeting Jacob and he wants a tour. And so I gave him a tour. And after that, we kind of got to know each other and he asked me to review some papers for CHI conferences and things. And then he asked me to come work for him years ago, 20 years ago now. So yeah, it's been quite an interesting journey. Yeah.
You mentioned SIGCHI. What is SIGCHI? Just so we can spread the word. Oh, it's a school. Yeah. Thanks for asking for me to expand on that. It's kind of an older organization. It's still around, but it stands for Special Interest Group for Computer-Human Interaction. And it was where we used to nerd out a lot back in the early 90s. That and UPA, Usability Professionals Association, which has since been renamed UXPA. So we always had these kind of support groups
in UX back in the day, and some of them still survive, but many more today. - Yeah, I can imagine, especially in the early days that it was probably not a lot of folks were in this field. So getting a chance to really talk about the work that you do and not just the work that you do, but being able to explain that work to others, I imagine that it was really great to have that support.
Yeah, it totally was. And it was, it was like, it was interesting, Therese, because back then it was kind of like, it was kind of a support group because we were always talking about things like fighting the good fight, you know, and it's been fun to see over the years, the change that has happened. Like we still do kind of fight the good fight, but when you talk to somebody who's kind of newer to the field and doing the work, it's,
in UX, they're not really talking about that as much. They're just talking about how to do the work. And this idea that it's us against the developers or us against product is kind of, I don't think it's gone, but our mentality is
kinder and more open and more, I mean, we were all, I shouldn't say kinder because we always were trying to do the right thing back then, but it just was, there's more of an understanding of what UX is, which I think paves a road for all functional groups to work better together.
Yeah. You mentioned that there have been changes over the years. Like what have you learned about how teams do research specifically? Because I know you mentioned that research was something we're very passionate about even back then. What has changed over time? Yeah. UX research is done now.
A ton. I think a ton has changed. Like, well, I mean, back then we did, like I mentioned UPA, which is now UXPA, and we always shared our research methods. Like we do these talks and we'd talk about how you do international or how you share research or how you, how you catalog your research and databases and things. And
I think we were still sort of figuring it out and trying to... It was like throwing spaghetti against a wall, where today what's happening more is the spaghetti's already been on the wall. And now we're figuring out how to name the spaghetti and how to throw it better and how to make it stick. So it's like things like terminology and almost branding and systemizing our work has...
has occurred, which is exciting. Things like design thinking, like empathizing and brainstorming and iterating. We've always done that, but we didn't call it something and we didn't make it simple for people to understand, for people who weren't in our field to understand. Or like Lean UX, you know, like naming it something and prescribing an order in which to do things has really helped us
I think, grow and speak about what we do more knowledgeably in a way that others can understand. So I think that's huge. Yeah. And it's also a metaphor with the spaghetti because I think to take it even a step further, we're keeping the spaghetti fresh too, ensuring that we're updating research and making sure that we're revisiting older research to see what's still relevant, what's still good.
Yeah. Yeah, that's a really, really good one. I mean, the spaghetti, fresh spaghetti is important. You know, sometimes we want to check out that fermented spaghetti. One of the pet peeves I have always had in research is,
is leaving data on the table. Like when you learn or you do a research study and you write your research goals, let's say, or your research questions with your team, and we're always so busy that we will say we need to learn these three things. So we do, let's say, a usability test and we learn that stuff and we
act on it and we move forward. Well, when you do a research study, there's a whole bunch of other stuff that the user has done or users have done that a lot of times we don't
really analyze or discuss or track. And I remember when I worked for IBM, like I would have people asking me to do a research study and I'd be like, okay, we already have that data because we already did a study like that. Can you read this first? And then see, and it was like, I wouldn't say nine times out of 10, but a lot of times people would say, yeah, I guess I don't need to do this study because we do already know that. Or let's do this different study and use that one as a jumping off point.
And I think that's part of what we're calling research ops today, research operations, and, you know, really tracking those things in a way that can be shared and found again.
Yeah. What exactly is research ops if you had to define it concisely? Oh boy, you're making me do it concisely. Okay. Well, if you had to, how would you define research ops then? I think research ops is about the components related to doing research. And so it's basically looking at
how do we plan research? How do we refine and improve our craft of researching? How do we also fit research in on projects? How do we make sure that research and how we do it is actually going to thrive and survive at our organization and in the world? And how do we educate and teach others about how to do research
when we are doing research, what that research is and what we learned in research studies. So I feel like it's a lot of folks kind of think research is about administration, like admin type tasks.
And it certainly is like, you know, research ops will include your research repository, skeletons for doing a field study. What are the forms we use to make sure people have consented to participating in a study? So it's it is a lot of admin stuff.
But I feel like the ops part is even more than that. Like the operations, like I learned about operations back in business school when I was getting my MBA and they really kind of pounded in the history of operations and Eli Whitney and the cotton gin industry.
in the late 1800s and then later in World War II, figuring out how do we really make things easy for people and expedite their work. So I feel like when we think about operations, it's really about efficiencies and economies and ensuring that what we're doing is making people work in a way that's productive and fast
And with that, ensuring that we open up time for them for strategic thinking. So it's kind of the same principle that we talk about when we think about design systems. I know you wrote a nice article about that not long ago. And when you think about a design system, someone might first say, hey, you're going to make me use all these designs that you're taking away my creativity. But
But the idea behind it is really consistency for our users, but also these kind of simpler widgets could be designed and refined and even coded and coded in a good way and put out there. And then the designers are their minds, their time can be opened up to think about more strategic things. I think that's true with research too. Like,
Even just the craft of research and understanding which method to use when and how to deal with the politics of those times when you write a research goal and you start to design a study with somebody. And in the middle of that study design, someone might say, oh, and can you also learn how many people would want to do this?
You know, when you realize that is a research question that is not going to work in the study that we're doing. So we need to kind of track those things. So basically the idea of like the operations of research and we want to be able to have systems in place so researchers can.
Tackle those kinds of problems instead of having to spend their time writing a script or, you know, finding that form to make sure that we're, you know, getting getting consent in a in a an ethical way and a fair way. Right. OK, so essentially you're systematizing much like a design system systematizes the process of creating interfaces in that you have this system.
this place, this place could be a lot of things, but it could be an intranet or it could be, maybe it's a shared drive of some sort, but you basically have this place that you can use as a resource and you have things perhaps like the repository, like you said, the research repository, the insights and saving them for the future. But also what you mentioned, which is the administrative details that often come with administering a study such as consent form, such as
maybe screening surveys that people can customize to their needs. You got it. Yeah. So it's tools, it's capabilities, it's also your budget, it's also your roles and your staff and communication. So it's really everything. It's really everything related to the system of doing user research. Okay. So given that it is research ops,
Who should be in charge of research ops? Should it be the researchers or since, like you said, there's so many other moving parts, you've got budget, you've got the people in charge, you've got the actual maybe organizational structure. So who should be the person to oversee that? Or is it many people who oversee that? It's a it's a great question, Teresa. I don't I don't think there's like one answer to that.
When you think about it, some organizations don't even have a researcher and some have tons of researchers that could be organized in a matrix way or a central way. And so it's going to really depend a lot on the organization, its UX maturity, its size, its roles. I think in an ideal world, you're going to have people whose job is research ops, like managing research ops.
They may even do something even more granular, like manage an insight repository. Like that's all they do is take these insights that we learn about how people think over time and they may manage that. So I think in an ideal world, we've got a
pretty big staff that can really someone be in charge of the research repository and someone be in charge of some of these other things but I don't think that happens all that frequently so what what happens pretty frequently I know from our classes is we'll have people who are just a team of one and they're supposed to be doing
interaction design, information architecture, visual design, research. And to them, they may feel like, oh my gosh, the idea of organizing research ops will put me over the edge. However, it's usually going to help them. So even a team of one should find ways to be efficient with their work, to reuse something they did well, right?
And then as an organization grows and a team grows, you should have roles, people who are responsible for doing these sorts of things. Okay. Yeah. You mentioned design ops. I remember sitting in on that class fairly recently as well. And
It is interesting thinking about how design ops itself can be its own functional team, of course, depending on the size of the organization and how many UX professionals are in that organization. So I guess my next question is, how does it relate or not relate to design ops? Since it seems like there is already, and just for some context, folks listening in, design ops would be the operational team
aspect of doing design, how we do design the people behind those designs, as well as the processes behind those designs and really breaking that down, designing that very process, which is pretty meta. So on that note,
Where does research ops fit in or does it not fit in? Yeah, yeah. It's a good question, Therese, too. I think like I actually kind of defer a bit to my colleague, Kate Kaplan, who's been doing so much research on design ops and teaches a really good class that we offer on that. And we've talked about this and she believes, and I'm with her on this, that research ops is really a
a part of design ops. Because if we think design operations is really to, is enabling consistent quality design, really, I feel the only way you can do that is with sound, thorough design.
good research. And so that to me is really, it's really just a part of design ops. You know, she calls like design ops, like I think she calls it a curated potluck dinner or something like that. Yeah. And so I think that we, you know, I think that we're going to be like some of the main courses on that potluck table, research ops is. So it's a part of it.
Okay, that makes sense. And yeah, I would definitely not recommend trying to, you know, design your process of designing if research is not part of that process. So yeah. So I guess on the topic of things I would not do, and maybe things maybe you would not do, what are some common research ops mistakes?
That you see. That's a great question, Therese. So I've been talking to quite a few practitioners recently about research ops and what they're doing and not doing in preparation for a class that we're developing called Research Ops. It's going to debut on September 1st this year.
this year, 2021. Woohoo! Yahoo is right. And some of the things that I've seen, I don't know that I'd call them trends, but I'd certainly call them themes, are things that folks are not doing that I think would help them. And one of them is not creating a research roadmap for
They kind of like what will happen is folks will say, well, we're working in agile and I just kind of as needed, I will do research. You know, I'll just we'll do it as needed. And when we can get people to kind of listen and that just doesn't really work. Right. That just doesn't really work. What what you really want to do is talk to your product owner or whomever is in charge of scheduling, basically, and figure out what's
where you can fit research in. And what I like to do is think about which kind of areas and features are most important to the business, to the organization, and to the users. But then I like to look at another plane. And that plane is how hard do we think this thing is going to be to design? You know, you never really know that. You have to make an educated guess on that. But an educated guess is better than no guess at all.
And so then you kind of decide based on the importance and how difficult something might be to design and try to put those design phases, those design sprints, whatever you call it at your organization, try to put them early on and well before the development sprint or before anyone's going to write an inch of code.
And with that, you've allowed yourself enough time to actually iterate the design. You're not creating these blocks that we create all the time or we accidentally create or are created for us that make it impossible to do multiple rounds of research or to get anyone to listen to what we learn in research.
So we want early research on stuff that's important, that's hard to design. And we want to iterate that design as many times as we can until and then have people write the code on it. And by the way, the developers, anyone writing code should be as much as they can involved in the research as we go. So I feel like that's a pretty big mistake. It has to do a lot with
planning and scheduling. And I think that that's one of the hardest parts of doing UX. I mean, doing UX research is that idea that I have to convince someone to do the research and to listen to me. And it's less frequently about budget. It's more about time. So I think schedule, I mean, no roadmap is a big issue. And another one which is kind of related to that is really
trying to design research ops or even just a research repository, trying to kind of design it either alone or with just the UX folks.
Because then what happens is it's only a good tool for the UX team and everyone else might be working in, you know, whatever. So, you know, Airtable and we decide to work in Dovetail or we're working in Considerly or something like that. And no one like knows that tool and they don't go to that tool or...
or maybe they're not well integrated. And so it makes it like even harder. It puts up a barrier that doesn't really need to be there. So I think doing it alone is another thing that is important.
kind of problematic today. Yeah, I love that point you made too about the tool sometimes itself being the barrier where maybe somebody is interested in the research findings, they're interested in learning, but as soon as they encounter this new tool, maybe they suddenly feel like an outsider and like this isn't intended for them. And even if they are given the privileges to access that, it's still
another tool that person now has to learn. And that can be intimidating and limits kind of who gets exposure to that. Exactly. Totally. So if you had to share maybe a couple approaches that someone could take when putting together something like a research repository or research artifact repository, what advice could you give when starting to create these types of systems?
I think like choosing a system, using a system is about looking at the content. It's like designing a website, actually. It's like looking at the content that you have and the content you believe you should have and then evaluating the tool based on that.
So we know for research, we need things like scheduling. We need communication about when we're going to do research, how and when that's going to be done. We need to track our overall reports, but then we need to track separate insights.
and maybe even prioritize those things and then give a status about them. So tagging is massive. Like, you know, dovetail is really cool for that kind of stuff. Once you kind of learn it, there's some, you know, some tricks to all of these tools that you have to overcome. But there are a lot of different tools that I think will,
they'll work today. That's one of the reasons I wasn't dying to talk about tools because they just change so frequently. And there are so many different features on all of them that what we say today might not be true in three weeks. So I think looking at, you know, what you think you need from a tool is what's most important.
Right. Totally agree. And I guess for those folks who are now looking at their research process, maybe thinking, I've never thought of my research process this way before. How do you even start when trying to improve a UX research process? What would you advise?
It's hard to be self-aware, isn't it? Like you don't necessarily know what's wrong with your current process if you're the one who kind of created it and is living in it. I think a little bit of outreach within your organization with your designers, developers, product, whomever else might be involved to figure out what is it that they seem to know and not know and need and not need.
And some of that's going to come from asking them and some of that's going to come from more observing, you know, just like we know that's how we do research. We ask questions and then we also observe. So I think some of that could be really helpful to figure out where the gaps are rather than kind of looking out there and going, okay, what's the perfect example? You know, how does Facebook do this? That's not necessarily going to apply to you. So I think looking within trying to figure out what works and doesn't work and
and try to find some places where it's gonna make a big impact. Like if you know, we're doing tons of research and it seems like people aren't really involved in it. And one of the reasons is either don't understand it or they don't know when it's happening.
And maybe you focus there first. You focus on making an easy to understand schedule and writing quick summaries, boilerplate of what these basic research methods are that we do. So people can, you know, maybe watch a video. You can make a video of it or something like that. I think find some pain points right now and try to fix those to start. So you can see some wins and then also folks, stakeholders around your organizations can see some wins and understand why this is worthwhile.
Yeah, I think that's helpful. I know that a lot of folks like me tend to see something like this, get really excited and perhaps a little overambitious.
And to use a metaphor that one of my colleagues used at one point, trying to remember exactly who it was, but the metaphor was don't boil the ocean, right? Try not to, you know, completely revolutionize your research process, especially if, you know, there is a good standard process, maybe identifying some of those key moments can lead to the biggest return on that investment of time and effort. Yeah, totally. Yeah.
I think it's true. Boiling the ocean is too much and it'll make you crazy. You want to be happy too. You want to be able to feel like you made some progress. And that's an important reality to doing this kind of work. If you're trying to really make a big change, you know, you're going to be a change agent in your organization. You need to feel good about it and you need to keep your energy up. And if you keep feeling like you're failing, you're
you're not going to do a good job. So make it possible. Yeah. Make it possible. Achievable goals. Yeah. And, and make it enjoyable to kind of make it something that, you know, you enjoy contributing to and that others enjoy contributing to so that it's something that ultimately can be useful to others. And yeah.
Totally. I mean, I think like what's cool about that idea, too, is like it's kind of easy to make research findings enjoyable by just opening up the world of what users do and say. You know, like people say the craziest, funniest, interesting things. And they you know, if you've never done a usability test with your team.
you know, the first time you do that, you should just cherish it because they're going to love it. They're going to be like, wow, can you believe that? And this person is really someone we would be selling this to. And I can't believe what they just said. I never would have thought of that before. And, you know, just pulling out interesting quotes and interesting things they did. Like one of my favorite things used to be to grab a huge stack of videos, like, and these were videos, these were NTSC tapes that...
I would grab like 20 of them and rewind them all to the same point. And we do what we call the master design studio. And I'd have everybody just watch like three minutes of 20 different people doing a task. And then we'd start sketching. And this was in my
93. And we do stuff like this today, but our technology is better and we're much better about organizing it and talking about it and things like that. But these things make it so fun and it makes it fun for the team to move forward. And it also makes them more confident. If you start to do a design and you go back and look at a couple of research studies right before, you feel really grounded. You feel really like, okay, yeah, these are the
that I need to solve. And it humanizes it too. Yeah, I like that. Thinking of it as a tool of empowerment, not just for the researchers, but for everyone who wants to solve a design problem and feeling like they are equipped with resources
the information and the evidence that they need in order to make the best design possible. Absolutely. Yes. You're so articulate, Therese. Thank you. You are. So, Cara, it's already time. I can't believe that time has flown by so fast. Me neither. Wow, Therese, you made it fun. Thank you. Not scary at all. I'm glad to hear. Well, if others would like to follow you and your work or want to learn anything else about research ops, where could you point people to?
Sure. Well, I'm on LinkedIn. I'm pretty active on LinkedIn. Feel free to find me there. I'm on Twitter, but less is active at Cara Ann there.
For ResearchOps, we'll certainly be writing quite a bit. We've got our class coming up and there's a whole community of people that have been really moving this field forward. The Cool Slack channel and the leaders there, Kate Tousey, Ruth Ellison, Bridget Melser, Holly Cole, Emma Bolton. There are tons of people that have been doing awesome work in the ResearchOps area. So you could also check their blogs and their work out too.
For sure. And we'll include all the links to that in the show notes. So check that out if you want more info. But otherwise, we're looking forward to your Research Ops class debuting September 1st. So if you are interested, definitely check that out at our website, nngroup.com. We'd love to have you there. Awesome. Well, thanks, Kara. Have a great rest of your day. And you, Therese. Thank you.
Thanks for listening to episode 12, the last episode of this season. Next month, we'll be starting season two of our podcast, all thanks to you, our listeners.
We are so grateful for your support. And if you want to show that support, please leave a rating and subscribe no matter what platform you are on. Also, you can find all of our reference courses and links in the episode notes. But if you want to get updates emailed to you on articles and videos that we publish or any upcoming courses as they get released,
Please sign up for our free email newsletter on our website at nngroup.com. That's n-n-g-r-o-u-p dot com. Until next time, or next season, and remember, keep it simple.