34. Data-Driven Decision-Making and Intranet Design (feat. Christian Knoebel and Charlie Kreitzberg, Princeton University)
43:24 Share

This is the Nielsen Norman Group UX Podcast. I'm Therese Fessenden. As we start a new year, we often like to reflect on our progress and improvements. In fact, at the end of this episode, we're going to share some exciting news about improvements we plan to make to this podcast, so stay tuned for that. But when it comes to resolving to improve in general, there are just so many ways to do that.

Last month, Dr. John Pagonis talked about how important it is to combine qualitative and quantitative research and to close the loop. Today, we're going to feature a team that did just that and how it led them to not only improving their public-facing website, but it also won them a Nielsen Norman Group Intranet Design Annual Award, which recognizes the world's best intranet designs.

Now intranets, meaning internally facing websites for employees or other organization members, these aren't often the subject of most design awards. And I get it, it is fun to award the splashy public-facing websites. But these intranets are incredibly challenging to design given how many different kinds of users they often serve.

and the resources that are often available to rise to those design challenges. Often it's pretty slim. So we're excited to share an interview we had last year with two members of Princeton University's award-winning intranet team, Charlie Kreitzberg, Senior UX Advisor at Princeton University, and Christian Noble, Princeton's Director of Digital Strategy. ♪

In this episode, my colleague UX specialist Tim Neusesser interviews Christian and Charlie, and they share with us how they got involved with UX, what they think makes their team successful, and how they created two complementing frameworks that allowed them to make the most of their mixed methods research and further, make some pretty incredible design choices.

Yeah, thank you, Charlie. Thank you, Christian, for being on the show today. It's really an honor to have both of you here. And maybe we can start off and you can just tell us a little bit about yourselves and how you ended up at Princeton University. Okay, perhaps I should jump in first.

I followed a very tortuous path to get there. And it goes way, way back to the 1960s when computers were giant things with disks that were 36 inches around. And I was 16 years old and I was a music student actually. And I went up to Dartmouth College for the summer

And the first night that I was there, I locked myself out of my dorm room and started wandering around the campus looking for help. And I went to the only building that had a light on, which turned out to be the computer center. And I had never seen a computer before in my life. And I was absolutely gobsmacked by this thing. I think it was instant love.

And I really spent the rest of the summer teaching myself programming, working on the computer there, which was open to all the students. Actually, Dartmouth had created the first computer literacy program in the United States and invented a language called BASIC, which you could teach yourself. And so I started doing music on the computer. And that really led to a career in computing.

I went on to get an undergraduate degree in computing and a graduate degree in computer science. But the thing that I really would like to share with you, Tim, here is that when I started working with computers, it actually changed my mind. I had a different way of looking at the world, and I wanted to bring that to other people, and that's what brought me into UX.

I was just so excited by this technology and the effect it could have on people that I decided to switch into psychology and I went to get my doctorate in cognitive psychology so I could begin to explore the relationship between computer technology and people.

people and that was basically the path that I took. When I graduated, I started my own company and I ran that for 25 years. We did UX in the 1980s and 1990s and into the 2000s. So it was just a very exciting time and we were working really hard to teach people about this. It wasn't until 2016

when I really kind of burned out running the company and I said, "I don't want to be doing this anymore." And this opportunity at Princeton opened up and the opportunity to work in such a great institution was something I just wanted to grab. And so that's when I came in and met Christian. Christian, how did you end up at Princeton University?

Well, I think if Charlie's path was meant to be, mine was definitely not meant to be. You know those squiggly lines that we always think about when we think about career paths? I think mine has certainly been like that. I started in media, actually working for newspapers. Well, when newspapers were at the very, very beginning, the late 90s.

transitioning to the web. So I started at the Wall Street Journal, got hired there as a senior editor in the Wall Street Journal is what was then the online journal, but still today, deweysj.com, was in its first year. And it was great because it was the Wild West. We were making stuff up every day. And I remember, I really think back on those days very fondly because everybody there just trying to publish the news in this like...

like really brand new, very untested format in mode. None of us knew what we were doing. We're making it up. It was awesome. It was a lot of fun. Eventually, well, I stayed there for about, I don't know, eight, nine years, went to the New York Times, also worked for the Philadelphia Inquirer. In fact, actually, let me go back to the journal. It was, the journal had invited Jacob Nielsen to come in and talk to us about usability and

We were in such a nascent stage. Any information we could get was like gold for us. And I just remember sitting there listening to Jacob and thinking like, oh, wow, this is great. This just really opened my eyes up to this field called UX. And I've been, I want to say a fan or a lurker sort of ever since. And when I came here to Print

Princeton, met Charlie, all my UX dreams came true, I guess, so to speak. Because here was something I really kind of loved but was not practicing. And to be involved with

a UX team and to really now be able to learn about it through actual, you know, doing was just a really a lot of fun and kinds of work that Charlie and I have done together, me from sort of a product side, because really, I think, you know, we didn't call it back in the day when I first started working in, you know, the internet.

And it certainly was not, you know, the internet was not a sure thing in those days. We didn't really call it product, but I think as time has gone on now, now we do see the kind of work that I do do as products. So to have this real appreciation for UX and,

meet Charlie, who was doing UX and both of us coming together kind of as a team and trying to attack the same problem from slightly different angles, I think has been just a really terrific experience and a lot of fun. And hopefully we're doing good work and hopefully it's something that our users really love and we're serving them really well. So that's kind of how I got here.

When I first came to Princeton, I met Christian and another one of our colleagues, Jessica Monaco, who's also a wonderful UX designer and visual designer. And our first project was actually to work on an intranet, and that project did not succeed. We got started, but it never really came to fruition. But it was something we really, really wanted to do.

And I'm not exactly sure why we tend to spark so well off each other. One reason is I think we're both very direct and honest people. There's not a lot of facade, at least when we're, we'd like to think we have a little bit of political savvy, but the reality is we tend to say exactly what we think. And sometimes we think very different things and that's an enormous strength.

Because nothing makes me happier than when I field an idea and somebody says, I have a better way of doing it. And that expands my mind in another way of looking at things.

So that's been a very productive and very pleasant relationship for now these six years. - It's really admirable what you were able to accomplish at Princeton, and Charlie, you mentioned the intranet that first didn't come to fruition, but how did you reach that from first starting this project, which didn't come to fruition, to actually then designing one of the best intranets in the world? - Well, I think Christian really was the motivator

on our second attempt. He spent two years going around the university, interviewing people, trying to understand what people wanted. And I kept coming to him and saying, I want to be part of the team. I want to be part of the team. He kept saying, yeah, well, we'll get there and we'll get you in there. And it took two years. But I think one of the things at core is that both of us are absolutely committed

more than anything else to the products that we're designing and producing. We want to create beautiful, excellent, wonderful products with wonderful user experiences. And there's really nothing that's more important than that. And that, I think, is one of the things that holds us together and has made this work. And I should also say that on the Internet project, we were just delighted to receive the award, but

I should also say that it wasn't just us two. We had a wonderful team that we were working with from the Web Development Services Group at Princeton

Jessica Monaco was also a designer there, and we had programmers and phenomenal project management. Also, my colleague Susan Sparigan did a lot of usability testing and supported us with some of the research. It really was a group effort, and the team was just completely aligned around this and

And I don't remember one meeting where we had, you know, conflict or, you know, an unpleasant experience. We just, whatever the issues were, we addressed them, we worked them through in a constructive way. And I think that really did make a big difference in the end product. Yeah, I couldn't agree more because, you know, it is a team effort. And I think what's really critical is

And this applies to the work you do early and then how you make that work a reality. And that is to keep your ears open, to really, really listen, let somebody else speak, keep your own mouth shut.

So for all the research work that I did, I really made sure that I got to every, not every, you know, in a quantity, but certainly in a quality, from a quality aspect, the different user groups that, or customer groups that we were, we had to serve. And just let them tell me what,

We had a little method. It's not like it was just completely freeform. We had a method, but to just get them to talk about their pains and their gains, as we typically talk about on the product side, and really find out where we needed to play this integral part. And that's because there was no doubt

There was a need for an intranet, but there was no demand for an intranet. And we knew we had to approach this as if it was a brand new SaaS product that say, I mean, we're at least analogous to that. Nobody knew they needed until it was, you know, they saw it and it was put in front of them. And we knew it had to be like really super good because otherwise adoption wasn't going to work. And.

So that means not simply listening to customers, also listening to stakeholders and doing what you can to align those needs. That's really, really critical. You do need the support of your stakeholders as well as providing something great for customers. And then in the development process, when you're sitting there with the team, the same thing, you need to do the same thing. You need to shut up and listen because you have in the room a bunch of people. They're very good at what they do. They know what to do.

And it's really important to make sure that they have a voice in what they're creating, no matter what their job is. So you may be a developer, but you know what we're not going to do? Just limit your voice to development. You have experience. You know customers from different perspectives. You need to have an input into that. And where we got to those points, and of course, they come up in every project where there are differences of opinion. It's just about how

having these really super healthy arguments to make sure everybody can just kind of really sit there and promote and advocate for their position, defend it where they need to, and eventually come to a place where we know what we've got, you know, because we've really woodshedded the idea, you know, as far as it can go based on the research and, of course, based on everybody's opinions and experience and abilities.

to create what comes out of that is something that's really, really super useful for the end user and also meets those stakeholder needs. Yeah, if I could add one thing to that, Christian, but team alignment is absolutely, I think, the most important thing if you're going to create a good user experience. I have been literally on hundreds or maybe thousands of UX projects having run this UX company for 25 years.

And what I found was that what usually made projects go south was that the team was not aligned around a common vision and that they really couldn't

adjust their own priorities and their own worldviews. So that's become a really important thing. And then at the end of the day, if you do have disagreement and you do have conflict, you have to have a way of resolving it, which for me is that, you know, if Christian and I disagree, Christian wins because he's the product manager and he has the larger responsibility, whereas I'm a UX lead and I'm one of a number of factors

that's responsible for the way this is going to be produced. But there's also the organizational strategy, and there are the business processes, and there are the technical limitations and technical objectives, and all of that has to be factored into it.

So we were very fortunate in that way. And with our development team, actually the director of web development services, Jill Maraca, came in there and was hands-on for the entire time of the project. And so if a, you know,

decision had to be made that we couldn't do it as a team, it would either be Christian or Jill, depending on whether it was a technical decision or a product decision. We met a couple of months ago and we spoke about the Princeton EDU website. You were telling me how you used a survey on your website to gain relevant insights.

Could you maybe tell us a little bit more about that project? Sure. Yeah, the survey was for us to understand what our different audience groups were doing on our site. Like what content was each audience group interested in?

The university has a lot more than your average set of audience groups. I mean, we, our small list is 16. And I mean, you can imagine what many of them are faculty, staff, undergraduate students, graduate students, parents, perspectives, and it really can go on and on. And a university website, just, you know, the main EDU site, like

any other organization, any other institution, there's the front door. That's where all those groups are walking through. So we had to make sure that our IA. So information architecture, how you organize the content on your site, and it's directly related to the navigation. And so the findability of content, the way people who come to your site understand where things are and how they can find them is directly related to how you organize that content. And we speak of that

about organization as an information architecture or IA. It's so interesting to see your dynamics, your team dynamics, how you describe how you work together and

When you were talking about your team, what were some criteria that helped you figure out what to design? And I believe you mentioned a rubric that your team followed. Could you maybe share a little bit about that, Charlie? Yeah, I'd love to. It was really interesting because Christian and I each came to this project with something we had developed ourselves that were quite different.

but that turned out to work really, really well together, Christian's pair lab and my 5D rubric. So as I've mentioned, I have been on hundreds and hundreds of UX projects, and I've gotten really clear about the fact that team alignment, if that fails, then you're very unlikely to emerge with a decent UX at the end of the project.

because there are so many different decisions that have to be made, and if they're not made in a consistent kind of way, as one of my colleagues said, it's like getting nibbled to death by ducks. Each decision takes a little bit out of the integrity of your design. So I became very interested in the fact that we don't have a common definition of UX.

I've done a little research on this. I ask people to define UX. I get the most wonderful definitions, but when you look at it, they're not operational. They're not workable. So I went back to some of the work that had been done. Whitney Quisenberry, who

produced something called the 5Es, which was a faceted view of user experience. You might be familiar, some of the listeners, with Peter Morvell's Honeycomb, which was very popular for a number of years. And these all tried to take this complex, multidimensional thing called user experience and produce a definition that looked at the various dimensions and facets.

And I felt that could go another step, so I did a whole lot of interviewing of people to try to find out what did they consider the components of UX. And out of that, I developed a schema called the 5D rubric, 5D standing for five dimensions. So the five dimensions that we found most important in describing UX comprehensively are the first one is empowerment.

The second one is efficiency. The third one is intuition or intuitiveness, or sometimes we think of that as usability. The fourth one is engagement. And the final one is trust.

And these are very complex dimensions. There's a lot to say about them, and they can be customized to any specific project. But what we found is that with those five dimensions, people feel satisfied that we're getting all of the information that they want to give us about their feelings and their experience.

That then became a definition that we were able to share with the entire team, with users, and which could support also assessment and metrics. We developed a bunch of online, very nice little assessment tools, which gave you both qualitative and quantitative measures. And I am very much of a qualitative person when it comes to analytics. Being a psychologist, I want to interview people.

I want to learn about what people have to say. So I am very much of a qualitative person in the way that I like to look at things. And when you look at the way that UX is traditionally measured, it's often with something like the system usability scale, the SUS, or the net promoter scale, the NPS, which does it on a scale of 1 to 10. System usability scale gives you a number from 1 to 100%.

I look at that and I say, what does it mean? I find something that has a value of 70 or 75. It doesn't tell me what to do.

I don't even know what the difference is between a 70 and an 80 in terms of any sort of absolute measure of UX goodness. And so I actually have been turning to places like Amazon where they use reviews, they use rating scales, they use dimensions. These are all rubrics. And what's most valuable are the comments that the people make.

And we have assessment tools now that we use Google Forms for or Microsoft Forms. They're cheap, they're free, they auto-score, and they take people less than five minutes. And so that was the 5D rubric, which we are finding to be very, very valuable. And we paired that with Christian's Pear Lab, which was another way of gaining insight to what people were thinking.

Maybe, Christian, you could explain a bit about that. Yeah, thanks, Charlie. And you want the qualitative side. I need the quantitative side. I need the numbers. The numbers are really important to me. And listen, I share that view of yours that what is an 80, right? Because if you don't have some context in it, if it's not in some way a measure that you can really, really wrap your head around, right?

it's hard to really work with. And it's easy for people like, say, to look at that and manipulate it one way or another and stuff. So I share that view. And what I was trying to do when I created this thing, the problem I was trying to solve that I had for many years in my experiences developing products was, what do I do? I mean, I don't know. I mean, okay, I do know. I mean, I have my intuition. I've got a bunch of data.

You get a bunch of ideas or a bunch of features, and then somebody's going to ask you, what are you going to do first? What's MVP? And I think intuition is great. And I use it myself, but it only gets you so far.

And the chance of getting it wrong, sadly, is high. And it can get higher when you have different inputs that are sort of dragging you in directions. Certainly, every project has a squeaky wheel of one form or another. Maybe it's a stakeholder, maybe it's a customer. And those squeaky wheels, they demand that oil and they tend to pull you or push you in different directions, which may not really be great for your customer or your product at the end of the day.

So I thought about this a lot over the years and I came up with sort of an algorithm. Yeah, well, not sort of, but it is an algorithm that I was then able to create a tool around, which at the end prioritizes essentially ideas, but for me, product features, because I started the beginning and after doing all the research, I have a list of feature ideas I need to know.

What is it that my end users, my customers, my audiences really, really, really value over the things that they don't value? Where does each part land? And you can...

Give to somebody a survey. You can say, okay, here are 15 things. Here are 20 things. Put them in rank order by your most favorite to least favorite. And you know what? The top of that list is really probably pretty accurate. The bottom of that list is probably pretty accurate of what they don't want. And then you have this big, giant middle of mush, like that instant oatmeal that you eat in the morning. It's like, yeah, okay, what do I do with this? And it's hard to make decisions off of that. And I guess in some sense, the same with

product feature prioritization is true, you've really got to do what you can to get into the minds of people in a way that, for me, is quantifiable. So what eventually became Paralab was an algorithm where I took ideas from social science and psychology, which is social science, some machine learning, and a couple of other things, and learned from those things if I

What is the human mind really doing when it's trying to make choices, when it's trying to prioritize? And it turns out that our brains aren't really good when faced with big, complex choice. The more you can reduce that complexity, the better we get at saying what it is we like and what it is we don't like. And it turns out that the easiest choice is a choice between two things, a pair.

And so what Pear Lab does is it takes a bunch of ideas and it splits them up into pairs and it puts them in front of customers, stakeholders, and it asks them, hey, here's two ideas. Which do you like more and by how much? And then I take all that data. And by the way, it's also figured out what really, really helps me is if I can segment that data by different customer groups or, you know, you could do it by anything. Even I keep mentioning stakeholders and

and customers, which has been really, really useful to me because Parallel can show you, or it shows me, where they agree, where they disagree. It uncovers hidden opportunities where you could possibly find things. I guess if we were in a for-profit environment, which we're not at Princeton, but where you could monetize things that you didn't know were priorities for certain groups, that can be a path. And it also

puts out groups of, hey, here are your high priorities, here are your medium priorities, here are your low priority things. And they're comparable too. So I can even say idea one is preferred by some segment group or by everybody 10 times more than say idea three and so forth. And you can walk into a meeting with that. I've walked into meetings with that data in hand and having a language around the numbers

has changed the conversation and helped really bring everybody on board. I think when we talk about inside Princeton, our intranet, how did we get to a point where we were able to get buy-in from stakeholders and satisfy customers? It's having those conversations with stakeholders

information or data or numbers, you know, this quantitative data in which you can really discuss it in plain language. It's been incredibly, incredibly helpful. Even with Inside Princeton, the name, we did that with Pear Lab and we had, you know, a whole bunch of different names and sent them out to different people. And this one popped up to the top and everybody likes it. It's easy. It's memorable. People know how to get to that URL, which is really a critical step in the user journey.

So it was a really good kind of thing to be able to do quantitatively. It was. And in fact, it's got a qualitative aspect to it because it also crowdsources ideas. And in the case of finding the name, for example, and we also found this with the feature prioritization, people suggested names and we feed them back into the study and then they get to vote on them again. And so because we know, you know, when UX, what do we say in UX? I am not the customer, right? And that applies to

I think very, very broadly,

I don't know all the answers. I don't know all the, you know, all the saying in this case, I don't know all the possible good ideas for names that we could have named this thing. Uh, but my customers do and they suggested it. And then we threw some back in, like we had a one bulletin helm, for example, which is a name of a tree. Well, it was a tree because it was chopped down a long time ago on the Princeton campus, which was a very, very early bulletin board. And I think it was in the 19th century and students would tack up, uh,

posters on this elm. This actually turned out to be a dead elm. They just used the trunk. And somebody knew that and said, hey, if you're building an intranet, why don't we harken back to 100 plus years ago, something that was really important in Princeton's culture. We fed it back in and it did okay. I think it ended up in the middle of the pack inside Princeton turned out to be number one. So having that quantitative view up front

Being able to prioritize and compare things so that you can speak to people about them. And then after we go through the process of building, having Charlie's 5D rubric at the end, which to me says, hey, did we do this right?

And where did we go wrong? We can then feed that back into the development process. And I think this is one of the things Charlie taught me, which really was a huge eye-opener. And it applies to the 5D rubric, but I think the way we approach UX, thanks to Charlie, is when you go through your UX exercises and you find defects, you stop and create a loop.

And Charlie, you're probably much better at articulating this than me, and maybe I should just shut up, but at least this is my view of what Charlie does.

Take those defects, fix them, put them back in the loop and run them through UX and just keep doing that and keep doing that and keep doing that until you are really, really, really sure that you have figured out the solution to those defects and then you can move on. The 5D rubric helps us really think through what those defects are and feed them back into the loop. I don't know, Charlie, did I articulate that

uh accurately i think that is the basic that sort of uh iteration and continuous improvement is the basis of user-centered design in ux i mean we had one really good example of this when you know we we pulled together our first version of the intranet and we had done all of this great user research and we thought we had our assumptions clear and we understood user priorities

And then we took the 5D rubric, which seems to be a really useful operational definition. And basically we had

We gave them the early version. We had them use it for a week or so. And then we asked them to fill out the online forms. And then we brought them together in a focus group to discuss the comments that they made and what they had to say. And we realized in the process of doing that that we had made some serious errors of assumption. And some of the designs we built were not what the users really wanted.

And that was amazingly clarifying. And we went back and we said, wow, we missed that. We did that wrong. Let's fix that. And we brought that back to the team. And we said, here's what we found out. And here's how we think we should fix it. And we worked through the designs and we rebuilt it and we went back out there and people were delighted.

And that's because their concerns and their issues were dealt with in a structured way and that the analyses that we did gave us the qualitative information. It wasn't just, oh, they think it's a 40 out of 100, but they said for every one of these dimensions, this is what I like, this is what I don't like, this is what I wish it would do.

And that guided us. And that's the guidance that we as UX designers need an awful lot of. Thank you, Charlie and Christian, for sharing all these insights and sharing all these stories. And it's so interesting to see how you use those two frameworks

that you developed, Charlie in the 5D rubric and Christian Pear Lab. How will you use these two frameworks to get quantitative and qualitative data, not only to work on your design, but also to align your team? That's what you mentioned earlier, that this team alignment was a key part of your success.

to actually design something like your outstanding and your awarded intranet at Princeton. Christian, you said you got this goldmine of information about different user groups. I remember there was a surprising finding about your undergrad students, how they used your website. Oh, sure. They don't.

Okay, so one of the surprising things we learned is they really don't, except in certain cases. And then we followed this up with some interviews, too, to confirm what we saw. And maybe you want to talk about this. Analytics tells you the what. Analytics doesn't tell you the why. So actually, I'll give you the whole picture here where the analytics plus the UX combined to explain.

really inform us about what undergraduates were, I should say, not doing on the site. And that is, they come to the site as perspectives. That is definitely clear. We saw that in the data. They matriculate and they stop using the site.

What they do, we found out through user interviews, is very quickly after they start their first semester, is they learn from their peers what apps and services they need in order to navigate the world of being a student, especially being a first-year student. And they cobble together essentially this digital footprint. That digital footprint does not include the course site. And the exception, or we call, I'm sorry, Princeton.edu, we call it the course site.

The exception is when they appear on it. Maybe we wrote a story about them. Maybe a professor of theirs appeared or a friend of theirs appeared. So there's a real social angle involved.

to the site for them. But it's very, very, very limited in scope. And they go on their merry way through their four years, really not coming back to look at Princeton.edu except in these other circumstances. And the other things we found out, I think, which has been eye-opening, why we got to an intranet was we were able to really, really clearly see what employees and specifically staff and faculty as separate segments were doing

And that was the first, you know, I think the genesis of the idea of, oh, you know, we have a real need here because the content on Princeton.edu is primarily geared for external audiences. And yet here we had this really, really clear set of behaviors that internal audiences were using the site for. And we wondered, okay, if they're doing this here, can we do something better for them?

and give them essentially their own presence. And that's what eventually became Inside Princeton. So today, and this is in large part thanks to that initial study, that pop-up study with the Google Analytics integration, we're now able to provide essentially a site for external audiences, princeton.edu, and then a site for internal audiences. The content's different, and they're working for the audiences that they're

that they're designed for. Maybe one last question. If the audience had to leave with a single piece of wisdom from this episode, Charlie, maybe you go first. What would you want it to be? What is one single piece of wisdom? Here's what I would like to say to them.

We have entered a new era of information technology. Technology is everywhere. Every single process, everything we do, there are little interactions that take place for people all through their days, all through their lives. Every time there is an interaction, we create a user experience. And that has made user experience really central.

We have been to some extent a craft with our own language. We're really good at talking to each other. And I think we do really good work, um, as, as, as a, uh, as a discipline. Uh, but we have not fully integrated into the larger processes of product development. Um, and, um,

you know, product configuration and all of those kinds of things. Now UX is everybody's responsibility. And what we really need to do is to find ways to operationalize user experience so that people who are not UX professionals can become full participants because UX is everybody's responsibility, not just the UX department. How about you, Christian? What would you like to give our audience on their way?

I think it's around listening, and it has two dimensions in my mind. So first is listening, and I think the other way to look at it is basically keep your mouth shut. And the reason is that you as a single person, even as a team,

Your view of the world, your ability, your intuition, all these really, really great things that you bring to your product, to your project that can get you really far can only get you so far. If you're not listening to your customers, if you're not listening to your stakeholders, you're going to miss something. You're going to miss something really, really important. Let them do the talking. And I mean, listen, we talk a lot about

empathy in the process of UX. And empathy really does start with listening, at least in my mind that it does. So keep your ears open and your mouth shut in that area. And the other one is to really partner, partner closely with UX and listen to your UX folks. I think Charlie and I, we work well together, but I think the things we've been able to accomplish, I think are because we talk every week. We have a standing meeting every week

And we talk through stuff and we talk to the projects that we're working on. We talk through other things that may not be related to the projects that we're working on. For me,

coming from the perspective of a product owner. I need UX really, really badly because in so many instances, UX acts to me as my conscience. Charlie gives me a gut check so often. He says, are you sure about that? Are you sure about that? And then as we start to talk through things,

He'll challenge me and he'll get me to think. And really where I think that UX has its greatest value is that it gets people to think, to really think through problems, think through them thoroughly so that at the end of the day, when you get to that end point, you have success.

a much, much, much better result. If you want to know more for Paralab, you can go to paralab.io and I have a bunch of case studies on there you can look through. And Charlie, where can people go to learn about 5D? Well, I think we're on the Princeton site at ux.princeton.edu. And also I'm

putting together a site, 5d-rubric.org, so that I can provide the material to other people who might want to use it. Christian and Charlie, thank you so much. You're an inspiring example of how product and UX can work together and build incredible products. So thanks for coming on the NNG podcast and hopefully talk soon. Tim and Therese, thank you for the opportunity. I mean,

Nielsen Norman Group has had an enormous influence on making UX really accessible to a very, very large group of people. And, you know, the consistently very high quality, which I use all the time, I refer people to the materials that you're producing because they're always good. And that's really great.

Because we need more people to understand this UX maturity, I think, is the future of information technology. I agree. And, you know, let me mention one more thing about you. I mentioned listening to Jacob years ago at the Journal, not long after I bought Don Norman's book, The Design of Everyday Things. And I'm telling you, I have never...

looked at a doorknob the same since I read that book and it's on my bookshelf at home. It's this eye-opener. I mean, just the stuff that all you guys are doing is really enriching for me. I don't know for many, many other people. And I'm also very, very thankful that you wanted to have us on. Thank you very much. That was Charlie Kreitzberg and Christian Noble.

We've included some links to Charlie's 5D Rubric and Christian's Pear Lab in the show notes, as well as a link to our Intranet Design Annual Report, which features not only the Princeton team's great intranet, but many other winners as well. We also have some links to other intranet and research-related resources in the show notes, which you can probably find on whatever podcast platform you're listening to.

Now, since we're talking about podcast platforms, now is a good time for our announcement. This is our last audio-only episode. But have no fear because starting our next episode, we'll be uploading a video version for every installment. So you'll be able to find all of those at our YouTube channel, which if you don't already subscribe, just search for NNGroup.

And whatever platform you're on, please do subscribe or follow so that you can be notified whether that's a video or audio episode whenever we publish the next installment. But to learn more about UX in general, check out our website or sign up for our email newsletter at www.nngroup.com. That's N-N-G-R-O-U-P dot com.

This show was hosted by Tim Noiselesser and produced by me, Therese Fessenden. All editing and post-production is by Jonas Zellner. Music is by Tiny Music and Dresden the Flamingo. That's it for today's show. Until next time, remember, keep it simple.