High-performance Hiring & Interviewing with former Amazon Worldwide GM of Prime Membership, Anna Collins
57:54 Share

Hello, Acquired LPs, and welcome to another episode of the LP Show. David in his San Francisco home and me up in Seattle. Today's episode is one that has been requested a few times in the Slack and a topic of great discussion. When you dive deeper into these company building topics and stop talking about just the

news of the day of who got funded or what company is going through trouble or the day-to-day of running a company, one of the most important things is how do you choose who to bring into your company and what process do you put them through to see if they're the right person. And we have never discussed interviewing on the LP show. As David and I were talking about how should we do an LP episode on...

interviewing, I was telling David about the story of this awesome entrepreneur in residence, Anna Collins, who'd been working with us at PSL, and she hopped on this interview loop to help us out. Just had, in our post-interview discussion, the most thoughtful feedback, the best questions. All of us were joking and looking at each other and saying, why did any of us even do this interview loop? And can Anna just do the whole thing for us? So here we are today to do an LP episode on the very first time on how to conduct the best interview you can

a key part of building a high performance team with the best person I can imagine to help us do it. So Anna, thank you for being with us today. Thanks Ben, it's great to be here.

To give folks a sense of Anna's background, I mentioned I personally know her because she's been working on new startup ideas with us. But before that, Anna has a prolific career. She is a fantastic leader and human, having started and scaled new businesses at multi-billion dollar companies in healthcare, edtech, advertising, gaming, CPG, retail, and of course, technology businesses. She was the worldwide general manager of Amazon Prime.

Before that, she was recruited by Microsoft to build out and scale the global search advertising business from concept all the way through growth and ultimately to $1.6 billion in revenue.

At CVS Health, she was responsible for leading the internet channel strategy and the acquisition culminating in the launch of CVS.com. She holds an A/B in economics and an MBA from Harvard University. While not changing the world, Anna coaches her sons Henry and Cooper in basketball and keeps up with her wonderful wife, Debbie,

A quote that closes out Anna's bio reads, family first, love wins. And Anna, I just wanted to include that because I saw it on a bio of yours somewhere and thought it'd be a great way to close that out. Thanks, Ben. Appreciate it. You have such a prolific background. Why don't we dive into some of the early history of that before we get to these sort of nitty gritty of tactical interviewing questions.

One of your earliest experiences as a three-sport athlete was on the Harvard basketball team. I just wanted you to share with us, because I know a little bit of the story, what that was like for you and why that was a special time in your life.

Yeah, it was a special time because when I went to Harvard as a freshman and joined, we were last place in the Ivy League. The first year, we won three games and lost 21. And so we were losing basically all the time. And we weren't just losing, we were getting crushed by other teams, 20 points, 50 points. And it was the first year of a new coach that came, Kathy Delaney-Smith. She had a vision of winning, and that was part of why I wanted to be on that team.

and turning the program around. And we went from worst to first in four years. So in my senior year, we won the Ivy League and had reversed the freshman year record, and we were 21-3. And it was the first time Harvard basketball ever won an Ivy League championship, men or women's. And that was a big journey going from worst to first and a lot of heartache and

and sorrow and a lot of learning and how to experiment and be a team together. And during that journey in basketball, I had a practice that if we did not win a game, I did not look at my individual stats. And that's because it didn't matter what I did. It didn't matter how many rebounds or how many points.

If we didn't win the game, it didn't matter. So I had a practice of only looking at my stats if we won the game. That gives you sort of the notion of how I feel about team versus individual and what matters. Well, speaking of team versus individual, I know you were in the Air Force. I'm curious to hear a little bit about your path of being in the Air Force and how that ended up leading you into the world of technology.

I can share a little bit about the Air Force, the basketball. It had more to do with the lead into technology with our friend Mike Galigan. The Air Force, yeah, I had the opportunity. I was picked to go to an engineering and science seminar at the Naval Academy. I got exposure to the opportunity of the service academies as well as then the ROTC Reserve Officer Training Corps programs and scholarships. And so, yeah.

I had a choice when I was graduating. I had both scholarships and appointments to the Naval Academy and the Air Force Academy. And you can take those scholarships anywhere. I also had other options with academic and athletic scholarships, so I was super fortunate. But I wanted the experience of serving. Serving mattered to me. Growing up in Michigan had that service component.

and value as a family. And then second, the opportunity to lead, to manage people and be responsible for resources and people at an early age. Really at 21, you're thrown in as an officer and you get that opportunity. And I knew that I wanted that. So those components all together had me really go after the ROTC scholarship. And I did that at MIT, cross-enrolled at MIT as I went to Harvard.

At some point, you made this sort of choice, hey, I'm going to go into the world of business. I don't think the Air Force has been this amazing experience, but that isn't my career. In college, I studied economics, and that was the closest thing to business at Harvard. And I thought, hey, I'm going to go to business school or law school, I'm not sure. With the Air Force, I didn't know how long I would stay, but

As I was in the Air Force, I figured out I was gay in my early to mid-20s. And that was when it was before don't ask, don't tell. It's just you're a criminal if you're gay. And so I was in finance. That's where the Air Force matched me to as a career, which ended up being a good career for me and background. So I wanted to get out when I figured that out after my four years of active service. And I had an echo time of that reserve that I owed.

But I didn't feel comfortable staying in. For me, it was integrity to not be my full self. And so I was just scared. And I got out after that first four years. Otherwise, I may have stayed in a little bit longer and potentially have gone to graduate school and then served again longer in the service. But for me, it was urgent urgency to get out based on being a criminal. Wow.

Well, I want to zoom forward a little bit through the early business experience you had at Johnson & Johnson and then launching CVS.com in the late 90s to where you decided to join Mike Galgan at Aquantive. And that was a heck of a journey. I mean, Aquantive...

you know, especially in those sort of late 90s and early 2000s. For folks that don't know, this is a company, it was one of the very earliest pioneers in the internet advertising space. It IPO'd and then sold to Microsoft as a multi-billion dollar acquisition. And boy, is there a story in there that we'll dive into on this show at some point. But Anna, I want to zoom in on sort of the leadership component. What did you learn being an executive during that time period and how fast things were moving?

It was the quintessential chaos, and it was pre-IPO when I joined in August 99. One thing is being flexible and adaptable. So even in the role, I interviewed and was recruited as VP of business development. But the week before I joined, Mike, who was acting president at the time, said, hey, we really need a leader for this media industry.

group that has 30 plus individuals in it and a manager to help figure out how to scale that portion of the business because we're growing like crazy and it's not working basically. And would you instead come into that operating role and be VP of media as your starting position? This is a week before I started. And I said, sure, that'd be great because I prefer an operating role. And I know we're moving quickly through your background here, but you've just done so much. And I think it's really relevant context for

you know, our discussion on interviewing. So speaking of customer focused, you also, you spent some time at Amazon and you were the worldwide general manager of Amazon Prime. And Amazon is sort of famous for their, not only interview process, but the, what do they call them? The principles, the 18? Leadership principles. Yeah. 14. 14 leadership principles. Yeah. And again, to be clear, I was one of the leaders in Amazon Prime and I was the global leader for

Amazon Prime membership, which did include retention, engagement, the first two Prime days, Prime member spent, pricing. And so, yes, that's super fun to do. Again, a big team of people to be thankful and grateful, again, for that experience. Amazon was a tremendous place and is a tremendous company and learned a lot and got the opportunity to work with a lot of amazing people there.

You skipped over the Microsoft portion, which I spent about seven years at Microsoft, which again, I hired. I'll just skip back to Microsoft in building out that search ad business from scratch. I hired with my team of people, over 500 people across the globe in 18 months.

And created brand new roles to do that. And so, again, that was that was actually the most phenomenal hiring practice that I that I've ever done again, both in scale and in speed.

And I could talk about that. But going back to Amazon, Amazon's leadership principles and how they operate, Amazon is like a mashup of Harvard Business School and the military because of how standardized they operate and with a great adherence to process and the way things are done. And so that's one of the reasons that

And this applies across the business, whether it's creating a new business, working backwards, doing the working backwards process with a customer press release and frequently asked questions or ERFAQ, or it's the hiring process and what that process is and how it's done and what the components are and how the preparation is for that, starting with the job description, what are the appropriate requirements

Seven, the most important seven leadership principles, for example, for that particular role, perhaps. And who should be on a loop? How do you do the pre-brief before the loop? How do you assign roles? And what you're basically doing is you're giving everyone a different perspective.

probably two leadership principles to interview by. And then everybody, you're basically getting a portion, everyone's doing a portion of the interview data collection. And so you get a full picture of the person. This is exactly the same process, by the way, that we did at Microsoft. So it was not different at Amazon than Microsoft.

Amazon has done a better job of branding it. Yeah, potentially as far as the communication on that. But I think also in the way that it's practiced. In other words, there might be more variation of how, I want to say religiously, but how well the process is followed in different, say, divisions or groups at Microsoft. Again, it's changed since I've been there, but it may have changed since I've been there. But during the period I was there.

But Microsoft has their seven values. At the time, 33 competencies. It's a similar thing. And we did the same. So how I interviewed at Amazon was not different from how I did it at Microsoft with my teams, as an example. We used different values slash leadership principles, but the actual method was the same.

There's a bar raiser, a bar raiser for an Amazon, which is the person or the role on the loop that is there to assess, to do two things. One is assess the long-term potential of that candidate. So are they a good fit for Amazon or the company? Do they match those values? And do they have the potential to grow? If you're hiring in at X level, can they grow two or three levels beyond that?

So are they good long-term fit? And number two, are they just a good cultural fit for the company? Because the hiring manager always has the immediate pressure of,

Needing to fill that role to get the job done the business done and so this this bar raiser role is this long-term potential and then also calibration is this person at that level if you're hiring in as a for example frontline manager do they have the right experience and competency and capability that would in proficiency to operate at that level so you're calibrating for that level and

Microsoft has a role called an as-appropriate. It's the same role. In other words, I was an as-appropriate. Yeah, the as-app. As-app, yeah, exactly. But that would be an example of how those practices are really the same, but the adherence and standardization of those is greater at Amazon.

And with the as appropriate, my understanding of the place where that term comes from is they're the last person on the loop and they only join as appropriate if the rest of the group is sort of trending up and they need that one person to come in and sort of weigh in. Was that your experience? And then also in the bar raiser, is that sort of the same way that the bar raiser would be like the last person on the loop? So two things. One at Microsoft, again, this is where we always had it. We had loops and

The experience, because you're going to construct the loop and say, you know, you want to do it to make it convenient and fast for the candidate as well as the company.

And so to the degree possible, you would have them all stacked together. So you start at the beginning of the day and at the end of the day. And based on schedules, it's too complex to say, you know, the as appropriate is at the end. Yes, or the bar is just at the end. If you could do that, that's great. It doesn't have to be that way, though. And it's kind of with scheduling. It's virtually impossible to always do that. So what's more important is that that role is on there. Second, you don't get to know the other person's feedback until after you've done your

feedback as an interviewer. And so it's about at Microsoft, you could see the other interview feedback as an as appropriate and then be able to use that to probe on concerns. And so that would be one of the things that as an as appropriate as the rest of the loop happened, you could not only have whatever you were planning to your assigned interview,

values or leadership principles in the Amazon world, but you could also have other concerns that you want to probe on that have come up through the loop to date. And so that's the other piece of it.

We want to thank our longtime friend of the show, Vanta, the leading trust management platform. Vanta, of course, automates your security reviews and compliance efforts. So frameworks like SOC 2, ISO 27001, GDPR, and HIPAA compliance and monitoring, Vanta takes care of these otherwise incredibly time and resource draining efforts for your organization and makes them fast and simple.

Yep, Vanta is the perfect example of the quote that we talk about all the time here on Acquired. Jeff Bezos, his idea that a company should only focus on what actually makes your beer taste better, i.e. spend your time and resources only on what's actually going to move the needle for your product and your customers and outsource everything else that doesn't. Every company needs compliance and trust with their vendors and customers. If you're a

It plays a major role in enabling revenue because customers and partners demand it, but yet it adds zero flavor to your actual product. Vanta takes care of all of it for you. No more spreadsheets, no fragmented tools, no manual reviews to cobble together your security and compliance requirements. It is one single software pane of glass.

that connects to all of your services via APIs and eliminates countless hours of work for your organization. There are now AI capabilities to make this even more powerful, and they even integrate with over 300 external tools. Plus, they let customers build private integrations with their internal systems.

And perhaps most importantly, your security reviews are now real-time instead of static, so you can monitor and share with your customers and partners to give them added confidence. So whether you're a startup or a large enterprise and your company is ready to automate compliance and streamline security reviews like

Like Vanta's 7,000 customers around the globe. And go back to making your beer taste better. Head on over to vanta.com slash acquired and just tell them that Ben and David sent you. And thanks to friend of the show, Christina, Vanta's CEO, all acquired listeners get $1,000 of free credit. Vanta.com slash acquired.

You said, you know, Amazon being kind of a cross of Harvard Business School and the military. How did you think about creating such a process-driven culture, which you have to do when you're hiring thousands and thousands of people? You need a standardized process.

with the making hiring decisions is, you know, these are human decisions. It's not like there's a right answer. What was the appropriate balance of like a rigid process or a rigorous process and still having room for like understanding that human element? A couple of things. One is having process. It's kind of like when you, if you ask someone, um,

about boundaries and creativity. You actually need structure for creativity. So basketball is played in a court that is 90 feet long. A free throw is 15 feet. It has a lot of rules and structure to it. But yet you could have Steph Curry or LeBron James or Larry Bird, Michael Jordan, different styles of basketball within that structure. So I believe, and music, same thing, you have

the staff, the same notes, different, you know, four-four time. And you have Mozart to Madonna to Lady Gaga. So I believe that structure is,

is what is required for creativity and for humans to actually connect. And if you have chaos, that's the opposite of being able to have an ability to connect and a good candidate experience. So it's all about being human. You're right. And you need the structure to help facilitate that and to make it productive. Because if you have no, if you don't have this kind of structure and rules, let's just go out to lunch together. Right.

Right. Then you don't know at the end of it. You don't know. It's similar actually to PSL's structure for assessing a business idea. You have a way to do voice of customer interviews. If everyone just, oh, let's just go talk to a bunch of customers, but you don't know what kind of data you're looking for. You don't have a hypothesis. Then you could conduct 10 voice of customer interviews and have nothing at the end.

Yeah, and I've certainly made that mistake, especially earlier in my career of trying to do interviews without a clear plan. And then I ended up spending an hour of both of our time and have no data at the end.

And I think that's a, you know, that's very common for people early in their career. The other thing is it leads to a bunch of subjective bias when you're just saying, oh, you know, is this person, because again, we all have these biases around, you know, do they like things I like? Are they interesting? Are they engaging? Someone might be introverted versus extroverted. You start judging based on style.

and other factors than what are the capabilities. And so the interview is about assessing fit, fit for role, fit for the company, fit for the potential to grow, fit for that level. And so you're assessing fit. What are the skills, functional knowledge, experience, capability, competencies that

And those competencies include the EQ stuff and the business judgment, like write a lot as a leadership principal at Amazon. That is about judgment. Are you right? A lot when you end judgment is both quantitative and qualitative getting back to the subjective. So how do you assess someone's judgment?

And that's important whether they're a frontline, first level developer or they're a director leading a big business.

So, what I think is important about that structure is part of that also is reminding people that this is a two-way, the candidate experience too. So, they're assessing, is this role, is this company, is this opportunity right for me? And so, again, you want to show up with the right game and intention and organization so that they get their questions answered as well. Okay.

I love that analogy of like the process is like the basketball court. Like you need with no basketball court, no rules. Like it's impossible to have creativity and greatness, but you can still have creativity and greatness within the process. Absolutely. Absolutely. And being authentic to your point, you know, to me that, that actually showing up and, and having a plan and then being able to connect with the person while you're at, while you're engaging in a dialogue around, um,

their experience and you're also sharing about what the opportunity and the role and the company is like is important. Well, Anna, there's so much more we could touch on in your background, including the fact that you were the president and COO of Bulletproof, which I'm sure lots of people are familiar with, disruptive sort of consumer health and wellness brand. Let's weave that into the meat of the episode here. So talking about the interview itself,

Whatever is most useful for you, whether you want to talk about it like Anna Collins is starting a startup right now and how you would do this or if you were at a bigger company, how you would do this. I want to talk about like you're about to hire some people and you need to figure out how to do that. What do you do first? What's your playbook? How should how should listeners think about this?

I guess a couple of things. One is, you know, we started to talk about it with the leadership principles and competency. So the first thing is it's, you know, the interviewing or bringing people onto the bus is

is the is one part of the people process. So there's like, what people are on the bus? How do you bring the new people onto the bus? What seat on the bus? How do you develop people and move them from one seat to the next seat on the bus? And then when is it time for someone to leave the bus, get off the bus, all of that, that larger people process is

is rooted and grounded in, like the basketball court and rules for the basketball game, there are structure and guidelines for the people process. And for me, those start with the vision, mission, and values of the company. So, for example, at Bulletproof, when I got there, again, Dave, the founder and CEO, had done a phenomenal job of growing the business and

to a very large business at the time I joined, but it was still very bootstrapped and chaotic from how it was operating. And so when I asked 10 people, what's the vision? What's the mission? What are the values? You'd get different answers. And there were some essence things of that that showed up like it was disruptive, like it was a bleeding edge, innovative kind of brand.

And it's a better for you, right? A brand that is about nutrition and it provides content and products to help people be healthier. But how that was being communicated to both customers and employees was different and it wasn't, again, clear. And so one of the things I did was in the first 90 days was start a process where

to define the vision, mission, and values with the employee input, customer input, feedback, and then ran a process with Dave and the leadership team to define the vision, mission, values. And so it's clear the mission of Bulletproof is to create products and provide information that radically improve lives. And everybody can say that and know that. And values like gratitude and innovation and customer focus

for example, are critical. And so you say, okay, here are the values, here are the vision mission, and then bake those into an interview process that says, okay, now we're going to interview for these things. We're going to interview, and those are critical in addition to skills and functional knowledge for whatever role it is. Those are critical things to weave in, just like for Amazon, they are with defined leadership principles. Those same vision mission values get

woven into the people processes for developing and assessing employees as well. With that, the skills and functional knowledge are really being assessed when the candidate in the front end part of the selection process for candidates. And so by the time

Well, I should say phone interviews and the initial interviews by the hiring manager and the recruiter and looking at. So you're really in your testing and the actual, I'll say, final loop is about, I'd say, confirming those skills and functional knowledge and then assessing overall performance.

it's part of that overall final assessment on that as well as the experience and capability and competencies in full. But you have a good, you know, you're 80% sure the person has the skills and functional knowledge by the time they walk in the door for the full group. And you're still pressing and assessing for those, but your front end of the process has already done a lot of

both in the interviews, those pre-interviews done assessing for that, right? And now you're going for this deeper knowing about what their capability is and their proficiency level of those things and their fit for operating within that company according to the values. Getting back to actually defining the role, it's important. And when you're saying, oh gosh, what is this, the job description of the role? And getting back to it doesn't need to be

But it needs to be thoughtful in saying, what are the key capabilities and leadership principles that are important for this role? And not just for this role individually or standalone, but with the team. So when I'm hiring, for example, when I was at Microsoft and I was starting, I was the only person as I came in and they're like, write, create the business plan for the search advertisements. There was nothing. It was me in a room by myself.

And as I created the plan and I started figuring out what are the different teams I'm going to need and what kind of leaders, right?

And I needed to then I was going to have to recruit industry experience in because there was no search advertising experience in the company. Right. So for me. And then I also needed Microsoft DNA in the team and the current display advertising business. So I knew I needed a mix. And so I start to say, OK, what do I need for experience, both in industry experience and the different levels of leaders and kinds of roles? And, you know, one of the first people I recruited externally into the company was.

former Avenue A Quantive mentee of mine, Randy Wooten, and I hired him as a senior director of this premium operations service operations team to build out. And I was recruiting him back from an international assignment in London where he was running the Atlas business for A Quantive. And

what was important, I was picking him because I needed international experience that I didn't have at the time. And so that was something I wanted in that role as an example. So you're also looking at and saying, what do I need for experience that the team doesn't have, that I don't have, that's complementary, as well as, if you will, the brass tacks of capability for that specific functional knowledge. So you're looking at broader. And then you're also clear about what's

What's critical, what's not negotiable for that role and what's not? Because there is no unicorn. There are no unicorns. You have to actually, people are imperfect. We all are. And so you have to say, what am I willing to give up and trade off in a candidate for this role? And you do that in the job description then. You have enough forethought when you're writing the JD to know what the critical ones are.

That's right. And with that, you're some of it's, you know, literally in the job description for the capabilities or experience. And some of it, again, is required. Some is preferred next to like what's non-negotiable. And then as you're also learning, as you're seeing candidates and you're talking to people, you get usually you also learn and get a clearer picture of like what's absolutely critical and also what kind of talent needs.

is in the marketplace that you, that, that, you know, you have available in a, in a timeframe. I have yet to make a hire at, at PSL, uh, where I didn't have to iterate the, the JD after meeting candidates. And I've always wondered, like, is this a failure on me as, as to define the role or is that actually just part of the process, particularly when you

have very few roles per year where like you need to understand what people are like to inform what the actual job description can be rather than sort of perfectly birthing the JD and then having that go out to the marketplace. Is that necessary to iterate? It depends, you know, especially in the, it's,

circumstance that you described, I think that sounds like that makes sense. But if you go the other way and say, we needed to build teams and we had to launch the ad business in France, which is the first country we picked after Singapore to keep the wheels on.

We had to have these capabilities across these different roles, and we had also more than one for media specialists, search media specialists and search media analysts were two roles that we made up. We made over 14 different roles for these teams, created new roles. We had to create them and then hire at scale. So 30, not one, and do it in eight weeks.

So we didn't have the luxury of saying, we're going to do what you did for the one or two that are very unique kinds of roles. So in some cases, you can do that. Or so, for example, at Bulletproof, when I went, I didn't have any CPG experience, consumer packaged goods experience. So I wanted for sure when I was hiring my VP of supply chain, my VP of sales, my CFO, they had to have CPG experience.

Especially as we were going into retail distribution, which again, I didn't have. I had a lot of e-commerce in retail, but I didn't have natural food, better for you products and retail mass retail distribution in a mass way. So knowing what you don't have, just like I did at Microsoft. And, you know, there's a difference between is it only one of is there's only one BP supply chain or are there 30 media specialists or, you know, search analysts involved?

I was thinking when you recruited Randy into your new business unit for search ads at Microsoft, that was a case, I think a lot of startup, particularly founders, find themselves in this position where you know you have a need and you have a former colleague who you've worked with, who you kind of like know would be perfect for it. Like, how did you think about that moment where, on the one hand, that's like so overwhelming. You've worked together before, you know it's like this person's great, you know they're gonna be great, let's get them in there, like get them working.

How do you balance that with like, oh, well, maybe there might be somebody better out there. Should I run a broader process for this? Or is it better to really wait though? Like, I know this person is going to do great. There's no free pass. It wasn't like, oh, I thought of Randy and then he got the job. That wasn't what happened. And in fact, he had never worked for me directly in the Avenue A equine business. So...

He wasn't on one of my teams, so I wasn't intimate with his work, although I knew what he had accomplished and I saw him as a high performing individual and leader. And so that was clear to me. He had the experience that I wanted. I was picking him. I knew of him, but I was picking him more by what his results and accomplishments and reputation were.

Then my direct and I also looked at other people. He wasn't the only person. So it wasn't I just said, OK, here's the only guy. He was the best guy in the field of people. And then I worked really hard to recruit him, which wasn't easy. But but he had said, hey, if you ever have something, I would love to work with you. So I said, hey, I'm playing that card, you know. And oh, by the way, he went through a rigorous process.

like we're talking about at Microsoft to get hired in at a senior level there. And of course he did super well and was a key leader in that team and continued to excel at Microsoft and beyond.

This question of having worked with someone before is an interesting one because it's related to a thing that I rely really heavily on in interviewing, which is references. I always find that what you can learn in an hour in an interview completely pales in comparison to a trusted reference. But I also know that that's dangerous because there's lots of circumstances, including bias, that could cause that person's perceived performance in their last job to not be relevant at all to what they could do at your company. So how do you weight

that? A couple of things. One is constructing a loop of five people in your company to collect their own data independently in this targeted way across the capabilities, competencies, leadership principles that matter for the role for your company and for your company is the essence of what you're going to use. You also, if you have prior work experience, I've hired another individual, Jeff Hull. I hired

three times, twice at Amazon and once at Bulletproof. He's back at Amazon now after the Bulletproof experience. But

So when you have experience working with somebody, but still he went through, he was sweating, right? You know, the interviews at Bulletproof, it wasn't like, oh, he had a pass because he had worked for me before. He still had to nail those interviews and be like, oh, is he right for this team, for this company, for this role? And so the other thing is what you said, Ben, is really important, which is what's the fit

for this particular opportunity position. And just because they were a great fit at that other company for that particular thing doesn't mean they're going to be a great fit. It could, but you want to really, that's what you need to suss out is what are those key capabilities and cultural aspects that make or don't make a good fit for that candidate in this role or position.

And the more someone is a, you know, quote unquote athlete or general athlete and has demonstrated flexibility and adaptability, you know, looking for those kinds of signals, those

evidence points where they've demonstrated these things that matter to you, that's what you really want to look for, whether it's in a reference. And the same thing when you're having a reference, hey, is this a great guy or gal? No, it's what specifically did they do? And you're also looking for contra evidence that says, well, what are the challenges in development and growth areas? Because we all have strengths and opportunities. And again, it's important that the opportunities are

The strengths match what you need and the opportunities aren't going to be barriers in that particular position or company. But there's no panacea. And just to be clear, I've made a lot of mistakes. I'm not 100%. I still, I've made bad hires.

And I've been part of making bad hires, both direct reports to me as well as extended on teams. There's no 100% on this. If you're right a majority of the time and get anywhere close to 80% of the time, then you're phenomenal.

It always strikes me in hiring how similar the process is to investing, really any type of investing. People obviously do talk about this, but I don't think enough. Like it's the type of decision and the type of impact that it can have. And your ability, like nobody is ever going to be right 100% of the time. Warren Buffett isn't right 100% of the time.

I'm sure, you know, Jeff Bezos in hiring isn't right 100% of the time. I keep going back to, I really love what you said earlier about the basketball court analogy. I think that makes so much sense for both of these. Yeah, no, I appreciate that, which just reminds me another, I have some rules of thumb that I use. And one of those rules of thumb is,

When in doubt, throw them out. I got that as I was getting trained as an as appropriate at Microsoft. Michael Dwan was a leader there who wasn't as appropriate and put me through. It's not just you get trained, but you also do an apprenticeship. This is similar for a bar raiser at Amazon. You're

You're doing shadowing and apprenticing really through a process for a while until you and then are getting shadowed by someone else. And they're like, OK, you're at the bar where now you can fly on your own and do this. So we trust you as a company to go do this thing and basically be a barrier to coming in. If it's not, you know, you're holding the line basically for the company for both the calibration and the potential long term hire for that for that company.

So when in doubt, throw them out. So that means don't be in on the, there's no on the fence, which even gets to everybody on a loop should say hire or no hire based on the competencies and values of

or leadership principles that I'm hiring on. In other words, I don't know what you, you could say you're a hire based on what you interviewed for, but I'm a no hire based on what I interviewed for. And then also you can always get, and that's job one for every interview on the loop is to go after and assess with the key questions, what they're, what they're looking for.

And then they can also, if they have time, get other data, if you will, or say, I also had my gut was off or my spice sensor. I had this question or I don't understand why. And you can add that as extra flavor. But you're making a decision based on what you assess based on your questions going after your assigned competencies and values. So hire or no hire. There's no maybe a hire, maybe, maybe not hire option.

And then remember that you're assessing Fib, you're also creating a candidate experience they're assessing. And so your job, one, again, is to get that data because otherwise there's no match, but they're also collecting data and you want to give them information about the opportunity in the company. Do you ever go into sell mode in an interview? Absolutely. And the faster I get my data, if I think, yes, this is it, then the faster I can go and sell.

But even if I know in my interview or think that they're not a fit for that role, I'll be looking for are they a fit somewhere else. If I think they're not a fit for the role and a fit for somewhere else in the company, I'll do that and assess for that. If I think they're not a fit for the role or the company, I'll still want it to be a good candidate experience because they are out in the world. They're a customer. They could be a customer. They could be a future candidate.

you know, again, I want that to be a good experience regardless. I'm not saying I always do a good job of that or that I can't be intimidating during interviews. I've heard that feedback too. But I work to be connected, authentic, and also create a good candidate experience where I can. Yeah, this is something that Mike brought up to me. He was saying it's such a fallacy that people think that when they think about employer brand, that it's created by the people that they hired. It's like, I think his example was we hired a thousand people or something at Aquantive. But like,

Like we interviewed another 50,000 that we didn't hire. Our brand is actually the sum of what they think of us, not what the small number of people who work at our company think of us in terms of talking about our company out in the world. And I think it's an oft overlooked thing, especially in startup land where people are just obsessed with efficiency and making the hire and moving on and building and selling. A lot of people, I think...

miss the fact that all those rejections that you're making out there, every one of those people has an anecdote of the way that they interacted with your company. Yes. And this also gets to the timeliness of scheduling the interview, communication back and forth before, after, collecting information, how was the experience? That's the other, you don't know. So that's certainly...

asking, getting a survey back, getting feedback back on what it was and then being able to use that to improve the process. And that's an important part as well of it. One of the things we didn't talk about specifically is the kind of interview questions and behavioral, the kinds of questions and the way that Amazon and Microsoft and that I've been trained even before that to do interviews is this behavioral approach.

questioning or behavioral interviewing, which says, what is the past? It's basically what you're doing is getting an example of past experience or behavior where they demonstrated this skill or knowledge or capability or, again, critical thinking or judgment. And you're asking a question that will demonstrate where they're actually showing or they're telling you about when they demonstrated this capability or skill set before.

And so that a lot of those questions are they start with tell me about a time when, you know, blank, blank, blank. It's not yes or no. And it's and typically, again, an interview, if you're doing it, you only have a chance to ask one or two or maybe three questions. And then you're probing within that question a lot to really get to specifics. That's the kind or type of interview method, if you will, that.

How have you learned to handle the jumping off point of that question? You're walking in to tell me about a time when you...

I don't know, made a hard decision, whatever. Actually, I want to hear Anna's, what's your favorite sort of question? I have a portfolio of questions. I have hundreds of questions that I, in my, well, one of the things, again, that benefit of having been around the block a few times is I do have this data bank of questions and answers where I've asked and I have my data set of like what kinds of answers and what I think are,

you know, good answers and what are not. It's just starting out with how I start out interviews. I do say, you know, Hey, I'm going to, who I am, my role. And I'm really happy to meet you. We're here to talk about X, Y, Z opportunity. You know, thanks for making time to come in today or to talk with us today. So I let them know like who I am, you know, what we're doing, setting it up. And that starts or that starts. And I say, and I'm going to take, I usually I'm typing. I said, I'm going to take notes.

And hopefully that's not disruptive. I'll try to make eye contact. I'm usually pretty good at it. But and you can also take notes if you want. And so I try to set up the interview like that and setting that up. I think a nice thing to work to do. And also I'll make time at the end to answer your questions, but I'll start out with some questions so that they know what to expect.

And then I usually start out with a couple of what I call softball questions that also give me some information about how they've prepared, how interested they are. And so, and again, I've asked these questions for decades now at all companies and all different levels, whether they're college grads or senior executives. And those questions are,

One, what percent fit on a scale of zero to 100? What percent fit are you for this opportunity and why? What are the three to five key factors that are why you say you're X percent fit? From that question, you know, some will say 80 percent, 75 percent.

90%, 100%. And what it tells you, again, if someone... First of all, I say opportunity. I don't say role. I don't say job. And when I say opportunity, I mean opportunity in the broadest sense because that's including the specific role. It's talking about that business, that sector, because it could be a division of Microsoft or it could be a startup company. It could be, but it's talking about the role and the company and the people. It's everything. And so what they say...

And how they match. So what do they know about the role? How have they reflected that role to their own experience? And how have they reflected the company and the business and the product service, whatever it is, all of that. And how enthusiastic are they? That'll show up as well. And how prepared are they? We'll show up in that. And the next question I ask is, on a scale of one to 10, what level of interest do you have in the opportunity and why? What are the three to five?

key factors in your answer. And I say, and 10 is you'd pay me money to have this job. And one is I couldn't pay enough money. And also do people answer the question? Will they give me a number in both those examples? It's another piece of feedback. Have you gotten answers from one to 10? Yes, I have. Mostly on the five to 10. Have you ever hired anybody who gave you a less than five number? No.

The number is the least important. Yeah. Although if someone gives you a 10 or a 100, I usually coach them to give me something out, which I have received both those answers before more than once. But it's really the thinking that goes, that's the most important thing, right? Not the actual, not the actual answer.

And I'd imagine that guides you to a place where, you know, they're talking about why they're interested in the roles. Then you have context of their perception of the company, their perception of the opportunity, their perception. Like I would assume your next set of questions, no matter what value you're interviewing for, sort of get shaped from there.

Actually, no. The questions, I have my questions that I'm diving specifically. Now, if I will collect information and I'll use it and I might ask additional questions, but I have a job to do. So if I'm interviewing and I'm interviewing for, does someone have good judgment or their critical thinking or their management leadership, then I have a set of questions that I have or a question or two that I'm going to ask that I have to ask to get that information.

in my mind. Now, again, I could use another one. I could segue. It's true. I could segue potentially using those, but then mostly I'll say, okay, we're going to, you know, we're going to move on to, you know, thanks for that. Or sometimes they'll share, you know, great stuff about their, why they're

prior experience is a super great match for the stroll. I'm like, gosh, that's great. Or that was a super accomplishment. I also try to really let the, let the candidate know that I'm listening in an authentic way, which is like, you know, people share what they've done. It's like, it's impressive. You're like, Oh God, that's great. That's awesome. Congratulations. Thanks for sharing.

And keep going. So again, I'm not perfect at that, but I pay attention. I try to do that as I'm going along. Yeah, let's jump into one of these questions where you have a job to do. It's a question that, you know, you've premeditated. You've said, OK, I'm mapping the fact that I'm interviewing for this characteristic to this question. Let's say that characteristic has good judgment.

What one or a few questions do you ask to tease that out without being obvious and direct and asking the person, tell me about a time where you exhibited good judgment? So a couple of things. One is I actually, it doesn't matter if the person knows that I am actually, you know, assessing their judgment. It's not like a trick, right? I'm not trying to trick. Yeah, that's the other thing. I'm not actually trying to trick or be devious or deceptive. That's not my intent during all this. All at the same time, I also don't want to be like, oh yeah, what's true.

What level of judgment do you have? Because that makes people uncomfortable. It's not actually the best way to get at it. So again, you could ask, those are battery questions, but one of my favorite questions for assessing judgment is,

Tell me about a decision that you made in the last year or two, a significant decision. One of the biggest decisions I say, you know, we've made small decisions every day, business around different things, but I want you to give me a significant decision, a business decision that you made in the last year or two that you would change if you could. Why do you pick that decision? Tell me specifically about the decision, what you would change and what you would do differently. And then what are you looking for in answers to that? Yeah.

Well, you're looking for, again, when you're assessing a question, there is the STAR method is sort of this typical behavioral interviewing method for, you know, what's the situation that they're starting with describing the situation. And then they're telling you what actions or what they actually did. Right. And then how they actually did it and sort of the why behind what they did.

how they did what they did and what kind of results they got after they took those actions. So the situation or task, the actual action and sort of why behind the action and the actual results that they got in a, in a very sort of as qualitative or quantitative kind of results that they actually got, what you're looking for in that, you know, for me, I'm looking in the judgment, how, so first of all, what, what scale and scope of the decision. So one of the things about assessing, um,

is for whatever, there's an entry level position or it's a senior level executive is scope and scale, right? So scope and scale and capability and impact. So, and what do they demonstrate it? And so when you ask for one of the most significant decisions, if they pick a very small scope kind of decision, then you're like, okay, well, that was not significant decision. So you're like, well, that's not in good judgment that you pick that decision, right?

right? Or that's the biggest scope and scale you've ever had, right? So then you pick that decision. Yeah, right. One of two things is wrong. Either you don't have experience making big decisions, or you do and you just exhibited bad judgment by not telling me about it. And again, you're not trying to trick someone, you're really trying to set them up, which is why I say, and if they still, and sometimes people, I'll give them do overs, if they pick something, I'm like, well, that, you know, it's not really what I'm looking for. Can you, is there something else? Or so again, I'm not trying to, I want the candidate, if

They're there in the process at this point. No, I'm trying to help them be successful. I'm not trying to trick them. I actually want them to be a fit, right? But I'm trying to figure out, I'm trying to get the information that helps me

know that or not to the best you know of my ability and so that's what i'm doing through that process and i'll say oh that's not you know is there another one or i'll give them an example i'll say when i was you know building a business at a search business at microsoft or when i decided whatever at the southern whatever is appropriate i will pick something out of mind to even help them give an example of this is what i'm looking for if they're struggling so

My point is you're looking for what the, to demonstrate the capability, what level of capability is it? Is it, does it fit for the role that you're assessing, looking for? Another one that I, it's like a teamwork, a collaboration or earn trust in Amazon leadership principles is I call it the, and Microsoft has one similar one. It's again, it's for a teamwork, but it's an adverse, I call it adverse situation question, which is,

Tell me about a time when you had a problem, a disagreement with a peer and a big disagreement, not like we have small disagreements all the time, but a big disagreement, a work disagreement. And tell me about...

One of your biggest ones that you've ever had in your whole career. Tell me what it was, what happened, what did you do, what did they do and what happened? And then I have them walk me through specifically in that. And again, you'll see, well, how did they work through that? What was their point of view? Did they understand the other party's point of view? Were they able to articulate it or not?

And, again, how did it resolve? What did they learn? What would they do differently, if anything? Again, you start to get to self-awareness, self-critical, vocally self-critical capability as well as their ability to team and understand and work with other points of view. Well, Anna, what didn't we ask you about the world of interviewing that would be helpful for folks to know?

The only thing that comes to mind for me is we didn't really touch on the debrief. Like, how do you get all the data in a room and then make a decision? Oh, yeah. There's a pre-brief, pre-guidance. The hiring manager owns the pre-brief. That's the other thing. I think accountability for hiring that maybe that's something, again, we didn't talk about. So this whole the work and effort they go into preparing for hiring.

First of all, the role, the interview loop, the resume, certainly the process upfront and sourcing and the ownership of the process. Recruiting does not own hiring or they are part of it. The hiring manager, hiring managers and the team that that hiring manager picks to partner on that process, but the hiring manager owns the process. The hiring manager is the one who's responsible and accountable and needs to be the quarterback also of the process.

It's another role of the as appropriate at Microsoft or Barraiser at Amazon is to make sure that hiring managers are doing what they're supposed to be doing and give that feedback directly to hiring managers if they're not owning it and being the quarterback for the process. And

running a process that is effective as well as efficient. And so that does have to do with pre-brief. So there's, you're going to send out and assign roles. You're going to prepare the loop ahead of time and

When there's a new role, having a conversation or a pre-brief about the role, what you're looking for, and everyone's part on the loop is important. And so, again, an email and then an actual meeting. It could be a short meeting, like a 30-minute meeting, again, to do that. And then the debrief after, where everyone has to get their feedback in within 24 hours. Again, you want to be efficient. And then you schedule a debrief.

With everybody to talk about. Both pre-brief and debrief hiring manager should be leading both of those? Yes, yes, yes. And when you walk into the room for the debrief, everyone's weighed in with their information. Presumably the hiring manager has looked at everyone's feedback, but has anyone else seen it?

Yes. After you have written your feedback, after someone has submitted their feedback, then everybody should be able to read whatever feedback's in. And it's actually another great process point that you really do want everybody, if they can, to read the feedback before they get in the room for the debrief. So then, again, it goes much more efficiently. If they can't, they can do it, you know, real time. But

And that's where they're going to share their point of view on their, on their, what they were interviewing for their science. There's even another point in here that you're probably assuming from being in high performing recruiting organizations for a while, but I can tell you it's definitely not the case for a lot of folks, which is that everybody who is on the loop writes down their feedback after the loop, right? Yeah.

Yes. And this is a practice. It's one of those things where, yeah, it's extra work. Like, you know, and it's also one of the things as I, you know, I loved it when I was, you know, as a senior leader in any organization, but people like, oh, I can't wait to have you. Like, you don't do anything. You just think. I'm like, are you kidding me? I'm doing stuff, doing stuff all the time. Yes, I think some of the time, but there are no jobs that are just thinking jobs. There are a lot of, you know, all jobs have doing and thinking in them. So, yeah.

That's one of my favorite two by twos to do with especially new college grads. That's a little digression, but it gets to this point of, yeah, it's a lot of work doing selecting people. And guess what? It's the most important thing. I actually think it's one of the most important things a business does. And you started out the podcast that way guys. And I really, I do like who you bring on the bus and what's seen on the bus as part of your team. That's, that's it. And that's where it's usually the biggest expense companies have. I mean, all of this, right. People, people,

So it's super important and it's worth putting in the right effort up front to get the right people on the bus and the right scene on the bus. And yes, writing down the feedback, submitting, it doesn't need to be a narrative, a novel. Again, it can be shorthand, but it needs to be understood, which is why at the high performing interviewing teams that do this at big and small companies, just write the feedback and then edit it right away and send it in when they're done with the interview.

If it's not clear and you need to like, are you busy day, but it's, you know, end of the day, you're going from meeting to meeting or interview to meeting to say, yeah, I'm going to do it at night or the next morning and let it settle a little bit to say, yeah, here's my hire or no hire based on what I, you know, what I heard. Well, that's a good place as any to leave it. Well, Anna, thank you so much for your time. Yeah. Thank you so much. This is great. Thanks guys. Super fun. Take care. Likewise. Stay safe.