Improve your decision-making, frameworks for learning, backcasting, and more | Annie Duke (#60 rebroadcast)
02:32:27 Share

Hey everyone, welcome to The Drive Podcast. I'm your host, Peter Attia. This podcast, my website, and my weekly newsletter all focus on the goal of translating the science of longevity into something accessible for everyone. Our goal is to provide the best content in health and wellness, and we've established a great team of analysts to make this happen.

It is extremely important to me to provide all of this content without relying on paid ads. To do this, our work is made entirely possible by our members. And in return, we offer exclusive member-only content and benefits above and beyond what is available for free.

If you want to take your knowledge of this space to the next level, it's our goal to ensure members get back much more than the price of a subscription. If you want to learn more about the benefits of our premium membership, head over to peteratiamd.com forward slash subscribe.

Welcome to a special episode of The Drive. For this week's episode, we want to rebroadcast a previous episode that I did with Annie Duke that first aired in July of 2019. I reached out to Annie, a decision strategist and World Series of Poker champion, to have a conversation after reading her bestselling book, Thinking in Bets, Making Smarter Decisions When You Don't Have All the Facts.

It was one of those reads that I just couldn't put down once I picked it up, and I wanted to share it with everyone. In this conversation, we go deep into certainty, decision-making, and of course, poker. We talk about why poker is, in many ways, the model system for decision-making in the real world, i.e., in a world where we have incomplete information. We also discuss backcasting, a topic I have since written and spoken at length about, and the importance of doing a premortem, not just a postmortem.

We discuss the decision matrix, meaning how you can look at and evaluate decisions after the fact when you have to be critical of both the decision-making process and the outcome. I think this episode remains just as important today as it did to me on the day when we recorded it over five years ago, and it contains a lot of valuable and actionable information that you can carry into your day-to-day decision-making and behaviors. So without further delay, please enjoy or re-enjoy my conversation with Annie Duke.

Annie, thanks so much for coming all the way up to New York. I'd like to say just to see me, but now you've let the cat out of the bag that I'm the second excuse to be here today. Yes. When I get requests, I try to pack them into the same day for efficiency purposes. Are you from Philly originally? I'm not from Philly, but my father is from Philly originally. So yeah, my dad grew up in Philadelphia and

graduated from West Philly High, actually. So he's born and bred. And I live in Philadelphia now. It's actually my third time through. So there's...

Clearly some kind of magnetic center to Philadelphia for me. Are you an Eagles fan? I am. I'm sorry. I can't. I'm so angry about them beating the Patriots two years ago. I'm just so upset about it. Oh, please. Oh, I'm so sorry for you that they didn't get their sixth Super Bowl that day and they had to wait two more years. I don't know. It just bothered me. All right. So wait, are you a Patriots fan? I am. Okay. So interestingly enough, I am a Red Sox fan because I grew up in New England and it's

Kind of just an accident of sort of the timing of my own life that I'm an Eagles fan, which has to do with I was never much interested in football when I was growing up, which isn't that crazy for when I was growing up in terms of the Patriots because they were awful. Just horrible. It was horrible. So I like the Celtics because Bird and Kevin McHale and obviously the Red Sox. It was like Fred Lynn and Carl Yastrzemski and Carlton Fisk and like that amazing team.

So, and we used to go like once a year. So I would like to remind you as you're so upset that they didn't get number six. On that day. Of what it was like to be a Red Sox fan. I know, I know. Well, you brought up Fisk, right? So it's like, yeah. Was it the 79 team or the 76 team? I think it was 76. 76. Someone's going to correct me, but yeah. Where it's like almost. No, I know. That's, it's hard to watch. You know what it is? I'm a Belichick fan. So people sort of say, oh, you're a Patriots fan. Like, wow, wow, wow. But.

It's not that I disproportionately like anyone on that team, although I think Tom Brady is, I think, unequivocally the greatest quarterback of all time at this point.

But to me, it's just, it's Belichick. Like I'm obsessed with the way this guy operates. Like you always get asked the question, like if you could meet anybody famous, who would it be? I mean, at the top of my list would have to be him. Like, and again, it wouldn't be in the setting of him giving glib answers in a news conference. Cause that wouldn't be that much fun. Although it's funny as hell to watch. If I could spend a day on his boat with him, really understanding the discipline that he brings to what he does, it would just

It would just amaze me. There's a beautiful article that was written probably five years ago. It was just told in a really funny way because it was the gist of the article was everybody's got a Belichick story and it's players, coaches, all telling these stories of him that, you

highlight this single-minded focus of winning that is kind of amazing to me even though from a life ethos standpoint I don't really care that much about sports anymore I've sort of there was a day when it really upset me if my team lost now it's sort of oh that sucks well that

It's good that you've managed to come to that point. Oh my God. I could tell you stories about when I was in high school. If my team lost, it was like, if you want to get at least one degree, just one degree from Bill Belichick, you should have Michael Lombardi on gridiron genius. It's a great book about leadership. He worked with Bill Walsh, worked with,

Belichick. He really kind of talks about what the leadership qualities of those two particular coaches. He's really great. You should look him up. Do you know him? I do, yes. Well, then I'm going to ask you to make that introduction. I will make that introduction. I met Bill Walsh about a month before he died, and it was the most sort of simple... He was at Stanford, and I guess he was getting some medical care, and I was there...

I lived in the Bay Area at the time, but I still swim at the Stanford pool. And one day I was just finishing a swim and I was walking back to the parking lot and I can't believe it. But Bill Walsh is like walking alone, like on this part of the campus. I normally would never interrupt somebody to be the fan boy because I just feel like it's sort of, I don't know, that's not my style to do it, but I couldn't help it. And I was, oh my God.

I just want to shake your hand. And, and of course, like a total idiot, I just started telling him all of my favorite moments of his coaching career as though he doesn't remember. Like, do you remember the drive at the end of the second Bengal Super Bowl when there were only two minutes and four seconds left? And remember when Taylor, and he was just the kindest. He totally entertained all of my nonsense.

Like, yeah, those were some great times. Those are some great memories. And sadly, he died like a month later. I mean, he was sick in that moment. He was totally able to. Anyway, so but that said, I'm not happy about the Eagles. And we'll just let that slide.

Literally, it was like two years later. Here's the thing. Belichick is in no danger of not getting into the Hall of Fame. Neither is Tom Brady. This is what I'm trying to remind you of the Red Sox. Letting the Eagles have their one moment. Yeah, well, now the Red Sox can't stop doing it. It's also sort of... Yeah, because I'd like to remind you that the Eagles did get their first try in like ever. Yeah.

at the Super Bowl against the Patriots in the first place and lost. So like, come on, it's like crumbs, crumbs for the peasants. So first of all, I read your book. I don't know how long ago it's been, but I mean, it was probably 10 minutes before I reached out to you on Twitter. I loved it. And I think we all sort of read books and apply our own filter to them. And of course, one of the filters is on this business of decision making. And in

In medicine, we make decisions all the time and we make decisions with incomplete information. And I remember explaining to somebody once something that I think I either heard you talk about on a podcast after the fact and describe it more eloquently. And I think you also alluded to it in the book, which is people always think of chess as this perfect game for decision making. It's actually my wife I was having this discussion with. I said, no,

Actually chess is not a great example because everybody has perfect knowledge at every moment in the game So it's not to discount the importance of strategy in chess people who are brilliant at chess are brilliant But it's not the perfect analog for decision making in life for that poker is much better Because you have incomplete information and that tends to mimic what we see in life and in medicine I think you almost never have complete information and that's sort of one of the things that

is both frustrating because how can you make a serious decision when you don't know all of the facts, but at the same time I think makes it intriguing and interesting and gives you this sense of there's a craft that can be mastered here that is less sort of amenable to robotic decision-making. Chess is missing this particular element which decision-making in general has which has to do with incomplete information. So in chess,

I can see the player's whole position. And what that means is that I should be able to calculate out if I'm considering a move, I know what all the possible moves are that the player can make. And then I can think of what all the possible moves are that I can make a response and so on and so forth. So that I should be able to calculate out with absolute perfect knowledge here what the right move is at any given time. It's missing another element though that I think is really important to think about, which is a very strong influence of luck. So in chess...

the pieces stay where they are. Nobody takes dice, you roll an 11 and two pawns come off the board, or if it's snake eyes, checkmate. Or maybe something good happens, you roll a six and you get an extra queen. That doesn't happen in chess, so that there's this, we know that the pieces are only ever going to move through an act of skill.

And this idea of the strong influence of luck is really, really important. Meaning, if I were to do the same thing in the same situation, and then post-decision, are there other things that could reasonably happen that I don't have control over? I don't have control over which thing is actually going to occur. It's actually a really important element to the way we make decisions. If you think about it in poker, if I put my money in with aces,

and you put your money in with fives, you know, 80-ish percent to win there, slightly better. That means 80% of the time I'm going to win. But that doesn't mean that you can't win. 20% of the time you can win. And I have no control either way. I don't know if on that particular iteration, I'm going to see the 80% or the 20%. That really crosses all decision making. So you can tell me this would be my...

intuition is not a medical practitioner, but not only are you dealing with incomplete information in medicine, but even if you did have complete information, if you apply a treatment in a situation and it happens to work,

There's all sorts of stuff going on, I assume that's relatively stochastic that we don't really understand, where if we were to do that same thing again, it may not work in the next situation. So we always want to think about that because that has a very big effect on our decision making. Yeah, you're absolutely correct. And just to hear you explain it that way, that might even be the more important factor actually than the...

absence of complete information. Now the definition of a chaotic system versus just a stochastic system is even more profound where even a slight change in the initial condition can produce a profound change in the outcome. We don't even have to posit that. We can just leave it at the same initial conditions can produce different outcomes. Right. We can kind of unpeel it in this particular way.

Let's say that we're dealing in a world where we had perfect information. I think it's important to think about this to understand why chess is really so bad in terms of thinking about these kinds of decisions.

It's an interesting decision-making problem. I'm not trying to slam chess. I think it's a great game and it teaches critical thinking skills and it teaches you to think many moves ahead and to think about what other people know. And there's all sorts of great benefits that it has. It just happens to be missing a couple of things that I think are really important. But let's say that we had perfect information. For example, if I have a coin and I've weighed it,

so that I know that the weight is correct, I've examined it and I can see that it's two-sided, then I now have perfect information. I know exactly what I need to know about the coin to understand how often it's going to flip heads and how often it's going to land on tails. But what I can't know is what it will land on the next try.

So that's that issue of luck. So if you flip and it lands heads, it's reasonable for me to say it could have landed tails. And if we did that again, that tails is totally a possibility, regardless of what I happen to have seen on this particular flip.

So and then what we can do is say, OK, so even in conditions of perfect information, we don't really know which outcome we're going to get. We might know the frequency, how often that might occur, how often we could expect it to occur. But we don't know for sure what will happen this time. And then you can take it further and say, but what if you haven't examined the coin? And based on some set of outcomes, you're now trying to derive what the coin looks like.

Now that becomes even harder. And that's where you get into something that looks very little like chess. Why is it, do you think, that as humans, we, at least most people, most of us, don't seem innately wired to understand probability theory? Because I studied math as an undergrad and I spent so much time in it, I think I take for granted now how...

Easy it is for me to think probabilistically I sort of view the world as a stochastic map But I'm pretty sure I wasn't born that way and I'm pretty sure that nothing maybe this isn't correct but I doubt that we were Evolutionarily wired to do that or I feel like we would see it more often in the way people behave and I feel like the absence of that in the general Person's point of view who hasn't been forced to be trained in that way speaks to that

This strikes me as one of the more difficult things in explaining anything. This touches everything from politics to business to science, because it's too easy to say,

Well, that didn't happen. Therefore, that couldn't have happened as opposed to describing it the way you did, which is no, actually, it could have happened. And if the same initial circumstance happens again, by the way, this thing could happen. I would imagine if we were to take the distribution of how naturally probabilistic thinking comes to somebody, I'm guessing that you were probably born closer to the right tail.

That being said, I guarantee you that if I followed you around for a day, I could find all sorts of spots where you're not thinking probabilistically. And I could in particular find all sorts of spots where...

Your initial gut reaction is not probabilistic and you have to stop yourself. I'm positive I'll give you an example of a game. I played that's not an exact example of this, but it humbled the hell out of me And I always go back and play this with other people So the idea of the game was to teach people what a 95 confidence interval looks like And this is back when I was at a consulting firm called mckinsey and company and I was a part of the risk practice Our whole job was walking around and helping companies manage risk

So we did this at an offsite one day to help us kind of figure this out. So the moderator goes up with 20 questions. Have you played this game? So I have, and this is one of Phil Tetlock's favorite who wrote Expert Political Judgment and Super Forecasting, which I highly recommend anybody interested in probabilistic

I ended up reading super forecasting years after this. Yeah. So, you know, he's got this in there. Yeah. So I'll tell it just for listeners. So you take, and I still have it by the way, I still have the 20 questions. I was terrible at this. I'll tell you what my score was after. And I'm so pissed about it too. Like how bad I was. Mine was like zero. So go ahead.

And I've really tried. It's just like the questions that I got were really out of my knowledge base. And it was like, I was trying to have really wide. It was a disaster. Okay. So what the moderator does is they give you 20 questions. Each one has a quantitative answer.

Answer that is known. That is known, yeah. And the question for you, the participant, is for each question, give me a bracket that contains the answer. So, for example, what is the average height of a male in the United States? An appropriate answer would be somewhere between...

four foot six and six foot one, because that would be a reasonable guess at a bracket that contains the average height of a male. Right. So the way that I've heard is that you want the bracket to contain it 95% of the time. Which means after you've done all 20, you should have exactly one incorrect and 19 correct. And the purpose of saying 95% confidence is you

If they said to you, you have to have 100% confidence, you can sandbag it. One inch tall. Yeah, they're somewhere from one inch tall to a million miles tall. And the questions, in fairness to people like you and I who have flunked the test, the questions are hard. So did you get, how many votes did Abraham Lincoln receive? I didn't get that one. I got like, what is the distance between Jupiter and its nearest moon? Right, I know. It's like, I don't have any goddamn clue. But...

How many people were voting during Lincoln's? I don't know. And I had this really wide range and I was great. But it's such a great exercise because you think, okay, I've got this under control. I'm going to just try my best to get everyone correct.

With a reasonable guess and I'll probably get one wrong and I think in the end I got 12 right and 8 wrong which of course is like a 60% confidence interval on things that are perfectly knowable and so yes, I completely agree with you and of course your initial example is a better one, which is how many times do I make

either a decision or have a gut reaction that I have to actively talk myself out of. The reason why I bring that up is that I think that there's a couple of things that we want to sort of keep in mind. And then if you want to talk about sort of like from a

like an evolutionary story, why it is that we're not particularly wired to think probabilistically we can get into that. But what I want to get across is this, that first of all, we're all sort of like sitting, we have proclivities, you know, where we're sitting on a distribution, somewhere on the distribution in terms of sort of how we're born, how we're wired to think probabilistically.

And that doesn't mean that we should say either we're there or we're not. I mean, the idea is that wherever we sit on the distribution or whatever our personal distribution looks like, we just want to shift it to the right. So wherever you are, you want to accomplish two things. One is it would be really good if you thought probabilistically a little bit more.

Why? Because the world is probabilistic. And so the more accurately you can think about what the outcomes of your decisions might be, because they're not deterministic. When you make a decision, there's many, many things that could occur with some likelihood of each of those things occurring. And understanding that particular fact is incredibly important to good decision making. So the more that you can shift yourself a little bit more on the side of probabilistic thinking, the better off you are, no matter where you start it.

And then the second thing we want to accomplish, which is what I was trying to get at at the catching yourself part, is to understand that we are wired this way. So it's really hard at all times to think that you don't know things for certain, to sort of live in this counterfactual world. And it's frankly kind of impractical. It would be hard to like drive, for example, if we were living counterfactually all the time. So the idea is that when we do make an error to catch it faster, right?

Because for most things, when we make an error and we have too much confidence or we're thinking about something as deterministic when we should be thinking about it as probabilistic, for most things, we don't catch it at all. So if we can just catch it a little bit more often and if we can catch it a little bit more quickly, we're a lot better off. And these small shifts, if you can get a couple percentage points better,

way better off in your decision making. So let's go back to kind of where you started thinking about this, which was through the lens of playing competitive poker. How did you end up playing poker? Okay.

Chaos theory. So let me just say that when I started playing poker, it wasn't all over television because I think that otherwise that would be a really simple answer. Like I was in college and I wanted to make money and I saw poker on TV and I started playing online. I think would be the origin story of the majority of poker players today. Not my origin story because I'm way too old for that. Where did you go to college? Columbia. Okay. Yeah. So you're here in the city. So I'm here in the city and

there's no internet. I mean, I think there's some sort of, I don't know, Al Gore was inventing it or something. I don't know, but it didn't exist.

So the first thing that happens on my way to becoming a professional poker player is that my brother, when he was in high school, got very, very interested in chess. Started really studying it, became incredibly good at the game. He got accepted to Columbia, decided to take a year deferment to live in New York and study with a grandmaster. During that year, he started playing poker.

All sorts of things happened. He lost his college nest egg. He eventually did very well. But in this particular year, he was learning, let's say. And he was learning maybe on the money that had been saved for college. It was a little bit of money. It wasn't a lot. Anyway, he had started playing. And then I followed to New York to go to Columbia a year later. I actually went. I didn't defer. And by the end of that year, my brother was very good. So

So now we're both living in new york at the same time. He actually doesn't end up going much later He ended up going I think for like six months, but he ends up never finishing college and he's playing poker professionally And i'm going and watching him sometimes so i'm just sort of sitting behind what does that mean at this era of time? These are like smoky rooms in new york city or yeah, there was a place called the mayfair which had a game and uh

They had poker and they had backgammon. Before that, my brother was playing at a place called the Bar Point. For people who don't know, that's a backgammon term. And it was like a backgammon club. People also played gin there. It was on 14th and 6th.

And in the back room, there was a poker game. This was before I came around. He was sort of already out of this place by the time that I came around. Although I did spend a summer here where I got to see him sort of playing in that particular place. So he found out about that through like his interest in chess ends up there in the back room. There's a game that starts on Friday night and ends on Monday morning and

And by that, I mean, that's how long the people played. They didn't get up and leave and come back. We can have a whole other discussion about sleep deprivation and decision making, but we'll save that for another time. Yeah, this was in the early 80s. So I believe there was cocaine involved. Yeah, I'm sure.

But at any rate, so he starts playing in that game. And that's actually where he... So he cuts his teeth. That's his apprenticeship. That's where he's cutting his teeth. That's where he's sort of learning how to play. That's where he's losing this college money. And eventually... So he's playing in that game. He's not doing very well. And he starts figuring out, oh...

Actually, it's sort of in the same way he approached chess. He sort of found whatever books he could find and he found a few other people who were kind of really interested in learning the game and he sort of starts working the game. And eventually he basically sort of breaks that game. Like he becomes so good that I can't remember whether they kicked him out or not. But in the meantime, when he had lost his college money and what he starts doing because now he needs money to be able to play.

is he became like the errand person for the game and so they'd be like hey Howard will you go get me a sandwich like at the deli downstairs or go get me some pizza or go get me cigarettes or go whatever and he'd go and do an errand and then he'd get a tip and then he'd save his tips up and then get into the game eventually he didn't need to be the errand person anymore because he turned the tip money into enough money for him to eventually send himself to college for that brief period but

But so he moved from there to the Mayfair, which was... And the Mayfair is where? Is it in Gramercy Park? Okay. Somewhere near there? It's either Gramercy or Chelsea. I think it's Gramercy Park. I think it's near Gramercy Park. It's a nice place.

Then he was playing there for a while, and he falls in at that point with a group that really becomes very famous. Dan Harrington, main event champion. Eric Seidel, who I think has nine world championship bracelets from the World Series of Poker. It might be more than that now. But he's won, I think, like $40 million playing poker. I mean, he's amazing.

Steve Zolotow, also bracelet holder, was in that group. Jason Lester. And these guys are all coming out of this one group. So this is sort of like Daniel Coyle writes about these centers of excellence, or I forget what his term is for them, but these little places that just produce so much greatness, like all these soccer players that come out of this place in Brazil and all these tennis players that come out of this place in Russia. Yeah.

This became the place where everybody's sharpening everybody in this one place. Exactly. If you're a tourist and you hear about this game, you don't get to go and play, obviously, right? Like they'll take your money if you want, but... Yeah, anybody can play and anybody does play. The thing about poker that's part of this stuff about like the hidden information and luck is that that kind of uncertainty leaves open the...

array of explanations for why you might not be doing well. So if I play chess against you a few times, it becomes very clear to me why I'm losing. I'm losing because you're better than I am. I have nothing else to protect my ego. Like I can't put a hat on it and say, no, it's because of something else. See, that's such a great point that's worth pausing on it as well for a moment. There are very few sports or activities that we play that

That are as pure as something like chess where as you said there is zero excuse whereas I can't Think of another activity. I'm sure there are several where if you really want to you can't oh the ref screwed me or all the wind I mean, did you see that or like I'm into archery, right? It's like oh, did you see that animal move out of the way at the last second? He heard the arrow coming or whereas in reality. No, you took a lousy shot, right? I mean, they're all on a spectrum and one thing that we forget is that

there's an equanimity to luck. So if you're playing tennis and there's wind, your opponent is also playing with that wind. If you're in a game, although we like to think so, the refs generally are making bad calls, just sort of period, some good, some bad. And they're not like screwing a particular team, although systematically going against a particular team, although it sort of feels like that. But what we notice is the wind blew our ball.

An interesting thing that happens in sports that are one-on-one, like I play tennis, is that you can blame it on the style matchup. That it's not that like you're not particularly good. That's a great new excuse. I hadn't even really thought of it. You should use that. Like, oh, that was a bad style for me. I mean, I'm actually better than them, but they're lobbing. They were doing a lot of lobbing. My

My high balls weren't really on that day, but I would crush anybody else. So yeah, exactly. So poker basically is just a fertile ground for that thinking. Right. Which means that it doesn't cause you to kind of self-select out. As you said, this was like a nest of what was going to become the best players in the world. And of course, people were still coming back. So he ends up at the Mayfair Wellness College. Well, I guess there's another element here, which is the variable reinforcement of winning.

Wow. So then you've got the whole dopamine side of this coupled with this psychology of I can post hoc rationalize my way all day long. I can rationalize my losses, pump up my winnings from an ego standpoint. And the next win is coming at a point I can't predict. I mean, that is a recipe for success.

get me into this. Actually, let me just bite into that for a second. So this is actually really interesting. So let me just say I didn't play poker at all in college. Like I never played. It wasn't until graduate school that I played three times or something. So I go to graduate school at Penn. I'm studying cognitive science. This becomes important to me later as I'm thinking about the game of poker, this issue that you just brought up, the difference between a variable reinforcement schedule and a fixed reinforcement schedule.

What we can think about is we can talk about ratio or interval schedule. So a ratio schedule would be by number. So if you put a rat in a Skinner box, and for those who don't know, Skinner box has got the lever, and then either there's positive reinforcement like a food pellet comes out or maybe negative reinforcement like a shock. So let's make it positive and say it's a pellet. A ratio schedule is by number. So it would be like every fifth press the rat gets...

a pellet. And an interval schedule is by time. Every five minutes, the rat gets a pellet. And if it's fixed,

What it means is it's actually every five presses or every five minutes. And depending on whether it's ratio or interval, you get different behaviors from the rat. So what happens on the ratio schedule is that the rat has this very even pressing that goes on. This very even, somewhat quick, but not mad press, press, press, press, food.

Press, press, press, press. And then if it's every five minutes, the rat twiddles its thumbs until it's almost five minutes and then it goes crazy. So you get this burst of activity. It's almost like they don't want to miss a second.

They want to make sure that they get that food exactly. They don't want to give up one second where they could have had food. And so you get this big burst of activity. Then you can also take a ratio or interval schedule and instead of it being fixed, you can make it variable. And what that means is it's now on average is the way that you can think about that. So let's say on average, every 20 lever presses, you'll get food or on average, every five minutes, you'll get food.

So now instead of this big burst of activity on the interval schedule on average every five minutes because they're not sort of waiting and then, oh, I think it's about five minutes and they go. Now they're pressing pretty steadily the whole time because they're not sure anymore, right? Is it going to be the second or not? And then on the variable ratio schedule, you get the same thing. You get a lot of pressing very quickly because they don't know is it the next press or not.

So that's all kind of interesting and you'll see different activity depending on whether it's variable or not. What's interesting is when you try to extinguish the behavior. This is what's really key. So when you extinguish a behavior, what you do is you just withdraw the reward. So now you've trained the rat, it's pressing the lever, and then you just stop giving the rat food and the question, what does it do? When it's fixed, it stops doing the thing super fast.

It's like, hey, wait a minute. Like I've been pressing 100 presses of this stupid lever. You haven't given me any food. I give up. I have figured this out. There's no food coming my way. Same thing if you're doing it by time. But when you make it variable, they just don't stop. And it's particularly bad when it's a variable ratio schedule. So I think about it like this. So

Anybody who's ever watched people sitting at a bank of slot machines in Las Vegas, those are on a variable ratio schedule that you'll get a reward. It's going to be on average a certain number of plays. You could get a reward five plays in a row. You could go...

80 plays without really getting much in return. And you kind of don't know because it's on average. And what happens to people sitting at those slot machines is that they'll be going along and there'll be no reward. And you'll hear them say a thing. I'm due. I'm due. And what that means is that it doesn't matter what you do to that machine, you could just turn it off.

What they think is, no, it must be the next press. It must be the next press. And this is exactly what happens to the rats in the Skinner box. You train them that on average, every 20 presses, they get food. And that means sometimes they're getting three pellets in a row. And sometimes they're going 80 presses without getting food. That's obviously out at the tail, no pun intended, but it's just an average. And now you would draw it and they just never stop pressing the lever.

And it looks just like what humans do. And I sort of always imagine the little rats in the cages saying I'm due. So this is exactly the way that something like poker works. If we're thinking about sort of an edge, what that means is that over time, I'm going to get a certain percentage of your money. But in a particular moment, you could win a lot of hands in a row.

In a particular moment, you could not win so many hands in a row. And unless you're a really good statistical aggregator, right? Unless you're sort of stepping back and really looking at what does this look like over the long run, it's really hard to see what the underlying distribution is. It's hard to see whether you're winning or losing to that situation. Well, there's a slight difference, right? Because if I understand correctly, the slot machine is driven by an underlying probability distribution curve.

So the person sitting there playing it like the Skinner rat is kind of right in that they are due, they just don't know when. Whereas the person playing poker, there is no certainty that they will ever win again necessarily.

In other words, it's both luck and some skill. So they could take probabilistic good hands and misplay them and lose on a potential hand that could have been their due, right? Yes. I'm thinking about that. Yeah. Yeah. So it's sort of like in some ways the person at the poker table has an even harder problem than the person at the slot machine. Yes. Because the slot machine is pure luck. So the slot machine simplifies the problem because...

The machine is set to a particular distribution and the probability is set about how much you're losing per dollar. And so if I sit down, the underlying probability is knowable. That's right. And your only task is to pull the lever or push the button. Whereas in poker, you still actually have to do what we're going to get to, which is the skill of this game. Right. And what you're trying to figure out is...

I mean, obviously this isn't true of everybody, but when people sit down at a slot machine, people aren't thinking I won because I'm so skillful. I mean, occasionally they are if they think they have a system. I don't think that's true. Most people are saying like, I'm doing this because I'm going to get lucky and maybe I'm due and whatnot. So I think that you have less illusion about it. So now in poker...

You lay on top of the fact that it's stochastic, that obviously it's a skill game. And in the long run, with enough iterations, your results will be solely determined by skill. Right, because the probability starts to approach equal distribution for everybody, assuming there's no cheating. Well, it would be true even if there's cheating for this reason. What that would mean is that whatever skill elements that you were applying, you wouldn't be able to overcome that.

So all I mean by this is that you have an expected value. That expected value is determined by your play compared to your opponent's play. And when you say skill, do you mean two skills? The first being the ability to look at all the cards that are available and calculate the probability of cards that are not being shown and the

The skill of being able to bluff or are you referring more to the latter than the former? Skill would be the umbrella of all the things that are within your control in the game that have an effect on the outcome. So first of all, let me just step back a second. Sometimes people think that a game isn't skill if in the short run there's a really strong influence of luck. So people will say...

Well, baseball is a game of skill, but poker isn't a game of skill because they're looking at, well, but in poker, I could have the best hand. I could play it perfectly and I can lose to you because of the turn of a card. So therefore, isn't it just luck?

And how can you tell if something is a game of skill? Because I'm obviously making the assertion that in the long run, your results are going to be solely determined by skill. I think the way you said it actually is probably the best definition I've ever heard, which is in the long run, if two different people can play the exact same game over and over and over again and have dramatically different results under the same stochastic input, doesn't that imply that there is a difference that is in the control of the player?

And isn't that the definition of skill? Yes. Because of actions that they take, their distribution is different. Right. So for example, if you and I made up a new game, which would be a really boring game called toss two dice and the person who gets the higher number wins.

If you and I played this game for the next week, we sat here at my table and we played over and over and over again. In the end, we would end up the same. There'd be nothing that we could do that would change the outcome. And therefore, there is no skill in that game. There's no skill in that game. But if we can bet on it. That's right. And baseball, of course, is such an obvious example of skill because while there is luck involved, within a matter of minutes, you would tell the difference between me and a professional baseball player doing anything on a baseball field. There's a really great test.

that you can do to figure out if something is skill. And then let me just get back to baseball for a second to show what's happening, why people are sort of misperceiving with poker, whether it's a skill game or a luck game. So the first thing is you just want to apply a test. Can you lose on purpose, assuming you followed the rules of the game? Now, it's not symmetric. I can

I cannot apply a test of can I win on purpose, but I can apply a test of can I lose on purpose. And I'll tell you why. So tic-tac-toe, let's agree that that's a skill game. It's bottom feeding skill game, but it is. Right. But if you and I play, neither of us can win on purpose.

Assuming that we both know how to play the game because we're going to tie every single time. But one of us can lose on purpose. Very easy to lose on purpose at that game. So we don't want to apply the win on purpose because that's hard, particularly as you narrow the skill gap. Then you can't do that. But I can lose on purpose. And in a game like baseball...

You can lose on purpose in a game like basketball, in a game like chess, in any of these games, you can lose on purpose. We can think of other acts of skill. Like if you're a salesperson, you could make it so that on purpose, you never closed a sale. As an example, it's a game of skill. If you were a lawyer, you could probably make sure you lost every case. I mean, hopefully you don't do this, but, and in poker, you can indeed, I can lose on purpose very easily. I can't win on purpose.

but I can lose on purpose. And that's true of baseball as well. Somebody can lose on purpose, but they cannot win on purpose. That you do not control. That at a basic level is a really good test of whether something is a skill game or something is a luck game. Where I think the confusion comes in is that as you narrow the skill gap, luck starts to play a much bigger role. It starts to kind of show itself that influence of, hey, there's different ways that this could turn out.

and sometimes you're going to see one thing, and sometimes you're going to see another thing, and you kind of don't have control over which thing you happen to see a lot of the time. We start to see that influence as we narrow the skill gap. That's a great point, right? That's why you look at virtually every Super Bowl these days. Most of them come down to the last series, the last possession. I don't remember these stats, but it could be as high as like two-thirds of Super Bowls come down to the last possession. And if you think about that,

You could easily do what I do when looking at the Eagles beating the Patriots and say it came down to that one stupid play when Brady went back for the pass and boom, the ball gets tripped out of his hand. And it's like, if that didn't happen, he absolutely would have connected with Gronk. They absolutely would have won. And whether or not I'm being delusional or not, you can't deny the point that those teams were so damn near perfect. If you took the Red Sox...

and you had them play a Little League team, there'd be basically no influence of luck, right? Like what luck is going to go the way of the Little League team? I mean, with the Red Sox, what luck is going to go against them? Aside from all their players all at once get injured and can't play anymore, they could put in their third stringers and the Little League team is crushed. But if you put the Red Sox against the Yankees...

This is why we have to have seven games and it's probably still not enough.

to try to figure out who's supposed to advance to the next round. And in fact, we know it's very rare in a series that a team wins for nothing. It's very often going to the sixth or seventh game. Why? Because it's very narrow. And it's very often things like you just said. Who wins the coin toss in a football game? That has a huge effect on what the outcome is. Who just gets to go first? Especially in overtime. Oh my gosh, that's a huge influence of luck. Why? Because you've narrowed the skill gap so much.

But if you took that NFL football team and you put them against peewee players... Or even a college team. Or even a college team. It's unbelievable that gap...

No one would care who won the coin toss in overtime. The NFL team, I mean, I don't know how they would get to overtime, but you just magically make overtime appear. You put them in there and the coin toss will have no effect whatsoever. So I think this is something for people to really understand. And so if we think about a game like poker and a lot of these decision-making processes that we do in life, the skill gap is actually quite narrow. So once I've taught you in poker, things like you know the ranking of the hands, right?

Like a flush beats a straight at you. You know you can bluff. You know sort of about betting. You can read your hand. All these things. That's so much of the building blocks of the game in terms of the skill. And how long does it take? Let's hit pause for one second. And just for the person listening to this who maybe doesn't know the ins and outs of poker. When we're talking about poker, you're talking about a variant of poker called Texas Hold'em, correct?

Not necessarily, but we can. Okay, that's what you played, correct? And that's what most of the pros are playing when we turn TV on and stuff like that? So what you see most of the pros playing when you turn the television on is Texas Hold'em. In terms of what most of them are playing, it's not necessarily that game. That's one of the games that they play.

So give us the brief overview of how that game works so that we can understand what those building blocks are that you're talking about, which is everybody within a day could know in detail what you're explaining quickly about how this game works. So this isn't true of all poker games, but it's true.

true of almost all poker games, enough that we can pretend that it's kind of true across the board, that the goal is to make a five-card hand. Again, there are some exceptions to this. I just want to make that clear. But I'm capturing most of poker. If I say you want to make a five-card hand, those five-card hands range from none of the five cards match up. And so all you care about is the top card and the hand. In most poker games, aces are high. And

In some poker games, they're low, but mostly they're high. Sometimes they're high and low. It depends. So you're trying to make a five card hand. Those five card hands range from very bad, meaning none of the cards match up to each other in any way, to really, really good. And really, really good would be a straight, which would be five cards in rank order. Five, six, seven, eight, nine. That's

That would be a straight. 10, Jack, Queen, King, Ace. That would be a straight. You can have a flush. That's when all five of the cards are of the same suit. So you have five hearts or five spades or whatever. Full houses where you have three of one rank, King, King, King, and then two of another rank,

10, 10. So your five cards would be king, king, king, 10, 10. So anyway, people can go and look up the hand rankings. The highest rank is a straight flush? It's a royal straight flush. So that would be an ace, king, queen, jack, 10, all of the same suit. Probability of that hand? I'm not sure what it is, but a straight flush...

It's one in 6,000 something. It hardly ever happens. So let's start there. So basically, almost any poker game you play, you're trying to make a five card hand. And there's a very clear set of rules about what the hierarchy and ranking are. There is no ambiguity. There's no ambiguity when you're playing. Now, different games have different rankings because you can play low. So in low, you're actually trying to make...

Sort of wet and high would be a bad hand. Okay, so then the other thing about most poker games is for most poker games, you're trying to make a five-card hand out of either seven or nine possible cards. That just is also true of most poker games, not all. So in Texas Hold'em, it's a particular form of poker where you have seven cards to work with to try to make your best five. So you're sort of like tossing two of them. But...

Some of the cards everybody gets to use. They're called community cards. So we can think about the variance of poker. So when you watch something like the Cincinnati Kid, for example, Steve McQueen, in that particular game, people are dealt their own five cards. And so you don't get to use any of the five cards that I have. So our cards are unique from each other and you don't get to see any of my cards.

Then you can have variants which are called stud games where all of our cards are private, but you get to see some of my cards. So in a stud game, at the end of the hand, I'll have three cards that are face down.

four cards that are face up to you and I'm still making my best five of those seven but I don't get to use yours it just gives me more information it just gives you more information so once you can see the four cards that are up that gives you some definition to the kinds of hands that I can have so as an example because I only have three down if my four up cards are a spade a club a heart and a diamond

I can't possibly have a flush because the most I could have of any suit now is only four. So you get certain information about the possibilities for what I could be holding because you get to see some of my cards. But yet you still don't get to use any of my cards. Whereas in Texas Hold'em... Now that's a stud game. But it's common. But it's common. So what happens is you get dealt two cards that are private to you. So these are face down. You don't get to see those particular cards.

And then in a particular way, five cards eventually will end up in the middle. So notice that that's seven, but those five cards that are sitting in the middle, both of us get to use. Now, what that means is that these five cards in the middle define for me what your possible holdings are. So that now changes for me what I think that the strength of my hand is. I'll give you a simple example. It just turns out if you have two private cards,

the most of a suit that I could have in my hand would be two. So I could have like two hearts, which means that in order for me to ever make a flush, which is five of the same suit, like five hearts, I need to have three more of those out on the board.

So if I look on the board and you look on the board and we both see that there aren't three of a suit that's on the board, it eliminates the possibility of a flush from the other person's hand. That becomes important because if I have a hand that is vulnerable to a flush, like if I have three of a kind or two pair or something like that,

That now increases how I think about the strength of my own hand. And you get to look at your cards as soon as they're dealt, the two? As soon as they're dealt. You get to see your private cards. Right. So you know what your private cards are the whole time. And then these community cards are coming down in the middle. And those community cards are defining for you what you think the other player might have. Now, this is a lot of where the skill in poker comes from, is understanding what those community cards mean.

Understanding how they define what the other player has how they might relate in a logical way to what the other player has What the probability is that the other players cards match up in some way to those five cards in the middle? Yeah, it's both right? It's what do those cards mean about what you have and what do they not mean about what other players have right and it's interesting because

Depending on how many players are at the table. And what's the typical number of players that are at the table? In Hold'em? Yeah. It's generally nine. Sometimes it's eight. The higher the limit, they tend to limit the number of players more. So if you're playing in a much lower limit game, you might see 10 people at a table. If you're playing in a much higher limit game, it's more likely to be eight. Which is still kind of staggering to me because now... So you have this math of here's what I have, here's what's common. Yeah.

Based on what's common, what can I impute about everybody else? But now I actually have to pay attention to the behavior of the other people. So I can do the probabilistic calculation broadly. I'm doing two probabilistic calculations. The implications of these cards on me, the implications of these cards on people for whom I don't know the other two cards. But now I have to do eight different behavioral checks. Right. So one thing about the way they relate to your own hand is probabilistic and one is a...

So the thing that isn't probabilistic is these relate to my cards, my hand in a certain way. So if I'm looking at the middle hand, like I know what I have.

Hopefully. But what is probabilistic, and this is what's really important, is what does that mean for whether I have the best hand or not? So that I don't know because that's relative to your hand because there isn't an absolute ranking. It's not like, oh, I have a flush, I get to win automatically. Now, that comes up. Sometimes if I look at the center of the board, I have the very best possible hand that a person could hold. That's very rare that that occurs. Most of the time,

I'm somewhere in between the worst possible hand that a person could hold and the best possible hand that a person can hold. So I have to figure out where I'm sitting on that spectrum, right? And it's probabilistic in nature, whether I'm likely to be beating you or not. And I can think about that in...

the sense of sort of just base rates, like if I'm holding this strength hand, how often would I be winning? But then I also have to adjust those probabilities for what you're doing and how you're acting. And how you're acting is going to tell me something. Two people could act in the exact same way, but depending on my model of who they are, it could mean very different things. So I could bet strongly with a hand. And if you call, that could mean that you have a particular range of hands that

And if another person calls, their range of hands that they might call with could be narrower or wider. They could be willing to call with either a hand that's much worse than the bottom end of the range of hands that you'd be willing to call there with, or they could actually be calling where their bottom range is better than what your bottom range is. And that's going to depend on that person.

Does a newcomer to a game have an advantage or a disadvantage? Let's say you and six of your friends have a regular game going and I show up as a first timer. You guys have never played with me. Does that give me an advantage or disadvantage?

in terms of the point you just made about ranging me? All things being equal. So let's pretend you know nothing about this. Until we play enough, you don't know if I'm good or not. So let's assume I know the fundamentals of the game. I would say all things being equal against the game, you're at a disadvantage because you don't know how to range six people. But me personally, I would prefer to play in terms of my ability to range you. I'm going to be better with somebody I've already played with before. It depends on what question you're asking.

And in fact, there's actually been many, many cases in poker where someone has come in and they're doing something that's just really unusual, that is not something that would be expected.

and they do really well a certain period of time, and then the market basically kind of figures out what they're doing. Either they adjust or they don't. But I've seen a lot of players come in and win a lot. In some cases, it's just attributable to their short-term bursts of luck.

But in some cases, they're actually doing something really unexpected and people are actually misranging them. This usually is because they're playing more aggressively with hands that people normally wouldn't play that aggressively with. And then people sort of figure it out and they adjust. And very often that person starts losing after that. Sometimes they adjust and they do other things. But often it has to do with this problem of

sort of what do you have for example like in cycling a totally unknown cyclist can win a stage of a race because they lead out they break out and

The peloton sort of says well, there's no way this person can do such and such because we don't know anything about them And all of a sudden they let them get away Now they're not going to be able to do that the next day because all of a sudden the peloton's going to say well Wait a minute. Wait a minute Now we're going to treat you like we would treat anybody who has the potential to win and your moves now fit into a different frame Right, exactly. That's exactly what happens in poker. The first thing that you're trying to do is figure out

I know sort of in an absolute sense what I have, but I don't know where that sits relative to what you have. And then once I sort of figure that out, and that's probabilistic in nature, I now have to figure out for you what's the best line of play for me. And tell me how the flow of the game goes from here. So the five cards common, we each have our two. How does the betting work? And what changes with the showing of cards or movement of common cards? So you start off, you get dealt two cards.

And there's a round of betting. And you can choose at that point. And this is actually a really important concept in poker. I actually tweeted about it a little while ago because I think people should be talking about it more, which is you're totally allowed to fold. Remember?

Remember when Steve Dubner and Steven Levitt wrote about this in Free Economics, like the benefit of quitting. We do not spend enough time encouraging people to quit. No, we spend a lot of time encouraging people to stick to it. And it's actually for everything that you stick to, there's huge opportunity costs involved, right? There's all the other things that you could have done. And if you're sticking to something that you're negative expectancy to, and I'm not saying money, I mean, I'm saying you might be negative expected happiness to it or negative expected health to it or negative expected whatever.

that you're giving up the opportunity to go do something. I mean, I'm a big fan of quit fast. So what I like, my theory is do lots and lots of things that are basically a free roll where there's very little downside to it. Go try out piano, right? It's like a little bit of your time. You can't die doing it. Just go see if you like it. But then quit fast. Just quit fast and move on to something else. And then if you find out, oh, I really love this. I really want to spend my time doing it. Now stick to it.

So I'm a huge fan of that. But in poker, this idea is very natural for a poker player because you're doing a lot of the quitting part. You're just saying like, no, this isn't a hand that I mathematically want to play. I don't think I realized that. And that actually might answer this question, which is one of the things I've never understood about gambling in general is you lose much more than you win, presumably. No, not in poker. Ah, because of what you just said. Because you can cut your losses. Okay. That answers the question because my question was going to be, I think it answers my question.

Given the power of loss aversion, how the hell do people keep playing? Because you'd think if you only win once out of every 10 hands, you're

That's nine experiences that are each more painful than the pleasure of the win. But of course, you don't view those nine as losses. You just view them as non-wins. You do lose real hands. I mean, you get to the end of a hand and you lose real money. But a lot of the times, if you really feel like you know the table well, you know the dynamic well, you just cut your losses. And you're like, I'm going to let these two guys fight it out over here. Right, exactly. And I'm going to only go to the mat when I really have a shot to win. And presumably losing that still falls into the loss aversion pain.

By that point, the pre-test probability is such that you have a better shot at winning. So it comes back to this kind of variable ratio reward schedule. You're winning enough big pots.

If you're playing this particular form of poker, which is limit, let me just say something about poker. It comes in two styles. One is limited betting. So in limited betting, it means that, for example, if we're playing, maybe we have to bet in like $10 increment. So I can bet $10 and then if you want to raise the bet, you can make it $20. And then if I want to raise it, I could raise it to $30. In other words, we're raising in $10 increments and we can't do anything more than that.

What comes along with that is there's limited exposure to the person, but obviously limits what I can win from you on a particular hand.

And then there's No Limit, which is what you really see on television, which is the super exciting. Someone shoves a million in chips on. Like when I think of poker, I think of rounders. I'm sure there's been a million movies. Which is No Limit. No Limit and snapping Oreos. I want to be around John Malkovich. I want Oreos. And I want to hear a Russian accent. Like that's all right. Anytime that movie's on anywhere, like if I turn on Netflix and somehow it pops up, I'm watching it.

Right. Yeah. That's no limit, which is like that super sexy. Oh my God. It's incredible. I'm going to lay everything on the table. Can I bet my lung? You know, but there's also something called limit. So in limit, a really good limit player at the end of an eight hour session is

a very good limit player is going to have won about 56% of the time. This is a really, really excellent player. Now, go talk to anybody on Wall Street. They can get 100 billion AUM if they say for every bet has a 56% chance of winning, right? I mean, this is a huge ad. So I'm saying this is a really, really, really good limit player.

What that means, though, is that the person that they're playing against is going to win 44% of the time. So think about how much reinforcement they're getting. So sometimes they're going to win three sessions in a row, right? I mean, 44%, that's a lot. This is a great player. That's hard to believe, even when you bring it back to the context of what we keep using as a great comparison, which is sports. Think about this. If the New England Patriots played, let's see, who was the worst team in the league last year? I don't even remember.

If you put the best team against the worst team, it would not be 56-44. It wouldn't even be close. It would be 85-15 or 90-10, even in professional sports. So that tells you how much narrower the gap is here in poker. This is specifically in this limit version of poker. Now, in no limit, you can get these bigger spreads. Ah, okay.

The 56% is dollars, not number of times in which a person wins? 56% of the time is at the end of an eight-hour session, how many times have I won? Have you won? Okay. At the end of that session. Okay.

So over eight hours. Now, just to allow you to understand, it's just a matter of the number of iterations, because sort of in the same way that if the Patriots are 56% against another team, if they play enough times, they're 100% to have won the series. So a poker player who's going to win 56% of the time at the end of eight hours, at the end of 1500 hours will approach 100% that they'll be in the money at the end of the year.

And this is in this limit version of poker. Now, when we get to no limit, basically what's happening is that you're realizing, you're more likely to realize you're... You're going to see huge volatility and you have enormously asymmetric outcomes at times. Right. So here's an interesting thing, actually, because of that.

is that when I started playing poker... I love that we haven't even got to you playing poker yet, but that's okay, because we're going to get there. When I started playing poker, the pool of players was small. So it wasn't on TV, it wasn't on the internet. There just wasn't this huge market of people who were like, oh, I'm going to go and play poker. So what that meant was that this limit version of poker was really what was being played.

in casinos by professionals. And the reason was that when you had someone come in who was not such a great player, making sure they were getting that frequent reinforcement was really, really, really important.

So you can think about it this way. If you have kind of an unlimited supply of new players coming into the game, when you come in, I should just take all your money right then because who cares if you get depressed and go away? Who cares if I hit you over the head and it plays more like chess?

And you're like, well, I'm not going to play anymore because it's not even close. It's not even close. That's ridiculous. Because there's literally 17 people standing behind you waiting to take your seat. So now what I can do is I can just maximize in that moment. I can just take as much of your money as quickly as possible because there's going to be somebody else willing to come into the game. When I first started playing, that was not true. When I first started playing, if someone came in the game who was really playing at negative expected value, you didn't want to take a sledgehammer to them.

You wanted to make sure that they were enjoying themselves, by the way. And so playing these limit games, which were kind of like a softer way that the edge played out over a longer period of time, was kind of the way to go. Because then they won enough. It was like they won enough of the time to feel pretty good. And the thing is that...

I think that people focus when they're thinking about poker on the expected value being about money. And when this person came in and they were losing money, it wasn't that awful for them. When somebody goes to see a sports game, they play money. They get something in return for it, right? They get to watch a good game. When somebody goes to a restaurant...

and they order food, they're giving money. - Yeah, there's a guarantee expected loss. - Right, but they're getting, for them, what they're getting in return is of value. So generally, when these people would come into the game who would be playing at negative expected value if you only thought about money, I think that most of them were actually playing at positive expectancy when you looked overall at what their overall experience was.

Now, the thing is that no matter who you are, it's not fun for somebody to hit you over the head with a sledgehammer. Like here, just give me all your money. It happened in two seconds. Now go away. But when they would come in and they'd get to play with really good players and they'd be learning the game, which is exciting. And they'd feel like, you know, I'm playing at table one at the Bellagio or Aria or whatever. And they're winning enough to feel good.

They're probably learning and this is entertainment for them. Now it's no longer zero sum anymore. It's like, is it zero sum as far as the money is concerned? Sure. But it's not zero sum overall when you look at like, what's the scope of what people value? Because there's all this entertainment happening as well. And I remember there was this guy, I won't say his last name, but this guy named Jay and

who came and played, he came in the late 90s. And he, you know, he's probably mid to late 70s. He had made a bunch of money on Wall Street, moved to Las Vegas to retire. His health was okay. His mind was super sharp.

And I think that he just sort of decided this is what he wanted to do for his retirement. He had a ton of money. And so he started playing in these high limit games with the high limit players at that time. And it was a small group. And he'd come in and play. And the whole group would break.

for dinner. And we'd all go with Jay to go eat at a restaurant in the Mirage. It was at this time, this was before the Bellagio was built. We'd all go eat dinner and he would sometimes have parties at his house and we'd go over to the party at his house or he'd get invited to somebody else's house. And this was his social life. And he was losing a lot of money, but he was happy.

He was having fun. He had this whole group of people and he got to have this fun conversation and he got to learn this game and so on and so forth. And I think this is kind of how he spent the last four or five years of his life. And he was for sure winning to that. So I feel like people sort of think it all sounds very cutthroat.

But it's not. And this is something that certainly I think that the players in the 90s, for sure, were really thinking about, which is what's the value add? How are you bringing value for this? We know how they're bringing value to us. Wow, that's so interesting. I wouldn't have guessed that. But that's a nice symbiosis. There's a really, really famous player named Chip Reese. He sadly passed away about very unexpectedly. He was young.

passed away probably about 15 years ago now. Maybe the greatest poker mind ever. I mean, certainly one of them. And the thing about Chip was he just made everybody happy around him. He made a ton of money playing backgammon. He made lots of money playing poker. And he had the superpower of these people who were very negative expectancy, if you were thinking about money, would be calling him up like dying to play with him because he was nice. It was just a really good time. I think about it as the same thing. Why do you go to a movie?

Well, okay, if we just think about it as money, it seems like a losing proposition. But of course, that's not what we should be really thinking about that as. And so I think that that gets lost a little bit as the game grows and becomes really, really, really big. And so if you don't have a particularly good time and you're sad and you leave, there's somebody else just to take your place.

And I think that that interesting aspect of how it really wasn't very zero-sum at all kind of goes away. I think it's come back a little bit now because the poker economy has contracted. And I think they're sort of coming back to that idea of how does this become fun for everybody. It was a really interesting aspect of the game back then. So when did you figure out you had a knack for this? It's hard to say because I feel like it's all relative. You know, I retired in 2012 and pretty much anything...

that I can think of that's competitive. Poker's really changed the ability for people to be able to

run batches of hands, for example, after the fact to sort of think, oh, here is the situation that I was in. Now I'm going to go and I'm going to run this and sort of see what the right answer would have been so that I can take that into the next time I play. The kind of analytics that people have, the ability for them to discover what the equilibrium point is on a particular decision or whatever, it's just beyond what I can even comprehend. So I left before I would have had to put that work in

If I did put the work in under those conditions, I'm not sure that I would do well. I mean, I know that if I sat down today without doing that work, I would get completely crushed. So in that sense, you can look at me and say, well, I didn't have a knack for it. If I tried to play against today's players...

So you're saying that a professional player today will take every hand of a game and they'll have the simulation after the fact of that hand? Well, not necessarily.

Well, not necessarily every hand, but they can pick hands and they can actually run the same way that you do it in baseball. Well, I was going to say, this sounds a lot like moneyball, which is the advent of analytics to the game. Right, exactly. So they can sit there and they can say, they can just set the parameters. If I am in a particular situation where I think that I have to bluff, and if I give you this range of hands, how often do you fold? If I give you this range of hands, how often do you fold? And they can figure out exactly at what point

if I bet a particular amount, am I winning enough to that bet that I can now try to bluff in that situation that maybe I would have never thought in a million years was a situation that I would be winning to if I were to bluff there because I just didn't have the data.

So I feel like knack is such a strange word because I feel like it's knack relative to what? Well, to your peers at the time. Yeah. So those peers changed. So I started playing in Montana. What year is this? Oh my gosh. I started playing. I turned professional in 94. So it's right around that time. And I was playing against people in Montana who...

A lot of them were retirees who were, this is kind of what they did. Some people have their bridge game or some people are playing golf or some people are playing poker and that's kind of what they wanted to do. So while I think that pretty much everybody who plays poker would like to get better and enjoys the game more, some people are more hungry to do that than others and people are getting different things out of the game. And the people that I was playing with

We're not thinking, let me become a world-class player. At that time, particularly because I had really great mentors, my understanding of the mathematics of that game was so beyond their understanding. I don't think that was part of their enjoyment. So the day that I stepped into that room, I had a knack, certainly compared to the people that I was playing against. I started playing two years before that in that game, and that was sort of where I was learning, and then in 94...

I decided I was a professional poker player and I moved to Las Vegas and I started playing there. And this is how many years after the game in New York? Or was it only your brother that was playing? Only my brother was playing. Yeah, I never played in New York. I don't think I ever played a hand of poker in New York. So you get to Vegas in 96? No, I got to Vegas in 94. Oh, that was the Declaration of Turning Pro. Okay, got it.

Does it become less fun when it's your job now? Does it amplify the stakes? Eventually. But no, not at this time. Poker is a really complicated game. Very, very complex. And I know that there's just so much that I don't know about the game. And I'm totally aware that if I tried to sit down and play against today's players, that they would bury me. Because there's this whole layer of the game that I haven't discovered. But while I was particularly in those early years in the 90s,

I had started off in Montana playing only Limit Hold'em. Now I go and I start to learn more about that game. So I'm starting to sort of peel back the layers of that game. And like most things that you do, as you peel those layers back, what you start to see is that there was a whole bunch of stuff underneath it that you didn't even know existed.

Right. So you think that the onion is a certain size and you peel a layer back and you realize, oh, wait, no, there's like so many more layers under there than I ever could have thought. I was telling this to somebody the other day. My head of research always says this is I don't know if he's paraphrasing somebody or if he coined the term. But the further you get from shore, the deeper the water. Somebody asked me a question about a topic. I don't remember what it was, but I said.

And I wasn't being facetious. I really mean it. I know less about this subject today than I did 10 years ago. And they said, how is that possible? And I said, well, to be completely accurate, I mean, on a relative basis, obviously, on an absolute basis, I know more than I did 10 years ago about subject Y. But my knowledge of how much broader it is today has dwarfed

my absolute knowledge. So on a relative basis, I've gone backwards. Yeah, exactly. That's what's happening. So I'm learning this game, limit, hold them, learning things about poker in general. And then I start expanding my repertoire of the types of games that I play. So I start to learn the Omaha games. I start to learn the stud games. I start to learn pot limit and no limit. So those are different betting structures. And I'm immersed. It's like I'm living and breathing and talking poker.

all the time with peers, with mentors, so on and so forth. So it's not really like a job in that sense. Certainly not during that first eight or 10 years.

I'm playing I'm learning and every second is learning So when did you see sort of robots in the book? But you also figure out it seems pretty early that there are at the risk of oversimplifying two different types of players there are the players who When they win it's because of how well they've done things when they lose it's because of bad luck and that's basically the narrative and

And you seem, and again, I don't know how quickly you came to this understanding, but at some point you seem to come to the understanding of, wait a minute, sometimes I'm winning when I make the wrong bet. And sometimes I'm losing when I make the right bet. And that last one, by the way, must be a hard pill to swallow.

You end up sort of in your own mind formulating a very common tool we use today, which is this decision matrix of, did I use the right process? Yes or no. Did I get the right outcome? Yes or no. That's a lovely two by two, but it's one that you have two squares of that two by two never get talked about, which are the ones that are at odds with each other. Let me really dig into something you just said. So I agree that we're not seeing the whole two by two.

Where I'm going to quibble is this. Correct me if I'm wrong, but I think what you said was you could have made really good decisions and had a good situation and ended up having a bad outcome. And that's the really hard pill to swallow. And I actually think it's the reverse. To me, I think that's the story you tell when you're opening your book, which bringing it back to the New England Patriots is when the tables are turned, right? It's the Pete Carroll story.

Pass call on the two-yard line or whatever in the Super Bowl that gets intercepted which right again by all metrics if you looked at this like you were a robot looking at reams of data the Outcome of that call was positive. That should have been a touchdown, right? That's a question of resulting when we're evaluating other people's decisions particularly people that we don't compare to particularly in a situation where the decision is complicated

We have this simplifier, the shortcut called resulting. So what we do is we say, if I know what the quality of the outcome is, I can work backwards from that to the quality of the decision. So this is what happens to Pete Carroll. It's the last play of Super Bowl 2015. Everybody expects he's down by four to the Patriots, as always. It's always the Patriots. Back to the beginning of the conversation. At any rate, everybody thinks he's going to hand the ball off to Marshawn Lynch.

Instead, he has Russell Wilson, the quarterback, pass the ball. This is very unexpected. The math is very, very complicated here. There's some game theory in there. There's options theory, all sorts of things that people are interested in reading about it. They can go look at Benjamin Morris on 538. Suffice it to say, it is a good play. We'll link to all of this in our show notes, by the way, because we love nerding out on this exact sort of background. So he passes the ball. It's intercepted.

They lose the Super Bowl this way. Malcolm Butler, my hero. Just, I mean. I can't imagine. Can you imagine what I was doing to my TV? I remember looking at my daughter going, earmuffs.

And it was like, fuck, yeah. I mean, I am screaming. My wife's like, all right, I'm going to take the kids out. I'm like, yeah, yeah, yeah, that's fine. Yeah, exactly. So this is what we do. The thing about understanding the quality of the decision is it's complex. It's really hard. You need to know a whole bunch of different things. You need to know, given the decision that was made, what were the possible outcomes.

how often would those outcomes occur? If you think about what are the consequences of each of those outcomes, how much do I prefer each of those outcomes so I can understand what the probability of getting a preferred outcome is? If I have an outcome that I don't prefer, like an interception, how often does that happen? And then you also have to compare it, do that exact process with the other possibilities, the other decisions that you could have made, and now actually compare to see how do I get the highest expected value, the highest chance of actually winning the Super Bowl.

And it's all very complicated and involves what's the interception rate, which people don't know off the top of their head. Or you have to think about if there's only 26 seconds left, how do I get three plays off instead of two? And it's very hard. Suffice it to say, if you pass the ball either on the first or the second play there, because it's second down, 26 seconds left, one timeout. If you pass the ball on either the first or second play there,

It means that you can do that plus hand the ball off to Marshawn Lynch. If all you do is hand the ball off to Marshawn Lynch twice, you don't get that third play. That third play is incredibly valuable, and the cost of the third play is whatever the interception rate is, and the interception rate there is less than 2%. And there's a time management piece to it as well. A pass that is not caught is an immediate time stop.

You could have a bizarre scenario under which you hand it off to Marshawn Lynch. It turns into a bit of a scrum. You have to burn one of your timeouts. Well, you're going to have to no matter what. You only have one left, if I recall. Right. So you have one left. So that's why if you hand it off to Marshawn Lynch...

on the first two plays, assuming the first one fails, that's the end of your timeout. So because you can get the clock to naturally stop on a pass, that's why if you pass on the first or second play, you can get three plays instead of two. But that's all very complex.

So what we tend to do in those situations is we say, that's all really hard and I don't know what any of that is and it's all very opaque to me. So if I understand what the quality of the outcome is, was it good or bad in this particular case, it was disastrous, then I know what the quality of the decision is. So that's what's happening in that Pete Carroll thing. That's called resulting. Resulting is a heuristic, simplifier, right?

Works really well in chess. If I lose to you in chess, I played worse than you. Works very poorly in poker. If I lose to you in poker, who knows? And certainly very poorly in football. We don't really know, depending on how the play worked out, whether it was a good or bad decision. But what's interesting is that people don't do this resulting thing.

in situations where they feel like the quality of the decision is known. And I say feel like because... In quotes, yeah. Right, because they don't necessarily know it. But remember that what I said is resulting is a simplifier. The quality of the decision, it's complicated, it's hard to figure out. So what happens when we feel like we do know the quality of the decision? So I'll give you an example. If I come home one day and I say to you, I got in a car accident.

Do you know if it's my fault or not? No clue. And you won't use the fact that I got in a car accident to work backwards like you do with Pete Carroll because you understand that there are certain things that really are knowable about driving. And you can ask me what those are. Did you break the rules of the road? Were you drinking? You can ask me a whole bunch of things.

So there's all sorts of places like that where we feel like we understand what the decision quality is. And so if we have a bad outcome, we actually tolerate it. This occurs all the time in business as well, right? If somebody makes a decision, let's say they execute on a decision that was made in a committee.

where the whole team agreed that this was the right way to go, and then it doesn't work out, people aren't like, idiot. But if they do something on their own that is not something that everybody always does...

then all of a sudden, if it doesn't work out, they're really under the gun. So we have this issue of transparency that occurs. So now let me get back to this roundabout way. So first of all, let me just give you a good example of the transparency problem. Okay, bear with me on this thought experiment for a second. You don't have ways, it's broken. And you're driving to the airport with your spouse. Obviously, you have to get there in time. And

You get in the car and you go the normal way that you go, that you've always gone. And something happens. There's a big accident on the road and your car is not moving. And you get to the airport and you just missed your flight. Are you getting blamed for that? Is that your fault? Is your spouse mad at you? No, not in my case. I can imagine in some people's cases. But mostly, no, it doesn't feel like it.

Okay, so now you get in your car again, you don't have waves, whatever your phone's broken, you get in your car, and you announce to your spouse, I have a shortcut. I was just talking to Morgan over there. And they told me about this great shortcut. So I'm going to take the shortcut because I've checked it out. I looked on a map, it's better. And now you go to the airport, and there is an accident, car is not moving, you make it to the airport, and you've just missed your flight.

Is your spouse mad at you? Again, mine would not be because she's superhuman. But yes, I think many people in that situation are absolutely getting yelled at. By the way, that's a superhuman spouse. It's like, I cannot believe your stupid shortcut. So this is one of those cases where, so we have this come up all the time. And when it comes up in so many different ways, it comes up in medicine all the time. Because that's such a great example, right? If Marshawn Lynch had been handed the ball and he had fumbled the ball,

same exact outcome, people would be pissed at Lynch. Nobody would be pissed at Pete Carroll. That is exactly right. And they might not even be pissed at Marshawn Lynch. They might just think that the Patriots line is too good. Yeah, the safety or whoever came and stripped the ball out was just exceptional.

Right. And why is that? What's the difference? Because in both cases, you fail. Expected versus not expected. Expected versus not expected. And this comes up, I guarantee you, this comes up in your business all the time. It comes up in your personal life all the time. It's one of the reasons why, for example, if someone's really unhappy in their job, they won't be aggressive enough about going and finding another position. And the reason is that...

The unhappiness in their job has now sort of become expected. Like they're not really blaming themselves for that, but if they go and find a new position and they're not happy there, they're going to feel like it's their fault, as an example. This happens in business strategy. This is the way that teams end up operating, so on and so forth. Okay, so this becomes a really big problem. So every human being understands this difference between kind of transparent and opaque decisions. Now here's my question for you. What decisions do you think are the most transparent to you? Your own decisions.

These have the most transparency to us. So what happens with our own decisions is that we allow uncertainty to bubble to the surface as the explanation for a bad outcome more often than we would for somebody else's decisions.

Why? Because we feel like we know what the quality of our decision is, number one. And number two, we don't really want to question the quality of our decision. So you can think about that sort of in general. Do you hand it off to Marshawn Lynch or do you pass it? It's a question of when are we allowing uncertainty as the explanation for a poor outcome? So it's innovative, unexpected.

opaque, all of those things. We don't allow the uncertainty in the door. But when we feel like we know, because obviously you're supposed to hand it off to Marshawn Lynch, now we let uncertainty into the door. We do this for ourselves as well. So now this brings me full circle to this thing about when we think about this matrix and you say, well, you might make a really good decision and have a bad outcome. That's the most painful.

By the way, that's just my subjective overlay. I don't know that there's any decision-making expert out there that would argue one way or the other. Well, I think that people do think that. So I just want to sort of pick at something. So let's just start again. So what you said was there are these two quadrants, right? Right, that we would think of as discordant. Right. So one of them is you make a bad decision

But you get the right outcome. You get the right outcome. And then you said even more painful is you make a good decision and you get a bad outcome. Now, what I think is interesting is that I would actually argue that that doesn't have a lot of pain associated with it. And let me explain why. Imagine you're looking at this in the aftermath of the outcome. Okay. So in one case, you've gotten a good outcome. In one case, you've gotten a bad outcome. So let me ask you to do the thought experiment. Where do you want to dig in more?

If you get a good outcome, are you looking to dig in to try to find out if you made a bad decision that resulted in that good outcome? Or are you just being like, good outcome, yay me.

But if you have a bad outcome, are you eager? Honestly, I think it depends. Like using sports as an example, I think is a different example versus in medicine. So I've talked about this before on the podcast. In medicine, there's a, or at least in surgery, there's a conference that's typically done called the morbidity and mortality conference where you, as the name suggests, every week you discuss all the morbidities and all the mortalities of the previous week.

And so that is an objective definition. A person...

can basically arrive dead in the hospital from, let's say, a gunshot wound, they still get presented in the conference, even though, let's say you did nothing. They literally showed up in the emergency room, you did CPR for a minute, they were all basically already dead and call time of death. That still shows up, even though there's no blame to be assigned within the confines of how medicine was practiced. So those ones don't get a lot of discussion. The ones that get a lot of discussion are you made a decision that seemed like the right decision at the time,

And it seems defensible, but the outcome was bad. Now, of course, as you've alluded to, these are so complicated because there's still the luck component. There's all these other things that go into it. So I guess if your question is, where do you learn the most versus what causes the most consternation? Those aren't necessarily the same, are they? The point that I'm making is not so much about where do you learn the most, because I think you learn from all four quadrants.

And it's really important to learn from all four quadrants, right? I made a good decision. I got a good outcome. I'd like to understand when I have made a good decision that has a really good expected value. And I want to be able to see that. I want to understand when I made a bad decision that had a bad outcome and where I had negative expected value. And then I can understand that so I can adjust my decision making going forward. And likewise, I'd like to understand when I made a bad decision that had a good outcome and vice versa.

What you just described is actually the whole shebang in terms of the reinforcement of this bad kind of thinking. Because what I'm guessing does not occur every week is the, wow, this patient did way better than expected. But we made the wrong decision. Let's have a conference about that. So let me give you a super simple example to try to get at this. Because I think that this is actually a really big problem, particularly as

I feel like now people have now understood the importance of process language. So if we think about any business movie from the 1980s, they're like, I'm results driven, right? Now you sort of look back at that and you're a little horrified because I think we all understand that we don't want to be results oriented. We kind of get that that's a problem.

And we'd all like to aspire to be Sam Hinckley and trust the process. And we know we're supposed to say process, process, process. But here's the problem. So now let's do this as a two by two. Okay. You have a status quo decision, the expected decision, a consensus decision, and you have a good outcome. Is anybody hailing you as the biggest genius of all time? No, but they're saying good job. You have a status quo decision, right?

that has a bad outcome? Is anybody excoriating you? No, they're saying tough luck. What could you do? You have an unexpected, an opaque, a non-status quo, a non-consensus decision that has a good outcome and they're hailing you as the biggest genius of all time.

you have a non-status quo decision that has a bad outcome and everybody's calling you an idiot. So people understand this. So now let's think about what does that do to somebody's decision making? Well, first of all, they want to avoid the bad outcome. This is where loss aversion comes in. We don't want to be the idiot. That's the one quadrant that you want to avoid. Right. Well, in general,

We want to avoid the whole right side of the table because the right side of the table contains the idiot section. So let's think about how can we avoid the whole right side of the table. Well, one way we can avoid sort of the bad outcomes altogether is to kind of fall down on the not deciding thing. Just sort of not doing anything that's going to cause a big loss or a big win. Either you could do it by omission, you could sort of not act.

You could make very, very low volatility choices. There's things that you could do there to play these strategies. The other thing that you can do, so you can avoid deciding it all, or you can make sure that you're making things that don't really create big losses, so you're not going to end up in the room.

But the other way to deal with that whole right side is to focus on the lower right quadrant and say, I never want to be the idiot in the room. And so therefore, I'm going to make status quo decisions. So let's think about this. For example, it doesn't matter if 80% or so of ear infections are viral. If a patient comes into my office, I'm giving them antibiotics because I don't want to get

yelled at if this happens to be the time. I want to be able to cover my butt and that way nobody can yell at me. So now let's think about this. So what ends up happening is that

We want people to be thinking, I just want to make the best decisions and I don't really want to be afraid of the bad outcomes. I particularly don't want people to be afraid of this idiot quadrant because we would like people to be innovative and we'd like them to be thinking outside the box and we'd like them to sort of push against what the status quo is because that's how obviously we move forward as a society, as a business, as an individual.

But this is what happens. Everybody has a morbidity conference. So even if in the morbidity conference you're talking about process,

you're in the room because of morbidity. So everybody knows I get in the room because of something bad. So here's a simple thought experiment that I use. Imagine that we're investing in real estate, like we have a real estate investing group. And we have a model of the market, obviously. And we have limited resources that we can deploy. And so we invest in a particular project. And at

It ends up doing 15% worse than we expected. We're all in a room. Now, we might be in that room talking about our model. True. We might be in the room talking about our decision process.

But we're having a long meeting in a room about the fact that the project did 15% worse than we expected. Now we do the exact same thing. We invest in this particular project and it does 15% better than expected. Is the same conversation happening? No. So I don't care how much the word process comes out of your mouth. Everybody knows they're supposed to be afraid of bad outcomes then.

And here's the really big problem. And it's true, we can talk about it in terms of medicine as well, but this particular thought experiment that has to do with the real estate, it matters just as much as if you've overestimated the market, which is how you'd end up 15% on the downside, than if you've underestimated the market. Because

your goal is to efficiently allocate your resources within the market that you're allocating the resources in. And if your model's wrong, I don't care if it's wrong, if it's underestimating or overestimating, that's a problem for efficient allocation of resources. Not only that, but when it's 15% to the good, it's a signal that there may be risk that you didn't identify.

So, for example, you could have the mean right, but you may have the variant. That's a really big problem because that has really big implications for what percentage of your bankroll or the total money that you have, the total capital that you have that you're supposed to deploy to that particular project. That's a very important point, by the way, which doesn't, I think, get talked about enough even in that circle, which is people are very fixated on return on capital growth.

but they generally aren't as appreciative of risk-adjusted return on capital. So Rayrock versus Rock is this concept that seems to

be missing from the way a lot of people think about that. So it's good that you brought that point up because there could be a number of problems with that issue, not just maybe you didn't allocate enough, maybe you allocated the right amount or too much. It didn't appreciate what was at risk. Right, exactly. So we want to be seeing those signals, right? Because we'd like to understand it if we underestimated the vol. And in both cases, your model could be totally right. And you can just be a couple of standard deviations away. Or there's a variety of reasons on that particular question.

time that it could have come in low or high that don't have anything to do with your model. But you kind of want to dig in either way because it's something unexpected happened. The other thing, by the way, that happens is that when you have a downside event and you do get in the room, people rarely ask, should we have done worse? They ask, could we have done better? And now again, you're reinforcing for people, I care when things go badly. When things go badly, I want to avoid them. Now, poker happens to be this very pure place where

where should I have done worse, regardless of whether I won or lost, is actually a very important question. Because sometimes you have a hand, I'm thinking about what you have, and let's say that I put you in a range where you're very weak.

And I feel that I'm very strong in comparison to you. So because you're weak, I'm playing the hand. We would call it slow or small where I'm... You're trying to draw as much out as possible. Which means that I'm not betting really big. Now at the end of the hand, you hit a lucky card at the end of the hand, the very last card. I end up losing the hand. But when we turn the hands over, it turns out that you were much stronger than I thought.

In this case, I should have lost more because, yes, you hit a lucky card, but I should have been extracting much more money from you because you would have tolerated it had I had your hand right. Now, it may turn out that there was no way for me to get your hand right, but I'm supposed to explore that because now what I know is that if I knew what your cards were, I would have lost a lot more money than I did, that I actually played the hand very poorly.

For what the hand was that you actually had I actually underplayed my hand and that story that example you use I think to me is one of the best examples of why the title of the book makes sense, which is This isn't a book about poker. Actually. It's a book that uses poker as a model system So for example, let's talk about biology. Why do we do so much biology in? C elegans a worm

Or in this type of a mouse or this yeast, do we care that much about mice and yeast and worms? No, they are model systems that allow us to do things in cleaner ways. And I think that's sort of the beauty of poker, which for the listener, they might think they're talking about poker so much. Why? Well, I think you have to understand enough about poker to understand what you just said.

Because what you just said is it's not often in life we will get that much clarity after the fact, but we can still bring that level of critical thought to our decisions. To me, that is the... If you remember nothing else from this podcast, this is the part I'd want someone to remember. I think of poker as the C.Elegans of decision-making in life. I love that. That's great. And we can think about this on the winning side as well, right? Like I could...

win a hand from you where I really should have won a lot less. Again, if I had had perfect information, I shouldn't have actually won as much as I did. Because, for example, I could misread your hand and I could play a hand really strongly and then I can be the one who hits a lucky card.

And actually, if I had perfect information, I would not have ever put that much money at risk. So we want to explore whether the outcome is good or bad. We want to have equanimity toward the direction that we're exploring. Could I have gotten even more out of you? Could I have done even better? Could I have done worse? And the other thing that besides that direction, by the way, is we want to think orthogonally, like maybe you lost for reasons that had nothing to do with your decision process, or maybe you won for reasons that you totally didn't predict.

This happens in financial markets all the time. You invest in, say, a company because you think that a particular product of theirs is going to do gangbusters or something, and then that product fails, but they happen to acquire another company. By the way, don't take credit for that if that wasn't in your model, but you happen to have won. So we want to be thinking up-down, orthogonal, in all these directions. So now here becomes the problem. We know...

that people don't like to be in this bottom white, they don't like to be in this place of, okay, something really bad happened. And if they do get in that something really bad happened place, they want to be able to say,

but it's not my fault. It's a bad luck situation. It's like, well, my decision process was good and everybody had consensus and I handed the ball off to Marshawn Lynch. What could I do? That was what I was supposed to do. So you're driving people into this place where they're very likely to be trying to build consensus, not necessarily of the good kind, right? Of the false kind.

They may be using data not to find the truth, but to tell a data story that supports them in case they happen to get into the room. So we really like to say the data told us so. They may just be doing what everybody has always done. They may be slow to decide because they need buy-in or the

right kind of sign off too often, or they need a higher level of certainty than they really should in order to make the decision. All sorts of bad things come out of this. And what really ends up coming out of it is that people tend to move slowly. They tend to default toward omissions versus commissions. In other words, not deciding versus deciding. And they tend not to like to

innovate too much. Now is this, my guess is there's a different propensity for this across different fields. So for example, have you ever had a chance to talk to Pete Carroll? I haven't. Just thinking about everything you said, it would be very interesting to have that discussion with him and say, it'd be hard for him to do this, but go back in time to before you made that decision.

I'm pretty sure the only thing Pete Carroll was thinking about was winning. I can't imagine there was even an ounce of thought in his mind that said, whatever decision I make, I want to make sure it's defensible at the press conference.

I just can't imagine he would have thought that, right? I agree. And I think that's what makes Pete Carroll such a great coach. I was just about to say, but wouldn't you see more Pete Carroll-like thinking in something like sports? Or is it less about sports and more about being the head coach versus the defensive coordinator?

who in the end is still making a very important decision, but answers to a coach. Of course, the coach answers to the owner, but I'm not sure I'm articulating the question well, but... No, I actually think that you are. So first of all, we know across the NFL that the slowness to adopt the analytics has been

And that's because in this particular case, the people who are judging you are going to be like the GMs or the fans or that kind of thing. And if they're judging you, they're going to be judging you for the unusual decisions. So I think actually if we go to Belichick, we can kind of see if we go back to this idea of this feeling of transparency, people are forgiving when there's a feeling of transparency. So if you're Pete Carroll and,

at that point in his career, people are much more likely and remember even with him, they really bashed him, but they're much more likely to sort of give you the benefit of the doubt. So I actually saw a great talk by Toby Moskowitz. He's really interesting guy, does a lot of stuff in sports analytics and quant and this kind of thing. And he was talking about Belichick and he said when you look at his decision-making when he was with the Browns, it was dismal compared to the analytics.

And it's not actually until he wins two Super Bowls that you start to see this decision making that's really starting to align with the analytics. It's really starting to be like, I don't care if nobody's going to understand this decision. I'm Bill Belichick and that's the decision I'm going to make. But it's after two Super Bowls that he gets there. Why? Because the fans are now tolerant of it.

They're willing to tolerate the stuff that they don't understand. Why did you go out for it on fourth and five there? That's insane. No, well, because you're Belichick. So what we want to think about is when we think about these, what you said, the morbidity conference, what is that encouraging in the people that you're trying to lead? You're encouraging the behavior that they naturally bend toward, which is I want to stay out of this lower right quadrant.

So I don't want anything bad to happen to me. But if it does happen to me, I'm going to make sure there's so much CYA there that it's going to be okay, that nobody's going to sit there and say, I made a crazy decision. And you're doing that because you're not. You're not exploring both directions, but particularly because you're not getting in the room when something unexpectedly good happens.

And here's the issue. When something unexpectedly good happens, there's so much to learn from that. Is that different, by the way, than when something good happens that shouldn't have happened based on what people didn't know? Now I'm getting sort of Rumsfeld-y in a bit. But okay, I'm making this up. So this could be complete nonsense. So I'm hoping you'll be able to come up with a real example. Remember the whole Y2K thing? Everybody thought...

Oh my God, this could be a free for all, blah, blah, blah. It turned out to be the biggest nothing burger in the history of civilization. Like I have friends in college whose entire job after college was getting ready for Y2K. So nothing happened. Fine. Now there's two ways nothing happened. One is nothing happened because there was really nothing going on and we just didn't understand that. And we did a whole bunch of stuff to be prepared for it.

And in the end, it didn't matter. There's another thing, which is, oh, no, it was a real problem. And all of that work that was done to make sure airplanes didn't fall out of the sky, it worked. Those are kind of different. And I'm using the problem there is there's work involved versus decision being made. So maybe this isn't a great example. I guess what I'm trying to get at is you can have great outcomes because you're

There was going to be a great outcome regardless of your decision. And then you can have great outcomes where you actually made the wrong decision and you got the right outcome. And then there's, you made the right decision and the right outcome. I'm almost wondering, like, are there other historical examples in wars? I'm trying to think of like D-Day or...

Was that invasion on that day the right thing to do? I mean, history will forever be changed as a result of it in the right direction. But has anyone ever gone back and said, well, you know what? Based on the weather pattern, like Eisenhower got lucky. That shouldn't have worked that day. And that was the wrong decision. They should have waited a day or something to that effect. Yes. So number one, with Y2K, there's a fourth thing that you can say. So I agree with all three things that you pointed out. It could have been good decision, good outcome. It could have been...

It would have happened no matter what we did. And so we were irrelevant. It could be that we had a good outcome and it was all sort of a waste of time. So it was a bad decision to do anything. It could be it doesn't really matter because when you sort of look at what the range of possibilities were, given the unknown information, the downside was so bad.

that you were actually supposed to protect against that regardless. And were you going to waste a whole bunch of human productivity doing it? Well, it's not really a waste because it just acts as a hedge against just in case it's the worst case scenario. And particularly with something like that, we would want to protect ourselves against that. But unless you dig into the win, you don't find that stuff out. And I think that that's where the problem is.

So with the Eisenhower example, what you said, it's such a great model for how do we think about our own outcomes in our own lives? So this is what tends to happen to us. And this is why I was saying, oh, it's actually not painful at all when you have a bad outcome to go in and look. It's what we naturally do. Something horrible happens to me. I'm looking in there. I'm trying to find all the luck in there. I'm exploring what were my other options? Was there something else that could have worked out?

better? How often was that decision going to work out poorly? And I'm so excited. I'm so excited to go in and discover that I didn't do anything wrong. I'm looking for that when it's a bad outcome. When it's a good outcome, when we land on the beaches of Normandy and it's a pivot point in the war and we win.

People aren't looking as hard to try to figure out, to really explore the counterfactuals. Well, what if I had done this? What if I had done this? What if it had turned out this way? So one of the first steps to being a really good decision maker, because we have this asymmetry in how willing we are to explore these outcomes, and it's what you just said. We're all living our lives having a morbidity conference. Now, the morbidity conference that we're having in our head is specifically to go find the luck in the whole thing.

But that's the only conference we're having. We're never having the my life is great conference. Let me go figure out if that's because of my own decision making or because of luck or if there was a better way or I could have actually opened up a whole other set of outcomes. Like nobody's doing that. That is such an interesting point. You are so correct in that. And as you're telling that story, I'm thinking of a sort of silly example in my life, but it's one that anyone who knows me knows I take very seriously, which is how much I love archery.

And I shoot a lot. So when I am not in New York, obviously, when I'm back in San Diego, I'm shooting constantly. And I'm always trying to push the limit of my capabilities, which means you're losing arrows. You're just invariably, if you're 60, 70 yards away, trying to hit something the size of a softball, you're going to miss every once in a while. I'm generally losing an arrow to the tune of one a day.

Now, these arrows are very expensive, so this is becoming quite a problem. Every one I've missed, I can tell you a hundred reasons why. And not bad luck. I can tell you the mistake I've made. I don't think I've once ever given thought to the perfect shots. I don't think I've ever spent time thinking about that, ever. I think...

90% of my energy goes into thinking about the 10% of the shots I didn't like. And I thought that's the right way to do it. Isn't that making me a better shooter? What I'm hearing you say is that's part of making you a better shooter. It's also going to encourage, I mean, if you think about sort of societally, if it's the bad outcomes that are triggering the dives, first of all, you're losing half of the opportunities to learn. If not more than half, because you might even be making more positive outcomes.

outcomes than less. Right, right. Well, let's assume that the chances of having an unexpectedly good outcome are symmetric. And let's make it that the thing that triggers us doing a dive is that it's unexpected and you can decide if it's like a one standard deviation away or the standard deviation and a half or whatever it is. We can decide and that would obviously depend on the uncertainty and how much ball there is and so on and so forth. But we'd figure that out and let's assume it's symmetric.

So by only digging into the downside, we're doing two things. One is we're losing half of our opportunities to learn. And the second thing is that we're reinforcing the risk aversion. We're saying, hey, what we really care about is downside outcomes, so you better avoid those. And three is we're

completely quashing anybody ever doing anything unexpected where because if you do something unexpected the chances that somebody's going to allow for luck or that you yourself are going to allow for luck is the explanation is just going to kind of go poof and we don't like that so there's all sorts of bad stuff that comes out of it so when we get an outcome we want to basically do what you were just doing with the beaches of Normandy

which is to say, here's the outcome. Before I make any conclusions about it, let me think about what are the other possible things that could have occurred.

So I have to think about what are the things that didn't happen but could have, those counterfactuals. And then I need to think about what was my preference for those things, how much did I like each of those things, and how often were those things going to happen. So it's just basically thinking about what was the payoff for those things. Obviously, you can compare those to cost, but let's just simplify it. What were the possibilities? What were my preferences for those possibilities?

And what was the probability of each of those possibilities occurring? And then, so that's the first step with the beaches of Normandy. It ended up being a particular way, but given that they decided to do what they did on that particular day, what are the other ways that it could have turned out with what probability would that have turned out and how much would we have liked those things depending on what our goal was, which in this case was to win the war. And then now we can take that and we can iterate it and say, what if they had waited two days?

What if they had done it two days before? What if they had done it in a different way? They had gone from land instead of sea. Now we can start thinking about if they had chosen other options.

then how would that have compared in terms of what the expected value was for the option that they actually chose? And you can do this for anything. So you can do this for the archery. I assume that a bullseye is pretty unexpected. At certain ranges, it's more surprising than it is not. Right. What you can say is you could actually go in and start forecasting it. I'm going in at a particular range given what I think that my skill level is. What percentage of the shots do I think I'm going to get a bullseye? So forecast it.

Say this is the percentage of times that I think I'm going to get a bullseye. Now, if it's a particular day and you're well below that, you would say, "Hmm, I wonder what happened today? What was the situation today where I was well below what my expectation was for the number of bullseyes?"

But now, if you're way above, you do the exact same thing. And this is what we don't do. I can guarantee you, because I know this is the way that people think. If you think, oh, I don't know how many times you shoot at the target. I've done so much of the work you're describing without the important piece. But at 50 yards, for example, I know, just based on what I do, that I should be able to put 8 out of 10 inside of a certain diameter. So...

I spend a lot of time thinking about when I'm hitting six out of 10 inside of that range, what's going on. But there's days I'm hitting 10 out of 10. This is what you do. You walk off and you dance. You're like, look at me. I don't think about why. You're like, whoa, check me out. I'm great.

And the thing is that, of course, you may have stepped up in skill, but wouldn't you like to know exactly what the adjustment, what was it that actually clicked for you that day? What is it that changed? And I know it's not that I stepped up in skill because it can vanish the next day. So it's clearly something that is transient. Well, one of the reasons why it may be transient

Because I haven't identified it and marked it and incorporated right because it could be totally transient It could be that you get to a certain skill level and this is gonna be your range. It's gonna be six to ten Oh, well by transient I just mean what I'm really saying is I'm sure that I did something better on that day But it's transient because I didn't identify it and make it a part of my permanent It could be that if we could look at me under the six out of ten day versus the ten out of ten day

I mean, I'm positive I'm doing something different. There is no way equipment, wind, or any other factor other than me explains that. But by only comparing eight to six, not six to 10, I'm missing a greater disparity, which in theory should call out the more obvious deltas. It's not just that you're missing half the time, it's you're using half the discrimination. It's like, it's easier to see that there are two lines if they're a foot apart versus a millimeter apart. So if I'm

If I had to say, like if someone asked me, what's the single biggest factor between a player who doesn't make a ton of progress or make slower progress compared to a player who really, really makes a lot of progress and becomes elite in poker? I would say that's it. That when you listen to someone like Eric Seidel talk about poker, you would hardly know whether they won or lost the hand.

all they're doing is exploring what are the counterfactuals? Like, what are the other lines that I could have chosen? Did I have the person's hand right? What if I had bet a little bit more? What if I had bet a little bit less? What if I hadn't played the hand at all?

What would have happened if I played this hand that I didn't play? And you would hardly know. Like, if you listened to them, you would have no idea. In fact, probably you would assume they had lost. Right, because it's so foreign to hear somebody be like that. Because they're exploring so much. You'd be like, whoa, you must have really lost that tournament. And they'd be like, no, I won the tournament. And you move on. And

And what you hear from players who aren't making that kind of progress is it goes two ways. When they talk about the hands they won, it's just sort of like I was great and they don't really spend a lot of time on it. When they talk about the hands they lost, it's a lot of exploration of the luck elements. And the thing about poker, because the luck element in a given instance, like in the short run, is so prominent that it's so easy to focus on the bad card that came.

So you'll have conversations with people where they'll talk about like, oh, I had this hand and this.

This person, it's called sucking out. They sucked out on me, meaning that they happened to hit a good card. And to somebody who's thinking, who spent a lot of time around someone like Eric Seidel, you're looking at it and saying, well, you should have won that hand way beforehand, or maybe you shouldn't have played that hand at all. So for example, like if I have a particular hand where you end up hitting a really good card, it may be that I should have made you fold before then.

Now, sometimes it's, you were gonna be sticky, you were gonna stay around, I had mathematically way the best of it, and whatever, bad luck happened. But sometimes it's, I should have made you fold a lot earlier, I should have made you pay more for the privilege.

of hitting that card that actually the way that I bet the hand meant that you were making money to my bet when I could have made you lose money to my bet regardless of whether you were going to beat me on that particular hand in the long run. I'd like to make you make losing decisions. Maybe I shouldn't have played the hand at all.

There's all sorts of other things that you can be exploring in those situations and they're not getting explored. Instead, what's getting focused on is I can't believe that person sucked out on me. So I have this framework for learning things that require skill and I see a parallel now that I've never seen before. So my framework, and I use swimming as the analogy for this because I learned to swim as an adult. So this was really the first time I explored this framework, but I do think it applies to many things, languages, you pick it, frankly. So the framework is we start out as adults

unconsciously incompetent. So you take a person like me at 31 who doesn't know how to swim, you put them in the water. There's no ambiguity about how bad I am, but I don't know why. All I can tell you is I can't swim. I can't actually tell you why I can't swim. So the first step of learning how to swim is becoming consciously incompetent. You actually have to understand why you're really bad at this. Oh, once you start to understand why

Archimedes principle. That's like a very important part of understanding how to swim. The more of you that's underwater, the more buoyancy there is to hold you up. Once you start to understand how Bernoulli works, oh, all of a sudden there's this difference about the velocity of a fluid over a body, et cetera. And then by the way, you're no better as a swimmer in those two boxes, right? The difference between- I just want to know the percentage of people who are learning swimming who cite Archimedes and Bernoulli. Yeah.

I just love that. That's the way that you're I don't know if I still have them But I would love it if I did I used to journal so relentlessly about swimming I had reams of journals that I would write in every single day of My free body diagrams of my body in different swimming positions and trying to figure out ways to make this less bad When I got into this I became so obsessed that I swam twice a day every day

This model makes so much sense to me. Well, you don't actually start swimming until you reach the third box, which is now you are consciously competent.

Now, the problem is you can't do that for long. So you tend to vacillate a lot between the second and third box, which is consciously competent, and then you go back to being consciously incompetent. But as long as you can maintain focus, you will vacillate between those, and usually it's your physical fatigue that will push you from the third box back to the second box. Now...

When you become a really good swimmer like Michael Phelps you transcend to this fourth box. You are now Unconsciously competent everything becomes sort of autonomic at that level now thinking about this through decisions as you were telling a story about Eric I'm thinking about the first level. So let's just call it four levels now The first level is you only examine losses through the lens of luck and

So that's our default. I would argue that takes no effort, just as being unconsciously incompetent is our default. Layer two or level two thinking is thinking

Only examine my losses through the lens of luck and skill. That's great. So you're taking some responsibility taking some responsibility Only trigger by the loss then you have a level three, which is I'll do that But I will also now examine my wins through skill and then the fourth level which I'm guessing is exactly where someone like Eric is is no I'm going to look at wins and losses through luck and

Yeah, so what I would say is I think that the thing about examining wins, I do think that examining wins through the lens of skill is pretty natural.

We don't spend a lot of time on it. It's kind of an assumption of skill. I would put wins down in level one. Let's put it this way. I would focus more on, I would actually think about it not just in terms of, are you examining the quadrants that are discordant?

In a symmetric way. So what I would say is that on the level one thinking is I lean on the discordance on the loss side and I lean on the concordance on the win side. And I just kind of do that naturally. And I don't spend a lot of time thinking about either of them. That makes sense. I don't spend a lot of time thinking about either of them.

Then I think that level two is you're still leaning on concordance on the win side. You're not examining those very much, but you do start to take the losses. At least you examine the discordant losses. Right. You examine the concordant losses now. So you examine the losses that are actually due to your own undoing. Then I'll get to the fourth level. Then I think that level three is...

I'm looking in all four quadrants. So I'm looking at wins that are due to skill, that are due to my good decision-making. I'm looking at wins that are due to my bad decision-making. I'm looking at losses that are due to my bad decision-making. I'm looking at losses that are due to my good decision-making. And I'm really trying to sort of look at those all equally. Where I think the fourth level comes in is when it's concordant on the win side,

that you don't accept it. In other words, that you say, I made good decisions and I won, but it does not mean there wasn't a better way. So let me actually start to really clean up in there and say, it's not about I made a decision, it was a winning decision, I was positive expected value for it,

And so I'm done. It's now I so care that I got onto what we would call the primary line of play, that I really came up with the best possible solution that I could, given what I knew, that I won't be satisfied with just finding out that there was concordance. Like, I want to go beyond that on the win side. I think what we're going to do in the show notes is literally take everything you just said and turn it into a printable one pager that every one of us should carry with us. Because that's that is a very...

elegant way to think about that. I promise you, I've never done what you just said. Like I've never once had a level four thought in my life. Well, first of all, let me just say that I only thought about it because you gave me the levels, but it's that last bit. So right now I'm writing a workbook very close to finished with it, where in chapter three, I really dig into that piece, which

which is when you have a good outcome and it turns out it's from good decisions, what are you doing with that? Are you done? Or are you trying to figure out if there was even better one beyond that? And here's the reason why we don't like to do that because

When we think about what sort of creates the fabric of our identity, our beliefs are the threads that our identity is woven out of. By doing a level four thought, you risk they're undermining that. Tearing a hole. Tearing a hole in it. Because one of the things that we want to do is feel like that fabric is strong. Like our beliefs are good and we're competent. I mean, this idea of competence and being able to bank credit.

for the way that things have turned out in our life is so core to what forms our identity that when you don't stop at, "My decisions were good and I had a good outcome," when you actually dig around in there, you're risking now turning something that was a win

into a loss. So this is where I love your idea of the levels. So if we think about that level one, like I won, it must be good. Why wouldn't you go further than that and say, well, maybe I won because of bad luck. You're turning a win into a loss. And in this particular case, we're taking win as the outcome. I had a good outcome. And you sort of want to just stop there because you don't want to dig down and find out you might have made a bad decision because now you turned a win into a loss.

So that's where when you get to level two, you're willing to do that. You're willing to say, I want a long run win. And what that means is I might have to turn this outcome that was a win into a loss. When you get to level four, what you say is,

okay, I had a good outcome. I looked and I saw that my decision-making was pretty good, but now I'm willing to turn that win into a loss in order to get long run better. Yeah, you were basically completely exposed at that point. I can imagine how your book became immediate fodder for every hedge fund manager and so forth. Because again, I think the advantage of that type of investing bears some resemblance to poker, which is

You can't really hide your decisions. Your outcomes are very clear. And which is not to say there's not a huge element of luck, just as there is with poker, but it's basically an industry where your decisions are exposed. Every decision you make in the end, you're going to be able to trace it to an outcome and, and,

Frankly, unless you're not in the business of making money, you have to be wed to good outcomes. So the poker player wants to do well. The hedge fund manager wants to do well. There's nowhere to hide, I guess, is what I'm trying to get at. I agree in theory that there's nowhere to hide. The interesting thing that I think is how long people can hide from it. That's what I find so fascinating.

is you look at poker and you're like you're getting answers right away right like you win or lose so here's the problem it's like okay at the end of the year somebody lost right but why how far are you willing to go to say I just got bad luck that's the question and I'm just really fascinated that people can go really far they can really convince themselves that they just got bad luck

And I think that that's true. Obviously, if you're running a hedge fund, you have investors that you're accountable to.

But we all know of hedge fund managers where they had a couple of good years and then people are sticking with them. It's like you have a couple of good years to start. There's this path dependence to this. And it's true of poker. Like if a poker player comes out of the gate in the first six months and loses everything in sight, they're very unlikely if they continue to lose, they're going to say, oh, I must be really bad at this. But for every poker player who comes out of the gate with a terrible first six months, there's a poker player

I actually remember way back when, a long time ago, there was a new player who came on the scene and they won a World Series of Poker bracelet. I mean, they were really young, I mean, early 20s.

And nobody had ever heard of them before. And they won. And I thought, oh, you know, that's nice. Because that happens, right? Obviously. Then they won a second one. And I was like, oh, that might be just the ruin of them. Now, why would I say that? Like, they had won a second World Series of Poker Bracelet because I understood this path dependence, right? That...

If you come out of the gate and you do really well, your ability to fool yourself afterwards for a very long time... It becomes infinite. It becomes almost infinite. And you see that in the hedge fund world all the time. Obviously, if an investor comes along and they don't show any results, by the way, sometimes unfairly, it becomes very hard for them to recover from that. Sometimes unfairly, because maybe it just went against them for that year or whatever.

But when it's the reverse, when they come out of the gate strong, you can fool yourself for a really, really, really, really long time. And that's the thing that I find so fascinating. That's what really kind of lights me up in trying to figure out the capacity for self-deception is so a bottomless well. Is there an industry that you look at where you just see on average it is more amenable to higher level people?

Thinking the way we just described the four levels in poker, like this framework, is there a place where either through some externality of the way the industry is set up and the rate of time being short between decision and outcome and some sort of transparency that we have a space where.

there's a greater incentive to rise to that level? First of all, let me just say this, that obviously we're only talking about places where in the shorter run, luck is going to have a strong influence. So when you look at sports that are really highly under the influence of skill, people are getting selected out. Nobody's fooling themselves. Well, actually, it was just bad luck that I didn't get to be a professional NFL linebacker

I mean, it might be if you got an injury, but I'm 5'5 and I weigh 130. Such bad luck that I wasn't. People are just like, no, I just wasn't good enough. So we know that in cases where it's very transparent that if you're not doing well, we can trace it back to skill. So the more chess-like we are, the less the capacity for us to fool ourselves. If I lose to you in chess, it's very hard for me to say that I got bad luck because I don't know, where's the luck? I don't know, go find it for me. There's a little bit of luck, but not much.

So the more on the chess range of things we are, the less the capacity to do that. So first of all, we've got to be in an environment where there's a strong short run influence of luck. Now, the tighter the feedback loop, the more feedback you're getting, the more that somebody needs to rise to the occasion. So if you were to look at what's the capacity, which individuals are going to thrive more in those systems,

definitely the ones who are much more willing to dig in and dig around. So people who are in higher frequency trading, for example, where those feedback loops or the time horizons are shorter, I think are going to be more likely to be thinking in this way. And if they're investing their own money.

Ah, that's a very interesting point. It's got to have Nassim, right? You got to have skin in the game. Are there poker players, by the way, that play with other people's money? And by that, I mean not money that they won from somebody else, but yes, I see. Okay. Yeah. So there's backing situations. Yeah. There's the same thing. The less you're trading on sort of reputation and the more that you're trading on your own, your own dollars. Also, I think that you're more likely to end up in a spot where you're thinking that way. So basically you have tons of skin in the game. There's a high element of luck and skill.

And the feedback loop is short. Those are three situations that will tend to push you towards either abject failure or getting better, moving up the ladder of this thinking. It's a little chicken and egg. So I don't know. Is it the people who succeed are more naturally kind of more naturally tend to have that kind of thinking and obviously it gets reinforced or

Or what would eric do if he wasn't the best poker player in the world? Well, I know what he would do because he was an options trader Okay, so again options trading you could argue that it's sort of similar, right? It's incredibly similar. It's almost the same But would he go and be a good ceo? Of a blue chip company like if he all of a sudden tomorrow assume he had the domain knowledge Okay, so he could go in and be the ceo of general electric

or Home Depot, or Exxon Mobil, or something where you could put a chip in his head that gives him the domain expertise to understand exploration and production or whatever. But could you move that industry or that company within its industry relative to its peers by bringing that layer of thinking? Yes. So that's kind of what I want to say. Is there a little bit of a chicken and an egg issue? Because I have met people across industries where there are people who are thinking this way.

where there are people who are really hungry for it. Now, in terms of this idea of like, what are you doing when you find out it was pretty good decisions and good outcomes? Or are you only doing postmortems and not thinking about it in terms of expectancy? That's a problem that you see across all industries. But also across industries, you see people who are really hungry to try to incorporate that into their solution space. How am I dealing with this? How am I running my teams? So if you just take the investing world, I...

I see people who think that way all the way from people who are trading options to people who are doing seed round venture capital. Much different than late stage venture capital where you might, the time horizon now is faster. I mean, not the same speed as options. But if you're like in the seed round, we're talking horizons, it could be a decade.

There are fund managers who are in long-short funds where they might put on seven positions in a year. You're hardly getting any data. And they're really looking to think this way because what's interesting is that while...

trading options might foster this kind of thinking better because you're getting feedback more quickly while you might end up getting sort of selected out by the people who you're accountable to quicker while maybe that particular type of mind would tend to get drawn towards, say, options trading better.

The issue is that in absence of the world telling you quickly, this type of thinking actually becomes much more important. Yeah, it becomes more important and more difficult. Right, because when you spread the feedback loops out, now the way that I always think about it is uncertainty gives you the leeway to allow bias to

to come in. And that's the difference between the chess and poker. In chess, because you've taken a lot of uncertainty away, you don't have the leeway to fool yourself. That is an absolute gold coin. But in poker, you do. So we can sort of think about if you look at the difference between, say, trading options and something that has a much longer time horizon, the longer the time horizon, the more capacity you have, the more leeway the world is giving you to fool yourself.

So then this kind of thinking, really thinking in this systematic way, really modeling the world in this way becomes much, much, much more important. It becomes a bigger imperative for you to be poking, for you to be thinking counterfactually, for you to have people challenging your ideas, for you to be

trying to think about, am I doing this because it's status quo? And if it doesn't go well, I won't get blamed. Am I innovating enough? Am I allowing people to challenge my ideas? Am I challenging ideas myself enough? Am I looking at that discordant box that has to do with good outcomes?

Or am I only willing to look at the discord when it's a bad outcome? When the bad outcome is concordant, am I like super excited about that? Or am I imagining, well, maybe there was a better way?

How am I digging into all of those spots? And then if you're in a leadership role, how am I propagating or am I not propagating these problems across the team? Am I telling the team through my actions that they should be terrified of bad results? As I'm thinking about the way that the world is giving me the leeway to be able to sort of have my beliefs lead me around on a leash.

Am I infecting my team with that same problem? Because now I can be creating this whole other problem. And it's such a long time horizon that maybe the world's going to tell me that this was actually problematic.

You should be digging into that as if you're getting the answer tomorrow, as if you're playing chess and you're going to find out right away. And you should be doing everything you can to try to make sure that you don't find out you hung your rook. This to me is why swimming is harder than riding a bike. You can teach anybody to ride a bike relatively quickly for a couple of reasons. One, the feedback loop is very quick. Like anytime you're out of balance, I mean, we just naturally internally, our inner ears allow us to sense things.

that disequilibrium instantaneously. So you get the fastest feedback loop imaginable and the consequences are quite high. It hurts significantly when you're not in balance. Contrary to popular belief, swimming is about balance. It's the exact same thing. You just happen to be in a medium that is so much more dense than air that the feedback loop is much slower and the consequences are much more gentle. It doesn't hurt.

when you're out of balance. In fact, if you fell every time on a bicycle that you were out of balance while learning to swim, you'd never ride a bike again. You'd be so black and blue. So in that sense, swimming is gentler, meaning you have more excuses. It's longer time horizon.

but it's that much more important that you do the examination. So if anybody can mindlessly ride a bike very soon after being shown how to do it, you can't mindlessly swim for years after learning how to swim. It's that hard. Swimming then becomes the example of where it's more important that you bring that level of thinking if you want to make any progress. I agree. And let me add something to this, which correct me if I'm wrong, but I think this is in the same space as you're thinking.

Sort of thinking about this soft landing issue, the wider the skill gap between you and the people that you're competing against, there's so much more cushion for a soft landing. So this is something that I sort of think about is by not...

exploring that, sure, I made winning decisions and I won, by not willing to go beyond that. Imagine if you're in a situation where, let's say we're in some sort of zero-sum game, so we're trading against each other. And if I were making perfect decisions against you, every time that we put a dollar on the table, I would get a quarter.

But because of the decisions that I'm making, every time we put a dollar on the table, I'm getting a nickel. So if we think about those quadrants, right, when I look at I have a good outcome, I won a nickel. And my decision making was good because I was positive expectancy, right? I was winning a nickel. I was winning 5%.

But by not exploring, I don't see the 20 cents that's sitting there that I could get every single time. And that's the problem with a soft landing. That's the problem with having that cushion is it doesn't let you know

That there's something better that you could be doing. I know we got to wrap because we both have to get moving. But there's one topic that you very loosely, I think we were going to go down the path of it by name, but then we didn't. But I want to bring it up because it's such an important concept in my world. And it's something I have done myself.

for such a long time, but never had a great name for it. And when I read your book, I was like, oh, awesome. I'm going to be plagiarizing this all day long. And it's the idea of backcasting. Oh, yeah. A moment ago, you used the term forecasting. Right. We're all about forecasting. Well, I want to be doing this in Q4, and therefore I need to do this today and this tomorrow and this tomorrow. And I'll tell you how I use it, and then I'll... And let me just say for the record, backcasting is not a word that I coined. Oh, all right. I don't

I don't want to reference you who references someone else. You can say you read about it in my book. I didn't invent that word. All right. Thank you for clarifying that. So prior to reading your book, I would just refer to it as reverse engineering, but that never really had any resonance with people. But now when I say backcasting, it forces me to explain the contrast between backcasting and forecasting, whereas reverse engineering doesn't have a striking of a contrast. So the place that I use it

is in understanding the exoskeleton of the body. So the actual physical structural demise of ourselves. So we're all sitting here thinking about exercise and everybody knows they should exercise, but nobody really can give you a great reason for why. We've been sitting here talking about the outliers, professional athletes, but outside of people who get paid to do something with their body, what are the rest of us doing?

It's amazing how often you push people on this. They can't give you a great answer. And because I ask everybody under the sun this question, I'll tell you, all answers basically fall into these three buckets. So if you're asking someone who's not getting paid to do a sport, why do you exercise? 90% of the answers are, I like the way it makes me look.

I like the way it makes me feel. And I like the freedom and flexibility it gives me with what I eat. Some combination of those three. There are other answers that show up a lot less frequently, such as I love the social interaction that comes with it and things like that. Now, I would argue all of those are fine reasons to exercise, but I don't think they're good enough. The good enough one only comes when you start thinking about backcasting, which is what do you want your body to function like the day before you take your last breath? Really think about this.

So I've taken this to an extreme and people have heard me talk about this on the podcast before but I come up with this idea of the centenarian olympics, which is Instead of training for like an olympics that's now where you would actually have to pull vault and run 100 meters in under 10 seconds Think about what an olympic games would look like of hundred year olds and ask yourself What do you want to be able to do in that olympic game? So I have like 18 things I want to be able to do when i'm 100 if I get to be 100

And they're all quite mundane, but they're all very pragmatic and practical. One of them is being able to do a 30-pound goblet squat, which, again, I could do today blindfolded on one leg 100,000 times. But at 100, that's actually going to be quite a task. And it's important because that replicates picking up a toddler, which presumably if I'm 100, I've got great grandkids. I want to be able to pick them up.

I want to be able to stand up off the floor using no support and only a single point of my own, meaning one arm and both my legs and get up off the floor without help. Can I do that today? Of course. Can you do that today with your eyes shut? Very few hundred year olds can do that. So I've kind of 18 of these things I want to be able to do. And that is backcasting. It's going to the finish line and saying, if I can do that at a hundred, what was I able to do at 90?

If I can do that at 90, but it must have been true at 60 and before you know it, I realized, wow, if I want to do that at a hundred, now that I'm 46, I actually have to be able to do all of these other things. So there's this whole matrix that comes out of it. That's why that concept for me just resonated so much because I really

I don't think medicine spends enough time backcasting. Yeah, so let me try to add another benefit of backcasting. So just to be clear, backcasting is, so when we forecast what we think is, there's a goal that I would like to reach, how do I get there? And then generally what you're thinking are what are the things that I'm doing in order to get to this goal? When you backcast, what you say is, it's the day afterwards. So I have some goal, I'd like to be able to do X in a year. And now it's a year and a day and I achieve my goal.

And what you ask now is you look back and you say, how did I actually achieve this? And you can also do something called a premortem. So a premortem would be, I have this goal that I'd like to achieve. I'd like to achieve it in a year. It's now a year and a day and I failed. Why did I fail? Why did I fail? Now, here's an added benefit that I would like to offer to you. When we forecast, when we think about, here's my goal, how do I get there? We're thinking about the things we're doing.

Here are the decisions that I make along the way. So we're thinking about all the things that are within our decision-making power, within our control, that are in the sort of skill side of the equation. But what happens when you back cast and you say, I actually succeeded, how did I get here? But particularly what happens when you do a premortem and you say, I failed, how did I get here? Now you allow luck into the equation. So as an example, if I'm thinking about when I'm 100,

I would like to be able to do a 30 pound goblet squat, which I've never heard before, but now I'll say that because I, whatever. Anyway, I learned that, but I'm now a hundred. It's the day I've turned a hundred and I cannot do that thing. Why? Well, I can think of some things like I didn't actually exercise enough, but I can also think of, I tripped and I fell and I broke my hip. Now tripped and fell that can happen to anybody. Um,

Probably not a matter of skill. Just something can happen. Like you could hurt yourself. But the break my hip might have to do with my bone density. That's going to cause you to explore other things that maybe wouldn't be as obvious. Because you saw nobody's thinking. You don't see that bad luck can occur if you're not working backwards. And then to your point, once you work backwards, you can now ask yourself some questions because you see the luck. You can say, okay, if it's on the good side,

Can I increase the chances that that thing happens to me? That I see is luck is going to intervene in my way. Can I increase the chances that's going to happen for me? Okay, so you could ask that. But then on the downside, which is really important to explore, let's say the breaking the hip thing, can I decrease the chances that I break my hip? Yes. What are the things that I could do? If the answer is no, then you can say, are there any hedges available to me? Can I hedge against this bad luck thing happening to me? Sometimes the answer to that is also no.

Regardless, what you can say is, okay, I've got my hedges set up. I'm doing what I can to decrease the chances that it occurs, but obviously it still could. So let me think now, before that bad luck intervenes, what am I going to do in response? Because if I plan for that in advance, I'm going to be making decisions

right now in a calmer state of mind where I'm not pants on fire, freaking out, rather than being reactive to what the world might deliver me at the time. So if I've already thought about it, if I already have a plan in place,

I'm not wasting time. I'm going to be much more calm. I'm going to think about it. So this is one of the big difference between working forwards and working backwards is that when we work forwards and we're thinking about how do we execute on any kind of strategic plan, because we can think about, I want to be able to do this thing when I'm 100 and we're developing a strategic plan for how we get there.

We really focus on what are the things that we're going to do that are going to get us there. And we don't say, what are the ways that luck might intervene? But when you work backwards, you naturally start to see the luck. And then you can start to explore all of these questions that then makes your strategic plan much better. It makes you explore different stuff. And does your workbook cover exactly this type of decision tree that we just went through? Yes. When is this coming out? A year from now.

So I'm just finishing up the draft now and then you have to lay it out and you know how that goes with books. So basically what I'm doing with the workbook is I'm taking these concepts that I talk about broadly and sort of bigger idea way and thinking in bets and I'm saying, okay,

How do you practically make a decision? How do you build out a decision tree? How do you think about the way that luck might intervene? How do you think about your own decision making? How do you see the luck? How do you communicate with other people such that you're making sure that you're getting the purest feedback?

from them, that you're not biasing their feedback, that you're actually hungry for the information and you're seeking the feedback, which is by the way, the first step, which most of us don't do. Most of us aren't seeking the feedback. So where you're hungry for the feedback, you're actually thinking about what does the future look like? What could the outcomes be? How am I going to react to those things?

How do I understand the sum of my experience and what that's supposed to teach me? So it was really trying to think about how do I allow people who don't do this for a living, who don't spend their time building out probabilistic decision trees, and this is not the way that they look at the world, how

How do I take somebody like that and show them the value and teach them how to actually think within this framework? Well, I think that means we're going to have to sit down and talk about a year. I hope so. I want to also just say one thing just for the listeners, because I would like you to put it in the show notes. So I'm going to say it right here. A lot of this stuff that we've talked about with luck and skill, I really want to recommend. Well, first of all, in the forecasting stuff, let me just, again, recommend super forecasting, which I know is going to end up in the show notes, but

There's a really, really wonderful book by Michael Malbison that's called The Success Equation. And it's all about the influence of luck and skill in our lives. It's got a lot of sports stuff in it, but just a lot of really digging down and drilling down into understanding luck and skill.

and how you sort of understand what the difference is between the range of activities and sort of where you can see more influence of skill, where you can see more influence of luck, how you can sort of parse those apart. This huge influence on our lives is luck. And he really drills down into that in a way that

if people are really interested in the conversation that we had today, then I guarantee they would be really interested in the success equation. And Malbison also, I mean, you can just search him and you can see a whole bunch of videos of his where he talks about luck and skill. A lot of his writing talks about luck and skill and sort of separating the two and understanding the different influences and

Yeah, I just can't recommend that book more. We'll put together a pretty robust set of notes here as we always do. And it will point to everything we've spoken about. Plus, I'm sure after the fact over the next couple of weeks, you and I'll have back and you'll think of something else that you want to hear. So I'd love to make this the sort of the compendium for all these things. And then, of course, I can't wait to sort of talk about this workbook. It sounds like it's

kind of an amazing tool. I mean, again, I would say like in the last few hours, I've realized that I'm not nearly as good a decision maker as even I think I am. And I don't say that to be sort of arrogant. I just think, look, I understand

understand decisions at least as well as anybody else has been my bias. And I'm at least as probabilistically savvy, but I realized like there's tons of blind spots that I'm still experiencing and I'm not even close to approaching the potential that, that a person would have to make better decisions. So that said, I find myself somewhat overwhelmed by how complicated this is. I'm already thinking like Travis is going to put together some great

that will hold us over until your book comes out. But having a book that could be sort of a guidepost to a decision matrix would be fantastic. And let me just, for people who have listened to this and felt sort of overwhelmed, it's funny because this happens to be the chapter that I'm writing right now, is that it's really, really good to have this fragility

framework to be thinking about decisions and to really recognize the probabilistic nature of the world and the way that things turn out for you and understanding what the structure of a really good decision is. That being said, that doesn't mean that you have to sit down with a slide rule for every decision that you make. And in fact, for the majority of decisions that you make in your life, you can actually decide quite fast.

And what's interesting is that the way to really understand when can I decide fast is to understand this broader framework. This broader framework that would allow you to take the bigger decisions in your life and actually start to think about them in this way where you're really forecasting them is actually the way to understand when can I go really fast.

And what it will allow you to do is stop taking 15 minutes to decide what to order in a restaurant because you have to understand this framework in order to sort of get when you can go fast and when you can go slow. Because when I do talk to people about this kind of framework, they do feel a little bit overwhelmed and they're like,

"Oh my gosh, how am I ever going to make a decision again? Like, I'm going to need a supercomputer in order to do it." It's very few decisions that you actually need to take a ton of time on. Mostly you can decide fast. And on the ones that are bigger decisions, embracing the probabilistic nature actually gets you out of the analysis paralysis. Because what you realize is once you've identified your options and you've got an option that you see is better relative to the other ones, you stop focusing on achieving absolute certainty

and start focusing on achieving relative to the other things that I could do, this one definitely looks better to me. And because I recognize that there's all sorts of stuff I don't know, and there's all sorts of different ways the world can turn out, and as long as I'm taking the things that occur in my life and trying to learn from them in a real way, I'm not going to be so terrified of sort of the downside. And that really frees you up. You've said that so elegantly, and I think that reiterates this sort of

The notion that intuitively people get it right. You go fast on the straightaway and slow on the curves. This framework shows you where the curves are. Yeah. And that frees you up to go faster in the straightaways. Yes. And interestingly enough, it frees you up relative to what you were doing before to go faster on the curves as well.

Absolutely. To not try to figure out in that moment, if I don't know the exact right path to take, I'm paralyzed and I just need to slam the brakes on. But to recognize when, well, this path is going to be good enough. I'll spare all the listeners the race car analogy that is like zooming into my cortex at the moment because there is a perfect analogy to driving around a corner in a race car, but I will spare everybody that. You can put it in the show notes.

There we go. Well, Annie, thanks so much. This was really great. I really appreciate you responding to my random request to sit down and talk about this. And I can't wait to do it again when the new book comes out. Well, thank you. I'm excited to come back when the workbook is done. But this was like an amazing conversation, very different than most of the podcasts I've done. So I really appreciate it. That's the way we do things here. We go deep. Yeah. Thank you for listening to this week's episode of The Drive. It's extremely important to me to provide all of this content without relying on paid ads.

to do this our work is made entirely possible by our members and in return we offer exclusive member only content and benefits above and beyond what is available for free so if you want to take your knowledge of this space to the next level it's our goal to ensure members get back much more than the price of the subscription

Premium membership includes several benefits. First, comprehensive podcast show notes that detail every topic, paper, person, and thing that we discuss in each episode. And the word on the street is nobody's show notes rival ours.

Second, monthly Ask Me Anything or AMA episodes. These episodes are comprised of detailed responses to subscriber questions, typically focused on a single topic and are designed to offer a great deal of clarity and detail on topics of special interest to our members. You'll also get access to the show notes for these episodes, of course.

Third, delivery of our premium newsletter, which is put together by our dedicated team of research analysts. This newsletter covers a wide range of topics related to longevity and provides much more detail than our free weekly newsletter. Fourth, access to our private podcast feed that provides you with access to every episode, including AMA's sans the spiel you're listening to now and in your regular podcast feed.

Fifth, the Qualies, an additional member-only podcast we put together that serves as a highlight reel featuring the best excerpts from previous episodes of The Drive. This is a great way to catch up on previous episodes without having to go back and listen to each one of them. And finally, other benefits that are added along the way. If you want to learn more and access these member-only benefits, you can head over to peteratiamd.com forward slash subscribe.

You can also find me on YouTube, Instagram, and Twitter, all with the handle PeterAttiaMD. You can also leave us a review on Apple Podcasts or whatever podcast player you use. This podcast is for general informational purposes only and does not constitute the practice of medicine, nursing, or other professional healthcare services, including the giving of medical advice. No doctor-patient relationship is formed.

The use of this information and the materials linked to this podcast is at the user's own risk. The content on this podcast is not intended to be a substitute for professional medical advice, diagnosis, or treatment. Users should not disregard or delay in obtaining medical advice from any medical condition they have, and they should seek the assistance of their healthcare professionals for any such conditions.

Finally, I take all conflicts of interest very seriously. For all of my disclosures and the companies I invest in or advise, please visit peteratiamd.com forward slash about where I keep an up-to-date and active list of all disclosures.