Shownotes Transcript
I started getting a few goosebumps and then I said, well, ask ChatGPT why that's the answer. It explained it. Oh, yeah. It's so good at that.
I've had great teachers that I've learned from. I had a librarian at my elementary school. I have a great tennis coach. And, you know, the best way to get unconfused about something is to find somebody who really deeply understands it. And I call that getting unconfused. Welcome to Unconfuse Me. I'm Bill Gates.
The role of technology in education isn't always obvious, but the potential and the possibilities are inspiring. When I would talk to people about loving to learn on the internet, and it was just at the point where, you know, putting up videos was kind of getting mainstream in about 2010, and people said, wow, you've got to use Khan Academy. My guest today is a teacher. He's CEO of Khan Academy, Saul Khan.
Khan Academy is an amazing tool. I was so impressed that Saul was able to teach such a broad range of subjects himself in the early days. Khan Academy's got 140 million users. A lot of students have benefited. So I'm excited to have him here today. Welcome, Saul. Thanks for having me.
Well, to start, I found out when you were younger, you were a singer in a heavy metal band and a cartoonist for the school paper. So which is the better skill, drawing or singing? Depends what you're trying to do, but I'll say drawing.
Yeah. Well, in your lectures, you do have to draw stuff. And it's nice. You erase a bunch of times if you don't like what you did. No, so every now and then, if you watch enough of the videos, I try to flex my art a little bit. You know, Sal can draw a cartoon. Sal can do a little bit of shading and cross-stitch and all that. No, I definitely, you know, when I was a kid, I would just draw for hours. And, you know, now I realize it does give you a certain, like, one, you never get bored.
It does, I think, build a certain curiosity and creativity just drawing all day. Well, Saul, remind us how you got into all of this. I think you were at a hedge fund at one point. Yeah, you go back to 2004. My original background was in tech. When I go to business school, I go into finance. I was an analyst at a hedge fund in Boston. I had just gotten married, family visiting me from New Orleans, which is where I was born and raised. It just came out of conversation. My 12-year-old cousin, Nadia, was having trouble with math.
I said, "Hey, Nadia, when you go back to New Orleans, I'm happy to tutor you." She agreed, started tutoring her after work every day. Long story short, that made a difference. It went from her being a weak math student or perceiving herself to be a weak math student to being a very strong one, started tutoring her brothers.
Words present my family, free tutoring's going on. Before I know it, I'm tutoring 10, 15 cousins. By 2009, there was a couple of, almost 100,000 folks who were using it back then. That's when I took the plunge, set it up as a not-for-profit, a mission-free world-class education for anyone, anywhere. And then it spread. Oh, yeah. A lot of the vision of Khan Academy in the early days, yes, there was an accessibility piece of it. Yeah, we can make videos. We can share with everyone. But the pedagogical aspect wasn't just access. It was that people
people could finally leverage technology to learn at their own pace. If you only got a 70 or 80 percent on some concept, the class doesn't need to move on to the next one and build on that gap. The students should be allowed to work on that. If you just let people have a strong foundation, the next few layers in math come pretty easy. With a weak foundation, no matter how bright or hardworking you are, they're very, very difficult. That was the thesis, I think over the last 10 years on this journey.
We've got a ton of efficacy studies in different contexts, different countries, and they all kind of say the same story that if students engage, let's call it 30 to 60 minutes a week, even that what I would call pretty low level of engagement, they're growing pretty dramatically, 20, 30% more than expected or in some cases more. Amazing. And so I think the question over the next couple of years is how do we get more students to engage at that level? Yeah, I think that challenge, a lot of our conversations have been on
that your tools are a miracle for the 10 to 15% most motivated, but then how do we draw in that other 85%? And I do think in the last year or two as you've engaged with teachers and districts, we're starting to see that it can be a tool basically for all students.
That's the goal. But to your point, I actually think 80%, 90% of hopefully 100% of students, if they're able to engage in the right way, especially if you catch them early, they can have that foundation. I think the main issue is just a lot of kids, frankly, just check out by the time they're in middle school. So in learning, you've got
The classroom environment can use that time in a certain way, and you've got whatever time you get with a student where they're kind of by themselves and assume they have a device. How do you see the time in the classroom? Is the availability of your tools shifting what people do in the classroom, and do they use it real time in the classroom? We were all starting 10 years ago, 12 years ago. It was only more fluent school districts that would have one-to-one laptops. The rest of them had to share. The good thing is over the last 10 years because of E-rate, a lot of that's been addressed. It
It's unusual now to see a school where you don't have one-to-one laptops in, say, a math class. And so, yeah, all the students will take out their laptops and do 20 minutes while the teacher walks around. They'll take 10 kids aside, do a focus intervention with them, do a worked example with them. The other 20 keep working. Then they'll take another 10 aside. But when they do that consistently, it makes a huge difference. And then for the students who either need gap filling or the students who are ready to race ahead,
they can do so. That's what they really like about Khan Academy. Not only do you get that practice and feedback in the moment, but the kids who want to race ahead can and the kids who need to do some gap-filling can too. And the dashboard that helps the teacher see the status of all the students, that's a Khan Academy thing that the teacher account gives them a really clear indication of what's going on. I mean, we all remember growing up in a classroom that the only time of real measurement is at the end of the unit. And then you get the test and
The test is graded. You get a 90%. I get an 80%. All right, let's move on. No one even attempts to improve that. And so what we realize is that now with tools like Khan Academy, you can give real-time information. Well, you can give real-time feedback to the student, but you can also give real-time information to the teacher so that they can intervene and they can do things. I hope in the future that that practice is the assessment, that you don't even have to take separate assessments. But yeah, the idea is let's give the teachers real-time information.
Let's not wait until it's too late. Let's talk about AI. You and I have both been lucky enough over the last six months or so to have engagement with the mix of both Microsoft and OpenAI and have early access.
And I remember some of the best examples of how to get the AI to do fun things were when you came up and saw me and we were brainstorming about this. You were the one who said, hey, if you tell it to write a speech like various politicians, including Trump and others, that it's stunning how it captures the voice of different people differently.
So, you know, give us the how you first were using the AI and then, you know, this is super timely because you've just recently come out with the Conmigo, if I'm saying that right. That's right. Very good. Yeah, I mean, it was OpenAI folks reached out and they said, hey, you know, we're a couple of weeks away from having
the first training of our model. And they wanted to reach out to Khan Academy for two reasons. One was they said, we want to make this really good at AP biology. And I only found out later, I don't know if this is true, they told me you gave them the challenge. That's right. In June, they kept showing me this thing. And I was like, yeah, it's kind of an idiot savant. I don't think it's practical. Why don't you
see if it can do with the AP biology exam, and I'm not going to pay any attention until you can, like, get a five. And I thought, okay, that'll give me three years to work on HIV and malaria and these guys. And then it was so bizarre because Sam Altman and Greg Brockman in Lake August said, hey, we want to come show you this thing. And so it was early September of
when there were like 30 people at my house. And, you know, I've said it's the most stunning demo
I've ever seen in my life. I mean, right up there with seeing the Xerox PARC graphics user interface that set the agenda for Microsoft for about 15 years, this demo was so surprising to me. The emergent depth that as they scaled up the training set, its fluency and you have to say understanding that computers could not read in the sense humans do and it couldn't write
in the sense humans do. And now, you know, lots of footnotes about hallucination and things like that. But I'm still personally in a state of shock at, wow, it is so good. And okay, therefore,
let's see where we can put it to good use. Yeah, 100 percent. Thank you for that challenge to them. I think they came to us because we have a large library of AP biology questions, et cetera. Like, hey, can we use that to either evaluate or train?
And at the time, I was like, well, what's in it for Khan Academy? I'm in it. And they're like, well, maybe you get access to the model. And I, too, was skeptical. I'd seen GPT-3 at that point. And it was cool, but I don't see how we're going to apply it. Two weeks later, they showed us the AP bio question. He said, so Sal, what's the answer to this? And I was like, OK, I think it's C. And it said, oh, the answer's C. And I was like, oh, that's interesting. I started getting a few goosebumps. And then I said, well, ask it why that's the answer.
explained it. Oh, yeah. It's so good at that. Oh, yeah. And I mean, I think what folks need to realize, because everyone had that moment with ChatGPT, but it was like that, but more because GPT-4 is even better. Way better, yeah. And I said, can you say why the other answers aren't correct? Did it. Then I said, oh, then I started like, I was almost shaking. Then I said, can you write 10 more questions like this? Bam, bam, bam. And the first 10 I saw were all
pretty good. I'm like, "Oh, yeah, yeah, yeah, that's all legit." And then the implications for Khan Academy started to go through my mind. And then we did start to get into some of its hallucinations and some of its math errors in those early days. But that weekend, they gave me and our chief learning officer access on Slack
And we just, like, I couldn't sleep. I was just having these in-the-rabbit-hole conversations with it. And then we had a hackathon for our team. We got about 40 people on our team under NDA. And we said, like, just come up with stuff. We were having the debates that everyone was having around...
Well, you know, the information is not 100% airtight. The math isn't great right now. The costs are not trivial. You know, can it introduce bias? What's the safety? Can you, how can you use, you know, the use of people's information, etc.?
But then we were starting to get it to work well as something that just helps you answer questions while you're watching a video, to work well as a tutor. And then every, honestly, every 10 minutes we thought about it, we're like, wait, it could also do this. It could also do that. It can also do that. And so we said, well, what if we could make it so you could talk to any historical character? What if you can make it so it gets into a debate with you? What if you can make it so it doesn't write your paper, but it writes it with you? What if it could do lesson plans for teachers? It could be the end of static curricula. I mean, just the imagination kept going. And
And by December, January, we had our team in full rapid prototyping mode. And yeah, just recently we launched Conmigo. And so far we're starting to titrate it out to the world, giving people access to it. But the feedback is tremendous.
Very promising. What we're hearing overwhelmingly from social media and the press is that they're really happy that we've engaged in this and that we're taking a safe approach where parents and teachers can monitor it. We have a moderation filter. OpenAI has also gone through great pains to make sure that things stay appropriate. You and I have talked extensively about the math issue and we've done some things that I think make it quite robust on top of the things that the model does.
And the costs are coming down dramatically. - Yeah, that's really impressive. - Even ChatGPT isn't bad, and then GPT-4 is dramatically better. I mean, it makes mistakes, to be clear. They both make mistakes. But one of the things we realized is, when you do math, especially if I'm tutoring you, let's say you do some work,
I don't just immediately say, correct, incorrect. I say, well, let me see this. Okay, let me see what he did. Okay, okay, okay. Okay, okay, yeah, yeah, good job. And one of the hacks, I don't think it's actually a hack. It's actually, I think, a principle we're doing, which is we weren't getting good results when we just asked it to decide when it acts as a tutor whether a student is right or wrong. But as soon as we said, you know what? Construct your thought, and these thoughts are private to you. Write that down first.
and then evaluate the student's response to your thoughts, and then say something publicly to the student, then the math improved dramatically. So it's, you know, it is funny. As soon as one day you think like, oh, this is so not like a human being. And so, and then the next year it's like, wow, that's kind of how we operate. We kind of need that thought before we can talk. The way I think of it is it's like a human that's not very good with context.
that the math context we know is a special context of check your answer. And it's also sometimes, even in conversations, if you get it into a mode where it's telling jokes, with humans, we have kind of a look or, you know, the new question is quite different. It thinks it's supposed to just keep telling jokes. And you almost have to do a reset to get it out of this, hey, everything is a joke-type mode. So,
It's like kind of a very naive person in terms of all these different contexts we're in. And my favorite one is where it tries to do Sudoku, which it can't do. And you pointed out, hey, that's not a good solution. And it says, oh, I must have mistyped. Because it's like, where is the typewriter? Exactly.
You have a typewriter? I like it. It gets defensive. It gets defensive. Learn that. It sees how humans deal with being accused of getting my dog a homework or something like that. So, yeah, it's kind of human but in a naive, untrained way.
So if we think about the next 10 years, you know, I'm sort of back to my optimism that I had when the Gates Foundation first got into education that both the absolute level of learning and the gap with lower income minority students that with these new tools can both close the gap and raise up the overall level of achievement. After the pandemic,
Tens of billions of dollars came in from the federal government to address learning loss. Everyone looked at these decades old studies around high-dosage tutoring and said, "Oh, we can kind of afford to do tutoring now."
And so they hired a bunch of tutoring companies and this and that. And it's not the tutoring company's fault. But what you're seeing now is just another example that we often see in the school system. Like, wow, it was a good idea, but we're not really seeing it move the dial. And the retrospective people are saying, well, in order for it to really work, it's got to be connected to what the kids are doing in school. And ideally, it should be happening while they're doing it in school.
And that's almost impossible logistically to do and expensive if you wanted to do it with live tutors. But Conmigo, the tutor that we're putting in Khan Academy, it in the moment can act like a fairly good human tutor. There's moments with it that I think would pass the Turing test where you think that there's a good human on the other side of the chat. And I think that has a good
chance of driving engagement? Because not only can it tutor, you can ask it, why do you need to learn this? And it says, well, what do you care about? And you say, well, I care about this or that. I say, well, if you care about this and this, you definitely need to learn this. And here's some other things to think about. And so it honestly answers it better than most humans would, that that is going to just drive a whole nother level of engagement, a whole
other level of, hey, a student is stuck in algebra because they have some gaps from seventh grade. That's very hard for one teacher with 30 kids to diagnose. But now to be able to intelligently with traditional tutoring interactions, say, hey,
hey, are you having trouble with the negative number here? Let's review that a little bit. I think that can drive a ton of engagement. I think, honestly, just feeling more connected can drive engagement. I'm actually really excited about the next phase of Conmigo is going to be Conmigo facilitating interactions inside of a classroom. With a group. With groups. So imagine a teacher just says, hey, Conmigo, take my class of 30, put them into 10 groups of three,
And I want every group to do this problem. And then I want the students to talk to each other about it. And you're the facilitator.
We've run those types of experiments and they're surprisingly fun where you can do that. And it's not just in math, you can do that in science. It pushes the students to, so why do you think, how does entropy and chemistry connect to entropy and information theory, computer science, and all of these types of things. And the humanities, and I do think all of these are correlated too. When students get confidence in one part of their academics, it drives all the other ones. But now a teacher can say, hey, kind
Canmigo asks all my students to give their reflection on this issue and immediately it can tell the teacher, hey, and this is science fiction. I think this is like six months away. It can say, hey, this is what everyone did. Here's some exemplars. Some of your kids tend to be confused here. This student seems a little bit disengaged. So I imagine that
I mean, I think we're going to see it next year. While the teacher is lecturing or whatever they're doing, if you're confused by something they just said, you just ask the bot in real time. It's like whispering to your friend without disrupting the classroom. But then the teacher will know, hey, three kids, Conmigo will tell the teacher, hey, three kids just asked me this question. I answered it for you. They're good now. But-
Hey, you might want to double-click on that a little bit. We have another nonprofit called Schoolhouse.World, which is about peer-to-peer tutoring. And we're already using AI to evaluate the transcripts and give the tutors feedback on what they could do better. I could see it being a little bit of a real-time, like, hey, you haven't called on this student, or here's a worked example for you. So I think it's going to be a wild couple of years, but hopefully positive. And one thing I always underestimated about
is how valuable it is for most students to have kind of a social experience. How do you create a conceptually rich but socially engaging way of talking about these math things? No, 100%. Some of my best memories are
sitting with friends and working on a particularly hard problem and somebody has an epiphany. And so, yeah, the more that we can try to replicate those things, the more that, you know, the model we use is it's a tutor for every student and it could be a teaching assistant or an army of teaching assistants for every teacher. And I think in every teacher on the planet would love to have
One, two, three amazing teaching assistants in that classroom that say, hey, let's facilitate these breakouts. Let's do focused interventions with different students. Let's make them explain the math to each other. One challenge we've always had is that when we bring new technology into the classroom, if we don't do it well, the teacher feels like, oh, you're trying to denigrate my creativity or freedom or your...
You're suggesting I'm not capable on my own, and yet we all know teachers are heroic, one of the most important hardest jobs in the world. So do you feel like we're doing better drawing both their input and showing that, okay, they are 100 percent at the center of the best solution?
Yeah. And look, I think not only is this about making sure that everyone understands their critical role, but it's also that one of the reasons why you have such a high attrition in the profession, I mean, we could talk about pay and compensation, all of that. But I think the biggest thing is it can be a very lonely and a very tiring job. I mean, most of us
with, quote, desk jobs, you know, like we don't have to be on continuously. Yes. But if you're a teacher... Nobody's trying to disrupt. No one's trying to disrupt or like, you know, chewing gum or looking at you. I mean, it is... And even when we're presenting, we're usually presenting to people who at least pretend to be interested. They're told to be, yes. Exactly. They got the memo. Well, you know, in a classroom, the kids aren't always giving you their most attentive look. And that...
is so draining energy-wise. And so we imagine the classroom of the future, and a lot of teachers are already doing this, is make the students do most of the things most of the time and that the teacher is the architect, is the conductor,
of what's going on. And I don't think anyone in an orchestra would think that the conductor is not an important role. The conductor is making sure that the orchestra is happening together and that everything is fitting together, that it's controlled chaos. And then I think everyone benefits. The kids have more fun, they learn more, and then the teachers have more fun. Instead of one teacher with 30, you should have two teachers with 60 or three teachers with 90 so that they also feel less lonely, which you can start to do if you take the lecture out of the classroom.
What do you think is going to happen in AI in education and work? Well, in the education piece, I actually think reading and writing where...
The software tools have not been that great. Very few students get feedback on an essay that this could be clearer, you really skip this piece in the reasoning. I do think the AI will be like a great high school teacher who really marks your essay and you go back and think, "Okay, I need to step up there."
The hardest question to answer is the job market. Now, we're not going to ever have too many teachers because in the worst case, we shrink class sizes and give more attention. The demand is pretty infinite there. I do a lot of my work in countries where the shortage of doctors and teachers is so acute
that the idea of, okay, the AI is too good. We don't really run into that. But the job market, I think that one, it's very hard to predict how it'll shift things. I don't think students should feel like, okay, I need to learn less. Like I don't need to multiply because computers were so good at multiplication. The new areas of job demand and how we shape this is going to be fascinating to see.
If you were to advise a parent or a student who's, let's say, a teenager is figuring out what to do with their life, given everything we talked about, AI, job market, what would you tell them? It's great if you learn enough that you find something that's particularly fascinating to you. You know, I have an orientation towards the sciences. I still think science.
you know, helping these AI things be better. We need that. The climate work we need is very, very scientifically intense. So anything in medicine, whether, you know, I have one child who wants to do political policy, including healthcare. I have one who's going to be a doctor. I have one who wants to be the CEO of the hospital. So all of them can contribute in very different ways. And what they pick kind of matches everything.
the differences in what they enjoy, what they're really good at. I would love it if some really smart people would go into politics. I know that's not attractive, but I think we need a new generation of science literate and reaching across the divide we have. God knows, you know, young people better come in and help us out there. So, Bill, what teachers really pop out in your mind that really affected you?
Well, early on in math, I was kind of lazy. And a teacher in eighth grade said, you know, how come you're so lazy? You could be really good at this. And I said, but, you know, we're not doing anything interesting. He would give me books to read and help really push me quite a bit. It made a huge difference that, you know, he sort of thought I was wasting my time. It changed my whole view of education that I kind of had this view that
The less effort you put in, the cooler you were. That's right. Very good. Do you have a particular teacher that helped motivate you? I have several that, probably three or four that pop in my mind. There's a second grade, Ms. Kraus and Ms. Roussel, it was this enrichment gifted program. They were the first teachers that really said, "Hey, what are you interested in?" That was the first time I experienced personalized education.
Jefferson Parish School System in Louisiana, I've got to give credit. It was a really great experience. Then I remember Miss Ellis in fifth grade. She ran her fifth grade social studies class, like what I now know as a college seminar. She would just peel an orange and just ask us questions. And I used to look forward to it. And then I also, Dr. Jairo Santania, when I was in high school, I took courses at the University of New Orleans, which is a local college there. And he was my differential equations teacher. And when he found out that I didn't have a computer at home,
He's like, well, you need a computer. And so he got me a research job at the University of New Orleans when I was 15 years old. And that's the first time I could, like, I really got to work with real computers. And if I didn't have that experience, I don't know what would have happened after that. If you had one magic wish to master a subject without any studying, which would you choose? I would say it would either be quantum physics or piano. That's a...
I'm embarrassed that even though I know all these computer languages and I took Latin and Greek, I don't speak any real languages other than English. So maybe if it was easy to learn and I could just use my wish on it, I might choose Chinese because I do think helping China and the US get along is going to be pretty important.
So I said before you came, if you had a record that you wanted to bring, please share it with me. Yeah. So what we're going to play is Bob Marley's Redemption Song, which is-- you mentioned my high school rock star days. Redemption Song was one of the first songs that I learned on guitar. It always spoke to me. It's about bondage of many kinds, both physical and mental bondage, and kind of breaking free of it. Yeah, so I'll play it. Cool.
So great. This is just a classic. Oh.
So if you do a web search on Sal Khan, you might get some of this guy. So do you ever get confused with Salman Khan? I do. In fact, the earlier days of Khan Academy, I got letters from some of his fans saying, you know, I've always been in love with you, and now I didn't know you could do math and all that. And it's funny because, you know, us South Asians like a good math.
you know, kind of cheesy thing. So when I went to India in 2050, actually, I saw you there. We had a live on like national television interview between the two of us. Oh, you're kidding. Just by virtue of us having the same name. But, you know, I do watch more Bollywood movies than folks suspect. Uh-oh. You know, my wife grew up in Pakistan. So people think that she made me watch the Bollywood movies, but it's usually, it's actually the other way around. I'm the one that made her watch the Bollywood. So I, you know, I was very aware of him. So it was kind of a fun experience.
thing to be in the same room and share the same name. Do you watch them in Hindi or English? I can understand Hindi, which I have no business understanding. My family, our mother tongue is Bengali. Oh, Bengali? Yeah. Oh, wow. And your wife? My wife's mother tongue, they're originally Gujarati, but she spoke Urdu growing up in Pakistan. But I learned Hindi mainly hearing my mom gossip on the phone when I was growing up.
And then Bollywood movies. That's my India now. Like, you know, a little bit Hindi and Urdu spoken is almost the same thing. So I know enough to get by. You and I both want educations to be a lot better and really for all types of students. It's an exciting time. And it seems like you've got as much energy for Khan Academy today as you did when it first got started.
Yeah, I've always run optimistic and, you know, I've always had a reasonable amount of energy. But yeah, I've told the team, like, what's about to happen in the next few years, it feels like we're in the middle of a science fiction book. And, you know, hopefully it'll be all for good. And I'm pretty hopeful. Well, thanks for coming down. Thanks for being on a learning journey with me in a cause that we both believe in. Thank you. Hopefully I've unconfused you. Yeah, yeah, definitely.
Unconfused Me is a production of the Gates Notes. Special thanks to my guest today, Saul Kahn. No, I've never been mistaken for Salman Kahn.