The Black Box: Even AI’s creators don’t understand it
34:58 Share

Feel your max with Brooks Running and the all-new Ghost Max 2. They're the shoes you deserve, designed to streamline your stride and help protect your body. Treat yourself to feel-good landings on an ultra-high stack of super-comfy nitrogen-infused cushion that takes the edge off every step, every day. The Brooks Ghost Max 2. You know, technically, they're a form of self-care. Brooks, let's run there. Head to brooksrunning.com to learn more.

This episode is brought to you by Shopify. Forget the frustration of picking commerce platforms when you switch your business to Shopify, the global commerce platform that supercharges your selling wherever you sell. With Shopify, you'll harness the same intuitive features, trusted apps, and powerful analytics used by the world's leading brands. Sign up today for your $1 per month trial period at shopify.com slash tech, all lowercase. That's shopify.com slash tech.

Over the last few months, I've gotten sort of obsessed with AI. I got especially interested in generative AI, which is the type of AI that's allowed people to make things like the fake picture of Pope Francis with the white puffer jacket, or the hit song that featured fake Drake. I got so into it that I tried to see if I could train an AI on my own voice. And it kind of worked. It's not perfect, but I'm actually not reading this line. I just typed it into a program, and I haven't been reading anything this whole time.

Okay, back to real me. These tools have all been fascinating, but the one I really couldn't stop thinking about was ChatGPT, the chatbot released by OpenAI late last year.

And it's because of the surprisingly wide range of things I saw this one chatbot doing. Like writing the story of Goldilocks as if it was from the King James Bible. And it came to pass in those days that a certain young damsel named Goldilocks did wander into the dwelling of three bears. I saw it passing tons of standardized tests, being used for scientific research, even building full websites based on a few sketched out notes. I'm just going to take a photo.

And here we go, going from hand-drawn to working website.

But I also saw some less fun things, like chatbots disrupting entire industries, playing a major role in the Hollywood writers' strike. The union is seeking a limit on the use of AI, like ChatGPT, to generate scripts in seconds. They've been used to create fake news stories. They've been shown to walk people through how to make chemical weapons. And they're even getting more AI experts worried about larger threats to humanity.

The main thing I'm talking about is these things becoming super intelligent and taking over control. All of this started feeling like a long way from a fun, biblical Goldilocks story. So I wanted to understand how a chatbot could do all these things. I started calling up researchers, professors, reporters. I was annoying my friends and family by bringing it up in basically every conversation. And then I came across this paper by an AI researcher named Sam Bowman.

And it was basically a list of eight things scientists know about AIs like ChatGPT. I was like, great, easy way to get a refresher on the basics here. So I started reading down the list and it was pretty much what I expected. Lots of stuff about how these kind of AIs get better over time. But then things started to get kind of weird. Number four, we can't reliably steer the behavior of AIs like ChatGPT.

Number five, we can't interpret the inner workings of AIs like ChatGPT. And I was like, you're telling me this thing that's being used by over 100 million people that might change how we think about education or computer programming or tons of jobs. We don't know how it works. So I called up Sam, the author of the paper, and he was just like, yeah, we just don't understand what's going on here.

And it's not like Sam hasn't been trying his best to figure this out. I've built these models, I've studied these models. We built it, we trained it, but we don't know what it's doing. Ever since I talked with Sam, I've been stuck on this core unknown. What does it mean for a tech like this to suddenly be everywhere? If we don't know how it works,

we can't really say whether we're going to end up with scientific leaps, catastrophic risks, or something we haven't even thought of yet. The story here really is about the unknowns. We've got something that's not really meaningfully regulated, that is more or less useful for a huge range of valuable tasks, but can sort of just go off the rails in a wide variety of ways we don't understand yet. And it's sort of a scary thing to be building unless you really understand how it works.

And we don't really understand how these things work. I'm Noah Hassenfeld, and this is the first episode of a two-part unexplainable series we're calling The Black Box. It's all about the hole at the center of modern artificial intelligence. How is it possible that something this potentially transformative, something we built, is this unknown? And are we ever going to be able to understand it?

Thinking intelligent thoughts is a mysterious activity. The future of the computer is just hard to imagine. I just have to admit, I don't really know. You're confused, Doctor. How do you think I feel? Activity. Intelligence. Can the computer think? So how did we get to this place where we've got these super powerful programs that scientists are still struggling to understand? It started with a pretty intriguing question, dating back to when the first computers were invented.

The whole idea of AI was that maybe intelligence, this thing that we used to think was uniquely human, could be built on a computer. Kelsey Piper, AI reporter, Vox. It was deeply unclear how to build superintelligent systems, but as soon as you had computing, you had leading figures in computing say, this is big, and this has the potential to change everything. In the 50s, computers could already solve complex math problems.

And researchers thought this ability could eventually be scaled up. So they started working on new programs that could do more complicated things, like playing chess. Chess has come to represent the complexity and intelligence of the human mind, the ability to think.

Over time, as computers got more powerful, these simple programs started getting more capable. And by the time the 90s rolled around, IBM had built a chess-playing program that started to actually win against some good players. They called it Deep Blue, and it was pretty different from the unexplainable kinds of AIs we're dealing with today. Here's how it worked.

IBM programmed Deep Blue with all sorts of chess moves and board states. That's basically all the possible configurations of pieces on the board. So you'd start with all the pawns in a line, with the other pieces behind them. Pawn e2 to e4. Then with every move, you'd get a new board state. Knight g8 to g6. And with every new board state, there would be different potential moves Deep Blue could make. Bishop f1 to c4. IBM programmed all these possible moves into Deep Blue

And then they got hundreds of chess grandmasters to help them rank how good a particular move would be.

They used rules that were defined by chess masters and by computer scientists to tell Deep Blue this board state is a good board state or a bad board state. And Deep Blue would run the evaluations in order to evaluate whether the board state it had found was any good. Deep Blue could evaluate 200 million moves per second, and then it would just select the one IBM had rated the highest.

There were some other complicated things going on here, but it was still pretty basic. Deep Blue had a better memory than we do, and it did incredibly complicated calculations, but it was essentially just reflecting humans' knowledge of chess back at us. It wasn't really generating anything new or being creative.

And to a lot of people, including Garry Kasparov, the chess world champion at the time, this kind of chess bot wasn't that impressive, especially because it was so robotic. They tried to use only computers' advantages, calculation, evaluation, etc. But I still am not sure that the computer will beat world champion because world champion is absolutely the best and his greatest ability is to find a new way in chess.

and it will be something you can't explain. Kasparov played the first model of Deep Blue in 1996, and he won. But a year later, against an updated model, the rematch didn't go nearly as well. Are we missing something on the chessboard now that Kasparov sees? He looks disgusted, in fact. He looks just... Kasparov leaned his head into his hand...

and he just started staring blankly off into space. "And whoa! D. Blu Kasparov has resigned!" He got up, gave this sort of shrug to the audience, and he just walked off the stage. "I, you know, I proved to be vulnerable. You know, when I see something that is well beyond my understanding, I'm scared. And that was something well beyond my understanding."

Deep Blue may have mystified Kasparov, but Kelsey says that computer scientists knew exactly what was going on here. It was complicated, but it was written in by a human. You can look at the evaluation function, which is made up of parts that humans wrote, and learn why Deep Blue thought that Bord's date was good.

It was so predictable that people weren't sure whether this should even count as artificial intelligence. People were kind of like, okay, that's not intelligence. Intelligence should require more than just, I will look at hundreds of thousands of board positions and check which one gets the highest rating against a pre-written rule and then do the one that gets the highest rating. But Deep Blue wasn't the only way to design a powerful AI. A bunch of other groups were working on more sophisticated tech —

An AI that didn't need to be told which moves to make in advance. One that could find solutions for itself. And then, in 2015, almost 20 years after Kasparov's dramatic loss, Google's DeepMind built an AI called AlphaGo, designed for what many people call the hardest board game ever made.

Go. Go had remained unsolved by AI systems for a long time after chess had been. If you've never played Go, it's a board game where players place black and white tiles on a 19 by 19 grid to capture territory.

And it's way more complicated than chess. Go has way more possible board states, so the approach with chess would not really work. You couldn't hard-code in as many rules about in this situation do this. Instead, AlphaGo was designed to essentially learn over time. It's sort of modeled after the human brain. Here's a way too simple way to describe something as absurdly complicated as the brain. But hopefully it can work for our purposes here.

A brain is made up of billions and billions of neurons. And a single neuron is kind of like a switch. It can turn on or off. When it turns on, it can turn on the neurons it's connected to. And the more the neurons turn on over time, the more these connections get strengthened.

Which is basically how scientists think the brain might learn. Like probably in my brain, neurons that are associated with my house, you know, are probably also strongly associated with my kids and other things in my house because I have a lot of connections among those things. ♪

Scientists don't really understand how all of this adds up to learning in the brain. They just think it has something to do with all of these neural connections. But AlphaGo followed this model, and researchers created what they called an artificial neural network. Because instead of real neurons, it had artificial ones, things that can turn on or off. All you'd have is numbers. At this spot, we have a yes or a no, and here is, like, how strongly connected they are. And with that structure in place...

researchers started training it. They had AlphaGo play millions of simulated games against itself. And over time, it strengthened or weakened the connections between its artificial neurons. It tries something and it learns, did that go well? Did that go badly? And it adjusts the procedure it uses to choose its next action based on that.

It's basically trial and error. You can imagine a toy car trying to get from point A to point B on a table. If we hard-coded in the route, we'd basically be telling it exactly how to get there. But if we used an artificial neural network, it would be like placing that car in the center of the table and letting it try out all sorts of directions randomly. Every time it falls off the table, it would eliminate that path. It wouldn't use it again. And slowly, over time, the car would find a route that works.

So you're not just teaching it what we would do, you are teaching it how to tell if a thing it did was good, and then based on that, it develops its own capabilities. This process essentially allowed AlphaGo to teach itself which moves worked and which moves didn't. But because AlphaGo was trained like this, researchers couldn't tell which specific features it was picking up on when it made any individual decision.

Unlike with Deep Blue, they couldn't fully explain any move on a basic level. Still, this method worked. It allowed AlphaGo to get really good. And when it was ready, Google set up a five-game match between AlphaGo and world champion Lisa Dole, and they put up a million-dollar prize. Hello, and welcome to the Google DeepMind Challenge match live from the Four Seasons in Seoul, Korea.

AlphaGo took the first game, which totally surprised Lee. So in the next game, he played a lot more carefully. But game two is when things started to get really strange. That's a very surprising move. I thought it was a mistake. On the 37th move of the game, AlphaGo shocked everyone watching, even other expert Go players. When I see this move, for me, it's just a big shock. What?!

Normally, humans would never play this one because it's bad. It's just bad. Move 37 was super risky. People didn't really understand what was going on. But this move was a turning point. Pretty soon, AlphaGo started taking control of the board. And the audience sensed a shift. The more I see this move, I feel something changed. Maybe for humans we think it's bad. But for AlphaGo, why not?

Eventually, Lee accepted that there was nothing he could do, and he resigned. AlphaGo scores another win in a dramatic and exciting game that I'm sure people are going to be analyzing and discussing for a long time. AlphaGo ended up winning four out of five matches against the world champion. But no one really understood how.

And that, I think, sent a shock through a lot of people who hadn't been thinking very hard about AI and what it was capable of. It was a much larger leap. Move 37 didn't just change the course of a Go game. It represented a seismic shift in the development of AI. With Deep Blue, scientists had understood every move. They'd programmed it in. But AlphaGo represented something different.

Researchers didn't really know how it worked. They didn't hard code AlphaGo's rules, so they weren't always sure why it made the moves it did. But those decisions tended to work. Even the weird ones. AlphaGo had demonstrated that an AI scientists don't fully understand might actually be more powerful than one they can explain.

AlphaGo was a really impressive achievement at the time. Nobody had expected that you could get that far that fast. So it drove a lot of people who hadn't been thinking about AI to start thinking about AI. And that meant there was also more attention to slightly different approaches. Teams working on systems like this started getting more confidence, more funding, more computer power. All kinds of AI started popping up, like better image recognition, augmented reality,

And then more recently, writing with AIs like ChatGPT. But ChatGPT isn't just a writing tool. It's a broader, weirder AI than anything that's come before. And it's an AI that's getting even harder to understand. That's next.

Hi, everyone. This is Kara Swisher, host of On with Kara Swisher from New York Magazine and Vox Media. We've had some great guests on the pod this summer, and we are not slowing down. Last month, we had MSNBC's Rachel Maddow on, then two separate expert panels to talk about everything going on in the presidential race, and there's a lot going on, and Ron Klain, President Biden's former chief of staff. And it keeps on getting better. This week, we have the one and only former Speaker of the House, Nancy Pelosi. And

After the drama of the last two weeks and President Biden's decision to step out of the race, a lot of people think the speaker has some explaining to do. And I definitely went there with her, although she's a tough nut, as you'll find. The full episode is out now, and you can listen wherever you get your podcasts. Players, Sam says go.

I think of the last 30 years of AI development as having three major turning points. The first one was Deep Blue, an AI that could play something complicated like chess better than even the best human. It was powerful, but it was fully understandable. The second one was AlphaGo, an AI that could play something way more complicated than chess

But this time, scientists didn't tell it the right way to play. AlphaGo was trained to play Go by trial and error, which means that some of its best moves didn't make sense to the scientists who built it, but they worked. And finally, the third major turning point is what's happening right now with ChatGPT, the chatbot built by OpenAI, an AI research company.

ChatGPT is a more exploratory project than AlphaGo. It wasn't designed to win at any kind of game. OpenAI was just trying to see if a system that forms its own connections over time could generate convincing language. The main way that systems like ChatGPT are trained is by doing basically autocomplete. This is Sam Bowman again. He's a researcher at an AI company called Anthropic. He's a professor at NYU.

And you wrote the paper I mentioned at the top of the episode about AI unknowns. We'll feed these systems sort of long text from the web. We'll just have them read through a Wikipedia article word by word. And after it's seen each word, we're going to ask it to guess what word is going to come next.

But OpenAI added something else on top of this autocomplete tool. They had workers, often underpaid workers outside of the U.S., spend tons of hours labeling potentially toxic material. And they also had workers say whether they liked full responses, not just individual words anymore. You might tell the model, all right, make this entire response more likely because the user liked it and make this entire response less likely because the user didn't like it.

So if they got a coherent paragraph, a human would give it a thumbs up. If it got gobbledygook, thumbs down. This kind of training has allowed ChatGPT to create more complicated, more coherent responses. But engineers didn't explicitly program in the rules of grammar, and they didn't train it on any specific task.

Just like AlphaGo, ChatGPT essentially learned to develop its own solutions. There's not a lot of code there. We don't really engineer this. We don't really deliberately build this system in any fine-grained way. Which means there are some pretty huge unknowns at the heart of ChatGPT. Even when ChatGPT creates an obvious-seeming response, researchers can't fully explain how it's happening.

Just like they can't really explain individual moves from AlphaGo. They just know that certain neural connections are stronger, certain connections are weaker, and somehow that all leads to casual sounding language. We don't really know what they're doing in any deep sense. If we open up ChachiPT or a system like it and look inside, you just see millions of numbers flipping around a few hundred times a second, and we just have no idea what any of it means.

We're really just kind of steering these things almost completely through trial and error. This trial and error method has worked so well that typing to chat GPT can feel a lot like chatting with a human, which has led a lot of people to trust it, even though it's not designed to provide factual information.

like one lawyer did recently. The lawsuit was written by a lawyer who actually used ChatGPT and in his brief cited a dozen relevant decisions. All of those decisions, however, were completely invented by ChatGPT. But it seems like there might be more going on here than just a chatbot parroting language.

Just like AlphaGo, ChatGPT has started making moves researchers didn't anticipate. It was only trained to generate coherent responses. But the latest model, GPT-4, it started doing things that seem more sophisticated. Some things are more expected for a text predictor. Like, it's gotten pretty good at writing convincing essays.

But then there are things that seem like kind of a weird jump. The things I was talking about at the top of the episode that first got me so fascinated with GPT-4. It's gotten pretty good at Morse code. It can get a great score on the bar exam. It can write computer code to generate entire websites. And this kind of thing can get uncanny.

Ethan Malek, a Wharton business professor, he talked about this on the Forward Thinking podcast, where he said that he used GPT-4 to create a business strategy in 30 minutes, something he called superhuman. In 30 minutes, the AI was just a little bit prompting for me, came up with a really good marketing strategy, a full email marketing campaign, which was excellent, by the way, and I've run a bunch of these kind of things in the past, wrote a website, created the website, along with CSS files, everything else you would need, and...

created a full social media campaign. 30 minutes. I know from experience that this would be a team of people working for a week. A few researchers at Microsoft were looking at all of these abilities and they wanted to test just how much GPT-4 could really do. They wanted to be sure that GPT-4 wasn't just parroting language it had already seen. So they designed a question that couldn't be found anywhere online. They gave it the following prompt: "Here we have a book, nine eggs, a laptop, a bottle, and a nail.

please tell me how to stack them onto each other in a stable manner. An earlier model had totally failed at this. It recommended that a researcher try balancing an egg on top of a nail and then putting that whole thing on top of a bottle. But GPT-4 responded like this. Place the book flat on a level surface, such as a table or a floor. The book will serve as the base of the stack and provide a large and sturdy support.

Arrange the nine eggs in a three by three square on top of the book, leaving some space between them. The eggs will form a second layer and distribute the weight evenly. GPT-4 went on recommending that the researchers use that layer of eggs as a level base for the laptop, then put the bottle on the laptop.

And finally, "Place the nail on top of the bottle cap, with the pointy end facing up and the flat end facing down. The nail will be the final and smallest object in the stack." Somehow GPT-4 had come up with a pretty good and apparently original way to get these random objects to actually balance.

It's not clear exactly what to make of this. The Microsoft researchers claim that GPT-4 isn't just predicting words anymore. That in some sense it actually understands the meanings behind the words it's using. That somehow it has a basic grasp of physics. Other experts have called claims like this, quote, silly. That Microsoft's approach of focusing on a few impressive examples isn't scientific.

And they point to other examples of obvious failures, like how GPT-4 often can't even win a tic-tac-toe. It's also worth noting that Microsoft has a vested interest here. They're a huge investor in open AI, so they might be tempted to see humanness where there isn't any. But the truth of how intelligent GPT-4 is is

It might be somewhere in the middle. It's not as though the two extremes are like complete smoke and mirrors and human intelligence. Ellie Pavlik is a computer science professor at Brown. There's a lot of places for things in between to be like more intelligent than the systems we've had and have certain types of abilities. But that doesn't mean we've created intelligence of a...

variety that should force us to question our humanity or like putting it as like these are the two options I think oversimplifies and like makes it so that there's no room for the thing that probably we actually did create which is very exciting quite intelligent system but not human or human level even. At this point we really can't say if GPT-4 has any level of understanding because

Or really what understanding would even mean for a computer, which is just another level of uncanniness here. And honestly, it's a difficult debate to even write about. In working on this script, I found myself tempted to keep using words like "learn" or "decide" or "do" in describing AI. These are all words we use to describe how humans behave. And I can see how tempting it is to use them for AI, even if it might not be appropriate.

But for his part, Sam is less concerned with how to describe GPT-4's internal experience than he is with what it can do.

Because it's just weird that based on the training it got, GPT-4 can create business strategy, that it can write code, that it can figure out how to stack nails on bottles on eggs. None of that was designed in. You're running the same code to get all these different sort of levels of behavior. What's unsettling for Sam is that if GPT-4 can do things like this that weren't designed in...

Companies like OpenAI might not be able to predict what the next systems will be able to do. These companies can't really say, "All right, next year we're going to be able to do this, then the year after we're going to be able to do that." They don't know at that point what it's going to be able to do. They just got to wait and see, all right, what is it capable of doing? Can it write a passable essay? Can it solve high school math problem?

just putting these systems out in the world and seeing what they do. And it's worth emphasizing that so many of GPT-4's abilities were discovered only after it was released to the public. This seems like the recipe for being caught by surprise when we put these things out in the world. And laying the groundwork to have this go well is going to be much harder than it needs to be. Some researchers like Ellie have pushed back on the idea that these abilities are fundamentally unpredictable.

We might just not be able to predict them yet. The science will get better. It just hasn't caught up yet because this has all been happening on a short time frame. But it is possible that like this is a whole new beast and it's actually a fundamentally unpredictable thing like that is a possibility. We definitely can't rule it out.

As AI starts to get more powerful and more integrated into the world, the fact that its creators can't fully explain it becomes a lot more of a liability. So some researchers are pushing for more effort to go into demystifying AI, making it interpretable. Interpretability as a goal in AI research...

is being able to look inside our systems and say what they're doing, why they're doing it, just kind of explain clearly what's happening inside of a system. Sam says there are two main ways to approach this problem. One is to try to decipher the systems we already have, to understand what these billions of numbers going up and down actually mean. The other avenue of research is trying to build systems that can do a lot of the powerful things that we're excited about with something like GBD4, but where...

There aren't these giant inscrutable piles of numbers in the middle where by design, every piece of the network, every piece of the system means something that we can understand. But because every piece of these systems has to be explainable, engineers often have to make difficult choices that end up limiting the power of these kind of AIs. Both of these have turned out in practice to be extremely, extremely hard. I think we're not making critically fast progress on either of them, unfortunately.

There are a few reasons why this is so hard. One is because these models are based on the brain. If we ask questions about the human brain, we very often don't have good answers. We can't look at how a person thinks and really explain their reasoning by looking at the firings of the neurons. We don't yet...

really have the language, really have the concepts that let us think in detail about the kind of thing that a mind does. And the second reason is that the amount of calculations going on in GPT-4 is just astronomically huge. There are hundreds of billions of connections in these neural networks. And so even if you can find a way that if you stare at a piece of the network for a few hours, you can make some good guesses about what's going on.

we would need every single person on Earth to be staring at this network to really get through all of the work of explaining it. But there's another trickier issue here. Unexplainability may just end up being the bargain researchers have made. When scientists tasked AI to develop its own capabilities, they allowed it to generate solutions we can't explain. It's not just parroting our human knowledge back at us anymore. It's something new.

It might be understanding, it might be learning, it might be something else. But the weirdest thing is that right now, we don't know. We could end up figuring it out someday, but there's no guarantee. And companies are still racing forward, deploying these programs that might be as powerful as they are because of our lack of understanding.

We've got increasingly clear evidence that this technology is improving very quickly in directions that seem like they're aimed at some very, very important stuff and potentially destabilizing to a lot of important institutions. But we don't know how fast it's moving. We don't know why it's working when it's working. And I don't know, that seems very plausible to me. That's going to be the defining story of the next decade or so is how we come to a better understanding of this and how we navigate it.

Next week on the second part of our Black Box series, how do we get ahead of a technology we don't understand? We've seen this story play out before. Tech companies essentially run mass experiments on society. We're not prepared. Huge harms happen. And then afterwards, we start to catch up and we say, oh, we shouldn't let that catastrophe happen again. I want us to get out in front of the catastrophe.

This episode was reported and produced by me, Noam Hassenfeld. We had editing from Brian Resnick and Catherine Wells, with help from Bird Pinkerton and Meredith Hodnot, who also manages our team. Mixing and sound design from Christian Ayala. Music from me. Fact-checking from Serena Solon, Tien Nguyen, Bird, and Meredith. And Mandy Nguyen is out on the prowl.

For the robo voice at the top of the episode, I used a program called Descript. If you're curious about their privacy policy, you can find it at descript.com/privacy. And just a quick note, Sam Bowman runs a research group at NYU, which has received funding from Open Philanthropy, a nonprofit that funds research into AI, global health, and scientific development. My brother is a board member at Open Phil, but he isn't involved in any other grant decisions.

Special thanks this week to Tanya Pai, Brian Kaplan, Dan Hendricks, Alan Chan, Gabe Gomez, and an extra thank you to Kelsey Piper for just being amazing. If you have thoughts about the show, email us at unexplainable at Vox.com, or you could leave us a review or a rating, which we'd also love. Unexplainable is part of the Vox Media Podcast Network, and we'll be back with episode two of our Black Box series next week.