The Zoom Election + Google DeepMind's Math Olympiad + HatGPT! Olympics Edition
01:01:34 Share

BP added more than $130 billion to the U.S. economy over the past two years by making investments from coast to coast. Investments like building EV charging hubs in Washington state and starting up new infrastructure in the Gulf of Mexico. It's and, not or. See what doing both means for energy nationwide at bp.com slash investing in America.

Let me ask you this. Do you have Olympics fever? I do. I love the Olympics. Do you? Yeah, I really enjoy the Olympics as well. I love my favorite story out of the Olympics so far is the pommel horse guy.

Do you know who I'm talking about? Wait, no, who's the pommel horse guy? This is this guy, Steven Nedorazic. He's on the men's artistic gymnastics team for the United States, and he is a pommel horse specialist. You know pommel horse? It's like the thing that you sort of spin around on. So most people in artistic gymnastics do like multiple events, but this guy, he's a one-trick pony. He is literally...

And I love him because he confuses the nerd jock binary, right? Like he's a nerd. He studies electrical engineering. He wears these like nerdy glasses, does Rubik's cubes. But he also, you know, lifts the U.S. to a medal in artistic gymnastics. But, you know, I was I was curious, like where Pommel Horse came from.

like as a sport because it's just sort of a weird one. Where does it come from? So I looked it up and it originates from the Romans who used wooden horses to teach mounting and dismounting. Oh. Yeah. And I love that because you know at some point there was some little gay Trojan that was like, what if we did a few little flips and spins when we got on our horses and then charged into battle? Yeah.

What if we had some panache? Yes. And the opposing armies would just stand down because they'd be like, we can't compete with that. That's too good. Your forces are simply too beautiful to slaughter in battle.

We surrender. Exactly. That's wonderful. But then that got me thinking about, like, what are the Olympic sports that used to be in the Olympics but are no longer? Like, what are the best sort of... Well, I have one. What's yours? Well, you know the wrestling used to be naked. And it's not clear to me why we stopped that, actually. Because, let me say this, you want to get the ratings up? Do they have to be wearing the singlets?

I'm Kevin Roos, a tech columnist for The New York Times. I'm Casey Newton from Platformer. And this is Hard Fork. This week, how 2024 became the Zoom election. Then, Google DeepMind's Pushmeet Kohli on how his team's AI hit a silver medal score at the International Mathematical Olympiad. And finally, it's time for an All-Olympics hat GPT. And we're going for the gold.

Well, Casey, it continues to be a big week for politics. Obviously, the election has been a source of big news stories over the past week. And my fascination over the past week when it comes to how tech is being used in this election has been with these sort of Zoom mega rallies. Have you been following these?

I have. It has been fascinating. Kevin, it says if all of the Democrats recently took a page from the 90s hip hop artist Rex in effect, who said, all I want to do is a zoom, zoom, zoom.

Yes. So I've been just fascinated by this because I think every presidential election sort of has kind of a defining technology, right? Like in 2008, Barack Obama ran. It was sort of the Facebook election. In 2016, obviously, we have Donald Trump who sort of uses Twitter to get his message out. And a lot of people credit that with his victory.

And now in 2024, we have Zoom, which is sort of emerging as this critical piece of campaign infrastructure in a way that I don't think a lot of people saw coming. No, and I mean, certainly I did not. Zoom has been around for a long time. I would say most of the people I know feel like they spend enough time on video conferencing as it is and are not leaping at the chance to do more of it.

And so when, as you point out, Kevin, this seemed to become one of the main technologies that Democrats are using to organize over the past few weeks, I really was surprised. Yeah, and I think we should talk about it today because I think it raises some interesting questions about how applicable this Zoom phenomenon really is beyond these Harris supporters and why Republicans aren't embracing these things as much as Democrats are.

So, Kevin, to get started, why don't you lay out what have been the biggest Zooms so far that have been part of this movement? So it started with this group, Win With Black Women, which organized sort of the first of these Zoom mega rallies. Shortly after Joe Biden dropped out of the race, they organized a call to

A ton of people showed up, like more than 44,000 people. They raised a bunch of money. And after that, basically every affinity group started doing their own version of the Zoom. And we should say, like, these are largely independent groups. They're not part of the official group.

campaign apparatus. But there was one for Black men. There was a big one for white women, for Kamala Harris. That one actually broke Zoom. It was so big, more than 150,000 people showed up. That raised a ton of money. And

you know, they had to like stream it through other platforms because they were, there were just too many people coming to the zoom, um, South Asian women, South Asian men, Latino men. They've all had their affinity group zooms for Kamala Harris. And then earlier this week, there was the one that I actually checked out, uh, white dudes for Harris. Uh, and how did you decide to attend the white dudes for Harris call out of all the ones you could have gone to? Yeah.

Well, obviously, I am a white dude. And while I am not involved with or supporting the campaign, like I'm, I thought it was important to report on how these organizing groups are doing on zoom, and how they're handling, you know, the massive influx of people who want to come to these things.

Yeah, I mean, I heard that there were nearly 60,000 people on White Dudes for Harris, which would be most of the podcasters in this country, I imagine. So, Kevin, tell us a little bit about what it was like to go to White Dudes for Harris. So...

The first thing I should say about the white dudes for Kamala, uh, zoom is that it was three and a half hours long. This was a, an endurance activity. I did not actually make it through the whole three and a half hours, but I did, um, watch a bit of it. And it was just kind of funny and strange. Like, uh, the, the celebrities on the call, they are not people who I think go on TV from their homes for a living. And so there's zoom rooms are like all very bad, like,

Jeff Bridges from the big Lebowski came on. He was like in this very poorly lit, like,

what looked like a garage. Joseph Gordon-Levitt also joined from like a very poorly lit room. You could like see his like video game arcade set up in the back. Like Lance Bass from NSYNC joined from his house and his kids were like running around. So it was just like, it was very funny and that everyone's kind of struggling to mute and unmute themselves like the way that everyone has always struggled to mute and unmute themselves. So it just kind of felt like

like the world's weirdest, most chaotic work meeting. But there was an agenda item, which was to raise a bunch of money for the Kamala Harris campaign. Yeah. You know, the Jeff Bridges thing is such a great reminder that most actors actually do not do their own cinematography. And so when they're called upon to do it, they can really struggle.

But so, okay, so you have all these sort of famous people and they're sort of coming up to the mic. And is there a sense of like, are they whipping each other into a frenzy? And like, when does the actual fundraising happen during this?

It sort of looks like an old school sort of telethon, right? Like there's, you know, if you donate to the campaign by like scanning this QR code, your name and the amount of your donation kind of scrolls across a ticker at the bottom of the screen. There's also this kind of like thermometer graphic on the right side of the screen that's sort of filling up as more donations come in. So these are fundraisers, and I think they've been quite successful at

that. I was very skeptical when I started on this call that this was going to be compelling. And, you know, I think on one level, like it's not all that compelling as content goes. But I think I was persuaded that at least these are very good ways to help campaigns or at least this one campaign raise money from small dollar donors.

I wonder, too, if, you know, because, you know, celebrities have long played a role in presidential campaigns and, you know, they might make an appearance at a rally and say a few words before the candidate comes on. It feels different to me when you're looking at that person in their home, you're on Zoom, which is maybe the work software that you're using to talk to your colleagues. And I wonder, Kevin, do you think there is an intimacy to Zoom that

that is making people turn to it because it is making them feel more into this movement and maybe likelier to open their wallets yeah i think there's absolutely something to the kind of intimacy or at least the faux intimacy of a zoom call that does feel very different i spoke to a bunch of the organizers of these calls and just try to figure out like like why is this such a phenomenon in the year 2024 why are people still excited about showing up at these zoom meetings

And part of what they said is that it feels participatory in a way that, say, like watching a broadcast does not like your your little box is there on the screen. You can put things in the chat, you know, with these huge zooms with tens of thousands of people on them, like you're probably not going to get called on. Like, but there's still a feeling that you are kind of in the virtual room with your maybe some of your political heroes, maybe a celebrity that you really like.

You know, people that you don't normally get to sort of have this sort of virtual connection with. So it feels small and intimate, even if the room is, you know, the size of a small stadium. And what can you tell about who is organizing these events? Does this idea get credited to any one individual or what should we know about that? Yeah.

Yeah, so everyone basically credits this group, Win With Black Women, for kind of kicking off the use of these mega zooms because they were the first to do it. And I spoke to Holly Holiday, who was one of the people involved with Win With Black Women, who said that this was actually a routine recurring Zoom call that they've been having for the last four years. Right. This is not a recent phenomenon anymore.

So this didn't come out of nowhere. They've been sort of building momentum on these calls. But, you know, their calls maybe used to have 100, 200, maybe 500 people on them if they were talking about something really important.

But I think the Kamala Harris entry into the race really did energize that and led to a huge spike in the number of people who are joining these calls. Yeah. So let's nerd out on the technical details a bit here, because I didn't realize you could get that many people on a Zoom call. And I feel like, you know, back when Zoom really came into its heyday in the year 2020, it was amazing.

I don't think it was possible to get that many people on a Zoom. So what have they been up to over there since then, Kev? Yeah, so it's a really interesting sort of web infrastructure story because you're right. Like four years ago during the 2020 campaign, there were some uses of Zoom for politics. Like Joe Biden did a few Zoom fundraisers, things like that. And...

Clearly, in the four years since then, Zoom has been building out its infrastructure and scaling up to the point that now you can have a Zoom webinar or an event with as many as 100,000 people in it. That's sort of the largest enterprise account tier ever.

But as we talked about, some of these events, or at least one of them, the white women for Kamala Harris call, actually got more than 100,000 people on it. And so I talked to Shannon Watts, who was one of the organizers of that call, who said that during the actual call, as Zoom was sort of breaking, they were sort of frantically messaging people that, you know,

were at Zoom saying, like, can we get more capacity? Can we get more capacity? And actually, Shannon posted a message on X that she got from a Zoom employee saying basically, okay, we've talked to the engineers and we can bump you up to accommodate as many as 200,000 people

on the call and actually told them that this would make them the largest Zoom event in Zoom history. So I have not been able to get in contact with Zoom. They are not responding to press emails. I think maybe they're a little shy about sort of touting too much connection to the Kamala Harris campaign. But

it does appear that behind the scenes they've been working very hard to increase the capacity of these Zoom meetings, which is not a simple thing. Like, this is very computationally demanding, and my understanding is that it has taken quite a bit of work for them to accommodate all of the interest in these big Zooms.

I'm also curious, though, Kevin, like, how do you get 150,000 white women to show up in one place at the same time? How are these organizers getting the word out? Because I think that's the sort of complementary tech piece here, right, is the distribution actually reaching these potential voters and getting them to show up somewhere.

Yeah, some of it is happening the ways you'd expect. It's happening on social media. People are posting, you know, the links and sharing the links to the web stream of these calls. But they're also using email. They're texting people, sort of basic sort of voter and donor outreach, things that campaigns have been doing for years.

And these calls are just being organized by people who already have a lot of experience in political organizing. So they know who their sort of celebrity supporters are and how to get in touch with them. Like these are not random Kamala Harris fans organizing these calls. These are people with a lot of experience in online organizing.

You know, it's a really interesting part of the discussion to me because I feel like we've talked so much over the past year about how hard it is to have political discussions online relative to maybe four years ago, right? Meta is playing down politics on threads and Instagram X is a shadow of its former self. And so the fact that you can get hundreds of thousands of people to come to these events on pretty short notice, I think speaks maybe to...

The surprising resilience of that system or the ability of these organizers to find ways around it?

Yeah, I think especially if you promised them that they'll see some celebrities on their Zoom screen, people will show up to these things on pretty short notice. It's much cheaper and easier to plan a Zoom rally than an actual rally where you need to rent a building and get the permits and get security and deal with all these celebrities and their handlers. You can just send Jeff Bridges or Lance Bass or Bradley Whitford a Zoom link and say, show up here. They're a lot more likely to come. Yeah.

Let's talk about the differences between the Democrats and the Republicans here, because you may have noticed we have not talked about any equivalent Republican Zooms. We looked around. We've had trouble sort of finding anything that Republicans are doing in this space. So, Kevin, how would you kind of compare what Republicans are doing on the organizing front to what Democrats seem to be doing so far this year?

Yeah, I mean, this was one of my big questions is like, are we going to start seeing a bunch of pro-Trump mega zooms? And we might. I think that the campaigns are definitely like taking note of how successful these rallies are going as fundraisers. But I was talking with a Republican strategist the other day about this, and he made a really interesting point, which is that in 2020,

Zoom got kind of Democrat coded. Like it was seen as what you would do if you were like afraid because of COVID to go out and go to a rally. And Republicans in 2020 were much more likely to want to ignore social distancing guidelines and go out to big rallies with a bunch of other people. And we did have a bunch of big Trump rallies in 2020.

And Zoom kind of became this sort of, I don't know, thing that they were a little bit opposed to just kind of on the grounds that it sounded like something Democrats were doing. And so as a result, there has not been as much attempt to organize on the right around Zoom. Now, obviously, we should say, like,

Republicans, the Trump campaign, they are doing a lot of digital outreach. They do spend a lot of money on ads, on social media platforms. They also have a bunch of pro-Trump influencers who are seeding their messages out on various platforms.

But from what I can tell, there is not sort of this groundswell of these sort of mega zooms that are happening in support of Trump. And we may not see that replicated on the Republican side. No. And in fact, they are making fun of the Zoom calls. Donald Trump Jr. said ahead of the white dude's Zoom call, quote, they should give it a more fitting name like Cucks for Kamala. That is what Don Jr. posted on X. And I just have to say, I think that's rude, Kevin.

Yeah, I mean, these things have gotten some mockery and pushback. One person I talked to compared it to a group therapy session for liberals, you know, where you're just sort of going around talking about your inner angst and why you're supporting Kamala Harris. And then there was a person who was on the white women call who...

who was sort of doing kind of a gentle parenting style speech that got made fun of on the right. People said it sounded like the HR woman who's about to lay you off. So not universal acceptance on these. What do you mean a gentle parenting speech? Like she was telling people how to parent? No, do you know gentle parenting? I mean, I can sort of imagine it, but if it's something specific, then I don't. Yeah, it's sort of this movement, you know,

where you're supposed to just like affirm your child. You're supposed to say, you know, oh, that's a really big feeling you're having. And let's like, let's talk about the feeling. So she was sort of proceeding on the Zoom call in this way that people said, sort of sounded like she was trying to gentle parent the other attendees on the call. And people reacted strongly against that. So like, I don't want to give the impression that like,

The Harris campaign and its surrogates are out there like breaking new ground when it comes to digital campaigning. Zoom has been around for years. It's not our new tool, but I think it reflects the enthusiasm among Democrats right now. This is sort of I have a few thoughts about these Zoom rallies. One is that I think this is not going to become sort of.

vital campaign infrastructure in every race. I think a lot of what's working about these is very specific to the Harris campaign, where you have a new candidate who people are sort of being introduced to, even though she's been the vice president, introduced to as a presidential candidate only about 100 days before the election. You have sort of this compressed timeline, and there's just a lot of enthusiasm. I think these Zoom mega rallies work

when you have a bunch of people who are already enthusiastic, I don't think they're a way to sort of drum up enthusiasm. So I would be shocked if these become, you know, mega zooms for, you know, some random state Senate race in some state. Like, I don't think this is going to be replicable at this scale in a lot of other races. Yeah.

Do we think that X, to any degree, has become the Republican alternative to Zoom for online organizing? Ron DeSantis launched his campaign on Spaces, which is X's sort of audio live streaming tool. And there are some still yet to be scheduled live streaming town halls with Trump and RFK Jr., who's running as an independent.

That was reported in May that that might happen. So is X maybe the Zoom alternative for the Republicans? Yeah, I think that's a good analogy. And certainly we've seen like a lot of sort of big name Republicans appearing on spaces and doing chats with Elon Musk and things like that.

They're also podcasts, YouTube channels. They have a lot of places where they can get a wide audience for their message. But yeah, they don't seem to be embracing these Zooms that I can see yet. But I wouldn't discount their ability to raise large sums of money. Obviously, they were very successful in 2016 and 2020 raising money through Facebook ads and other platforms.

So I think it's too much to say that Zoom is the only important technology in the 2024 election cycle, but it does seem to be something that has been raising a ton of money for the Kamala Harris campaign.

So here's the big question that I've been wanting to ask you about all of this, Kevin, which is the year 2020 was the year of the COVID lockdown. It's when most of us were introduced to Zoom as an everyday tool. Why wasn't 2020 the Zoom election?

Yeah, it's also the question that I had when I started looking into this. And I think there are sort of three main answers to why this didn't take off in 2020. The first is just the technical reason, right? You couldn't have 100,000, 150,000 people on a Zoom call. You just literally could not do that. They did not have the infrastructure to make that possible at the time.

The second reason is you also had this sort of security problem. You remember Zoom bombing? Yes, I was actually a Zoom bombing victim in 2020. Wait, tell the story. Well, so, you know, when lockdown started, a couple of friends and I would just kind of host a public Zoom and we'd send the link out and just be like, hey, we're like all hanging out after work if you want to talk to, you know, some random tech people. And somebody, you know, took over the Zoom and started showing pornography, Kevin. Oh, boy. Yeah.

Well, and I'm deeply sorry for my role in that. Listen, I was bored. It was either showing porn on your Zoom or baking sourdough, and I chose violence. Well, so, yes, there was the Zoom bombing issue. There were a lot of fears that if you held a big political rally on Zoom and advertised it and passed the link around, someone would come in and start saying crazy things.

And of course, Zoom did do a few things to increase security back in 2020, but I think people were still pretty wary of it. So that's another possible reason that these rallies didn't take off in a big way back then.

And then I think something that I heard from sort of political organizers who tried to do big Zoom events in 2020 and weren't able to is just that there was so much Zoom fatigue at the time, right? People were using Zoom all day, every day for work, a lot of them. And so if someone sent you a link to a Zoom where, you know, a candidate or a surrogate is going to be talking, you were just kind of like, oh, great. Now one more thing I got to like stare into my computer for. Yeah.

And I have to say, to this day, when people ask me to do any kind of virtual speaking or talk to a classroom, the amount of unprocessed COVID trauma that just leaps to the forefront of my mind is overwhelming. I truly still do not like going on video chat because of the 2020 flashbacks it gives me.

But for a lot of Democrats, that has not been the case. Yep. So what do we think this means for online organizing going forward, Kevin? Is this just kind of like a one-off quirk of the 2024 election? Or do you think we're learning something here that political campaigns will apply in the future?

I mean, I think they are going to be an important part of campaigns going forward because there is this feeling of intimacy and participation and authenticity. You know, you do get to see a little, you know, more of your political heroes with their hair down, you know, at home in their living rooms or their home offices, which is pretty appealing to some people. I would actually love to see this trend go further. I want to see a Zoom presidential debate.

And not only do I want to see one, I want to see many of them. Like imagine in a campaign, if instead of doing these like handful of big televised debates that are on national TV and the campaigns, you spend weeks preparing for them and, you know, Lester Holt moderates or whatever. Imagine just a,

You know, if there are 100 sort of mini debates happening on Zoom where they could just... Yes. I mean, the technology is so easy. Imagine you could just, you know, oh, you're having a disagreement about border policy. Hop on a Zoom. Let's debate it out for 10 minutes. And then everyone can sort of decide how that went.

But I think this is sort of the direction that things should move. What do you think? Well, I would love to discuss this with you more, Kevin, but I have to hop on a gay dudes for Charlie XCX call. So I'll be over on that Zoom if anybody needs me. All right, have fun. When we come back, how Google DeepMind struck silver in a competition with young mathletes.

This podcast is supported by FX's Shogun. Set in the year 1600, Lord Yoshitoronaga is fighting for his life as his enemies unite against him. With 26 Emmy nominations, including Outstanding Drama Series, Shogun is the most Emmy-nominated series of the year. Starring Emmy nominees Hiroyuki Sanada, Anna Soai, Tadanobu Asano, Takahiro Hira, and Nestor Carbonell. Shogun is available for your Emmy consideration

at fxnetworks.com slash fyc. I'm Julian Barnes. I'm an intelligence reporter at The New York Times. I try to find out what the U.S. government is keeping secret. Governments keep secrets for all kinds of reasons. They might be embarrassed by the information. They might think the public can't understand it. But we at The New York Times think that democracy works best when the public is informed.

It takes a lot of time to find people willing to talk about those secrets. Many people with information have a certain agenda or have a certain angle, and that's why it requires talking to a lot of people to make sure that we're not misled and that we give a complete story to our readers. If The New York Times was not reporting these stories, some of them might never come to light. If you want to support this kind of work, you can do that by subscribing to The New York Times.

Well, Kevin, we have Olympic fever on the show this week. And to start with, we want to talk about the International Mathematical Olympiad. Now, this is not one of the Olympics that they show on TV, right? With Bob Costas? No, but they should because they are arguably more important.

Kevin to the future, even than running and jumping off that pommel horse. Yeah. So the International Math Olympiad, from what I understand, and I say this as a person who did not do very well in math and did not study it beyond high school calculus, was

My understanding is this is the pinnacle of elite mathematics for young mathematicians, that this is a notoriously difficult test that is given out every year and that you can actually win a medal for solving problems correctly. That's right. For a while now, people who work on AI have looked at this competition. They've said, "This will be really hard to get an AI system to succeed at this thing. I wonder how long it will take."

And the folks over at Google DeepMind took that as a challenge. And recently, we learned that when they put their own system to this test, they were able to solve four of the six problems, the equivalent of a silver medal in the competition. Yeah, this was a really big deal in the world of mathematics, from what I understand. I think it's fair to say that no one expected or very few people expected an AI system to be able to get the equivalent of a silver medal this quickly. Yeah.

That's right. And in fact, Google DeepMind's score was only one point shy of what would have been needed for a gold medal. So they are already operating almost at the level of what the smartest young mathletes can do. So like you just said, Kevin, this is a really big breakthrough. And for me, it raises a lot of questions about where we go from here. Now that we're able to use these systems to

solve these really novel problems that require a lot of creativity, can we actually start to generalize? Can we move beyond simple math and start to solve really hard problems in other domains? Can we solve problems related to novel physics or chemistry or maybe some of the big issues facing us in the world?

So we have a lot we want to find out today about how these models work, what this means for the future of AI in general. And so to answer those questions, we've invited Pushmeet Kohli to join us. Pushmeet is the vice president of research for Google DeepMind, as well as the overall head of AI for science. So let's bring him in. Let's bring him in. ♪

Pushmeet Kohli, welcome to Hard Fork. Thanks for having me here. Yeah, well, so I want to start by something that I read about your team recently in the New York Times, which is that Google DeepMind has a ceremonial gong in its office and that the gong has been banged a lot recently. So tell us about this gong and what's been leading you to bang it.

Yeah, so I think this gong is a traditional, especially in our reinforcement learning team. And they have used it whenever the reinforcement learning agent does something dramatic, right? It discovers something new or solves an interesting problem. And

a couple of weeks back, there was the International Maths Olympiad and we asked our most advanced maths agent to try to solve some of these problems. And these are incredibly hard mathematic problems that

some of the best young mathematicians all over the world come to be tested on. And before this, it was felt that the current generation of AI systems were not there yet in terms of being able to solve these problems.

But our agent managed to solve a bunch of these problems. And whenever it solved one, the team hit the gong. I love this. I think newsrooms should adopt ceremonial gongs too. Like if someone breaks a big scoop, you just ring the gong. I think that would add some whimsy to the newsroom. Yeah, absolutely. Yeah.

Now, one of the knocks against large language models has been for the past couple of years that they are bad at math, which has been surprising to a lot of people because people think, you know, computers, they must be good at math. But then, you know, they see examples of people putting what seem like extremely simple arithmetic or algebra problems into a system like ChatGPT or Gemini and out comes this like very confident but totally wrong answer, right?

So why is math such a challenge for large language models? And then how did you come up with a solution to that?

Yeah, so the interesting thing about maths is that it has a set of rules. It's defined by a set of rules, right? You cannot just make things up in maths, right? Contrasted with natural language or the current generation of large language models, they are probabilistic models. They think about what is the probability of the next word or the next token appearing, right? And many things are possible. People sort of make things up.

right? And so the model is just thinking about in probabilistic terms. In mathematics, when you're reasoning about what comes next, you have to be very precise. There are specific rules, and which makes it an extremely challenging thing, because you have to be very precise in what you say. And if you have to now write a very long proof, you have to be precise in every step of that proof, which makes it extremely challenging.

Right. So, you know, my understanding is that to achieve the success that you had this time around, you used two different AI models. What can you tell us about what that breakthrough was that let you perform as well in the Olympiad as you did? And please keep in mind that Kevin and I know almost nothing about math. So there were two systems called AlphaProof and AlphaGeometry.

Alpha geometry was a more precise system which was focused on solving geometry problems. It has a language in which it describes geometry problems and it is able to do very effective search, fast search in that language. Alpha proof, our other system, the real key result was that it is able to understand problems in this much more expressive mathematical language.

a language called Lean. Like you must have sort of heard of programming languages, C++ and Java and Python and so on. So for maths, there is this very specific language called Lean in which you can formalize, you can specify what the maths problem is. And AlphaProof works in this language

to write proofs for those problems, right? And by writing these proofs in a very precise way, it is able to sort of make progress and then verify that it actually gave the right answer, not just hallucinated it. - And are these technically large language models or are they some different kind of AI system?

So the system underlying it has the same basic machinery, but how it trains and learns and how it sort of conducts the search is different.

So it's more specialized towards the maths problem. And the other sort of interesting thing about it is, is while solving the problem, it actually learns how can it become better and better at solving this specific problem. So these are such hard problems that it tries to specialize on those problems while it's trying to solve it.

And how do you train a model like AlphaProof or AlphaGeometry? Do you just feed it a bunch of problems from advanced math textbooks and say, okay, learn all this stuff and then go solve these proofs? Yeah, so that's another very interesting question. Now, the thing that I was telling you about that this agent AlphaProof solves problems in this language called Lean. And the tricky thing is that we don't have a lot of training data

of maths in this language called Lean. So what you would do first is we collected a lot of maths problems in English, then sort of build a translator which could translate these problems into this more specific and formal language called Lean.

That gave us a lot of translations. And then we asked this agent to solve these millions of resulting problems. So it was trained on all these millions of different sort of problems. And every time it was successful in doing so, it got a reward.

You know, I've been thinking about giving Kevin a reward every time he asks a good question just to see what that would do for the podcast. So we'll keep you posted about that. But I do want to talk about how you actually went about solving them because, you know, me not knowing anything else, I think, well, you go to your Gemini.Google.com, you copy and paste the prompt, you know, into the box and you hit enter. And then, you know, a few seconds or a few hours later, you get your result. But I'm guessing that is not actually what happened.

Yeah, it's a bit more sort of complicated at the moment. Eventually, we hope that Gemini will be at the place where you can sort of take one of these problems and put it there and we'll be able to solve it. But in this particular case, this is the real sort of cutting edge of machine learning and AI.

We took these problems in mathematics, like in English, then a mathematician sort of translated it into a lean version, which is a more formal definition of the problem. And that problem was given to the agent. And the agent then spent hours

And in some cases, for some of the hardest problem days, thinking about the problem, solving multiple variants of the problem to see if it can crack it in one way or the other way, and then eventually found the proofs for these problems. And what are you able to see along the way? Like, is it generating outputs that you're able to review? Or is it all just sort of, you know, just a word? It just spits out a number at the end. Yeah.

It's very interesting. You're asking what it's very interesting. What it does, it essentially tries to solve variants of the problem. So it will say, OK, let me, there is this very hard problem. Can I

make an easier version of that problem? For example, I have to prove a statement about numbers. Can I prove this statement about even numbers? Is the problem easier for even numbers, right? Or rational numbers? Or for this, let me think about proving it for this special case when the numbers are less than something, right? And

And then over time, it sort of starts proving these. And then once it is successful, it changes its learning sort of strategy to be able to sort of prove the full problem. So that was very interesting to see.

I want to talk about how long this system took. You just said that your system had taken as much as several days to solve some of these problems from the Math Olympiad. When real human students are competing in the Math Olympiad, they had about four and a half hours for each exam, so maybe like an hour and a half per question. But your model is slower than that, which I think would strike some people as odd, given that it's a

It's a computer, and computers are supposed to be fast at things, especially with this much compute in them. So what explains the disparity between the time that alpha proof takes to solve one of these problems versus a very talented human mathematician?

Yeah, it just tells you the difficulty of the problem. Now, like, of course, we could sort of increase the amount of compute, right? And those times would be smaller. But in this particular attempt, we were not trying to really minimize the amount of time. What we were trying to do is maximize what

types of problems can even be solved regardless of the time. Because before this, even if you give the system not days, but months, it will not be able to solve the problem.

And am I right, Pushmeet, that while some of the problems did take up to three days, that some other problems your system was able to solve in just a few minutes? Few seconds even, right? So for the geometry problems, our specialized system, Alpha Geometry, it operates in the language that's specifically designed for expressing geometry problems. And in that language, it's able to sort of solve these problems in a matter of seconds. Yeah.

Sometimes these systems solve problems in ways that humans usually wouldn't come up with. There's a certain difference in how they go about solving a problem. I remember when AlphaGo was competing against the world Go champion, Lee Sedol. There was this famous Move 37 that everyone was talking about where the computer sort of makes this move and all the commentators and everyone who understands the game of Go is like,

What was that? Like, that was a move that no one expected, that no human Go player would have made, but that ended up being sort of the correct move in that situation. Were there any sort of ways that this model was able to solve these math problems that struck you as either strange or just unhuman-like? Yeah.

You're absolutely right. There can be many counterintuitive strategies that the agent can be taking. Now, we are still sort of digesting all the information that the agent worked out. So we have not done the full analysis yet, but it was very interesting to see all the interesting sub-variants of the problem that this agent was sort of working on. So some of them

Maybe a mathematician might also solve it in that way. But definitely in certain cases, the way it constructed the proof was quite different. I'm curious, Pushmeet, about the two problems that your models were not able to solve correctly. Is there a theory among the team for why those problems were harder?

Yeah, so that's a very good question. So there are different types of mathematics problems that appear in the Olympiad. There are algebra problems, number theory problems, geometry problems, and combinatorics problems. And the two problems that we weren't able to solve were combinatoric problems. They require reasoning about large solution spaces and so on. So, I mean, we are now in

analyzing what made those problems challenging and then are working on improving the agent. Yeah. And Kevin, you wouldn't know this, but combinatorics is the branch of mathematics dealing with combinations of objects belonging to a finite set in accordance with certain constraints, such as those of graph theory. Thank you, Casey. That I'm sure was just ad-libbed off the top of your head and not just read from a Wikipedia page. I appreciate that. No, I'm a huge combinatorics guy. Pushmeat, what?

One of the big questions in AI is whether it will soon be able to advance the frontier in scientific fields by solving novel problems, maybe by coming up with a proof for an unproven theorem. How close do you think we are to that now?

Yeah, so I think with regards to alpha proof, this is still sort of a bit away from research level mathematics, but we hope it will become an interesting tool for mathematicians to use very sort of soon. Aside from alpha proof, in the last three years, some of our other sort of models have already made new discoveries in mathematics.

But those are more specialized sort of models which were trying to solve a specific problem like there is this problem of matrix multiplication. Can you multiply matrices faster? This is like a very fundamental problem in computer science and mathematics. And we were able to show that you can come up with a new algorithm which is faster than what the computer scientists knew about after sort of decades of work on that problem.

And we have similar sort of results for other problems as well. But in a general way, alpha proof is probably the most general and state-of-the-art system.

I'm curious how elite mathematicians are feeling about AlphaProof and all the successes that you've had with this AI system. I remember that after the big AlphaGo competition, where the system that DeepMind had developed beat the world champion Go player, Lee Sedol actually retired shortly after that from professional Go because he basically said there was no point in trying to beat the machine anymore. He was

it sounded like a little bit depressed about the fact that he had been sort of usurped in this way. Do you think mathematicians will see this silver medal on the IMO and go, well, why am I doing this? And just take an early retirement, hang up their calculators? So this is a very interesting thing. Now, even in the case of Go, what we saw is that

After the match, when Go players started analyzing AlphaGo's strategies, they discovered a lot of new theory about Go that they had not sort of seen. Like Move 37, you mentioned that earlier. But like really exploring this unexplored part of the game re-energized the whole community. Now, in the context of maths, if you think about it,

It's not a game, right? We are trying to understand the world and it's a very hard problem. And what alpha proof or systems like this give you is basically a very powerful tool to assist in that big thing that mathematicians and scientists are trying to do, which is try to make sense of the world. Hmm.

I'm curious if we could just zoom out a little bit because obviously there are a lot of people who are excited or nervous about the progress happening in AI, but who aren't elite math Olympiad participants and don't really see the obvious connection between what you all are doing

at Google DeepMind and their own lives. Can you just sort of paint a picture for us of a future where AI becomes very, very good, if not better than all living humans at complex math? What would that mean for society? What would that mean for scientific progress? Basically, why should people care about this?

So like, first of all, I should say, like, in your description, I am one of those people in the sense that like, my background is a I'm a computer scientist. You're not a mathlete? No, I'm not. I'm not a mathematician. I'm a computer scientist. And if you give me some of those IMO problems, I would struggle. And I have no I have no hesitation in saying that.

Those are very hard sort of problems. Now, the interesting thing is, what are the long-term sort of consequences of this progress? It might seem, oh, it's a conceptual thing, like some maths sort of contest, but it's

in the real world, it has major implications. The same techniques of giving proofs of these mathematical statements, the same type of technology can be used for understanding and proving the behavior of software systems, for instance. And if you look at

Basically, the world today, it's run by software. This will have implications for cybersecurity. Like we will be able to sort of prove correctness, sort of find bugs in software systems much more effectively. And so there are so many downstream positive use cases of the same technology.

Well, it sounds like there will soon be many more reasons to bang the gong over there. All right. Pushme, thank you so much for coming on. Great to talk to you. Yes. Thank you so much. This was super fun. Yeah, it was fun. It was fun. I loved the discussion. You guys are great. When we come back, we're going to talk about a different kind of Olympics. The real ones happening in Paris. We'll play a special all Olympics edition of Hat GPT. When it comes to making plans, you are the best.

from those delicious barbecues to special birthdays and unforgettable family get-togethers. The same way you plan for those important moments, start planning to protect you and your loved ones from a natural disaster like a hurricane, flood, wildfire, or tornado. Sign up for local weather and emergency alerts, prepare an emergency kit, and make a family communications plan. Get started today at ready.gov slash plan. A public service message brought to you by FEMA and the Ad Council.

Okay, so this week we're playing a special version of Hatch EPT. That's right, Kevin. The Olympics are happening and we are both super into them. And so we thought, why not take one of our favorite games and give it an Olympian twist? And to be clear, is this the math Olympics that we're doing or is this the regular sports Olympics? No, this is what I think some people might call the real Olympics. Okay.

Okay. No shade to the mathletes out there. Not at all. Your competitions are very real, too. Absolutely. But these are the ones going on in Paris. That's right. Okay. So Hat GPT, of course, is the game where we pick news stories out of a hat and we start talking about them until one of us gets sick of hearing the other person and says, stop generating. All right. Let's do it. Let's win a gold medal in Hat GPT. ♪

So I'm going to pull out of a hat because you're out of the studio this week, and I'll just read them, and then we can do our thing. First up, an AI version of Al Michaels will deliver Olympic recaps on Peacock. This comes from The Verge. It's about a new feature that NBC has developed called Your Daily Olympic Recap.

app or in the Peacock app, you basically put in your name and, uh, pick the sports that are interesting to you. And then, uh, Al Michaels or an AI version of Al Michaels, the legendary broadcaster will give you daily personalized recaps of these Olympic events. Uh, Casey, have you tried this? I actually have because I was so curious what it would be like to see, uh,

Is it fun? Is it good?

Um, it's okay. You know, you get to personalize it a bit. I think my main review would be his voice generally sounds pretty good. Like if you sort of weren't paying that much attention, you might not be able to figure out that it's AI. But the sort of getting in and out of the clips is very stuttery and weird. And it feels like the Peacock app is having a stroke every time I watch these clips. So the AI performance

piece of it is good and the rest of it has been sort of bad. That was my experience and I believe we have a clip of it, Kevin, that I can play for you. Please. Hi, Casey.

Welcome to your daily Olympic recap, your personal rundown of yesterday's most thrilling Olympic moments. Suni Lee impressed in multiple events, including a strong display on uneven bars by sticking her landing. Over on the balance beam, Simone Biles did not let a slight wobble prevent her from recording another high score to strengthen the American's lead.

So I think one reason why this works is that there are probably only 500 words that you need as an Olympic commentator. And L. Michaels has already said all of them over decades of announcing the Olympics. So like he said balance beam before he said, you know, gymnastics before. So they already have like a word bank of him perplexed.

saying like 80% of everything he needs to say in this. Yeah, I did see in one article that I read about this, that they are actually having like human editors sort of review these clips, especially for some of the pronunciations of the athletes names, because that is not in the Al Michaels word bank necessarily. So they have not been able to fully automate Al Michaels, but they're, you know, maybe 80% of the way there.

I wish they would just go further with the personalization. Like, you know, you hear him say, hi, Casey, but you notice he doesn't ask about my day or really try to learn much about my life. So I think that's just room for improvement over there at NBC. Yeah, Casey, it's Al Michaels. You really didn't get a lot done today. I'm looking at your productivity stats. Maybe you should watch less Olympics tomorrow and actually do your goddamn job. All right, stop generating. All right, next one.

This is another AI Olympic story. This one is called Dear Google, Who Wants an AI Written Fan Letter? Comes from TechCrunch. And this one is about the backlash to an ad that's been running during the Olympics for Google's Gemini language model, in which a dad basically asks Gemini to help his daughter write a fan letter to her favorite athlete, the runner Sidney McLaughlin-Levrone. And maybe we should just watch this ad together. Yeah, let's pull it up.

My little girl's always been a runner. High knees. There you go. She wants to show Sydney some love, and I'm pretty good with words, but this has to be just right. So, Gemini, help my daughter write a letter telling Sydney how inspiring she is. And be sure to mention that my daughter plans on breaking her world record one day. She says, sorry, not sorry.

So, Casey, this ad got a lot of backlash from people who said, what could be less inspiring than using an AI language model to write a letter to your favorite athlete? This sucks so bad. Like, really?

Really, really bad. It's so ill conceived. Like when I think about all of the difficult conversations that people have with each other where you might want to bounce it off somebody first, the idea of saying, hey, like you're really cool and you inspire me is like the lowest thing on that list ever.

right? And the whole reason that you want to tell someone something is because you have something personal to say, not some AI version of it. But I'll tell you, Kevin, the best part about this whole ad is after this one minute of just absolute nonsense and, you know, sort of ill-conceived ideas about how to use AI, they never even show you what the AI says. So you can't even see what the output was in order to judge whether it was good or not. So it's like, not only

Not only does Google have a terrible idea about how to use AI, they won't even show you what the AI made. Yeah, it truly amazes me that this one made it out of the like Google marketing meeting brainstorm about their Olympics ads. Like one of the things that feels the worst about AI is how it can make these sort of heartfelt expressions seem totally formulaic and just like the machine extrusions of these language models.

And what better way to show that off than, you know, than a girl who is inspired by her favorite Olympian and instead of writing a heartfelt letter, plugs a few words into Gemini and off she goes. You know, what's great is that eventually there probably actually will just be something in Gmail that says, oh, by the way, this fan letter that you've gotten, it was definitely written by AI. It was actually written by our AI. And then you just archive it without reading it. Yeah.

Yeah, it does. Did you write letters to celebrities as a kid? Um, yeah, I wrote like letters to like authors of children's books and stuff and would occasionally, you know, get a little note back. And it was very exciting. Yeah, I did too. There was a book at the library that you could check out that actually had the addresses of a bunch of like celebrities and athletes. And it was all their agents. Like I was not even then under the impression that the celebrities themselves were, were, uh,

were reading these things and responding. But you'd sometimes get, you know, a letter back from Dan Marino of the Miami Dolphins or Whoopi Goldberg, you know, would send you a little like signed picture or something. So I did that. I probably could have saved myself some time using a language model, but it would have been a lot less fun. Did you really write to Whoopi Goldberg?

I did. She sent me a photo. You just saw a ghost or what happened? She was, I, I just blanketed. I, every celebrity in this book who I had heard of, I sent a letter. I mean, this was, this was what passed for entertainment in my small town. What did you say to Whoopi Goldberg? I just, I, I think I said, you know, I, I liked Sister Act and, um, Great movie. Could I please have a signed photo? And she did. Wow. Or her agent did or her, her,

mail processing service did. Shout out to Whoopi Goldberg. That's a really heartwarming story, actually. All right, stop generating. Okay, next story. Canada sends two Olympic staff members home for spying on New Zealand's soccer practice with drone. This one comes from The Athletic.

A drone was flown over a training session for New Zealand's Olympic women's soccer team in Saint-Étienne, France. The New Zealand Olympic Committee said that the New Zealand women's soccer team reported the drone to police and that they detained the operator, who was a staff member for the Canadian team. So, Casey, the soccer teams are spying on each other using drones. What do we make of this? Kevin, in the words of the Canadian national anthem, Oh, Canada!

What is happening here? If there's one thing that you can have been able to count on in this world, it is that Canadians are good, solid, upstanding, forthright citizens. In fact, the prime minister of their country recently came on this podcast and I think was a model citizen. So the idea that, you know, his Olympic team has people on it that are spying on soccer teams with drones. Frankly, it shocks me and I think it's outrageous. I just don't.

really get what you would understand by flying the drone over the practice. It's like, oh, they're scrimmaging. Oh, they're practicing a new formation. Like, is that really going to turn the tide of the game? What's also amazing is that the Canadian team is, like, incredible, right? Like, I believe they won the gold medal at the last Summer Olympics.

And the New Zealand team, by contrast, is like toward the bottom of the pack. So the idea that the Canadians would use these dirty tricks when they're already so far ahead of their competition. Once again, you got to ask yourself what's going on up there. Yeah. All right. Well, Kevin, let's stop droning on and stop generating. Hey. All right. Next up.

A viral Olympics selfie is sparking hopes of cross-border harmony. This one comes from CNN. Photos of North and South Korean table tennis players posing for a selfie on the medal podium have gotten people across the globe talking, with some hailing it as a show of rare cross-border harmony. This came after a recent competition between South Korea and North Korea. South Korea won bronze and North Korea won silver in the mixed doubles table tennis event.

And one of the players from South Korea celebrated by whipping out a phone and taking a selfie on the podium. The photo has apparently gone viral on Korean social media because these two countries, North Korea and South Korea, are still technically at war with each other. And this may be a moment of rare diplomacy between the two nations. What do you think? I love these. And let me say, this is what the Olympics are actually for.

Right. Like, yes, we love to see how far the human body can be pushed. But also we want to spread a spirit of, you know, peace and harmony throughout the world. So while I don't know that one selfie is going to be what leads North and South Korea to bury the hatchet, definitely feels like a step in the right direction. And, you know, maybe maybe a sign that social media, which is, I assume, where these folks wanted to post their selfie, can do a little bit of good every once in a while.

Yeah. Do you know what I learned while looking this up? Until this Olympics, Olympians were not allowed to take selfies on the medal podium. What? Yes. In the past, athletes were banned from having personal belongings, including their phones, on the field of play, which includes the podium. That's tyranny. You know, if I won a medal at the Olympics, damn right I want to take a selfie.

Absolutely. Let the people take their selfies. I'm glad they changed that rule. So they not only changed the rule, but Samsung has begun sponsoring this effort. They now supply the winners of medals at the Paris Olympics with a new Samsung Galaxy Z Flip 6 phone to take what they're calling...

victory selfies on the podium. So a good marketing opportunity for them too. All right. Well, now I don't like it anymore. And never forget that Samsung Galaxy phones exploded on planes and got banned from them. So be careful when Samsung puts that thing in your hand on the podium, Olympians. All right. Stop generating.

All right, this next one. Okay, this one is not specifically about the Olympics, but is instead about something that went big on the internet after the opening ceremony. This one's from People, the esteemed tech news publication, which had a story called, Olympics viewers react after opening ceremony features a menage a trois bit.

Casey, did you see the opening ceremony and it's menage a trois bit? I did because you're making me realize, you know, earlier you were asking me what my favorite event is. And this question made me realize that my favorite event is the opening ceremonies because the opening ceremonies of the Olympics are where gay culture is allowed to express itself the most, Kevin. And boy, did Paris not disappoint on that front. So I didn't actually see this one. What happened? What were people so mad about?

Well, and I'm going to pull it up because people has a nice description of it that I think will be helpful to read. So basically, this scene shows three people decorated in colorful outfits running up a staircase together and embracing briefly before heading into a bedroom. Wink, wink. And when they get there, they touch each other's faces a little before closing the door on the camera person.

So as one person put on X casual polyamory during the Olympics opening ceremony, I'm here for it. Wow. So this was actually like kind of a promotion of the poly lifestyle or was this something that people sort of took out of context?

No, this was like actually like the conservative criticism of France as a country is that it's always trying to like those liberal gates. Yes. Yes. Promulgating its liberty and nontraditional values. And I get like, I love it. Like the whole point of these events being held at different countries around the world is so you can get a taste of the culture and France. They really have leaned into that liberty, my friend. And that means that, yes, there will be three ways during the opening ceremony. Yeah.

Finally, representation for the polycule on the Olympics opening ceremony. Heck, there might be a four-way. Well, I know a lot of Bay Area tech employees will be excited about that one.

Yes. And I think, you know, if the Olympics ever were held, like in the Bay Area, I think the organizers of the opening ceremony will be sad that France beat us to the punch when it came to that particular night. All right. Stop dintering. That is our special Olympics edition of Hat GPT. Casey, thanks for playing. Thank you, Kevin. And of course, good luck to all of the athletes for Team USA and all the other athletes around the world. Yeah.

Good luck to all the Olympians, human and non-human.

This podcast is supported by Progressive Insurance. Whether you love true crime or comedy, celebrity interviews or news, you call the shots on what's in your podcast queue. And guess what? Now you can call them on your auto insurance too with the Name Your Price tool from Progressive. It works just the way it sounds. You tell Progressive how much you want to pay for car insurance, and they'll show you coverage options that fit your budget. Get your quote today at Progressive.com to join the over 28 million drivers who trust Progressive.

Progressive Casualty Insurance Company & Affiliates. Price and coverage match limited by state law.

Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Jen Poyant. We're fact-checked by Caitlin Love. Today's show was engineered by Alyssa Moxley. Original music by Alicia Bietube, Marion Lozano, Diane Wong, Rowan Nemisto, and Dan Powell. Our audience editor is Nelga Logli. Video production by Ryan Manning and Chris Schott. You can watch this full episode on YouTube at youtube.com slash hardfork.

Special thanks to Paula Schumann, Pui Wing Tam, Dahlia Haddad, and Jeffrey Miranda. You can email us at heartforkatnytimes.com with all your international math Olympiad problem solutions. Imagine earning a degree that prepares you with real skills for the real world. Capella University's programs teach skills relevant to your career so you can apply what you learn right away. Learn how Capella can make a difference in your life at capella.edu.